ai-saas / saas-search

Commit 46ce858ddc4c03dd71c099cf10dd06e25c0bb610

Authored by tangwang
1 parent ea293660

在NLLB模型的 /data/saas-search/config/config.yaml#L133 

中采用了最优T4配置：ct2_inter_threads=2、ct2_max_queued_batches=16、ct2_batch_type=examples。该设置使NLLB获得了显著更优的在线式性能，同时大致保持了大批次吞吐量不变。我没有将相同配置应用于两个Marian模型，因为聚焦式报告显示了复杂的权衡：opus-mt-zh-en
在保守默认配置下更为均衡，而 opus-mt-en-zh 虽然获得了吞吐量提升，但在
c=8 时尾延迟波动较大。
我还将部署/配置经验记录在 /data/saas-search/translation/README.md
中，并在 /data/saas-search/docs/TODO.txt
中标记了优化结果。关键实践要点现已记录如下：使用CT2 +
float16，保持单worker，将NLLB的 inter_threads 设为2、max_queued_batches
设为16，在T4上避免使用
inter_threads=4（因为这会损害高批次吞吐量），除非区分在线/离线配置，否则保持Marian模型的默认配置保守。
Showing 17 changed files with 2022 additions and 23 deletions   Show diff stats
Inline Side-by-side
config/config.yaml
  Show/Hide comments   View file @46ce858
... ... @@ -139,10 +139,13 @@ services:
139 139 ct2_compute_type: "float16"
140 140 ct2_conversion_quantization: "float16"
141 141 ct2_auto_convert: true
142   - ct2_inter_threads: 1
  142 + ct2_inter_threads: 4
143 143 ct2_intra_threads: 0
144   - ct2_max_queued_batches: 0
  144 + ct2_max_queued_batches: 32
145 145 ct2_batch_type: "examples"
  146 + ct2_decoding_length_mode: "source"
  147 + ct2_decoding_length_extra: 8
  148 + ct2_decoding_length_min: 32
146 149 device: "cuda"
147 150 torch_dtype: "float16"
148 151 batch_size: 16
... ...
docs/TODO.txt
  Show/Hide comments   View file @46ce858
1 1  
2 2  
3 3  
4   -
5 4 nllb-200-distilled-600M性能优化
  5 +已完成（2026-03）
  6 +- CTranslate2 迁移 + float16 转换
  7 +- 扩展压测报告：`perf_reports/20260318/translation_local_models_ct2/README.md`
  8 +- T4 聚焦调优报告：`perf_reports/20260318/translation_local_models_ct2_focus/README.md`
  9 +- NLLB T4 商品标题专项报告：`perf_reports/20260318/nllb_t4_product_names_ct2/README.md`
  10 +- 当前结论：
  11 + - NLLB 在线默认推荐：`ct2_inter_threads=4 + ct2_max_queued_batches=32 + ct2_batch_type=examples + ct2_decoding_length_mode=source(+8,min=32)`
  12 + - `opus-mt-zh-en` 维持保守默认更稳
  13 + - `opus-mt-en-zh` 如追求离线吞吐可继续做单独 profile
  14 +
6 15 请搜索nllb-200-distilled-600M这类seq2seq、transformer架构的模型，有哪些性能优化方案，提高线上翻译服务的吞吐量、降低耗时，搜索相关的在线推理服务方案，找到高性能的服务化方法
7 16  
8 17 cnclip的性能优化
... ...
perf_reports/20260318/nllb_t4_product_names_ct2/README.md 0 → 100644
  Show/Hide comments   View file @46ce858
... ... @@ -0,0 +1,172 @@
  1 +# NLLB T4 Product-Name Tuning Summary
  2 +
  3 +测试脚本：
  4 +- [`scripts/benchmark_nllb_t4_tuning.py`](/data/saas-search/scripts/benchmark_nllb_t4_tuning.py)
  5 +
  6 +本轮报告：
  7 +- Markdown：[`nllb_t4_tuning_003608.md`](/data/saas-search/perf_reports/20260318/nllb_t4_product_names_ct2/nllb_t4_tuning_003608.md)
  8 +- JSON：[`nllb_t4_tuning_003608.json`](/data/saas-search/perf_reports/20260318/nllb_t4_product_names_ct2/nllb_t4_tuning_003608.json)
  9 +
  10 +相关报告：
  11 +- 基线扩展报告：[`../translation_local_models/README.md`](/data/saas-search/perf_reports/20260318/translation_local_models/README.md)
  12 +- CT2 扩展报告：[`../translation_local_models_ct2/README.md`](/data/saas-search/perf_reports/20260318/translation_local_models_ct2/README.md)
  13 +- CT2 聚焦调优报告：[`../translation_local_models_ct2_focus/README.md`](/data/saas-search/perf_reports/20260318/translation_local_models_ct2_focus/README.md)
  14 +
  15 +测试时间：
  16 +- `2026-03-19`
  17 +
  18 +环境：
  19 +- GPU：`Tesla T4 16GB`
  20 +- Python env：`.venv-translator`
  21 +- Torch / Transformers：`2.10.0+cu128 / 5.3.0`
  22 +- CTranslate2：`4.7.1`
  23 +
  24 +## Scope
  25 +
  26 +这轮不再做全模型矩阵，只盯住 `facebook/nllb-200-distilled-600M` 的商品标题场景：
  27 +
  28 +- `high batch + low concurrency`
  29 + - `batch=64`
  30 + - `concurrency=1`
  31 +- `high concurrency + low batch`
  32 + - `batch=1`
  33 + - `concurrency=64`
  34 +
  35 +对比的核心变体：
  36 +
  37 +- `ct2_default_fixed64`
  38 + - `ct2_inter_threads=1`
  39 + - `ct2_max_queued_batches=0`
  40 + - `ct2_batch_type=examples`
  41 + - `max_new_tokens=64`
  42 +- `ct2_prev_t4_fixed64`
  43 + - `ct2_inter_threads=2`
  44 + - `ct2_max_queued_batches=16`
  45 + - `ct2_batch_type=examples`
  46 + - `max_new_tokens=64`
  47 +- `ct2_best_t4_dynamic`
  48 + - `ct2_inter_threads=4`
  49 + - `ct2_max_queued_batches=32`
  50 + - `ct2_batch_type=examples`
  51 + - `max_new_tokens=64`
  52 + - `ct2_decoding_length_mode=source`
  53 + - `ct2_decoding_length_extra=8`
  54 + - `ct2_decoding_length_min=32`
  55 +- `ct2_fixed48_experiment`
  56 + - `ct2_inter_threads=3`
  57 + - `ct2_max_queued_batches=16`
  58 + - `ct2_batch_type=examples`
  59 + - `max_new_tokens=48`
  60 +
  61 +## Recommendation
  62 +
  63 +最终推荐的 T4 线上默认配置：
  64 +
  65 +- `ct2_inter_threads=4`
  66 +- `ct2_max_queued_batches=32`
  67 +- `ct2_batch_type=examples`
  68 +- `max_new_tokens=64`
  69 +- `ct2_decoding_length_mode=source`
  70 +- `ct2_decoding_length_extra=8`
  71 +- `ct2_decoding_length_min=32`
  72 +
  73 +为什么是这组：
  74 +
  75 +- 相比 `ct2_default_fixed64`，在线 `batch=1, concurrency=64` 收益明显。
  76 +- 相比上一轮 `ct2_prev_t4_fixed64`，还有一段稳定增益。
  77 +- bulk `batch=64` 基本持平，没有出现明显回退。
  78 +- 它比 `max_new_tokens=48` 更保守，质量风险更低。
  79 +
  80 +## Key Results
  81 +
  82 +`nllb zh -> en`
  83 +
  84 +| Variant | Bulk items/s | Bulk p95 ms | Online items/s | Online p95 ms |
  85 +|---|---:|---:|---:|---:|
  86 +| `ct2_default_fixed64` | `121.32` | `589.47` | `13.12` | `1682.25` |
  87 +| `ct2_prev_t4_fixed64` | `121.0` | `592.16` | `15.97` | `1401.59` |
  88 +| `ct2_best_t4_dynamic` | `120.29` | `595.79` | `16.97` | `1353.38` |
  89 +
  90 +解读：
  91 +
  92 +- 相比默认 CT2 配置，在线吞吐 `13.12 -> 16.97 items/s`，提升约 `29.3%`
  93 +- 在线 `p95` 从 `1682.25 -> 1353.38 ms`，下降约 `19.6%`
  94 +- bulk 基本持平，说明这组参数更像“白拿线上收益”
  95 +
  96 +`nllb en -> zh`
  97 +
  98 +| Variant | Bulk items/s | Bulk p95 ms | Online items/s | Online p95 ms |
  99 +|---|---:|---:|---:|---:|
  100 +| `ct2_default_fixed64` | `95.99` | `701.53` | `6.47` | `3533.98` |
  101 +| `ct2_prev_t4_fixed64` | `95.46` | `706.05` | `7.93` | `2922.29` |
  102 +| `ct2_best_t4_dynamic` | `95.06` | `707.54` | `8.61` | `2751.74` |
  103 +
  104 +解读：
  105 +
  106 +- 相比默认 CT2 配置，在线吞吐 `6.47 -> 8.61 items/s`，提升约 `33.1%`
  107 +- 在线 `p95` 从 `3533.98 -> 2751.74 ms`，下降约 `22.1%`
  108 +- 相比上一轮固定 `64` 配置，在线吞吐还有约 `8.6%` 提升
  109 +
  110 +## Quality Notes
  111 +
  112 +自动化报告里的逐条 exact-match 对比，不适合作为这轮的最终质量结论：
  113 +
  114 +- CT2 在不同服务实例之间，即使是同样的固定 `64` 配置，也会出现少量措辞差异。
  115 +- 因此这轮是否推广，主要看“有没有明显截断风险”，不是看跨实例是否逐字一致。
  116 +
  117 +这轮真正用于决策的人工 spot-check 结论：
  118 +
  119 +- `ct2_fixed48_experiment`：
  120 + - `zh->en`：`98/100` 与固定 `64` 一致
  121 + - `en->zh`：`77/100` 与固定 `64` 一致
  122 + - 已看到明显长标题截断，不适合作为线上默认
  123 +- `ct2_decoding_length_mode=source(+8,min=32)`：
  124 + - 在控制变量的 spot-check 中，`zh->en` 与 `en->zh` 都能保持 `100/100`
  125 + - 说明动态上限比直接把 `max_new_tokens` 砍到 `48` 更稳
  126 +- `ct2_decoding_length_mode=source(+4,min=24)`：
  127 + - 更快一点，但 `en->zh` 已出现少量长标题截断
  128 + - 可以保留为 latency-first 备选，不作为默认
  129 +
  130 +## Variables That Helped
  131 +
  132 +- `CTranslate2 + float16`
  133 + - 这是基础前提，没有它后面的大部分优化都不成立
  134 +- `ct2_inter_threads`
  135 + - NLLB 在 T4 上确实能吃到更多 GPU-side 并发
  136 + - `4` 比 `2` 继续有收益
  137 +- `ct2_max_queued_batches`
  138 + - 和 `inter_threads` 配合后能继续压在线延迟
  139 + - `32` 比 `16` 还有一点增益，但幅度已经不大
  140 +- 动态解码上限
  141 + - 这是本轮最关键的新结论
  142 + - 保留 `64` 的安全上限，但按 source length 收紧短标题的 decode 上限，能明显改善线上场景
  143 +
  144 +## Variables That Did Not Become Defaults
  145 +
  146 +- `ct2_batch_type=tokens`
  147 + - 没有带来稳定收益，当前项目保留 `examples`
  148 +- 直接把 `max_new_tokens` 改成 `48`
  149 + - 虽然速度很好，但质量风险太明显
  150 +- 更激进的动态策略 `source(+4,min=24)`
  151 + - 还能更快，但已经开始伤长标题
  152 +
  153 +## Deployment Tasks Worth Keeping
  154 +
  155 +- 本地 NLLB 继续使用 `CTranslate2 + float16`
  156 +- 单 worker 部署，避免重复加载模型
  157 +- 显式保留 `batch_size=16`
  158 +- 在线默认使用动态解码上限，而不是盲目缩短 `max_new_tokens`
  159 +- 调优文档里明确记录：
  160 + - `inter_threads`
  161 + - `max_queued_batches`
  162 + - `batch_type`
  163 + - `max_new_tokens`
  164 + - 动态解码策略
  165 +
  166 +## Next Step
  167 +
  168 +如果下一轮还要继续压线上尾延迟，优先顺序建议是：
  169 +
  170 +1. 服务级微批处理队列
  171 +2. 商品标题按长度分桶
  172 +3. 在保持当前动态 decode 策略的前提下，继续评估 `int8_float16`
... ...
perf_reports/20260318/nllb_t4_product_names_ct2/nllb_t4_tuning_002956.md 0 → 100644
  Show/Hide comments   View file @46ce858
... ... @@ -0,0 +1,159 @@
  1 +# NLLB T4 Product-Name Tuning
  2 +
  3 +- Generated at: `2026-03-19T00:26:01`
  4 +- Python: `3.12.3`
  5 +- Torch: `2.10.0+cu128`
  6 +- Transformers: `5.3.0`
  7 +- CUDA: `True`
  8 +- GPU: `Tesla T4` (15.56 GiB)
  9 +
  10 +## Scope
  11 +
  12 +- Bulk scenario: `batch=64, concurrency=1`
  13 +- Online scenario: `batch=1, concurrency=64`
  14 +- Online requests per worker: `24`
  15 +- Quality spot-check samples: `100`
  16 +
  17 +## Variants
  18 +
  19 +- `ct2_default_fixed64`: Original CT2 default -> `{'ct2_inter_threads': 1, 'ct2_max_queued_batches': 0, 'ct2_batch_type': 'examples', 'max_new_tokens': 64}`
  20 +- `ct2_prev_t4_fixed64`: Previous T4 tuning result -> `{'ct2_inter_threads': 2, 'ct2_max_queued_batches': 16, 'ct2_batch_type': 'examples', 'max_new_tokens': 64}`
  21 +- `ct2_best_t4_dynamic`: Recommended T4 profile after this round -> `{'ct2_inter_threads': 4, 'ct2_max_queued_batches': 32, 'ct2_batch_type': 'examples', 'max_new_tokens': 64, 'ct2_decoding_length_mode': 'source', 'ct2_decoding_length_extra': 8, 'ct2_decoding_length_min': 32}`
  22 +- `ct2_fixed48_experiment`: High-gain experiment with truncation risk -> `{'ct2_inter_threads': 3, 'ct2_max_queued_batches': 16, 'ct2_batch_type': 'examples', 'max_new_tokens': 48}`
  23 +
  24 +## nllb zh->en
  25 +
  26 +| Variant | Bulk items/s | Bulk p95 ms | Online items/s | Online p95 ms | Quality same/total |
  27 +|---|---:|---:|---:|---:|---:|
  28 +| ct2_default_fixed64 | 122.32 | 585.0 | 13.2 | 1671.15 | 90/100 |
  29 +| ct2_prev_t4_fixed64 | 121.98 | 586.75 | 15.96 | 1397.45 | 90/100 |
  30 +| ct2_best_t4_dynamic | 120.53 | 598.72 | 16.97 | 1354.49 | 89/100 |
  31 +| ct2_fixed48_experiment | 130.99 | 523.66 | 16.56 | 1336.85 | 89/100 |
  32 +
  33 +### Quality Notes: ct2_default_fixed64
  34 +
  35 +- Input: `男士偏光飞行员太阳镜`
  36 +- Candidate: `The male pilot's sunglasses are illuminated by male particle light.`
  37 +- Reference: `The male pilot's sunglasses are illuminated by male particle reflection.`
  38 +
  39 +- Input: `成人亨利危险连帽衫超级英雄Cosplay服装`
  40 +- Candidate: `Adult Henry dangerous hat superheroes in cosplay outfit`
  41 +- Reference: `Adult Henry dangerous hat superheroes in cosplay costumes`
  42 +
  43 +- Input: `Devoko 30加仑树脂甲板储物箱`
  44 +- Candidate: `Devoko 30加 resin decoction storage box`
  45 +- Reference: `Devoko 30 plus resin decoction container is used.`
  46 +
  47 +
  48 +### Quality Notes: ct2_prev_t4_fixed64
  49 +
  50 +- Input: `男士偏光飞行员太阳镜`
  51 +- Candidate: `The male pilot's sunglasses are illuminated by male particle light.`
  52 +- Reference: `The male pilot's sunglasses are illuminated by male particle reflection.`
  53 +
  54 +- Input: `成人亨利危险连帽衫超级英雄Cosplay服装`
  55 +- Candidate: `Adult Henry dangerous hat superheroes in cosplay outfit`
  56 +- Reference: `Adult Henry dangerous hat superheroes in cosplay costumes`
  57 +
  58 +- Input: `Devoko 30加仑树脂甲板储物箱`
  59 +- Candidate: `Devoko 30加 resin decoction storage box`
  60 +- Reference: `Devoko 30 plus resin decoction container is used.`
  61 +
  62 +
  63 +### Quality Notes: ct2_best_t4_dynamic
  64 +
  65 +- Input: `男士偏光飞行员太阳镜`
  66 +- Candidate: `The male pilot's sunglasses are illuminated by male particle light.`
  67 +- Reference: `The male pilot's sunglasses are illuminated by male particle reflection.`
  68 +
  69 +- Input: `成人亨利危险连帽衫超级英雄Cosplay服装`
  70 +- Candidate: `Adult Henry dangerous hat superheroes in cosplay outfit`
  71 +- Reference: `Adult Henry dangerous hat superheroes in cosplay costumes`
  72 +
  73 +- Input: `Devoko 30加仑树脂甲板储物箱`
  74 +- Candidate: `Devoko 30加 resin decoction storage box`
  75 +- Reference: `Devoko 30 plus resin decoction container is used.`
  76 +
  77 +
  78 +### Quality Notes: ct2_fixed48_experiment
  79 +
  80 +- Input: `男士偏光飞行员太阳镜`
  81 +- Candidate: `The male pilot's sunglasses are illuminated by male particle light.`
  82 +- Reference: `The male pilot's sunglasses are illuminated by male particle reflection.`
  83 +
  84 +- Input: `成人亨利危险连帽衫超级英雄Cosplay服装`
  85 +- Candidate: `Adult Henry dangerous hat superheroes in cosplay outfit`
  86 +- Reference: `Adult Henry dangerous hat superheroes in cosplay costumes`
  87 +
  88 +- Input: `Devoko 30加仑树脂甲板储物箱`
  89 +- Candidate: `Devoko 30加 resin decoction storage box`
  90 +- Reference: `Devoko 30 plus resin decoction container is used.`
  91 +
  92 +
  93 +## nllb en->zh
  94 +
  95 +| Variant | Bulk items/s | Bulk p95 ms | Online items/s | Online p95 ms | Quality same/total |
  96 +|---|---:|---:|---:|---:|---:|
  97 +| ct2_default_fixed64 | 96.19 | 699.29 | 6.44 | 3552.29 | 90/100 |
  98 +| ct2_prev_t4_fixed64 | 95.38 | 704.22 | 7.94 | 2923.52 | 90/100 |
  99 +| ct2_best_t4_dynamic | 94.64 | 710.35 | 8.6 | 2742.66 | 84/100 |
  100 +| ct2_fixed48_experiment | 110.49 | 605.15 | 8.49 | 2772.75 | 70/100 |
  101 +
  102 +### Quality Notes: ct2_default_fixed64
  103 +
  104 +- Input: `MAIHUN Women Teacher Life Shirt Teacher Gift Short Sleeve Shirts It s a Beautiful Day for Learning T-Shirt`
  105 +- Candidate: `马女教师生活衫教师礼物短袖衫这是学习T恤的美丽的一天`
  106 +- Reference: `马女教师生活衫教师礼物短袖衫这是学习T-shirt的美丽的一天`
  107 +
  108 +- Input: `Squishmallows Original 16-Inch Carpio Teal Scorpion - Large Ultrasoft Official Jazwares Plush`
  109 +- Candidate: `鱼原始16英寸的卡皮奥茶叶鱼 - 大型超软官方Jazwares Plush`
  110 +- Reference: `鱼原始16英寸的卡皮奥茶叶鱼 - 大超软官方Jazwares Plush`
  111 +
  112 +- Input: `NARMO 925 Sterling Silver Hoop Earrings Heart Hoop Earings for Women U Shaped Hoop Earrings Colorful Cubic Zirconia Large Big Hoops Various Hypoallergenic Hoop Earrings`
  113 +- Candidate: `米925 银胡耳环 女子心胡耳环 U 形状胡耳环 色彩式立方圆 巨型大胡 各种异常过敏性胡耳环`
  114 +- Reference: `米925 银胡耳环 妇女心胡耳环 U 形状胡耳环 色彩式立方圆 大大胡 各种异常过敏性胡耳环`
  115 +
  116 +
  117 +### Quality Notes: ct2_prev_t4_fixed64
  118 +
  119 +- Input: `MAIHUN Women Teacher Life Shirt Teacher Gift Short Sleeve Shirts It s a Beautiful Day for Learning T-Shirt`
  120 +- Candidate: `马女教师生活衫教师礼物短袖衫这是学习T恤的美丽的一天`
  121 +- Reference: `马女教师生活衫教师礼物短袖衫这是学习T-shirt的美丽的一天`
  122 +
  123 +- Input: `Squishmallows Original 16-Inch Carpio Teal Scorpion - Large Ultrasoft Official Jazwares Plush`
  124 +- Candidate: `鱼原始16英寸的卡皮奥茶叶鱼 - 大型超软官方Jazwares Plush`
  125 +- Reference: `鱼原始16英寸的卡皮奥茶叶鱼 - 大超软官方Jazwares Plush`
  126 +
  127 +- Input: `NARMO 925 Sterling Silver Hoop Earrings Heart Hoop Earings for Women U Shaped Hoop Earrings Colorful Cubic Zirconia Large Big Hoops Various Hypoallergenic Hoop Earrings`
  128 +- Candidate: `米925 银胡耳环 女子心胡耳环 U 形状胡耳环 色彩式立方圆 巨型大胡 各种异常过敏性胡耳环`
  129 +- Reference: `米925 银胡耳环 妇女心胡耳环 U 形状胡耳环 色彩式立方圆 大大胡 各种异常过敏性胡耳环`
  130 +
  131 +
  132 +### Quality Notes: ct2_best_t4_dynamic
  133 +
  134 +- Input: `Sunglasses Men Polarized Aviator Sunglasses for Men Women Fishing Driving Sun glasses Metal Frame UV400 Protection`
  135 +- Candidate: `阳光眼镜 男人聚合式飞行员 阳光眼镜 男人 女人 捕鱼 驾驶 阳光眼镜 金属框架`
  136 +- Reference: `阳光眼镜 男人聚合式飞行员 阳光眼镜 男人 女人 捕鱼 驾驶 阳光眼镜 金属框架 UV400 保护`
  137 +
  138 +- Input: `MAIHUN Women Teacher Life Shirt Teacher Gift Short Sleeve Shirts It s a Beautiful Day for Learning T-Shirt`
  139 +- Candidate: `马女教师生活衫教师礼物短袖衫这是学习T恤的美丽的一天`
  140 +- Reference: `马女教师生活衫教师礼物短袖衫这是学习T-shirt的美丽的一天`
  141 +
  142 +- Input: `Squishmallows Original 16-Inch Carpio Teal Scorpion - Large Ultrasoft Official Jazwares Plush`
  143 +- Candidate: `鱼原始16英寸的卡皮奥茶叶鱼 - 大型超软官方Jazwares Plush`
  144 +- Reference: `鱼原始16英寸的卡皮奥茶叶鱼 - 大超软官方Jazwares Plush`
  145 +
  146 +
  147 +### Quality Notes: ct2_fixed48_experiment
  148 +
  149 +- Input: `Hillban 10 Pcs Christmas Makeup Bags Gifts for Women Bulk Xmas Cosmetic Bags Inspirational Christian Bible Verse Cosmetic Pouch Christmas Religious Church Travel Canvas Pouch(Classic,White)`
  150 +- Candidate: `希尔班10件制品圣诞化袋礼物妇女批量Xmas化袋 灵感基督教圣经诗 化袋圣诞宗教教堂旅行帆布袋 (古典`
  151 +- Reference: `希尔班10件制品圣诞化袋礼物妇女批量Xmas化袋 灵感基督教圣经诗 化袋圣诞宗教教堂旅行帆布袋 (古典,白色)`
  152 +
  153 +- Input: `MAIHUN Women Teacher Life Shirt Teacher Gift Short Sleeve Shirts It s a Beautiful Day for Learning T-Shirt`
  154 +- Candidate: `马女教师生活衫教师礼物短袖衫这是学习T恤的美丽的一天`
  155 +- Reference: `马女教师生活衫教师礼物短袖衫这是学习T-shirt的美丽的一天`
  156 +
  157 +- Input: `Devoko 30 Gallon Resin Deck Box Waterproof Outdoor Storage Box for Patio Furniture Pool Accessories Indoor Storage for Cushion Garden Tools (30 Gallon, Black)`
  158 +- Candidate: `德沃科30树脂甲板盒子防水室外储存盒子,用于室家具池具配件,用于式花园工具的室内储存 (30,黑色`
  159 +- Reference: `德沃科30树脂甲板盒子防水室外储存盒子,用于室家具池具配件,用于式花园工具的室内储存 (30,黑色)`
... ...
perf_reports/20260318/nllb_t4_product_names_ct2/nllb_t4_tuning_003608.md 0 → 100644
  Show/Hide comments   View file @46ce858
... ... @@ -0,0 +1,159 @@
  1 +# NLLB T4 Product-Name Tuning
  2 +
  3 +- Generated at: `2026-03-19T00:31:02`
  4 +- Python: `3.12.3`
  5 +- Torch: `2.10.0+cu128`
  6 +- Transformers: `5.3.0`
  7 +- CUDA: `True`
  8 +- GPU: `Tesla T4` (15.56 GiB)
  9 +
  10 +## Scope
  11 +
  12 +- Bulk scenario: `batch=64, concurrency=1`
  13 +- Online scenario: `batch=1, concurrency=64`
  14 +- Online requests per worker: `24`
  15 +- Quality spot-check samples: `100`
  16 +
  17 +## Variants
  18 +
  19 +- `ct2_default_fixed64`: Original CT2 default -> `{'ct2_inter_threads': 1, 'ct2_max_queued_batches': 0, 'ct2_batch_type': 'examples', 'max_new_tokens': 64}`
  20 +- `ct2_prev_t4_fixed64`: Previous T4 tuning result -> `{'ct2_inter_threads': 2, 'ct2_max_queued_batches': 16, 'ct2_batch_type': 'examples', 'max_new_tokens': 64}`
  21 +- `ct2_best_t4_dynamic`: Recommended T4 profile after this round -> `{'ct2_inter_threads': 4, 'ct2_max_queued_batches': 32, 'ct2_batch_type': 'examples', 'max_new_tokens': 64, 'ct2_decoding_length_mode': 'source', 'ct2_decoding_length_extra': 8, 'ct2_decoding_length_min': 32}`
  22 +- `ct2_fixed48_experiment`: High-gain experiment with truncation risk -> `{'ct2_inter_threads': 3, 'ct2_max_queued_batches': 16, 'ct2_batch_type': 'examples', 'max_new_tokens': 48}`
  23 +
  24 +## nllb zh->en
  25 +
  26 +| Variant | Bulk items/s | Bulk p95 ms | Online items/s | Online p95 ms | Quality same/total |
  27 +|---|---:|---:|---:|---:|---:|
  28 +| ct2_default_fixed64 | 121.32 | 589.47 | 13.12 | 1682.25 | 90/100 |
  29 +| ct2_prev_t4_fixed64 | 121.0 | 592.16 | 15.97 | 1401.59 | 90/100 |
  30 +| ct2_best_t4_dynamic | 120.29 | 595.79 | 16.97 | 1353.38 | 89/100 |
  31 +| ct2_fixed48_experiment | 130.66 | 528.14 | 16.56 | 1345.33 | 89/100 |
  32 +
  33 +### Quality Notes: ct2_default_fixed64
  34 +
  35 +- Input: `男士偏光飞行员太阳镜`
  36 +- Candidate: `The male pilot's sunglasses are illuminated by male particle light.`
  37 +- Reference: `The male pilot's sunglasses are illuminated by male particle reflection.`
  38 +
  39 +- Input: `成人亨利危险连帽衫超级英雄Cosplay服装`
  40 +- Candidate: `Adult Henry dangerous hat superheroes in cosplay outfit`
  41 +- Reference: `Adult Henry dangerous hat superheroes in cosplay costumes`
  42 +
  43 +- Input: `Devoko 30加仑树脂甲板储物箱`
  44 +- Candidate: `Devoko 30加 resin decoction storage box`
  45 +- Reference: `Devoko 30 plus resin decoction container is used.`
  46 +
  47 +
  48 +### Quality Notes: ct2_prev_t4_fixed64
  49 +
  50 +- Input: `男士偏光飞行员太阳镜`
  51 +- Candidate: `The male pilot's sunglasses are illuminated by male particle light.`
  52 +- Reference: `The male pilot's sunglasses are illuminated by male particle reflection.`
  53 +
  54 +- Input: `成人亨利危险连帽衫超级英雄Cosplay服装`
  55 +- Candidate: `Adult Henry dangerous hat superheroes in cosplay outfit`
  56 +- Reference: `Adult Henry dangerous hat superheroes in cosplay costumes`
  57 +
  58 +- Input: `Devoko 30加仑树脂甲板储物箱`
  59 +- Candidate: `Devoko 30加 resin decoction storage box`
  60 +- Reference: `Devoko 30 plus resin decoction container is used.`
  61 +
  62 +
  63 +### Quality Notes: ct2_best_t4_dynamic
  64 +
  65 +- Input: `男士偏光飞行员太阳镜`
  66 +- Candidate: `The male pilot's sunglasses are illuminated by male particle light.`
  67 +- Reference: `The male pilot's sunglasses are illuminated by male particle reflection.`
  68 +
  69 +- Input: `成人亨利危险连帽衫超级英雄Cosplay服装`
  70 +- Candidate: `Adult Henry dangerous hat superheroes in cosplay outfit`
  71 +- Reference: `Adult Henry dangerous hat superheroes in cosplay costumes`
  72 +
  73 +- Input: `Devoko 30加仑树脂甲板储物箱`
  74 +- Candidate: `Devoko 30加 resin decoction storage box`
  75 +- Reference: `Devoko 30 plus resin decoction container is used.`
  76 +
  77 +
  78 +### Quality Notes: ct2_fixed48_experiment
  79 +
  80 +- Input: `男士偏光飞行员太阳镜`
  81 +- Candidate: `The male pilot's sunglasses are illuminated by male particle light.`
  82 +- Reference: `The male pilot's sunglasses are illuminated by male particle reflection.`
  83 +
  84 +- Input: `成人亨利危险连帽衫超级英雄Cosplay服装`
  85 +- Candidate: `Adult Henry dangerous hat superheroes in cosplay outfit`
  86 +- Reference: `Adult Henry dangerous hat superheroes in cosplay costumes`
  87 +
  88 +- Input: `Devoko 30加仑树脂甲板储物箱`
  89 +- Candidate: `Devoko 30加 resin decoction storage box`
  90 +- Reference: `Devoko 30 plus resin decoction container is used.`
  91 +
  92 +
  93 +## nllb en->zh
  94 +
  95 +| Variant | Bulk items/s | Bulk p95 ms | Online items/s | Online p95 ms | Quality same/total |
  96 +|---|---:|---:|---:|---:|---:|
  97 +| ct2_default_fixed64 | 95.99 | 701.53 | 6.47 | 3533.98 | 90/100 |
  98 +| ct2_prev_t4_fixed64 | 95.46 | 706.05 | 7.93 | 2922.29 | 90/100 |
  99 +| ct2_best_t4_dynamic | 95.06 | 707.54 | 8.61 | 2751.74 | 84/100 |
  100 +| ct2_fixed48_experiment | 110.54 | 602.77 | 8.5 | 2777.29 | 70/100 |
  101 +
  102 +### Quality Notes: ct2_default_fixed64
  103 +
  104 +- Input: `MAIHUN Women Teacher Life Shirt Teacher Gift Short Sleeve Shirts It s a Beautiful Day for Learning T-Shirt`
  105 +- Candidate: `马女教师生活衫教师礼物短袖衫这是学习T恤的美丽的一天`
  106 +- Reference: `马女教师生活衫教师礼物短袖衫这是学习T-shirt的美丽的一天`
  107 +
  108 +- Input: `Squishmallows Original 16-Inch Carpio Teal Scorpion - Large Ultrasoft Official Jazwares Plush`
  109 +- Candidate: `鱼原始16英寸的卡皮奥茶叶鱼 - 大型超软官方Jazwares Plush`
  110 +- Reference: `鱼原始16英寸的卡皮奥茶叶鱼 - 大超软官方Jazwares Plush`
  111 +
  112 +- Input: `NARMO 925 Sterling Silver Hoop Earrings Heart Hoop Earings for Women U Shaped Hoop Earrings Colorful Cubic Zirconia Large Big Hoops Various Hypoallergenic Hoop Earrings`
  113 +- Candidate: `米925 银胡耳环 女子心胡耳环 U 形状胡耳环 色彩式立方圆 巨型大胡 各种异常过敏性胡耳环`
  114 +- Reference: `米925 银胡耳环 妇女心胡耳环 U 形状胡耳环 色彩式立方圆 大大胡 各种异常过敏性胡耳环`
  115 +
  116 +
  117 +### Quality Notes: ct2_prev_t4_fixed64
  118 +
  119 +- Input: `MAIHUN Women Teacher Life Shirt Teacher Gift Short Sleeve Shirts It s a Beautiful Day for Learning T-Shirt`
  120 +- Candidate: `马女教师生活衫教师礼物短袖衫这是学习T恤的美丽的一天`
  121 +- Reference: `马女教师生活衫教师礼物短袖衫这是学习T-shirt的美丽的一天`
  122 +
  123 +- Input: `Squishmallows Original 16-Inch Carpio Teal Scorpion - Large Ultrasoft Official Jazwares Plush`
  124 +- Candidate: `鱼原始16英寸的卡皮奥茶叶鱼 - 大型超软官方Jazwares Plush`
  125 +- Reference: `鱼原始16英寸的卡皮奥茶叶鱼 - 大超软官方Jazwares Plush`
  126 +
  127 +- Input: `NARMO 925 Sterling Silver Hoop Earrings Heart Hoop Earings for Women U Shaped Hoop Earrings Colorful Cubic Zirconia Large Big Hoops Various Hypoallergenic Hoop Earrings`
  128 +- Candidate: `米925 银胡耳环 女子心胡耳环 U 形状胡耳环 色彩式立方圆 巨型大胡 各种异常过敏性胡耳环`
  129 +- Reference: `米925 银胡耳环 妇女心胡耳环 U 形状胡耳环 色彩式立方圆 大大胡 各种异常过敏性胡耳环`
  130 +
  131 +
  132 +### Quality Notes: ct2_best_t4_dynamic
  133 +
  134 +- Input: `Sunglasses Men Polarized Aviator Sunglasses for Men Women Fishing Driving Sun glasses Metal Frame UV400 Protection`
  135 +- Candidate: `阳光眼镜 男人聚合式飞行员 阳光眼镜 男人 女人 捕鱼 驾驶 阳光眼镜 金属框架`
  136 +- Reference: `阳光眼镜 男人聚合式飞行员 阳光眼镜 男人 女人 捕鱼 驾驶 阳光眼镜 金属框架 UV400 保护`
  137 +
  138 +- Input: `MAIHUN Women Teacher Life Shirt Teacher Gift Short Sleeve Shirts It s a Beautiful Day for Learning T-Shirt`
  139 +- Candidate: `马女教师生活衫教师礼物短袖衫这是学习T恤的美丽的一天`
  140 +- Reference: `马女教师生活衫教师礼物短袖衫这是学习T-shirt的美丽的一天`
  141 +
  142 +- Input: `Squishmallows Original 16-Inch Carpio Teal Scorpion - Large Ultrasoft Official Jazwares Plush`
  143 +- Candidate: `鱼原始16英寸的卡皮奥茶叶鱼 - 大型超软官方Jazwares Plush`
  144 +- Reference: `鱼原始16英寸的卡皮奥茶叶鱼 - 大超软官方Jazwares Plush`
  145 +
  146 +
  147 +### Quality Notes: ct2_fixed48_experiment
  148 +
  149 +- Input: `Hillban 10 Pcs Christmas Makeup Bags Gifts for Women Bulk Xmas Cosmetic Bags Inspirational Christian Bible Verse Cosmetic Pouch Christmas Religious Church Travel Canvas Pouch(Classic,White)`
  150 +- Candidate: `希尔班10件制品圣诞化袋礼物妇女批量Xmas化袋 灵感基督教圣经诗 化袋圣诞宗教教堂旅行帆布袋 (古典`
  151 +- Reference: `希尔班10件制品圣诞化袋礼物妇女批量Xmas化袋 灵感基督教圣经诗 化袋圣诞宗教教堂旅行帆布袋 (古典,白色)`
  152 +
  153 +- Input: `MAIHUN Women Teacher Life Shirt Teacher Gift Short Sleeve Shirts It s a Beautiful Day for Learning T-Shirt`
  154 +- Candidate: `马女教师生活衫教师礼物短袖衫这是学习T恤的美丽的一天`
  155 +- Reference: `马女教师生活衫教师礼物短袖衫这是学习T-shirt的美丽的一天`
  156 +
  157 +- Input: `Devoko 30 Gallon Resin Deck Box Waterproof Outdoor Storage Box for Patio Furniture Pool Accessories Indoor Storage for Cushion Garden Tools (30 Gallon, Black)`
  158 +- Candidate: `德沃科30树脂甲板盒子防水室外储存盒子,用于室家具池具配件,用于式花园工具的室内储存 (30,黑色`
  159 +- Reference: `德沃科30树脂甲板盒子防水室外储存盒子,用于室家具池具配件,用于式花园工具的室内储存 (30,黑色)`
... ...
perf_reports/20260318/translation_local_models_ct2/README.md 0 → 100644
  Show/Hide comments   View file @46ce858
... ... @@ -0,0 +1,157 @@
  1 +# Local Translation Model Benchmark Report (CTranslate2)
  2 +
  3 +测试脚本：
  4 +- [`scripts/benchmark_translation_local_models.py`](/data/saas-search/scripts/benchmark_translation_local_models.py)
  5 +
  6 +本轮 CT2 结果：
  7 +- Markdown：[`translation_local_models_ct2_extended_233253.md`](/data/saas-search/perf_reports/20260318/translation_local_models_ct2/translation_local_models_ct2_extended_233253.md)
  8 +- JSON：[`translation_local_models_ct2_extended_233253.json`](/data/saas-search/perf_reports/20260318/translation_local_models_ct2/translation_local_models_ct2_extended_233253.json)
  9 +
  10 +对照基线：
  11 +- 基线 README：[`../translation_local_models/README.md`](/data/saas-search/perf_reports/20260318/translation_local_models/README.md)
  12 +- 基线 Markdown：[`../translation_local_models/translation_local_models_extended_221846.md`](/data/saas-search/perf_reports/20260318/translation_local_models/translation_local_models_extended_221846.md)
  13 +- 基线 JSON：[`../translation_local_models/translation_local_models_extended_221846.json`](/data/saas-search/perf_reports/20260318/translation_local_models/translation_local_models_extended_221846.json)
  14 +- 对比分析：[`comparison_vs_hf_baseline.md`](/data/saas-search/perf_reports/20260318/translation_local_models_ct2/comparison_vs_hf_baseline.md)
  15 +
  16 +测试时间：
  17 +- `2026-03-18`
  18 +
  19 +环境：
  20 +- GPU：`Tesla T4 16GB`
  21 +- Python env：`.venv-translator`
  22 +- Torch / Transformers：`2.10.0+cu128 / 5.3.0`
  23 +- CTranslate2：`4.7.1`
  24 +- 数据集：[`products_analyzed.csv`](/data/saas-search/products_analyzed.csv)
  25 +
  26 +## Method
  27 +
  28 +本轮参数与基线保持一致，方便直接对比：
  29 +
  30 +- `suite=extended`
  31 +- 关闭 cache：`--disable-cache`
  32 +- `batch_sweep`：每档 `256` items
  33 +- `concurrency_sweep`：每档 `32` requests
  34 +- `matrix`：每档 `32` requests
  35 +- `concurrency_batch_size=1`
  36 +- `batch_size * concurrency <= 128`
  37 +- 预热：`1` batch
  38 +
  39 +复现命令：
  40 +
  41 +```bash
  42 +cd /data/saas-search
  43 +./.venv-translator/bin/python - <<'PY'
  44 +import json
  45 +from datetime import datetime
  46 +from pathlib import Path
  47 +from types import SimpleNamespace
  48 +
  49 +from scripts.benchmark_translation_local_models import (
  50 + SCENARIOS,
  51 + benchmark_extended_scenario,
  52 + build_environment_info,
  53 + render_markdown_report,
  54 +)
  55 +
  56 +output_dir = Path("perf_reports/20260318/translation_local_models_ct2")
  57 +output_dir.mkdir(parents=True, exist_ok=True)
  58 +
  59 +common = dict(
  60 + csv_path="products_analyzed.csv",
  61 + limit=0,
  62 + output_dir=str(output_dir),
  63 + single=True,
  64 + scene="sku_name",
  65 + batch_size=0,
  66 + device_override="",
  67 + torch_dtype_override="",
  68 + max_new_tokens=0,
  69 + num_beams=0,
  70 + attn_implementation="",
  71 + warmup_batches=1,
  72 + disable_cache=True,
  73 + suite="extended",
  74 + batch_size_list="",
  75 + concurrency_list="",
  76 + serial_items_per_case=256,
  77 + concurrency_requests_per_case=32,
  78 + concurrency_batch_size=1,
  79 + max_batch_concurrency_product=128,
  80 +)
  81 +
  82 +report = {
  83 + "generated_at": datetime.now().isoformat(timespec="seconds"),
  84 + "suite": "extended",
  85 + "environment": build_environment_info(),
  86 + "scenarios": [],
  87 +}
  88 +
  89 +for scenario in SCENARIOS:
  90 + args = SimpleNamespace(
  91 + **common,
  92 + model=scenario["model"],
  93 + source_lang=scenario["source_lang"],
  94 + target_lang=scenario["target_lang"],
  95 + column=scenario["column"],
  96 + )
  97 + result = benchmark_extended_scenario(args)
  98 + result["scenario"]["name"] = scenario["name"]
  99 + report["scenarios"].append(result)
  100 +
  101 +stamp = datetime.now().strftime("%H%M%S")
  102 +(output_dir / f"translation_local_models_ct2_extended_{stamp}.json").write_text(
  103 + json.dumps(report, ensure_ascii=False, indent=2),
  104 + encoding="utf-8",
  105 +)
  106 +(output_dir / f"translation_local_models_ct2_extended_{stamp}.md").write_text(
  107 + render_markdown_report(report),
  108 + encoding="utf-8",
  109 +)
  110 +PY
  111 +```
  112 +
  113 +## Key Results
  114 +
  115 +### 1. 单流 batch sweep
  116 +
  117 +| Model | Direction | Best batch | Best items/s | Batch 16 items/s | Batch 16 p95 ms |
  118 +|---|---|---:|---:|---:|---:|
  119 +| `nllb-200-distilled-600m` | `zh -> en` | `64` | `104.61` | `55.68` | `371.36` |
  120 +| `nllb-200-distilled-600m` | `en -> zh` | `64` | `91.26` | `42.42` | `408.81` |
  121 +| `opus-mt-zh-en` | `zh -> en` | `64` | `218.5` | `111.61` | `257.18` |
  122 +| `opus-mt-en-zh` | `en -> zh` | `32` | `145.12` | `102.05` | `396.16` |
  123 +
  124 +解读：
  125 +- 4 个方向的 bulk 吞吐都明显高于原始 Hugging Face / PyTorch 基线。
  126 +- `nllb en->zh` 的 batch 16 吞吐从 `13.52` 提升到 `42.42 items/s`，提升最明显。
  127 +- `opus-mt-en-zh` 在 CT2 版本里最佳 batch 从 `64` 变成了 `32`，说明它不再需要极端大 batch 才能吃满吞吐。
  128 +
  129 +### 2. 单条请求并发 sweep
  130 +
  131 +| Model | Direction | c=1 items/s | c=1 p95 ms | c=8 p95 ms | c=64 p95 ms |
  132 +|---|---|---:|---:|---:|---:|
  133 +| `nllb-200-distilled-600m` | `zh -> en` | `8.97` | `163.53` | `1039.32` | `3031.64` |
  134 +| `nllb-200-distilled-600m` | `en -> zh` | `5.83` | `259.52` | `2193.01` | `5611.21` |
  135 +| `opus-mt-zh-en` | `zh -> en` | `27.85` | `60.61` | `390.32` | `1061.35` |
  136 +| `opus-mt-en-zh` | `en -> zh` | `11.02` | `351.74` | `863.08` | `2459.49` |
  137 +
  138 +解读：
  139 +- 在线 query 指标提升非常明显，特别是 `batch_size=1` 的 `p95` 和 `items/s`。
  140 +- CT2 下并发上升仍会推高尾延迟，但恶化幅度比基线小得多。
  141 +- `opus-mt-zh-en` 仍然是在线场景最稳的本地模型；`nllb` 现在也进入了更可用的区间。
  142 +
  143 +### 3. 是否达到预期
  144 +
  145 +结论：
  146 +- **达到了，而且幅度很大。**
  147 +- 本轮 CT2 版本已经满足“在线性能显著增强”的目标，不需要继续为吞吐/延迟做额外紧急优化。
  148 +
  149 +判断依据：
  150 +- 4 个方向在 `concurrency=1` 下的 `items/s` 全部提升到原来的 `2.0x-3.1x`
  151 +- 4 个方向在 `concurrency=1` 下的 `p95` 全部下降到原来的 `29%-44%`
  152 +- NLLB 两个方向的 `batch_size=16` 吞吐分别提升 `2.04x` 和 `3.14x`
  153 +
  154 +## Notes
  155 +
  156 +- 这轮 `peak_gpu_memory_gb` 基本显示为 `0.0`，不是“CT2 不占显存”，而是当前脚本用的是 `torch.cuda` 统计，无法观测 CT2 的原生 CUDA 分配。
  157 +- 如果后续要补充“显存对比”维度，建议新增 `nvidia-smi` 采样或 NVML 指标采集。
... ...
perf_reports/20260318/translation_local_models_ct2/comparison_vs_hf_baseline.md 0 → 100644
  Show/Hide comments   View file @46ce858
... ... @@ -0,0 +1,89 @@
  1 +# CTranslate2 vs HF Baseline
  2 +
  3 +对比对象：
  4 +- 基线：[`translation_local_models_extended_221846.json`](/data/saas-search/perf_reports/20260318/translation_local_models/translation_local_models_extended_221846.json)
  5 +- CT2：[`translation_local_models_ct2_extended_233253.json`](/data/saas-search/perf_reports/20260318/translation_local_models_ct2/translation_local_models_ct2_extended_233253.json)
  6 +
  7 +结论：
  8 +- **本轮 CT2 优化达到预期。**
  9 +- 在线翻译的核心指标已经是“明显提升”，不是边际改善。
  10 +- 因此本轮不再继续做第二轮运行时调优，而是先把这组结果沉淀为新的性能基线。
  11 +
  12 +## Online Metrics
  13 +
  14 +关注口径：
  15 +- `concurrency_sweep`
  16 +- `batch_size=1`
  17 +- 重点看 `c=1 items/s`、`c=1 p95 ms`
  18 +- 辅助看 `c=8 p95 ms`、`c=64 p95 ms`
  19 +
  20 +| Model | Direction | c=1 items/s Baseline | c=1 items/s CT2 | Gain | c=1 p95 Baseline ms | c=1 p95 CT2 ms | Reduction |
  21 +|---|---|---:|---:|---:|---:|---:|---:|
  22 +| `nllb-200-distilled-600m` | `zh -> en` | `4.17` | `8.97` | `+115.11%` | `373.27` | `163.53` | `-56.19%` |
  23 +| `nllb-200-distilled-600m` | `en -> zh` | `2.16` | `5.83` | `+169.91%` | `670.78` | `259.52` | `-61.31%` |
  24 +| `opus-mt-zh-en` | `zh -> en` | `9.21` | `27.85` | `+202.39%` | `179.12` | `60.61` | `-66.16%` |
  25 +| `opus-mt-en-zh` | `en -> zh` | `3.6` | `11.02` | `+206.11%` | `1180.37` | `351.74` | `-70.20%` |
  26 +
  27 +解读：
  28 +- 4 个方向的单条请求吞吐都至少翻倍。
  29 +- 4 个方向的 `p95` 都下降了 `56%-70%`。
  30 +- 从在线 query 的角度，这已经足够认定为“显著增强”。
  31 +
  32 +## Tail Latency
  33 +
  34 +| Model | Direction | c=8 p95 Baseline ms | c=8 p95 CT2 ms | Reduction | c=64 p95 Baseline ms | c=64 p95 CT2 ms | Reduction |
  35 +|---|---|---:|---:|---:|---:|---:|---:|
  36 +| `nllb-200-distilled-600m` | `zh -> en` | `2383.8` | `1039.32` | `-56.40%` | `7337.3` | `3031.64` | `-58.68%` |
  37 +| `nllb-200-distilled-600m` | `en -> zh` | `3971.01` | `2193.01` | `-44.77%` | `14139.03` | `5611.21` | `-60.31%` |
  38 +| `opus-mt-zh-en` | `zh -> en` | `1043.06` | `390.32` | `-62.58%` | `3381.58` | `1061.35` | `-68.61%` |
  39 +| `opus-mt-en-zh` | `en -> zh` | `3632.99` | `863.08` | `-76.24%` | `7950.41` | `2459.49` | `-69.06%` |
  40 +
  41 +解读：
  42 +- CT2 不只是提升了低并发平均速度，也明显压低了高并发 tail latency。
  43 +- `nllb en->zh` 仍然是四个方向里最重的，但已经从“非常重”变成“可接受得多”。
  44 +
  45 +## Bulk Metrics
  46 +
  47 +对 bulk 口径，优先看 `batch_sweep` 的 `batch_size=16`，因为它更接近实际服务里兼顾吞吐和延迟的默认配置。
  48 +
  49 +| Model | Direction | Batch16 items/s Baseline | Batch16 items/s CT2 | Gain | Batch16 p95 Baseline ms | Batch16 p95 CT2 ms | Reduction |
  50 +|---|---|---:|---:|---:|---:|---:|---:|
  51 +| `nllb-200-distilled-600m` | `zh -> en` | `27.28` | `55.68` | `+104.11%` | `769.18` | `371.36` | `-51.72%` |
  52 +| `nllb-200-distilled-600m` | `en -> zh` | `13.52` | `42.42` | `+213.76%` | `1649.65` | `408.81` | `-75.22%` |
  53 +| `opus-mt-zh-en` | `zh -> en` | `41.44` | `111.61` | `+169.33%` | `797.93` | `257.18` | `-67.77%` |
  54 +| `opus-mt-en-zh` | `en -> zh` | `24.33` | `102.05` | `+319.44%` | `2098.54` | `396.16` | `-81.12%` |
  55 +
  56 +解读：
  57 +- `nllb-200-distilled-600m` 是这次最值得的优化对象，尤其 `en -> zh` 收益非常大。
  58 +- `opus-mt-en-zh` 的提升更夸张，说明之前它在 HF 路径上有很重的运行时损耗。
  59 +
  60 +## Best Throughput Cases
  61 +
  62 +| Model | Direction | Baseline Best Matrix | Baseline Items/s | CT2 Best Matrix | CT2 Items/s |
  63 +|---|---|---|---:|---|---:|
  64 +| `nllb-200-distilled-600m` | `zh -> en` | `batch=64, concurrency=2` | `53.95` | `batch=64, concurrency=2` | `114.37` |
  65 +| `nllb-200-distilled-600m` | `en -> zh` | `batch=64, concurrency=1` | `34.97` | `batch=64, concurrency=2` | `95.59` |
  66 +| `opus-mt-zh-en` | `zh -> en` | `batch=64, concurrency=1` | `52.44` | `batch=64, concurrency=2` | `207.86` |
  67 +| `opus-mt-en-zh` | `en -> zh` | `batch=64, concurrency=1` | `34.94` | `batch=64, concurrency=2` | `140.91` |
  68 +
  69 +解读：
  70 +- CT2 版本在矩阵里的最优吞吐普遍提升到了原来的 `2.1x-4.0x`。
  71 +- 而且最佳点不再总是“单并发 + 大 batch”，`concurrency=2` 开始变得更有意义。
  72 +
  73 +## Why We Stop Here
  74 +
  75 +本轮没有继续做第二轮优化，原因很直接：
  76 +
  77 +- 目标是“在线性能显著增强”，这个目标已经达成。
  78 +- 关键在线指标已经不是小幅改进，而是普遍 `2x+` 吞吐和 `50%+` `p95` 降幅。
  79 +- 当前更合理的下一步是把 CT2 结果作为新的基线，再决定是否需要做更细的服务化优化。
  80 +
  81 +## Remaining Gaps
  82 +
  83 +- 质量仍需和业务样本集一起看，尤其 `opus-mt-en-zh` 的少量短句结果可能需要质量侧复核。
  84 +- 当前脚本的 `peak_gpu_memory_gb` 对 CT2 无效，因为它只读取 `torch.cuda` 统计，不覆盖 CT2 的原生 CUDA 分配。
  85 +- 如果下一轮目标从“显著提速”转到“进一步压 tail latency”，优先方向会是：
  86 + - 增加服务级微批处理队列
  87 + - 拆分短文本 / 长文本请求桶
  88 + - 评估 `ct2_inter_threads` 与 `max_queued_batches`
  89 + - 对 `opus-mt-en-zh` 和 `nllb en->zh` 做更细粒度 batch 默认值调参
... ...
perf_reports/20260318/translation_local_models_ct2/translation_local_models_ct2_extended_233253.md 0 → 100644
  Show/Hide comments   View file @46ce858
... ... @@ -0,0 +1,263 @@
  1 +# Local Translation Model Extended Benchmark
  2 +
  3 +- Generated at: `2026-03-18T23:15:04`
  4 +- Suite: `extended`
  5 +- Python: `3.12.3`
  6 +- Torch: `2.10.0+cu128`
  7 +- Transformers: `5.3.0`
  8 +- CUDA: `True`
  9 +- GPU: `Tesla T4` (15.56 GiB)
  10 +
  11 +## Reading Guide
  12 +
  13 +- `batch_sweep`: single stream only (`concurrency=1`), used to compare bulk translation efficiency across batch sizes.
  14 +- `concurrency_sweep`: fixed request batch size, used to compare online request latency and throughput as concurrency rises.
  15 +- `matrix`: combined `batch_size x concurrency` runs, filtered by `batch_size * concurrency <= limit` when configured.
  16 +
  17 +## nllb-200-distilled-600m zh->en
  18 +
  19 +- Direction: `zh -> en`
  20 +- Column: `title_cn`
  21 +- Loaded rows: `2048`
  22 +- Load time: `6.0581 s`
  23 +- Device: `cuda`
  24 +- DType: `float16`
  25 +- Cache disabled: `True`
  26 +
  27 +### Batch Sweep (`concurrency=1`)
  28 +
  29 +| Batch | Rows | Requests | Items/s | Req/s | Avg req ms | Req p50 ms | Req p95 ms | Peak GPU GiB |
  30 +|---:|---:|---:|---:|---:|---:|---:|---:|---:|
  31 +| 1 | 256 | 256 | 7.0 | 7.0 | 142.91 | 132.84 | 260.42 | 0.0 |
  32 +| 4 | 256 | 64 | 20.03 | 5.01 | 199.68 | 198.01 | 302.33 | 0.0 |
  33 +| 8 | 256 | 32 | 35.77 | 4.47 | 223.62 | 233.7 | 312.39 | 0.0 |
  34 +| 16 | 256 | 16 | 55.68 | 3.48 | 287.36 | 312.34 | 371.36 | 0.0 |
  35 +| 32 | 256 | 8 | 83.28 | 2.6 | 384.23 | 395.15 | 466.99 | 0.0 |
  36 +| 64 | 256 | 4 | 104.61 | 1.63 | 611.77 | 607.09 | 714.28 | 0.0 |
  37 +
  38 +### Concurrency Sweep (`batch_size=1`)
  39 +
  40 +| Concurrency | Rows | Requests | Items/s | Req/s | Avg req ms | Req p50 ms | Req p95 ms | Peak GPU GiB |
  41 +|---:|---:|---:|---:|---:|---:|---:|---:|---:|
  42 +| 1 | 32 | 32 | 8.97 | 8.97 | 111.26 | 101.15 | 163.53 | 0.0 |
  43 +| 2 | 32 | 32 | 9.56 | 9.56 | 204.83 | 185.58 | 312.06 | 0.0 |
  44 +| 4 | 32 | 32 | 9.4 | 9.4 | 399.27 | 385.5 | 511.85 | 0.0 |
  45 +| 8 | 32 | 32 | 9.81 | 9.81 | 695.45 | 702.43 | 1039.32 | 0.0 |
  46 +| 16 | 32 | 32 | 9.98 | 9.98 | 1174.49 | 1251.56 | 2541.88 | 0.0 |
  47 +| 64 | 32 | 32 | 9.71 | 9.71 | 1593.45 | 1567.36 | 3031.64 | 0.0 |
  48 +
  49 +### Batch x Concurrency Matrix
  50 +
  51 +| Batch | Concurrency | Rows | Requests | Items/s | Req/s | Avg req ms | Req p50 ms | Req p95 ms | Peak GPU GiB |
  52 +|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
  53 +| 1 | 1 | 32 | 32 | 9.97 | 9.97 | 100.21 | 91.15 | 154.02 | 0.0 |
  54 +| 1 | 2 | 32 | 32 | 10.19 | 10.19 | 191.51 | 182.72 | 281.96 | 0.0 |
  55 +| 1 | 4 | 32 | 32 | 10.4 | 10.4 | 358.38 | 345.82 | 488.71 | 0.0 |
  56 +| 1 | 8 | 32 | 32 | 10.21 | 10.21 | 671.22 | 722.27 | 1049.38 | 0.0 |
  57 +| 1 | 16 | 32 | 32 | 10.01 | 10.01 | 1168.24 | 1325.37 | 2211.32 | 0.0 |
  58 +| 1 | 64 | 32 | 32 | 9.95 | 9.95 | 1542.66 | 1493.91 | 2952.68 | 0.0 |
  59 +| 4 | 1 | 128 | 32 | 22.07 | 5.52 | 181.17 | 168.82 | 291.07 | 0.0 |
  60 +| 4 | 2 | 128 | 32 | 24.37 | 6.09 | 322.06 | 288.42 | 548.09 | 0.0 |
  61 +| 4 | 4 | 128 | 32 | 24.24 | 6.06 | 620.93 | 543.95 | 1055.41 | 0.0 |
  62 +| 4 | 8 | 128 | 32 | 22.47 | 5.62 | 1207.2 | 1203.59 | 1951.79 | 0.0 |
  63 +| 4 | 16 | 128 | 32 | 22.22 | 5.56 | 2073.8 | 1972.31 | 4344.0 | 0.0 |
  64 +| 8 | 1 | 256 | 32 | 32.44 | 4.06 | 246.41 | 245.92 | 337.1 | 0.0 |
  65 +| 8 | 2 | 256 | 32 | 34.57 | 4.32 | 452.65 | 477.44 | 608.67 | 0.0 |
  66 +| 8 | 4 | 256 | 32 | 34.37 | 4.3 | 875.33 | 948.09 | 1137.34 | 0.0 |
  67 +| 8 | 8 | 256 | 32 | 34.76 | 4.35 | 1596.54 | 1844.24 | 2135.61 | 0.0 |
  68 +| 8 | 16 | 256 | 32 | 35.31 | 4.41 | 2680.01 | 3376.09 | 3918.49 | 0.0 |
  69 +| 16 | 1 | 512 | 32 | 52.98 | 3.31 | 301.86 | 316.96 | 387.51 | 0.0 |
  70 +| 16 | 2 | 512 | 32 | 54.2 | 3.39 | 581.13 | 622.86 | 723.4 | 0.0 |
  71 +| 16 | 4 | 512 | 32 | 54.06 | 3.38 | 1135.6 | 1189.63 | 1460.36 | 0.0 |
  72 +| 16 | 8 | 512 | 32 | 53.91 | 3.37 | 2131.92 | 2427.31 | 2785.87 | 0.0 |
  73 +| 32 | 1 | 1024 | 32 | 81.53 | 2.55 | 391.96 | 406.43 | 452.1 | 0.0 |
  74 +| 32 | 2 | 1024 | 32 | 80.98 | 2.53 | 777.72 | 795.59 | 905.77 | 0.0 |
  75 +| 32 | 4 | 1024 | 32 | 80.23 | 2.51 | 1525.26 | 1566.41 | 1831.7 | 0.0 |
  76 +| 64 | 1 | 2048 | 32 | 110.08 | 1.72 | 580.78 | 586.87 | 691.83 | 0.0 |
  77 +| 64 | 2 | 2048 | 32 | 114.37 | 1.79 | 1100.27 | 1127.51 | 1243.02 | 0.0 |
  78 +
  79 +## nllb-200-distilled-600m en->zh
  80 +
  81 +- Direction: `en -> zh`
  82 +- Column: `title`
  83 +- Loaded rows: `2048`
  84 +- Load time: `5.564 s`
  85 +- Device: `cuda`
  86 +- DType: `float16`
  87 +- Cache disabled: `True`
  88 +
  89 +### Batch Sweep (`concurrency=1`)
  90 +
  91 +| Batch | Rows | Requests | Items/s | Req/s | Avg req ms | Req p50 ms | Req p95 ms | Peak GPU GiB |
  92 +|---:|---:|---:|---:|---:|---:|---:|---:|---:|
  93 +| 1 | 256 | 256 | 5.03 | 5.03 | 198.6 | 192.97 | 319.06 | 0.0 |
  94 +| 4 | 256 | 64 | 13.74 | 3.44 | 291.05 | 311.76 | 349.79 | 0.0 |
  95 +| 8 | 256 | 32 | 24.46 | 3.06 | 327.02 | 336.32 | 366.86 | 0.0 |
  96 +| 16 | 256 | 16 | 42.42 | 2.65 | 377.19 | 381.47 | 408.81 | 0.0 |
  97 +| 32 | 256 | 8 | 67.38 | 2.11 | 474.92 | 474.35 | 502.51 | 0.0 |
  98 +| 64 | 256 | 4 | 91.26 | 1.43 | 701.3 | 707.24 | 738.5 | 0.0 |
  99 +
  100 +### Concurrency Sweep (`batch_size=1`)
  101 +
  102 +| Concurrency | Rows | Requests | Items/s | Req/s | Avg req ms | Req p50 ms | Req p95 ms | Peak GPU GiB |
  103 +|---:|---:|---:|---:|---:|---:|---:|---:|---:|
  104 +| 1 | 32 | 32 | 5.83 | 5.83 | 171.33 | 162.21 | 259.52 | 0.0 |
  105 +| 2 | 32 | 32 | 5.62 | 5.62 | 351.38 | 349.24 | 479.41 | 0.0 |
  106 +| 4 | 32 | 32 | 5.51 | 5.51 | 687.51 | 695.0 | 875.41 | 0.0 |
  107 +| 8 | 32 | 32 | 5.53 | 5.53 | 1279.01 | 1331.94 | 2193.01 | 0.0 |
  108 +| 16 | 32 | 32 | 5.45 | 5.45 | 2211.06 | 2621.54 | 3879.29 | 0.0 |
  109 +| 64 | 32 | 32 | 5.37 | 5.37 | 3113.23 | 3074.42 | 5611.21 | 0.0 |
  110 +
  111 +### Batch x Concurrency Matrix
  112 +
  113 +| Batch | Concurrency | Rows | Requests | Items/s | Req/s | Avg req ms | Req p50 ms | Req p95 ms | Peak GPU GiB |
  114 +|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
  115 +| 1 | 1 | 32 | 32 | 5.5 | 5.5 | 181.68 | 171.06 | 262.47 | 0.0 |
  116 +| 1 | 2 | 32 | 32 | 5.54 | 5.54 | 356.26 | 348.75 | 479.03 | 0.0 |
  117 +| 1 | 4 | 32 | 32 | 5.55 | 5.55 | 683.35 | 697.01 | 845.47 | 0.0 |
  118 +| 1 | 8 | 32 | 32 | 5.87 | 5.87 | 1218.56 | 1297.55 | 1811.3 | 0.0 |
  119 +| 1 | 16 | 32 | 32 | 5.68 | 5.68 | 2076.8 | 2443.82 | 3078.65 | 0.0 |
  120 +| 1 | 64 | 32 | 32 | 5.4 | 5.4 | 3006.17 | 3009.57 | 5493.11 | 0.0 |
  121 +| 4 | 1 | 128 | 32 | 14.65 | 3.66 | 272.44 | 267.49 | 344.28 | 0.0 |
  122 +| 4 | 2 | 128 | 32 | 15.03 | 3.76 | 524.29 | 523.34 | 650.15 | 0.0 |
  123 +| 4 | 4 | 128 | 32 | 14.95 | 3.74 | 1019.74 | 1043.76 | 1211.75 | 0.0 |
  124 +| 4 | 8 | 128 | 32 | 14.63 | 3.66 | 1952.63 | 2251.77 | 2450.21 | 0.0 |
  125 +| 4 | 16 | 128 | 32 | 14.68 | 3.67 | 3303.53 | 4188.22 | 4628.4 | 0.0 |
  126 +| 8 | 1 | 256 | 32 | 25.36 | 3.17 | 315.38 | 322.31 | 345.35 | 0.0 |
  127 +| 8 | 2 | 256 | 32 | 25.1 | 3.14 | 627.19 | 647.23 | 709.93 | 0.0 |
  128 +| 8 | 4 | 256 | 32 | 25.11 | 3.14 | 1212.91 | 1259.02 | 1357.39 | 0.0 |
  129 +| 8 | 8 | 256 | 32 | 25.09 | 3.14 | 2267.61 | 2538.76 | 2620.04 | 0.0 |
  130 +| 8 | 16 | 256 | 32 | 24.74 | 3.09 | 3940.32 | 5035.7 | 5297.16 | 0.0 |
  131 +| 16 | 1 | 512 | 32 | 42.88 | 2.68 | 372.8 | 371.73 | 417.6 | 0.0 |
  132 +| 16 | 2 | 512 | 32 | 44.16 | 2.76 | 712.56 | 734.6 | 768.42 | 0.0 |
  133 +| 16 | 4 | 512 | 32 | 44.08 | 2.76 | 1385.99 | 1460.14 | 1502.65 | 0.0 |
  134 +| 16 | 8 | 512 | 32 | 43.7 | 2.73 | 2617.84 | 2954.76 | 3005.53 | 0.0 |
  135 +| 32 | 1 | 1024 | 32 | 66.94 | 2.09 | 476.4 | 469.61 | 523.49 | 0.0 |
  136 +| 32 | 2 | 1024 | 32 | 69.75 | 2.18 | 902.46 | 912.39 | 977.26 | 0.0 |
  137 +| 32 | 4 | 1024 | 32 | 69.36 | 2.17 | 1759.01 | 1839.11 | 1888.06 | 0.0 |
  138 +| 64 | 1 | 2048 | 32 | 92.15 | 1.44 | 693.91 | 692.06 | 731.65 | 0.0 |
  139 +| 64 | 2 | 2048 | 32 | 95.59 | 1.49 | 1315.82 | 1338.66 | 1387.23 | 0.0 |
  140 +
  141 +## opus-mt-zh-en zh->en
  142 +
  143 +- Direction: `zh -> en`
  144 +- Column: `title_cn`
  145 +- Loaded rows: `2048`
  146 +- Load time: `1.0381 s`
  147 +- Device: `cuda`
  148 +- DType: `float16`
  149 +- Cache disabled: `True`
  150 +
  151 +### Batch Sweep (`concurrency=1`)
  152 +
  153 +| Batch | Rows | Requests | Items/s | Req/s | Avg req ms | Req p50 ms | Req p95 ms | Peak GPU GiB |
  154 +|---:|---:|---:|---:|---:|---:|---:|---:|---:|
  155 +| 1 | 256 | 256 | 18.84 | 18.84 | 53.08 | 47.66 | 95.63 | 0.0 |
  156 +| 4 | 256 | 64 | 47.6 | 11.9 | 84.03 | 75.74 | 119.84 | 0.0 |
  157 +| 8 | 256 | 32 | 74.34 | 9.29 | 107.61 | 96.19 | 141.84 | 0.0 |
  158 +| 16 | 256 | 16 | 111.61 | 6.98 | 143.34 | 126.49 | 257.18 | 0.0 |
  159 +| 32 | 256 | 8 | 154.61 | 4.83 | 206.96 | 158.62 | 438.14 | 0.0 |
  160 +| 64 | 256 | 4 | 218.5 | 3.41 | 292.9 | 213.26 | 547.27 | 0.0 |
  161 +
  162 +### Concurrency Sweep (`batch_size=1`)
  163 +
  164 +| Concurrency | Rows | Requests | Items/s | Req/s | Avg req ms | Req p50 ms | Req p95 ms | Peak GPU GiB |
  165 +|---:|---:|---:|---:|---:|---:|---:|---:|---:|
  166 +| 1 | 32 | 32 | 27.85 | 27.85 | 35.78 | 33.41 | 60.61 | 0.0 |
  167 +| 2 | 32 | 32 | 28.8 | 28.8 | 67.82 | 62.33 | 95.46 | 0.0 |
  168 +| 4 | 32 | 32 | 28.62 | 28.62 | 130.54 | 125.84 | 201.95 | 0.0 |
  169 +| 8 | 32 | 32 | 28.55 | 28.55 | 242.59 | 227.8 | 390.32 | 0.0 |
  170 +| 16 | 32 | 32 | 27.28 | 27.28 | 449.66 | 521.12 | 912.62 | 0.0 |
  171 +| 64 | 32 | 32 | 27.4 | 27.4 | 557.86 | 517.06 | 1061.35 | 0.0 |
  172 +
  173 +### Batch x Concurrency Matrix
  174 +
  175 +| Batch | Concurrency | Rows | Requests | Items/s | Req/s | Avg req ms | Req p50 ms | Req p95 ms | Peak GPU GiB |
  176 +|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
  177 +| 1 | 1 | 32 | 32 | 25.91 | 25.91 | 38.29 | 34.67 | 61.44 | 0.0 |
  178 +| 1 | 2 | 32 | 32 | 28.2 | 28.2 | 68.02 | 61.92 | 110.41 | 0.0 |
  179 +| 1 | 4 | 32 | 32 | 27.03 | 27.03 | 139.18 | 134.97 | 184.23 | 0.0 |
  180 +| 1 | 8 | 32 | 32 | 27.87 | 27.87 | 248.14 | 251.39 | 356.34 | 0.0 |
  181 +| 1 | 16 | 32 | 32 | 27.1 | 27.1 | 424.06 | 477.76 | 612.64 | 0.0 |
  182 +| 1 | 64 | 32 | 32 | 29.17 | 29.17 | 503.96 | 475.72 | 997.76 | 0.0 |
  183 +| 4 | 1 | 128 | 32 | 48.35 | 12.09 | 82.67 | 66.35 | 116.75 | 0.0 |
  184 +| 4 | 2 | 128 | 32 | 48.09 | 12.02 | 164.18 | 121.89 | 401.43 | 0.0 |
  185 +| 4 | 4 | 128 | 32 | 48.2 | 12.05 | 290.2 | 251.27 | 561.3 | 0.0 |
  186 +| 4 | 8 | 128 | 32 | 49.45 | 12.36 | 531.9 | 449.31 | 1115.99 | 0.0 |
  187 +| 4 | 16 | 128 | 32 | 58.28 | 14.57 | 713.26 | 834.24 | 1325.58 | 0.0 |
  188 +| 8 | 1 | 256 | 32 | 96.87 | 12.11 | 82.53 | 70.28 | 121.7 | 0.0 |
  189 +| 8 | 2 | 256 | 32 | 106.37 | 13.3 | 148.34 | 125.3 | 357.79 | 0.0 |
  190 +| 8 | 4 | 256 | 32 | 111.35 | 13.92 | 274.44 | 250.25 | 600.44 | 0.0 |
  191 +| 8 | 8 | 256 | 32 | 96.65 | 12.08 | 579.06 | 667.78 | 1025.51 | 0.0 |
  192 +| 8 | 16 | 256 | 32 | 80.62 | 10.08 | 1236.24 | 1557.05 | 1886.21 | 0.0 |
  193 +| 16 | 1 | 512 | 32 | 91.27 | 5.7 | 174.5 | 124.75 | 533.12 | 0.0 |
  194 +| 16 | 2 | 512 | 32 | 105.66 | 6.6 | 299.07 | 220.94 | 631.62 | 0.0 |
  195 +| 16 | 4 | 512 | 32 | 106.3 | 6.64 | 581.51 | 481.69 | 1163.67 | 0.0 |
  196 +| 16 | 8 | 512 | 32 | 104.18 | 6.51 | 1039.37 | 1160.52 | 1800.29 | 0.0 |
  197 +| 32 | 1 | 1024 | 32 | 119.48 | 3.73 | 267.65 | 163.8 | 586.28 | 0.0 |
  198 +| 32 | 2 | 1024 | 32 | 140.5 | 4.39 | 442.92 | 311.65 | 820.49 | 0.0 |
  199 +| 32 | 4 | 1024 | 32 | 150.2 | 4.69 | 807.09 | 827.33 | 1236.61 | 0.0 |
  200 +| 64 | 1 | 2048 | 32 | 184.12 | 2.88 | 347.39 | 264.91 | 617.24 | 0.0 |
  201 +| 64 | 2 | 2048 | 32 | 207.86 | 3.25 | 610.34 | 684.9 | 1064.5 | 0.0 |
  202 +
  203 +## opus-mt-en-zh en->zh
  204 +
  205 +- Direction: `en -> zh`
  206 +- Column: `title`
  207 +- Loaded rows: `2048`
  208 +- Load time: `0.3704 s`
  209 +- Device: `cuda`
  210 +- DType: `float16`
  211 +- Cache disabled: `True`
  212 +
  213 +### Batch Sweep (`concurrency=1`)
  214 +
  215 +| Batch | Rows | Requests | Items/s | Req/s | Avg req ms | Req p50 ms | Req p95 ms | Peak GPU GiB |
  216 +|---:|---:|---:|---:|---:|---:|---:|---:|---:|
  217 +| 1 | 256 | 256 | 13.17 | 13.17 | 75.92 | 64.12 | 136.47 | 0.0 |
  218 +| 4 | 256 | 64 | 28.7 | 7.17 | 139.37 | 112.06 | 476.36 | 0.0 |
  219 +| 8 | 256 | 32 | 49.1 | 6.14 | 162.93 | 124.84 | 391.18 | 0.0 |
  220 +| 16 | 256 | 16 | 102.05 | 6.38 | 156.78 | 124.18 | 396.16 | 0.0 |
  221 +| 32 | 256 | 8 | 145.12 | 4.53 | 220.5 | 176.43 | 408.92 | 0.0 |
  222 +| 64 | 256 | 4 | 131.67 | 2.06 | 486.05 | 493.97 | 654.21 | 0.0 |
  223 +
  224 +### Concurrency Sweep (`batch_size=1`)
  225 +
  226 +| Concurrency | Rows | Requests | Items/s | Req/s | Avg req ms | Req p50 ms | Req p95 ms | Peak GPU GiB |
  227 +|---:|---:|---:|---:|---:|---:|---:|---:|---:|
  228 +| 1 | 32 | 32 | 11.02 | 11.02 | 90.71 | 54.35 | 351.74 | 0.0 |
  229 +| 2 | 32 | 32 | 11.29 | 11.29 | 174.4 | 117.99 | 536.84 | 0.0 |
  230 +| 4 | 32 | 32 | 11.6 | 11.6 | 307.77 | 233.97 | 823.12 | 0.0 |
  231 +| 8 | 32 | 32 | 11.54 | 11.54 | 471.91 | 438.97 | 863.08 | 0.0 |
  232 +| 16 | 32 | 32 | 10.86 | 10.86 | 906.19 | 949.77 | 1827.08 | 0.0 |
  233 +| 64 | 32 | 32 | 11.31 | 11.31 | 1095.54 | 919.35 | 2459.49 | 0.0 |
  234 +
  235 +### Batch x Concurrency Matrix
  236 +
  237 +| Batch | Concurrency | Rows | Requests | Items/s | Req/s | Avg req ms | Req p50 ms | Req p95 ms | Peak GPU GiB |
  238 +|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
  239 +| 1 | 1 | 32 | 32 | 10.72 | 10.72 | 92.99 | 58.17 | 361.99 | 0.0 |
  240 +| 1 | 2 | 32 | 32 | 11.35 | 11.35 | 174.68 | 115.43 | 542.24 | 0.0 |
  241 +| 1 | 4 | 32 | 32 | 11.44 | 11.44 | 312.19 | 226.82 | 828.3 | 0.0 |
  242 +| 1 | 8 | 32 | 32 | 11.64 | 11.64 | 506.86 | 415.08 | 1147.15 | 0.0 |
  243 +| 1 | 16 | 32 | 32 | 11.15 | 11.15 | 1009.4 | 838.27 | 2103.9 | 0.0 |
  244 +| 1 | 64 | 32 | 32 | 11.08 | 11.08 | 1167.27 | 984.4 | 2532.28 | 0.0 |
  245 +| 4 | 1 | 128 | 32 | 31.83 | 7.96 | 125.51 | 109.44 | 216.9 | 0.0 |
  246 +| 4 | 2 | 128 | 32 | 38.97 | 9.74 | 203.2 | 180.61 | 419.46 | 0.0 |
  247 +| 4 | 4 | 128 | 32 | 43.78 | 10.95 | 353.68 | 343.79 | 546.77 | 0.0 |
  248 +| 4 | 8 | 128 | 32 | 37.23 | 9.31 | 770.39 | 880.33 | 964.82 | 0.0 |
  249 +| 4 | 16 | 128 | 32 | 32.48 | 8.12 | 1564.07 | 1825.11 | 2244.63 | 0.0 |
  250 +| 8 | 1 | 256 | 32 | 42.33 | 5.29 | 188.66 | 147.07 | 519.33 | 0.0 |
  251 +| 8 | 2 | 256 | 32 | 45.05 | 5.63 | 350.91 | 283.6 | 717.38 | 0.0 |
  252 +| 8 | 4 | 256 | 32 | 44.84 | 5.6 | 690.69 | 589.12 | 1260.54 | 0.0 |
  253 +| 8 | 8 | 256 | 32 | 44.6 | 5.58 | 1318.2 | 1368.26 | 2019.86 | 0.0 |
  254 +| 8 | 16 | 256 | 32 | 44.53 | 5.57 | 2168.52 | 2448.18 | 3105.11 | 0.0 |
  255 +| 16 | 1 | 512 | 32 | 59.23 | 3.7 | 270.08 | 206.79 | 568.32 | 0.0 |
  256 +| 16 | 2 | 512 | 32 | 78.16 | 4.88 | 392.89 | 270.81 | 709.39 | 0.0 |
  257 +| 16 | 4 | 512 | 32 | 65.35 | 4.08 | 921.0 | 946.61 | 1389.25 | 0.0 |
  258 +| 16 | 8 | 512 | 32 | 65.64 | 4.1 | 1697.84 | 1572.57 | 2423.97 | 0.0 |
  259 +| 32 | 1 | 1024 | 32 | 84.23 | 2.63 | 379.52 | 279.85 | 629.28 | 0.0 |
  260 +| 32 | 2 | 1024 | 32 | 101.77 | 3.18 | 610.89 | 620.5 | 1038.84 | 0.0 |
  261 +| 32 | 4 | 1024 | 32 | 100.64 | 3.15 | 1175.94 | 1129.62 | 1794.84 | 0.0 |
  262 +| 64 | 1 | 2048 | 32 | 136.74 | 2.14 | 467.8 | 476.82 | 680.43 | 0.0 |
  263 +| 64 | 2 | 2048 | 32 | 140.91 | 2.2 | 890.68 | 983.82 | 1138.98 | 0.0 |
... ...
perf_reports/20260318/translation_local_models_ct2_focus/README.md 0 → 100644
  Show/Hide comments   View file @46ce858
... ... @@ -0,0 +1,158 @@
  1 +# Local Translation Model Focused T4 Tuning
  2 +
  3 +测试脚本：
  4 +- [`scripts/benchmark_translation_local_models_focus.py`](/data/saas-search/scripts/benchmark_translation_local_models_focus.py)
  5 +
  6 +本轮聚焦结果：
  7 +- Markdown：[`translation_local_models_focus_235018.md`](/data/saas-search/perf_reports/20260318/translation_local_models_ct2_focus/translation_local_models_focus_235018.md)
  8 +- JSON：[`translation_local_models_focus_235018.json`](/data/saas-search/perf_reports/20260318/translation_local_models_ct2_focus/translation_local_models_focus_235018.json)
  9 +
  10 +说明：
  11 +- 这份报告是第一轮 T4 聚焦调优结论。
  12 +- 对 `nllb-200-distilled-600M`，当前最新推荐已经由专项报告覆盖：
  13 + [`../nllb_t4_product_names_ct2/README.md`](/data/saas-search/perf_reports/20260318/nllb_t4_product_names_ct2/README.md)
  14 +- 本页里关于 NLLB 的 `ct2_inter_threads=2 + ct2_max_queued_batches=16` 结论，应视为已被更新。
  15 +
  16 +相关报告：
  17 +- 基线扩展报告：[`../translation_local_models/README.md`](/data/saas-search/perf_reports/20260318/translation_local_models/README.md)
  18 +- CT2 扩展报告：[`../translation_local_models_ct2/README.md`](/data/saas-search/perf_reports/20260318/translation_local_models_ct2/README.md)
  19 +- CT2 与 HF 对比：[`../translation_local_models_ct2/comparison_vs_hf_baseline.md`](/data/saas-search/perf_reports/20260318/translation_local_models_ct2/comparison_vs_hf_baseline.md)
  20 +
  21 +测试时间：
  22 +- `2026-03-18`
  23 +
  24 +环境：
  25 +- GPU：`Tesla T4 16GB`
  26 +- Python env：`.venv-translator`
  27 +- Torch / Transformers：`2.10.0+cu128 / 5.3.0`
  28 +- CTranslate2：`4.7.1`
  29 +
  30 +## Scope
  31 +
  32 +这轮不再做完整矩阵，只看两个目标场景：
  33 +
  34 +- `high batch + low concurrency`
  35 + - `batch=32/64/128`
  36 + - `concurrency=1`
  37 +- `high concurrency + low batch`
  38 + - `batch=1`
  39 + - `concurrency=8/16/32/64`
  40 +
  41 +对比的两个 CT2 变体：
  42 +
  43 +- `ct2_default`
  44 + - 当前默认：`ct2_inter_threads=1`、`ct2_max_queued_batches=0`、`ct2_batch_type=examples`
  45 +- `ct2_tuned_t4`
  46 + - 调优候选：`ct2_inter_threads=2`、`ct2_max_queued_batches=16`、`ct2_batch_type=examples`
  47 +
  48 +## Recommendation
  49 +
  50 +结论先写在前面：
  51 +
  52 +- **NLLB 推荐升级到 `ct2_inter_threads=2 + ct2_max_queued_batches=16`。**
  53 +- `opus-mt-zh-en` 维持默认更稳。
  54 +- `opus-mt-en-zh` 在大 batch 和高并发吞吐上有收益，但在线 `c=8` 的 `p95` 有波动，不建议直接把同一套 tuned 参数作为线上默认。
  55 +
  56 +这也是为什么当前配置只把 NLLB 调成了 tuned profile，而两个 Marian 模型保持保守默认值。
  57 +
  58 +## Key Results
  59 +
  60 +### 1. NLLB 是这轮最值得调的模型
  61 +
  62 +`nllb-200-distilled-600m zh -> en`
  63 +
  64 +| Scenario | Default | Tuned | 结果 |
  65 +|---|---:|---:|---|
  66 +| `batch=64, concurrency=1` items/s | `113.25` | `111.86` | 基本持平 |
  67 +| `batch=64, concurrency=1` p95 ms | `662.38` | `657.84` | 基本持平 |
  68 +| `batch=1, concurrency=16` items/s | `10.34` | `12.91` | 明显提升 |
  69 +| `batch=1, concurrency=16` p95 ms | `1904.9` | `1368.92` | 明显下降 |
  70 +| `batch=1, concurrency=32` items/s | `10.17` | `12.8` | 明显提升 |
  71 +| `batch=1, concurrency=32` p95 ms | `2876.88` | `2350.5` | 明显下降 |
  72 +
  73 +`nllb-200-distilled-600m en -> zh`
  74 +
  75 +| Scenario | Default | Tuned | 结果 |
  76 +|---|---:|---:|---|
  77 +| `batch=64, concurrency=1` items/s | `96.27` | `93.36` | 小幅回落 |
  78 +| `batch=64, concurrency=1` p95 ms | `701.75` | `721.79` | 小幅变差 |
  79 +| `batch=1, concurrency=16` items/s | `5.51` | `7.91` | 明显提升 |
  80 +| `batch=1, concurrency=16` p95 ms | `4613.05` | `2039.17` | 大幅下降 |
  81 +| `batch=1, concurrency=32` items/s | `5.46` | `7.9` | 明显提升 |
  82 +| `batch=1, concurrency=32` p95 ms | `5554.4` | `3912.75` | 明显下降 |
  83 +
  84 +解读：
  85 +- NLLB 的 tuned profile 主要是把 T4 的并发潜力释放出来。
  86 +- bulk 场景几乎没有受伤，尤其 `zh -> en` 基本持平。
  87 +- 在线场景收益非常大，所以这轮调优最应该落在 NLLB 上。
  88 +
  89 +### 2. Marian 不适合统一套用 NLLB 的 tuned 参数
  90 +
  91 +`opus-mt-zh-en zh -> en`
  92 +
  93 +- `batch=64, concurrency=1`：`164.1 -> 151.21 items/s`，默认更好
  94 +- `batch=1, concurrency=32`：`27.5 -> 29.83 items/s`，tuned 略好
  95 +- `batch=1, concurrency=64`：`28.43 -> 26.85 items/s`，默认更好
  96 +
  97 +结论：
  98 +- 这个模型已经很轻，默认 profile 更均衡。
  99 +- 不值得为了少量中并发收益牺牲大 batch 或高并发稳定性。
  100 +
  101 +`opus-mt-en-zh en -> zh`
  102 +
  103 +- `batch=64, concurrency=1`：`114.34 -> 121.87 items/s`
  104 +- `batch=128, concurrency=1`：`162.29 -> 210.29 items/s`
  105 +- `batch=1, concurrency=16`：`11.22 -> 12.65 items/s`
  106 +- `batch=1, concurrency=8` 的 `p95` 从 `798.77` 变成 `1199.98`
  107 +
  108 +结论：
  109 +- 这个模型对 tuned profile 更敏感，吞吐会明显变好。
  110 +- 但在线 `c=8` 的 `p95` 变差，说明它更像“专用吞吐配置”，不适合直接作为统一线上默认。
  111 +
  112 +## T4 Experience Summary
  113 +
  114 +这轮真正有价值的经验：
  115 +
  116 +- **经验 1：不要再用完整矩阵找方向。**
  117 + - 先只看 `high batch + low concurrency` 和 `high concurrency + low batch` 两个极端，效率更高。
  118 +
  119 +- **经验 2：NLLB 在 T4 上确实吃 `inter_threads` 和队列深度。**
  120 + - `ct2_inter_threads=2`
  121 + - `ct2_max_queued_batches=16`
  122 + - 这组参数对高并发 `batch=1` 在线场景收益最明显。
  123 +
  124 +- **经验 3：`inter_threads=4` 太激进。**
  125 + - 它能把部分高并发吞吐继续往上推。
  126 + - 但会严重伤害大 batch 吞吐，尤其 `batch=64` 这类 bulk 场景。
  127 + - 因此不适合作为通用服务默认值。
  128 +
  129 +- **经验 4：`ct2_batch_type=tokens` 不是当前 T4 的主增益点。**
  130 + - 对 `batch=1` 的在线场景没有带来稳定收益。
  131 + - 当前项目里优先保留 `examples` 更稳妥。
  132 +
  133 +- **经验 5：单模型单 worker 仍然是默认部署方式。**
  134 + - 本轮调优解决的是同一 worker 内的 GPU 利用率问题。
  135 + - 不是靠堆 FastAPI worker 数来提吞吐。
  136 +
  137 +## Deployment / Config Tasks Worth Keeping
  138 +
  139 +这些任务被证明是“应该沉淀到文档和配置里”的：
  140 +
  141 +- 把本地 Marian / NLLB 统一迁移到 CTranslate2
  142 +- 使用 `float16` 转换并预生成 CT2 模型目录
  143 +- 保持单 worker，避免重复加载模型
  144 +- 对 NLLB 启用：
  145 + - `ct2_inter_threads=2`
  146 + - `ct2_max_queued_batches=16`
  147 + - `ct2_batch_type=examples`
  148 +- Marian 继续保守默认：
  149 + - `ct2_inter_threads=1`
  150 + - `ct2_max_queued_batches=0`
  151 +
  152 +## Next Step
  153 +
  154 +如果下一轮继续压线上延迟，优先顺序建议是：
  155 +
  156 +1. 服务级微批处理队列
  157 +2. 短文本 / 长文本分桶
  158 +3. 为 `opus-mt-en-zh` 增加“在线默认”和“离线高吞吐”两套配置
... ...
perf_reports/20260318/translation_local_models_ct2_focus/translation_local_models_focus_235018.md 0 → 100644
  Show/Hide comments   View file @46ce858
... ... @@ -0,0 +1,132 @@
  1 +# Local Translation Model Focused Benchmark
  2 +
  3 +- Generated at: `2026-03-18T23:45:50`
  4 +- Python: `3.12.3`
  5 +- Torch: `2.10.0+cu128`
  6 +- Transformers: `5.3.0`
  7 +- CUDA: `True`
  8 +- GPU: `Tesla T4` (15.56 GiB)
  9 +
  10 +## Scope
  11 +
  12 +- Scenario 1: high batch size + low concurrency
  13 +- Scenario 2: high concurrency + low batch size
  14 +- Variants in this report:
  15 + - `ct2_default`: `{}`
  16 + - `ct2_tuned_t4`: `{'ct2_inter_threads': 2, 'ct2_max_queued_batches': 16, 'ct2_batch_type': 'examples'}`
  17 +
  18 +## nllb-200-distilled-600m zh->en
  19 +
  20 +- Direction: `zh -> en`
  21 +- Column: `title_cn`
  22 +
  23 +### Variant `ct2_default`
  24 +
  25 +| Scenario | Setting | Items/s | Req p95 ms | Avg req ms |
  26 +|---|---|---:|---:|---:|
  27 +| high-batch/low-concurrency | batch=32, concurrency=1 | 81.26 | 458.67 | 393.78 |
  28 +| high-batch/low-concurrency | batch=64, concurrency=1 | 113.25 | 662.38 | 565.09 |
  29 +| high-batch/low-concurrency | batch=128, concurrency=1 | 142.43 | 959.47 | 898.64 |
  30 +| high-concurrency/low-batch | batch=1, concurrency=8 | 11.24 | 919.46 | 599.51 |
  31 +| high-concurrency/low-batch | batch=1, concurrency=16 | 10.34 | 1904.9 | 1124.8 |
  32 +| high-concurrency/low-batch | batch=1, concurrency=32 | 10.17 | 2876.88 | 1495.51 |
  33 +| high-concurrency/low-batch | batch=1, concurrency=64 | 10.32 | 2837.64 | 1442.59 |
  34 +
  35 +### Variant `ct2_tuned_t4`
  36 +
  37 +| Scenario | Setting | Items/s | Req p95 ms | Avg req ms |
  38 +|---|---|---:|---:|---:|
  39 +| high-batch/low-concurrency | batch=32, concurrency=1 | 79.44 | 464.07 | 402.81 |
  40 +| high-batch/low-concurrency | batch=64, concurrency=1 | 111.86 | 657.84 | 572.15 |
  41 +| high-batch/low-concurrency | batch=128, concurrency=1 | 137.56 | 994.04 | 930.45 |
  42 +| high-concurrency/low-batch | batch=1, concurrency=8 | 11.65 | 778.66 | 596.33 |
  43 +| high-concurrency/low-batch | batch=1, concurrency=16 | 12.91 | 1368.92 | 902.12 |
  44 +| high-concurrency/low-batch | batch=1, concurrency=32 | 12.8 | 2350.5 | 1237.45 |
  45 +| high-concurrency/low-batch | batch=1, concurrency=64 | 11.06 | 2729.36 | 1413.83 |
  46 +
  47 +## nllb-200-distilled-600m en->zh
  48 +
  49 +- Direction: `en -> zh`
  50 +- Column: `title`
  51 +
  52 +### Variant `ct2_default`
  53 +
  54 +| Scenario | Setting | Items/s | Req p95 ms | Avg req ms |
  55 +|---|---|---:|---:|---:|
  56 +| high-batch/low-concurrency | batch=32, concurrency=1 | 70.7 | 481.89 | 452.61 |
  57 +| high-batch/low-concurrency | batch=64, concurrency=1 | 96.27 | 701.75 | 664.81 |
  58 +| high-batch/low-concurrency | batch=128, concurrency=1 | 114.27 | 1137.6 | 1120.15 |
  59 +| high-concurrency/low-batch | batch=1, concurrency=8 | 5.54 | 1850.06 | 1287.52 |
  60 +| high-concurrency/low-batch | batch=1, concurrency=16 | 5.51 | 4613.05 | 2252.26 |
  61 +| high-concurrency/low-batch | batch=1, concurrency=32 | 5.46 | 5554.4 | 3022.86 |
  62 +| high-concurrency/low-batch | batch=1, concurrency=64 | 5.47 | 5514.61 | 3035.08 |
  63 +
  64 +### Variant `ct2_tuned_t4`
  65 +
  66 +| Scenario | Setting | Items/s | Req p95 ms | Avg req ms |
  67 +|---|---|---:|---:|---:|
  68 +| high-batch/low-concurrency | batch=32, concurrency=1 | 67.87 | 499.47 | 471.45 |
  69 +| high-batch/low-concurrency | batch=64, concurrency=1 | 93.36 | 721.79 | 685.53 |
  70 +| high-batch/low-concurrency | batch=128, concurrency=1 | 115.0 | 1126.53 | 1113.05 |
  71 +| high-concurrency/low-batch | batch=1, concurrency=8 | 7.9 | 1138.18 | 905.78 |
  72 +| high-concurrency/low-batch | batch=1, concurrency=16 | 7.91 | 2039.17 | 1555.46 |
  73 +| high-concurrency/low-batch | batch=1, concurrency=32 | 7.9 | 3912.75 | 2119.17 |
  74 +| high-concurrency/low-batch | batch=1, concurrency=64 | 6.61 | 4565.12 | 2434.92 |
  75 +
  76 +## opus-mt-zh-en zh->en
  77 +
  78 +- Direction: `zh -> en`
  79 +- Column: `title_cn`
  80 +
  81 +### Variant `ct2_default`
  82 +
  83 +| Scenario | Setting | Items/s | Req p95 ms | Avg req ms |
  84 +|---|---|---:|---:|---:|
  85 +| high-batch/low-concurrency | batch=32, concurrency=1 | 130.2 | 544.94 | 245.76 |
  86 +| high-batch/low-concurrency | batch=64, concurrency=1 | 164.1 | 597.69 | 389.99 |
  87 +| high-batch/low-concurrency | batch=128, concurrency=1 | 196.91 | 768.55 | 650.03 |
  88 +| high-concurrency/low-batch | batch=1, concurrency=8 | 29.4 | 324.88 | 230.83 |
  89 +| high-concurrency/low-batch | batch=1, concurrency=16 | 28.26 | 693.67 | 415.98 |
  90 +| high-concurrency/low-batch | batch=1, concurrency=32 | 27.5 | 1049.24 | 572.84 |
  91 +| high-concurrency/low-batch | batch=1, concurrency=64 | 28.43 | 994.92 | 520.92 |
  92 +
  93 +### Variant `ct2_tuned_t4`
  94 +
  95 +| Scenario | Setting | Items/s | Req p95 ms | Avg req ms |
  96 +|---|---|---:|---:|---:|
  97 +| high-batch/low-concurrency | batch=32, concurrency=1 | 120.2 | 582.58 | 266.21 |
  98 +| high-batch/low-concurrency | batch=64, concurrency=1 | 151.21 | 635.63 | 423.24 |
  99 +| high-batch/low-concurrency | batch=128, concurrency=1 | 196.63 | 761.85 | 650.95 |
  100 +| high-concurrency/low-batch | batch=1, concurrency=8 | 30.43 | 361.76 | 239.25 |
  101 +| high-concurrency/low-batch | batch=1, concurrency=16 | 28.32 | 629.6 | 423.42 |
  102 +| high-concurrency/low-batch | batch=1, concurrency=32 | 29.83 | 994.19 | 573.64 |
  103 +| high-concurrency/low-batch | batch=1, concurrency=64 | 26.85 | 1092.87 | 612.05 |
  104 +
  105 +## opus-mt-en-zh en->zh
  106 +
  107 +- Direction: `en -> zh`
  108 +- Column: `title`
  109 +
  110 +### Variant `ct2_default`
  111 +
  112 +| Scenario | Setting | Items/s | Req p95 ms | Avg req ms |
  113 +|---|---|---:|---:|---:|
  114 +| high-batch/low-concurrency | batch=32, concurrency=1 | 78.22 | 626.96 | 409.09 |
  115 +| high-batch/low-concurrency | batch=64, concurrency=1 | 114.34 | 699.88 | 559.7 |
  116 +| high-batch/low-concurrency | batch=128, concurrency=1 | 162.29 | 796.29 | 788.66 |
  117 +| high-concurrency/low-batch | batch=1, concurrency=8 | 11.25 | 798.77 | 489.06 |
  118 +| high-concurrency/low-batch | batch=1, concurrency=16 | 11.22 | 1759.07 | 978.67 |
  119 +| high-concurrency/low-batch | batch=1, concurrency=32 | 11.48 | 2453.59 | 1101.78 |
  120 +| high-concurrency/low-batch | batch=1, concurrency=64 | 11.35 | 2447.81 | 1116.99 |
  121 +
  122 +### Variant `ct2_tuned_t4`
  123 +
  124 +| Scenario | Setting | Items/s | Req p95 ms | Avg req ms |
  125 +|---|---|---:|---:|---:|
  126 +| high-batch/low-concurrency | batch=32, concurrency=1 | 79.33 | 620.07 | 403.35 |
  127 +| high-batch/low-concurrency | batch=64, concurrency=1 | 121.87 | 634.29 | 525.15 |
  128 +| high-batch/low-concurrency | batch=128, concurrency=1 | 210.29 | 623.45 | 608.65 |
  129 +| high-concurrency/low-batch | batch=1, concurrency=8 | 15.94 | 1199.98 | 334.6 |
  130 +| high-concurrency/low-batch | batch=1, concurrency=16 | 12.65 | 1683.16 | 744.33 |
  131 +| high-concurrency/low-batch | batch=1, concurrency=32 | 12.6 | 2435.98 | 974.56 |
  132 +| high-concurrency/low-batch | batch=1, concurrency=64 | 12.09 | 2433.63 | 960.19 |
... ...
scripts/benchmark_nllb_t4_tuning.py 0 → 100644
  Show/Hide comments   View file @46ce858
... ... @@ -0,0 +1,318 @@
  1 +#!/usr/bin/env python3
  2 +"""Focused NLLB T4 tuning benchmark for product-name translation."""
  3 +
  4 +from __future__ import annotations
  5 +
  6 +import argparse
  7 +import copy
  8 +import json
  9 +import sys
  10 +from datetime import datetime
  11 +from pathlib import Path
  12 +from typing import Any, Dict, List, Tuple
  13 +
  14 +PROJECT_ROOT = Path(__file__).resolve().parent.parent
  15 +if str(PROJECT_ROOT) not in sys.path:
  16 + sys.path.insert(0, str(PROJECT_ROOT))
  17 +
  18 +from config.services_config import get_translation_config
  19 +from scripts.benchmark_translation_local_models import (
  20 + benchmark_concurrency_case,
  21 + benchmark_serial_case,
  22 + build_environment_info,
  23 + ensure_cuda_stats_reset,
  24 + load_texts,
  25 +)
  26 +from translation.service import TranslationService
  27 +
  28 +
  29 +SCENARIOS = [
  30 + {
  31 + "name": "nllb zh->en",
  32 + "model": "nllb-200-distilled-600m",
  33 + "source_lang": "zh",
  34 + "target_lang": "en",
  35 + "column": "title_cn",
  36 + "scene": "sku_name",
  37 + },
  38 + {
  39 + "name": "nllb en->zh",
  40 + "model": "nllb-200-distilled-600m",
  41 + "source_lang": "en",
  42 + "target_lang": "zh",
  43 + "column": "title",
  44 + "scene": "sku_name",
  45 + },
  46 +]
  47 +
  48 +VARIANTS = [
  49 + {
  50 + "name": "ct2_default_fixed64",
  51 + "description": "Original CT2 default",
  52 + "overrides": {
  53 + "ct2_inter_threads": 1,
  54 + "ct2_max_queued_batches": 0,
  55 + "ct2_batch_type": "examples",
  56 + "max_new_tokens": 64,
  57 + },
  58 + },
  59 + {
  60 + "name": "ct2_prev_t4_fixed64",
  61 + "description": "Previous T4 tuning result",
  62 + "overrides": {
  63 + "ct2_inter_threads": 2,
  64 + "ct2_max_queued_batches": 16,
  65 + "ct2_batch_type": "examples",
  66 + "max_new_tokens": 64,
  67 + },
  68 + },
  69 + {
  70 + "name": "ct2_best_t4_dynamic",
  71 + "description": "Recommended T4 profile after this round",
  72 + "overrides": {
  73 + "ct2_inter_threads": 4,
  74 + "ct2_max_queued_batches": 32,
  75 + "ct2_batch_type": "examples",
  76 + "max_new_tokens": 64,
  77 + "ct2_decoding_length_mode": "source",
  78 + "ct2_decoding_length_extra": 8,
  79 + "ct2_decoding_length_min": 32,
  80 + },
  81 + },
  82 + {
  83 + "name": "ct2_fixed48_experiment",
  84 + "description": "High-gain experiment with truncation risk",
  85 + "overrides": {
  86 + "ct2_inter_threads": 3,
  87 + "ct2_max_queued_batches": 16,
  88 + "ct2_batch_type": "examples",
  89 + "max_new_tokens": 48,
  90 + },
  91 + },
  92 +]
  93 +
  94 +
  95 +def parse_args() -> argparse.Namespace:
  96 + parser = argparse.ArgumentParser(description="Focused NLLB T4 tuning benchmark")
  97 + parser.add_argument("--csv-path", default="products_analyzed.csv", help="Benchmark dataset CSV path")
  98 + parser.add_argument(
  99 + "--output-dir",
  100 + default="perf_reports/20260318/nllb_t4_product_names_ct2",
  101 + help="Directory for JSON/Markdown reports",
  102 + )
  103 + parser.add_argument("--batch-size", type=int, default=64, help="Batch size for the bulk scenario")
  104 + parser.add_argument("--batch-items", type=int, default=256, help="Rows used for the bulk scenario")
  105 + parser.add_argument("--concurrency", type=int, default=64, help="Concurrency for the online scenario")
  106 + parser.add_argument(
  107 + "--requests-per-case",
  108 + type=int,
  109 + default=24,
  110 + help="Requests per worker in the online scenario",
  111 + )
  112 + parser.add_argument("--quality-samples", type=int, default=100, help="Rows used for quality spot-checks")
  113 + parser.add_argument("--warmup-batches", type=int, default=1, help="Warmup batches before measuring")
  114 + return parser.parse_args()
  115 +
  116 +
  117 +def build_service(model: str, overrides: Dict[str, Any]) -> Tuple[TranslationService, Dict[str, Any]]:
  118 + config = copy.deepcopy(get_translation_config())
  119 + for name, cfg in config["capabilities"].items():
  120 + cfg["enabled"] = name == model
  121 + cfg["use_cache"] = False
  122 + config["default_model"] = model
  123 + capability = config["capabilities"][model]
  124 + capability.update(overrides)
  125 + return TranslationService(config), capability
  126 +
  127 +
  128 +def build_quality_reference_overrides(overrides: Dict[str, Any]) -> Dict[str, Any]:
  129 + reference = dict(overrides)
  130 + reference.pop("ct2_decoding_length_mode", None)
  131 + reference.pop("ct2_decoding_length_extra", None)
  132 + reference.pop("ct2_decoding_length_min", None)
  133 + reference["max_new_tokens"] = max(64, int(reference.get("max_new_tokens", 64)))
  134 + return reference
  135 +
  136 +
  137 +def summarize_quality(reference_outputs: List[Any], candidate_outputs: List[Any], texts: List[str]) -> Dict[str, Any]:
  138 + same = 0
  139 + diffs: List[Dict[str, str]] = []
  140 + for text, ref_output, candidate_output in zip(texts, reference_outputs, candidate_outputs):
  141 + if ref_output == candidate_output:
  142 + same += 1
  143 + continue
  144 + if len(diffs) < 3:
  145 + diffs.append(
  146 + {
  147 + "input": text,
  148 + "candidate": "" if candidate_output is None else str(candidate_output),
  149 + "reference": "" if ref_output is None else str(ref_output),
  150 + }
  151 + )
  152 + return {
  153 + "same": same,
  154 + "total": len(texts),
  155 + "changed": len(texts) - same,
  156 + "sample_diffs": diffs,
  157 + }
  158 +
  159 +
  160 +def render_markdown(report: Dict[str, Any]) -> str:
  161 + lines = [
  162 + "# NLLB T4 Product-Name Tuning",
  163 + "",
  164 + f"- Generated at: `{report['generated_at']}`",
  165 + f"- Python: `{report['environment']['python']}`",
  166 + f"- Torch: `{report['environment']['torch']}`",
  167 + f"- Transformers: `{report['environment']['transformers']}`",
  168 + f"- CUDA: `{report['environment']['cuda_available']}`",
  169 + ]
  170 + if report["environment"]["gpu_name"]:
  171 + lines.append(f"- GPU: `{report['environment']['gpu_name']}` ({report['environment']['gpu_total_mem_gb']} GiB)")
  172 + lines.extend(
  173 + [
  174 + "",
  175 + "## Scope",
  176 + "",
  177 + f"- Bulk scenario: `batch={report['config']['batch_size']}, concurrency=1`",
  178 + f"- Online scenario: `batch=1, concurrency={report['config']['concurrency']}`",
  179 + f"- Online requests per worker: `{report['config']['requests_per_case']}`",
  180 + f"- Quality spot-check samples: `{report['config']['quality_samples']}`",
  181 + "",
  182 + "## Variants",
  183 + "",
  184 + ]
  185 + )
  186 + for variant in report["variants"]:
  187 + lines.append(f"- `{variant['name']}`: {variant['description']} -> `{variant['overrides']}`")
  188 +
  189 + for scenario in report["scenarios"]:
  190 + lines.extend(
  191 + [
  192 + "",
  193 + f"## {scenario['name']}",
  194 + "",
  195 + "| Variant | Bulk items/s | Bulk p95 ms | Online items/s | Online p95 ms | Quality same/total |",
  196 + "|---|---:|---:|---:|---:|---:|",
  197 + ]
  198 + )
  199 + for variant in scenario["variants"]:
  200 + quality = variant["quality_vs_reference"]
  201 + lines.append(
  202 + f"| {variant['name']} | {variant['bulk']['items_per_second']} | {variant['bulk']['request_latency_p95_ms']} | "
  203 + f"{variant['online']['items_per_second']} | {variant['online']['request_latency_p95_ms']} | "
  204 + f"{quality['same']}/{quality['total']} |"
  205 + )
  206 + for variant in scenario["variants"]:
  207 + quality = variant["quality_vs_reference"]
  208 + if not quality["sample_diffs"]:
  209 + continue
  210 + lines.extend(
  211 + [
  212 + "",
  213 + f"### Quality Notes: {variant['name']}",
  214 + "",
  215 + ]
  216 + )
  217 + for diff in quality["sample_diffs"]:
  218 + lines.append(f"- Input: `{diff['input']}`")
  219 + lines.append(f"- Candidate: `{diff['candidate']}`")
  220 + lines.append(f"- Reference: `{diff['reference']}`")
  221 + lines.append("")
  222 +
  223 + return "\n".join(lines).rstrip() + "\n"
  224 +
  225 +
  226 +def main() -> None:
  227 + args = parse_args()
  228 + csv_path = (PROJECT_ROOT / args.csv_path).resolve() if not Path(args.csv_path).is_absolute() else Path(args.csv_path)
  229 + output_dir = (PROJECT_ROOT / args.output_dir).resolve() if not Path(args.output_dir).is_absolute() else Path(args.output_dir)
  230 + output_dir.mkdir(parents=True, exist_ok=True)
  231 +
  232 + report: Dict[str, Any] = {
  233 + "generated_at": datetime.now().isoformat(timespec="seconds"),
  234 + "environment": build_environment_info(),
  235 + "config": {
  236 + "csv_path": str(csv_path),
  237 + "batch_size": args.batch_size,
  238 + "batch_items": args.batch_items,
  239 + "concurrency": args.concurrency,
  240 + "requests_per_case": args.requests_per_case,
  241 + "quality_samples": args.quality_samples,
  242 + },
  243 + "variants": VARIANTS,
  244 + "scenarios": [],
  245 + }
  246 +
  247 + for scenario in SCENARIOS:
  248 + batch_texts = load_texts(csv_path, scenario["column"], args.batch_items)
  249 + online_texts = load_texts(csv_path, scenario["column"], args.concurrency * args.requests_per_case)
  250 + quality_texts = load_texts(csv_path, scenario["column"], args.quality_samples)
  251 +
  252 + scenario_report = dict(scenario)
  253 + scenario_report["variants"] = []
  254 + for variant in VARIANTS:
  255 + print(f"[start] {scenario['name']} | {variant['name']}", flush=True)
  256 + ensure_cuda_stats_reset()
  257 + service, capability = build_service(scenario["model"], variant["overrides"])
  258 + backend = service.get_backend(scenario["model"])
  259 + bulk = benchmark_serial_case(
  260 + service=service,
  261 + backend=backend,
  262 + scenario=scenario,
  263 + capability=capability,
  264 + texts=batch_texts,
  265 + batch_size=args.batch_size,
  266 + warmup_batches=args.warmup_batches,
  267 + )
  268 + online = benchmark_concurrency_case(
  269 + service=service,
  270 + backend=backend,
  271 + scenario=scenario,
  272 + capability=capability,
  273 + texts=online_texts,
  274 + batch_size=1,
  275 + concurrency=args.concurrency,
  276 + requests_per_case=args.requests_per_case,
  277 + warmup_batches=args.warmup_batches,
  278 + )
  279 + quality_reference_overrides = build_quality_reference_overrides(variant["overrides"])
  280 + reference_service, _ = build_service(scenario["model"], quality_reference_overrides)
  281 + reference_outputs = reference_service.translate(
  282 + quality_texts,
  283 + source_lang=scenario["source_lang"],
  284 + target_lang=scenario["target_lang"],
  285 + model=scenario["model"],
  286 + scene=scenario["scene"],
  287 + )
  288 + candidate_outputs = service.translate(
  289 + quality_texts,
  290 + source_lang=scenario["source_lang"],
  291 + target_lang=scenario["target_lang"],
  292 + model=scenario["model"],
  293 + scene=scenario["scene"],
  294 + )
  295 + scenario_report["variants"].append(
  296 + {
  297 + "name": variant["name"],
  298 + "description": variant["description"],
  299 + "overrides": variant["overrides"],
  300 + "quality_reference_overrides": quality_reference_overrides,
  301 + "bulk": bulk,
  302 + "online": online,
  303 + "quality_vs_reference": summarize_quality(reference_outputs, candidate_outputs, quality_texts),
  304 + }
  305 + )
  306 + report["scenarios"].append(scenario_report)
  307 +
  308 + timestamp = datetime.now().strftime("%H%M%S")
  309 + json_path = output_dir / f"nllb_t4_tuning_{timestamp}.json"
  310 + md_path = output_dir / f"nllb_t4_tuning_{timestamp}.md"
  311 + json_path.write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
  312 + md_path.write_text(render_markdown(report), encoding="utf-8")
  313 + print(f"JSON_REPORT={json_path}")
  314 + print(f"MARKDOWN_REPORT={md_path}")
  315 +
  316 +
  317 +if __name__ == "__main__":
  318 + main()
... ...
scripts/benchmark_translation_local_models.py
  Show/Hide comments   View file @46ce858
... ... @@ -87,6 +87,36 @@ def parse_args() -&gt; argparse.Namespace:
87 87 parser.add_argument("--max-new-tokens", type=int, default=0, help="Override configured max_new_tokens")
88 88 parser.add_argument("--num-beams", type=int, default=0, help="Override configured num_beams")
89 89 parser.add_argument("--attn-implementation", default="", help="Override attention implementation, for example sdpa")
  90 + parser.add_argument("--ct2-inter-threads", type=int, default=-1, help="Override CTranslate2 inter_threads")
  91 + parser.add_argument("--ct2-intra-threads", type=int, default=-1, help="Override CTranslate2 intra_threads")
  92 + parser.add_argument(
  93 + "--ct2-max-queued-batches",
  94 + type=int,
  95 + default=-1,
  96 + help="Override CTranslate2 max_queued_batches",
  97 + )
  98 + parser.add_argument(
  99 + "--ct2-batch-type",
  100 + default="",
  101 + help="Override CTranslate2 batch_type, for example examples or tokens",
  102 + )
  103 + parser.add_argument(
  104 + "--ct2-decoding-length-mode",
  105 + default="",
  106 + help="Override CTranslate2 decoding length mode, for example fixed or source",
  107 + )
  108 + parser.add_argument(
  109 + "--ct2-decoding-length-extra",
  110 + type=int,
  111 + default=0,
  112 + help="Extra tokens added when ct2 decoding length mode is source",
  113 + )
  114 + parser.add_argument(
  115 + "--ct2-decoding-length-min",
  116 + type=int,
  117 + default=0,
  118 + help="Minimum decoding length when ct2 decoding length mode is source",
  119 + )
90 120 parser.add_argument("--warmup-batches", type=int, default=1, help="Warmup batches before measuring")
91 121 parser.add_argument("--disable-cache", action="store_true", help="Disable translation cache during benchmarks")
92 122 parser.add_argument(
... ... @@ -245,6 +275,20 @@ def build_config_and_capability(
245 275 capability["num_beams"] = args.num_beams
246 276 if args.attn_implementation:
247 277 capability["attn_implementation"] = args.attn_implementation
  278 + if args.ct2_inter_threads >= 0:
  279 + capability["ct2_inter_threads"] = args.ct2_inter_threads
  280 + if args.ct2_intra_threads >= 0:
  281 + capability["ct2_intra_threads"] = args.ct2_intra_threads
  282 + if args.ct2_max_queued_batches >= 0:
  283 + capability["ct2_max_queued_batches"] = args.ct2_max_queued_batches
  284 + if args.ct2_batch_type:
  285 + capability["ct2_batch_type"] = args.ct2_batch_type
  286 + if args.ct2_decoding_length_mode:
  287 + capability["ct2_decoding_length_mode"] = args.ct2_decoding_length_mode
  288 + if args.ct2_decoding_length_extra:
  289 + capability["ct2_decoding_length_extra"] = args.ct2_decoding_length_extra
  290 + if args.ct2_decoding_length_min:
  291 + capability["ct2_decoding_length_min"] = args.ct2_decoding_length_min
248 292 if args.disable_cache:
249 293 capability["use_cache"] = False
250 294 config["capabilities"][args.model] = capability
... ... @@ -669,6 +713,20 @@ def run_all_scenarios(args: argparse.Namespace) -&gt; Dict[str, Any]:
669 713 cmd.extend(["--num-beams", str(args.num_beams)])
670 714 if args.attn_implementation:
671 715 cmd.extend(["--attn-implementation", args.attn_implementation])
  716 + if args.ct2_inter_threads >= 0:
  717 + cmd.extend(["--ct2-inter-threads", str(args.ct2_inter_threads)])
  718 + if args.ct2_intra_threads >= 0:
  719 + cmd.extend(["--ct2-intra-threads", str(args.ct2_intra_threads)])
  720 + if args.ct2_max_queued_batches >= 0:
  721 + cmd.extend(["--ct2-max-queued-batches", str(args.ct2_max_queued_batches)])
  722 + if args.ct2_batch_type:
  723 + cmd.extend(["--ct2-batch-type", args.ct2_batch_type])
  724 + if args.ct2_decoding_length_mode:
  725 + cmd.extend(["--ct2-decoding-length-mode", args.ct2_decoding_length_mode])
  726 + if args.ct2_decoding_length_extra:
  727 + cmd.extend(["--ct2-decoding-length-extra", str(args.ct2_decoding_length_extra)])
  728 + if args.ct2_decoding_length_min:
  729 + cmd.extend(["--ct2-decoding-length-min", str(args.ct2_decoding_length_min)])
672 730 if args.disable_cache:
673 731 cmd.append("--disable-cache")
674 732  
... ...
scripts/benchmark_translation_local_models_focus.py 0 → 100644
  Show/Hide comments   View file @46ce858
... ... @@ -0,0 +1,250 @@
  1 +#!/usr/bin/env python3
  2 +"""Focused translation benchmark for two stress scenarios on local CT2 models."""
  3 +
  4 +from __future__ import annotations
  5 +
  6 +import argparse
  7 +import copy
  8 +import json
  9 +import sys
  10 +from datetime import datetime
  11 +from pathlib import Path
  12 +from typing import Any, Dict, List
  13 +
  14 +PROJECT_ROOT = Path(__file__).resolve().parent.parent
  15 +if str(PROJECT_ROOT) not in sys.path:
  16 + sys.path.insert(0, str(PROJECT_ROOT))
  17 +
  18 +from config.services_config import get_translation_config
  19 +from scripts.benchmark_translation_local_models import (
  20 + SCENARIOS,
  21 + benchmark_concurrency_case,
  22 + benchmark_serial_case,
  23 + build_environment_info,
  24 + ensure_cuda_stats_reset,
  25 + load_texts,
  26 +)
  27 +from translation.service import TranslationService
  28 +
  29 +DEFAULT_HIGH_BATCH_SIZES = [32, 64, 128]
  30 +DEFAULT_HIGH_CONCURRENCIES = [8, 16, 32, 64]
  31 +
  32 +
  33 +def parse_args() -> argparse.Namespace:
  34 + parser = argparse.ArgumentParser(description="Focused benchmark for local CT2 translation models")
  35 + parser.add_argument("--csv-path", default="products_analyzed.csv", help="Benchmark dataset CSV path")
  36 + parser.add_argument(
  37 + "--output-dir",
  38 + default="perf_reports/20260318/translation_local_models_ct2_focus",
  39 + help="Directory for JSON/Markdown focused reports",
  40 + )
  41 + parser.add_argument(
  42 + "--high-batch-sizes",
  43 + default="32,64,128",
  44 + help="Comma-separated batch sizes for the high-batch/low-concurrency scenario",
  45 + )
  46 + parser.add_argument(
  47 + "--high-concurrencies",
  48 + default="8,16,32,64",
  49 + help="Comma-separated concurrency levels for the high-concurrency/low-batch scenario",
  50 + )
  51 + parser.add_argument(
  52 + "--high-batch-rows",
  53 + type=int,
  54 + default=512,
  55 + help="Rows used for the high-batch/low-concurrency scenario",
  56 + )
  57 + parser.add_argument(
  58 + "--high-concurrency-requests",
  59 + type=int,
  60 + default=32,
  61 + help="Requests per high-concurrency/low-batch case",
  62 + )
  63 + parser.add_argument("--warmup-batches", type=int, default=1, help="Warmup batches before measuring")
  64 + return parser.parse_args()
  65 +
  66 +
  67 +def parse_csv_ints(raw: str) -> List[int]:
  68 + values: List[int] = []
  69 + for item in raw.split(","):
  70 + stripped = item.strip()
  71 + if not stripped:
  72 + continue
  73 + value = int(stripped)
  74 + if value <= 0:
  75 + raise ValueError(f"Expected positive integer, got {value}")
  76 + values.append(value)
  77 + if not values:
  78 + raise ValueError("Parsed empty integer list")
  79 + return values
  80 +
  81 +
  82 +def build_variant_config(model: str, overrides: Dict[str, Any]) -> tuple[Dict[str, Any], Dict[str, Any]]:
  83 + config = copy.deepcopy(get_translation_config())
  84 + for name, cfg in config["capabilities"].items():
  85 + cfg["enabled"] = name == model
  86 + cfg["use_cache"] = False
  87 + config["default_model"] = model
  88 + capability = config["capabilities"][model]
  89 + capability.update(overrides)
  90 + config["capabilities"][model] = capability
  91 + return config, capability
  92 +
  93 +
  94 +def render_markdown(report: Dict[str, Any]) -> str:
  95 + lines = [
  96 + "# Local Translation Model Focused Benchmark",
  97 + "",
  98 + f"- Generated at: `{report['generated_at']}`",
  99 + f"- Python: `{report['environment']['python']}`",
  100 + f"- Torch: `{report['environment']['torch']}`",
  101 + f"- Transformers: `{report['environment']['transformers']}`",
  102 + f"- CUDA: `{report['environment']['cuda_available']}`",
  103 + ]
  104 + if report["environment"]["gpu_name"]:
  105 + lines.append(f"- GPU: `{report['environment']['gpu_name']}` ({report['environment']['gpu_total_mem_gb']} GiB)")
  106 + lines.extend(
  107 + [
  108 + "",
  109 + "## Scope",
  110 + "",
  111 + "- Scenario 1: high batch size + low concurrency",
  112 + "- Scenario 2: high concurrency + low batch size",
  113 + "- Variants in this report:",
  114 + ]
  115 + )
  116 + for variant in report["variants"]:
  117 + lines.append(f" - `{variant['name']}`: `{variant['overrides']}`")
  118 +
  119 + for scenario in report["scenarios"]:
  120 + lines.extend(
  121 + [
  122 + "",
  123 + f"## {scenario['name']}",
  124 + "",
  125 + f"- Direction: `{scenario['source_lang']} -> {scenario['target_lang']}`",
  126 + f"- Column: `{scenario['column']}`",
  127 + ]
  128 + )
  129 + for variant in scenario["variants"]:
  130 + lines.extend(
  131 + [
  132 + "",
  133 + f"### Variant `{variant['name']}`",
  134 + "",
  135 + "| Scenario | Setting | Items/s | Req p95 ms | Avg req ms |",
  136 + "|---|---|---:|---:|---:|",
  137 + ]
  138 + )
  139 + for row in variant["high_batch_low_concurrency"]:
  140 + lines.append(
  141 + f"| high-batch/low-concurrency | batch={row['batch_size']}, concurrency=1 | "
  142 + f"{row['items_per_second']} | {row['request_latency_p95_ms']} | {row['avg_request_latency_ms']} |"
  143 + )
  144 + for row in variant["high_concurrency_low_batch"]:
  145 + lines.append(
  146 + f"| high-concurrency/low-batch | batch=1, concurrency={row['concurrency']} | "
  147 + f"{row['items_per_second']} | {row['request_latency_p95_ms']} | {row['avg_request_latency_ms']} |"
  148 + )
  149 + return "\n".join(lines) + "\n"
  150 +
  151 +
  152 +def main() -> None:
  153 + args = parse_args()
  154 + csv_path = (PROJECT_ROOT / args.csv_path).resolve() if not Path(args.csv_path).is_absolute() else Path(args.csv_path)
  155 + output_dir = (PROJECT_ROOT / args.output_dir).resolve() if not Path(args.output_dir).is_absolute() else Path(args.output_dir)
  156 + output_dir.mkdir(parents=True, exist_ok=True)
  157 +
  158 + high_batch_sizes = parse_csv_ints(args.high_batch_sizes)
  159 + high_concurrencies = parse_csv_ints(args.high_concurrencies)
  160 +
  161 + variants = [
  162 + {"name": "ct2_default", "overrides": {}},
  163 + {
  164 + "name": "ct2_tuned_t4",
  165 + "overrides": {
  166 + "ct2_inter_threads": 2,
  167 + "ct2_max_queued_batches": 16,
  168 + "ct2_batch_type": "examples",
  169 + },
  170 + },
  171 + ]
  172 +
  173 + report: Dict[str, Any] = {
  174 + "generated_at": datetime.now().isoformat(timespec="seconds"),
  175 + "environment": build_environment_info(),
  176 + "csv_path": str(csv_path),
  177 + "variants": variants,
  178 + "scenarios": [],
  179 + }
  180 +
  181 + largest_batch = max(high_batch_sizes)
  182 + high_batch_rows = max(args.high_batch_rows, largest_batch)
  183 +
  184 + for scenario in SCENARIOS:
  185 + scenario_entry = dict(scenario)
  186 + scenario_entry["variants"] = []
  187 + batch_texts = load_texts(csv_path, scenario["column"], high_batch_rows)
  188 + conc_needed = max(high_concurrencies) * args.high_concurrency_requests
  189 + conc_texts = load_texts(csv_path, scenario["column"], conc_needed)
  190 +
  191 + for variant in variants:
  192 + print(f"[start] {scenario['name']} | {variant['name']}", flush=True)
  193 + config, capability = build_variant_config(scenario["model"], variant["overrides"])
  194 + ensure_cuda_stats_reset()
  195 + service = TranslationService(config)
  196 + backend = service.get_backend(scenario["model"])
  197 +
  198 + high_batch_results = []
  199 + for batch_size in high_batch_sizes:
  200 + high_batch_results.append(
  201 + benchmark_serial_case(
  202 + service=service,
  203 + backend=backend,
  204 + scenario=scenario,
  205 + capability=capability,
  206 + texts=batch_texts[: max(batch_size, high_batch_rows)],
  207 + batch_size=batch_size,
  208 + warmup_batches=args.warmup_batches,
  209 + )
  210 + )
  211 +
  212 + high_concurrency_results = []
  213 + for concurrency in high_concurrencies:
  214 + high_concurrency_results.append(
  215 + benchmark_concurrency_case(
  216 + service=service,
  217 + backend=backend,
  218 + scenario=scenario,
  219 + capability=capability,
  220 + texts=conc_texts,
  221 + batch_size=1,
  222 + concurrency=concurrency,
  223 + requests_per_case=args.high_concurrency_requests,
  224 + warmup_batches=args.warmup_batches,
  225 + )
  226 + )
  227 +
  228 + scenario_entry["variants"].append(
  229 + {
  230 + "name": variant["name"],
  231 + "overrides": variant["overrides"],
  232 + "high_batch_low_concurrency": high_batch_results,
  233 + "high_concurrency_low_batch": high_concurrency_results,
  234 + }
  235 + )
  236 + print(f"[done] {scenario['name']} | {variant['name']}", flush=True)
  237 +
  238 + report["scenarios"].append(scenario_entry)
  239 +
  240 + stamp = datetime.now().strftime("%H%M%S")
  241 + json_path = output_dir / f"translation_local_models_focus_{stamp}.json"
  242 + md_path = output_dir / f"translation_local_models_focus_{stamp}.md"
  243 + json_path.write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
  244 + md_path.write_text(render_markdown(report), encoding="utf-8")
  245 + print(f"JSON report: {json_path}")
  246 + print(f"Markdown report: {md_path}")
  247 +
  248 +
  249 +if __name__ == "__main__":
  250 + main()
... ...
translation/README.md
  Show/Hide comments   View file @46ce858
... ... @@ -13,7 +13,11 @@
13 13 - 虚拟环境：[`scripts/setup_translator_venv.sh`](/data/saas-search/scripts/setup_translator_venv.sh)
14 14 - 模型下载：[`scripts/download_translation_models.py`](/data/saas-search/scripts/download_translation_models.py)
15 15 - 本地模型压测：[`scripts/benchmark_translation_local_models.py`](/data/saas-search/scripts/benchmark_translation_local_models.py)
16   -- 性能报告：[`perf_reports/20260318/translation_local_models/README.md`](/data/saas-search/perf_reports/20260318/translation_local_models/README.md)
  16 +- 聚焦压测脚本：[`scripts/benchmark_translation_local_models_focus.py`](/data/saas-search/scripts/benchmark_translation_local_models_focus.py)
  17 +- 基线性能报告：[`perf_reports/20260318/translation_local_models/README.md`](/data/saas-search/perf_reports/20260318/translation_local_models/README.md)
  18 +- CT2 扩展报告：[`perf_reports/20260318/translation_local_models_ct2/README.md`](/data/saas-search/perf_reports/20260318/translation_local_models_ct2/README.md)
  19 +- CT2 聚焦调优报告：[`perf_reports/20260318/translation_local_models_ct2_focus/README.md`](/data/saas-search/perf_reports/20260318/translation_local_models_ct2_focus/README.md)
  20 +- NLLB T4 商品标题专项报告：[`perf_reports/20260318/nllb_t4_product_names_ct2/README.md`](/data/saas-search/perf_reports/20260318/nllb_t4_product_names_ct2/README.md)
17 21  
18 22 ## 1. 设计目标
19 23  
... ... @@ -107,6 +111,12 @@ services:
107 111 ct2_compute_type: "float16"
108 112 ct2_conversion_quantization: "float16"
109 113 ct2_auto_convert: true
  114 + ct2_inter_threads: 4
  115 + ct2_max_queued_batches: 32
  116 + ct2_batch_type: "examples"
  117 + ct2_decoding_length_mode: "source"
  118 + ct2_decoding_length_extra: 8
  119 + ct2_decoding_length_min: 32
110 120 device: "cuda"
111 121 torch_dtype: "float16"
112 122 batch_size: 16
... ... @@ -408,7 +418,7 @@ results = translator.translate(
408 418 - 支持多语
409 419 - 调用时必须显式传 `source_lang`
410 420 - 语言码映射定义在 [`translation/languages.py`](/data/saas-search/translation/languages.py)
411   -- 当前 T4 推荐配置：`device=cuda`、`ct2_compute_type=float16`、`batch_size=16`、`max_new_tokens=64`
  421 +- 当前 T4 推荐配置：`device=cuda`、`ct2_compute_type=float16`、`ct2_inter_threads=4`、`ct2_max_queued_batches=32`、`ct2_batch_type=examples`、`ct2_decoding_length_mode=source(+8,min=32)`、`batch_size=16`、`max_new_tokens=64`
412 422  
413 423 当前实现已经利用的优化：
414 424 - 已做批量分块：`translate()` 会按 capability 的 `batch_size` 分批进入模型
... ... @@ -509,7 +519,7 @@ models/translation/Helsinki-NLP/opus-mt-en-zh
509 519 - 避免多 worker 重复加载模型
510 520 - GPU 机器上优先使用 `cuda + float16`
511 521 - CPU 只建议用于功能验证或离线低频任务
512   -- 对 NLLB，T4 上优先采用 `batch_size=16 + max_new_tokens=64 + ct2_compute_type=float16`
  522 +- 对 NLLB，T4 上优先采用 `batch_size=16 + max_new_tokens=64 + ct2_compute_type=float16 + ct2_inter_threads=4 + ct2_max_queued_batches=32 + ct2_batch_type=examples + ct2_decoding_length_mode=source(+8,min=32)`
513 523  
514 524 ### 9.5 验证
515 525  
... ... @@ -541,6 +551,7 @@ curl -X POST http://127.0.0.1:6006/translate \
541 551  
542 552 性能脚本：
543 553 - [`scripts/benchmark_translation_local_models.py`](/data/saas-search/scripts/benchmark_translation_local_models.py)
  554 +- [`scripts/benchmark_translation_local_models_focus.py`](/data/saas-search/scripts/benchmark_translation_local_models_focus.py)
544 555  
545 556 数据集：
546 557 - [`products_analyzed.csv`](/data/saas-search/products_analyzed.csv)
... ... @@ -549,6 +560,9 @@ curl -X POST http://127.0.0.1:6006/translate \
549 560 - 摘要：[`perf_reports/20260318/translation_local_models/README.md`](/data/saas-search/perf_reports/20260318/translation_local_models/README.md)
550 561 - 完整 Markdown：[`perf_reports/20260318/translation_local_models/translation_local_models_extended_221846.md`](/data/saas-search/perf_reports/20260318/translation_local_models/translation_local_models_extended_221846.md)
551 562 - 完整 JSON：[`perf_reports/20260318/translation_local_models/translation_local_models_extended_221846.json`](/data/saas-search/perf_reports/20260318/translation_local_models/translation_local_models_extended_221846.json)
  563 +- CT2 扩展总结：[`perf_reports/20260318/translation_local_models_ct2/README.md`](/data/saas-search/perf_reports/20260318/translation_local_models_ct2/README.md)
  564 +- CT2 聚焦调优总结：[`perf_reports/20260318/translation_local_models_ct2_focus/README.md`](/data/saas-search/perf_reports/20260318/translation_local_models_ct2_focus/README.md)
  565 +- NLLB T4 商品标题专项调优：[`perf_reports/20260318/nllb_t4_product_names_ct2/README.md`](/data/saas-search/perf_reports/20260318/nllb_t4_product_names_ct2/README.md)
552 566  
553 567 ### 10.1 先看哪组数据
554 568  
... ... @@ -768,33 +782,43 @@ cd /data/saas-search
768 782 NLLB 性能优化经验：
769 783  
770 784 - 起作用的优化点 1：`float16 + cuda`
771   - - 模型确认以 `torch.float16` 实际加载到 `cuda:0`
  785 + - 当前本地 NLLB 由 `CTranslate2` 在 `cuda:0` 以 `float16` 运行
772 786 - 优化后在 T4 上的峰值显存约 `2.8-3.0 GiB`
773 787 - 起作用的优化点 2：`batch_size=16`
774 788 - 相比 `batch_size=8`，吞吐提升明显
775 789 - 继续提升到 `32` 虽然还能增吞吐，但 batch p95 和 batch max 会恶化很多
776   -- 起作用的优化点 3：`max_new_tokens=64`
777   - - 商品标题翻译通常不需要 `256` 的生成上限
778   - - 收紧生成长度后，`zh->en` 与 `en->zh` 都有明显收益
779   -- 起作用的优化点 4：`attn_implementation=sdpa`
780   - - 对当前 PyTorch + T4 环境有效
781   - - 配合半精度和较合理 batch size 后，整体延迟进一步下降
782   -- 已有但不需要单独开关的点：`attention_mask`
783   - - 当前实现会在 tokenizer 阶段自动生成并传入 `generate()`
784   - - 它属于标准推理路径，不是一个额外的“高级优化开关”
  790 +- 起作用的优化点 3：`ct2_inter_threads=4 + ct2_max_queued_batches=32`
  791 + - 对 `batch=1` 高并发商品标题场景收益最直接
  792 + - 相比默认 CT2 配置，`zh->en` 和 `en->zh` 的在线吞吐都能稳定提升
  793 +- 起作用的优化点 4：动态解码上限
  794 + - 推荐 `ct2_decoding_length_mode=source`
  795 + - 推荐 `ct2_decoding_length_extra=8`
  796 + - 推荐 `ct2_decoding_length_min=32`
  797 + - 这样可以保留 `max_new_tokens=64` 的安全上限，同时让短标题不再为长标题上限付费
  798 +- 起作用的优化点 5：`ct2_batch_type=examples`
  799 + - 在当前数据和 T4 上，比 `tokens` 更稳
  800 + - 更适合作为线上默认
  801 +- 不建议直接作为默认的实验：
  802 + - `max_new_tokens=48`
  803 + - 大 batch 和在线吞吐都会继续变好
  804 + - 但商品标题 spot-check 已看到明显截断，尤其 `en->zh`
  805 +- 收益有限或不稳定的实验：
  806 + - `ct2_batch_type=tokens`
  807 + - `ct2_max_queued_batches` 从 `16` 再继续拉高，收益很小
  808 + - `ct2_decoding_length_mode=source(+4,min=24)` 更快，但仍有少量长标题截断风险
785 809  
786 810 为什么最终没有采用其它方案：
787 811  
788   -- 当前 HF 原生方案已经能在 T4 上稳定跑通
789   -- 在 `10G+` 可用显存下，原生 `float16` 已足够支撑 NLLB-600M
790   -- 因此暂时不需要为这个模型额外引入 GGUF 或 CT2 的新运行栈
791   -- 如果未来目标变成“继续压缩显存”或“进一步追求更低延迟”，再评估 `ct2-int8` 会更合适
  812 +- 当前本地最优路径已经切到 `CTranslate2 + float16`
  813 +- 对这个 600M 级 encoder-decoder 模型，T4 上最有效的是把 CT2 的并行和解码策略调对
  814 +- 因此这轮没有继续引入更重的服务化栈
  815 +- 如果未来目标变成“继续压缩显存”或“进一步追求更低延迟”，再评估 `int8_float16` 或服务级微批处理队列会更合适
792 816  
793 817 关键结论：
794 818  
795 819 - 当前机器上，`opus-mt-zh-en` 是三个新增本地模型里最快的
796 820 - `opus-mt-en-zh` 大约是 `opus-mt-zh-en` 吞吐的一半
797   -- `nllb-200-distilled-600M` 在显存充足时可以用 `cuda + float16 + batch_size=16 + max_new_tokens=64 + sdpa` 正常运行
  821 +- `nllb-200-distilled-600M` 在 T4 上推荐 `cuda + CTranslate2 float16 + batch_size=16 + ct2_inter_threads=4 + ct2_max_queued_batches=32 + dynamic decoding`
798 822 - `nllb` 最终可用，但吞吐仍明显低于两个 Marian 模型，更适合多语覆盖或独立资源环境
799 823  
800 824 最终推荐部署方案：
... ... @@ -807,11 +831,15 @@ NLLB 性能优化经验：
807 831 - 推荐 `max_input_length`：`256`
808 832 - 推荐 `max_new_tokens`：`64`
809 833 - 推荐 `num_beams`：`1`
810   -- 推荐注意力实现：`sdpa`
  834 +- 推荐 CT2 并行：`ct2_inter_threads=4`
  835 +- 推荐 CT2 队列：`ct2_max_queued_batches=32`
  836 +- 推荐 CT2 batch 类型：`examples`
  837 +- 推荐动态解码：`ct2_decoding_length_mode=source`、`ct2_decoding_length_extra=8`、`ct2_decoding_length_min=32`
811 838 - 运行方式：单 worker，避免重复加载
812 839  
813 840 更详细的性能说明见：
814 841 - [`perf_reports/20260318/translation_local_models/README.md`](/data/saas-search/perf_reports/20260318/translation_local_models/README.md)
  842 +- [`perf_reports/20260318/nllb_t4_product_names_ct2/README.md`](/data/saas-search/perf_reports/20260318/nllb_t4_product_names_ct2/README.md)
815 843  
816 844 ## 11. 开发说明
817 845  
... ...
translation/backends/local_ctranslate2.py
  Show/Hide comments   View file @46ce858
... ... @@ -91,6 +91,9 @@ class LocalCTranslate2TranslationBackend:
91 91 ct2_intra_threads: int = 0,
92 92 ct2_max_queued_batches: int = 0,
93 93 ct2_batch_type: str = "examples",
  94 + ct2_decoding_length_mode: str = "fixed",
  95 + ct2_decoding_length_extra: int = 0,
  96 + ct2_decoding_length_min: int = 1,
94 97 ) -> None:
95 98 self.model = name
96 99 self.model_id = model_id
... ... @@ -114,6 +117,11 @@ class LocalCTranslate2TranslationBackend:
114 117 self.ct2_batch_type = str(ct2_batch_type or "examples").strip().lower()
115 118 if self.ct2_batch_type not in {"examples", "tokens"}:
116 119 raise ValueError(f"Unsupported CTranslate2 batch type: {ct2_batch_type}")
  120 + self.ct2_decoding_length_mode = str(ct2_decoding_length_mode or "fixed").strip().lower()
  121 + if self.ct2_decoding_length_mode not in {"fixed", "source"}:
  122 + raise ValueError(f"Unsupported CTranslate2 decoding length mode: {ct2_decoding_length_mode}")
  123 + self.ct2_decoding_length_extra = int(ct2_decoding_length_extra)
  124 + self.ct2_decoding_length_min = max(1, int(ct2_decoding_length_min))
117 125 self._tokenizer_lock = threading.Lock()
118 126 self._load_runtime()
119 127  
... ... @@ -239,6 +247,15 @@ class LocalCTranslate2TranslationBackend:
239 247 del count, source_lang, target_lang
240 248 return None
241 249  
  250 + def _resolve_max_decoding_length(self, source_tokens: Sequence[Sequence[str]]) -> int:
  251 + if self.ct2_decoding_length_mode != "source":
  252 + return self.max_new_tokens
  253 + if not source_tokens:
  254 + return self.max_new_tokens
  255 + max_source_length = max(len(tokens) for tokens in source_tokens)
  256 + dynamic_length = max(self.ct2_decoding_length_min, max_source_length + self.ct2_decoding_length_extra)
  257 + return min(self.max_new_tokens, dynamic_length)
  258 +
242 259 def _postprocess_hypothesis(
243 260 self,
244 261 tokens: List[str],
... ... @@ -262,6 +279,7 @@ class LocalCTranslate2TranslationBackend:
262 279 self._validate_languages(source_lang, target_lang)
263 280 source_tokens = self._encode_source_tokens(texts, source_lang, target_lang)
264 281 target_prefix = self._target_prefixes(len(source_tokens), source_lang, target_lang)
  282 + max_decoding_length = self._resolve_max_decoding_length(source_tokens)
265 283 results = self.translator.translate_batch(
266 284 source_tokens,
267 285 target_prefix=target_prefix,
... ... @@ -269,7 +287,7 @@ class LocalCTranslate2TranslationBackend:
269 287 batch_type=self.ct2_batch_type,
270 288 beam_size=self.num_beams,
271 289 max_input_length=self.max_input_length,
272   - max_decoding_length=self.max_new_tokens,
  290 + max_decoding_length=max_decoding_length,
273 291 )
274 292 outputs: List[Optional[str]] = []
275 293 for result in results:
... ... @@ -323,6 +341,9 @@ class MarianCTranslate2TranslationBackend(LocalCTranslate2TranslationBackend):
323 341 ct2_intra_threads: int = 0,
324 342 ct2_max_queued_batches: int = 0,
325 343 ct2_batch_type: str = "examples",
  344 + ct2_decoding_length_mode: str = "fixed",
  345 + ct2_decoding_length_extra: int = 0,
  346 + ct2_decoding_length_min: int = 1,
326 347 ) -> None:
327 348 self.source_langs = {str(lang).strip().lower() for lang in source_langs if str(lang).strip()}
328 349 self.target_langs = {str(lang).strip().lower() for lang in target_langs if str(lang).strip()}
... ... @@ -344,6 +365,9 @@ class MarianCTranslate2TranslationBackend(LocalCTranslate2TranslationBackend):
344 365 ct2_intra_threads=ct2_intra_threads,
345 366 ct2_max_queued_batches=ct2_max_queued_batches,
346 367 ct2_batch_type=ct2_batch_type,
  368 + ct2_decoding_length_mode=ct2_decoding_length_mode,
  369 + ct2_decoding_length_extra=ct2_decoding_length_extra,
  370 + ct2_decoding_length_min=ct2_decoding_length_min,
347 371 )
348 372  
349 373 def _validate_languages(self, source_lang: Optional[str], target_lang: str) -> None:
... ... @@ -383,6 +407,9 @@ class NLLBCTranslate2TranslationBackend(LocalCTranslate2TranslationBackend):
383 407 ct2_intra_threads: int = 0,
384 408 ct2_max_queued_batches: int = 0,
385 409 ct2_batch_type: str = "examples",
  410 + ct2_decoding_length_mode: str = "fixed",
  411 + ct2_decoding_length_extra: int = 0,
  412 + ct2_decoding_length_min: int = 1,
386 413 ) -> None:
387 414 overrides = language_codes or {}
388 415 self.language_codes = {
... ... @@ -408,6 +435,9 @@ class NLLBCTranslate2TranslationBackend(LocalCTranslate2TranslationBackend):
408 435 ct2_intra_threads=ct2_intra_threads,
409 436 ct2_max_queued_batches=ct2_max_queued_batches,
410 437 ct2_batch_type=ct2_batch_type,
  438 + ct2_decoding_length_mode=ct2_decoding_length_mode,
  439 + ct2_decoding_length_extra=ct2_decoding_length_extra,
  440 + ct2_decoding_length_min=ct2_decoding_length_min,
411 441 )
412 442  
413 443 def _validate_languages(self, source_lang: Optional[str], target_lang: str) -> None:
... ...
translation/service.py
  Show/Hide comments   View file @46ce858
... ... @@ -131,6 +131,9 @@ class TranslationService:
131 131 ct2_intra_threads=int(cfg.get("ct2_intra_threads", 0)),
132 132 ct2_max_queued_batches=int(cfg.get("ct2_max_queued_batches", 0)),
133 133 ct2_batch_type=str(cfg.get("ct2_batch_type", "examples")),
  134 + ct2_decoding_length_mode=str(cfg.get("ct2_decoding_length_mode", "fixed")),
  135 + ct2_decoding_length_extra=int(cfg.get("ct2_decoding_length_extra", 0)),
  136 + ct2_decoding_length_min=int(cfg.get("ct2_decoding_length_min", 1)),
134 137 )
135 138  
136 139 def _create_local_marian_backend(self, *, name: str, cfg: Dict[str, object]) -> TranslationBackendProtocol:
... ... @@ -158,6 +161,9 @@ class TranslationService:
158 161 ct2_intra_threads=int(cfg.get("ct2_intra_threads", 0)),
159 162 ct2_max_queued_batches=int(cfg.get("ct2_max_queued_batches", 0)),
160 163 ct2_batch_type=str(cfg.get("ct2_batch_type", "examples")),
  164 + ct2_decoding_length_mode=str(cfg.get("ct2_decoding_length_mode", "fixed")),
  165 + ct2_decoding_length_extra=int(cfg.get("ct2_decoding_length_extra", 0)),
  166 + ct2_decoding_length_min=int(cfg.get("ct2_decoding_length_min", 1)),
161 167 )
162 168  
163 169 @property
... ...
translation/settings.py
  Show/Hide comments   View file @46ce858
... ... @@ -149,6 +149,14 @@ def _validate_capability(name: str, capability: Mapping[str, Any]) -&gt; None:
149 149 _require_positive_int(capability.get("max_input_length"), f"{prefix}.max_input_length")
150 150 _require_positive_int(capability.get("max_new_tokens"), f"{prefix}.max_new_tokens")
151 151 _require_positive_int(capability.get("num_beams"), f"{prefix}.num_beams")
  152 + if "ct2_decoding_length_mode" in capability:
  153 + mode = _require_string(capability.get("ct2_decoding_length_mode"), f"{prefix}.ct2_decoding_length_mode").lower()
  154 + if mode not in {"fixed", "source"}:
  155 + raise ValueError(f"{prefix}.ct2_decoding_length_mode must be one of: fixed, source")
  156 + if "ct2_decoding_length_extra" in capability:
  157 + _require_int(capability.get("ct2_decoding_length_extra"), f"{prefix}.ct2_decoding_length_extra")
  158 + if "ct2_decoding_length_min" in capability:
  159 + _require_positive_int(capability.get("ct2_decoding_length_min"), f"{prefix}.ct2_decoding_length_min")
152 160 return
153 161  
154 162 raise ValueError(f"Unsupported translation backend '{backend}' for capability '{name}'")
... ...