Reranker benchmark: qwen3_vllm vs qwen3_vllm_score × instruction_format

Date: 2026-03-25
Host: single GPU (Tesla T4, ~16 GiB), CUDA 12.8 (see nvidia-smi during run).

Configuration (from config/config.yaml)

Shared across both backends for this run:

Key	Value
model_name	Qwen/Qwen3-Reranker-0.6B
max_model_len	160
infer_batch_size	100
sort_by_doc_length	true
enable_prefix_caching	true
enforce_eager	false
dtype	float16
tensor_parallel_size	1
gpu_memory_utilization	0.20
instruction	Rank products by query with category & style match prioritized

qwen3_vllm uses vLLM generate + logprobs (.venv-reranker).
qwen3_vllm_score uses vLLM LLM.score() (.venv-reranker-score, pinned vLLM stack per reranker/README.md).

Methodology

• Script: python scripts/benchmark_reranker_random_titles.py 100,200,400,600,800,1000 --repeat 5 with --seed 99 (see note below), --quiet-runs, --timeout 360.
• Titles: default file /home/ubuntu/rerank_test/titles.1.8w (one title per line).
• Query: default 健身女生T恤短袖.
• Each scenario: 3 warm-up requests at n=400 (not timed), then 5 timed runs per n.
• Metric: client wall time for POST /rerank (localhost), milliseconds.
• After each services.rerank.backend / instruction_format change: ./restart.sh reranker, then GET /health until backend and instruction_format matched the intended scenario (extended reranker/server.py to expose instruction_format when the backend defines _instruction_format).

Note on RNG seed: With --seed 42, some runs occasionally lost one sample at n=600 (non-200 or transport error). All figures below use --seed 99 so every cell has 5/5 successful runs and comparable sampled titles.

Raw artifacts

JSON aggregates (means, stdev, raw values_ms): same directory, qwen3_vllm_{compact,standard}.json, qwen3_vllm_score_{compact,standard}.json.

Results — mean latency (ms)

backend	instruction_format	n=100	n=200	n=400	n=600	n=800	n=1000
qwen3_vllm	compact	213.5	418.0	861.4	1263.4	1744.3	2162.2
qwen3_vllm	standard	254.9	475.4	909.7	1353.2	1912.5	2406.7
qwen3_vllm_score	compact	239.2	480.2	966.2	1433.5	1937.2	2428.4
qwen3_vllm_score	standard	299.6	591.8	1178.9	1773.7	2341.6	2931.7

Short interpretation

1. compact vs standard: For both backends, compact is faster on this setup (shorter / different chat template vs fixed yes/no system prompt + user block — see reranker/backends/qwen3_vllm.py / qwen3_vllm_score.py).
2. qwen3_vllm vs qwen3_vllm_score: At n=1000, qwen3_vllm + compact is the fastest row (~2162 ms mean); qwen3_vllm_score + standard is the slowest (~2932 ms). Ordering can change on other GPUs / vLLM versions / batching.
3. Repo default after tests: services.rerank.backend: qwen3_vllm_score, instruction_format: compact on both qwen3_vllm and qwen3_vllm_score blocks (patch script keeps them aligned).

Tooling added / changed

• reranker/server.py: /health includes instruction_format when the active backend sets _instruction_format.
• scripts/benchmark_reranker_random_titles.py: --tag, --json-summary-out, --quiet-runs.
• scripts/patch_rerank_vllm_benchmark_config.py: surgical YAML patch (preserves newlines).
• scripts/run_reranker_vllm_instruction_benchmark.sh: full matrix driver (continues if a benchmark exits non-zero; uses --timeout 360).