Blame view

scripts/evaluation/eval_framework/prompts.py 20.3 KB
c81b0fc1   tangwang   scripts/evaluatio...
1
2
3
4
  """LLM prompt templates for relevance judging (keep wording changes here)."""
  
  from __future__ import annotations
  
a345b01f   tangwang   eval framework
5
  from typing import Sequence
c81b0fc1   tangwang   scripts/evaluatio...
6
  
cdd8ee3a   tangwang   eval框架日志独立
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
  
  _QUERY_INTENT_ANALYSIS_TEMPLATE_EN = """You are an intent analysis expert for a fashion e-commerce search system.
  
  Given a user's search query, analyze the shopping intent behind the query in the context of fashion and apparel e-commerce, and summarize the user's core search need in one concise sentence.
  Also provide the Chinese translation and English translation of the query.
  
  Requirements:
  - Keep the intent analysis concise and easy to understand, using 1 to 3 short sentences.
  - Stay grounded in the original query and summarize the user's likely shopping intent without adding unnecessary context.
  - When the query is vague or ambiguous, take a conservative approach and keep the analysis close to the original wording.
  - Chinese translation: if the original query is already in Chinese, keep it unchanged.
  - English translation: if the original query is already in English, keep it unchanged.
  - Do not output anything other than the required three-line format.
  
  Output format (strictly exactly three lines):
  Intent: concise analysis of the user's search intent
  Query中文翻译: Chinese translation of the query
  Query English translation: English translation of the query
  
  Now analyze the following query:
  
  Query: {query}
  """
  
  _QUERY_INTENT_ANALYSIS_RESULT_TEMPLATE_ZH = """
  你是一个服装品类电商搜索意图分析专家。
  
  给定用户输入的搜索词,请在服装品类电商场景下,分析该搜索词背后的购物意图,并用一句话简要描述用户的核心搜索需求。
  同时,提供该搜索词的中文翻译和英文翻译。
  
  要求:
  - 意图分析应简洁易懂,用 1  3 句短句概括用户的搜索意图。
  - 结合 query 本身,尽量贴近用户原始搜索需求进行总结,不添加不必要的背景、延伸或臆测。
  - 如果 query 不够明确或有歧义,应保守处理,尽量保持与原词表达一致。
  - 中文翻译:如果原始 query 本身就是中文,则按原样输出。
  - 英文翻译:如果原始 query 本身就是英文,则按原样输出。
  - 除指定格式外,不要输出任何额外说明。
  
  输出格式(严格按三行输出):
  Intent: 对用户搜索意图的简洁分析
  Query中文翻译: query 的中文翻译
  Query English translation: query 的英文翻译
  
  现在请分析以下搜索词:
  
  Query: {query}
  """
  
a345b01f   tangwang   eval framework
55
  _CLASSIFY_TEMPLATE_EN = """You are a relevance judgment assistant for a fashion e-commerce search system.
46d94a05   tangwang   评估标准修改
56
57
58
59
  Given a user query and the information for each product, assign a relevance label to each product.
  
  Your goal is to judge relevance from the perspective of e-commerce search ranking.
  The key question is whether the user would view the product as the intended item, or as an acceptable substitute.
3984ec64   tangwang   evalution 标注标准优化
60
61
62
  
  ## Relevance Labels
  
46d94a05   tangwang   评估标准修改
63
64
  ### Exact Match
  The product satisfies the users core shopping intent: the core product type matches, and all explicitly stated key attributes in the query are supported by the product information, with no obvious conflict.
3984ec64   tangwang   evalution 标注标准优化
65
  
3ac1f8d1   tangwang   评估标准优化
66
67
  Typical use cases:
  - The query contains only a product type, and the product is exactly that type.
46d94a05   tangwang   评估标准修改
68
  - The query contains product type + attributes, and the product matches both the type and all explicitly stated attributes.
3984ec64   tangwang   evalution 标注标准优化
69
  
46d94a05   tangwang   评估标准修改
70
71
  ### High Relevant
  The product satisfies the users main intent: the core product type matches, but some explicitly requested attributes are missing from the product information, cannot be confirmed, or show minor / non-critical deviations. The product is still a good substitute for the users core need.
3984ec64   tangwang   evalution 标注标准优化
72
  
46d94a05   tangwang   评估标准修改
73
74
75
76
  Use High Relevant in the following cases:
  - The core product type matches, but some requested attributes are missing, not mentioned, or cannot be verified.
  - The core product type matches, but attributes such as color, material, style, fit, or length have minor deviations, as long as the deviation does not materially undermine the users main shopping intent.
  - The product is not the users ideal target, but in an e-commerce shopping context, it would still be considered an acceptable and strong substitute.
3984ec64   tangwang   evalution 标注标准优化
77
  
46d94a05   tangwang   评估标准修改
78
79
80
81
82
83
84
  Typical examples:
  - Query: red slim-fit T-shirt
    Product: womens T-shirt
     Color and fit cannot be confirmed.
  - Query: red slim-fit T-shirt
    Product: blue slim-fit T-shirt
     Product type and fit match, but the color is different.
3984ec64   tangwang   evalution 标注标准优化
85
  
46d94a05   tangwang   评估标准修改
86
87
88
  Detailed case:
  - Query: cotton long-sleeve shirt
  - Product: J.VER Men's Linen Shirt Casual Button Down Long Sleeve Solid Plain Collar Summer Beach Shirt with Pocket”
3ac1f8d1   tangwang   评估标准优化
89
90
  
  Analysis:
46d94a05   tangwang   评估标准修改
91
92
93
94
  - Material mismatch: the query explicitly requires cotton, while the product is linen, so it cannot be labeled as Exact Match.
  - However, the core category still matches: both are long-sleeve shirts.
  - In e-commerce search, users may still click this item because the style and wearing scenario are similar.
  - Therefore, it should be labeled as High Relevant: not the exact target, but a good substitute.
3984ec64   tangwang   evalution 标注标准优化
95
  
46d94a05   tangwang   评估标准修改
96
97
98
  Detailed case:
  - Query: black mid-length skirt
  - Product: New spring autumn loose slimming full long floral skirt pleated skirt
3984ec64   tangwang   evalution 标注标准优化
99
  
46d94a05   tangwang   评估标准修改
100
101
102
103
104
105
  Analysis:
  - Category match: the product is a skirt, so the category matches.
  - Color mismatch: the product description does not indicate black and explicitly mentions floral, which is substantially different from plain black.
  - Length deviation: the user asks for mid-length, while the product title emphasizes long skirt, which is somewhat longer.
  - However, the core category skirt still matches, and style features such as slimming and full skirt may still fit some preferences of users searching for a mid-length skirt. Also, long versus mid-length is a deviation, but not a severe contradiction.
  - Therefore, this should be labeled as High Relevant: the core type matches, but there are several non-fatal attribute deviations.
3984ec64   tangwang   evalution 标注标准优化
106
  
46d94a05   tangwang   评估标准修改
107
108
  ### Low Relevant
  The product has a noticeable gap from the users core target, but still shares some similarity with the query in style, scenario, function, or broader category. A small portion of users may still view it as a barely acceptable substitute. It is not the intended item, but still has some relevance.
3984ec64   tangwang   evalution 标注标准优化
109
  
46d94a05   tangwang   评估标准修改
110
111
112
113
  Use Low Relevant in the following cases:
  - The core product type does not match, but the two types are still very close in style, wearing scenario, or function, so there is still some substitutability.
  - The core product type matches, but the product differs from the users ideal target on multiple attributes; it still has some relevance, but is no longer a strong substitute.
  - An important query requirement is clearly violated, but the product still retains a limited reason to be clicked.
3ac1f8d1   tangwang   评估标准优化
114
  
46d94a05   tangwang   评估标准修改
115
116
117
118
  Typical cases:
  - Query: black mid-length skirt
    Product: New high-waisted V-neck mid-length dress elegant printed black sexy dress
     The core product type differs (skirt vs dress), but both belong to closely related apparel types and share a similar mid-length style, so it is Low Relevant.
3984ec64   tangwang   evalution 标注标准优化
119
  
46d94a05   tangwang   评估标准修改
120
121
122
  - Query: jeans
    Product: casual pants
     The core product type is different, but both belong to the broader pants category, and the style / wearing scenario may still be close enough to be a weak substitute.
3984ec64   tangwang   evalution 标注标准优化
123
  
46d94a05   tangwang   评估标准修改
124
125
126
127
128
129
130
131
132
  ### Irrelevant
  The product does not satisfy the users main shopping intent, and the likelihood of user engagement is very low.
  
  Typical situations:
  - The core product type does not match the query and is not a close substitute in style, scenario, or function.
  - The product belongs to a roughly related broader category, but not to an interchangeable subtype explicitly requested in the query, and the style or usage scenario differs significantly.
  - The core product type matches, but the product clearly violates an explicit and important requirement in the query, with little or no acceptable substitutability.
  
  Typical examples:
4d000c94   tangwang   融合公式调参
133
134
135
136
137
138
139
140
141
142
143
144
  
  1. **Different core product type with no substitutability**
  - Query: "pants", Product: "shoes"
  - Query: "boots", Product: "sneakers"
  
  2. **Close core product type but clear conflict on key attributes**
  - Query: "slim-fit pants", Product: "loose wide-leg pants"
  - Query: "sleeveless dress", Product: "long-sleeve dress"
  - Query: "loose-fitting hoodie", Product: "slim-fit T-shirt"
  
  3. **Same broad category but significant difference in style or usage scenario**
  - Query: "jeans", Product: "sweatpants / dress pants"
46d94a05   tangwang   评估标准修改
145
146
147
148
149
150
151
152
153
154
155
  
  ## Judgment Principles
  
  1. **Product type is the highest-priority factor.**
     If the query explicitly specifies a concrete product type, the result must match that product type in order to be labeled as Exact Match or High Relevant.
     Different product types should usually be labeled as Low Relevant or Irrelevant.
  
     - **Low Relevant**: use only when the two product types are very close in style, scenario, or function, and the user may still treat one as a barely acceptable substitute for the other.
     - **Irrelevant**: all other product type mismatch cases.
  
  2. **Similar or related product types are usually not directly interchangeable when the query is explicit, but their closeness should determine whether the label is Low Relevant or Irrelevant.**
3984ec64   tangwang   evalution 标注标准优化
156
     For example:
46d94a05   tangwang   评估标准修改
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
     - **May be Low Relevant due to strong similarity in style / scenario**: dress vs skirt, long skirt vs mid-length skirt, jeans vs casual pants, sneakers vs skate shoes.
     - **Should be Irrelevant due to substantial difference in style / scenario**: pants vs shoes, T-shirt vs hat, boots vs sneakers, jeans vs suit pants, backpack vs handbag.
  
  3. **Once the core product type matches, evaluate attributes.**
     - All explicit attributes match  **Exact Match**
     - Some attributes are missing, not mentioned, cannot be verified, or show only minor deviations  **High Relevant**
     - There are multiple attribute deviations, or an important attribute is clearly violated, but the product still retains some substitutability  **Low Relevant**
     - There is a clear and important hard conflict, and substitutability is extremely low  **Irrelevant**
  
  4. **Strictly distinguish among not mentioned / cannot confirm, minor deviation, and explicit contradiction.**
     - If an attribute is not mentioned or cannot be verified, prefer **High Relevant**.
     - If an attribute shows a minor deviation, such as different color, different material, or slightly different length, it should usually be labeled **High Relevant**.
     - If an attribute is explicitly opposite to the query requirement, such as sleeveless vs long-sleeve or slim-fit vs loose wide-leg, decide between **Low Relevant** and **Irrelevant** based on the severity of the conflict and practical substitutability.
     - If the conflict directly breaks the users main shopping goal, it should usually be labeled **Irrelevant**.
  
  5. **Substitutability should be judged from real shopping intent, not just surface-level textual similarity.**
     The question is whether the user would realistically accept the product in a shopping scenario.
     - Good substitute  **High Relevant**
     - Barely acceptable substitute  **Low Relevant**
     - Hardly substitutable at all  **Irrelevant**
  
  6. **When product information is insufficient, do not treat cannot confirm as conflict.**
     If a product does not mention an attribute, that does not mean the attribute is definitely violated.
     Therefore:
     - If the attribute is not mentioned or cannot be confirmed, prefer **High Relevant**;
     - Only treat it as a conflict when the product information clearly shows the opposite of the query requirement.
3984ec64   tangwang   evalution 标注标准优化
183
  
1c2ba48e   tangwang   eval tagger
184
185
  Query: {query}
  {intent_suffix}
3984ec64   tangwang   evalution 标注标准优化
186
187
188
189
190
  
  Products:
  {lines}
  
  ## Output Format
46d94a05   tangwang   评估标准修改
191
192
193
194
195
  Output exactly {n} lines.
  Each line must be exactly one of the following:
  Exact Match
  High Relevant
  Low Relevant
3984ec64   tangwang   evalution 标注标准优化
196
197
  Irrelevant
  
1c2ba48e   tangwang   eval tagger
198
  Now, assign a relevance label to each result with respect to the query "{query}".
46d94a05   tangwang   评估标准修改
199
200
  The output lines must correspond to the products above in the same order.
  Do not output anything else.
3984ec64   tangwang   evalution 标注标准优化
201
202
  """
  
a345b01f   tangwang   eval framework
203
  _CLASSIFY_TEMPLATE_ZH = """你是一个服饰电商搜索系统中的相关性判断助手。
3ac1f8d1   tangwang   评估标准优化
204
  给定用户查询词以及每个商品的信息,请为每个商品分配一个相关性标签。
3984ec64   tangwang   evalution 标注标准优化
205
  
46d94a05   tangwang   评估标准修改
206
207
208
  你的目标是从电商搜索排序的角度,判断商品是否满足用户的购物意图。
  判断时应优先考虑“用户是否会把该商品视为目标商品,或可接受的替代品”。
  
3984ec64   tangwang   evalution 标注标准优化
209
210
211
  ## 相关性标签
  
  ### 完全相关
46d94a05   tangwang   评估标准修改
212
  商品满足用户的核心购物意图:核心商品类型匹配,且查询中所有明确提及的关键属性均有商品信息支持。
3984ec64   tangwang   evalution 标注标准优化
213
  
3ac1f8d1   tangwang   评估标准优化
214
  典型适用场景:
46d94a05   tangwang   评估标准修改
215
216
217
218
219
  - 查询仅包含商品类型,商品即为该类型。
  - 查询包含“商品类型 + 属性”,商品在类型及所有明确属性上均符合。
  
  ### 基本相关
  商品满足用户的主要意图:核心商品类型匹配,但查询中明确提出的部分要求未在商品信息中体现、无法确认,或存在轻微偏差 / 非关键偏差。该商品仍是满足用户核心需求的良好替代品。
3984ec64   tangwang   evalution 标注标准优化
220
  
46d94a05   tangwang   评估标准修改
221
222
223
224
  在以下情况使用“基本相关”:
  - 核心商品类型匹配,但部分属性缺失、未提及或无法确认。
  - 核心商品类型匹配,但颜色、材质、风格、版型、长度等属性存在轻微偏差,只要这种偏差不会明显破坏用户的主要购买意图。
  - 商品不是用户最理想的目标,但在电商购物场景下仍可能被视为可接受、且较优的替代品。
3984ec64   tangwang   evalution 标注标准优化
225
226
  
  典型情况:
46d94a05   tangwang   评估标准修改
227
228
229
230
  - 查询:“红色修身T恤”,商品:“女士T恤”
     颜色、版型无法确认。
  - 查询:“红色修身T恤”,商品:“蓝色修身T恤”
     商品类型和版型匹配,但颜色不同。
3984ec64   tangwang   evalution 标注标准优化
231
  
3ac1f8d1   tangwang   评估标准优化
232
233
234
  详细案例:
  - 查询:“棉质长袖衬衫”
  - 商品:“J.VER男式亚麻衬衫休闲纽扣长袖衬衫纯色平领夏季沙滩衬衫带口袋”
3984ec64   tangwang   evalution 标注标准优化
235
  
3ac1f8d1   tangwang   评估标准优化
236
  分析:
46d94a05   tangwang   评估标准修改
237
  - 材质不符:Query 明确指定“棉质”,而商品为“亚麻”,因此不能判为“完全相关”。
3ac1f8d1   tangwang   评估标准优化
238
239
  - 但核心品类仍然匹配:两者都是“长袖衬衫”。
  - 在电商搜索中,用户仍可能因为款式、穿着场景相近而点击该商品。
46d94a05   tangwang   评估标准修改
240
  - 因此应判为“基本相关”,即“非精确目标,但属于良好替代品”。
3984ec64   tangwang   evalution 标注标准优化
241
  
bdb65283   tangwang   标注框架 批量标注
242
243
244
245
246
  详细案例:
  - 查询:“黑色中长半身裙”
  - 商品:“春秋季新款宽松显瘦大摆长裙碎花半身裙褶皱设计裙”
  
  分析:
46d94a05   tangwang   评估标准修改
247
248
249
250
251
252
253
254
255
256
257
258
259
  - 品类匹配:商品是“半身裙”,品类符合。
  - 颜色不匹配:商品描述未提及黑色,且明确包含“碎花”,与纯黑差异较大。
  - 长度存在偏差:用户要求“中长”,而商品标题强调“长裙”,长度偏长。
  - 但核心品类“半身裙”匹配,“显瘦”“大摆”等风格特征仍可能符合部分搜索“中长半身裙”用户的潜在偏好;同时“长裙”和“中长”虽有偏差,但不构成严重对立。
  - 因此应判为“基本相关”:核心品类匹配,但存在若干非致命属性偏差。
  
  ### 弱相关
  商品与用户的核心目标存在明显差距,但仍与查询在风格、场景、功能或大类上具有一定相似性,可能被少量用户视为勉强可接受的替代品。属于“非目标,但仍有一定关联”。
  
  在以下情况使用“弱相关”:
  - 核心商品类型不一致,但两者在风格、穿着场景或功能上非常接近,仍具有一定替代性。
  - 核心商品类型匹配,但在多个属性上与用户理想目标差距较大,虽仍有一定关联性,但已不是高质量替代品。
  - 查询要求中的某个重要属性被明显违背,但商品仍保留少量被点击的理由。
bdb65283   tangwang   标注框架 批量标注
260
261
  
  典型情况:
46d94a05   tangwang   评估标准修改
262
263
  - 查询:“黑色中长半身裙”,商品:“新款高腰V领中长款连衣裙 优雅印花黑色性感连衣裙”
     核心商品类型“半身裙”与“连衣裙”不同,但两者同属裙装,且款式上均为“中长款”,在穿搭场景上接近,因此属于“弱相关”。
bdb65283   tangwang   标注框架 批量标注
264
  
46d94a05   tangwang   评估标准修改
265
266
267
268
269
270
271
272
273
274
  - 查询:“牛仔裤”,商品:“休闲裤”
     核心商品类型不同,但同属裤装大类,风格和穿着场景可能接近,可作为较弱替代品。
  
  ### 不相关
  商品未满足用户的主要购物意图,用户点击动机极低。
  
  主要表现为以下情形之一:
  - 核心商品类型与查询不匹配,且不属于风格 / 场景 / 功能接近的可替代品。
  - 商品虽属于大致相关的大类,但与查询明确指定的具体子类不可互换,且风格或场景差异大。
  - 核心商品类型匹配,但商品明显违背了查询中一个明确且重要的要求,且几乎不具备可接受的替代性。
3984ec64   tangwang   evalution 标注标准优化
275
276
  
  典型情况:
4d000c94   tangwang   融合公式调参
277
278
  
  **1. 核心品类不同,且无替代性**
46d94a05   tangwang   评估标准修改
279
  - 查询:“裤子”,商品:“鞋子”
46d94a05   tangwang   评估标准修改
280
  - 查询:“靴子”,商品:“运动鞋”
4d000c94   tangwang   融合公式调参
281
282
283
284
285
286
287
288
  
  **2. 核心品类相近,但关键属性明显冲突**
  - 查询:“紧身裤”,商品:“阔腿裤”
  - 查询:“无袖连衣裙”,商品:“长袖连衣裙”
  - 查询:“宽松卫衣”,商品:“修身T恤”
  
  **3. 核心品类同属大类,但风格、场景差异巨大**
  - 查询:“牛仔裤”,商品:“运动裤 / 西裤”
3984ec64   tangwang   evalution 标注标准优化
289
  
3ac1f8d1   tangwang   评估标准优化
290
  ## 判断原则
3984ec64   tangwang   evalution 标注标准优化
291
  
46d94a05   tangwang   评估标准修改
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
  1. **商品类型是最高优先级因素。**
     如果查询明确指定了具体商品类型,那么结果必须匹配该商品类型,才可能判为“完全相关”或“基本相关”。
     不同商品类型通常应判为“弱相关”或“不相关”。
  
     - **弱相关**:仅当两种商品类型在风格、场景、功能上非常接近,用户有一定概率将其视为勉强可接受的替代品时使用。
     - **不相关**:其他所有商品类型不匹配的情况。
  
  2. **相似或相关的商品类型,在查询明确时通常不可直接互换,但要根据接近程度区分“弱相关”与“不相关”。**
     例如:
     - **风格 / 场景高度接近,可判为弱相关**:连衣裙 vs 半身裙、长裙 vs 中长裙、牛仔裤 vs 休闲裤、运动鞋 vs 板鞋。
     - **风格 / 场景差异大,应判为不相关**:裤子 vs 鞋子、T vs 帽子、靴子 vs 运动鞋、牛仔裤 vs 西装裤、双肩包 vs 手提包。
  
  3. **当核心商品类型匹配后,再评估属性。**
     - 所有明确属性都匹配  **完全相关**
     - 部分属性缺失、未提及、无法确认,或存在轻微偏差  **基本相关**
     - 存在多个属性偏差,或某个重要属性被明显违背,但商品仍保留一定替代性  **弱相关**
     - 存在明确且重要的强冲突,且替代性极低  **不相关**
  
  4. **要严格区分“未提及 / 无法确认”“轻微偏差”“明确冲突”。**
     - 如果某属性没有提及,或无法验证,优先判为“基本相关”。
     - 如果某属性存在轻微偏差,如颜色不同、材质不同、长度略有差异,通常判为“基本相关”。
     - 如果某属性与查询要求明确相反,如无袖 vs 长袖、修身 vs 宽松阔腿,则要根据冲突严重性与替代性,在“弱相关”与“不相关”之间判断。
     - 若该冲突会直接破坏用户的主要购买目标,通常判为“不相关”。
  
  5. **“是否可替代”应从真实电商购物意图出发判断。**
     不是只看字面相似,而要看用户在购物场景下是否可能接受该商品。
     - 良好替代品  **基本相关**
     - 勉强替代品  **弱相关**
     - 几乎不可替代  **不相关**
  
  6. **若商品信息不足,不要把“无法确认”误判为“冲突”。**
     商品未写明某属性,不等于该属性一定不符合。
     因此:
     - 未提及 / 无法确认,优先按“基本相关”处理;
     - 只有当商品信息明确显示与查询要求相反时,才视为属性冲突。
3984ec64   tangwang   evalution 标注标准优化
327
  
1c2ba48e   tangwang   eval tagger
328
329
  查询:{query}
  {intent_suffix}
3984ec64   tangwang   evalution 标注标准优化
330
  
3ac1f8d1   tangwang   评估标准优化
331
  商品:
3984ec64   tangwang   evalution 标注标准优化
332
333
334
  {lines}
  
  ## 输出格式
46d94a05   tangwang   评估标准修改
335
  严格输出 {n} 行,每行只能是以下四者之一:
3ac1f8d1   tangwang   评估标准优化
336
  完全相关
46d94a05   tangwang   评估标准修改
337
338
  基本相关
  弱相关
3ac1f8d1   tangwang   评估标准优化
339
  不相关
3984ec64   tangwang   evalution 标注标准优化
340
  
1c2ba48e   tangwang   eval tagger
341
  现在请根据 query{query}”,为每个结果标注相关性标签。输出行必须与上方商品顺序一一对应,不要输出任何其他内容。
3984ec64   tangwang   evalution 标注标准优化
342
343
344
  """
  
  
cdd8ee3a   tangwang   eval框架日志独立
345
346
347
348
349
350
351
352
353
354
  def intent_analysis_prompt(query: str) -> str:
      return _QUERY_INTENT_ANALYSIS_TEMPLATE_EN.format(query=query)
  
  
  def classify_prompt(
      query: str,
      numbered_doc_lines: Sequence[str],
      *,
      query_intent_block: str = "",
  ) -> str:
c81b0fc1   tangwang   scripts/evaluatio...
355
356
      lines = "\n".join(numbered_doc_lines)
      n = len(numbered_doc_lines)
cdd8ee3a   tangwang   eval框架日志独立
357
358
      intent_suffix = f"\n{query_intent_block.strip()}" if query_intent_block and query_intent_block.strip() else ""
      return _CLASSIFY_TEMPLATE_EN.format(query=query, intent_suffix=intent_suffix, lines=lines, n=n)