Blame view

scripts/evaluation/eval_framework/prompts.py 20.2 KB
c81b0fc1   tangwang   scripts/evaluatio...
1
2
3
4
  """LLM prompt templates for relevance judging (keep wording changes here)."""
  
  from __future__ import annotations
  
a345b01f   tangwang   eval framework
5
  from typing import Sequence
c81b0fc1   tangwang   scripts/evaluatio...
6
  
cdd8ee3a   tangwang   eval框架日志独立
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
  
  _QUERY_INTENT_ANALYSIS_TEMPLATE_EN = """You are an intent analysis expert for a fashion e-commerce search system.
  
  Given a user's search query, analyze the shopping intent behind the query in the context of fashion and apparel e-commerce, and summarize the user's core search need in one concise sentence.
  Also provide the Chinese translation and English translation of the query.
  
  Requirements:
  - Keep the intent analysis concise and easy to understand, using 1 to 3 short sentences.
  - Stay grounded in the original query and summarize the user's likely shopping intent without adding unnecessary context.
  - When the query is vague or ambiguous, take a conservative approach and keep the analysis close to the original wording.
  - Chinese translation: if the original query is already in Chinese, keep it unchanged.
  - English translation: if the original query is already in English, keep it unchanged.
  - Do not output anything other than the required three-line format.
  
  Output format (strictly exactly three lines):
  Intent: concise analysis of the user's search intent
  Query中文翻译: Chinese translation of the query
  Query English translation: English translation of the query
  
  Now analyze the following query:
  
  Query: {query}
  """
  
  _QUERY_INTENT_ANALYSIS_RESULT_TEMPLATE_ZH = """
  你是一个服装品类电商搜索意图分析专家。
  
  给定用户输入的搜索词,请在服装品类电商场景下,分析该搜索词背后的购物意图,并用一句话简要描述用户的核心搜索需求。
  同时,提供该搜索词的中文翻译和英文翻译。
  
  要求:
  - 意图分析应简洁易懂,用 1  3 句短句概括用户的搜索意图。
  - 结合 query 本身,尽量贴近用户原始搜索需求进行总结,不添加不必要的背景、延伸或臆测。
  - 如果 query 不够明确或有歧义,应保守处理,尽量保持与原词表达一致。
  - 中文翻译:如果原始 query 本身就是中文,则按原样输出。
  - 英文翻译:如果原始 query 本身就是英文,则按原样输出。
  - 除指定格式外,不要输出任何额外说明。
  
  输出格式(严格按三行输出):
  Intent: 对用户搜索意图的简洁分析
  Query中文翻译: query 的中文翻译
  Query English translation: query 的英文翻译
  
  现在请分析以下搜索词:
  
  Query: {query}
  """
  
a345b01f   tangwang   eval framework
55
  _CLASSIFY_TEMPLATE_EN = """You are a relevance judgment assistant for a fashion e-commerce search system.
46d94a05   tangwang   评估标准修改
56
57
58
59
  Given a user query and the information for each product, assign a relevance label to each product.
  
  Your goal is to judge relevance from the perspective of e-commerce search ranking.
  The key question is whether the user would view the product as the intended item, or as an acceptable substitute.
3984ec64   tangwang   evalution 标注标准优化
60
61
62
  
  ## Relevance Labels
  
46d94a05   tangwang   评估标准修改
63
64
  ### Exact Match
  The product satisfies the users core shopping intent: the core product type matches, and all explicitly stated key attributes in the query are supported by the product information, with no obvious conflict.
3984ec64   tangwang   evalution 标注标准优化
65
  
3ac1f8d1   tangwang   评估标准优化
66
67
  Typical use cases:
  - The query contains only a product type, and the product is exactly that type.
46d94a05   tangwang   评估标准修改
68
  - The query contains product type + attributes, and the product matches both the type and all explicitly stated attributes.
3984ec64   tangwang   evalution 标注标准优化
69
  
46d94a05   tangwang   评估标准修改
70
71
  ### High Relevant
  The product satisfies the users main intent: the core product type matches, but some explicitly requested attributes are missing from the product information, cannot be confirmed, or show minor / non-critical deviations. The product is still a good substitute for the users core need.
3984ec64   tangwang   evalution 标注标准优化
72
  
46d94a05   tangwang   评估标准修改
73
74
75
76
  Use High Relevant in the following cases:
  - The core product type matches, but some requested attributes are missing, not mentioned, or cannot be verified.
  - The core product type matches, but attributes such as color, material, style, fit, or length have minor deviations, as long as the deviation does not materially undermine the users main shopping intent.
  - The product is not the users ideal target, but in an e-commerce shopping context, it would still be considered an acceptable and strong substitute.
3984ec64   tangwang   evalution 标注标准优化
77
  
46d94a05   tangwang   评估标准修改
78
79
80
81
82
83
84
  Typical examples:
  - Query: red slim-fit T-shirt
    Product: womens T-shirt
     Color and fit cannot be confirmed.
  - Query: red slim-fit T-shirt
    Product: blue slim-fit T-shirt
     Product type and fit match, but the color is different.
3984ec64   tangwang   evalution 标注标准优化
85
  
46d94a05   tangwang   评估标准修改
86
87
88
  Detailed case:
  - Query: cotton long-sleeve shirt
  - Product: J.VER Men's Linen Shirt Casual Button Down Long Sleeve Solid Plain Collar Summer Beach Shirt with Pocket”
3ac1f8d1   tangwang   评估标准优化
89
90
  
  Analysis:
46d94a05   tangwang   评估标准修改
91
92
93
94
  - Material mismatch: the query explicitly requires cotton, while the product is linen, so it cannot be labeled as Exact Match.
  - However, the core category still matches: both are long-sleeve shirts.
  - In e-commerce search, users may still click this item because the style and wearing scenario are similar.
  - Therefore, it should be labeled as High Relevant: not the exact target, but a good substitute.
3984ec64   tangwang   evalution 标注标准优化
95
  
46d94a05   tangwang   评估标准修改
96
97
98
  Detailed case:
  - Query: black mid-length skirt
  - Product: New spring autumn loose slimming full long floral skirt pleated skirt
3984ec64   tangwang   evalution 标注标准优化
99
  
46d94a05   tangwang   评估标准修改
100
101
102
103
104
105
  Analysis:
  - Category match: the product is a skirt, so the category matches.
  - Color mismatch: the product description does not indicate black and explicitly mentions floral, which is substantially different from plain black.
  - Length deviation: the user asks for mid-length, while the product title emphasizes long skirt, which is somewhat longer.
  - However, the core category skirt still matches, and style features such as slimming and full skirt may still fit some preferences of users searching for a mid-length skirt. Also, long versus mid-length is a deviation, but not a severe contradiction.
  - Therefore, this should be labeled as High Relevant: the core type matches, but there are several non-fatal attribute deviations.
3984ec64   tangwang   evalution 标注标准优化
106
  
46d94a05   tangwang   评估标准修改
107
108
  ### Low Relevant
  The product has a noticeable gap from the users core target, but still shares some similarity with the query in style, scenario, function, or broader category. A small portion of users may still view it as a barely acceptable substitute. It is not the intended item, but still has some relevance.
3984ec64   tangwang   evalution 标注标准优化
109
  
46d94a05   tangwang   评估标准修改
110
111
112
113
  Use Low Relevant in the following cases:
  - The core product type does not match, but the two types are still very close in style, wearing scenario, or function, so there is still some substitutability.
  - The core product type matches, but the product differs from the users ideal target on multiple attributes; it still has some relevance, but is no longer a strong substitute.
  - An important query requirement is clearly violated, but the product still retains a limited reason to be clicked.
3ac1f8d1   tangwang   评估标准优化
114
  
46d94a05   tangwang   评估标准修改
115
116
117
118
  Typical cases:
  - Query: black mid-length skirt
    Product: New high-waisted V-neck mid-length dress elegant printed black sexy dress
     The core product type differs (skirt vs dress), but both belong to closely related apparel types and share a similar mid-length style, so it is Low Relevant.
3984ec64   tangwang   evalution 标注标准优化
119
  
46d94a05   tangwang   评估标准修改
120
121
122
  - Query: jeans
    Product: casual pants
     The core product type is different, but both belong to the broader pants category, and the style / wearing scenario may still be close enough to be a weak substitute.
3984ec64   tangwang   evalution 标注标准优化
123
  
46d94a05   tangwang   评估标准修改
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
  ### Irrelevant
  The product does not satisfy the users main shopping intent, and the likelihood of user engagement is very low.
  
  Typical situations:
  - The core product type does not match the query and is not a close substitute in style, scenario, or function.
  - The product belongs to a roughly related broader category, but not to an interchangeable subtype explicitly requested in the query, and the style or usage scenario differs significantly.
  - The core product type matches, but the product clearly violates an explicit and important requirement in the query, with little or no acceptable substitutability.
  
  Typical examples:
  - Query: pants
    Product: shoes
     Wrong product type.
  - Query: slim-fit pants
    Product: loose wide-leg pants
     Clear contradiction in fit, with extremely low substitutability.
  - Query: sleeveless dress
    Product: long-sleeve dress
     Clear contradiction in sleeve type.
  - Query: jeans
    Product: sweatpants
     Different core category, with significantly different style and wearing scenario.
  - Query: boots
    Product: sneakers
     Different core category, different function, and different usage scenario.
  
  ## Judgment Principles
  
  1. **Product type is the highest-priority factor.**
     If the query explicitly specifies a concrete product type, the result must match that product type in order to be labeled as Exact Match or High Relevant.
     Different product types should usually be labeled as Low Relevant or Irrelevant.
  
     - **Low Relevant**: use only when the two product types are very close in style, scenario, or function, and the user may still treat one as a barely acceptable substitute for the other.
     - **Irrelevant**: all other product type mismatch cases.
  
  2. **Similar or related product types are usually not directly interchangeable when the query is explicit, but their closeness should determine whether the label is Low Relevant or Irrelevant.**
3984ec64   tangwang   evalution 标注标准优化
159
     For example:
46d94a05   tangwang   评估标准修改
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
     - **May be Low Relevant due to strong similarity in style / scenario**: dress vs skirt, long skirt vs mid-length skirt, jeans vs casual pants, sneakers vs skate shoes.
     - **Should be Irrelevant due to substantial difference in style / scenario**: pants vs shoes, T-shirt vs hat, boots vs sneakers, jeans vs suit pants, backpack vs handbag.
  
  3. **Once the core product type matches, evaluate attributes.**
     - All explicit attributes match  **Exact Match**
     - Some attributes are missing, not mentioned, cannot be verified, or show only minor deviations  **High Relevant**
     - There are multiple attribute deviations, or an important attribute is clearly violated, but the product still retains some substitutability  **Low Relevant**
     - There is a clear and important hard conflict, and substitutability is extremely low  **Irrelevant**
  
  4. **Strictly distinguish among not mentioned / cannot confirm, minor deviation, and explicit contradiction.**
     - If an attribute is not mentioned or cannot be verified, prefer **High Relevant**.
     - If an attribute shows a minor deviation, such as different color, different material, or slightly different length, it should usually be labeled **High Relevant**.
     - If an attribute is explicitly opposite to the query requirement, such as sleeveless vs long-sleeve or slim-fit vs loose wide-leg, decide between **Low Relevant** and **Irrelevant** based on the severity of the conflict and practical substitutability.
     - If the conflict directly breaks the users main shopping goal, it should usually be labeled **Irrelevant**.
  
  5. **Substitutability should be judged from real shopping intent, not just surface-level textual similarity.**
     The question is whether the user would realistically accept the product in a shopping scenario.
     - Good substitute  **High Relevant**
     - Barely acceptable substitute  **Low Relevant**
     - Hardly substitutable at all  **Irrelevant**
  
  6. **When product information is insufficient, do not treat cannot confirm as conflict.**
     If a product does not mention an attribute, that does not mean the attribute is definitely violated.
     Therefore:
     - If the attribute is not mentioned or cannot be confirmed, prefer **High Relevant**;
     - Only treat it as a conflict when the product information clearly shows the opposite of the query requirement.
3984ec64   tangwang   evalution 标注标准优化
186
  
cdd8ee3a   tangwang   eval框架日志独立
187
  Query: {query}{intent_suffix}
3984ec64   tangwang   evalution 标注标准优化
188
189
190
191
192
  
  Products:
  {lines}
  
  ## Output Format
46d94a05   tangwang   评估标准修改
193
194
195
196
197
  Output exactly {n} lines.
  Each line must be exactly one of the following:
  Exact Match
  High Relevant
  Low Relevant
3984ec64   tangwang   evalution 标注标准优化
198
199
  Irrelevant
  
46d94a05   tangwang   评估标准修改
200
201
  The output lines must correspond to the products above in the same order.
  Do not output anything else.
3984ec64   tangwang   evalution 标注标准优化
202
203
  """
  
a345b01f   tangwang   eval framework
204
  _CLASSIFY_TEMPLATE_ZH = """你是一个服饰电商搜索系统中的相关性判断助手。
3ac1f8d1   tangwang   评估标准优化
205
  给定用户查询词以及每个商品的信息,请为每个商品分配一个相关性标签。
3984ec64   tangwang   evalution 标注标准优化
206
  
46d94a05   tangwang   评估标准修改
207
208
209
  你的目标是从电商搜索排序的角度,判断商品是否满足用户的购物意图。
  判断时应优先考虑“用户是否会把该商品视为目标商品,或可接受的替代品”。
  
3984ec64   tangwang   evalution 标注标准优化
210
211
212
  ## 相关性标签
  
  ### 完全相关
46d94a05   tangwang   评估标准修改
213
  商品满足用户的核心购物意图:核心商品类型匹配,且查询中所有明确提及的关键属性均有商品信息支持。
3984ec64   tangwang   evalution 标注标准优化
214
  
3ac1f8d1   tangwang   评估标准优化
215
  典型适用场景:
46d94a05   tangwang   评估标准修改
216
217
218
219
220
  - 查询仅包含商品类型,商品即为该类型。
  - 查询包含“商品类型 + 属性”,商品在类型及所有明确属性上均符合。
  
  ### 基本相关
  商品满足用户的主要意图:核心商品类型匹配,但查询中明确提出的部分要求未在商品信息中体现、无法确认,或存在轻微偏差 / 非关键偏差。该商品仍是满足用户核心需求的良好替代品。
3984ec64   tangwang   evalution 标注标准优化
221
  
46d94a05   tangwang   评估标准修改
222
223
224
225
  在以下情况使用“基本相关”:
  - 核心商品类型匹配,但部分属性缺失、未提及或无法确认。
  - 核心商品类型匹配,但颜色、材质、风格、版型、长度等属性存在轻微偏差,只要这种偏差不会明显破坏用户的主要购买意图。
  - 商品不是用户最理想的目标,但在电商购物场景下仍可能被视为可接受、且较优的替代品。
3984ec64   tangwang   evalution 标注标准优化
226
227
  
  典型情况:
46d94a05   tangwang   评估标准修改
228
229
230
231
  - 查询:“红色修身T恤”,商品:“女士T恤”
     颜色、版型无法确认。
  - 查询:“红色修身T恤”,商品:“蓝色修身T恤”
     商品类型和版型匹配,但颜色不同。
3984ec64   tangwang   evalution 标注标准优化
232
  
3ac1f8d1   tangwang   评估标准优化
233
234
235
  详细案例:
  - 查询:“棉质长袖衬衫”
  - 商品:“J.VER男式亚麻衬衫休闲纽扣长袖衬衫纯色平领夏季沙滩衬衫带口袋”
3984ec64   tangwang   evalution 标注标准优化
236
  
3ac1f8d1   tangwang   评估标准优化
237
  分析:
46d94a05   tangwang   评估标准修改
238
  - 材质不符:Query 明确指定“棉质”,而商品为“亚麻”,因此不能判为“完全相关”。
3ac1f8d1   tangwang   评估标准优化
239
240
  - 但核心品类仍然匹配:两者都是“长袖衬衫”。
  - 在电商搜索中,用户仍可能因为款式、穿着场景相近而点击该商品。
46d94a05   tangwang   评估标准修改
241
  - 因此应判为“基本相关”,即“非精确目标,但属于良好替代品”。
3984ec64   tangwang   evalution 标注标准优化
242
  
bdb65283   tangwang   标注框架 批量标注
243
244
245
246
247
  详细案例:
  - 查询:“黑色中长半身裙”
  - 商品:“春秋季新款宽松显瘦大摆长裙碎花半身裙褶皱设计裙”
  
  分析:
46d94a05   tangwang   评估标准修改
248
249
250
251
252
253
254
255
256
257
258
259
260
  - 品类匹配:商品是“半身裙”,品类符合。
  - 颜色不匹配:商品描述未提及黑色,且明确包含“碎花”,与纯黑差异较大。
  - 长度存在偏差:用户要求“中长”,而商品标题强调“长裙”,长度偏长。
  - 但核心品类“半身裙”匹配,“显瘦”“大摆”等风格特征仍可能符合部分搜索“中长半身裙”用户的潜在偏好;同时“长裙”和“中长”虽有偏差,但不构成严重对立。
  - 因此应判为“基本相关”:核心品类匹配,但存在若干非致命属性偏差。
  
  ### 弱相关
  商品与用户的核心目标存在明显差距,但仍与查询在风格、场景、功能或大类上具有一定相似性,可能被少量用户视为勉强可接受的替代品。属于“非目标,但仍有一定关联”。
  
  在以下情况使用“弱相关”:
  - 核心商品类型不一致,但两者在风格、穿着场景或功能上非常接近,仍具有一定替代性。
  - 核心商品类型匹配,但在多个属性上与用户理想目标差距较大,虽仍有一定关联性,但已不是高质量替代品。
  - 查询要求中的某个重要属性被明显违背,但商品仍保留少量被点击的理由。
bdb65283   tangwang   标注框架 批量标注
261
262
  
  典型情况:
46d94a05   tangwang   评估标准修改
263
264
  - 查询:“黑色中长半身裙”,商品:“新款高腰V领中长款连衣裙 优雅印花黑色性感连衣裙”
     核心商品类型“半身裙”与“连衣裙”不同,但两者同属裙装,且款式上均为“中长款”,在穿搭场景上接近,因此属于“弱相关”。
bdb65283   tangwang   标注框架 批量标注
265
  
46d94a05   tangwang   评估标准修改
266
267
268
269
270
271
272
273
274
275
  - 查询:“牛仔裤”,商品:“休闲裤”
     核心商品类型不同,但同属裤装大类,风格和穿着场景可能接近,可作为较弱替代品。
  
  ### 不相关
  商品未满足用户的主要购物意图,用户点击动机极低。
  
  主要表现为以下情形之一:
  - 核心商品类型与查询不匹配,且不属于风格 / 场景 / 功能接近的可替代品。
  - 商品虽属于大致相关的大类,但与查询明确指定的具体子类不可互换,且风格或场景差异大。
  - 核心商品类型匹配,但商品明显违背了查询中一个明确且重要的要求,且几乎不具备可接受的替代性。
3984ec64   tangwang   evalution 标注标准优化
276
277
  
  典型情况:
46d94a05   tangwang   评估标准修改
278
279
280
281
282
283
284
285
286
287
  - 查询:“裤子”,商品:“鞋子”
     商品类型错误。
  - 查询:“修身裤”,商品:“宽松阔腿裤”
     与版型要求明显冲突,替代性极低。
  - 查询:“无袖连衣裙”,商品:“长袖连衣裙”
     与袖型要求明显冲突。
  - 查询:“牛仔裤”,商品:“运动裤”
     核心品类不同(牛仔裤 vs 运动裤),风格和场景差异大。
  - 查询:“靴子”,商品:“运动鞋”
     核心品类不同,功能和适用场景差异大。
3984ec64   tangwang   evalution 标注标准优化
288
  
3ac1f8d1   tangwang   评估标准优化
289
  ## 判断原则
3984ec64   tangwang   evalution 标注标准优化
290
  
46d94a05   tangwang   评估标准修改
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
  1. **商品类型是最高优先级因素。**
     如果查询明确指定了具体商品类型,那么结果必须匹配该商品类型,才可能判为“完全相关”或“基本相关”。
     不同商品类型通常应判为“弱相关”或“不相关”。
  
     - **弱相关**:仅当两种商品类型在风格、场景、功能上非常接近,用户有一定概率将其视为勉强可接受的替代品时使用。
     - **不相关**:其他所有商品类型不匹配的情况。
  
  2. **相似或相关的商品类型,在查询明确时通常不可直接互换,但要根据接近程度区分“弱相关”与“不相关”。**
     例如:
     - **风格 / 场景高度接近,可判为弱相关**:连衣裙 vs 半身裙、长裙 vs 中长裙、牛仔裤 vs 休闲裤、运动鞋 vs 板鞋。
     - **风格 / 场景差异大,应判为不相关**:裤子 vs 鞋子、T vs 帽子、靴子 vs 运动鞋、牛仔裤 vs 西装裤、双肩包 vs 手提包。
  
  3. **当核心商品类型匹配后,再评估属性。**
     - 所有明确属性都匹配  **完全相关**
     - 部分属性缺失、未提及、无法确认,或存在轻微偏差  **基本相关**
     - 存在多个属性偏差,或某个重要属性被明显违背,但商品仍保留一定替代性  **弱相关**
     - 存在明确且重要的强冲突,且替代性极低  **不相关**
  
  4. **要严格区分“未提及 / 无法确认”“轻微偏差”“明确冲突”。**
     - 如果某属性没有提及,或无法验证,优先判为“基本相关”。
     - 如果某属性存在轻微偏差,如颜色不同、材质不同、长度略有差异,通常判为“基本相关”。
     - 如果某属性与查询要求明确相反,如无袖 vs 长袖、修身 vs 宽松阔腿,则要根据冲突严重性与替代性,在“弱相关”与“不相关”之间判断。
     - 若该冲突会直接破坏用户的主要购买目标,通常判为“不相关”。
  
  5. **“是否可替代”应从真实电商购物意图出发判断。**
     不是只看字面相似,而要看用户在购物场景下是否可能接受该商品。
     - 良好替代品  **基本相关**
     - 勉强替代品  **弱相关**
     - 几乎不可替代  **不相关**
  
  6. **若商品信息不足,不要把“无法确认”误判为“冲突”。**
     商品未写明某属性,不等于该属性一定不符合。
     因此:
     - 未提及 / 无法确认,优先按“基本相关”处理;
     - 只有当商品信息明确显示与查询要求相反时,才视为属性冲突。
3984ec64   tangwang   evalution 标注标准优化
326
  
cdd8ee3a   tangwang   eval框架日志独立
327
  查询:{query}{intent_suffix}
3984ec64   tangwang   evalution 标注标准优化
328
  
3ac1f8d1   tangwang   评估标准优化
329
  商品:
3984ec64   tangwang   evalution 标注标准优化
330
331
332
  {lines}
  
  ## 输出格式
46d94a05   tangwang   评估标准修改
333
  严格输出 {n} 行,每行只能是以下四者之一:
3ac1f8d1   tangwang   评估标准优化
334
  完全相关
46d94a05   tangwang   评估标准修改
335
336
  基本相关
  弱相关
3ac1f8d1   tangwang   评估标准优化
337
  不相关
3984ec64   tangwang   evalution 标注标准优化
338
  
3ac1f8d1   tangwang   评估标准优化
339
340
  输出行必须与上方商品顺序一一对应。
  不要输出任何其他内容。
3984ec64   tangwang   evalution 标注标准优化
341
342
343
  """
  
  
cdd8ee3a   tangwang   eval框架日志独立
344
345
346
347
348
349
350
351
352
353
  def intent_analysis_prompt(query: str) -> str:
      return _QUERY_INTENT_ANALYSIS_TEMPLATE_EN.format(query=query)
  
  
  def classify_prompt(
      query: str,
      numbered_doc_lines: Sequence[str],
      *,
      query_intent_block: str = "",
  ) -> str:
c81b0fc1   tangwang   scripts/evaluatio...
354
355
      lines = "\n".join(numbered_doc_lines)
      n = len(numbered_doc_lines)
cdd8ee3a   tangwang   eval框架日志独立
356
357
      intent_suffix = f"\n{query_intent_block.strip()}" if query_intent_block and query_intent_block.strip() else ""
      return _CLASSIFY_TEMPLATE_EN.format(query=query, intent_suffix=intent_suffix, lines=lines, n=n)