639bee0a
tangwang
issues整理(评估框架<R...
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
|
_QUERY_INTENT_ANALYSIS_TEMPLATE_EN = """You are an intent analysis expert for a fashion e-commerce search system.
Given a user's search query, analyze the shopping intent behind the query in the context of fashion and apparel e-commerce, and summarize the user's core search need in one concise sentence.
Also provide the Chinese translation and English translation of the query.
Requirements:
- Keep the intent analysis concise and easy to understand, using 1 to 3 short sentences.
- Stay grounded in the original query and summarize the user's likely shopping intent without adding unnecessary context.
- When the query is vague or ambiguous, take a conservative approach and keep the analysis close to the original wording.
- Chinese translation: if the original query is already in Chinese, keep it unchanged.
- English translation: if the original query is already in English, keep it unchanged.
- Do not output anything other than the required three-line format.
Output format (strictly exactly three lines):
Intent: concise analysis of the user's search intent
Query中文翻译: Chinese translation of the query
Query English translation: English translation of the query
Now analyze the following query:
Query: {query}
"""
_QUERY_INTENT_ANALYSIS_RESULT_TEMPLATE_ZH = """
你是一个服装品类电商搜索意图分析专家。
给定用户输入的搜索词,请在服装品类电商场景下,分析该搜索词背后的购物意图,并用一句话简要描述用户的核心搜索需求。
同时,提供该搜索词的中文翻译和英文翻译。
要求:
- 意图分析应简洁易懂,用 1 到 3 句短句概括用户的搜索意图。
- 结合 query 本身,尽量贴近用户原始搜索需求进行总结,不添加不必要的背景、延伸或臆测。
- 如果 query 不够明确或有歧义,应保守处理,尽量保持与原词表达一致。
- 中文翻译:如果原始 query 本身就是中文,则按原样输出。
- 英文翻译:如果原始 query 本身就是英文,则按原样输出。
- 除指定格式外,不要输出任何额外说明。
输出格式(严格按三行输出):
Intent: 对用户搜索意图的简洁分析
Query中文翻译: query 的中文翻译
Query English translation: query 的英文翻译
现在请分析以下搜索词:
Query: {query}
"""
_CLASSIFY_TEMPLATE_EN = """You are a relevance judgment assistant for a fashion e-commerce search system.
Given a user query and the information for each product, assign a relevance label to each product.
Your goal is to judge relevance from the perspective of e-commerce search ranking.
The key question is whether the user would view the product as the intended item, or as an acceptable substitute.
## Relevance Labels
### Exact Match
The product satisfies the user’s core shopping intent: the core product type matches, and all explicitly stated key attributes in the query are supported by the product information, with no obvious conflict.
Typical use cases:
- The query contains only a product type, and the product is exactly that type.
- The query contains “product type + attributes”, and the product matches both the type and all explicitly stated attributes.
### High Relevant
The product satisfies the user’s main intent: the core product type matches, but some explicitly requested attributes are missing from the product information, cannot be confirmed, or show minor / non-critical deviations. The product is still a good substitute for the user’s core need.
Use “High Relevant” in the following cases:
- The core product type matches, but some requested attributes are missing, not mentioned, or cannot be verified.
- The core product type matches, but attributes such as color, material, style, fit, or length have minor deviations, as long as the deviation does not materially undermine the user’s main shopping intent.
- The product is not the user’s ideal target, but in an e-commerce shopping context, it would still be considered an acceptable and strong substitute.
Typical examples:
- Query: “red slim-fit T-shirt”
Product: “women’s T-shirt”
→ Color and fit cannot be confirmed.
- Query: “red slim-fit T-shirt”
Product: “blue slim-fit T-shirt”
→ Product type and fit match, but the color is different.
Detailed case:
- Query: “cotton long-sleeve shirt”
- Product: “J.VER Men's Linen Shirt Casual Button Down Long Sleeve Solid Plain Collar Summer Beach Shirt with Pocket”
Analysis:
- Material mismatch: the query explicitly requires “cotton”, while the product is “linen”, so it cannot be labeled as “Exact Match”.
- However, the core category still matches: both are long-sleeve shirts.
- In e-commerce search, users may still click this item because the style and wearing scenario are similar.
- Therefore, it should be labeled as “High Relevant”: not the exact target, but a good substitute.
Detailed case:
- Query: “black mid-length skirt”
- Product: “New spring autumn loose slimming full long floral skirt pleated skirt”
Analysis:
- Category match: the product is a skirt, so the category matches.
- Color mismatch: the product description does not indicate black and explicitly mentions “floral”, which is substantially different from plain black.
- Length deviation: the user asks for “mid-length”, while the product title emphasizes “long skirt”, which is somewhat longer.
- However, the core category “skirt” still matches, and style features such as “slimming” and “full skirt” may still fit some preferences of users searching for a mid-length skirt. Also, “long” versus “mid-length” is a deviation, but not a severe contradiction.
- Therefore, this should be labeled as “High Relevant”: the core type matches, but there are several non-fatal attribute deviations.
### Low Relevant
The product has a noticeable gap from the user’s core target, but still shares some similarity with the query in style, scenario, function, or broader category. A small portion of users may still view it as a barely acceptable substitute. It is not the intended item, but still has some relevance.
Use “Low Relevant” in the following cases:
- The core product type does not match, but the two types are still very close in style, wearing scenario, or function, so there is still some substitutability.
- The core product type matches, but the product differs from the user’s ideal target on multiple attributes; it still has some relevance, but is no longer a strong substitute.
- An important query requirement is clearly violated, but the product still retains a limited reason to be clicked.
Typical cases:
- Query: “black mid-length skirt”
Product: “New high-waisted V-neck mid-length dress elegant printed black sexy dress”
→ The core product type differs (“skirt” vs “dress”), but both belong to closely related apparel types and share a similar mid-length style, so it is “Low Relevant”.
- Query: “jeans”
Product: “casual pants”
→ The core product type is different, but both belong to the broader pants category, and the style / wearing scenario may still be close enough to be a weak substitute.
### Irrelevant
The product does not satisfy the user’s main shopping intent, and the likelihood of user engagement is very low.
Typical situations:
- The core product type does not match the query and is not a close substitute in style, scenario, or function.
- The product belongs to a roughly related broader category, but not to an interchangeable subtype explicitly requested in the query, and the style or usage scenario differs significantly.
- The core product type matches, but the product clearly violates an explicit and important requirement in the query, with little or no acceptable substitutability.
Typical examples:
1. **Different core product type with no substitutability**
- Query: "pants", Product: "shoes"
- Query: "boots", Product: "sneakers"
2. **Close core product type but clear conflict on key attributes**
- Query: "slim-fit pants", Product: "loose wide-leg pants"
- Query: "sleeveless dress", Product: "long-sleeve dress"
- Query: "loose-fitting hoodie", Product: "slim-fit T-shirt"
3. **Same broad category but significant difference in style or usage scenario**
- Query: "jeans", Product: "sweatpants / dress pants"
## Judgment Principles
1. **Product type is the highest-priority factor.**
If the query explicitly specifies a concrete product type, the result must match that product type in order to be labeled as “Exact Match” or “High Relevant”.
Different product types should usually be labeled as “Low Relevant” or “Irrelevant”.
- **Low Relevant**: use only when the two product types are very close in style, scenario, or function, and the user may still treat one as a barely acceptable substitute for the other.
- **Irrelevant**: all other product type mismatch cases.
2. **Similar or related product types are usually not directly interchangeable when the query is explicit, but their closeness should determine whether the label is “Low Relevant” or “Irrelevant”.**
For example:
- **May be Low Relevant due to strong similarity in style / scenario**: dress vs skirt, long skirt vs mid-length skirt, jeans vs casual pants, sneakers vs skate shoes.
- **Should be Irrelevant due to substantial difference in style / scenario**: pants vs shoes, T-shirt vs hat, boots vs sneakers, jeans vs suit pants, backpack vs handbag.
3. **Once the core product type matches, evaluate attributes.**
- All explicit attributes match → **Exact Match**
- Some attributes are missing, not mentioned, cannot be verified, or show only minor deviations → **High Relevant**
- There are multiple attribute deviations, or an important attribute is clearly violated, but the product still retains some substitutability → **Low Relevant**
- There is a clear and important hard conflict, and substitutability is extremely low → **Irrelevant**
4. **Strictly distinguish among “not mentioned / cannot confirm”, “minor deviation”, and “explicit contradiction”.**
- If an attribute is not mentioned or cannot be verified, prefer **High Relevant**.
- If an attribute shows a minor deviation, such as different color, different material, or slightly different length, it should usually be labeled **High Relevant**.
- If an attribute is explicitly opposite to the query requirement, such as sleeveless vs long-sleeve or slim-fit vs loose wide-leg, decide between **Low Relevant** and **Irrelevant** based on the severity of the conflict and practical substitutability.
- If the conflict directly breaks the user’s main shopping goal, it should usually be labeled **Irrelevant**.
5. **Substitutability should be judged from real shopping intent, not just surface-level textual similarity.**
The question is whether the user would realistically accept the product in a shopping scenario.
- Good substitute → **High Relevant**
- Barely acceptable substitute → **Low Relevant**
- Hardly substitutable at all → **Irrelevant**
|