商品数据源入ES配置规范.md
8.92 KB
根据您提供的内容，我将其整理为规范的Markdown格式：
ES索引配置文档
1. 全局配置
1.1 文本字段相关性设定
需要修改所有text字段相关性算法-BM25算法的默认参数：
"similarity": {
"default": {
"type": "BM25",
"b": "0.0",
"k1": "0.0"
}
}
1.2 索引分片设定
number_of_replicas：0/1number_of_shards：设置建议 分片数
1.3 索引刷新时间设定
refresh_interval：默认30S，根据客户需要进行调整json "refresh_interval": "30s"
2. 单个字段配置
| 分析方式 | 字段预处理和ES输入格式要求 | 对应ES mapping配置 | 备注 |
|---|---|---|---|
| 电商通用分析-中文 | - | json { "type": "text", "analyzer": "index_ansj", "search_analyzer": "query_ansj" } |
- |
| 文本-多语言向量化 | 调用"文本向量化"模块得到1024维向量 | json { "type": "dense_vector", "dims": 1024, "index": true, "similarity": "dot_product" } |
1. 依赖"文本向量化"模块2. 如果定期全量，需要对向量化结果做缓存 |
| 图片-向量化 | 调用"图片向量化"模块得到1024维向量 | json { "type": "nested", "properties": { "vector": { "type": "dense_vector", "dims": 1024, "similarity": "dot_product" }, "url": { "type": "text" } } } |
1. 依赖"图片向量化"模块2. 如果定期全量，需要对向量化结果做缓存 |
| 关键词 | ES输入格式：list或者单个值 | json {"type": "keyword"} |
- |
| 电商通用分析-英文 | - | json {"type": "text", "analyzer": "english"} |
- |
| 电商通用分析-阿拉伯文 | - | json {"type": "text", "analyzer": "arabic"} |
- |
| 电商通用分析-西班牙文 | - | json {"type": "text", "analyzer": "spanish"} |
- |
| 电商通用分析-俄文 | - | json {"type": "text", "analyzer": "russian"} |
- |
| 电商通用分析-日文 | - | json {"type": "text", "analyzer": "japanese"} |
- |
| 数值-整数 | - | json {"type": "long"} |
- |
| 数值-浮点型 | - | json {"type": "float"} |
- |
| 分值 | 输入是float，配置处理方式：log, pow, sigmoid等 | TODO：给代码, log | - |
| 子串 | - | 暂时不支持 | - |
| ngram匹配或前缀匹配或边缘前缀匹配 | - | 暂时不支持 | 以后根据需要再添加 |
这样整理后，文档结构更加清晰，表格格式规范，便于阅读和理解。
参考 opensearch：
数据接口 文本相关性字段 向量相关性字段
- 模块提取 文本向量化 import sys import torch from sentence_transformers import SentenceTransformer import time import threading from modelscope import snapshot_download from transformers import AutoModel import os from openai import OpenAI from config.logging_config import get_app_logger
Get logger for this module
logger = get_app_logger(name)
class BgeEncoder: _instance = None _lock = threading.Lock()
def __new__(cls, model_dir='Xorbits/bge-m3'):
with cls._lock:
if cls._instance is None:
cls._instance = super(BgeEncoder, cls).__new__(cls)
logger.info("[BgeEncoder] Creating a new instance with model directory: %s", model_dir)
cls._instance.model = SentenceTransformer(snapshot_download(model_dir))
logger.info("[BgeEncoder] New instance has been created")
return cls._instance
def encode(self, sentences, normalize_embeddings=True, device='cuda'):
# Move model to specified device
if device == 'gpu':
device = 'cuda'
self.model = self.model.to(device)
embeddings = self.model.encode(sentences, normalize_embeddings=normalize_embeddings, device=device, show_progress_bar=False)
return embeddings
图片向量化 import sys import os import io import requests import torch import numpy as np from PIL import Image import logging import threading from typing import List, Optional, Union from config.logging_config import get_app_logger import cn_clip.clip as clip from cn_clip.clip import load_from_name
Get logger for this module
logger = get_app_logger(name)
DEFAULT_MODEL_NAME = "ViT-L-14-336" # ["ViT-B-16", "ViT-L-14", "ViT-L-14-336", "ViT-H-14", "RN50"]
DEFAULT_MODEL_NAME = "ViT-H-14" MODEL_DOWNLOAD_DIR = "/data/tw/uat/EsSearcher"
class CLIPImageEncoder: """CLIP Image Encoder for generating image embeddings using cn_clip"""
_instance = None
_lock = threading.Lock()
def __new__(cls, model_name=DEFAULT_MODEL_NAME, device=None):
with cls._lock:
if cls._instance is None:
cls._instance = super(CLIPImageEncoder, cls).__new__(cls)
logger.info(f"[CLIPImageEncoder] Creating new instance with model: {model_name}")
cls._instance._initialize_model(model_name, device)
return cls._instance
def _initialize_model(self, model_name, device):
"""Initialize the CLIP model using cn_clip"""
try:
self.device = device if device else ("cuda" if torch.cuda.is_available() else "cpu")
self.model, self.preprocess = load_from_name(model_name, device=self.device, download_root=MODEL_DOWNLOAD_DIR)
self.model.eval()
self.model_name = model_name
logger.info(f"[CLIPImageEncoder] Model {model_name} initialized successfully on device {self.device}")
except Exception as e:
logger.error(f"[CLIPImageEncoder] Failed to initialize model: {str(e)}")
raise
def validate_image(self, image_data: bytes) -> Image.Image:
"""Validate image data and return PIL Image if valid"""
try:
image_stream = io.BytesIO(image_data)
image = Image.open(image_stream)
image.verify()
image_stream.seek(0)
image = Image.open(image_stream)
if image.mode != 'RGB':
image = image.convert('RGB')
return image
except Exception as e:
raise ValueError(f"Invalid image data: {str(e)}")
def download_image(self, url: str, timeout: int = 10) -> bytes:
"""Download image from URL"""
try:
if url.startswith(('http://', 'https://')):
response = requests.get(url, timeout=timeout)
if response.status_code != 200:
raise ValueError(f"HTTP {response.status_code}")
return response.content
else:
# Local file path
with open(url, 'rb') as f:
return f.read()
except Exception as e:
raise ValueError(f"Failed to download image from {url}: {str(e)}")
def preprocess_image(self, image: Image.Image, max_size: int = 1024) -> Image.Image:
"""Preprocess image for CLIP model"""
# Resize if too large
if max(image.size) > max_size:
ratio = max_size / max(image.size)
new_size = tuple(int(dim * ratio) for dim in image.size)
image = image.resize(new_size, Image.Resampling.LANCZOS)
return image
def encode_text(self, text):
"""Encode text to embedding vector using cn_clip"""
text_data = clip.tokenize([text] if type(text) == str else text).to(self.device)
with torch.no_grad():
text_features = self.model.encode_text(text_data)
text_features /= text_features.norm(dim=-1, keepdim=True)
return text_features
def encode_image(self, image: Image.Image) -> Optional[np.ndarray]:
"""Encode image to embedding vector using cn_clip"""
if not isinstance(image, Image.Image):
raise ValueError("CLIPImageEncoder.encode_image Input must be a PIL.Image")
try:
infer_data = self.preprocess(image).unsqueeze(0).to(self.device)
with torch.no_grad():
image_features = self.model.encode_image(infer_data)
image_features /= image_features.norm(dim=-1, keepdim=True)
return image_features.cpu().numpy().astype('float32')[0]
except Exception as e:
logger.error(f"Failed to process image. Reason: {str(e)}")
return None
def encode_image_from_url(self, url: str) -> Optional[np.ndarray]:
"""Complete pipeline: download, validate, preprocess and encode image from URL"""
try:
# Download image
image_data = self.download_image(url)
# Validate image
image = self.validate_image(image_data)
# Preprocess image
image = self.preprocess_image(image)
# Encode image
embedding = self.encode_image(image)
return embedding
except Exception as e:
logger.error(f"Error processing image from URL {url}: {str(e)}")
return None