tangwang / SearchEngine

08 Dec, 2025

1 commit

07 Dec, 2025

2 commits

  • 0064e946   feat: 增量索引服务、租户配置和翻译功能集成 ... Browse File » 
    主要功能：
1. 增量数据获取服务
   - 新增 IncrementalIndexerService 提供单个SPU数据获取
   - 新增 /indexer/spu/{spu_id} API接口
   - 服务启动时预加载分类映射等公共数据
   - 提取 SPUDocumentTransformer 统一全量和增量转换逻辑
   - 支持根据租户配置进行语言处理和翻译

3. 租户配置系统
   - 租户配置合并到统一配置文件 config/config.yaml
   - 支持每个租户独立配置主语言和翻译选项
   - 租户162配置为翻译关闭（用于测试）

4. 翻译功能集成
   - 翻译提示词作为DeepL API的context参数传递
   - 支持中英文提示词配置
   - 索引场景：同步翻译，使用缓存
   - 查询场景：异步翻译，立即返回

测试：
- 新增 indexer/test_indexing.py 和 query/test_translation.py
- 验证租户162翻译关闭功能
- 验证全量和增量索引功能
    tangwang
     
  • 6e0e310c   1. Translator 类增强 ... Browse File » 
    添加 ThreadPoolExecutor 线程池用于异步翻译
修改 translate_multi() 方法，支持 async_mode 参数（默认 True）
添加 _get_cached_translation() 方法，快速获取缓存
添加 _translate_async() 方法，异步执行翻译任务
2. 异步翻译逻辑
命中缓存：立即返回缓存结果
未命中缓存：
异步启动翻译任务（不阻塞）
返回 None（本次查询不使用）
翻译完成后自动存入缓存
下次查询时可直接使用缓存结果
3. QueryParser 更新
调用 translate_multi() 时使用 async_mode=True
过滤掉 None 值（未完成的翻译）
优化日志输出，区分缓存命中和异步翻译
工作流程
首次查询：未命中缓存 → 异步翻译 → 返回空翻译 → 不阻塞
翻译完成：结果存入缓存
后续查询：命中缓存 → 立即返回 → 快速响应
    tangwang
     

14 Nov, 2025

1 commit

13 Nov, 2025

1 commit

11 Nov, 2025

1 commit

  • 16c42787   feat: implement request-scoped context management with structured logging ... Browse File » 
    ## 🎯 Major Features
- Request context management system for complete request visibility
- Structured JSON logging with automatic daily rotation
- Performance monitoring with detailed stage timing breakdowns
- Query analysis result storage and intermediate result tracking
- Error and warning collection with context correlation

## 🔧 Technical Improvements
- **Context Management**: Request-level context with reqid/uid correlation
- **Performance Monitoring**: Automatic timing for all search pipeline stages
- **Structured Logging**: JSON format logs with request context injection
- **Query Enhancement**: Complete query analysis tracking and storage
- **Error Handling**: Enhanced error tracking with context information

## 🐛 Bug Fixes
- Fixed DeepL API endpoint (paid vs free API confusion)
- Fixed vector generation (GPU memory cleanup)
- Fixed logger parameter passing format (reqid/uid handling)
- Fixed translation and embedding functionality

## 🌟 API Improvements
- Simplified API interface (8→5 parameters, 37.5% reduction)
- Made internal functionality transparent to users
- Added performance info to API responses
- Enhanced request correlation and tracking

## 📁 New Infrastructure
- Comprehensive test suite (unit, integration, API tests)
- CI/CD pipeline with automated quality checks
- Performance monitoring and testing tools
- Documentation and example usage guides

## 🔒 Security & Reliability
- Thread-safe context management for concurrent requests
- Automatic log rotation and structured output
- Error isolation with detailed context information
- Complete request lifecycle tracking

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
    tangwang
     

08 Nov, 2025

2 commits