# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

This is a **Search Engine SaaS** platform for e-commerce product search, specifically designed for Shoplazza (店匠) independent sites. It's a multi-tenant configurable search system built on Elasticsearch with AI-powered search capabilities.

**Tech Stack:**
- Elasticsearch 8.x as the search engine backend
- MySQL (Shoplazza database) as the primary data source
- Python 3.10 with PyTorch/CUDA support
- BGE-M3 model for text embeddings (1024-dim vectors)
- CN-CLIP (ViT-H-14) for image embeddings
- FastAPI for REST API layer

## Development Environment

**Required Environment Setup:**
```bash
source /home/tw/miniconda3/etc/profile.d/conda.sh
conda activate searchengine
```

**Database Configuration:**
```
host: 120.79.247.228
port: 3316
database: saas
username: saas
password: P89cZHS5d7dFyc9R
```

## Common Development Commands

### Environment Setup
```bash
# Complete environment setup
./setup.sh

# Install Python dependencies
pip install -r requirements.txt
```

### Data Management
```bash
# Generate test data (Tenant1 Mock + Tenant2 CSV)
./scripts/mock_data.sh

# Ingest data to Elasticsearch
./scripts/ingest.sh <tenant_id> [recreate]  # e.g., ./scripts/ingest.sh 1 true
python main.py ingest data.csv --limit 1000 --batch-size 50
```

### Running Services
```bash
# Start all services (production)
./run.sh

# Start development server with auto-reload
./scripts/start_backend.sh
python main.py serve --host 0.0.0.0 --port 6002 --reload

# Start frontend debugging UI
./scripts/start_frontend.sh
```

### Testing
```bash
# Run all tests
python -m pytest tests/

# Run specific test types
python -m pytest tests/unit/          # Unit tests
python -m pytest tests/integration/   # Integration tests
python -m pytest -m "api"             # API tests only

# Test search from command line
python main.py search "query" --tenant-id 1 --size 10
```

### Development Utilities
```bash
# Stop all services
./scripts/stop.sh

# Test environment (for CI/development)
./scripts/start_test_environment.sh
./scripts/stop_test_environment.sh

# Install server dependencies
./scripts/install_server_deps.sh
```

## Architecture Overview

### Core Components
```
/data/tw/SearchEngine/
├── api/              # FastAPI REST API service (port 6002)
├── config/           # Configuration management system
├── indexer/          # MySQL → Elasticsearch data pipeline
├── search/           # Search engine and ranking logic
├── query/            # Query parsing, translation, rewriting
├── embeddings/       # ML models (BGE-M3, CN-CLIP)
├── scripts/          # Automation and utility scripts
├── utils/            # Shared utilities (ES client, etc.)
├── frontend/         # Simple debugging UI
├── mappings/         # Elasticsearch index mappings
└── tests/            # Unit and integration tests
```

### Data Flow Architecture
**Pipeline**: MySQL → Indexer → Elasticsearch → API → Frontend

1. **Data Source Layer**:
   - Shoplazza MySQL database with `shoplazza_product_sku` and `shoplazza_product_spu` tables
   - Tenant-specific extension tables for custom attributes and multi-language fields

2. **Indexing Layer** (`indexer/`):
   - Reads from MySQL, applies transformations with embeddings
   - Uses `DataTransformer` and `IndexingPipeline` for batch processing
   - Supports both full and incremental indexing with embedding caching

3. **Query Processing Layer** (`query/`):
   - `QueryParser`: Handles query rewriting, translation, and text embedding conversion
   - Multi-language support with automatic detection and translation
   - Boolean logic parsing with operator precedence: `()` > `ANDNOT` > `AND` > `OR` > `RANK`

4. **Search Engine Layer** (`search/`):
   - `Searcher`: Executes hybrid searches combining BM25 and dense vectors
   - Configurable ranking expressions with function_score support
   - Multi-tenant isolation via `tenant_id` field

5. **API Layer** (`api/`):
   - FastAPI service on port 6002 with multi-tenant support
   - Text search: `POST /search/`
   - Image search: `POST /image-search/`
   - Tenant identification via `X-Tenant-ID` header

### Multi-Tenant Configuration System

The system uses centralized configuration through `config/config.yaml`:

1. **Field Configuration** (`config/field_types.py`):
   - Defines field types: TEXT, KEYWORD, EMBEDDING, INT, DOUBLE, etc.
   - Specifies analyzers: Chinese (ansj), English, Arabic, Spanish, Russian, Japanese
   - Required fields and preprocessing rules

2. **Index Configuration** (`mappings/search_products.json`):
   - Unified index structure shared by all tenants
   - Elasticsearch field mappings and analyzer configurations
   - BM25 similarity with modified parameters (`b=0.0, k1=0.0`)

3. **Query Configuration** (`search/query_config.py`):
   - Query domain definitions (default, category_name, title, brand_name, etc.)
   - Ranking expressions and function_score configurations
   - Translation and embedding settings

### Embedding Models

**Text Embedding** (`embeddings/bge_encoder.py`):
- Uses BGE-M3 model (`Xorbits/bge-m3`)
- Singleton pattern with thread-safe initialization
- Generates 1024-dimensional vectors with GPU/CUDA support
- Configurable caching to avoid recomputation

**Image Embedding** (`embeddings/clip_encoder.py`):
- Uses CN-CLIP model (ViT-H-14)
- Downloads and preprocesses images from URLs
- Supports both local and remote image processing
- Generates 1024-dimensional vectors

### Search and Ranking

**Hybrid Search Approach**:
- Combines traditional BM25 text relevance with dense vector similarity
- Supports text embeddings (BGE-M3) and image embeddings (CN-CLIP)
- Configurable ranking expressions like: `static_bm25() + 0.2*text_embedding_relevance() + general_score*2 + timeliness(end_time)`

**Boolean Search Support**:
- Full boolean logic with AND, OR, ANDNOT, RANK operators
- Parentheses for complex query structures
- Configurable operator precedence

**Faceted Search**:
- Terms and range faceting support
- Multi-dimensional filtering capabilities
- Configurable facet fields and aggregations

## Testing Infrastructure

**Test Framework**: pytest with async support

**Test Structure**:
- `tests/conftest.py`: Comprehensive test fixtures and configuration
- `tests/unit/`: Unit tests for individual components
- `tests/integration/`: Integration tests for system workflows
- Test markers: `@pytest.mark.unit`, `@pytest.mark.integration`, `@pytest.mark.api`

**Test Data**:
- Tenant1: Mock data with 10,000 product records
- Tenant2: CSV-based test dataset
- Automated test data generation via `scripts/mock_data.sh`

**Key Test Fixtures** (from `conftest.py`):
- `sample_search_config`: Complete configuration for testing
- `mock_es_client`: Mocked Elasticsearch client
- `test_searcher`: Searcher instance with mock dependencies
- `temp_config_file`: Temporary YAML configuration for tests

## API Endpoints

**Main API** (FastAPI on port 6002):
- `POST /search/` - Text search with multi-language support
- `POST /image-search/` - Image search using CN-CLIP embeddings
- Health check and management endpoints
- Multi-tenant support via `X-Tenant-ID` header

**API Features**:
- Hybrid search combining text and vector similarity
- Configurable ranking and filtering
- Faceted search with aggregations
- Multi-language query processing and translation
- Real-time search with configurable result sizes

## Key Implementation Details

1. **Environment Variables**: All sensitive configuration stored in `.env` (template: `.env.example`)
2. **Configuration Management**: Dynamic config loading through `config/config_loader.py`
3. **Error Handling**: Comprehensive error handling with proper HTTP status codes
4. **Performance**: Batch processing for indexing, embedding caching, and connection pooling
5. **Logging**: Structured logging with request tracing for debugging
6. **Security**: Tenant isolation at the index level with proper access controls

## Database Tables

**Main Tables**:
- `shoplazza_product_sku` - SKU level product data with pricing and inventory
- `shoplazza_product_spu` - SPU level product data with categories and attributes
- Tenant extension tables for custom fields and multi-language content

**Data Processing**:
- Full data sync handled by separate Java project (not in this repo)
- This repository includes test implementations for development and debugging
- Extension tables joined with main tables during indexing process