# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

This is a comprehensive **Recommendation System** built for a B2B e-commerce platform. The system generates offline recommendation indices including item-to-item similarity (i2i) and interest aggregation indices, supporting online recommendation services with high-performance algorithms.

**Tech Stack**: Python 3.x, Pandas, NumPy, NetworkX, Gensim, C++ (Swing algorithm), Redis, Elasticsearch, MySQL

## 🏗️ System Architecture

### High-Level Components

```
recommendation/
├── offline_tasks/          # Main offline processing engine
├── graphembedding/        # Graph-based embedding algorithms  
├── refers/                # Reference materials and data
├── config.py             # Global configuration
└── requirements.txt      # Python dependencies
```

### Core Modules

#### 1. Offline Tasks (`/offline_tasks/`)

**Primary Purpose**: Generate recommendation indices through various ML algorithms

**Key Features**:
- **4 i2i similarity algorithms**: Swing (C++ & Python), Session W2V, DeepWalk, Content-based
- **11-dimension interest aggregation**: Platform, client, category, supplier dimensions
- **Automated pipeline**: One-command execution with memory monitoring
- **High-performance C++ integration**: 10-100x faster Swing implementation

**Directory Structure**:
```
offline_tasks/
├── scripts/               # All algorithm implementations
│   ├── fetch_item_attributes.py    # Preprocessing: item metadata
│   ├── generate_session.py         # Preprocessing: user sessions
│   ├── i2i_swing.py                # Swing algorithm (Python)
│   ├── i2i_session_w2v.py          # Session Word2Vec
│   ├── i2i_deepwalk.py             # DeepWalk with tag-based walks
│   ├── i2i_content_similar.py      # Content-based similarity
│   ├── interest_aggregation.py     # Multi-dimensional aggregation
│   └── load_index_to_redis.py      # Redis import
├── collaboration/         # C++ Swing algorithm (high-performance)
│   ├── src/                        # C++ source files
│   ├── run.sh                      # Build and execute script
│   └── output/                     # C++ algorithm outputs
├── config/
│   └── offline_config.py           # Configuration file
├── doc/                          # Comprehensive documentation
├── output/                       # Generated indices
├── logs/                         # Execution logs
├── run.sh                        # Main execution script (⭐ RECOMMENDED)
└── README.md                     # Module documentation
```

#### 2. Graph Embedding (`/graphembedding/`)

**Purpose**: Advanced graph-based embedding algorithms for content-aware recommendations

**Components**:
- **DeepWalk**: Enhanced with tag-based random walks for diversity
- **Session W2V**: Session-based word embeddings
- **Improvements**: Tag-based walks, Softmax sampling, multi-process support

#### 3. Configuration (`/config.py`)

**Global Settings**:
- **Elasticsearch**: Host, credentials, index configuration
- **Redis**: Cache configuration, timeouts, expiration policies
- **Database**: External database connection parameters

## 🚀 Development Workflow

### Quick Start

```bash
# 1. Install dependencies
cd /data/tw/recommendation/offline_tasks
bash install.sh

# 2. Test connections
python3 test_connection.py

# 3. Run full pipeline (recommended)
bash run.sh

# 4. Run individual algorithms
python3 scripts/i2i_swing.py --lookback_days 730 --debug
python3 scripts/interest_aggregation.py --lookback_days 730 --top_n 1000
```

### Common Development Commands

**Setup and Installation:**
```bash
# Install Python dependencies
pip install -r requirements.txt

# Build C++ Swing algorithm
cd offline_tasks/collaboration && make

# Activate conda environment (required)
conda activate tw
```

**Testing:**
```bash
# Test database and Redis connections
python3 offline_tasks/test_connection.py

# Test Elasticsearch connection
python3 offline_tasks/scripts/test_es_connection.py
```

**Build and Compilation:**
```bash
# Build C++ algorithms
cd offline_tasks/collaboration
make clean && make

# Clean build artifacts
make clean
```

**Running Individual Components:**
```bash
# Generate session data
python3 offline_tasks/scripts/generate_session.py --lookback_days 730 --debug

# Run C++ Swing algorithm
cd offline_tasks/collaboration && bash run.sh

# Load indices to Redis
python3 offline_tasks/scripts/load_index_to_redis.py --redis-host localhost --redis-port 6379
```

### Algorithm Execution Order

The system follows this optimized execution pipeline:

1. **Preprocessing Tasks** (Run once per session)
   - `fetch_item_attributes.py` → Item metadata mapping
   - `generate_session.py` → User behavior sessions

2. **Core Algorithms**
   - C++ Swing (`collaboration/run.sh`) → High-performance similarity
   - Python Swing (`i2i_swing.py`) → Time-aware similarity
   - Session W2V (`i2i_session_w2v.py`) → Sequence-based similarity
   - DeepWalk (`i2i_deepwalk.py`) → Graph-based embeddings
   - Content Similarity (`i2i_content_similar.py`) → Attribute-based

3. **Post-processing**
   - Interest Aggregation (`interest_aggregation.py`) → Multi-dimensional indices
   - Redis Import (`load_index_to_redis.py`) → Online serving

### Key Configuration Files

#### Main Configuration (`offline_config.py`)
```python
# Critical settings
DEFAULT_LOOKBACK_DAYS = 730    # Historical data window
DEFAULT_I2I_TOP_N = 50         # Similar items per product
DEFAULT_INTEREST_TOP_N = 1000   # Aggregated items per dimension

# Algorithm parameters
I2I_CONFIG = {
    'swing': {'alpha': 0.5, 'threshold1': 0.5, 'threshold2': 0.5},
    'session_w2v': {'vector_size': 128, 'window_size': 5},
    'deepwalk': {'num_walks': 10, 'walk_length': 40}
}

# Behavior weights for different user actions
behavior_weights = {
    'purchase': 10.0,
    'contactFactory': 5.0,
    'addToCart': 3.0,
    'addToPool': 2.0
}
```

#### Database Configuration (`config.py`)
```python
# External database
DB_CONFIG = {
    'host': 'selectdb-cn-wuf3vsokg05-public.selectdbfe.rds.aliyuncs.com',
    'port': '9030',
    'database': 'datacenter',
    'username': 'readonly',
    'password': 'essa1234'
}

# Redis for online serving
REDIS_CONFIG = {
    'host': 'localhost',
    'port': 6379,
    'cache_expire_days': 180
}
```

## 🔧 Key Algorithms & Features

### 1. Swing Algorithm (Dual Implementation)

**C++ Version** (Production):
- **Performance**: 10-100x faster than Python
- **Use Case**: Large-scale production processing
- **Output**: Raw similarity scores
- **Location**: `collaboration/`

**Python Version** (Development/Enhanced):
- **Features**: Time decay, daily session support
- **Use Case**: Development, debugging, parameter tuning
- **Output**: Normalized scores with readable names
- **Location**: `scripts/i2i_swing.py`

### 2. DeepWalk with Tag Enhancement

**Innovative Features**:
- **Tag-based walks**: 20% probability of content-guided walks
- **Softmax sampling**: Temperature-controlled diversity
- **Multi-process**: Parallel walk generation
- **Purpose**: Solves recommendation homogeneity issues

### 3. Interest Aggregation

**Multi-dimensional Support**:
- **7 single dimensions**: platform, client_platform, supplier, category_level1-4
- **4 combined dimensions**: platform_client, platform_category2/3, client_category2
- **3 list types**: hot (popular), cart (cart additions), new (recent), global (overall)

## 📊 Data Pipeline

### Input Data Sources
- **User Behavior**: Purchase, contact, cart, pool interactions
- **Item Metadata**: Categories, suppliers, attributes
- **Session Data**: Time-weighted user behavior sequences

### Output Formats
```
# i2i Similarity (item-to-item)
item_id \t similar_id1:score1,similar_id2:score2,...

# Interest Aggregation  
dimension:value \t item_id1,item_id2,item_id3,...

# Redis Keys
item:similar:swing_cpp:12345
interest:hot:platform:pc
```

### Storage Architecture
- **Redis**: Fast online serving (400MB memory footprint)
- **Elasticsearch**: Vector similarity search
- **Local Files**: Raw algorithm outputs for debugging

## 🐛 Development Guidelines

### Adding New Algorithms

1. **Create script in `scripts/`**:
   ```python
   import from db_service, config.offline_config, debug_utils
   Follow existing pattern: fetch_data → process → save_output
   ```

2. **Update configuration** in `offline_config.py`:
   ```python
   NEW_ALGORITHM_CONFIG = {
       'param1': value1,
       'param2': value2
   }
   ```

3. **Add to execution pipeline** in `run.sh` or `run_all.py`

### Debugging Practices

- **Use debug mode**: `--debug` flag for readable outputs
- **Check logs**: `logs/run_all_YYYYMMDD.log`
- **Validate data**: `debug_utils.py` provides data validation
- **Monitor memory**: System includes memory monitoring

### Performance Optimization

- **Database optimization**: Preprocessing reduces queries by 80-90%
- **C++ integration**: Critical for production performance
- **Parallel processing**: Multi-threaded algorithms
- **Memory management**: Configurable thresholds and monitoring

### Code Quality

This codebase does not have formal linting or testing frameworks configured. When making changes:

- **Python**: Follow PEP 8 style guidelines
- **C++**: Use the existing coding style in collaboration/src/
- **No formal unit tests**: Test functionality manually using the debug modes
- **Manual testing**: Use `--debug` flags for readable outputs during development

## 🔄 Maintenance & Operations

### Daily Execution
```bash
# Recommended production command
0 2 * * * cd /data/tw/recommendation/offline_tasks && bash run.sh
```

### Monitoring
- **Logs**: `logs/` directory with date-based rotation
- **Memory**: Built-in memory monitoring with kill thresholds
- **Output Validation**: Automated data quality checks
- **Error Handling**: Comprehensive logging and recovery

### Backup Strategy
- **Output files**: Daily snapshots in `output/`
- **Configuration**: Version-controlled configs
- **Logs**: 180-day retention with cleanup

## 🎯 Key Architecture Decisions

### 1. Hybrid Algorithm Approach
- **Problem**: Python Swing too slow for production (can take hours)
- **Solution**: C++ core for performance + Python for flexibility and debugging
- **Benefit**: C++ version is 10-100x faster, Python version provides enhanced features and readability

### 2. Preprocessing Optimization
- **Problem**: Repeated database queries across algorithms
- **Solution**: Centralized metadata and session generation via `fetch_item_attributes.py` and `generate_session.py`
- **Benefit**: 80-90% reduction in database load

### 3. Multi-dimensional Interest Aggregation
- **Problem**: Need for flexible recommendation personalization
- **Solution**: 11 dimensions with 3 list types each
- **Benefit**: Supports diverse business scenarios

### 4. Tag-enhanced DeepWalk
- **Problem**: Recommendation homogeneity
- **Solution**: Content-aware random walks
- **Benefit**: Improved diversity and serendipity

### 5. Environment Management
- **Problem**: Dependency isolation and reproducibility
- **Solution**: Conda environment named `tw`
- **Benefit**: Consistent Python environment across development and production

## 📚 Documentation Resources

### Core Documentation
- **[offline_tasks/doc/详细设计文档.md](offline_tasks/doc/详细设计文档.md)** - Complete system architecture
- **[offline_tasks/doc/离线索引数据规范.md](offline_tasks/doc/离线索引数据规范.md)** - Data format specifications
- **[offline_tasks/doc/Redis数据规范.md](offline_tasks/doc/Redis数据规范.md)** - Redis integration guide
- **[offline_tasks/README.md](offline_tasks/README.md)** - Quick start guide

### Algorithm Documentation
- **[graphembedding/deepwalk/README.md](graphembedding/deepwalk/README.md)** - DeepWalk with tag enhancements
- **[collaboration/README.md](collaboration/README.md)** - C++ Swing algorithm
- **[collaboration/Swing快速开始.md](collaboration/Swing快速开始.md)** - Swing implementation guide

## 🚨 Important Notes for Development

1. **Environment**: Uses Conda environment `tw` - activate before running
2. **Database**: Read-only access to external database
3. **Redis**: Local instance for development, configurable for production
4. **Memory**: Algorithms are memory-intensive - monitor usage
5. **Output**: All files include date stamps for versioning
6. **Testing**: Always test with small datasets before production runs

## 🔗 Related Components

- **Online Services**: Redis-based recommendation serving
- **Elasticsearch**: Vector similarity search capabilities  
- **Frontend APIs**: Recommendation interfaces for different platforms
- **Monitoring**: Performance metrics and error tracking

---

**Last Updated**: 2024-12-10  
**Maintained by**: Recommendation System Team  
**Status**: Production-ready with active development