# RFCP Iteration 3.4.0 — Large Radius Support (20-50km)

## Goal

Enable 50km radius calculations without OOM by implementing memory-efficient processing patterns.

**Current limitation:** > 10-20km radius causes OOM (5+ GB RAM usage)
**Target:** 50km radius with < 4GB RAM peak

---

## Phase 1: Memory-Mapped Terrain

### 1.1 Terrain mmap Loading

Change terrain_service to use memory-mapped files instead of loading full arrays into RAM.

**File:** `backend/app/services/terrain_service.py`

```python
# Before (loads ~25 MB per tile into RAM):
terrain = np.fromfile(f, dtype='>i2').reshape((rows, cols))

# After (near-zero RAM, OS pages from disk):
terrain = np.memmap(f, dtype='>i2', mode='r', shape=(rows, cols))
```

**Expected impact:** -200-400 MB RAM per tile

### 1.2 Terrain Disk Cache

- Save downloaded .hgt files to persistent disk cache
- Don't keep raw arrays in memory after initial processing
- Implement LRU eviction if cache exceeds 2GB
- Location: `~/.rfcp/terrain_cache/`

---

## Phase 2: Tile-Based Processing

### 2.1 Split Large Calculations

If radius > 10km, split calculation area into 5km sub-tiles.

**File:** `backend/app/services/coverage_service.py` (or new `tile_processor.py`)

```python
def calculate_coverage_tiled(site, radius_m, resolution_m, settings):
    """Tile-based calculation for large radius."""
    
    # Small radius — use existing single-pass
    if radius_m <= 10000:
        return calculate_coverage_single(site, radius_m, resolution_m, settings)
    
    # Large radius — split into tiles
    TILE_SIZE = 5000  # 5km tiles
    tiles = generate_tile_grid(site.lat, site.lon, radius_m, TILE_SIZE)
    
    all_results = []
    
    for i, tile in enumerate(tiles):
        log(f"Processing tile {i+1}/{len(tiles)}: {tile.bbox}")
        
        # Load data for this tile only
        tile_terrain = load_terrain_for_bbox(tile.bbox)
        tile_buildings = load_buildings_for_bbox(tile.bbox)
        
        # Calculate coverage for tile
        tile_points = generate_grid_for_tile(tile, resolution_m)
        tile_results = calculate_points(tile_points, site, settings,
                                        tile_terrain, tile_buildings)
        
        all_results.extend(tile_results)
        
        # Free memory
        del tile_terrain, tile_buildings
        gc.collect()
        
        # Report progress
        progress = (i + 1) / len(tiles) * 100
        yield_progress(progress, f"Tile {i+1}/{len(tiles)}")
    
    return merge_and_dedupe_results(all_results)


def generate_tile_grid(center_lat, center_lon, radius_m, tile_size_m):
    """Generate grid of tiles covering the calculation area."""
    tiles = []
    
    # Calculate bbox of full area
    lat_delta = radius_m / 111000
    lon_delta = radius_m / (111000 * cos(radians(center_lat)))
    
    # Generate tile grid
    n_tiles = ceil(radius_m * 2 / tile_size_m)
    
    for i in range(n_tiles):
        for j in range(n_tiles):
            tile_bbox = calculate_tile_bbox(center_lat, center_lon, 
                                           i, j, n_tiles, tile_size_m)
            
            # Only include tiles that intersect with coverage circle
            if tile_intersects_circle(tile_bbox, center_lat, center_lon, radius_m):
                tiles.append(Tile(bbox=tile_bbox, index=(i, j)))
    
    return tiles
```

### 2.2 Progressive Results via WebSocket

Send results per-tile as they complete, so user sees coverage growing.

**File:** `backend/app/api/websocket.py`

```python
async def calculate_coverage_ws(websocket, params):
    for tile_results in calculate_coverage_tiled_generator(params):
        # Send partial results
        await websocket.send_json({
            "type": "partial_results",
            "points": tile_results.points,
            "progress": tile_results.progress,
            "tile": tile_results.tile_index,
            "status": f"Tile {tile_results.tile_index} complete"
        })
    
    # Final message
    await websocket.send_json({
        "type": "complete",
        "total_points": total_points,
        "computation_time": elapsed
    })
```

---

## Phase 3: SQLite Cache for OSM Data

### 3.1 Create Local Database

Replace in-memory OSM cache with SQLite database with spatial indexing.

**File:** `backend/app/services/cache_db.py` (NEW)

```python
import sqlite3
import json

class OSMCacheDB:
    def __init__(self, db_path="~/.rfcp/osm_cache.db"):
        self.conn = sqlite3.connect(db_path)
        self._init_tables()
    
    def _init_tables(self):
        self.conn.executescript("""
            CREATE TABLE IF NOT EXISTS buildings (
                id INTEGER PRIMARY KEY,
                osm_id TEXT UNIQUE,
                lat REAL NOT NULL,
                lon REAL NOT NULL,
                height REAL DEFAULT 10.0,
                geometry TEXT,  -- GeoJSON
                cell_key TEXT,  -- grid cell for batch loading
                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
            );
            
            CREATE INDEX IF NOT EXISTS idx_buildings_lat ON buildings(lat);
            CREATE INDEX IF NOT EXISTS idx_buildings_lon ON buildings(lon);
            CREATE INDEX IF NOT EXISTS idx_buildings_cell ON buildings(cell_key);
            
            CREATE TABLE IF NOT EXISTS vegetation (
                id INTEGER PRIMARY KEY,
                osm_id TEXT UNIQUE,
                lat REAL NOT NULL,
                lon REAL NOT NULL,
                type TEXT,
                geometry TEXT,
                cell_key TEXT
            );
            
            CREATE INDEX IF NOT EXISTS idx_veg_lat ON vegetation(lat);
            CREATE INDEX IF NOT EXISTS idx_veg_lon ON vegetation(lon);
            
            -- Metadata for cache invalidation
            CREATE TABLE IF NOT EXISTS cache_meta (
                cell_key TEXT PRIMARY KEY,
                data_type TEXT,
                fetched_at TIMESTAMP,
                item_count INTEGER
            );
        """)
        self.conn.commit()
    
    def query_buildings_bbox(self, min_lat, max_lat, min_lon, max_lon, limit=20000):
        """Query buildings within bounding box."""
        cursor = self.conn.execute("""
            SELECT osm_id, lat, lon, height, geometry
            FROM buildings
            WHERE lat BETWEEN ? AND ?
            AND lon BETWEEN ? AND ?
            LIMIT ?
        """, (min_lat, max_lat, min_lon, max_lon, limit))
        
        return [self._row_to_building(row) for row in cursor]
    
    def insert_buildings(self, buildings, cell_key):
        """Bulk insert buildings from OSM fetch."""
        self.conn.executemany("""
            INSERT OR IGNORE INTO buildings 
            (osm_id, lat, lon, height, geometry, cell_key)
            VALUES (?, ?, ?, ?, ?, ?)
        """, [
            (b['id'], b['lat'], b['lon'], b.get('height', 10), 
             json.dumps(b.get('geometry')), cell_key)
            for b in buildings
        ])
        self.conn.commit()
    
    def is_cell_cached(self, cell_key, data_type, max_age_hours=24):
        """Check if cell data is cached and fresh."""
        cursor = self.conn.execute("""
            SELECT fetched_at FROM cache_meta
            WHERE cell_key = ? AND data_type = ?
            AND fetched_at > datetime('now', ?)
        """, (cell_key, data_type, f'-{max_age_hours} hours'))
        
        return cursor.fetchone() is not None
```

### 3.2 Update OSM Client

Modify OSM client to use SQLite cache.

**File:** `backend/app/services/osm_client.py`

```python
class OSMClient:
    def __init__(self):
        self.cache_db = OSMCacheDB()
    
    def get_buildings(self, bbox, max_count=20000):
        min_lat, min_lon, max_lat, max_lon = bbox
        cell_key = self._bbox_to_cell_key(bbox)
        
        # Check cache first
        if self.cache_db.is_cell_cached(cell_key, 'buildings'):
            return self.cache_db.query_buildings_bbox(
                min_lat, max_lat, min_lon, max_lon, max_count
            )
        
        # Fetch from Overpass API
        buildings = self._fetch_from_overpass(bbox, 'buildings')
        
        # Store in cache
        self.cache_db.insert_buildings(buildings, cell_key)
        
        return buildings[:max_count]
```

---

## Phase 4: Worker Memory Optimization

### 4.1 Per-Tile Building Loading

Workers receive only tile bbox and query buildings themselves (or receive pre-filtered list).

```python
def _pool_worker_tiled(args):
    """Worker that loads buildings for its tile only."""
    tile_bbox, terrain_shm_refs, config = args
    
    # Load only buildings for this tile
    cache_db = OSMCacheDB()
    buildings = cache_db.query_buildings_bbox(*tile_bbox, limit=5000)
    
    # Much smaller memory footprint per worker
    # ...rest of calculation
```

### 4.2 Adaptive Worker Count

Reduce workers for large radius to prevent combined memory explosion.

```python
def get_worker_count_for_radius(radius_m, base_workers):
    """Scale down workers for large calculations."""
    if radius_m > 30000:
        return min(base_workers, 2)
    elif radius_m > 20000:
        return min(base_workers, 3)
    elif radius_m > 10000:
        return min(base_workers, 4)
    return base_workers
```

---

## Phase 5: Frontend Progressive Rendering

### 5.1 Accumulate Partial Results

**File:** `frontend/src/store/coverage.ts`

```typescript
interface CoverageState {
    points: CoveragePoint[];
    isCalculating: boolean;
    progress: number;
    // NEW:
    partialResults: CoveragePoint[];
    tilesCompleted: number;
    totalTiles: number;
}

// Handle partial results
case 'partial_results':
    set(state => ({
        partialResults: [...state.partialResults, ...message.points],
        progress: message.progress,
        tilesCompleted: state.tilesCompleted + 1
    }));
    break;

case 'complete':
    set(state => ({
        points: state.partialResults,  // Finalize
        partialResults: [],
        isCalculating: false
    }));
    break;
```

### 5.2 Incremental Heatmap Render

**File:** `frontend/src/components/map/CoverageHeatmap.tsx`

```typescript
function CoverageHeatmap() {
    const { points, partialResults, isCalculating } = useCoverageStore();
    
    // Show partial results while calculating
    const displayPoints = isCalculating ? partialResults : points;
    
    // Throttle re-renders during streaming (every 500 points)
    const throttledPoints = useThrottle(displayPoints, 500);
    
    return <HeatmapLayer points={throttledPoints} />;
}
```

---

## Implementation Order

### Priority 1 — Biggest Impact
1. **Tile-based processing** (Phase 2.1) — enables large radius
2. **SQLite cache** (Phase 3) — reduces memory, speeds up repeated calcs

### Priority 2 — Memory Reduction
3. **Terrain mmap** (Phase 1.1) — easy win, minimal code change
4. **Per-tile building loading** (Phase 4.1)

### Priority 3 — UX Improvement
5. **Progressive WebSocket** (Phase 2.2)
6. **Frontend streaming** (Phase 5)

### Priority 4 — Polish
7. **Terrain disk cache** (Phase 1.2)
8. **Adaptive worker count** (Phase 4.2)

---

## Success Criteria

| Radius | Max Time | Max RAM |
|--------|----------|---------|
| 20 km  | < 3 min  | < 3 GB  |
| 30 km  | < 5 min  | < 3.5 GB |
| 50 km  | < 10 min | < 4 GB  |

- No OOM crashes at any radius up to 50km
- Progressive results visible within 30s of starting
- Cache reuse speeds up repeated calculations 5-10x

---

## Files to Modify

### Backend (Python)

| File | Changes |
|------|---------|
| `terrain_service.py` | mmap loading, disk cache |
| `coverage_service.py` | tile-based routing |
| `parallel_coverage_service.py` | adaptive workers |
| `osm_client.py` | SQLite integration |
| `websocket.py` | streaming results |
| **NEW** `tile_processor.py` | tile generation & processing |
| **NEW** `cache_db.py` | SQLite cache layer |

### Frontend (TypeScript)

| File | Changes |
|------|---------|
| `store/coverage.ts` | partial results handling |
| `CoverageHeatmap.tsx` | incremental rendering |
| `App.tsx` | progress for tiled calc |

---

## Testing

```bash
# Test 20km radius
curl -X POST http://localhost:8888/api/coverage/calculate \
  -H "Content-Type: application/json" \
  -d '{"radius": 20000, "resolution": 500, "preset": "standard"}'

# Monitor memory
watch -n 1 'ps aux | grep rfcp-server | awk "{print \$6/1024\" MB\"}"'

# Test 50km radius
curl -X POST http://localhost:8888/api/coverage/calculate \
  -H "Content-Type: application/json" \
  -d '{"radius": 50000, "resolution": 1000, "preset": "standard"}'
```

---

## Notes

- Tile size 5km is a balance — smaller = more overhead, larger = more memory
- SQLite R-tree extension would be faster but requires compilation
- For Rust version, all of this will be native and faster

---

*"Think in tiles, stream results, cache everything"* 🗺️