# RFCP 3.9.0 — SRTM1 Real Terrain Data Integration

## Context

RFCP currently downloads terrain tiles from an elevation API at runtime.
This works but has limitations:
- Requires internet connection
- Unknown data source quality
- No offline capability (critical for tactical/field use)
- No control over resolution or caching

Goal: Replace with SRTM1 (30m resolution) HGT files, offline-first architecture.

## SRTM1 Data Format

HGT files are dead simple:
- 1°×1° tiles, named by southwest corner: `N48E033.hgt`
- 3601×3601 grid of signed 16-bit integers (big-endian)
- Each value = elevation in meters
- File size: exactly 25,934,402 bytes (3601 × 3601 × 2)
- Row order: north to south (first row = northernmost)
- Column order: west to east
- Adjacent tiles overlap by 1 pixel on shared edges
- Void/no-data value: -32768

Compressed (.hgt.zip): ~10-15 MB per tile typically.

## Architecture

### Tile Storage Layout

```
{app_data}/terrain/
├── srtm1/                    # 30m resolution tiles
│   ├── N48E033.hgt           # Uncompressed for fast access
│   ├── N48E034.hgt
│   ├── N48E035.hgt
│   └── ...
├── tile_index.json           # Metadata: available tiles, checksums, dates
└── downloads/                # Temporary download staging
```

On Windows, `{app_data}` = the application's data directory.
For PyInstaller exe: `data/terrain/` relative to exe location.
The path must be configurable (environment variable or config file).

### Tile Manager (new file: `terrain_manager.py`)

```python
class SRTMTileManager:
    """Manages SRTM1 HGT tile storage, loading, and caching."""
    
    def __init__(self, terrain_dir: str):
        self.terrain_dir = Path(terrain_dir)
        self.srtm1_dir = self.terrain_dir / "srtm1"
        self.srtm1_dir.mkdir(parents=True, exist_ok=True)
        
        # In-memory cache: tile_name -> numpy array
        self._tile_cache: Dict[str, np.ndarray] = {}
        self._max_cache_tiles = 16  # ~16 tiles = ~400 MB RAM
    
    def get_tile_name(self, lat: float, lon: float) -> str:
        """Convert lat/lon to SRTM tile name."""
        # Floor to get southwest corner
        lat_int = int(lat) if lat >= 0 else int(lat) - 1
        lon_int = int(lon) if lon >= 0 else int(lon) - 1
        
        lat_prefix = "N" if lat_int >= 0 else "S"
        lon_prefix = "E" if lon_int >= 0 else "W"
        
        return f"{lat_prefix}{abs(lat_int):02d}{lon_prefix}{abs(lon_int):03d}"
    
    def get_required_tiles(self, center_lat, center_lon, radius_km) -> List[str]:
        """Determine which tiles are needed for a coverage calculation."""
        # Calculate bounding box from center + radius
        # Return list of tile names
    
    def has_tile(self, tile_name: str) -> bool:
        """Check if tile exists locally."""
        return (self.srtm1_dir / f"{tile_name}.hgt").exists()
    
    def load_tile(self, tile_name: str) -> Optional[np.ndarray]:
        """Load tile from disk into memory. Returns 3601x3601 int16 array."""
        if tile_name in self._tile_cache:
            return self._tile_cache[tile_name]
        
        hgt_path = self.srtm1_dir / f"{tile_name}.hgt"
        if not hgt_path.exists():
            return None
        
        # Read raw HGT: big-endian signed 16-bit
        data = np.fromfile(str(hgt_path), dtype='>i2')
        tile = data.reshape((3601, 3601))
        
        # Replace void values
        tile = tile.astype(np.float32)
        tile[tile == -32768] = np.nan
        
        # Cache management (LRU-style: evict oldest if full)
        if len(self._tile_cache) >= self._max_cache_tiles:
            oldest_key = next(iter(self._tile_cache))
            del self._tile_cache[oldest_key]
        
        self._tile_cache[tile_name] = tile
        return tile
    
    def get_elevation(self, lat: float, lon: float) -> Optional[float]:
        """Get elevation at a single point with bilinear interpolation."""
        tile_name = self.get_tile_name(lat, lon)
        tile = self.load_tile(tile_name)
        if tile is None:
            return None
        return self._bilinear_sample(tile, lat, lon)
    
    def get_elevations_batch(self, lats: np.ndarray, lons: np.ndarray) -> np.ndarray:
        """Get elevations for array of points. Vectorized."""
        # Group points by tile
        # Load needed tiles
        # Vectorized bilinear interpolation per tile
        # Return array of elevations
    
    async def download_tile(self, tile_name: str) -> bool:
        """Download a single tile from remote source (if online)."""
        # Try multiple sources in order:
        # 1. Own server (future: UMTC sync endpoint)
        # 2. srtm.fasma.org (no auth required)  
        # 3. viewfinderpanoramas.org (no auth, void-filled)
        # Returns True if successful
    
    def get_missing_tiles(self, center_lat, center_lon, radius_km) -> List[str]:
        """Check which needed tiles are not available locally."""
        required = self.get_required_tiles(center_lat, center_lon, radius_km)
        return [t for t in required if not self.has_tile(t)]
```

### Bilinear Interpolation (CRITICAL for accuracy)

Current system uses nearest-neighbor (pick closest grid cell).
SRTM1 at 30m means nearest-neighbor can have 15m positional error.
Bilinear interpolation reduces this to sub-meter accuracy.

```python
def _bilinear_sample(self, tile: np.ndarray, lat: float, lon: float) -> float:
    """Sample elevation with bilinear interpolation."""
    # Tile southwest corner
    lat_int = int(lat) if lat >= 0 else int(lat) - 1
    lon_int = int(lon) if lon >= 0 else int(lon) - 1
    
    # Fractional position within tile (0.0 to 1.0)
    lat_frac = lat - lat_int  # 0 = south edge, 1 = north edge
    lon_frac = lon - lon_int  # 0 = west edge, 1 = east edge
    
    # Convert to row/col (note: rows go north to south!)
    row_exact = (1.0 - lat_frac) * 3600.0  # 0 = north, 3600 = south
    col_exact = lon_frac * 3600.0           # 0 = west, 3600 = east
    
    # Four surrounding grid points
    r0 = int(row_exact)
    c0 = int(col_exact)
    r1 = min(r0 + 1, 3600)
    c1 = min(c0 + 1, 3600)
    
    # Fractional position between grid points
    dr = row_exact - r0
    dc = col_exact - c0
    
    # Bilinear interpolation
    z00 = tile[r0, c0]
    z01 = tile[r0, c1]
    z10 = tile[r1, c0]
    z11 = tile[r1, c1]
    
    # Handle NaN (void) values
    if np.isnan(z00) or np.isnan(z01) or np.isnan(z10) or np.isnan(z11):
        # Fall back to nearest non-NaN
        valid = [(z00, 0, 0), (z01, 0, 1), (z10, 1, 0), (z11, 1, 1)]
        valid = [(z, r, c) for z, r, c in valid if not np.isnan(z)]
        return valid[0][0] if valid else 0.0
    
    elevation = (z00 * (1 - dr) * (1 - dc) +
                 z01 * (1 - dr) * dc +
                 z10 * dr * (1 - dc) +
                 z11 * dr * dc)
    
    return float(elevation)
```

### Vectorized Batch Elevation (for GPU pipeline)

This replaces the current `_batch_elevation_lookup` in gpu_service.py.
Must handle multi-tile seamlessly.

```python
def get_elevations_batch(self, lats: np.ndarray, lons: np.ndarray) -> np.ndarray:
    """Vectorized elevation lookup with bilinear interpolation.
    
    Handles points spanning multiple tiles efficiently.
    Groups points by tile, processes each tile with full NumPy vectorization.
    """
    elevations = np.zeros(len(lats), dtype=np.float32)
    
    # Compute tile indices for each point
    lat_ints = np.where(lats >= 0, np.floor(lats).astype(int), 
                        np.floor(lats).astype(int))
    lon_ints = np.where(lons >= 0, np.floor(lons).astype(int),
                        np.floor(lons).astype(int))
    
    # Group by tile
    tile_keys = lat_ints * 1000 + lon_ints  # unique key per tile
    unique_keys = np.unique(tile_keys)
    
    for key in unique_keys:
        mask = tile_keys == key
        lat_int = int(key // 1000)
        lon_int = int(key % 1000)
        if lon_int > 500:  # handle negative longitudes
            lon_int -= 1000
        
        tile_name = self._make_tile_name(lat_int, lon_int)
        tile = self.load_tile(tile_name)
        
        if tile is None:
            elevations[mask] = 0.0  # no data
            continue
        
        # Vectorized bilinear for all points in this tile
        tile_lats = lats[mask]
        tile_lons = lons[mask]
        
        lat_frac = tile_lats - lat_int
        lon_frac = tile_lons - lon_int
        
        row_exact = (1.0 - lat_frac) * 3600.0
        col_exact = lon_frac * 3600.0
        
        r0 = np.clip(row_exact.astype(int), 0, 3599)
        c0 = np.clip(col_exact.astype(int), 0, 3599)
        r1 = np.clip(r0 + 1, 0, 3600)
        c1 = np.clip(c0 + 1, 0, 3600)
        
        dr = row_exact - r0
        dc = col_exact - c0
        
        z00 = tile[r0, c0]
        z01 = tile[r0, c1]
        z10 = tile[r1, c0]
        z11 = tile[r1, c1]
        
        result = (z00 * (1 - dr) * (1 - dc) +
                  z01 * (1 - dr) * dc +
                  z10 * dr * (1 - dc) +
                  z11 * dr * dc)
        
        # Handle NaN voids
        nan_mask = np.isnan(result)
        if nan_mask.any():
            result[nan_mask] = 0.0
        
        elevations[mask] = result
    
    return elevations
```

## Integration Points

### 1. Replace terrain_service.py elevation lookup

Current terrain service downloads elevation data from an API.
Replace with SRTMTileManager calls:

```python
# OLD: 
elevation = await self.terrain_service.get_elevation(lat, lon)

# NEW:
elevation = self.tile_manager.get_elevation(lat, lon)
# Or for batch (GPU pipeline Phase 2.6):
elevations = self.tile_manager.get_elevations_batch(lats_array, lons_array)
```

### 2. Replace _batch_elevation_lookup in gpu_service.py

The vectorized elevation lookup in gpu_service.py currently loads tiles
and does nearest-neighbor sampling. Replace with tile_manager.get_elevations_batch()
which does bilinear interpolation.

### 3. Coverage service pre-check

Before starting calculation, check if all needed tiles are available:

```python
missing = self.tile_manager.get_missing_tiles(site_lat, site_lon, radius_km)
if missing:
    if has_internet:
        # Try to download missing tiles
        for tile_name in missing:
            await self.tile_manager.download_tile(tile_name)
    else:
        # Return warning to frontend
        return {"warning": f"Missing terrain tiles: {missing}. Using flat terrain."}
```

### 4. Frontend notification

When tiles are missing, show a warning banner:
"⚠ Terrain data not available for this area. Coverage accuracy reduced."

When tiles are being downloaded:
"⬇ Downloading terrain data... (N48E033.hgt, 12.5 MB)"

### 5. Terrain Profile Viewer

The terrain profile viewer should use the same tile_manager
for consistent elevation data. With bilinear interpolation,
profiles will be much smoother and more accurate.

## Download Sources (Priority Order)

For auto-download when online:

1. **srtm.fasma.org** (no auth, direct HGT.zip download)
   URL: `https://srtm.fasma.org/N48E033.SRTMGL1.hgt.zip`
   - Free, no registration
   - SRTM1 (30m) data
   - May be slow or unreliable
   
2. **viewfinderpanoramas.org** (no auth, void-filled data)
   URL: `http://viewfinderpanoramas.org/dem1/{region}/{tile}.hgt.zip`
   - Free, no registration  
   - Void areas filled from topographic maps
   - Better quality in mountainous areas
   - File naming might differ by region

3. **Future: UMTC sync server**
   URL: `https://rfcp.{your-domain}/api/terrain/tiles/{tile_name}.hgt`
   - Self-hosted on your infrastructure
   - Accessible via WireGuard mesh
   - Can pre-populate with full Ukraine dataset

## Offline Bundle Strategy

For installer / field deployment:

### Option A: Region packs
Pre-package tiles by operational area:
- `terrain-dnipro.zip` — 4 tiles around Dnipro area (~100 MB)
- `terrain-ukraine-east.zip` — ~50 tiles, eastern Ukraine (~1.2 GB)
- `terrain-ukraine-full.zip` — ~171 tiles, all Ukraine (~4.3 GB)

### Option B: On-demand with cache
Ship empty, download tiles as needed on first calculation.
Cache permanently. Works well for development/testing.

### Option C: Live USB bundle  
For tactical deployment, include full Ukraine terrain data
on the live USB alongside the application. 4.3 GB is acceptable
for a USB drive.

Recommend: **Option B for now** (development), **Option C for deployment**.

## File Changes

### New Files
- `backend/app/services/terrain_manager.py` — SRTMTileManager class

### Modified Files  
- `backend/app/services/terrain_service.py` — Replace API calls with tile_manager
- `backend/app/services/gpu_service.py` — Replace _batch_elevation_lookup
- `backend/app/services/coverage_service.py` — Add missing tile pre-check
- `backend/app/main.py` — Initialize tile_manager on startup

### Config
- Add `TERRAIN_DIR` environment variable / config option
- Default: `./data/terrain` relative to backend exe

## Testing

```powershell
# Build and test
cd D:\root\rfcp\backend
pyinstaller ..\installer\rfcp-server-gpu.spec --noconfirm
.\dist\rfcp-server\rfcp-server.exe
```

### Test 1: First run (no tiles cached)
- Start app, trigger calculation
- Should attempt to download required tile(s)
- If online: downloads, caches, calculates
- If offline: warning, flat terrain fallback

### Test 2: Cached tiles
- Run same calculation again
- Tile loaded from disk cache, no download
- Should be fast (tile load from disk < 100ms)

### Test 3: Accuracy comparison  
- Compare elevation at known points (e.g., Dnipro city center)
- Cross-reference with Google Earth elevation
- Expected accuracy: ±5m horizontal, ±16m vertical (SRTM spec)

### Test 4: Multi-tile calculation
- Set radius to 50km+ to span multiple tiles
- Verify seamless stitching at tile boundaries
- No elevation jumps or artifacts at edges

### Test 5: Terrain profile
- Draw terrain profile across tile boundary
- Should be smooth, no discontinuity
- Compare with Google Earth profile for same path

### Test 6: Performance
- Tile load time from disk: <100ms
- Batch elevation lookup (6000 points): <50ms
- Should not regress overall calculation time
- Memory: ~25 MB per loaded tile, max 16 tiles = 400 MB

## What NOT to Change

- Don't modify GPU pipeline architecture (Phase 2.5/2.6/2.7)
- Don't change propagation model math
- Don't change API endpoints or response format
- Don't change frontend map or heatmap rendering
- Don't change OSM building/vegetation fetching
- Don't change PyInstaller build process (just add data dir)

## Success Criteria

- [ ] SRTM1 tiles load correctly (3601×3601, 30m resolution)
- [ ] Bilinear interpolation working (smoother than nearest-neighbor)
- [ ] Offline mode works with pre-cached tiles
- [ ] Auto-download works when online
- [ ] Missing tile warning shown to user
- [ ] Multi-tile seamless stitching
- [ ] Terrain profile accuracy matches Google Earth within 20m
- [ ] No performance regression (calculation time same or faster)
- [ ] Tile cache directory configurable