437 lines
15 KiB
Markdown
437 lines
15 KiB
Markdown
# RFCP 3.9.0 — SRTM1 Real Terrain Data Integration
|
||
|
||
## Context
|
||
|
||
RFCP currently downloads terrain tiles from an elevation API at runtime.
|
||
This works but has limitations:
|
||
- Requires internet connection
|
||
- Unknown data source quality
|
||
- No offline capability (critical for tactical/field use)
|
||
- No control over resolution or caching
|
||
|
||
Goal: Replace with SRTM1 (30m resolution) HGT files, offline-first architecture.
|
||
|
||
## SRTM1 Data Format
|
||
|
||
HGT files are dead simple:
|
||
- 1°×1° tiles, named by southwest corner: `N48E033.hgt`
|
||
- 3601×3601 grid of signed 16-bit integers (big-endian)
|
||
- Each value = elevation in meters
|
||
- File size: exactly 25,934,402 bytes (3601 × 3601 × 2)
|
||
- Row order: north to south (first row = northernmost)
|
||
- Column order: west to east
|
||
- Adjacent tiles overlap by 1 pixel on shared edges
|
||
- Void/no-data value: -32768
|
||
|
||
Compressed (.hgt.zip): ~10-15 MB per tile typically.
|
||
|
||
## Architecture
|
||
|
||
### Tile Storage Layout
|
||
|
||
```
|
||
{app_data}/terrain/
|
||
├── srtm1/ # 30m resolution tiles
|
||
│ ├── N48E033.hgt # Uncompressed for fast access
|
||
│ ├── N48E034.hgt
|
||
│ ├── N48E035.hgt
|
||
│ └── ...
|
||
├── tile_index.json # Metadata: available tiles, checksums, dates
|
||
└── downloads/ # Temporary download staging
|
||
```
|
||
|
||
On Windows, `{app_data}` = the application's data directory.
|
||
For PyInstaller exe: `data/terrain/` relative to exe location.
|
||
The path must be configurable (environment variable or config file).
|
||
|
||
### Tile Manager (new file: `terrain_manager.py`)
|
||
|
||
```python
|
||
class SRTMTileManager:
|
||
"""Manages SRTM1 HGT tile storage, loading, and caching."""
|
||
|
||
def __init__(self, terrain_dir: str):
|
||
self.terrain_dir = Path(terrain_dir)
|
||
self.srtm1_dir = self.terrain_dir / "srtm1"
|
||
self.srtm1_dir.mkdir(parents=True, exist_ok=True)
|
||
|
||
# In-memory cache: tile_name -> numpy array
|
||
self._tile_cache: Dict[str, np.ndarray] = {}
|
||
self._max_cache_tiles = 16 # ~16 tiles = ~400 MB RAM
|
||
|
||
def get_tile_name(self, lat: float, lon: float) -> str:
|
||
"""Convert lat/lon to SRTM tile name."""
|
||
# Floor to get southwest corner
|
||
lat_int = int(lat) if lat >= 0 else int(lat) - 1
|
||
lon_int = int(lon) if lon >= 0 else int(lon) - 1
|
||
|
||
lat_prefix = "N" if lat_int >= 0 else "S"
|
||
lon_prefix = "E" if lon_int >= 0 else "W"
|
||
|
||
return f"{lat_prefix}{abs(lat_int):02d}{lon_prefix}{abs(lon_int):03d}"
|
||
|
||
def get_required_tiles(self, center_lat, center_lon, radius_km) -> List[str]:
|
||
"""Determine which tiles are needed for a coverage calculation."""
|
||
# Calculate bounding box from center + radius
|
||
# Return list of tile names
|
||
|
||
def has_tile(self, tile_name: str) -> bool:
|
||
"""Check if tile exists locally."""
|
||
return (self.srtm1_dir / f"{tile_name}.hgt").exists()
|
||
|
||
def load_tile(self, tile_name: str) -> Optional[np.ndarray]:
|
||
"""Load tile from disk into memory. Returns 3601x3601 int16 array."""
|
||
if tile_name in self._tile_cache:
|
||
return self._tile_cache[tile_name]
|
||
|
||
hgt_path = self.srtm1_dir / f"{tile_name}.hgt"
|
||
if not hgt_path.exists():
|
||
return None
|
||
|
||
# Read raw HGT: big-endian signed 16-bit
|
||
data = np.fromfile(str(hgt_path), dtype='>i2')
|
||
tile = data.reshape((3601, 3601))
|
||
|
||
# Replace void values
|
||
tile = tile.astype(np.float32)
|
||
tile[tile == -32768] = np.nan
|
||
|
||
# Cache management (LRU-style: evict oldest if full)
|
||
if len(self._tile_cache) >= self._max_cache_tiles:
|
||
oldest_key = next(iter(self._tile_cache))
|
||
del self._tile_cache[oldest_key]
|
||
|
||
self._tile_cache[tile_name] = tile
|
||
return tile
|
||
|
||
def get_elevation(self, lat: float, lon: float) -> Optional[float]:
|
||
"""Get elevation at a single point with bilinear interpolation."""
|
||
tile_name = self.get_tile_name(lat, lon)
|
||
tile = self.load_tile(tile_name)
|
||
if tile is None:
|
||
return None
|
||
return self._bilinear_sample(tile, lat, lon)
|
||
|
||
def get_elevations_batch(self, lats: np.ndarray, lons: np.ndarray) -> np.ndarray:
|
||
"""Get elevations for array of points. Vectorized."""
|
||
# Group points by tile
|
||
# Load needed tiles
|
||
# Vectorized bilinear interpolation per tile
|
||
# Return array of elevations
|
||
|
||
async def download_tile(self, tile_name: str) -> bool:
|
||
"""Download a single tile from remote source (if online)."""
|
||
# Try multiple sources in order:
|
||
# 1. Own server (future: UMTC sync endpoint)
|
||
# 2. srtm.fasma.org (no auth required)
|
||
# 3. viewfinderpanoramas.org (no auth, void-filled)
|
||
# Returns True if successful
|
||
|
||
def get_missing_tiles(self, center_lat, center_lon, radius_km) -> List[str]:
|
||
"""Check which needed tiles are not available locally."""
|
||
required = self.get_required_tiles(center_lat, center_lon, radius_km)
|
||
return [t for t in required if not self.has_tile(t)]
|
||
```
|
||
|
||
### Bilinear Interpolation (CRITICAL for accuracy)
|
||
|
||
Current system uses nearest-neighbor (pick closest grid cell).
|
||
SRTM1 at 30m means nearest-neighbor can have 15m positional error.
|
||
Bilinear interpolation reduces this to sub-meter accuracy.
|
||
|
||
```python
|
||
def _bilinear_sample(self, tile: np.ndarray, lat: float, lon: float) -> float:
|
||
"""Sample elevation with bilinear interpolation."""
|
||
# Tile southwest corner
|
||
lat_int = int(lat) if lat >= 0 else int(lat) - 1
|
||
lon_int = int(lon) if lon >= 0 else int(lon) - 1
|
||
|
||
# Fractional position within tile (0.0 to 1.0)
|
||
lat_frac = lat - lat_int # 0 = south edge, 1 = north edge
|
||
lon_frac = lon - lon_int # 0 = west edge, 1 = east edge
|
||
|
||
# Convert to row/col (note: rows go north to south!)
|
||
row_exact = (1.0 - lat_frac) * 3600.0 # 0 = north, 3600 = south
|
||
col_exact = lon_frac * 3600.0 # 0 = west, 3600 = east
|
||
|
||
# Four surrounding grid points
|
||
r0 = int(row_exact)
|
||
c0 = int(col_exact)
|
||
r1 = min(r0 + 1, 3600)
|
||
c1 = min(c0 + 1, 3600)
|
||
|
||
# Fractional position between grid points
|
||
dr = row_exact - r0
|
||
dc = col_exact - c0
|
||
|
||
# Bilinear interpolation
|
||
z00 = tile[r0, c0]
|
||
z01 = tile[r0, c1]
|
||
z10 = tile[r1, c0]
|
||
z11 = tile[r1, c1]
|
||
|
||
# Handle NaN (void) values
|
||
if np.isnan(z00) or np.isnan(z01) or np.isnan(z10) or np.isnan(z11):
|
||
# Fall back to nearest non-NaN
|
||
valid = [(z00, 0, 0), (z01, 0, 1), (z10, 1, 0), (z11, 1, 1)]
|
||
valid = [(z, r, c) for z, r, c in valid if not np.isnan(z)]
|
||
return valid[0][0] if valid else 0.0
|
||
|
||
elevation = (z00 * (1 - dr) * (1 - dc) +
|
||
z01 * (1 - dr) * dc +
|
||
z10 * dr * (1 - dc) +
|
||
z11 * dr * dc)
|
||
|
||
return float(elevation)
|
||
```
|
||
|
||
### Vectorized Batch Elevation (for GPU pipeline)
|
||
|
||
This replaces the current `_batch_elevation_lookup` in gpu_service.py.
|
||
Must handle multi-tile seamlessly.
|
||
|
||
```python
|
||
def get_elevations_batch(self, lats: np.ndarray, lons: np.ndarray) -> np.ndarray:
|
||
"""Vectorized elevation lookup with bilinear interpolation.
|
||
|
||
Handles points spanning multiple tiles efficiently.
|
||
Groups points by tile, processes each tile with full NumPy vectorization.
|
||
"""
|
||
elevations = np.zeros(len(lats), dtype=np.float32)
|
||
|
||
# Compute tile indices for each point
|
||
lat_ints = np.where(lats >= 0, np.floor(lats).astype(int),
|
||
np.floor(lats).astype(int))
|
||
lon_ints = np.where(lons >= 0, np.floor(lons).astype(int),
|
||
np.floor(lons).astype(int))
|
||
|
||
# Group by tile
|
||
tile_keys = lat_ints * 1000 + lon_ints # unique key per tile
|
||
unique_keys = np.unique(tile_keys)
|
||
|
||
for key in unique_keys:
|
||
mask = tile_keys == key
|
||
lat_int = int(key // 1000)
|
||
lon_int = int(key % 1000)
|
||
if lon_int > 500: # handle negative longitudes
|
||
lon_int -= 1000
|
||
|
||
tile_name = self._make_tile_name(lat_int, lon_int)
|
||
tile = self.load_tile(tile_name)
|
||
|
||
if tile is None:
|
||
elevations[mask] = 0.0 # no data
|
||
continue
|
||
|
||
# Vectorized bilinear for all points in this tile
|
||
tile_lats = lats[mask]
|
||
tile_lons = lons[mask]
|
||
|
||
lat_frac = tile_lats - lat_int
|
||
lon_frac = tile_lons - lon_int
|
||
|
||
row_exact = (1.0 - lat_frac) * 3600.0
|
||
col_exact = lon_frac * 3600.0
|
||
|
||
r0 = np.clip(row_exact.astype(int), 0, 3599)
|
||
c0 = np.clip(col_exact.astype(int), 0, 3599)
|
||
r1 = np.clip(r0 + 1, 0, 3600)
|
||
c1 = np.clip(c0 + 1, 0, 3600)
|
||
|
||
dr = row_exact - r0
|
||
dc = col_exact - c0
|
||
|
||
z00 = tile[r0, c0]
|
||
z01 = tile[r0, c1]
|
||
z10 = tile[r1, c0]
|
||
z11 = tile[r1, c1]
|
||
|
||
result = (z00 * (1 - dr) * (1 - dc) +
|
||
z01 * (1 - dr) * dc +
|
||
z10 * dr * (1 - dc) +
|
||
z11 * dr * dc)
|
||
|
||
# Handle NaN voids
|
||
nan_mask = np.isnan(result)
|
||
if nan_mask.any():
|
||
result[nan_mask] = 0.0
|
||
|
||
elevations[mask] = result
|
||
|
||
return elevations
|
||
```
|
||
|
||
## Integration Points
|
||
|
||
### 1. Replace terrain_service.py elevation lookup
|
||
|
||
Current terrain service downloads elevation data from an API.
|
||
Replace with SRTMTileManager calls:
|
||
|
||
```python
|
||
# OLD:
|
||
elevation = await self.terrain_service.get_elevation(lat, lon)
|
||
|
||
# NEW:
|
||
elevation = self.tile_manager.get_elevation(lat, lon)
|
||
# Or for batch (GPU pipeline Phase 2.6):
|
||
elevations = self.tile_manager.get_elevations_batch(lats_array, lons_array)
|
||
```
|
||
|
||
### 2. Replace _batch_elevation_lookup in gpu_service.py
|
||
|
||
The vectorized elevation lookup in gpu_service.py currently loads tiles
|
||
and does nearest-neighbor sampling. Replace with tile_manager.get_elevations_batch()
|
||
which does bilinear interpolation.
|
||
|
||
### 3. Coverage service pre-check
|
||
|
||
Before starting calculation, check if all needed tiles are available:
|
||
|
||
```python
|
||
missing = self.tile_manager.get_missing_tiles(site_lat, site_lon, radius_km)
|
||
if missing:
|
||
if has_internet:
|
||
# Try to download missing tiles
|
||
for tile_name in missing:
|
||
await self.tile_manager.download_tile(tile_name)
|
||
else:
|
||
# Return warning to frontend
|
||
return {"warning": f"Missing terrain tiles: {missing}. Using flat terrain."}
|
||
```
|
||
|
||
### 4. Frontend notification
|
||
|
||
When tiles are missing, show a warning banner:
|
||
"⚠ Terrain data not available for this area. Coverage accuracy reduced."
|
||
|
||
When tiles are being downloaded:
|
||
"⬇ Downloading terrain data... (N48E033.hgt, 12.5 MB)"
|
||
|
||
### 5. Terrain Profile Viewer
|
||
|
||
The terrain profile viewer should use the same tile_manager
|
||
for consistent elevation data. With bilinear interpolation,
|
||
profiles will be much smoother and more accurate.
|
||
|
||
## Download Sources (Priority Order)
|
||
|
||
For auto-download when online:
|
||
|
||
1. **srtm.fasma.org** (no auth, direct HGT.zip download)
|
||
URL: `https://srtm.fasma.org/N48E033.SRTMGL1.hgt.zip`
|
||
- Free, no registration
|
||
- SRTM1 (30m) data
|
||
- May be slow or unreliable
|
||
|
||
2. **viewfinderpanoramas.org** (no auth, void-filled data)
|
||
URL: `http://viewfinderpanoramas.org/dem1/{region}/{tile}.hgt.zip`
|
||
- Free, no registration
|
||
- Void areas filled from topographic maps
|
||
- Better quality in mountainous areas
|
||
- File naming might differ by region
|
||
|
||
3. **Future: UMTC sync server**
|
||
URL: `https://rfcp.{your-domain}/api/terrain/tiles/{tile_name}.hgt`
|
||
- Self-hosted on your infrastructure
|
||
- Accessible via WireGuard mesh
|
||
- Can pre-populate with full Ukraine dataset
|
||
|
||
## Offline Bundle Strategy
|
||
|
||
For installer / field deployment:
|
||
|
||
### Option A: Region packs
|
||
Pre-package tiles by operational area:
|
||
- `terrain-dnipro.zip` — 4 tiles around Dnipro area (~100 MB)
|
||
- `terrain-ukraine-east.zip` — ~50 tiles, eastern Ukraine (~1.2 GB)
|
||
- `terrain-ukraine-full.zip` — ~171 tiles, all Ukraine (~4.3 GB)
|
||
|
||
### Option B: On-demand with cache
|
||
Ship empty, download tiles as needed on first calculation.
|
||
Cache permanently. Works well for development/testing.
|
||
|
||
### Option C: Live USB bundle
|
||
For tactical deployment, include full Ukraine terrain data
|
||
on the live USB alongside the application. 4.3 GB is acceptable
|
||
for a USB drive.
|
||
|
||
Recommend: **Option B for now** (development), **Option C for deployment**.
|
||
|
||
## File Changes
|
||
|
||
### New Files
|
||
- `backend/app/services/terrain_manager.py` — SRTMTileManager class
|
||
|
||
### Modified Files
|
||
- `backend/app/services/terrain_service.py` — Replace API calls with tile_manager
|
||
- `backend/app/services/gpu_service.py` — Replace _batch_elevation_lookup
|
||
- `backend/app/services/coverage_service.py` — Add missing tile pre-check
|
||
- `backend/app/main.py` — Initialize tile_manager on startup
|
||
|
||
### Config
|
||
- Add `TERRAIN_DIR` environment variable / config option
|
||
- Default: `./data/terrain` relative to backend exe
|
||
|
||
## Testing
|
||
|
||
```powershell
|
||
# Build and test
|
||
cd D:\root\rfcp\backend
|
||
pyinstaller ..\installer\rfcp-server-gpu.spec --noconfirm
|
||
.\dist\rfcp-server\rfcp-server.exe
|
||
```
|
||
|
||
### Test 1: First run (no tiles cached)
|
||
- Start app, trigger calculation
|
||
- Should attempt to download required tile(s)
|
||
- If online: downloads, caches, calculates
|
||
- If offline: warning, flat terrain fallback
|
||
|
||
### Test 2: Cached tiles
|
||
- Run same calculation again
|
||
- Tile loaded from disk cache, no download
|
||
- Should be fast (tile load from disk < 100ms)
|
||
|
||
### Test 3: Accuracy comparison
|
||
- Compare elevation at known points (e.g., Dnipro city center)
|
||
- Cross-reference with Google Earth elevation
|
||
- Expected accuracy: ±5m horizontal, ±16m vertical (SRTM spec)
|
||
|
||
### Test 4: Multi-tile calculation
|
||
- Set radius to 50km+ to span multiple tiles
|
||
- Verify seamless stitching at tile boundaries
|
||
- No elevation jumps or artifacts at edges
|
||
|
||
### Test 5: Terrain profile
|
||
- Draw terrain profile across tile boundary
|
||
- Should be smooth, no discontinuity
|
||
- Compare with Google Earth profile for same path
|
||
|
||
### Test 6: Performance
|
||
- Tile load time from disk: <100ms
|
||
- Batch elevation lookup (6000 points): <50ms
|
||
- Should not regress overall calculation time
|
||
- Memory: ~25 MB per loaded tile, max 16 tiles = 400 MB
|
||
|
||
## What NOT to Change
|
||
|
||
- Don't modify GPU pipeline architecture (Phase 2.5/2.6/2.7)
|
||
- Don't change propagation model math
|
||
- Don't change API endpoints or response format
|
||
- Don't change frontend map or heatmap rendering
|
||
- Don't change OSM building/vegetation fetching
|
||
- Don't change PyInstaller build process (just add data dir)
|
||
|
||
## Success Criteria
|
||
|
||
- [ ] SRTM1 tiles load correctly (3601×3601, 30m resolution)
|
||
- [ ] Bilinear interpolation working (smoother than nearest-neighbor)
|
||
- [ ] Offline mode works with pre-cached tiles
|
||
- [ ] Auto-download works when online
|
||
- [ ] Missing tile warning shown to user
|
||
- [ ] Multi-tile seamless stitching
|
||
- [ ] Terrain profile accuracy matches Google Earth within 20m
|
||
- [ ] No performance regression (calculation time same or faster)
|
||
- [ ] Tile cache directory configurable
|