15 KiB
15 KiB
RFCP 3.9.0 — SRTM1 Real Terrain Data Integration
Context
RFCP currently downloads terrain tiles from an elevation API at runtime. This works but has limitations:
- Requires internet connection
- Unknown data source quality
- No offline capability (critical for tactical/field use)
- No control over resolution or caching
Goal: Replace with SRTM1 (30m resolution) HGT files, offline-first architecture.
SRTM1 Data Format
HGT files are dead simple:
- 1°×1° tiles, named by southwest corner:
N48E033.hgt - 3601×3601 grid of signed 16-bit integers (big-endian)
- Each value = elevation in meters
- File size: exactly 25,934,402 bytes (3601 × 3601 × 2)
- Row order: north to south (first row = northernmost)
- Column order: west to east
- Adjacent tiles overlap by 1 pixel on shared edges
- Void/no-data value: -32768
Compressed (.hgt.zip): ~10-15 MB per tile typically.
Architecture
Tile Storage Layout
{app_data}/terrain/
├── srtm1/ # 30m resolution tiles
│ ├── N48E033.hgt # Uncompressed for fast access
│ ├── N48E034.hgt
│ ├── N48E035.hgt
│ └── ...
├── tile_index.json # Metadata: available tiles, checksums, dates
└── downloads/ # Temporary download staging
On Windows, {app_data} = the application's data directory.
For PyInstaller exe: data/terrain/ relative to exe location.
The path must be configurable (environment variable or config file).
Tile Manager (new file: terrain_manager.py)
class SRTMTileManager:
"""Manages SRTM1 HGT tile storage, loading, and caching."""
def __init__(self, terrain_dir: str):
self.terrain_dir = Path(terrain_dir)
self.srtm1_dir = self.terrain_dir / "srtm1"
self.srtm1_dir.mkdir(parents=True, exist_ok=True)
# In-memory cache: tile_name -> numpy array
self._tile_cache: Dict[str, np.ndarray] = {}
self._max_cache_tiles = 16 # ~16 tiles = ~400 MB RAM
def get_tile_name(self, lat: float, lon: float) -> str:
"""Convert lat/lon to SRTM tile name."""
# Floor to get southwest corner
lat_int = int(lat) if lat >= 0 else int(lat) - 1
lon_int = int(lon) if lon >= 0 else int(lon) - 1
lat_prefix = "N" if lat_int >= 0 else "S"
lon_prefix = "E" if lon_int >= 0 else "W"
return f"{lat_prefix}{abs(lat_int):02d}{lon_prefix}{abs(lon_int):03d}"
def get_required_tiles(self, center_lat, center_lon, radius_km) -> List[str]:
"""Determine which tiles are needed for a coverage calculation."""
# Calculate bounding box from center + radius
# Return list of tile names
def has_tile(self, tile_name: str) -> bool:
"""Check if tile exists locally."""
return (self.srtm1_dir / f"{tile_name}.hgt").exists()
def load_tile(self, tile_name: str) -> Optional[np.ndarray]:
"""Load tile from disk into memory. Returns 3601x3601 int16 array."""
if tile_name in self._tile_cache:
return self._tile_cache[tile_name]
hgt_path = self.srtm1_dir / f"{tile_name}.hgt"
if not hgt_path.exists():
return None
# Read raw HGT: big-endian signed 16-bit
data = np.fromfile(str(hgt_path), dtype='>i2')
tile = data.reshape((3601, 3601))
# Replace void values
tile = tile.astype(np.float32)
tile[tile == -32768] = np.nan
# Cache management (LRU-style: evict oldest if full)
if len(self._tile_cache) >= self._max_cache_tiles:
oldest_key = next(iter(self._tile_cache))
del self._tile_cache[oldest_key]
self._tile_cache[tile_name] = tile
return tile
def get_elevation(self, lat: float, lon: float) -> Optional[float]:
"""Get elevation at a single point with bilinear interpolation."""
tile_name = self.get_tile_name(lat, lon)
tile = self.load_tile(tile_name)
if tile is None:
return None
return self._bilinear_sample(tile, lat, lon)
def get_elevations_batch(self, lats: np.ndarray, lons: np.ndarray) -> np.ndarray:
"""Get elevations for array of points. Vectorized."""
# Group points by tile
# Load needed tiles
# Vectorized bilinear interpolation per tile
# Return array of elevations
async def download_tile(self, tile_name: str) -> bool:
"""Download a single tile from remote source (if online)."""
# Try multiple sources in order:
# 1. Own server (future: UMTC sync endpoint)
# 2. srtm.fasma.org (no auth required)
# 3. viewfinderpanoramas.org (no auth, void-filled)
# Returns True if successful
def get_missing_tiles(self, center_lat, center_lon, radius_km) -> List[str]:
"""Check which needed tiles are not available locally."""
required = self.get_required_tiles(center_lat, center_lon, radius_km)
return [t for t in required if not self.has_tile(t)]
Bilinear Interpolation (CRITICAL for accuracy)
Current system uses nearest-neighbor (pick closest grid cell). SRTM1 at 30m means nearest-neighbor can have 15m positional error. Bilinear interpolation reduces this to sub-meter accuracy.
def _bilinear_sample(self, tile: np.ndarray, lat: float, lon: float) -> float:
"""Sample elevation with bilinear interpolation."""
# Tile southwest corner
lat_int = int(lat) if lat >= 0 else int(lat) - 1
lon_int = int(lon) if lon >= 0 else int(lon) - 1
# Fractional position within tile (0.0 to 1.0)
lat_frac = lat - lat_int # 0 = south edge, 1 = north edge
lon_frac = lon - lon_int # 0 = west edge, 1 = east edge
# Convert to row/col (note: rows go north to south!)
row_exact = (1.0 - lat_frac) * 3600.0 # 0 = north, 3600 = south
col_exact = lon_frac * 3600.0 # 0 = west, 3600 = east
# Four surrounding grid points
r0 = int(row_exact)
c0 = int(col_exact)
r1 = min(r0 + 1, 3600)
c1 = min(c0 + 1, 3600)
# Fractional position between grid points
dr = row_exact - r0
dc = col_exact - c0
# Bilinear interpolation
z00 = tile[r0, c0]
z01 = tile[r0, c1]
z10 = tile[r1, c0]
z11 = tile[r1, c1]
# Handle NaN (void) values
if np.isnan(z00) or np.isnan(z01) or np.isnan(z10) or np.isnan(z11):
# Fall back to nearest non-NaN
valid = [(z00, 0, 0), (z01, 0, 1), (z10, 1, 0), (z11, 1, 1)]
valid = [(z, r, c) for z, r, c in valid if not np.isnan(z)]
return valid[0][0] if valid else 0.0
elevation = (z00 * (1 - dr) * (1 - dc) +
z01 * (1 - dr) * dc +
z10 * dr * (1 - dc) +
z11 * dr * dc)
return float(elevation)
Vectorized Batch Elevation (for GPU pipeline)
This replaces the current _batch_elevation_lookup in gpu_service.py.
Must handle multi-tile seamlessly.
def get_elevations_batch(self, lats: np.ndarray, lons: np.ndarray) -> np.ndarray:
"""Vectorized elevation lookup with bilinear interpolation.
Handles points spanning multiple tiles efficiently.
Groups points by tile, processes each tile with full NumPy vectorization.
"""
elevations = np.zeros(len(lats), dtype=np.float32)
# Compute tile indices for each point
lat_ints = np.where(lats >= 0, np.floor(lats).astype(int),
np.floor(lats).astype(int))
lon_ints = np.where(lons >= 0, np.floor(lons).astype(int),
np.floor(lons).astype(int))
# Group by tile
tile_keys = lat_ints * 1000 + lon_ints # unique key per tile
unique_keys = np.unique(tile_keys)
for key in unique_keys:
mask = tile_keys == key
lat_int = int(key // 1000)
lon_int = int(key % 1000)
if lon_int > 500: # handle negative longitudes
lon_int -= 1000
tile_name = self._make_tile_name(lat_int, lon_int)
tile = self.load_tile(tile_name)
if tile is None:
elevations[mask] = 0.0 # no data
continue
# Vectorized bilinear for all points in this tile
tile_lats = lats[mask]
tile_lons = lons[mask]
lat_frac = tile_lats - lat_int
lon_frac = tile_lons - lon_int
row_exact = (1.0 - lat_frac) * 3600.0
col_exact = lon_frac * 3600.0
r0 = np.clip(row_exact.astype(int), 0, 3599)
c0 = np.clip(col_exact.astype(int), 0, 3599)
r1 = np.clip(r0 + 1, 0, 3600)
c1 = np.clip(c0 + 1, 0, 3600)
dr = row_exact - r0
dc = col_exact - c0
z00 = tile[r0, c0]
z01 = tile[r0, c1]
z10 = tile[r1, c0]
z11 = tile[r1, c1]
result = (z00 * (1 - dr) * (1 - dc) +
z01 * (1 - dr) * dc +
z10 * dr * (1 - dc) +
z11 * dr * dc)
# Handle NaN voids
nan_mask = np.isnan(result)
if nan_mask.any():
result[nan_mask] = 0.0
elevations[mask] = result
return elevations
Integration Points
1. Replace terrain_service.py elevation lookup
Current terrain service downloads elevation data from an API. Replace with SRTMTileManager calls:
# OLD:
elevation = await self.terrain_service.get_elevation(lat, lon)
# NEW:
elevation = self.tile_manager.get_elevation(lat, lon)
# Or for batch (GPU pipeline Phase 2.6):
elevations = self.tile_manager.get_elevations_batch(lats_array, lons_array)
2. Replace _batch_elevation_lookup in gpu_service.py
The vectorized elevation lookup in gpu_service.py currently loads tiles and does nearest-neighbor sampling. Replace with tile_manager.get_elevations_batch() which does bilinear interpolation.
3. Coverage service pre-check
Before starting calculation, check if all needed tiles are available:
missing = self.tile_manager.get_missing_tiles(site_lat, site_lon, radius_km)
if missing:
if has_internet:
# Try to download missing tiles
for tile_name in missing:
await self.tile_manager.download_tile(tile_name)
else:
# Return warning to frontend
return {"warning": f"Missing terrain tiles: {missing}. Using flat terrain."}
4. Frontend notification
When tiles are missing, show a warning banner: "⚠ Terrain data not available for this area. Coverage accuracy reduced."
When tiles are being downloaded: "⬇ Downloading terrain data... (N48E033.hgt, 12.5 MB)"
5. Terrain Profile Viewer
The terrain profile viewer should use the same tile_manager for consistent elevation data. With bilinear interpolation, profiles will be much smoother and more accurate.
Download Sources (Priority Order)
For auto-download when online:
-
srtm.fasma.org (no auth, direct HGT.zip download) URL:
https://srtm.fasma.org/N48E033.SRTMGL1.hgt.zip- Free, no registration
- SRTM1 (30m) data
- May be slow or unreliable
-
viewfinderpanoramas.org (no auth, void-filled data) URL:
http://viewfinderpanoramas.org/dem1/{region}/{tile}.hgt.zip- Free, no registration
- Void areas filled from topographic maps
- Better quality in mountainous areas
- File naming might differ by region
-
Future: UMTC sync server URL:
https://rfcp.{your-domain}/api/terrain/tiles/{tile_name}.hgt- Self-hosted on your infrastructure
- Accessible via WireGuard mesh
- Can pre-populate with full Ukraine dataset
Offline Bundle Strategy
For installer / field deployment:
Option A: Region packs
Pre-package tiles by operational area:
terrain-dnipro.zip— 4 tiles around Dnipro area (~100 MB)terrain-ukraine-east.zip— ~50 tiles, eastern Ukraine (~1.2 GB)terrain-ukraine-full.zip— ~171 tiles, all Ukraine (~4.3 GB)
Option B: On-demand with cache
Ship empty, download tiles as needed on first calculation. Cache permanently. Works well for development/testing.
Option C: Live USB bundle
For tactical deployment, include full Ukraine terrain data on the live USB alongside the application. 4.3 GB is acceptable for a USB drive.
Recommend: Option B for now (development), Option C for deployment.
File Changes
New Files
backend/app/services/terrain_manager.py— SRTMTileManager class
Modified Files
backend/app/services/terrain_service.py— Replace API calls with tile_managerbackend/app/services/gpu_service.py— Replace _batch_elevation_lookupbackend/app/services/coverage_service.py— Add missing tile pre-checkbackend/app/main.py— Initialize tile_manager on startup
Config
- Add
TERRAIN_DIRenvironment variable / config option - Default:
./data/terrainrelative to backend exe
Testing
# Build and test
cd D:\root\rfcp\backend
pyinstaller ..\installer\rfcp-server-gpu.spec --noconfirm
.\dist\rfcp-server\rfcp-server.exe
Test 1: First run (no tiles cached)
- Start app, trigger calculation
- Should attempt to download required tile(s)
- If online: downloads, caches, calculates
- If offline: warning, flat terrain fallback
Test 2: Cached tiles
- Run same calculation again
- Tile loaded from disk cache, no download
- Should be fast (tile load from disk < 100ms)
Test 3: Accuracy comparison
- Compare elevation at known points (e.g., Dnipro city center)
- Cross-reference with Google Earth elevation
- Expected accuracy: ±5m horizontal, ±16m vertical (SRTM spec)
Test 4: Multi-tile calculation
- Set radius to 50km+ to span multiple tiles
- Verify seamless stitching at tile boundaries
- No elevation jumps or artifacts at edges
Test 5: Terrain profile
- Draw terrain profile across tile boundary
- Should be smooth, no discontinuity
- Compare with Google Earth profile for same path
Test 6: Performance
- Tile load time from disk: <100ms
- Batch elevation lookup (6000 points): <50ms
- Should not regress overall calculation time
- Memory: ~25 MB per loaded tile, max 16 tiles = 400 MB
What NOT to Change
- Don't modify GPU pipeline architecture (Phase 2.5/2.6/2.7)
- Don't change propagation model math
- Don't change API endpoints or response format
- Don't change frontend map or heatmap rendering
- Don't change OSM building/vegetation fetching
- Don't change PyInstaller build process (just add data dir)
Success Criteria
- SRTM1 tiles load correctly (3601×3601, 30m resolution)
- Bilinear interpolation working (smoother than nearest-neighbor)
- Offline mode works with pre-cached tiles
- Auto-download works when online
- Missing tile warning shown to user
- Multi-tile seamless stitching
- Terrain profile accuracy matches Google Earth within 20m
- No performance regression (calculation time same or faster)
- Tile cache directory configurable