@mytec: iter3.10 start, baseline rc ready
This commit is contained in:
436
docs/devlog/gpu_supp/RFCP-3.9.0-SRTM-Terrain-Integration.md
Normal file
436
docs/devlog/gpu_supp/RFCP-3.9.0-SRTM-Terrain-Integration.md
Normal file
@@ -0,0 +1,436 @@
|
||||
# RFCP 3.9.0 — SRTM1 Real Terrain Data Integration
|
||||
|
||||
## Context
|
||||
|
||||
RFCP currently downloads terrain tiles from an elevation API at runtime.
|
||||
This works but has limitations:
|
||||
- Requires internet connection
|
||||
- Unknown data source quality
|
||||
- No offline capability (critical for tactical/field use)
|
||||
- No control over resolution or caching
|
||||
|
||||
Goal: Replace with SRTM1 (30m resolution) HGT files, offline-first architecture.
|
||||
|
||||
## SRTM1 Data Format
|
||||
|
||||
HGT files are dead simple:
|
||||
- 1°×1° tiles, named by southwest corner: `N48E033.hgt`
|
||||
- 3601×3601 grid of signed 16-bit integers (big-endian)
|
||||
- Each value = elevation in meters
|
||||
- File size: exactly 25,934,402 bytes (3601 × 3601 × 2)
|
||||
- Row order: north to south (first row = northernmost)
|
||||
- Column order: west to east
|
||||
- Adjacent tiles overlap by 1 pixel on shared edges
|
||||
- Void/no-data value: -32768
|
||||
|
||||
Compressed (.hgt.zip): ~10-15 MB per tile typically.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Tile Storage Layout
|
||||
|
||||
```
|
||||
{app_data}/terrain/
|
||||
├── srtm1/ # 30m resolution tiles
|
||||
│ ├── N48E033.hgt # Uncompressed for fast access
|
||||
│ ├── N48E034.hgt
|
||||
│ ├── N48E035.hgt
|
||||
│ └── ...
|
||||
├── tile_index.json # Metadata: available tiles, checksums, dates
|
||||
└── downloads/ # Temporary download staging
|
||||
```
|
||||
|
||||
On Windows, `{app_data}` = the application's data directory.
|
||||
For PyInstaller exe: `data/terrain/` relative to exe location.
|
||||
The path must be configurable (environment variable or config file).
|
||||
|
||||
### Tile Manager (new file: `terrain_manager.py`)
|
||||
|
||||
```python
|
||||
class SRTMTileManager:
|
||||
"""Manages SRTM1 HGT tile storage, loading, and caching."""
|
||||
|
||||
def __init__(self, terrain_dir: str):
|
||||
self.terrain_dir = Path(terrain_dir)
|
||||
self.srtm1_dir = self.terrain_dir / "srtm1"
|
||||
self.srtm1_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# In-memory cache: tile_name -> numpy array
|
||||
self._tile_cache: Dict[str, np.ndarray] = {}
|
||||
self._max_cache_tiles = 16 # ~16 tiles = ~400 MB RAM
|
||||
|
||||
def get_tile_name(self, lat: float, lon: float) -> str:
|
||||
"""Convert lat/lon to SRTM tile name."""
|
||||
# Floor to get southwest corner
|
||||
lat_int = int(lat) if lat >= 0 else int(lat) - 1
|
||||
lon_int = int(lon) if lon >= 0 else int(lon) - 1
|
||||
|
||||
lat_prefix = "N" if lat_int >= 0 else "S"
|
||||
lon_prefix = "E" if lon_int >= 0 else "W"
|
||||
|
||||
return f"{lat_prefix}{abs(lat_int):02d}{lon_prefix}{abs(lon_int):03d}"
|
||||
|
||||
def get_required_tiles(self, center_lat, center_lon, radius_km) -> List[str]:
|
||||
"""Determine which tiles are needed for a coverage calculation."""
|
||||
# Calculate bounding box from center + radius
|
||||
# Return list of tile names
|
||||
|
||||
def has_tile(self, tile_name: str) -> bool:
|
||||
"""Check if tile exists locally."""
|
||||
return (self.srtm1_dir / f"{tile_name}.hgt").exists()
|
||||
|
||||
def load_tile(self, tile_name: str) -> Optional[np.ndarray]:
|
||||
"""Load tile from disk into memory. Returns 3601x3601 int16 array."""
|
||||
if tile_name in self._tile_cache:
|
||||
return self._tile_cache[tile_name]
|
||||
|
||||
hgt_path = self.srtm1_dir / f"{tile_name}.hgt"
|
||||
if not hgt_path.exists():
|
||||
return None
|
||||
|
||||
# Read raw HGT: big-endian signed 16-bit
|
||||
data = np.fromfile(str(hgt_path), dtype='>i2')
|
||||
tile = data.reshape((3601, 3601))
|
||||
|
||||
# Replace void values
|
||||
tile = tile.astype(np.float32)
|
||||
tile[tile == -32768] = np.nan
|
||||
|
||||
# Cache management (LRU-style: evict oldest if full)
|
||||
if len(self._tile_cache) >= self._max_cache_tiles:
|
||||
oldest_key = next(iter(self._tile_cache))
|
||||
del self._tile_cache[oldest_key]
|
||||
|
||||
self._tile_cache[tile_name] = tile
|
||||
return tile
|
||||
|
||||
def get_elevation(self, lat: float, lon: float) -> Optional[float]:
|
||||
"""Get elevation at a single point with bilinear interpolation."""
|
||||
tile_name = self.get_tile_name(lat, lon)
|
||||
tile = self.load_tile(tile_name)
|
||||
if tile is None:
|
||||
return None
|
||||
return self._bilinear_sample(tile, lat, lon)
|
||||
|
||||
def get_elevations_batch(self, lats: np.ndarray, lons: np.ndarray) -> np.ndarray:
|
||||
"""Get elevations for array of points. Vectorized."""
|
||||
# Group points by tile
|
||||
# Load needed tiles
|
||||
# Vectorized bilinear interpolation per tile
|
||||
# Return array of elevations
|
||||
|
||||
async def download_tile(self, tile_name: str) -> bool:
|
||||
"""Download a single tile from remote source (if online)."""
|
||||
# Try multiple sources in order:
|
||||
# 1. Own server (future: UMTC sync endpoint)
|
||||
# 2. srtm.fasma.org (no auth required)
|
||||
# 3. viewfinderpanoramas.org (no auth, void-filled)
|
||||
# Returns True if successful
|
||||
|
||||
def get_missing_tiles(self, center_lat, center_lon, radius_km) -> List[str]:
|
||||
"""Check which needed tiles are not available locally."""
|
||||
required = self.get_required_tiles(center_lat, center_lon, radius_km)
|
||||
return [t for t in required if not self.has_tile(t)]
|
||||
```
|
||||
|
||||
### Bilinear Interpolation (CRITICAL for accuracy)
|
||||
|
||||
Current system uses nearest-neighbor (pick closest grid cell).
|
||||
SRTM1 at 30m means nearest-neighbor can have 15m positional error.
|
||||
Bilinear interpolation reduces this to sub-meter accuracy.
|
||||
|
||||
```python
|
||||
def _bilinear_sample(self, tile: np.ndarray, lat: float, lon: float) -> float:
|
||||
"""Sample elevation with bilinear interpolation."""
|
||||
# Tile southwest corner
|
||||
lat_int = int(lat) if lat >= 0 else int(lat) - 1
|
||||
lon_int = int(lon) if lon >= 0 else int(lon) - 1
|
||||
|
||||
# Fractional position within tile (0.0 to 1.0)
|
||||
lat_frac = lat - lat_int # 0 = south edge, 1 = north edge
|
||||
lon_frac = lon - lon_int # 0 = west edge, 1 = east edge
|
||||
|
||||
# Convert to row/col (note: rows go north to south!)
|
||||
row_exact = (1.0 - lat_frac) * 3600.0 # 0 = north, 3600 = south
|
||||
col_exact = lon_frac * 3600.0 # 0 = west, 3600 = east
|
||||
|
||||
# Four surrounding grid points
|
||||
r0 = int(row_exact)
|
||||
c0 = int(col_exact)
|
||||
r1 = min(r0 + 1, 3600)
|
||||
c1 = min(c0 + 1, 3600)
|
||||
|
||||
# Fractional position between grid points
|
||||
dr = row_exact - r0
|
||||
dc = col_exact - c0
|
||||
|
||||
# Bilinear interpolation
|
||||
z00 = tile[r0, c0]
|
||||
z01 = tile[r0, c1]
|
||||
z10 = tile[r1, c0]
|
||||
z11 = tile[r1, c1]
|
||||
|
||||
# Handle NaN (void) values
|
||||
if np.isnan(z00) or np.isnan(z01) or np.isnan(z10) or np.isnan(z11):
|
||||
# Fall back to nearest non-NaN
|
||||
valid = [(z00, 0, 0), (z01, 0, 1), (z10, 1, 0), (z11, 1, 1)]
|
||||
valid = [(z, r, c) for z, r, c in valid if not np.isnan(z)]
|
||||
return valid[0][0] if valid else 0.0
|
||||
|
||||
elevation = (z00 * (1 - dr) * (1 - dc) +
|
||||
z01 * (1 - dr) * dc +
|
||||
z10 * dr * (1 - dc) +
|
||||
z11 * dr * dc)
|
||||
|
||||
return float(elevation)
|
||||
```
|
||||
|
||||
### Vectorized Batch Elevation (for GPU pipeline)
|
||||
|
||||
This replaces the current `_batch_elevation_lookup` in gpu_service.py.
|
||||
Must handle multi-tile seamlessly.
|
||||
|
||||
```python
|
||||
def get_elevations_batch(self, lats: np.ndarray, lons: np.ndarray) -> np.ndarray:
|
||||
"""Vectorized elevation lookup with bilinear interpolation.
|
||||
|
||||
Handles points spanning multiple tiles efficiently.
|
||||
Groups points by tile, processes each tile with full NumPy vectorization.
|
||||
"""
|
||||
elevations = np.zeros(len(lats), dtype=np.float32)
|
||||
|
||||
# Compute tile indices for each point
|
||||
lat_ints = np.where(lats >= 0, np.floor(lats).astype(int),
|
||||
np.floor(lats).astype(int))
|
||||
lon_ints = np.where(lons >= 0, np.floor(lons).astype(int),
|
||||
np.floor(lons).astype(int))
|
||||
|
||||
# Group by tile
|
||||
tile_keys = lat_ints * 1000 + lon_ints # unique key per tile
|
||||
unique_keys = np.unique(tile_keys)
|
||||
|
||||
for key in unique_keys:
|
||||
mask = tile_keys == key
|
||||
lat_int = int(key // 1000)
|
||||
lon_int = int(key % 1000)
|
||||
if lon_int > 500: # handle negative longitudes
|
||||
lon_int -= 1000
|
||||
|
||||
tile_name = self._make_tile_name(lat_int, lon_int)
|
||||
tile = self.load_tile(tile_name)
|
||||
|
||||
if tile is None:
|
||||
elevations[mask] = 0.0 # no data
|
||||
continue
|
||||
|
||||
# Vectorized bilinear for all points in this tile
|
||||
tile_lats = lats[mask]
|
||||
tile_lons = lons[mask]
|
||||
|
||||
lat_frac = tile_lats - lat_int
|
||||
lon_frac = tile_lons - lon_int
|
||||
|
||||
row_exact = (1.0 - lat_frac) * 3600.0
|
||||
col_exact = lon_frac * 3600.0
|
||||
|
||||
r0 = np.clip(row_exact.astype(int), 0, 3599)
|
||||
c0 = np.clip(col_exact.astype(int), 0, 3599)
|
||||
r1 = np.clip(r0 + 1, 0, 3600)
|
||||
c1 = np.clip(c0 + 1, 0, 3600)
|
||||
|
||||
dr = row_exact - r0
|
||||
dc = col_exact - c0
|
||||
|
||||
z00 = tile[r0, c0]
|
||||
z01 = tile[r0, c1]
|
||||
z10 = tile[r1, c0]
|
||||
z11 = tile[r1, c1]
|
||||
|
||||
result = (z00 * (1 - dr) * (1 - dc) +
|
||||
z01 * (1 - dr) * dc +
|
||||
z10 * dr * (1 - dc) +
|
||||
z11 * dr * dc)
|
||||
|
||||
# Handle NaN voids
|
||||
nan_mask = np.isnan(result)
|
||||
if nan_mask.any():
|
||||
result[nan_mask] = 0.0
|
||||
|
||||
elevations[mask] = result
|
||||
|
||||
return elevations
|
||||
```
|
||||
|
||||
## Integration Points
|
||||
|
||||
### 1. Replace terrain_service.py elevation lookup
|
||||
|
||||
Current terrain service downloads elevation data from an API.
|
||||
Replace with SRTMTileManager calls:
|
||||
|
||||
```python
|
||||
# OLD:
|
||||
elevation = await self.terrain_service.get_elevation(lat, lon)
|
||||
|
||||
# NEW:
|
||||
elevation = self.tile_manager.get_elevation(lat, lon)
|
||||
# Or for batch (GPU pipeline Phase 2.6):
|
||||
elevations = self.tile_manager.get_elevations_batch(lats_array, lons_array)
|
||||
```
|
||||
|
||||
### 2. Replace _batch_elevation_lookup in gpu_service.py
|
||||
|
||||
The vectorized elevation lookup in gpu_service.py currently loads tiles
|
||||
and does nearest-neighbor sampling. Replace with tile_manager.get_elevations_batch()
|
||||
which does bilinear interpolation.
|
||||
|
||||
### 3. Coverage service pre-check
|
||||
|
||||
Before starting calculation, check if all needed tiles are available:
|
||||
|
||||
```python
|
||||
missing = self.tile_manager.get_missing_tiles(site_lat, site_lon, radius_km)
|
||||
if missing:
|
||||
if has_internet:
|
||||
# Try to download missing tiles
|
||||
for tile_name in missing:
|
||||
await self.tile_manager.download_tile(tile_name)
|
||||
else:
|
||||
# Return warning to frontend
|
||||
return {"warning": f"Missing terrain tiles: {missing}. Using flat terrain."}
|
||||
```
|
||||
|
||||
### 4. Frontend notification
|
||||
|
||||
When tiles are missing, show a warning banner:
|
||||
"⚠ Terrain data not available for this area. Coverage accuracy reduced."
|
||||
|
||||
When tiles are being downloaded:
|
||||
"⬇ Downloading terrain data... (N48E033.hgt, 12.5 MB)"
|
||||
|
||||
### 5. Terrain Profile Viewer
|
||||
|
||||
The terrain profile viewer should use the same tile_manager
|
||||
for consistent elevation data. With bilinear interpolation,
|
||||
profiles will be much smoother and more accurate.
|
||||
|
||||
## Download Sources (Priority Order)
|
||||
|
||||
For auto-download when online:
|
||||
|
||||
1. **srtm.fasma.org** (no auth, direct HGT.zip download)
|
||||
URL: `https://srtm.fasma.org/N48E033.SRTMGL1.hgt.zip`
|
||||
- Free, no registration
|
||||
- SRTM1 (30m) data
|
||||
- May be slow or unreliable
|
||||
|
||||
2. **viewfinderpanoramas.org** (no auth, void-filled data)
|
||||
URL: `http://viewfinderpanoramas.org/dem1/{region}/{tile}.hgt.zip`
|
||||
- Free, no registration
|
||||
- Void areas filled from topographic maps
|
||||
- Better quality in mountainous areas
|
||||
- File naming might differ by region
|
||||
|
||||
3. **Future: UMTC sync server**
|
||||
URL: `https://rfcp.{your-domain}/api/terrain/tiles/{tile_name}.hgt`
|
||||
- Self-hosted on your infrastructure
|
||||
- Accessible via WireGuard mesh
|
||||
- Can pre-populate with full Ukraine dataset
|
||||
|
||||
## Offline Bundle Strategy
|
||||
|
||||
For installer / field deployment:
|
||||
|
||||
### Option A: Region packs
|
||||
Pre-package tiles by operational area:
|
||||
- `terrain-dnipro.zip` — 4 tiles around Dnipro area (~100 MB)
|
||||
- `terrain-ukraine-east.zip` — ~50 tiles, eastern Ukraine (~1.2 GB)
|
||||
- `terrain-ukraine-full.zip` — ~171 tiles, all Ukraine (~4.3 GB)
|
||||
|
||||
### Option B: On-demand with cache
|
||||
Ship empty, download tiles as needed on first calculation.
|
||||
Cache permanently. Works well for development/testing.
|
||||
|
||||
### Option C: Live USB bundle
|
||||
For tactical deployment, include full Ukraine terrain data
|
||||
on the live USB alongside the application. 4.3 GB is acceptable
|
||||
for a USB drive.
|
||||
|
||||
Recommend: **Option B for now** (development), **Option C for deployment**.
|
||||
|
||||
## File Changes
|
||||
|
||||
### New Files
|
||||
- `backend/app/services/terrain_manager.py` — SRTMTileManager class
|
||||
|
||||
### Modified Files
|
||||
- `backend/app/services/terrain_service.py` — Replace API calls with tile_manager
|
||||
- `backend/app/services/gpu_service.py` — Replace _batch_elevation_lookup
|
||||
- `backend/app/services/coverage_service.py` — Add missing tile pre-check
|
||||
- `backend/app/main.py` — Initialize tile_manager on startup
|
||||
|
||||
### Config
|
||||
- Add `TERRAIN_DIR` environment variable / config option
|
||||
- Default: `./data/terrain` relative to backend exe
|
||||
|
||||
## Testing
|
||||
|
||||
```powershell
|
||||
# Build and test
|
||||
cd D:\root\rfcp\backend
|
||||
pyinstaller ..\installer\rfcp-server-gpu.spec --noconfirm
|
||||
.\dist\rfcp-server\rfcp-server.exe
|
||||
```
|
||||
|
||||
### Test 1: First run (no tiles cached)
|
||||
- Start app, trigger calculation
|
||||
- Should attempt to download required tile(s)
|
||||
- If online: downloads, caches, calculates
|
||||
- If offline: warning, flat terrain fallback
|
||||
|
||||
### Test 2: Cached tiles
|
||||
- Run same calculation again
|
||||
- Tile loaded from disk cache, no download
|
||||
- Should be fast (tile load from disk < 100ms)
|
||||
|
||||
### Test 3: Accuracy comparison
|
||||
- Compare elevation at known points (e.g., Dnipro city center)
|
||||
- Cross-reference with Google Earth elevation
|
||||
- Expected accuracy: ±5m horizontal, ±16m vertical (SRTM spec)
|
||||
|
||||
### Test 4: Multi-tile calculation
|
||||
- Set radius to 50km+ to span multiple tiles
|
||||
- Verify seamless stitching at tile boundaries
|
||||
- No elevation jumps or artifacts at edges
|
||||
|
||||
### Test 5: Terrain profile
|
||||
- Draw terrain profile across tile boundary
|
||||
- Should be smooth, no discontinuity
|
||||
- Compare with Google Earth profile for same path
|
||||
|
||||
### Test 6: Performance
|
||||
- Tile load time from disk: <100ms
|
||||
- Batch elevation lookup (6000 points): <50ms
|
||||
- Should not regress overall calculation time
|
||||
- Memory: ~25 MB per loaded tile, max 16 tiles = 400 MB
|
||||
|
||||
## What NOT to Change
|
||||
|
||||
- Don't modify GPU pipeline architecture (Phase 2.5/2.6/2.7)
|
||||
- Don't change propagation model math
|
||||
- Don't change API endpoints or response format
|
||||
- Don't change frontend map or heatmap rendering
|
||||
- Don't change OSM building/vegetation fetching
|
||||
- Don't change PyInstaller build process (just add data dir)
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] SRTM1 tiles load correctly (3601×3601, 30m resolution)
|
||||
- [ ] Bilinear interpolation working (smoother than nearest-neighbor)
|
||||
- [ ] Offline mode works with pre-cached tiles
|
||||
- [ ] Auto-download works when online
|
||||
- [ ] Missing tile warning shown to user
|
||||
- [ ] Multi-tile seamless stitching
|
||||
- [ ] Terrain profile accuracy matches Google Earth within 20m
|
||||
- [ ] No performance regression (calculation time same or faster)
|
||||
- [ ] Tile cache directory configurable
|
||||
Reference in New Issue
Block a user