commit 3a08501b0db4b69a6196d4be8fbdfaf90441765f Author: Aelith Date: Tue Jun 16 13:18:39 2026 -0400 Initial commit: Nerdhalla Valheim World Map - Phase 1: POI database (11,309 locations, SQLite with spatial queries) - Phase 2: Tile decoder + Leaflet.js web viewer (3 map layers, search, categories) - Phase 3: .db/.fch parser WIP for fog-of-war overlay - Full README with roadmap, architecture, and usage docs diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..f7141a1 --- /dev/null +++ b/.gitignore @@ -0,0 +1,18 @@ +# Large data files (stored in /tmp/, not in repo) +*.json +*.db +*.bin.gz + +# Python cache +__pycache__/ +*.pyc +*.pyo + +# OS files +.DS_Store +Thumbs.db + +# Environment +.env +.venv +venv/ diff --git a/README.md b/README.md new file mode 100644 index 0000000..eb24fc4 --- /dev/null +++ b/README.md @@ -0,0 +1,287 @@ +# Nerdhalla Valheim World Map + +> **Seed:** `yzZ5fr2tGa` | **World:** 24000 × 24000 | **Biomes:** 9 | **POIs:** 11,309 + +An interactive web map of the Nerdhalla Valheim world, built from a `valheim-map.world` export. Includes biome/height map tiles, a searchable POI database, and a roadmap for fog-of-war integration. + +--- + +## Quick Start + +```bash +# Serve the map viewer +cd ~/projects/valheim-map/output && python3 -m http.server 8765 +# Open: http://72.60.69.120:8765/index.html +``` + +**Live:** http://72.60.69.120:8765/index.html (port 8765, ufw allowed) + +--- + +## Project Structure + +``` +~/projects/valheim-map/ +├── src/ # Source scripts +│ ├── valheim_db.py # Phase 1: Build POI SQLite database from locations.json +│ ├── decode_valheim_tiles.py # Phase 2: Decode .bin.gz tiles → PNG images +│ ├── build_viewer.py # Phase 2: Generate Leaflet.js web viewer +│ ├── parse_valheim_db.py # Phase 3: Attempted .db parser (WIP) +│ └── parse_valheim_zpackage.py # Phase 3: ZPackage format exploration (WIP) +├── data/ # Data files (small) +│ ├── valheim_poi.db # SQLite POI database (55 MB) +│ └── Nerdhalla.fwl # World metadata file (499 bytes) +├── output/ # Generated map assets +│ ├── index.html # Leaflet.js web viewer (5.4 MB) +│ ├── world_composite.png # Full world map, biome+height (3.7 MB) +│ ├── world_biome.png # Full world biome classification (374 KB) +│ ├── world_height.png # Full world heightmap (2.4 MB) +│ ├── XX-YY_*.png # Individual tile images (16 tiles × 3 layers) +│ └── explored.png # (future) Explored/unexplored overlay +├── notes/ # Reference notes +│ └── tile-format-readme.txt # Original valheim-map.world tile format spec +└── README.md # This file +``` + +**Large files not in repo** (stored in /tmp/): +- `/tmp/valheim_locations.json` — 52 MB raw location export +- `/tmp/Nerdhalla.db` — 85 MB world save from Nerdcade AMP container +- `/tmp/Nerdhalla.json` — 1.5 GB JSON conversion (valheim-save-tools) +- `/tmp/valheim_tiles/tiles/*.bin.gz` — 16 tile files, ~100 MB total + +--- + +## Roadmap + +### ✅ Phase 1 — POI Database (Complete) + +**Script:** `src/valheim_db.py` + +Parsed `locations.json` (52 MB) into a queryable SQLite database with spatial support. + +**Tables:** +- `locations` — 11,309 entries (prefab name, category, coordinates, dungeon/loot flags) +- `important_contents` — 24,407 entries (veggisirs, surtling cores, treasure, spawners) +- `random_contents` — 457,617 entries (dungeon interior loot) +- `dungeon_components` — 49,338 entries (room layouts) + +**Spatial queries work:** +- "Highest density of tar pits?" → Zone (1500, 3500) has 11 in 1000u radius +- "Most Surtling Cores?" → Crypt3 @ (2039, -1162) has 20 cores +- "Best base location for tar farming?" → (1210, -4159) — 5 tar pits in 200m, Plains biome + +### ✅ Phase 2 — Web Map Viewer (Complete) + +**Scripts:** `src/decode_valheim_tiles.py`, `src/build_viewer.py` + +Decoded 16 tile files (4×4 grid, 1024×1024 samples each) into PNG images and built a Leaflet.js viewer. + +**Features:** +- **3 map layers:** Composite (biome+height shading), Biome (9-color classification), Height (grayscale elevation) +- **11,309 POI markers** — color-coded by category, clickable with loot popups +- **Category legend** — toggle groups on/off (boss, dungeon, vegvisir, vendor, resource, etc.) +- **Search** — by prefab name or category +- **Coordinate display** — Valheim world coords in popups + +**Layer toggle buttons** (bottom-left): Composite | Biome | Height + +### ⏳ Phase 3 — Fog-of-War Overlay (Blocked) + +**Goal:** Show explored vs unexplored areas on the map. + +**Blocked by:** The explored map data is stored per-player in `.fch` character files on each player's local machine, NOT in the world `.db` file. The `.db` contains 1.8M ZDOs (world objects, terrain, structures) but no explored bitmap. + +**What we have:** +- `Nerdhalla.db` (85 MB) — pulled from Nerdcade AMP container, parsed via valheim-save-tools +- `Nerdhalla.fwl` (499 bytes) — world settings (seed, presets, difficulty) +- ZPackage parser WIP in `src/parse_valheim_zpackage.py` + +**What's needed:** Player `.fch` files from: +- Windows: `%USERPROFILE%\AppData\LocalLow\IronGate\Valheim\characters\` +- Linux: `~/.config/unity3d/IronGate/Valheim/characters/` + +### 🔮 Phase 4 — Cartography Table Integration (Planned) + +**Goal:** A shared map of truth for the server. + +**Approach options (in order of preference):** + +1. **ServerSideMap mod (BepInEx)** — Gold standard. Strips map data from client files, hosts on server in real-time. As you explore, fog clears for everyone. Pin sharing with dedup. Requires BepInEx on AMP Valheim instance + all players install the mod. + +2. **Periodic merge script** — Collect `.fch` files from players, merge explored bitmaps (OR operation), deduplicate pins (same name + within 15m), write back clean files. Run on cron. + +3. **Cartography table automation** — Lightweight BepInEx plugin that auto-triggers "Record Discoveries" near the table. Still manual but removes forgetfulness. + +**Pin cleaning strategy:** +- Dedup by name + proximity (15m radius) +- "Last writer wins" for conflicting pins +- Ghost pin prevention: track pin origin, only remove if no active player has it + +--- + +## Technical Details + +### Tile Format + +From the valheim-map.world export Readme: + +``` +struct MapSample { + uint16_t biome; // 2 bytes + float height; // 4 bytes + float forestFactor; // 4 bytes +} // 10 bytes per sample +``` + +- 1024 × 1024 samples per tile = 10,485,760 bytes uncompressed +- 4 × 4 tiles = 16 tiles, 24000 × 24000 Valheim units +- Gzip compressed individually + +**Biome enum:** None=0, Meadows=1, Swamp=2, Mountain=4, BlackForest=8, Plains=16, AshLands=32, DeepNorth=64, Ocean=256, Mistlands=512 + +### Coordinate Mapping + +``` +Valheim coords: (-12000, -12000) to (+12000, +12000) +Image coords: (0, 0) to (4096, 4096) + +pixel_x = (valheim_x + 12000) / 24000 * 4096 +pixel_y = (valheim_z + 12000) / 24000 * 4096 +``` + +### Zone System + +From the [Valheim Wiki](https://valheim.fandom.com/wiki/Zones): +- Zones are 64m × 64m +- Generated: 9×9 around each player (288m radius) +- Loaded: 5×5 (160m radius) +- Active: 3×3 (96m radius) +- Current: the zone(s) containing the player + +### Database Schema + +```sql +-- Core locations table +CREATE TABLE locations ( + id INTEGER PRIMARY KEY, + prefab_name TEXT NOT NULL, + category TEXT NOT NULL, + pos_x REAL, pos_y REAL, pos_z REAL, + has_dungeon INTEGER DEFAULT 0, + has_important INTEGER DEFAULT 0, + has_random INTEGER DEFAULT 0 +); + +-- Important contents (veggisirs, cores, treasure) +CREATE TABLE important_contents ( + id INTEGER PRIMARY KEY, + location_id INTEGER NOT NULL, + friendly_name TEXT, + count INTEGER, + icon TEXT, + FOREIGN KEY (location_id) REFERENCES locations(id) +); + +-- Random contents (dungeon interior loot) +CREATE TABLE random_contents ( + id INTEGER PRIMARY KEY, + location_id INTEGER NOT NULL, + type TEXT, parent TEXT, name TEXT, interior INTEGER, + FOREIGN KEY (location_id) REFERENCES locations(id) +); + +-- Dungeon components (room layouts) +CREATE TABLE dungeon_components ( + id INTEGER PRIMARY KEY, + location_id INTEGER NOT NULL, + delta_x REAL, delta_y REAL, rotation_y REAL, + parent_type TEXT, parent_name TEXT, + FOREIGN KEY (location_id) REFERENCES locations(id) +); +``` + +### .db File Format (ZPackage) + +The world `.db` file uses Unity's ZPackage binary serialization format: + +``` +int32: world_version +double: net_time +int64: my_id +uint32: next_uid +int32: num_zdos +ZDO[]: zone data objects (1,800,945 in Nerdhalla) +Zones: zone management data +RandomEvent: random event state +SHA512: 64-byte hash +``` + +Each ZDO contains: uid, prefab hash, owner, data type, revision, key-value pairs (floats, vectors, strings, byte arrays), position, rotation. + +**Current world version:** 37 (valheim-save-tools max: 34 — parses with warning) + +--- + +## Usage Examples + +### Query the POI database + +```bash +sqlite3 ~/projects/valheim-map/data/valheim_poi.db + +# Find all tar pits +SELECT pos_x, pos_z FROM locations WHERE prefab_name LIKE '%TarPit%'; + +# Find boss altars +SELECT prefab_name, pos_x, pos_z FROM locations WHERE category='boss'; + +# Find dungeons with most surtling cores +SELECT l.prefab_name, l.pos_x, l.pos_z, SUM(ic.count) as total +FROM locations l JOIN important_contents ic ON l.id = ic.location_id +WHERE ic.friendly_name LIKE '%SurtlingCore%' +GROUP BY l.id ORDER BY total DESC LIMIT 10; + +# Find vegvisirs +SELECT ic.friendly_name, l.pos_x, l.pos_z +FROM locations l JOIN important_contents ic ON l.id = ic.location_id +WHERE ic.friendly_name LIKE '%Vegvisir%'; +``` + +### Rebuild from scratch + +```bash +# 1. Build POI database +python3 src/valheim_db.py + +# 2. Decode tiles to PNG +python3 src/decode_valheim_tiles.py + +# 3. Build web viewer +python3 src/build_viewer.py + +# 4. Serve +cd output && python3 -m http.server 8765 +``` + +--- + +## Data Sources + +| Source | File | Size | Origin | +|--------|------|------|--------| +| valheim-map.world export | `locations.json` | 52 MB | ROG (E:\Downloads\MapData_yzZ5fr2tGa\) | +| valheim-map.world export | `tiles/*.bin.gz` | ~100 MB | ROG | +| AMP Valheim container | `Nerdhalla.db` | 85 MB | Nerdcade (`docker cp AMP_Nerdhalla01:/AMP/Valheim/896660/Saves/worlds_local/`) | +| AMP Valheim container | `Nerdhalla.fwl` | 499 B | Nerdcade | + +--- + +## Future Work + +- [ ] **Fog-of-war overlay** — get `.fch` files from players, extract explored bitmap +- [ ] **ServerSideMap mod** — install BepInEx on AMP Valheim, deploy shared map +- [ ] **Pin cleaning/syncing** — dedup, merge, ghost pin prevention +- [ ] **Auto-update cron** — periodically pull `.db` from Nerdcade, regenerate map +- [ ] **Forgejo repo** — host this project in a self-hosted git instance +- [ ] **FoundryVTT export** — convert POI data for tabletop use +- [ ] **Mobile-friendly** — responsive layout for phone viewing diff --git a/data/Nerdhalla.fwl b/data/Nerdhalla.fwl new file mode 100644 index 0000000..f0a2c59 Binary files /dev/null and b/data/Nerdhalla.fwl differ diff --git a/notes/tile-format-readme.txt b/notes/tile-format-readme.txt new file mode 100644 index 0000000..e9ee0b1 --- /dev/null +++ b/notes/tile-format-readme.txt @@ -0,0 +1,19 @@ +Tile format (from valheim-map.world export Readme.txt): + +Each tile is a gzip-compressed binary file containing a flat array of: + struct MapSample { + uint16_t biome; // 2 bytes - biome enum + float height; // 4 bytes - height in Valheim units + float forestFactor; // 4 bytes - forest density + } // = 10 bytes per sample + +1024 x 1024 samples per tile = 10,485,760 bytes uncompressed +4x4 tiles = 16 tiles total +World: 24000 x 24000 Valheim units + +Biome enum: + None=0, Meadows=1, Swamp=2, Mountain=4, BlackForest=8, + Plains=16, AshLands=32, DeepNorth=64, Ocean=256, Mistlands=512 + +locations.json: list of every location with prefab name, position, contents +map.json: world metadata (seed, tile size, version) diff --git a/output/00-00_biome.png b/output/00-00_biome.png new file mode 100644 index 0000000..e70246d Binary files /dev/null and b/output/00-00_biome.png differ diff --git a/output/00-00_composite.png b/output/00-00_composite.png new file mode 100644 index 0000000..346fbca Binary files /dev/null and b/output/00-00_composite.png differ diff --git a/output/00-00_height.png b/output/00-00_height.png new file mode 100644 index 0000000..8ad6790 Binary files /dev/null and b/output/00-00_height.png differ diff --git a/output/00-01_biome.png b/output/00-01_biome.png new file mode 100644 index 0000000..c8ee9ce Binary files /dev/null and b/output/00-01_biome.png differ diff --git a/output/00-01_composite.png b/output/00-01_composite.png new file mode 100644 index 0000000..f3bd959 Binary files /dev/null and b/output/00-01_composite.png differ diff --git a/output/00-01_height.png b/output/00-01_height.png new file mode 100644 index 0000000..f68843d Binary files /dev/null and b/output/00-01_height.png differ diff --git a/output/00-02_biome.png b/output/00-02_biome.png new file mode 100644 index 0000000..b19e32f Binary files /dev/null and b/output/00-02_biome.png differ diff --git a/output/00-02_composite.png b/output/00-02_composite.png new file mode 100644 index 0000000..b037fd6 Binary files /dev/null and b/output/00-02_composite.png differ diff --git a/output/00-02_height.png b/output/00-02_height.png new file mode 100644 index 0000000..cbe247c Binary files /dev/null and b/output/00-02_height.png differ diff --git a/output/00-03_biome.png b/output/00-03_biome.png new file mode 100644 index 0000000..ca22585 Binary files /dev/null and b/output/00-03_biome.png differ diff --git a/output/00-03_composite.png b/output/00-03_composite.png new file mode 100644 index 0000000..2d74bed Binary files /dev/null and b/output/00-03_composite.png differ diff --git a/output/00-03_height.png b/output/00-03_height.png new file mode 100644 index 0000000..1db74b7 Binary files /dev/null and b/output/00-03_height.png differ diff --git a/output/01-00_biome.png b/output/01-00_biome.png new file mode 100644 index 0000000..7b61993 Binary files /dev/null and b/output/01-00_biome.png differ diff --git a/output/01-00_composite.png b/output/01-00_composite.png new file mode 100644 index 0000000..bc4c66b Binary files /dev/null and b/output/01-00_composite.png differ diff --git a/output/01-00_height.png b/output/01-00_height.png new file mode 100644 index 0000000..3b85215 Binary files /dev/null and b/output/01-00_height.png differ diff --git a/output/01-01_biome.png b/output/01-01_biome.png new file mode 100644 index 0000000..4c0eaa9 Binary files /dev/null and b/output/01-01_biome.png differ diff --git a/output/01-01_composite.png b/output/01-01_composite.png new file mode 100644 index 0000000..4b49e98 Binary files /dev/null and b/output/01-01_composite.png differ diff --git a/output/01-01_height.png b/output/01-01_height.png new file mode 100644 index 0000000..d5eaa14 Binary files /dev/null and b/output/01-01_height.png differ diff --git a/output/01-02_biome.png b/output/01-02_biome.png new file mode 100644 index 0000000..c0f8061 Binary files /dev/null and b/output/01-02_biome.png differ diff --git a/output/01-02_composite.png b/output/01-02_composite.png new file mode 100644 index 0000000..ed9987e Binary files /dev/null and b/output/01-02_composite.png differ diff --git a/output/01-02_height.png b/output/01-02_height.png new file mode 100644 index 0000000..eea2ff9 Binary files /dev/null and b/output/01-02_height.png differ diff --git a/output/01-03_biome.png b/output/01-03_biome.png new file mode 100644 index 0000000..c4d1d8b Binary files /dev/null and b/output/01-03_biome.png differ diff --git a/output/01-03_composite.png b/output/01-03_composite.png new file mode 100644 index 0000000..92c1c66 Binary files /dev/null and b/output/01-03_composite.png differ diff --git a/output/01-03_height.png b/output/01-03_height.png new file mode 100644 index 0000000..3a29e48 Binary files /dev/null and b/output/01-03_height.png differ diff --git a/output/02-00_biome.png b/output/02-00_biome.png new file mode 100644 index 0000000..84e48eb Binary files /dev/null and b/output/02-00_biome.png differ diff --git a/output/02-00_composite.png b/output/02-00_composite.png new file mode 100644 index 0000000..03ed67e Binary files /dev/null and b/output/02-00_composite.png differ diff --git a/output/02-00_height.png b/output/02-00_height.png new file mode 100644 index 0000000..1c580d8 Binary files /dev/null and b/output/02-00_height.png differ diff --git a/output/02-01_biome.png b/output/02-01_biome.png new file mode 100644 index 0000000..2b92422 Binary files /dev/null and b/output/02-01_biome.png differ diff --git a/output/02-01_composite.png b/output/02-01_composite.png new file mode 100644 index 0000000..5ad0c18 Binary files /dev/null and b/output/02-01_composite.png differ diff --git a/output/02-01_height.png b/output/02-01_height.png new file mode 100644 index 0000000..ca95491 Binary files /dev/null and b/output/02-01_height.png differ diff --git a/output/02-02_biome.png b/output/02-02_biome.png new file mode 100644 index 0000000..9df0359 Binary files /dev/null and b/output/02-02_biome.png differ diff --git a/output/02-02_composite.png b/output/02-02_composite.png new file mode 100644 index 0000000..498d222 Binary files /dev/null and b/output/02-02_composite.png differ diff --git a/output/02-02_height.png b/output/02-02_height.png new file mode 100644 index 0000000..f6ba3be Binary files /dev/null and b/output/02-02_height.png differ diff --git a/output/02-03_biome.png b/output/02-03_biome.png new file mode 100644 index 0000000..0d52144 Binary files /dev/null and b/output/02-03_biome.png differ diff --git a/output/02-03_composite.png b/output/02-03_composite.png new file mode 100644 index 0000000..dd96931 Binary files /dev/null and b/output/02-03_composite.png differ diff --git a/output/02-03_height.png b/output/02-03_height.png new file mode 100644 index 0000000..59b4544 Binary files /dev/null and b/output/02-03_height.png differ diff --git a/output/03-00_biome.png b/output/03-00_biome.png new file mode 100644 index 0000000..210a464 Binary files /dev/null and b/output/03-00_biome.png differ diff --git a/output/03-00_composite.png b/output/03-00_composite.png new file mode 100644 index 0000000..896406f Binary files /dev/null and b/output/03-00_composite.png differ diff --git a/output/03-00_height.png b/output/03-00_height.png new file mode 100644 index 0000000..a79a5a1 Binary files /dev/null and b/output/03-00_height.png differ diff --git a/output/03-01_biome.png b/output/03-01_biome.png new file mode 100644 index 0000000..1d29e3c Binary files /dev/null and b/output/03-01_biome.png differ diff --git a/output/03-01_composite.png b/output/03-01_composite.png new file mode 100644 index 0000000..43ff549 Binary files /dev/null and b/output/03-01_composite.png differ diff --git a/output/03-01_height.png b/output/03-01_height.png new file mode 100644 index 0000000..7759576 Binary files /dev/null and b/output/03-01_height.png differ diff --git a/output/03-02_biome.png b/output/03-02_biome.png new file mode 100644 index 0000000..513b255 Binary files /dev/null and b/output/03-02_biome.png differ diff --git a/output/03-02_composite.png b/output/03-02_composite.png new file mode 100644 index 0000000..3b108ac Binary files /dev/null and b/output/03-02_composite.png differ diff --git a/output/03-02_height.png b/output/03-02_height.png new file mode 100644 index 0000000..5b9ee8b Binary files /dev/null and b/output/03-02_height.png differ diff --git a/output/03-03_biome.png b/output/03-03_biome.png new file mode 100644 index 0000000..272ff40 Binary files /dev/null and b/output/03-03_biome.png differ diff --git a/output/03-03_composite.png b/output/03-03_composite.png new file mode 100644 index 0000000..03593dd Binary files /dev/null and b/output/03-03_composite.png differ diff --git a/output/03-03_height.png b/output/03-03_height.png new file mode 100644 index 0000000..ed2d3d7 Binary files /dev/null and b/output/03-03_height.png differ diff --git a/output/index.html b/output/index.html new file mode 100644 index 0000000..954b00a --- /dev/null +++ b/output/index.html @@ -0,0 +1,252 @@ + + + + + +Nerdhalla — Valheim World Map + + + + + +
+ +
+ 🔍 + +
+
+ +
+

🌍 Nerdhalla

+
SeedyzZ5fr2tGa
+
World24000 x 24000
+
POIs11309
+
Items24407
+
Biomes9
+
+
+ Click a marker for details
+ Search by name or category +
+
+ +
+ + + +
+ +
+ + + + \ No newline at end of file diff --git a/output/world_biome.png b/output/world_biome.png new file mode 100644 index 0000000..2cc5f1e Binary files /dev/null and b/output/world_biome.png differ diff --git a/output/world_composite.png b/output/world_composite.png new file mode 100644 index 0000000..3bd7a18 Binary files /dev/null and b/output/world_composite.png differ diff --git a/output/world_height.png b/output/world_height.png new file mode 100644 index 0000000..6b2e00c Binary files /dev/null and b/output/world_height.png differ diff --git a/src/build_viewer.py b/src/build_viewer.py new file mode 100644 index 0000000..bfb38a4 --- /dev/null +++ b/src/build_viewer.py @@ -0,0 +1,344 @@ +#!/usr/bin/env python3 +""" +Build a Leaflet.js web viewer for the Nerdhalla Valheim world. +""" +import sqlite3, json, os + +DB_PATH = "/tmp/valheim_poi.db" +OUTPUT_DIR = "/tmp/valheim_map_output" +VIEWER_PATH = os.path.join(OUTPUT_DIR, "index.html") + +WORLD_SIZE = 24000 +PIXELS_TOTAL = 4096 + +def valheim_to_pixel(vx, vz): + px = (vx + WORLD_SIZE/2) / WORLD_SIZE * PIXELS_TOTAL + py = (vz + WORLD_SIZE/2) / WORLD_SIZE * PIXELS_TOTAL + return px, py + +def query_pois(): + conn = sqlite3.connect(DB_PATH) + c = conn.cursor() + c.execute(""" + SELECT id, prefab_name, category, pos_x, pos_z, has_dungeon, has_important + FROM locations WHERE pos_x IS NOT NULL AND pos_z IS NOT NULL + ORDER BY category, prefab_name + """) + pois = [] + for row in c.fetchall(): + loc_id, name, cat, x, z, has_dungeon, has_important = row + px, py = valheim_to_pixel(x, z) + pois.append({ + 'id': loc_id, 'name': name, 'category': cat, + 'x': round(x, 1), 'z': round(z, 1), + 'px': round(px, 1), 'py': round(py, 1), + 'has_dungeon': bool(has_dungeon), 'has_important': bool(has_important), + }) + + c.execute(""" + SELECT l.id, l.prefab_name, l.pos_x, l.pos_z, ic.friendly_name, ic.count + FROM locations l JOIN important_contents ic ON l.id = ic.location_id + WHERE l.pos_x IS NOT NULL AND l.pos_z IS NOT NULL + ORDER BY ic.count DESC + """) + contents = [] + for row in c.fetchall(): + loc_id, name, x, z, fname, count = row + px, py = valheim_to_pixel(x, z) + contents.append({ + 'loc_id': loc_id, 'loc_name': name, 'item': fname, 'count': count, + 'x': round(x, 1), 'z': round(z, 1), + 'px': round(px, 1), 'py': round(py, 1), + }) + conn.close() + return pois, contents + +def build_viewer(pois, contents): + pois_json = json.dumps(pois) + contents_json = json.dumps(contents) + + cat_colors = { + 'boss': '#ff4444', 'dungeon': '#ff8800', 'vegvisir': '#44aaff', + 'altar': '#aa88ff', 'vendor': '#ffdd00', 'structure': '#88cc88', + 'resource': '#ff66aa', 'spawner': '#ff44ff', 'point_of_interest': '#44ffaa', + 'other': '#888888', + } + cat_colors_json = json.dumps(cat_colors) + + cat_labels = { + 'boss': 'Boss Altars', 'dungeon': 'Dungeons', 'vegvisir': 'Vegvisirs', + 'vendor': 'Vendors', 'altar': 'Altars / Stones', 'structure': 'Structures', + 'resource': 'Resources', 'spawner': 'Spawners', 'point_of_interest': 'POIs', + 'other': 'Other', + } + cat_labels_json = json.dumps(cat_labels) + + # Build category counts + cat_counts = {} + for p in pois: + cat_counts[p['category']] = cat_counts.get(p['category'], 0) + 1 + cat_counts_json = json.dumps(cat_counts) + + html = """ + + + + +Nerdhalla — Valheim World Map + + + + + +
+ +
+ 🔍 + +
+
+ +
+

🌍 Nerdhalla

+
SeedyzZ5fr2tGa
+
World24000 x 24000
+
POIs""" + str(len(pois)) + """
+
Items""" + str(len(contents)) + """
+
Biomes9
+
+
+ Click a marker for details
+ Search by name or category +
+
+ +
+ + + +
+ +
+ + + +""" + + with open(VIEWER_PATH, 'w') as f: + f.write(html) + print(f"Viewer written to {VIEWER_PATH}") + +if __name__ == '__main__': + print("Building Valheim map viewer...") + pois, contents = query_pois() + print(f"Loaded {len(pois)} POIs and {len(contents)} important contents") + build_viewer(pois, contents) + print("Done!") diff --git a/src/decode_valheim_tiles.py b/src/decode_valheim_tiles.py new file mode 100644 index 0000000..6e8bd7a --- /dev/null +++ b/src/decode_valheim_tiles.py @@ -0,0 +1,254 @@ +#!/usr/bin/env python3 +""" +Decode Valheim map tiles (from valheim-map.world export) into PNG images. +Tile format per sample: uint16_t biome + float height + float forestFactor = 10 bytes. +1024 x 1024 samples per tile, 4x4 tiles = 24000x24000 world. +""" +import struct, gzip, os, json, math +from PIL import Image + +TILES_DIR = "/tmp/valheim_tiles/tiles" +OUTPUT_DIR = "/tmp/valheim_map_output" +MAP_JSON_PATH = "/tmp/valheim_map_data/map.json" + +# Biome colors (distinct, map-friendly) +BIOME_COLORS = { + 0: (30, 30, 40), # None (deep ocean/void) + 1: (107, 142, 35), # Meadows - olive green + 2: (85, 107, 47), # Swamp - dark olive + 4: (139, 137, 137), # Mountain - grey + 8: (34, 100, 34), # BlackForest - dark green + 16: (210, 180, 60), # Plains - gold + 32: (139, 69, 19), # AshLands - dark brown/rust + 64: (200, 220, 255), # DeepNorth - ice blue + 256: (25, 60, 120), # Ocean - deep blue + 512: (120, 80, 150), # Mistlands - purple +} + +# Biome to height-based tint mapping for compositing +BIOME_HEIGHT_KEYS = { + 1: 'meadows', + 2: 'swamp', + 4: 'mountain', + 8: 'blackforest', + 16: 'plains', + 32: 'ashlands', + 64: 'deepnorth', + 256: 'ocean', + 512: 'mistlands', +} + +def read_tile(tile_path): + """Read a compressed tile and return (biomes, heights, forest_factors) as 2D arrays. + + Format: flat array of struct { uint16_t biome; float height; float forestFactor; } + = 10 bytes per sample. No header. 1024*1024 = 1,048,576 samples = 10,485,760 bytes. + """ + with gzip.open(tile_path, 'rb') as f: + raw = f.read() + + total_samples = 1024 * 1024 + expected = total_samples * 10 + actual = len(raw) + + if actual != expected: + print(f" NOTE: Got {actual} bytes, expected {expected} (off by {actual - expected})") + + biomes = [[0]*1024 for _ in range(1024)] + heights = [[0.0]*1024 for _ in range(1024)] + forests = [[0.0]*1024 for _ in range(1024)] + + n = min(total_samples, actual // 10) + for i in range(n): + offset = i * 10 + biome, height, forest = struct.unpack(' -1000] + if not all_heights: + return Image.new('L', (1024, 1024), 128) + h_min, h_max = min(all_heights), max(all_heights) + h_range = h_max - h_min if h_max > h_min else 1 + + img = Image.new('L', (1024, 1024)) + pixels = img.load() + for y in range(1024): + for x in range(1024): + h = tile_heights[y][x] + if math.isnan(h) or h < -1000: + val = 0 + else: + val = int((h - h_min) / h_range * 255) + pixels[x, y] = max(0, min(255, val)) + return img + +def make_composite_image(tile_biomes, tile_heights): + """Create a combined biome+height map using height-shaded biome colors.""" + all_heights = [h for row in tile_heights for h in row if not math.isnan(h) and h > -1000] + if not all_heights: + return make_biome_image(tile_biomes) + h_min, h_max = min(all_heights), max(all_heights) + h_range = h_max - h_min if h_max > h_min else 1 + + img = Image.new('RGB', (1024, 1024)) + pixels = img.load() + for y in range(1024): + for x in range(1024): + biome = tile_biomes[y][x] + h = tile_heights[y][x] + + # Find dominant biome + found = 0 + for b in [512, 256, 64, 32, 16, 8, 4, 2, 1]: + if biome & b: + found = b + break + + base = BIOME_COLORS.get(found, (50, 50, 50)) + + # Height shading (darker = lower, lighter = higher) + if math.isnan(h) or h < -1000: + shade = 0.5 + else: + shade = 0.4 + 0.6 * ((h - h_min) / h_range) + + r = int(base[0] * shade) + g = int(base[1] * shade) + b = int(base[2] * shade) + pixels[x, y] = (max(0, min(255, r)), + max(0, min(255, g)), + max(0, min(255, b))) + return img + +def make_biome_and_height_tiles(): + """Process all 16 tiles into PNG images.""" + os.makedirs(OUTPUT_DIR, exist_ok=True) + + tile_names = sorted([f for f in os.listdir(TILES_DIR) if f.endswith('.bin.gz')]) + print(f"Found {len(tile_names)} tiles") + + for tile_name in tile_names: + tile_path = os.path.join(TILES_DIR, tile_name) + base = tile_name.replace('.bin.gz', '') + print(f"\nDecoding {tile_name}...") + + biomes, heights, forests = read_tile(tile_path) + print(f" Biomes: {len(set(b for row in biomes for b in row))} unique values") + + # Biome image + biome_img = make_biome_image(biomes) + biome_path = os.path.join(OUTPUT_DIR, f"{base}_biome.png") + biome_img.save(biome_path) + print(f" Biome: {biome_path}") + + # Height image + height_img = make_height_image(heights) + height_path = os.path.join(OUTPUT_DIR, f"{base}_height.png") + height_img.save(height_path) + print(f" Height: {height_path}") + + # Composite image + composite_img = make_composite_image(biomes, heights) + composite_path = os.path.join(OUTPUT_DIR, f"{base}_composite.png") + composite_img.save(composite_path) + print(f" Composite: {composite_path}") + + print(f"\nAll tiles processed. Output in {OUTPUT_DIR}/") + +def stitch_full_map(): + """Stitch the 4x4 tiles into a full world map.""" + tile_names = sorted([f for f in os.listdir(TILES_DIR) if f.endswith('.bin.gz')]) + # Parse tile coords - they're xx-yy.bin.gz + tiles = {} + for tn in tile_names: + parts = tn.replace('.bin.gz', '').split('-') + x, y = int(parts[0]), int(parts[1]) + tiles[(x, y)] = tn + + full_biome = Image.new('RGB', (4096, 4096)) + full_height = Image.new('L', (4096, 4096)) + full_composite = Image.new('RGB', (4096, 4096)) + + for (tx, ty), tn in sorted(tiles.items()): + print(f"Stitching tile {tn} at position ({tx}, {ty})...") + biomes, heights, _ = read_tile(os.path.join(TILES_DIR, tn)) + + biome_img = make_biome_image(biomes) + full_biome.paste(biome_img, (tx * 1024, ty * 1024)) + + height_img = make_height_image(heights) + full_height.paste(height_img, (tx * 1024, ty * 1024)) + + composite_img = make_composite_image(biomes, heights) + full_composite.paste(composite_img, (tx * 1024, ty * 1024)) + + world_biome_path = os.path.join(OUTPUT_DIR, "world_biome.png") + world_height_path = os.path.join(OUTPUT_DIR, "world_height.png") + world_composite_path = os.path.join(OUTPUT_DIR, "world_composite.png") + + full_biome.save(world_biome_path) + full_height.save(world_height_path) + full_composite.save(world_composite_path) + + print(f"\nFull world maps saved:") + print(f" Biome: {world_biome_path}") + print(f" Height: {world_height_path}") + print(f" Composite: {world_composite_path}") + +if __name__ == '__main__': + print("=" * 60) + print("Valheim Tile Decoder") + print("=" * 60) + print(f"Reading tiles from: {TILES_DIR}") + + # First check if PIL is available + try: + from PIL import Image + except ImportError: + print("ERROR: Pillow not installed. Install with: pip install Pillow") + sys.exit(1) + + # Check first tile structure + sample_tile = sorted([f for f in os.listdir(TILES_DIR) if f.endswith('.bin.gz')])[0] + with gzip.open(os.path.join(TILES_DIR, sample_tile), 'rb') as f: + raw = f.read() + print(f"Sample tile {sample_tile}: {len(raw)} bytes uncompressed") + print(f" Header (first 4 bytes): {raw[:4].hex()} = {struct.unpack('= expected_bitmap: + print(f"Data is at least {expected_bitmap} bytes - treating as bitmap") + bitmap = raw[:expected_bitmap] + else: + print(f"Data is {len(raw)} bytes, less than expected {expected_bitmap}") + print("Trying to interpret as int32 array...") + + # Read as int32s and see what we get + vals = struct.unpack_from(f'<{len(raw)//4}I', raw) + print(f"First 20 int32 values: {vals[:20]}") + print(f"Max value: {max(vals)}, Min value: {min(vals)}") + + # If values are 0 or 1, it might be a per-pixel explored flag + if max(vals) <= 1: + print("Values are 0/1 - treating as per-pixel explored flags") + bitmap_bytes = bytearray() + for i in range(0, len(vals), 8): + byte = 0 + for j in range(8): + if i + j < len(vals) and vals[i + j]: + byte |= (1 << j) + bitmap_bytes.append(byte) + bitmap = bytes(bitmap_bytes) + else: + # Try treating raw bytes as bitmap directly + bitmap = raw[:expected_bitmap] + + return bitmap + +def bitmap_to_image(bitmap, width=4096, height=4096): + """Convert explored bitmap to a grayscale image (white=explored, black=unexplored).""" + if bitmap is None: + return None + + img = Image.new('L', (width, height)) + pixels = img.load() + + bytes_per_row = (width + 7) // 8 + + for y in range(height): + row_start = y * bytes_per_row + for x in range(width): + byte_idx = row_start + (x // 8) + bit_idx = x % 8 + if byte_idx < len(bitmap): + explored = (bitmap[byte_idx] >> bit_idx) & 1 + pixels[x, y] = 255 if explored else 0 + else: + pixels[x, y] = 0 + + return img + +def create_fog_overlay(explored_img): + """Create a fog-of-war overlay: explored=transparent, unexplored=dark overlay.""" + if explored_img is None: + return None + + # Create RGBA image: unexplored areas get a dark overlay + overlay = Image.new('RGBA', (4096, 4096)) + overlay_pixels = overlay.load() + explored_pixels = explored_img.load() + + for y in range(4096): + for x in range(4096): + if explored_pixels[x, y] > 128: + # Explored - transparent + overlay_pixels[x, y] = (0, 0, 0, 0) + else: + # Unexplored - dark semi-transparent + overlay_pixels[x, y] = (0, 0, 0, 200) + + return overlay + +def main(): + print("=" * 60) + print("Valheim .db File Parser — Explored Map Data") + print("=" * 60) + + print(f"\nReading: {DB_PATH}") + print(f"File size: {os.path.getsize(DB_PATH)} bytes") + + save_data = read_save_file(DB_PATH) + + print(f"\nWorld count: {save_data['world_count']}") + for i, world in enumerate(save_data['worlds']): + print(f"\nWorld {i}:") + print(f" World key: {world['world_key'].hex()}") + print(f" Map data bool: {world['map_data_bool']}") + print(f" Map data length: {world['map_data_len']}") + md = world['map_data'] + print(f" num1: {md['num1']}") + print(f" num2: {md['num2']}") + print(f" Explored raw size: {len(md['explored_raw'])} bytes") + + # Show first few bytes of explored data + print(f" First 32 bytes: {md['explored_raw'][:32].hex()}") + + # Extract explored bitmap + print("\n" + "=" * 60) + print("Extracting explored bitmap...") + bitmap = extract_explored_bitmap(save_data) + + if bitmap: + print(f"\nBitmap size: {len(bitmap)} bytes") + print(f"Expected for 4096x4096: {4096*4096//8} bytes") + + # Convert to image + explored_img = bitmap_to_image(bitmap) + if explored_img: + explored_path = os.path.join(OUTPUT_DIR, "explored.png") + explored_img.save(explored_path) + print(f"\nExplored map saved: {explored_path}") + + # Count explored vs unexplored pixels + explored_pixels = sum(1 for y in range(4096) for x in range(4096) if explored_img.getpixel((x, y)) > 128) + total = 4096 * 4096 + pct = explored_pixels / total * 100 + print(f"Explored: {explored_pixels:,} / {total:,} pixels ({pct:.1f}%)") + + # Create fog overlay + overlay = create_fog_overlay(explored_img) + if overlay: + overlay_path = os.path.join(OUTPUT_DIR, "fog_overlay.png") + overlay.save(overlay_path) + print(f"Fog overlay saved: {overlay_path}") + + # Also create a composite with the world map + composite = Image.open(os.path.join(OUTPUT_DIR, "world_composite.png")).convert('RGBA') + composite.paste(overlay, (0, 0), overlay) + fogged_path = os.path.join(OUTPUT_DIR, "world_fogged.png") + composite.save(fogged_path) + print(f"Fogged world map saved: {fogged_path}") + else: + print("Could not extract explored bitmap") + # Try alternative: read raw bytes and dump structure + print("\nDumping raw structure for analysis...") + with open(DB_PATH, 'rb') as f: + raw = f.read() + + # Show first 200 bytes + print(f"First 200 bytes hex dump:") + for i in range(0, min(200, len(raw)), 16): + hex_str = ' '.join(f'{b:02x}' for b in raw[i:i+16]) + ascii_str = ''.join(chr(b) if 32 <= b < 127 else '.' for b in raw[i:i+16]) + print(f" {i:04x}: {hex_str:48s} {ascii_str}") + +if __name__ == '__main__': + main() diff --git a/src/parse_valheim_zpackage.py b/src/parse_valheim_zpackage.py new file mode 100644 index 0000000..8beb947 --- /dev/null +++ b/src/parse_valheim_zpackage.py @@ -0,0 +1,283 @@ +#!/usr/bin/env python3 +""" +Python ZPackage parser for Valheim .db files. +Extracts cartography table data (explored map bitmap + pins). +""" +import struct, gzip, io, sys, os +from PIL import Image + +DB_PATH = "/tmp/Nerdhalla.db" +OUTPUT_DIR = "/tmp/valheim_map_output" + +# Known prefab hashes for cartography table +# Valheim uses fnv-1a 64-bit hashes for prefab names +CARTOGRAPHY_TABLE_HASH = None # We'll search for it + +def fnv1a_64(data): + """FNV-1a 64-bit hash (Valheim's prefab hashing).""" + h = 0xcbf29ce484222325 + for b in data: + h ^= b + h = (h * 0x100000001b3) & 0xffffffffffffffff + return h + +# Pre-calculate common prefab hashes +PREFAB_HASHES = { + 'CartographyTable': None, +} + +class ZPackage: + """Read-only ZPackage parser (Valheim binary serialization format).""" + + def __init__(self, data): + self.data = data + self.pos = 0 + + def read_int32(self): + v = struct.unpack_from(' 0: + print(f" Scanned {i}/{num_zdos}...") + + zdo_start = zp.tell() + + # ZDO header + uid = zp.read_int64() + prefab = zp.read_int32() + owner = zp.read_int32() + data_type = zp.read_int32() + revision = zp.read_int32() + + # Check if this is a cartography table + if prefab == cart_hash or prefab == piece_hash: + print(f"\n *** FOUND Cartography Table at ZDO {i} ***") + print(f" UID: {uid}, Prefab hash: {prefab}, Owner: {owner}") + print(f" Data type: {data_type}, Revision: {revision}") + cart_found = (i, uid, prefab, zp.tell()) + break + + # Skip ZDO data based on data_type + if data_type == 0: + # No data + pass + elif data_type == 1: + # Has persistent data + _ = zp.read_int32() # data length + # Skip the data + # Actually we need to parse it properly + # For now, just skip based on remaining data + pass + elif data_type == 2: + # Has ZDO data (key-value pairs) + num_keys = zp.read_byte() + for _ in range(num_keys): + key_hash = zp.read_int32() + value_type = zp.read_byte() + if value_type == 0: # float + zp.skip(4) + elif value_type == 1: # Vector3 + zp.skip(12) + elif value_type == 2: # Quaternion + zp.skip(16) + elif value_type == 3: # int + zp.skip(4) + elif value_type == 4: # long + zp.skip(8) + elif value_type == 5: # string + _ = zp.read_string() + elif value_type == 6: # byte array + length = zp.read_int32() + zp.skip(length) + else: + print(f" Unknown value type {value_type} at {zp.tell()}") + break + elif data_type == 3: + # Both persistent and ZDO data + _ = zp.read_int32() # persistent data length + num_keys = zp.read_byte() + for _ in range(num_keys): + key_hash = zp.read_int32() + value_type = zp.read_byte() + if value_type == 0: + zp.skip(4) + elif value_type == 1: + zp.skip(12) + elif value_type == 2: + zp.skip(16) + elif value_type == 3: + zp.skip(4) + elif value_type == 4: + zp.skip(8) + elif value_type == 5: + _ = zp.read_string() + elif value_type == 6: + length = zp.read_int32() + zp.skip(length) + else: + break + + # Skip sector data + # Each ZDO has sector info: 2 shorts (sector x, y) + 3 floats (position) + 4 floats (rotation) + # Actually this varies by version. Let's just skip to next ZDO by finding it. + + # For world version >= 33, ZDOs have a compact format + # Let's just try to find the next ZDO by looking for the pattern + + if cart_found: + print(f"\nCartography table found at ZDO index {cart_found[0]}") + else: + print(f"\nCartography table not found in {num_zdos} ZDOs") + print("The explored map data might be stored differently.") + print("Let me check the character save format instead...") + + return cart_found + + +def main(): + print("=" * 60) + print("Valheim Cartography Table Extractor") + print("=" * 60) + + result = parse_db(DB_PATH) + + if result: + print("\nNext steps: extract the explored bitmap from the ZDO data") + else: + print("\nThe cartography table data may be stored in a different format.") + print("Let me check the Zones section of the .db file...") + + # The explored map data might be in the Zones section + # Or it might be stored per-player in .fch files + print("\nAlternative approach: parse the .db Zones section for explored data") + +if __name__ == '__main__': + main() diff --git a/src/valheim_db.py b/src/valheim_db.py new file mode 100644 index 0000000..2eb4080 --- /dev/null +++ b/src/valheim_db.py @@ -0,0 +1,266 @@ +#!/usr/bin/env python3 +"""Valheim World POI Database Builder +Parses locations.json into a queryable SQLite database with spatial support. +""" +import json, sqlite3, re, sys +from collections import Counter + +DB_PATH = "/tmp/valheim_poi.db" +JSON_PATH = "/tmp/valheim_locations.json" + +# ── Parse position strings like "(1.8, 81.4, 0.0)" ── +def parse_pos(s): + """Returns (x, y_height, z) — x/z are map coords, y is altitude.""" + m = re.match(r'\(?(-?[\d.]+),\s*(-?[\d.]+),\s*(-?[\d.]+)\)?', str(s)) + if m: + return float(m.group(1)), float(m.group(2)), float(m.group(3)) + return None, None, None + +def classify_prefab(name): + """Classify a prefab into a category for querying.""" + categories = { + 'boss': ['Eikthyrnir', 'Bonemass', 'GDKing', 'GoblinKing', 'Moder', 'Yagluth', 'SeekerQueen'], + 'dungeon': ['Crypt', 'SunkenCrypt', 'Cave', 'DvergrTown', 'Mistlands_Dungeon'], + 'vegvisir': ['Vegvisir'], + 'altar': ['OfferingPlace', 'Runestone', 'StoneHenge'], + 'vendor': ['Vendor_BlackForest', 'Haldor', 'Hildir'], + 'structure': ['Castle', 'Tower', 'Ruin', 'Fortress', 'Camp'], + 'resource': ['TarPit', 'Obsidian', 'Silver', 'Copper', 'Tin', 'DragonEgg', 'Mistlands_Guards'], + 'spawner': ['Spawner', 'SpawnArea'], + 'point_of_interest': ['StartTemple', 'FulingVillage', 'VikingBroad', 'StoneBridge'], + } + for cat, patterns in categories.items(): + for p in patterns: + if p.lower() in name.lower(): + return cat + return 'other' + +def create_db(conn): + c = conn.cursor() + c.executescript(""" + DROP TABLE IF EXISTS locations; + DROP TABLE IF EXISTS important_contents; + DROP TABLE IF EXISTS random_contents; + DROP TABLE IF EXISTS dungeon_components; + DROP TABLE IF EXISTS prefab_categories; + + CREATE TABLE locations ( + id INTEGER PRIMARY KEY, + prefab_name TEXT NOT NULL, + category TEXT NOT NULL, + pos_x REAL, + pos_y REAL, -- altitude/height + pos_z REAL, -- map depth + has_dungeon INTEGER DEFAULT 0, + has_important INTEGER DEFAULT 0, + has_random INTEGER DEFAULT 0 + ); + + CREATE TABLE important_contents ( + id INTEGER PRIMARY KEY, + location_id INTEGER NOT NULL, + friendly_name TEXT, + count INTEGER, + icon TEXT, + FOREIGN KEY (location_id) REFERENCES locations(id) + ); + + CREATE TABLE random_contents ( + id INTEGER PRIMARY KEY, + location_id INTEGER NOT NULL, + type TEXT, + parent TEXT, + name TEXT, + interior INTEGER, + FOREIGN KEY (location_id) REFERENCES locations(id) + ); + + CREATE TABLE dungeon_components ( + id INTEGER PRIMARY KEY, + location_id INTEGER NOT NULL, + delta_x REAL, + delta_y REAL, + rotation_y REAL, + parent_type TEXT, + parent_name TEXT, + FOREIGN KEY (location_id) REFERENCES locations(id) + ); + + CREATE INDEX idx_locations_prefab ON locations(prefab_name); + CREATE INDEX idx_locations_category ON locations(category); + CREATE INDEX idx_locations_pos ON locations(pos_x, pos_z); + CREATE INDEX idx_important_name ON important_contents(friendly_name); + CREATE INDEX idx_random_type ON random_contents(type, parent); + """) + conn.commit() + +def parse_and_insert(conn): + c = conn.cursor() + loc_id = 0 + imp_id = 0 + rnd_id = 0 + cmp_id = 0 + + prefab_counter = Counter() + category_counter = Counter() + imp_counter = Counter() + + print("Parsing locations.json...") + with open(JSON_PATH, 'r', encoding='utf-8') as f: + data = json.load(f) + + total = len(data) + print(f"Total locations: {total}") + + for i, loc in enumerate(data): + prefab = loc.get('PrefabName', 'Unknown') + pos_str = loc.get('Position', '') + x, y, z = parse_pos(pos_str) + category = classify_prefab(prefab) + + prefab_counter[prefab] += 1 + category_counter[category] += 1 + + has_dungeon = 1 if 'DungeonComponents' in loc and loc['DungeonComponents'] else 0 + has_important = 1 if 'ImportantContents' in loc and loc['ImportantContents'] else 0 + has_random = 1 if 'RandomContents' in loc and loc['RandomContents'] else 0 + + loc_id += 1 + c.execute(""" + INSERT INTO locations (id, prefab_name, category, pos_x, pos_y, pos_z, + has_dungeon, has_important, has_random) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) + """, (loc_id, prefab, category, x, y, z, has_dungeon, has_important, has_random)) + + # Important contents + for ic in loc.get('ImportantContents', []): + imp_id += 1 + friendly = ic.get('FriendlyName', '') + count = ic.get('Count', 1) + icon = ic.get('Icon', '') + imp_counter[friendly] += count + c.execute(""" + INSERT INTO important_contents (id, location_id, friendly_name, count, icon) + VALUES (?, ?, ?, ?, ?) + """, (imp_id, loc_id, friendly, count, icon)) + + # Random contents + for rc in loc.get('RandomContents', []): + rnd_id += 1 + c.execute(""" + INSERT INTO random_contents (id, location_id, type, parent, name, interior) + VALUES (?, ?, ?, ?, ?, ?) + """, (rnd_id, loc_id, rc.get('Type',''), rc.get('Parent',''), + rc.get('Name',''), rc.get('Interior',0))) + + # Dungeon components + for dc in loc.get('DungeonComponents', []): + cmp_id += 1 + ik = dc.get('ImageKey', {}) + c.execute(""" + INSERT INTO dungeon_components (id, location_id, delta_x, delta_y, + rotation_y, parent_type, parent_name) + VALUES (?, ?, ?, ?, ?, ?, ?) + """, (cmp_id, loc_id, dc.get('DeltaX',0), dc.get('DeltaY',0), + dc.get('RotationY',0), ik.get('Type',''), ik.get('Parent',''))) + + if (i + 1) % 5000 == 0: + print(f" Processed {i+1}/{total}...") + conn.commit() + + conn.commit() + + print("\n=== Summary ===") + print(f"Locations: {loc_id}") + print(f"Important contents: {imp_id}") + print(f"Random contents: {rnd_id}") + print(f"Dungeon components: {cmp_id}") + + print("\n--- By Category ---") + for cat, cnt in sorted(category_counter.items(), key=lambda x: -x[1]): + print(f" {cat}: {cnt}") + + print("\n--- Top 20 Prefabs ---") + for prefab, cnt in prefab_counter.most_common(20): + print(f" {prefab}: {cnt}") + + print("\n--- Top 20 Important Contents ---") + for name, cnt in imp_counter.most_common(20): + print(f" {name}: {cnt}") + + return conn + +def demo_queries(conn): + c = conn.cursor() + print("\n" + "="*60) + print("DEMO QUERIES") + print("="*60) + + # 1. Boss locations + print("\n--- Boss Altars ---") + for row in c.execute(""" + SELECT prefab_name, pos_x, pos_z FROM locations + WHERE category='boss' ORDER BY prefab_name + """): + print(f" {row[0]}: ({row[1]:.0f}, {row[2]:.0f})") + + # 2. Tar pits + print("\n--- Tar Pits ---") + for row in c.execute(""" + SELECT pos_x, pos_z FROM locations WHERE prefab_name LIKE '%TarPit%' + """): + print(f" Tar Pit at ({row[0]:.0f}, {row[1]:.0f})") + + # 3. Surtling cores in dungeons + print("\n--- Surtling Core Locations ---") + for row in c.execute(""" + SELECT l.prefab_name, l.pos_x, l.pos_z, SUM(ic.count) as total + FROM locations l JOIN important_contents ic ON l.id = ic.location_id + WHERE ic.friendly_name LIKE '%SurtlingCore%' + GROUP BY l.id ORDER BY total DESC LIMIT 10 + """): + print(f" {row[0]} @ ({row[1]:.0f},{row[2]:.0f}): {row[3]} cores") + + # 4. Vegvisirs + print("\n--- Vegvisirs (Boss Hint Stones) ---") + for row in c.execute(""" + SELECT ic.friendly_name, l.pos_x, l.pos_z + FROM locations l JOIN important_contents ic ON l.id = ic.location_id + WHERE ic.friendly_name LIKE '%Vegvisir%' + """): + print(f" {row[0]} at ({row[1]:.0f}, {row[2]:.0f})") + + # 5. Tar pit density - find clusters + print("\n--- Tar Pit Cluster Analysis ---") + rows = c.execute(""" + SELECT pos_x, pos_z FROM locations WHERE prefab_name LIKE '%TarPit%' + """).fetchall() + if rows: + # Find highest density zone (grid-based) + grid = {} + grid_size = 1000 # 1000 unit cells + for x, z in rows: + gx, gz = int(x // grid_size), int(z // grid_size) + grid[(gx, gz)] = grid.get((gx, gz), 0) + 1 + top = sorted(grid.items(), key=lambda x: -x[1])[:5] + for (gx, gz), cnt in top: + center_x = gx * grid_size + grid_size // 2 + center_z = gz * grid_size + grid_size // 2 + print(f" Zone ({center_x}, {center_z}): {cnt} tar pits in {grid_size}u radius") + +if __name__ == '__main__': + import os + if os.path.exists(DB_PATH): + os.remove(DB_PATH) + + conn = sqlite3.connect(DB_PATH) + conn.execute("PRAGMA journal_mode=WAL") + conn.execute("PRAGMA synchronous=NORMAL") + + create_db(conn) + parse_and_insert(conn) + demo_queries(conn) + + conn.close() + print(f"\nDatabase saved to {DB_PATH}") + print(f"Size: {os.path.getsize(DB_PATH) / 1024 / 1024:.1f} MB")