master-server replay + trampler RE: protocol, hashes, footprints, map renderer

- master_scrape.py: live master-server (ger.hologryph.com) ClientMessage replay over the two-socket /login + /connect handshake (PlayFab ticket auth). Pulled compartment defs, shop prices, research tree, storage, characters, expedition -> extracted/master_*.json - PlayFab confirmed auth-only for this title (Economy disabled); docs corrected - trampler_hashes.py: blueprint hash algo MD5(UTF8(compact-JSON)); CompartmentsHash(#1) and ConnectionsHash(#3) verified & generatable from scratch - walkerdto_to_blueprint.py: WalkerDto(expedition) -> WalkerBlueprintDto, enum int<->name, verified by storage->WS->storage round-trip - render_trampler.py: per-floor map from CompartmentsDatabase cell footprints (rotation solved via overlap check) + doors/hatches from Connections + turret arcs + cargo C1-C8 in game order - docs/MASTER_SERVER.md, docs/TRAMPLER.md; ghidra address-offset bug fixed (no -0x1000)
2026-06-16 00:35:17 +02:00
parent 3df0797acc
commit fc6b270fa8
29 changed files with 61574 additions and 0 deletions
--- a/docs/BACKEND_PLAYFAB.md
+++ b/docs/BACKEND_PLAYFAB.md
@@ -1,5 +1,14 @@
 # Backend = PlayFab (Azure) — acquiring server-authoritative data

+> **CORRECTIONS (2026-06-15, from a live playtest + code-verified RE — these override claims below):**
+> - **PlayFab is auth-only for this title.** Economy v2 is disabled (`403 "Catalog is not enabled
+>   for this title"`); the classic catalog/inventory are empty. The economy (compartment stats,
+>   prices, research) is **not** in PlayFab — it's on the master server. PlayFab does
+>   `LoginWithSteam` + profile + one `TitleData` walker blueprint (`TramplerBlueprint3`). TitleId = **56693**.
+> - **Master server is NOT gRPC/SignalR and NOT cleartext `ws://:80`.** It's a custom per-call
+>   WebSocket RPC over **`wss://:443`**, JSON, auth = PlayFab `SessionTicket` as a query param of the
+>   `Login` op. Full spec + scraper: **[MASTER_SERVER.md](MASTER_SERVER.md)** (`reverse/master_scrape.py`).
+
 ## Observed boot behavior — no active playtest (GROUND TRUTH, reported by user)

 When no playtest is running, the exact, observed sequence is:
--- a/docs/MASTER_SERVER.md
+++ b/docs/MASTER_SERVER.md
@@ -0,0 +1,75 @@
+# Master server (`*.hologryph.com`) — protocol & scrape (WORKING)
+
+Reverse-engineered from a fresh `il2cpp/dump.cs` + Ghidra decompile of the **2026-06-15 build
+(Steam buildid 23737037, GameAssembly.dll 143 MB)** and **verified live** — `reverse/master_scrape.py`
+pulls the full economy + stats from an external process (no game, no BattlEye interaction).
+
+> Supersedes earlier guesses: PlayFab is auth-only (see [BACKEND_PLAYFAB.md](BACKEND_PLAYFAB.md));
+> the master server is **not** SignalR/gRPC and **not** cleartext `ws://:80`. The June-5 build used a
+> per-call WebSocket-per-op model; **today's build changed to the two-socket `ClientMessage` protocol
+> below** — re-RE was required after the playtest updated the client.
+
+## Transport: `ClientMessage` over `wss://`, two sockets
+
+Base region URL (live): `wss://ger.hologryph.com/gameclient/` (also eus/westus/sin/bra/aus/sko/uae;
+client picks one via region selection). Everything is JSON **text** frames of:
+
+```
+ClientMessage = {"Id": <long>, "ClientMessageType": <int>, "Action": <int>, "Message": <string|null>}
+  ClientMessageType: Error=0, Ping=1, Action=2, Event=3   (requests use 2; Ping must be echoed)
+  Action (ClientAction enum): Login=0, Connect=1, GetCharacters=2, GetStorage=20, GetNews=21,
+    GetCompartmentDefinitions=23, GetDatabaseGuid=26, GetDefaultWalkerBlueprints=28, GetServers=29,
+    GetResearchTree=32, GetShopItems=39, ... (full list in il2cpp/dump.cs `enum ClientAction`)
+  Message: the request payload as a JSON *string* (nested), or null for parameterless ops.
+```
+Serialization = Newtonsoft default: **PascalCase fields, enums as integers**. Each request gets an
+incrementing `Id`; the reply is a `ClientMessage` with the **same `Id`** (`_pendingRequests`
+correlates them), so replies may interleave with `Event(3)` pushes and `Ping(1)`.
+
+Reply `Message` is either the **DTO directly** (e.g. Login → `LoginUserResultDto`) or an
+`OperationResult<T>` = `{IsSucceed,Error,Status,Result}` — `master_scrape.py` normalizes both.
+
+## The handshake (this is the part that 1008s if wrong)
+
+**Two sequential WebSocket connections:**
+
+1. **`/login`** — connect `wss://<region>/gameclient/login` (NO auth header). Send
+   `ClientMessage{Action=Login(0), Message=JSON(LoginRequest{SessionTicket, Title, Platform, ClientVersion})}`:
+   - `SessionTicket` = the **PlayFab** SessionTicket (from `LoginWithSteam`; see playfab_scrape.py)
+   - `Title` = PlayFab TitleId = `56693`; `Platform` = `1` (STEAM, int); `ClientVersion` = the build's
+     **git commit hash** (e.g. `fbe5e4370daf86d54933ff4786b3312acd8c2d98`, from `Player.log`'s
+     `Send Login … ver:'…'`; changes every build — server may version-gate → `Status=5 VERSION_MISMATCH`).
+   - Reply `Message` = `LoginUserResultDto{User{Id,PlayFabId,PlatformId,DisplayName}, SessionTicket}`.
+     **Keep that `SessionTicket`** (the server session ticket). Close the socket.
+2. **`/connect`** — connect `wss://<region>/gameclient/connect` **with HTTP upgrade header
+   `Authorization: <server SessionTicket>`** (raw, no `Bearer `). Send `ClientMessage{Action=Connect(1),
+   Message:"False"}` (the `isDebug` bool). This is the **persistent** socket; send all data ops over it.
+
+Why the earlier attempts got `1008 "User is unauthorized!"`: connecting straight to `/connect` with no
+`Authorization` header (the auth is the header, set from the ticket that only `/login` mints). There is
+**no `SetRequestHeader` for `/login`** (its ticket rides inside the Login message instead).
+
+## Data ops (over the `/connect` socket)
+
+Send `ClientMessage{Action=<n>, Message:null}`; reply `Message` carries the result. Verified live:
+
+| Action | n | Result |
+|---|---|---|
+| GetCompartmentDefinitions | 23 | `List<CompartmentDefinitionDto>` — EpbId, HP, Weight, VisualWeight, **Properties[{Key,Value}]**, CrownPrice, T1/T2/T3_MetalPrice (126 items) |
+| GetShopItems | 39 | `List<ShopItemDto>` — DefinitionName, Amount, BuyPrice[PriceDto] (54 items) |
+| GetResearchTree | 32 | `ResearchTreeJsonDto{Roots[3], Nodes[98]}` |
+| GetDefaultWalkerBlueprints | 28 | default walker blueprints |
+| GetServers / GetNews / GetDatabaseGuid / GetCharacters / GetStorage | 29/21/26/2/20 | server list / news / db guid / your characters / wallet+inventory |
+
+`PriceDto = {ItemDefinition (currency item id), Amount}`; names resolved via `extracted/item_names.json`.
+
+## How to run
+
+```bash
+venv/bin/python reverse/master_scrape.py --selftest        # offline
+venv/bin/python reverse/master_scrape.py                   # dry run (prints the exact frames)
+venv/bin/python reverse/master_scrape.py --go --data       # login + global data  -> extracted/master_*.json
+venv/bin/python reverse/master_scrape.py --go --user       # + GetStorage / GetCharacters
+```
+Token from `reverse/.secrets/playfab_token.json` (gitignored): needs `SessionTicket` (fresh PlayFab
+login) + `ClientVersion` (current build's commit hash). Output → `extracted/master_<Action>.json`.
--- a/docs/TRAMPLER.md
+++ b/docs/TRAMPLER.md
@@ -0,0 +1,83 @@
+# Trampler / walker blueprints — structure, hashes, footprints, rendering
+
+All findings from this session. Data-derived unless marked otherwise. Tooling in `reverse/`.
+
+## Where blueprints come from
+- **Your own trampler**: master-server `GetTrampler` (Action 16, `Message`=characterId) or PlayFab
+  `TitleData` (`TramplerBlueprint3`, a default). Shape = `WalkerBlueprintDto`.
+- **Another crew member's / the host's**: `GetExpedition` (Action 7) → `ExpeditionDto.Trampler`
+  (a `WalkerDto`), or `GetExpeditionWalker(characterId)`. `WalkerDto` is the same build in a
+  slightly different wrapper (see conversion below).
+- **Local saved walkers**: `.wbt` files in `…/Hologryph/Sand/Data/Walkers/*.wbt` (slightly different
+  serialization — parsed separately; not covered here).
+
+## `WalkerBlueprintDto` (the blueprint)
+```
+Id, UniqueId, Version,
+Chassis: CompartmentBlueprintDto,
+Compartments: [CompartmentBlueprintDto],   # ORDER IS MEANINGFUL = the game's order (matches Id field)
+Connections: [ConnectionBlueprintDto],
+CompartmentsHash, DefinitionsHash, ConnectionsHash   # 3 MD5 hashes
+```
+- `CompartmentBlueprintDto = {Id, EpbId, CellCoordinate{x,y,z}, DecorationsInfo, Rotation,
+  CompartmentHash, DefinitionHash}`  — `CellCoordinate` is the **origin/anchor cell**; `y` = floor.
+- `ConnectionBlueprintDto = {Id, EpbId(null), ActualDirection{x,y,z}, GridCoordinate{x,y,z},
+  SlotType, State}`.
+- Compartment **order = the game's order** (verified: cargo C1–C8 matched in-game). Same as the `Id`
+  field; arrays preserve it.
+
+## `WalkerDto` → `WalkerBlueprintDto` conversion  (`reverse/walkerdto_to_blueprint.py`)
+`WalkerDto = {Id, FirstNameIndex, SecondNameIndex, BlueprintId, BlueprintUniqueId, Chassis,
+Compartments:[CompartmentDto], Connections:[ConnectionDto]}` where `CompartmentDto={Id, Blueprint}`
+and `ConnectionDto={Id, Blueprint}`.
+- The WS form serializes **enums as ints and omits null EpbId**; the blueprint/hash ("storage") form
+  uses **enum NAME strings** and includes `EpbId:null`. Convert: unwrap `.Blueprint`, map enums
+  int→name, add `EpbId:null`. Enum maps (from dump.cs): `ConnectionSlotType`
+  DOOR0/HATCH1/STRUCTURE2/BALCONY3/DECK4; `ConnectionState` DEFAULT0/DOOR1/OPEN2;
+  `ConnectionsCount` FULL0/PARTIAL1/ERROR2.
+- Verified by a **storage→WS→storage round-trip** on the sample (byte-identical, hashes reproduce).
+
+## The 3 hashes  (`reverse/trampler_hashes.py`)
+`MD5Utility.ComputeHash(obj) = MD5( UTF8( JsonConvert.SerializeObject(obj) ) )` → **uppercase hex,
+no separators** (`X2`). Newtonsoft compact: no whitespace, PascalCase, declaration-order keys, nulls
+included, enums as **name strings**.
+- **`CompartmentsHash = MD5(compact-JSON(Compartments))`** — VERIFIED.
+- **`ConnectionsHash  = MD5(compact-JSON(Connections))`** — VERIFIED (this was the previously-hard #3).
+- `DefinitionsHash` = MD5 of the actual `CompartmentDefinitionDto`s — PROVISIONAL (hashes the
+  definitions, not blueprint fields; verify live vs a current `GetTrampler` + `GetCompartmentDefinitions`).
+
+## Footprints  (`extracted/CompartmentsDatabase.json` → `compartments[].cells[]`)
+Each compartment def has `cells[] = {position{x,y,z}, sockets{dir:{slotType:…}}, volumeOccupied,
+requireSupport}` in the compartment's **local frame**. World cells = rotate local `(x,z)` by the
+blueprint `Rotation`, add origin; `y` world = origin.y + local.y.
+- **Rotation convention (verified by 0-overlap check across 56 solid cells):**
+  `90→(z,-x), 180→(-x,-z), 270→(-z,x)`.
+- `volumeOccupied=true` = solid cell; `false` = non-solid (e.g. the reactor's north half-circle dome).
+- Reactor `Round_Metal_2x1` is actually a 2-wide tower **rising y0→y5** with a 2×2 solid base + a
+  1-row dome to the north — the EpbId "2x1" is not the real footprint.
+
+## Connections (doors / hatches)  — blueprint `Connections[]`
+Per slot: `GridCoordinate` (world cell), `ActualDirection` (face), `SlotType`, `State`.
+- **DOOR** (horizontal, `dir.y==0`): `State` DOOR/OPEN = real door, DEFAULT = wall.
+- **HATCH** (vertical, `dir.y==±1`): `State` DOOR = hatch between floors.
+- STRUCTURE / DECK / BALCONY = structural/walkable connections.
+
+## Weapons
+- `BattleRam_2x2` = **rams** (no firing arc — battering rams). `TurretSlot_*` = turret mounts;
+  `…OpenCorner…`/`…MetalCorner…` = **45° corner** mounts, others = straight/cardinal.
+- **Firing facing is NOT in the data** (weapon `Properties` empty; `CompartmentsDatabase` has no
+  aim/forward/angle field). The renderer derives facing from the **`Rotation` field** + a
+  user-supplied calibration (CALIBRATION, not pure data): straight rot0 = South, corner rot0 = SW,
+  +90° = clockwise → `face = base − rotation` (base: straight 90°, corner 135°), arc = ±90°.
+
+## Renderer  (`reverse/render_trampler.py`)
+Per-floor (y −1…3) PNG: real footprints (solid vs faded non-solid), doors/hatches, single-tile guns
+with ±90° firing wedges, rams, cargo numbered **C1–C8** in JSON order. Outputs
+`extracted/host_trampler_*.png`. Reactor column above deck 3 and legs below the base are clipped.
+
+## Other scrape tooling (this session)
+- `reverse/master_scrape.py` — master-server replay (two-socket `/login` + `/connect`; see
+  [MASTER_SERVER.md](MASTER_SERVER.md)).
+- `reverse/{capture_hosts,ws_scrape,playfab_scrape,noise_filter,resolve_decomp}.py`.
+- Ghidra: `analyzeHeadless … -postScript decomp_targets.py|disasm_targets.py` (offset bug fixed:
+  rva = il2cppdumper `Address` directly, **no −0x1000**).