The patcher uploads the patched ROM and then fires a platform scan to
register it. When a second scan runs against the same platform around the
same time (a filesystem-watcher rescan, a scheduled rescan, or another
manual scan on a multi-worker setup), both scans could see the new file as
absent from the DB and each insert it, producing two identical library
entries for one patched file.
A platform folder can't physically hold two entries with the same name, so
a ROM is uniquely identified by (platform_id, fs_name). Enforce that with a
unique index instead of the previous plain index, which makes the duplicate
impossible. The scan's early ROM insert now adopts the row created by a
concurrent scan (catching the integrity error and skipping) instead of
failing, and ROM rename pre-checks for a name collision so it returns a
clean 409 rather than hitting the constraint.
Includes a migration that removes any pre-existing duplicates (keeping the
lowest id; dependents cascade) before upgrading the index to unique.
Fixes#3590
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0135UV8Xn2XHkRhjzhm9UptP
The /api/roms list endpoint did several O(library) computations on every
request. On a 100k-rom library each request took 4-5s. This addresses the
dominant costs, all measured on a real 100k-rom MariaDB.
- Cache rom_id_index: the full ordered id list backing virtual scroll was
recomputed (the sibling-dedup window over the whole library) on every
request, even limit=1, and shipped uncached. Memoise the unscoped scan
under the same versioned cache as the other sidecars. 2815ms -> 7ms on hit.
- Slim the sibling-dedup query: the inner derived table materialized all of
Rom (including JSON metadata blobs) for 100k rows, and carried a wide unused
fs_name_no_ext through the window's temp table (spilling the sort to disk),
plus a pointless inner ORDER BY. Select only the columns the window needs.
2.79s -> 0.86s, identical results, no schema change.
- Rewrite with_char_index: replace row_number() over the whole library (full
materialization + double filesort) with a per-letter COUNT and an
accumulate. Identical output, drops a filesort layer.
- Add idx_roms_sibling_cover covering index for the sibling_roms view
self-join, so the 7-way metadata-id OR resolves from the index instead of
reading wide rows per parent. ~8x on dense pages warm, far more cold.
AI assistance: written with Claude Code (diagnosis, query rewrites, migration,
tests).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Show a "where you left off" screenshot on the Home continue-playing rail and
the live-activity board, with a small cover-art thumbnail (PIP) in the corner
so the game stays identifiable. Both render at the image's natural aspect.
Backend:
- New shared util `continue_playing_screenshot(rom, latest_save)` resolving the
image in priority order: latest save's screenshot, then title screen, then
first gameplay screenshot (None → frontend falls back to cover art).
- `SimpleRomSchema.screenshot_path` populated only on the `last_played` query;
`get_latest_saves_for_roms` batch handler (+ tests).
- ActivityEntry / ActivityEntrySchema gain `screenshot_path`, computed from the
session player's latest save in both the socket and REST heartbeat paths.
Frontend:
- New shared `CoverArtPip.vue` (bottom-right 2D cover thumbnail), reused by
GameCard and ActivityCard.
- Home continue-playing rail uses `screenshot_path` + PIP, natural aspect (no
forced hero/style).
- Activity board: screenshot-forward cover + PIP, and a wrapping flex layout so
cards share a uniform height with natural-ratio widths (gallery-card
behavior).
- GameCover only keys the measured ratio by rom id for the rom's own cover, so
a `coverSrc` override (screenshot) never pollutes the gallery's ratio cache.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop the name_sort_key_custom flag/migration in favour of a flagless rule: a
key is "custom" when it no longer equals compute(name). Apply that consistently
across all three write paths so a manual sort key survives renames while a
derived key keeps following the name:
- @validates re-derives on name assignment only when the stored key still
matches the derived value; direct name_sort_key assignment stores a
normalized custom key (or reverts to derived when cleared). Handles both
kwarg orders at construction.
- update_rom mirrors the same check for the bulk update() path it bypasses.
- The edit endpoint only writes the key when the user actually changed the
field, delegating the untouched case to update_rom.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collapse the separate `sort_name` column into `name_sort_key`, which is now
the single user-settable sort field: always normalized and indexed for fast
ordering, derived from `name` by default, and overridable. A new
`name_sort_key_custom` boolean marks user/metadata overrides so they survive
renames and rescans.
- Drop the `roms.sort_name` column; repurpose migration 0085 to add
`name_sort_key_custom`.
- Derive the key via `@validates("name")` unless pinned custom; the edit
dialog, unmatch flow, and ES-DE gamelist <sortname> set custom keys.
- update_rom / scan_rom keep the columns in sync explicitly (bulk update and
construction bypass / reorder the validator).
- Frontend: edit field drives name_sort_key (empty when auto), api sends the
override only when custom, regenerated types updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Centralize the *_no_tags / *_no_ext / *_extension columns (derived from a
file name) behind @validates hooks instead of computing them by hand at
every write site:
- Add pure helpers (compute_file_name_parts and friends) to models.base;
the filesystem base handler now delegates to them.
- Add @validates on Rom (fs_name), BaseAsset (file_name, inherited by all
asset subclasses), and Firmware (file_name).
- update_rom keeps the fs_name-derived columns in sync on bulk update(),
which also fixes the rename path never updating fs_extension.
- Drop the now-redundant computations at the scan/rename call sites.
Also fix the migration backfill loop and a pre-existing list[str | None]
type mismatch surfaced in scan_handler. Add tests for the helpers, the
validators, and the update_rom bulk-sync path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements RFC 8628-style device authorization so clients
(argosy-launcher, grout) can pair by display instead of manually
copying tokens. Device posts to an open /api/auth/device/init with
its identifier and requested scopes; the server returns device_code
+ user_code + QR URL. User scans QR, lands at /pair/device, approves
(optionally editing name/scopes/expiry); the device's next poll on
/api/auth/device/token returns a ClientToken bound 1:1 to a newly-
created (or deduped) Device record. Downstream endpoints
(/play-sessions, /sync/negotiate) infer device_id from the bound
token so the client doesn't have to ship it on every call.
- Migrations 0080/0081: devices.client_device_identifier (unique
per user) and client_tokens.device_id FK (ON DELETE SET NULL)
- Five new endpoints under /api/auth/device (init/pending/approve/
deny/token) with Redis-backed state, per-IP rate limits, and
RFC-compliant error codes (authorization_pending, slow_down,
expired_token, access_denied)
- HybridAuthBackend surfaces bound device_id on request.state and
bumps devices.last_seen with a 5-minute debounce
- /api/users/me returns current_device_id for bound tokens so a
device can identify itself from its token alone
- Frontend approval screen at /pair/device with editable scopes/
name/expiry (defaults to Never), 3s auto-close countdown
- ClientApiTokens settings list shows bound-device chip
- 20 i18n keys added to all 17 locales; generated models updated
- 52 new tests across 13 classes; full suite 1334 passed
Planning and review assisted by Claude Code.
saves responses now include one device_syncs entry per device that has
synced a save, not just the caller's, so clients can tell which devices
hold a save. is_current is computed per entry and the caller's own entry
is ordered first for backward compatibility.
add a saves.origin_device_id column (migration 0081) recording the
device that created a save, set on initial upload only, surfaced as
origin_device_id on the save schema.
Three sync callsites (endpoints/sync.py, sync_watcher.py, and both
branches of tasks/sync_push_pull_task.py) ran get_saves(...) and then
discarded archival null-slot rows in a Python list comprehension. On
libraries with many archival/web-UI uploads that's a strict waste:
those rows are pulled from MariaDB, hydrated into Save model instances,
and then immediately filtered out.
Add a slot_not_null bool kwarg to DBSavesHandler.get_saves and apply
the filter in the SQL query. Update all four callsites to use it and
drop the Python-side comprehension. Default stays False so unrelated
callers keep the current behavior.
get_all_saves() materialized every Save row across all users into a
single .all() list. On instances with very large libraries that's a
real RAM ceiling and pins every row for the lifetime of the recompute
run.
Replace it with get_saves_after_id(after_id, limit) and have the
recompute task drive keyset pagination in PAGE_SIZE-row chunks. SQLAlchemy
streaming via .execution_options(yield_per=...) is incompatible with the
per-call session lifetime that @begin_session enforces (the session
exits before the consumer iterates), so keyset paging from the caller is
the cleanest fit.
Behavior is unchanged: same row coverage, same idempotency, same
counters. Memory usage drops from O(all saves) to O(PAGE_SIZE).
Cleanup pass on save-sync addressing three independent failure modes
that interact in production data: content_hash drift between client
and server, null-slot archival saves leaking into sync flows, and
content-hash dedupe collapsing legitimately-distinct slots.
Bug fixes
- compute_content_hash dispatched on zipfile.is_zipfile(relative_path),
which silently returned False whenever the process's CWD wasn't
ASSETS_BASE_PATH. Every zip save fell through to the raw-MD5 branch,
persisting hashes that disagreed with clients computing the intended
per-entry zip-hash. Resolve to a full path before the dispatch.
- _build_negotiate_plan, sync_push_pull_task, and sync_watcher all
treated null-slot saves as sync-eligible. Null-slot saves represent
web-UI / archival uploads; including them in negotiate plans matched
them against device pushes by filename and overwrote archival data.
Filter null-slot saves at all three call sites.
- get_save_by_content_hash matched on (rom_id, user_id, content_hash)
only, so identical bytes uploaded to different slots collapsed into
one record. Scope the lookup by slot when provided so clone-save-
to-new-slot creates a distinct row per slot.
- get_save_by_filename matched on (rom_id, user_id, file_name) only.
When two uploads to different slots happened in the same wall-clock
second (the datetime tag is per-second), the second upload UPDATED
the first record's slot instead of creating a distinct row. Scope
the filename lookup by slot too.
One-shot recovery
- New recompute_save_content_hashes manual task walks every Save row,
recomputes via the fixed dispatch, and updates rows whose values
differ. Idempotent; safe to re-run.
- Backend startup runs a COUNT(content_hash IS NULL) query and, if
any rows exist, enqueues the recompute task on the low-priority
RQ queue. The API process moves on; the worker handles the
recompute out-of-band. Subsequent restarts find zero NULL hashes
and skip. Admins can also trigger the task manually.
Test infrastructure
- Added tests/_zipfile_shim.reload_zipfile() mirroring the pattern
from utils/zip_cache.py for the same zipfile-inflate64 + CPython
3.13.5 incompatibility. Test fixtures that build ZIPs call it
immediately before opening the archive.
Add order_by and order_dir parameters to get_saves() for flexible
sorting. Supports "updated_at" and "created_at" fields with "asc" or
"desc" direction (default: desc). Enables ascending order for pruning
scenarios.
- Add device registration and save synchronization
- Implement slot-based save organization with datetime tagging
- Add conflict detection for multi-device sync scenarios
- Add content hash computation for save deduplication
- Support ZIP inner-file hashing for consistent deduplication
- Add confirm_download endpoint for sync state management
- Add overwrite parameter to bypass conflict checks
