Adopt master's ROM schema design (sibling_roms + files, batched
get_files_for_roms / get_siblings_for_roms) while preserving the v2-branch
features master lacks: per-user is_main_sibling on siblings and audio_meta
on rom files.
Conflict resolution:
- responses/rom.py: keep master's sibling_roms/files fields; re-graft
is_main_sibling via SiblingRomSchema.from_rom(rom, is_main_sibling=...);
restore the eager-relationship fallback in
SimpleRomSchema.from_orm_with_request (None sentinel) so the v2
/{id}/simple endpoint still returns siblings/files.
- roms_handler.py: get_siblings_for_roms now left-joins RomUser and returns
(Rom, is_main_sibling) tuples; keep both branch and master file helpers.
- drop the redundant branch-only sibling_ids field and
get_sibling_data_for_roms.
- generated types resolved to match (sibling_roms + files; RomFileSchema
keeps audio_meta and gains archive_members).
- update v2 components and the RelatedGameCard mock to read sibling_roms.
- fix stale exclude={"siblings"} -> "sibling_roms" in scan emit payloads.
- re-chain the audio_meta migration as 0083 (after master's 0082) to keep a
single Alembic head.
- package.json: union of branch tooling + master dependency bumps; lock
regenerated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
saves responses now include one device_syncs entry per device that has
synced a save, not just the caller's, so clients can tell which devices
hold a save. is_current is computed per entry and the caller's own entry
is ordered first for backward compatibility.
add a saves.origin_device_id column (migration 0081) recording the
device that created a save, set on initial upload only, surfaced as
origin_device_id on the save schema.
The scan loaders no longer eager-load `Rom.files` (#3425 + follow-ups), so
the hash-based metadata lookups can't rely on `rom.files` being populated —
`hasheous`/`ss` `lookup_rom` read `RomFile.is_top_level`, which dereferences
`RomFile.rom.full_path` and would raise `DetachedInstanceError` once the
session closed.
Add `DBRomsHandler.rom_files_for_rom_id`, which loads a ROM's files on demand
with the `RomFile.rom` backref eager-loaded (`load_only(fs_path, fs_name)`).
The scan path uses it as a fallback only when the filesystem walk yielded no
files (e.g. an unchanged rescan), behind a per-ROM `functools.cache` helper so
the playmatch/hasheous/ss lookups share a single DB fetch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
filter_roms feeds both the gallery/list endpoint (SimpleRomSchema, no
files) and the feed endpoints (which iterate rom.files / is_top_level).
The cleanup commit's unconditional selectinload(Rom.files) + joinedload
made the gallery/list and filter-value paths pay for files they never
serialize.
Gate the files load behind a new `include_files` flag (default False),
mirroring the existing `include_file_stats` opt-in, and plumb it through
get_roms_scalar. The 9 feed endpoints that actually read rom.files opt
in; the gallery/list, filter-values, identifiers, smart-collection, and
the three feeds that don't touch files (webrcade, fpkgi, kekatsu) skip
the load entirely — keeping the gallery query at zero file cost.
https://claude.ai/code/session_01PSXKmejPRzdxLFMN6P2QQ4
Restore the "platform only" contract of `get_roms_by_fs_name` (per its
docstring) by dropping the `selectinload(Rom.files)` + `joinedload`. That
load only existed for `scan_rom`'s rare `fs_rom["files"] or rom.files`
fallback, but it forced files (and a per-file join back to roms) for every
ROM in a scan batch — expensive on large platforms, and only used when the
filesystem scan yielded no files.
Instead, fetch the persisted files on demand: `scan_rom` now resolves match
files via a small helper that returns the filesystem-scanned files, falling
back to `db_rom_handler.get_rom_files_by_rom_id(rom.id)` only when there are
none. The new getter eager-loads the `RomFile.rom` backref so `is_top_level`
keeps working on the detached results (the rare path was already latently
broken on master, which loaded files without the backref).
https://claude.ai/code/session_01PSXKmejPRzdxLFMN6P2QQ4
Replace the `_link_rom_files_to_parent` post-fetch hook with the
declarative loader pattern PR #3425 originally removed, restoring
`joinedload(RomFile.rom).load_only(Rom.fs_path, Rom.fs_name)` on the two
queries that still load `Rom.files` (`with_details` and
`get_roms_by_fs_name`).
#3425 dropped that joinedload everywhere as part of denormalizing file
stats into the `multi_file` / `top_level_file_count` column properties.
But `is_top_level` / `file_name_for_download` (multi-file downloads, 3DS
QR codes, metadata matching) still read `RomFile.rom.full_path`, so the
two file-loading paths were over-cleaned, causing a `DetachedInstanceError`
(500) on multi-file downloads once the session closed.
The gallery query (`filter_roms`) dropped `Rom.files` entirely and is
untouched, so the performance win from #3425 is preserved; the restored
join only adds an index-backed PK lookup of two columns to the existing
files `selectin` on the detail/scan paths.
https://claude.ai/code/session_01PSXKmejPRzdxLFMN6P2QQ4
PR #3425 dropped `lazy="joined"` from `RomFile.rom` and removed the
`joinedload(RomFile.rom)` from the ROM loaders to speed up the gallery
query. That left the `RomFile.rom` backref unpopulated. Single-file
downloads only read `RomFile.full_path` (built from `file_path`/
`file_name`), so they kept working, but multi-file (game folder)
downloads call `file_name_for_download()` / `is_top_level`, which read
`self.rom.full_path`. With no eager-loaded backref, that triggered a
lazy load on a detached instance once the handler session closed,
raising `DetachedInstanceError` and returning a 500.
Rather than reverting the loader changes (and the gallery gains), wire
the `RomFile.rom` backref up in Python from the parent ROM we already
hold in memory, via `set_committed_value`. This is zero extra DB cost
and only runs on the detail/download paths (`with_details` and
`get_roms_by_fs_name`); the optimized `filter_roms` gallery query is
untouched.
https://claude.ai/code/session_01PSXKmejPRzdxLFMN6P2QQ4
Three sync callsites (endpoints/sync.py, sync_watcher.py, and both
branches of tasks/sync_push_pull_task.py) ran get_saves(...) and then
discarded archival null-slot rows in a Python list comprehension. On
libraries with many archival/web-UI uploads that's a strict waste:
those rows are pulled from MariaDB, hydrated into Save model instances,
and then immediately filtered out.
Add a slot_not_null bool kwarg to DBSavesHandler.get_saves and apply
the filter in the SQL query. Update all four callsites to use it and
drop the Python-side comprehension. Default stays False so unrelated
callers keep the current behavior.
get_all_saves() materialized every Save row across all users into a
single .all() list. On instances with very large libraries that's a
real RAM ceiling and pins every row for the lifetime of the recompute
run.
Replace it with get_saves_after_id(after_id, limit) and have the
recompute task drive keyset pagination in PAGE_SIZE-row chunks. SQLAlchemy
streaming via .execution_options(yield_per=...) is incompatible with the
per-call session lifetime that @begin_session enforces (the session
exits before the consumer iterates), so keyset paging from the caller is
the cleanest fit.
Behavior is unchanged: same row coverage, same idempotency, same
counters. Memory usage drops from O(all saves) to O(PAGE_SIZE).
Cleanup pass on save-sync addressing three independent failure modes
that interact in production data: content_hash drift between client
and server, null-slot archival saves leaking into sync flows, and
content-hash dedupe collapsing legitimately-distinct slots.
Bug fixes
- compute_content_hash dispatched on zipfile.is_zipfile(relative_path),
which silently returned False whenever the process's CWD wasn't
ASSETS_BASE_PATH. Every zip save fell through to the raw-MD5 branch,
persisting hashes that disagreed with clients computing the intended
per-entry zip-hash. Resolve to a full path before the dispatch.
- _build_negotiate_plan, sync_push_pull_task, and sync_watcher all
treated null-slot saves as sync-eligible. Null-slot saves represent
web-UI / archival uploads; including them in negotiate plans matched
them against device pushes by filename and overwrote archival data.
Filter null-slot saves at all three call sites.
- get_save_by_content_hash matched on (rom_id, user_id, content_hash)
only, so identical bytes uploaded to different slots collapsed into
one record. Scope the lookup by slot when provided so clone-save-
to-new-slot creates a distinct row per slot.
- get_save_by_filename matched on (rom_id, user_id, file_name) only.
When two uploads to different slots happened in the same wall-clock
second (the datetime tag is per-second), the second upload UPDATED
the first record's slot instead of creating a distinct row. Scope
the filename lookup by slot too.
One-shot recovery
- New recompute_save_content_hashes manual task walks every Save row,
recomputes via the fixed dispatch, and updates rows whose values
differ. Idempotent; safe to re-run.
- Backend startup runs a COUNT(content_hash IS NULL) query and, if
any rows exist, enqueues the recompute task on the low-priority
RQ queue. The API process moves on; the worker handles the
recompute out-of-band. Subsequent restarts find zero NULL hashes
and skip. Admins can also trigger the task manually.
Test infrastructure
- Added tests/_zipfile_shim.reload_zipfile() mirroring the pattern
from utils/zip_cache.py for the same zipfile-inflate64 + CPython
3.13.5 incompatibility. Test fixtures that build ZIPs call it
immediately before opening the archive.
- Adjust padding in CollectionPickerRow and NewCollectionRow for a more compact layout.
- Introduce GameCard component in ManageCollectionsDialog for better visual representation of single ROMs.
- Implement SiblingBadge in GameCard to display sibling versions of ROMs.
- Add MainSiblingToggle and VersionSwitcher components to GameHeader for managing default versions and switching between ROM versions.
- Refactor CollectionSettingsDrawer to remove delete functionality, moving it to Collection.vue for consistency with other gallery surfaces.
- Enhance CollectionsIndex with specific empty state messages based on search and filter criteria.
- Update UserInterface settings to remove showSiblings toggle, integrating sibling display into groupRoms functionality.
- Refactor ScanPlatform.vue to always use RVirtualScroller for listing ROMs, improving performance and layout consistency.
- Introduce a maximum viewport height for the virtual scroller to prevent excessive DOM growth.
- Update ScanPlatformRow.vue to make each ROM row a clickable router link, enhancing navigation to ROM details.
- Add a synthetic "All" option in RSelect for metadata sources, allowing users to toggle all items in multi-select mode.
- Split metadata providers into general and specific categories for clearer organization in the Scan view.
- Improve styling and layout of the scan configuration card for better user experience.
The paginated ROM list eager-loaded sibling_roms via selectinload, which
hydrated full Rom ORM instances (including heavy JSON metadata columns)
for every sibling even though only an existence/count check was needed
on the frontend. On large collections this dominated request latency.
Split sibling handling by response shape:
- SimpleRomSchema (list): siblings is now list[int]; populated per page
by a single SELECT against the sibling_roms view projecting only
(rom_id, sibling_rom_id) — no Rom row hydration.
- DetailedRomSchema (detail): keeps full SiblingRomSchema objects, with
load_only on (id, name, fs_name_no_tags, fs_name_no_ext) so sibling
rows stop dragging in JSON metadata.
Frontend usage already only consumes siblings.length on list views; the
detail-page VersionSwitcher continues to receive the richer schema.
Drop the migration and the multi_file / top_level_file_count columns on
roms; express both as deferred column_property correlated subqueries
against rom_files instead. The gallery list and detail queries opt in
via undefer, so they get the values computed in the same SELECT via
indexed subqueries (rom_id index already in place); other code paths
that don't read the flags pay nothing.
This keeps the gallery perf win (no rom_files load for cards) without
introducing schema state that has to stay in sync with rom_files at
write time.
The gallery list endpoint was eager-loading every rom_file row for each
paginated ROM via selectinload, then re-joining each row back to its
parent rom for the is_top_level computation. For platforms with extracted
multi-file ROMs (Xbox 360 ~1394 files/ROM, Switch ~199 files/ROM), this
made /api/roms time out at 120s even with a rom_id index.
Cards never displayed individual files — only the has_simple_single_file
/ has_nested_single_file / has_multiple_files booleans that derive from
the file list. Denormalize the underlying state onto roms as multi_file
(folder-based vs single-file) and top_level_file_count, recompute the
booleans from those columns, drop the selectinload from filter_roms, and
move the files field from SimpleRomSchema to DetailedRomSchema so the
gallery payload no longer ships file rows.
Also drop the redundant joinedload(RomFile.rom) and switch the relation
to lazy="select" so subsequent file.rom accesses resolve from the
session identity map instead of re-JOINing the parent rom per file row.
ShowQRCode.vue's folder-based DS/3DS fallback now fetches the detailed
rom on demand, since SimpleRom no longer carries files.
- Deleted the MetadataSections.vue component, which handled metadata display for various providers.
- Updated AlphaStrip.vue to include a direction prop for sorting letters in ascending or descending order.
- Enhanced GalleryShell.vue to manage grid sorting direction and integrate it with the toolbar.
- Modified GalleryToolbar.vue to add sorting direction controls for grid mode.
- Adjusted PlatformTile.vue to simplify the playable indicator display.
- Refactored useGalleryVirtualItems to normalize letter sorting with new bucket logic.
- Added subtitle support to RTextField for enhanced context display.
- Updated CollectionsIndex.vue to implement grid sorting functionality.
- Refined PlatformsIndex.vue to separate list and grid sorting states, improving overall sorting logic.
- Introduced a new fixture `multi_file_rom` in `conftest.py` to create a ROM with multiple files for testing purposes.
- Updated `test_manual.py` and `test_soundtrack.py` to utilize the new fixture for testing manual uploads and soundtrack functionalities.
- Enhanced the audio tag extraction logic in `audio_tags.py` to handle oversized audio files and added a function to check allowed audio file extensions.
- Modified `nginx.py` to support inline file serving for audio files, allowing for better streaming capabilities.
- Improved error handling in the Vue components for soundtrack management, including user feedback for playback errors and metadata loading issues.
- Refactored the soundtrack player store to use local storage for volume and mute settings, simplifying state management.
- Added new localization strings for soundtrack player actions and error messages in both English (US and GB) locale files.
- Use CollectionSchema instead of ReturnType<typeof collectionsStore.getCollection>
in AddRoms.vue and RemoveRoms.vue (simpler, per gantoine review)
- Wrap bulk INSERT in a savepoint so a concurrent duplicate-key violation
is caught via IntegrityError and ignored rather than aborting the transaction
- Only bump Collection.updated_at in remove_roms_from_collection when rows
were actually deleted (rowcount > 0), matching add_roms_to_collection behavior
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace full rom_ids list replacement with atomic POST/DELETE endpoints
that add or remove individual ROMs from a collection. This prevents
concurrent rapid clicks from overwriting each other (last-write-wins).
Also fix missing session.flush() in add_rom_user() and add collection
endpoint tests.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Commit 3991e1b6e removed `@with_details` from `get_roms_by_fs_name` but
left the body using the `query` parameter that decorator was supposed
to inject, so every scan hit `'NoneType' object has no attribute
'filter'` and crashed the platform identification task.
Make the function self-contained: build `select(Rom)` directly and
eager-load only `Rom.platform`, the one relationship the scan loop
actually needs (via `rom.platform_slug` / `rom.platform.fs_slug`).
Keeps the prior commit's intent of avoiding the heavy `with_details`
eager-load on every batch.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Collapse the two parallel id lists and their mirrored chunked-update
loops into a `flips: dict[bool, list[int]]` keyed by desired state, and
drop unused rom assignments in the related tests.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The scan was spending excessive time on large platforms even when all ROMs
were already scanned. Root causes: per-ROM UPDATE queries for skipped ROMs
(10k individual writes), missing composite index on (platform_id, fs_name)
causing full table scans, NOT IN clauses with 10k+ values in
mark_missing_roms(), and redundant filesystem reads.
Changes:
- Add bulk_mark_present() for batch-updating skipped ROMs in one query
- Move skip detection from _identify_rom to the batch loop so skipped ROMs
never enter the async scan pipeline, and report progress for them
- Add composite index idx_roms_platform_id_fs_name via migration 0077
- Rewrite mark_missing_roms() with flip-based approach: mark all missing,
then un-mark present ones in chunks of 1000
- Cache filesystem reads in scan_platforms() to avoid double directory
traversal (precounting + scanning)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
