The /api/roms filter_values dict is cached in Redis keyed only by a
runtime data-version that survives app upgrades. When a filter field is
added (e.g. tags), entries written by the previous version lack the new
key, so the response fails validation (filter_values.tags "Field
required") and returns 500 until the entry's 7-day TTL elapses or a scan
invalidates it.
Embed a schema version in the cache key so reads after an upgrade ignore
entries written under the old shape and recompute instead. Pre-namespace
keys are bypassed for free by the changed key format. Bump
ROM_FILTERS_CACHE_SCHEMA_VERSION whenever the cached dict gains or loses
a field.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The single-rom endpoints (`/{id}/simple`, `/{id}`) build SiblingRomSchema
from the eager-loaded `sibling_roms` relationship. `_visible_siblings`
filters those siblings with `perms.can_see_rom(s.id, s.platform_id)`, but
the `with_details` loader restricts sibling columns via `load_only` and did
not include `platform_id`. Reading the deferred attribute after the session
closed triggered a lazy load on a detached instance, raising
DetachedInstanceError.
Add `Rom.platform_id` to the sibling `load_only` so the permission check
reads an already-loaded column.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a "Tag" gallery filter so a library can be narrowed by the custom
tags parsed from filenames (No-Intro / Redump nomenclature, e.g. Proto,
Beta, Demo), mirroring the existing data-derived multi-value filters
(regions, languages). Closes#3546.
Backend:
- New `_filter_by_tags` over `Rom.tags` with any/all/none logic; wired
through filter_roms, get_roms_scalar, the /roms endpoint (tags +
tags_logic), and the unscoped cache gate.
- Tags are data-derived, so the selectable list is surfaced via
filter_values: added to _collect_filter_values, with_filter_values,
get_rom_filters, and RomFiltersDict.
- Smart collections apply the criterion via get_smart_collection_roms.
Frontend (v1 + v2):
- Wired into the galleryFilter store, rom API service, request cache key,
both filter drawers, v2 URL persistence, the v2 refetch watcher, and
smart-collection serialization/summary + create dialog.
- Added the `platform.tag` key to all locales (distinct from the existing
`platform.tags` section heading).
Tests:
- Backend handler (any/all/none + unknown-value), endpoint, filter_values,
and smart-collection coverage.
- Frontend smartCollectionCriteria serialization/summary coverage.
AI assistance: implemented with Claude Code (Anthropic).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was the only CRLF-encoded file in the backend; every other Python
source uses LF. The earlier minimal-diff workaround left it mixed, so
convert it fully to LF (the repo standard) for internal consistency.
No logic changes in this commit. The only behavioural change to this file
in the PR remains the two lines added to get_smart_collection_roms
(metadata_providers + metadata_providers_logic).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Addresses review feedback on #3615:
- Backend: merge the new filter column map with stats_handler's
`_METADATA_SOURCE_COLUMNS` into a single `METADATA_SOURCE_COLUMNS` in
models/rom.py, imported by both. It's the union of the two prior sets,
so the stats coverage breakdown now also reports `libretro`.
- Frontend: drop the duplicated provider list from utils; derive the
filter options from the `METADATA_PROVIDERS` registry via
`METADATA_PROVIDER_FILTER_OPTIONS` (added a `name` brand field). Both
filter drawers import from the registry now.
- collections_handler.py: revert the stray CRLF->LF normalization so the
diff is just the two smart-collection lines.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds a "Metadata provider" gallery filter so a library can be narrowed by
which providers each ROM matched (IGDB, MobyGames, ScreenScraper, etc.),
mirroring the existing multi-value filters (player count, age rating).
Backend:
- New `_filter_by_metadata_providers` keyed off each provider's id column
on Rom, with any/all/none logic; wired through filter_roms,
get_roms_scalar, the /roms endpoint, and the unscoped cache gate.
- Smart collections apply the criterion via get_smart_collection_roms.
- Unknown provider slugs are ignored (no-op) so stale bookmarks or
hand-edited URLs don't error.
Frontend (v1 + v2):
- Shared METADATA_PROVIDER_OPTIONS registry; provider list is a fixed set
rather than data-derived, so options are static.
- Wired into the galleryFilter store, rom API service, request cache key,
both filter drawers, v2 URL persistence, the v2 refetch watcher, and
smart-collection serialization/summary.
- Added the `platform.metadata-provider` key to all locales.
Tests:
- Backend handler, endpoint, and smart-collection coverage (any/all/none
plus unknown-slug).
- Frontend smartCollectionCriteria serialization/summary coverage.
Also normalized collections_handler.py to LF (it was the only CRLF file in
the backend, which tripped git-diff-check on the edited lines).
AI assistance: implemented with Claude Code (Anthropic).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolves the CI blocker and a cluster of opt-out visibility "fail-open"
gaps surfaced in review of the granular permission system.
Security / correctness:
- admin oauth_scopes projection keeps canonical FULL_SCOPES order
(order_scopes) instead of sorting alphabetically, fixing the red
test_user.py::test_admin on MariaDB + Postgres.
- default-group hides no longer fail open: the resolver resolves the
effective (own-or-default) group before the hidden-entity lookup.
- /roms/by-hash and /roms/by-metadata-provider now 404-mask hidden roms.
- USERS-entity grant no longer enables admin creation: add_user and
invite-link require a real admin to mint admin accounts.
Visibility leaks closed on secondary read paths:
- feeds, sibling roms (list query + single-rom schemas), /stats counts
and per-platform breakdowns, collection rom_ids/rom_count, search_rom.
Hardening / cleanups:
- firmware/platform PUT 404-mask hidden entities; group rename conflict
returns 400 not 500; guard against removing the last default group;
kiosk read-only enforced at the fine layer; add_hidden_entity rejects
non-cascading entity types.
Frontend:
- permissionGroups.ensureLoaded coalesces concurrent callers on one
in-flight request; permissions.setGrants resets isAdmin/hidden;
CreateUserDialog no longer orphans a user when group assignment fails;
HiddenGamesPicker search rows are native buttons (keyboard/gamepad);
invite-role labels and group swatch aria-label use i18n; drop dead code
(originalRole, unused permissionsApi export).
AI assistance: changes authored with Claude Code (Claude Opus), driven by
the Copilot review and a multi-agent adversarial review, then verified
(backend pytest, frontend typecheck/vitest, i18n parity, trunk).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The /api/roms list endpoint did several O(library) computations on every
request. On a 100k-rom library each request took 4-5s. This addresses the
dominant costs, all measured on a real 100k-rom MariaDB.
- Cache rom_id_index: the full ordered id list backing virtual scroll was
recomputed (the sibling-dedup window over the whole library) on every
request, even limit=1, and shipped uncached. Memoise the unscoped scan
under the same versioned cache as the other sidecars. 2815ms -> 7ms on hit.
- Slim the sibling-dedup query: the inner derived table materialized all of
Rom (including JSON metadata blobs) for 100k rows, and carried a wide unused
fs_name_no_ext through the window's temp table (spilling the sort to disk),
plus a pointless inner ORDER BY. Select only the columns the window needs.
2.79s -> 0.86s, identical results, no schema change.
- Rewrite with_char_index: replace row_number() over the whole library (full
materialization + double filesort) with a per-letter COUNT and an
accumulate. Identical output, drops a filesort layer.
- Add idx_roms_sibling_cover covering index for the sibling_roms view
self-join, so the 7-way metadata-id OR resolves from the index instead of
reading wide rows per parent. ~8x on dense pages warm, far more cold.
AI assistance: written with Claude Code (diagnosis, query rewrites, migration,
tests).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Added new permission handling in `backend/handler/auth/dependencies.py` to support fine-grained, DB-backed permission checks.
- Enhanced user role update logic in `backend/endpoints/user.py` to prevent demotion of the last admin.
- Introduced `hidden_platform_ids` and `hidden_rom_ids` parameters in various database handlers to manage visibility based on admin settings.
- Created new endpoints for managing permission groups, user memberships, and hidden entities in `backend/tests/endpoints/test_permissions_admin.py`.
- Added tests for permissions visibility and CRUD operations in `backend/tests/endpoints/test_permissions_visibility.py` and `backend/tests/endpoints/test_permissions_me.py`.
- Updated archive handling in `backend/utils/archives.py` to improve error logging and timeout management during extraction.
- Introduced a new permission model with `PermissionGroup`, `UserPermissionOverride`, and `HiddenEntity` to manage access control.
- Added `DBPermissionsHandler` for handling permission-related database operations.
- Updated `User` model to include a foreign key to `PermissionGroup` and modified `oauth_scopes` to derive from the new permission model.
- Implemented tests to ensure the new permission model maintains parity with legacy access controls.
- Created documentation outlining the new permission system architecture and migration strategy.
Show a "where you left off" screenshot on the Home continue-playing rail and
the live-activity board, with a small cover-art thumbnail (PIP) in the corner
so the game stays identifiable. Both render at the image's natural aspect.
Backend:
- New shared util `continue_playing_screenshot(rom, latest_save)` resolving the
image in priority order: latest save's screenshot, then title screen, then
first gameplay screenshot (None → frontend falls back to cover art).
- `SimpleRomSchema.screenshot_path` populated only on the `last_played` query;
`get_latest_saves_for_roms` batch handler (+ tests).
- ActivityEntry / ActivityEntrySchema gain `screenshot_path`, computed from the
session player's latest save in both the socket and REST heartbeat paths.
Frontend:
- New shared `CoverArtPip.vue` (bottom-right 2D cover thumbnail), reused by
GameCard and ActivityCard.
- Home continue-playing rail uses `screenshot_path` + PIP, natural aspect (no
forced hero/style).
- Activity board: screenshot-forward cover + PIP, and a wrapping flex layout so
cards share a uniform height with natural-ratio widths (gallery-card
behavior).
- GameCover only keys the measured ratio by rom id for the rom's own cover, so
a `coverSrc` override (screenshot) never pollutes the gallery's ratio cache.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Introduced new API endpoints for updating visibility of saves and states.
- Added `is_public` property to `SaveSchema` and `StateSchema`.
- Created new models for user saves and states with visibility attributes.
- Updated the `SaveDataTab` component to differentiate between "Mine" and "Community" sections.
- Implemented visibility toggle functionality for user saves and states.
- Enhanced localization files to include new strings for visibility actions.
Drop the name_sort_key_custom flag/migration in favour of a flagless rule: a
key is "custom" when it no longer equals compute(name). Apply that consistently
across all three write paths so a manual sort key survives renames while a
derived key keeps following the name:
- @validates re-derives on name assignment only when the stored key still
matches the derived value; direct name_sort_key assignment stores a
normalized custom key (or reverts to derived when cleared). Handles both
kwarg orders at construction.
- update_rom mirrors the same check for the bulk update() path it bypasses.
- The edit endpoint only writes the key when the user actually changed the
field, delegating the untouched case to update_rom.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collapse the separate `sort_name` column into `name_sort_key`, which is now
the single user-settable sort field: always normalized and indexed for fast
ordering, derived from `name` by default, and overridable. A new
`name_sort_key_custom` boolean marks user/metadata overrides so they survive
renames and rescans.
- Drop the `roms.sort_name` column; repurpose migration 0085 to add
`name_sort_key_custom`.
- Derive the key via `@validates("name")` unless pinned custom; the edit
dialog, unmatch flow, and ES-DE gamelist <sortname> set custom keys.
- update_rom / scan_rom keep the columns in sync explicitly (bulk update and
construction bypass / reorder the validator).
- Frontend: edit field drives name_sort_key (empty when auto), api sends the
override only when custom, regenerated types updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Added RDropzone component for handling file uploads with a customizable interface.
- Integrated RDropzone into Patcher and Upload views, replacing previous drop zone implementations.
- Enhanced ScreenshotsTab with additional functionality for community screenshots, including visibility toggles and owner display.
- Updated styles for improved user experience and responsiveness.
- Created Storybook stories for RDropzone to demonstrate its usage and interaction.
Centralize the *_no_tags / *_no_ext / *_extension columns (derived from a
file name) behind @validates hooks instead of computing them by hand at
every write site:
- Add pure helpers (compute_file_name_parts and friends) to models.base;
the filesystem base handler now delegates to them.
- Add @validates on Rom (fs_name), BaseAsset (file_name, inherited by all
asset subclasses), and Firmware (file_name).
- update_rom keeps the fs_name-derived columns in sync on bulk update(),
which also fixes the rename path never updating fs_extension.
- Drop the now-redundant computations at the scan/rename call sites.
Also fix the migration backfill loop and a pre-existing list[str | None]
type mismatch surfaced in scan_handler. Add tests for the helpers, the
validators, and the update_rom bulk-sync path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements RFC 8628-style device authorization so clients
(argosy-launcher, grout) can pair by display instead of manually
copying tokens. Device posts to an open /api/auth/device/init with
its identifier and requested scopes; the server returns device_code
+ user_code + QR URL. User scans QR, lands at /pair/device, approves
(optionally editing name/scopes/expiry); the device's next poll on
/api/auth/device/token returns a ClientToken bound 1:1 to a newly-
created (or deduped) Device record. Downstream endpoints
(/play-sessions, /sync/negotiate) infer device_id from the bound
token so the client doesn't have to ship it on every call.
- Migrations 0080/0081: devices.client_device_identifier (unique
per user) and client_tokens.device_id FK (ON DELETE SET NULL)
- Five new endpoints under /api/auth/device (init/pending/approve/
deny/token) with Redis-backed state, per-IP rate limits, and
RFC-compliant error codes (authorization_pending, slow_down,
expired_token, access_denied)
- HybridAuthBackend surfaces bound device_id on request.state and
bumps devices.last_seen with a 5-minute debounce
- /api/users/me returns current_device_id for bound tokens so a
device can identify itself from its token alone
- Frontend approval screen at /pair/device with editable scopes/
name/expiry (defaults to Never), 3s auto-close countdown
- ClientApiTokens settings list shows bound-device chip
- 20 i18n keys added to all 17 locales; generated models updated
- 52 new tests across 13 classes; full suite 1334 passed
Planning and review assisted by Claude Code.
Adds a few new indexes to handle full-text searches instead of doing
`ILIKE` matching, improving performance substantially.
Alongside that, a few other things were done in order to improve search
performance, such as caching filter values so they're not computed on
each request to /api/roms. Overall, this should have a very noticeable
impact on large collections when using the search feature.
- Added "esbuild" version "^0.28.1" to frontend package overrides.
- Updated "exclude-newer-package" in pyproject.toml to include "vcrpy" with a date of "2026-06-17".
- Modified uv.lock to reflect the new "vcrpy" version "8.2.1" and removed platform-specific markers for dependencies.
Adopt master's ROM schema design (sibling_roms + files, batched
get_files_for_roms / get_siblings_for_roms) while preserving the v2-branch
features master lacks: per-user is_main_sibling on siblings and audio_meta
on rom files.
Conflict resolution:
- responses/rom.py: keep master's sibling_roms/files fields; re-graft
is_main_sibling via SiblingRomSchema.from_rom(rom, is_main_sibling=...);
restore the eager-relationship fallback in
SimpleRomSchema.from_orm_with_request (None sentinel) so the v2
/{id}/simple endpoint still returns siblings/files.
- roms_handler.py: get_siblings_for_roms now left-joins RomUser and returns
(Rom, is_main_sibling) tuples; keep both branch and master file helpers.
- drop the redundant branch-only sibling_ids field and
get_sibling_data_for_roms.
- generated types resolved to match (sibling_roms + files; RomFileSchema
keeps audio_meta and gains archive_members).
- update v2 components and the RelatedGameCard mock to read sibling_roms.
- fix stale exclude={"siblings"} -> "sibling_roms" in scan emit payloads.
- re-chain the audio_meta migration as 0083 (after master's 0082) to keep a
single Alembic head.
- package.json: union of branch tooling + master dependency bumps; lock
regenerated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
saves responses now include one device_syncs entry per device that has
synced a save, not just the caller's, so clients can tell which devices
hold a save. is_current is computed per entry and the caller's own entry
is ordered first for backward compatibility.
add a saves.origin_device_id column (migration 0081) recording the
device that created a save, set on initial upload only, surfaced as
origin_device_id on the save schema.
The scan loaders no longer eager-load `Rom.files` (#3425 + follow-ups), so
the hash-based metadata lookups can't rely on `rom.files` being populated —
`hasheous`/`ss` `lookup_rom` read `RomFile.is_top_level`, which dereferences
`RomFile.rom.full_path` and would raise `DetachedInstanceError` once the
session closed.
Add `DBRomsHandler.rom_files_for_rom_id`, which loads a ROM's files on demand
with the `RomFile.rom` backref eager-loaded (`load_only(fs_path, fs_name)`).
The scan path uses it as a fallback only when the filesystem walk yielded no
files (e.g. an unchanged rescan), behind a per-ROM `functools.cache` helper so
the playmatch/hasheous/ss lookups share a single DB fetch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
filter_roms feeds both the gallery/list endpoint (SimpleRomSchema, no
files) and the feed endpoints (which iterate rom.files / is_top_level).
The cleanup commit's unconditional selectinload(Rom.files) + joinedload
made the gallery/list and filter-value paths pay for files they never
serialize.
Gate the files load behind a new `include_files` flag (default False),
mirroring the existing `include_file_stats` opt-in, and plumb it through
get_roms_scalar. The 9 feed endpoints that actually read rom.files opt
in; the gallery/list, filter-values, identifiers, smart-collection, and
the three feeds that don't touch files (webrcade, fpkgi, kekatsu) skip
the load entirely — keeping the gallery query at zero file cost.
https://claude.ai/code/session_01PSXKmejPRzdxLFMN6P2QQ4
Restore the "platform only" contract of `get_roms_by_fs_name` (per its
docstring) by dropping the `selectinload(Rom.files)` + `joinedload`. That
load only existed for `scan_rom`'s rare `fs_rom["files"] or rom.files`
fallback, but it forced files (and a per-file join back to roms) for every
ROM in a scan batch — expensive on large platforms, and only used when the
filesystem scan yielded no files.
Instead, fetch the persisted files on demand: `scan_rom` now resolves match
files via a small helper that returns the filesystem-scanned files, falling
back to `db_rom_handler.get_rom_files_by_rom_id(rom.id)` only when there are
none. The new getter eager-loads the `RomFile.rom` backref so `is_top_level`
keeps working on the detached results (the rare path was already latently
broken on master, which loaded files without the backref).
https://claude.ai/code/session_01PSXKmejPRzdxLFMN6P2QQ4
Replace the `_link_rom_files_to_parent` post-fetch hook with the
declarative loader pattern PR #3425 originally removed, restoring
`joinedload(RomFile.rom).load_only(Rom.fs_path, Rom.fs_name)` on the two
queries that still load `Rom.files` (`with_details` and
`get_roms_by_fs_name`).
#3425 dropped that joinedload everywhere as part of denormalizing file
stats into the `multi_file` / `top_level_file_count` column properties.
But `is_top_level` / `file_name_for_download` (multi-file downloads, 3DS
QR codes, metadata matching) still read `RomFile.rom.full_path`, so the
two file-loading paths were over-cleaned, causing a `DetachedInstanceError`
(500) on multi-file downloads once the session closed.
The gallery query (`filter_roms`) dropped `Rom.files` entirely and is
untouched, so the performance win from #3425 is preserved; the restored
join only adds an index-backed PK lookup of two columns to the existing
files `selectin` on the detail/scan paths.
https://claude.ai/code/session_01PSXKmejPRzdxLFMN6P2QQ4
PR #3425 dropped `lazy="joined"` from `RomFile.rom` and removed the
`joinedload(RomFile.rom)` from the ROM loaders to speed up the gallery
query. That left the `RomFile.rom` backref unpopulated. Single-file
downloads only read `RomFile.full_path` (built from `file_path`/
`file_name`), so they kept working, but multi-file (game folder)
downloads call `file_name_for_download()` / `is_top_level`, which read
`self.rom.full_path`. With no eager-loaded backref, that triggered a
lazy load on a detached instance once the handler session closed,
raising `DetachedInstanceError` and returning a 500.
Rather than reverting the loader changes (and the gallery gains), wire
the `RomFile.rom` backref up in Python from the parent ROM we already
hold in memory, via `set_committed_value`. This is zero extra DB cost
and only runs on the detail/download paths (`with_details` and
`get_roms_by_fs_name`); the optimized `filter_roms` gallery query is
untouched.
https://claude.ai/code/session_01PSXKmejPRzdxLFMN6P2QQ4
Three sync callsites (endpoints/sync.py, sync_watcher.py, and both
branches of tasks/sync_push_pull_task.py) ran get_saves(...) and then
discarded archival null-slot rows in a Python list comprehension. On
libraries with many archival/web-UI uploads that's a strict waste:
those rows are pulled from MariaDB, hydrated into Save model instances,
and then immediately filtered out.
Add a slot_not_null bool kwarg to DBSavesHandler.get_saves and apply
the filter in the SQL query. Update all four callsites to use it and
drop the Python-side comprehension. Default stays False so unrelated
callers keep the current behavior.
get_all_saves() materialized every Save row across all users into a
single .all() list. On instances with very large libraries that's a
real RAM ceiling and pins every row for the lifetime of the recompute
run.
Replace it with get_saves_after_id(after_id, limit) and have the
recompute task drive keyset pagination in PAGE_SIZE-row chunks. SQLAlchemy
streaming via .execution_options(yield_per=...) is incompatible with the
per-call session lifetime that @begin_session enforces (the session
exits before the consumer iterates), so keyset paging from the caller is
the cleanest fit.
Behavior is unchanged: same row coverage, same idempotency, same
counters. Memory usage drops from O(all saves) to O(PAGE_SIZE).
