The scan loaders no longer eager-load `Rom.files` (#3425 + follow-ups), so
the hash-based metadata lookups can't rely on `rom.files` being populated —
`hasheous`/`ss` `lookup_rom` read `RomFile.is_top_level`, which dereferences
`RomFile.rom.full_path` and would raise `DetachedInstanceError` once the
session closed.
Add `DBRomsHandler.rom_files_for_rom_id`, which loads a ROM's files on demand
with the `RomFile.rom` backref eager-loaded (`load_only(fs_path, fs_name)`).
The scan path uses it as a fallback only when the filesystem walk yielded no
files (e.g. an unchanged rescan), behind a per-ROM `functools.cache` helper so
the playmatch/hasheous/ss lookups share a single DB fetch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The five pkgj feed tests created a ROM but no RomFile, so the per-file
feeds emitted only a header and the "in response.text" assertions never
actually verified output (pre-existing failures, also red on master).
Add a top-level `.pkg` GAME file (games feeds) or a DLC-category file
(dlc feeds), mirroring the pkgi_ps3 test, so the feeds produce rows.
This also gives real coverage of the new `include_files=True` path that
these feeds rely on.
https://claude.ai/code/session_01PSXKmejPRzdxLFMN6P2QQ4
filter_roms feeds both the gallery/list endpoint (SimpleRomSchema, no
files) and the feed endpoints (which iterate rom.files / is_top_level).
The cleanup commit's unconditional selectinload(Rom.files) + joinedload
made the gallery/list and filter-value paths pay for files they never
serialize.
Gate the files load behind a new `include_files` flag (default False),
mirroring the existing `include_file_stats` opt-in, and plumb it through
get_roms_scalar. The 9 feed endpoints that actually read rom.files opt
in; the gallery/list, filter-values, identifiers, smart-collection, and
the three feeds that don't touch files (webrcade, fpkgi, kekatsu) skip
the load entirely — keeping the gallery query at zero file cost.
https://claude.ai/code/session_01PSXKmejPRzdxLFMN6P2QQ4
The SS metadata handler pre-encoded the name-search term with quote()
before handing it to the service layer, which percent-encodes the query
again via yarl's with_query(). This double-encoded any character that
needs URL-encoding (e.g. "+" -> "%2B" -> "%252B"), so the request URL
carried a doubly-escaped term.
Pass the raw (unidecode-transliterated but un-percent-encoded) term to
search_games() in both _search_rom() and get_matched_roms_by_name() and
let the URL builder encode it exactly once. The scan now sends e.g.
recherche=...%2B... instead of ...%252B...
This is a request-correctness fix. It does not, on its own, make every
previously-unmatched title match: ScreenScraper's jeuRecherche normalizes
punctuation and applies its own relevance ranking, so some titles still
return no results for the full filename-derived term (verified directly
against the API). Improving name-search robustness is a separate concern.
Add TestSearchTermEncoding regression tests covering the un-pre-encoded
term, preserved unidecode transliteration, and a single-encoded request
URL (%2B, never %252B).
Written primarily by Claude Code.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Restore the "platform only" contract of `get_roms_by_fs_name` (per its
docstring) by dropping the `selectinload(Rom.files)` + `joinedload`. That
load only existed for `scan_rom`'s rare `fs_rom["files"] or rom.files`
fallback, but it forced files (and a per-file join back to roms) for every
ROM in a scan batch — expensive on large platforms, and only used when the
filesystem scan yielded no files.
Instead, fetch the persisted files on demand: `scan_rom` now resolves match
files via a small helper that returns the filesystem-scanned files, falling
back to `db_rom_handler.get_rom_files_by_rom_id(rom.id)` only when there are
none. The new getter eager-loads the `RomFile.rom` backref so `is_top_level`
keeps working on the detached results (the rare path was already latently
broken on master, which loaded files without the backref).
https://claude.ai/code/session_01PSXKmejPRzdxLFMN6P2QQ4
Replace the `_link_rom_files_to_parent` post-fetch hook with the
declarative loader pattern PR #3425 originally removed, restoring
`joinedload(RomFile.rom).load_only(Rom.fs_path, Rom.fs_name)` on the two
queries that still load `Rom.files` (`with_details` and
`get_roms_by_fs_name`).
#3425 dropped that joinedload everywhere as part of denormalizing file
stats into the `multi_file` / `top_level_file_count` column properties.
But `is_top_level` / `file_name_for_download` (multi-file downloads, 3DS
QR codes, metadata matching) still read `RomFile.rom.full_path`, so the
two file-loading paths were over-cleaned, causing a `DetachedInstanceError`
(500) on multi-file downloads once the session closed.
The gallery query (`filter_roms`) dropped `Rom.files` entirely and is
untouched, so the performance win from #3425 is preserved; the restored
join only adds an index-backed PK lookup of two columns to the existing
files `selectin` on the detail/scan paths.
https://claude.ai/code/session_01PSXKmejPRzdxLFMN6P2QQ4
Add a shared `multi_file_rom` fixture (a game folder with multiple
RomFile rows) and an endpoint-level test that downloads it via
`GET /api/roms/{id}/content/{file_name}`. This exercises the multi-file
download path end-to-end, which builds each mod_zip manifest entry from
`file.rom.full_path` after the handler session has closed — the exact
path that 500'd with `DetachedInstanceError` before the backref fix.
The download endpoint had no test coverage for multi-file ROMs (the
`rom` fixture has no RomFile rows), which is why the regression slipped
through. Reuse the new fixture in the handler-level regression test too.
https://claude.ai/code/session_01PSXKmejPRzdxLFMN6P2QQ4
PR #3425 dropped `lazy="joined"` from `RomFile.rom` and removed the
`joinedload(RomFile.rom)` from the ROM loaders to speed up the gallery
query. That left the `RomFile.rom` backref unpopulated. Single-file
downloads only read `RomFile.full_path` (built from `file_path`/
`file_name`), so they kept working, but multi-file (game folder)
downloads call `file_name_for_download()` / `is_top_level`, which read
`self.rom.full_path`. With no eager-loaded backref, that triggered a
lazy load on a detached instance once the handler session closed,
raising `DetachedInstanceError` and returning a 500.
Rather than reverting the loader changes (and the gallery gains), wire
the `RomFile.rom` backref up in Python from the parent ROM we already
hold in memory, via `set_committed_value`. This is zero extra DB cost
and only runs on the detail/download paths (`with_details` and
`get_roms_by_fs_name`); the optimized `filter_roms` gallery query is
untouched.
https://claude.ai/code/session_01PSXKmejPRzdxLFMN6P2QQ4
New Nintendo 3DS games never matched ScreenScraper because the platform
was missing from SCREENSAVER_PLATFORM_LIST. With no entry, get_platform()
returns ss_id=None and scan_handler skips the entire ScreenScraper lookup
(hash and filename) for the platform, reporting everything as unmatched.
ScreenScraper has no separate New 3DS system; New 3DS games live under the
regular Nintendo 3DS system (ID 17). Alias New Nintendo 3DS to that system,
matching the existing Famicom->NES, Super Famicom->SNES, and DSi->DS aliases.
Fixes#3464
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
IGDB catalogues a console and its regional twin as two separate
platforms (SNES/Super Famicom, NES/Famicom). RomM locked each IGDB
search to a single platform id, so a region-exclusive title catalogued
under only the twin — e.g. the Japan-only Super Famicom game
"Rudra no Hihou" (platform 58) scanned from an `snes` folder
(platform 19) — was filtered out before name matching ran and never
matched.
Include a platform's regional twin in the IGDB platform filter so both
are searched. A non-twin platform keeps the exact existing query
(`platforms=[19]`); a twin produces an OR group
(`(platforms=[19] | platforms=[58])`), leaving all other platforms and
recorded cassettes unchanged.
Written primarily by Claude Code.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
IGDB scans dropped games whose filename uses a localized (non-English)
title even when that title exists in IGDB's alternative_names. The
alternative_name wildcard search surfaced the correct game, but
_search_rom() rebuilt its name->game candidate dict using only the
primary English name, so the Jaro-Winkler re-check scored the localized
term below threshold and discarded the match (issue #3435).
Add _index_games_by_searchable_name(), which indexes each game by its
primary name plus alternative_names and game_localizations titles, and
use it for both candidate-building passes in _search_rom(). Primary
names keep precedence (lowest-igdb-id tiebreak); alternative/
localization titles fill in only names not already claimed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The archive branch of get_rom_files (introduced in #3412) was missing
the RAHasherService.calculate_hash call that exists in the non-archive
branch, causing all archive-format ROMs to produce an empty ra_hash
during scanning regardless of platform.
The RA hash call is now made for archive ROMs, mirroring the existing
non-archive behaviour. The RA_BUFFER_HASH_UNSUPPORTED skip logic in
RAHasherService already handles disc-based platforms (PSX, PS2, PSP,
Saturn, Dreamcast, etc.) so those continue to be excluded automatically.
Also improves handling of folder-based multi-file ROMs whose directories
contain compressed files. RAHasher cannot process archives via the /*
glob and fails with "Could not open file". The fix mirrors the existing
CHD folder logic: for cartridge platforms the largest archive in the
folder is passed directly to RAHasher for buffer hashing; for disc
platforms the call is skipped as buffer hashing is unsupported.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- genesis_plus_gx_wide added alongside genesis_plus_gx for all Sega
MD/MS/GG/CD platforms
- bsnes added alongside snes9x for SNES platforms
- freeintv added for Intellivision
- azahar added for Nintendo 3DS (3ds, new-nintendo-3ds) and marked as
requiring threads
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
RQ 2.x Job.set_id rejects ':' in a job_id, so the bootstrap enqueue raised
ValueError that the broad except swallowed -- the content_hash recompute task
never ran, leaving legacy saves with stale/NULL hashes. Drop the colon, remove
the unsupported unique=True (would TypeError the worker once enqueued), and
replace the dead DuplicateJobError branch with a real Job.exists guard.
The server datetime-tags every slot upload's filename (archival spec), so a
slot accrues many rows and the stored file_name never equals the client's
untagged canonical name. Keying negotiate's server-save map on file_name meant
every client save missed -> perpetual "upload", and every tagged server row
went unmatched -> perpetual "download", with save rows growing unbounded.
Pair on (rom_id, slot), collapsing each slot to its newest row, so
compare_save_state actually runs and content hashes decide the action.
Tests: real upload->negotiate round-trip (lets _apply_datetime_tag run, client
reports the untagged name) and a 3-device convergence test; both fail against
the old file_name keying.
When sending a hash lookup to ScreenScraper, romnom was always set to the
archive filename on disk (e.g. Mario.zip). For single-file archives, the hash
is computed from the internal file (e.g. mario.n64), so sending the archive
name sends slightly incorrect info to ss.fr during a KO scrape.
When archive_members has exactly one entry, romnom now uses that member's
name. Multi-file archives and non-archive files continue to use the filesystem
filename unchanged.
Closes#3444
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
FSAssetsHandler.compute_content_hash and _compute_zip_hash were
building full paths via f"{self.base_path}/{file_path}". self.base_path
is already a pathlib.Path (resolved by FSHandler.__init__), so the
f-string forced it to str, hard-coded the separator, and re-parsed --
fine on Linux but a footgun if a caller ever sneaks a leading slash or
the path needs Path semantics elsewhere.
Switch both spots to self.base_path / file_path, which is what every
other FSHandler subclass in this module already does (e.g.
FSRomsHandler, FSResourcesHandler, FSSyncHandler all join Path objects
directly).
Three sync callsites (endpoints/sync.py, sync_watcher.py, and both
branches of tasks/sync_push_pull_task.py) ran get_saves(...) and then
discarded archival null-slot rows in a Python list comprehension. On
libraries with many archival/web-UI uploads that's a strict waste:
those rows are pulled from MariaDB, hydrated into Save model instances,
and then immediately filtered out.
Add a slot_not_null bool kwarg to DBSavesHandler.get_saves and apply
the filter in the SQL query. Update all four callsites to use it and
drop the Python-side comprehension. Default stays False so unrelated
callers keep the current behavior.
