Folder-stored disc games (the standard Redump .cue + .bin layout, .gdi
sets, multi-bin, etc.) never matched RetroAchievements. roms_handler
builds an `ra_path` ending in `/*` when the folder has no .chd, and
RAHasherService only rescued that glob for folders containing archives.
A folder of plain uncompressed tracks fell through, so the literal `*`
reached RAHasher via create_subprocess_exec (no shell to expand it),
which failed with "Could not open track" and stored an empty ra_hash.
Resolve the `/*` glob to a single real file before spawning RAHasher:
prefer a disc descriptor (.cue/.gdi/.m3u), which RAHasher follows to the
referenced tracks, otherwise fall back to the largest file in the folder
(raw .iso/.bin, or the main file of a multi-file cartridge set). This
mirrors the existing "pick the largest .chd" handling for CHD folders.
Fixes#3497.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ScreenScraper enforces a per-account *thread* (concurrency) cap rather than
a request rate. Requests can take several seconds, so spacing out request
starts at 1/s could still leave multiple requests overlapping in flight,
exceeding the cap and getting rejected with ScreenScraper's custom HTTP codes.
- Add ConcurrencyLimiter: a runtime-resizable, loop-agnostic limiter that
bounds simultaneous in-flight operations (held for the whole request via
async context manager), instead of spacing request starts.
- Switch the ScreenScraper service from the req/s RateLimiter to a module-level
ConcurrencyLimiter defaulting to a single thread.
- Recognize contributor/donor accounts: parse `ssuser.maxthreads` from each
response and raise the concurrency allowance to match, so supporters scrape
with their full thread count instead of the conservative default.
Adds unit tests for the limiter (blocking, wake-on-release, runtime resize)
and for the ScreenScraper slot-holding and thread-allowance updates.
https://claude.ai/code/session_01133QQuWvq8Zm25DZMP9PVr
Wire the existing RateLimiter into the IGDB, ScreenScraper, MobyGames and
RetroAchievements services so requests are spaced under each provider's
documented req/s cap, instead of only reacting to HTTP 429 after the fact.
- IGDB: 4 req/s (documented hard limit)
- MobyGames: 1 req/s (free-tier burst cap)
- ScreenScraper: 1 req/s (free-tier throttle)
- RetroAchievements: 4 req/s (conservative; no published hard limit)
A slot is acquired before both the initial request and the timeout/429
retry. The reactive 2s 429 backoff is kept as a fallback. Tests neutralize
the shared limiters via an autouse fixture and assert acquire() is awaited.
https://claude.ai/code/session_01133QQuWvq8Zm25DZMP9PVr
The archive branch of get_rom_files (introduced in #3412) was missing
the RAHasherService.calculate_hash call that exists in the non-archive
branch, causing all archive-format ROMs to produce an empty ra_hash
during scanning regardless of platform.
The RA hash call is now made for archive ROMs, mirroring the existing
non-archive behaviour. The RA_BUFFER_HASH_UNSUPPORTED skip logic in
RAHasherService already handles disc-based platforms (PSX, PS2, PSP,
Saturn, Dreamcast, etc.) so those continue to be excluded automatically.
Also improves handling of folder-based multi-file ROMs whose directories
contain compressed files. RAHasher cannot process archives via the /*
glob and fails with "Could not open file". The fix mirrors the existing
CHD folder logic: for cartridge platforms the largest archive in the
folder is passed directly to RAHasher for buffer hashing; for disc
platforms the call is skipped as buffer hashing is unsupported.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Fix several issues in ScreenScraper API request/response handling:
- Correctly handle SS-specific HTTP error codes (KO responses, 429, 431,
and the SS-quirk of returning 401 when server CPU >60%).
- Construct requests with proper parameter encoding so jeuInfos lookups
and search queries return the expected results.
- Store media URLs returned by SS as-is, preserving the dev credential
query parameters required for media playback. Removing them broke
downstream media fetches.
To keep dev credentials out of log output, add a redacting formatter in
the logger pipeline that scrubs ssid/sspassword/devid/devpassword query
parameters from any URL it sees.
Test coverage added for the new HTTP error paths and the as-is URL
storage behaviour.
Callers now pass the full platform dict and rom.fs_extension; the service
normalizes the extension (optional leading dot, case-insensitive) before
checking the compressed-archive skip set, so ROMs stored with bare
extensions like "zip" correctly hit the skip path.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
RAHasher was being spawned for every hashable ROM regardless of file
type. When the source file is a zip/7z/tar and the RA platform needs
an on-disk disc image (PSX, PS2, PSP, Saturn, Dreamcast, Sega CD,
3DO, PC-FX, Neo Geo CD, TurboGrafx CD, Atari Jaguar CD, Wii), the
subprocess fails with "Unsupported console for buffer hash: {id}"
after paying full process-spawn overhead per ROM — a serious slowdown
when indexing large zipped collections (e.g. myrient PS2/PSP sets).
calculate_hash now short-circuits those combinations with a debug log
and no subprocess. Raw disc images (.iso, .chd, .cue/.bin) and
archives on cartridge platforms still go through RAHasher as before.
Also centralize COMPRESSED_FILE_EXTENSIONS in utils/filesystem.py so
roms_handler (is_compressed_file / hashing), rahasher (skip logic),
and feeds (PKGi passthrough) share one source of truth. The shared
set adds .rar, which is_compressed_file now recognizes too.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds the libretro thumbnail repository as a first-class artwork source so
region-correct box art (PAL/Europe, Japan, etc.) can be matched directly
to ROM filenames, addressing rommapp/romm#3239.
Implementation follows the SGDB handler pattern (artwork-only, no game
metadata): MetadataSource enum entry, scan-time fetch wired into the
SCAN_ARTWORK_PRIORITY loop, /search/roms integration, MatchRom dialog
chip + cover selection, and a heartbeat flag.
Matching is exact case-insensitive against the directory listing first
(so a ROM named "(Europe)" lands on the (Europe) artwork), with a
JaroWinkler fuzzy fallback at 0.8 that strips parenthetical tags from
both sides. Listings are cached in Redis with a 24h TTL.
`libretro_id` is persisted on the Rom model as the SHA1 hex of the
matched libretro filename — stable across scans, distinct per region,
indexed for lookup. Migration 0077 adds the column.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add a new service adapter for the IGDB API, to separate concerns with
RomM's handler for metadata. This adapter is agnostic to the handler and
only provides methods to interact with the API, and correctly return
typed responses.
The API authorization was also improved to not rely on decorating each
method that makes requests, but instead using an `aiohttp` middleware
to automatically add the required headers to each request.
Utils `mark_expanded` and `mark_list_expanded` where added to help
narrow the types of IGDB's expandable fields when we know they are
expanded, for `mypy` type checking.
Add a new service adapter for the MobyGames API, to separate concerns
with RomM's handler for metadata.
This adapter is agnostic to the handler and only provides methods to
interact with the API, and correctly return typed responses.
