Commit Graph

115 Commits

Author SHA1 Message Date
Spinnich
01f0b1d2b5 feat(hashing): compute raw CHD hashes and route disc-data SHA1 to Hasheous
CHD files now follow the same hash logic as all other file types — CRC32,
MD5, and SHA1 are computed from raw container bytes. This allows
ScreenScraper to log KO entries for unrecognised CHD files, which it
could not do when only the disc-data SHA1 was being computed.

The CHD header SHA1 (disc-data SHA1) is separately extracted and stored
in a new chd_sha1_hash field on RomFile, with a migration adding the
column to rom_files. Hasheous receives only this disc-data SHA1 (no
CRC/MD5) since it indexes disc-based games by disc-data SHA1, not raw
file hashes.

The RAHasher multi-file path now passes the largest CHD directly instead
of a /* wildcard, which RAHasher cannot expand. Hash computations are
wrapped in asyncio.to_thread to avoid blocking the event loop during
large reads.

Hash-lookup metadata handlers (ScreenScraper, Hasheous, Playmatch) now
fall back to rom.files (stored DB hashes) when fs_rom files are not
rehashed, fixing hash-based matching for UNMATCHED and UPDATE scan types.

The Disc SHA-1 is displayed in the ROM detail view for both single-file
(FileInfo.vue) and multi-file (FileSelectItem.vue) CHD games.
2026-05-17 08:01:05 -04:00
Georges-Antoine Assi
5e3a2707b0 cleanup 2026-05-03 19:39:19 -04:00
copilot-swe-agent[bot]
9593c30292 Address PR review: normalize exclusion sets, avoid duplicates, add multi-dot test for get_rom_files
Agent-Logs-Url: https://github.com/rommapp/romm/sessions/8cbbc2ca-a3e3-4c61-9e47-f8544d59231a

Co-authored-by: gantoine <3247106+gantoine@users.noreply.github.com>
2026-05-03 23:34:30 +00:00
copilot-swe-agent[bot]
101629628e Simplify extension exclusion to use ends-with check instead of sub-extension iteration
Agent-Logs-Url: https://github.com/rommapp/romm/sessions/a81b2023-a243-4721-bc5e-c6fa1a473a79

Co-authored-by: gantoine <3247106+gantoine@users.noreply.github.com>
2026-05-03 22:46:21 +00:00
copilot-swe-agent[bot]
55cd0cfc4f Support compound suffix exclusions like "hash.txt" for multi-dot filenames
Agent-Logs-Url: https://github.com/rommapp/romm/sessions/d1c69638-bfa0-480e-8050-d565b234ea44

Co-authored-by: gantoine <3247106+gantoine@users.noreply.github.com>
2026-05-03 01:29:04 +00:00
copilot-swe-agent[bot]
21de7e21f8 Fix file exclusion for multi-dot filenames (e.g. game.nds.hash.txt)
Agent-Logs-Url: https://github.com/rommapp/romm/sessions/2f711770-100b-4e9e-a66e-ab1a74f025f8

Co-authored-by: gantoine <3247106+gantoine@users.noreply.github.com>
2026-05-02 16:50:30 +00:00
Georges-Antoine Assi
962a9bfa7e one more 2026-04-30 14:39:52 -04:00
Georges-Antoine Assi
fc8d69dc0c update tests 2026-04-30 14:18:49 -04:00
Georges-Antoine Assi
96c3634b80 refactor: split HIGH_PRIO_STRUCTURE_PATH into STRUCTURE_PATH_A/B
Replace the single HIGH_PRIO_STRUCTURE_PATH config attribute with two
glob patterns (STRUCTURE_PATH_A = roms/*, STRUCTURE_PATH_B = */roms) and
update all call sites to detect Structure B via glob.glob, defaulting to
Structure A when no match is found.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 12:52:22 -04:00
Georges-Antoine Assi
e8a6e9f01d final fixes 2026-04-12 18:43:24 -04:00
Georges-Antoine Assi
d45afb5dde more fixes 2026-04-12 18:32:15 -04:00
Georges-Antoine Assi
628d8d8bae refactor: pass RAGamesPlatform dict into calculate_hash, normalize extension
Callers now pass the full platform dict and rom.fs_extension; the service
normalizes the extension (optional leading dot, case-insensitive) before
checking the compressed-archive skip set, so ROMs stored with bare
extensions like "zip" correctly hit the skip path.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 18:05:44 -04:00
Georges-Antoine Assi
8f1b8f41d7 perf: skip RAHasher subprocess for archived disc-platform ROMs
RAHasher was being spawned for every hashable ROM regardless of file
type. When the source file is a zip/7z/tar and the RA platform needs
an on-disk disc image (PSX, PS2, PSP, Saturn, Dreamcast, Sega CD,
3DO, PC-FX, Neo Geo CD, TurboGrafx CD, Atari Jaguar CD, Wii), the
subprocess fails with "Unsupported console for buffer hash: {id}"
after paying full process-spawn overhead per ROM — a serious slowdown
when indexing large zipped collections (e.g. myrient PS2/PSP sets).

calculate_hash now short-circuits those combinations with a debug log
and no subprocess. Raw disc images (.iso, .chd, .cue/.bin) and
archives on cartridge platforms still go through RAHasher as before.

Also centralize COMPRESSED_FILE_EXTENSIONS in utils/filesystem.py so
roms_handler (is_compressed_file / hashing), rahasher (skip logic),
and feeds (PKGi passthrough) share one source of truth. The shared
set adds .rar, which is_compressed_file now recognizes too.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 17:18:14 -04:00
Georges-Antoine Assi
f9f3dfd927 changes from bot review 2026-04-12 09:50:54 -04:00
Georges-Antoine Assi
ec8583016b mega ton of fixes for 4.8 2026-04-03 10:54:31 -04:00
Georges-Antoine Assi
744d92d6d1 lower fs name 2026-03-11 21:05:01 -04:00
copilot-swe-agent[bot]
24fe5b941f refactor: move get_pico8_cover_url to FSRomsHandler, use validate_path for safe path construction
Co-authored-by: gantoine <3247106+gantoine@users.noreply.github.com>
2026-03-11 22:17:22 +00:00
Georges-Antoine Assi
ee8b55e6ef last set of changes 2026-03-07 09:56:17 -05:00
Georges-Antoine Assi
8a56e9b333 [ROMM-3026] Region/language shortcodes should be case sensitive 2026-02-18 10:19:12 -05:00
Georges-Antoine Assi
f867968f37 refactor get_rom_files return value 2025-12-30 11:42:38 -05:00
Georges-Antoine Assi
0971026f95 Add support for version tag 2025-12-30 11:37:06 -05:00
Zurdi
0d9a2e9380 Update backend/handler/filesystem/roms_handler.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-11-20 15:21:02 +01:00
zurdi
c82caa81b8 Add calculate hashes option to scanning process and update translations 2025-11-20 10:49:22 +00:00
sftwninja
7581c0a8e8 fix: Address Gemini PR comments 2025-11-17 01:55:55 -06:00
sftwninja
90a5a66a12 Use internal SHA1 hash if CHD file is v5 2025-11-16 23:41:32 -06:00
Georges-Antoine Assi
f8b0ae63a1 fix scanning multi file games with ssfr 2025-10-31 10:50:51 -04:00
Georges-Antoine Assi
24a5acce5d [ROMM-2552] Rom hashes should only include top-level nested files 2025-10-18 18:05:57 -04:00
Georges-Antoine Assi
b5776be475 Split rom.multi into more specific fields 2025-09-25 18:48:27 -04:00
Georges-Antoine Assi
3c4113f8a8 Merge branch 'master' into flashpoint-metadata-handler 2025-09-11 21:27:48 -04:00
Michael Manganiello
e4e3928d1b misc: Apply import sorting 2025-09-04 11:17:00 -03:00
Georges-Antoine Assi
ef2546ec08 fix base handler filename 2025-08-27 12:40:16 -04:00
Georges-Antoine Assi
82f527b3ad Remove check for non extension 2025-08-20 20:48:30 -04:00
Georges-Antoine Assi
8fb4769776 changes from code review 2025-08-13 14:03:45 -04:00
Georges-Antoine Assi
81a04e4ab4 Always hash the largest file in compressed multi file roms 2025-08-13 13:43:07 -04:00
Georges-Antoine Assi
b1609096de fix file_type typo 2025-08-08 15:00:18 -04:00
Georges-Antoine Assi
c47be7240b changs from bot review 2025-08-07 18:15:49 -04:00
Georges-Antoine Assi
9e37d87610 fix complex game names 2025-08-07 18:02:04 -04:00
Georges-Antoine Assi
926fe26fd9 [exp] use cli 7z to cacl hash files 2025-08-07 13:12:32 -04:00
Georges-Antoine Assi
8061db40b2 fix tests and cleanup 2025-08-02 17:55:43 -04:00
Georges-Antoine Assi
9fd1adbbfb fixup icon names 2025-07-24 23:20:07 -04:00
Georges-Antoine Assi
c2c80fbfc3 fix tests 2025-07-24 17:33:14 -04:00
Georges-Antoine Assi
59a2e5d5aa move away from explicit LIST use 2025-07-24 16:41:17 -04:00
Georges-Antoine Assi
595c52bcab Unify singular slugs 2025-07-23 21:52:52 -04:00
Georges-Antoine Assi
d2792ba0ad Use validate_path for getting rom_files 2025-07-20 19:58:39 -04:00
Georges-Antoine Assi
51652d5084 make all path stuff async 2025-07-18 10:53:55 -04:00
Georges-Antoine Assi
ff7fdbd9a4 fix moving files and folders 2025-07-17 21:31:27 -04:00
Georges-Antoine Assi
c93687a0f4 handle some errors better 2025-07-17 18:15:44 -04:00
Georges-Antoine Assi
877be2fe35 add test for roms handler 2025-07-17 17:38:56 -04:00
Georges-Antoine Assi
ea50b2c401 fix the firmware endpoint 2025-07-17 14:59:20 -04:00
Georges-Antoine Assi
aaf6741e93 Create safe filesystem handler 2025-07-17 12:30:57 -04:00