The scanning process could try to fetch thousands of roms from the
database, one by one, which is very inefficient, especially when only a
few new roms are added to the library, as the overhead of database calls
is very high compared to metadata API calls.
This change introduces a new method in the roms handler to retrieve roms
by their file names in bulk, which is used during the scan process to
fetch a batch of roms at once, instead of one by one.
It also avoids the `@with_details` decorator when fetching roms during
the scanning process, as those details are not needed, and they were
adding unnecessary joins and data decoding.
IGDB provides alternative names for games, which we are currently not
considering when checking for an exact match.
This change starts considering alternative names, in addition to the
game's name and slug, when checking for an exact match.
This change includes more categories when matching IGDB games. While
testing, some games were incorrectly matched to the wrong game, and the
reason was that the game was a Port (e.g. `Arkanoid`, `Contra`,
`Double Dragon`, `Metal Gear` for NES), a Remake (e.g.
`Adventure Island` for NES), or a Remaster.
At the moment, 7zip files are generating memory issues and even OOM
errors on user installations. This is because the current stable release
of `py7zr` does not support decompression streaming, and RomM needs to
decompress the each 7zip file in the library into memory to be able to
calculate hashes.
This change introduces a `py7zr` fork I created to have a stable commit
SHA to refer to in case upstream gets any forced pushes. It includes the
contents of the pull request the `py7zr` creator is working on to
support decompression streaming [1].
The way decompression streaming is implemented in `py7zr` is different
than the other compression utilities. Instead of being able to provide a
`bytes` iterator, we need to provide a `Py7zIO` implementation that
will call a callback on each read and write operation.
[1] https://github.com/miurahr/py7zr/pull/620
The short-term goal is to completely typehint the IGDB API responses. This
first change adds the base structures and enums RomM currently uses.
The `ExpandableField` type will allow us to model the expansion
mechanism the IGDB API provides, where a field can include either an ID,
or the full nested structure.
The `tar` decompression function was failing for some users, with error
message:
```
'NoneType' object does not support the context manager protocol
```
As explained in the official documentation [1], the `extractfile` method
returns `None` if the member is not a regular file or a link. This
change skips any member that is not a regular file.
[1] https://docs.python.org/3/library/tarfile.html#tarfile.TarFile.extractfile
Instead of using just strings, this change converts the scopes to a
`StrEnum`, to be compatible with places where a string is expected. This
avoids typos when using these scopes, simplifies searching for usages,
and improves type hints.
An extra change was the fix to the Firmware download endpoint, which
wasn't respecting the `DISABLE_DOWNLOAD_ENDPOINT_AUTH` flag.
The previous implementation was calling `resize_cover_to_small` within
the context manager that was writing the image to the filesystem. This
was causing `PIL` to raise an error because it could not identify the
open and temporarily created file as a valid image.
Instead of saving the original image to the filesystem and then resizing
it, we now open the image in memory, resize it, and then save it to the
filesystem. We also avoid reading the `BytesIO` object twice by saving
small and big images from the same initial `Image` object.
Fixes#1191.
This change replaces the creation of Redis URL, from a simple string
interpolation, to using `yarl.URL`. The main benefit, besides not
forgetting to set all five different variables on every Redis client
initialization, is that user credentials are correctly URL-encoded, if
present.
Up until now, if a password had special characters, it could break the
generated URL.
This change also introduces support for a `REDIS_SSL` setting, which
allows the user to specify if the Redis connection should use SSL or not.
This change improves memory usage, by only keeping a single archive's
member file in memory at a time during 7zip decompression.
The `py7zr` library does not support streaming decompression yet, so
this change is the best we can do for now.
Potential fix for #1211, but it won't improve memory usage for
single-file 7zip archives.
