Files
romm/backend/utils/filesystem.py
Michael Manganiello f20a9ffe34 fix: Avoid recursive os.walk calls
`os.walk` is a generator that can iteratively navigate from the
specified path, top-bottom. However, most of the calls to `os.walk` in
the project cast the call to `list()`, which makes it traverse the path
and recursively find all nested directories.

This is commonly not needed, as we end up just using a `[0]` index to
only access the root path.

This change adds a few utils that simplifies listing files/directories,
and by default does it non-recursively. Performance gains shouldn't be
noticeable in systems with high-speed storage, but we can avoid the edge
cases of users having too many nested directories, by avoiding unneeded
I/O.
2024-07-13 15:30:04 -03:00

30 lines
978 B
Python

import os
from collections.abc import Iterator
from pathlib import Path
def iter_files(path: str, recursive: bool = False) -> Iterator[tuple[Path, str]]:
"""List files in a directory.
Yields tuples where the first element is the path to the directory where the file is located,
and the second element is the name of the file.
"""
for root, _, files in os.walk(path, topdown=True):
for file in files:
yield Path(root), file
if not recursive:
break
def iter_directories(path: str, recursive: bool = False) -> Iterator[tuple[Path, str]]:
"""List directories in a directory.
Yields tuples where the first element is the path to the directory where the directory is located,
and the second element is the name of the directory.
"""
for root, dirs, _ in os.walk(path, topdown=True):
for directory in dirs:
yield Path(root), directory
if not recursive:
break