In Python, circular imports occur when two or more modules depend on each other directly or indirectly. I’ve encountered this problem frequently when my project was getting larger and more complex.
The Pitfall of Incomplete Python Modularization
In many projects, there’s typically a utils module containing various utility functions used across different parts of the codebase. My project followed this pattern with the following folder structure:
my_project
├── __init__.py
├── cli.py
├── ...
├── utils
│ ├── __init__.py
│ ├── eval.py
│ ├── transforms.py
│ └── ...
...In utils/__init__.py, I imported all utility functions from individual files so I could use them simply with from my_project.utils import ... throughout the project:
my_project/utils/__init__.py
from .eval import *
from .transforms import *
...While this approach brings convenience, it creates a coupling problem. Imagine a class A uses some utility functions, but other utilities need to import A to work properly. Even if A isn’t directly involved with those utility functions, the implicit dependencies propagating through utils/__init__.py can cause circular imports.
Solution: Keep Things Decoupled!
The solution is to modularize as much as possible. By keeping modules independent, we reduce the risk of circular imports, and of course, improve the overall codebase design as well.
In my case above, we can avoid grouping unrelated utility functions together in the same module by __init__.py. Remove the from xxx import * statements and import only specific functions you need in each module. When you need functions in submodule transforms, use from my_project.utils.transforms import specific_function instead of from my_project.utils import specific_function.
Avoid putting things that are conceptually similar but mechanically different in the same module. For example, I initially placed a callback HATNetworkCapacity and a metric HATNetworkCapacityMetric in the same Python file, which led to circular imports. Although both belong to the “metrics” concept, they are completely different things in the code. When I put them in separate modules: HATNetworkCapacity in metrics/ and HATNetworkCapacityMetric in utils/metrics.py, the circular import issue was resolved.