-
Notifications
You must be signed in to change notification settings - Fork 53
Description
Summary
When using PyO3’s PyModule::add_submodule, we create a module-like attribute such as main_mod.sub_mod. As noted in its document
Note that this doesn’t define a package, so this won’t allow Python code to directly import submodules by using from my_module import submodule. For more information, see #759 and #1517.
at runtime this object:
- is attached as an attribute on
main_mod, - is a
PyModuleinstance, - but is not importable via
import main_mod.sub_modbecausemain_modis not a package.
This structure appears regardless of whether the project uses a “pure Rust” layout or a mixed Python/Rust layout: as long as the relationship is created via add_submodule, main_mod.sub_mod is an attribute, not an importable submodule.
The question is: how should such an object be represented in .pyi stubs without lying about the import semantics?
Runtime structure (PyO3)
Minimal PyO3 example:
use pyo3::prelude::*;
use pyo3::types::PyModule;
#[pymodule]
fn main_mod(py: Python<'_>, m: &PyModule) -> PyResult<()> {
let sub = PyModule::new(py, "sub_mod")?;
// add functions / classes to `sub` here...
m.add_submodule(sub)?;
Ok(())
}From Python’s point of view:
import main_mod
main_mod.sub_mod # OK: attribute, a module object
from main_mod import sub_mod # OK: attribute import
import main_mod.sub_mod
# ImportError: 'main_mod' is not a packageSo main_mod.sub_mod is not a proper submodule name for the import system; it is just an attribute that happens to be a module object.
This is true for:
- a single extension module (
main_mod.so, “pure Rust layout”), and - mixed Python/Rust packages,
as long as the relationship is created via PyModule::add_submodule.
Why usual stub layouts don’t work
The usual stub layouts for a submodule:
main_mod/sub_mod.pyimain_mod/sub_mod/__init__.pyi
both tell type checkers that an importable module named main_mod.sub_mod exists.
If we provide stubs in either of these forms, then:
import main_mod.sub_mod # type-checks fine according to the stubbut at runtime this always fails with:
ImportError: 'main_mod' is not a packageFor an object created via PyModule::add_submodule,
modeling it as a separate module in stubs is inherently incorrect, because it claims an import that Python can never perform.
So the goal is to find a representation that:
- gives good type information and IDE completion, and
- does not pretend that
import main_mod.sub_modis supported.
Proposed representation: typed attribute (class or Protocol)
Instead of declaring a separate module stub, we can represent sub_mod as a typed attribute of main_mod in its .pyi:
# main_mod.pyi (or main_mod/__init__.pyi)
class _SubMod:
# Public API of the `sub_mod` object
def foo(self, x: int) -> int: ...
def bar(self, s: str) -> str: ...
# etc.
sub_mod: _SubModAlternatively, if we prefer structural typing:
from typing import Protocol
class _SubMod(Protocol):
def foo(self, x: int) -> int: ...
def bar(self, s: str) -> str: ...
sub_mod: _SubModThis has the following properties:
-
It is honest about the runtime:
sub_modis an attribute ofmain_mod, not a separately importable module. -
Type checkers and IDEs still provide full support for the actual usage pattern:
from main_mod import sub_mod sub_mod.foo(1) sub_mod.bar("x")
-
It works the same way for both pure Rust and mixed Python/Rust layouts, because it only depends on how
add_submodulebehaves, not on packaging.
Proposed documentation / tooling guidance
Given that PyModule::add_submodule always creates non-importable “submodule-like” attributes, I’d like to propose:
-
Add documentation (in PyO3 and/or maturin) recommending that objects created via
add_submodulebe represented in stubs as typed attributes, not as separate submodules.-
Concretely, something like:
# main_mod.pyi class _SubMod: ... sub_mod: _SubMod
-
-
If maturin (or other tooling) ever generates stubs automatically for PyO3 modules, then:
- For children registered via
add_submodule, it should emit a typed attribute in the parent module’s.pyirather than generating a separate stub module.
- For children registered via
Open questions
- Would it be acceptable to add a short “Stub files for
add_submodule” section to the PyO3 docs showing this pattern? - Are there known edge cases where type checkers behave poorly with this “typed attribute representing a module-like object” pattern?