Skip to content

Stub representation for PyO3 PyModule::add_submodule (non-importable “submodules”) #351

@termoshtt

Description

@termoshtt

Summary

When using PyO3’s PyModule::add_submodule, we create a module-like attribute such as main_mod.sub_mod. As noted in its document

Note that this doesn’t define a package, so this won’t allow Python code to directly import submodules by using from my_module import submodule. For more information, see #759 and #1517.

at runtime this object:

  • is attached as an attribute on main_mod,
  • is a PyModule instance,
  • but is not importable via import main_mod.sub_mod because main_mod is not a package.

This structure appears regardless of whether the project uses a “pure Rust” layout or a mixed Python/Rust layout: as long as the relationship is created via add_submodule, main_mod.sub_mod is an attribute, not an importable submodule.

The question is: how should such an object be represented in .pyi stubs without lying about the import semantics?


Runtime structure (PyO3)

Minimal PyO3 example:

use pyo3::prelude::*;
use pyo3::types::PyModule;

#[pymodule]
fn main_mod(py: Python<'_>, m: &PyModule) -> PyResult<()> {
    let sub = PyModule::new(py, "sub_mod")?;
    // add functions / classes to `sub` here...
    m.add_submodule(sub)?;
    Ok(())
}

From Python’s point of view:

import main_mod

main_mod.sub_mod             # OK: attribute, a module object
from main_mod import sub_mod # OK: attribute import

import main_mod.sub_mod
# ImportError: 'main_mod' is not a package

So main_mod.sub_mod is not a proper submodule name for the import system; it is just an attribute that happens to be a module object.

This is true for:

  • a single extension module (main_mod.so, “pure Rust layout”), and
  • mixed Python/Rust packages,

as long as the relationship is created via PyModule::add_submodule.


Why usual stub layouts don’t work

The usual stub layouts for a submodule:

  • main_mod/sub_mod.pyi
  • main_mod/sub_mod/__init__.pyi

both tell type checkers that an importable module named main_mod.sub_mod exists.

If we provide stubs in either of these forms, then:

import main_mod.sub_mod  # type-checks fine according to the stub

but at runtime this always fails with:

ImportError: 'main_mod' is not a package

For an object created via PyModule::add_submodule,
modeling it as a separate module in stubs is inherently incorrect, because it claims an import that Python can never perform.

So the goal is to find a representation that:

  • gives good type information and IDE completion, and
  • does not pretend that import main_mod.sub_mod is supported.

Proposed representation: typed attribute (class or Protocol)

Instead of declaring a separate module stub, we can represent sub_mod as a typed attribute of main_mod in its .pyi:

# main_mod.pyi (or main_mod/__init__.pyi)

class _SubMod:
    # Public API of the `sub_mod` object
    def foo(self, x: int) -> int: ...
    def bar(self, s: str) -> str: ...
    # etc.

sub_mod: _SubMod

Alternatively, if we prefer structural typing:

from typing import Protocol

class _SubMod(Protocol):
    def foo(self, x: int) -> int: ...
    def bar(self, s: str) -> str: ...

sub_mod: _SubMod

This has the following properties:

  • It is honest about the runtime: sub_mod is an attribute of main_mod, not a separately importable module.

  • Type checkers and IDEs still provide full support for the actual usage pattern:

    from main_mod import sub_mod
    
    sub_mod.foo(1)
    sub_mod.bar("x")
  • It works the same way for both pure Rust and mixed Python/Rust layouts, because it only depends on how add_submodule behaves, not on packaging.


Proposed documentation / tooling guidance

Given that PyModule::add_submodule always creates non-importable “submodule-like” attributes, I’d like to propose:

  • Add documentation (in PyO3 and/or maturin) recommending that objects created via add_submodule be represented in stubs as typed attributes, not as separate submodules.

    • Concretely, something like:

      # main_mod.pyi
      
      class _SubMod: ...
      sub_mod: _SubMod
  • If maturin (or other tooling) ever generates stubs automatically for PyO3 modules, then:

    • For children registered via add_submodule, it should emit a typed attribute in the parent module’s .pyi rather than generating a separate stub module.

Open questions

  • Would it be acceptable to add a short “Stub files for add_submodule” section to the PyO3 docs showing this pattern?
  • Are there known edge cases where type checkers behave poorly with this “typed attribute representing a module-like object” pattern?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions