DAGrunner documentation
Sphinx syntax (reST) and subsequently building documentation with it can be fraught with frustration and burden. It is for this reason that it was decided to experiment with an alternative approach, which is to take advantage of native github markdown rendering support.
A script has been written to scrape and generate markdown reference documentation from our python files. This python script is available at docs/gen_docs. This is a WIP and will likely improve over time. It may also become pulled out into its own repository if deemed useful elsewhere. For DAGrunner, writing to the current working directory (docs), this would be done as follows:
gen_docs dagrunner .
This will generate files for your package with the following naming <package>.<subpackage>.<module>.md
Additionally an index file is generated <package>_index.md
For DAGrunner, see here
Whilst the above demonstrates manually building the documentation. This is unlikely to be necessary.
When making a pull request to DAGrunner, each push will trigger a github action to build the documentation and push the built documentation to your branch. This way, the built documentation itself can be reviewed critically as part of the changes included in your pull request.
From the README.md, we can access the reference documentation by following the link:
Here is a preview of the index that the above link takes us to, giving us an overview of package:
Taking a look at one of the modules (dagrunner.execute_graph):
Syntax of the dagrunner.execute_graph.plugin_executor docstring (see here):
Note that any github flavoured markdown syntax is supported and module/function/class/method docstrings are parsed and included in the resulting reference documentation files.
Documentation is handled much like any other Python library’s docs. While many projects embed reStructuredText that Sphinx converts into .rst files, we instead embed Markdown directly in the source and generate .md files without using Sphinx. This leverages native Markdown rendering in places like GitHub and most IDEs.
A custom script located at docs/gen_docs creates the Markdown reference docs from the Python files. To build the docs, run:
gen_docs dagrunner . The command produces a Markdown file for each module using the pattern ...md.
It also creates an index file named _index.md, which provides an overview and links to all modules, classes, and functions.
For the DAGrunner project, see the generated docs here.
In practice, you rarely need to run this manually. When you open a pull request for DAGrunner, a GitHub Action automatically runs the script, builds the documentation, and pushes the resulting Markdown files to your branch. This lets reviewers examine the updated docs alongside the code changes.