How I Write MkDocs Plugins

As I mentioned in How I Setup This Blog, MkDocs is a core part of this blog's publishing workflow and I had to get my hands dirty to get everything working just right.

In this post, I'll guide you through my development workflow for writing and debugging MkDocs plugins and hooks.

Already know how to setup the development environment? Skip to the next section.

Development Environment Setup¶

For 95% of my development work, I use VSCode, but since I can't live without a Unix shell, I typically work within a Windows Subsystem for Linux (WSL) environment.

So, in a WSL prompt, I do a git clone of my peddamat/mkdocs-obsidian:

me in ~
[I] ❯ git clone https://github.com/peddamat/mkdocs-obsidian.git
Cloning into 'mk'...
remote: Enumerating objects: 817, done.
remote: Counting objects: 100% (284/284), done.
remote: Compressing objects: 100% (120/120), done.
remote: Total 817 (delta 127), reused 276 (delta 125), pack-reused 533
Receiving objects: 100% (817/817), 686.36 KiB | 2.29 MiB/s, done.
Resolving deltas: 100% (420/420), done.

Then I fire up a VSCode instance (DIRECTLY FROM THE WSL PROMPT!)

me in ~
[I] ❯ cd mkdocs-obsidian
[I] ❯ code .
me in mkdocs-obsidian on  main took 2s

VSCode starts up in this nifty "WSL" mode, making interaction with the WSL environment completely seamless:

Pull Obsidian Vault¶

Since I store my Obsidian vault in a separate private repository (I explain the reasoning in How I Setup This Blog), I now clone it into a subdirectory called "notes".

[I] ❯ git clone git@github.com:peddamat/notes.git
Cloning into 'notes'...
remote: Enumerating objects: 1521, done.
remote: Counting objects: 100% (276/276), done.
remote: Compressing objects: 100% (183/183), done.
remote: Total 1521 (delta 171), reused 194 (delta 91), pack-reused 1245
Receiving objects: 100% (1521/1521), 37.05 MiB | 15.31 MiB/s, done.
Resolving deltas: 100% (698/698), done.
me in mkdocs-obsidian on  main took 4s

Local Webserver Setup¶

Then, from a VSCode terminal, I fire up a local development instance of my blog using:

[I] ❯ pipenv run mkdocs serve 
...
INFO     -  Documentation built in 3.84 seconds
INFO     -  [21:37:00] Serving on http://127.0.0.1:8000/

Making my blog accessible at http://127.0.0.1:8000/ while I work on it.

At this point I'm ready to start developing.

Get Plugin Code¶

In this example, we'll be working on the mkdocs-roamlinks-plugin , so let's begin by doing a git clone of the code into the mkdocs-obsidian directory:

[I] ❯ git clone git@github.com:peddamat/mkdocs-roamlinks-plugin.git
Cloning into 'mkdocs-roamlinks-plugin'...
remote: Enumerating objects: 101, done.
remote: Counting objects: 100% (57/57), done.
remote: Compressing objects: 100% (31/31), done.
remote: Total 101 (delta 33), reused 43 (delta 26), pack-reused 44
Receiving objects: 100% (101/101), 19.70 KiB | 3.94 MiB/s, done.
Resolving deltas: 100% (47/47), done.
me in mkdocs-obsidian on  main [?] took 2s

Info

If you'll notice, I'm pulling the code from my own personal fork of mkdocs-roamlinks-plugin. It's good practice to always fork a repo before you git clone it (so you can avoid having to Google the syntax for removing and adding a remote in git).

Install Plugin¶

Now that you've got a local copy of the plugin code:

We need to tell pipenv to install this local copy of the plugin code, rather than what it previously installed into the packages folder.

This is done by using the -e flag (Note: the -e flag also works with plain pip install):

[I] ❯ pipenv install -e mkdocs-roamlinks-plugin
Installing -e mkdocs-roamlinks-plugin…
Looking in indexes: https://pypi.python.org/simple
Obtaining file:///home/me/mkdocs-obsidian/mkdocs-roamlinks-plugin
...
Installing collected packages: mkdocs-roamlinks-plugin
  Attempting uninstall: mkdocs-roamlinks-plugin
    Found existing installation: mkdocs-roamlinks-plugin 0.2.0
    Uninstalling mkdocs-roamlinks-plugin-0.2.0:
      Successfully uninstalled mkdocs-roamlinks-plugin-0.2.0
  Running setup.py develop for mkdocs-roamlinks-plugin
Successfully installed mkdocs-roamlinks-plugin

Adding -e mkdocs-roamlinks-plugin to Pipfile's [packages]…
Pipfile.lock (43fcfc) out of date, updating to (e1b7fa)…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (e1b7fa)!
Installing dependencies from Pipfile.lock (e1b7fa)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 58/58 — 00:00:27
me in mkdocs-obsidian on  main [!?] took 52s

Now you have everything you need to start hacking on the plugin.

One thing to note, however, mkdocs serve does not automatically refresh the site as you make changes to the plugin's code. You will need to kill and rerun mkdocs serve as you work.

Alternatively...¶

If you prefer discipline over speed, you can forgo making a local clone of your plugin, and simply work directly from your GitHub fork.

You do this by:

[I] ❯ pipenv install git+https://github.com/peddamat/mkdocs-roamlinks-plugin@master

Installing git+https://github.com/peddamat/mkdocs-roamlinks-plugin@master…
⠏WARNING: pipenv requires an #egg fragment for version controlled dependencies. Please install remote dependency in the form git+https://github.com/peddamat/mkdocs-roamlinks-plugin#egg=<package-name>.

Oops, I forgot the part about the #egg=<package-name>, as will you.

You can find the <package-name> bit in the plugin's setup.py:

setup.py

from setuptools import setup, find_packages

setup(
    name='mkdocs-roamlinks-plugin',
...
    entry_points={
        'mkdocs.plugins': [
            'roamlinks = mkdocs_roamlinks_plugin.plugin:RoamLinksPlugin',
        ]
    }
)

It will be the bit at the end there.

Anyways, add that and:

[I] ❯ pipenv install git+https://github.com/peddamat/mkdocs-roamlinks-plugin@master#egg=RoamLinksPlugin
Installing git+https://github.com/peddamat/mkdocs-roamlinks-plugin@master#egg=RoamLinksPlugin…
⠙Warning: You installed a VCS dependency in non–editable mode. This will work fine, but sub-dependencies will not be resolved by $ pipenv lock.
  To enable this sub–dependency functionality, specify that this dependency is editable.
Looking in indexes: https://pypi.python.org/simple
Collecting RoamLinksPlugin
  Cloning https://github.com/peddamat/mkdocs-roamlinks-plugin (to revision master) to /tmp/pip-install-tmytxsvo/RoamLinksPlugin
...
Building wheels for collected packages: mkdocs-roamlinks-plugin, mkdocs-roamlinks-plugin
  Building wheel for mkdocs-roamlinks-plugin (setup.py): started
  Building wheel for mkdocs-roamlinks-plugin (setup.py): finished with status 'done'
  Created wheel for mkdocs-roamlinks-plugin: filename=mkdocs_roamlinks_plugin-0.2.0-py3-none-any.whl size=5000 sha256=54562b59cb08d33050dc798f20af767b179d3fbbe431f47100fb71cfd2fa661b
  Stored in directory: /tmp/pip-ephem-wheel-cache-3go_fqjk/wheels/db/eb/02/34dd609165555dbc911de65c19533bc9fc01f05b99bafef267
  Building wheel for mkdocs-roamlinks-plugin (setup.py): started
  Building wheel for mkdocs-roamlinks-plugin (setup.py): finished with status 'done'
  Created wheel for mkdocs-roamlinks-plugin: filename=mkdocs_roamlinks_plugin-0.2.0-py3-none-any.whl size=5000 sha256=54562b59cb08d33050dc798f20af767b179d3fbbe431f47100fb71cfd2fa661b
  Stored in directory: /tmp/pip-ephem-wheel-cache-3go_fqjk/wheels/09/48/26/db3e2583dc622b5b0abf779173bd857e464b75bc67a5f03a4f
Successfully built mkdocs-roamlinks-plugin mkdocs-roamlinks-plugin

Adding git+https://github.com/peddamat/mkdocs-roamlinks-plugin@master#egg=RoamLinksPlugin to Pipfile's [packages]…
Pipfile.lock (bdb602) out of date, updating to (31e69d)…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (31e69d)!
Installing dependencies from Pipfile.lock (31e69d)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 57/57 — 00:00:18
To activate this project's virtualenv, run the following:
 $ pipenv shell
me in mkdocs-obsidian on  main [!?] took 37s

Now, after committing and pushing your code to GitHub, you can do a pipenv update which will pull and install your latest commits:

$ pipenv update                           
Running $ pipenv lock then $ pipenv sync.
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (43fcfc)!
Installing dependencies from Pipfile.lock (43fcfc)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 57/57 — 00:00:22
To activate this project's virtualenv, run the following:
 $ pipenv shell
All dependencies are now up-to-date!

Debugging A Plugin¶

One plugin that does some heavy lifting is the mkdocs-roamlinks plugin, which resolves and converts internal links, including [[wikilink]]-style links popular among Obsidian users.

However, I found myself with a persnickety issue when linking files with parenthesis in their filenames, for example, Let's Write An Article (Part 1).md or Let's Write An Article (Part 2).md.

The resulting links were came out like Let's Write An Article (Part 1)), a subtle error, but an error nonetheless.

So I fixed the issue and submitted a pull request: here

In this article, I'll walk you through how I went about debugging that issue.

Unit Testing¶

One of the best ways to debug a plugin is to write some test cases.

If your plugin doesn't already have test cases, let me walk you through how to get started.

Add Pytest¶

In this example we'll be using Pytest, but feel free to explore other options.

First, add an extra_requires entry after install_requires in the plugin's setup.py:

install_requires=[
    'mkdocs>=1.0.4',
],
extras_require={
    'dev': [ 'pytest']
},

Create "tests" folder¶

Next, create a folder called "tests". Within this "tests" folder, create a file named __init__.py and another named test_plugin.py:

mkdir tests
touch tests/__init__.py
touch tests/test_plugin.py

It should look something like:

[I] ➜ tree
.
...
├── setup.py
└── tests
    ├── __init__.py
    └── test_plugin.py

Write Unit Tests¶

Write your fixtures and test cases in test_plugin.py. Here's an excerpt from mkdocs-roamlinks-plugin:

import os
import tempfile
import pytest

from mkdocs.structure.files import File
from mkdocs.structure.pages import Page
from mkdocs_roamlinks_plugin.plugin import RoamLinksPlugin

@pytest.fixture
def temp_directory():
    with tempfile.TemporaryDirectory() as temp_dir:
        yield temp_dir

@pytest.fixture
def config(temp_directory):
    return {"docs_dir": temp_directory}

@pytest.fixture
def site_navigation():
    return []

...

@pytest.fixture
def converter(temp_directory, config, site_navigation, page):
    def c(markdown):
        plugin = RoamLinksPlugin()
        return plugin.on_page_markdown(markdown, page, config, site_navigation)

    return c

## Test Cases

def test_converts_basic_link(converter):
    assert converter("[[Git Flow]]") == "[Git Flow](<../software/git_flow.md>)"

def test_converts_link_with_slash(converter):
    assert converter("[software/Git Flow](<../../software/Git Flow.md>)") == "[software/Git Flow](<../software/Git Flow.md>)"

def test_converts_link_with_anchor_only(converter):
    assert converter("[#Heading identifiers](<#heading-identifiers>)") == "[#Heading identifiers](<#heading-identifiers>)"    
...

The bulk of the excerpt above are test fixtures, which you can learn about here.

At the bottom of the excerpt you can see three test cases.

Execute Tests¶

Once you've fleshed out your test_plugin.py, execute the test runner by running the command pytest from within the tests folder, which looks something like:

$ pytest
================== test session starts ==================
platform darwin -- Python 3.9.7, pytest-7.2.2, pluggy-1.0.0
rootdir: /Users/me/Source/mkdocs-roamlinks-plugin
plugins: anyio-3.6.1
collected 19 items

test_plugin.py ...................                [100%]

================== 19 passed in 0.37s ===================

Writing An Extension¶

MkDocs provides two different ways to extend its core functionality: hooks and plugins, with hooks being a lightweight version of plugins.

Unless you want to redistribute your extension, my advice is to simply write a hook. Or, start your plugin as a hook until it's ready to be repackaged.

Getting Started¶

If you're a "first principles" sort of person (guilty), the best place to start learning the codebase is from mkdocs/mkdocs/commands.

This is where the MkDocs command-line commands are defined, i.e. mkdocs build & mkdocs serve. The code for the former being defined in build.py and for the latter in serve.py.

mkdocs/structure/files.py#L303 is where File objects are instantiated.
mkdocs/commands/build.py#L297 is where the on_files event gets called.
mkdocs/commands/build.py#L239 is where the page is written using the abs_dest_path property of the file.
mkdocs/structure/pages.py#L326 - Documentation file xxx which is not found in the documentation files.

Once you get a feel for how things are put together, the next best place to visit is the plugin reference, specifically, the Events section, which provides this great diagram illustrating the basic execution flow:

Writing Hooks¶

We're going to examine two hooks from my mkdocs-obsidian repo.

To get started, add a hooks section to the bottom of your mkdocs.yml:

hooks:
  - hooks/flatten_filenames.py
  - hooks/only_include_published.py

Next, create a "hooks" directory:

$ tree
.
├── hooks
│   ├── flatten_filenames.py
│   └── only_include_published.py
...

`hooks/flatten_filenames.py`¶

import logging, re
import mkdocs.plugins

log = logging.getLogger('mkdocs')

# Rewrite using `python-slugify`
def on_files(files, config):
    for f in files:
        if f.is_documentation_page() or f.is_media_file():
            f.abs_dest_path = f.abs_dest_path.replace(" ", "-").lower()
            f.abs_dest_path = f.abs_dest_path.replace("(", "").lower()
            f.abs_dest_path = f.abs_dest_path.replace(")", "").lower()
            f.dest_path = f.dest_path.replace(" - ", "-").lower()
            f.dest_path = f.dest_path.replace(" ", "-").lower()
            f.dest_path = f.dest_path.replace("(", "").lower()
            f.dest_path = f.dest_path.replace(")", "").lower()
            f.url = f.dest_path.replace("%20", "-").lower()

    return files

`hooks/only_include_published.py`¶

import os, logging, typing, frontmatter
from typing import Optional

import mkdocs.plugins
from mkdocs.structure.pages import Page
from mkdocs.structure.files import Files
from mkdocs.config.defaults import MkDocsConfig

log = logging.getLogger('mkdocs')

def is_page_published(meta: typing.Dict) -> bool:
    if 'publish' in meta:
        return meta['publish'] == True

def on_files(files: Files, *, config: MkDocsConfig) -> Optional[Files]:
    base_docs_url = config["docs_dir"]

    for file in files.documentation_pages():
        ...
    return files

def on_post_page(output: str, *, page: Page, config: MkDocsConfig) -> Optional[str]:
    if not is_page_published(page.meta):
        return ''

    return output

Writing Plugins¶

tbd

Last update: 2023-04-01