Mermaid and Pydantic

Introduction

In my previous article, I was writing about Mermaid and that I wanted to experiment with generating documentation from the actual sources.

I was recently able to focus properly on this and I think I had a blast breakthrough at least for myself.

The state now

Last time I wrote that I was still investigating how and what, especially because of the compression that takes place when you do a graph TopDown(TD). There is a very simple solution to that, make it Left to Right!

Now all I needed to add to the flavor is parse the bits and pieces of our configuration data. As mentioned I use Ansible for that, I generate the configuration I need and use a jinja template to actually parse the data and print whatever I need wherever I need it. I cannot share details ofcourse because that is work related, but I am sure you can use your imagination on your own defined configuration and what items you need to graph something.

Oh, and I tossed the subgraphs entirely. I did have a look at the architecture drawing that one can make, but that seems still a bit too difficult to get a proper model out of that.

Model

I mentioned the word model in the previous section on purpose, our configuration is repeatable, as any modern configuration should be when you use it a gazillion times. That also means that you can wrap it in a model.

Pydantic

This is where pydantic comes in, a coworker of mine did a demo of this recently and I watched it afterwards (it was given on my day off, but we love to share internally so I could still view it later on). He is from a different group but they too have a configuration that is repeatable and perfectly fits a model.

What is Pydantic

I gave a talk about Pydantic recently myself and I used the phrase: It is a strict and quick validator of a given model, over a defined configuration.

Perhaps that does not give the right merit to the tool, since FastAPI uses it to validate in and output on the fly with the tool, but for me this works perfectly.

How does it work

Basically you stricly denote your configuration, and if you for example use yaml this has a certain layout. I will try to give an example a bit lower in the article. This layout and configuration items (yaml entries, like lists, dicts, a combination of them, etc.) if used well, are always matching a certain criteria.

Like with ansible, something can be ‘state: present’, or ‘state: absent’. If you would wrap that in a Pydantic scheme, it will become: ‘state: Literal[“present”,“absent”]’. That means, that if the validation traverses your configuration, and finds a state keyword, it should match either present or absent. All other values are wrong and your validator should fail. You can also have the flag: ’enabled: true’, or false. That reads in pydantic like: ’enabled: bool’, since it is either true or false. if it is an integer (all digits), then you can state: ‘version: int’ for example. You can use regular expressions as well, so if you know how a keyword’s value should look like, you can push it through a regular expression and validate that what you think must be defined is actually defined.

but, not everything is Required right ? That is true, so using the version as example, if that is an optional parameter in your configuration, you can define that like ‘version: int | None = None’, and it will be either if it exists an integer, or ignored (/optional) it is is not defined.

Sections

So, not all configuration has just one layer, moest configuration has lists, dicts, a combination of them, can you validate that as well? Yes you can. You can point a certain part of your configuration, to an ‘upstream’ validation. So instead of telling that ‘version: int’ is what should happen, imagine it is a list. you can duplicate your codeblock and name it ‘VersionCheck’ for example. Then you do this in the lower config item: ‘version: VersionCheck’. You have that new structure that is named VersionCheck ABOVE (bottom up thus) the normal validator, and define how the version contents should be. Perhaps it shows like this:

version:
  name: This is our version
  major: 1
  minor: 0
  patch: p0

That does not work if you tell ‘version: int’ right? So imagine you created that new VersionCheck, you can then point version to that validation object, and do this:

name: str
major: int
minor: int
patch: str (or regex that ^p\d is what you expect).

The version tag is actually missing, because you ‘decent’ into the version hierarchy when you reference it. That way you can loop over lists, dicts, etc pretty easily.

How did you implement this?

I cheated a bit and read his model and adopted it to our configuration and we optimized it a bit to use it in our CI/CD stream. I use Ansible (yes again) to construct the configuration that I modified my colleague’s wrapper for and use that to parse the data.

I cannot share details on how we did that at work, but if you are really curious I am considering writing a post on it, so that you can have an idea.