Introduction
In a previous article, I wrote something about Azure Devops and how I see it. I also wanted to play with automated documentation generation, which basically means that I am considering a template, and get bits and pieces from a configuration that I also use to deploy stuff, into a markdown file and present that on a ‘host’.
One of the things that ‘annoyed’ me is that documentation always lags behind. Why? Because you need to modify it to remain relevant. See it is an instruction manual for your car, as long as the car itself remains the same, the manual remains relevant. But if you lets say, change the navigation unit in it, you either need to write a new part for that in the documentation (and add it as addendum or something), or reprint the instructions, else it will not fit anymore.
This requires manual labor. And a part of my job is to automate our deployments. Manual, automate, that does not compute right? It does in some context, but in this particular context it does not.
I wanted to experiment with a system that would auto rebuild the documentation based on the actual sources. Since we run an Ansible environment, we have a codebase and configuration in place. Why not use them together and extract the bits and pieces we need?
What I found initially
Is that my predecessor, thanks Liam, already took care of a couple of those things. He wrote playbooks that extract a couple of configuration parameters and transform that into a temporary Markdown file and release that as artifact which gets used before publishing the wiki for example.
That gave me a nice boost, practically visible on what is possible, I decided to start the experiment there. But that is mainly text? What about graphics? Mermaid to the rescue!
Mermaid
I saw, I think a year ago, a mentioning of mermaidjs thing from a colleague and also saw it in Joplin, and then at various other places. Back then I did not take particular note of it, as I was busy making automated deployments for our infrastructure.
Now that that dust settled, and I was going to experiment with automated documentation I searched back in my mind, and recalled Mermaid. What does it do?
What is mermaid?
Mermaid is basically a diagramming and/or charting tool as you wish. With a simple set of instructions, you can create a diagram from text based input. Awesome right? To give a little example, in markdown you could specify this:
```mermaid
flowchart TD
Topitem->Secondlayer_1
Topitem->Secondlayer_2
Secondlayer_1 --> Bottomlayer_1_1
Secondlayer_1 --> Bottomlayer_1_2
Secondlayer_2 --> Bottomlayer_2_1
Secondlayer_2 --> Bottomlayer_2_2
```
flowchart TD Topitem --> Secondlayer_1 Topitem --> Secondlayer_2 Secondlayer_1 --> Bottomlayer_1_1 Secondlayer_1 --> Bottomlayer_1_2 Secondlayer_2 --> Bottomlayer_2_1 Secondlayer_2 --> Bottomlayer_2_2
This is ofcourse all very basic, the mermaid processor can work with ‘ids’ that you can give a name, and use a large set of different graphical layouts as well as use markup to make clear what the id means. See here for more information on how this could work for you.
And now what?
Well, did I mention that I use ‘draw.io’ when I want to make a drawing and/or diagram? Do you notice the overlap there ? Most often a drawing of how an enviroment looks like is nothing more then a Diagram with some markup. Machine A connects to Network A with IP A and B on both sides, it uses protocol XYZ between them to talk to eachother. They have a certain storage backend to storage host A, etc. That is just a flowchart presented differently. In my eyes then. Send me a message in case you disagree or see it differently.
So. We have this annoying thing that you need to update documentation when you change something, which most often is still required, but also some parts that you can extract from ‘your configuration’ and update automatically. Like a drawing of the environment! And you know as I do, that drawings are most often updated last, and/or just forgotten.
I started to experiment with a hardcoded diagram, that looks a bit like the above example but then more related to our environment.
My environment
In a VMWare environment you normally have a vCenter, one or more clusters, and each cluster has one or more ESXi hosts. Those hosts have some sort of storage layer between them, specific network and vlans assigned to them etc. I combined that data in a diagram. And it looked like a nice start, but not really there yet. In a small scale setup this would make a network diagram automatically, and means you can automate the network drawings away for these infrastructure components (not limited to though!). But we dont have a small scale setup. We have quite a large estate running. This made it quickly difficult to read.
From within DrawIO I would do something like the following to present this (strongly simplified, and condensed, you get the drill):
Subgraphs?
Because it became difficult to read if you add enough (valuable) data. I tried using sub graphs, so I created a vCenter on top, linked that to a box with hosts on it, linked it to a storage backend, linked to a network with a couple of vlans (distributed switch reference in case you are familiar).
Roughly this looks like:
flowchart TD; subgraph A_vCenter vCenter --> A_Cluster vCenter --> A_Storage vCenter --> A_Network end subgraph A_Cluster Cluster_A --> Host_A Cluster_A --> Host_B Cluster_A --> Host_C end subgraph A_Storage Storage_A --> Volume_A Storage_A --> Volume_B end subgraph A_Network Network_A --> Vlan_A_123 Network_A --> Vlan_B_456 end
```mermaid
flowchart TD;
subgraph A_vCenter
vCenter --> A_Cluster
vCenter --> A_Storage
vCenter --> A_Network
end
subgraph A_Cluster
Cluster_A --> Host_A
Cluster_A --> Host_B
Cluster_A --> Host_C
end
subgraph A_Storage
Storage_A --> Volume_A
Storage_A --> Volume_B
end
subgraph A_Network
Network_A --> Vlan_A_123
Network_A --> Vlan_B_456
end
```
This is still readable in the above example, but if you have lets say 500 vlans connected to a cluster, just
because you can, it comes difficult to read. If you have multiple clusters under a vCenter, it comes difficult
to read, especially if you want to connect a cluster to A_Storage and another one to B_Storage
and perhaps want to mention which interfaces they use as the line-text. To give an example, I tried drawing that
below:
flowchart TD; subgraph A_vCenter vCenter --> A_Cluster vCenter --> B_Cluster end subgraph A_Cluster Cluster_A --> Host_A_A(Host_A_A) Cluster_A --> Host_A_B(Host_A_B) Cluster_A --> Host_A_C(Host_A_C) Cluster_A --> A_Storage Cluster_A --> A_Network end subgraph B_Cluster Cluster_B --> Host_B_A Cluster_B --> Host_B_B Cluster_B --> Host_B_C Cluster_B --> B_Storage Cluster_B --> B_Network end subgraph A_Storage Storage_A --> Volume_A_A[(Volume_A_A)] Storage_A --> Volume_A_B[(Volume_A_B)] end subgraph B_Storage Storage_B -->|HBA_B_A_1| Volume_B_A[(Volume_B_A)] Storage_B -->|HBA_B_A_2| Volume_B_B[(Volume_B_B)] end subgraph A_Network Network_A --> Vlan_A_A_123 Network_A --> Vlan_A_B_456 end subgraph B_Network Network_B -->|Iface_B_A_1| Vlan_B_A_1 Network_B -->|Iface_B_A_2| Vlan_B_A_2 Network_B -->|Iface_B_A_3| Vlan_B_A_3 Network_B -->|Iface_B_A_4| Vlan_B_A_4 Network_B -->|Iface_B_A_5| Vlan_B_A_5 Network_B -->|Iface_B_A_456| Vlan_B_B_456 end
```mermaid
flowchart TD;
subgraph A_vCenter
vCenter --> A_Cluster
vCenter --> B_Cluster
end
subgraph A_Cluster
Cluster_A --> Host_A_A(Host_A_A)
Cluster_A --> Host_A_B(Host_A_B)
Cluster_A --> Host_A_C(Host_A_C)
Cluster_A --> A_Storage
Cluster_A --> A_Network
end
subgraph B_Cluster
Cluster_B --> Host_B_A
Cluster_B --> Host_B_B
Cluster_B --> Host_B_C
Cluster_B --> B_Storage
Cluster_B --> B_Network
end
subgraph A_Storage
Storage_A --> Volume_A_A[(Volume_A_A)]
Storage_A --> Volume_A_B[(Volume_A_B)]
end
subgraph B_Storage
Storage_B -->|HBA_B_A_1| Volume_B_A[(Volume_B_A)]
Storage_B -->|HBA_B_A_2| Volume_B_B[(Volume_B_B)]
end
subgraph A_Network
Network_A --> Vlan_A_A_123
Network_A --> Vlan_A_B_456
end
subgraph B_Network
Network_B -->|Iface_B_A_1| Vlan_B_A_1
Network_B -->|Iface_B_A_2| Vlan_B_A_2
Network_B -->|Iface_B_A_3| Vlan_B_A_3
Network_B -->|Iface_B_A_4| Vlan_B_A_4
Network_B -->|Iface_B_A_5| Vlan_B_A_5
Network_B -->|Iface_B_A_456| Vlan_B_B_456
end
```
And this is still very minimal, in a regular setup, you have many more details that you might want to add. For now I am playing with the idea to generate multiple pages of data and zoom in om a specific set of data per page render. This doesn’t automate away the generation of always up to date environment drawings though.
Manually
Also this is still all done manually, which is also not the idea. What I want to do is write an Ansible playbook, or re-use existing ones, and grab data from it and use the ID and Name of a parameter to form the documentation and create that before the artifacts are generated and used to generate the Wiki.
Briljant, now that works we have finally a automated network drawing?
No not really, the above examples are using the plain mermaid engine in for example
GoHugo. See Here how to do that.
But.. as more often Azure does a different thing. You can render a more limited
subset of the Mermaid application within Azure, using the :::mermaid code block.
This looks similar to the ``` blocks from Markdown, but isn’t entirely
the same. Also it appears that you cannot use all features, nor multiple in a row.
So the above examples will not be possible within Azure at this moment. Having
said that, I also heard that the feature was much more limited before so it got
traction anyway. I hope it will be build out to a much more full set of the
regular mermaid implementation and that you can also use it similar to the examples
on the web. It would make life much easier for a lot of people. Note that the above
examples are, as far as I know and could test, usable, just limited to one per page.
If you have ideas about this, and/or would like to discuss this, you know where to find me (See the contact page on top).