ADR: Kobol vs Pegasus

The decision at hand is whether to run all development tools directly on my local workstation (kobol) or inside a Vagrantbox (pegasus). This ARD only affects "normal" development workstations. The Virtualization solution of choice is Virtualbox with Vagrant. My dedicated VM server is running Proxmox.

Status = Done (2022-05-16)

Update (2022-07-13) → Decision withdrawn and reversed in favor of kobol (see "Decision")

Requirements Overview

Setup with minimal manual interactions and minimal development tools installed directly onto my workstation.
Encapsulated as much as possible inside Docker Containers and/or Vagrantboxes to ensure (broken) updates can easily be rolled back and the setup is portable in case I have to reset my workstation or have to switch to another workstation.
1. Add or update packages without any risk → avoid having to repair broken installations (dumping the broken setup and reverting to a working one is preferred).
2. Instead of using apt-get install use Docker and create a Symlink in /usr/bin to a wrapper-script which delegates commands to a Docker contaner.
Use the same setup for the workstation I use at work (as far as possible → Windows vs Linux → at least ssh keys won’t be not the same!)

Quality Goals

The whole setup is Infrastructure as Code.
Run as much as possible in Containers.

Decision

The setup of choice uses a Vagrantbox for all development tools. This The main reasons are the simple setup and the clearly defined baseline which allows to easily revert to a working setup in case the setup gets messy.

Decision Update

Update (2022-07-13) → Although the desicion to favor a Vagrantbox makes sence for several reasons, working with this VM Turned out to be rather uncomfortable. When Vagrantboxes are involved, the Vagrantbox is not usable because it cannot handle VMs. For this usecase one has to switch to the host machine. By doing this not all features and tools are available or are not installed the same way / at the same location. Because of this discrepancy the decision is cancelled and turned in favor of working directly on the host machine. This emphasizes the need to run as much as possible in Docker containers and enforces the disciplin to follow infrastructure as code principles.

Decision Details

These details are part of the original decision which now is invalid.

General takeaways are:

Every new tool installation should get its own git branch. This way rolling back becomes as easy as possible.
Run as much as possible in containers! This way separating tools from themselves and repairing broken configs becomes much easier, even when the development tools are not encapsulated in a Vagrantbox.

Run everything directly on physical machine "kobol" (in Docker) Encapsulate everything inside Vagrantbox "pegasus" Info / Comment

Run everything directly on physical machine "kobol" (in Docker)	Encapsulate everything inside Vagrantbox "pegasus"	Info / Comment
Terraform can manage Docker Containers and Vagrantboxes.	Terraform can only manage Docker Containers. Terraform cannot connect to Vagrant on host (and use this to spin up VMs).	No real usecase so far → When do I need to spin up multiple Vagrantboxes on my host machine?
Docker image leftovers remain on host machine.	Docker image leftovers are gone when VM is removed.
Broken configs must be repaired, simple rollback more complicated (when not encapsuled in Docker).	No need to repair broken configs. Just revert to an older version. Especially when new features are developed in Git branches.
No need to Sync SSH Keys into VM	SSH Keys must be synced into VM	This is not really a pain either
No (additional) issues with network / SSH when connection to other nodes on the network.	Some connections to other nodes on the network don’t work as expected (especially from Docker Container inside Vagrantbox).	Rundeck in a Docker Container from inside the Vagrantbox has trouble connecting to my prometheus-RasPi
Setup is not easily portable. Scripts can be used on other machines but one might have to update the scripts according to the new machine.	Dependencies must be met (`~/.ssh`, `~/.tmp`, `~/.work` directories, `.gitconfig file`) → so the setup is not portable without adjustments. Escpecially not to Windows machines.
No difference between IntelliJ terminal and "regular" terminal.	Some GUI tools are still running on kobol directly (IntelliJ, Postman). IntelliJ terminal differs from terminal inside Vagrantbox (not the same toolset as inside the VM although scripts depend on it.).
No need to SSH into a Vagrantbox for terminal commands	Need to SSH into a Vagrantbox for terminal commands	This is not really a pain either
Resources and services not strictly isolated from one another. Heavy crashes can take down the host system as well. In a worst case scenario the host machine must be cleaned up and repaired manually.	Stricter isolation of resources and services for development tools. Harder to crash whole system. In case of a bad crash, VM can be reset from scratch easily.
Running updates of tools, which are not wrapped in a Docker Container, comes with a higher risk and higher expense in case of rollback.	Running any update comes without any risk because reverting to a working setup can be done easily. This becomes even easier when every update has a dedicated git branch.
Staging ground (dedicated VM?) for new tools needed to avoid leftovers on the host machine.	No staging ground to test new tools needed. The VM can be trashed and setup from scratch without these tools leafing no trace of any PoC.
No port forwarding needed	Dedicated port-forwarding needed. In case more ports than the currently forwarded ports are needed, a VM restart is necessary.	This is not really a pain either
	Duplicate Git Installation. Git is installed on the host anyway (e.g. to allow IntelliJ to use Git)
Some tools still have to run on the host machine (node_exporter, portainer, cAdvisor) to provide monitoring information to Prometheus	Same monitoring tools should run inside the VM as well. But collecting these information cannot be done easily because the VM is not reachable by its name from everywhere on my local network. Name resolution for Vagrantboxes only works from my workstation (= the VM host)
Some utility tooly have to run on the host (export Firefox bookmarks regularly for FTP upload)	Not possible from VM because the Firefox of choice is installed on the host and not reachable from the VM.
Fast startup, ready when Laptop is up-and-running	Wait for full VM startup every time. Takes even more time when VM is deleted and recreated because provisioning needs some time.	Docker handling is the same for both setups. Docker downloads missing images and does not care if the image is downloaded to the host or the guest VM.
Keeping environments up to date keeps a lot more effort. Everything must be commited to a git repo. So running everything in Docker is a must. Still there might be differences because the OS is not necessarily the same.	Multiple (Linux) machines can be set up exactly the same way. Every machine is kept up to date by pulling from the remote git repository.
New SSH keys can be used right away	When adding new SSH keys (on the host since the host is the master) the VM must be destroyed and re-provisioned from scratch
Security breaches like Log4Shell might take over the whole system.	Security breaches like Log4Shell don’t take over the whole system. Just the VM. Cleanup can be done easily by dumping the VM.

Terraform can manage Docker Containers and Vagrantboxes.

Terraform can only manage Docker Containers. Terraform cannot connect to Vagrant on host (and use this to spin up VMs).

No real usecase so far → When do I need to spin up multiple Vagrantboxes on my host machine?

Docker image leftovers remain on host machine.

Docker image leftovers are gone when VM is removed.

Broken configs must be repaired, simple rollback more complicated (when not encapsuled in Docker).

No need to repair broken configs. Just revert to an older version. Especially when new features are developed in Git branches.

No need to Sync SSH Keys into VM

SSH Keys must be synced into VM

This is not really a pain either

No (additional) issues with network / SSH when connection to other nodes on the network.

Some connections to other nodes on the network don’t work as expected (especially from Docker Container inside Vagrantbox).

Rundeck in a Docker Container from inside the Vagrantbox has trouble connecting to my prometheus-RasPi

Setup is not easily portable. Scripts can be used on other machines but one might have to update the scripts according to the new machine.

Dependencies must be met (~/.ssh, ~/.tmp, ~/.work directories, .gitconfig file) → so the setup is not portable without adjustments. Escpecially not to Windows machines.

No difference between IntelliJ terminal and "regular" terminal.

Some GUI tools are still running on kobol directly (IntelliJ, Postman). IntelliJ terminal differs from terminal inside Vagrantbox (not the same toolset as inside the VM although scripts depend on it.).

No need to SSH into a Vagrantbox for terminal commands

Need to SSH into a Vagrantbox for terminal commands

This is not really a pain either

Resources and services not strictly isolated from one another. Heavy crashes can take down the host system as well. In a worst case scenario the host machine must be cleaned up and repaired manually.

Stricter isolation of resources and services for development tools. Harder to crash whole system. In case of a bad crash, VM can be reset from scratch easily.

Running updates of tools, which are not wrapped in a Docker Container, comes with a higher risk and higher expense in case of rollback.

Running any update comes without any risk because reverting to a working setup can be done easily. This becomes even easier when every update has a dedicated git branch.

Staging ground (dedicated VM?) for new tools needed to avoid leftovers on the host machine.

No staging ground to test new tools needed. The VM can be trashed and setup from scratch without these tools leafing no trace of any PoC.

No port forwarding needed

Dedicated port-forwarding needed. In case more ports than the currently forwarded ports are needed, a VM restart is necessary.

This is not really a pain either

Duplicate Git Installation. Git is installed on the host anyway (e.g. to allow IntelliJ to use Git)

Some tools still have to run on the host machine (node_exporter, portainer, cAdvisor) to provide monitoring information to Prometheus

Same monitoring tools should run inside the VM as well. But collecting these information cannot be done easily because the VM is not reachable by its name from everywhere on my local network. Name resolution for Vagrantboxes only works from my workstation (= the VM host)

Some utility tooly have to run on the host (export Firefox bookmarks regularly for FTP upload)

Not possible from VM because the Firefox of choice is installed on the host and not reachable from the VM.

Fast startup, ready when Laptop is up-and-running

Wait for full VM startup every time. Takes even more time when VM is deleted and recreated because provisioning needs some time.

Docker handling is the same for both setups. Docker downloads missing images and does not care if the image is downloaded to the host or the guest VM.

Keeping environments up to date keeps a lot more effort. Everything must be commited to a git repo. So running everything in Docker is a must. Still there might be differences because the OS is not necessarily the same.

Multiple (Linux) machines can be set up exactly the same way. Every machine is kept up to date by pulling from the remote git repository.

New SSH keys can be used right away

When adding new SSH keys (on the host since the host is the master) the VM must be destroyed and re-provisioned from scratch

Security breaches like Log4Shell might take over the whole system.

Security breaches like Log4Shell don’t take over the whole system. Just the VM. Cleanup can be done easily by dumping the VM.