“To make a great film you need three things – the script, the script and the script.”

Alfred Hitchcock

There is a nice trick I use when I’m writing Kubernetes Jobs of embedding scripts in configmaps. Jobs are a Kubernetes resource type which spin up a one-shot container and run it to exit, with an optional bounded runtime and the usual Kubernetes resource limits. Jobs are often used during deployments, upgrades, for running regular maintenance, and other purposes. Jobs may be complex applications like a Database migration tool, but many are simple shell scripts.

We use Terraform/Terragrunt a lot for managing our environments via Infrastructure-as-Code (IaC) and we use a weekly Kubernetes job to run all our terraform plans and produce a “drift report” of the differences between our Terraform plans and the Real World.

The terraform drift script is a shellscript with some embedded python, and we run it using the excellent devops container image devopsinfra/docker-terragrunt. Rather than build a custom container image from this upsteam image, we mount the drift script into the container using a configmap. Configmaps are Kubernetes resources which may contain arbitrary data and can be mounted into a container as if they were a file. The script definition looks like:

apiVersion: v1
kind: ConfigMap
metadata:
  name: terragrunt-drift-script
  namespace: automation-st2
data:
  drift_detection.sh: |
     <<< PUT YOUR SCRIPT HERE >>>

In this case our configmap is mounted into the container as follows:

 apiVersion: batch/v1
 kind: Job
<<< metadata: block is provided by Stackstorm when it runs the job >>>
 spec:
   template:
     spec:
       serviceAccountName: terragrunt
       ttlSecondsAfterFinished: 43200
       containers:
       - name: terragrunt
	 image: devopsinfra/docker-terragrunt:gcp-tf-1.3.2-tg-0.39.1
	 command: ["bash","/opt/terragrunt/drift_detection.sh"]
	 env:
	   # Which project to run the plans for
	   - name: PROJECT
	     value: {project}
	   # Which GCS bucket to file the reports into
	   - name: REPORTS
	     value: {reports}
	 volumeMounts:
	   - name: terragrunt-drift-script
	     mountPath: /opt/terragrunt/drift_detection.sh
	     subPath: drift_detection.sh

       volumes:
	 - name: terragrunt-drift-script
	   configMap:
	     # Provide the name of the ConfigMap containing the files you want
	     # to add to the container
	     name: terragrunt-drift-script

       restartPolicy: Never
   backoffLimit: 1
   

Then the command line to run this script is passed in as the container startup command.

The advantages of doing this are:

• it is very much faster to develop the script since you don’t have to rebuild the container all the time
• we can use the upstream container for multiple scripts without having to build and maintain a specific container for each one
• the script is much more visible as it is not embedded inside an opaque container image
• It is easier to upgrade the upstream container (in addition, whilst in production the version should be locked down, other environments may use latest tag)

Our script configmaps are managed by an ArgoCD job so they get updated just by checking the changes into GIT, and are run weekly or daily by the workflow engine Stackstorm.