Skip to content

Managing a MicroK8s cluster

The Kubernetes documentation includes a great conceptual explanation of Kubernetes Objects:

Kubernetes objects are persistent entities in the Kubernetes system. Kubernetes uses these entities to represent the state of your cluster. Specifically, they can describe:

  • What containerized applications are running (and on which nodes)
  • The resources available to those applications
  • The policies around how those applications behave, such as restart policies, upgrades, and fault-tolerance

A Kubernetes object is a "record of intent"--once you create the object, the Kubernetes system will constantly work to ensure that object exists. By creating an object, you're effectively telling the Kubernetes system what you want your cluster's workload to look like; this is your cluster's desired state.

We'll use a few different methods for working with Kubernetes Objects, including:

MicroK8s addons

MicroK8s addons are extra services that can be enabled in MicroK8s, creating Objects, effectively functioning as a Kubernetes Operator. We will use addons to enable:

  • dashboard: The standard Kubernetes dashboard
  • dns: CoreDNS
  • ingress: A simple ingress controller for external access
  • metallb: The MetalLB LoadBalancer
  • hostpath-storage: A default storage class

We can do that in one line (here providing metallb with an IP address range to use for LoadBalancers) like this:

microk8s enable dashboard dns ingress metallb:192.168.1.192/27 hostpath-storage

kubectl

kubectl is the CLI for communicating with the Kubernetes control plane, and is able to perform a large set of operations including create, get, describe, and delete.

We will use kubectl primarily for four operations:

  1. To patch addons
  2. To create secrets
  3. To create tokens
  4. To run kustomize (see Kustomize below)

An example of something to patch is the ingress controller, to enable SSL passthrough for services that need to handle TLS themselves.

k patch daemonset -n ingress nginx-ingress-microk8s-controller \
    --type=json \
    -p='[{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value": "--enable-ssl-passthrough"}]'

Secrets can be created of different kinds, including tls and Opaque (generic) secrets. For example, to create an Opaque secret containing TLS certificate and key files from a ./certs/ directory, you can run:

k create secret generic -n kube-system kubernetes-dashboard-certs --from-file=./certs

Tokens can be created to provide user credentials. For example, to generate a token for admin-user you can run:

k create token admin-user

NOTE: In the above commands, microk8s kubectl has been aliased to k.

Kustomize

Kustomize is a standalone tool for customizing Kubernetes objects using kustomization files. Through those files, Kustomize allows you to:

  • generate resources from other sources
  • set cross-cutting fields for resources
  • compose and customize collections of resources

The Kustomize GitHub page features a great explanation of usage. We will use kustomize to modify and create resources on the cluster, including:

  • Creating a LoadBalancer Service for the cluster
  • Creating Ingress resources for the dashboard addon
  • Deploying Tekton Pipelines

A simple example of a Kustomization with a production overlay is the LoadBalancer Service. We define that Service with the following resource definition, k8s/nginx-ingress/base/ingress-service.yaml:

We declare that resource in our kustomization file, k8s/nginx-ingress/base/kustomization.yaml:

# https://argo-cd.readthedocs.io/en/stable/operator-manual/installation/#kustomize
# https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-manifests/
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: ingress
resources:
# Deploy the ingress controller
# https://raw.githubusercontent.com/nginxinc/kubernetes-ingress/v3.0.0/deployments/daemon-set/nginx-ingress.yaml
  - nginx-ingress.yaml
# Get access to the ingress controller
  - https://raw.githubusercontent.com/nginxinc/kubernetes-ingress/v3.0.0/deployments/service/loadbalancer.yaml

Note both of these files live in a base/ directory. In a neighboring overlays/prod directory, we create our trivial production overlay, k8s/nginx-ingress/overlays/prod/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: ingress
resources:
  - ../../base

In this case, prod does not modify base at all, but it can and generally will, for example applying labels to indicate the configuration and to modify resources.

With the kustomizations defined, we can apply our prod overlay to the cluster as follows:

k apply -k k8s/nginx-ingress/overlays/prod

The implementations of all kustomizations can be viewed in the source code.

Juju

Juju is an open source framework that uses Charmed Operators, or Charms, to deploy cloud infrastructure and operations.

Our use case for using Juju is to install Kubeflow, a machine learning toolkit for Kubernetes. Specifically, we will use it to deploy Charmed Kubeflow's kubeflow-lite bundle.

For installation steps, please refer to Charmed Kubeflow guidance.