+ - 0:00:00
Notes for current slide
Notes for next slide

Cloud Native
Continuous Deployment
with GitLab, Helm, and
Linode Kubernetes Engine


shared/title.md

1/233

Intros

  • Hello! I'm JΓ©rΓ΄me Petazzoni

    (@jpetazzo on Twitter)

  • I worked at Docker from ~2011 to 2018

  • I'm now doing consulting, training, etc. on Docker & Kubernetes

    (check out container.training!)

  • I'll show you how to deploy a complete CI/CD pipeline on LKE!

    (Linode Kubernetes Engine 😎)

logistics.md

2/233

Accessing these slides now

  • We recommend that you open these slides in your browser:

    https://2021-03-lke.container.training/

  • Use arrows to move to next/previous slide

    (up, down, left, right, page up, page down)

  • Type a slide number + ENTER to go to that slide

  • The slide number is also visible in the URL bar

    (e.g. .../#123 for slide 123)

shared/about-slides.md

3/233

Accessing these slides later

shared/about-slides.md

4/233

These slides are open source

  • You are welcome to use, re-use, share these slides

  • These slides are written in markdown

  • The sources of these slides are available in a public GitHub repository:

    https://github.com/jpetazzo/container.training

  • Typos? Mistakes? Questions? Feel free to hover over the bottom of the slide ...

πŸ‘‡ Try it! The source file will be shown and you can view it on GitHub and fork and edit it.

shared/about-slides.md

5/233

Extra details

  • This slide has a little magnifying glass in the top left corner

  • This magnifying glass indicates slides that provide extra details

  • Feel free to skip them if:

    • you are in a hurry

    • you are new to this and want to avoid cognitive overload

    • you want only the most essential information

  • You can review these slides another time if you want, they'll be waiting for you ☺

shared/about-slides.md

6/233

Image separating from the next module

9/233

Get ready!

(automatically generated title slide)

10/233

Get ready!

  • We're going to set up a whole Continous Deployment pipeline

  • ... for Kubernetes apps

  • ... on a Kubernetes cluster

  • Ingredients: cert-manager, GitLab, Helm, Linode DNS, LKE, Traefik

lke/intro.md

11/233

Philosophy

  • "Do one thing, do it well"
12/233

Philosophy

  • "Do one thing, do it well"

  • ... But a CD pipeline is a complex system with interconnected parts!

  • GitLab is no exception to that rule

  • Let's have a look at its components!

lke/intro.md

13/233

GitLab components

  • GitLab dependencies listed in the GitLab official Helm chart

  • External dependencies:

    cert-manager, grafana, minio, nginx-ingress, postgresql, prometheus, redis, registry, shared-secrets

    (these dependencies correspond to external charts not created by GitLab)

  • Internal dependencies:

    geo-logcursor, gitaly, gitlab-exporter, gitlab-grafana, gitlab-pages, gitlab-shell, kas, mailroom, migrations, operator, praefect, sidekiq, task-runner, webservice

    (these dependencies correspond to subcharts embedded in the GitLab chart)

lke/intro.md

14/233

Philosophy

  • Use the GitLab chart to deploy everything that is specific to GitLab

  • Deploy cluster-wide components separately

    (cert-manager, ExternalDNS, Ingress Controller...)

lke/intro.md

15/233

What we're going to do

  • Spin up an LKE cluster

  • Run a simple test app

  • Install a few extras

    (the cluster-wide components mentioned earlier)

  • Set up GitLab

  • Push an app with a CD pipeline to GitLab

lke/intro.md

16/233

What you need to know

  • If you just want to follow along and watch...

    • container basics (what's an image, what's a container...)

    • Kubernetes basics (what are Deployments, Namespaces, Pods, Services)

  • If you want to run this on your own Kubernetes cluster...

    • intermediate Kubernetes concepts (annotations, Ingresses)

    • Helm basic concepts (how to install/upgrade releases; how to set "values")

    • basic Kubernetes troubleshooting commands (view logs, events)

  • There will be a lot of explanations and reminders along the way

lke/intro.md

17/233

What you need to have

If you want to run this on your own...

  • A Linode account

  • A domain name that you will point to Linode DNS

    (I got cloudnative.party for $5)

  • Local tools to control your Kubernetes cluster:

  • Patience, as many operations will require us to wait a few minutes!

lke/intro.md

18/233

Do I really need a Linode account?

  • Can I use a local cluster, e.g. with Minikube?

    It will be very difficult to get valid TLS certs with a local cluster.

    Also, GitLab needs quite a bit of resources.

  • Can I use another Kubernetes provider?

    You certainly can: Kubernetes is a standard platform!

    But you'll have to adjust a few things.

    (I'll try my best to tell you what as we go along.)

lke/intro.md

19/233

Why do I need a domain name?

  • Because accessing gitlab.cloudnative.party is easier than 102.34.55.67

  • Because we'll need TLS certificates

    (and it's very easy to obtain certs with Let's Encrypt when we have a domain)

  • We'll illustrate automatic DNS configuration with ExternalDNS, too!

    (Kubernetes will automatically create DNS entries in our domain)

lke/intro.md

20/233

Nice-to-haves

Here are a few tools that I like...

  • linode-cli to manage Linode resources from the command line

  • stern to comfortably view logs of Kubernetes pods

  • k9s to manage Kubernetes resources with that retro BBS look and feel 😎

  • kube-ps1 to keep track of which Kubernetes cluster and namespace we're working on

  • kubectx to easily switch between clusters, contexts, and namespaces

lke/intro.md

21/233

Warning βš οΈπŸ’Έ

lke/intro.md

22/233

Image separating from the next module

23/233

Our sample application

(automatically generated title slide)

24/233

Our sample application

  • I'm going to run our demo app locally, with Docker

    (you don't have to do that; do it if you like!)

  • Clone the repository:
    git clone https://github.com/jpetazzo/container.training

(You can also fork the repository on GitHub and clone your fork if you prefer that.)

shared/sampleapp.md

25/233

Downloading and running the application

Let's start this before we look around, as downloading will take a little time...

  • Go to the dockercoins directory, in the cloned repo:

    cd container.training/dockercoins
  • Use Compose to build and run all containers:

    docker-compose up

Compose tells Docker to build all container images (pulling the corresponding base images), then starts all containers, and displays aggregated logs.

shared/sampleapp.md

26/233

What's this application?

27/233

What's this application?

  • It is a DockerCoin miner! πŸ’°πŸ³πŸ“¦πŸš’
28/233

What's this application?

  • It is a DockerCoin miner! πŸ’°πŸ³πŸ“¦πŸš’

  • No, you can't buy coffee with DockerCoins

29/233

What's this application?

  • It is a DockerCoin miner! πŸ’°πŸ³πŸ“¦πŸš’

  • No, you can't buy coffee with DockerCoins

  • How DockerCoins works:

    • generate a few random bytes

    • hash these bytes

    • increment a counter (to keep track of speed)

    • repeat forever!

30/233

What's this application?

  • It is a DockerCoin miner! πŸ’°πŸ³πŸ“¦πŸš’

  • No, you can't buy coffee with DockerCoins

  • How DockerCoins works:

    • generate a few random bytes

    • hash these bytes

    • increment a counter (to keep track of speed)

    • repeat forever!

  • DockerCoins is not a cryptocurrency

    (the only common points are "randomness," "hashing," and "coins" in the name)

shared/sampleapp.md

31/233

DockerCoins in the microservices era

  • DockerCoins is made of 5 services:

    • rng = web service generating random bytes

    • hasher = web service computing hash of POSTed data

    • worker = background process calling rng and hasher

    • webui = web interface to watch progress

    • redis = data store (holds a counter updated by worker)

  • These 5 services are visible in the application's Compose file, docker-compose.yml

shared/sampleapp.md

32/233

How DockerCoins works

  • worker invokes web service rng to generate random bytes

  • worker invokes web service hasher to hash these bytes

  • worker does this in an infinite loop

  • every second, worker updates redis to indicate how many loops were done

  • webui queries redis, and computes and exposes "hashing speed" in our browser

(See diagram on next slide!)

shared/sampleapp.md

33/233

Service discovery in container-land

How does each service find out the address of the other ones?

35/233

Service discovery in container-land

How does each service find out the address of the other ones?

  • We do not hard-code IP addresses in the code

  • We do not hard-code FQDNs in the code, either

  • We just connect to a service name, and container-magic does the rest

    (And by container-magic, we mean "a crafty, dynamic, embedded DNS server")

shared/sampleapp.md

36/233

Example in worker/worker.py

redis = Redis("redis")
def get_random_bytes():
r = requests.get("http://rng/32")
return r.content
def hash_bytes(data):
r = requests.post("http://hasher/",
data=data,
headers={"Content-Type": "application/octet-stream"})

(Full source code available here)

shared/sampleapp.md

37/233
  • Containers can have network aliases (resolvable through DNS)

  • Compose file version 2+ makes each container reachable through its service name

  • Compose file version 1 required "links" sections to accomplish this

  • Network aliases are automatically namespaced

    • you can have multiple apps declaring and using a service named database

    • containers in the blue app will resolve database to the IP of the blue database

    • containers in the green app will resolve database to the IP of the green database

shared/sampleapp.md

38/233

Show me the code!

  • You can check the GitHub repository with all the materials of this workshop:
    https://github.com/jpetazzo/container.training

  • The application is in the dockercoins subdirectory

  • The Compose file (docker-compose.yml) lists all 5 services

  • redis is using an official image from the Docker Hub

  • hasher, rng, worker, webui are each built from a Dockerfile

  • Each service's Dockerfile and source code is in its own directory

    (hasher is in the hasher directory, rng is in the rng directory, etc.)

shared/sampleapp.md

39/233

Compose file format version

This is relevant only if you have used Compose before 2016...

  • Compose 1.6 introduced support for a new Compose file format (aka "v2")

  • Services are no longer at the top level, but under a services section

  • There has to be a version key at the top level, with value "2" (as a string, not an integer)

  • Containers are placed on a dedicated network, making links unnecessary

  • There are other minor differences, but upgrade is easy and straightforward

shared/sampleapp.md

40/233

Our application at work

  • On the left-hand side, the "rainbow strip" shows the container names

  • On the right-hand side, we see the output of our containers

  • We can see the worker service making requests to rng and hasher

  • For rng and hasher, we see HTTP access logs

shared/sampleapp.md

41/233

Connecting to the web UI

  • "Logs are exciting and fun!" (No-one, ever)

  • The webui container exposes a web dashboard; let's view it

  • With a web browser, connect to node1 on port 8000

  • Remember: the nodeX aliases are valid only on the nodes themselves

  • In your browser, you need to enter the IP address of your node

A drawing area should show up, and after a few seconds, a blue graph will appear.

shared/sampleapp.md

42/233

Why does the speed seem irregular?

  • It looks like the speed is approximately 4 hashes/second

  • Or more precisely: 4 hashes/second, with regular dips down to zero

  • Why?

43/233

Why does the speed seem irregular?

  • It looks like the speed is approximately 4 hashes/second

  • Or more precisely: 4 hashes/second, with regular dips down to zero

  • Why?

  • The app actually has a constant, steady speed: 3.33 hashes/second
    (which corresponds to 1 hash every 0.3 seconds, for reasons)

  • Yes, and?

shared/sampleapp.md

44/233

The reason why this graph is not awesome

  • The worker doesn't update the counter after every loop, but up to once per second

  • The speed is computed by the browser, checking the counter about once per second

  • Between two consecutive updates, the counter will increase either by 4, or by 0

  • The perceived speed will therefore be 4 - 4 - 4 - 0 - 4 - 4 - 0 etc.

  • What can we conclude from this?

45/233

The reason why this graph is not awesome

  • The worker doesn't update the counter after every loop, but up to once per second

  • The speed is computed by the browser, checking the counter about once per second

  • Between two consecutive updates, the counter will increase either by 4, or by 0

  • The perceived speed will therefore be 4 - 4 - 4 - 0 - 4 - 4 - 0 etc.

  • What can we conclude from this?

  • "I'm clearly incapable of writing good frontend code!" πŸ˜€ β€” JΓ©rΓ΄me

shared/sampleapp.md

46/233

Stopping the application

  • If we interrupt Compose (with ^C), it will politely ask the Docker Engine to stop the app

  • The Docker Engine will send a TERM signal to the containers

  • If the containers do not exit in a timely manner, the Engine sends a KILL signal

  • Stop the application by hitting ^C
47/233

Stopping the application

  • If we interrupt Compose (with ^C), it will politely ask the Docker Engine to stop the app

  • The Docker Engine will send a TERM signal to the containers

  • If the containers do not exit in a timely manner, the Engine sends a KILL signal

  • Stop the application by hitting ^C

Some containers exit immediately, others take longer.

The containers that do not handle SIGTERM end up being killed after a 10s timeout. If we are very impatient, we can hit ^C a second time!

shared/sampleapp.md

48/233

Clean up

  • Before moving on, let's remove those containers
  • Tell Compose to remove everything:
    docker-compose down

shared/composedown.md

49/233

Image separating from the next module

50/233

Deploying our LKE cluster

(automatically generated title slide)

51/233

Deploying our LKE cluster

  • If we wanted to deploy Kubernetes manually, what would we need to do?

    (not that I recommend doing that...)

  • Control plane (etcd, API server, scheduler, controllers)

  • Nodes (VMs with a container engine + the Kubelet agent; CNI setup)

  • High availability (etcd clustering, API load balancer)

  • Security (CA and TLS certificates everywhere)

  • Cloud integration (to provision LoadBalancer services, storage...)

And that's just to get a basic cluster!

lke/deploy-cluster.md

52/233

The best way to deploy Kubernetes

The best way to deploy Kubernetes is to get someone else to do it for us.

(Me, ever since I've been working with Kubernetes)

lke/deploy-cluster.md

53/233

Managed Kubernetes

  • Cloud provider runs the control plane

    (including etcd, API load balancer, TLS setup, cloud integration)

  • We run nodes

    (the cloud provider generally gives us an easy way to provision them)

  • Get started in minutes

  • We're going to use Linode Kubernetes Engine

lke/deploy-cluster.md

54/233

Creating a cluster

  • With the web console:

    https://cloud.linode.com/kubernetes/clusters

  • Pick the region of your choice

  • Pick the latest available Kubernetes version

  • Pick 3 nodes with 8 GB of RAM

  • Click! ✨

  • Wait a few minutes... ⌚️

  • Download the kubeconfig file πŸ’Ύ

lke/deploy-cluster.md

55/233

With the CLI

  • View available regions with linode-cli regions list

  • View available server types with linode-cli linodes types

  • View available Kubernetes versions with linode-cli lke versions-list

  • Create cluster:

    linode-cli lke cluster-create --label=hello-lke --region=us-east \
    --k8s_version=1.20 --node_pools.type=g6-standard-4 --node_pools.count=3
  • Note the cluster ID (e.g.: 12345)

  • Download the kubeconfig file:

    linode-cli lke kubeconfig-view 12345 --text --no-headers | base64 -d

lke/deploy-cluster.md

56/233

Communicating with the cluster

  • All the Kubernetes tools (kubectl, but also helm etc) use the same config file

  • That file is (by default) $HOME/.kube/config

  • It can hold multiple cluster definitions (or contexts)

  • Or, we can have multiple config files and switch between them:

    • by adding the --kubeconfig flag each time we invoke a tool (πŸ™„)

    • or by setting the KUBECONFIG environment variable (☺️)

lke/deploy-cluster.md

57/233

Using the kubeconfig file

Option 1:

  • move the kubeconfig file to e.g. ~/.kube/config.lke

  • set the environment variable: export KUBECONFIG=~/.kube/config.lke

Option 2:

  • directly move the kubeconfig file to ~/.kube/config

  • do not do that if you already have a file there!

Option 3:

  • merge the new kubeconfig file with our existing file

lke/deploy-cluster.md

58/233

Merging kubeconfig

  • Assuming that we want to merge ~/.kube/config and ~/.kube/config.lke ...

  • Move our existing kubeconfig file:

    cp ~/.kube/config ~/.kube/config.old
  • Merge both files:

    KUBECONFIG=~/.kube/config.old:~/.kube/config.lke kubectl config \
    view --raw > ~/.kube/config
  • Check that everything is there:

    kubectl config get-contexts

lke/deploy-cluster.md

59/233

Are we there yet?

  • Let's check if our control plane is available:

    kubectl get services

    β†’ This should show the kubernetes ClusterIP service

  • Look for our nodes:

    kubectl get nodes

    β†’ This should show 3 nodes (or whatever amount we picked earlier)

  • If the nodes aren't visible yet, give them a minute to join the cluster

lke/deploy-cluster.md

60/233

Image separating from the next module

61/233

Quick Kubernetes review

(automatically generated title slide)

62/233

Quick Kubernetes review

  • Let's deploy a simple HTTP server

  • And expose it to the outside world!

  • Feel free to skip this section if you're familiar with Kubernetes

lke/kubernetes-review.md

63/233

Creating a container

  • On Kubernetes, one doesn't simply run a container

  • We need to create a "Pod"

  • A Pod will be a group of containers running together

    (often, it will be a group of one container)

  • We can create a standalone Pod, but generally, we'll use a controller

    (for instance: Deployment, Replica Set, Daemon Set, Job, Stateful Set...)

  • The controller will take care of scaling and recreating the Pod if needed

    (note that within a Pod, containers can also be restarted automatically if needed)

lke/kubernetes-review.md

64/233

A controller, you said?

  • We're going to use one of the most common controllers: a Deployment

  • Deployments...

    • can be scaled (will create the requested number of Pods)

    • will recreate Pods if e.g. they get evicted or their Node is down

    • handle rolling updates

  • Deployments actually delegate a lot of these tasks to Replica Sets

  • We will generally have the following hierarchy:

    Deployment β†’ Replica Set β†’ Pod

lke/kubernetes-review.md

65/233

Creating a Deployment

  • Without further ado:

    kubectl create deployment web --image=nginx
  • Check what happened:

    kubectl get all
  • Wait until the NGINX Pod is "Running"!

  • Note: kubectl create deployment is great when getting started...

  • ... But later, we will probably write YAML instead!

lke/kubernetes-review.md

66/233

Exposing the Deployment

  • We need to create a Service

  • We can use kubectl expose for that

    (but, again, we will probably use YAML later!)

  • For internal use, we can use the default Service type, ClusterIP:

    kubectl expose deployment web --port=80
  • For external use, we can use a Service of type LoadBalancer:

    kubectl expose deployment web --port=80 --type=LoadBalancer

lke/kubernetes-review.md

67/233

Changing the Service type

  • We can kubectl delete service web and recreate it

  • Or, kubectl edit service web and dive into the YAML

  • Or, kubectl patch service web --patch '{"spec": {"type": "LoadBalancer"}}'

  • ... These are just a few "classic" methods; there are many ways to do this!

lke/kubernetes-review.md

68/233

Deployment β†’ Pod

  • Can we check exactly what's going on when the Pod is created?

  • Option 1: watch kubectl get all

    • displays all object types
    • refreshes every 2 seconds
    • puts a high load on the API server when there are many objects
  • Option 2: kubectl get pods --watch --output-watch-events

    • can only display one type of object
    • will show all modifications happening (Γ  la tail -f)
    • doesn't put a high load on the API server (except for initial display)

lke/kubernetes-review.md

69/233

Recreating the Deployment

  • Let's delete our Deployment:

    kubectl delete deployment web
  • Watch Pod updates:

    kubectl get pods --watch --output-watch-events
  • Recreate the Deployment and see what Pods do:

    kubectl create deployment web --image=nginx

lke/kubernetes-review.md

70/233

Service stability

  • Our Service still works even though we deleted and re-created the Deployment

  • It wouldn't have worked while the Deployment was deleted, though

  • A Service is a stable endpoint

71/233

:T: Warming up with a quick Kubernetes review

:Q: In Kubernetes, what is a Pod? :A: βœ”οΈA basic unit of scaling that can contain one or more containers :A: An abstraction for an application and its dependencies :A: It's just a fancy name for "container" but they're the same :A: A group of cluster nodes used for scheduling purposes

:Q: In Kubernetes, what is a Replica Set? :A: βœ”οΈA controller used to create one or multiple identical Pods :A: A numeric parameter in a Pod specification, used to scale that Pod :A: A group of containers running on the same node :A: A group of containers running on different nodes

:Q: In Kubernetes, what is a Deployment? :A: βœ”οΈA controller that can manage Replica Sets corresponding to different configurations :A: A manifest telling Kubernetes how to deploy an app and its dependencies :A: A list of instructions executed in a container to configure that container :A: A basic unit of work for the Kubernetes scheduler

lke/kubernetes-review.md

19,000 words

They say, "a picture is worth one thousand words."

The following 19 slides show what really happens when we run:

kubectl create deployment web --image=nginx

k8s/deploymentslideshow.md

72/233

Image separating from the next module

92/233

Accessing internal services

(automatically generated title slide)

93/233

Accessing internal services

  • How can we temporarily access a service without exposing it to everyone?

  • kubectl proxy: gives us access to the API, which includes a proxy for HTTP resources

  • kubectl port-forward: allows forwarding of TCP ports to arbitrary pods, services, ...

k8s/accessinternal.md

94/233

kubectl proxy in theory

  • Running kubectl proxy gives us access to the entire Kubernetes API

  • The API includes routes to proxy HTTP traffic

  • These routes look like the following:

    /api/v1/namespaces/<namespace>/services/<service>/proxy

  • We just add the URI to the end of the request, for instance:

    /api/v1/namespaces/<namespace>/services/<service>/proxy/index.html

  • We can access services and pods this way

k8s/accessinternal.md

95/233

kubectl proxy in practice

  • Let's access the web service through kubectl proxy
  • Run an API proxy in the background:

    kubectl proxy &
  • Access the web service:

    curl localhost:8001/api/v1/namespaces/default/services/web/proxy/
  • Terminate the proxy:

    kill %1

k8s/accessinternal.md

96/233

kubectl port-forward in theory

  • What if we want to access a TCP service?

  • We can use kubectl port-forward instead

  • It will create a TCP relay to forward connections to a specific port

    (of a pod, service, deployment...)

  • The syntax is:

    kubectl port-forward service/name_of_service local_port:remote_port

  • If only one port number is specified, it is used for both local and remote ports

k8s/accessinternal.md

97/233

kubectl port-forward in practice

  • Let's access our remote NGINX server
  • Forward connections from local port 1234 to remote port 80:

    kubectl port-forward svc/web 1234:80 &
  • Connect to the NGINX server:

    curl localhost:1234
  • Terminate the port forwarder:
    kill %1
98/233

:EN:- Securely accessing internal services :FR:- Accès sécurisé aux services internes

:T: Accessing internal services from our local machine

:Q: What's the advantage of "kubectl port-forward" compared to a NodePort? :A: It can forward arbitrary protocols :A: It doesn't require Kubernetes API credentials :A: It offers deterministic load balancing (instead of random) :A: βœ”οΈIt doesn't expose the service to the public

:Q: What's the security concept behind "kubectl port-forward"? :A: βœ”οΈWe authenticate with the Kubernetes API, and it forwards connections on our behalf :A: It detects our source IP address, and only allows connections coming from it :A: It uses end-to-end mTLS (mutual TLS) to authenticate our connections :A: There is no security (as long as it's running, anyone can connect from anywhere)

k8s/accessinternal.md

Image separating from the next module

99/233

DNS, Ingress, Metrics

(automatically generated title slide)

100/233

DNS, Ingress, Metrics

  • We got a basic app up and running

  • We accessed it over a raw IP address

  • Can we do better?

    (i.e. access it with a domain name!)

  • How much resources is it using?

lke/what-is-missing.md

101/233

DNS

  • We'd like to associate a fancy name to that LoadBalancer Service

    (e.g. nginx.cloudnative.party β†’ A.B.C.D)

    • option 1: manually add a DNS record

    • option 2: find a way to create DNS records automatically

  • We will install ExternalDNS to automate DNS records creatoin

  • ExternalDNS supports Linode DNS and dozens of other providers

lke/what-is-missing.md

102/233

Ingress

  • What if we have multiple web services to expose?

  • We could create one LoadBalancer Service for each of them

  • This would create a lot of cloud load balancers

    (and they typically incur a cost, even if it's a small one)

  • Instead, we can use an Ingress Controller

  • Ingress Controller = HTTP load balancer / reverse proxy

  • Put all our HTTP services behind a single LoadBalancer Service

  • Can also do fancy "content-based" routing (using headers, request path...)

  • We will install Traefik as our Ingress Controller

lke/what-is-missing.md

103/233

Metrics

  • How much resources are we using right now?

  • When will we need to scale up our cluster?

  • We need metrics!

  • We're going to install the metrics server

  • It's a very basic metrics system

    (no retention, no graphs, no alerting...)

  • But it's lightweight, and it is used internally by Kubernetes for autoscaling

lke/what-is-missing.md

104/233

What's next

  • We're going to install all these components

  • Very often, things can be installed with a simple YAML file

  • Very often, that YAML file needs to be customized a little bit

    (add command-line parameters, provide API tokens...)

  • Instead, we're going to use Helm charts

  • Helm charts give us a way to customize what we deploy

  • Helm can also keep track of what we install

    (for easier uninstall and updates)

lke/what-is-missing.md

105/233

Image separating from the next module

106/233

Managing stacks with Helm

(automatically generated title slide)

107/233

Managing stacks with Helm

  • Helm is a (kind of!) package manager for Kubernetes

  • We can use it to:

    • find existing packages (called "charts") created by other folks

    • install these packages, configuring them for our particular setup

    • package our own things (for distribution or for internal use)

    • manage the lifecycle of these installs (rollback to previous version etc.)

  • It's a "CNCF graduate project", indicating a certain level of maturity

    (more on that later)

k8s/helm-intro.md

108/233

From kubectl run to YAML

  • We can create resources with one-line commands

    (kubectl run, kubectl createa deployment, kubectl expose...)

  • We can also create resources by loading YAML files

    (with kubectl apply -f, kubectl create -f...)

  • There can be multiple resources in a single YAML files

    (making them convenient to deploy entire stacks)

  • However, these YAML bundles often need to be customized

    (e.g.: number of replicas, image version to use, features to enable...)

k8s/helm-intro.md

109/233

Beyond YAML

  • Very often, after putting together our first app.yaml, we end up with:

    • app-prod.yaml

    • app-staging.yaml

    • app-dev.yaml

    • instructions indicating to users "please tweak this and that in the YAML"

  • That's where using something like CUE, Kustomize, or Helm can help!

  • Now we can do something like this:

    helm install app ... --set this.parameter=that.value

k8s/helm-intro.md

110/233

Other features of Helm

  • With Helm, we create "charts"

  • These charts can be used internally or distributed publicly

  • Public charts can be indexed through the Artifact Hub

  • This gives us a way to find and install other folks' charts

  • Helm also gives us ways to manage the lifecycle of what we install:

    • keep track of what we have installed

    • upgrade versions, change parameters, roll back, uninstall

  • Furthermore, even if it's not "the" standard, it's definitely "a" standard!

k8s/helm-intro.md

111/233

CNCF graduation status

  • On April 30th 2020, Helm was the 10th project to graduate within the CNCF

    πŸŽ‰

    (alongside Containerd, Prometheus, and Kubernetes itself)

  • This is an acknowledgement by the CNCF for projects that

    demonstrate thriving adoption, an open governance process,
    and a strong commitment to community, sustainability, and inclusivity.

  • See CNCF announcement and Helm announcement

k8s/helm-intro.md

112/233

Helm concepts

  • helm is a CLI tool

  • It is used to find, install, upgrade charts

  • A chart is an archive containing templatized YAML bundles

  • Charts are versioned

  • Charts can be stored on private or public repositories

k8s/helm-intro.md

113/233

Differences between charts and packages

  • A package (deb, rpm...) contains binaries, libraries, etc.

  • A chart contains YAML manifests

    (the binaries, libraries, etc. are in the images referenced by the chart)

  • On most distributions, a package can only be installed once

    (installing another version replaces the installed one)

  • A chart can be installed multiple times

  • Each installation is called a release

  • This allows to install e.g. 10 instances of MongoDB

    (with potentially different versions and configurations)

k8s/helm-intro.md

114/233

Wait a minute ...

But, on my Debian system, I have Python 2 and Python 3.
Also, I have multiple versions of the Postgres database engine!

Yes!

But they have different package names:

  • python2.7, python3.8

  • postgresql-10, postgresql-11

Good to know: the Postgres package in Debian includes provisions to deploy multiple Postgres servers on the same system, but it's an exception (and it's a lot of work done by the package maintainer, not by the dpkg or apt tools).

k8s/helm-intro.md

115/233

Helm 2 vs Helm 3

  • Helm 3 was released November 13, 2019

  • Charts remain compatible between Helm 2 and Helm 3

  • The CLI is very similar (with minor changes to some commands)

  • The main difference is that Helm 2 uses tiller, a server-side component

  • Helm 3 doesn't use tiller at all, making it simpler (yay!)

k8s/helm-intro.md

116/233

With or without tiller

  • With Helm 3:

    • the helm CLI communicates directly with the Kubernetes API

    • it creates resources (deployments, services...) with our credentials

  • With Helm 2:

    • the helm CLI communicates with tiller, telling tiller what to do

    • tiller then communicates with the Kubernetes API, using its own credentials

  • This indirect model caused significant permissions headaches

    (tiller required very broad permissions to function)

  • tiller was removed in Helm 3 to simplify the security aspects

k8s/helm-intro.md

117/233

Installing Helm

  • If the helm CLI is not installed in your environment, install it
  • Check if helm is installed:

    helm
  • If it's not installed, run the following command:

    curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get-helm-3 \
    | bash

(To install Helm 2, replace get-helm-3 with get.)

k8s/helm-intro.md

118/233

Only if using Helm 2 ...

  • We need to install Tiller and give it some permissions

  • Tiller is composed of a service and a deployment in the kube-system namespace

  • They can be managed (installed, upgraded...) with the helm CLI

  • Deploy Tiller:
    helm init

At the end of the install process, you will see:

Happy Helming!

k8s/helm-intro.md

119/233

Only if using Helm 2 ...

  • Tiller needs permissions to create Kubernetes resources

  • In a more realistic deployment, you might create per-user or per-team service accounts, roles, and role bindings

  • Grant cluster-admin role to kube-system:default service account:
    kubectl create clusterrolebinding add-on-cluster-admin \
    --clusterrole=cluster-admin --serviceaccount=kube-system:default

(Defining the exact roles and permissions on your cluster requires a deeper knowledge of Kubernetes' RBAC model. The command above is fine for personal and development clusters.)

k8s/helm-intro.md

120/233

Charts and repositories

  • A repository (or repo in short) is a collection of charts

  • It's just a bunch of files

    (they can be hosted by a static HTTP server, or on a local directory)

  • We can add "repos" to Helm, giving them a nickname

  • The nickname is used when referring to charts on that repo

    (for instance, if we try to install hello/world, that means the chart world on the repo hello; and that repo hello might be something like https://blahblah.hello.io/charts/)

k8s/helm-intro.md

121/233

How to find charts, the old way

  • Helm 2 came with one pre-configured repo, the "stable" repo

    (located at https://charts.helm.sh/stable)

  • Helm 3 doesn't have any pre-configured repo

  • The "stable" repo mentioned above is now being deprecated

  • The new approach is to have fully decentralized repos

  • Repos can be indexed in the Artifact Hub

    (which supersedes the Helm Hub)

k8s/helm-intro.md

122/233

How to find charts, the new way

  • Go to the Artifact Hub (https://artifacthub.io)

  • Or use helm search hub ... from the CLI

  • Let's try to find a Helm chart for something called "OWASP Juice Shop"!

    (it is a famous demo app used in security challenges)

k8s/helm-intro.md

123/233

Finding charts from the CLI

  • We can use helm search hub <keyword>
  • Look for the OWASP Juice Shop app:

    helm search hub owasp juice
  • Since the URLs are truncated, try with the YAML output:

    helm search hub owasp juice -o yaml

Then go to β†’ https://artifacthub.io/packages/helm/seccurecodebox/juice-shop

k8s/helm-intro.md

124/233

Finding charts on the web

  • We can also use the Artifact Hub search feature
  • Go to https://artifacthub.io/

  • In the search box on top, enter "owasp juice"

  • Click on the "juice-shop" result (not "multi-juicer" or "juicy-ctf")

k8s/helm-intro.md

125/233

Installing the chart

  • Click on the "Install" button, it will show instructions
  • First, add the repository for that chart:

    helm repo add juice https://charts.securecodebox.io
  • Then, install the chart:

    helm install my-juice-shop juice/juice-shop

Note: it is also possible to install directly a chart, with --repo https://...

k8s/helm-intro.md

126/233

Charts and releases

  • "Installing a chart" means creating a release

  • In the previous exemple, the release was named "my-juice-shop"

  • We can also use --generate-name to ask Helm to generate a name for us

  • List the releases:

    helm list
  • Check that we have a my-juice-shop-... Pod up and running:

    kubectl get pods

k8s/helm-intro.md

127/233

Searching and installing with Helm 2

  • Helm 2 doesn't have support for the Helm Hub

  • The helm search command only takes a search string argument

    (e.g. helm search juice-shop)

  • With Helm 2, the name is optional:

    helm install juice/juice-shop will automatically generate a name

    helm install --name my-juice-shop juice/juice-shop will specify a name

k8s/helm-intro.md

128/233

Viewing resources of a release

  • This specific chart labels all its resources with a release label

  • We can use a selector to see these resources

  • List all the resources created by this release:
    kubectl get all --selector=app.kubernetes.io/instance=my-juice-shop

Note: this label wasn't added automatically by Helm.
It is defined in that chart. In other words, not all charts will provide this label.

k8s/helm-intro.md

129/233

Configuring a release

  • By default, juice/juice-shop creates a service of type ClusterIP

  • We would like to change that to a NodePort

  • We could use kubectl edit service my-juice-shop, but ...

    ... our changes would get overwritten next time we update that chart!

  • Instead, we are going to set a value

  • Values are parameters that the chart can use to change its behavior

  • Values have default values

  • Each chart is free to define its own values and their defaults

k8s/helm-intro.md

130/233

Checking possible values

  • We can inspect a chart with helm show or helm inspect
  • Look at the README for the app:

    helm show readme juice/juice-shop
  • Look at the values and their defaults:

    helm show values juice/juice-shop

The values may or may not have useful comments.

The readme may or may not have (accurate) explanations for the values.

(If we're unlucky, there won't be any indication about how to use the values!)

k8s/helm-intro.md

131/233

Setting values

  • Values can be set when installing a chart, or when upgrading it

  • We are going to update my-juice-shop to change the type of the service

  • Update my-juice-shop:
    helm upgrade my-juice-shop juice/juice-shop --set service.type=NodePort

Note that we have to specify the chart that we use (juice/my-juice-shop), even if we just want to update some values.

We can set multiple values. If we want to set many values, we can use -f/--values and pass a YAML file with all the values.

All unspecified values will take the default values defined in the chart.

k8s/helm-intro.md

132/233

Connecting to the Juice Shop

  • Let's check the app that we just installed
  • Check the node port allocated to the service:

    kubectl get service my-juice-shop
    PORT=$(kubectl get service my-juice-shop -o jsonpath={..nodePort})
  • Connect to it:

    curl localhost:$PORT/
133/233

:EN:- Helm concepts :EN:- Installing software with Helm :EN:- Helm 2, Helm 3, and the Helm Hub

:FR:- Fonctionnement gΓ©nΓ©ral de Helm :FR:- Installer des composants via Helm :FR:- Helm 2, Helm 3, et le Helm Hub

:T: Getting started with Helm and its concepts

:Q: Which comparison is the most adequate? :A: Helm is a firewall, charts are access lists :A: βœ”οΈHelm is a package manager, charts are packages :A: Helm is an artefact repository, charts are artefacts :A: Helm is a CI/CD platform, charts are CI/CD pipelines

:Q: What's required to distribute a Helm chart? :A: A Helm commercial license :A: A Docker registry :A: An account on the Helm Hub :A: βœ”οΈAn HTTP server

k8s/helm-intro.md

Image separating from the next module

134/233

ExternalDNS

(automatically generated title slide)

135/233

ExternalDNS

  • ExternalDNS will automatically create DNS records from Kubernetes resources

    • Services (with the annotation external-dns.alpha.kubernetes.io/hostname)

    • Ingresses (automatically)

  • It requires a domain name (obviously)

  • ... And that domain name should be configurable through an API

  • As of April 2021, it supports a few dozens of providers

  • We're going to use Linode DNS

lke/external-dns.md

136/233

Prep work

  • We need a domain name

    (if you need a cheap one, look e.g. at GANDI; there are many options below $10)

  • That domain name should be configured to point to Linode DNS servers

    (ns1.linode.com to ns5.linode.com)

  • We need to generate a Linode API token with DNS API access

  • Pro-tip: reduce the default TTL of the domain to 5 minutes!

lke/external-dns.md

137/233

Deploying ExternalDNS

  • The ExternalDNS documentation has a tutorial for Linode

  • ... It's basically a lot of YAML!

  • That's where using a Helm chart will be very helpful

  • There are a few ExternalDNS charts available out there

  • We will use the one from Bitnami

    (these folks maintain a lot of great Helm charts!)

lke/external-dns.md

138/233

How we'll install things with Helm

  • We will install each chart in its own namespace

    (this is not mandatory, but it helps to see what belongs to what)

  • We will use helm upgrade --install instead of helm install

    (that way, if we want to change something, we can just re-run the command)

  • We will use the --create-namespace and --namespace ... options

  • To keep things boring and predictible, if we are installing chart xyz:

    • we will install it in namespace xyz

    • we will name the release xyz as well

lke/external-dns.md

139/233

Installing ExternalDNS

  • First, let's add the Bitnami repo:

    helm repo add bitnami https://charts.bitnami.com/bitnami
  • Then, install ExternalDNS:

    LINODE_API_TOKEN=1234abcd...6789
    helm upgrade --install external-dns bitnami/external-dns \
    --namespace external-dns --create-namespace \
    --set provider=linode \
    --set linode.apiToken=$LINODE_API_TOKEN

    (Make sure to update your API token above!)

lke/external-dns.md

140/233

Testing ExternalDNS

  • Let's annotate our NGINX service to expose it with a DNS record:

    kubectl annotate service web \
    external-dns.alpha.kubernetes.io/hostname=nginx.cloudnative.party

    (make sure to use your domain name above, otherwise that won't work!)

  • Check ExternalDNS logs:

    kubectl logs -n external-dns -l app.kubernetes.io/name=external-dns
  • It might take a few minutes for ExternalDNS to start, patience!

  • Then try to access nginx.cloudnative.party (or whatever domain you picked)

lke/external-dns.md

141/233

Image separating from the next module

142/233

Installing Traefik

(automatically generated title slide)

143/233

Installing Traefik

  • Traefik is going to be our Ingress Controller

  • Let's install it with a Helm chart, in its own namespace

  • First, let's add the Traefik chart repository:

    helm repo add traefik https://helm.traefik.io/traefik
  • Then, install the chart:

    helm upgrade --install traefik traefik/traefik \
    --create-namespace --namespace traefik \
    --set "ports.websecure.tls.enabled=true"

    (that option that we added enables HTTPS, it will be useful later!)

lke/traefik.md

144/233

Testing Traefik

  • Let's create an Ingress resource!

  • If we're using Kubernetes 1.20 or later, we can simply do this:

    kubectl create ingress web \
    --rule=ingress-is-fun.cloudnative.party/*=web:80

    (make sure to update and use your own domain)

  • Check that the Ingress was correctly created:

    kubectl get ingress
    kubectl describe ingress
  • If we're using Kubernetes 1.19 or earlier, we'll need some YAML

lke/traefik.md

145/233

Creating an Ingress with YAML

  • This is how we do it with YAML:
    kubectl apply -f- <<EOF
    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
    name: web
    spec:
    rules:
    - host: ingress-is-fun.cloudnative.party
    http:
    paths:
    - path: /
    backend:
    serviceName: web
    servicePort: 80
    EOF

lke/traefik.md

146/233

Ingress versions...

  • Note how we used the v1beta1 Ingress version on the previous YAML

    (to be compatible with older Kubernetes versions)

  • This YAML will give you deprecation warnings on recent version of Kubernetes

    (since the Ingress spec is now at version v1)

  • Don't worry too much about the deprecation warnings

    (on Kubernetes, deprecation happens over a long time window, typically 1 year)

  • You will have time to revisit and worry later! πŸ˜…

lke/traefik.md

147/233

Does it work?

  • Try to connect to the Ingress host name

    (in my example, http://ingress-is-fun.cloudnative.party/)

  • Normally, it doesn't work (yet) πŸ€”

  • Let's look at kubectl get ingress again

  • ExternalDNS is trying to create records mapping HOSTS to ADDRESS

  • But the ADDRESS field is currently empty!

  • We need to tell Traefik to fill that ADDRESS field

lke/traefik.md

148/233

Reconfiguring Traefik

  • There is a "magic" flag to tell Traefik to update the address status field

  • Let's update our Traefik install:

    helm upgrade --install traefik traefik/traefik \
    --create-namespace --namespace traefik \
    --set "ports.websecure.tls.enabled=true" \
    --set "providers.kubernetesIngress.publishedService.enabled=true"

lke/traefik.md

149/233

Checking what we did

  • Check the output of kubectl get ingress

    (there should be an address now)

  • Check the logs of ExternalDNS

    (there should be a mention of the new DNS record)

  • Try again to connect to the HTTP address

    (now it should work)

  • Note that some of these operations might take a minute or two

    (be patient!)

150/233

:T: Installing the Traefik Ingress Controller

:Q: What's the job of an Ingress Controller? :A: Prevent unauthorized access to Kubernetes services :A: Firewall inbound traffic on the Kubernetes API :A: βœ”οΈHandle inbound HTTP traffic for Kubernetes services :A: Keep track of the location of Kubernetes operators

:Q: What happens when we create an "Ingress resource"? :A: A web service is automatically deployed and scaled on our cluster :A: Kubernetes starts tracking the location of our users :A: Traffic coming from the specified addresses will be allowed :A: βœ”οΈA load balancer is configured with HTTP traffic rules

lke/traefik.md

Image separating from the next module

151/233

Installing metrics-server

(automatically generated title slide)

152/233

Installing metrics-server

  • We've installed a few things on our cluster so far

  • How much resources (CPU, RAM) are we using?

  • We need metrics!

  • If metrics-server is installed, we can get Nodes metrics like this:

    kubectl top nodes
  • At the moment, this should show us error: Metrics API not available

  • How do we fix this?

lke/metrics-server.md

153/233

Many ways to get metrics

  • We could use a SAAS like Datadog, New Relic...

  • We could use a self-hosted solution like Prometheus

  • Or we could use metrics-server

  • What's special about metrics-server?

lke/metrics-server.md

154/233

Pros/cons

Cons:

  • no data retention (no history data, just instant numbers)

  • only CPU and RAM of nodes and pods (no disk or network usage or I/O...)

Pros:

  • very lightweight

  • doesn't require storage

  • used by Kubernetes autoscaling

lke/metrics-server.md

155/233

Why metrics-server

  • We may install something fancier later

    (think: Prometheus with Grafana)

  • But metrics-server will work in minutes

  • It will barely use resources on our cluster

  • It's required for autoscaling anyway

lke/metrics-server.md

156/233

How metric-server works

  • It runs a single Pod

  • That Pod will fetch metrics from all our Nodes

  • It will expose them through the Kubernetes API agregation layer

    (we won't say much more about that agregation layer; that's fairly advanced stuff!)

lke/metrics-server.md

157/233

Installing metrics-server

  • In a lot of places, this is done with a little bit of custom YAML

    (derived from the official installation instructions)

  • We're going to use Helm one more time:

    helm upgrade --install metrics-server bitnami/metrics-server \
    --create-namespace --namespace metrics-server \
    --set apiService.create=true \
    --set extraArgs.kubelet-insecure-tls=true \
    --set extraArgs.kubelet-preferred-address-types=InternalIP
  • What are these options for?

lke/metrics-server.md

158/233

Installation options

  • apiService.create=true

    register metrics-server with the Kubernetes agregation layer

    (create an entry that will show up in kubectl get apiservices)

  • extraArgs.kubelet-insecure-tls=true

    when connecting to nodes to collect their metrics, don't check kubelet TLS certs

    (because most kubelet certs include the node name, but not its IP address)

  • extraArgs.kubelet-preferred-address-types=InternalIP

    when connecting to nodes, use their internal IP address instead of node name

    (because the latter requires an internal DNS, which is rarely configured)

lke/metrics-server.md

159/233

Testing metrics-server

  • After a minute or two, metrics-server should be up

  • We should now be able to check Nodes resource usage:

    kubectl top nodes
  • And Pods resource usage, too:

    kubectl top pods --all-namespaces

lke/metrics-server.md

160/233

Keep some padding

  • The RAM usage that we see should correspond more or less to the Resident Set Size

  • Our pods also need some extra space for buffers, caches...

  • Do not aim for 100% memory usage!

  • Some more realistic targets:

    50% (for workloads with disk I/O and leveraging caching)

    90% (on very big nodes with mostly CPU-bound workloads)

    75% (anywhere in between!)

lke/metrics-server.md

161/233

Image separating from the next module

162/233

Prometheus and Grafana

(automatically generated title slide)

163/233

Prometheus and Grafana

  • What if we want metrics retention, view graphs, trends?

  • A very popular combo is Prometheus+Grafana:

    • Prometheus as the "metrics engine"

    • Grafana to display comprehensive dashboards

  • Prometheus also has an alert-manager component to trigger alerts

    (we won't talk about that one)

lke/prometheus.md

164/233

Installing Prometheus and Grafana

  • A complete metrics stack needs at least:

    • the Prometheus server (collects metrics and stores them efficiently)

    • a collection of exporters (exposing metrics to Prometheus)

    • Grafana

    • a collection of Grafana dashboards (building them from scratch is tedious)

  • The Helm chart kube-prometheus-stack combines all these elements

  • ... So we're going to use it to deploy our metrics stack!

lke/prometheus.md

165/233

Installing kube-prometheus-stack

  • Let's install that stack directly from its repo

    (without doing helm repo add first)

  • Otherwise, keep the same naming strategy:

    helm upgrade --install kube-prometheus-stack kube-prometheus-stack \
    --namespace kube-prometheus-stack --create-namespace \
    --repo https://prometheus-community.github.io/helm-charts
  • This will take a minute...

  • Then check what was installed:

    kubectl get all --namespace kube-prometheus-stack

lke/prometheus.md

166/233

Exposing Grafana

  • Let's create an Ingress for Grafana

    kubectl create ingress --namespace kube-prometheus-stack grafana \
    --rule=grafana.cloudnative.party/*=kube-prometheus-stack-grafana:80

    (as usual, make sure to use your domain name above)

  • Connect to Grafana

    (remember that the DNS record might take a few minutes to come up)

lke/prometheus.md

167/233

Grafana credentials

  • What could the login and password be?

  • Let's look at the Secrets available in the namespace:

    kubectl get secrets --namespace kube-prometheus-stack
  • There is a kube-prometheus-stack-grafana that looks promising!

  • Decode the Secret:

    kubectl get secret --namespace kube-prometheus-stack \
    kube-prometheus-stack-grafana -o json | jq '.data | map_values(@base64d)'
  • If you don't have the jq tool mentioned above, don't worry...

168/233

Grafana credentials

  • What could the login and password be?

  • Let's look at the Secrets available in the namespace:

    kubectl get secrets --namespace kube-prometheus-stack
  • There is a kube-prometheus-stack-grafana that looks promising!

  • Decode the Secret:

    kubectl get secret --namespace kube-prometheus-stack \
    kube-prometheus-stack-grafana -o json | jq '.data | map_values(@base64d)'
  • If you don't have the jq tool mentioned above, don't worry...

  • The login/password is hardcoded to admin/prom-operator 😬

lke/prometheus.md

169/233

Grafana dashboards

  • Once logged in, click on the "Dashboards" icon on the left

    (it's the one that looks like four squares)

  • Then click on the "Manage" entry

  • Then click on "Kubernetes / Compute Resources / Cluster"

  • This gives us a breakdown of resource usage by Namespace

  • Feel free to explore the other dashboards!

170/233

:T: Observing our cluster with Prometheus and Grafana

:Q: What's the relationship between Prometheus and Grafana? :A: Prometheus collects and graphs metrics; Grafana sends alerts :A: βœ”οΈPrometheus collects metrics; Grafana displays them on dashboards :A: Prometheus collects and graphs metrics; Grafana is its configuration interface :A: Grafana collects and graphs metrics; Prometheus sends alerts

lke/prometheus.md

Image separating from the next module

171/233

cert-manager

(automatically generated title slide)

172/233

cert-manager

  • cert-managerΒΉ facilitates certificate signing through the Kubernetes API:

    • we create a Certificate object (that's a CRD)

    • cert-manager creates a private key

    • it signs that key ...

    • ... or interacts with a certificate authority to obtain the signature

    • it stores the resulting key+cert in a Secret resource

  • These Secret resources can be used in many places (Ingress, mTLS, ...)

ΒΉAlways lower case, words separated with a dash; see the style guide

k8s/cert-manager.md

173/233

Getting signatures

  • cert-manager can use multiple Issuers (another CRD), including:

  • Multiple issuers can be configured simultaneously

  • Issuers can be available in a single namespace, or in the whole cluster

    (then we use the ClusterIssuer CRD)

k8s/cert-manager.md

174/233

cert-manager in action

  • We will install cert-manager

  • We will create a ClusterIssuer to obtain certificates with Let's Encrypt

    (this will involve setting up an Ingress Controller)

  • We will create a Certificate request

  • cert-manager will honor that request and create a TLS Secret

k8s/cert-manager.md

175/233

Installing cert-manager

  • It can be installed with a YAML manifest, or with Helm
  • Let's install the cert-manager Helm chart with this one-liner:
    helm install cert-manager cert-manager \
    --repo https://charts.jetstack.io \
    --create-namespace --namespace cert-manager \
    --set installCRDs=true
  • If you prefer to install with a single YAML file, that's fine too!

    (see the documentation for instructions)

k8s/cert-manager.md

176/233

ClusterIssuer manifest

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
# Remember to update this if you use this manifest to obtain real certificates :)
email: hello@example.com
server: https://acme-staging-v02.api.letsencrypt.org/directory
# To use the production environment, use the following line instead:
#server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: issuer-letsencrypt-staging
solvers:
- http01:
ingress:
class: traefik

k8s/cert-manager.md

177/233

Creating the ClusterIssuer

  • Create the ClusterIssuer:
    kubectl apply cm-clusterissuer.yaml

k8s/cert-manager.md

178/233

Certificate manifest

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: xyz.A.B.C.D.nip.io
spec:
secretName: xyz.A.B.C.D.nip.io
dnsNames:
- xyz.A.B.C.D.nip.io
issuerRef:
name: letsencrypt-staging
kind: ClusterIssuer
  • The name, secretName, and dnsNames don't have to match

  • There can be multiple dnsNames

  • The issuerRef must match the ClusterIssuer that we created earlier

k8s/cert-manager.md

179/233

Creating the Certificate

  • Edit the Certificate to update the domain name

    (make sure to replace A.B.C.D with the IP address of one of your nodes!)

  • Create the Certificate:

    kubectl apply -f cm-certificate.yaml

k8s/cert-manager.md

180/233

What's happening?

  • cert-manager will create:

    • the secret key

    • a Pod, a Service, and an Ingress to complete the HTTP challenge

  • then it waits for the challenge to complete

  • View the resources created by cert-manager:
    kubectl get pods,services,ingresses \
    --selector=acme.cert-manager.io/http01-solver=true

k8s/cert-manager.md

181/233

HTTP challenge

  • The CA (in this case, Let's Encrypt) will fetch a particular URL:

    http://<our-domain>/.well-known/acme-challenge/<token>

  • Check the path of the Ingress in particular:
    kubectl describe ingress
    --selector=acme.cert-manager.io/http01-solver=true

k8s/cert-manager.md

182/233

And then...

  • A little bit later, we will have a kubernetes.io/tls Secret:

    kubectl get secrets
  • Note that this might take a few minutes, because of the DNS integration!

k8s/cert-manager.md

183/233

Using the secret

  • For bonus points, try to use the secret in an Ingress!

  • This is what the manifest would look like:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: xyz
spec:
tls:
- secretName: xyz.A.B.C.D.nip.io
hosts:
- xyz.A.B.C.D.nip.io
rules:
...

k8s/cert-manager.md

184/233

Automatic TLS Ingress with annotations

  • It is also possible to annotate Ingress resources for cert-manager

  • If we annotate an Ingress resource with cert-manager.io/cluster-issuer=xxx:

    • cert-manager will detect that annotation

    • it will obtain a certificate using the specified ClusterIssuer (xxx)

    • it will store the key and certificate in the specified Secret

  • Note: the Ingress still needs the tls section with secretName and hosts

185/233

:EN:- Obtaining certificates with cert-manager :FR:- Obtenir des certificats avec cert-manager

:T: Obtaining TLS certificates with cert-manager

k8s/cert-manager.md

Image separating from the next module

186/233

CI/CD with GitLab

(automatically generated title slide)

187/233

CI/CD with GitLab

  • In this section, we will see how to set up a CI/CD pipeline with GitLab

    (using a "self-hosted" GitLab; i.e. running on our Kubernetes cluster)

  • The big picture:

    • each time we push code to GitLab, it will be deployed in a staging environment

    • each time we push the production tag, it will be deployed in production

k8s/gitlab.md

188/233

Disclaimers

  • We'll use GitLab here as an exemple, but there are many other options

    (e.g. some combination of Argo, Harbor, Tekton ...)

  • There are also hosted options

    (e.g. GitHub Actions and many others)

  • We'll use a specific pipeline and workflow, but it's purely arbitrary

    (treat it as a source of inspiration, not a model to be copied!)

k8s/gitlab.md

189/233

Workflow overview

  • Push code to GitLab's git server

  • GitLab notices the .gitlab-ci.yml file, which defines our pipeline

  • Our pipeline can have multiple stages executed sequentially

    (e.g. lint, build, test, deploy ...)

  • Each stage can have multiple jobs executed in parallel

    (e.g. build images in parallel)

  • Each job will be executed in an independent runner pod

k8s/gitlab.md

190/233

Pipeline overview

  • Our repository holds source code, Dockerfiles, and a Helm chart

  • Lint stage will check the Helm chart validity

  • Build stage will build container images

    (and push them to GitLab's integrated registry)

  • Deploy stage will deploy the Helm chart, using these images

  • Pushes to production will deploy to "the" production namespace

  • Pushes to other tags/branches will deploy to a namespace created on the fly

  • We will discuss shortcomings and alternatives and the end of this chapter!

k8s/gitlab.md

191/233

Lots of requirements

  • We need a lot of components to pull this off:

    • a domain name

    • a storage class

    • a TLS-capable ingress controller

    • the cert-manager operator

    • GitLab itself

    • the GitLab pipeline

  • Wow, why?!?

k8s/gitlab.md

192/233

I find your lack of TLS disturbing

  • We need a container registry (obviously!)

  • Docker (and other container engines) require TLS on the registry

    (with valid certificates)

  • A few options:

    • use a "real" TLS certificate (e.g. obtained with Let's Encrypt)

    • use a self-signed TLS certificate

    • communicate with the registry over localhost (TLS isn't required then)

k8s/gitlab.md

193/233

Why not self-signed certs?

  • When using self-signed certs, we need to either:

    • add the cert (or CA) to trusted certs

    • disable cert validation

  • This needs to be done on every client connecting to the registry:

    • CI/CD pipeline (building and pushing images)

    • container engine (deploying the images)

    • other tools (e.g. container security scanner)

  • It's doable, but it's a lot of hacks (especially when adding more tools!)

k8s/gitlab.md

194/233

Why not localhost?

  • TLS is usually not required when the registry is on localhost

  • We could expose the registry e.g. on a NodePort

  • ... And then tweak the CI/CD pipeline to use that instead

  • This is great when obtaining valid certs is difficult:

    • air-gapped or internal environments (that can't use Let's Encrypt)

    • no domain name available

  • Downside: the registry isn't easily or safely available from outside

    (the NodePort essentially defeats TLS)

k8s/gitlab.md

195/233

Can we use nip.io?

  • We will use Let's Encrypt

  • Let's Encrypt has a quota of certificates per domain

    (in 2020, that was 50 certificates per week per domain)

  • So if we all use nip.io, we will probably run into that limit

  • But you can try and see if it works!

k8s/gitlab.md

196/233

Install GitLab itself

  • We will deploy GitLab with its official Helm chart

  • It will still require a bunch of parameters and customization

  • Brace!

k8s/gitlab.md

197/233

Installing the GitLab chart

helm repo add gitlab https://charts.gitlab.io/
DOMAIN=cloudnative.party
ISSUER=letsencrypt-production
helm upgrade --install gitlab gitlab/gitlab \
--create-namespace --namespace gitlab \
--set global.hosts.domain=$DOMAIN \
--set certmanager.install=false \
--set nginx-ingress.enabled=false \
--set global.ingress.class=traefik \
--set global.ingress.provider=traefik \
--set global.ingress.configureCertmanager=false \
--set global.ingress.annotations."cert-manager\.io/cluster-issuer"=$ISSUER \
--set gitlab.webservice.ingress.tls.secretName=gitlab-gitlab-tls \
--set registry.ingress.tls.secretName=gitlab-registry-tls \
--set minio.ingress.tls.secretName=gitlab-minio-tls

😰 Can we talk about all these parameters?

k8s/gitlab.md

198/233

Breaking down all these parameters

  • certmanager.install=false

    do not install cert-manager, we already have it

  • nginx-ingress.enabled=false

    do not install the NGINX ingress controller, we already have Traefik

  • global.ingress.class=traefik, global.ingress.provider=traefik

    these merely enable creation of Ingress resources

  • global.ingress.configureCertmanager=false

    do not create a cert-manager Issuer or ClusterIssuer, we have ours

k8s/gitlab.md

199/233

More parameters

  • global.ingress.annotations."cert-manager\.io/cluster-issuer"=$ISSUER

    this annotation tells cert-manager to automatically issue certs

  • gitlab.webservice.ingress.tls.secretName=gitlab-gitlab-tls,
    registry.ingress.tls.secretName=gitlab-registry-tls,
    minio.ingress.tls.secretName=gitlab-minio-tls

    these annotations enable TLS in the Ingress controller

k8s/gitlab.md

200/233

Wait for GitLab to come up

  • Let's watch what's happening in the GitLab namespace:

    watch kubectl get all --namespace gitlab
  • We want to wait for all the Pods to be "Running" or "Completed"

  • This will take a few minutes (10-15 minutes for me)

  • Don't worry if you see Pods crashing and restarting

    (it happens when they are waiting on a dependency which isn't up yet)

k8s/gitlab.md

201/233

Things that could go wrong

  • Symptom: Pods remain "Pending" or "ContainerCreating" for a while

  • Investigate these pods (with kubectl describe pod ...)

  • Also look at events:

    kubectl get events \
    --field-selector=type=Warning --sort-by=metadata.creationTimestamp
  • Make sure your cluster is big enough

    (I use 3 g6-standard-4 nodes)

k8s/gitlab.md

202/233

Log into GitLab

  • First, let's check that we can connect to GitLab (with TLS):

    https://gitlab.$DOMAIN

  • It's asking us for a login and password!

  • The login is root, and the password is stored in a Secret:

    kubectl get secrets --namespace=gitlab gitlab-gitlab-initial-root-password \
    -o jsonpath={.data.password} | base64 -d

k8s/gitlab.md

203/233

Configure GitLab

  • For simplicity, we're going to use that "root" user

    (but later, you can create multiple users, teams, etc.)

  • First, let's add our SSH key

    (top-right user menu β†’ settings, then SSH keys on the left)

  • Then, create a project

    (using the + menu next to the search bar on top)

  • Let's call it kubecoin

    (you can change it, but you'll have to adjust Git paths later on)

k8s/gitlab.md

204/233

Try to push our repository

  • This is the repository that we're going to use:

    https://github.com/jpetazzo/kubecoin

  • Let's clone that repository locally first:

    git clone https://github.com/jpetazzo/kubecoin
  • Add our GitLab instance as a remote:

    git remote add gitlab git@gitlab.$DOMAIN:root/kubecoin.git
  • Try to push:

    git push -u gitlab

k8s/gitlab.md

205/233

Connection refused?

  • Normally, we get the following error:

    port 22: Connection refused

  • Why? πŸ€”

206/233

Connection refused?

  • Normally, we get the following error:

    port 22: Connection refused

  • Why? πŸ€”

  • What does gitlab.$DOMAIN point to?

207/233

Connection refused?

  • Normally, we get the following error:

    port 22: Connection refused

  • Why? πŸ€”

  • What does gitlab.$DOMAIN point to?

  • Our Ingress Controller! (i.e. Traefik) πŸ’‘

  • Our Ingress Controller has nothing to do with port 22

  • So how do we solve this?

k8s/gitlab.md

208/233

Routing port 22

  • Whatever is on gitlab.$DOMAIN needs to have the following "routing":

    • port 80 β†’ GitLab web service

    • port 443 β†’ GitLab web service, with TLS

    • port 22 β†’ GitLab shell service

  • Currently, Traefik is managing gitlab.$DOMAIN

  • We are going to tell Traefik to:

    • accept connections on port 22

    • send them to GitLab

k8s/gitlab.md

209/233

TCP routing

  • The technique that we are going to use is specific to Traefik

  • Other Ingress Controllers may or may not have similar features

  • When they have similar features, they will be enabled very differently

k8s/gitlab.md

210/233

Telling Traefik to open port 22

  • Let's reconfigure Traefik:

    helm upgrade --install traefik traefik/traefik \
    --create-namespace --namespace traefik \
    --set "ports.websecure.tls.enabled=true" \
    --set "providers.kubernetesIngress.publishedService.enabled=true" \
    --set "ports.ssh.port=2222" \
    --set "ports.ssh.exposedPort=22" \
    --set "ports.ssh.expose=true" \
    --set "ports.ssh.protocol=TCP"
  • This creates a new "port" on Traefik, called "ssh", listening on port 22

  • Internally, Traefik listens on port 2222 (for permission reasons)

  • Note: Traefik docs also call these ports "entrypoints"

    (these entrypoints are totally unrelated to the ENTRYPOINT in Dockerfiles)

k8s/gitlab.md

211/233

Knocking on port 22

  • What happens if we try to connect to that port 22 right now?

    curl gitlab.$DOMAIN:22
  • We hit GitLab's web service!

  • We need to tell Traefik what to do with connections to that port 22

  • For that, we will create a "TCP route"

k8s/gitlab.md

212/233

Traefik TCP route

The following custom resource tells Traefik to route the ssh port that we created earlier, to the gitlab-gitlab-shell service belonging to GitLab.

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
name: gitlab-shell
namespace: gitlab
spec:
entryPoints:
- ssh
routes:
- match: HostSNI(`*`)
services:
- name: gitlab-gitlab-shell
port: 22

The HostSNI wildcard is the magic option to define a "default route".

k8s/gitlab.md

213/233

Creating the TCP route

Since our manifest has backticks, we must pay attention to quoting:

kubectl apply -f- << "EOF"
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
name: gitlab-shell
namespace: gitlab
spec:
entryPoints:
- ssh
routes:
- match: HostSNI(`*`)
services:
- name: gitlab-gitlab-shell
port: 22
EOF

k8s/gitlab.md

214/233

Knocking on port 22, again

  • Let's see what happens if we try port 22 now:

    curl gitlab.$DOMAIN:22
  • This should tell us something like Received HTTP/0.9 when not allowed

    (because we're no longer talking to an HTTP server, but to SSH!)

  • Try with SSH:

    ssh git@gitlab.$DOMAIN
  • After accepting the key fingerprint, we should see Welcome to GitLab, @root!

k8s/gitlab.md

215/233

Pushing again

  • Now we can try to push our repository again:

    git push -u gitlab
  • Reload the project page in GitLab

  • We should see our repository!

k8s/gitlab.md

216/233

CI/CD

  • Click on the CI/CD tab on the left

    (the one with the shuttle / space rocket icon)

  • Our pipeline was detected...

  • But it failed πŸ˜•

  • Let's click on one of the failed jobs

  • This is a permission issue!

k8s/gitlab.md

217/233

Fixing permissions

  • GitLab needs to do a few of things in our cluster:

    • create Pods to build our container images with BuildKit

    • create Namespaces to deploy staging and production versions of our app

    • create and update resources in these Namespaces

  • For the time being, we're going to grant broad permissions

    (and we will revisit and discuss what to do later)

k8s/gitlab.md

218/233

Granting permissions

  • Let's give cluster-admin permissions to the GitLab ServiceAccount:

    kubectl create clusterrolebinding gitlab \
    --clusterrole=cluster-admin --serviceaccount=gitlab:default
  • Then retry the CI/CD pipeline

  • The build steps will now succeed; but the deploy steps will fail

  • We need to set the REGISTRY_USER and REGISTRY_PASSWORD variables

  • Let's explain what this is about!

k8s/gitlab.md

219/233

GitLab container registry access

  • A registry access token is created for the duration of the CI/CD pipeline

    (it is exposed through the $CI_JOB_TOKEN environment variable)

  • This token gives access only to a specific repository in the registry

  • It is valid only during the execution of the CI/CD pipeline

  • We can (and we do!) use it to push images to the registry

  • We cannot use it to pull images when running in staging or production

    (because Kubernetes might need to pull images after the token expires)

  • We need to create a separate read-only registry access token

k8s/gitlab.md

220/233

Creating the registry access token

  • Let's go to "Settings" (the cog wheel on the left) / "Access Tokens"

  • Create a token with read_registry permission

  • Save the token name and the token value

  • Then go to "Settings" / "CI/CD"

  • In the "Variables" section, add two variables:

    • REGISTRY_USER β†’ token name
    • REGISTRY_PASSWORD β†’ token value
  • Make sure that they are not protected!

    (otherwise, they won't be available in non-default tags and branches)

k8s/gitlab.md

221/233

Trying again

  • Go back to the CI/CD pipeline view, and hit "Retry"

  • The deploy stage should now work correctly! πŸŽ‰

k8s/gitlab.md

222/233

Our CI/CD pipeline

  • Let's have a look at the .gitlab-ci.yml file

  • We have multiple stages:

    • lint (currently doesn't do much, it's mostly as an example)

    • build (currently uses BuildKit)

    • deploy

  • "Deploy" behaves differently in staging and production

  • Let's investigate that!

k8s/gitlab.md

223/233

Staging vs production

  • In our pipeline, "production" means "a tag or branch named production"

    (see the except: and only: sections)

  • Everything else is "staging"

  • In "staging":

    • we build and push images
    • we create a staging Namespace and deploy a copy of the app there
  • In "production":

    • we do not build anything
    • we deploy (or update) a copy of the app in the production Namespace

k8s/gitlab.md

224/233

Namespace naming

  • GitLab will create Namespaces named gl-<user>-<project>-<hash>

  • At the end of the deployment, the web UI will be available at:

    http://<user>-<project>-<githash>-gitlab.<domain>

  • The "production" Namespace will be <user>-<project>

  • And it will be available on its own domain as well:

    http://<project>-<githash>-gitlab.<domain>

k8s/gitlab.md

225/233

Production

  • git tag -f production && git push -f --tags

  • Our CI/CD pipeline will deploy on the production URL

    (http://<user>-<project>-gitlab.<domain>)

  • It will do it only if that same git commit was pushed to staging first

    (because the "production" pipeline skips the build phase)

k8s/gitlab.md

226/233

Let's talk about build

  • There are many ways to build container images on Kubernetes

  • And they all suck Many of them have inconveniencing issues

  • Let's do a quick review!

k8s/gitlab.md

227/233

Docker-based approaches

  • Bind-mount the Docker socket

    • very easy, but requires Docker Engine
    • build resource usage "evades" Kubernetes scheduler
    • insecure
  • Docker-in-Docker in a pod

    • requires privileged pod
    • insecure
    • approaches like rootless or sysbox might help in the future
  • External build host

    • more secure
    • requires resources outside of the Kubernetes cluster

k8s/gitlab.md

228/233

Non-privileged builders

  • Kaniko

    • each build runs in its own containers or pod
    • no caching by default
    • registry-based caching is possible
  • BuildKit / docker buildx

    • can leverage Docker Engine or long-running Kubernetes worker pod
    • supports distributed, multi-arch build farms
    • basic caching out of the box
    • can also leverage registry-based caching

k8s/gitlab.md

229/233

Other approaches

  • Ditch the Dockerfile!

  • bazel

  • jib

  • ko

  • etc.

k8s/gitlab.md

230/233

Discussion

  • Our CI/CD workflow is just one of the many possibilities

  • It would be nice to add some actual unit or e2e tests

  • Map the production namespace to a "real" domain name

  • Automatically remove older staging environments

    (see e.g. kube-janitor)

  • Deploy production to a separate cluster

  • Better segregate permissions

    (don't give cluster-admin to the GitLab pipeline)

k8s/gitlab.md

231/233

Why not use GitLab's Kubernetes integration?

  • "All-in-one" approach

    (deploys its own Ingress, cert-manager, Prometheus, and much more)

  • I wanted to show you something flexible and customizable instead

  • But feel free to explore it now that we have shown the basics!

232/233

:EN:- CI/CD with GitLab :FR:- CI/CD avec GitLab

k8s/gitlab.md

That's all, folks!
Thank you ✨

end

shared/thankyou.md

233/233

Intros

  • Hello! I'm JΓ©rΓ΄me Petazzoni

    (@jpetazzo on Twitter)

  • I worked at Docker from ~2011 to 2018

  • I'm now doing consulting, training, etc. on Docker & Kubernetes

    (check out container.training!)

  • I'll show you how to deploy a complete CI/CD pipeline on LKE!

    (Linode Kubernetes Engine 😎)

logistics.md

2/233
Paused

Help

Keyboard shortcuts

↑, ←, Pg Up, k Go to previous slide
↓, β†’, Pg Dn, Space, j Go to next slide
Home Go to first slide
End Go to last slide
Number + Return Go to specific slide
b / m / f Toggle blackout / mirrored / fullscreen mode
c Clone slideshow
p Toggle presenter mode
t Restart the presentation timer
?, h Toggle this help
Esc Back to slideshow