Cloud Native
Continuous Deployment
with GitLab, Helm, and
Linode Kubernetes Engine
Hello! I'm JΓ©rΓ΄me Petazzoni
(@jpetazzo on Twitter)
I worked at Docker from ~2011 to 2018
I'm now doing consulting, training, etc. on Docker & Kubernetes
(check out container.training!)
I'll show you how to deploy a complete CI/CD pipeline on LKE!
(Linode Kubernetes Engine π)
We recommend that you open these slides in your browser:
Use arrows to move to next/previous slide
(up, down, left, right, page up, page down)
Type a slide number + ENTER to go to that slide
The slide number is also visible in the URL bar
(e.g. .../#123 for slide 123)
Slides will remain online so you can review them later if needed
(let's say we'll keep them online at least 1 year, how about that?)
You can download the slides using that URL:
https://2021-03-lke.container.training/slides.zip
(then open the file lke.yml.html
)
You will find new versions of these slides on:
You are welcome to use, re-use, share these slides
These slides are written in markdown
The sources of these slides are available in a public GitHub repository:
Typos? Mistakes? Questions? Feel free to hover over the bottom of the slide ...
π Try it! The source file will be shown and you can view it on GitHub and fork and edit it.
This slide has a little magnifying glass in the top left corner
This magnifying glass indicates slides that provide extra details
Feel free to skip them if:
you are in a hurry
you are new to this and want to avoid cognitive overload
you want only the most essential information
You can review these slides another time if you want, they'll be waiting for you βΊ
(auto-generated TOC)
(auto-generated TOC)
Get ready!
(automatically generated title slide)
We're going to set up a whole Continous Deployment pipeline
... for Kubernetes apps
... on a Kubernetes cluster
Ingredients: cert-manager, GitLab, Helm, Linode DNS, LKE, Traefik
"Do one thing, do it well"
... But a CD pipeline is a complex system with interconnected parts!
GitLab is no exception to that rule
Let's have a look at its components!
GitLab dependencies listed in the GitLab official Helm chart
External dependencies:
cert-manager, grafana, minio, nginx-ingress, postgresql, prometheus, redis, registry, shared-secrets
(these dependencies correspond to external charts not created by GitLab)
Internal dependencies:
geo-logcursor, gitaly, gitlab-exporter, gitlab-grafana, gitlab-pages, gitlab-shell, kas, mailroom, migrations, operator, praefect, sidekiq, task-runner, webservice
(these dependencies correspond to subcharts embedded in the GitLab chart)
Use the GitLab chart to deploy everything that is specific to GitLab
Deploy cluster-wide components separately
(cert-manager, ExternalDNS, Ingress Controller...)
Spin up an LKE cluster
Run a simple test app
Install a few extras
(the cluster-wide components mentioned earlier)
Set up GitLab
Push an app with a CD pipeline to GitLab
If you just want to follow along and watch...
container basics (what's an image, what's a container...)
Kubernetes basics (what are Deployments, Namespaces, Pods, Services)
If you want to run this on your own Kubernetes cluster...
intermediate Kubernetes concepts (annotations, Ingresses)
Helm basic concepts (how to install/upgrade releases; how to set "values")
basic Kubernetes troubleshooting commands (view logs, events)
There will be a lot of explanations and reminders along the way
If you want to run this on your own...
A Linode account
A domain name that you will point to Linode DNS
(I got cloudnative.party for $5)
Local tools to control your Kubernetes cluster:
Patience, as many operations will require us to wait a few minutes!
Can I use a local cluster, e.g. with Minikube?
It will be very difficult to get valid TLS certs with a local cluster.
Also, GitLab needs quite a bit of resources.
Can I use another Kubernetes provider?
You certainly can: Kubernetes is a standard platform!
But you'll have to adjust a few things.
(I'll try my best to tell you what as we go along.)
Because accessing gitlab.cloudnative.party is easier than 102.34.55.67
Because we'll need TLS certificates
(and it's very easy to obtain certs with Let's Encrypt when we have a domain)
We'll illustrate automatic DNS configuration with ExternalDNS, too!
(Kubernetes will automatically create DNS entries in our domain)
Here are a few tools that I like...
linode-cli to manage Linode resources from the command line
stern to comfortably view logs of Kubernetes pods
k9s to manage Kubernetes resources with that retro BBS look and feel π
kube-ps1 to keep track of which Kubernetes cluster and namespace we're working on
kubectx to easily switch between clusters, contexts, and namespaces
We're going to spin up cloud resources
Remember to shut them down when you're down!
In the immortal words of Cloud Economist Corey Quinn:
Our sample application
(automatically generated title slide)
I'm going to run our demo app locally, with Docker
(you don't have to do that; do it if you like!)
git clone https://github.com/jpetazzo/container.training
(You can also fork the repository on GitHub and clone your fork if you prefer that.)
Let's start this before we look around, as downloading will take a little time...
Go to the dockercoins
directory, in the cloned repo:
cd container.training/dockercoins
Use Compose to build and run all containers:
docker-compose up
Compose tells Docker to build all container images (pulling the corresponding base images), then starts all containers, and displays aggregated logs.
It is a DockerCoin miner! π°π³π¦π’
No, you can't buy coffee with DockerCoins
It is a DockerCoin miner! π°π³π¦π’
No, you can't buy coffee with DockerCoins
How DockerCoins works:
generate a few random bytes
hash these bytes
increment a counter (to keep track of speed)
repeat forever!
It is a DockerCoin miner! π°π³π¦π’
No, you can't buy coffee with DockerCoins
How DockerCoins works:
generate a few random bytes
hash these bytes
increment a counter (to keep track of speed)
repeat forever!
DockerCoins is not a cryptocurrency
(the only common points are "randomness," "hashing," and "coins" in the name)
DockerCoins is made of 5 services:
rng
= web service generating random bytes
hasher
= web service computing hash of POSTed data
worker
= background process calling rng
and hasher
webui
= web interface to watch progress
redis
= data store (holds a counter updated by worker
)
These 5 services are visible in the application's Compose file, docker-compose.yml
worker
invokes web service rng
to generate random bytes
worker
invokes web service hasher
to hash these bytes
worker
does this in an infinite loop
every second, worker
updates redis
to indicate how many loops were done
webui
queries redis
, and computes and exposes "hashing speed" in our browser
(See diagram on next slide!)
How does each service find out the address of the other ones?
How does each service find out the address of the other ones?
We do not hard-code IP addresses in the code
We do not hard-code FQDNs in the code, either
We just connect to a service name, and container-magic does the rest
(And by container-magic, we mean "a crafty, dynamic, embedded DNS server")
worker/worker.py
redis = Redis("redis")def get_random_bytes(): r = requests.get("http://rng/32") return r.contentdef hash_bytes(data): r = requests.post("http://hasher/", data=data, headers={"Content-Type": "application/octet-stream"})
(Full source code available here)
Containers can have network aliases (resolvable through DNS)
Compose file version 2+ makes each container reachable through its service name
Compose file version 1 required "links" sections to accomplish this
Network aliases are automatically namespaced
you can have multiple apps declaring and using a service named database
containers in the blue app will resolve database
to the IP of the blue database
containers in the green app will resolve database
to the IP of the green database
You can check the GitHub repository with all the materials of this workshop:
https://github.com/jpetazzo/container.training
The application is in the dockercoins subdirectory
The Compose file (docker-compose.yml) lists all 5 services
redis
is using an official image from the Docker Hub
hasher
, rng
, worker
, webui
are each built from a Dockerfile
Each service's Dockerfile and source code is in its own directory
(hasher
is in the hasher directory,
rng
is in the rng
directory, etc.)
This is relevant only if you have used Compose before 2016...
Compose 1.6 introduced support for a new Compose file format (aka "v2")
Services are no longer at the top level, but under a services
section
There has to be a version
key at the top level, with value "2"
(as a string, not an integer)
Containers are placed on a dedicated network, making links unnecessary
There are other minor differences, but upgrade is easy and straightforward
On the left-hand side, the "rainbow strip" shows the container names
On the right-hand side, we see the output of our containers
We can see the worker
service making requests to rng
and hasher
For rng
and hasher
, we see HTTP access logs
"Logs are exciting and fun!" (No-one, ever)
The webui
container exposes a web dashboard; let's view it
With a web browser, connect to node1
on port 8000
Remember: the nodeX
aliases are valid only on the nodes themselves
In your browser, you need to enter the IP address of your node
A drawing area should show up, and after a few seconds, a blue graph will appear.
It looks like the speed is approximately 4 hashes/second
Or more precisely: 4 hashes/second, with regular dips down to zero
Why?
It looks like the speed is approximately 4 hashes/second
Or more precisely: 4 hashes/second, with regular dips down to zero
Why?
The app actually has a constant, steady speed: 3.33 hashes/second
(which corresponds to 1 hash every 0.3 seconds, for reasons)
Yes, and?
The worker doesn't update the counter after every loop, but up to once per second
The speed is computed by the browser, checking the counter about once per second
Between two consecutive updates, the counter will increase either by 4, or by 0
The perceived speed will therefore be 4 - 4 - 4 - 0 - 4 - 4 - 0 etc.
What can we conclude from this?
The worker doesn't update the counter after every loop, but up to once per second
The speed is computed by the browser, checking the counter about once per second
Between two consecutive updates, the counter will increase either by 4, or by 0
The perceived speed will therefore be 4 - 4 - 4 - 0 - 4 - 4 - 0 etc.
What can we conclude from this?
If we interrupt Compose (with ^C
), it will politely ask the Docker Engine to stop the app
The Docker Engine will send a TERM
signal to the containers
If the containers do not exit in a timely manner, the Engine sends a KILL
signal
^C
If we interrupt Compose (with ^C
), it will politely ask the Docker Engine to stop the app
The Docker Engine will send a TERM
signal to the containers
If the containers do not exit in a timely manner, the Engine sends a KILL
signal
^C
Some containers exit immediately, others take longer.
The containers that do not handle SIGTERM
end up being killed after a 10s timeout. If we are very impatient, we can hit ^C
a second time!
docker-compose down
Deploying our LKE cluster
(automatically generated title slide)
If we wanted to deploy Kubernetes manually, what would we need to do?
(not that I recommend doing that...)
Control plane (etcd, API server, scheduler, controllers)
Nodes (VMs with a container engine + the Kubelet agent; CNI setup)
High availability (etcd clustering, API load balancer)
Security (CA and TLS certificates everywhere)
Cloud integration (to provision LoadBalancer services, storage...)
And that's just to get a basic cluster!
The best way to deploy Kubernetes is to get someone else to do it for us.
(Me, ever since I've been working with Kubernetes)
Cloud provider runs the control plane
(including etcd, API load balancer, TLS setup, cloud integration)
We run nodes
(the cloud provider generally gives us an easy way to provision them)
Get started in minutes
We're going to use Linode Kubernetes Engine
With the web console:
Pick the region of your choice
Pick the latest available Kubernetes version
Pick 3 nodes with 8 GB of RAM
Click! β¨
Wait a few minutes... βοΈ
Download the kubeconfig file πΎ
View available regions with linode-cli regions list
View available server types with linode-cli linodes types
View available Kubernetes versions with linode-cli lke versions-list
Create cluster:
linode-cli lke cluster-create --label=hello-lke --region=us-east \ --k8s_version=1.20 --node_pools.type=g6-standard-4 --node_pools.count=3
Note the cluster ID (e.g.: 12345)
Download the kubeconfig file:
linode-cli lke kubeconfig-view 12345 --text --no-headers | base64 -d
All the Kubernetes tools (kubectl
, but also helm
etc) use the same config file
That file is (by default) $HOME/.kube/config
It can hold multiple cluster definitions (or contexts)
Or, we can have multiple config files and switch between them:
by adding the --kubeconfig
flag each time we invoke a tool (π)
or by setting the KUBECONFIG
environment variable (βΊοΈ)
Option 1:
move the kubeconfig file to e.g. ~/.kube/config.lke
set the environment variable: export KUBECONFIG=~/.kube/config.lke
Option 2:
directly move the kubeconfig file to ~/.kube/config
do not do that if you already have a file there!
Option 3:
Assuming that we want to merge ~/.kube/config
and ~/.kube/config.lke
...
Move our existing kubeconfig file:
cp ~/.kube/config ~/.kube/config.old
Merge both files:
KUBECONFIG=~/.kube/config.old:~/.kube/config.lke kubectl config \ view --raw > ~/.kube/config
Check that everything is there:
kubectl config get-contexts
Let's check if our control plane is available:
kubectl get services
β This should show the kubernetes
ClusterIP
service
Look for our nodes:
kubectl get nodes
β This should show 3 nodes (or whatever amount we picked earlier)
If the nodes aren't visible yet, give them a minute to join the cluster
Quick Kubernetes review
(automatically generated title slide)
Let's deploy a simple HTTP server
And expose it to the outside world!
Feel free to skip this section if you're familiar with Kubernetes
On Kubernetes, one doesn't simply run a container
We need to create a "Pod"
A Pod will be a group of containers running together
(often, it will be a group of one container)
We can create a standalone Pod, but generally, we'll use a controller
(for instance: Deployment, Replica Set, Daemon Set, Job, Stateful Set...)
The controller will take care of scaling and recreating the Pod if needed
(note that within a Pod, containers can also be restarted automatically if needed)
We're going to use one of the most common controllers: a Deployment
Deployments...
can be scaled (will create the requested number of Pods)
will recreate Pods if e.g. they get evicted or their Node is down
handle rolling updates
Deployments actually delegate a lot of these tasks to Replica Sets
We will generally have the following hierarchy:
Deployment β Replica Set β Pod
Without further ado:
kubectl create deployment web --image=nginx
Check what happened:
kubectl get all
Wait until the NGINX Pod is "Running"!
Note: kubectl create deployment
is great when getting started...
... But later, we will probably write YAML instead!
We need to create a Service
We can use kubectl expose
for that
(but, again, we will probably use YAML later!)
For internal use, we can use the default Service type, ClusterIP:
kubectl expose deployment web --port=80
For external use, we can use a Service of type LoadBalancer:
kubectl expose deployment web --port=80 --type=LoadBalancer
We can kubectl delete service web
and recreate it
Or, kubectl edit service web
and dive into the YAML
Or, kubectl patch service web --patch '{"spec": {"type": "LoadBalancer"}}'
... These are just a few "classic" methods; there are many ways to do this!
Can we check exactly what's going on when the Pod is created?
Option 1: watch kubectl get all
Option 2: kubectl get pods --watch --output-watch-events
tail -f
)Let's delete our Deployment:
kubectl delete deployment web
Watch Pod updates:
kubectl get pods --watch --output-watch-events
Recreate the Deployment and see what Pods do:
kubectl create deployment web --image=nginx
Our Service still works even though we deleted and re-created the Deployment
It wouldn't have worked while the Deployment was deleted, though
A Service is a stable endpoint
:T: Warming up with a quick Kubernetes review
:Q: In Kubernetes, what is a Pod? :A: βοΈA basic unit of scaling that can contain one or more containers :A: An abstraction for an application and its dependencies :A: It's just a fancy name for "container" but they're the same :A: A group of cluster nodes used for scheduling purposes
:Q: In Kubernetes, what is a Replica Set? :A: βοΈA controller used to create one or multiple identical Pods :A: A numeric parameter in a Pod specification, used to scale that Pod :A: A group of containers running on the same node :A: A group of containers running on different nodes
:Q: In Kubernetes, what is a Deployment? :A: βοΈA controller that can manage Replica Sets corresponding to different configurations :A: A manifest telling Kubernetes how to deploy an app and its dependencies :A: A list of instructions executed in a container to configure that container :A: A basic unit of work for the Kubernetes scheduler
They say, "a picture is worth one thousand words."
The following 19 slides show what really happens when we run:
kubectl create deployment web --image=nginx
Accessing internal services
(automatically generated title slide)
How can we temporarily access a service without exposing it to everyone?
kubectl proxy
: gives us access to the API, which includes a proxy for HTTP resources
kubectl port-forward
: allows forwarding of TCP ports to arbitrary pods, services, ...
kubectl proxy
in theoryRunning kubectl proxy
gives us access to the entire Kubernetes API
The API includes routes to proxy HTTP traffic
These routes look like the following:
/api/v1/namespaces/<namespace>/services/<service>/proxy
We just add the URI to the end of the request, for instance:
/api/v1/namespaces/<namespace>/services/<service>/proxy/index.html
We can access services
and pods
this way
kubectl proxy
in practiceweb
service through kubectl proxy
Run an API proxy in the background:
kubectl proxy &
Access the web
service:
curl localhost:8001/api/v1/namespaces/default/services/web/proxy/
Terminate the proxy:
kill %1
kubectl port-forward
in theoryWhat if we want to access a TCP service?
We can use kubectl port-forward
instead
It will create a TCP relay to forward connections to a specific port
(of a pod, service, deployment...)
The syntax is:
kubectl port-forward service/name_of_service local_port:remote_port
If only one port number is specified, it is used for both local and remote ports
kubectl port-forward
in practiceForward connections from local port 1234 to remote port 80:
kubectl port-forward svc/web 1234:80 &
Connect to the NGINX server:
curl localhost:1234
kill %1
:EN:- Securely accessing internal services :FR:- Accès sécurisé aux services internes
:T: Accessing internal services from our local machine
:Q: What's the advantage of "kubectl port-forward" compared to a NodePort? :A: It can forward arbitrary protocols :A: It doesn't require Kubernetes API credentials :A: It offers deterministic load balancing (instead of random) :A: βοΈIt doesn't expose the service to the public
:Q: What's the security concept behind "kubectl port-forward"? :A: βοΈWe authenticate with the Kubernetes API, and it forwards connections on our behalf :A: It detects our source IP address, and only allows connections coming from it :A: It uses end-to-end mTLS (mutual TLS) to authenticate our connections :A: There is no security (as long as it's running, anyone can connect from anywhere)
DNS, Ingress, Metrics
(automatically generated title slide)
We got a basic app up and running
We accessed it over a raw IP address
Can we do better?
(i.e. access it with a domain name!)
How much resources is it using?
We'd like to associate a fancy name to that LoadBalancer Service
(e.g. nginx.cloudnative.party
β A.B.C.D
)
option 1: manually add a DNS record
option 2: find a way to create DNS records automatically
We will install ExternalDNS to automate DNS records creatoin
ExternalDNS supports Linode DNS and dozens of other providers
What if we have multiple web services to expose?
We could create one LoadBalancer Service for each of them
This would create a lot of cloud load balancers
(and they typically incur a cost, even if it's a small one)
Instead, we can use an Ingress Controller
Ingress Controller = HTTP load balancer / reverse proxy
Put all our HTTP services behind a single LoadBalancer Service
Can also do fancy "content-based" routing (using headers, request path...)
We will install Traefik as our Ingress Controller
How much resources are we using right now?
When will we need to scale up our cluster?
We need metrics!
We're going to install the metrics server
It's a very basic metrics system
(no retention, no graphs, no alerting...)
But it's lightweight, and it is used internally by Kubernetes for autoscaling
We're going to install all these components
Very often, things can be installed with a simple YAML file
Very often, that YAML file needs to be customized a little bit
(add command-line parameters, provide API tokens...)
Instead, we're going to use Helm charts
Helm charts give us a way to customize what we deploy
Helm can also keep track of what we install
(for easier uninstall and updates)
Managing stacks with Helm
(automatically generated title slide)
Helm is a (kind of!) package manager for Kubernetes
We can use it to:
find existing packages (called "charts") created by other folks
install these packages, configuring them for our particular setup
package our own things (for distribution or for internal use)
manage the lifecycle of these installs (rollback to previous version etc.)
It's a "CNCF graduate project", indicating a certain level of maturity
(more on that later)
kubectl run
to YAMLWe can create resources with one-line commands
(kubectl run
, kubectl createa deployment
, kubectl expose
...)
We can also create resources by loading YAML files
(with kubectl apply -f
, kubectl create -f
...)
There can be multiple resources in a single YAML files
(making them convenient to deploy entire stacks)
However, these YAML bundles often need to be customized
(e.g.: number of replicas, image version to use, features to enable...)
Very often, after putting together our first app.yaml
, we end up with:
app-prod.yaml
app-staging.yaml
app-dev.yaml
instructions indicating to users "please tweak this and that in the YAML"
That's where using something like CUE, Kustomize, or Helm can help!
Now we can do something like this:
helm install app ... --set this.parameter=that.value
With Helm, we create "charts"
These charts can be used internally or distributed publicly
Public charts can be indexed through the Artifact Hub
This gives us a way to find and install other folks' charts
Helm also gives us ways to manage the lifecycle of what we install:
keep track of what we have installed
upgrade versions, change parameters, roll back, uninstall
Furthermore, even if it's not "the" standard, it's definitely "a" standard!
On April 30th 2020, Helm was the 10th project to graduate within the CNCF
π
(alongside Containerd, Prometheus, and Kubernetes itself)
This is an acknowledgement by the CNCF for projects that
demonstrate thriving adoption, an open governance process,
and a strong commitment to community, sustainability, and inclusivity.
See CNCF announcement and Helm announcement
helm
is a CLI tool
It is used to find, install, upgrade charts
A chart is an archive containing templatized YAML bundles
Charts are versioned
Charts can be stored on private or public repositories
A package (deb, rpm...) contains binaries, libraries, etc.
A chart contains YAML manifests
(the binaries, libraries, etc. are in the images referenced by the chart)
On most distributions, a package can only be installed once
(installing another version replaces the installed one)
A chart can be installed multiple times
Each installation is called a release
This allows to install e.g. 10 instances of MongoDB
(with potentially different versions and configurations)
But, on my Debian system, I have Python 2 and Python 3.
Also, I have multiple versions of the Postgres database engine!
Yes!
But they have different package names:
python2.7
, python3.8
postgresql-10
, postgresql-11
Good to know: the Postgres package in Debian includes
provisions to deploy multiple Postgres servers on the
same system, but it's an exception (and it's a lot of
work done by the package maintainer, not by the dpkg
or apt
tools).
Helm 3 was released November 13, 2019
Charts remain compatible between Helm 2 and Helm 3
The CLI is very similar (with minor changes to some commands)
The main difference is that Helm 2 uses tiller
, a server-side component
Helm 3 doesn't use tiller
at all, making it simpler (yay!)
tiller
With Helm 3:
the helm
CLI communicates directly with the Kubernetes API
it creates resources (deployments, services...) with our credentials
With Helm 2:
the helm
CLI communicates with tiller
, telling tiller
what to do
tiller
then communicates with the Kubernetes API, using its own credentials
This indirect model caused significant permissions headaches
(tiller
required very broad permissions to function)
tiller
was removed in Helm 3 to simplify the security aspects
helm
CLI is not installed in your environment, install itCheck if helm
is installed:
helm
If it's not installed, run the following command:
curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get-helm-3 \| bash
(To install Helm 2, replace get-helm-3
with get
.)
We need to install Tiller and give it some permissions
Tiller is composed of a service and a deployment in the kube-system
namespace
They can be managed (installed, upgraded...) with the helm
CLI
helm init
At the end of the install process, you will see:
Happy Helming!
Tiller needs permissions to create Kubernetes resources
In a more realistic deployment, you might create per-user or per-team service accounts, roles, and role bindings
cluster-admin
role to kube-system:default
service account:kubectl create clusterrolebinding add-on-cluster-admin \ --clusterrole=cluster-admin --serviceaccount=kube-system:default
(Defining the exact roles and permissions on your cluster requires a deeper knowledge of Kubernetes' RBAC model. The command above is fine for personal and development clusters.)
A repository (or repo in short) is a collection of charts
It's just a bunch of files
(they can be hosted by a static HTTP server, or on a local directory)
We can add "repos" to Helm, giving them a nickname
The nickname is used when referring to charts on that repo
(for instance, if we try to install hello/world
, that
means the chart world
on the repo hello
; and that repo
hello
might be something like https://blahblah.hello.io/charts/)
Helm 2 came with one pre-configured repo, the "stable" repo
(located at https://charts.helm.sh/stable)
Helm 3 doesn't have any pre-configured repo
The "stable" repo mentioned above is now being deprecated
The new approach is to have fully decentralized repos
Repos can be indexed in the Artifact Hub
(which supersedes the Helm Hub)
Go to the Artifact Hub (https://artifacthub.io)
Or use helm search hub ...
from the CLI
Let's try to find a Helm chart for something called "OWASP Juice Shop"!
(it is a famous demo app used in security challenges)
helm search hub <keyword>
Look for the OWASP Juice Shop app:
helm search hub owasp juice
Since the URLs are truncated, try with the YAML output:
helm search hub owasp juice -o yaml
Then go to β https://artifacthub.io/packages/helm/seccurecodebox/juice-shop
Go to https://artifacthub.io/
In the search box on top, enter "owasp juice"
Click on the "juice-shop" result (not "multi-juicer" or "juicy-ctf")
First, add the repository for that chart:
helm repo add juice https://charts.securecodebox.io
Then, install the chart:
helm install my-juice-shop juice/juice-shop
Note: it is also possible to install directly a chart, with --repo https://...
"Installing a chart" means creating a release
In the previous exemple, the release was named "my-juice-shop"
We can also use --generate-name
to ask Helm to generate a name for us
List the releases:
helm list
Check that we have a my-juice-shop-...
Pod up and running:
kubectl get pods
Helm 2 doesn't have support for the Helm Hub
The helm search
command only takes a search string argument
(e.g. helm search juice-shop
)
With Helm 2, the name is optional:
helm install juice/juice-shop
will automatically generate a name
helm install --name my-juice-shop juice/juice-shop
will specify a name
This specific chart labels all its resources with a release
label
We can use a selector to see these resources
kubectl get all --selector=app.kubernetes.io/instance=my-juice-shop
Note: this label wasn't added automatically by Helm.
It is defined in that chart. In other words, not all charts will provide this label.
By default, juice/juice-shop
creates a service of type ClusterIP
We would like to change that to a NodePort
We could use kubectl edit service my-juice-shop
, but ...
... our changes would get overwritten next time we update that chart!
Instead, we are going to set a value
Values are parameters that the chart can use to change its behavior
Values have default values
Each chart is free to define its own values and their defaults
helm show
or helm inspect
Look at the README for the app:
helm show readme juice/juice-shop
Look at the values and their defaults:
helm show values juice/juice-shop
The values
may or may not have useful comments.
The readme
may or may not have (accurate) explanations for the values.
(If we're unlucky, there won't be any indication about how to use the values!)
Values can be set when installing a chart, or when upgrading it
We are going to update my-juice-shop
to change the type of the service
my-juice-shop
:helm upgrade my-juice-shop juice/juice-shop --set service.type=NodePort
Note that we have to specify the chart that we use (juice/my-juice-shop
),
even if we just want to update some values.
We can set multiple values. If we want to set many values, we can use -f
/--values
and pass a YAML file with all the values.
All unspecified values will take the default values defined in the chart.
Check the node port allocated to the service:
kubectl get service my-juice-shopPORT=$(kubectl get service my-juice-shop -o jsonpath={..nodePort})
Connect to it:
curl localhost:$PORT/
:EN:- Helm concepts :EN:- Installing software with Helm :EN:- Helm 2, Helm 3, and the Helm Hub
:FR:- Fonctionnement gΓ©nΓ©ral de Helm :FR:- Installer des composants via Helm :FR:- Helm 2, Helm 3, et le Helm Hub
:T: Getting started with Helm and its concepts
:Q: Which comparison is the most adequate? :A: Helm is a firewall, charts are access lists :A: βοΈHelm is a package manager, charts are packages :A: Helm is an artefact repository, charts are artefacts :A: Helm is a CI/CD platform, charts are CI/CD pipelines
:Q: What's required to distribute a Helm chart? :A: A Helm commercial license :A: A Docker registry :A: An account on the Helm Hub :A: βοΈAn HTTP server
(automatically generated title slide)
ExternalDNS will automatically create DNS records from Kubernetes resources
Services (with the annotation external-dns.alpha.kubernetes.io/hostname
)
Ingresses (automatically)
It requires a domain name (obviously)
... And that domain name should be configurable through an API
As of April 2021, it supports a few dozens of providers
We're going to use Linode DNS
We need a domain name
(if you need a cheap one, look e.g. at GANDI; there are many options below $10)
That domain name should be configured to point to Linode DNS servers
(ns1.linode.com to ns5.linode.com)
We need to generate a Linode API token with DNS API access
Pro-tip: reduce the default TTL of the domain to 5 minutes!
The ExternalDNS documentation has a tutorial for Linode
... It's basically a lot of YAML!
That's where using a Helm chart will be very helpful
There are a few ExternalDNS charts available out there
We will use the one from Bitnami
(these folks maintain a lot of great Helm charts!)
We will install each chart in its own namespace
(this is not mandatory, but it helps to see what belongs to what)
We will use helm upgrade --install
instead of helm install
(that way, if we want to change something, we can just re-run the command)
We will use the --create-namespace
and --namespace ...
options
To keep things boring and predictible, if we are installing chart xyz
:
we will install it in namespace xyz
we will name the release xyz
as well
First, let's add the Bitnami repo:
helm repo add bitnami https://charts.bitnami.com/bitnami
Then, install ExternalDNS:
LINODE_API_TOKEN=1234abcd...6789helm upgrade --install external-dns bitnami/external-dns \ --namespace external-dns --create-namespace \ --set provider=linode \ --set linode.apiToken=$LINODE_API_TOKEN
(Make sure to update your API token above!)
Let's annotate our NGINX service to expose it with a DNS record:
kubectl annotate service web \ external-dns.alpha.kubernetes.io/hostname=nginx.cloudnative.party
(make sure to use your domain name above, otherwise that won't work!)
Check ExternalDNS logs:
kubectl logs -n external-dns -l app.kubernetes.io/name=external-dns
It might take a few minutes for ExternalDNS to start, patience!
Then try to access nginx.cloudnative.party
(or whatever domain you picked)
Installing Traefik
(automatically generated title slide)
Traefik is going to be our Ingress Controller
Let's install it with a Helm chart, in its own namespace
First, let's add the Traefik chart repository:
helm repo add traefik https://helm.traefik.io/traefik
Then, install the chart:
helm upgrade --install traefik traefik/traefik \ --create-namespace --namespace traefik \ --set "ports.websecure.tls.enabled=true"
(that option that we added enables HTTPS, it will be useful later!)
Let's create an Ingress resource!
If we're using Kubernetes 1.20 or later, we can simply do this:
kubectl create ingress web \ --rule=ingress-is-fun.cloudnative.party/*=web:80
(make sure to update and use your own domain)
Check that the Ingress was correctly created:
kubectl get ingresskubectl describe ingress
If we're using Kubernetes 1.19 or earlier, we'll need some YAML
kubectl apply -f- <<EOFapiVersion: networking.k8s.io/v1beta1kind: Ingressmetadata: name: webspec: rules: - host: ingress-is-fun.cloudnative.party http: paths: - path: / backend: serviceName: web servicePort: 80EOF
Note how we used the v1beta1
Ingress version on the previous YAML
(to be compatible with older Kubernetes versions)
This YAML will give you deprecation warnings on recent version of Kubernetes
(since the Ingress spec is now at version v1
)
Don't worry too much about the deprecation warnings
(on Kubernetes, deprecation happens over a long time window, typically 1 year)
You will have time to revisit and worry later! π
Try to connect to the Ingress host name
(in my example, http://ingress-is-fun.cloudnative.party/)
Normally, it doesn't work (yet) π€
Let's look at kubectl get ingress
again
ExternalDNS is trying to create records mapping HOSTS to ADDRESS
But the ADDRESS field is currently empty!
We need to tell Traefik to fill that ADDRESS field
There is a "magic" flag to tell Traefik to update the address status field
Let's update our Traefik install:
helm upgrade --install traefik traefik/traefik \ --create-namespace --namespace traefik \ --set "ports.websecure.tls.enabled=true" \ --set "providers.kubernetesIngress.publishedService.enabled=true"
Check the output of kubectl get ingress
(there should be an address now)
Check the logs of ExternalDNS
(there should be a mention of the new DNS record)
Try again to connect to the HTTP address
(now it should work)
Note that some of these operations might take a minute or two
(be patient!)
:T: Installing the Traefik Ingress Controller
:Q: What's the job of an Ingress Controller? :A: Prevent unauthorized access to Kubernetes services :A: Firewall inbound traffic on the Kubernetes API :A: βοΈHandle inbound HTTP traffic for Kubernetes services :A: Keep track of the location of Kubernetes operators
:Q: What happens when we create an "Ingress resource"? :A: A web service is automatically deployed and scaled on our cluster :A: Kubernetes starts tracking the location of our users :A: Traffic coming from the specified addresses will be allowed :A: βοΈA load balancer is configured with HTTP traffic rules
Installing metrics-server
(automatically generated title slide)
We've installed a few things on our cluster so far
How much resources (CPU, RAM) are we using?
We need metrics!
If metrics-server is installed, we can get Nodes metrics like this:
kubectl top nodes
At the moment, this should show us error: Metrics API not available
How do we fix this?
We could use a SAAS like Datadog, New Relic...
We could use a self-hosted solution like Prometheus
Or we could use metrics-server
What's special about metrics-server?
Cons:
no data retention (no history data, just instant numbers)
only CPU and RAM of nodes and pods (no disk or network usage or I/O...)
Pros:
very lightweight
doesn't require storage
used by Kubernetes autoscaling
We may install something fancier later
(think: Prometheus with Grafana)
But metrics-server will work in minutes
It will barely use resources on our cluster
It's required for autoscaling anyway
It runs a single Pod
That Pod will fetch metrics from all our Nodes
It will expose them through the Kubernetes API agregation layer
(we won't say much more about that agregation layer; that's fairly advanced stuff!)
In a lot of places, this is done with a little bit of custom YAML
(derived from the official installation instructions)
We're going to use Helm one more time:
helm upgrade --install metrics-server bitnami/metrics-server \ --create-namespace --namespace metrics-server \ --set apiService.create=true \ --set extraArgs.kubelet-insecure-tls=true \ --set extraArgs.kubelet-preferred-address-types=InternalIP
What are these options for?
apiService.create=true
register metrics-server
with the Kubernetes agregation layer
(create an entry that will show up in kubectl get apiservices
)
extraArgs.kubelet-insecure-tls=true
when connecting to nodes to collect their metrics, don't check kubelet TLS certs
(because most kubelet certs include the node name, but not its IP address)
extraArgs.kubelet-preferred-address-types=InternalIP
when connecting to nodes, use their internal IP address instead of node name
(because the latter requires an internal DNS, which is rarely configured)
After a minute or two, metrics-server should be up
We should now be able to check Nodes resource usage:
kubectl top nodes
And Pods resource usage, too:
kubectl top pods --all-namespaces
The RAM usage that we see should correspond more or less to the Resident Set Size
Our pods also need some extra space for buffers, caches...
Do not aim for 100% memory usage!
Some more realistic targets:
50% (for workloads with disk I/O and leveraging caching)
90% (on very big nodes with mostly CPU-bound workloads)
75% (anywhere in between!)
Prometheus and Grafana
(automatically generated title slide)
What if we want metrics retention, view graphs, trends?
A very popular combo is Prometheus+Grafana:
Prometheus as the "metrics engine"
Grafana to display comprehensive dashboards
Prometheus also has an alert-manager component to trigger alerts
(we won't talk about that one)
A complete metrics stack needs at least:
the Prometheus server (collects metrics and stores them efficiently)
a collection of exporters (exposing metrics to Prometheus)
Grafana
a collection of Grafana dashboards (building them from scratch is tedious)
The Helm chart kube-prometheus-stack
combines all these elements
... So we're going to use it to deploy our metrics stack!
kube-prometheus-stack
Let's install that stack directly from its repo
(without doing helm repo add
first)
Otherwise, keep the same naming strategy:
helm upgrade --install kube-prometheus-stack kube-prometheus-stack \ --namespace kube-prometheus-stack --create-namespace \ --repo https://prometheus-community.github.io/helm-charts
This will take a minute...
Then check what was installed:
kubectl get all --namespace kube-prometheus-stack
Let's create an Ingress for Grafana
kubectl create ingress --namespace kube-prometheus-stack grafana \ --rule=grafana.cloudnative.party/*=kube-prometheus-stack-grafana:80
(as usual, make sure to use your domain name above)
Connect to Grafana
(remember that the DNS record might take a few minutes to come up)
What could the login and password be?
Let's look at the Secrets available in the namespace:
kubectl get secrets --namespace kube-prometheus-stack
There is a kube-prometheus-stack-grafana
that looks promising!
Decode the Secret:
kubectl get secret --namespace kube-prometheus-stack \ kube-prometheus-stack-grafana -o json | jq '.data | map_values(@base64d)'
If you don't have the jq
tool mentioned above, don't worry...
What could the login and password be?
Let's look at the Secrets available in the namespace:
kubectl get secrets --namespace kube-prometheus-stack
There is a kube-prometheus-stack-grafana
that looks promising!
Decode the Secret:
kubectl get secret --namespace kube-prometheus-stack \ kube-prometheus-stack-grafana -o json | jq '.data | map_values(@base64d)'
If you don't have the jq
tool mentioned above, don't worry...
The login/password is hardcoded to admin
/prom-operator
π¬
Once logged in, click on the "Dashboards" icon on the left
(it's the one that looks like four squares)
Then click on the "Manage" entry
Then click on "Kubernetes / Compute Resources / Cluster"
This gives us a breakdown of resource usage by Namespace
Feel free to explore the other dashboards!
:T: Observing our cluster with Prometheus and Grafana
:Q: What's the relationship between Prometheus and Grafana? :A: Prometheus collects and graphs metrics; Grafana sends alerts :A: βοΈPrometheus collects metrics; Grafana displays them on dashboards :A: Prometheus collects and graphs metrics; Grafana is its configuration interface :A: Grafana collects and graphs metrics; Prometheus sends alerts
cert-manager
(automatically generated title slide)
cert-managerΒΉ facilitates certificate signing through the Kubernetes API:
we create a Certificate object (that's a CRD)
cert-manager creates a private key
it signs that key ...
... or interacts with a certificate authority to obtain the signature
it stores the resulting key+cert in a Secret resource
These Secret resources can be used in many places (Ingress, mTLS, ...)
ΒΉAlways lower case, words separated with a dash; see the style guide
cert-manager can use multiple Issuers (another CRD), including:
self-signed
cert-manager acting as a CA
the ACME protocol (notably used by Let's Encrypt)
Multiple issuers can be configured simultaneously
Issuers can be available in a single namespace, or in the whole cluster
(then we use the ClusterIssuer CRD)
We will install cert-manager
We will create a ClusterIssuer to obtain certificates with Let's Encrypt
(this will involve setting up an Ingress Controller)
We will create a Certificate request
cert-manager will honor that request and create a TLS Secret
helm install cert-manager cert-manager \ --repo https://charts.jetstack.io \ --create-namespace --namespace cert-manager \ --set installCRDs=true
If you prefer to install with a single YAML file, that's fine too!
(see the documentation for instructions)
apiVersion: cert-manager.io/v1kind: ClusterIssuermetadata: name: letsencrypt-stagingspec: acme: # Remember to update this if you use this manifest to obtain real certificates :) email: hello@example.com server: https://acme-staging-v02.api.letsencrypt.org/directory # To use the production environment, use the following line instead: #server: https://acme-v02.api.letsencrypt.org/directory privateKeySecretRef: name: issuer-letsencrypt-staging solvers: - http01: ingress: class: traefik
Download the file k8s/cm-clusterissuer.yaml
(or copy-paste from the previous slide)
kubectl apply cm-clusterissuer.yaml
apiVersion: cert-manager.io/v1kind: Certificatemetadata: name: xyz.A.B.C.D.nip.iospec: secretName: xyz.A.B.C.D.nip.io dnsNames: - xyz.A.B.C.D.nip.io issuerRef: name: letsencrypt-staging kind: ClusterIssuer
The name
, secretName
, and dnsNames
don't have to match
There can be multiple dnsNames
The issuerRef
must match the ClusterIssuer that we created earlier
Download the file k8s/cm-certificate.yaml
(or copy-paste from the previous slide)
Edit the Certificate to update the domain name
(make sure to replace A.B.C.D with the IP address of one of your nodes!)
Create the Certificate:
kubectl apply -f cm-certificate.yaml
cert-manager will create:
the secret key
a Pod, a Service, and an Ingress to complete the HTTP challenge
then it waits for the challenge to complete
kubectl get pods,services,ingresses \ --selector=acme.cert-manager.io/http01-solver=true
The CA (in this case, Let's Encrypt) will fetch a particular URL:
http://<our-domain>/.well-known/acme-challenge/<token>
kubectl describe ingress --selector=acme.cert-manager.io/http01-solver=true
A little bit later, we will have a kubernetes.io/tls
Secret:
kubectl get secrets
Note that this might take a few minutes, because of the DNS integration!
For bonus points, try to use the secret in an Ingress!
This is what the manifest would look like:
apiVersion: networking.k8s.io/v1beta1kind: Ingressmetadata: name: xyzspec: tls: - secretName: xyz.A.B.C.D.nip.io hosts: - xyz.A.B.C.D.nip.io rules: ...
It is also possible to annotate Ingress resources for cert-manager
If we annotate an Ingress resource with cert-manager.io/cluster-issuer=xxx
:
cert-manager will detect that annotation
it will obtain a certificate using the specified ClusterIssuer (xxx
)
it will store the key and certificate in the specified Secret
Note: the Ingress still needs the tls
section with secretName
and hosts
:EN:- Obtaining certificates with cert-manager :FR:- Obtenir des certificats avec cert-manager
:T: Obtaining TLS certificates with cert-manager
CI/CD with GitLab
(automatically generated title slide)
In this section, we will see how to set up a CI/CD pipeline with GitLab
(using a "self-hosted" GitLab; i.e. running on our Kubernetes cluster)
The big picture:
each time we push code to GitLab, it will be deployed in a staging environment
each time we push the production
tag, it will be deployed in production
We'll use GitLab here as an exemple, but there are many other options
(e.g. some combination of Argo, Harbor, Tekton ...)
There are also hosted options
(e.g. GitHub Actions and many others)
We'll use a specific pipeline and workflow, but it's purely arbitrary
(treat it as a source of inspiration, not a model to be copied!)
Push code to GitLab's git server
GitLab notices the .gitlab-ci.yml
file, which defines our pipeline
Our pipeline can have multiple stages executed sequentially
(e.g. lint, build, test, deploy ...)
Each stage can have multiple jobs executed in parallel
(e.g. build images in parallel)
Each job will be executed in an independent runner pod
Our repository holds source code, Dockerfiles, and a Helm chart
Lint stage will check the Helm chart validity
Build stage will build container images
(and push them to GitLab's integrated registry)
Deploy stage will deploy the Helm chart, using these images
Pushes to production
will deploy to "the" production namespace
Pushes to other tags/branches will deploy to a namespace created on the fly
We will discuss shortcomings and alternatives and the end of this chapter!
We need a lot of components to pull this off:
a domain name
a storage class
a TLS-capable ingress controller
the cert-manager operator
GitLab itself
the GitLab pipeline
Wow, why?!?
We need a container registry (obviously!)
Docker (and other container engines) require TLS on the registry
(with valid certificates)
A few options:
use a "real" TLS certificate (e.g. obtained with Let's Encrypt)
use a self-signed TLS certificate
communicate with the registry over localhost (TLS isn't required then)
When using self-signed certs, we need to either:
add the cert (or CA) to trusted certs
disable cert validation
This needs to be done on every client connecting to the registry:
CI/CD pipeline (building and pushing images)
container engine (deploying the images)
other tools (e.g. container security scanner)
It's doable, but it's a lot of hacks (especially when adding more tools!)
TLS is usually not required when the registry is on localhost
We could expose the registry e.g. on a NodePort
... And then tweak the CI/CD pipeline to use that instead
This is great when obtaining valid certs is difficult:
air-gapped or internal environments (that can't use Let's Encrypt)
no domain name available
Downside: the registry isn't easily or safely available from outside
(the NodePort
essentially defeats TLS)
nip.io
?We will use Let's Encrypt
Let's Encrypt has a quota of certificates per domain
(in 2020, that was 50 certificates per week per domain)
So if we all use nip.io
, we will probably run into that limit
But you can try and see if it works!
We will deploy GitLab with its official Helm chart
It will still require a bunch of parameters and customization
Brace!
helm repo add gitlab https://charts.gitlab.io/DOMAIN=cloudnative.partyISSUER=letsencrypt-productionhelm upgrade --install gitlab gitlab/gitlab \ --create-namespace --namespace gitlab \ --set global.hosts.domain=$DOMAIN \ --set certmanager.install=false \ --set nginx-ingress.enabled=false \ --set global.ingress.class=traefik \ --set global.ingress.provider=traefik \ --set global.ingress.configureCertmanager=false \ --set global.ingress.annotations."cert-manager\.io/cluster-issuer"=$ISSUER \ --set gitlab.webservice.ingress.tls.secretName=gitlab-gitlab-tls \ --set registry.ingress.tls.secretName=gitlab-registry-tls \ --set minio.ingress.tls.secretName=gitlab-minio-tls
π° Can we talk about all these parameters?
certmanager.install=false
do not install cert-manager, we already have it
nginx-ingress.enabled=false
do not install the NGINX ingress controller, we already have Traefik
global.ingress.class=traefik
, global.ingress.provider=traefik
these merely enable creation of Ingress resources
global.ingress.configureCertmanager=false
do not create a cert-manager Issuer or ClusterIssuer, we have ours
global.ingress.annotations."cert-manager\.io/cluster-issuer"=$ISSUER
this annotation tells cert-manager to automatically issue certs
gitlab.webservice.ingress.tls.secretName=gitlab-gitlab-tls
,
registry.ingress.tls.secretName=gitlab-registry-tls
,
minio.ingress.tls.secretName=gitlab-minio-tls
these annotations enable TLS in the Ingress controller
Let's watch what's happening in the GitLab namespace:
watch kubectl get all --namespace gitlab
We want to wait for all the Pods to be "Running" or "Completed"
This will take a few minutes (10-15 minutes for me)
Don't worry if you see Pods crashing and restarting
(it happens when they are waiting on a dependency which isn't up yet)
Symptom: Pods remain "Pending" or "ContainerCreating" for a while
Investigate these pods (with kubectl describe pod ...
)
Also look at events:
kubectl get events \ --field-selector=type=Warning --sort-by=metadata.creationTimestamp
Make sure your cluster is big enough
(I use 3 g6-standard-4
nodes)
First, let's check that we can connect to GitLab (with TLS):
https://gitlab.$DOMAIN
It's asking us for a login and password!
The login is root
, and the password is stored in a Secret:
kubectl get secrets --namespace=gitlab gitlab-gitlab-initial-root-password \ -o jsonpath={.data.password} | base64 -d
For simplicity, we're going to use that "root" user
(but later, you can create multiple users, teams, etc.)
First, let's add our SSH key
(top-right user menu β settings, then SSH keys on the left)
Then, create a project
(using the + menu next to the search bar on top)
Let's call it kubecoin
(you can change it, but you'll have to adjust Git paths later on)
This is the repository that we're going to use:
Let's clone that repository locally first:
git clone https://github.com/jpetazzo/kubecoin
Add our GitLab instance as a remote:
git remote add gitlab git@gitlab.$DOMAIN:root/kubecoin.git
Try to push:
git push -u gitlab
Normally, we get the following error:
port 22: Connection refused
Why? π€
Normally, we get the following error:
port 22: Connection refused
Why? π€
What does gitlab.$DOMAIN
point to?
Normally, we get the following error:
port 22: Connection refused
Why? π€
What does gitlab.$DOMAIN
point to?
Our Ingress Controller! (i.e. Traefik) π‘
Our Ingress Controller has nothing to do with port 22
So how do we solve this?
Whatever is on gitlab.$DOMAIN
needs to have the following "routing":
port 80 β GitLab web service
port 443 β GitLab web service, with TLS
port 22 β GitLab shell service
Currently, Traefik is managing gitlab.$DOMAIN
We are going to tell Traefik to:
accept connections on port 22
send them to GitLab
The technique that we are going to use is specific to Traefik
Other Ingress Controllers may or may not have similar features
When they have similar features, they will be enabled very differently
Let's reconfigure Traefik:
helm upgrade --install traefik traefik/traefik \ --create-namespace --namespace traefik \ --set "ports.websecure.tls.enabled=true" \ --set "providers.kubernetesIngress.publishedService.enabled=true" \ --set "ports.ssh.port=2222" \ --set "ports.ssh.exposedPort=22" \ --set "ports.ssh.expose=true" \ --set "ports.ssh.protocol=TCP"
This creates a new "port" on Traefik, called "ssh", listening on port 22
Internally, Traefik listens on port 2222 (for permission reasons)
Note: Traefik docs also call these ports "entrypoints"
(these entrypoints are totally unrelated to the ENTRYPOINT
in Dockerfiles)
What happens if we try to connect to that port 22 right now?
curl gitlab.$DOMAIN:22
We hit GitLab's web service!
We need to tell Traefik what to do with connections to that port 22
For that, we will create a "TCP route"
The following custom resource tells Traefik to route the ssh
port that we
created earlier, to the gitlab-gitlab-shell
service belonging to GitLab.
apiVersion: traefik.containo.us/v1alpha1kind: IngressRouteTCPmetadata: name: gitlab-shell namespace: gitlabspec: entryPoints: - ssh routes: - match: HostSNI(`*`) services: - name: gitlab-gitlab-shell port: 22
The HostSNI
wildcard is the magic option to define a "default route".
Since our manifest has backticks, we must pay attention to quoting:
kubectl apply -f- << "EOF"apiVersion: traefik.containo.us/v1alpha1kind: IngressRouteTCPmetadata: name: gitlab-shell namespace: gitlabspec: entryPoints: - ssh routes: - match: HostSNI(`*`) services: - name: gitlab-gitlab-shell port: 22EOF
Let's see what happens if we try port 22 now:
curl gitlab.$DOMAIN:22
This should tell us something like Received HTTP/0.9 when not allowed
(because we're no longer talking to an HTTP server, but to SSH!)
Try with SSH:
ssh git@gitlab.$DOMAIN
After accepting the key fingerprint, we should see Welcome to GitLab, @root!
Now we can try to push our repository again:
git push -u gitlab
Reload the project page in GitLab
We should see our repository!
Click on the CI/CD tab on the left
(the one with the shuttle / space rocket icon)
Our pipeline was detected...
But it failed π
Let's click on one of the failed jobs
This is a permission issue!
GitLab needs to do a few of things in our cluster:
create Pods to build our container images with BuildKit
create Namespaces to deploy staging and production versions of our app
create and update resources in these Namespaces
For the time being, we're going to grant broad permissions
(and we will revisit and discuss what to do later)
Let's give cluster-admin
permissions to the GitLab ServiceAccount:
kubectl create clusterrolebinding gitlab \ --clusterrole=cluster-admin --serviceaccount=gitlab:default
Then retry the CI/CD pipeline
The build steps will now succeed; but the deploy steps will fail
We need to set the REGISTRY_USER
and REGISTRY_PASSWORD
variables
Let's explain what this is about!
A registry access token is created for the duration of the CI/CD pipeline
(it is exposed through the $CI_JOB_TOKEN
environment variable)
This token gives access only to a specific repository in the registry
It is valid only during the execution of the CI/CD pipeline
We can (and we do!) use it to push images to the registry
We cannot use it to pull images when running in staging or production
(because Kubernetes might need to pull images after the token expires)
We need to create a separate read-only registry access token
Let's go to "Settings" (the cog wheel on the left) / "Access Tokens"
Create a token with read_registry
permission
Save the token name and the token value
Then go to "Settings" / "CI/CD"
In the "Variables" section, add two variables:
REGISTRY_USER
β token nameREGISTRY_PASSWORD
β token valueMake sure that they are not protected!
(otherwise, they won't be available in non-default tags and branches)
Go back to the CI/CD pipeline view, and hit "Retry"
The deploy stage should now work correctly! π
Let's have a look at the .gitlab-ci.yml file
We have multiple stages:
lint (currently doesn't do much, it's mostly as an example)
build (currently uses BuildKit)
deploy
"Deploy" behaves differently in staging and production
Let's investigate that!
In our pipeline, "production" means "a tag or branch named production
"
(see the except:
and only:
sections)
Everything else is "staging"
In "staging":
In "production":
GitLab will create Namespaces named gl-<user>-<project>-<hash>
At the end of the deployment, the web UI will be available at:
http://<user>-<project>-<githash>-gitlab.<domain>
The "production" Namespace will be <user>-<project>
And it will be available on its own domain as well:
http://<project>-<githash>-gitlab.<domain>
git tag -f production && git push -f --tags
Our CI/CD pipeline will deploy on the production URL
(http://<user>-<project>-gitlab.<domain>
)
It will do it only if that same git commit was pushed to staging first
(because the "production" pipeline skips the build phase)
There are many ways to build container images on Kubernetes
And they all suck Many of them have inconveniencing issues
Let's do a quick review!
Bind-mount the Docker socket
Docker-in-Docker in a pod
External build host
Kaniko
BuildKit / docker buildx
Our CI/CD workflow is just one of the many possibilities
It would be nice to add some actual unit or e2e tests
Map the production namespace to a "real" domain name
Automatically remove older staging environments
(see e.g. kube-janitor)
Deploy production to a separate cluster
Better segregate permissions
(don't give cluster-admin
to the GitLab pipeline)
"All-in-one" approach
(deploys its own Ingress, cert-manager, Prometheus, and much more)
I wanted to show you something flexible and customizable instead
But feel free to explore it now that we have shown the basics!
:EN:- CI/CD with GitLab :FR:- CI/CD avec GitLab
Hello! I'm JΓ©rΓ΄me Petazzoni
(@jpetazzo on Twitter)
I worked at Docker from ~2011 to 2018
I'm now doing consulting, training, etc. on Docker & Kubernetes
(check out container.training!)
I'll show you how to deploy a complete CI/CD pipeline on LKE!
(Linode Kubernetes Engine π)
Keyboard shortcuts
β, β, Pg Up, k | Go to previous slide |
β, β, Pg Dn, Space, j | Go to next slide |
Home | Go to first slide |
End | Go to last slide |
Number + Return | Go to specific slide |
b / m / f | Toggle blackout / mirrored / fullscreen mode |
c | Clone slideshow |
p | Toggle presenter mode |
t | Restart the presentation timer |
?, h | Toggle this help |
Esc | Back to slideshow |