Project

General

Profile

Actions

Task #9565

open

Select a CI/CD for deploying helm charts/docker containers etc.

Added by Nico Schottelius 3 months ago. Updated 2 months ago.

Status:
In Progress
Priority:
Normal
Target version:
-
Start date:
07/26/2021
Due date:
% Done:

0%

Estimated time:
PM Check date:

Description

  • Basically: git push && pipeline that does the rest
  • Input from your experiences is appreciated

Choices

Flux v2

  • Overall nice
  • Does not cover the build phase
  • Has nice multi cluster support

drone

  • Unclear on how / where to store the output
  • K8S support seems to be fragile

Jenkins

  • The "standard"
  • Very heavy (4GB+ memory)

Buildbot

  • Old
  • Static workers (easy to configure via k8s)
  • Seems to be fast and easy to setup
  • Python based
  • Website and documentation down as of 2021-08-08

GoCD

TL;DR: Does not even start in an IPv6 k8s cluster

Non-working installation:

helm upgrade --install --set server.service.type=ClusterIP,server.ingress.enabled=false gocd gocd/gocd

Gitlab

TL;DR: has a lot included, maybe too much

ArgoCD and(?) argoflow

  • Rather complicated / big ecosystem
  • Design to be cloud native
  • Dependencies nicely solved
    • in order or via DAG

Argo flow

  • Output is very S3 centered
    • We could use this, even though it seems overkill
    • This might be a practical requirement
    • Might be able to ignore this feature
  • Argo flow tries to access /var/run/docker.sock directly - which does not exist for crio based environments
    • MountVolume.SetUp failed for volume "docker-sock" : hostPath type check failed: /var/run/docker.sock is not a socket file

Flows

DNS Update

Questions:

  • Should we create a stand-alone zone repository?
    • Would be very small
    • Can only clone head/last commit
  • If using git pull inside the container, we need to pass along credentials
    • possible in a secret

Flow v1

  • We change a zone file in git and push it somewhere
  • A new helm chart is being created
  • (maybe in between) bump the chartversion field?
    • only if knot was able to run it?
  • The new helm chart is uploaded to the chartmuseum
  • The pods/services are notified about a new version
    • How?
      • Configmap change?
      • git pull?

Flow v2: pull from git repo

  • The helm chart is given a git repo (+possible secret)
  • The pod tries reloading every minute
    • if checkconf works: restart
    • else: reject
  • A webhook in gitea might be used to trigger the DNS server instances
    • Faster deploy
    • Question is where to, whether we have 1 hook per cluster, etc.

Disadvantage: need to build our own container (?)

  • In theory a custom container could do that in a pod

Flow v3: push pipeline

  • In theory we want every zone change to create a new version number
  • Actually we have this already with the git commit

Nothing to be done here.

Actions #1

Updated by Nico Schottelius 3 months ago

  • Description updated (diff)
Actions #2

Updated by Nico Schottelius 3 months ago

  • Description updated (diff)
Actions #3

Updated by Nico Schottelius 3 months ago

  • Description updated (diff)
Actions #4

Updated by Nico Schottelius 3 months ago

  • Description updated (diff)
Actions #5

Updated by Nico Schottelius 3 months ago

  • Description updated (diff)
Actions #6

Updated by Amal Elshihaby 3 months ago

I prefer using travisCI or circleCI, they are light and easy to maintained.
I think too that they works good with Kubernetes

Actions #7

Updated by Mondi Ravi 3 months ago

I do not have any preference -- I would probably go with Jenkins just because it is widely adopted.

GoCD looks promising also, though I've not used it personally.

Actions #8

Updated by Nico Schottelius 2 months ago

  • Project changed from 45 to Open Infrastructure
Actions #9

Updated by Nico Schottelius 2 months ago

  • Description updated (diff)
Actions #10

Updated by Nico Schottelius 2 months ago

  • Description updated (diff)
Actions #11

Updated by Nico Schottelius 2 months ago

  • Description updated (diff)

Argoflow notes

loop:

      dag:
        tasks:
          - name: print-message
            template: whalesay
            arguments:
              parameters:
                - name: message
                  value: "{{item}}" 
            withItems:
              - "hello world" 
              - "goodbye world" 

Sequence

     dag:
        tasks:
          - name: print-message
            template: whalesay
            arguments:
              parameters:
                - name: message
                  value: "{{item}}" 
            withSequence:
              count: 5

  • A steps template allows you to run a series of steps in sequence.
  • A suspend template allows you to automatically suspend a workflow, e.g. while waiting on manual approval, or while an external system does some work.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: exit-handler-
spec:
  entrypoint: main
  templates:
    - name: main
      dag:
        tasks:
          - name: a
            template: whalesay
            onExit: tidy-up

    - name: whalesay
      container:
        image: docker/whalesay

    - name: tidy-up
      container:
        image: docker/whalesay
        command: [ cowsay ]
        args: [ "tidy up!" ]

Parameters

  • Similar to helm values
    - name: main
      inputs:
        parameters:
          - name: message
      container:
        image: docker/whalesay
        command: [ cowsay ]
        args: [ "{{inputs.parameters.message}}" ]

Chaining in & out via a file:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: parameters-
spec:
  entrypoint: main
  templates:
    - name: main
      dag:
        tasks:
          - name: generate-parameter
            template: whalesay
          - name: consume-parameter
            template: print-message
            dependencies:
              - generate-parameter
            arguments:
              parameters:
                - name: message
                  value: "{{tasks.generate-parameter.outputs.parameters.hello-param}}" 

    - name: whalesay
      container:
        image: docker/whalesay
        command: [ sh, -c ]
        args: [ "echo -n hello world > /tmp/hello_world.txt" ]
      outputs:
        parameters:
          - name: hello-param
            valueFrom:
              path: /tmp/hello_world.txt

    - name: print-message
      inputs:
        parameters:
          - name: message
      container:
        image: docker/whalesay
        command: [ cowsay ]
        args: [ "{{inputs.parameters.message}}" ]

Workflowtemplate

  • Basically a workflow stored in k8s that can be reused

CronWorkflow

  • as the name says

Webhooks

  • Very easy to create w/ input
  • Using workflowtemplate
Actions #12

Updated by Nico Schottelius 2 months ago

  • Description updated (diff)
Actions #13

Updated by Nico Schottelius 2 months ago

GoCD test

helm upgrade --install --set server.service.type=ClusterIP,server.ingress.enabled=false gocd gocd/gocd

Hangs in creating

[16:00] nb3:generic% kubectl describe pods gocd-server-5b8fb6b58f-54qc8
Name:           gocd-server-5b8fb6b58f-54qc8
Namespace:      default
Priority:       0
Node:           server60/2a0a:e5c0:13:0:225:b3ff:fe20:3736
Start Time:     Sun, 08 Aug 2021 15:58:49 +0200
Labels:         app=gocd
                component=server
                pod-template-hash=5b8fb6b58f
                release=gocd
Annotations:    cni.projectcalico.org/podIP: 2a0a:e5c0:13:e1:ddc1:7d11:9a1f:95a5/128
                cni.projectcalico.org/podIPs: 2a0a:e5c0:13:e1:ddc1:7d11:9a1f:95a5/128
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  ReplicaSet/gocd-server-5b8fb6b58f
Containers:
  gocd-server:
    Container ID:   
    Image:          gocd/gocd-server:v21.2.0
    Image ID:       
    Port:           8153/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Liveness:       http-get http://:8153/go/api/v1/health delay=90s timeout=1s period=15s #success=1 #failure=10
    Readiness:      http-get http://:8153/go/api/v1/health delay=90s timeout=1s period=15s #success=1 #failure=10
    Environment:
      GOCD_PLUGIN_INSTALL_kubernetes-elastic-agents:        https://github.com/gocd/kubernetes-elastic-agents/releases/download/v3.7.1-230/kubernetes-elastic-agent-3.7.1-230.jar
      GOCD_PLUGIN_INSTALL_docker-registry-artifact-plugin:  https://github.com/gocd/docker-registry-artifact-plugin/releases/download/v1.1.0-104/docker-registry-artifact-plugin-1.1.0-104.jar
    Mounts:
      /docker-entrypoint.d from goserver-vol (rw,path="scripts")
      /godata from goserver-vol (rw,path="godata")
      /home/go from goserver-vol (rw,path="homego")
      /preconfigure_server.sh from config-vol (rw,path="preconfigure_server.sh")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2m9th (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config-vol:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      gocd
    Optional:  false
  goserver-vol:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  gocd-server
    ReadOnly:   false
  kube-api-access-2m9th:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  100s  default-scheduler  Successfully assigned default/gocd-server-5b8fb6b58f-54qc8 to server60
  Normal  Pulled     97s   kubelet            Container image "gocd/gocd-server:v21.2.0" already present on machine
  Normal  Created    97s   kubelet            Created container gocd-server
  Normal  Started    97s   kubelet            Started container gocd-server
Actions #14

Updated by Nico Schottelius 2 months ago

  • Description updated (diff)
Actions #15

Updated by Nico Schottelius 2 months ago

  • Description updated (diff)
Actions #16

Updated by Nico Schottelius 2 months ago

  • Description updated (diff)
Actions #17

Updated by Nico Schottelius 2 months ago

  • Description updated (diff)
Actions #18

Updated by Nico Schottelius 2 months ago

  • Description updated (diff)
Actions #19

Updated by Nico Schottelius 2 months ago

  • Description updated (diff)
Actions

Also available in: Atom PDF