changed 4 years ago 278 views

K8S notes for realworldapp

  1. Install MicroK8s on Linux
sudo snap install microk8s --classic

If you are not root

sudo usermod -a -G microk8s $USER
sudo chown -f -R $USER ~/.kube

You will also need to re-enter the session for the group update to take place:

su - $USER
  1. Check the status while Kubernetes starts
microk8s status --wait-ready
  1. Turn on the services you want
microk8s enable dashboard dns registry istio ingress

Try microk8s enable --help for a list of available services and optional features. microk8s disable <name> turns off a service.

  1. Start using Kubernetes
microk8s kubectl get all --all-namespaces

If you mainly use MicroK8s you can make our kubectl the default one on your command-line with alias mkctl=“microk8s kubectl”. Since it is a standard upstream kubectl, you can also drive other Kubernetes clusters with it by pointing to the respective kubeconfig
file via the --kubeconfig argument.

  1. Access the Kubernetes dashboard
microk8s dashboard-proxy
  1. Start and stop Kubernetes to save battery
    Kubernetes is a collection of system services that talk to each other all the time. If you don’t need them running in the background then you will save battery by stopping them. microk8s start and microk8s stop will do the work for you.

  2. If you need to clean the installation (remove everything)

sudo snap remove microk8s --purge
  1. Allo download from private registry

For exemple I had to create the file /etc/docker/daemon.json
with following content

{
    "insecure-registries" : ["localhost:32000", "registry.gitlab.inria.fr"] 
}

and then restart docker

sudo service docker restart

Kubebernetes sur le projet

  1. Mettre une config dans kubernetes (équivalent au consul)
microk8s kubectl create configmap mysql-config \
	--from-literal=mysql.service.name=mysql \
	--from-literal=mysql.db.name=tlc
  1. Mettre un secret dans kubernetes (équivalent au vault)
microk8s kubectl create secret generic db-security \
	--from-literal=db.user.name=foo \
	--from-literal=db.user.password=bar
  1. Support image download from private repository (not required in this lab)
sudo nano -w /var/snap/microk8s/current/args/containerd-template.toml

change the end of the file

    [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
        endpoint = ["https://registry-1.docker.io", ]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."localhost:32000"]
        endpoint = ["http://localhost:32000"]

to add the two last following lines

    [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
        endpoint = ["https://registry-1.docker.io", ]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."localhost:32000"]
        endpoint = ["http://localhost:32000"]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."registry.gitlab.inria.fr"]
        endpoint = ["https://registry.gitlab.inria.fr"]
microk8s stop
microk8s start

and add you registry login password as a secret. Update the information with you login and your password

microk8s kubectl create secret docker-registry my-private-registry1   --docker-server=registry.gitlab.inria.fr   --docker-username="logingitlab"   --docker-password="xxxx"   --docker-email=youranonymouslogin@inria.fr
microk8s kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "my-private-registry1"}]}'
  1. Create a secret
microk8s kubectl create secret generic my-jwt-private-key  --from-literal=my.jwt.privatekey=foo
microk8s kubectl create secret generic my-jwt-public-key  --from-literal=rest.my_jwt_publickey=bar
  1. Create the image for your services

Image for the back
In the node-express-realworld-example-app folder, create a Dockerfile file with the following content

FROM node:10

RUN mkdir /app
WORKDIR /app

COPY package.json .

RUN npm install --production

COPY . .

CMD ["npm", "start"]

Next in this folder build the image

docker build -t  localhost:32000/realworldnode .

Image for the front

Prepare the front for production

Edit the file src/environments/environment.prod.ts
Replace 'https://conduit.productionready.io/api\’

with \‘/api\’

You can edit this file using nano

nano -w src/environments/environment.prod.ts

In the node-express-realworld-example-app folder, create a Dockerfile file with the following content

FROM node:10-alpine as build-step
WORKDIR /app
COPY package.json /app
RUN npm install
COPY . /app
RUN npm run build

FROM bunkerity/bunkerized-nginx as prod
COPY --from=build-step /app/dist/ /www

Next in this folder build the image

docker build -t  localhost:32000/realworldangular .
  1. push the images to the public registry or localhost:32000 registry
docker push localhost:32000/realworldnode 
docker push  localhost:32000/realworldangular 
  1. Create the manifest for the back, the front and mongodb
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    devweb.service: mongorealworld
  name: mongorealworld
spec:
  replicas: 1
  selector:
    matchLabels:
      devweb.service: mongorealworld
  strategy: {}
  template:
    metadata:
      labels:
        devweb.network/realworld: "true"
        devweb.service: mongorealworld
    spec:
      containers:
#        - env:
#            - name: MONGO_INITDB_ROOT_USERNAME
#              value: root
#            - name: MONGO_INITDB_ROOT_PASSWORD
#              value: root
        - image: mongo:latest
          name: mongorealworld
          resources: {}
      restartPolicy: Always
---

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
        devweb.network/realworld: "true"
        devweb.service: realworldnode
  name: realworldnode
spec:
  replicas: 1
  selector:
    matchLabels:
      devweb.service: realworldnode
  strategy: {}
  template:
    metadata:
      labels:
        devweb.network/realworld: "true"
        devweb.service: realworldnode 
    spec:
      containers:
        - image: localhost:32000/realworldnode:latest 
          env:
            - name: MONGODB_URI
              value: mongodb://mongorealworld/conduit
            - name: NODE_ENV
              value: production
            - name: SECRET
              value: demo
          name: realworldnode
          ports:
            - containerPort: 3000
          resources: {}
#          volumeMounts:
#            - mountPath: /work/config
#              name: orangeconfigfolder
#            - mountPath: /logs
#              name: logsvolume
#      volumes:
#        - name: logsvolume
#          hostPath:
#            path: /logs # Path of the directory on the Host (node)
#            type: Directory # DirectoryOrCreate, FileOrCreate
#        - name: orangeconfigfolder
#          hostPath:
#            path: /root/compose/captcha-orange-server-config # Path of the directory on the Host (node)
#            type: Directory # DirectoryOrCreate, FileOrCreate
      restartPolicy: Always

---

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    devweb.service: realworldangular
  name: realworldangular
spec:
  replicas: 1
  selector:
    matchLabels:
      devweb.service: realworldangular
  strategy: {}
  template:
    metadata:
      labels:
        devweb.network/realworld: "true"
        devweb.service: realworldangular
    spec:
      containers:
        - env:
            - name: SERVE_FILES
              value: 'yes'
            - name: PROXY_REAL_IP
              value: 'yes'
            - name: AUTO_LETS_ENCRYPT
              value: 'no'
            - name: REDIRECT_HTTP_TO_HTTPS
              value: 'no'
            - name: HTTP2
              value: 'no'
            - name: USE_DNSBL
              value: 'no'
            - name: USE_WHITELIST_IP
              value: 'no'
            - name: USE_BLACKLIST_IP
              value: 'no'
            - name: USE_LIMIT_REQ
              value: 'no'
            - name: USE_FAIL2BAN
              value: 'no'
            - name: USE_CLAMAV_UPLOAD
              value: 'no'
            - name: USE_CLAMAV_SCAN
              value: 'no'
            - name: USE_ANTIBOT
              value: 'no'
            - name: USE_MODSECURITY
              value: 'no'
            - name: BLOCK_USER_AGENT
              value: 'no'
#            - name: MONGO_INITDB_ROOT_PASSWORD
#              value: root
          image: localhost:32000/realworldangular:latest
          ports:
            - containerPort: 8080
          name: realworldangular
          resources: {}
      restartPolicy: Always      

---

apiVersion: v1
kind: Service
metadata:
  name: mongorealworld
  namespace: default
spec:
  selector:
    devweb.service: mongorealworld
  ports:
  - port: 27017

---

apiVersion: v1
kind: Service
metadata:
  labels:
    devweb.service: realworldnode
  name: realworldnode
spec:
  ports:
    - name: "3000"
      port: 3000
      targetPort: 3000
  selector:
    devweb.service: realworldnode
---

apiVersion: v1
kind: Service
metadata:
  name: realworldangular
  namespace: default
spec:
  selector:
    devweb.service: realworldangular
  ports:
    - name: "8080"
      port: 8080
      targetPort: 8080

---
  1. Deploy the service

First you have to update the different volume path and image version you want to use in the yaml file.

next.

microk8s kubectl apply -f services.yaml

start the K8S proxy to monitor the deployment.

next create a simple ingress for this service. Create a simple ingress.yml file with the following content.

nano -w ingress.yml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: realworld-ingress
  namespace: default
spec:
  rules:
  - host: www.monservice.fr
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: realworldangular
            port:
              number: 8080
      - path: /api
        pathType: Prefix
        backend:
          service:
            name: realworldnode
            port:
              number: 3000

Next, apply this ingress.

microk8s kubectl apply -f ingress.yml

Remarks

k8s yaml file could be templated following this approach
https://learnk8s.io/templating-yaml-with-code

Remove https to dashboard and external access

sudo microk8s.kubectl -n kube-system edit deploy kubernetes-dashboard -o yaml 

When Kubernetes dashboard is installed using the recommended settings, both authentication and HTTPS are enabled. Sometimes, you may want to disable authentication or HTTPS. For example, if Kubernetes dashboard is served behind a proxy, then it’s unnecessary to e
nable authentication when the proxy has its own authentication enabled. Kubernetes dashboard uses auto-generated certificates for HTTPS, which may cause problems for HTTP client to access. This post shows how to disable authentication and HTTPS in Kubernetes dash
board.

First, we need to modify deployment of Kubernetes dashboard to remove the argument --auto-generate-certificates and add following extra arguments:

–enable-skip-login
–disable-settings-authorizer
–enable-insecure-login
–insecure-bind-address=0.0.0.0

After this change, Kubernetes dashboard server now starts on port 9090 for HTTP. Then we need to modify livenessProbe to use HTTP as the scheme and 9090 as the port. Port 9090 also needs to be added as containerPort.

Finally, we need to modify service of Kubernetes dashboard to open port 80 and use 9090 as the target port.

sudo microk8s.kubectl -n kube-system edit svc kubernetes-dashboard -o yaml 

next create a simple ingress for this service

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dashboard-ingress
#  annotations:
#    nginx.ingress.kubernetes.io/rewrite-target: /
  namespace: kube-system
spec:
  rules:
  - host: realapp.foo.fr
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: kubernetes-dashboard
#            namespace: kube-system
            port:
              number: 80

To get the login token

token=$(microk8s kubectl -n kube-system get secret | grep default-token | cut -d " " -f1)
microk8s kubectl -n kube-system describe secret $token

Configure network policies

I just create a template of network policy (network.yaml).
https://kubernetes.io/docs/concepts/services-networking/network-policies/

Add a full spring property file as a config

microk8s kubectl create configmap api --from-file= src/main/resources/application.yml