Lingxian Kong, senior developer at Catalyst Cloud, offers this tutorial.

At Catalyst Cloud, we’re planning to deploy Magnum in our OpenStack based public cloud in New Zealand.

Magnum is an OpenStack service offering container clusters as-a-service, with support for Docker Swarm, Kubernetes, Mesos or DC/OS (Catalyst Cloud will support Kubernetes at the first step). Users of the service can deploy clusters of thousands of nodes in minutes and access them securely using their native APIs.

One of the feature requests coming from our existing customers is integration with OpenStack Keystone for both authentication and authorization, so that existing users within a tenant can access a Kubernetes cluster created by the tenant administrator in Magnum without too much extra user-management configuration inside the Kubernetes cluster.

The development team at Catalyst Cloud tested the integration between Keystone and Kubernetes, after some bug fixes and improvements, it works perfectly fine!

Before I walk you through it, you might want to look at a blog post about Keystone authentication in Kubernetes clusters by Saverio Proto from SWITCH. Here in this tutorial, we’ll talk about authentication and authorization, making the authorization configuration is very easy and flexible, especially after this pull request merged.  One last note before starting: we’d also like to give huge thanks to OpenLab — all of our tests are performed on infrastructure that they provided.

Prerequisites

  • An OpenStack deployment. I’m using Devstack to set up the OpenStack cloud of Queens release for the test.
  • We’re assuming that users for this tutorial already use OpenStack.
    In the real world, there are several types of users:

    • Cloud operators responsible for cloud operation and maintenance
    • Kubernetes cluster admin users who can create/delete Kubernetes cluster in OpenStack cloud and who are also administrators of the Kubernetes cluster
    • Kubernetes cluster normal users who can use the cluster resources, including, maybe different users with different authorities
  • Kubernetes version >= v1.9.3. For testing purposes, I won’t use Magnum in this post, the Kubernetes cluster admin can create the cluster in VMs by using kubeadm.
  • kubectl version >=v1.8.0. As Saverio Proto said in his blog post, kubectl has been capable of reading the openstack env variables since v1.8.0

Basic workflow

From the perspective of OpenStack cloud operator

As a cloud operator, there are some tasks to perform before exposing the Keystone authentication and authorization functionality to Kubernetes cluster admin:

  • Necessary Keystone roles for Kubernetes cluster operations need to be created for different users, e.g. kube-adm, kube-editor, kube-viewer
    • kube-adm role can create/update/delete Kubernetes cluster, can also associate roles to other normal users within the tenant
    • kube-editor can create/update/delete/watch Kubernetes cluster resources
    • kube-viewer can only have read access to Kubernetes cluster resources
source ~/openstack_admin_credentials
for role in "k8s-admin" "k8s-viewer" "k8s-editor"; do openstack role create $role; done
openstack role add --user demo --project demo k8s-viewer
openstack user create demo_editor --project demo --password password
openstack role add --user demo_editor --project demo k8s-editor
openstack user create demo_admin --project demo --password password
openstack role add --user demo_admin --project demo k8s-admin

From the perspective of a Kubernetes cluster admin

In this example, demo_admin user is a Kubernetes cluster admin in the demo tenant. demo_admin user is responsible for creating Kubernetes cluster inside the VMs, it could be done easily with Saverio Proto’s repo, I also have an ugly Ansible script repo here.

After the Kubernetes cluster is up and running, the cluster admin should make some configuration for Keystone authentication and authorization to work.

  1. Define Keystone authorization policy file.
    cat << EOF > /etc/kubernetes/pki/webhookpolicy.json
    [
      {
        "resource": {
          "verbs": ["get", "list", "watch"],
          "resources": ["pods"],
          "version": "*",
          "namespace": "default"
        },
        "match": [
          {
            "type": "role",
            "values": ["k8s-admin", "k8s-viewer", "k8s-editor"]
          },
          {
            "type": "project",
            "values": ["demo"]
          }
        ]
      },
      {
        "resource": {
          "verbs": ["create", "update", "delete"],
          "resources": ["pods"],
          "version": "*",
          "namespace": "default"
        },
        "match": [
          {
            "type": "role",
            "values": ["k8s-admin", "k8s-editor"]
          },
          {
            "type": "project",
            "values": ["demo"]
          }
        ]
      }
    ]
    EOF
    

    As an example, the above policy file definition is pretty straightforward. The Kubernetes cluster can only be accessed by the users in demo tenant, users with k8s-admin or k8s-editor role have both write and read permission to the pod resource, but users with k8s-viewer role can only get/list/watch pods.

  2. Deploy k8s-keystone-auth service. The implementation of k8s-keystone-auth service locates in the OpenStack cloud provider repo for Kubernetes. It’s running as a static pod (managed by kubelet) in the Kubernetes cluster.
    cat << EOF > /etc/kubernetes/manifests/k8s-keystone-auth.yaml
    ---
    apiVersion: v1
    kind: Pod
    metadata:
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ""
      labels:
        component: k8s-keystone-auth
        tier: control-plane
      name: k8s-keystone-auth
      namespace: kube-system
    spec:
      containers:
        - name: k8s-keystone-auth
          image: lingxiankong/k8s-keystone-auth:authorization-improved
          imagePullPolicy: Always
          args:
            - ./bin/k8s-keystone-auth
            - --tls-cert-file
            - /etc/kubernetes/pki/apiserver.crt
            - --tls-private-key-file
            - /etc/kubernetes/pki/apiserver.key
            - --keystone-policy-file
            - /etc/kubernetes/pki/webhookpolicy.json
            - --keystone-url
            - http://10.0.19.138/identity/v3
          volumeMounts:
            - mountPath: /etc/kubernetes/pki
              name: k8s-certs
              readOnly: true
            - mountPath: /etc/ssl/certs
              name: ca-certs
              readOnly: true
          resources:
            requests:
              cpu: 200m
          ports:
            - containerPort: 8443
              hostPort: 8443
              name: https
              protocol: TCP
      hostNetwork: true
      volumes:
      - hostPath:
          path: /etc/kubernetes/pki
          type: DirectoryOrCreate
        name: k8s-certs
      - hostPath:
          path: /etc/ssl/certs
          type: DirectoryOrCreate
        name: ca-certs
    status: {}
    EOF
    

    The image is built using the Dockerfile in the cloud-provider-openstack repo, you can build your own image if needed. Please replace the keystone-url param pointing to your own OpenStack cloud.

  3. Configure authentication and authorization webhook for Kubernetes API server, then wait for the API server to run.
    cat << EOF > /etc/kubernetes/pki/webhookconfig.yaml
    ---
    apiVersion: v1
    kind: Config
    preferences: {}
    clusters:
      - cluster:
          insecure-skip-tls-verify: true
          server: https://localhost:8443/webhook
        name: webhook
    users:
      - name: webhook
    contexts:
      - context:
          cluster: webhook
          user: webhook
        name: webhook
    current-context: webhook
    EOF
    sed -i "/authorization-mode/c \ \ \ \ - --authorization-mode=Node,Webhook,RBAC" /etc/kubernetes/manifests/kube-apiserver.yaml
    sed -i '/image:/ i \ \ \ \ - --authentication-token-webhook-config-file=/etc/kubernetes/pki/webhookconfig.yaml' /etc/kubernetes/manifests/kube-apiserver.yaml
    sed -i '/image:/ i \ \ \ \ - --authorization-webhook-config-file=/etc/kubernetes/pki/webhookconfig.yaml' /etc/kubernetes/manifests/kube-apiserver.yaml
    

Now the Kubernetes cluster is ready for use by users within the demo tenant in OpenStack.

From the perspective of a Kubernetes cluster user

Cluster users only need to config the kubectl to work with OpenStack environment variables.

kubectl config set-credentials openstackuser --auth-provider=openstack
kubectl config set-context --cluster=kubernetes --user=openstackuser [email protected] --namespace=default
kubectl config use-context [email protected]
  1. demo user can list pods but can not create pods.
    $ source ~/openrc_demo
    $ kubectl get pods
    No resources found.
    $ cat << EOF > ~/test_pod.yaml
    ---
    apiVersion: v1
    kind: Pod
    metadata:
      name: busybox-test
      namespace: default
    spec:
      containers:
      - name: busybox
        image: busybox
        command:
          - sleep
          - "3600"
        imagePullPolicy: IfNotPresent
      restartPolicy: Never
    EOF
    $ kubectl create -f ~/test_pod.yaml
    Error from server (Forbidden): error when creating "test_pod.yaml": pods is forbidden: User "demo" cannot create pods in the namespace "default"
    
  2. demo_editor user can create and list pods.
    $ source ~/openrc_demoeditor
    $ kubectl create -f ~/test_pod.yaml
    pod "busybox-test" created
    $ kubectl get pods
    NAME           READY     STATUS    RESTARTS   AGE
    busybox-test   1/1       Running   0          3s
    
  3. users from other tenants don’t have access to the Kubernetes cluster
    $ source ~/openrc_alt_demo
    $ kubectl get pods
    Error from server (Forbidden): pods is forbidden: User "alt_demo" cannot list pods in the namespace "default"
    

All in one – live demo

asciicast

Future work

At Catalyst Cloud, we’re working on automating all the manual steps in Magnum, so in the near future, the Kubernetes cluster admin will only need to run a single command to create the cluster and Magnum will perform the rest automatically. We’ll also continue to work with sig-openstack group to keep improving stability and performance of the integration.

Author Lingxian Kong will be heading up four sessions at the Vancouver Summit, check them out here.