You've successfully subscribed to WorldRemit Technology Blog
Great! Next, complete checkout for full access to WorldRemit Technology Blog
Welcome back! You've successfully signed in.
Success! Your account is fully activated, you now have access to all content.
Success! Your billing info is updated.
Billing info update failed.
Kong Validating Admission Controller Webhook with cert-manager

Kong Validating Admission Controller Webhook with cert-manager

. 4 min read

| Image by Patrice Audet via Unsplash Copyright- free

I wanted to put out a quick guide on how to get the Kong Validating Admission Controller (VAC) working with cert-manager.

For those who don’t know, Kong can be deployed as an Ingress Controller on Kubernetes, which means that as applications are deployed and new services are created, Kong will automatically configure itself to serve traffic to these services. This also allows us to configure services within Kong using k8s manifests.

The Kong Ingress Controller ships with an Admission Controller for KongPlugin and KongConsumer resources in the API group. The Admission Controller is a piece of code that intercepts requests to the Kubernetes API server before the object is persisted, but after the request has been authenticated and authorised.

If your organisation allows teams to configure their services in Kong, you would want to use their VAC to make sure no bad config gets passed to Kong.

Let's setup the VAC Webhook with Kong. A quick search reveals Kong has instructions on how to do this - Open-Source API Management and Microservice Management.

The Admission Controller needs a TLS certificate and key pair which you need to generate as part of the deployment.

The above documentation is a good start, however Kong only provides certificate instructions on how to either:

  • Generate the certificate manually that is suitable for testing and playing with the VAC, but the year is 2021... please don’t use this method for Production...
  • Take advantage of the built-in Kubernetes CA, using the k8s CA allows us to skip most of the commands from the manual step, but we still need to manually validate the certificates which is still not good enough.

None of the above provides us with a solid renewal solution, and if your cert is bad, all requests that have anything to do with the apiGroup attached to the ValidatingWebhookConfiguration will fail.

Here comes cert-manager, the automation-friendly solution. What does it do? Cert Manager is a Kubernetes add-on to automate the management and issuance of TLS certificates from various issuing sources.

I will skip the installation section and assume you have cert-manager installed and correctly set up in your cluster.

After removing all of this, let’s get Kong to work properly with cert-manager.

Browsing through the cert-manager documents, we can quickly come up with the yaml required to generate a self-signed certificate request.

kind: Issuer
  name: selfsigned-issuer
  selfSigned: {}
kind: Certificate
  name: kong-validation-webhook
  commonName: kong-validation-webhook.kong.svc
    - kong-validation-webhook.kong.svc.cluster.local
    - kong-validation-webhook.kong.svc
    kind: Issuer
    name: selfsigned-issuer
  secretName: kong-validation-webhook
  duration: 8760h # 1y
  renewBefore: 730h # 1m

Cert-manager issues certificates with a default duration of 90 days, which normally would be fine, but as we will see later, there is an issue with Kong when it comes to updating the cert files, so we set it to 1 year, the default for anything that communicates with the API in k8s -

After applying the above to the cluster, you will see a new secret name in your Kong namespace with the name of kong-validation-webhook.

Before we configure the actual webhook, we need to modify the Kong Ingress Controller and add the following yaml to our Controller deployment.

  value: ":8080"
  - mountPath: /admission-webhook
    name: validation-webhook
  - name: validation-webhook
      secretName: kong-validation-webhook

Or just patch it directly as in the image below.

kubectl patch deploy -n kong ingress-kong \\
    -p '{"spec":  \\
    {"template":  \\
    {"spec":      \\
    {"containers":  \\
    [{"name":"ingress-controller","env":  \\
    [{"name":"CONTROLLER_ADMISSION_WEBHOOK_LISTEN","value":":8080"}],  \\
    "volumeMounts":[{"name":"validation-webhook","mountPath":"/admission-webhook"}]}], \\
    "volumes":[{"secret":{"secretName":"kong-validation-webhook"}, \\
    "name":"validation-webhook"}]}}}}' \\
deployment.extensions/ingress-kong patched  \\

Once the secret is created and the deployment updated, we can issue the actual webhook as shown in the image below.

# <
kind: ValidatingWebhookConfiguration
  name: kong-validations
  annotations: kong/kong-validation-webhook
  - name:
    failurePolicy: Fail
    sideEffects: None
    admissionReviewVersions: ["v1beta1", "v1"]
      - apiGroups:
          - "*"
          - CREATE
          - UPDATE
          - kongconsumers
          - kongplugins
      - apiGroups:
          - ""
          - "v1"
          - CREATE
          - UPDATE
          - secrets
        name: kong-validation-webhook
        namespace: kong
      # caBundle: Cg==

Note that we do not need the caBundle as we are generating it with cert-manager.

Make sure you deploy the ValidatingWebhookConfiguration after the certificate is generated and deployment is updated. If you first deploy the VAC, any deploy that is relatable to the resources listed above will fail, possibly others will fail too, due to certificate incompatibility.

Error from server (InternalError): error when creating "STDIN":
Internal error occurred: failed calling webhook "": 
Post https://kong-validation-webhook.kong.svc:443/?timeout=30s: x509: 
certificate signed by unknown authority (possibly because of "x509: 
invalid signature: parent certificate cannot sign this kind of certificate" \
while trying to verify candidate authority certificate "kong-validation-webhook.kong.svc")

After everything has been set up, we can see Kong now rejects the invalid configurations as shown in the image below.

❯ echo "
kind: KongPlugin
  name: request-id
  foo: bar
  header_name: my-request-id
plugin: correlation-id
" | aws-okta exec wr-login.devops -- kubectl apply -f -
Error from server: error when creating "STDIN": 
admission webhook "" denied the request: 
400 Bad Request
{"fields":{"config":{"foo":"unknown field"}},
"name":"schema violation","code":2,
"message":"schema violation ( unknown field)"}
exit status 1

You will be forgiven if you think that's all, but there is a lurking issue that will become apparent in a couple of months, the certificate renewal will fail as Kong won't receive the updated certificate on disk without redeploying, and by the time you redeploy you will most likely see the same certificate error as before.

The "workaround" for this is to set a long cert duration (this is the duration and renewBefore in the cert-manager certificate request yaml). This was not ideal from a security perspective but I wasn't aware of a better temporary solution to the problem.

Thanks for taking the time to read! Hopefully this short guide will help you setup Kong VAC Webhook with cert-manager and allow you to better manage certificates. We should always strive to squash manual processes when configuring production ready workloads.

Have a great day and stay safe!