Simplify PKI in Kubernetes Environments and Accelerate DevOps

I recently worked with a client on their public key infrastructure (PKI). After a few short discussions, it became evident that they were ill-prepared. They had significant challenges related to managing SSL/TLS certificates, which serve as machine identities in their diverse infrastructure. Like most organizations, they had started their digital transformation journey. They were grappling with changes like moving from on-premises to the cloud and rearchitecting applications from the ground up. In addition, operational pain and quality of talent had long plagued the organization, which resulted in a confusing amalgamation of technology across various on-premises and cloud providers. 

Sound familiar?

Furthermore, as we zeroed in on the PKI environment it too was in disarray. After several rounds of discussions, the root cause behind this became clear. They had built several layers of complex PKI infrastructure at all points of the infrastructure - both on-premises and in the cloud. There were no standardized controls or approaches in use. This was costing developer and DevOps engineering time. In addition, security teams felt things were getting out of control since many certificates were not controlled or made visible to InfoSec and PKI teams. And, security teams couldn’t inspect traffic since they didn’t have access to the private keys. Information security was concerned that an attacker could go undetected by using encrypted tunnels.

This is how their PKI was set up:

As you can see, they were using many different sources on-premises, ranging from Let’s Encrypt to GlobalSign, in addition to Amazon Certificate manager and Azure Key Vault. They also had many instances of AWS and Azure.

My goal for this client was to remove unnecessary complexity and provide building blocks for developers to request certificates quickly via automation, while also addressing security concerns.

Here is how I tackled the challenge. 

Streamlining Certificate Management to Address DevOps Security Challenges

Rather than reinvent the wheel, I turned to NIST for a blueprint that we could practically follow. Sparing you all the details, I read through several documents surrounding cryptography and landed a new guidance document: NIST 1800-16B: Securing Web Transactions: TLS Server Certificate Management.

After studying this detailed guidance, I extrapolated several clear, immediate recommendations around these themes: 

• Simplify processes by offering a certificate service to developers
• Provide security teams control and visibility over certificates in use

My recommendations were to:
1. Standardize CAs across on-premises and cloud providers
2. Use Jetstack cert-manager to automate certificate life cycles for Kubernetes clusters
3. Use Venafi as a Service to connect certificate authorities to DevOps workflows via native integrations, enforce policy (e.g. certificate attributes and sources), and provide visibility for audits and compliance

Ultimately, these choices helped to streamline certificate management workflows in multicloud Kubernetes environments and also acted as a unified API that would allow interactions with different CAs. In the section that follows, I’ll provide detailed step-by-step instructions for how to address points #2 and #3 above, using GlobalSign as a CA.

How to Connect Kubernetes to DevOpsACCELERATE and GlobalSign

Technologies Used

• Jetstack cert-manager for Kubernetes (and OpenShift)
• Venafi as a Service DevOpsACCELERATE

Step 1. Get a free DevOpsACCELERATE Account: https://www.venafi.com/cloud

Step 2. Configure CA Accounts 

On the left-hand menu select “CA Accounts” and set up your CA account(s).  

Select add new account for each provider.

The end result should look like this:

Step 3. Set Up Issuing Templates
Issuing templates connect a certificate authority and certificate attributes together – aka security policy. You’ll likely have multiple issuing templates since you will need publicly-trusted certificates with longer validity periods as well as private certificates for back-end infrastructure, such as containers and service meshes.
 

Note: Make sure you use .*.yourdomain.com to match your domain if you want to make sure you validate domains before issuance (Recommended). Example below:

Step 4. Configure Projects and Zones

In this example we are going to create a Cloud Project and grant the team the ability to issue both public and private certificates.

In order to issue certificates we first create a Project, which allows us to segment certificate issuance for an application.
 

Select “Create A New Project” and give the project a name, e.g. “Cloud.”

Then, add a Zone. A Zone represents an environment, e.g., development, staging, production. 

Since this is the first time using Venafi as a Service DevOpsACCELERATE, I’ll create zones and tie them to the issuing templates. This connects an application’s environment to the certificate source and attributes as defined in the issuing template (aka security policy).

Step 5. Grab Zone IDs to Use for Issuing

Once you’ve finished the previous step, your project should be listed. Click on the project. In this case our project name is “Cloud” so I clicked on the Project named “Cloud.”
 

Once opened, you will see each Zone that is tied to our issuing templates (this defines the CA and certificate attributes). Click on each tab – Development, Production, Staging – and copy the Zone ID’s. These will be used in our yaml files to create the issuers in cert-manager. An example is highlighted in the subsequent screenshot:

Step 6. Grab the API key and Store It for Use Later

This is what we are going to need to make the certificate request to the Venafi as a Service API broker.

Now that we have the information we need to get started issuing certificates., lets iInstall Jetstack cert-manager so we can start requesting certificates from the various CA providers to use in our Kubernetes environment

Step 7. Install Jetstack cert-manager

First, let’s create a kubernetes namespace specifically for cert-manager.

kubectl create namespace cert-manager
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v0.14.2/cert-manager.yaml

If all went well, you should see something like this:.
 

Run the following to check on status, running means we are ready to move on the next step:

 kubectl get pods --namespace cert-manager
 

Step 8. Set Up API Key as a Secret Within Kubernetes

This is the API that you grabbed in Step 6 above.

If you are using a separate namespace, just change the name from default.

kubectl create secret generic cloud-secrets --namespace='default' --from-literal=apikey='1a637c2b-125f-0000-82ea-c7b60ed00000'

Should return:

Step 9. Set Up the Issuers That You Want to Configure

In our case we have one issuer, so we are going to need to create the issuer.
Please note the name and zone id in the file.

kubectl create -f globalsign-cloud-issuer.yaml

kubectl describe issuer globalsign-venafi-issuer --namespace=default

kubectl describe issuer globalsign-venafi-issuer --namespace=default

Step 10. Try it Out!

Now that we have installed cert-manager, we can start requesting certificates. Note also that cert-manager automatically attempts to renew certificates before they expire which is very handy.

In my example, I had to request a certificate from our internal CA and one from our external to be used. Example request format is below.

Running the request:
The below command will run the request to get a certificate.

kubectl apply -f examplecert.yml

View your request here:
https://ui.venafi.cloud/certificates/inventory

Step 11. Troubleshooting
The first step to troubleshooting is to get a status on your existing certificates. So we will run our kubectl get certificates command. This will allow us to see what is currently in inventory and the status.

kubectl get certificates

Since the flag is false (expected to be true) we will dive a little deeper and get the appropriate status messages using:

kubectl describe certificates globalsign-devops.amplisight.io

Once the command is run, you will see there is a listing for the certificate request in the message and the current status.  Above you will see that each certificate request gets assigned a unique id which is in the format of <certificate name>-Unique ID.  

In order to see more details on potential errors on issuance you must run the following command:

kubectl describe certificaterequest globalsign-devops.amplisight.io-23434489

In our case, we were just a little (or a lot!) impatient and needed to allow the issuer time to issue the certificate.

Take the Next Step

To learn more about GlobalSign’s PKI for DevOps or Venafi as a Service DevOpsACCELERATE, we invite you to watch this short webinar demo and visit: 
https://www.globalsign.com/en-sg/lp/venafi
https://www.venafi.com/cloud