Creating, Using and Editing Kubernetes Secrets

Secrets in Kubernetes are essentially key-value pairs stored in K8s' API server's database. They are by default not encrypted so anyone with access to the API can retrieve, change or delete a secret. Thus, when storing sensitive information in K8s' secrets store, you must enable encryption-at-rest.

Good uses for secrets:

• Storing SSL certificates
• Storing container registry authentication information
• Storing sensitive access keys to 3rd party distributed configuration tools

Kubernetes documentation is fairly good, so I am not going to rehash it here, but I did want to focus on a common scenario:

You have a microservice which needs to read a configuration from a 3rd-party Distributed Configuration Service (DCS). To access this DCS, the microservice needs a key.

Creating

Let's see how this can be done using the generic secret type. The following kubectl command will create the secret.

kubectl create secret generic dcscreds --from-literal=clientid=abc --from-literal=secret=123

The first 4 words in the statement are specific to kubectl. The last word, generic, states we want to create a generic type of secret. Other types include TLS and container registry secrets. The dcscreds in the above statement is the name of the secret which you'll reference in your pods later to pull the secrets. Lastly, there are two values we're storing in the secret. Secrets can be created from different sources:

• env file (such as Docker's .env)
• actual files with the values
• literal strings

In the above example we're using literal strings. I feel like that actually makes the most sense since the other two options mean you're reading from files which could potentially be checked into source control... and we shouldn't be checking credentials into source control to begin with. The --from-literal= parameter picks the third option from the list above, and is then followed by the key=value pair clientid=abc.

Let's check the secret with

kubectl get secrets

One of the secrets should be

NAME                    TYPE                  DATA    AGE
dcscreds                Opaque                2       1m

Using

How do we reference this secret in our deployment YML files now? Let's say we want to inject a secret as an environment variable into our microservice so that it can access the DCS. Here's an example deployment YML:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cache-deployment
  namespace: integration
  labels:
    app: cache
spec:
  template:
    metadata:
      name: cache
      labels:
        app: cache
    spec:
      containers:
        - name: cache
          image: redis
          env:
            - name: CLIENTID
              valueFrom:
                secretKeyRef:
                  name: dcscreds
                  key: clientid
                  optional: false
            - name: PASSWORD
              valueFrom:
                secretKeyRef:
                  name: dcscreds
                  key: secret
                  optional: false
  selector:
    matchLabels:
      app: cache

In the above deployment you can see the env has two values. The env.name of the value is what you'd refer to inside your application, while the env.valueFrom.secretKeyRef.name is the name of secret we created above, followed by the env.valueFrom.secretKeyRef.key which refers to the key of the key=value pair. The optional flag will fail to create the deployment if the secret/key does not exist.

Great, but, what if the secret changes?

Editing

To edit a secret, we can use the following command:

kubectl edit secrets dcscreds

This will open the secrets YML in the operating system's default text editor. It will look something like this:

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  clientid: YWJj
  secret: MTIz
kind: Secret
metadata:
  creationTimestamp: "2022-05-14T20:49:15Z"
  name: dcscreds
  namespace: default
  resourceVersion: "34419738"
  uid: 5d489368-27d6-4e57-b16b-b936109b027f
type: Opaque

Values in data are base64 encoded. You can use a base64 decoder to see what they are. If you need to encode new values, you'll have to encode them. As an example, let's say the secret changes from 123 to 456. Here's a helpful command to encode that:

echo -n '456' | base64

The ouput would look like

NDU2

Once you have that base64 encoded value, simply replace it in the secrets YAML file:

...
data:
  clientId: YWJj
  secret: NDU2
...

Save the file, close it, and kubectl will update the secret in Kubernetes. Done and done!

Deleting a secret is done the same way as deleting other entities:

kubectl delete secret dcscreds

---

Resources

• Secrets
• Encryption at Rest