Kubernetes for non-ops folks - Kubernetes Objects and manifests - an introduction

What are k8s objects?

Objects in k8s are entities that k8s uses to represent the state of your cluster. They are used to describe:

• what applications are running on which nodes in the cluster
• what resources are available to those applications
• what policies exist around the behaviour of those applications

Like I mentioned in the description of the control loop in Controller Manager, these object specs determine the desired state.

How do I work with them?

In order to work with k8s objects, you need to use the k8s API. You do this using the kubectl CLI. If you wish to programmatically interact with the API, you could use one of the k8s client libraries.

How do I describe an object to k8s?

In order to describe a k8s object, you have to create a manifest file, which is a yaml file with something called spec, short for specification of the object. The spec is where you describe the characteristics of the resource - the desired state.

The other side of this is the status, which describes the current state of the object, supplied and maintained by the k8s system. The control plane actively manages every object’s state to match the desired state that you supplied in the spec.

So the control loop whenever a spec is updated is about making the status look like what’s described in the spec.

Example

Let’s look at a manifest file example. The manifest file is a plain text file, most commonly found in YAML format by convention. You could write it in JSON too, if you really want. No one’s going to stop you. You then have to use kubectl to send this spec or desired state to the control plane. kubectl talks to the API Server, converts the YAML to JSON behind the scenes and via the API, requests the control plane to achieve the desired state.

The control plane obliges.

The following is a manifest taken from the example at k8s.io

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2 # tells deployment to run 2 pods matching the template
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

You might have noticed that before the spec field in the manifest, there are some other fields that weren’t spoken about earlier. Notice the name and the kind fields. We’ll cover them soon.

In order to let the control plane know about your desired state, you use kubectl and pass the path to the manifest file.

kubectl apply path/to/my/manifest.yaml

Et Voila! If all goes well, kubectl prints something like this on the command line.

deployment.apps/nginx-deployment created

Anatomy of a manifest file - mandatory fields

Some of the fields in the manifest you saw earlier are mandatory fields and cannot be omitted.

• apiVersion - the version of k8s API you are talking to
• metadata - data that helps uniquely identify your object in the cluster - name, uid, namespace etc.
• kind - there are few different kinds of objects you can create
• spec - desired state of your object

Checkout the K8s API Reference to figure out all the different fields for different kinds of k8s objects and all the gory details.

So what are some objects in k8s and what do they mean?

There are several different types of objects that you’ll come across in k8s. Many of these objects are watched by controllers of the same name which then ensure that your application maintains the desired state.

Before we go ahead I’d like to introduce a concept that is used a lot in the kubernetes world or in most distributed systems world:

A workload is an application running on kubernetes. Whatever your workload is, whether it is a single component or is comprised of many components, k8s runs them inside a set of pods.

That helps me start the definitions of the objects.

Pod

A pod represents a set of running containers on your cluster! They are the most fundamental deployable units of computing that you can create and manage in your k8s cluster.

If you look at the definition of the word pod in a dictionary, you’ll find things like:

a seed vessel of a leguminous plant such as the pea, splitting open on both sides when ripe

a detachable or self-contained unit on an aircraft, spacecraft, vehicle, or vessel, having a particular function: the torpedo’s sensor pod

a small herd or school of marine animals, especially whales

A pod in k8s, is a fundamental deployment unit that comprises one or more containers, with shared storage and network resources and a specification for how to run the containers.

Why more than one container?

A pod may need to run multiple containers because they are closely related in a way that is necessary. These containers are coupled in a way that their contents are always co-located and co-scheduled and run in a shared context; aka a namespace or cgroups. This is based on the needs of the application and its design.

Pods may contain something called an Init Container or ephemeral containers, or more recently sidecar containers! Fascinated? These are pretty advanced topics at this point, so you don’t need to think about them now, but if you are curious, search and find out.

Running multiple containers in a pod is a relatively advanced use case. Use this pattern where it is absolutely necessary - always remember

just because you can, doesn’t mean you should.

ReplicaSets

ReplicaSets are used to maintain a stable set of replica Pods running at any given time in the a k8s cluster to ensure high availability. This is used to guarantee the availability of a specified number of identical pods.

How do they work?

Like other objects, you define ReplicaSets in a yaml file with all the necessary fields. You have to let ReplicaSet know how to identify Pods and the number of Replicas and describe the pod in the pod template.

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: frontend
  labels:
    app: guestbook
    tier: frontend
spec:
  # modify replicas according to your case
  replicas: 3
  selector:
    matchLabels:
      tier: frontend
  template:
    metadata:
      labels:
        tier: frontend
    spec:
      containers:
      - name: php-redis
        image: us-docker.pkg.dev/google-samples/containers/gke/gb-frontend:v5

When do you use one?

The official recommendation from the k8s.io docs is that a Deployment allows you to do much more than just specify you need a certain number of replicas for your pod and takes over providing a declarative approach to updating your pods. Thus in fact, you may never need to use a ReplicaSet explicitly in production and can solely rely on defining and updating Deployments instead. 👍

Deployment

Deployment is the most common way to describe and run an application on your cluster. It provides a way to declaratively update your application on k8s.

A desired state is described in a Deployment, and the Deployment controller changes the actual state to the desired state.

The deployment describes your service, the number of replicas of which has to run at any given time, the name of the containers etc.

They are used for:

• Rolling out a ReplicaSet
• Declaring new state of the pods - i.e. for the purpose of updating your applications
• rolling back to an earlier deployment revision
• scaling up the deployment based on needs
• pausing the rollout of a deployment
• cleaning up older replicasets

As this is a introductory series for non operations folks to get an overview, I’m not going to go into the details of each. But all of the interaction to k8s is done via kubectl and the intent is shared with the cluster through the deployment manifest file.

DaemonSets

As the name suggests they are used to run daemons.

In computer science, a daemon is a process that runs in the background.

DaemonSet ensures that a set of nodes would run a copy of a pod.

When you delete a DaemonSet, all the pods created as part of it will be cleaned up. And when the nodes running those daemons are killed, the daemon pods will be garbage collected too.

This is how we have solutions for storage, logs collection, monitoring, etc running on k8s.

PersistentVolume

Storage is a different problem to compute. When talking about k8s most people are concerned about the compute part of their microservices and often forget that there could be stateful applications that need to persist data somewhere. PersistentVolume provides an API for users and admins to abstract details of how storage is provisioned and consumed in k8s.

As we have covered Pods, you already know that pods can be created and destroyed at random and as often as needed. Thus storing data in a pod, is like accepting that the data is going to be volatile. So if you want the data that is being dealt with by your pod persisted somewhere, then you’d have to resort to Persistent Volumes.

How can I use a persistent volume?

In simple steps:

1. provision a persistent volume
2. Then create a persistent volume claim - i.e. a request for storage by the user - claiming a certain size of storage
3. k8s then binds a PVC (persistent volume claim) to a PV that meets the criteria as defined by the claim
4. Use the PVC in your pod spec yaml to mount the PV and start accessing it for storage!

I am not going to go further into the details as this is an overview of some core k8s concepts.

So if you want to know more, head over to the docs on this.

Service

In k8s, a Service is a concept that exposes an application over the network within or beyond your k8s cluster, backed by one or more Pods. As we know by now, pods are ephemeral - could be created and destroyed in a short period of time.

Every time a pod comes up it could have a different IP address in the cluster. A Service is the abstraction that provides a stable endpoint, i.e. an IP address and a DNS name for a set of pods, based on how you define them, allowing other set of pods or components or end-users to interact with these pods, without having to know their exact individual IP addresses.

This provides some benefits which I think you might have already guessed:

• Services make your applications discoverable. They have a stable IP and DNS entry, making it easier to consume
• They work like load balancers when you have a pod with replicas. They abstract away the fact that replicas are serving requests. The clients do not need to connect directly to a pod, they connect to a service.

This abstraction enables you to expose a service in many different ways, which k8s refers to as service types, enabling you to expose a service internally within the cluster, on a static port on all nodes, create a load balancer that exposes the service externally and so on.

StatefulSet

A StatefulSet is an interesting one, it is used to describe and manage an application that stores state. When your app manages state, the order and uniqueness of the pod instances become important.

This is particularly useful when:

• your application needs to have persistent unique network identifiers
• does something with persistent storage - like manages a database or datastore
• needs to ensure graceful deployment and scaling

Examples

Some real world examples to demonstrate the use-cases better:

1. Your application is a Database. If you have a SQL database to run in the cluster where you need to ensure there is one primary and a secondary instance and their replicas. You need to ensure these instances have persistent identities so that requests can reach the right instance at any given time.
2. If your application is a Distributed file system, like HDFS (Hadoop distributed file system), each node, needs a persistent identity and storage to manage the file system metadata and the data blocks.
3. Another example is that you are running messaging system like Kafka on k8s. Your Kafka brokers need a stable identity and persistent storage for the cluster to function
4. Running a monitoring system like Prometheus on your k8s cluster where yet again, you have to share the persistent storage across the cluster.

Gotchas and limitations

• Deleting a StatefulSet from your cluster doesn’t delete the Persistent Storage associated with it. Which is ideal in most cases. You wouldn’t want to lose the data just because someone deleted the thing that represented the a stateful application
• The prescribed way to terminate pods of a StatefulSet is to scale it down to 0 prior to deletion

For more details, check out the Limitations section of the StatefulSet docs.

Namespace

Namespaces are a means to divide cluster resources among multiple users or teams. They are a grouping and isolation mechanism that makes it easier to manage cluster resources and organise complex environments.

A very common use-case is when a cluster is shared by multiple teams - some people refer to this as multi-tenancy - but that is entirely contextual - i.e. it depends on what a tenant in your cluster mean.

The idea is that namespaces provide isolated little worlds within a cluster. So an example I can think of is that two different teams can have services of the same name, however belonging to different namespaces and doing different things! In a way this isolation reduces the chances of naming conflicts. Namespaces are most useful probably when enforcing limits and quotas for different groups.

Also worth mentioning that there are some default namespaces - called Initial namespaces as a k8s cluster starts with a few namespaces.

ConfigMaps

I think the name gives away what this thing does.

ConfigMaps = Configuration + Map

It is a dictionary datastore in k8s that allows you to separately store configuration from your application code. If you are a software engineer, you have already separated concerns when developing your application, and injected dependencies only when required. ConfigMaps enable your containerised apps to be built once and deployed to different environments.

ConfigMaps are generally defined in yaml like all other k8s objects, but the special thing is instead of using the spec keyword to define it, we use data!

You can then go on and define simple key-values or define a collection of key-values spread over multiple lines.

The real beauty is that k8s allows you to consume ConfigMaps in many different ways.

1. Inside your container command
2. As environment variables for a container
3. As a file in a read-only volume for the application
4. Using k8s API inside the pod that reads from the ConfigMap

ConfigMaps are not meant to store large data - this is not a database. Please be mindful of what you put in there. If your application configuration is read during container initialisation, you don’t want to be reading and parsing huge amounts of data from your ConfigMap.

Also ConfigMaps do not encrypt the configuration data. So please do not store sensitive information here.

In essence ConfigMaps are a shared key-value store of key configuration that maybe required by your pod or pods within a namespace.

Secrets

Some configuration is meant to be stored securely because it is sensitive information, like passwords, API keys, or certificates. Secrets are designed to handle data that needs to stay confidential, they are also locked down to a specific set of pods who have the right keys to the data.

Secrets are base64 encoded by default and can be encrypted in transit and at rest based on your k8s cluster configuration. Just because it is base64 encoded, doesn’t make it encrypted. Encoding and encryption are two very different things. You can also define fine-grained access permissions, i.e. who or what can access a secret.

Just as you wouldn’t store secrets in ConfigMaps, you shouldn’t store non-sensitive information in Secrets.

Just because you use Secrets doesn’t make your k8s cluster invulnerable. Ensure that you have good security implemented on all aspects of the cluster, like for starters, you have configured proper role based access control for your k8s cluster. After all Secrets are only secure if your cluster is secure.

Secrets are by default scoped to the a single namespace - like most other resources, they can only be accessed within the namespace they were defined for.

Job

The first time I heard this term, I was confused as to why this existed and why it was different from concepts like Deployment or simply a pod.

A job represents a one-off task that runs and does what it has to do and then terminates in the end. A job creates one or more pods and will continue to retry until the pods have completed successfully. Once the pods complete successfully, that’s the exit condition of the retry mechanism of the Job, which is different from what we have seen with Deployments. Deployments are suited for scalable or long running workloads - it doesn’t track a pod’s completion but ensures that the pod is always running.

So if a job can start one or more pods, surely it can start multiple pods in parallel to get things executed in parallel too. Jobs are designed to keep track of how many tasks or pods have completed successfully or failed and guarantee that they run until they are completed successfully.

As a concept I remember what a job is like so:

Ingress

Ingress is a resource that manages external access to services inside k8s cluster - usually via HTTP or HTTPS. It behaves like a reverse proxy and provides routing rules to map external traffic to an appropriate service inside the k8s cluster.

Earlier we saw that the concept of Service exposes an application within or beyond the cluster. However, Ingress is similar but takes it further.

1. Provides a unified way to expose multiple services through a single external IP address - in a way it is a reverse proxy
2. Provides routing to services based on URLs - path or hostnames
3. Simplifies HTTP configuration by handling SSL/TLS termination centrally
   • Instead of configuring SSL/TLS for each Service individually, you set up a certificate at the Ingress level.
   • Certificates can be renewed or replaced at the Ingress controller level instead of doing it per service
   • Ingress controller even does certificate negotiation and renewal automation - using Cert Manager
4. Supports custom headers, rate limiting, authentication integration through annotations and configurations

It is a bit like an API gateway.

References

• Objects in k8s
• Deployments in Kubernetes
• Linux namespaces - man page
• What are StatefulSets in k8s