35 Advanced Kubernetes Interview Questions for 5+ Experienced

The control plane is the brain of the Kubernetes cluster. Its main components are:

• `kube-apiserver`: Exposes the Kubernetes API. It is the front end of the control plane and handles all internal and external requests.
• `etcd`: A consistent and highly-available key-value store used as Kubernetes’ backing store for all cluster data (the desired state).
• `kube-scheduler`: Watches for newly created Pods with no assigned node and selects a node for them to run on based on resource requirements, policies, and affinity specifications.
• `kube-controller-manager`: Runs controller processes. These controllers watch the cluster’s state through the API server and work to move the current state towards the desired state. Examples include the Node controller, Replication controller, etc.

`etcd` is a distributed key-value store that Kubernetes uses as its single source of truth for all cluster configuration and state. Every object you create (like Pods, Deployments, Services) is stored in `etcd` as a key-value pair.

It is critical because it stores the desired state of the entire cluster. All control plane components watch `etcd` (via the API server) for changes and work to reconcile the actual state with this desired state. A failure or data loss in `etcd` means the loss of the entire cluster’s state, making it a critical component to back up.

The reconciliation loop is the core concept that drives Kubernetes. It’s an endless control loop performed by controllers. The process is:

1. A controller **observes** the current state of the cluster.
2. It compares this current state with the **desired state** stored in `etcd`.
3. If there is a difference, the controller takes **action** to make the current state match the desired state.

For example, a ReplicaSet controller sees that the desired replica count is 3 but only 2 pods are running. It then takes action by creating a new Pod to reconcile the state.

• `livenessProbe`: Checks if a container is still running and healthy. If the probe fails, the `kubelet` will kill the container and restart it according to its restart policy. Use this to recover from deadlocks or unrecoverable application states.
• `readinessProbe`: Checks if a container is ready to start accepting traffic. If the probe fails, the container’s IP address is removed from the endpoints of any matching Services. Use this to signal that your application is temporarily busy or still initializing and shouldn’t receive requests.
• `startupProbe`: Checks if an application within a container has started successfully. All other probes are disabled until this one succeeds. This is useful for slow-starting applications to prevent them from being killed by a `livenessProbe` before they are even up and running.

Kubernetes assigns a QoS class to each Pod to determine its scheduling and eviction priority. The class is determined by the CPU and memory `requests` and `limits` set on its containers.

• `Guaranteed`: Highest priority. Assigned when every container in the Pod has both a memory and CPU request and limit, and they are equal. These pods are the last to be evicted.
• `Burstable`: Medium priority. Assigned when at least one container has a memory or CPU request, but they don’t meet the criteria for `Guaranteed`.
• `BestEffort`: Lowest priority. Assigned when no container in the Pod has any memory or CPU requests set. These pods are the first to be evicted during node pressure.

• `Deployment`: Manages a set of identical, stateless Pods. It’s ideal for web servers or APIs. It handles rolling updates and rollbacks and ensures a specified number of replicas are running. Pods are interchangeable.
• `StatefulSet`: Manages stateful applications that require stable, unique network identifiers and persistent storage. Pods are created and scaled in a predictable, ordered way (e.g., `web-0`, `web-1`) and are not interchangeable. It’s used for databases like ZooKeeper or etcd.
• `DaemonSet`: Ensures that a copy of a Pod runs on all (or a specified subset of) nodes in the cluster. It’s used for cluster-level daemons like logging agents (e.g., Fluentd), monitoring agents (e.g., Prometheus Node Exporter), or network plugins.

• HPA: Scales the number of Pod replicas in a Deployment or ReplicaSet based on observed CPU or memory usage (or custom metrics). It’s for scaling *out* and *in*.
• VPA: Adjusts the CPU and memory `requests` and `limits` for the containers in a Pod to match actual usage. It scales *up* and *down*. A key limitation is that it often requires restarting the Pod to apply the new resource values.

They cannot be used together on the same metrics (CPU/memory) for the same set of pods.

• Taints and Tolerations work together to ensure that pods are not scheduled onto inappropriate nodes. A `Taint` is applied to a node to repel pods. A `Toleration` is applied to a pod to allow it to be scheduled on a node with a matching taint. It’s a “push” mechanism from the node’s perspective.
• Node Affinity is a property of a pod that “pulls” it towards a set of nodes with certain labels. `requiredDuringSchedulingIgnoredDuringExecution` is a hard requirement, while `preferredDuringSchedulingIgnoredDuringExecution` is a soft preference.

In short, taints/tolerations are about “repelling” pods from nodes, while node affinity is about “attracting” pods to nodes.

A Pod Disruption Budget limits the number of pods of a replicated application that are simultaneously down from voluntary disruptions (like a node drain for maintenance or a cluster upgrade). It does *not* protect against involuntary disruptions like a hardware failure.

You can specify either `minAvailable` or `maxUnavailable` to ensure that your application maintains a certain level of availability during planned maintenance activities.

Init containers are special containers that run to completion *before* the main application containers in a Pod are started. They are executed sequentially. If any init container fails, the Pod is restarted.

They are used for setup tasks that must complete before the main app starts, such as:

• Waiting for a dependent service (like a database) to become available.
• Fetching configuration or secrets from an external source.
• Performing database migrations or setting up schema.

• `ClusterIP` (default): Exposes the Service on an internal IP address within the cluster. This makes the Service reachable only from within the cluster. This is the standard type for internal service-to-service communication.
• `NodePort`: Exposes the Service on a static port on each node’s IP address. A `ClusterIP` service is automatically created as well. This allows you to access the service from outside the cluster by targeting `:`.
• `LoadBalancer`: Exposes the Service externally using a cloud provider’s load balancer (e.g., an ELB on AWS). The cloud provider creates a load balancer that forwards traffic to the Service’s `NodePort`. This is the standard way to expose a service to the internet.

An **`Ingress`** is an API object that manages external access to services within a cluster, typically HTTP(S). It provides Layer 7 routing capabilities like host-based or path-based routing.

An Ingress is not a service itself. It requires an **Ingress Controller** (e.g., NGINX, Traefik) to be running in the cluster to fulfill the Ingress rules. A `LoadBalancer` Service works at Layer 4 and simply exposes a single service on a dedicated IP address. An Ingress Controller, often exposed via a single `LoadBalancer` service, can manage routing for *many* different services from a single entry point, which is much more efficient and cost-effective.

A **CNI (Container Network Interface)** plugin is responsible for providing networking for containers. When the `kubelet` creates a pod, it calls the CNI plugin to wire up the pod’s network interface, assign it an IP address, and connect it to the cluster network so it can communicate with other pods.

Kubernetes does not have a built-in network solution; it relies on these third-party plugins. Two popular examples are:

• Calico: A popular plugin known for its high performance and its support for fine-grained Network Policies.
• Flannel: A simpler overlay network that is easy to set up and is a good choice for less complex environments.

A **Network Policy** is a specification of how groups of pods are allowed to communicate with each other and other network endpoints. They act as a firewall at the pod level.

By default, all pods in a cluster can communicate with all other pods. Network Policies are crucial for security because they allow you to implement a “zero-trust” network model. You can create rules to enforce segmentation, for example, allowing pods in the `frontend` namespace to talk to pods in the `backend` namespace on a specific port, while denying all other traffic.

`kube-proxy` is a network proxy that runs on every node in the cluster. Its job is to maintain network rules on nodes that allow for network communication to your Pods from network sessions inside or outside of your cluster. It watches the API server for changes to Service and Endpoint objects and updates the networking rules (e.g., using `iptables` or `IPVS`) on the node to forward traffic sent to a Service’s virtual IP to the correct backend Pods.

A Headless Service is a regular service where you explicitly set the `clusterIP` to `None`. This type of service does not get a stable `ClusterIP`. Instead, when you perform a DNS lookup for the service, the DNS server returns a list of the IP addresses of all the individual Pods that back the service. This is useful for stateful applications (often used with `StatefulSets`) where you need to connect directly to a specific pod instance rather than a random one behind a virtual IP.

• A **`PersistentVolume` (PV)** is a piece of storage in the cluster that has been provisioned by an administrator. It’s a resource in the cluster, just like a node.
• A **`PersistentVolumeClaim` (PVC)** is a request for storage by a user. It’s similar to how a Pod consumes node resources. The PVC specifies a size and an access mode. Kubernetes then binds the PVC to a suitable available PV.
• A **`StorageClass`** provides a way for administrators to describe the “classes” of storage they offer. When a PVC requests a certain storage class, a **dynamic provisioner** can automatically create a new PV for it, eliminating the need for pre-provisioning PVs manually.

• `ReadWriteOnce` (RWO): The volume can be mounted as read-write by a single node.
• `ReadOnlyMany` (ROX): The volume can be mounted as read-only by many nodes simultaneously.
• `ReadWriteMany` (RWX): The volume can be mounted as read-write by many nodes simultaneously.

The availability of these modes depends on the underlying storage provider (e.g., an AWS EBS volume only supports RWO, while an NFS share supports RWX).

A `StatefulSet` can use a `volumeClaimTemplates` section in its specification. For each replica in the `StatefulSet`, this template will be used to dynamically provision a unique `PersistentVolumeClaim`. This PVC is then automatically bound to a new `PersistentVolume`. The name of the PVC is stable and tied to the pod’s stable identity (e.g., `data-web-0`). If the pod `web-0` is rescheduled to a new node, it will be re-attached to the same persistent volume, ensuring its state is preserved.

RBAC (Role-Based Access Control) is the standard mechanism for controlling authorization in Kubernetes.

• A **`Role`** contains a set of rules that define permissions (verbs like `get`, `list`, `create`) on specific resources (like `pods`, `deployments`). A `Role` is namespaced.
• A **`ClusterRole`** is the same as a `Role` but is non-namespaced. It can be used to grant permissions on cluster-scoped resources (like `nodes`) or on namespaced resources across all namespaces.
• A **`RoleBinding`** grants the permissions defined in a `Role` to a set of subjects (users, groups, or service accounts) within a specific namespace.
• A **`ClusterRoleBinding`** grants the permissions defined in a `ClusterRole` to subjects across the entire cluster.

A **Service Account** provides an identity for processes that run in a Pod. By default, pods are assigned the `default` service account in their namespace.

It’s a security best practice to create a specific service account for each application and grant it only the minimum required permissions via a `Role` and `RoleBinding`. This follows the principle of least privilege. For example, if an application only needs to list pods, you would create a service account for it and bind it to a Role that only allows the `get` and `list` verbs on the `pods` resource, instead of using the overly permissive default service account.

Pod Security Policies (PSPs) were a cluster-level resource for controlling security-sensitive aspects of a pod’s specification, such as running as root or accessing the host network. They were complex to manage and were deprecated in version 1.21 and removed in 1.25.

They have been replaced by **Pod Security Admission (PSA)**. PSA is a built-in admission controller that enforces the Pod Security Standards (`Privileged`, `Baseline`, `Restricted`) at the namespace level. It’s simpler to use and is now the standard way to enforce pod security in a cluster.

Kubernetes `Secrets` provide a way to store sensitive data like passwords or API keys. However, they are only Base64 encoded, not encrypted by default in `etcd`. Best practices include:

1. Enable Encryption at Rest: Configure encryption for secrets in `etcd`.
2. Use RBAC: Restrict `get`, `list`, and `watch` access to Secrets to only the pods and users that absolutely need them.
3. External Secrets Management: The most secure approach is to use an external secrets store like HashiCorp Vault or AWS/GCP Secret Manager. You can use a tool like the External Secrets Operator to sync these external secrets into Kubernetes `Secret` objects securely.

The **Operator Pattern** is a method of packaging, deploying, and managing a Kubernetes application. An Operator is a custom controller that uses the Kubernetes API to manage a complex, stateful application on behalf of a human operator.

It encodes domain-specific operational knowledge into software. For example, a Prometheus Operator would know how to deploy a Prometheus instance, set up its configuration, scale it, and handle backups automatically. It achieves this by defining its own **Custom Resource Definitions (CRDs)** (e.g., a `Prometheus` resource) and running a reconciliation loop to ensure the live state matches the state defined in the custom resource.

A CRD is a way to extend the Kubernetes API by adding your own custom object types. Once you create a CRD, the Kubernetes API server will create a new RESTful resource path for it, and you can manage these custom objects using `kubectl`, just like built-in objects like `Pods` and `Deployments`. CRDs are the foundation of the Operator Pattern.

A **Service Mesh** is a dedicated infrastructure layer for making service-to-service communication safe, fast, and reliable. It works by injecting a “sidecar” proxy (like Envoy) alongside each application container. All network traffic between services is routed through these proxies.

This allows you to get powerful features without changing your application code, such as:

• Observability: Automatic metrics, logs, and traces for all traffic.
• Traffic Management: Sophisticated routing rules, canary deployments, and A/B testing.
• Security: Automatic mutual TLS (mTLS) encryption for all traffic and fine-grained authorization policies.

A Pod in the `Pending` state means it has been accepted by the cluster but cannot be scheduled onto a node or its container images cannot be pulled. Common reasons include:

• Insufficient Resources: The cluster doesn’t have enough available CPU or memory to satisfy the Pod’s `requests`.
• Scheduling Constraints: The Pod cannot be scheduled due to taints/tolerations, node affinity rules, or other constraints.
• PV/PVC Issues: The Pod is waiting for a `PersistentVolumeClaim` to be bound.

I would start by using `kubectl describe pod ` to look at the Events section, which usually provides a clear reason from the scheduler.

A `CrashLoopBackOff` status means that a container in the Pod is starting, crashing, and then being restarted by Kubernetes, over and over again. To troubleshoot:

1. Use `kubectl logs ` to check the application’s logs for any error messages.
2. If the crash is too fast to capture logs, use `kubectl logs –previous` to get the logs from the previous failed container instance.
3. Use `kubectl describe pod ` to check for issues like incorrect configuration, command arguments, or failed probes.

Helm is a package manager for Kubernetes. It allows you to define, install, and upgrade complex Kubernetes applications as a single, versioned unit called a “Chart.” A Helm Chart is a collection of templated YAML files. It solves the problem of managing and deploying applications with many interdependent Kubernetes resources, making deployments repeatable and manageable.

A rolling update is the default deployment strategy for a `Deployment` object. When you update the Pod template (e.g., by changing the image tag), the Deployment controller gradually replaces old Pods with new ones.

You can control this process with `maxSurge` (how many extra pods can be created above the desired count) and `maxUnavailable` (how many pods can be unavailable during the update). This ensures that the update happens with zero downtime and that application availability is maintained.

The Downward API allows you to expose metadata about a Pod and the cluster environment to the containers running inside that Pod. You can expose this information as either environment variables or files mounted into the container. This is useful for applications that need to be self-aware of their own identity (like their name, namespace, or IP address) within the Kubernetes environment.

You can use a `ResourceQuota` object. A `ResourceQuota` provides constraints that limit the aggregate resource consumption for all objects within a namespace. You can set quotas for:

• Total compute resources (CPU and memory `requests` and `limits`).
• Total storage resources (number of PVCs or total storage capacity).
• Total object counts (number of pods, services, secrets, etc.).

A finalizer is a special key in an object’s metadata that tells Kubernetes to wait until specific cleanup actions are complete before fully deleting the object. When you try to delete an object with a finalizer, the object is put into a “terminating” state. A controller that is responsible for that finalizer must perform its cleanup logic (e.g., deleting external resources associated with the object) and then remove the finalizer key from the object. Once all finalizers are removed, Kubernetes will complete the deletion.

The cluster autoscaler is a component that automatically adjusts the size of your Kubernetes cluster by adding or removing nodes. It works by periodically checking for:

• Pods in a `Pending` state that cannot be scheduled due to insufficient resources. If it finds any, it will add a new node to the cluster.
• Nodes that are underutilized for a certain period. If a node’s resource utilization is below a threshold and its pods can be safely rescheduled elsewhere, the cluster autoscaler will drain the node and terminate it.

This policy is used for pods that are running with `hostNetwork: true`. By default, such pods would use the host’s DNS settings. Setting `dnsPolicy` to `ClusterFirstWithHostNet` explicitly tells the pod to use Kubernetes’ internal DNS service (for resolving cluster services like `myservice.mynamespace.svc.cluster.local`) first, and then fall back to the host’s DNS configuration if the lookup fails. This allows a pod to benefit from both direct access to the host network and the ability to discover services within the cluster.