Kubernetes has become the de facto standard for containerized application deployment, but its rise also attracts malicious actors. Kubernetes security is now a top priority as the attack surface expands with increasing adoption. This comprehensive guide explores the state of Kubernetes security, analyzing the major risks, vulnerabilities, and essential best practices to fortify your cloud-native infrastructure.
[lwptoc]
The explosive growth of Kubernetes has not been without consequences - it is increasingly becoming a target for various attacks. The situation is compounded by the fact that a typical Kubernetes cluster accumulates many components necessary for its full operation. This integration complicates the infrastructure, expanding the range of directions for attacks by malicious actors.
According to the Red Hat 2023 State of Kubernetes Security Report, security issues have a significant impact on businesses:
Slowed Deployments: 67% of organizations reported delaying or halting deployments due to security concerns. Security becoming an afterthought negates the agility benefits of containerization.
Financial Losses: 37% of companies experienced revenue or customer loss due to a Kubernetes security incident. Security breaches can disrupt critical projects and damage customer trust.
Employee Impact: Security incidents can even lead to employee terminations (21%) and organizational fines (25%).
The report highlights the pervasiveness of security threats throughout the application lifecycle:
All Phases Affected: While runtime is the most common attack point (49%), the build and deployment phases are almost equally vulnerable (nearly 45%).
Security Misconfigurations: A worrying 45% of respondents admitted to experiencing misconfiguration incidents, highlighting the need for robust security practices.
Unpatched Vulnerabilities: Another 42% discovered major vulnerabilities requiring remediation, indicating a lack of thorough security testing.
What Are the Main Kubernetes Security Risks?
Container Escape is one of the most frequently exploited vulnerabilities in Kubernetes. Implementing security standards for pods (e.g. seccomp mode enabled by default starting in Kubernetes 1.22) and using Linux security modules like AppArmor and SELinux can help mitigate this risk.
The 2023 Sysdig Cloud-Native Security and Usage Report is based on data collected from billions of containers, thousands of cloud accounts, and hundreds of thousands of applications their customers ran over the past year. It highlights these main security threats:
Supply Chain Risks: 87% of images have high or critical vulnerabilities. Teams must prioritize as there are so many. Interestingly, 85% of critical vulnerabilities have fixes available but don't make it to the container runtime for various reasons. Teams spend time on non-applicable threats while real risks go unaddressed. They recommend prioritizing vulnerabilities that actually impact the runtime environment.
Shortened Container Lifecycles: 72% of containers live less than 5 minutes, up 28% from 44% in 2021. Such short lifespans make debugging applications or infrastructure issues difficult.
Security Issues Continue Impacting Business
67% of companies delayed or slowed deployment due to security concerns. As organizations adopt cloud-native like Kubernetes and microservices architectures, unforeseen security challenges often arise. When security is an afterthought, the agility gained from containerization is negated. Some are overwhelmed by security needs across the application lifecycle.
Security Incidents Impact Employees and Revenue
21% of respondents said an incident led to employee termination, and over a third experienced revenue or customer loss. Compliance violations or data breaches can result in talent loss, experience loss, regulatory fines, and negative publicity. 37% cited revenue/customer loss from a Kubernetes security incident, which could delay projects, product releases, and lose market share.
Incidents Impact All Lifecycle Phases
90% experienced at least one security incident in the last year across all phases - 49% in runtime, but nearly as many in build/deploy. Kubernetes prioritized developer productivity over security, so hardening like SELinux is challenging to customize and integrate. 45% had a misconfiguration incident, 42% a major vulnerability, and 27% failed an audit.
Security Remains a Top Concern
38% think security investment is inadequate as containers/Kubernetes add complexity. Containers emphasize agility, so security testing may not be prioritized as much as deployment speed. Properly investing means understanding Kubernetes risks and implementing controls across all layers.
Decentralized Security Responsibility
Less than a third consider the security team responsible for Kubernetes security. Securing Kubernetes requires different teams to own parts of the lifecycle - DevOps for infrastructure, security for policies/controls, developers for app/image security, operations for access controls, etc.
Embracing DevSecOps
45% have advanced DevSecOps integration with automated security practices across the lifecycle. 39% more are early-stage. However, 17% still separate DevOps and security, likely making them more reactive.
Top Concerns: Misconfigurations and Vulnerabilities
Over 50% worry about misconfigurations and vulnerabilities due to Kubernetes customizability. The complex environments with rapid scaling make consistent security posture challenging. The shared kernel means one vulnerability can impact multiple containers/hosts. Automating security scanning for common issues like privileged containers, vulnerable dependencies, and insecure defaults can mitigate risks.
The Majority Address Misconfiguration Concerns
Exposed/unprotected sensitive data is the most worrying misconfiguration at 32%. With configuration a leading cause for concern, respondents did not single out any one misconfiguration as significantly more worrisome than others. This underscores the need for a comprehensive approach to securing all Kubernetes components. Encouragingly, organizations are taking steps to address risks - 75% who worried about default configurations are working to remediate them.
Consequences Can Be Serious
Ransomware from misconfigurations is the top cited concern at 40%. Human error behind most breaches can increase breach cost and detection time. 41% worry most about ransomware, and 53% of those experienced a ransomware attack last year. For each cited consequence, a larger percentage had actually experienced it, like 46% with data deletion despite only 34% worried about it. This may indicate being inundated with concerns, lack of resources, or decentralized security ownership.
Software Supply Chain Risks
35% most worry about supply chain software vulnerabilities. Supply chain security is a hot topic after incidents like SolarWinds and Log4Shell. Respondents are concerned about vulnerabilities and open source usage, which are understandable risks if components contain flaws or are unmaintained.
Their concerns are well-founded - over half experienced virtually every supplied chain issue, with vulnerable components at 69% and CI/CD weaknesses at 68% most common.
Scanning and Attestation Are Key Controls
Nearly half identified artifact signing as most important for supply chain security. Attestation ensures software meets standards and is uncompromised, building trust across the supply chain participants like internal teams, vendors, and open source.
Scanning and attestation are two of the most important security controls for software supply chain security:
47% Vulnerability scanning
43% Security attestation (image signing, deployment signing, pipeline attestation, etc.)
40% Access and authentication
34% Configuration management
31% CI/CD integration and security automation
29% Registry governance
20% IDE scanning
Popular Open Source Security Tools
KubeLinter at 37% and Kube-hunter at 32% are the top used open source Kubernetes security tools. Open Policy Agent, a policy engine for Kubernetes and more, ties Kube-hunter at 32% usage.
37% KubeLinter
32% Kube-hunter
32% Open Policy Agent (OPA)
29% Kube-bench
19% Falco
14% Terrascan
14% StackRox
10% Clair
9% Kyverno
8% Checkov
Common Mistakes in Kubernetes Security
The Fairwinds Security, Cost and Reliability Workload Results report evaluates statistics from over 150,000 workloads across hundreds of companies using Fairwinds Insights. Its authors also believe Kubernetes security is deteriorating. The stats show users often neglect best practices:
Enabled Risky Linux Capabilities: Like CHOWN, DAC_OVERRIDE, FSETID, FOWNER etc. Some are enabled by default though most workloads don't need them. Only 10% of companies disabled them in 2022, down from 42% in 2021. A third run almost all workloads in an insecure mode. This is compounded by security-enhancing settings like allowPrivilegeEscalation being disabled by default.
44% run 71%+ of workloads as root, up 22% from last year - puzzling given the CVE count in this area.
62% of organizations have at least half their workloads potentially vulnerable to image vulnerabilities, up significantly.
Outdated Helm Charts: 46% have half or more workloads on outdated Helm charts, up from 33%.
Privilege Excess: Sysdig data shows 90% of privileges go unused despite cloud security and zero trust best practices.
Have Security Concerns About Your Kubernetes Environment? Contact Gart Solutions for Expert DevOps and Cloud-Native Security Guidance.
How Can You Improve Kubernetes Security?
When security is an afterthought, organizations risk negating the core benefit of faster application development and deployment by not ensuring their cloud-native environments are built, deployed, and managed securely. Our findings show events in the build and deploy stages significantly impact security, underscored by the prevalence of misconfigurations and vulnerabilities across organizations. Security must therefore shift left, imperceptibly embedding into DevOps workflows instead of being "bolted on" when an application nears production deployment.
Use Kubernetes-Native Security Architectures and Controls
Kubernetes-native security leverages the rich declarative data and native controls in Kubernetes to deliver key security benefits. Analyzing Kubernetes' declarative data yields better risk-based insights into configuration management, compliance, segmentation, and Kubernetes-specific vulnerabilities. Using the same infrastructure and controls for application development and security reduces the learning curve and supports faster analysis and troubleshooting. It also eliminates operational conflicts by ensuring security gains the same automation and scalability advantages Kubernetes provides infrastructure.
Security Across the Full Lifecycle
Security has long been viewed as inhibiting business, especially by developers and DevOps teams mandated to deliver code quickly. With containers and Kubernetes, security should accelerate business by helping developers build strong security into assets from the start. Look for a platform that incorporates DevOps best practices and internal controls into its configuration checks. It should also assess Kubernetes' configuration for a secure posture, so developers can focus on feature delivery.
Bridge DevOps and SecOps
Given no clear role or team is solely responsible for container/Kubernetes security at most organizations, your tooling must bridge teams from Security and Ops to DevOps and Development. To be effective, the platform must have security controls relevant to containerized, Kubernetes environments. It should also assess risk appropriately - telling a developer to fix all 39 vulnerabilities with a CVSS score ≥7 is inefficient. Identifying the 3 deployments exposed to that vulnerability, showing why they're risky, will significantly improve posture.
Other Tips:
Follow least privilege when assigning roles/privileges to users and service accounts to limit attacker ability to gain excessive access if breached. Use specialized tooling to find and remove excessive privileges.
Use defense in depth techniques to hinder lateral movement and data exfiltration by attackers.
Continuously scan manifest files, registries, and clusters for vulnerabilities.
Regularly update cluster software. After vulnerability disclosures, attackers hunt for unpatched clusters. The Log4J vulnerability saw over 840,000 attempted exploits within 72 hours of disclosure.
Conclusion
As a leading DevOps provider, Gart Solutions specializes in helping organizations bolster their Kubernetes security posture and implement robust best practices across the entire cloud-native stack. Our team of seasoned experts can assess your current security risks, vulnerabilities, and misconfigurations, then provide tailored solutions to safeguard your containerized applications and Kubernetes clusters.
Don't leave your business exposed to emerging threats. Reach out to Gart Solutions today to ensure your mission-critical cloud-native infrastructure remains secure and compliant in 2024 and beyond.
Containerization has revolutionized the way applications are developed, deployed, and managed. It allows for greater scalability, flexibility, and efficiency in running software across different environments. Two of the most popular container orchestration platforms are OpenShift and Kubernetes.
Both OpenShift and Kubernetes offer similar benefits, such as automating container deployment, scaling, and management, as well as providing high availability and fault tolerance. However, there are some differences in terms of features, capabilities, and ease of use.
In this blog post, we will delve deeper into the core features and components of OpenShift vs Kubernetes, their architectures, and the advantages they offer for container orchestration. We will also compare and contrast the two platforms, exploring their key differences and use cases where each excels. This comparison will provide insights into choosing the right platform based on specific requirements and project goals.
Comparison Table OpenShift vs Kubernetes
FeatureOpenShiftKubernetesContainer OrchestrationYesYesPlatform ProviderRed HatCloud Native Computing Foundation (CNCF)InstallationEasy, with OpenShift installerRequires manual setupManagementRobust management capabilitiesBasic management functionalitiesScalingBuilt-in scaling featuresScaling options availableDeploymentAutomated deployment processesManual deployment processesSecurityAdvanced security featuresBasic security featuresAccess ControlRole-based access control (RBAC)Role-based access control (RBAC)Community SupportSupported by Red Hat communityBroad community supportEcosystemIntegrated with Red Hat ecosystemBroad ecosystem of tools and servicesUse CasesEnterprise deploymentsGeneral-purpose deploymentsTable comparing OpenShift and Kubernetes
Understanding OpenShift
OpenShift is a container platform developed by Red Hat, a leading provider of open-source solutions. It aims to provide a comprehensive platform for organizations to develop, deploy, and manage containerized applications efficiently.
OpenShift originated from Red Hat's earlier Platform-as-a-Service (PaaS) offering called OpenShift Online, which was released in 2011. Over time, OpenShift evolved into a more robust and flexible platform that embraced containerization and adopted Kubernetes as its orchestration engine. Today, it is available as OpenShift Container Platform for on-premises deployments and OpenShift Dedicated for managed cloud environments.
Core features and components of OpenShift
OpenShift offers a rich set of features and components that enhance the Kubernetes experience. Some of the core features and components include:
Container Runtime
Build and Deployment Automation
Integrated Developer Experience
Image Registry
Service Mesh
OpenShift supports multiple container runtimes, including Docker and CRI-O, providing flexibility in choosing the underlying technology.
OpenShift includes tools for automating the build and deployment processes, such as Source-to-Image (S2I) and Kubernetes-native build capabilities.
OpenShift provides developers with a user-friendly interface and tools for easy application development, testing, and collaboration.
OpenShift includes an integrated image registry to store and manage container images, allowing for versioning, security scanning, and efficient image distribution.
OpenShift integrates with popular service mesh solutions like Istio, enabling advanced traffic management, security, and observability for microservices architectures.
Overview of OpenShift's architecture
OpenShift's architecture is built on top of Kubernetes and extends it with additional components. At its core, OpenShift employs the same concepts of pods, services, and deployments as Kubernetes, but it introduces higher-level abstractions to simplify application management. These abstractions include Projects, Applications, and Routes.
OpenShift uses a master-node architecture where the master components handle the orchestration and management of the cluster, and the worker nodes execute the containers. The master components include the API server, controller manager, and scheduler, while the worker nodes host the pods and execute the containerized applications.
Advantages of using OpenShift for container orchestration
OpenShift provides an intuitive web-based console and command-line tools that simplify application development, testing, and collaboration, making it easier for developers to work with containers.
It's integrated tools for build automation, deployment, and scaling help streamline operations and reduce manual effort, enabling faster application delivery. Also OpenShift offers built-in security features such as role-based access control (RBAC), image scanning, and network policies, ensuring secure container environments. It also helps organizations comply with regulatory requirements.
OpenShift leverages Kubernetes' scaling capabilities to scale applications horizontally based on demand, ensuring high availability and efficient resource utilization.
OpenShift provides a robust platform for deploying and managing complex enterprise applications, enabling seamless scaling, fault tolerance, and easy integration with existing systems.
It's automation capabilities, integrated tools, and container-based workflows make it an ideal platform for implementing DevOps practices and CI/CD pipelines.
Understanding Kubernetes
In 2014, Google released Kubernetes as an open-source project, allowing organizations to benefit from its container orchestration capabilities. Since then, Kubernetes has gained immense popularity and has become the industry standard for managing containers at scale.
Kubernetes offers a comprehensive set of features and components that enable efficient container orchestration. Some of the key features and components of Kubernetes include:
Pod
The basic building block of Kubernetes, a pod represents one or more containers that are tightly coupled and share the same resources, such as network and storage.
Service
Kubernetes services provide a stable network endpoint to access pods, allowing for load balancing and service discovery within the cluster.
Replication Controller/ReplicaSet
These components manage the desired number of pod replicas, ensuring high availability and fault tolerance by automatically scaling and replacing failed instances.
Deployment
Deployments provide declarative updates to pods and replica sets, enabling seamless rollouts and rollbacks of application changes.
Namespace
Kubernetes namespaces provide a logical separation and isolation of resources within a cluster, allowing multiple teams or projects to share the same cluster securely.
Overview of Kubernetes' architecture
Kubernetes follows a master-worker architecture, where the master components manage and control the cluster, while the worker nodes execute the containers. The key components of the Kubernetes architecture include:
API Server: The central control point for managing and interacting with the cluster. It exposes the Kubernetes API, allowing users to control the cluster and its resources.
Controller Manager: This component ensures that the desired state of the cluster matches the actual state by monitoring and managing various controllers, such as the ReplicaSet controller and Deployment controller.
Scheduler: The scheduler assigns pods to worker nodes based on resource requirements, constraints, and availability, ensuring efficient utilization of cluster resources.
etcd: A distributed key-value store that stores the cluster's configuration data, providing reliable data storage and consistency for the entire cluster.
Kubelet: The primary agent running on each worker node, responsible for managing and executing containers as part of pods.
Advantages of using Kubernetes for container orchestration
Kubernetes allows organizations to scale applications horizontally by adding or removing containers, ensuring high availability and efficient resource utilization. Kubernetes' declarative approach to deployments enables seamless application rollouts and rollbacks, reducing the risk of errors during updates.
This project monitors the health of containers and automatically restarts or replaces failed instances, ensuring the continuous availability of applications. Kubernetes optimizes resource allocation, allowing multiple applications to run efficiently on a cluster while effectively managing resource utilization.
Kubernetes is cloud-agnostic, enabling applications to run consistently across different cloud providers or on-premises environments, promoting portability and avoiding vendor lock-in.
Use cases and industries where Kubernetes excels
It's important to note that Kubernetes' flexibility and scalability make it applicable to a wide range of industries and use cases.
As an example, let's take a look at how GART implemented IoT device management using Kubernetes. They leveraged Kubernetes to effectively manage and scale their IoT edge devices. By utilizing Kubernetes, they achieved seamless deployment of containerized applications on edge devices, centralized management and monitoring of the devices, and streamlined software updates across their IoT network. For more details about their use case, you can visit the following link: IoT Device Management using Kubernetes.
What Are the Similarities Between OpenShift and Kubernetes?
Container Orchestration
Scalability and High Availability
API-Driven Architecture
Container Runtime Compatibility
Both OpenShift and Kubernetes excel in container orchestration, providing the ability to automate the deployment, scaling, and management of containerized applications.
Both platforms offer features for scaling applications horizontally, ensuring high availability and efficient resource utilization.
OpenShift and Kubernetes follow an API-driven approach, allowing users to interact with the platform and manage resources through a unified API.
Both OpenShift and Kubernetes support popular container runtimes such as Docker, allowing flexibility in choosing the underlying technology.
What Are the Key Differences in Features and Capabilities?
OpenShift provides additional features and tools beyond Kubernetes, including integrated build automation, image registry, service mesh integration, and a user-friendly developer interface.
OpenShift focuses on providing a more user-friendly experience with its web-based console and streamlined developer workflows, making it easier for developers to work with containers compared to Kubernetes.
OpenShift originated as a PaaS offering and retains some of its PaaS capabilities, providing higher-level abstractions and simplifications for application development and deployment.
Container management and orchestration capabilities
OpenShift builds upon Kubernetes and enhances its container management capabilities with additional tools and features tailored for enterprise environments.
Kubernetes provides a robust and extensible platform for container management and orchestration, offering a wide range of capabilities and a large ecosystem of tools and extensions.
Deployment and scaling options
OpenShift provides a user-friendly interface and integrated tools for automating the build and deployment process, allowing for easy application scaling and rollouts.
Kubernetes offers various deployment options, including Deployments and StatefulSets, enabling declarative updates and efficient scaling of applications.
Resource management and monitoring
OpenShift provides built-in monitoring capabilities, including logging, metrics, and health checks, allowing for effective resource management and troubleshooting.
Kubernetes offers resource management features, such as resource quotas and limits, enabling organizations to efficiently allocate and manage resources within the cluster.
User interface and ease of use
OpenShift focuses on providing a user-friendly web-based console and developer-friendly tools, making it easier for users to navigate and interact with the platform.
Kubernetes offers a command-line interface (CLI) and a web-based dashboard for managing and monitoring the cluster, but it may require a steeper learning curve compared to OpenShift.
Community support and ecosystem
OpenShift benefits from the strong community support of Kubernetes, as it is built upon Kubernetes. Additionally, OpenShift has its own active community and ecosystem, providing extensions, integrations, and support resources.
Kubernetes has a vast and active open-source community, with a wide range of tools, extensions, and documentation available. It enjoys broad industry adoption and has a mature ecosystem.
Conclusion: OpenShift vs Kubernetes
When comparing OpenShift vs Kubernetes, it's important to consider your specific needs and requirements. OpenShift offers additional enterprise-focused features, while Kubernetes provides a robust foundation for container orchestration. Both platforms excel in their own ways, and the choice ultimately depends on factors such as scalability, ease of use, and integration capabilities. By understanding the similarities and differences between OpenShift and Kubernetes, you can make an informed decision that aligns with your container orchestration goals.
Kubernetes is becoming the standard for development, while the entry barrier remains quite high. We've compiled a list of recommendations for application developers who are migrating their apps to the orchestrator. Knowing the listed points will help you avoid potential problems and not create limitations where Kubernetes has advantages. Kubernetes Migration.
Who is this text for?
It's for developers who don't have DevOps expertise in their team – no full-time specialists. They want to move to Kubernetes because the future belongs to microservices, and Kubernetes is the best solution for container orchestration and accelerating development through code delivery automation to environments. At the same time, they may have locally run something in Docker but haven't developed anything for Kubernetes yet.
Such a team can outsource Kubernetes support – hire a contractor company, an individual specialist, or, for example, use the relevant services of an IT infrastructure provider. For instance, Gart Solutions offers DevOps as a Service – a service where the company's experienced DevOps specialists will take on any infrastructure project under full or partial supervision: they'll move your applications to Kubernetes, implement DevOps practices, and accelerate your time-to-market.
Regardless, there are nuances that developers should be aware of at the architecture planning and development stage – before the project falls into the hands of DevOps specialists (and they force you to modify the code to work in the cluster). We highlight these nuances in the text.
The text will be useful for those who are writing code from scratch and planning to run it in Kubernetes, as well as for those who already have a ready application that needs to be migrated to Kubernetes. In the latter case, you can go through the list and understand where you should check if your code meets the orchestrator's requirements.
The Basics
At the start, make sure you're familiar with the "golden" standard The Twelve-Factor App. This is a public guide describing the architectural principles of modern web applications. Most likely, you're already applying some of the points. Check how well your application adheres to these development standards. For the infrastructure side of the move, from audit to cutover, see our step-by-step Kubernetes migration guide.
The Twelve-Factor App
Codebase: One codebase, tracked in a version control system, serves all deployments.
Dependencies: Explicitly declare and isolate dependencies.
Configuration: Store configuration in the environment - don't embed in code.
Backing Services: Treat backing services as attached resources.
Build, Release, Run: Strictly separate the build, release, and run stages.
Processes: Run the application as one or more stateless processes.
Port Binding: Expose services through port binding.
Concurrency: Scale the application by adding processes.
Disposability: Maximize reliability with fast startup and graceful shutdown.
Dev/Prod Parity: Keep development, staging, and production environments as similar as possible.
Logs: Treat logs as event streams.
Admin Processes: Run administrative tasks as one-off processes.
Now let's move on to the highlighted recommendations and nuances.
Prefer Stateless Applications
Why?
Implementing fault tolerance for stateful applications will require significantly more effort and expertise.
Normal behavior for Kubernetes is to shut down and restart nodes. This happens during auto-healing when a node stops responding and is recreated, or during auto-scaling down (e.g., some nodes are no longer loaded, and the orchestrator excludes them to save resources).
Since nodes and pods in Kubernetes can be dynamically removed and recreated, the application should be ready for this. It should not write any data that needs to be preserved to the container it is running in.
What to do?
You need to organize the application so that data is written to databases, files to S3 storage, and, say, cache to Redis or Memcache. By having the application store data "on the side," we significantly facilitate cluster scaling under load when additional nodes need to be added, and replication.
In a stateful application where data is stored in an attached Volume (roughly speaking, in a folder next to the application), everything becomes more complicated. When scaling a stateful application, you'll have to order "volumes," ensure they're properly attached and created in the right zone. And what do you do with this "volume" when a replica needs to be removed?
Yes, some business applications should be run as stateful. However, in this case, they need to be made more manageable in Kubernetes. You need an Operator, specific agents inside performing the necessary actions... A prime example here is the postgres-operator. All of this is far more labor-intensive than simply throwing the code into a container, specifying five replicas, and watching it all work without additional dancing.
Ensure Availability of Endpoints for Application Health Checks
Why?
We've already noted that Kubernetes itself ensures the application is maintained in the desired state. This includes checking the application's operation, restarting faulty pods, disabling load, restarting on less loaded nodes, terminating pods exceeding the set resource consumption limits.
For the cluster to correctly monitor the application's state, you should ensure the availability of endpoints for health checks, the so-called liveness and readiness probes. These are important Kubernetes mechanisms that essentially poke the container with a stick – check the application's viability (whether it's working properly).
What to do?
The liveness probes mechanism helps determine when it's time to restart the container so the application doesn't get stuck and continues to work.
Readiness probes are not so radical: they allow Kubernetes to understand when the container is ready to accept traffic. In case of an unsuccessful check, the pod will simply be excluded from load balancing – no new requests will reach it, but no forced termination will occur.
You can use this capability to allow the application to "digest" the incoming stream of requests without pod failure. Once a few readiness checks pass successfully, the replica pod will return to load balancing and start receiving requests again. From the nginx-ingress side, this looks like excluding the replica's address from the upstream.
Such checks are a useful Kubernetes feature, but if liveness probes are configured incorrectly, they can harm the application's operation. For example, if you try to deploy an application update that fails the liveness/readiness checks, it will roll back in the pipeline or lead to performance degradation (in case of correct pod configuration). There are also known cases of cascading pod restarts, when Kubernetes "collapses" one pod after another due to liveness probe failures.
You can disable checks as a feature, and you should do so if you don't fully understand their specifics and implications. Otherwise, it's important to specify the necessary endpoints and inform DevOps about them.
If you have an HTTP endpoint that can be an exhaustive indicator, you can configure both liveness and readiness probes to work with it. By using the same endpoint, make sure your pod will be restarted if this endpoint cannot return a correct response.
Aim for More Predictable and Uniform Application Resource Consumption
Why?
Almost always, containers inside Kubernetes pods are resource-limited within certain (sometimes quite small) values. Scaling in the cluster happens horizontally by increasing the number of replicas, not the size of a single replica. In Kubernetes, we deal with memory limits and CPU limits. Mistakes in CPU limits can lead to throttling, exhausting the available CPU time allotted to the container. And if we promise more memory than is available, as the load increases and hits the ceiling, Kubernetes will start evicting the lowest priority pods from the node. Of course, limits are configurable. You can always find a value at which Kubernetes won't "kill" pods due to memory constraints, but nevertheless, more predictable resource consumption by the application is a best practice. The more uniform the application's consumption, the tighter you can schedule the load.
What to do
Evaluate your application: estimate approximately how many requests it processes, how much memory it occupies. How many pods need to be run for the load to be evenly distributed among them? A situation where one pod consistently consumes more than the others is unfavorable for the user. Such a pod will be constantly restarted by Kubernetes, jeopardizing the application's fault-tolerant operation. Expanding resource limits for potential peak loads is also not an option. In that case, resources will remain idle whenever there is no load, wasting money. In parallel with setting limits, it's important to monitor pod metrics. This could be kube-Prometheus-Stack, VictoriaMetrics, or at least Metrics Server (more suitable for very basic scaling; in its console, you can view stats from kubelet—how much pods are consuming). Monitoring will help identify problem areas in production and reconsider the resource distribution logic.
There is a rather specific nuance regarding CPU time that Kubernetes application developers should keep in mind to avoid deployment issues and having to rewrite code to meet SRE requirements. Let's say a container is allocated 500 milli-CPUs—roughly 0.5 CPU time of a single core over 100 ms of real time. Simplified, if the application utilizes CPU time in several continuous threads (let's say four) and "eats up" all 500 milli-CPUs in 25 ms, it will be frozen by the system for the remaining 75 ms until the next quota period. Staging databases run in Kubernetes with small limits exemplify this behavior: under load, queries that would normally take 5 ms plummet to 100 ms. If the response graphs show load continuously increasing and then latency spiking, you've likely encountered this nuance. Address resource management—give more resources to replicas or increase the number of replicas to reduce load on each one.
Get a sample of IT Audit
Sign up now
Get on email
Loading...
Thank you!
You have successfully joined our subscriber list.
Leverage Kubernetes ConfigMaps, Secrets, and Environment Variables
Kubernetes has several objects that can greatly simplify life for developers. Study them to save yourself time in the future.
ConfigMaps are Kubernetes objects used to store non-confidential data as key-value pairs. Pods can use them as environment variables or as configuration files in volumes. For example, you're developing an application that can run locally (for direct development) and, say, in the cloud. You create an environment variable for your app—e.g., DATABASE_HOST—which the app will use to connect to the database. You set this variable locally to localhost. But when running the app in the cloud, you need to specify a different value—say, the hostname of an external database.
Environment variables allow using the same Docker image for both local use and cloud deployment. No need to rebuild the image for each individual parameter. Since the parameter is dynamic and can change, you can specify it via an environment variable.
The same applies to application config files. These files store certain settings for the app to function correctly. Usually, when building a Docker image, you specify a default config or a config file to load into Docker. Different environments (dev, prod, test, etc.) require different settings that sometimes need to be changed, for testing, for example.
Instead of building separate Docker images for each environment and config, you can mount config files into Docker when starting the pod. The app will then pick up the specified configuration files, and you'll use one Docker image for the pods.
Typically, iftheconfig files are large,youuse a Volume to mount them into Docker as files. If the config files contain short values, environment variables are more convenient. It all depends on yourapplication's requirements.
Another useful Kubernetes abstraction is Secrets. Secrets are similar to ConfigMaps but are meant for storing confidential data—passwords, tokens, keys, etc. Using Secrets means you don't need to include secret data in your application code. They can be created independently of the pods that use them, reducing the risk of data exposure. Secrets can be used as files in volumes mounted in one or multiple pod containers. They can also serve as environment variables for containers.
Disclaimer: In this point, we're only describing out-of-the-box Kubernetes functionality. We're not covering more specialized solutions for working with secrets, such as Vault.
Knowing about these Kubernetes features, a developer won't need to rebuild the entire container if something changes in the prepared configuration, for example, a password change.
Ensure Graceful Container Shutdown with SIGTERM
Why?
There are situations where Kubernetes "kills" an application before it has a chance to release resources. This is not an ideal scenario. It's better if the application can respond to incoming requests without accepting new ones, complete a transaction, or save data to a database before shutting down.
What to Do
A successful practice here is for the application to handle the SIGTERM signal. When a container is being terminated, the PID 1 process first receives the SIGTERM signal, and then the application is given some time for graceful termination (the default is 30 seconds). Subsequently, if the container has not terminated itself, it receives SIGKILL—the forced termination signal. The application should not continue accepting connections after receiving SIGTERM.
Many frameworks (e.g., Django) can handle this out of the box. Your application may already handle SIGTERM appropriately. Ensure that this is the case.
Here are a few more important points:
The Application Should Not Depend on Which Pod a Request Goes To
When moving an application to Kubernetes, we expectedly encounter auto-scaling—you may have migrated for this very reason. Depending on the load, the orchestrator adds or removes application replicas. The application mustn't depend on which pod a client request goes to, for example, static pods. Either the state needs to be synchronized to provide an identical response from any pod, or your backend should be able to work across multiple replicas without corrupting data.
Your Application Should Be Behind a Reverse Proxy and Serve Links Over HTTPS
Kubernetes has the Ingress entity, which essentially provides a reverse proxy for the application, typically an nginx with cluster automation. For the application, it's enough to work over HTTP and understand that the external link will be over HTTPS. Remember that in Kubernetes, the application is behind a reverse proxy, not directly exposed to the internet, and links should be served over HTTPS. When the application returns an HTTP link, Ingress rewrites it to HTTPS, which can lead to looping and a redirect error. Usually, you can avoid such a conflict by simply toggling a parameter in the library you're using, checking that the application is behind a reverse proxy. But if you're writing an application from scratch, it's important to remember how Ingress works as a reverse proxy.
Leave SSL Certificate Management to Kubernetes
Developers don't need to think about how to "hook up" certificate addition in Kubernetes. Reinventing the wheel here is unnecessary and even harmful. For this, a separate service is typically used in the orchestrator — cert-manager, which can be additionally installed. The Ingress Controller in Kubernetes allows using SSL certificates for TLS traffic termination. You can use both Let's Encrypt and pre-issued certificates. If needed, you can create a special secret for storing issued SSL certificates.