Most of the initial Kubernetes cluster deployments were based on Virtual Machines. After all, Kubernetes is just a bunch of software components such as API server, controller manager, scheduler, etcd, kublet, kube-proxy, along with container run-time, and other plugins to run on a base OS. Most enterprises already had virtualized environments and most of the public clouds were based on virtualization technology as well, so it was very natural to start deploying Kubernetes clusters onto VMs. However, as more Kubernetes clusters went into production, more people started asking: Do I really have to run Kubernetes on virtual infrastructure? Is there a reason not to bypass virtualization and directly run Kubernetes on bare metal machines?
The answer, as so often is the case, is it depends - there is no one-size-fits-all answer for this question. There are both pros and cons for deploying Kubernetes on VM based or bare metal infrastructure:
Performance: If performance is your big concern then bare metal probably wins the day. Virtualization will introduce a tax on the performance side. Typically the virtualization tax will introduce 5-10% performance overhead due to context switching and the extra hypervisor layer. Also, when multiple VMs end up being placed on the same physical machine, virtualization may also suffer from the noisy neighbor or oversubscription problem and further degrade the performance.
Some workloads need direct exclusive hardware access, e.g., AI/ML workloads might benefit from access to GPUs, or a big data workload might need high IOPS storage access. Although hypervisors typically provide some PCI passthrough and direct storage access capability, it can be difficult to configure and still adds 1-2% overhead over bare metal. Edge computing use cases have emerged that require small form factor Kubernetes clusters on hardware platforms with limited hardware horsepower, limited bandwidth, and limited remote management capability.
So again, for situations where performance or footprint is the big thing, Bare metal wins!
Cost: If your primary concern is cost then you might prefer a bare metal solution, as virtualization sometimes adds additional cost to infrastrutcure. On AWS, the VM and bare metal machines are priced the same, e.g., m5.24xlarge and m5.metal, both are priced at $4.608/hr on us-east-1, both have 96 cores, 345 ECU, 384GB memory. However. if you have on-prem private clouds like VMware vSphere, it does come with license cost on top of bare metal infrastructure. There can be additional management cost for virtualization operations as well. So, if cost is your primary concern, Bare metal wins!
Operations: Virtualization adds a nice layer of abstraction over compute, storage, and network, and all modern virtualization platforms and public clouds also provide convenient management features. Those features, built up over more than a decade, include features such as VM image, thin provisioning, snapshot, rollback, virtualized SDN for network micro-segmentation, and native persistent volume support using virtualized attachable storage. With bare metal, on the other hand, administrators have to deal with aged management tools and protocols such as IPMI, Redfish, and PXE for provisioning automation, many of which are not consistent between different vendors. Also if everything else in your environment is on virtual infrastructure then consistency can drive general operational benefit.
Bare metal does not provide an easy way to do snapshot and rollback and may need physical switch-based SDN solutions and external SAN/NFS for persistent volume support in Kubernetes. However, things are improving. Kubernetes cluster-api based orchestration with bare metal as a provider can make provisioning automation easier. Through some CNI and CSI plugins, Kubernetes can have some native network and storage solutions without relying on external systems. For example, Calico can provide Layer 3 SDN natively in Kubernetes on flat network; Storage solutions like Portworx, OpenEBS, Rook can provide persistent volume support using hosts' local disks clustered together via distributed file system. It's not as comprehensive as in the fully virtualized environment, but is improving.
Long story short, at least for now, Virtualization wins!
Wrapping it all up, as with so many things, there is no clear answer as to whether Kubernetes should be deployed on virtual infrastructure or bare metal. It all depends on your needs and your use case. What's important is to understand what those needs are and find the right solutions that cover you across the needs you have.
##
About the Author
Saad Malik is passionate about building products in the areas of cloud, virtualization, containers, and distributed systems. In his fifteen years of experience, Saad has shipped multiple new products in enterprise, service provider, and consumer technologies. He is a hardcore Trekkie and enjoys building autonomous drone tracking software.
