I’m a Cloud Solutions Architect. My job is amazing. I get to talk to customers about the “Art of the Possible”: what open source technologies are available to them from Red Hat, what capabilities they provide, and how they fit into their long term strategies and shorter term tactical projects. I also spend time working with partners and my peers to educate them in our technologies and help them recommend solutions to their customers.
I was first exposed to Kubernetes when I joined Red Hat 2 years ago. Back then, OpenShift felt revolutionary. I've learned a lot more about Kubernetes, spending time lurking in the Special Interest Group (SIG) meetings for the last year. What is remarkable to me about Kubernetes is the pace of innovation. I encourage anyone with more than a passing interest in Kubernetes to pick a SIG channel in Slack and participate in the process. The community is amazing - open and welcoming to new contributors.
One of the questions I have been hearing a lot from customers, partners and my peers is about how OpenShift compares to the managed Kubernetes services provided by most of the major public cloud providers. Google introduced the world to Kubernetes, and their Google Cloud Platform has a managed service called “Google Kubernetes Engine” (GKE). Amazon recently announced their new service “Amazon Elastic Container Service for Kubernetes” (EKS) and Microsoft has also announced a new Kubernetes based offering called “Azure Container Service” (AKS).
It’s complicated to assess what solutions are right to solve the particular challenges in front of you, particularly if you are building applications leveraging containers. My focus in this post is to outline the differences between these new managed Kubernetes services and OpenShift to help clarify capabilities of each.
I have no interest in spreading uncertainty about these platforms. The larger and more robust the Kubernetes ecosystem becomes, the better things are for customers and vendors alike. I’ve spent a considerable amount of time trying to make this an objective analysis, and to avoid tone that makes it feel like an attack on other platforms. This is a point in time snapshot. IN the coming weeks and months there will be new capabilities that I have missed. The community is working hard to add features to the Kubernetes platform, and every software vendor continues to add value for their customers, I fully expect additional features to be added to each solution. If you feel like I’ve missed something, or misrepresented facts in this story, please let me know.
Amazon’s EKS runs in the AWS cloud. It is currently in preview/beta, GA is not announced at this point. AKS (Azure Container Services) is newly announced and built to run in the Azure environment. GKE runs on Google Cloud Platform. This is one of the largest differences between the public cloud managed services offerings and OpenShift. Each is purpose built for their specific environment. OpenShift Container Platform is built to allow customers to deploy to a multitude of platforms and provide the same experience for developers and operations teams in each.
A more direct comparison to the public cloud offerings is to compare to OpenShift Dedicated. OpenShift Dedicated is a managed service offering from Red Hat. Customers pay for usage in the OpenShift Dedicated environment and the infrastructure is managed for them. OpenShift Dedicated customers deploy applications, manage access and integration points to the platform, but the day to day operations are handled by Red Hat’s operations team. OpenShift Dedicated runs on AWS, Google Cloud Platform, and in the very near future Azure.
Technical Features Differences
There are a considerable number of features available in OpenShift (both OpenShift Dedicated and OpenShift Container Platform) beyond the capabilities of Kubernetes. Customers buying into a managed Kubernetes offering are in the same position as customers who install vanilla Kubernetes — they will have to solve the problems Red Hat has solved for with additional solutions.
For customers consuming a public cloud managed Kubernetes offering, it means paying for additional services from the provider. There is nothing wrong with that approach, but it is important to recognize the differences between a vanilla Kubernetes deployment and successful operation of container-based applications.
I want to clearly describe the capabilities OpenShift provides, and the services a consumer of a managed Kubernetes service would need to consider to have similar capabilities.
- OpenShift Container Platform can be deployed on physical systems, virtual machines, private cloud infrastructure, public cloud environments. Each public cloud managed service offering is tied to their respective cloud environment.
- OpenShift provides consistency in Operations and Developer Experience. This means a deployment that spans multiple private and public cloud environments can be used by developers and supported by operations teams in exactly the same way. Each public cloud service will need to be consumed and managed in a manner tied to the underlying platform.
- Red Hat Enterprise Linux (RHEL) is included. When a customer buys support for OpenShift Dedicated or OpenShift Container Platform, they are getting full support for RHEL as well. In other platforms, operating system support would need to be purchased and managed separately.
- Red Hat supports host Operating System all the way up to application services with OpenShift Container Platform entitlements. This means all of the elements from the Operating System to the service layer is supported. In the managed Kubernetes model, support extends only to the underlying Kubernetes infrastructure.
- OpenShift includes a built-in SDN. OpenShift SDN is a VXLAN-backed SDN solution that is available by default. It’s not completely clear how networking will be managed in the public cloud Kubernetes offerings. My assumption is that each provider will have implement a Container Network Interface (CNI) plugin to work with their internal network offerings but it is unclear at this point what will be provided.
- Routing capabilities are provided by OpenShift. In the managed Kubernetes services, customers will need to decide how they are going to handle routing — leverage networking constructs native to each public cloud or install, configure and manage their own software for this purpose.
- OpenShift provides an Image Registry. While many customers will initially leverage Docker Hub or other options available on the web, I know from experience that most customers will want to build their own registry for approved images and that’s why OpenShift provides the ability to install a registry right away. In public cloud offerings, customers will need to choose a registry solution and configure their environment to use it.
- Aggregated Logging is installed with OpenShift. During the installation of OpenShift, customers can choose to install Elasticsearch, Fluentd and Kibana (EFK) to for log aggregation, and the installation process will take care of building and configuring EFK automatically. In a managed Kubernetes environment, customers need to decide how they will handle logging of Kubernetes and the applications deployed in Kubernetes.
- Operational metrics aggregation and presentation are available to be installed and configured easily in OpenShift. Similar to the Log Aggregation capabilities, the OpenShift installation process provides a simple way to deploy metrics for monitoring applications and OpenShift itself. In OpenShift 3.7, customers may choose between deploying Hawkular metrics, or Prometheus. Customers leveraging a public cloud Kubernetes service will need to decide how they will manage metrics in their environment.
- OpenShift provides integrated tools for developer self-service (web, CLI & IDE), container builds (Source to Image), CI (Jenkins) and CD (Jenkins Pipelines). OpenShift customers can choose to integrate their existing CI/CD tooling into OpenShift workflows, or define Jenkins Pipelines and have OpenShift automatically deploy Jenkins containers to run the pipelines and then retire the containers once the pipelines have completed. This type of CI/CD integration is missing from public cloud managed Kubernetes services.
- Red Hat’s middleware portfolio is tested to run, scale and work reliably on OpenShift and is already running at scale in some of the most demanding environments on the planet. The managed Kubernetes services are new and our middleware portfolio is not certified yet to run on these platforms.
- Role Based Access Control (RBAC) capabilities are tightly integrated into all features of OpenShift. This is one of the most requested features for Kubernetes, and one of the first things Red Hat built for OpenShift. Customers need to be able to control access to resources inside of application platforms. RBAC is built into the fabric of OpenShift. At this stage, it’s not clear how managed Kubernetes services will implement RBAC. As upstream Kubernetes implements RBAC capabilities perhaps each public cloud provider will build integration with their own Identity Management service offerings.
- OpenShift provides a simple way to integrate Kubernetes with software-defined storage through Red Hat’s Container Native Storage integration. The need for software-defined storage becomes readily apparent to customers as they begin to deploy applications that expect a container orchestration platform. OpenShift provides an easy way to deploy Gluster and integrate the deployment with Kubernetes. It’s not clear at this stage how the public cloud Kubernetes offerings will deal with storage.
- Red Hat’s ISV program provides a significant advantage for customers looking to build and extend their Kubernetes deployment. There are currently 141 internal and 3rd party components that are certified and supported to integrate with OpenShift. This means that valid subscriptions for OpenShift support provides support for a large number of certified integrations. Integration with public cloud provider Kubernetes services should be technically feasible, but the level of support to be provided for those integrations is not defined in the current SLAs.
- OpenShift provides a UI which gives developers and administrators access to deploy and manage applications. The OpenShift UI now includes a Service Catalog based on Open Service Broker & Kubernetes Service Catalog project. Customers choosing public cloud provider managed Kubernetes will need to decide how to implement and manage a UI & service catalog.
- Red Hat security tools integrate with OpenShift. This means SELinux is built into the foundation of OpenShift deployments, and OpenSCAP tooling can be used to manage OpenShift environments. In a public cloud managed Kubernetes service, customers will need to choose security solutions to implement and install, configure, and manage on their own.
Support is critical when you’re considering a managed service offering, but it is not clear what level of support for Kubernetes will be provided on any of the public cloud platforms. All of these hosted Kubernetes services are new, and the services are evolving and maturing quickly. I'll describe current state of support for those considering adoption of these services.
Amazon has not described the Service Level Agreement (SLA) for their Kubernetes service. Microsoft’s Kubernetes on AKS is a free service. There is no SLA outside of support for underlying VMs. Troubleshooting Kubernetes problems will be up to your operations teams. Google does not define the type of support provided for Kubernetes Engine.
None of these services provide support for the container itself or the operating system hosting the containers. OpenShift Dedicated services includes support of infrastructure and runtime for applications. There is no question about the level of support you will receive for the OpenShift platform — it’s spelled out in the Terms of Service. In addition to Kubernetes, the OpenShift Dedicated team supports all of the other features of OpenShift as well.
There are significant differences in the costing of each solution. Cost comparisons between the services is complicated. The capital cost of a platform depends on a customer’s relationship with a vendor, the scale of initial deployment, expected growth and the discounts they enjoy. The operating cost is much more difficult to quantify, and depends on staffing numbers, in-house skills, and operational efficiencies.
OpenShift can be deployed to bare metal, virtualization infrastructure, private clouds, and public clouds. This allows customers to build one container orchestration platform across all of these environments. That means developers have a singular deployment experience no matter where the application ends up being deployed. Operations teams have one container platform to manage. For large scale enterprises, this can be a very big deal. Less training is required, fewer changes to operational processes and workflows. Those efficiencies have a trickle down effect, increasing Mean Time Between Failure (MTBF) and lowering Mean Time to Resolution (MTTR).
From a pricing perspective, not all the details of these new managed Kubernetes services are resolved. Amazon has not released details of EKS pricing. Microsoft and Google both charge for usage of the compute, storage, and networking resources consumed by the cluster nodes. Google will not charge for the Kubernetes masters. OpenShift is priced based on the size of your deployment.
In addition to the base Kubernetes service, customers need to consider the additional services required to make Kubernetes an Enterprise platform. To match capabilities of OpenShift in the public cloud environments, you’ll also need to consider adding:
* load balancers
* application gateways
* container registry
* service fabric
* log analytics
* policy services
* developer tools
* software defined storage
Your particular environment may or may not require these capabilities, but I thought it worth underscoring that these are capabilities provided out of the box by OpenShift.
Thanks for your time. I hope this was useful to you, and please - let me know if I've successfully made this an objective review of the platforms. It's an exciting time for infrastructure teams. Kubernetes is changing the way we think about containers and creating genuine disruption in our industry.