A comparison of different approaches to give developer teams access to Kubernetes.
With the DevOps approach becoming the new normal, developers have to consider the production runtime environment when developing new software. This implies that the developers have to deal with technologies that are rather infrastructure than application technologies. Nowadays, this often means Kubernetes and containers. While this already poses a new challenge to development teams, there emerges an additional practical problem: If developers are expected to directly work with the target infrastructure, i.e. Kubernetes, how can they get access to it so they can easily develop, test and deploy with it.
In general, there are two distinct approaches to provide developers with a Kubernetes environment for development purposes: Imitating the Kubernetes environment on the local computer and using a real Kubernetes cloud environment also during development.
Developers can run Kubernetes locally, which is a good starting point for smaller applications. For more complex software (microservices, machine learning,…) or larger teams, this approach is not suitable as it is limited to the local computing resources and is impossible to centrally maintain.
Alternatively, every developer can get an own k8s cluster in the cloud. While this provides them with a very realistic environment and basically unlimited computing power, it is also difficult to maintain and control. This solution will also lead to a waste of computing resources and thus to higher cost. It should therefore just be used by smaller teams of Kubernetes experts and if computing costs do not play an important role.
Finally, developers in a team can share a common Kubernetes cluster. This has the same advantages of realism and computing power as the previous approach but additionally makes it easier to centrally maintain and control. The major difficulty here is to isolate users from each other and to provide them k8s namespaces. However, this can be done by connecting a Kubernetes cluster to DevSpace Cloud, which cares for the user isolation, management and namespace provisioning. This solution is thus also meant for complex applications and larger teams of developers without k8s experience.
1. Run Kubernetes locally on the developer’s machine
The first option is to imitate the Kubernetes cloud environment by running Kubernetes on the local machine of every developer. This can be done with solutions such as minikube or k3s which need to be installed on the computers of the developers and then provide a Kubernetes experience to users that is similar to regular Kubernetes environments in the cloud.
By using these tools, development with a local Kubernetes environment can be a relatively easy solution. Together with the fact that no cloud resources are needed and thus no additional expenses are required from the business, it is very easy to experiment with such a setup, which makes the entry-barrier very low even for individual developers.
Another advantage of this approach is that developers have their own, completely isolated environments that are under their full control.
This is at the same time, however, also a major disadvantage: While you want your developers to interact with Kubernetes, you do not want them to manage it completely on their own as this always requires some configuration effort and deeper knowledge about Kubernetes. If you don’t want to teach every developer on your team how to configure Kubernetes, someone will have to manage and set up isolated k8s instances on every developer’s laptop and keep them updated. Especially in teams with more than 4 or 5 people and different hardware or operating systems, this will become a very cumbersome or even impossible task.
Moreover, running an application in a local Kubernetes environment requires quite a lot of computing resources. Depending on the complexity of the application, the required resources might just not be locally available, e.g. if you have a larger microservice-based application or an AI software that requires GPUs. If this is the case, the local approach will automatically become infeasible.
Using local Kubernetes solutions can be a good starting point to develop with Kubernetes as it can be easily tested by the developers without dealing with a cloud environment at all. As your team and your application grow over time, you might want or even have to move to another solution that requires less knowledge and effort on the developers’ side and provides more central control and oversight.
2. Provide developers access to Kubernetes in the cloud
The alternative solution to develop with Kubernetes is to provide all developers in the team some kind of access to a cloud-based Kubernetes environment.
Again, there are two general approaches to do that: Providing an individual cluster for every developer or sharing a single cluster among a developer team.
2.1 Give every developer an own Kubernetes cluster
The obvious solution to give developers access to Kubernetes in the cloud is to simply give every developer an own Kubernetes cluster.
The main advantage of this approach is that the developers are as well isolated from each other as they are with local development. They cannot interfere with each other and if something breaks, it is easy to find the cause.
Usually, the same cloud environment is used for development and production, e.g. the same private cloud or the same public cloud provider. For this, the development with this environment is very realistic compared to the production environment because cloud-specific configurations or features can also be used during development. As a result, tests executed in this environment are very reliable and software that runs in the development environment is very likely to also run in production. Another advantage of this approach is that the application may be very complex and may require a lot or specific computing resources, such as GPUs, because these resources can be provided in the cloud.
However, this approach also has some disadvantages: At first, it is usually not possible to give every developer an own Kubernetes cluster in a private cloud, which makes this approach rather suitable for public clouds, where very different machine types and sizes are available. A bigger problem, however, is that many individual k8s clusters require a lot of maintenance. Similar to the problem with local Kubernetes solutions, the individual clusters can either be managed by the developers themselves, but then they need to have the necessary knowledge and cannot focus on their actual work of developing new software, or by admins who then will have a lot of work maintaining many clusters in parallel. This problem is also related to the question of how to provide access to the cluster to every developer. Here, every developer, for example, needs to install kubectl and needs an account for the public cloud provider.
Moreover, running many Kubernetes clusters and having many users with access to the cloud makes it very complicated to keep control and oversight over the whole system. Especially regarding the cost for computing resources, this can be a problem as there may easily emerge the tendency to allocate too many resources. For example, if developers have the choice to get more resources to run their applications or tests marginally faster, they may tend to take the extra resources even though this may be not financially reasonable. Additionally, sometimes a lot of resources are just needed for a short time. Then, it would be best to scale down afterward, but this downscaling is a task that can be easily forgotten. In general, running many separate Kubernetes clusters is not efficient from a resource perspective as the basic k8s functionality needs to run several times although this is actually not necessary.
For this, giving every developer in a team an own Kubernetes cluster is an approach that is rather suited for small teams in which every developer has some Kubernetes experience already. However, as this solution is certainly not computing resource efficient and hardly feasible for larger teams, it is generally inferior to the alternative approach for developing in the cloud: Sharing a cluster.
2.2 Share one cluster among your developers
It is also possible to just use one or very few Kubernetes clusters and share them with several developers in a team by giving each developer an own namespace in the cluster. (For simplicity, I will only write about “one cluster” in the following. However, it does not necessarily have to be exactly one cluster, but could also be a small, limited number of clusters.)
Since the development is also cloud-based, this approach has the same advantages of realism and almost unlimited resources as the previously described one-cluster-per-developer approach. Additionally, it is also possible to use a private cloud as only one k8s cluster is needed for the whole development team.
Unlike the approach with multiple clusters, it is very resource efficient to run Kubernetes just once and share its basic functionality with the whole team. This is also the typical approach for production systems, i.e. you would not run individual Kubernetes clusters for each part of a larger application but rather have everything running in one or a few clusters. Since there is just one cluster, it is also much easier to keep oversight and to control the resource cost by simply setting limits in Kubernetes.
However, this approach also has some limitations: The individual users of the cluster need to be isolated from each other to prevent complete anarchy. Without the proper setting for multi-tenancy, one user could use up all computing resources so that no other developer can use the cluster anymore. It would also be possible to break something in a way that affects other users, which makes it much harder to trace the source of an error.
With the shared cluster approach, every developer needs to get a namespace. This namespace has to be created by a cluster admin before it can be used by the developer (if not every developer is a cluster admin, which is not recommended). However, this can slow down the development process as the admin has to be contacted every time a new namespace is needed. In the worst case, when the admin is not available, the developers are not able to work at all.
Overall, the approach of sharing one Kubernetes cluster for development with the whole team is a computing resource efficient and scalable way of developing with Kubernetes in the cloud. The problem of user isolation, however, needs to be properly solved for this approach. Fortunately, there is a solution that addresses exactly this challenge: With DevSpace Cloud, it is possible to transform any Kubernetes cluster into a cluster that supports multi-tenancy with automatic user isolation, resource limitations and on-demand provisioning of namespaces for developers. (For technical details and a comprehensive list of features, read this article)
DevSpace Cloud is available as a SaaS-solution or on-premise. In any case, you simply connect your Kubernetes cluster and can invite additional users to the cluster whose permissions can be set in a graphical UI.
Use Cases and Comparison of the Options
Running Kubernetes on your local machine with tools such as minikube is a feasible approach for small projects that can be run on a limited infrastructure. Since Kubernetes is running on the local computer of every team member, this form of development is rather suited for teams in which all developers have some experience with Kubernetes and can actually care for the environment themselves because you could otherwise easily run into a maintenance nightmare.
The intuitive solution for cloud development of providing every developer with an own Kubernetes cluster is also more appropriate for small teams with k8s expertise that do not use a private cloud. Since it requires more computing resources than other approaches, it is the most expensive and thus rather a temporary solution. It is therefore generally inferior to the last option.
The last alternative is to share one cluster with the developers in a team. This can be done in both private and public clouds and is more efficient and thus cheaper considering the computing resources. However, to prevent chaos and implement it efficiently in your development flow, you need to solve the multi-tenancy issue. If you use solutions such as DevSpace Cloud for this, this approach is suitable for larger projects and teams without previous Kubernetes knowledge.
While local development does not require any additional resources, you will have to pay for computing resources in a public cloud. To learn more about how to minimize the cost for cloud computing resources during development, read this post.
The local development approach is certainly the cheapest way of developing with Kubernetes. Since it is limited to smaller, less complex applications and cannot be centrally managed, it is rather meant for small teams and individuals. In contrast to this, running an individual cluster for every developer is pretty inefficient from a resource and maintenance perspective and can thus only be recommended for smaller teams with reliable Kubernetes experts. Finally, a shared cluster for a team with proper multi-tenancy settings provides an efficient way that can be used by companies of all sizes and thus is the most professional way of developing software in Kubernetes.
If you want to truly develop software cloud-natively with Kubernetes, developers need to have access to Kubernetes during development. I hope this post was helpful for you to find out in which way you can achieve this best for your use case.