Kontakt

Kubernetes for stateless workloads

kubernetes.io

Kubernetes is, in our opinion, the emerging standard for container orchestration and a robust open source basis for building PAAS. Since fall 2017, we're using it ourselves for all our internal and external services, that's why we can wholeheartedly recommend to ADOPT Kubernetes.

Benefits

  • Provides a unified API for smaller and bigger workloads, with the option of scaling the applications from small to big.
  • kubectl allows developers to self-service and debug (e.g. log into the containers)
  • Can be used on a single host (using minikube), or on a fleet of many services.
  • Is the emerging de-facto standard for containerized applications.
  • consists of very clever concepts/atoms (like Deployment, Pod, Ingress, Service), which are pluggable in flexible ways
  • Helm being used as package manager.
  • Using Google Container Engine, there exists a professional hosted service for critical workloads.
  • Supports Role-Based Access Control (RBAC).
  • Supported by lots of external applications.

Drawbacks

  • the yaml configuration can be quite verbose
  • Has some learning curve, but a great interactive tutorial
  • The initial setup is nontrivial; and it is hard to figure out how to best install a working cluster.

Alternatives

  • We've used multiple dokku servers in the past, but it was more and more difficult to know where each application resided; and it did not provide the resiliency we needed.
  • We want to control our own infrastructure, that's why cloud-based PAAS platforms like Heroku are not used by us.

Kubernetes at Sandstorm

We at Sandstorm use Kubernetes in a little special way, which is outlined below:

  • We installed Kubernetes using Kismatic, which worked well for us. We're using weave net as overlay network. Before, we've evaluated the use of Canonical Juju, and Rancher 2.0; but Kismatic was the first one really working well.
  • As we currently only have two Kubernetes servers, we are not yet running in high-availability mode. Instead, we manually pin pods to the worker nodes for now. Because of this, we can use the hostPath persistent volumes, which are very easy to understand.
  • We're heavily using ingress-nginx, e.g. for things like SSO; and kube-lego for Let's Encrypt SSL certificates.
  • As Docker Registry and CI system, we are using GitLab.
  • "Big" stateful services like MySQL/Postgres databases are not hosted by Kubernetes, but colocated on the same infrastructure (installed using apt etc) - for much of the reasoning explained in this blog post.
  • For user authentication, we're using client certificates together with RBAC.
  • Backups are automatically done using CronJob and Restic.
  • Monitoring is done using Prometheus and Grafana.

In the mid term term, we plan the following infrastructure additions:

  • Add a third server. Once this is done, we can in principle tolerate one failure.
  • Set up MySQL (Galera) Cluster and Postgres Replication, to have a HA DB cluster.
  • For resilient storage, the following options exist (this blog post structures it nicely):
    • GlusterFS - currently our favourite; seems like high performance and HA, and suppots ReadWriteMany.
    • OpenEBS - looks interesting, but is only a block storage; thus it does not seem to support ReadWriteMany mounts (which we need).
    • Rook / CephFS - seems to be way more hard to deploy (compared to GlusterFS)