For some time now, I’m also doing a project that anyone can use to create your own Kubernetes cluster, that is entirely driven by the Flux project. What’s probably the best part of it, by using a free layer infrastructure from Oracle Cloud, you can run this cluster completely for free! So this gives you a fully functional Kubernetes cluster, managed by a great GitOps tool, and without paying a penny. Check the repo here: https://github.com/piontec/free-oci-kubernetes/

The post How to Get a Fully Functional, GitOps Driven Kubernetes Cluster for Free appeared first on Tailored Cloud.

When I bought my machine, it wasn’t working and was sold as “unknown, non-working condition”. This model of espresso machine is super popular, and you can find a lot of info about how it works and how to repair every single piece of it. So, I won’t write everything again over here, but will make it easier for you and link all the resources I used. My main source of knowledge was a blog dedicated to modding and fixing the Gaggia Classic (mainly), which is absolutely great, detailed, and insightful, but available only in Polish. If you don’t speak this language, I suggest trying auto-translation, as the blog is totally worth it.

Using this set of links, you can basically repair anything you want in your Gaggia as well

• home page of the “Moja Gaggia, Moje Mody” blog
• how, where and which seals to change in your machine
• fixing/cleaning the 3-way valve
• how is the Ulka vibartion pump built and how to fix it
• how to check of the heaters of your boiler are OK
• big list of spare parts
• electric connections diagram

The post Repairing the Gaggia Classic appeared first on Tailored Cloud.

I think that in order to better understand the modding process of an espresso machine, you need to first understand how it’s working.
What I bought for my project was a roughly 20 years old Gaggia Classic, in a quite rare black Coffee version. Still, in regard to how it’s built, it’s the same as Gaggia Classic v1. What was really interesting to me, is that this machine uses no electronics at all: everything is driven directly using the AC power and switches on the front of the machine – really nice and simple solution!

How coffee is made

When a machine is turned on, it turns on both heaters integrated with the water boiler. Depending on whether the machine is in ‘coffee brewing’ mode or ‘steaming’ mode, there’s one of two thermocouples included in the circuit with the heaters (one disconnects when the boiler gets to ‘coffee brewing’ temperature, the other disconnects when it gets to a higher ‘steaming’ temperature). By default, they are internally connected (on), but when they reach a certain temperature, they break (turn off internally) the circuit and boiler heaters stop working. When the water in the boiler cools down a bit, the thermocouple switches back on and heaters start again. That way, the machine executes its first function: heating water. By the way, this process is what is causing some problems in espresso brewing: the difference between off-and-back-on temperatures (called hysteresis) of the heating circuit can be as high as 10 °C, which will already impact the taste of espresso. The PID controller we want to add will keep our temperature within 1-2 °C of the target value.

Unless you flip the brewing switch, nothing else happens. When you do, the brewing process looks like this:

• The pump starts pumping the water from the water tank to the hose that goes to the Over-Pressure Valve (OPV). The pressure of this water is roughly 15 bar.
• Next, the water goes through OPV. In general, you can have OPV in a form of a simple plastic wedge, that is pushed against the water current by a spring or a full regulated OPV. My advice: don’t even buy machines that have the simple one. Gaggia Classic v1 has a regulated OPV valve. This valve has 1 input and 2 outputs, so to speak. The water flows in and pushes internally on a spring-loaded rubber seal. If the pressure is low, the seal holds and all the water goes out through the first output, which is used to brew coffee. When the pressure of the water is high enough, the spring is compressed, the seal opens and some water gets out the 2nd overflow output of the valve. This output leads back to the water container, where there’s no resistance on the way, so there’s also no pressure buildup and any excess water can flow this way. That allows the OPV to limit the water pressure to a constant value – the one that is needed to open the overflow output. By regulating a nut within the OPV you can preload the spring as hard as needed and that way also regulate the pressure at which the overflow opens. Still, to calibrate the OPV precisely, you need to connect a pressure gauge somewhere. One option to do that is to block water output out of the coffee group using a blind sieve, then split the water hose and connect your gauge there. The other, simpler way, is to connect the pressure gauge to your normal portafilter, blocking water outflow using the gauge directly. So, here’s the 2nd step of the espresso process: setting the water pressure to a correct value, typically 9 bar.

  

• After the OPV, the water passes through a 3-way valve. In theory, this part is not really needed, but you still want to have it. The way this valve works is that it has 3 connections: from the OPV (so the water you want to brew your coffee with), one exits to the coffee group (where you want your water to go to make a coffee) and the second output to the drip tray. The 3-way valve works by enabling one or (actually xor the other of paths water can flow. When the AC power is enabled, the valve allows water to flow between OPV and the coffee group, but blocks output to the drip tray. When it’s not energized by AC, it allows the water to flow between the coffee group and the drip tray. Why is it useful? Just after finishing the brew, there’s still quite high pressure in the coffee group and your portafilter. If you remove your portafilter from the group while there’s still some pressure, droplets and bits of coffee will start flying around your kitchen! To avoid that, as soon as you turn off the water pump, also the 3-way valve is turned off and allows the extra pressure to be released to the drip tray through the 2nd path.

  

• Now we’re almost done. We have water at the right temperature and pressure, we’re protected from a mess by the 3-way valve, we can just push out the hot water from the boiler into the portafilter – it is the fresh water coming from the water tank that is pushing out the hot water from the boiler onto your coffee. So, put your ground coffee into the portafilter and let the water flow!

The post How a single boiler espresso machine works? appeared first on Tailored Cloud.

Still, I want to drop at least a short update here.

Over the last few months, I tried the projects I mentioned in the previous post. I also have a decision about when to use which:

– if you need wireless connectivity, MQTT integration and such – use the bleeding edge

– if you want the most advanced features, yet no connectivity and a very opinionated main developer – go with gaggiuino.

Still, no matter which you choose, you need an operational Gaggia machine first. I have an almost ready short entry about the best resources I found on the Internet when I was fixing mine.

The post Short espresso machine project update appeared first on Tailored Cloud.

For a few years, I was a moderately happy owner of an Ascaso Dream espresso machine and i2 mini grinder. They create a nice set and the coffee was… OK. Espresso was hit-and-miss, but mostly miss, and it always frustrated me that an espresso at a regular coffee bar, like Costa, is much better.

If my espresso quality is subpar, then what is required to get a good espresso? I started to check again what is considered important when making espresso. The non-ultimate list is:

• first and foremost, a good grinder that can get you the needed fine-grained coffee powder of uniform distribution
• precise water temperature control (and good water quality in general)
• water pressure control (should be 9 bar, at least as a start value)
• even distribution, tamping and extraction of coffee puck in the basket (this can be a hard one)
• repeatability and control over everything mentioned above.

After a long break, I got back to a polish coffee forum, where I wanted to check if someone figured out how to make better coffee using hardware I have. I learned there that one folk has basically remade his Ascaso Arc (same hardware, different case than my Dream), added an AT Mega based MCU and turned it into a beast. That started me off. I started to read about how espresso machines are made and how they work, and I found a few open source projects on GitHub, where people create a firmware for home class espresso machines. That was it – at this point, I knew what my next big hobby project is.

This blog entry starts a series of note-to-future-self entries, where I want to document where I found stuff needed to fix and make the open source espresso machine based on Gaggia Classic.

The firmware projects I found are:

• https://github.com/Zer0-bit/gaggiuino
• https://github.com/rancilio-pid/ranciliopid and its fork https://github.com/medlor/bleeding-edge-ranciliopid

The first project on the list is aimed at Gaggia Classic machine, while the second one supports a few machines, mainly Rancilio Silvia, but also Gaggia Classic. I read about the projects, compared them, checked on the features and decided to go with ‘ranciliopid’ or its fork. So, I created a plan and arranged it in stages:

• stage 0
  • buy a used Gaggia Classic
  • learn how it’s built, repair it (if needed), clean it and make nice
  • make it work as the original design intended to
  • add precision baskets, precision shower filter and an NPF (naked portafilter – it doesn’t change the taste of coffee, but allows you to better see how your extraction works and looks awesome – I always wanted to have one )
• stage 1 – add ESP32 MCU in the “full” scenario, with features including
  • digital control through relays of all the main electric components: 3-way valve, pump and heaters, and the TSIC 306 temperature sensor, which allows for:
    • very precise temperature control with PID controller (with 1 °C resolution)
    • timed shots: the machine stops after the set number of seconds
    • pre-infusion: when a brew starts, the machine first pumps water for a brief moment of time (like 2 s), then takes a short pause, then starts the actual extraction – this presoaking of coffee allows getting rid of the air trapped in between the coffee powder and should make your extraction more even
    • backflush: a cleaning mode, where brew group is first pressurized (with a blind sieve/basket attached), then expelled through the 3-way valve to the drip tray
  • add an OLED screen
  • configure MQTT integration
  • insulate the boiler of the machine to eliminate the gigantic heat loss of the naked metal boiler and speed up the heating process on start
  • add a water level sensor: to tell you when you run out of water, so you’re not surprised mid-brew
  • figure out how to place all of that in the machine, add necessary case for the electronics
• stage 2 – add scales
  • Good espresso needs to have a correct extraction ration and extraction time. Extraction ratio is the ratio between the weight of your final product – the espresso in your glass – to the weight of coffee beans used to make it. There’s no single correct value, but in general people agree that your extraction ratio should be somewhere between 2-3x (so from 10 g of freshly ground coffee powder you should get 20-30 g of espresso). As for time, the typical value is 25 s, but it also can vary between 20-30 s. So, to make your coffee right, you need to find (dial in) your grinding settings for a specific coffee to land within these ranges. The basic option is to set your brew time to a constant value, like 25 s, then change your grinding setting until you get the 2-3x extraction ratio. But to be able to tell the ratio, you have to put a scale under your espresso glass when you pull the shot. Now, the firmware that I want to use (and gaggiuino is working on it as well) has a killer feature: you can integrate a scale into your drip tray and connect it directly to the ESP32. That way, you can start making your coffee by weight: your machine will stop the extraction when you reach a certain weight of espresso and will tell you how much time that needed. This is my next stage after the basic functionality.
• stage 3 – features that need to be created and I would like to help contribute
  • Pressure profiling: this is the hot topic in espresso world (as far as I’ve seen). The idea is that you add 2 hardware elements: a water pressure sensor to check the pressure of water flowing through your coffee puck and an AC light dimmer, which you can use to cut down the power of the water pump. By cutting the power of the pump, you make the pump lower the extraction pressure. Controlling the pump using data from the pressure sensor, you can create a pressure profile of water during your extraction. Ideas that I’ve seen so far are based on linear change, so for example your brew starts at 9 atm but then falls down to 6 atm at the end of brew. Still, everything is possible – you just have to create a code that can execute that.
  • Drip tray overflow warning. Since the scale I want to integrate into my machine is actually based under the drip tray, it should be possible to get the weight of the tray when it’s almost full of water, then display a warning for me that I should empty it. Sounds simple and useful.
  • Automatic extraction rate brewing. As I explained above, you can either brew your espresso by time (for a given number of seconds) or until it reaches a certain weight (like until we get 30 g of espresso). The problem that I see with that second method is that my initial coffee powder weight might vary. I’d like to try to integrate a digital scale connected to the espresso machine’s MCU, that weights how much coffee powder has my grinder produced. Having this information in the MCU, I can stop the extraction when the ratio reaches a specific configured extraction rate, like 2,5 x.

Wrapping up

This is going to be a long project for me, but I’m actually writing this after already (almost) completing my stage 1. In this blog series, I want to save a set of “notes for my future self”, so that I have a written reference when I need to service my own machine again.

In the next few entries, I’ll try to start by explaining how an espresso machine works and how you can repair it if you have to.

The post Smart Espresso Machine – starting a new hobby project appeared first on Tailored Cloud.

The post Cloud-native observability series: season finale available now! appeared first on Tailored Cloud.

In part 2, I’m discussing how to build a simple micro services app that will be used as a ‘lab rat’ for our experiments with the cloud-native stack later.

In part 3, I’m describing how you can deploy your application (just a binary) as a full fledged Kubernetes concept, up to the point where we can pack everything into a Helm chart.

Enjoy!

The post Building applications for the cloud-native stack – continuation. appeared first on Tailored Cloud.

It is a year since a wrote a blog post here. A lot has changed over this year in my professional life. After quitting my previous job, I was looking for new opportunities and eventually decided to join the cool team at Giant Swarm as a platform architect. I have to say this was a great choice! My focus is now on enabling our customers to run applications on top of Kubernetes clusters.

Still, I didn’t want to stop blogging. Fortunately, I can now connect everything as part of my new job! So, I’m switching my focus slightly to cloud-native applications running on Kubernetes than just Kubernetes. This is also the topic of a blog series called “A Few Dozen Lines of Code”, the first part of which was just published on our company blog!

This series is meant to show you how using an opinionated cloud-native stack of tools installed on your Kubernetes cluster can make your dev and operations team happier than ever Stay tuned for next parts, they should come shortly.

So, this entry is kind of “Schrödinger’s entry”. It is a new entry and it is not – as I’m just pointing you to my article on the company blog. Still, I will inform you if anything written by me is published there. Meanwhile, you can check other engineering articles published there, it’s good content.

The post Schrödinger’s come back to blogging: using the cloud-native stack series. appeared first on Tailored Cloud.

• SmartNat is a Kubernetes ingress controller for exposing a massive number of TCP/UDP services to the outside world using just 1 server
• available on GitHub: https://github.com/DevFactory/smartnat

SmartNat – Kubernetes ingress controller for TCP/UDP services

Some time ago I wrote posts about writing a very simple Kubernetes controller and using operator framework to create a more complete one. Well, at the same time, I was starting to work on one at the company I’m working for. Meanwhile, I convinced management at the company to release the project as an OpenSource – and here it is!

The project is called SmartNat. It’s a Kubernetes ingress controller for TCP/UDP services that allows you to drive external traffic to your Services. It’s kind of Service with NodePort, but on strong steroids. It runs on a separate instance (well, you can run however you like, but this makes the most sense) and interconnects external (usually public) network with the subnet used by your Kubernetes cluster. SmartNat allows you to use multiple network interfaces, each one having multiple IP addresses to forward traffic from an external network to your services on a port-by-port basis. That way using just a single server or instance you can easily expose hundreds or even thousands of Services. The important property is that all of this is done using L3/L4 tools only, so SmartNat helps where HTTP based Ingresses can’t. Additionally, SmartNat supports simple traffic filtering of traffic coming from external subnet and also HA mode.

If you’re interested, check the project on github: https://github.com/DevFactory/smartnat.

The post SmartNat – dirt cheap Kubernetes ingress controller for TCP/UDP services appeared first on Tailored Cloud.

• unless you need a really low-level control or are writing a specialized controller, do use one of the helper libraries like  the operator-sdk or kubebuilder to avoid writing a lot of boilerplate code,
• handling events and objects in the controller is about synchronizing the state between API object and the system, not about reacting to events,
• try to be stateless, as keeping state is your problem,
• when you write a Kubernetes controller, be sure you know what your state is and how to make it durable,
• remember your controller can be restarted any time,
• use Custom Resource Definition to create any API object you need,
• DeepCopy() API objects if you’re changing them,
• if you’re creating new API objects, always set their OwnerReference.

Introduction

Some time ago I posted an entry about how to write a Kubernetes controller. It was a very minimalistic example of a controller, which aimed only at putting the main pieces in place. Still, I wanted to explore the topic further and write something more realistic and useful. As you can learn in the previous blog entry, despite the simple idea of how a controller operates, writing one requires a lot of boilerplate code. Fortunately, there are already libraries/frameworks that are trying to remove this boilerplate stuff and get you on track much faster. I had a look at two of them: the operator-sdk and kubebuilder. As I found the operator-sdk first, I started with it. In this blog entry, I want to show you how to write a Kubernetes controller with operator-sdk.

My idea was pretty simple. I wanted to write an operator, which is useful, but still very simple, so it can serve as a tutorial material, showing how to build operators. This turned into a Netperf Operator, a kubernetes tool that allows you to run the good old network performance benchmarking tool called netperf (yes, click this URL and admire pure HTML at its best!). I wanted to be able to run netperf, a client-server application, between 2 kubernetes pods, preferably running on different cluster nodes. Such tests are pretty valuable as they are the only way you can check your real pod-to-pod network performance, including the impact of all the networking layers, like overlay networks and such. Normally, running netperf requires you to start a server at one place and a netperf client at the other end of the tested connection. I wanted to make this a one-step process, managed by an operator. And of course, learn and how you how to write a Kubernetes controller. You can check the resulting project on my github page: netperf-operator.

Oh, but why do I call it “operator”, not “controller”? These concepts are very close: people tend to name Kubernetes controllers specialized in running and configuring a single application  “an operator”. There are already many awesome operators. Prometheus-operator is just one of them, but it really shows the power of an idea of running an application integrated with your kubernetes cluster and managed “the cluster way”. You can find other operators on the awesome-operators github page – be sure to check them.

How does kubernetes controller/operator work and run?

Well, this is mainly a reminder, but still a crucial one, so let’s say it once again. A very simplified controller’s life looks like this:

while true {
  receiveInfoAboutAPIObjects()
  synchronizeRealStateToMatchFetchedInfo()
}

There are a few consequences coming from the simple pseudocode above. The first one is that you need to receive notifications about the state and its changes of kubernetes objects that you’re interested in. Thankfully, the operator-sdk comes to the rescue here. It takes care of the synchronization with kubernetes API server and the notification loop. It also allows you to decide what type of API objects you want to observe.

Another important property is that the synchronization loop is not really event-driven, but state-driven. It means that your controller won’t receive events only when something changes, for example, a Pod dies or is created. You will get them then, but moreover, you will also get periodic refreshers: updates that show you the complete required state of the Kubernetes object, no matter if it changed or not since the last update. This is a very important property and you have to get it right. It took me some time to stick to it, so let me rephrase it. The notification loop doesn’t tell what change you need to perform in a system, but how the system should look like. It declares and describes the desired state, not the change that’s needed to produce it. Figuring out how to get to the desired state is a task for the controller.

Let me give you an example here. Let’s suppose we’re writing ReplicaSet controller. The ReplicaSet controller takes a pod configuration template and a desired number of pods to create. Its task is to ensure that the declared number of the pods are always running in the system. Let’s now assume we have created a ReplicaSet object with a Pod template and we set the pod count to 5 and also already 5 pods are running in the cluster. Now, if any of the pods die, the ReplicaSet controller gets a pod status update. But its reaction is not just “create a new pod”. Remember, a controller doesn’t react directly to the event, but checks for a difference between the configured and the real state. So, the controller checks that there are currently 4 pods running in the system, the required count is 5, so the solution is to start 1 new pod. But the same synchronization logic is also run periodically, even if there are no pod events in the system. Imagine now, that a whole node dies in the cluster and this node was running 2 out of 5 of our pods. Now, the cluster is not solving the problem of “how to notify the controller that 2 pods have died”. Instead, the controller will run the synchronization loop and check that there are now only 3 pods in the system and 5 are required, so the solution to reconcile the state is to start 2 new pods. So, we’re checking for the desired state and make it a reality, not simply react to events like.

One more important thing is that you should expect your controller/operator process to be terminated at any time. Most probably your controller will be run just as a pod in the cluster, and pods are mortal and should be easy to restart and recreate. This means, that if you need to keep a state, you basically have to keep it outside of your controller so that after a controller pod restart you can resume controller’s operations.

How to write a Kubernetes controller with operator-sdk – bootstrapping

After the general concepts presented above, we’re ready to write a Kubernetes controller and bootstrap a new operator project. This topic is nicely covered on the project’s github page. Definitely, have a look there for a reference. I bootstrapped my project with

operator-sdk new netperf-operator --api-version=app.example.com/v1alpha1 --kind=Netperf

Custom Resources and Custom Resource Definitions

OK, but let’s pause for a second. You might be wondering how the controller pattern is really useful, except for controllers that are already in the Kubernetes, like Deployment or StatefulSet controller. After all, not everything is about pods and services, that are native objects in Kubernetes. But can you write a controller that handles some other objects? The great thing is that Kubernetes API and objects system is easily extensible. The same way the cluster gets information for standard controllers from your description of Pods, Deployments or Services, it can also handle any Custom Resource. It just needs to be configured using a Custom Resource Definition. Using this mechanism, you can introduce any object you need into your cluster and configure it using YAML files and “kubectl” or API calls. You can find the CRD for netperf-operator here. The whole declaration is really in these few lines:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: netperfs.app.example.com
spec:
  group: app.example.com
  names:
    kind: Netperf
    listKind: NetperfList
    plural: netperfs
    singular: netperf
  scope: Namespaced
  version: v1alpha1

It just defines the name of your Custom Resource (singular and plural), API object kind (Netperf) and that your CRD is scoped to a single namespace. And that’s it, you’re ready to create your own new “Netperf” objects. As you can see, we’re not giving here any data schema that we expect from the Custom Resource object. We handle that in the controller code; for CRD definition the object schema is irrelevant. Still, there’s a new feature in kubernetes 1.11 that allows for validating a Custom Resource by embedding Custom Resource validation schema in CRD definition, but it’s a new thing and just beta in 1.11.

So, the next step is to define your Custom Resource schema in code. Basic stuff is already generated by the “operator-sdk new” command used to bootstrap the project (check this directory). The important part is that we have to add the Spec and Status parts of the Custom Resource (CR) object. Spec is the specification, so basically, an input describing the object. Status shows, well, the status of the object. Here’s how the full definition looks like, but the most important part is:

type Netperf struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata"`
	Spec              NetperfSpec   `json:"spec"`
	Status            NetperfStatus `json:"status,omitempty"`
}

type NetperfSpec struct {
	ServerNode string `json:"serverNode"`
	ClientNode string `json:"clientNode"`
}

type NetperfStatus struct {
	Status          string  `json:"status"`
	ServerPod       string  `json:"serverPod"`
	ClientPod       string  `json:"clientPod"`
	SpeedBitsPerSec float64 `json:"speedBitsPerSec"`
}

Lines 1-6 were already generated. We’re just providing the NetperfSpec and NetperfStatus definition. In Spec, I’m expecting just 2 input variables: names of kubernetes nodes where the controller should run the netperf server and client pods. With Spec like that, we can define a netperf test between any 2 cluster nodes we want. The Status is responsible for showing what are the names of ServerPod and ClientPod, the overall Status of the current CR (a single netperf test) and the final test result in bps.

When your definition is done, you have to run the command:

operator-sdk generate k8s

This will generate some helper functions for your CR, like deep copying the object instances. Now your definition of the Custom Resource is done and you can already use it!

The controller loop – checking what needs to be done

Do you remember how much code it took to start a control loop without any controller library? You might be surprised when you check the main.go file. Basically, the bootstrap code is this:

resource := "app.example.com/v1alpha1"
kind := "Netperf"
namespace, err := k8sutil.GetWatchNamespace()
if err != nil {
	logrus.Fatalf("Failed to get watch namespace: %v", err)
}
resyncPeriod := 5
sdk.Watch(resource, kind, namespace, resyncPeriod)
sdk.Watch("v1", "Pod", namespace, resyncPeriod)
sdk.Handle(stub.NewHandler(operator.NewNetperf(realkube.NewRealProvider())))
sdk.Run(context.TODO())

We’re declaring API resource type and kind, how frequently we want to get object status update from the API server and then in lines 8 and 9 we just start to watch changes on our “Netperf” and “Pod” objects. Why “Pod”? Because our Netperf Operator creates new pods with netperf client and server and it needs to react to changes and control their lifetime. After that, we register the Handler (line 10, the stub was generated during the bootstrap step as well) and then you Run the whole thing in line 11. The generated handler stub is just a generic handler of any type of objects. These generic objects are switched into domain-specific object types as soon as possible and handled accordingly with (full source):

func (h *Handler) Handle(ctx context.Context, event sdk.Event) error {
	switch event.Object.(type) {
	case *v1alpha1.Netperf:
		netperf := event.Object.(*v1alpha1.Netperf)
		return h.operator.HandleNetperf(netperf, event.Deleted)
	case *v1.Pod:
		pod := event.Object.(*v1.Pod)
		return h.operator.HandlePod(pod, event.Deleted)
	default:
		logrus.Warnf("unknown event received: %s", event)
	}
	return nil
}

OK, now we have everything needed to write a custom Kubernetes controller in place: Custom Resource Definition (CRD), Netperf Custom Resource (CR) and the control loop, where we can easily react to the incoming information about the system state we have to create. Now the “only” thing left is to add our business logic that does that.

Business logic, a.k.a. how to run Netperf on Kubernetes using API calls

I won’t describe the whole code line by line here, you can check the source here. But I want to give an overview that will hopefully make the code easier to understand.

Handling state

The main concept is that we need to react to events in a different way, depending on what we have already completed and the state Netperf object is in. It’s basically a finite state machine: our action depends on the state we’re in and the event we receive. We assume a single Netperf object can be in one of the following states:

• NetperfPhaseInitial: resource was created, but no actions were taken yet;
• NetperfPhaseServer: server pod is being created, but the client is not yet started;
• NetperfPhaseTest: the client pod is created as well and the test should be in progress;
• NetperfPhaseDone: the client pod has stopped, we collect the result and stop both server and client pods; the test is complete;
• NetperfPhaseError: something bad happened and the controller was not able to complete the test; we can’t proceed and finish with an error.

Check the following image for visualization of states (circles), events (black font) and actions (violet font) we have to handle.

As you can see, to make everything work, we have to keep state – our current action depends on what previously already was applied to a Netperf object. That’s a major issue. We can’t just keep the state as the object’s state in the RAM memory, as we have to deal with failures. No matter if we want to run our controller on Kubernetes itself, as a Pod, or outside the cluster, as a standalone process – we have to handle restarts. So, the solution is to offload the problem of keeping state to some external durable storage. If our state is big, we could use some external key-value storage, like Redis or Etcd. Fortunately, our whole state is just a single variable with the state’s name. In this case, we can just keep the state within the Netperf object itself – the kubernetes API server will be our durable storage. And in the same go, we provide a better feedback to our users, who can check what’s the state of a Netperf test. When you write a Kubernetes controller, make sure you know what your state is and how to make it durable.

Still, the bottom line is: your process can terminate at any moment. Write a kubernetes controller assuming it can be terminated and restarted at any time.

Oh, one more thing: in your code, you should never change the object that you received in the control loop. Always make a copy of the object first – that’s why you have this generated DeepCopy() function. Then, save the copy with API server call.

Control flow overview

Let’s briefly go over what Netperf operator does for events in particular states:

• We start with HandleNetperf() method, which just checks if the event is about a Netperf object being deleted or created/updated (since we synchronize the state to match the API object, there’s actually no need to tell a create from an update event),
  • If it was a create/update, we go to handleNetperfUpdateEvent(), where we make sure that the Server pod is started and existing. Remember, we can receive this event multiple times, not just once or only when a new Netperf object is created! So, we check for the current Netperf object state. If it is “Initial” or “Server”, we run startServerPod(), which makes sure the pod either already exists or is created (but not blindly creates the pod). In other states, we ignore the update request, as to get to these states the server pod must have been already created.
  • If the delete flag was set,  we call deleteNetperfPods(), which basically does… nothing! Our netperf pods will be automatically deleted by the API server when the Netperf object that created them is deleted. This is possible because we correctly set the OwnerReference for our pods.
• The second entry point into our business logic controller is with HandlePod() method. Keep in mind that with this approach that method is called for every pod in the same namespace. So, we start with checking if the owner of the pod is an existing Netperf object. If not, our controller has nothing to care for it. If it is a related pod event, in handlePodUpdateEvent() we check if it is about a Server or a Client pod. Then, we call the event handler function for the specific kind of pod.
  • in handleServerPodEvent() we’re checking if the Server pod is already up and running. If not, we’re waiting. If it’s ready, we check if the client pod is already created for this Netperf object. Again, we’re not blindly creating a client pod, as it might have been already created. Instead, we check the current state and create the Client pod only if it doesn’t yet exist.
  • in handleClientPodEvent() we’re checking for the Client pod status. If it has already completed, we can get the output log from it, parse it and get the netperf speed result value. We’re also cleaning up both the Client and Server pod. Finally, we do the one last update of the Nerperf object, to include the test result in the Status part of the object definition.

Building and running the project

When you wrote a Kubernetes controller you naturally want to build and run it. Project building is described with a fairly complex workflow on the operator-sdk github page. This workflow includes building a docker image, then pushing it to the registry and after that deploying to a test cluster. This is nice when you want to test the full deployment cycle, but it’s terrible for development, as it takes a long time to build and deploy. On the github page of Netperf Operator, I described another approach for the development cycle, where you can just build and debug a local go binary, without even touching docker and deploying to cluster. Be sure to check it out. You can also find information about how to run the Netperf Operator there.

How to write a Kubernetes controller – a short summary

OK, this entry is a lengthy one, not mentioning the linked code. Still, I really think that using a library like operator-sdk makes the whole thing much easier and faster. And creating a custom controller using a Custom Resource Definition really opens up plenty of possibilities for your needs. After all, Kubernetes is “just a new operating system” – at some point you have to write your own application… or rather write a Kubernets controller!

The post Write a Kubernetes controller (operator) with operator-sdk appeared first on Tailored Cloud.