Posted on

Raspberry Pi 4 Microk8s Kubernetes Clustering Gotchas

3 node raspberry pi 4 cluster

I’ve been running a few docker workloads on various stand-alone raspberry pi 4 hosts. This has worked well, but I decided to up my game a bit and setup a Kubernetes cluster. Why? Kubernetes is the container orchestration technology that is taking over the cloud and figured it would be a good learning opportunity to figure out how all the bits play together.

For my workloads, I need a 64 bit OS and I am using raspberry pi 4 8GB boards with a power-over-ethernet (POE) hat. I am using Ubuntu Server 64 bit and I am using Microk8s for the Kubernetes runtime. The tutorials are straight forward and I am not going to rehash that, but instead call out the gotchas to look out for.

CoreDNS

For my infrastructure stuff, I use DHCP reservations with long leases and make an internal DNS entries. This is a lot easier to centrally manage that doing static address assignments. I knew I was going to need k8 DNS support, so I did the following….

microk8s enable dns

And then when I moved my docker hosted container into a pod it failed. After a little troubleshooting to make sure there wasn’t any network layer issues, and validating that I could resolve external DNS names, I knew the problem was CoreDNS wasn’t pointed at my internal DNS servers. There are a couple ways to fix this…

# pass the dns server ips when enabling coredns 
microk8s enable dns:dns1, dns2

# or you can edit the configuration after-the-fact
microk8s kubectl -n kube-system edit configmap/coredns

Private Registry

I wanted to run a private registry to start with. Why? ISP connections can fail and it is also a fast way for me to experiment. Microk8s is the container orchestration layer, and it is using Docker for the container runtime. Docker by default will attempt to use HTTPs when connecting to the registry, which breaks with the private registry. You will see an error such as “…http: server gave HTTP response to HTTPS client.”

I am running a 3 node cluster, and I setup the registry storage on node-01. So we have to make some configuration edits…

# edit the docker /etc/docker/daemon.json file and add the ip address or FQDN to the registry host. I did this on each node of the cluster
{
  "insecure-registries" : ["xx.xx.xx.xx:32000"]
}

# restart docker
sudo systemclt restart docker

# now edit the container template and use the same 
# ip address/FQDN. I did this step on each node in 
# the cluster to make sure everything was consistent.
# The point of a cluster is to let the cluster consider 
# all resources when picking a host, so each node needs 
# to be able to pull the docker images if there is a 
# redeployment, scaling event, etc.
sudo nano /var/snap/microk8s/current/args/containerd-template.toml

 [plugins."io.containerd.grpc.v1.cri".registry.mirrors."xx.xx.xx.xx:32000"]
          endpoint = ["http://xx.xx.xx.xx:32000"]

# after making the edit/saving, restart the microk8s node
microk8s stop
microk8s start

I ported the leo ntp time server monitoring to run in the microk8s cluster. It has worked flawlessly until it croaked. The entire cluster was jacked up. I was using channel=1.20/stable which was the latest release at that time. I have since rebuilt the cluster to use channel=1.21/stable and everything has been bullet proof.

Posted on

Machine Learning to Eat Free

This hack is brilliant:
In today’s digital age, a large Instagram audience is considered a valuable currency. I had also heard through the grapevine that I could monetize a large following — or in my desired case — use it to have my meals paid for. So I did just that.

I created an Instagram page that showcased pictures of New York City’s skylines, iconic spots, elegant skyscrapers — you name it. The page has amassed a following of over 25,000 users in the NYC area and it’s still rapidly growing.

Posted on

Why We Cannot Have Nice Things: Comment SPAM

Just finished trashing 183 spam comments.  The spam techniques were interesting – some used flattery to try to disguise the spam, others were just flat out proud of their spam.  I could install comment filtering tools, but that’s work and cost.  Time to change the comment settings.

Posted on

Hello WordPress

I’ve wanted to host a blog for a bit now, and was leaning towards using a WordPress host.  I’ll pay them a little fee, my small little blog gets on the ‘ole Interwebs, the hosting company gets the headache and I get to focus on writing about stuff.

However, with the easy access to cloud technologies and architectures, this would be a lost opportunity.  I’ve had to build and run an ecommerce platform before and it’s work. Even if you automate a lot of stuff, you still have the operational issues such as having to work thru a bad patch, preventing malicious traffic, scaling traffic, browser differences, etc.

I don’t want that level of a headache, but I figure I can create a site that can be highly scalable, fully automated, but at a basic wordpress hosting price point.  I’ll write up the details and gotcha’s in a future post.