Latest Posts (15 found)
Dizzy Zone 3 weeks ago

Redis is fast - I'll cache in Postgres

There are books & many articles online, like this one arguing for using Postgres for everything. I thought I’d take a look at one use case - using Postgres instead of Redis for caching. I work with APIs quite a bit, so I’d build a super simple HTTP server that responds with data from that cache. I’d start from Redis as this is something I frequently encounter at work, switch it out to Postgres using unlogged tables and see if there’s a difference. I’ll run the experiment on my homelab’s k8s cluster. The idea is to run Postgres or Redis on one node, limiting it to 2CPUs via k8s limits, as well as 8GiB of memory. On another node, I’ll run the web server itself and then spin a pod for the benchmark executed via k6 on the third. Both postgres and redis are used with the out of the box settings for the following images: I wrote a simple webserver, with 2 endpoints, a cache and a “Session” struct which we’ll store in the cache: For redis, I’ve implemented the cache using as follows: The postgres cache is implemented using the library: I’ll seed the redis and postgres with 30 million entries each, keeping record of the inserted uuids. From there, I’ll generate a subset of existing uuids to use while benchmarking. This allows for simulating both hits and misses. I’ll do a few runs of benchmarks for gets first, then sets and then a mixed run. Each run will execute for 2 minutes. I’ll look at the number of operations per second, latencies as well as memory and CPU usage during those times. To simulate a somewhat real scenario where only a subset of keys exist in the cache the set benchmark will have a 10% chance to update an existing key, whereas the get will have an 80% chance of picking an existing key. The mixed workload will have a 20% chance to execute a set scenario and 80% for the get scenario. Redis performed better than Postgres, which did not surprise me at all. The bottleneck was actually the HTTP server. The machine running the http server maxed out on CPU, with redis running comfortably with ~1280mCPU - short of the 2000mCPU limit imposed. Redis used ~3800MiB of RAM, which stayed flat across the runs. For postgres, the bottleneck was the CPU on postgres side. It consistently maxed out the 2 cores dedicated to it, while also using ~5000MiB of RAM. Redis also did better when it comes to latencies of the HTTP responses: Once again Redis performed better. The CPU usage stayed roughly the same as in the case of the GET experiment, with the RAM usage growing to ~4300MiB due to the new keys being inserted. The bottleneck stayed on the HTTP server side, with Redis using ~1280mCPU once again. Postgres once again was bottlenecked by the CPU, constantly using 100% of the 2 cores it was limited to. During the course of the run, the memory usage grew to ~5500MiB. During the test, the endpoints with the Redis cache implementation also had better latencies: The mixed benchmark also returned the predictable result of Redis reigning superior. As has been the story so far, the CPU stayed put at ~1280mCPU, RAM usage grew a bit due to the new keys being inserted. Postgres maxed out the two cores and reached around 6GiB of memory used. Latencies once again were better when using redis: In the benchmark, I’ve used an unlogged table for postgres but this has not seemed to help, or has it? If I rerun the same benchmark with a normal(logged) table we can look at the numbers. The unlogged table makes a huge difference for the write benchmark and a somewhat smaller but still significant one for the mixed workload. This is because the unlogged tables skip the write ahead log making them a lot faster for writes. There’s very little difference for the read performance though and I expect more runs would show the two test cases converging. Redis is faster than postgres when it comes to caching, there’s no doubt about it. It conveniently comes with a bunch of other useful functionality that one would expect from a cache, such as TTLs. It was also bottlenecked by the hardware, my service or a combination of both and could definitely show better numbers. Surely, we should all use Redis for our caching needs then, right? Well, I think I’ll still use postgres. Almost always, my projects need a database. Not having to add another dependency comes with its own benefits. If I need my keys to expire, I’ll add a column for it, and a cron job to remove those keys from the table. As far as speed goes - 7425 requests per second is still a lot. That’s more than half a billion requests per day. All on hardware that’s 10 years old and using laptop CPUs. Not many projects will reach this scale and if they do I can just upgrade the postgres instance or if need be spin up a redis then. Having an interface for your cache so you can easily switch out the underlying store is definitely something I’ll keep doing exactly for this purpose. Thanks for reading!

performance Performance database Database
0 views
Dizzy Zone 2 months ago

n8n and large files

I’ve self hosted an n8n to automate video uploads to youtube. The idea was pretty simple - when I put an file in a specific dir on my nas, n8n would pick it up, as well as a file. The metadata file would contain things like the title and description, and it would upload this to youtube. I started creating this automation and pretty soon had something that resembled the finished workflow. I started testing it out. However, I ran into a problem. I had a node in my workflow, and whenever I manually executed it, it would take way too long to finish - a couple of minutes. My first thought was OK - the video files are pretty large, and we’re likely loading them into memory. Then again, it should not take that long, as the video I was testing with was about 200MB in size. I tried playing around with it a bit more but could not get it to work properly. The spare time I had ran out, so I left it as is to continue with it when I next have time for it. A few days pass, and I notice that the s3 bucket I use for off-site backups started rapidly increasing in size. My backup script takes in a bunch of directories in my NAS that I consider critical, things like my database dumps, configuration files and similar. These are then backed up using kopia to an s3 bucket. The backup is pretty small, usually. The databases are a hundred MB, other files in the backup take a similar amount of space. It was very unusual to see the backup gaining a couple of gigabytes a day. So I went to investigate. I looked at the backup, and noticed that my database dump went from 104 MB on one day, to 2.37GB the next. I quickly determined that the culprit was the table in the n8n database by first running the query: This helped me determine which database was the culprit. I then executed the following on the relevant database: Which listed as the one to blame. How so? Well… apparently, when you use the node in n8n, the data for the run is saved as is in the table, meaning that all executions saved the mp4 file to the database. This is also the reason why the node took so long to execute. To alleviate this, you should set the env var to as per n8n docs . Removing the relevant workflow runs from n8n and running a postgres vacuum shrunk the database dump back to 100MB. The step is now also taking seconds, not minutes to complete. Having this data stored in the database allows for the n8n runs to be repeatable, but doesn’t play very nice if you’re working with larger files, such as video. I haven’t tinkered as much yet, but I guess using the filesystem mode would mean that if you move the file in the filesystem, you’re not going to be able to fully reproduce a workflow run in n8n. However, if you are self hosting, working with large files and don’t really care about the ability to reproduce a run, it’s probably best to use the filesystem mode.

performance Performance database Database
0 views
Dizzy Zone 2 months ago

Malicious Node install script on Google search

Sometimes I have to install Node on my machine for work, or a personal project. This occurs rarely, so I keep forgetting how to do it. So I did what I usually do, and Googled how to install nvm. To my surprise there’s a sponsored result, which immediately triggers a red flag: This link leads to a repo in Github. It has the following readme - I’ve omitted the domain and path: If we look at the script in question: It prompts for your pass, stores it in a file and downloads a binary. It then executes that binary and your system is compromised. This attack seems to target quite a few google keywords, as I’ve tried other queries related to installing Node and quite a few of them show the sponsored malware. I’ve reported the repository and the ad as malicious and hopefully Github/Google will take it down. Nevertheless - it serves as a reminder that it’s as important as ever to stay vigilant and never execute arbitrary scripts on your machines. Stay safe!

security Security nodejs Node.js
0 views
Dizzy Zone 3 months ago

Wrapping Go errors with caller info

I find Go’s error handling refreshingly simple & clear. Nowadays, all I do is wrap my errors using with some additional context. The pattern I tend to use most of the time just includes a function/method name, like so: If I now call , the error - when logged or printed - will look like this: . This gives just enough context to figure out what happened, and where. I thought to myself - this can be wrapped in a helper function, to automatically take the caller name and do this wrapping for me. Here’s an implementation: We can now use it like so: And it will return the following: . Keep in mind that this will have performance overhead: So it’s 4-5 times slower than a simple wrap using . One benefit of using is that refactoring is easier - you change the function/method name and don’t need to bother yourself with changing the string in the . The performance hit is likely not an issue for most programs, unless you’re wrapping tons of errors in hot loops or something like that. I’m a bit torn if I want to add this helper to my codebases and use it everywhere - seems like an extra dependency with little benefit, but I’ll have to give it a spin before I decide if this is something I’m ready to accept. Time will tell. Thanks for reading!

go Go programming Programming performance Performance
0 views
Dizzy Zone 6 months ago

BLAKE2b performance on Apple Silicon

For work, I was going to store some hashed tokens in a database. I was going to keep it simple and go with HMAC-SHA256 for it but having recently read Jean-Philippe Aumasson’s book “Serious Cryptography” I remembered that BLAKE2 should be quicker: BLAKE2 was designed with the following ideas in mind: … It should be faster than all previous hash standards Cool, I thought, let’s consider BLAKE2 then. First, let’s write a simple benchmark to see just how much faster BLAKE2 would be than HMAC-SHA256. Performance is not important for my use case as the hashing will almost certainly not be a bottleneck but I was curious. So I write a benchmark: Run it and the results are: OK… So BLAKE2 is slower than HMAC-SHA256. Yes, we have less allocations which is nice, but it does take quite a few more CPU cycles. My first thought is that it might indeed be faster but only if the input is like way, way longer. So I switch to the following token: Hmmmm… This is even worse. Now, it could be that we’re dealing with a non optimal implementation. SHA256 is implemented in the stdlib in Go and very likely well optimized, whereas the BLAKE2 implementation I’m using comes from the package. Perhaps we can find a better one. The first search result recommends which has been archived since 2018. Not promising, but let’s give it a shot. These are the results with the original token. So this is even slower… I quickly glance over the codebase of and notice all the architecture specific files but there aren’t any for ARM architecture. Let’s try on a machine with an AMD64 processor. Right, so it seems that the implementations I’m using are only optimized for AMD64. Or are they? One final test, I spin up an ARM based VPS on Hetzner to test it out. Note, that since the benchmark was not able to determine the exact CPU model the CPU line was omitted. So with this, it seems that the issue only occurs on Apple Silicon processors. In the benchmarks with other processors, the BLAKE2b does win. I’m not entirely sure what causes this as CPU architectures are not my strong suite. If you do - please let me know by posting a comment.

performance Performance
0 views
Dizzy Zone 7 months ago

State of my Homelab 2025

For many years now I’ve had at least one machine at home which would work as a server to host some apps. In the last couple of years I’ve been getting more into it which has led me to purchase additional hardware. I’ve decided it would be nice to document my journey so I’ll try to make a post like this - detailing the setup that I currently have with both the hardware and the software once a year. Since this is (almost) the first post on my homelab setup let me start with some background. I am a software developer who likes to dabble in infrastructure. I like hosting things that I use on my own servers. However, as much as I’d like to run them on racks due to the electricity costs in Europe this is prohibitively expensive. What is more - this requires space, which I do not want to sacrifice at home. Therefore, my homelab is on the small side - both physically and from the resource perspective. I also tend to try not to spend too much on hardware so I end up running old and/or second hand hardware. The homelab consists of 4 machines. These are: MSI Cubi 3 Silent NUC This NUC has been running for close to 6 years now. I originally bought it in 2019 with 16GB of RAM, but since upgraded it to 32GB. It still has the original SSD in it and has run faultlessly. I’ve an old blog post about this machine specifically, if you’re interested. Currently, this machine acts as k3s server. 2x Lenovo ThinkCentre M910 Tiny These I bought post lease for around 130 Euros each in 2023. I’ve no clue how much use they saw before becoming part of my homelab, but they have been running flawlessly for the last two years here. These things are tiny and powerful. They are also quiet. This was very important when I was living in an apartment. If I ever need to expand my homelab again, I’m sure to be looking for more post lease ThinkCentres. These machines are the agents on my k3s cluster. Mostly second-hand NAS Originally assembled at the end of 2023. The new parts on the NAS are the case and the storage drives. The rest of the components I scavenged off of local bulletin boards & local recycling companies working with electronic waste. The whole system(minus the storage) set me back around 250 Euros, with the case being half that. I would have gone for a bigger case(a used one) but living in an apartment space was at a premium and I could not find anything second-hand that would fit the specific place I had in mind for this machine. I would also have gone for HDDs were it not for the noise. This machine now runs TrueNAS SCALE and serves as durable storage for the k3s cluster. I currently live in a location where no wired internet is available. Therefore, I’ve a 4/5G CPE (H352-381) providing the main connectivity. Since the quality of the internet depends heavily on the load of the nearby cellular towers, I see quite dramatic slowdowns during busy hours, such as Sunday evenings. On working days, I’ll get 200-300Mbps down, 50mbps up. On weekends and busy evenings this can drop down to 50 down, 10 up. Knowing that wireless internet can be a bit of pain, I’ve opted to for a router with dual WAN support. I went for TP-Link’s ER605. The plan is to get a starlink at some point and plug that in. From there, I’ve 2x TP-Link Deco XE75 wireless access points - one per floor of the house. These have far exceeded my expectations. They provide excellent coverage even in a concrete house and far beyond it. There are also a couple of unmanaged switches somewhere along the way here as well. I also own a few shelly smart plugs, those are mostly used for measuring the power consumption of the homelab and for whenever I’m curious about a random device’s power consumption. Besides that, I have 2x Shelly H&T Gen 3 humidity and temperature sensors for monitoring these around the house. Over the years, I’ve grown quite conscious about how much electricity the devices around the house use. When choosing the hardware I always try to consider energy consumption, to a reasonable degree that is. The homelab uses approximately 49 watts of power in total with the usual load it’s seeing. The NAS is consuming ~22w of that, the Lenovo ThinkCentres take around ~8W each and the Cubi draws ~10W. At the current electricity prices here this comes to right around 7.5 euros a month. I’ve written in more detail about the power consumption of these machines in my previous blog post if you’re interested. The homelab neatly fits in a custom built rack with NAS sitting on the bottom shelf, and compute units above that. There’s a power strip attached to the side of it, as well as a small switch. Each of the machines is plugged into a shelly smart plug, which continuously emits updates of the usage to an MQTT server inside the k3s cluster. The rack sits in the utility room of my house. Here’s what it looks like: The idea here is to put a monitor on top in the future, as well as a keyboard to allow for easy access if the servers would start acting up. This, however, is still in the works. I’ve purposefully left some space to be able to shove more one litre computers if needed. The NAS machine runs TrueNAS Scale . It has a single 4-wide RAIDZ2. 2TB drives each give me a usable capacity of 3.4TB. While this might not sound like a lot, I don’t really store much media here. I store data for applications that I run in k8s and in my case these are usually tiny. I expose the dataset as an SMB share. The only additional application I end up running on the NAS is netdata for metric collection. This netdata instance only collects the metrics and forwards the data to a netdata parent running on the k3s cluster. I’ve written a blog post on how to do that previously . Periodically, a job on the k3s cluster will backup the important bits from NAS into an off-site s3 bucket. I use kopia for backups. The cluster consists of a single k3s server and two agents. With the exception of my NAS, all the other hosts run debian . On top of that, I have k3s installed. From there, I use helm with the helm secrets plugin to deploy a whole bunch of things on the kubernetes cluster. The helm charts are stored in a private git repo on github. Some of the first helm charts I installed on the cluster are the kubernetes csi driver to access the SMB share on the NAS and MetalLB for any services that I need exposed on the local network. I also deploy a PostgreSQL instance to serve as my main database. For each of the domains I own and use (including the one you’re currently on) I deploy three things - a cloudflared instance for the cloudflare tunnel which forwards traffic to an nginx instance. This then forwards requests further to either a specific kubernetes service or authelia for authentication. For monitoring I use Netdata. I use their helm chart in the k3s cluster, with a single parent and children running on each node. It will automatically pick up prometheus metrics from pods with the appropriate annotations present. I think it’s time to move away from the infrastructure and talk about the actual things I have running there: The whole cluster uses around 1.1 cores under normal load and around 19GB of RAM. While I’m generally happy with my homelab, there are a few points that I got wrong or would like to improve. First, I’d like to run postgres on a completely separate host, or more ideally on two separate hosts with automatic failover. For this, I’d like to use the two M910 Tiny’s I already have. This would require me to remove them from the k3s cluster. I’d like to replace them 2 small machines, possibly with the Intel’s N series processors. However, I’ve struggled to find second hand options for them. I’d use them as k3s servers with the whole cluster consisting of 3 servers. This would allow a failure of a single node with no downtime. Second, I wish to switch my NAS to an HDD array for extra storage. This will come in handy for future projects. I would also like to move away from SMB to NFS shares, as I’d expect those to play nicer with linux and mac machines at home. I chose SMB for the theoretical use-case of having a windows machine at home, but I’ve never gotten one. Third, I intend to move away from cloudflare tunnels. I’ll use a VPS and some self hosted tunnel alternative, I’m yet to decide which one. I’ll need to write an ansible playbook to set the server up to my liking so that I can freely move between VPS providers if/when needed. Fourth, there’s a few applications that are on my to self-host list: Fifth, integrate all the utility appliances at home, such as the recuperator and the heat pump with the dashboard mentioned above to allow for easy control in a single panel, instead of having to use multiple applications. Sixth, add a UPS to the homelab. This is a nice to have and might not happen any time soon but it’s something I could grab if the right deal comes along. Seventh, add an outdoor thermometer or a whole weather station to the setup and a prometheus exporter for it so it can be visualized in Netdata. Eight, actually mount a monitor on top of the rack. A keyboard would help there as well in case I need to directly hook into one of the machines. And probably a few other things as well… That’s what my homelab setup looks like. If you made it this far - thank you for reading. If you have any questions, do drop a comment below!

hardware Hardware devops DevOps
0 views
Dizzy Zone 7 months ago

My homelabs power consumption

My homelab consists of 4 machines currently. When choosing them I tried to be energy conscious - using hardware which would not consume too much electrical power, while still trying to maintain the up-front cost low. These are mostly older systems and I was unable to find decent power consumption numbers for them. 1x MSI Cubi 3 Silent NUC 2x Lenovo ThinkCentre M910 Tiny 1x Mostly second-hand NAS Since these are all currently plugged into shelly smart plugs, I’m able to tell the power usage of each of them separately. The plug measures the power consumption and reports that to EMQX via MQTT. From there, I have a custom prometheus collector implemented. It subscribes to the relevant topics on EMQX, and exposes a endpoint in the prometheus exposition format. This is periodically crawled by netdata, which is running in the same cluster. I created a dashboard in netdata which allows me to visualize the CPU usage of each machine and its power consumption. It looks something like this: I’m not going to measure the consumption of the systems when they are idle as there are quite a few things running on my cluster and I wouldn’t like to interrupt them. However, I’ll show the current CPU usage and the power draw from the plug. The CPU usage is what I’d call steady in my homelab, it rarely spikes and seems to be pretty consistent across the board. We’ll call this the baseline. I’ll then do a 10 minute stress test using the tool on each machine, to see what the power consumption looks like at that moment. Note that the CPU % shows the % of cpu used across all cores - 100% indicates that all cores are working at their max. The cost estimates assume a price of 0,211479 €/kWh - coming directly from my last bill and I’ll call 30 days a month. Here are the results for the baseline: And for the full load test: Thanks for reading!

hardware Hardware data-visualization Data Visualization
0 views
Dizzy Zone 7 months ago

On Umami

I’ve been using Umami analytics on this blog for quite some time now. I self host an instance on my homelab. I spent a bit of time researching self-hosted analytics and generally they had a few issues. First, I’d like the analytics platform to be privacy focused. No cookies, GDPR compliant, no PII. Umami checks this mark. Second, many of the alternatives had quite a bit of hardware resource overhead once hosted. They would either consume a ton of memory, the cpu usage would be high or require me to host something like ClickHouse to run them. Since this blog is not the new york times I feel like hosting a specific database for it would be overkill. My homelab is rather small, so keeping things minimal is how I manage to stretch it. Umami consumes around 1% of a CPU core and ~240MiB of RAM on my homelab. Third, postgres as the datastore. Postgres is my go to database, and I host tons of small tools that use it as the backend. I like having a single instance that can then be easily backed up & restored, without having to resort to a ton of different databases. Therefore, any analytics tool would have to use it. Fourth, the tracking script should be minimal in size, not to affect load times too badly. The blog itself is pretty lightweight, so any tracking script bloat would defeat that. The Umami script has a content lenght of 1482 bytes once gzipped, not too shabby. Generally, I’ve been happy with the choice. However, there is one thing in Umami that annoys me more than it probably should: the visit timer. Apparently, the visit time is only updated once a user navigates to another page on the blog. If they simply leave, there’s no visit duration stored whatsoever. This makes the visit time tracker completely useless. I’m not the first one to notice this but the issue has since been moved to a discussion which has seen no progress. Good news is there’s a few things one could do - perhaps add a custom event to track this? Or fork Umami, since it’s open source and fix it. Both of these fall strictly into my “can’t be arsed to do” category, so I guess it’s not that important. Thanks for reading! Perhaps there are other analytics tools that tick the boxes above? Let me know in the comments below.

devops DevOps open-source Open Source Analytics
0 views
Dizzy Zone 10 months ago

ML for related posts on Hugo

After reading the technicalwriting.dev post on embeddings I thought to myself - this seems like something that I can implement quite quickly and would serve as a good starting point for some hands on experience with machine learning. I want to start small, so something as simple as filling the related posts section of this blog seems like an ideal candidate to get my hands dirty. This blog is made with Hugo and uses the related content feature which provides a list of related posts based on the tags & keywords you use. While I have no quarrels with the mechanism, I thought this would be a good place to try and experiment with embeddings. Since this blog is static, and I’d like it to keep it that way wherever possible I figured I’ll need to pre-compute the related posts for each post whenever a Hugo build is issued. However, there seems to be no way to natively hook into the Hugo’s build pipeline, so I’ll have to handle this via a script. The script would build a matrix of posts and their embeddings, compare each post to the rest of the posts and rank them according to their cosine similarity. I’d then keep the top N of those. I would then need to get Hugo to render these as links to those posts. To achieve this, I’ll have to save the recommended posts as part of the original post, probably in the frontmatter, so that these are available when rendering the related posts section via Hugo. This pretty much lead me to the following architecture: I could write everything in Python and have a single CLI for this, but my intent is to have this as a part of a generic blog cli helper tool which I’d like to keep in Go. Ideally, I’d stick with Go for the API part as well but using Python here makes life easier due to the excellent ML support and plentiful examples. I might re-visit this choice at some point though. Keep in mind that any code you find here is just a script intended for personal use, it will not be the prettiest or most optimized code you’ve ever seen, but that’s not my goal here. ML models can be huge. I’m not aiming for a world-class solution here, and I’d like something that is reasonably small as I intend to run it on my homelab. My homelab runs on very modest hardware - second hand 1L computers with no dedicated GPUs, old CPUs and limited amounts of RAM, so I’d like something relatively small. Since I’m going to feed in full blog posts, I might choose something with a larger token limit. Other than that, I have very limited knowledge of what I should pay attention to as far as models for this specific task. After typing in a few search queries, I’ve stumbled upon the Massive Text Embedding Benchmark (MTEB) Leaderboard . I then glanced through the list, choosing jina-embeddings-v3 due to its small size(2.13GB listed memory usage) and good accuracy - that’s the hope, at least - it’s rank 26 on the leaderboard at the time of writing after all. From there, I’ve opted to quickly build a super small python HTTP API with a single endpoint. It takes a POST at with a payload of: And returns an embedding: I tested this locally and found that the embedding calculation can take some time. For longer texts, this might take as long as 10 seconds. While I don’t expect to run the process for all the posts in my blog frequently, I figured I’d need to add some persistence here. I intend to run this on my homelab, which consists of 1L post lease machines with no dedicated GPUs and old generations of CPUs so it would be even slower there. To keep things simple, I opted to add support for a postgres table that stores the hash of the input text as well as the resulting embedding. Upon receiving a request I’ll calculate the hash of the text and check for its existence in the database first allowing for quick responses if a text has already been processed. In reality this means that unless I edit a blog post, an embedding response for it should be rapid if I had it calculated in the past. I also added an optional API token to add some very crude auth support. I then wrote a Dockerfile for it, and proceeded to host it in my homelab. With it now accessible by my CLI, I had the API ready. Oh, and it’s currently using right around ~3.6GiB of memory: To calculate embeddings for a single blog post usually takes around 7 seconds on my hardware. The time scales with the length of the post. It also seems to consume a full CPU core when doing so: The full code for the API is available here . The folder structure of my blog posts looks something like this: The cli accepts a base path, in my case. It will walk through the directory, parsing each file as a post. It will store it in memory, in the following structure: Once a markdown file is read, I do a few operations on it. First, I strip the hugo shortcodes. Then I render the markdown document to HTML by using the blackfriday markdown processor . Afterwards, I strip out the HTML tags. This leaves me with a plain text, which is ready to be passed into the model for embedding calculation. I did this because I worried that the raw contents of the file with shortcodes might yield results that are not representative. If that’s true or not - I do not know. Now I need to calculate the embeddings. This is done in a following pass through, where I iterate over all the posts and fetch their embeddings from the api. The third and final pass over the posts calculates the cosine similarity between each of the posts. It then takes the top 3 most similar posts, and marks their paths in the frontmatter struct’s similar posts property. The markdown documents are then saved. This is all quite inefficient and I could optimize this, but I’m really not worried about the performance here. There are also a few caveats - first, there’s a script that updates the markdown files that I’m working on. I’m not very fond of this, as it could lead to loss of data. If I were to run the script and while it’s executing change something, the change would get overwritten by the script once its job is complete. However, since I’m the only person working on this blog, and I have it version controlled I deem this an acceptable risk. I’ve had to do it this way as I could not figure out an alternative to hook into the hugo’s build chain. If you’d like a deeper look at the code - the full code for the CLI is available here . With all my posts now containing a property in their frontmatter, all that’s left to do is to adjust the template I use to render the related posts section. The template will render the property if it’s present. If it’s not, it will fall back to the Hugo’s built-in related feature. Here’s what it looks like: With this, the blog is armed to provide ML related posts instead of relying on keywords and tags. Let’s take my previous post - Probabilistic Early Expiration in Go - and compare the tag based related posts vs the ML related posts. For this particular case, I feel like the tag based recommendations are semantically closer to what I would expect. Nevertheless, I’m planning on sticking with the ML generated ones at least for a short while to get to know them better! So, is it all worth it? Well, probably not. I wasn’t really doing this for any sort of tangible result. I don’t even have the metrics to verify that this indeed has any effect. However, this was a fun little experiment and a good entry to getting to know ML a little. I’ll probably do more and deeper dives into ML at some point and maybe re-visit this feature for a second iteration in the future. Thanks for reading! If you’ve got any comments or suggestions, do leave a comment below.

python Python web-development Web Development programming Programming html HTML ai AI
0 views
Dizzy Zone 1 years ago

Probabilistic Early Expiration in Go

I often end up in situations where I need to cache this or that. Often, these values are cached for a period of time. You’re probably familiar with the pattern. You try to get a value from cache, if you succeed, you return it to the caller and call it a day. If the value is not there, you fetch it(most likely from the database) or compute it and the put it in the cache. In most cases, this works great. However, if the key you’re using for your cache entry gets accessed frequently and the operation to compute the data takes a while you’ll end up in a situation where multiple parallel requests will simultaneously get a cache miss. All of these requests will independently load the from source and store the value in cache. This results in wasted resources and can even lead to a denial of service. Let me illustrate with an example. I’ll use redis for cache and a simple Go http server on top. Here’s the full code: Let’s put some load on the endpoint and see what happens. I’ll use vegeta for this. I run . Vegeta ends up making 500 requests every second for 30 seconds. I graph them as a histogram of HTTP result codes with buckets that span 100ms each. The result is the following graph. When we start the experiment the cache is empty - we have no value stored there. We get the initial stampede as a bunch of requests reach our server. All of them check the cache find nothing there, call the and store it in cache. Since the takes ~500ms to complete any requests made in the first 500ms end up calling . Once one of the requests manages to store the value in the cache all the following requests fetch it from cache and we start seeing responses with the status code of 200. The pattern then repeats every 3 seconds as the expiry mechanism on redis kicks in. In this toy example this doesn’t cause any issues but in a production environment this can lead to unnecessary load on your systems, degraded user experience or even a self induced denial of service. So how can we prevent this? Well, there’s a few ways. We could introduce a lock - any cache miss would result in code trying to achieve a lock. Distributed locking is not a trivial thing to do and often these have subtle edge cases that require delicate handling. We could also periodically re-compute the value using a background job but this requires an extra process to be running introducing yet another cog that needs to be maintained and monitored in our code. This approach might also not be feasible to do if you have dynamic cache keys. There is another approach, called probabilistic early expiration and this is something I’d like to explore further. This technique allows one to recompute the value based on a probability. When fetching the value from cache, you also compute if you need to regenerate the cache value based on a probability. The closer you are to the expiry of the existing value, the higher the probability. I’m basing the specific implementation on XFetch by A. Vattani, F.Chierichetti & K. Lowenstein in Optimal Probabilistic Cache Stampede Prevention . I’ll introduce a new endpoint on the HTTP server which will also perform the expensive calculation but this time use XFetch when caching. For XFetch to work, we need to store how long the expensive operation took(the delta) and when the cache key expires. To achieve that, I’ll introduce a struct that will hold these values as well as the message itself: I add a function to wrap the original message with these attributes & serialize it for storing in redis: Let’s also write a method to recompute and store the value in redis: To determine if we need to update the value based on the probability, we can add a method to : If we hook it all up we end up with the following handler: The handler works much like the first one, however, upon getting a cache hit we roll the dice. Depending on the outcome we either just return the value we just fetched, or update the value early. We’ll use the HTTP status codes to determine between the 3 cases: I start up vegeta once again this time running against the new endpoint and here’s the result: The tiny blue blobs there indicate when we actually ended up updating the cache value early. We no longer see cache misses after the initial warm up period. To avoid the initial spike you could pre-store the cached value if this is important for your use case. If you’d like to be more aggressive with your caching and refresh the value more frequently, you can play with the beta parameter. Here’s what the same experiment looks like with the beta param set to 2: We’re now seeing probabilistic updates way more frequently. All in all this is a neat little technique that can help with avoiding cache stampedes. Keep in mind though, this only works if you are periodically fetching the same key from the cache - otherwise you won’t see much benefit. Got another way of dealing with cache stampedes? Noticed a mistake? Let me know in the comments below!

backend Backend performance Performance
0 views
Dizzy Zone 1 years ago

SQLC & dynamic queries

SQLC has become my go-to tool for interacting with databases in Go. It gives you full control over your queries since you end up writing SQL yourself. It then generates models and type safe code to interact with those queries. I won’t go over the basics here, if you feel like it you can try their interactive playground . Frequently I end up needing to filter the data by a set of fields in the database. This set of fields is often determined by the caller, be it via REST API or other means. This means that the code I’m writing has to support dynamic queries, where we query by a subset of fields. Let’s see an example. Assume my API returns some car data and I store it in the following table: The user might want to filter by brand, or by model. Or by brand and model. Or by brand, color, model, state and body type. You get the point, there’s a whole bunch of permutations here and SQLC is not great at handling this. I usually approach it with the following SQLC query: It might not be the prettiest solution, but it has worked for me quite well. There are a couple of downsides though. First, the param struct that the SQLC generates contains quite a bunch of fields: You’ll have to handle this in your code, to set the proper ones if the user provides the relevant input. Second, people always ask me if there is a cost associated with having such a query versus using something like a query builder to build the query with specific params. To this, I always answer: “Probably, but it’s unlikely you’ll notice”. I have decided to try and benchmark this, to see if there is a meaningful cost associated with it. I’ll use postgres & pgbench, since I’m only really interested in the query performance ignoring any code overhead. For schema, we’ll use the table above. We’ll seed the data with the following query: We’ll then have two different queries - one that we’d build with a query builder: And one where we’d have some extra overhead from our SQLC approach: I’ve also added a composite index for these specific queries: Armed with this, I’ve spun up an instance of postgres in docker on my machine, created the schema and generated the dataset. I then pre-warmed the cache by executing a couple of pgbench runs but not logging any results. From there, I ran pgbench 4 times: And these are the latencies for both queries after being run through plotly: There is very little difference between the two, with SQLC approach having a larger number of outliers. Despite this, I would state that there is no significant difference between the two. I then re-ran the experiment, only this time introduced a limit of 1 on both queries. My thought process was that this would perhaps allow the faster(in theory) query to shine, since we would not have to spend so much time transferring data. Here is the box plot: The SQLC approach does seem a tad slower here, but not by a lot. In conclusion, there is very little difference in terms of performance for these queries, meaning that using the a bunch of s to implement dynamic querying in SQLC is something you can do without fear of massive performance repercussions. Your mileage may vary and in extreme cases this might not be these - always benchmark your own use cases. The only price you pay for manually constructing the dynamic queries is the manual work involved in writing the queries & mapping of values from the request parameters to the query parameters. Thanks for reading! If you’ve enjoyed my babbling you might also like my post on SQL string constant gotcha .

go Go database Database sql Sql
0 views
Dizzy Zone 1 years ago

Enums in Go

I’ve seen many discussions about whether Go should add enum support to the language. I’m not going to bother arguing for or against but instead show how to make do with what we have in the language now. Enumerated types, or enums, represent finite sets of named values. They are usually introduced to signal that variables can only take one of the predefined values for the enum. For example, we could have an enum called with members , , . Usually, the members are represented as integer values, starting from zero. In this case, would correspond to 0, to 1 and to 2 with , and being the names of corresponding members. They help simplify the code as they are self-documenting and explicitly list all possible values for the given type. In many languages enums will also return compile errors if you’ll try to assign to an invalid value. However, since enums do not exist in Go, we do not have such guarantees. Usually, one of the first steps of defining an enum in Go is to define a custom type for it. There are 2 commonly used types for this purpose, and . Let’s start with strings: I actively avoid defining my enums in this style, since using strings increases the likelihood of errors. You don’t really know if the enum members are defined in uppercase, lowercase, title case or something else entirely. Besides, there is a high chance of miss-spelling the strings both in the definition and subsequent use and becomes . I also often use bitmasks so that might influence my judgement, but I’ll talk about bitmasks in a separate post at some point. For these reasons I prefer an int declaration: Keep in mind that this is highly subjective and people will have their own preferences. Also, the int definition does not read as nicely when displayed, but we will fix that later. In the colors example, I’m only using three colors, but what if we had 5, 10 or 20 of them? It would be quite tedious to assign values to each and every single one of them. Luckily, we can simplify this by using the keyword that Go provides: Iota acts as syntactic sugar, automatically incrementing the value for each successive integer constant in a constant declaration. If we’d like to start at another number, we can achieve this with the following: You can also use a variety of expressions on iota but I hardly recommend that except for the most trivial of cases as this leads to code that is hard to read and comprehend. One of the common use cases of such expressions which is still readable is defining bitmasks: For more on iota, please refer to the Go spec . One thing to note is that you should be very careful when making changes to already established constant declarations with iota. It’s easy to cause headaches if you remove or change the order of members as these could have already been saved to a database or stored in some other way. Once you ingest those, what was once blue might become red so keep that in mind. While such declarations might suffice as an enum in some circumstances you usually will expect more from your enum. For starters, you’d like to be able to return the name of the member. Right now, a will print , but how would we print the name? How would I determine if is valid color or not? I’m also able to define a custom color by simply defining a variable . What if I’d like to marshal my enum to their string representation when returning this via an API? Let’s see how we can address some of these concerns. Since we’ve defined Color as a custom type we can implement the stringer interface on it to get the member names. We can now print the name by calling the method on any of the and get the names out. There are many ways one could implement this method but all of them have the same caveat - whenever I add a new color in my constant declaration, I will also need to modify the method. Should I forget to do so, I’ll have a bug on my hands. Luckily, we can leverage code generation with the stringer tool can help us. It can generate the code required for our Color enum to implement the stringer interface. You’ll need to have the stringer tool installed so run to do that. Afterwards, include the following directive, I usually plop it right above my enum type declaration: If you run you’ll see a file appear, with the stringer interface implemented, allowing you to access the names of the members like so: If we use our enum in a struct and marshal it it will be represented as int: Sometimes, this behavior might suit you. For instance, you might be OK if the value is stored as an integer however if you’re exposing this information to the end user it might make sense to display the color name instead. To achieve this, we can implement the method for our enum. I’m specifically implementing the over as the latter falls back to using internally in the std libs json library. This means that by implementing it we will get the color represented as string in the marshalled form for both JSON, XML and text representations and, depending on the implementation, perhaps other formats and libraries. If we’d like to accept string colors as input, we’ll have to do a bit more work. First, we’ll need to be able to determine if a given string is a valid color for our enum or not. To achieve this, let’s implement a function Once again, we could implement this in many different ways, but they will have the downside that if we’re ever expanding our enum, we’ll have to go into the and extend it to support our new members. There are tools that can generate this for us, and I’ll talk about them later. With this, we can implement the method and unmarshal an input with colors as strings like so: If an invalid color is provided, the unmarshalling will result in an error. Similarly, we could implement the and interfaces for database interactions: If you’re working with quite a few enums and need the custom marshalling and stringer/valuer/scanner interface implementations it can become quite tedious having to do all these steps for each of your enums. Everything that I’ve discussed so far can be generated with the go-enum library . With it, the enum definition becomes a bit different: If you run a file will be generated including all the custom marshalling, parsing and stringer/valuer/scanner implementations. This is a great tool if you work with multiple enums and have to do this often. Another alternative that leverages generics and avoids generation is the enum library . Both of these are valid options and it is up to the reader to choose one that suits your needs. I will go over my preferences at the end of this blog post. There’s one caveat with these enums and that’s the fact that one can just construct a new enum member by hand. There’s nothing preventing me from defining a and passing that to a function expecting a color: Firstly, I would like to say that there is no bulletproof way to protect from this, but there are some things you can do. If we define our enum as a struct in a separate package: You would obviously include ways to construct valid enums from outside the package by either the ID or name, the methods for serializing, stringifying and any other needs you have. However, this only provides an illusion of safety, since you can do any of the following: For 2) we could shift our enum by 1, and include an unknown value with the id of 0. We’d still have to handle this unknown value in any code dealing with colors: Since structs can not be declared const, we have to become inventive to cover ourselves from 1). We can define the colors as funcs: With this in place, you can no longer assign a color as another. An alternative approach would be to make the color type an unexported int and only export its members: To make this type even remotely useful, we could export and implement a Colors interface: You could then use it like this: In theory, you could still write a custom implementation of this interface and create a custom color like that but I think that is highly unlikely to happen. However, I’m not a big fan of these approaches as they seem a tad cumbersome while providing little in return. With all this said, I’d like to point out a few things that have been working quite well for me in practice and my general experience: This is just my opinion and it might not match the situation you’re in. Always choose what works best for you! If you have any suggestions or alternatives, I’d be glad to hear them in the comments below.

tutorial Tutorial go Go Xml json JSON programming Programming
1 views
Dizzy Zone 1 years ago

Streaming Netdata metrics from TrueNAS SCALE

When you install TrueNAS SCALE your NAS runs a Netdata. You can verify that by executing in the TrueNAS shell. I use Netdata to monitor my homelab and have a parent set up for long term metric storage. I’d love to configure the NAS to also push metrics to a parent, so that I can access them all from a single place. Normally, if you want to set up streaming in Netdata it’s enough to edit . However, I wouldn’t recommend doing this on a TrueNAS install for a couple of reasons. Firstly, this is not a recommended way of adjusting configuration and that is clearly evident when you open up the TrueNAS shell: Secondly, if you ignore these warnings and use the shell to adjust the configuration, you’ll still end up in a pickle. Trust me, I’ve tried. The Netdata version that comes preinstalled with TrueNAS SCALE( ) is . You can verify this by running in the TrueNAS shell. This version is quite old, over a year old at the time of writing and seems to have bug in the streaming protocol. When I set it up to stream to my Netdata parent, the Netdata daemon in TrueNAS started crashlooping. This meant that no metrics were visible on the TrueNAS Web UI or being streamed to the parent. At that point I realized I’d probably need to use the Netdata app on True NAS. The good thing about this app is that it uses the Netdata helm chart under the hood, meaning new versions of Netdata are available. Simply install it from the available applications on the TrueNAS Web UI. As far as configuration goes, there is no need to configure anything. Once the app is installed and running, open up a shell by hitting the button on the app menu: We’re now inside the Netdata container running inside of the kubernetes cluster running on your TrueNAS SCALE. Using your favorite text editor, open and paste the following: Replace with the host and port of your Netdata parent and with your Netdata streaming API key. The next thing you’ll probably want to do is assign this node a proper hostname. By default, Netdata uses the hostname of machine it’s running on and since we’re on a pod in kubernetes you’ll get something catchy like . Luckily, we can fix that. I will also make a few general adjustments to the Netdata configuration, switching it to in memory mode as well as disable some auxiliary features as that will get handled by the parent. This makes the installation lightweight, consuming less resources of our NAS. Open and paste the following: Exit the shell, stop and start the app again. You should now be able to see the new node on your Netdata parent. If you don’t have a parent but would like to be able to access the metrics from Netdata Cloud it is possible to claim it using the same TrueNAS app. To do so, first grab a few things from the cloud. You’ll need the claim token, and the room ID. You can get them by hitting the button in the Netdata Cloud. You’ll see something like this: Next, head to the TrueNAS SCALE Web UI apps section, find and hit install on Netdata. However, this time some extra configuration is required. Under the you’ll want to add 3 environment variables: After the installation is complete, you should see your node appear on Netdata cloud. I hope this helps! If you’ve got any questions shoot them in the comments below.

shell Shell tutorial Tutorial devops DevOps
0 views
Dizzy Zone 1 years ago

SQL string constant gotcha

I was working on a project that uses PostgreSQL and as part of my task I needed to write a migration. The migrations are written as plain SQL and applied using the migrate library. The migration itself was not that complex, some rows needed to be updated where a column matched one of the values in a list. In my rush, I’ve opted for a rather simple query that went something like this: When the migration got executed on the testing environment, no errors were reported and all looked well. However, after some time I noticed that the migration failed to update some rows that I had anticipated. The observant readers probably noticed that there’s a missing comma after in the previous snippet and this is indeed what caused the issue. What surprised me is that the query did not result in an error. Why is that? Let’s explore. I’ll create a simple table and insert a few entries. Running the following query returns 3 rows, as expected. However, if I fail to include a comma after the resulting query returns a single row: The query does not result in an error and still yields some results. To better understand what is happening we can further simplify this example to a simple SELECT: The original query failed to include the sessions with and browsers because brave and firefox got concatenated resulting in the following query: What I find odd is that this behavior only exists if you include at least one newline between the two string constants. This means that results in a syntax error. This behavior is documented in the PostgreSQL documentation : This slightly bizarre behavior is specified by SQL; PostgreSQL is following the standard. After scratching my head for a bit, I then went on to try and figure out a way to minimize the chance of this happening to me in the future. If you’re working with strings using and providing an array as a parameter to it should keep you safe: Should you omit a comma it will let you know that you’ve got a .

sql Sql database Database
0 views
Dizzy Zone 6 years ago

Moving from Jenkins to Drone

I’ve written in the past that this blog is a playground for me to try various tools and play with code around it. Jenkins has been my choice as the CI for it since the start, mostly since it was something I’m used to. However, I’ve also stated that running it on an old laptop with no real backups is a recipe for disaster. I have since rectified the issue by hiding the laptop under a box in a closet but that meant moving away from Jenkins to something that’s lighter and more portable. The choice is the self-hosted enterprise edition of Drone . Drone consists of two parts - server and agents. The server handles auth, users, secret management, handles hooks for source control and orchestrates the work for agents. The agents are responsible for actually executing the workflows they receive from the server. This basically means that you can have workers anywhere as long as they can reach the drone server. Drone is written in Go and is extremely lightweight. This means that it has an extremely low memory requirements which is of great advantage to me as I’m trying to keep the costs to a minimum. The server I’m running takes up around 15MB of memory while the worker takes 14MB. Everything is dockerized so it’s super easy to deploy as well. My setup consists of a server running in an AWS EC2 instance and a worker running at home in a MSI Cubi Silent NUC I’ve recently acquired. The Raspberry I’ve used for jenkins is also a great candidate to run a worker but due to the workloads I throw at it(lots of disk io - the sd card can’t handle that. Looking at you - javascript.) it’s less than ideal in my situation. I’ll keep it on hand just in case I need more workers. The old laptop could also be a candidate here for the future. That’s part of the joy with Drone - you can literally run it anywhere. Drone looks for a file in an enabled repository. It’s in this file that you specify your pipeline. What makes drone great is that you can actually run the pipeline locally, using the drone cli. It makes testing the builds super easy. That’s a huge contrast to what I’m used to with Jenkins(Disclaimer: I might just be a scrub, I’m not hating on Jenkins). What this also means is that you really don’t need to worry about storing the jobs themselves anywhere as they are just as safe as the rest of your code in your source control. Hopefully anyway. The steps in the pipeline are run in docker containers which are thrown out after the pipeline is done. It means that the jobs are nice and reproducible. And while I hate YAML the pipelines are quite easy to understand. Click for a look at an example . I like it. Drone seems to be on a really good path towards becoming an excellent CI tool. There’s things missing though. It seems a bit basic. Things like the lack of global secrets(they are now defined per-repo instead or I didn’t manage to find them) or proper documentation as the current one seems a bit lacking. Took me quite a while to get my first proper job running and I’ve only managed that after I’ve looked at a few examples on the internet, rather than the docs. There’s also the question of pricing. The website is not super clear on the pricing, but from what I gather, the enterprise edition is free as long as you run less than 15000 jobs per year. The pricing afterwards is per user at an unknown rate. Anyways, I should be covered as I probably run less than 500 jobs per year and do not plan on adding new ones any time soon. There’s also the lack of option to run multiple different jobs from the same repository, which leads to a pattern on many small repo’s appearing on my source control. I’m not too fond of that and wish there was a way to schedule quite a few jobs from a single repo. Nonetheless, once you get the hang of it Drone seems to be a powerful tool. The UI is crisp, deploying it is easy and it runs on a potato. Everything I like from my CI installation. As for the lacking features - I’m sure they are coming soon™. I’ll keep this space updated with my latest projects running on drone. Stay tuned! Have you used Drone? Do you prefer something else? Do let me know in the comments below.

go Go devops DevOps cloud Cloud ci-cd Ci Cd
0 views