Posts in Tutorial (20 found)
JSLegendDev Today

My Biggest Gripe With YouTube

3 years ago, I started a YouTube channel called JSLegendDev where I uploaded tutorials teaching the JavaScript programming language through the development of 2D games. The state of the space around the time I started was as follows : Tutorials inferior to an hour in length were not in demand. They made very little views. Tutorials divided into multiple parts where dead on arrival. You were guaranteed dwindling views on every new upload. To adapt, other content creators started uploading longer, multi-hour, often project based tutorials which translated to more views. Seeing the shift, I also decided to follow suit and uploaded tutorials reaching the 4-10 hour mark. I saw some success doing this. Therefore, I kept at it for a while. However, as time passed, I got tired of recording extremely long tutorials and they, in general, started to make less views. There are many hypotheses as to why YouTube’s algorithm started serving tutorial content less. The advent of AI could’ve been the likely cause but also a general shift in YouTube becoming more of an entertainment focused platform to the detriment of educational content. Something you now put on TV to relax. In the programming space, channel producing content that can be watched passively like tech news, tech drama, tech history, high level discussions, etc… continued to thrive. Seeing this new shift and because I was genuinely tired of making YouTube tutorials, I published my first scripted video titled “How do Devs Make Levels Without Game Engines” which was first published as an article. In that piece, I told the story of how I discovered a convenient way to design levels for my games using an external editor called Tiled in conjunction with my editor-less game framework. At the end of that video, I promoted a paid tutorial I made teaching the exact steps needed to achieve what was presented. The video ended up accumulating over 30k views, which was pretty great! It took far less effort to make compared to my multi-hour tutorials and I was able to make a few sales on my paid tutorial I mentioned within. Previously, I was very unsuccessful in selling any paid courses and I didn’t quite understand why. However, the answer now hit me like a truck. Why would anyone still have the appetite for a paid course after having invested the time following a free multi-hour course? Even if the subject of the paid offering was different, they would probably be too tired to commit to another one. Anyway, following in the footsteps of this first breakthrough, I uploaded another scripted video titled “You Can Now Make PS2 Games in JavaScript” which was again first published as an article. In that video, I told the story of how I discovered that you could make PS2 games in JavaScript and provided an overview of how the viewer could get started. Despite including very practical knowledge, the viewer was never expected to follow along and therefore could watch it passively. It was a resounding success, over 100k views! Unfortunately, I didn’t sell any courses in that video because I simply didn’t have the energy to both make the video and a course. The best business decision would have been to wait before uploading. I’ll go into more details later, but my biggest gripe with YouTube is that it’s no longer a great platform to build an audience but rather it’s only good for reach and here, I had wasted a lot of reach. After having made so many game development tutorials, I wanted to try my hand in creating an original game that I would sell on Steam. Once the project was starting to take shape, I had the idea of making a video about it to gauge interest as I wasn’t sure it would find an audience. Therefore, I had the idea of using the same format used in my two previous successful videos. However, rather than focusing on technical details, I instead would tell the story of how I came up with my game’s design covering the various iterations and challenges I faced while working on it. Therefore, I ended up uploading a video titled “Making a Small RPG” which again, was originally an article. It was also a resounding success reaching barely below 100k views! However, it came with a hidden cost. That cost was the tipping point that made me realize that YouTube is no longer a good platform to build an audience on. I naively thought that if the video performed well, this would translate to subscribers and an audience eager to hear more about the project, but this wasn’t the case. I had made a big mistake by not setting up a Steam page to direct viewers to before publishing the video. On my next upload concerning the project, the fall off in terms of views was brutal. I went from 98k views to below 10k. It became clear that YouTube was acting as a gatekeeper between me and the audience I thought I had built. After reflecting on the situation, I came to the following conclusion. The reason my 3 previous videos had performed well was due having certain characteristics that aligned with YouTube’s goal as a platform, which consists in making people watch videos for as long as possible so they can serve more ads. I listed them below : The subject of all three videos were remarkable which lead to people clicking on them. Something is remarkable when it obviously stands out as being interesting/noteworthy. For example, the subject of my video titled “You Can Now Make PS2 Games in JavaScript” is remarkable because the PS2 is a very popular, but now old console and you had to use a hard programming language called C++ to make games for it. Being able to now use JavaScript, a simpler but most importantly, a language originally designed for making websites and not games, makes the subject come across as immediately noteworthy. Therefore, remarkable. The use of storytelling made people eager to watch more of the video. This can be explained by the fact that we instinctively want to know what happens next in a compelling story. Finally, the length of the videos were all above 10 minutes and the 2 more successful ones were in the 15+ min range. This resulted in more absolute watch time compared to shorter content. For example, if 2 videos are both watched fully by the same audience. The shorter one will translate to less total time spent on the platform compared to the longer one. Therefore, YouTube will recommend the longer one instead because there’s an opportunity cost to doing otherwise. To understand the fall off, it’s important to first mention that usually, series on YouTube don’t work. The second video of a series ends up making less views than the first because it requires prior context before clicking. Thus reducing its appeal and limiting its reach. However, I knew this going in. I tried making the second video as independent as possible but in the end, a second video talking about the same subject was bound to be less remarkable. It didn’t help that because I summarized the content of the first video in the second one, a familiar viewer would have found it less engaging making the video further away from hitting criteria 2 and 3 that I outlined above. Consequently, I realized I had wasted my biggest marketing ammunition regarding my small RPG game as I had no way to contact the audience hit by the first video. Like with the one on making PS2 games in JavaScript, I had wasted tremendous reach. At this point, I realized my biggest gripe with YouTube was simply that I could not access my audience reliably. Therefore, was it really my audience? On one hand, YouTube allows someone without a following to reach millions but on the other, the link to those reached is fickle. I thought I was building an audience by gaining subscribers but instead, I was building a sand castle that could easily be carried away by the slightest algorithm waves. YouTube wasn’t always like this. People used to subscribe to channels and seek their content in their subscriptions tab. However, the platform effectively buried this model by conditioning users to seek recommended videos on the home page and deprioritizing the Subscriptions tab to the point that it barely looks like a clickable section. You have to click on the “Subscriptions” text to access your sub feed. Doesn’t look very clickable doesn’t it? I think that we’re now entering an era where YouTube is starting to treat content creators as interchangeable much like TikTok. They saw the success TikTok had, tried to replicate it with Shorts and now YouTube long form is getting affected as well. I fear that in the future, uploading to YouTube will look no different than making posts on Reddit. You might get views, you might get comments, but they’re self contained to a specific post with no following building up and no guarantee of your next posts having the same reach. The conclusion to all of this is that it’s not worth it to be a YouTuber. Relying on YouTube adsense and sponsorships (sponsors use views as a metric to determine how much to pay you) for your livelihood is simply not sustainable due to how fickle getting views on the platform is. Therefore, focusing so much on making YouTube content will most likely lead to your exploitation. That said, is quitting really the answer? Considering that YouTube can give you incredible reach even if you’re a nobody as long as you make content that is remarkable, engaging (for example, through storytelling) and long enough, it would be stupid to completely walk away, at least in my case. Therefore a new strategy appears on the horizon. It consists in building your audience outside of YouTube through a mailling list (Substack conveniently allows you to do so) and to strategically make occasional compelling YouTube content to tap into the platform’s reach potential. However, the key is to always direct viewers to the mailling list. Why is building an audience through email so important? because it allows you to have a direct and long lasting link with your audience. It also gives you independence from social media platforms. Even in the case of Substack, where this article is currently hosted, I can export my email list and move to another platform or email sending service without my subscribers even noticing. This shift implies that I no longer need to worry about pumping frequent content for YouTube because I’m not making money through them or worrying about doing so. By making YouTube content rarely, I get to keep most of my energy to build something compelling outside the platform like an actual game, writing interesting articles, making an in-depth course or other kinds of art/products. This plan seems to me as more sustainable and more healthy long term. That’s about all I’ve got to share. Hope this article was insightful. If you’re curious to see where this journey will lead, I recommend subscribing! I usually write about programming, game development and game design. Subscribe now You can check some of my previous articles below. Tutorials inferior to an hour in length were not in demand. They made very little views. Tutorials divided into multiple parts where dead on arrival. You were guaranteed dwindling views on every new upload. The video ended up accumulating over 30k views, which was pretty great! It took far less effort to make compared to my multi-hour tutorials and I was able to make a few sales on my paid tutorial I mentioned within. Previously, I was very unsuccessful in selling any paid courses and I didn’t quite understand why. However, the answer now hit me like a truck. Why would anyone still have the appetite for a paid course after having invested the time following a free multi-hour course? Even if the subject of the paid offering was different, they would probably be too tired to commit to another one. Anyway, following in the footsteps of this first breakthrough, I uploaded another scripted video titled “You Can Now Make PS2 Games in JavaScript” which was again first published as an article. In that video, I told the story of how I discovered that you could make PS2 games in JavaScript and provided an overview of how the viewer could get started. Despite including very practical knowledge, the viewer was never expected to follow along and therefore could watch it passively. It was a resounding success, over 100k views! Unfortunately, I didn’t sell any courses in that video because I simply didn’t have the energy to both make the video and a course. The best business decision would have been to wait before uploading. I’ll go into more details later, but my biggest gripe with YouTube is that it’s no longer a great platform to build an audience but rather it’s only good for reach and here, I had wasted a lot of reach. After having made so many game development tutorials, I wanted to try my hand in creating an original game that I would sell on Steam. Once the project was starting to take shape, I had the idea of making a video about it to gauge interest as I wasn’t sure it would find an audience. Therefore, I had the idea of using the same format used in my two previous successful videos. However, rather than focusing on technical details, I instead would tell the story of how I came up with my game’s design covering the various iterations and challenges I faced while working on it. Therefore, I ended up uploading a video titled “Making a Small RPG” which again, was originally an article. It was also a resounding success reaching barely below 100k views! However, it came with a hidden cost. That cost was the tipping point that made me realize that YouTube is no longer a good platform to build an audience on. I naively thought that if the video performed well, this would translate to subscribers and an audience eager to hear more about the project, but this wasn’t the case. I had made a big mistake by not setting up a Steam page to direct viewers to before publishing the video. On my next upload concerning the project, the fall off in terms of views was brutal. I went from 98k views to below 10k. It became clear that YouTube was acting as a gatekeeper between me and the audience I thought I had built. After reflecting on the situation, I came to the following conclusion. The reason my 3 previous videos had performed well was due having certain characteristics that aligned with YouTube’s goal as a platform, which consists in making people watch videos for as long as possible so they can serve more ads. I listed them below : The subject of all three videos were remarkable which lead to people clicking on them. Something is remarkable when it obviously stands out as being interesting/noteworthy. For example, the subject of my video titled “You Can Now Make PS2 Games in JavaScript” is remarkable because the PS2 is a very popular, but now old console and you had to use a hard programming language called C++ to make games for it. Being able to now use JavaScript, a simpler but most importantly, a language originally designed for making websites and not games, makes the subject come across as immediately noteworthy. Therefore, remarkable. The use of storytelling made people eager to watch more of the video. This can be explained by the fact that we instinctively want to know what happens next in a compelling story. Finally, the length of the videos were all above 10 minutes and the 2 more successful ones were in the 15+ min range. This resulted in more absolute watch time compared to shorter content. For example, if 2 videos are both watched fully by the same audience. The shorter one will translate to less total time spent on the platform compared to the longer one. Therefore, YouTube will recommend the longer one instead because there’s an opportunity cost to doing otherwise.

programming Programming javascript JavaScript c++ C++ tutorial Tutorial
0 views
miguelgrinberg.com 3 days ago

SQLAlchemy 2 In Practice - Solutions to the Exercises

To conclude with my SQLAlchemy 2 in Practice series, this article contains the solutions to all the exercises. If you'd like to support my work, I encourage you to buy this book, either directly from my store or on Amazon . Thank you!

tutorial Tutorial python Python database Database
0 views
@hannahilea 1 weeks ago

Text of the wild: Receive notifications from your BirdNET-Pi via ntfy

Tutorial for configuring a BirdNET-Pi with ntfy to send push notifications whenever a rare-to-you bird starts yelling

hardware Hardware tutorial Tutorial
0 views
Circus Scientist 1 weeks ago

Installing SmartPoi D1 Mini version with Arduino IDE V2

4. Go to Tools -> Boards -> Boards Manager and select esp8266 to install (may need to re-start Arduino IDE before it shows up) 5. Install the ESP8266 LittleFS Uploader program in Arduino: Step 1: Download the Plugin You need to put this file in a specific directory. If the folder doesn’t exist yet, you will need to create it. The workflow to actually upload files is identical to the old version: The console at the bottom will compile your file system image and push it straight to the flash memory! 5. Get SmartPoi from the SmartPoi Firmware Downloader website 6. Select options in Arduino IDE 2.0: 7. Compile and Upload 8. Do the LittleFS Filesystem Upload mentioned above (step 5.4) The post Installing SmartPoi D1 Mini version with Arduino IDE V2 appeared first on Circus Scientist . Download and install Arduino IDE V2 Go to Tools -> Manage Libraries and install FastLED 3.7.5 (ESP8266 version of SmartPoi will not work with the latest FastLED!) Go to File -> Preferences and input the following in “Additional boards manager URLs” (adding ESP8266 boards support) : http://arduino.esp8266.com/stable/package_esp8266com_index.json Open your web browser and go to the official GitHub releases page for the tool: GitHub: arduino-littlefs-upload Download the latest version ending in (for example: ). Windows: 1. Navigate to: 2. Look for a hidden folder named (note the dot at the front).3. Inside , create a new folder named .4. Move the file into that folder. macOS / Linux: Open Finder/File Manager and go to your home directory: (You may need to hit on Mac to see hidden folders). Create a folder named inside it. Drop the file into that folder. Restart Arduino IDE 2. In IDE 1.8, the tool lived in the Tools menu. In IDE 2, it lives in the Command Palette . Open the Command Palette by pressing: Windows/Linux: + + macOS: + + Type into the prompt. You should see the option: Upload LittleFS to Pico/ESP8266/ESP32 . Open your Arduino sketch. Go to Sketch > Show Sketch Folder . Create a folder named exactly alongside your file. Place whatever HTML, TXT, or config files you want inside it. Important: Select your D1 Mini board and port, and close the Serial Monitor (if open, it blocks the upload). Open the Command Palette ( ) and click Upload LittleFS to Pico/ESP8266/ESP32 . CPU Frequency: 160mhz Board: LOLIN(WEMOS) D1 R2 & Mini Flash Size: “4MB (FS:3MB OTA: ~512KB)” Debug Port: Serial (if you want to see serial ouput – optional) Select your port (COM1, USB0 …) Leave everything else on default settings

html HTML tutorial Tutorial json JSON hardware Hardware
0 views
マリウス 1 weeks ago

Photography Workflow with ~~Darktable on Linux~~ Lightroom on GrapheneOS

Disclaimer: I had initially prepared this post under the title Photography Workflow with Darktable on Linux , but after endless fights with Darktable I eventually decided to scrap that workflow altogether and look for an alternative. The workflow documented herein is unfortunately very far from the result I was striving for, yet it is sadly the best I can put together given the current state of open-source RAW development and photo editing software. After I gave Adobe the finger back in 2019 and moved my photography workflow to Capture One on a MacBook , I eventually had to reconsider this approach when I moved back to Linux on the desktop and replaced the device with a Linux laptop . I briefly tried running Capture One in a Windows VM on my laptop , but decided against it, as it was a huge PITA and lacked proper hardware acceleration. Initially I considered a fork of what is probably the best-known open-source RAW developer and photography workflow application out there, Darktable , called Ansel , but ultimately decided against it. The points that Ansel ’s author, Aurélien, brought up seemed like valid criticisms and demonstrated both his knowledge of and his passion for making Darktable a better tool. However, reading further through his website and his GitHub account, it became apparent that he might be the kind of misunderstood genius who has great ideas and ambition, but who would ultimately struggle to operate within, let alone lead the kind of community required to successfully maintain a fork of a piece of software this large. I therefore didn’t have high hopes of this lone cowboy keeping up with, let alone surpassing, the development efforts the Darktable community is currently putting in. Given that Ansel was explicitly billed as a hard-fork that would not remain compatible with the official Darktable release, going down that path felt too risky. Ansel would ultimately have to provide a migration path for existing Darktable users, as otherwise there would be little to no incentive for anyone with a functioning Darktable workflow already in place to put up with the effort. Instead, I decided to stick with Darktable . For about a year I tried to build a new workflow on top of it. The things I would miss the most from Capture One were the VSCO presets that I had brought over from Lightroom , and for which there didn’t seem to be any way to convert them into a format compatible with Darktable while producing roughly similar results. Luckily, João, a developer and photographer, made what he calls t3mujinpack , a collection of film emulation presets for Darktable . In a blog post , he provides details on which film stocks are included and how to make use of them in Darktable . His pack includes the presets I almost exclusively use from VSCO : Kodak’s Portra 160, 400 and 800. While the results aren’t 100% identical to what Capture One produces with the converted VSCO packs, neither are those exports identical to what Lightroom originally produced. Every piece of software has slight differences in its inner workings, so this is to be expected and can be adjusted for. During my travel through all of Spain in 2024 I decided to rely exclusively on Darktable for developing and editing the photos that I would ultimately upload to this site. That was a big mistake. I rarely say bad things about truly open-source software, because ultimately it is open-source, it’s driven by a community of volunteers, and everyone should be happy that these people do what they do. Also, given that it’s open-source, anyone is free to go ahead and improve what they deem worth improving. However, Darktable is, in my opinion, one of the few exceptions that seem to have derailed so badly that it’s fair to say it has reached a point of no return in terms of usability and jankiness . Let me explain by starting with one of the most annoying things: More often than not, Darktable crashes in the middle of editing sessions, apparently due to Wayland-related issues. However, since I’m also running GIMP and Blender , which I would argue do similar, or even slightly more complex things than Darktable , yet don’t run into such issues, I’d assume that this is not a problem with my Wayland setup specifically. I didn’t try to debug the issue further, as I was mainly focused on testing and establishing a workflow. Had Darktable otherwise worked perfectly fine for me and only run into this issue every once in a while, I would have dug deeper to find the root cause. Unfortunately, this was only one of many things that kept me from continuing to use Darktable . Besides the random crashes, Darktable is unbearably janky and slow. The UI feels like it’s about to fall over at any moment, regardless of whether ROCm acceleration is enabled or not. UI elements feel hacked together, the overall navigation is hostile towards regular users, and it’s impossible to find anything just by looking, because everything is hidden behind collapsed modules, tabs and a gazillion sliders and buttons. To give a single example of the sheer UI craziness that is Darktable : To rotate an image to the right (clockwise), you need to drag a slider to the left (counterclockwise). While on a touchscreen interface this might be more intuitive, when using a touchpad on a laptop or even a mouse it definitely doesn’t feel natural. After all, maybe a slider isn’t the best UI element for this operation to begin with? Another issue that I experienced was related to organizing photos. With over 4000 (RAW) photos in the library, Darktable becomes unbearable to work with. Aside from the spontaneous crashes and overall slow UI, finding specific photos in a library of that size is an excruciatingly painful task. Unlike Capture One and Lightroom , Darktable doesn’t easily support a workflow based on individual, smaller libraries, e.g. organized by location or event. There are ways to sort photos within Darktable ’s main library, but I couldn’t find an easy way to split them out into multiple small libraries. Assuming that you managed to find and edit the photos you were looking for, the headaches continue when you try to export them. It appears that Darktable is unable to export photos with pixel-perfect adherence to the crop aspect ratio . The implementation details and the proposed solution appear to be just as janky as everything else, and a quick search for in the Darktable GitHub repository uncovers a lot more of that same jankiness. I ended up running the following command over every photo exported by Darktable , just to obtain a properly shaped image, meaning I’d lose a few pixels here and there: As mentioned a long time back in an update , I ended up with a broken Darktable library, meaning that I lost all the adjustments that I did manage to export up until that point . Short story long, I eventually ditched Darktable for a plan B . After Darktable broke my library and I lost months’ worth of edits, I found myself back at square one. The idea of returning to Adobe felt like defeat, but when I looked at what was actually available for my setup, which is a Google Pixel Tablet running GrapheneOS , Adobe Lightroom for Android turned out to be the only realistic option that could handle RAW files and offer a non-destructive editing workflow. Adobe Lightroom Mobile is, on paper, a reasonably capable RAW editor for Android. It supports a wide range of camera RAW formats and offers the familiar tone curve, HSL sliders, color grading, masking, and healing tools that anyone coming from desktop Lightroom will recognize. It can read photos directly from the device’s own storage, edit them locally without an internet connection, and export to JPEG with full control over quality and output dimensions. In short, the feature set is there. The physical side of the workflow is straightforward. I attach a USB-C SD card reader to the Pixel Tablet, open a file manager, and copy the RAW files from the card into a dedicated folder on the tablet’s internal storage. From there I open Lightroom , import the photos from that folder into a local album, and work through them one by one. Once a photo is where I want it, I export it as a JPEG into the folder on the tablet’s storage. That folder is monitored by Syncthing , which synchronizes the finished exports to my other devices in the background. The performance of Adobe Lightroom on Android is, to put it mildly, terrible. Rendering a RAW preview after entering edit mode takes long enough that you find yourself staring at a loading indicator more often than at the actual photo. Scrolling through a grid of thumbnails is a choppy, stuttering affair that makes you wonder whether the application is doing something computationally expensive or is just poorly written. I acknowledge that the Pixel Tablet is an older budget device, yet Lightroom treats it as if it were running on hardware from 2005. Lightroom on Android is every bit as buggy as Adobe products traditionally are on macOS and Windows, but somehow worse, because the interface is also frequently broken in ways that make the application essentially unusable without restarting it. The UI will routinely enter a state where confirmation and action buttons either stop responding to taps, as if the touch layer has fallen out of sync with whatever is rendered on screen, or simply disappear altogether. The only resolution is to quit the app and reopen it, at which point you hope that the edit you were in the middle of survived. Entire features will similarly go dark without warning. The auto-straighten function, which should detect the horizon in a photo and level it, simply grays out and stops working at some point. No error, no indication as to why it has become unavailable, nothing. Again, restart the app, try again, maybe it works this time. These are not edge cases or exotic scenarios, but rather the normal operating experience of Adobe Lightroom Mobile . One of the things I was most concerned about before committing to this workflow was the prospect of Adobe silently uploading my photos to their cloud infrastructure. The desktop version of Lightroom has a long and well-documented history of syncing content to Adobe’s servers in ways that are easy to miss and difficult to fully disable. On Android, GrapheneOS gives you a tool that the desktop doesn’t: Per-application network permission revocation. I first disabled the cloud sync option within Lightroom ’s own settings, then went into GrapheneOS’s permission manager and removed the network permission from the Lightroom app entirely. It continues to function as a local RAW editor without any network access whatsoever. Photos stay on the device. Nothing leaves without my explicit say-so via Syncthing. Note: To keep things simple, I did not go into the fact that Lightroom is running inside an Android 16 Private Space , which also contains a sandboxed instance of Google Play Services and lets me create a virtual barrier between the rest of the FOSS apps on the Pixel Tablet and this spyware malware crap proprietary software. With this setup, however, importing data becomes slightly more tedious, as it requires the Google Files app to be able to read an attached USB-C storage device (SD card) from within the Private Space . The Google Files app is a giant UX disaster all by itself, into which, for the sake of our both’s time and mental health, I won’t dive into. One pleasant surprise was that I managed to import the VSCO Lightroom presets I purchased well over a decade ago into Lightroom Mobile on Android. The preset files still work, and the film emulations I had relied on for years, in particular the Kodak Portra series, show up in the presets panel and can be applied to photos. With Adobe being Adobe, however, this had to come with a catch. Lightroom Mobile is apparently incapable of remembering which preset was applied to a given photo. Open an edited photo that had a VSCO preset applied, and Lightroom will display a warning telling you it cannot find the preset, even though the preset is sitting right there in the presets list, available and functioning, ready to be applied to new photos. The edit itself is intact… well… at least sometimes. Other times, Lightroom simply loses the edits altogether. It’s the kind of bug that suggests the feature was never properly tested beyond the initial happy path, which is about what you’d expect from Adobe. To be frank, this workflow sucks compared to the one I had on macOS using Capture One . Lightroom is still the terrible POS it had always been, and paying money to a company like Adobe feels like funding a criminal organization. Unfortunately, there doesn’t appear to be a viable alternative, especially not one that’s libre . The remaining options would be to either pay into Apple’s walled garden by purchasing one of their newer iAmtheproduct devices and subscribing to Capture One Mobile , or to rely exclusively on Fuji’s in-camera film simulations (which sadly won’t work for the Sony ). Judging by the reviews of Capture One Mobile , however, the former option doesn’t appear too promising either. Looking at the situation in a more positive light, I nevertheless managed to replace the underlying stack on which my photography workflow runs with more privacy-respecting software ( GrapheneOS ). That’s at least something , although it seems this workflow won’t live that long either, given that Google keeps locking down their Pixel devices and GrapheneOS appears to be pivoting to Motorola-made hardware , who might not release a GrapheneOS-compatible Moto Pad anytime soon. Oh well. Pro tip: A USI 2.0 pen makes using Lightroom on a device like the Pixel Tablet significantly less painful, at least as long as the USI pen actually works properly, which sadly isn’t always the case with the Renaisser pen I own. If you’re looking for a more general review of the Google Pixel Tablet with GrapheneOS, look here .

tutorial Tutorial open-source Open Source
0 views
Martin Fowler 1 weeks ago

Maintainability sensors for coding agents

In her recent article about harness engineering for coding agent users, Birgitta Böckeler laid out a mental model for expanding a coding agent harness: a system of guides and sensors that increase the probability of good agent outputs and enable self-correction before issues reach human eyes. Birgitta has now started publishing an article where she walks though her experiences using sensors to keep a codebase maintainable. This part looks at static analysis with basic code linting.

tutorial Tutorial devops DevOps programming Programming
0 views
miguelgrinberg.com 2 weeks ago

SQLAlchemy 2 In Practice - Chapter 8: SQLAlchemy and the Web

This is the eighth and final chapter of my SQLAlchemy 2 in Practice book. If you'd like to support my work, I encourage you to buy this book, either directly from my store or on Amazon . Thank you! Whether you are building a traditional web application, or a web API that works alongside a web front end or smartphone app, SQLAlchemy is one of the best choices to add database support to a Python web server. In this chapter two example integrations with Flask and FastAPI will be demonstrated. These are two of the most popular Python web frameworks and should serve as examples even if you use another web framework.

tutorial Tutorial python Python
0 views
Michael Hoffmann 2 weeks ago

Nuxt Tip: Difference Between useFetch and event.$fetch

Understand the key distinction between useFetch and event.$fetch in Nuxt. Learn why using event.$fetch for server-side API calls is the recommended approach for better performance and SSR correctness.

web-development Web Development tutorial Tutorial
0 views
Kev Quirk 2 weeks ago

Replacing my ISP router with a UniFi Cloud Gateway Max

So I recently upgraded my home internet to full fibre, after which I also decided to upgrade my router as there were some things I wanted to do with my network that my ISP-provided router wasn't capable of. I replaced my mesh system with a UniFi one a couple years ago, so it made sense to stick with the UniFi brand and go with one of their routers, so £250 later, I had a Cloud Gateway Max on its way to me. I figured this would be a straightforward process, but my god was I wrong! So I took a backup of my Cloud Key 1 config and figured I could unplug that, plug in the Cloud Gateway, restore the config and be done. I assumed there would be a couple things I needed to tweak, but for the most part, it would be a simple 10 minute job. You see, dear reader, in order to configure the Cloud Gateway you need an internet connection. No internet connection, no configuration. So by unplugging my ISP router -thus killing the internet to my entire house - I couldn't even get to the point where I could enter my ISP credentials, let alone configure the bloody thing. Without the internet connection all I could configure was the IP and MAC of the router. Absolutely pointless! There may be a way of doing this without an internet connection, but I couldn't find it and it certainly wasn't obvious. So I had to reconnect my old rig - the ISP router, the Cloud Key, and access points. Then I hung the Cloud Gateway off the ISP router so it could get an internet connection. Luckily this worked and I was finally able to configure the thing. After which I disconnected the Cloud Key, assuming the access points would all fail over to the Cloud Gateway when I restored the config backup from the Cloud Key. You see, the config back from the Cloud Key is a completely different file format ( ) to what the Cloud Gateway was expecting ( ). What the actual fuck! Soooooo back online went the Cloud Key, and I had to remove all 4 access points from there, just so I could "adopt" them with the Cloud Gateway. Then I had to manually setup my SSIDs and DHCP so it all matched the old rig. But finally, after 3 hours of fucking around, a job that I thought would take 10 minutes was done. UniFi is really good kit and has lots of features, but I don't understand why it has to be so difficult to set up. It feels like UniFi is the Apple of the networking world - they do everything they can to keep you in their ecosystem and up sell. Want our wifi? You're gonna need one of our routers, or this arbitrary piece of hardware for that. Oh you want to move an AP to a new management device? Yeah, you can't just move it - you need to do these 5 steps instead. Had I not already spent over a thousand pound on this UniFi kit, I would have chucked it all on eBay and gone with something else, but alas WiFi Apple has me in their walled garden! Anyway, it was a painful process, but it's working. And to be fair to UniFi, once it is all setup, it's rock solid and feature rich. I won't be upgrading again any time soon though, that's for sure! Now I just need to familiarise myself with all the nifty features the Cloud Gateway offers, so I can improve my network. Fun times! A Cloud Key is a stupid piece of hardware that is needed in lieu of a UniFi router. It controls the wireless access points.  ↩ Thanks for reading this post via RSS. RSS is ace, and so are you. ❤️ You can reply to this post by email , or leave a comment . A Cloud Key is a stupid piece of hardware that is needed in lieu of a UniFi router. It controls the wireless access points.  ↩

Networking hardware Hardware tutorial Tutorial
0 views
Alex White's Blog 2 weeks ago

Installing JPilot on Arch

This post is a quick tip for anyone else running into issues installing the Palm Pilot desktop software, JPilot on Arch Linux. If you just try installing via , the build will fail as the dependency no longer builds on modern systems. The solution is to first install , then .

open-source Open Source linux Linux tutorial Tutorial
0 views
Maurycy 2 weeks ago

Hosting a website on an 8-bit microcontroller.

In today's episode of "dumb things to do with an AVR microcontroller": MCU website demo (may go down if this gets posted to HN) My victim is the AVR64DD32 which is quite similar to the Atmega328 of Arduino fame. Compared to the older Atmega, these are cheaper for the same memory, use a single programming pin and have nicer peripherals: So that's the computer (and a rather spacious one at that) but it'll need an internet connection to host a website. The obvious choice is Ethernet , but even the slowest version (10BASE-T) still runs at 10 megabits/second. Worse, it uses Manchester encoding: a zero is sent as "10" and a one as "01", so 10 megabits of data is actually 20 megabits at the wire. This is simply too fast for the AVR to generate. While it's processor can run at 24 MHz, but all the peripherals and IO pins max out at 12 MHz. (although some other 8-bit chips can manage it) The proper solution is to buy a dedicated ethernet chip from DigiKey, but then I'd be waiting weeks to finish this project. ... and ethernet is far from the only option: Serial Line Internet Protocol (RFC 1055) is a very old and very simple standard for running networks over serial: Before sending a packet, wrap it in 0xC0 bytes. If the packet contains any 0xC0 bytes, replace them with 0xDB 0xDC. To avoid ambiguity, any pre-existing 0xDB bytes are replaced with 0xDB 0xDD. This scheme was widely used for connecting to the internet over modems: A old-school dial up modem just runs a serial link over a phone line, and it's up the the computer to do anything with it. ... which is why SLIP is still supported by modern Linux: The hardware on the microcontroller's end is trivial: It does work with no external components, but I wanted some blinkenlights, and an idiot-proofing diode for when I inevitably connect the power backwards. Because it only draws a few milliwatts, it's perfectly fine to run the server of the serial adapter's 5 volt rail: it's really nice to only have one cable to deal with. Now it has an internet connection , but that's hardly a server. In order for my web page to get to your computer, it needs to pass through dozens of different networks. To do this, each packet has an IP header: 40 bytes that contain the address of the source and destination computers, and some other stuff I don't really care about. The protocol used to be a lot more complex, with features like packet fragmentation that require a lot of memory to handle correctly, but I don't have to: every modern operating system disables fragmentation and IPv6 removed it entirely. This makes implementing it very easy: Just swap around the source and destination of a recieved packet to generate the header for the response. (and reset the TTL counter) The other protocol, TCP is a lot harder : Implementing it requires the microcontroller to track connection states, periodically retransmit lost packets and handle a huge number edge cases. It took several days to get my custom implementation working well enough, and it's still got a few bugs. As for implementing HTTP, I didn't: The server always sends a hardcoded "response" back to the client. This works fine as long as there's only a single URL on the site. [Video of the page loading. See web or files directory: loading.mp4] Ok great , but what if I want to share it with friends? Unfortunately, to do that, it needs a publically routable IPv4 address. Not only are these expensive (there's a limited number) but it's impossible to get a good internet connection at my place. (no, Starlink is not good) I do have a machine with a publically routable address, but it's at a datacenter near Helsinki: I'd need a very long serial cable... Another cool thing Linux supports is wireguard, which creates a virtual network link over the internet. This works even if one of the machines is behind (CG)NAT or other annoyances. Problem solved: have the Linux router box connect to the VPS to get a proper internet connection? ... except the MCU still doesn't have it's own IP address: I could forward everything from my VPS's address to it, but that would break my normal website. Instead, I setup the server to proxy any requests under to the server using a local address block. This means that visitors aren't directly connecting to the MCU's TCP/IP stack... but hey, it's the same setup that the Vape Server uses and no one complained. (It also makes it slightly harder to break by sending SYN packets, but it's not exactly hard to DDoS a server connected over what's effectively dial-up) /mcu : The page hosted from the microcontroller. http://ewaste.fka.wtf/ : The Vape Server, a website hosted off a 32-bit MCU pulled from the trash.

programming Programming tutorial Tutorial hardware Hardware
0 views
Alex White's Blog 3 weeks ago

Tiny Visitor Counter

I created a tiny script for counting per-page visitors on your site. It's as simple as uploading the PHP file to your server and pointing a tag to it. Leveraging the script as an image is an attempt to seed out bots (since they typically don't render images). Here's a live version of the script: You can grab the script on my Codeberg . To setup with Pure Blog , upload the PHP file to your folder and add the following HTML to your page and post footer HTML under Settings->Site (this assumes the script was uploaded to ):

html HTML tutorial Tutorial php Php web-development Web Development
0 views
The Coder Cafe 3 weeks ago

Cache Use Cases Explained

☕ Welcome to The Coder Cafe! Today, we discuss cache use cases. When we think about caching, it’s pretty frequent to focus on where it happens; for example, client-side, server-side, or in a CDN. Yet, there’s a more important question that should be answered first: What’s the use case? In this post, we will break down two common cache use cases: reducing latency and improving capacity. And we will see why the line between the two is blurrier than it seems. Get cozy, grab a coffee, and let’s begin! A Cache for Latency Latency is the time between when a request is sent and when a response is received. A cache for latency exists to reduce the average latency of a service . The classic access pattern looks like this 1 : We check the cache first. On a cache hit, we return the data directly without touching the backend. On a miss, we go to the backend, return the result, and store it in the cache for future requests. Why does this reduce latency? The cache keeps data in memory, which is significantly faster to read from than a remote database that may involve network round-trips, disk I/O, and query execution. On a hit, all of that work is skipped. In Soft vs. Hard Dependency , we introduced two kinds of dependencies: A soft dependency is a non-critical dependency for the service to operate properly. A hard dependency is a critical dependency for the service to operate properly. A cache for latency is a soft dependency . If the cache becomes unavailable, requests fall through to the backend. The system keeps working, just at a higher latency. Keep this in mind, because it’s the key difference we’ll come back to. A cache for capacity exists to serve higher throughput than the backend can handle on its own. The access pattern is identical to the latency case: cache first, then backend on a miss. So what actually makes these two different? The difference is not in the code; it’s in what the backend can absorb. In a capacity scenario, the backend would be overwhelmed if it received all the traffic directly. The cache absorbs a large portion of the requests, keeping the backend load manageable. This changes the nature of the dependency . If the cache goes down, the backend is suddenly hit with all the traffic it was previously shielded from. Whether the system survives depends on the backend’s own capacity. If the backend can scale fast enough, the cache is still a soft dependency: there will be a rough period, but the system recovers. If the backend can’t cope with the load, the cache becomes a hard dependency . Without it, the system fails . Here’s a question worth asking: if the access pattern for both types is identical, how do we know which one we have? In most cases, caches are introduced to reduce latency. But here’s what can happen over time: Our system is stable. Cache hit rates are high, backend load is low. Traffic grows. The backend load stays low because the cache is absorbing most of it. Nothing breaks. No alerts fire. Six months pass. Nothing has changed, no code, no configuration, no architecture decision. And yet the cache is no longer reducing latency. It’s keeping the backend alive. The cache didn’t change. The code didn’t change. The system grew around the cache, and the cache quietly became load-bearing . The same risk appears when a cache goes cold. For example: A migration to a new cache instance A data format change that requires purging existing entries A cache restart after maintenance Any of these can produce a large wave of cache misses in a short window. If we were running a latency cache, we would see higher latency for a while. If we were running a capacity cache, we would see a traffic spike that the backend can’t absorb. The unsettling part is that the code is identical in both cases. The difference only becomes visible at failure time . The root problem is that teams often don’t know which type of cache they’re running . They built it for latency, and that’s still how they think about it, even as the system outgrows that assumption. A few approaches help here: Periodically ask: could the backend handle the current traffic if the cache were completely removed ? Load testing without the cache, or estimating backend capacity against current traffic levels, gives you a concrete answer. Treat cache hit rate as a meaningful operational signal , not just a performance metric. A sustained drop in hit rate means the backend is absorbing more traffic than usual. If that trend continues, it’s an early warning that you may be drifting toward a capacity problem. When migrating a cache or invalidating a large portion of its data, warm the new cache before routing live traffic to it. This prevents a cold-start burst from hitting the backend all at once. Finally, once we recognize that a cache is operating as a capacity cache , we should treat it accordingly. It’s no longer optional infrastructure and it deserves proper alerting and a clear plan for what happens if it goes down. AI is getting better every day. Are you? At The Coder Cafe, we serve fundamental concepts to make you an engineer that AI won’t replace. Written by a Google SWE, trusted by thousands of engineers worldwide. A cache for latency serves data from memory to reduce average response time. It is a soft dependency: if unavailable, the system degrades in latency but continues to work. A cache for capacity absorbs traffic that the backend couldn’t handle on its own. It can be a soft or a hard dependency, depending on whether the backend can absorb the load without it. Both types share the same access pattern, which makes them easy to confuse. A latency cache can silently become a capacity cache as traffic grows, without any code change. When a capacity cache goes cold or fails, the backend can be overwhelmed. Hit rate monitoring, periodic load testing, and cache warming are practical ways to manage this risk. Availability Models Safety and Liveness The PACELC Theorem The Three Types of Cache Cache stampede Even though variations exist. A Cache for Latency Latency is the time between when a request is sent and when a response is received. A cache for latency exists to reduce the average latency of a service . The classic access pattern looks like this 1 : We check the cache first. On a cache hit, we return the data directly without touching the backend. On a miss, we go to the backend, return the result, and store it in the cache for future requests. A soft dependency is a non-critical dependency for the service to operate properly. A hard dependency is a critical dependency for the service to operate properly. Our system is stable. Cache hit rates are high, backend load is low. Traffic grows. The backend load stays low because the cache is absorbing most of it. Nothing breaks. No alerts fire. Six months pass. Nothing has changed, no code, no configuration, no architecture decision. And yet the cache is no longer reducing latency. It’s keeping the backend alive. A migration to a new cache instance A data format change that requires purging existing entries A cache restart after maintenance Periodically ask: could the backend handle the current traffic if the cache were completely removed ? Load testing without the cache, or estimating backend capacity against current traffic levels, gives you a concrete answer. Treat cache hit rate as a meaningful operational signal , not just a performance metric. A sustained drop in hit rate means the backend is absorbing more traffic than usual. If that trend continues, it’s an early warning that you may be drifting toward a capacity problem. When migrating a cache or invalidating a large portion of its data, warm the new cache before routing live traffic to it. This prevents a cold-start burst from hitting the backend all at once. Finally, once we recognize that a cache is operating as a capacity cache , we should treat it accordingly. It’s no longer optional infrastructure and it deserves proper alerting and a clear plan for what happens if it goes down. A cache for latency serves data from memory to reduce average response time. It is a soft dependency: if unavailable, the system degrades in latency but continues to work. A cache for capacity absorbs traffic that the backend couldn’t handle on its own. It can be a soft or a hard dependency, depending on whether the backend can absorb the load without it. Both types share the same access pattern, which makes them easy to confuse. A latency cache can silently become a capacity cache as traffic grows, without any code change. When a capacity cache goes cold or fails, the backend can be overwhelmed. Hit rate monitoring, periodic load testing, and cache warming are practical ways to manage this risk. Availability Models Safety and Liveness The PACELC Theorem The Three Types of Cache Cache stampede

backend Backend tutorial Tutorial performance Performance
0 views
Circus Scientist 3 weeks ago

Upgrading cheap LED Juggling Balls

Many years ago I bought 10 LED juggling balls from Oddballs in the UK. At the time, the special they had on was pretty good for bulk orders, but even today the balls are pretty cheap, if a bit basic (11 Pounds UK). The balls, being cheap, did not last. They have replacable batteries and the screw-in plug to close and switch on is a terrible design. Recently one of my upgraded K8 juggling balls broke somehow (a short from being dropped too much?). I fixed it up but I decided I needed some spares in case this happens again just before a show. Luckily I had some broken Oddballs balls lying around. I used the super bright LED’s from the ball and upgraded them with a 110mAh battery and Jack Switch for switching on/off and charging. Components: The post Upgrading cheap LED Juggling Balls appeared first on Circus Scientist . Oddballs Juggling Ball 110mAh LiPo battery Jack Switch

hardware Hardware tutorial Tutorial Electronics
0 views
Jim Nielsen 3 weeks ago

Reminder: You Can Stitch Together Lots of Little HTML Pages With Navigations For Interactions

I wrote about building websites with LLMs — (L)ots of (L)ittle ht(M)l page(s) — and I think it’s time for a post-mortem on that approach: I’ve tweaked a few things from that original post but the underlying idea is still the same, which I would describe as: Avoid in-page interactions that require JavaScript in favor of multi-page navigations that rely on HTML and are enhanced with CSS view transitions (and a dash of JS if/where prudent). As an example, on my blog I have a “Menu”. It doesn’t “expand” or “slide out” or “pop in” or whatever else you can do with JS. Instead, it navigates to an entirely-new page that is focused on just the menu options of my site. I say “navigates” because it’s just a link — — and it functions like a link, but the navigation interaction is enhanced by CSS view transitions. Have a newer device with a modern browser? Great, you get a nicer effect. Have an older device, or an older browser, or JS disabled, Et al.? It’ll still work. If you can follow a link — which is the most fundamental thing a browser can do — it will work. So how’s it all work under the hood? In essence, all the pages have a link to the menu (except the menu page). When you navigate to the menu, that link is changed to an “X” which “closes” the menu. The closing is still just a link (back to ) but it’s enhanced with JS to actually do a “back” in the browser history. This makes it so “opening/closing” the menu doesn’t add an entry to your browser history. As a simplified example, the code looks like this: The checks whether we came to this page as a navigation (mostly likely from within the blog itself) or via a direct visit (i.e. somebody typed it into the URL bar, unlikely but possible) which is how I suss out whether there’s a meaningful run or not. Here’s a video of how it all works, if that’s your thing: While this solution seems simplistic, it was not a simple thing to arrive at. It required me to spend time thinking about what was essential to navigation, how that interaction could work across multiple pages, and how I could ensure page size stayed small so the interaction was both fast and robust while remaining intuitive to use. In other words, the approach shaped the design. Turns out, if you have a website and you think of the browser as a way to navigate documents — rather than a runtime to execute arbitrary code and fetch, compile, and present them — things can be a lot simpler than our tools often prime us to make them. Reply via: Email · Mastodon · Bluesky

frontend Frontend web-development Web Development css CSS tutorial Tutorial html HTML
0 views
the website of jyn 4 weeks ago

callgraph analysis

I recently wrote a post for work titled "Callgraph Analysis", about how to write a custom lint with access to the full call graph of a program. You can read it on my work's blog .

tutorial Tutorial programming Programming
0 views
Eli Bendersky's website 4 weeks ago

Scaling, stretching and shifting sinusoids

This is a brief and simple [1] explanation of how to adjust the standard sinusoid sin(x) to change its amplitude, frequency and phase shift. More precisely, given the general function: We’ll see how adjusting the parameters , and affect the shape of s(x) . Each section below covers one of these aspects mathematically, and you can use the demo at the bottom to experiment with the topic visually. Scaling is conceptually the simplest change; we adjust to increase or decrease the amplitude (maximal height) of s(x) . Setting A=2 will make the value twice as large (in both the positive and negative direction) as the original function. Stretching changes the frequency of sin(x) , which is inverse proportional to its period. The baseline function sin(x) has a period of 2\pi , meaning it repeats every 2\pi . In other words, sin(x)=sin(x+2\pi) for any . If we set w=2 , we get sin(2x) . This function repeats itself twice as fast as sin(x) , because is multiplied by 2 before being fed into the sinusoid. If changes by \pi , the sinusoid’s input changes by 2\pi . Therefore, the period of sin(2x) is \pi , the period of sin(4x) is \frac{\pi}{2} and so on. [2] More generally, the period of sin(wx) is \frac{2\pi}{w} . Play with the demo below to see this in action, by changing and observing how the waveform changes. If we know the period p we want, we can easily calculate the that gives us this period: The final parameter we discuss is ; it’s called the phase of the sinusoid. In the baseline sin(x) , . The sinusoid is 0 at x=0 , achieves its positive peak at x=\frac{\pi}{2} , crosses 0 again at x=\pi , negative peak at x=\frac{3\pi}{2} and returns to its original position at x=2\pi where the repetition begins. By adding a non-zero , we don’t affect the sinusoid’s amplitude or frequency, but we do shift it right or left along the axis. For example, suppose we use the function sin(x+\theta) with \theta=\frac{\pi}{2} . Then when x=0 , we have sin(\frac{\pi}{2}) , so the sinusoid is already at its positive peak; at x=\frac{\pi}{2} , the sinusoid crosses 0 into the negatives, etc. Everything happens earlier (by exactly the value of \theta=\frac{\pi}{2} ) than in the baseline sinusoid. In other words, we’ve shifted the function left by \frac{\pi}{2} . Similarly, when is negative, everything happens later, and the function is shifted right . We’ve now gone over all the parameters for the function: Use the demo below to adjust these parameters and observe their effect on the sinusoid: controls the scaling factor (amplitude). is the frequency and controls the repetition period controls the phase - how much the sinusoid is shifted left or right

tutorial Tutorial Science
0 views
Giles's blog 1 months ago

10Gb/s Ethernet: what I actually did to get it working in my home

Having learned enough about 10Gb/s Ethernet to be comfortable about setting it up in my house, it was time to bite the bullet: order it from the ISP, buy some kit, and get started. I already had 2.5Gb/s working. The apartment has structured cabling -- each room has one or more RJ45 sockets in the wall, and there's a patch panel downstairs by our front door that has a matching patch socket for each wall socket. So when we moved in, I simply set things up so that there was a 2.5Gb/s switch down by the patch panel, and wired everything together there. Most of our stuff works over WiFi, of course, but I needed a wired backbone to connect the excessive number of computers in my study both to each other, and to the outside world. What did I need to do? Simplifying a bit, I had this 2.5Gb/s setup: There are a few other things dotted around, of course -- extra APs and what-have-you -- but that's the core, and I'll focus on that to keep things simple. Would I be able to get it all upgraded to work with 10Gb/s? The most important question was the structured cabling in the walls; was it CAT-5E or CAT-6, or even CAT-6A? Remember from the last post, 10GBASE-T might work over short runs of -5E (even though officially it's not meant to be able to). It probably would run over -6, because that's generally OK up to 55 metres or so, and I don't think any of the runs in the house are longer than that. And it would be fine over -6A, which is good for 100-metre runs. I was unable to find out exactly which type I had (the parts of the cables that are visible to me don't have any kind of marking to say), so I decided to do a staged rollout. The first step was to set up the wired network within my study as 10Gb/s. There were two important things to wire up; my primary desktop, , and a Proxmox cluster I have running in an 11" rack. The setup I had was just one 2.5Gb/s switch sitting on top of the rack, linked to the wall, to the cluster machines, and to . Now, getting the Proxmox cluster up to high-speed internal networking was a non-starter. The machines there are all old ones -- it's essentially a retirement home for mini-PCs I used to use for other things 1 . They're mostly gigabit ethernet, with one 2.5Gb/s one. But getting up to 10Gb/s was an important goal, as that's where I do most of my work. I also wanted to have space for a second machine that I'm planning to set up to do training/inference without tying up 's GPU, and that would also need fast networking. I wanted to have things running reasonably cool (after all, the PC itself and its GPU pump out quite enough heat already when doing a training run ), so DAC felt like the right way to go. I bought a reasonably cheap managed 10Gb/s switch 2 , a MikroTik CRS305-1G-4S+IN , with a single 10GBASE-T adapter to allow me to connect it to the wall socket. I tend to name anything on my network with its own IP, so this became . Next, a 10Gb/s SFP+ PCIe card -- an Asus XG-C100F -- for and a DAC cable to connect the two. For the Proxmox cluster, I decided to stick with the old 2.5Gb/s unmanaged switch, a TRENDnet TEG-S5061 . I'd originally bought that one because it was the cheapest 2.5Gb/s on Amazon with decent reviews, and had completely forgotten that it had one major feature -- an SFP+ 10Gb/s port for the uplink! So another short DAC to connect that to the MikroTik, and the study network "backbone" was 10Gb/s. Of course, no two computers in there could actually communicate at that speed, as only was 10Gb/s-capable -- but I could have all of the Proxmox machines talking to at the same time at full speed. I did some tests with to make sure that it was all working as expected; I couldn't test very thoroughly, but I was able to get about 4Gb/s total throughput, which was reassuring: two machines at 1Gb/s plus one at 2.5Gb/s should be a touch less than 4.5Gb/s. The next step was to check the possibilities for the connection down to the patch panel. I bought a Ubiquiti 10G Ethernet dongle , and took my laptop, 3 , down there. The news was good! Running an test between and down the structured cabling, I was able to get just less than 10Gb/s from to , and about 7Gb/s from to . The slower receive speed at the end worried me, but when I checked it became obvious what was going on. I could see the kernel process running at 100%, so some single-core thing was maxing out. The Ethernet dongle was connected over USB, of course, and that meant it needed to do much more work on the CPU for each incoming "data has arrived" interrupt than a PCIe card like the one on . That meant that could only receive data at a rate that one core could handle, which happened to be 7Gb/s. is a ThinkPad optimised for lightness and long battery life, not CPU power, so single-core performance is not great, and it hit a wall. But the 10Gb/s speed in the other direction was enough to make me comfortable that the structured cabling could handle that speed, which was excellent news -- probably I had either short runs of CAT-6, or CAT-6A in there, though conceivably I was just getting very lucky with CAT-5E. The downside was the heat. The USB dongle got too hot to comfortably hold while it was running, and while I wasn't able to check the SFP+ module in the MikroTik during the test, when I came back upstairs again I touched it and it was even hotter. I decided that that was something to keep an eye on for later (and as you'll see, it did become a recurring theme). For now, it was time to do the rest of the upgrade. Downstairs at the patch panel, it was a simple choice. All of the connections were RJ45, of course, and I only needed four. So the MikroTik CRS304-4XG-IN was the obvious choice. The final place where I needed to do some upgrades was at the ISP end. The box that our provider gave us had just one 10Gb/s port -- a 10GBASE-T RJ45 one. Now, I don't generally trust ISP routers that much, so I've always had my own router sitting between them and the home network -- a dual-port mini-PC running a locked-down Arch installation 4 . My old one was dual-2.5Gb/s, so that needed an upgrade. I settled on a Protectli VP2440 , which has two SFP+ 10Gb/s cages, plus two normal 2.5Gb/s RJ45s. I didn't need the latter, but it was the cheapest option with 10Gb/s in their range, and I've always been very happy with their hardware and customer service. However, I was a little concerned about thermals. As I mentioned, the SFP+ module in the MikroTik in the study got very hot when I did my test. I'd need dual SFP+ modules for the Protectli -- one for the WAN port connected to the ISP box, and the other for the wall socket to go down to the patch panel. Might it overheat? The good thing about Protectli is that you can just ask them. I dropped them a line, and got a reply the next day from a customer support rep saying that he believed it would be fine, but he just wanted to double-check with one of their techs. The following day, he followed up to say that the tech had confirmed that it would be OK. Promising! And because of that, plus their 30-day money-back guarantee, I decided to go for it. A few days later, the new router arrived. I named it , set it up with my normal router Arch installation, plugged it into the ISP box and the wall... and it worked just fine! So the setup at this point was: At the same time I decided to move the main WiFi AP ( , a Ubiquiti U6 Enterprise ) that was previously next to the router over to my study -- so that was hanging off the TRENDnet switch. After a bit of bedding in, I decided I wanted to move back to the same place as the router -- it's more central so it provides better WiFi coverage from there. So I got another CRS304-4XG-IN -- the 10GBASE-T MikroTik switch, like the one by the patch panel -- so that the first part of the above topology became: All of this is sitting in a sideboard next to the dining table with no ventilation. That's probably close to a pathological case for hot-running network infrastructure like this, so... how about those thermals? I like to keep track of what is going on with my zoo of computers, so I run Telegraf on all of them. This collects stats like the CPU temperature, system load, disk space, CPU and network use, and so on. They send this to an InfluxDB instance on a Proxmox VM ( , if you're keeping track). When I set all of this up, I also wanted to monitor the switches. MikroTik switches expose their stats over SNMP, so with a bit of help from various LLMs I was able to augment the Telegraf config on to also scrape that data and send it to . I use Grafana to get all of this stuff into various dashboards, and one of them is the temperatures of the networking hardware. Firstly, -- the Protectli router with two SFP+ cages, each of which has a 10GBASE-T module. I receive separate temperatures for the CPU and for each SFP+ module: That's not exactly running cool, but TBH it's not too bad! I believe that the SFP+ cages are thermally coupled to the case (which is essentially one giant heatsink). So they're running a bit hotter than the machine as a whole, but it's not baking. Let's see how that does as the weather warms -- you can see that it's been going up over the last week or so as we had a bit of a heatwave here in Lisbon. How about , the MikroTik CRS304-4XG-IN switch -- all native 10GBASE-T, in the same sideboard as ? A bit hotter than I'd like -- above the tested ambient temperature of up to 70C, though of course this is internal rather than external; , which is right next to , having an internal temperature lower than 70C suggests that we're probably still OK, as its internal temperature can't be lower than ambient. I think that both of those could be improved, though. The sideboard they're in is unventilated, and it has the Ubiquiti U6 Enterprise WiFi AP in there too -- that runs pretty hot. So a sensible first step is probably to move the AP elsewhere, and if that's not enough, perhaps to add a USB fan to bring cooler air in through the back of the sideboard. Now, how about , the switch downstairs by the patch panel? It's also in a cupboard with no airflow, and while it's not sharing it with a router, there is a PoE injector and another WiFi AP, , in there (albeit a cooler-running one, a Ubiquiti U7 Lite ). Not too bad at all! Plenty of headroom there. Finally, let's go back upstairs to my study. If you remember, I have there, a MikroTik CRS305-1G-4S+IN -- a four-port SFP+ switch. I get just data for the switch itself and for the 10GBASE-T module -- the DACs don't report numbers. Check this out -- the right hand chart especially: Yikes! The switch itself is OK at a comfortable 48C, but that SFP+ module is hovering around 93C. That's internal rather than the "touch" temperature, but assuming they're close, it's definitely getting towards blistering temperatures if you touch it. I'm getting a stick-on mini-heatsink -- the type you can get for Raspberry Pis -- to see if that might help. It's also sitting on a 11" rack, so I might see if I can find a way to thermally couple it to that. But despite those somewhat concerning numbers, it's all working fine! I have a periodic network test running on , checking end-to-end out to Google's 8.8.8.8 nameservers, and I haven't seen a glitch. tests from to show negligible numbers of errors. It's a working system, so naturally I want to change things. What? TBH, I think I'll be able to limit my desire to tinker in the short term to just sorting those worrying thermal numbers. For and in the sideboard, I think that moving the WiFi AP out again will help. It's power-over-Ethernet, so I can just run one wire up the wall and hide the AP itself behind some art. For the almost-boiling-point SFP+ module on , the study switch, a stick-on Raspberry Pi heatsink is, as I said, probably a good starting point. If that isn't enough, perhaps one with a cooling fan. The actual amount of power being used there isn't much, just 3W or so -- it's only reaching such a high temperature because it's in such a small space. The more interesting question is, what will I do if and when it's time to take the next step up, to 40Gb/s or higher? As I said in my last post , 10GBASE-T is essentially the end of the RJ45, twisted pair world we've been in for the last 20+ years. CAT-8 cabling can, apparently, run up to 40Gb/s, but it comes with its own problems -- it's super-stiff, and hard to run around tight corners or to get into the limited space in the boxes behind wall sockets. I think that the right thing to do would probably be to switch to optical fibre. I did some initial research around this while I was still unsure if the existing cabling would work, and it seems like replacing each cable drop (that is, run from a wall socket to the patch panel) with at least a dual-fibre cable, one to send and one to receive, would work fine, potentially even up to 800Gb/s with the right setup. The wall sockets could be LC duplex, which are designed to be easy to connect (by fibre standards). If I wanted to really future-proof things, it might even make sense to run four-fibre or even eight-fibre cables, and leave all but two of each "dark". That would potentially leave even more space for improvement, and would actually cost very little extra -- the installation cost would be way higher than the cost of the cable. Still, at hundreds of Euros per cable drop, plus project overheads, I'm glad I don't have to do that now. A good decision to be able to punt down the line; who knows what will change between now and whenever my ISP starts offering even faster speeds? So let's wrap this up with the moment you've undoubtedly been waiting for... Not bad! Not quite the 10Gb/s advertised, but it's close -- and I've seen it get up to 9Gb/s from time to time (but unfortunately not screenshotted it). And to be clear, that was from -- so the speed was through all three of the switches, , and , and through the router. Direct tests from from the CLI version of the Ookla app 5 get similar results -- in fact, oddly, they tend to be about 5% slower than the ones from . Not sure what to make of that. I'll have to investigate further, but if anyone has any ideas about what might cause it, I'd love to hear them. So now, when I'm uploading models to Hugging Face and downloading others, syncing large environments, downloading the latest Arch ISO, and streaming music, while at the same time Sara is watching Netflix and my Dropbox is Dropboxing, everything can run smoothly. Nice! Mission accomplished. I hope this was an interesting read, and perhaps helpful for other people who are considering a similar upgrade. Now, time for me to go back to your regularly-scheduled all-AI, all-the-time content ;-) My OpenClaw instance, which runs there, has dubbed it "the Island of Misfit Computers".  ↩ I moved from a simple network to a multi-VLAN one at the same time as this upgrade, so managed switches have become useful -- if you're just doing an upgrade to 10Gb, you can do it all with unmanaged ones.  ↩ In case you're wondering about the naming strategy for machines on the network: What can I say. It passes the time.  ↩ It's largely old routers that populate the Proxmox cluster.  ↩ Their own one , not the more commonly-used OSS Python one , which isn't fast enough to handle speeds over about 5Gb/s.  ↩ The ISP connection came into the apartment in the living room. It went through a router/firewall machine I'd set up myself (more on that later), then via a 2.5Gb/s switch to the main WiFi AP and also to a wall socket. Down at the patch panel, I had a 2.5Gb/s switch, which was connected to the patch socket corresponding to the router's wall socket. Another connection from that switch went to the patch socket corresponding to the wall socket in my study. In the study, I had another 2.5Gb/s switch that handled internal networking. ISP box to WAN on the router. LAN on to wall socket. Patch panel socket corresponding to that wall socket to port 0 on the downstairs RJ45-only switch, . port 1 to the patch panel corresponding to my study's wall socket. (Other ports to other things I'm disregarding for simplicity.) Wall socket in the study to the RJ45 SFP+ module in port 0 on . port 1: DAC to an SFP+ network card on , my workstation. port 2: DAC to the SFP+ 10Gb/s uplink on the old TRENDnet 2.5Gb/s switch to handle the Proxmox cluster. ISP box to WAN on the router. LAN on to the new switch ( ) port 0. Port 1 on to the wall socket (thence down to the patch panel). Port 2 on to the WiFi AP via a PoE injector. My OpenClaw instance, which runs there, has dubbed it "the Island of Misfit Computers".  ↩ I moved from a simple network to a multi-VLAN one at the same time as this upgrade, so managed switches have become useful -- if you're just doing an upgrade to 10Gb, you can do it all with unmanaged ones.  ↩ In case you're wondering about the naming strategy for machines on the network: PCs, desktops, etc: name starts with P , for example or . Laptops: name starts with L . Basically just . Sara named her own work laptop, unrestricted by my convention, so it's called . Routers: name starts with R : , . Network infrastructure: name starts with N : , and . WiFi APs: name starts with W , eg. and . VMs on Proxmox: name starts with V : , , , etc. I also have a bare metal server on Hetzner, which I've named . It's largely old routers that populate the Proxmox cluster.  ↩ Their own one , not the more commonly-used OSS Python one , which isn't fast enough to handle speeds over about 5Gb/s.  ↩

python Python hardware Hardware tutorial Tutorial
1 views
Giles's blog 1 months ago

10Gb Ethernet: what I had to (re)learn

My ISP recently started offering a 10Gb option, and my "shiny new thing!" Pavlovian response immediately kicked in. So of course, I had to upgrade the wired networking in my home -- which meant I had to learn a few things to get it all working, and relearn a bunch of stuff I'd forgotten over the years. Wired networking for home and small offices hasn't really moved forward that much in the last 20-odd years. Back in 2006, gigabit Ethernet was standard for businesses, and most home users moved to it not long after. Perhaps due to the rise of WiFi for most "last few metres" connections, it's pretty much stagnated there, perhaps with a bit of a push towards 2.5Gb/s more recently. But with faster ISP connections arriving, I think things are starting to become a bit more interesting. Even the fastest WiFi 7 connections are only able to get up to around 6Gb/s to a single device -- and that's in an ideal "super-fast machine sitting right next to the AP in a shielded lab" setup. Here's what I had to drag up from my memory, and the new stuff I had to learn, in order to get this all working. I'll write about the background in this post, and then tomorrow I'll post about what I actually put in place. Let's start with a bit of the backstory. Bear with me, it's not just self-indulgent reminiscing! When I first started using networked computers, back in the early 90s, the most popular standard was 10BASE2 . We had this in the first office that I worked in, and in the university computer labs. In the back of your computer, you'd have a T-shaped connector like this: © Raimond Spekking / CC BY-SA 4.0 (via Wikimedia Commons ) The end facing the camera in that photo was the bit that went into your computer. Computers were daisy-chained together; you might have a server connected to workstation one, workstation one to workstation two, and so on, until you reached the last workstation. You'd have to cap the unused end of the T connectors at each end of the chain with a special terminator. Essentially it was a single coaxial cable, so every computer saw every bit that was sent along the bus. In turn, that meant that everyone was sharing the same bandwidth, a meagre 10Mb/s. The cool thing about Ethernet (compared to older networking technologies) was that the computers shared it without any need for coordination -- if two of them started "speaking" at the same time, they'd notice, and stop. They would then start again after a random back-off, so one of them would randomly wait for less time than the other and start first. The other would notice that "the line was busy" and would wait again for another chance. Of course, this limited the number of computers you could have on one network, as past around 20 or so, they'd spend all of their time interrupting each other and never actually be able to send anything -- and anyway, sharing 10Mb/s across a large number of computers would be an issue. On top of that, there was a hard cap of 30 machines per network. You'd use more specialised networking equipment to link different networks together -- bridges, switches and routers. More about switches later. By the time we started setting up networking in a house that I shared with friends, in around 1996 or so 1 , the most popular option had changed: now people were using 10BASE-T. Still 10Mb/s, but using the RJ45 connectors and twisted-pair cables that we've come to know and love. All of the computers would have a single cable going to a hub, in a star topology. You might link multiple hubs together to build larger networks. However, these hubs were still little more than a convenient form factor to electrically link all of the wires together into a single bus. You still had the problem that every computer could see every bit on the bus, and the same bandwidth-sharing and limits with the number of computers that you could handle as a result. Over the years after that, things moved on. Switches had been relatively expensive things; they would be used to interlink hubs, or 10BASE2 networks. They would learn (from seeing the source MAC address on incoming packets) which machines were sending to each of their ports, and use that to know where to send packets that came in on other ports. If, say, a switch learned that addresses A, B, and C were on port 1, then if a packet for one of those machines came in on port 2, it would know it could just send it out on port 1 and not on the others. That helped to address the bandwidth-sharing and the problems with collisions. Prices for switches got lower and lower, and eventually -- I think sometime between 2005 and 2010 -- they became so cheap that there was little point in bothering with hubs -- you'd just connect every computer directly to a switch. That meant that any two computers on the same switch could talk to each other at the full network speed, as packets would just be switched from port to port 2 . The connections between switches were still a bottleneck, of course, but that was much less of a problem. At the same time, speeds increased, from 10Mb/s to 100Mb and then finally to 1Gb/s, which was standard for business machines by 2005 or so -- I remember that when we bought our first computers for Resolver Systems back then, that's what they came with by default. Home computers weren't far behind -- and that's where we've been ever since. 3 Back to that bottleneck between the switches. Even back in the days of 10Mb/s networks, if you were managing a larger network, you would want a faster network to interlink them -- so, for example, if two computers on the same switch both wanted to access some external resource, they wouldn't be competing for the same 10Mb/s uplink. Once you went past small office-sized networks, that kind of thing started becoming important. ISPs and datacenters, of course, had the same problem in spades. What you would need was an uplink on the switch that could run at a faster data rate. So even when 1Gb/s Ethernet was too expensive for the connections to the computers themselves, you might have a switch with a 1Gb/s uplink to connect it to the larger network, and a bunch of 100Mb/s ports for the local stuff. Additionally, for larger networks you would have another problem -- physical distance. All of these RJ45-based networking technologies had a maximum cable length of 100m. You could extend that by putting a repeater (or even just a switch) every 100m or so as a "signal booster" -- but if, for example, you wanted to link two buildings, that could be tricky. You'd need to run both the data cable and power, and you'd need to have some way of getting access to the repeaters if they went wrong. Ethernet over fibre optic connections had been a standard thing for years, though, and it had much better range -- for single-mode, many kilometers. So while it was too fiddly for LANs, it made great sense as a backbone technology. What that meant, though, was that in order to set up some particular network topology, you might wind up having to get a whole bunch of different switches. For short connections between two of them, you might use an RJ45 uplink connection, while for longer ones you might want fibre. More complex topologies might need some entirely different mix of ports. To make this worse, there were a bunch of different fibre optic standards -- multi-mode and single mode fibres, different connectors, and so on. Rather than manufacturing a large range of different kinds of switches with all of the combinations that people needed, manufacturers separated out the physical layer of the transport from the switching hardware. A switch, instead of having specific RJ45 or fibre connectors for its ports, would have Small Form-factor Pluggable (SFP) "cages", essentially a new kind of socket. These allow people to mix and match different kinds of transceiver modules, which would slot into the cage to provide an actual usable interface -- one for RJ45 for gigabit Ethernet, or one for the particular kind of fibre connection they were using -- whatever configuration worked best for them. A typical switch for a larger network might have one or two of those for backbone connections, and then RJ45s for local connections. Over time, gigabit backbones were no longer enough, and SFP was followed by SFP+, which could handle 10Gb/s. Since then, there have been extensions for even faster speeds, way up to hundreds of Gb/s. Back in the day, this stuff was only important to network admins for medium-sized networks and larger, of course. But now, 10Gb Ethernet means that we've now hit the point where it matters even for home users, and that's because of thermals. Here's the problem. Somewhat loosely speaking, the faster a network connection on a particular kind of wiring, the hotter it runs. Over an RJ45/twisted pair connection, 10Mb/s Ethernet basically shed no heat, 100Mb/s a little more, even gigabit Ethernet just left your switches somewhat warm. The jump up to 10Gb over RJ45, called 10GBASE-T, makes things decidedly toasty -- you'll see just how toasty in tomorrow's post. There's also the issue of cabling. Because network speeds have been stable for some time -- Gigabit Ethernet being the standard for ~20 years -- most buildings with structured cabling (the kind of thing where there are RJ45 sockets in the walls wired together) will have the standard for that -- CAT-5E. Unfortunately 10Gb/s Ethernet won't officially work over it -- you might be lucky, especially with short cables, but in general it won't work, or if it does it won't be reliable. CAT-6 cabling helps -- it can handle 10Gb/s over runs up to about 55 metres. And the ideal is CAT-6A, which can handle 10Gb/s over the same 100 metre cable lengths that you'd expect for the older, slower setups. What this meant was that an interim standard was created. 10GBASE-T is hot and needs cables that people don't necessarily have, especially when you're talking about what's installed in the walls of their building. But if you run it a bit slower, you can do so over older cables and without melting them. That's why I didn't mention 2.5Gb/s Ethernet earlier (or indeed the rarer 5Gb/s). They were introduced as slowed-down versions of 10Gb/s to get it to work on existing infrastructure without major upgrades. And that's great, right up until the point your ISP emails you to say that they're offering 10Gb/s to your home now... So, what can you do to run 10Gb/s without melting things? Let's think about what an SFP or SFP+ module actually is. It slots into a cage on a switch. On one side, there's an electrical connection to the switch hardware, which is carrying the signal -- incoming and outgoing -- using a particular protocol 4 . The module does its magic, and on the other side we have -- say -- 10GBASE-T to an RJ45 socket, or a blinking laser with an appropriate interface for optical fibre. What would happen if you just had a dumb electrical cable to connect an SFP+ cage on one switch to another on another switch? That actually works pretty well! It's called a passive Direct Attach Copper (DAC) cable. The interfacing is a little more complicated than just a completely dumb wire -- the switch will want to query the module in the cage to find out some details about it, so you need a tiny bit of electronics -- but it's still really simple. On top of that, if you add a bit of amplification to the DAC, then you get an active DAC, which can double that kind of length (though these are relatively rare). The neat thing about DACs is that they run much cooler than 10GBASE-T, using about a third of the power. Of course, they lose out in terms of range. But for simple stuff within one room, and especially between switches in a rack, they work really well. The next step on top of DACs is that you can convert the underlying SFP(+) protocol directly to light, and send it down an optical fibre -- normally called an Active Optical Cable, or an AOC for short (though I've seen the rather confusing terminology "optical DAC" in various places). With that, you can normally get up to 100m. These are cheap and easy to use (because they're all-in-one units, so you don't have any fiddly alignment of the fibre to do), so they're the best option once you pass passive-DAC distances. After that, though, you really need to switch to the official standards, and go to more traditional fibre-optic setups. I've done much less research into those, so won't try to explain them. Either way, for the home, anything above this level is probably overkill right now... So: moving from the 2.5Gb/s networks that work smoothly with the same infrastructure we've been using for the last 20 years or so to 10Gb/s is a tricky step change. Suddenly, things that didn't matter -- thermal management, cable lengths, and so on -- become important. And there are solutions, but you need to start actually understanding things again rather than just plugging stuff in and assuming it will work. Fun! Time to put it into practice :-) In my next post, I'll show exactly the changes I had to make to get my existing 2.5Gb/s network ported over to 10Gb/s -- the hardware I wound up buying, how well it works, and (importantly) how hot it all runs. To share our blazingly fast bonded dual ISDN Internet connection -- 128Kb/s.  ↩ I remember feeling a little sad when that happened, because it meant that what I felt was coolest about Ethernet -- the back-off-and-retry thing -- was no longer all that important. And when the connections went full duplex (a single switch port could both send and receive at the same time over the same cable) it was finished.  ↩ If you're thinking "what about 2.5Gb/s?", I'll come back to that -- it's an interesting case.  ↩ SFF-8472 for SFP, then there's SFF-8431 and SFF-8432 for SFP+.  ↩ To share our blazingly fast bonded dual ISDN Internet connection -- 128Kb/s.  ↩ I remember feeling a little sad when that happened, because it meant that what I felt was coolest about Ethernet -- the back-off-and-retry thing -- was no longer all that important. And when the connections went full duplex (a single switch port could both send and receive at the same time over the same cable) it was finished.  ↩ If you're thinking "what about 2.5Gb/s?", I'll come back to that -- it's an interesting case.  ↩ SFF-8472 for SFP, then there's SFF-8431 and SFF-8432 for SFP+.  ↩

tutorial Tutorial Networking hardware Hardware
0 views