Posts in Devops (20 found)

Self-hosting my photos with Immich

For every cloud service I use, I want to have a local copy of my data for backup purposes and independence. Unfortunately, the tool stopped working in March 2025 when Google restricted the OAuth scopes, so I needed an alternative for my existing Google Photos setup. In this post, I describe how I have set up Immich , a self-hostable photo manager. Here is the end result: a few (live) photos from NixCon 2025 : I am running Immich on my Ryzen 7 Mini PC (ASRock DeskMini X600) , which consumes less than 10 W of power in idle and has plenty of resources for VMs (64 GB RAM, 1 TB disk). You can read more about it in my blog post from July 2024: When I saw the first reviews of the ASRock DeskMini X600 barebone, I was immediately interested in building a home-lab hypervisor (VM host) with it. Apparently, the DeskMini X600 uses less than 10W of power but supports latest-generation AMD CPUs like the Ryzen 7 8700G! Read more → I installed Proxmox , an Open Source virtualization platform, to divide this mini server into VMs, but you could of course also install Immich directly on any server. I created a VM (named “photos”) with 500 GB of disk space, 4 CPU cores and 4 GB of RAM. For the initial import, you could assign more CPU and RAM, but for normal usage, that’s enough. I (declaratively) installed NixOS on that VM as described in this blog post: For one of my network storage PC builds, I was looking for an alternative to Flatcar Container Linux and tried out NixOS again (after an almost 10 year break). There are many ways to install NixOS, and in this article I will outline how I like to install NixOS on physical hardware or virtual machines: over the network and fully declaratively. Read more → Afterwards, I enabled Immich, with this exact configuration: At this point, Immich is available on , but not over the network, because NixOS enables a firewall by default. I could enable the option, but I actually want Immich to only be available via my Tailscale VPN, for which I don’t need to open firewall access — instead, I use to forward traffic to : Because I have Tailscale’s MagicDNS and TLS certificate provisioning enabled, that means I can now open https://photos.example.ts.net in my browser on my PC, laptop or phone. At first, I tried importing my photos using the official Immich CLI: Unfortunately, the upload was not running reliably and had to be restarted manually a few times after running into a timeout. Later I realized that this was because the Immich server runs background jobs like thumbnail creation, metadata extraction or face detection, and these background jobs slow down the upload to the extent that the upload can fail with a timeout. The other issue was that even after the upload was done, I realized that Google Takeout archives for Google Photos contain metadata in separate JSON files next to the original image files: Unfortunately, these files are not considered by . Luckily, there is a great third-party tool called immich-go , which solves both of these issues! It pauses background tasks before uploading and restarts them afterwards, which works much better, and it does its best to understand Google Takeout archives. I ran as follows and it worked beautifully: My main source of new photos is my phone, so I installed the Immich app on my iPhone, logged into my Immich server via its Tailscale URL and enabled automatic backup of new photos via the icon at the top right. I am not 100% sure whether these settings are correct, but it seems like camera photos generally go into Live Photos, and Recent should cover other files…?! If anyone knows, please send an explanation (or a link!) and I will update the article. I also strongly recommend to disable notifications for Immich, because otherwise you get notifications whenever it uploads images in the background. These notifications are not required for background upload to work, as an Immich developer confirmed on Reddit . Open Settings → Apps → Immich → Notifications and un-tick the permission checkbox: Immich’s documentation on backups contains some good recommendations. The Immich developers recommend backing up the entire contents of , which is on NixOS. The subdirectory contains SQL dumps, whereas the 3 directories , and contain all user-uploaded data. Hence, I have set up a systemd timer that runs to copy onto my PC, which is enrolled in a 3-2-1 backup scheme . Immich (currently?) does not contain photo editing features, so to rotate or crop an image, I download the image and use GIMP . To share images, I still upload them to Google Photos (depending on who I share them with). The two most promising options in the space of self-hosted image management tools seem to be Immich and Ente . I got the impression that Immich is more popular in my bubble, and Ente made the impression on me that its scope is far larger than what I am looking for: Ente is a service that provides a fully open source, end-to-end encrypted platform for you to store your data in the cloud without needing to trust the service provider. On top of this platform, we have built two apps so far: Ente Photos (an alternative to Apple and Google Photos) and Ente Auth (a 2FA alternative to the deprecated Authy). I don’t need an end-to-end encrypted platform. I already have encryption on the transit layer (Tailscale) and disk layer (LUKS), no need for more complexity. Immich is a delightful app! It’s very fast and generally seems to work well. The initial import is smooth, but only if you use the right tool. Ideally, the official could be improved. Or maybe could be made the official one. I think the auto backup is too hard to configure on an iPhone, so that could also be improved. But aside from these initial stumbling blocks, I have no complaints.

0 views
Brain Baking 2 days ago

Using Energy Prediction To Better Plan Cron Jobs

Since the Belgian government mandated the use of digitized smart energy meters we’ve been more carefully monitoring our daily energy demand. Before, we’d simply chuck all the dishes in the machine and program it to run at night: no more noise when we’re around. But now, consuming energy at night is costing us much more. The trick is to take as little as possible from the grid, but also put as little as possible back. In short, consume (or store) energy when our solar panels produce it. That dishwasher will have to run at noon instead. The same principle applies to running demanding software: CPU or GPU-intensive tasks consume an awful amount of energy, so why run them when there’s less energy available locally, thus paying more? Traditionally, these kinds of background jobs are always scheduled at night using a simple cron expression like that says “At 03:00 AM, kick things in gear”. But we can do better. At 03:00 AM, our solar panels are asleep too. Why not run the job when the sun is shining? Probably because you don’t want to interfere with the heavy load of your software system during the day thanks to your end users. It’s usually not a good idea to start generating PDF files en masse , clogging up all available threads, severely slowing down the handling of incoming HTTP requests. But there’s still a big margin to improve the planning of the job: instead of saying “At 03:00 AM exactly ”, why can’t we say “Between 01:00 AM and 07:00 AM”? That’s still before the big HTTP rush, and in the early morning, chances are there’s more cheap energy available to you. Cooking up a simple version of this for home use is easy with the help of Home Assistant. The following historical graph shows our typical energy demand during the last week (dreadful Belgian weather included): Home Assistant history of P1 Energy Meter Demand from 24 Nov to 28 Nov. Care to guess what these spikes represent? Evenings. Turning on the stove, the oven, the lights, the TV obviously creates a big spike in energy consumption, and at the same time, the moon replacing the sun results in us taking instead of giving from the energy grid. This is the reason the government charges more then: if everybody creates spikes at the same time, there’s much more pressure on the general grid. But I can’t bake my fries at noon when I’m work and we aren’t supposed to watch TV when we’re working from home… That data is available through the Home Assistant API: . Use an authorization header with a Bearer token created in your Home Assistant profile. If you collect this for a few weeks and average the results you can make an estimated guess when demand will be going up or down. If you want things to get a bit more fancy, you can use the EMHASS Home Assistant plug-in that includes a power production forecast module. This thing uses machine learning and other APIs such as https://solcast.com/ that predicts solar power—or weather in general: the better the weather, the more power available to burn through (given you’ve got solar panels installed). EMHASS also internalizes your power consumption habits. Combined, its prediction model can help to better plan your jobs when energy demand is low and availability is high. You don’t need Home Assistant to do this, but the software does help smooth things over with centralized access to data using a streamlined API. Our energy consumption and generation is measured using HomeWizard’s P1 Meter that plugs into our provider’s digital meter and sends the data over to Home Assistant. That’s cool if you are running software in your own basement, but will hardly do on a bigger scale. Instead of monitoring your own energy usage, you can rely on grid data from the providers. In Europe, the European Network of Transmission System Operators for Electricity (ENTSO-E) provides APIs to access power statistics based on your region—including a day-ahead forecast! In USA, there’s the U.S. Energy Information Administration (EIA) providing the equivalent, also including a forecast, depending on the state. ENTSO-E returns a day-ahead pricing model while EIA returns consumption in megawatthours, but both statistics can be used for the same thing: to better plan that cron job. And that’s exactly what we at JobRunr managed to do. JobRunr is an open-source Java library for easy asynchronous background job scheduling that I’ve had the pleasure to work on the last year. Using JobRunr, planning a job with a cron expression is trivial: But we don’t want that thing to trigger at 3 AM, remember? Instead, we want it to trigger between an interval, when the energy prices are at their lowest, meaning when the CPU-intensive job will produce the least amount of CO2 . In JobRunr v8, we introduced the concept of Carbon Aware Job Processing that uses energy prediction of the aforementioned APIs to better plan your cron jobs. The configuration for this is ridiculously easy: (1) tell JobRunr which region you’re in, (2) adjust that cron. Done. Instead of , use : this means “plan at somewhere between an hour before 3 AM to four hours later than 3 AM, when the lowest amount of CO2 will be generated”. That string is not a valid cron expression but a custom extension on it we invented to minimize configuration. Behind the scene, JobRunr will look up the energy forecasts for your region and plan the job according to your specified time range. There are other ways to plan jobs (e.g. fire-and-forget, providing s instaed of a cron, …), but you get the gist. JobRunr’s dashboard can be consulted to inspect when the job is due for processing. Since the scheduled picks can sometimes be confusing—why did it plan this at 6 AM and not at 7?—the dashboard also visualizes the predictions. In the following screenshot, you can see being planned at 15:00 PM, with an initial interval between 09:39 and 17:39 (GMT+2): The JobRunr dashboard: a pending job, to be processed on Mon Jul 07 2025 at 15:00 PM. There’s also a practical guide that helps you get started if you’re interested in fooling around with the system. The idea here is simple: postpone firing up that CPU to the moments with more sunshine, when energy is more readily available, and when less CO2 will be generated 1 . If you’re living in Europe/Belgium, you’re probably already trying to optimize the energy consumption in your household the exact same way because of the digital meters. Why not applying this principle on a grander scale? Amazon offers EC2 Spot Instances to “optimize compute usage” which is also marketed as more sustainable, but this is not the same thing. Shifting your cloud workout to a Spot Instance will use “spare energy” that was already being generated. JobRunr, and hopefully soon other software that optimized jobs based on energy availability, plans using marginal changes. In theory, the decision can determine the fuel resource as high spikes force high-emission plants to burn more fuel. In always-on infrastructure, spare compute capacity is sold as the Spot product—there’s no marginal change. The environmental impact of planning your job to align with low grid carbon intensity is much higher—in a good way—compared to shifting cloud instance types from on-demand/reserved to Spot. Still, it’s better than nothing, I guess. If the recent outages of these big cloud providers have taught us anything, it’s that on-premise self-hosting is not dead yet. If you happen to be rocking Java, give JobRunr a try. And if you’re not, we challenge you to implement something similar and make the world a better place! You probably already noticed that in this article I’ve interchanged carbon intensity with energy availability. It’s a lot more complicated than that, but for the purpose of Carbon Aware Job Processing, we assume a strong relationship between the electricity price and CO2 emissions.  ↩︎ Related topics: / java / By Wouter Groeneveld on 28 November 2025.  Reply via email . You probably already noticed that in this article I’ve interchanged carbon intensity with energy availability. It’s a lot more complicated than that, but for the purpose of Carbon Aware Job Processing, we assume a strong relationship between the electricity price and CO2 emissions.  ↩︎

0 views
neilzone 4 days ago

Using a2dismod apache2's mod_status which exposed information via a .onion / Tor hidden service

Earlier this week, I received a vulnerability report. The report said that, when accessing the site/server via the .onion / Tor hidden service URL, it was possible to view information about the server, and live connections to it, because of . is an apache2 default module, which shows information about the apache2 server on /server-status. It is only available via localhost but, because of the default configuration of a Tor .onion/hidden service, which entails proxying to localhost, it was available. The report was absolutely valid, and I am grateful for it. Thank you, kind anonymous reporter. It was easily fixed, made all the more annoying because I knew about this issue (it has been discussed for years) but forgot to disable the module when I moved the webserver a few months ago. One to chalk up to experience I have had a security.txt file in place on the decoded.legal website for quite a while now, but I’ve never had anyone use it. I asked the person who reported it to me if they had contacted me via it, but no, they had not.

0 views
Ruslan Osipov 6 days ago

Turns out Windows has a package manager

I have a Windows 11 PC, and something that really annoyed me about Windows for decades is the inability to update all installed programs at once. It’s just oh-so-annoying to have to update a program manually, which is worse for things I don’t use often - meaning every time I open a program, I have to deal with update pop-ups. I was clearly living under a rock, because all the way in 2020 Microsoft introduced package manager which lets you install, and more importantly update packages. It’s as simple as opening a command line (ideally as administrator, so you don’t have to keep hitting yes on the permission prompt for every program), and runinng . Yup, that’s it. You’ll update the vast majority of software you have installed. Some software isn’t compatible, but when I ran the command for the first time, Windows updated a little over 20 packages, which included the apps I find myself having to update manually the most often. To avoid having to do this manually, I’ve used windows Task Scheduler to create a new weekly task which runs a file, which consists of a single line: I just had to make sure Run with the highest privileges is enabled in task settings. So long, pesky update reminders. My Windows apps will finally stay up-to-date, hopefully.

0 views
DHH 6 days ago

No backup, no cry

I haven't done a full-system backup since back in the olden days before Dropbox and Git. Every machine I now own is treated as a stateless, disposable unit that can be stolen, lost, or corrupted without consequences. The combination of full-disk encryption and distributed copies of all important data means there's just no stress if anything bad happens to the computer. But don't mistake this for just a "everything is in the cloud" argument. Yes, I use Dropbox and GitHub to hold all the data that I care about, but the beauty of these systems is that they work with local copies of that data, so with a couple of computers here and there, I always have a recent version of everything, in case either syncing service should go offline (or away!). The trick to making this regime work is to stick with it. This is especially true for Dropbox. It's where everything of importance needs to go: documents, images, whatever. And it's instantly distributed on all the machines I run. Everything outside of Dropbox is essentially treated as a temporary directory that's fully disposable. It's from this principle that I built Omarchy too. Given that I already had a way to restore all data and code onto a new machine in no time at all, it seemed so unreasonable that the configuration needed for a fully functional system still took hours on end. Now it's all encoded in an ISO setup that installs in two minutes on a fast computer. Now it's true that this method relies on both multiple computers and a fast internet connection. If you're stuck on a rock in the middle of nowhere, and you somehow haven't discovered the glory of Starlink, maybe just stick to your old full-disk backup ways. But if you live in the modern world, there ought to be no reason why a busted computer is a calamity of data loss or a long restore process.

1 views

Praise the Backup

Well, that was a fun weekend. I have spent half of my time reinstalling MacOS Sequoia , and trying to get it back the way it was, while trying to avoid losing important files. You see, on that chilly Saturday afternoon, I wanted to take care of my ageing computer, and tried an app that was supposed to clean the old files and “residue” from previously uninstalled apps. As a reader of this blog, you may know that I tend to use very few apps , but I try a lot of them . Trying a lot of apps means doing a lot of installs, and then a lot of uninstall processes. So, while experimenting with yet another app, it crashed in the middle of its cleaning work. And, because I was being dumb, I thought it would be a good idea to empty the bin at that moment. 5, 10, 15, 20, 25… These were the thousands of files being deleted permanently from the bin. Even with lots of old app files, the number still seemed rather high. I stopped the process only after losing thousands of files and realised that this cleaning app had put in the bin a lot of files and folders that it shouldn't have. A lot of files and folders. My blog files with all my Eleventy settings, all of it. Most of my system preference files. The app even deleted its own application files, which is why the app crashed, I believe. None of my other apps or extensions could be launched, error messages everywhere. I was having a lot of fun. I restarted my computer, hoping the powerful reboot spirits would once again act miraculously, but my dear old MacBook Air welcomed me as it was a brand-new Mac, almost fresh installation. Even my keyboard was set to the wrong layout (which made it truly fantastic to enter a password in such a moment of panic), my wallpaper was gone, the dock was featuring all the default apps, and I was logged off my iCloud account. Thankfully, this last part turned out to be a good thing because my personal and most important files, stored on iCloud Drive, were safe from whatever had happened on my machine. I also had a two-month-old backup on an external SSD, mirrored on JottaCloud . The cherry on top was that I couldn’t use the “Put Back” right-click action on the files left in the bin as they were not put there by the Finder, but by this third-party app. 1 There were 1200 files and folders left or so, most of them obscure preference files. Needless to say that I didn’t really bother taking hours of my weekend putting them back where they belonged, even if I knew how. I scavenged what I could, everything that seemed important — including a folder called “BBEdit Backups” (more on that later) — and used this opportunity to start anew. Since my last backup was two months old, needless to say that I had a decent amount of work to do putting everything back together, including the last four posts of the website you're reading — which had been vaporised from my computer. I had to reinstall all my apps, my preferences, my keyboard shortcuts, everything that I could, while I could still recall what they were in detail. I won’t blame the app that caused all of this, or my old computer, as much as I will blame myself. I should have been more careful about how to use it properly, I shouldn’t have decided to empty the bin at that moment, and I should have done better and more frequent backups: once every quarter is definitely not enough. The clean MacOS install experience itself was not great: It was very slow, annoying, and during all this time I worried about not being able to connect to my site again or make Eleventy work the same way it did (sorry if I get a little PTSD ). 2 Today, as I write this, my computer doesn’t really feel any faster; a clean install can only do so much on the last generation of Intel MacBook Airs. MacOS was a pain, and I was reminded of my Windows user days more than I expected. For example, I kept getting a message along the lines of “The widget blahblahblah is a different version from a previous version of that widget, do you want to open it?” and clicking “No” just brought back the pop-up window three or four more times before it eventually went away. The prompt even interrupted me while I was trying to type my complicated Wi-Fi password. Not once, not twice, but thrice. Now, everything seems fine. Eleventy works. Xmit works. BBEdit is just like it was. This whole experience made me realise three main things. Apologies if you see anything weird on this site: some little layout issues and typos that were fixed in the last two months may have returned. Please let me know if you see anything suspicious (or any of the usual typos). In the meantime, don’t be an idiot like me: take care of those backups. I won’t name the app in this post because I’m not 100% sure if the app was the sole guilty party in this affair, if guilty at all. Maybe I didn’t set it up right, maybe it’s all my fault!  ↩︎ As the song goes.  ↩︎ That BBEdit is, indeed, just too good . I’m not sure if I could have brought everything back so quickly and confidently without this app. The BBEdit automated backup folder, the one I found in the bin, really saved me. Many of the most recent versions of the Jolly Teapot text files were still there, so I didn’t have to import the text from the live website. Just when I thought I couldn’t love this app more than I already did. I’m proud of myself for thinking of creating a backup of my BBEdit preferences too. That I seriously needed to create a better backup system so that in the event of something like this happening again, whether a human error or an app shitting the bed, I would only have a week or two of files to recover, and not a whole nine weeks of them. I just created an Automator workflow to help me automate my backups and include more files. I considered using Time Machine on my external SSD, or using an app like Hazel , but for my minimal needs, this Automator workflow should do just fine. That I may have actually enjoyed all of this: the crash and this weird situation gave me an excuse to both operate a clean installation on my Mac and justify the purchase of a new one. I will probably wait until March for the next generation of MacBooks Air, but the regular M5 MacBook Pro has never looked so good. I won’t name the app in this post because I’m not 100% sure if the app was the sole guilty party in this affair, if guilty at all. Maybe I didn’t set it up right, maybe it’s all my fault!  ↩︎ As the song goes.  ↩︎

1 views

Automating agentic development

This week, I visited my friends at 2389 in Chicago. These are the folks who took my journal plugin for Claude Code and ran with the idea, creating botboard.biz , a social media platform for your team's coding agents. They also put together an actual research paper proving that both tools improve coding outcomes and reduce costs. Harper is one of the folks behind 2389...and the person who first suggested to me that maybe I could do something about our coding agents' propensity for saying things like: His initial suggestion was that maybe I could make a single-key keyboard that just sends Back in May, I made one of those . When I added keyboard support to the Easy button, I made sure not to disable the speaker. So it helpfully exclaimed "That was easy!" every time it sent: But...you still had to press the button. I'm pretty sure the button got used for at least a day before it was...retired. But the problem it was designed to solve is very, very real. And pretty frustrating. Yesterday morning, sitting in 2389's offices, we spent a bunch of time talking about automating ourselves out of a job. In that spirit, I finally dug enough into Claude Code hooks to build out the first version of Double Shot Latte , a Claude Code plugin that, hopefully, makes a thing of the past. DSL is implemented as a Claude Code "Stop" hook. Any time Claude thinks it should stop and ask for human interaction, it first runs this hook. The hook hands off the last couple of messages to another instance of Claude with a prompt asking it to judge whether Claude genuinely needs the human's help or whether it's just craving attention. It tries to err on the side of pushing Claude to keep working. To try to avoid situations where it misjudges Claude's ability to keep working without guidance, it bails out if Claude tries to stop three times in five minutes. Testing DSL was...a little bit tricky. I needed to find situations where Claude would work for a bit and then stop and ask for my approval to keep working. Naturally, I asked Claude for test scenarios. The first was "build a full ecommerce platform." Claude cranked for about 20 minutes before stopping. I thought the judge agent hadn't worked, but...Claude had actually fulfilled the entire spec and built out an ecommerce platform. (The actual implementation was nothing to write home about, but I'm genuinely not sure what it could have done next without a little more direction. The second attempt fared no better. On Claude's advice, I asked another Claude to build out an HTML widget toolkit. Once again, it cranked for a while. It built widgets. It wrote tests. It wrote a Storybook. And when it stopped for the first time...I couldn't actually fault it. Slightly unsure how to test things, I put this all aside for a bit to work on another project. I opened up Claude Code and typed Claude greeted me like it normally does. And instead of stopping there like it usually would, it noticed that there were uncommitted files in my working directory and started to dig through each of them trying to reverse engineer the current project. Success! (I hit to stop it so that I could tell it what I actually wanted.) Double Shot Latte will absolutely burn more tokens than you're burning now. You might want to think twice about using it unsupervised. If you want to put Claude Code into turbo mode, DSL is available on the Superpowers marketplace . If you don't yet have the Superpowers marketplace set up, you'll need to do that before you can install Double Shot Latte: Once you do have the marketplace installed, run this command inside Claude Code: Then, restart Claude Code so it can pick up the new hook.

0 views
Rik Huijzer 1 weeks ago

Google does not allow Google employees to use their new IDE

Gergely Orosz on X: > Amusing: Google does not allow its devs to use its newly launched IDE, Antigravity, for development. > > Classic example of “externalisation” at Google: they have an *internal* (increasingly different!) version of it, but they won’t use what they launched to everyone else > > (Devs with a @google .com address cannot sign up, and it was communicated internally as well to not use it)

0 views
Rik Huijzer 1 weeks ago

Making IPv6 work with Caddy and Hetzner

After a few hours of fiddling, this site is now properly accessible via IPv6. My configuration uses Caddy as the reverse proxy meaning that it forwards the requests to the right service based on the `Host` that the browser specifies. Thanks to this, one server can host hundreds if not thousands of websites from one IP. In my Caddyfile, I had specified the following ```caddyfile www.huijzer.xyz { redir https://huijzer.xyz permanent } huijzer.xyz { reverse_proxy 127.0.0.1:3000 } ``` And then I thought that for IPv6 maybe this last part should have been ```caddyfile huijzer.xyz { reverse_p...

0 views
Simon Willison 1 weeks ago

How I automate my Substack newsletter with content from my blog

I sent out my weekly-ish Substack newsletter this morning and took the opportunity to record a YouTube video demonstrating my process and describing the different components that make it work. There's a lot of digital duct tape involved, taking the content from Django+Heroku+PostgreSQL to GitHub Actions to SQLite+Datasette+Fly.io to JavaScript+Observable and finally to Substack. The core process is the same as I described back in 2023 . I have an Observable notebook called blog-to-newsletter which fetches content from my blog's database, filters out anything that has been in the newsletter before, formats what's left as HTML and offers a big "Copy rich text newsletter to clipboard" button. I click that button, paste the result into the Substack editor, tweak a few things and hit send. The whole process usually takes just a few minutes. I make very minor edits: That's the whole process! The most important cell in the Observable notebook is this one: This uses the JavaScript function to pull data from my blog's Datasette instance, using a very complex SQL query that is composed elsewhere in the notebook. Here's a link to see and execute that query directly in Datasette. It's 143 lines of convoluted SQL that assembles most of the HTML for the newsletter using SQLite string concatenation! An illustrative snippet: My blog's URLs look like - this SQL constructs that three letter month abbreviation from the month number using a substring operation. This is a terrible way to assemble HTML, but I've stuck with it because it amuses me. The rest of the Observable notebook takes that data, filters out anything that links to content mentioned in the previous newsletters and composes it into a block of HTML that can be copied using that big button. Here's the recipe it uses to turn HTML into rich text content on a clipboard suitable for Substack. I can't remember how I figured this out but it's very effective: My blog itself is a Django application hosted on Heroku, with data stored in Heroku PostgreSQL. Here's the source code for that Django application . I use the Django admin as my CMS. Datasette provides a JSON API over a SQLite database... which means something needs to convert that PostgreSQL database into a SQLite database that Datasette can use. My system for doing that lives in the simonw/simonwillisonblog-backup GitHub repository. It uses GitHub Actions on a schedule that executes every two hours, fetching the latest data from PostgreSQL and converting that to SQLite. My db-to-sqlite tool is responsible for that conversion. I call it like this : That command uses Heroku credentials in an environment variable to fetch the database connection URL for my blog's PostgreSQL database (and fixes a small difference in the URL scheme). can then export that data and write it to a SQLite database file called . The options specify the tables that should be included in the export. The repository does more than just that conversion: it also exports the resulting data to JSON files that live in the repository, which gives me a commit history of changes I make to my content. This is a cheap way to get a revision history of my blog content without having to mess around with detailed history tracking inside the Django application itself. At the end of my GitHub Actions workflow is this code that publishes the resulting database to Datasette running on Fly.io using the datasette publish fly plugin: As you can see, there are a lot of moving parts! Surprisingly it all mostly just works - I rarely have to intervene in the process, and the cost of those different components is pleasantly low. You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options . The core process is the same as I described back in 2023 . I have an Observable notebook called blog-to-newsletter which fetches content from my blog's database, filters out anything that has been in the newsletter before, formats what's left as HTML and offers a big "Copy rich text newsletter to clipboard" button. I click that button, paste the result into the Substack editor, tweak a few things and hit send. The whole process usually takes just a few minutes. I make very minor edits: I set the title and the subheading for the newsletter. This is often a direct copy of the title of the featured blog post. Substack turns YouTube URLs into embeds, which often isn't what I want - especially if I have a YouTube URL inside a code example. Blocks of preformatted text often have an extra blank line at the end, which I remove. Occasionally I'll make a content edit - removing a piece of content that doesn't fit the newsletter, or fixing a time reference like "yesterday" that doesn't make sense any more. I pick the featured image for the newsletter and add some tags.

0 views
iDiallo 1 weeks ago

How Do You Send an Email?

It's been over a year and I didn't receive a single notification email from my web-server. It could either mean that my $6 VPS is amazing and hasn't gone down once this past year. Or it could mean that my health check service has gone down. Well this year, I have received emails from readers to tell me my website was down. So after doing some digging, I discovered that my health checker works just fine, but all emails it sends are being rejected by gmail. Unless you use a third party service, you have little to no chance of sending an email that gets delivered. Every year, email services seem to become a tad bit more expensive. When I first started this website, sending emails to my subscribers was free on Mailchimp. Now it costs $45 a month. On Buttondown, as of this writing, it costs $29 a month. What are they doing that costs so much? It seems like sending emails is impossibly hard, something you can almost never do yourself. You have to rely on established services if you want any guarantee that your email will be delivered. But is it really that complicated? Emails, just like websites, use a basic communication protocol to function. For you to land on this website, your browser somehow communicated with my web server, did some negotiating, and then my server sent HTML data that your browser rendered on the page. But what about email? Is the process any different? The short answer is no. Email and the web work in remarkably similar fashion. Here's the short version: In order to send me an email, your email client takes the email address you provide, connects to my server, does some negotiating, and then my server accepts the email content you intended to send and saves it. My email client will then take that saved content and notify me that I have a new message from you. That's it. That's how email works. So what's the big fuss about? Why are email services charging $45 just to send ~1,500 emails? Why is it so expensive, while I can serve millions of requests a day on my web server for a fraction of the cost? The short answer is spam . But before we get to spam, let's get into the details I've omitted from the examples above. The negotiations. How similar email and web traffic really are? When you type a URL into your browser and hit enter, here's what happens: The entire exchange is direct, simple, and happens in milliseconds. Now let's look at email. The process is similar: Both HTTP and email use DNS to find servers, establish TCP connections, exchange data using text-based protocols, and deliver content to the end user. They're built on the same fundamental internet technologies. So if email is just as simple as serving a website, why does it cost so much more? The answer lies in a problem that both systems share but handle very differently. Unwanted third-party writes. Both web servers and email servers allow outside parties to send them data. Web servers accept form submissions, comments, API requests, and user-generated content. Email servers accept messages from any other email server on the internet. In both cases, this openness creates an opportunity for abuse. Spam isn't unique to email, it's everywhere. My blog used to get around 6,000 spam comments on a daily basis. On the greater internet, you will see spam comments on blogs, spam account registrations, spam API calls, spam form submissions, and yes, spam emails. The main difference is visibility. When spam protection works well, it's invisible. You visit websites every day without realizing that behind the scenes. CAPTCHAs are blocking bot submissions, rate limiters are rejecting suspicious traffic, and content filters are catching spam comments before they're published. You don't get to see the thousands of spam attempts that happen every day on my blog, because of some filtering I've implemented. On a well run web-server, the work is invisible. The same is true for email. A well-run email server silently: There is a massive amount of spam. In fact, spam accounts for roughly 45-50% of all email traffic globally . But when the system works, you simply don't see it. If we can combat spam on the web without charging exorbitant fees, email spam shouldn't be that different. The technical challenges are very similar. Yet a basic web server on a $5/month VPS can handle millions of requests with minimal spam-fighting overhead. Meanwhile, sending 1,500 emails costs $29-45 per month through commercial services. The difference isn't purely technical. It's about reputation, deliverability networks, and the ecosystem that has evolved around email. Email providers have created a cartel-like system where your ability to reach inboxes depends on your server's reputation, which is nearly impossible to establish as a newcomer. They've turned a technical problem (spam) into a business moat. And we're all paying for it. Email isn't inherently more complex or expensive than web hosting. Both the protocols and the infrastructure are similar, and the spam problem exists in both domains. The cost difference is mostly artificial. It's the result of an ecosystem that has consolidated around a few major providers who control deliverability. It doesn't help that Intuit owns Mailchimp now. Understanding this doesn't necessarily change the fact that you'll probably still need to pay for email services if you want reliable delivery. But it should make you question whether that $45 monthly bill is really justified by the technical costs involved. Or whether it's just the price of admission to a gatekept system. DNS Lookup : Your browser asks a DNS server, "What's the IP address for this domain?" The DNS server responds with something like . Connection : Your browser establishes a TCP connection with that IP address on port 80 (HTTP) or port 443 (HTTPS). Request : Your browser sends an HTTP request: "GET /blog-post HTTP/1.1" Response : My web server processes the request and sends back the HTML, CSS, and JavaScript that make up the page. Rendering : Your browser receives this data and renders it on your screen. DNS Lookup : Your email client takes my email address ( ) and asks a DNS server, "What's the mail server for example.com?" The DNS server responds with an MX (Mail Exchange) record pointing to my mail server's address. Connection : Your email client (or your email provider's server) establishes a TCP connection with my mail server on port 25 (SMTP) or port 587 (for authenticated SMTP). Negotiation (SMTP) : Your server says "HELO, I have a message for [email protected]." My server responds: "OK, send it." Transfer : Your server sends the email content, headers, body, attachments, using the Simple Mail Transfer Protocol (SMTP). Storage : My mail server accepts the message and stores it in my mailbox, which can be a simple text file on the server. Retrieval : Later, when I open my email client, it connects to my server using IMAP (port 993) or POP3 (port 110) and asks, "Any new messages?" My server responds with your email, and my client displays it. Checks sender reputation against blacklists Validates SPF, DKIM, and DMARC records Scans message content for spam signatures Filters out malicious attachments Quarantines suspicious senders Both require reputation systems Both need content filtering Both face distributed abuse Both require infrastructure to handle high volume

0 views
Rik Huijzer 1 weeks ago

Do Not Put Your Site Behind Cloudflare if You Don't Need To

At the time of writing 12:43 UTC on Tue 18 Nov, Cloudflare has taken many sites down. I'm trying to browse the web, but about half of the sites show an error: ![cloudflare.webp](/files/45b312b038ccdc65) Most of these sites are not even that big. I expect maybe a few thousand visitors per month. This demonstrates again a simple fact: if you put your site behind a centralized service, then this service is a single point of failure. Even large established companies make mistakes and can go down. Most people use Cloudflare because they have been scared into the idea that you need DDoS protecti...

0 views
Rik Huijzer 1 weeks ago

Generating an SSH key for a webserver

Assuming you have the SSH password for a webserver called say `case` and email `[email protected]`, you can generate a key as follows: ``` ssh-keygen -t rsa -b 4096 -C "[email protected]" -f ~/.ssh/case ``` Next, add the server, which has say username `user` at location `case.example.com`, to your `~/.ssh/config`: ``` Host case HostName case.example.com User user IdentityFile ~/.ssh/case ``` Then you can copy this key to the server ``` ssh-copy-id -i ~/.ssh/case [email protected] ``` and afterwards log in with ``` ssh case ```

0 views
Brain Baking 2 weeks ago

Migrating From Gitea To Codeberg

After the last week’s Gitea attack debacle , moving all things Git off the VPS became a top priority. In 2022, like many of you, I gave up GitHub and spun up two Gitea instances myself: a private one safely behind bars on the NAS and a public one where all my public GitHub projects were moved to. Three years later, I think it’s time to move again. Gitea was forked into Forgejo a few years ago because of yet another licensing drama but I just couldn’t be bothered by keeping everything up to date. So I didn’t. And then Gitea started acting annoying: artifact folders weren’t properly cleaned up even though I encouraged it to do so in the configuration up to the point that the files clogged up the entire disk. The result was crashes of nearly everything as there wasn’t even a few bytes space left to append to logs in . And then bots started scraping the hell out of the commit endpoints. Anyway, I liked Gitea/Forgejo’s ease of use, so migrating everything off-site to Codeberg seemed like the most obvious solution. I’ve been wanting to go back to a coding community for a while now. if you host your own Gitea instance, you isolate yourself from the rest of the open source world. Collaboration still is the easiest on GitHub as simply everyone is hanging out there but I’d rather stop coding entirely than feed Microsoft’s AI. Give Up GitHub, folks. The steps involving the migration were surprisingly easy and fast to execute: Congratulations, you’re now the proud owner of Codeberg repositories. From here, we now have to decide what to do with the old Gitea instance. Do you want to simply kill it? Do you want to mirror your repositories? Or temporarily forward using the same URLs? Since I don’t want to keep it around forever and wanted to stop it immediately, but not yet break all URLs, I rewrote the Nginx location to a redirect: This does redirect https://codeberg.org/wouterg/brainbaking to the new correct location https://codeberg.org/wouterg/brainbaking , but it does not fix that will not follow redirects by default. You can proxy the entire thing and add more headers to fix that, or tell Git to follow redirects instead, or just change the remote URL and be done with it: . I figured not a lot of folks have cloned copies of my repositories on their hard drives—and if you do, you’d probably go looking online for the correct version and remove/reclone the entire thing. Next, it is time to kill the Gitea instance: (and ). Set a reminder in your calender to remove the redirect and clean up your VPS in a month or two, just to be sure. I should have enough backups in case things go wrong but you never know. The final piece of the puzzle is a financial one. Codeberg is a non-profit organisation that relies entirely on donations to keep things spinning, and I reckon they also need resources to fight those pesky AI crawlers (they’re also using Anubis, by the way). Consider donating or even becoming an active member that also allows you to vote when strategic decisions are being made. Go library programmers, don’t forget to double-check your import paths. So-called “vanity imports” ease friction here as you can set up a redirect from there. should still work. I rely on vangen to generate simple HTML pages for import paths. In the future, I’d rather move these paths to to avoid cluttering up Hugo’s folder. By Wouter Groeneveld on 13 November 2025.  Reply via email . Create a Codeberg account. Generate a temporary access token on your to-be-defunct Gitea instance: see https://docs.gitea.com/development/api-usage for the exact command. For each repository to migrate: click on your profile, and instead of creating a new blank repository, select “from migration” or go to URL https://codeberg.org/repo/migrate . Select Gitea, fill in the Git endpoint and access token and press migrate . Optionally, also migrate LFS/Wikis/issues/whatever by checking the appropriate boxes. Re-archive the repositories that were publicly archived.

0 views
maxdeviant.com 2 weeks ago

Head in the Zed Cloud

For the past five months I've been leading the efforts to rebuild Zed 's cloud infrastructure. Our current backend—known as Collab—has been chugging along since basically the beginning of the company. We use Collab every day to work together on Zed in Zed. However, as Zed continues to grow and attracts more users, we knew that we needed a full reboot of our backend infrastructure to set us up for success for our future endeavors. Enter Zed Cloud. Like Zed itself, Zed Cloud is built in Rust 1 . This time around there is a slight twist: all of this is running on Cloudflare Workers , with our Rust code being compiled down to WebAssembly (Wasm). One of our goals with this rebuild was to reduce the amount of operational effort it takes to maintain our hosted services, so that we can focus more of our time and energy on building Zed itself. Cloudflare Workers allow us to easily scale up to meet demand without having to fuss over it too much. Additionally, Cloudflare offers an ever-growing amount of managed services that cover anything you might need for a production web service. Here are some of the Cloudflare services we're using today: Another one of our goals with this rebuild was to build a platform that was easy to test. To achieve this, we built our own platform framework on top of the Cloudflare Workers runtime APIs. At the heart of this framework is the trait: This trait allows us to write our code in a platform-agnostic way while still leveraging all of the functionality that Cloudflare Workers has to offer. Each one of these associated types corresponds to some aspect of the platform that we'll want to have control over in a test environment. For instance, if we have a service that needs to interact with the system clock and a Workers KV store, we would define it like this: There are two implementors of the trait: and . —as the name might suggest—is an implementation of the platform on top of the Cloudflare Workers runtime. This implementation targets Wasm and is what we run when developing locally (using Wrangler ) and in production. We have a crate 2 that contains bindings to the Cloudflare Workers JS runtime. You can think of as the glue between those bindings and the idiomatic Rust APIs exposed by the trait. The is used when running tests, and allows for simulating almost every part of the system in order to effectively test our code. Here's an example of a test for ingesting a webhook from Orb : In this test we're able to test the full end-to-end flow of: The call to advances the test simulator, in this case running the pending queue consumers. At the center of the is the , a crate that powers our in-house async runtime. The scheduler is shared between GPUI —Zed's UI framework—and the used in tests. This shared scheduler enables us to write tests that span the client and the server. So we can have a test that starts in a piece of Zed code, flows through Zed Cloud, and then asserts on the state of something in Zed after it receives the response from the backend. The work being done on Zed Cloud now is laying the foundation to support our future work around collaborative coding with DeltaDB . If you want to work with me on building out Zed Cloud, we are currently hiring for this role. We're looking for engineers with experience building and maintaining web APIs and platforms, solid web fundamentals, and who are excited about Rust. If you end up applying, you can mention this blog post in your application. I look forward to hearing from you! The codebase is currently 70k lines of Rust code and 5.7k lines of TypeScript. This is essentially our own version of . I'd like to switch to using directly, at some point. Hyperdrive for talking to Postgres Workers KV for ephemeral storage Cloudflare Queues for asynchronous job processing Receiving and validating an incoming webhook event to our webhook ingestion endpoint Putting the webhook event into a queue Consuming the webhook event in a background worker and processing it

1 views
neilzone 3 weeks ago

Downgrading Debian from testing to stable (trixie)

I have some machines running Debian testing. However, it looks questionable whether I would get through Cyber Essentials running testing (or sid/unstable) rather than stable. So I can either try to downgrade the machines, or else reinstall the OS. The official guidance is not to downgrade: No, it isn’t supported Instead, to wipe and reinstall. I am fully prepared to wipe and reinstall if needed - in other words, I’ve taken and tested backups - so I didn’t really have much to lose by trying a downgrade. The worst case scenario is that I end up wiping and reinstalling anyway. I took, and tested, backups first. I reverted my sources.list entry: I then pinned the stable repos: And then I did the downgrade, which took a few minutes: I actually ran it twice. And… it mostly worked. On one machine, I had to fix a few bits by hand: On another machine, it finished cleanly. I also had to set up my accounts in Thunderbird again, because of a change in profile syntax. Overall, this went a lot more smoothly than I had expected. (Obviously, perhaps, YMMV…)

0 views
neilzone 3 weeks ago

Migrating Mastodon (glitch-soc fork) to another Intel NUC

Some brief notes on a successful migration of my Mastodon server, running the glitch-soc fork) from one Intel NUC to another. Mostly in case I need to do it again! I set up the machine with Debian 13 stable with LVM and LUKS, and then ran my usual set-up / hardening script. I followed the official Mastodon installation instructions to get started. Once I had created the mastodon user, I then rsync’d the directory from my old server to my new server, rather than going through the installation. I also dumped and rsync’d from my old machine the postgresql database, and the redis database, following the Mastodon migration instructions . I moved across the nginx config, and the entirety of , although I could have generated new certs easily enough. I used the new systemd unit files. I then followed the glitch-soc update instructions . I finally adjusted DNS and added some temporary NAT rules to redirect traffic while DNS changes were propogating. When I set up my Mastodon server in early 2018, I started with a Raspberry Pi 3. I moved to a Raspberry Pi 4 ( some brief notes ) in 2021, I think. When I had about 5,000 followers, my Raspberry Pi-based setup started to struggle, and I moved to an old Intel NUC (i5-3427U, 8GB RAM). I don’t recall exactly when that was. That was great, but it has started to struggle recently (just over 9,000 followers, and a busy feed), and I have been running out of disk space on it, even with regular media purge cycles. So now it is running on a bit less old Intel NUC (i7-5557U CPU @ 3.10GHz, 16GB RAM), with some more space for my bad jokes.

0 views
devansh 3 weeks ago

Hitchhiker's Guide to Attack Surface Management

I first heard about the word "ASM" (i.e., Attack Surface Management) probably in late 2018, and I thought it must be some complex infrastructure for tracking assets of an organization. Looking back, I realize I almost had a similar stack for discovering, tracking, and detecting obscure assets of organizations, and I was using it for my bug hunting adventures. I feel my stack was kinda goated, as I was able to find obscure assets of Apple, Facebook, Shopify, Twitter, and many other Fortune 100 companies, and reported hundreds of bugs, all through automation. Back in the day, projects like ProjectDiscovery were not present, so if I had to write an effective port scanner, I had to do it from scratch. (Masscan and nmap were present, but I had my fair share of issues using them, this is a story for another time). I used to write DNS resolvers (massdns had a high error rate), port scanners, web scrapers, directory brute-force utilities, wordlists, lots of JavaScript parsing logic using regex, and a hell of a lot of other things. I used to have up to 50+ self-developed tools for bug-bounty recon stuff and another 60-something helper scripts written in bash. I used to orchestrate (gluing together with duct tape is a better word) and slap together scripts like a workflow, and save the output in text files. Whenever I dealt with a large number of domains, I used to distribute the load over multiple servers (server spin-up + SSH into it + SCP for pushing and pulling files from it). The setup was very fragile and error-prone, and I spent countless nights trying to debug errors in the workflows. But it was all worth it. I learned the art of Attack Surface Management without even trying to learn about it. I was just a teenager trying to make quick bucks through bug hunting, and this fragile, duct-taped system was my edge. Fast forward to today, I have now spent almost a decade in the bug bounty scene. I joined HackerOne in 2020 (to present) as a vulnerability triager, where I have triaged and reviewed tens of thousands of vulnerability submissions. Fair to say, I have seen a lot of things, from doomsday level 0-days, to reports related to leaked credentials which could have led to entire infrastructure compromise, just because some dev pushed an AWS secret key in git logs, to things where some organizations were not even aware they were running Jenkins servers on some obscure subdomain which could have allowed RCE and then lateral movement to other layers of infrastructure. A lot of these issues I have seen were totally avoidable, only if organizations followed some basic attack surface management techniques. If I search "Guide to ASM" on Internet, almost none of the supposed guides are real resources. They funnel you to their own ASM solution, and the guide is just present there to provide you with some surface-level information, and is mostly a marketing gimmick. This is precisely why I decided to write something where I try to cover everything I learned and know about ASM, and how to protect your organization's assets before bad actors could get to them. This is going to be a rough and raw guide, and will not lead you to a funnel where I am trying to sell my own ASM SaaS to you. I have nothing to sell, other than offering what I know. But in case you are an organization who needs help implementing the things I am mentioning below, you can reach out to me via X or email (both available on the homepage of this blog). This guide will provide you with insights into exactly how big your attack surface really is. CISOs can look at it and see if their organizations have all of these covered, security researchers and bug hunters can look at this and maybe find new ideas related to where to look during recon. Devs can look at it and see if they are unintentionally leaving any door open for hackers. If you are into security, it has something to offer you. Attack surface is one of those terms getting thrown around in security circles so much that it's become almost meaningless noise. In theory, it sounds simple enough, right. Attack surface is every single potential entry point, interaction vector, or exploitable interface an attacker could use to compromise your systems, steal your data, or generally wreck your day. But here's the thing, it's the sum total of everything you've exposed to the internet. Every API endpoint you forgot about, every subdomain some dev spun up for "testing purposes" five years ago and then abandoned, every IoT device plugged into your network, every employee laptop connecting from a coffee shop, every third-party vendor with a backdoor into your environment, every cloud storage bucket with permissions that make no sense, every Slack channel, every git commit leaking credentials, every paste on Pastebin containing your database passwords. Most organizations think about attack surface in incredibly narrow terms. They think if they have a website, an email server, and maybe some VPN endpoints, they've got "good visibility" into their assets. That's just plain wrong. Straight up wrong. Your actual attack surface would terrify you if you actually understood it. You run , and is your main domain. You probably know about , , maybe . But what about that your intern from 2015 spun up and just never bothered to delete. It's not documented anywhere. Nobody remembers it exists. Domain attack surface goes way beyond what's sitting in your asset management system. Every subdomain is a potential entry point. Most of these subdomains are completely forgotten. Subdomain enumeration is reconnaissance 101 for attackers and bug hunters. It's not rocket science. Setting up a tool that actively monitors through active and passive sources for new subdomains and generates alerts is honestly an hour's worth of work. You can use tools like Subfinder, Amass, or just mine Certificate Transparency logs to discover every single subdomain connected to your domain. Certificate Transparency logs were designed to increase security by making certificate issuance public, and they've become an absolute reconnaissance goldmine. Every time you get an SSL certificate for , that information is sitting in public logs for anyone to find. Attackers systematically enumerate these subdomains using Certificate Transparency log searches, DNS brute-forcing with massive wordlists, reverse DNS lookups to map IP ranges back to domains, historical DNS data from services like SecurityTrails, and zone transfer exploitation if your DNS is misconfigured. Attackers are looking for old development environments still running vulnerable software, staging servers with production data sitting on them, forgotten admin panels, API endpoints without authentication, internal tools accidentally exposed, and test environments with default credentials nobody changed. Every subdomain is an asset. Every asset is a potential vulnerability. Every vulnerability is an entry point. Domains and subdomains are just the starting point though. Once you've figured out all the subdomains belonging to your organization, the next step is to take a hard look at IP address space, which is another absolutely massive component of your attack surface. Organizations own, sometimes lease, IP ranges, sometimes small /24 blocks, sometimes massive /16 ranges, and every single IP address in those blocks and ranges that responds to external traffic is part of your attack surface. And attackers enumerate them all if you won't. They use WHOIS lookups to identify your IP ranges, port scanning to find what services are running where, service fingerprinting to identify exact software versions, and banner grabbing to extract configuration information. If you have a /24 network with 256 IP addresses and even 10% of those IPs are running services, you've got 25 potential attack vectors. Scale that to a /20 or /16 and you're looking at thousands of potential entry points. And attackers aren't just looking at the IPs you know about. They're looking at adjacent IP ranges you might have acquired through mergers, historical IP allocations that haven't been properly decommissioned, and shared IP ranges where your servers coexist with others. Traditional infrastructure was complicated enough, and now we have cloud. It's literally exploded organizations' attack surfaces in ways that are genuinely difficult to even comprehend. Every cloud service you spin up, be it an EC2 instance, S3 bucket, Lambda function, or API Gateway endpoint, all of this is a new attack vector. In my opinion and experience so far, I think the main issue with cloud infrastructure is that it's ephemeral and distributed. Resources get spun up and torn down constantly. Developers create instances for testing and forget about them. Auto-scaling groups generate new resources dynamically. Containerized workloads spin up massive Kubernetes clusters you have minimal visibility into. Your cloud attack surface could be literally anything. Examples are countless, but I'd categorize them into 8 different categories. Compute instances like EC2, Azure VMs, GCP Compute Engine instances exposed to the internet. Storage buckets like S3, Azure Blob Storage, GCP Cloud Storage with misconfigured permissions. Serverless stuff like Lambda functions with public URLs or overly permissive IAM roles. API endpoints like API Gateway, Azure API Management endpoints without proper authentication. Container registries like Docker images with embedded secrets or vulnerabilities. Kubernetes clusters with exposed API servers, misconfigured network policies, vulnerable ingress controllers. Managed databases like RDS, CosmosDB, Cloud SQL instances with weak access controls. IAM roles and service accounts with overly permissive identities that enable privilege escalation. I've seen instances in the past where a single misconfigured S3 bucket policy exposed terabytes of data. An overly permissive Lambda IAM role enabled lateral movement across an entire AWS account. A publicly accessible Kubernetes API server gave an attacker full cluster control. Honestly, cloud kinda scares me as well. And to top it off, multi-cloud infrastructure makes everything worse. If you're running AWS, Azure, and GCP together, you've just tripled your attack surface management complexity. Each cloud provider has different security models, different configuration profiles, and different attack vectors. Every application now uses APIs, and all applications nowadays are like a constellation of APIs talking to each other. Every API you use in your organization is your attack surface. The problem with APIs is that they're often deployed without the same security scrutiny as traditional web applications. Developers spin up API endpoints for specific features and those endpoints accumulate over time. Some of them are shadow APIs, meaning API endpoints which aren't documented anywhere. These endpoints are the equivalent of forgotten subdomains, and attackers can find them through analyzing JavaScript files for API endpoint references, fuzzing common API path patterns, examining mobile app traffic to discover backend APIs, and mining old documentation or code repositories for deprecated endpoints. Your API attack surface includes REST APIs exposed to the internet, GraphQL endpoints with overly broad query capabilities, WebSocket connections for real-time functionality, gRPC services for inter-service communication, and legacy SOAP APIs that never got decommissioned. If your organization has mobile apps, be it iOS, Android, or both, this is a direct window to your infrastructure and should be part of your attack surface management strategy. Mobile apps communicate with backend APIs and those API endpoints are discoverable by reversing the app. The reversed source of the app could reveal hard-coded API keys, tokens, and credentials. Using JADX plus APKTool plus Dex2jar is all a motivated attacker needs. Web servers often expose directories and files that weren't meant to be publicly accessible. Attackers systematically enumerate these using automated tools like ffuf, dirbuster, gobuster, and wfuzz with massive wordlists to discover hidden endpoints, configuration files, backup files, and administrative interfaces. Common exposed directories include admin panels, backup directories containing database dumps or source code, configuration files with database credentials and API keys, development directories with debug information, documentation directories revealing internal systems, upload directories for file storage, and old or forgotten directories from previous deployments. Your attack surface must include directories which are accidentally left accessible during deployments, staging servers with production data, backup directories with old source code versions, administrative interfaces without authentication, API documentation exposing endpoint details, and test directories with debug output enabled. Even if you've removed a directory from production, old cached versions may still be accessible through web caches or CDNs. Search engines also index these directories, making them discoverable through dorking techniques. If your organization is using IoT devices, and everyone uses these days, this should be part of your attack surface management strategy. They're invisible to traditional security tools. Your EDR solution doesn't protect IoT devices. Your vulnerability scanner can't inventory them. Your patch management system can't update them. Your IoT attack surface could include smart building systems like HVAC, lighting, access control. Security cameras and surveillance systems. Printers and copiers, which are computers with network access. Badge readers and physical access systems. Industrial control systems and SCADA devices. Medical devices in healthcare environments. Employee wearables and fitness trackers. Voice assistants and smart speakers. The problem with IoT devices is that they're often deployed without any security consideration. They have default credentials that never get changed, unpatched firmware with known vulnerabilities, no encryption for data in transit, weak authentication mechanisms, and insecure network configurations. Social media presence is an attack surface component that most organizations completely ignore. Attackers can use social media for reconnaissance by looking at employee profiles on LinkedIn to reveal organizational structure, technologies in use, and current projects. Twitter/X accounts can leak information about deployments, outages, and technology stack. Employee GitHub profiles expose email patterns and development practices. Company blogs can announce new features before security review. It could also be a direct attack vector. Attackers can use information from social media to craft convincing phishing attacks. Hijacked social media accounts can be used to spread malware or phishing links. Employees can accidentally share sensitive information. Fake accounts can impersonate your brand to defraud customers. Your employees' social media presence is part of your attack surface whether you like it or not. Third-party vendors, suppliers, contractors, or partners with access to your systems should be part of your attack surface. Supply chain attacks are becoming more and more common these days. Attackers can compromise a vendor with weaker security and then use that vendor's access to reach your environment. From there, they pivot from the vendor network to your systems. This isn't a hypothetical scenario, it has happened multiple times in the past. You might have heard about the SolarWinds attack, where attackers compromised SolarWinds' build system and distributed malware through software updates to thousands of customers. Another famous case study is the MOVEit vulnerability in MOVEit Transfer software, exploited by the Cl0p ransomware group, which affected over 2,700 organizations. These are examples of some high-profile supply chain security attacks. Your third-party attack surface could include things like VPNs, remote desktop connections, privileged access systems, third-party services with API keys to your systems, login credentials shared with vendors, SaaS applications storing your data, and external IT support with administrative access. It's obvious you can't directly control third-party security. You can audit them, have them pen-test their assets as part of your vendor compliance plan, and include security requirements in contracts, but ultimately their security posture is outside your control. And attackers know this. GitHub, GitLab, Bitbucket, they all are a massive attack surface. Attackers search through code repositories in hopes of finding hard-coded credentials like API keys, database passwords, and tokens. Private keys, SSH keys, TLS certificates, and encryption keys. Internal architecture documentation revealing infrastructure details in code comments. Configuration files with database connection strings and internal URLs. Deprecated code with vulnerabilities that's still in production. Even private repositories aren't safe. Attackers can compromise developer accounts to access private repositories, former employees retain access after leaving, and overly broad repository permissions grant access to too many people. Automated scanners continuously monitor public repositories for secrets. The moment a developer accidentally pushes credentials to a public repository, automated systems detect it within minutes. Attackers have already extracted and weaponized those credentials before the developer realizes the mistake. CI/CD pipelines are massive another attack vector. Especially in recent times, and not many organizations are giving attention to this attack vector. This should totally be part of your attack surface management. Attackers compromise GitHub Actions workflows with malicious code injection, Jenkins servers with weak authentication, GitLab CI/CD variables containing secrets, and build artifacts with embedded malware. The GitHub Actions supply chain attack, CVE-2025-30066, demonstrated this perfectly. Attackers compromised the Action used in over 23,000 repositories, injecting malicious code that leaked secrets from build logs. Jenkins specifically is a goldmine for attackers. An exposed Jenkins instance provides complete control over multiple critical servers, access to hardcoded AWS keys, Redis credentials, and BitBucket tokens, ability to manipulate builds and inject malicious code, and exfiltration of production database credentials containing PII. Modern collaboration tools are massive attack surface components that most organizations underestimate. Slack has hidden security risks despite being invite-only. Slack attack surface could include indefinite data retention where every message, channel, and file is stored forever unless admins configure retention periods. Public channels accessible to all users so one breached account opens the floodgates. Third-party integrations with excessive permissions accessing messages and user data. Former contractor access where individuals retain access long after projects end. Phishing and impersonation where it's easy to change names and pictures to impersonate senior personnel. In 2022, Slack leaked hashed passwords for five years affecting 0.5% of users. Slack channels commonly contain API keys, authentication tokens, database credentials, customer PII, financial data, internal system passwords, and confidential project information. The average cost of a breached record was $164 in 2022. When 1 in 166 messages in Slack contains confidential information, every new message adds another dollar to total risk exposure. With 5,000 employees sending 30 million Slack messages per year, that's substantial exposure. Trello board exposure is a significant attack surface. Trello attack vectors include public boards with sensitive information accidentally shared publicly, default public visibility where boards are created as public by default in some configurations, unsecured REST API allowing unauthenticated access to user data, and scraping attacks where attackers use email lists to enumerate Trello accounts. The 2024 Trello data breach exposed 15 million users' personal information when a threat actor named "emo" exploited an unsecured REST API using 500 million email addresses to compile detailed user profiles. Security researcher David Shear documented hundreds of public Trello boards exposing passwords, credentials, IT support customer access details, website admin logins, and client server management credentials. IT companies were using Trello to troubleshoot client requests and manage infrastructure, storing all credentials on public Trello boards. Jira misconfiguration is a widespread attack surface issue. Common misconfigurations include public dashboards and filters with "Everyone" access actually meaning public internet access, anonymous access enabled allowing unauthenticated users to browse, user picker functionality providing complete lists of usernames and email addresses, and project visibility allowing sensitive projects to be accessible without authentication. Confluence misconfiguration exposes internal documentation. Confluence attack surface components include anonymous access at site level allowing public access, public spaces where space admins grant anonymous permissions, inherited permissions where all content within a space inherits space-level access, and user profile visibility allowing anonymous users to view profiles of logged-in users. When anonymous access is enabled globally and space admins allow anonymous users to access their spaces, anyone on the internet can access that content. Confluence spaces often contain internal documentation with hardcoded credentials, financial information, project details, employee information, and API documentation with authentication details. Cloud storage misconfiguration is epidemic. Google Drive misconfiguration attack surface includes "Anyone with the link" sharing making files accessible without authentication, overly permissive sharing defaults making it easy to accidentally share publicly, inherited folder permissions exposing everything beneath, unmanaged third-party apps with excessive read/write/delete permissions, inactive user accounts where former employees retain access, and external ownership blind spots where externally-owned content is shared into the environment. Metomic's 2023 Google Scanner Report found that of 6.5 million Google Drive files analyzed, 40.2% contained sensitive information, 34.2% were shared externally, and 0.5% were publicly accessible, mostly unintentionally. In December 2023, Japanese game developer Ateam suffered a catastrophic Google Drive misconfiguration that exposed personal data of nearly 1 million people for over six years due to "Anyone with the link" settings. Based on Valence research, 22% of external data shares utilize open links, and 94% of these open link shares are inactive, forgotten files with public URLs floating around the internet. Dropbox, OneDrive, and Box share similar attack surface components including misconfigured sharing permissions, weak or missing password protection, overly broad access grants, third-party app integrations with excessive permissions, and lack of visibility into external sharing. Features that make file sharing convenient create data leakage risks when misconfigured. Pastebin and similar paste sites are both reconnaissance sources and attack vectors. Paste site attack surface includes public data dumps of stolen credentials, API keys, and database dumps posted publicly, malware hosting of obfuscated payloads, C2 communications where malware uses Pastebin for command and control, credential leakage from developers accidentally posting secrets, and bypassing security filters since Pastebin is legitimate so security tools don't block it. For organizations, leaked API keys or database credentials on Pastebin lead to unauthorized access, data exfiltration, and service disruption. Attackers continuously scan Pastebin for mentions of target organizations using automated tools. Security teams must actively monitor Pastebin and similar paste sites for company name mentions, email domain references, and specific keywords related to the organization. Because paste sites don't require registration or authentication and content is rarely removed, they've become permanent archives of leaked secrets. Container registries expose significant attack surface. Container registry attack surface includes secrets embedded in image layers where 30,000 unique secrets were found in 19,000 images, with 10% of scanned Docker images containing secrets, and 1,200 secrets, 4%, being active and valid. Immutable cached layers contain 85% of embedded secrets that can't be removed, exposed registries with 117 Docker registries accessible without authentication, unsecured registries allowing pull, push, and delete operations, and source code exposure where full application code is accessible by pulling images. GitGuardian's analysis of 200,000 publicly available Docker images revealed a staggering secret exposure problem. Even more alarming, 99% of images containing active secrets were pulled in 2024, demonstrating real-world exploitation. Unit 42's research identified 941 Docker registries exposed to the internet, with 117 accessible without authentication containing 2,956 repositories, 15,887 tags, and full source code and historical versions. Out of 117 unsecured registries, 80 allow pull operations to download images, 92 allow push operations to upload malicious images, and 7 allow delete operations for ransomware potential. Sysdig's analysis of over 250,000 Linux images on Docker Hub found 1,652 malicious images including cryptominers, most common, embedded secrets, second most prevalent, SSH keys and public keys for backdoor implants, API keys and authentication tokens, and database credentials. The secrets found in container images included AWS access keys, database passwords, SSH private keys, API tokens for cloud services, GitHub personal access tokens, and TLS certificates. Shadow IT includes unapproved SaaS applications like Dropbox, Google Drive, and personal cloud storage used for work. Personal devices like BYOD laptops, tablets, and smartphones accessing corporate data. Rogue cloud deployments where developers spin up AWS instances without approval. Unauthorized messaging apps like WhatsApp, Telegram, and Signal used for business communication. Unapproved IoT devices like smart speakers, wireless cameras, and fitness trackers on the corporate network. Gartner estimates that shadow IT makes up 30-40% of IT spending in large companies, and 76% of organizations surveyed experienced cyberattacks due to exploitation of unknown, unmanaged, or poorly managed assets. Shadow IT expands your attack surface because it's not protected by your security controls, it's not monitored by your security team, it's not included in your vulnerability scans, it's not patched by your IT department, and it often has weak or default credentials. And you can't secure what you don't know exists. Bring Your Own Device, BYOD, policies sound great for employee flexibility and cost savings. For security teams, they're a nightmare. BYOD expands your attack surface by introducing unmanaged endpoints like personal devices without EDR, antivirus, or encryption. Mixing personal and business use where work data is stored alongside personal apps with unknown security. Connecting from untrusted networks like public Wi-Fi and home networks with compromised routers. Installing unapproved applications with malware or excessive permissions. Lacking consistent security updates with devices running outdated operating systems. Common BYOD security issues include data leakage through personal cloud backup services, malware infections from personal app downloads, lost or stolen devices containing corporate data, family members using devices that access work systems, and lack of IT visibility and control. The 60% of small and mid-sized businesses that close within six months of a major cyberattack often have BYOD-related security gaps as contributing factors. Remote access infrastructure like VPNs and Remote Desktop Protocol, RDP, are among the most exploited attack vectors. SSL VPN appliances from vendors like Fortinet, SonicWall, Check Point, and Palo Alto are under constant attack. VPN attack vectors include authentication bypass vulnerabilities with CVEs allowing attackers to hijack active sessions, credential stuffing through brute-forcing VPN logins with leaked credentials, exploitation of unpatched vulnerabilities with critical CVEs in VPN appliances, and configuration weaknesses like default credentials, weak passwords, and lack of MFA. Real-world attacks demonstrate the risk. Check Point SSL VPN CVE-2024-24919 allowed authentication bypass for session hijacking. Fortinet SSL-VPN vulnerabilities were leveraged for lateral movement and privilege escalation. SonicWall CVE-2024-53704 allowed remote authentication bypass for SSL VPN. Once inside via VPN, attackers conduct network reconnaissance, lateral movement, and privilege escalation. RDP is worse. Sophos found that cybercriminals abused RDP in 90% of attacks they investigated. External remote services like RDP were the initial access vector in 65% of incident response cases. RDP attack vectors include exposed RDP ports with port 3389 open to the internet, weak authentication with simple passwords vulnerable to brute force, lack of MFA with no second factor for authentication, and credential reuse from compromised passwords in data breaches. In one Darktrace case, attackers compromised an organization four times in six months, each time through exposed RDP ports. The attack chain went successful RDP login, internal reconnaissance via WMI, lateral movement via PsExec, and objective achievement. The Palo Alto Unit 42 Incident Response report found RDP was the initial attack vector in 50% of ransomware deployment cases. Email infrastructure remains a primary attack vector. Your email attack surface includes mail servers like Exchange, Office 365, and Gmail with configuration weaknesses, email authentication with misconfigured SPF, DKIM, and DMARC records, phishing-susceptible users targeted through social engineering, email attachments and links as malware delivery mechanisms, and compromised accounts through credential stuffing or password reuse. Email authentication misconfiguration is particularly insidious. If your SPF, DKIM, and DMARC records are wrong or missing, attackers can spoof emails from your domain, your legitimate emails get marked as spam, and phishing emails impersonating your organization succeed. Email servers themselves are also targets. The NSA released guidance on Microsoft Exchange Server security specifically because Exchange servers are so frequently compromised. Container orchestration platforms like Kubernetes introduce massive attack surface complexity. The Kubernetes attack surface includes the Kubernetes API server with exposed or misconfigured API endpoints, container images with vulnerabilities in base images or application layers, container registries like Docker Hub, ECR, and GCR with weak access controls, pod security policies with overly permissive container configurations, network policies with insufficient micro-segmentation between pods, secrets management with hardcoded secrets or weak secret storage, and RBAC misconfigurations with overly broad service account permissions. Container security issues include containers running as root with excessive privileges, exposed Docker daemon sockets allowing container escape, vulnerable dependencies in container images, and lack of runtime security monitoring. The Docker daemon attack surface is particularly concerning. Running containers with privileged access or allowing docker.sock access can enable container escape and host compromise. Serverless computing like AWS Lambda, Azure Functions, and Google Cloud Functions promised to eliminate infrastructure management. Instead, it just created new attack surfaces. Serverless attack surface components include function code vulnerabilities like injection flaws and insecure dependencies, IAM misconfigurations with overly permissive Lambda execution roles, environment variables storing secrets as plain text, function URLs with publicly accessible endpoints without authentication, and event source mappings with untrusted input from various cloud services. The overabundance of event sources expands the attack surface. Lambda functions can be triggered by S3 events, API Gateway requests, DynamoDB streams, SNS topics, EventBridge schedules, IoT events, and dozens more. Each event source is a potential injection point. If function input validation is insufficient, attackers can manipulate event data to exploit the function. Real-world Lambda attacks include credential theft by exfiltrating IAM credentials from environment variables, lateral movement using over-permissioned roles to access other AWS resources, and data exfiltration by invoking functions to query and extract database contents. The Scarlet Eel adversary specifically targeted AWS Lambda for credential theft and lateral movement. Microservices architecture multiplies attack surface by decomposing monolithic applications into dozens or hundreds of independent services. Each microservice has its own attack surface including authentication mechanisms where each service needs to verify requests, authorization rules where each service enforces access controls, API endpoints for service-to-service communication channels, data stores where each service may have its own database, and network interfaces where each service exposes network ports. Microservices security challenges include east-west traffic vulnerabilities with service-to-service communication without encryption or authentication, authentication and authorization complexity from managing auth across 40 plus services multiplied by 3 environments equaling 240 configurations, service-to-service trust where services blindly trust internal traffic, network segmentation failures with flat networks allowing unrestricted pod-to-pod communication, and inconsistent security policies with different services having different security standards. One compromised microservice can enable lateral movement across the entire application. Without proper network segmentation and zero trust architecture, attackers pivot from service to service. How do you measure something this large, right. Attack surface measurement is complex. Attack surface metrics include the total number of assets with all discovered systems, applications, and devices, newly discovered assets found through continuous discovery, the number of exposed assets accessible from the internet, open ports and services with network services listening for connections, vulnerabilities by severity including critical, high, medium, and low CVEs, mean time to detect, MTTD, measuring how quickly new assets are discovered, mean time to remediate, MTTR, measuring how quickly vulnerabilities are fixed, shadow IT assets that are unknown or unmanaged, third-party exposure from vendor and partner access points, and attack surface change rate showing how rapidly the attack surface evolves. Academic research has produced formal attack surface measurement methods. Pratyusa Manadhata's foundational work defines attack surface as a three-tuple, System Attackability, Channel Attackability, Data Attackability. But in practice, most organizations struggle with basic attack surface visibility, let alone quantitative measurement. Your attack surface isn't static. It changes constantly. Changes happen because developers deploy new services and APIs, cloud auto-scaling spins up new instances, shadow IT appears as employees adopt unapproved tools, acquisitions bring new infrastructure into your environment, IoT devices get plugged into your network, and subdomains get created for new projects. Static, point-in-time assessments are obsolete. You need continuous asset discovery and monitoring. Continuous discovery methods include automated network scanning for regular scans to detect new devices, cloud API polling to query cloud provider APIs for resource changes, DNS monitoring to track new subdomains via Certificate Transparency logs, passive traffic analysis to observe network traffic and identify assets, integration with CMDB or ITSM to sync with configuration management databases, and cloud inventory automation using Infrastructure as Code to track deployments. Understanding your attack surface is step one. Reducing it is the goal. Attack surface reduction begins with asset elimination by removing unnecessary assets entirely. This includes decommissioning unused servers and applications, deleting abandoned subdomains and DNS records, shutting down forgotten development environments, disabling unused network services and ports, and removing unused user accounts and service identities. Access control hardening implements least privilege everywhere by enforcing multi-factor authentication, MFA, for all remote access, using role-based access control, RBAC, for cloud resources, implementing zero trust network architecture, restricting network access with micro-segmentation, and applying the principle of least privilege to IAM roles. Exposure minimization reduces what's visible to attackers by moving services behind VPNs or bastion hosts, using private IP ranges for internal services, implementing network address translation, NAT, for outbound access, restricting API endpoints to authorized sources only, and disabling unnecessary features and functionalities. Security hardening strengthens what remains by applying security patches promptly, using security configuration baselines, enabling encryption for data in transit and at rest, implementing Web Application Firewalls, WAF, for web apps, and deploying endpoint detection and response, EDR, on all devices. Monitoring and detection watch for attacks in progress by implementing real-time threat detection, enabling comprehensive logging and SIEM integration, deploying intrusion detection and prevention systems, IDS/IPS, monitoring for anomalous behavior patterns, and using threat intelligence feeds to identify known bad actors. Your attack surface is exponentially larger than you think it is. Every asset you know about probably has three you don't. Every known vulnerability probably has ten undiscovered ones. Every third-party integration probably grants more access than you realize. Every collaboration tool is leaking more data than you imagine. Every paste site contains more of your secrets than you want to admit. And attackers know this. They're not just looking at what you think you've secured. They're systematically enumerating every possible entry point. They're mining Certificate Transparency logs for forgotten subdomains. They're scanning every IP in your address space. They're reverse-engineering your mobile apps. They're buying employee credentials from data breach databases. They're compromising your vendors to reach you. They're scraping Pastebin for your leaked secrets. They're pulling your public Docker images and extracting the embedded credentials. They're accessing your misconfigured S3 buckets and exfiltrating terabytes of data. They're exploiting your exposed Jenkins instances to compromise your entire infrastructure. They're manipulating your AI agents to exfiltrate private Notion data. The asymmetry is brutal. You have to defend every single attack vector. They only need to find one that works. So what do you do. Start by accepting that you don't have complete visibility. Nobody does. But you can work toward better visibility through continuous discovery, automated asset management, and integration of security tools that help map your actual attack surface. Implement attack surface reduction aggressively. Every asset you eliminate is one less thing to defend. Every service you shut down is one less potential vulnerability. Every piece of shadow IT you discover and bring under management is one less blind spot. Every misconfigured cloud storage bucket you fix is terabytes of data no longer exposed. Every leaked secret you rotate is one less credential floating around the internet. Adopt zero trust architecture. Stop assuming that anything, internal services, microservices, authenticated users, collaboration tools, is inherently trustworthy. Verify everything. Monitor paste sites and code repositories. Your secrets are out there. Find them before attackers weaponize them. Secure your collaboration tools. Slack, Trello, Jira, Confluence, Notion, Google Drive, and Airtable are all leaking data. Lock them down. Fix your container security. Scan images for secrets. Use secret managers instead of environment variables. Secure your registries. Harden your CI/CD pipelines. Jenkins, GitHub Actions, and GitLab CI are high-value targets. Protect them. And test your assumptions with red team exercises and continuous security testing. Your attack surface is what an attacker can reach, not what you think you've secured. The attack surface problem isn't getting better. Cloud adoption, DevOps practices, remote work, IoT proliferation, supply chain complexity, collaboration tool sprawl, and container adoption are all expanding organizational attack surfaces faster than security teams can keep up. But understanding the problem is the first step toward managing it. And now you understand exactly how catastrophically large your attack surface actually is.

1 views
./techtipsy 3 weeks ago

The day IPv6 went away

I take pride in hosting my blog on a 13-year old ThinkPad acting as a home server , but sometimes it’s kind of a pain. It’s only fair that I cover the downsides of this setup in contrast to all the positives. Yesterday, I happened to notice that a connection to a backup endpoint was gone. Okay, happens sometimes. Then I went into the router and noticed that hey, that’s odd, there’s no WAN6 connection showing up. All gone. Just as if I had gone back to a crappy ISP that only provides IPv4 ! Restarting the interface did not work, but a full router restart worked. Since the IPv4 address and IPv6 prefix are all dynamic, that meant that my DNS entries had just gone stale. I do have a custom DNS auto-updater script for my DNS provider, but DNS propagation takes time. Luckily not a lot of time, my uptime checker only reported downtime of 5-15 minutes, depending on the domain. Here’s what it looked like on OpenWRT. Impact to my blog? Not really noticeable, since IPv4 kept trucking along. Perhaps a few IPv6-only readers may have noticed this. 1 I can always move to a cheap VPS or the cloud at a moments’ notice, but where’s the fun in that? I can produce AWS levels of uptime at home, thankyouverymuch ! I think I’ll now need to figure out some safeguards, even if it means scheduling a weekly router reboot if the WAN6 interface is not up for X amount of time. That, and better monitoring. if you are that person, say hi!  ↩︎ if you are that person, say hi!  ↩︎

0 views