Posts in Docker (18 found)
blog.philz.dev 1 months ago

Containerizing Agents

Simon Willison has been writing about using parallel coding agents ( blog ), and his post encouraged me to write down about my current workflow, which involves both parallelism, containerization, and web browsers. I’m spoiled by (and helped build) sketch.dev’s agent containerization , so, when I need to use other agents as well, I wrote a shell script to containerize them "just so." My workflow is that I run , and I find myself in a web browser, in the same git repo I was in, but now in a randomly named branch, in a container, in . The first pane is the agent, but there are other panes doing other stuff. When I'm done, I've got a branch to work with, and I merge/rebase/cherry-pick. Let's break up the pieces: First, my shell script is in my favorite shell scripting language, dependency-less python3. Python3 has the advantage of not requiring you to think about and is sufficiently available. Second, I have a customized Dockerfile with the dependencies my projects need. I don't minimize the container; I add all the things I want. Browsers, playwright, subtrace, tmux, etc. Third, U cross-mount my git repo itself into the container, and create a worktree inside the container. From the outside, this work tree is going to look "prunable", but that causes no harm, and there’s a new branch that corresponds to the agent’s worktree. I like worktrees more than remotes because they’re in the same namespace; you don’t need to "fetch" or "push" across them. It’s easy to lose changes when the container exits; I commit automatically on exit. It's also easy to lose the worktree if something calls on your behalf, but recovery is possible with and some fiddling. Fourth, I run tmux inside the container so that opening a shell in the container is as simple as opening a new pane. (Somehow, is too rich.) I'm used to sketch.dev's terminal pane to do the little git operation, take a look at a diff, run a server... tmux helps. Fifth, networking magic with Tailscale. I publish ports 8000-9999 (and 11111) on my tailnet, using the same randomly generated name as I've used for my container and my branch. You're inevitably working on a web app, and you inevitably need to actually look at it, and Docker networking is doable, but you have to pre-declare exposed ports, and avoid conflicts, and ... it's just not great for this use case. There are other solutions (ngrok, SSH port forwarding), but I already use Tailscale, so this works nicely. I originally started with tsnsrv , but then vibe-coded a custom thing that supports port ranges. is the userland networking library here, and the agents do a fine job one-shotting this stuff. Sixth, I use to expose my to my browser over the tailnet network. I'm used to having a browser-tab per agent, and this gives me that. (Terminal-based agents feel weird to me. Browsers are great at scrolling, expand/collapse widgets, cut and paste, word wrap of text, etc.) Seventh, I vibe-coded a headless browser tool called , which wraps the excellent chromedp library, which remote-controls a headless Chrome over it's debugging protocol. Getting the MCPs configured for playwright was finnicky, especially across multiple agents, and I'm experimenting with this command line tool to do the same. As I’ve written about before . Using agents in containers gives me two things I value: Isolation for parallel work. The agents can start processes and run tests and so forth without conflicting on ports or files. A bit more security. Even the Economist has now picked up on the Lethal Trifecta (or Simon Willison's original) . By explicitly choosing which environment variables I forward, and not sharing my cookies and my SSH keys, I’m exerting some control over what data and capabilities are exposed to the agent. We’re still playing with fire (can you break out of Colima? Sure! Can you edit my git repo? Sure! Break into my tailnet? Sorta.), but it’s a smaller, more controlled burn. If you want to try my nonsense, https://github.com/philz/ctr-agent . First, my shell script is in my favorite shell scripting language, dependency-less python3. Python3 has the advantage of not requiring you to think about and is sufficiently available. Second, I have a customized Dockerfile with the dependencies my projects need. I don't minimize the container; I add all the things I want. Browsers, playwright, subtrace, tmux, etc. Third, U cross-mount my git repo itself into the container, and create a worktree inside the container. From the outside, this work tree is going to look "prunable", but that causes no harm, and there’s a new branch that corresponds to the agent’s worktree. I like worktrees more than remotes because they’re in the same namespace; you don’t need to "fetch" or "push" across them. It’s easy to lose changes when the container exits; I commit automatically on exit. It's also easy to lose the worktree if something calls on your behalf, but recovery is possible with and some fiddling. Fourth, I run tmux inside the container so that opening a shell in the container is as simple as opening a new pane. (Somehow, is too rich.) I'm used to sketch.dev's terminal pane to do the little git operation, take a look at a diff, run a server... tmux helps. Fifth, networking magic with Tailscale. I publish ports 8000-9999 (and 11111) on my tailnet, using the same randomly generated name as I've used for my container and my branch. You're inevitably working on a web app, and you inevitably need to actually look at it, and Docker networking is doable, but you have to pre-declare exposed ports, and avoid conflicts, and ... it's just not great for this use case. There are other solutions (ngrok, SSH port forwarding), but I already use Tailscale, so this works nicely. I originally started with tsnsrv , but then vibe-coded a custom thing that supports port ranges. is the userland networking library here, and the agents do a fine job one-shotting this stuff. Sixth, I use to expose my to my browser over the tailnet network. I'm used to having a browser-tab per agent, and this gives me that. (Terminal-based agents feel weird to me. Browsers are great at scrolling, expand/collapse widgets, cut and paste, word wrap of text, etc.) Seventh, I vibe-coded a headless browser tool called , which wraps the excellent chromedp library, which remote-controls a headless Chrome over it's debugging protocol. Getting the MCPs configured for playwright was finnicky, especially across multiple agents, and I'm experimenting with this command line tool to do the same. Isolation for parallel work. The agents can start processes and run tests and so forth without conflicting on ports or files. A bit more security. Even the Economist has now picked up on the Lethal Trifecta (or Simon Willison's original) . By explicitly choosing which environment variables I forward, and not sharing my cookies and my SSH keys, I’m exerting some control over what data and capabilities are exposed to the agent. We’re still playing with fire (can you break out of Colima? Sure! Can you edit my git repo? Sure! Break into my tailnet? Sorta.), but it’s a smaller, more controlled burn.

devops DevOps shell Shell docker Docker
1 views
Karan Sharma 1 months ago

State of My Homelab 2025

For the past five years, I have maintained a homelab in various configurations. This journey has served as a practical exploration of different technologies, from Raspberry Pi clusters running K3s to a hybrid cloud setup and eventually a cloud-based Nomad setup . Each iteration provided valuable lessons, consistently highlighting the operational benefits of simplicity. This article details the current state of my homelab. A primary motivation for this build was to dip my toes into “actual” homelabbing—that is, maintaining a physical server at home. The main design goal was to build a dedicated, reliable, and performant server that is easy to maintain. This led me to move away from complex container orchestrators like Kubernetes in favor of a more straightforward Docker Compose workflow. I will cover the hardware build, software architecture, and the rationale behind the key decisions. After considerable research, I selected components to balance performance, power efficiency, and cost. The server is designed for 24/7 operation in a home environment, making noise and power consumption important considerations. My previous setups involved Kubernetes and Nomad, but the operational overhead proved unnecessary for my use case. I have since standardized on a Git-based, Docker Compose workflow that prioritizes simplicity and transparency. The core of the system is a Git repository that holds all configurations. Each service is defined as a self-contained “stack” in its own directory. The structure is organized by machine, making it easy to manage multiple environments: This modular approach allows me to manage each application’s configuration, including its and any related files, as an independent unit. Deployments are handled by a custom script, with a providing a convenient command-runner interface. The process is fundamentally simple: Each machine’s connection settings ( , , ) are defined in its file. This file can also contain and hooks for custom actions. The makes daily operations trivial: This system provides fine-grained control over deployments, with support for actions like , , , , and (which also removes persistent volumes). To keep the system consistent, I follow a few key patterns: The homelab comprises three distinct machines to provide isolation and redundancy. This distributed setup isolates my home network from the public internet and ensures that critical public services remain online even if the home server is down for maintenance. The following is a breakdown of the services, or “stacks,” running on each machine. A few key services that are central to the homelab are detailed further in the next section. I came across Technitium DNS after seeing a recommendation from @oddtazz , and it has been a revelation. For anyone who wants more than just basic ad blocking from their DNS server, it’s a game-changer. It serves as both a recursive and authoritative server, meaning I don’t need a separate tool like to resolve from root hints. The level of configuration is incredible—from DNSSEC, custom zones, and SOA records to fine-grained caching control. The UI is a bit dated, but that’s a minor point for me given the raw power it provides. It is a vastly underrated tool for any homelabber who wants to go beyond Pi-hole or AdGuard Home. For a long time, I felt that monitoring a homelab meant spinning up a full Prometheus and Grafana stack. Beszel is the perfect antidote to that complexity. It provides exactly what I need for basic node monitoring—CPU, memory, disk, and network usage—in a simple, lightweight package. It’s incredibly easy to set up and provides a clean, real-time view of my servers without the overhead of a more complex system. For a simple homelab monitoring setup, it’s hard to beat. While Beszel monitors the servers from the inside, Gatus watches them from the outside. Running on an independent Hetzner VM, its job is to ensure my services are reachable from the public internet. It validates HTTP status codes, response times, and more. This separation is crucial; if my entire home network goes down, Gatus is still online to send an alert to my phone. It’s the final piece of the puzzle for robust monitoring, ensuring I know when things are broken even if the monitoring service itself is part of the outage. Data integrity and recoverability are critical. My strategy is built on layers of redundancy and encryption. I chose BTRFS for its modern features: The two 4TB drives are mirrored in a RAID 1 array, providing redundancy against a single drive failure. The entire array is encrypted using LUKS2, with the key stored on the boot SSD for automatic mounting. This protects data at rest in case of physical theft or drive disposal. Mount options in : RAID does not protect against accidental deletion, file corruption, or catastrophic failure. My backup strategy follows the 3-2-1 rule. Daily, automated backups are managed by systemd timers running . Backups are encrypted and sent to Cloudflare R2, providing an off-site copy. R2 was chosen for its zero-cost egress, which is a significant advantage for restores. The backup script covers critical application data and the Docker Compose configurations: Each backup run reports its status to a healthchecks.io endpoint, which sends a push notification on failure. I must appreciate its generous free tier, which is more than sufficient for my needs. This homelab represents a shift in philosophy from exploring complexity to valuing simplicity and reliability. The upfront hardware investment of ~$1,200 is offset by eliminating recurring cloud hosting costs and providing complete control over my data and services. For those considering a homelab, my primary recommendation is to start with a simple, well-understood foundation. A reliable machine with a solid backup strategy is more valuable than a complex, hard-to-maintain cluster. The goal is to build a system that serves your needs, not one that you serve. CPU : The Ryzen 5 7600X provides a strong price-to-performance ratio. Its 6 cores offer ample headroom for concurrent containerized workloads and future experimentation. Storage : The boot drive is a 500GB NVMe for fast OS and application performance. The primary storage consists of two 4TB HDDs in a BTRFS RAID 1 configuration. To mitigate the risk of correlated failures, I chose drives from different manufacturers (WD and Seagate) purchased at different times. RAM : 32GB of DDR5-6000 provides sufficient memory for a growing number of services without risking contention. Case & PSU : The ASUS Prime AP201 is a compact MicroATX case with a clean aesthetic suitable for a home office. The Corsair SF750 (80+ Platinum) PSU was chosen for its efficiency and to provide capacity for a future GPU for local LLM or transcoding workloads. Sync : copies the specified stack’s directory from the local Git repository to a (e.g., ) on the target machine. Execute : runs the appropriate command on the remote machine. Data Persistence : Instead of using Docker named volumes, I use host bind mounts. All persistent data for a service is stored in a dedicated directory on the host, typically . This makes backups and data management more transparent. Reverse Proxy Network : The Caddy stack defines a shared Docker network called . Other stacks that need to be exposed to the internet are configured to join this network. This allows Caddy to discover and proxy them without exposing their ports on the host machine. I have written about this pattern in detail in a previous post . Port Exposure : Services behind the reverse proxy use the directive in their to make ports available to Caddy within the Docker network. I avoid binding ports directly with unless absolutely necessary. floyd-homelab-1 (Primary Server) : The core of the homelab, running on the AMD hardware detailed above. It runs data-intensive personal services (e.g., Immich, Paperless-ngx) and is accessible only via the Tailscale network. floyd-pub-1 (Public VPS) : A small cloud VPS that hosts public-facing services requiring high availability, such as DNS utilities, analytics, and notification relays. floyd-monitor-public (Monitoring VPS) : A small Hetzner VM running Gatus for health checks. Its independence ensures that I am alerted if the primary homelab or home network goes offline. Actual : A local-first personal finance and budgeting tool. Caddy : A powerful, enterprise-ready, open source web server with automatic HTTPS. Gitea : A Git service for personal projects. Glance : A dashboard for viewing all my feeds and data in one place. Immich : A photo and video backup solution, directly from my mobile phone. Karakeep : An app for bookmarking everything, with AI-based tagging and full-text search. Owntracks : A private location tracker for recording my own location data. Paperless-ngx : A document management system that transforms physical documents into a searchable online archive. Silverbullet : A Markdown-based knowledge management and note-taking tool. Caddy : Reverse proxy for the services on this node. Beszel-agent : The agent for the Beszel monitoring platform. Caddy : Reverse proxy for the services on this node. Cloak : A service to securely share sensitive text with others. Doggo : A command-line DNS Client for Humans, written in Golang. Ntfy : A self-hosted push notification service. prom2grafana : A tool to convert Prometheus metrics to Grafana dashboards and alert rules using AI. Umami : A simple, fast, privacy-focused alternative to Google Analytics. Checksumming : Protects against silent data corruption. Copy-on-Write : Enables instantaneous, low-cost snapshots. Transparent Compression : compression saves space without significant performance overhead.

go Go docker Docker devops DevOps hardware Hardware
0 views
Kix Panganiban 2 months ago

Cutting the cord on TV and movie subscriptions

In 2025, there's no longer a single subscription that you can pay for to watch any new movie or TV show that comes out. Netflix, Disney, HBO, and even Apple now push you to pay a separate subscription just to watch that one new show that everyone's talking about -- and I'm sick of it. Thanks to a friend of mine, I recently got intrigued by the idea of seedboxing again. In a nutshell, instead of spending $ to pay for 5 different streaming services, you pay a single fee to have someone in an area with lax torrenting laws host a VPS for you -- where you can run a torrent client and a Plex server, download content, and stream it to your devices. I tried a few seedbox services, but the pricing didn't really work for me. And since I'm in the Philippines, many of them suffer from high latency, and even raw download speeds can be spotty. So I put my work hat on and decided to try spinning up my own media server, and I chose this stack: https://github.com/Rick45/quick-arr-Stack For people just getting into home media servers like myself, this stack can essentially be run with just , with a few modifications to the env values as necessary. (For Windows users running this on WSL like me, you'll need to change all containers using networking to instead, and expose all ports one by one. Most of them only need one port, except for the Plex container, which lists them here .) Once it's up, you get: The quick arr Stack repo has a much longer and thorough explanation of each component, as well as how to configure them. Once it's all up and running -- you now have access to any TV show or movie that you want, without paying ridiculous subscription fees to all those streaming apps! Deluge -- the torrent client Plex Media Server -- this should be obvious unless you don't know what Plex is; it hosts all your downloaded content and allows you to access it via the Plex apps or through a web browser Radarr -- a tool for searching for movies, presented in a much nicer interface than manually searching for individual torrents Sonarr -- a tool for searching for TV shows, and as I understand it, a fork of Radarr Prowlarr -- converts search requests from Radarr and Sonarr into torrent downloads in Deluge Bazarr -- automatically downloads subtitles for your downloaded media Torrenting is illegal. That should be obvious. Check your local laws to make sure you're not breaking any. The stack includes an optional VPN client, which you could use if you want to be less detectable. You'll need to configure the right torrent trackers in Prowlarr. Some are great for movies, some for TV shows, and there are different ones for anime. There doesn't seem to be a single tracker that does it all. Even then, some trackers might not work. For example, l337's Cloudflare firewall is blocking Prowlarr. Not all movies and TV shows will be easy to find, so if you're looking for some obscure media, you might need to go with a Usenet tracker. This setup requires a pretty stable internet connection (with headroom for both your torrenting and your regular use), and tons of storage. Depending on how much media you're downloading, you'll probably need to delete watched series consistently or use extremely large drives. Diagnosing issues (Prowlarr can't see Sonarr! Plex isn't updating! Downloads aren't appearing in Deluge!) requires some understanding of Docker containers, Linux, and a bit of command-line work. It's certainly not impossible, but might be off-putting for beginners.

docker Docker tutorial Tutorial open-source Open Source
0 views
ryansouthgate.com 2 months ago

A secure & efficient Node/npm in Docker setup for frontend development

IntroI’ve been searching for a secure and efficient Node in Docker setup for a little while now. And I think I’ve found it! A couple of years ago, I’d install the latest LTS version of Node/npm on my Windows machine and be done with it. In my day job, I could be working on one of 4 different front end projects, and have my own front end projects I’d work on in my spare time.

docker Docker
0 views
//pauls dev blog 7 months ago

How To Secure Your Docker Environment By Using a Docker Socket Proxy

In my previous article, I talked about Docker security tips that are often overlooked while working with Docker. One critical flaw was mounting the Docker Socket directly into containers which can be a major vulnerability. I recommended using a Docker Socket Proxy that is a simple but powerful solution. In this article, we will dive deeper into this topic. We will explore the Docker Socket Proxy and learn why exposing it directly into a container is a dangerous practice. We will use it to harden our container infrastructure. Additionally, we will make use of the newly deployed Docker Socket Proxy in a containerized application (Portainer) to communicate with the Docker API in a safe way. Let’s make our containers smarter and safer. In any Docker environment, the Docker Socket ( ) refers to a special file within the Linux OS that allows direct communication with the Docker daemon . The Docker daemon is the core process that manages every container on the host system. This means that every application (or container) that has access to the Docker Socket can fully manage containers (e.g., create, stop, delete, or modify ), which results in full control over the entire Docker environment. If the Docker Socket is mounted inside a Docker container (which many containers like Portainer want for convenience), we are giving that container full control over the Docker Daemon. Now, if an attacker can gain access to our container, they can: This also works if the Docker socket is mounted as read-only because attackers can often find ways to execute commands through it. In contrast to the simple Docker Socket, the Docket Socket Proxy acts as a middleware between our containers and the Docker Socket. This results in not exposing the Docker Socket directly to all our containers. With the Docket Socket Proxy, we can: Using a Docker Socket Proxy in our Docker environment will introduce the following benefits: To set up our own Docker Socket Proxy, we will use the LinuxServer.io version of the Docket Socket Proxy to restrict the API access. To set it up, we first need to create a Compose file called : In this Compose file, we used several environment variables that permit or allow the Docker Socket Proxy to do certain tasks. If we want to allow/permit other functionalities, we can find the setting in this list and add it to our environment variables. Before starting the Docker Socket Proxy, we have to create an external network because it was specified as one in the Compose file. We can do this with the following command: Now that we have the network created, we can start the Compose file with the Docker Socket Proxy by executing: Assuming that the Docker Socket Proxy is running we can start to configure and deploy Portainer with the Docker Socket Proxy instead of directly accessing the Docker Socket. To do this, we have to create a Compose file named with the following contents: In this Compose file, there are two very important settings that differ from a normal Portainer deployment with direct access to the Docker Socket: 1. The command uses a TCP connection to the Socket proxy container: 2. The network that is used is the previously created external network To start the Portainer Docker container, we can execute: After some seconds, the command should be finished,d and Portainer is started. To access Portainer, we can switch to our browser and open: In this very short tutorial, we learned how to set up the Docker Socket Proxy container, use it within a Portainer instance to securely access the Docker API, and allow minimal exposure. This should add an extra layer of trust and security. But, why is this secure? So, in Conclusion, we can say that using a Docker Socket Proxy is a much safer alternative to the normal approach where we expose directly. Additionally, it is very easy to do, and we should take the extra step to secure our environment. Let's all make our environments more secure 😊🚀! I hope you find this tutorial helpful and can and will use the Docker Socket Proxy in your Docker environment. Also, I would love to hear your ideas and thoughts about the Docker Socket Proxy. If you can provide more information or have other important settings to set, don’t hesitate to comment here. If you have any questions, please write them down below. I try to answer them if possible. Feel free to connect with me on  Medium ,  LinkedIn ,  Twitter , BlueSky , and  GitHub . Delete and modify every container in our Docker environment Create new compromised Docker containers with run harmful software Escape from the container to take over the host Limit all actions that the container can do by granting only permissions that are necessary Allow only specific containers to use the Proxy instead of exposing the full socket to everyone Run a small, well-defined proxy container that is controlled and can be used to access the Docker API It minimizes the risk by restricting which actions can be done by applications. It prevents direct access to the Docker Daemon , which reduces attack surfaces in our environment It allows controlled and secure automation while still maintaining security. (for HTTPS) ✅ It prevents us from full Docker socket exposure. This restricts API access, so Portainer (or other containers) only gets the permissions it needs. ✅ It reduces attack surface because no direct socket access is possible and sensitive API actions like and are blocked. ✅ It works in Swarm mode because it uses an overlay network for secure communication between containers

docker Docker security Security devops DevOps
0 views
Asher Falcon 9 months ago

docker deployment

Recently I found myself wanting to deploy a docker-compose file to a server. I wanted to automate this process as much as possible, so I decided to write a shell script to do it. This post will go through the process of writing a shell script to deploy a docker-compose file to a server.

tutorial Tutorial shell Shell docker Docker devops DevOps
0 views
//pauls dev blog 9 months ago

How To Self-Host FreshRSS Reader Using Docker

Have you ever thought about self-hosting your own RSS reader? Yes, That's good, you're in the right place. In this guide, we will walk through everything you need to know to self-host your own version of FreshRss, a self-hosted, lightweight, easy-to-work-with, powerful, and customizable RSS Reader. RSS stands for Really Simple Syndication and is a tool that can help us to aggregate and read content from multiple websites in one place which is updated when new posts are published on any of the websites added as RSS feed. RSS readers can be used to stay updated and get the latest posts from blogs (also mine), news sites, or podcasts without visiting each site and without having an algorithm filtering these sites "for our interests". The news/articles are simply ordered chronologically. This can save time because all the content we want is available in one place and often they are offline usable so you can read without an active internet connection. Another important feature of an RSS reader is privacy because the RSS reader pulls the content directly from the website and this avoids tracking and adds. Unfortunately, not every website supports RSS feeds containing the complete article, but we can avoid being tracked for those who do. Nowadays, many different kinds of RSS readers exist, but most of them are hosted by different providers: As usual with external hosted services, more or the main content is blocked behind a Paywall. Because of that, I decided to host my very own RSS reader which I can use. My choice was FreshRSS because it is lightweight, easy to work with, very powerful, and can be customized to my needs. Additionally, is is a multi-user application where I can invite my colleagues/friend or share lists for anonymous reading. Another cool feature is that there is an API for clients, including mobiles! Internally, FreshRSS natively supports basic Web scraping based on XPath and JSON documents if there are websites that do not have any RSS (Atom) feed published. To follow every step within this tutorial and have a running FresshRSS service in the end we need to have a running Docker or Docker Swarm environment with a configured load balancer like Traefik. If we want to use Docker in Swarm mode to host our services we can follow this article to set it up: After using that article to set up our Docker Swarm we should install and configure a Traefik load balancer (or something similar: NGINX, Caddy, etc) which will grant our service Let's Encrypt SSL certificates and forward our services within our Docker Swarm environment. To learn about this (and other important services) we could follow this tutorial: If we don't want to host our own Docker Swarm but still want to use Docker and Traefik to host FreshRSS I have created the following tutorial which will show how to set up a Traefik Proxy that can be used to grant Let's Encrypt SSL certificate to services running in Docker containers: Before installing FreshRss we have to create a file called which will store every environment variable that is used (for our specific use case) in our Docker container. For more information about settings in the file see this GitHub reference . Now, that we have everything properly set up we can install FreshRSS on our server using either simple Deployment on one server or Docker Swarm deployment. FreshRSS will be installed with Docker Compose. The Compose file contains the service name, all mandatory/important settings for Traefik to have a unique URL and an SSL certificate. To install FreshRSS within your Docker Swarm you can paste the following code into your docker-compose.yml which will be explained afterward. Line 4: The rolling release, which has the same features as the official git     branch is used Line 5 - 8: This sets up log rotation for the container and sets 10 megabytes as a maximum for a log file using a maximum of three log files. Line 9 - 11: To persist all container data, two persistent volumes are used (data/extensions) Line 12 - 13: Set the used network to my Traefik network Line 15 - 17: The service will only be deployed to a Docker Swarm node if the label  is true. This can be achieved by executing the following command before deploying the docker-compose.yml to the stack: Line 18 - 29: Set up a standard configuration for a service deployed on Port 80 in a Docker Swarm with Traefik and Let's Encrypt certificates. In Lines 22 and 25 a URL is registered for this service: rss.paulsblog.dev Line 30: Enables using the file Line 31 - 34: Configure environment variables used by FreshRSS like the timezone (Line 32), the cronjob to update (Line 33), the production version (Line 34) Line 36 - 46: Sets the and environment variable which automatically passes arguments to the command line  . Only executed at the very first run. This means, that if you make changes, delete the FreshRSS service, the volume, and restart the service. Line 47 - 54: The used volumes and networks are defined because this has to be done in Compose files. As we will deploy this service into our Docker Swarm using a file we have to prepend the docker config command which will generate a new Compose file with all variables from the file. To have a simple command we will pipe the output the the docker stack command and deploy it. Because of this procedure, the resulting command to deploy the service in our Docker Swarm is: If you do not have a running Docker Swarm you can use this Compose file: There are only two differences between this and the Docker Swarm Compose files. The first is in  Lines 5 - 7  where a hostname ( ), container name ( ), and the restart policy is set:  . The other change is that  labels are removed from the deploy - keyword  and put to a higher order within the Compose file. This is done because deploy is only used in a Docker Swarm environment but labels can also be used in a simple Docker environment. To start the container simply use: After deploying our own instance of FreshRSS we can switch to our favorite web browser, open https://rss.paulsblog.dev (or your used domain), and log in with the credentials specified within the file (admin:freshrss123456). We should see something similar like this: The first thing we have to do is configure the FreshRss instance. To do this, we should press the cogwheel in the top right corner to open up the settings: Now we have to do two things: On the main page, you can click the small + icon next to Subscription Management to open up the following page: Before adding a feed we should create a category (e.g. Blogs). Then we can add an RSS feed for our most visited/interesting blog (-> https://www.paulsblog.dev/rss ). Pressing Add will open up the settings of the RSS feed page and we are able to configure it even further: On this page, we can add or change the description, check if the feed is working, and also set or update the category. If we Submit we can go back to the article overview and will see all articles that were fetched from the RSS feed: Clicking on an article will show us the complete article to read (if the author of the website has enabled full articles in RSS feed, otherwise there is only a small preview.) For example, on my page, the last article in the RSS feed before publishing this one could be completely read in FreshRSS: As mentioned earlier we could access our self-hosted FreshRSS service from our mobile device. For Android Users, I would recommend ReadDrops ( https://github.com/readrops/Readrops ) to access our FreshRSS. Readdrops can be found in the Google Play Store and is completely free to use: https://play.google.com/store/apps/details?id=com.readrops.app As I am not an IOS user, I do not know which app is best so you have to find one from this wonderful list in the FreshRSS GitHub . Congratulations to us, as we should now have installed our own FreshRSS blog (at least me). If you want to improve the user experience or do more with your FreshRSS instance you can: This is the end of this tutorial. Once you finish setting up your personal FreshRSS instance, share your experience with me. I would love to read it! Also, if you want to give feedback about this tutorial or have any alternative self hosted RSS readers, please comment here and explain the differences. And as always, if you have any questions, please ask them in the comments. I will answer them if possible. Feel free to connect with me on  Medium ,  LinkedIn ,  Twitter , and  GitHub . NewsBlur : NewsBlur is a "free" RSS reader that can be used on the web, iPad/iPhone, and Android and is able to follow 64 feeds. Like many others, it has a premium account that adds more "features". InnoReader : InnoReader is another "free" RSS reader that can be used to follow websites and read articles on their website. If you have an active subscription, you can also use their IOS or Android apps. With InnoRead you can follow 150 feeds. Feedly : Feedly is also a "free" RSS reader which offers a free plan that can be used on the web and IOS/Android and allows you to follow up to 100 feeds and organize them into 3 folders. Change the password for our admin. This is done in Account -> Profile Adjust the theme we want to use for our instance. This is done in Configuration -> Display Add my blog to your FreshRSS instance: https://www.paulsblog.dev/rss Add Krebs On Security Add LinuxServer.io Add BleepingComputer Invite your friends to share it

docker Docker json JSON tutorial Tutorial
0 views
//pauls dev blog 9 months ago

How To Move Docker's Data Directory To Free Up Disk Space

When working with services like Docker which stores large amounts of data in locations that often do not have enough space can be a very frustrating experience, especially, if we are running out of disk space on our Linux server. Recently, I had a case where the partition on a host using docker was running very low on disk space. This time, I deployed so many containers on the host that the storage on the root partition (~15GB) was nearly full. Luckily, I had an additional disk mounted at with plenty of disk space. So I was searching for a simple and effective solution to my problem which was moving Docker's default storage location to another directory on the disk mounted at to ensure that all docker services can operate without problems and prevent data loss. In this article, I want to explain all the steps I did to safely relocate Docker's data directory so that everyone can do this without breaking their containerized applications if this problem ever occurs. If Docker is installed its default behavior is that container images, volumes, and other runtime data (logs) will be stored in which will grow significantly over time because of: When hosting Linux servers, the root partition often has limited space because it should only contain the filesystem and configuration files. This results in having Docker using all left space very fast resulting in slowdowns, crashes, or failures if launching new Docker containers. The solution is to move the Docker's data directory to avoid these problems. Before we can move the Docker's data directory we should stop the Docker service to prevent any data loss while moving the files. Running this command will ensure that no active processes are using Docker’s current data directory while we move it. Unfortunately, this will stop all services running on our host. If we do this in production we have to plan it in a maintenance window! After the Docker service is stopped we can add (or set) a setting value in the configuration file which normally is located at . If we cannot find it there we should create it and add the following configuration: Now, to ensure that every already created container, all images, and all volumes keep working, we have to copy the contents of (the old path) to the new location using the following command: In this command, the flag preserves the permissions, symbolic links, and all corresponding metadata. The is used to not copy data from other mounted filesystems and the property is used to simply copy the directory and not create an extra subdirectory. The next step is restarting the Docker service: This command will restart the Docker service and should instruct Docker to use the new storage location which we defined in the . Finally, we should confirm that Docker is using the new directory and is working correctly. To check this, we can use the following command: This command should return the new directory path we set: Then, we should test that our containers, images, and volumes are working by simply listing them: As a last step, we can check the container by accessing them. For example, if we host a website we can simply access it with the browser. In some cases moving the Docker's data directory is not possible and we could use different techniques to free up disk space and move files. The most important command for freeing up disk space while using Docker is: Docker provides built-in commands to remove unused data and reclaim space: This built-in command from Docker is used to remove unused data and reclaim space. It will remove every stopped container, each not-used network, the dangling images, and all build caches (it won't delete the volumes!) . We should keep in mind that we use it with caution as it will delete every image that is not linked to a running Docker container! If is too dangerous we can remove the individual parts separately. Remove unused networks: Remove unused volumes: Remove dangling images: Another approach to moving the entire Docker directory, we could use Linux bind mounts to selectively move certain large parts in the Docker directory like images or volumes. To do this, we simply create a Linux bind mount using the new path as the source and the docker directory as the destination: This solution is very flexible because we are able to manage the storage without affecting the Docker runtime. Having not enough disk space in a Docker environment could lead to heavy problems as I already learned in this "incident" . By freeing up disk space regularly and moving Docker's data directory to a separate large partition it is possible to ensure a running system and our containerized applications. I hope this article gave you an easy step-by-step guide to apply this to your Docker environment if you encounter that your Docker data directory is consuming too much space. But remember, keeping an eye on your storage needs and proactively managing Docker's data, like enabling logrotate, will help you prevent these unexpected problems. As a best practice, I would recommend cleaning up unused Docker resources regularly and monitoring disk usage on a daily basis to avoid running into problems! To do this, we could use Grafana, Netdata, Checkmk (Nagios), Zabbix, MinMon, or simply a CronJob. However, I would love to hear your feedback about this tutorial. Furthermore, if you have any alternative approaches, please comment here and explain what you have done. Also, if you have any questions, please ask them in the comments. I try to answer them if possible. Feel free to connect with me on  Medium ,  LinkedIn ,  Twitter , BlueSky , and  GitHub . Log files and container data Persistent volumes containing application data (databases, files, etc) Docker images and layers

devops DevOps linux Linux docker Docker
0 views
./techtipsy 11 months ago

The IPv6 situation on Docker is good now!

Good news, everyone! Doing IPv6 networking stuff on Docker is actually good now! I’ve recently started reworking my home server setup to be more IPv6 compatible, and as part of that I learned that during the summer of 2024 Docker shipped an update that eliminated a lot of the configuration and tweaking previously necessary to support IPv6. There is no need to change the daemon configuration any longer, it just works on Docker Engine v27 and later. If your host has a working IPv6 setup and you want to listen to port 80 on both IPv4 and IPv6, then you don’t have to do anything special. However, the container will only have an IPv4 address internally. You can verify it by listing all the Docker networks via and running for the one associated with your container. For services like that log the source IP address, this is problematic, as every incoming IPv6 request will be logged with the Docker network gateway IP address, such as . If you want the container to have an IPv4 and an IPv6 address within the Docker network, you can create a new network and enable IPv6 in it. There are situations where it’s handy to have a static IP address for a container within the Docker network. If you need help coming up with an unique local IPv6 address range, you can use this tool. If you choose the host network driver, your container will operate within the same networking space as your container host. If the host handles both IPv4 and IPv6 networking, then your container will happily operate with both. However, due to reduced network isolation, this has some security implications that you must take into account. If you want your container to only accept connections on select interfaces, such as a Wireguard connection, then you will need to specify the IP addresses in the section. Here’s one example with both IPv4 and IPv6. I’ve given up on Podman. Before doing things the IPv6 way, Podman was functional for the most part, requiring a few tweaks to get things working. I have not managed to get Podman to play fair with IPv6. No matter what I did, I could not get it to listen to certain ports and access my services, the ports would always be filtered out. I’m genuinely happy to see that the IPv6 support has gotten better with Docker, and I hope that this short introduction helps those out there looking to do things the IPv6 way with containers.

docker Docker devops DevOps
0 views
//pauls dev blog 1 years ago

How To Setup Docker Registry In Kubernetes Using Traefik v2

More than a year ago I created a tutorial on How To Install A Private Docker Container Registry In Kubernetes: In this tutorial, I was using Traefik for exposing the Docker Registry which will allow to access the registry through HTTPS with a proper TLS certificate. Unfortunately, the tutorial does not work anymore because the of the Kubernetes used by any Traefik file has changed. Instead of using all IngressRoutes have to use . Because of this problem, I recreated setting up the Docker Container Registry tutorial in a simple way. For more information or explanation switch to my previous tutorial . First, we create a namespace that we use for our registry: To add the PVC we create a file and add: Then let's deploy the file to our Kubernetes using: Before deploying our Docker Container Registry we should create a secret that we use to authenticate when pushing/pulling. To simplify this step I created a script that can be used for this purpose. Create a new file and add: To generate the files execute: This will create two files ( and ) in your destination folder( ) with the needed strings. Now, to Install and Deploy the registry we will use Helm the Kubernetes Package Manager. First, we will add the Helm Repository and create a which will contain our specific data. 1. Create : 2. Add and Update Helm Repository 3. Install Docker Registry Now we can add the Traefik IngressRoute which will expose the Docker Container Registry. To do this we have to create a file called and add: This file can then be easily applied to our Kubernetes cluster by executing: The last step will be to test if the Docker Container Registry is working. To check this we can simply download any Docker image and push it to our newly set upped Container Registry. First, we pull an image from Docker Hub by executing: Then we tag the image with a custom name and add our Docker Registry domain name as prefix: Then we have to login into our Docker Container Registry: Now, we can try to push our personal NGINX image to our Private Container Registry by executing: If there are no errors, our Kubernetes Docker Container Registry is working and we can start using it. This simple update to my previously written tutorial, " How To Install A Private Docker Container Registry In Kubernetes ", should help you to deploy a private Docker Container Registry in your Kubernetes cluster with a newer version of Traefik. If you need further information on how to apply this tutorial or have any questions, please ask them in the comments. I will try to answer them if possible. Feel free to connect with me on  Medium ,  LinkedIn ,  Twitter , and  GitHub .

docker Docker devops DevOps
0 views
fasterthanli.me 2 years ago

Extra credit

We’ve achieved our goals already with this series: we have a web service written in Rust, built into a Docker image with nix, with a nice dev shell, that we can deploy to fly.io . But there’s always room for improvement, and so I wanted to talk about a few things we didn’t bother doing in the previous chapters. When we ran our app locally, we signed up for a MaxMindDB account, and had to download and unpack the “Country” Geolite2 database manually.

shell Shell rust Rust docker Docker
0 views
fasterthanli.me 2 years ago

Generating a docker image with nix

There it is. The final installment. Over the course of this series, we’ve built a very useful Rust web service that shows us colored ASCII art cats, and we’ve packaged it with docker, and deployed it to https://fly.io . We did all that without using at all, and then in the last few chapters, we’ve learned to use , and now it’s time to tell goodbye, along with this whole-ass :

rust Rust docker Docker
0 views
Karan Sharma 3 years ago

Logging on Nomad with Vector

Application orchestrators like Nomad , Kubernetes etc., allow you to deploy multiple applications on the same host. Logs are stored on the underlying node wherever the applications are run. However, it’s pretty common to treat these instances as ephemeral if they’re a part of an autoscaling group. Therefore depending on the node’s availability to search these logs is not a practical idea as the node can go down anytime. Moreover, in most cases, access to these nodes is limited to cluster administrators, not the maintainers (developers/application owners). In such cases, a log shipping agent must ship and aggregate logs from all cluster nodes and store them centrally (like Elasticsearch, Clickhouse, Loki). I’d recommend reading this excellent post by Adrian, who has explained how to set up a Vector logging pipeline for applications running with task driver and ship the logs to Loki. For the applications running using task driver, Nomad piggybacks to docker daemon for configuring logging options. Docker daemon supports many logging options; in my experience, the log driver works reliably well. However, this post is about tasks not using but any other driver (e.g. and ). Nomad doesn’t provide many configuration options for logging for these drivers. The biggest issue is that Nomad logs the application’s stream to the log directory as-is without annotating any metadata about the task. This means that if you’ve multiple applications running on one host, the log shipping agent will not be able to identify which application’s logs are being ingested. Consider this as an example. We’re running a simple web server using driver: Once the alloc is running, we can find its IP address using: On sending an HTTP request using we can see the logs that this webserver generated: Nomad stores the logs inside the applications’ allocation directory, inside . To see the logs for the above allocation ID, we can use: As you can see, these logs are precisely what the command generates. Ideally, Nomad should have enriched these logs with metadata about the allocation ID, job name, namespace, the node it’s running on, etc., as noted in this GitHub issue . However, since that’s not yet available, I brainstormed a few different options: The first approach was to run as a sidecar next to the main task. This is the simplest option, to begin with. Vector can be independently configured to add metadata for the logs collected from the allocation directory of the group. However, as with every sidecar deployment, there’s a lot of extra resource usage. For every 10 different groups, reserving resources for 10 vector agents quickly eats up your available CPU/Memory of the underlying node. A more critical downside, though, was asking every developer to also configure a Vector sidecar job. And to keep all these configs in sync to ensure they’re unified across namespaces is also another headache. Due to these reasons, I discarded this option early on. However, suppose your deployment scale (managing applications) is relatively smaller. In that case, this is actually not a bad idea. My next option was to listen to the Nomad events stream and generate a “vector” configuration template to collect logs and enrich them with metadata from the Events Stream. I developed v0.1 of nomad-vector-logger based on this concept. Since I’ve written a wrapper to collect events from Nomad using nomad-events-sink it was relatively straightforward to extend it to generate a config. However, after testing in prod for a few days, I noticed that relying on the stream is unreliable. Nomad events are not WebSocket based (as of yet). It’s a simple long polling mechanism which sends events to a Go channel as and when they happen. What happens when you miss an event? What happens when you run , which clears the events index? These were some of the challenges I faced with this v0.1 approach. There needs to be some sort of “reconcile” mechanism that periodically runs. A reconciliation loop that lists all allocations using the HTTP API can help whenever there are missing events. I also posted about the above program in Vector’s discord group (they’re super active+helpful folks) and discussed this daemon with them. They also suggested a simpler alternative: generating a CSV of running allocations instead of a config. Vector has support for Enrichment Tables which means that it can “lookup” a CSV file to find a particular row and enrich the log event with the information found from the CSV. This seemed a super cool idea, and I developed v0.2 using this. Super thankful to Vector maintainers for giving me this idea! However, this approach had a few “subtle” drawbacks that I found: The final v0.3.0 solution, which IMHO fixed all the above issues, was: Now that we’ve covered a few different approaches and the pros/cons of each let’s see how works. Essentially is meant to be deployed inside a Nomad cluster as a job. A system job in Nomad runs on each node. Whenever a new node gets added to the cluster, Nomad’s scheduler schedules a copy of this program on that new node automatically. This is the equivalent of a “Daemonset” in K8s. uses Nomad’s HTTP API to query all nodes’ running allocations. Once it gets the list, it adds it to an internal map and signals to generate a config. The final config that is templated out looks like this: For people unfamiliar with vector, it’s essentially doing 2 things: Vector pipeline will send this event to another “transformer” which can further process the log event (for eg parsing it as or JSON etc) and then finally send it to an upstream sink like Loki/Elasticsearch etc. Here’s an example of the before/after of a log line shown above in this post: Perfect! We’ve annotated the same log event with Nomad metadata, and Vector will be able to identify these logs. If you’re interested in a complete setup on deploying this to Nomad, take a look at dev setup which contains a Nomad jobspec to deploy as a sidecar with as the main task. Hope this post helped you start configuring a logging pipeline for applications running with non-docker task drivers. doesn’t support live-reloading if the CSV file changes. has support for watching a config file for changes or sending a to reload. However, that only works for vector’s own config files. Since the CSV file is an external file, vector cannot watch it for changes. I came up with an ugly bash script hack and compared the hash of the file in a loop and if it changed, then send a to vector. All I can say is it’s ugly, but it works . If you wish to see it, it’s available here in all it’s glory. The most significant issue was the chance of losing logs for the initial 10-20s of a new allocation. The above shell script had a because can be a bit CPU intensive to keep frequently doing. Vector sees a new allocation and starts ingesting events. It tries to look up the CSV row by the allocation ID, but it doesn’t find it yet in the CSV file, complains about it, and drops the log event. Thus, I had to drop the CSV idea in search to find another more reliable approach to this. For people interested in this approach, you can checkout the branch here . Skip Nomad events stream. Since I have to build a reconciliation loop anyway, listening to events is just extra work without tangible benefits. I used a background Goroutine channel to periodically refresh the list of allocations running on that node. Even if I fetched this list once every 30s or so, it’s OK because Vector will start ingesting logs once the config gets generated. It will start reading the file from the beginning . So logs aren’t lost even if I templated the config much later after the alloc began running. I added the support to delay the removal of allocation from the file. If an allocation is stopped (e.g., a new version is deployed or the job restarted), the program doesn’t immediately removes the allocation from the config file. The user can set a delay period which works like a cooling down period. In this period, one can assume that Vector would have finished sending all logs to the upstream. In case the application generates too many logs faster than what the upstream sink can accept (e.g. if the upstream Elasticsearch gets slower). Suppose we remove the allocation _ immediately whenever it stops. In that case, there’s a probability that Vector wouldn’t have read the file to the end . This cooling period helps to ensure that doesn’t happen. This is not a fool-proof situation but should cover most cases unless the upstream sink is dead for many hours. Get logs from a “file” source . The file path comes from (where all the logs for task are located) It adds a JSON object with relevant keys.

Bash devops DevOps docker Docker go Go json JSON
0 views
Fredrik Meyer 4 years ago

How to connect to localhost from a Docker container

At work I recently spent a lot of time trying to get a running Docker container to connect to a service running on . The setup is this: we have a Docker container that runs inside a Docker bridge network together with other Docker containers it can communicate with. On the host machine there is a service which listens on . For the purposes of this blog post, let’s just assume this service’s job is to stream messages to some server. I wanted one of the containers to connect to the service running on . In this post I will explain how I got it to work (only on Linux). The picture is something like this: Docker containers inside the same bridge network can easily communicate with each other by using their container ID as the host name. However, Docker Container A cannot easily talk to something on on the host machine. There is even someone on StackOverflow saying that this is not possible! Without network: host you can’t go back from a container to the host. Only host to container. This is the main ideology behind containers. They are isolated for both stability and security reasons. This is a truth with modifications. The way Docker manages its networking is by suitably modifying . Which means that we also can do the same! We are going to use to route traffic to the bridge gateway inside a container to traffic on on the host machine (the next steps are thanks to this very helpful StackOverflow answer). The IP of the bridge gateway can be found by running . To not have to worry about the actual IP, you can start the container with . This will map the host name to the IP of the bridge gateway. First we have to find the name of the Docker bridge network. This is not the same as the ID you see when you do , but almost. If you use the default bridge network, then the name should be . If you made your own network, the name will be (see the original code ). You can also get the name of the bridge by running . You can set the bridge name yourself when you create the network with the following command ( source ): (just remember that it can be at most 15 characters long) We need to allow routing from the bridge to localhost: Note that this has some mild security implications . The next step is to use to forward traffic to the bridge gateway to localhost. Once this is done, you can connect to the service on on the host machine by calling . Happy coding!

docker Docker tutorial Tutorial devops DevOps
0 views
Can ELMA 4 years ago

Using ngrok in Docker

You can run multiple services as a single app with . If you are using ngrok in your development environment, you may also need to make it a Docker service. Should you? Probably not. But if you want to, you will see how to do it in this article. First of all, create a folder named . You should then create a symbolic link in that folder that references to the actual ngrok executable. For instance, In the ngrok folder, you now have a symbolic link to the ngrok executable. Next, you should create a configuration file for ngrok. In the same ngrok folder, create a file like the one below: Then define your ngrok service in the file: Did you notice the section in the above file? You can find more about it in this article . You can access it on . You may also need to access the ngrok service from other services. Moreover, you may want to get the current ngrok address dynamically. Below is a sample Python code I wrote for this: This will return the public URL of the running ngrok instance according to the HTTP protocol you specified.

devops DevOps docker Docker tutorial Tutorial python Python
0 views
Can ELMA 4 years ago

Docker profiles: Scenario based services

Sometimes you need some services only in certain scenarios. Docker Compose has a handy feature to achieve this; profiles. Here's a sample file below for the examples in the following sections. There are profiles assigned to two of the services in the above example. The service is assigned to the profile, while the service is assigned to the profile . Unlike other services, and services do not have profiles. So they always start. But the and services will only be started when you activate their profiles. The below command would start the and services by default. The following command would also start the service along with the and services. That's because we enabled the profile: The below command would enable multiple profiles, and , at once and start the and services besides the and services. You can also use the environment variable to specify which profiles to enable:

docker Docker devops DevOps
0 views
ptrchm 4 years ago

Docker on macOS Without Performance Problems

The app I’m currently working on runs on Docker. It’s a medium-sized Rails monolith with a bunch of resource-heavy dependencies (Postgres, PostGIS, ElasticSearch, Redis, Next.js and a few more). Docker helps us make sure the whole stack works for everyone, every time. For those using Linux, it works without noticeable downsides. But on macOS, despite every possible performance tweak, Docker had been a huge pain. MacBook with Docker will always run hot, battery will drain in less than an hour, fan speed is high enough for the laptop to take off and I need an external SSD disk to fit all the images.

javascript JavaScript performance Performance docker Docker ruby Ruby
0 views