Hardening Rust Code For Production
We talked about patterns for defensive programming in Rust before, in which implicit invariants that aren’t enforced by the compiler lead to utter misery. But being careful isn’t enough! Even valid code can fail at runtime in ways that are hard to predict and control. That’s what we’re covering next. This article is for you if you want to… What happens when a Rust program panics? There is no single correct answer because is not a “single behavior.” For starters, there’s a difference between unwind and abort. invokes a closure, which captures the cause of an unwinding panic. But the Rustonomicon has the following to say about unwinding panics: We would encourage you to only do this sparingly . In particular, Rust’s current unwinding implementation is heavily optimized for the “doesn’t unwind” case. If a program doesn’t unwind, there should be no runtime cost for the program being ready to unwind. The alternative to unwinding is aborting the entire process. That does what it says on the tin: the program immediately terminates without unwinding the stack or running destructors. Halt and catch fire. Weirdly enough, that’s often the safer choice, especially when dealing with FFI boundaries or performance-critical code. That’s because unwinding across FFI boundaries is undefined behavior, and unwinding can be expensive in performance-sensitive code. To enable aborting on panic, add the following to your : And even if you did not explicitly configure this, catastrophic panics like stack overflows and out-of-memory errors always abort the process . That’s because unwinding in these situations is unsafe and can lead to undefined behavior. In practice, this shows up in two places: These failures are fundamentally different from ordinary panics in that they cannot be caught or recovered from. To handle them gracefully, you need to know exactly how and where your program will run, and design accordingly. For example, in the case of , avoid unbounded user input that could lead to excessive allocations. Another difference is between thread-level failures and process-level crashes. A common misunderstanding is that terminates the entire program, but in a multi-threaded application, that is not necessarily the case. For example, a background worker thread can panic while the main thread continues running. What sounds like a benefit can leave the system in a partially degraded state. This distinction becomes especially important in long-running systems (servers, workers, async runtimes, …). A panic in a request-handling thread might only abort that one request, while the rest of the service remains available. Here’s a small example using scoped threads ( Playground ): The interesting part of the output is this: Request 2 panics, but requests 1 and 3 still finish. The panic belongs to the worker thread. The main thread gets notified on but keeps running. 1 Whether this is acceptable depends on the system’s invariants. If a panic indicates a violated assumption confined to a small scope, like a single request, letting the process continue may be reasonable. But if it signals a global invariant violation, continuing execution can be outright dangerous. Panic behavior is part of your system’s failure model . Treating all panics as equivalent hides important distinctions and leads to fragile assumptions. Be explicit about whether a failure may take down a single task, a single thread, or the entire process. Never panic in an uncontrolled manner. If you maintain a library, you have less control over where your code runs and what a panic can take down. Consider enabling stricter Clippy lints such as and to catch common panic sources before they become part of your public API. Those lints can be noisy in applications, but they are often useful when panic freedom matters more than convenience. Now that you understand how panics work, let’s talk about operational hardening. When things go wrong, you want to know about it. But by default, Rust panics just print to and disappear into the void. In production systems, that’s not so great. You might prefer crash reporting or centralized failure handling, and that’s where panic hooks come in. A panic hook is a function that gets called whenever a panic occurs, giving you a chance to record the failure before the program terminates or unwinds. It will not make an invalid state safe again. Its job is to capture enough context to debug the failure, alert someone, and shut down cleanly when possible. Here’s a simple example of setting a panic hook: And here’s a panic hook that sends structured JSON data to a crash reporting service: What’s Inside ? The struct contains the panic message (via ) and the source location where the panic occurred (via ). Be aware that both can leak sensitive information: file paths may reveal internal directory structure, and panic messages might contain interpolated user data. And finally, here’s Sentry’s panic hook handler , which is even more sophisticated: Sentry’s panic hook: There’s a lot to learn from these few lines of code! Panic hooks are also your final opportunity to prevent information leaks. The sensitive data can come from two places: the panic payload and the panic location. The payload is whatever your code passed to , , , or an assertion. That means it can contain interpolated user input, internal state from output, request headers, tokens, email addresses, IP addresses, customer IDs, or other identifiers. The location can expose source file paths, workspace names, or CI/build machine directory layouts. A well-designed panic hook sanitizes these messages before they reach logs or crash reports. Better yet, avoid putting secrets or raw user data into panic messages in the first place. Prefer stable error codes, request IDs, or redacted domain types. Regexes can catch obvious patterns like email addresses and bearer tokens. UUIDs and IP addresses can also identify users. Treat those checks as your final fallback. You can look into crates like expunge or veil to automatically redact sensitive information from structs: Before the process terminates, you might want to flush logs, close network connections, or notify other systems that this instance is going down. Setting a hook is a great way to perform such cleanup operations. Panic Hooks Run in a Compromised Environment Be careful: one of the subsystems you want to interact with might be the cause of the panic you’re handling! For example, if your database connection pool panicked, trying to flush pending writes to that same pool will likely fail or hang. Keep cleanup operations fault-tolerant and avoid anything that can panic, block indefinitely, or depend on the subsystem that just failed. Panic hooks only run for unwinding panics. If your program aborts on panic, or if the panic is caused by a stack overflow or out-of-memory condition, your hook won’t execute. Never rely on panic hooks for correctness. They’re purely for observability and graceful degradation; don’t try to recover from logic errors as it is very hard to rely on a system’s fragile underpinnings at this stage. Okay, you handle errors gracefully and you know how your system behaves on panic. Panic behavior isn’t the only runtime failure mode you need to worry about. Here’s some simple recursive code. What is wrong with it? The problem is that recursion can quickly exhaust stack space. If you allow users to call this function with large inputs, it might crash your program. Rust does not guarantee tail-call optimization on stable Rust . Some compilers and languages can turn certain tail-recursive functions into loops, but you should not rely on that transformation in Rust. If recursion depth depends on user input or external data, rewrite the algorithm iteratively or put an explicit bound on the depth. It takes some experience, but for recursive algorithms where you’re not in control of the input size, it’s often safer to use an iterative approach: One of the most dangerous assumptions in Rust development is that debug and release builds are functionally equivalent. They’re not. In many ways, you’re shipping a different program than the one you tested. The most obvious difference is integer overflow behavior. Debug builds panic on overflow, while release builds silently wrap around. We covered that in Pitfalls of Safe Rust . But the differences run deeper than arithmetic. Release builds remove checks, enable optimizations, and may exercise different code paths behind . Unsafe code and FFI boundaries are especially sensitive to this: undefined behavior can appear harmless in debug mode and break only once the optimizer starts relying on Rust’s aliasing and validity rules. Here is a trivial example: In a debug build, trips the . In a release build, the assertion is gone. The subtraction can underflow and wrap around, turning an invalid discount into a huge number. If the check protects a real runtime invariant, use or return a instead of relying on . The fact that tests pass in debug mode does not prove that production behavior is correct. Run normal debug tests as the fast default, and add release-mode tests for critical integration tests, arithmetic-heavy code, unsafe or FFI-heavy code, and anything whose behavior depends on optimization or release-only configuration. Your code is only as safe as your dependencies. You should regularly audit your dependencies for known vulnerabilities. Two helpful tools for that are and . It’s recommended to run those as part of CI. mimalloc is a drop-in global allocator built by Microsoft. What’s special about it is that it also has a secure mode , which adds mitigations like guard pages, randomized allocation, and encrypted free lists to make some heap-corruption bugs harder to exploit. 2 Safe Rust already prevents most use-after-free and buffer-overflow bugs, and a secure allocator does not magically make memory-unsafe code safe. This is mostly defense-in-depth for programs with unsafe code, custom allocators, C/C++ dependencies, or FFI-heavy boundaries. To enable secure mode, put this in : Then use it as your global allocator: Now, all heap allocations in your Rust program will use mimalloc’s secure allocator. Measure the performance impact on your workload before rolling this out broadly; allocator choice can matter a lot for latency-sensitive services, games, packet processing, and other allocation-heavy programs. Even well-written Rust code can be compromised through its dependencies, environment, or C FFI boundaries. The idea is to reduce your blast radius. Now, how you do that depends on your deployment environment, but generally people use Docker and Linux, so I thought I’d share some techniques for those; specifically, how to build minimal container images and filesystem sandboxing. A minimal production image contains exactly what you put in it. Even if your service is compromised, the attacker has very limited tools at their disposal to do further damage. My recommendation is Google’s distroless images , but please do your own research 3 as I’m not an expert on this. Distroless images are minimal Debian-based images stripped of everything unnecessary, while still including TLS certificates and a non-root user. For a typical Rust web service, start with : it includes the C runtime libraries that a normal Debian-built Rust binary may dynamically link against, but no shell or package manager. (Check the latest version in the distroless README .) Here is an example Dockerfile using for dependency caching: Take this Dockerfile as a starting point, but please adapt it to your own project requirements. keeps dependency builds in a separate Docker layer, so changing your application code does not force all dependencies to rebuild. The important details are: use the same Rust version in all build stages, build with , scope workspace builds with when appropriate, and keep , , and editor files out of the build context via . For a deep dive on Docker images and build-time optimization, see Tips For Faster CI Builds . Keep the Debian suffix explicit instead of using the unversioned tag, and pin by digest if reproducible deploys matter to you. If you deliberately build a fully static musl binary, then or even can be a better fit. But don’t mix the two approaches: a glibc-linked binary needs a runtime image that provides the libraries it links against. A Note On Alpine Base Images Alpine base images are a well-known alternative, but they use musl instead of glibc. That can expose differences in DNS resolution, TLS/native dependencies, allocator behavior, and crates that assume a glibc-like environment. ( 1 2 3 ) That doesn’t mean Alpine or musl are wrong; just treat them as a deliberate target and test them like one. If you build on Debian and want a small runtime image, distroless is usually the less surprising default. Even inside a minimal container, your process still has access to any file the container mounts. Landlock is a Linux security module that lets a process restrict its own filesystem access. If your service is ever exploited, the attacker can only reach the files you explicitly allowed. 4 Landlock Is Deployment-Specific Landlock is Linux-only and requires kernel support. It landed in Linux 5.13, but older enterprise kernels, custom cloud images, or container hosts may not enable it. Check your actual deployment target. Also apply the sandbox only after you know which files your process needs. If your service executes helper binaries from , reads timezone data from , loads certificates, opens SQLite files, reads config from , or writes uploads to , those paths must be allowed explicitly. On non-Linux targets, look for equivalent sandboxing mechanisms instead of copying this exact snippet. Call as early as possible in , before spawning threads or accepting connections. The restrictions apply to the entire process from that point forward. The two approaches really go hand in hand: Don’t run as root in production, even inside a container. That’s one reason distroless images provide a user and why the example above uses the tag. If your service only needs to listen for HTTP traffic, prefer a high port like over running as root just to bind to port . Linux capabilities are another useful lever. Instead of giving a process full root privileges, grant only the specific capability it needs, such as for binding to low ports. If a process needs elevated privileges only during startup, drop them before accepting requests. The details vary by platform and orchestrator, so treat Linux containers as one concrete setup. For systemd services, Kubernetes, FreeBSD jails, macOS sandboxing, or Windows services, look up the equivalent least-privilege and sandboxing features for that environment. The big picture is that security hardening is about reducing the surface of things that can go wrong. Every capability your process holds unnecessarily is a liability and everything your code manages that could be delegated to the OS, init system, or container runtime probably should be. Miri is an interpreter for Rust’s mid-level intermediate representation (MIR) that can detect undefined behavior at runtime. It works by executing your Rust code in a special environment that tracks memory accesses, pointer validity, and other low-level details to catch issues that the compiler can’t statically guarantee against. More people should know about Miri, because it is really helpful for hard-to-detect race conditions in multi-threaded or async code; but it can do way more than that, of course. It has already detected a lot of real-world bugs , even in the standard library. Using it is as simple as running: This will run your tests under Miri’s interpreter. The docs also describe how to add miri to CI : (Make sure to check the latest instructions in the Miri repo, as the setup process may change over time.) If you’d like to learn more about Miri, there is a research paper from 2026 that goes into the design and implementation details: Miri: Practical Undefined Behavior Detection for Rust . A hardened service doesn’t just crash. Instead, it shuts down gracefully when asked. Aim to finish in-flight requests, flush your buffers, and release resources cleanly before you exit. The pattern is: listen for shutdown signals, stop accepting new work, drain existing work, then exit. Frameworks like Axum have built-in support for graceful shutdown . Use it! The key is handling signals like (sent by Kubernetes, systemd, or ) and (Ctrl+C). Here’s a minimal example using tokio-graceful-shutdown , which is a crate that provides good signal handling without much boilerplate. It introduces a concept of “subsystems” that can run concurrently and listen for shutdown requests. When an external service (database, API, cache) starts failing, you don’t want to keep hammering it with requests. A circuit breaker tracks failures and “trips” when a threshold is reached. For production use, consider crates like or the more actively maintained , which is based on failsafe. Unbounded resources are a common source of runtime failures. Everybody who was on call for a production service will tell you this. Set explicit limits on everything . SREs will thank you for it! Limits make your service more predictable, and they make misconfigurations obvious sooner. Common things you should limit include: Here are some examples of how to do this in practice: See Axum’s : Bound the number of items in every queue or channel in your system. Every unbounded resource is a potential DoS vector. Explicit limits turn those catastrophic failures into (annoying but harmless) graceful rejections. Ideally, your system should be able to recover from transient failures without human intervention. Health checks let load balancers and orchestrators know when something is wrong, so they can react. A typical setup has two endpoints, a liveness probe and a readiness probe. The liveness probe checks if the process is alive at all, while the readiness probe checks if the process is healthy enough to handle traffic. This could honestly be an entire article on its own, but here’s a quick example using Axum to illustrate the concept: What’s neat about it is that this maps directly to Kubernetes’ health check system: Do we really need both probes? Yes, because they serve different purposes: Finally, here are some more tools that help you catch problems before they hit production: The tools above help catch undefined behavior, memory safety issues, code coverage gaps, and performance bottlenecks. They are dynamic analysis tools that complement Rust’s static guarantees. This only holds for unwinding panics. If you compile with , or hit a stack overflow or out-of-memory failure, the whole process exits and never gets a chance to return . ↩ https://docs.rs/mimalloc-safe/latest/mimalloc_safe/ ↩ Data sources I found useful for this topic include this post and this comparison . ↩ This approach would have prevented a vulnerability in Meta’s crate , a tool for recording and displaying system data like hardware utilization and cgroup information on Linux. ↩ make your code resilient at runtime harden your Rust code for production know how Rust code can fail in unexpected ways and how to recover from that Panic Semantics Are Part of Your API Unwind vs. Abort Thread-Level vs. Process-Level Failures Observing Failures With Panic Hooks Example Panic Hooks Sanitizing Sensitive Data Cleanup Operations Limitations Stack Overflows And Runtime Behavior Release and Debug Builds Are Two Different Programs Testing Release Behavior Supply-Chain Security Secure Allocations With mimalloc Limit Your Runtime Attack Surface Minimal Docker Images Filesystem Sandboxing With Landlock Drop Privileges and Capabilities Miri: Detect Unsafe Code Issues Graceful Shutdown Handling Circuit Breakers for External Dependencies Resource Limits Request Body Size Limits Limit Queue Depth Set Timeouts on Everything External Health Checks and Self-Healing Runtime Hardening Tooling Panics that would unwind across an extern “C” boundary are defined to abort instead of unwinding, because letting unwinding cross that boundary is undefined behavior . And if a fails, it aborts the process . If that’s a problem, you need to proactively check for allocation sizes before allocating or avoid heap allocations altogether. Logs the panic information Preserves the previous panic hook behavior by calling Ensures the hook is only set once using minimal images limit what’s in the container Landlock limits what the process can touch at runtime. Upper bound on any user input (upload file size, parameter bounds, etc.) request body size timeouts on external calls concurrent connections to external services queue depth for background jobs thread count and DB connection pool size Kubernetes stops sending traffic (graceful degradation) if the readiness probe fails. It does not yet kill the pod. Kubernetes restarts your pod if the liveness probe fails (it’s self-healing!) – fuzz testing for Rust code – another fuzzer with Rust support – detects usage of unsafe code – runs Valgrind on Rust code to find memory errors – code coverage via rustc/LLVM source-based instrumentation ( ). It reports line and region coverage, works with and , and is a good default for new projects. – an older Rust coverage tool with strong Cargo and CI ergonomics. On Linux it defaults to a backend ( only); LLVM coverage is available through and is the default on macOS and Windows. Useful if its reports fit your workflow, but expect different platform and test-runner edge cases than . This only holds for unwinding panics. If you compile with , or hit a stack overflow or out-of-memory failure, the whole process exits and never gets a chance to return . ↩ https://docs.rs/mimalloc-safe/latest/mimalloc_safe/ ↩ Data sources I found useful for this topic include this post and this comparison . ↩ This approach would have prevented a vulnerability in Meta’s crate , a tool for recording and displaying system data like hardware utilization and cgroup information on Linux. ↩
Performance
Programming
JSON
Ci Cd
UI
Dart
Open Source
JavaScript
Hardware
AI
C++
CSS
Business
Security
Data
Linux
DevOps
Python
Shell
C#
API
Go
C
Rust