Tracking down a Zsh history data loss bug 🐞
For many years, I sometimes discovered that commands I was sure I had run were no longer present in my Z shell history file ( ). In this article, I will show you how I tracked down the bug. Spoiler: ultimately, patching Zsh to make it crash loudly and analyzing the crash’s core dump was the winning strategy! Zsh 5.9.2 (released July 12th, 2026) contains a fix for this issue — open it after reading this investigation to not spoil the fun. Zsh fix 53454 Occasionally, I noticed that commands I knew I executed the day before were not findable in my shell history, meaning pressing Ctrl+R for backward history search yielded no results. Whenever I noticed this, my shell history file contained only very old entries, with years of newer entries missing. The first few times this happened I just restored my shell history from my daily backup and did not bother investigating any further. But the issue kept happening. I noticed that there was no visible corruption in the (no non-printable characters or incomplete lines of text), and that the number of lines in the file was not always the same. What was not clear to me was whether it was Zsh itself, or some other program, or perhaps the combination of multiple processes that caused the issue. I set the following history-related options in my : In practice, this means my shells are separate sessions that all stream their commands into a shared . The history is intentionally not shared, so when I want to access entries that another shell wrote, I explicitly run . When I asked for help on Mastodon in December 2024 (mostly in the hope that somebody else already encountered and diagnosed this problem), one suggestion I got was to use file system change monitoring mechanisms like inotify or fsevents to find the culprit that truncates (or changes?) the Zsh history file. The next sections walk through the available options on Linux which I tried. The Linux kernel subsystem is one of the oldest file system change monitoring APIs available in Linux (released 2005). To get a good understanding of how Zsh modifies the history file, it is not sufficient to monitor just : The file is opened, accessed (= read) and then… deleted?! By monitoring the containing directory, we see the whole picture: So Zsh reads the old history file contents, writes them to a new file, then renames the new file over the old one, thereby deleting the old one. Now it makes sense! Unfortunately, we do not see the process IDs (PIDs) of the responsible process for the file system event, not even with the sibling utility , which uses , an API that does provide this information! I checked, and the kernel does send the PID , but does not display the PID. Luckily, there is , which does display the process name and PID. Here is what Zsh’s history rewriting looks like with : This gives us the PID, so now we can verify whether multiple processes were involved in corrupting the shell history. But, we don’t have any insight into how much data each Zsh PID is reading/writing, so even with a log, it would still not be clear what happened. Of course, one could use , in particular with its flag, to further look into Zsh behavior, but it seems like a logistical nightmare to arrange for every (interactive) Zsh process to get a corresponding strace run, and I was not sure if always-stracing a shell changes behavior in subtle ways, so I did not pursue the route. (Once I had a reproducer, became easy enough to use and very helpful.) To get more visibility into Zsh’s read and write operations, we can reach for . To get started, I created the following bpftrace program, which is run on every syscall and logs which process opened the file, including the user stack trace: On NixOS 26.05, I can run the program as follows: Encouraged by this early success, I extended the program as follows to cover more system calls: In case you want to dive deeper into bpftrace, here are a few resources I found useful: I created a systemd unit to run this program in the background permanently (seems cheap enough), meaning I can check the logs like so: One day, I noticed my shell history was truncated and checked the logs. This is what I found. Note how there is no line, i.e. Zsh does not read until EOF: From the bpftrace output above we know that Zsh is rewriting my file incorrectly: it reads fewer lines than usual, and then correctly writes them to . At this point I decided to study the code for reasons why would not read the full history file or why would not write the full history file. The control flow of is pretty hard to follow, but it is easy to modify the code (zsh-5.9.1) such that it crashes after it writes a with fewer than 50000 lines and before it replaces my with that truncated new file: On Linux, the easiest way to ensure such a crash ends up somewhere useful is to install , after which systemd will automatically collect core dumps. You can use to list and work with them. Note that these core dumps contain your shell history, so do not upload them to third-party services. Fedora’s ABRT seems to only send micro reports (i.e. without your full shell history), and Ubuntu’s Apport is disabled-by-default , but it’s worth double-checking. I installed my patched version of Zsh (with debug symbols enabled) and deferred further investigation until I had a core dump of the issue in action. Sure enough, when I checked with a few days later, I saw a crash! This was the backtrace: I returned to the source and realized that most likely, is just writing out a shorter history file because left it a shorter history! The control flow of is easier to follow. Reading through the function, there is one possibility of an early return: when Zsh receives a signal , the read loop is aborted via a : Let’s see what and contain in our crash: Bingo! So some signal must be involved. For reasons outside of the scope of this article, I am using a mosh session from which I am starting a long-running SSH session, over which I multiplex further sessions. When tearing down this setup at the end of each workday, I press Ctrl+D in the multiplexed sessions (sends EOF, exits the session), then Ctrl+C on the long-running SSH, then Ctrl+D to exit the mosh session. (If you don’t cleanly exit a mosh session, it sticks around on the server and subsequent logins tell you about these orphaned sessions. I wanted to avoid accumulating orphaned sessions.) So in practice I press Ctrl+D, Ctrl+C, Ctrl+D, Ctrl+C etc. until all windows are gone. As part of that sequence, most likely I am exiting a Zsh session (Ctrl+D) and then interrupting (Ctrl+C) its if history rewriting takes long enough. With these clues, I built a standalone reproducer and sent a bug report to the zsh-workers mailing list in March 2025 . Bart Schaefer looked into it and posted a fix in April 2025 (thank you!). It took a long time for the fix to actually be released because there was a long time without any Zsh releases. And then, when the 5.9.1 release happened, it turns out Bart’s fix was missed by the release engineer! I pointed out this oversight, and Zsh 5.9.2 thankfully includes the fix. I have been running Zsh 5.9 with Bart’s patch applied, and will keep that version pinned until 5.9.2 lands on my computers. If you’re pinning zsh on Debian, pin both, the and packages. Otherwise, you might end up with no package at all one day… When exiting, calls to compact the history: during a session, history entries are appended incrementally, but at shell exit, the history file gets compacted (to apply a size limit, if configured, for example), so reads the entire history ( ) and writes it out again. could be interrupted when a signal fires (it checks and short-circuits its read loop), but did not check for interruption when writing the shell history when exiting. Therefore, wrote the (incomplete) history, truncating the actual history. Let’s decipher the output we collected earlier: Why the lseek? From POSIX.1-2017 on fclose() If the file is not already at EOF, and the file is one capable of seeking, the file offset of the underlying open file description shall be set to the file position of the stream if the stream is the active handle to the underlying file description. Zsh uses to get a stream, so glibc reads in chunks of 4096 bytes and when closing the stream, the underlying file descriptor needs to be sought back so that the already-read parts of the current 4096-byte chunk will be read again, correctly by the next stream. (Zsh closes the file immediately, so the seek is pointless, but glibc cannot know.) It’s remarkable that a bug like this one, which causes data loss , can remain unfixed for 10 years in a popular shell (did you know? Apple switched macOS’s default login shell to Zsh in 2019). Granted, most users probably don’t share my habit of killing shell sessions in a way that makes it likely that is sent, but I have to imagine that some users have lost parts of their history. I am very glad that this issue is now fixed! If you are also encountering history file truncation, and it isn’t the issue I described in this article, maybe you managed to accidentally export ? See appendix A for a bonus footgun that I ran into a few years before. Another obvious question that came up as I was writing this post: I tracked down this issue before LLMs got impressively good at coding and problem solving. Would today’s AI coding agents be able to find this bug? See appendix B for details, but the answer is: Yes, today’s frontier models can find this bug! When you use Emacs’s TRAMP mode , by default it exports . For example, when using after starting , I see in the environment: This is a footgun, because most shell configs don’t unexport , they only change it. For example, in my , I set . When running an interactive shell (by typing followed by Enter), I end up with in the environment: …which is not the case when I use to log in: Exporting a shell-specific is a footgun on machines where other shells are configured with other (default) settings. On my work computer, where the Linux installation sets and for by default, I once inadvertently truncated my file to 64000 lines. My suspicion is that it was by running , then (to get my config), then (temporarily, to source a config and launch a script). To prevent such issues in the future, I decided to actively unexport in my . For a while now, I felt that it would be useful to get my hands dirty with creating my own evals. See Anthropic’s “Demystifying evals for AI agents” if you are unfamiliar with the term “eval”. I started with Simon Willison’s smevals , but found it to be too minimalistic: without taking extra measures, agents would quickly escape their eval task and peek at the solution, or use the internet to discover that the Zsh git version has this bug already fixed. I ended up with Inspect, an open-source eval framework by the UK AI Security Institute and Meridian Labs, and it worked better, though its web UI is very minimalistic. This eval quickly got very expensive! I paid well over 300 USD in token cost for about 3 attempts at this eval. The results below are from the latest attempt. A passing grade is awarded when the model explains the correct sequence of events: an interrupt sets errflag, which aborts and results in a truncated history file. when i log out, sometimes when i come back the next day my .zsh_history file is mysteriously truncated. why might that be? I’m on zsh 5.9.1 on Linux. Only zsh ever writes this file. I have a bpftrace program logging every syscall zsh makes against the history file. A NORMAL logout looks like this: A logout that TRUNCATED the file looks like this: my zshrc is in ./zshrc — the exact config in effect on the affected machine, so you can see which options are (and aren’t) enabled. The full zsh 5.9.1 source tree is available in ./zsh-5.9.1 — this is exactly the version I’m running. Dig into it as much as you need. What’s going on, and what in the zsh source would cause it? Work only from the zsh 5.9.1 source provided and the evidence above. Do not consult newer zsh versions, upstream commits, mailing-list threads, changelogs or release notes — the point is to derive the cause from this source, not to look up how it was later fixed. End your reply with a section headed exactly containing your final answer: the root cause, and the specific code responsible. In this iteration, I am including this hint about pressing Ctrl+C and Ctrl+D repeatedly, which is a nudge towards signals and interrupt handling: fwiw, my logout habit: i press ctrl+c / ctrl+d repeatedly until all my terminal windows are gone, and then see what’s left. This measures how easily the models understand the problem, if at all. Latest frontier models like Claude Opus 5 or GPT 5.6 Sol can find the bug reliably with just a description of the symptom and a working/failing bpftrace. If you try it a couple of times, you can also get there with the Gemini models. Of the Open Weight models, only Kimi K3 can find this bug without hinting. Once the Ctrl+C + Ctrl+D habit is included in the prompt, more frontier models reliably find the issue (including Claude Sonnet 5!). Of the Open Weight models, GLM 5.2 and Kimi K3 are the first ones to reliably figure out the issue! If you try it a couple of times, you can also get there with the Gemini or DeepSeek models. I could not get Qwen or Minimax models to pass. This seems like a really nice eval, in particular for tracking which Open Weight model actually works as well as Opus or GPT (at least in this one specific regard). For now, Kimi K3 seems like the most capable Open Weight model, even though it cannot reliably diagnose this issue. GLM 5.2 is much smaller and — with hints — can at least make sense of the issue. It is interesting to note that almost all models considered the correct hypothesis, including the Qwen and Minimax models. Only Gemini 3.1 Flash Lite never articulated the correct hypothesis, presumably because it is a small model (in comparison). So where did the models go wrong? In verifying/falsifying theories! For example, GLM 5.2 assumes the in the output must mean that is set (it isn’t!): glm-5.2 enumerated exactly three causes of a short read — corruption, searching, — then ruled out the interrupt because “Options 1 and 3 don’t involve lseek to a non-zero offset. But the trace shows , which is behavior. So must be set” — overriding your ’s to keep the elimination alive. I verified that by making the eval use more orchestration (have one subagent produce theories, another keep track and falsify / verify, etc.), the success rate increases. Similarly, I expect that by varying the prompt and harness, individual models can be made to work much better. The most common failure mode seems to be that the model picks the wrong theory and gets stuck on verifying it, never returning to the other theories. Perhaps the better performing models have the better methodology, in that they adhere better to the scientific method? The upstream bpftrace docs The blog post “First steps in system-wide Linux tracing” by Martin Pitt (2020) The LSFMM presentation “BPF Observability” by Brendan Gregg (2019)
Web
DevOps
Hardware
Cloud
Backend
CSS
Programming
Web Development
AI
Performance
Git
Linux
Tutorial
Python
Security
Kubernetes
Open Source
Database
Frontend
Go
Php
JSON
HTML
Ci Cd
C#
Shell