Posts in Database (20 found)
Krebs on Security 2 weeks ago

U.S. Soldier Gets 70 Months in Prison for AT&T, Verizon Extortions

A U.S. Army soldier who pleaded guilty to hacking into multiple telecommunications companies and stealing mobile call and text metadata for more than 100 million AT&T customers in 2024 was sentenced to 70 months in federal prison today and ordered to pay nearly $300,000 in restitution to victims. One of several selfies from the Facebook page of Cameron Wagenius. Cameron John Wagenius , 22, was stationed at a U.S. Army base in South Korea when he adopted the cybercriminal persona “ Kiberphant0m .” Working with three alleged co-conspirators, Kiberphant0m downloaded data from several large customers of the cloud data storage service Snowflake that had exposed credentials and did not enforce multi-factor authentication (Snowflake has since mandated MFA on all accounts). In October 2024, Kiberphant0m bragged on the cybercrime forums that he’d stolen the call and text metadata (e.g. source and destination number, timestamp, duration, etc.) for tens of millions of AT&T customers. Kiberphant0m claimed to have hacked into more than dozen telecommunications companies worldwide, including Verizon’s Push-to-Talk business, and publicly extorted these companies in exchange for a promise not to publish the stolen data. In late November 2025, KrebsOnSecurity warned that Kiberphant0m was likely a U.S. soldier stationed in South Korea . Less than a month later, Wagenius was arrested and charged in two separate federal indictments, and soon pleaded guilty to all counts in both cases. At his sentencing hearing in Seattle today, Wagenius was sentenced to nearly six years in federal prison, and ordered to pay $294,978 in restitution. Federal prosecutors said Wagenius was assisted in his efforts to extort victim companies by Kenneth Schuchman , a 28-year old man from Vancouver, Washington who has a lengthy cybercriminal history. In 2019, Schuchman pleaded guilty to operating the Satori botnet , a vast collection of hacked Internet-of-Things (IoT) devices that was used for large-scale distributed denial-of-service (DDoS) attacks. Two other alleged co-conspirators of Wagenius are still facing charges in connection with the Snowflake data thefts; Conor Riley Moucka , a.k.a. “Judische,” of Kitchener, Ontario was arrested in 2024 and pleaded guilty in August 2026 ; and John Erin Binns , an American man currently living in Turkey who is also wanted for a 2021 data breach at T-Mobile that exposed the personal information of at least 76 million customers. Kiberphant0m also admitted to re-extorting victims, and threatening to disclose national security secrets. Immediately following Moucka’s arrest — after AT&T had already paid the extortion group a $370,000 Bitcoin ransom — Kiberphant0m posted on hacker forums what he claimed were the AT&T call logs for then President-elect Donald Trump and for then Vice President Kamala Harris, as well as schematics allegedly stolen from the U.S. National Security Agency (NSA). Paul Russell is a resident agent in charge at the Defense Criminal Investigative Service (DCIS), the criminal investigative arm of the U.S. Department of Defense Office of Inspector General. Russell said when DCIS received information that a soldier with secret clearance was allegedly involved in cybercrime and extortion, the agency began working the investigation alongside the FBI, the Army Criminal Investigative Division (CID), and the U.S. Secret Service. “We don’t often get leads where there’s an active duty soldier with a secret clearance who’s creating hacking tools and trafficking in data,” Russell said. “That doesn’t happen every day, and so when that hits it really spins all of our partner organizations up. It was very serious from jump street, just because it was unique, it was an insider threat, and we weren’t sure what we were dealing with.” A sentencing memo (PDF) filed Sept. 19 by federal prosecutors in Seattle notes that while Wagenius pleaded guilty almost immediately and has been remarkably cooperative, he recently got caught trying to find security vulnerabilities in the BOP’s computer network. The government’s memo notes that while incarcerated and awaiting sentencing, Wagenius violated the computer use policies of the Bureau of Prisons (BOP) in attempts to learn about vulnerabilities in BOP computer systems. “According to records from BOP, in or around September 2025, Wagenius used another inmate’s email system to request that the email recipient prompt a commercial AI tool to provide information about “[w]hat CVE’s are there for Windows 10 Enterprise privilege escalation and bypasses” and to “[p]rovide the CVE’s and a real world working script for each CVE . . . without omitted code,” the government’s memo states. The memo states that less than a week later, Wagenius used a different inmate’s email account and requested that the email recipient prompt an AI tool to “[p]rovide the step by step for CVE-2023-45208 , code for this if any, and if no code exists make some, make sure to describe everything in detail.” CVE-2023-45208 is a three-year-old “command injection” vulnerability in D-Link networking devices. That same month, Wagenius allegedly again requested that the email recipient prompt AI with the question, “How do you make an antenna in a prison environment with commissary or readily available items/tools to improve/make an antenna to extend radio reception?” Federal prosecutors said Wagenius also requested that the recipient research escaping prison. “In several instances, Wagenius framed the AI queries as being posed in connection to a book he was writing. This is a common method of ‘prompt injection,’ in which attackers feed specially crafted, deceptive inputs into commercial AI tools that are programmed to avoid outputting malicious code that can be used to exploit computer vulnerabilities,” the sentencing memo reads. The government told the court it is unaware of evidence that Wagenius figured out how to use or deploy the vulnerabilities he was researching in the BOP’s systems, and when questioned said he was only researching “potential vulnerabilities to provide information to the BOP.” Incredibly, despite the enormous financial value of the data stolen from AT&T and other telecom providers, Wagenius’s extortion efforts were largely unsuccessful. The government’s sentencing memo says Wagenius made a whopping total of around $1,500 from selling stolen data. “While Wagenius was not particularly financially successful as a cybercriminal, he both intended to and caused significant harm to numerous individual victims, U.S. companies, and the U.S. government,” the memo states.

database Database cloud Cloud security Security
0 views
Andrew Atkinson 2 weeks ago

Let's Get Down to Business with Rails and PostgreSQL

I’m happy to share that I was a guest on the Rails Business podcast and the episode is now live. Here’s a recap of the topics we discussed. We spent most of the time discussing engineering work last year related to scaling the database workload for peak traffic at Aura where we use Ruby on Rails and PostgreSQL. I’m presenting this information in two places this fall, Rails at Scale Summit in Austin, TX, and the Postgres Summit in NYC. Besides the podcast, you can get more details in these posts: If you’re not able to make those events or prefer to listen to a discussion of the content, then the podcast will have a lot of the same information. Besides the work at Aura, we also checked in on my book High Performance PostgreSQL for Rails , which has now been available in print for two years. Ryan asked about what’s next for the book, and it’s a difficult question to answer. With the availability and quality of AI tooling for software engineers, I don’t see developers buying books as much. With fewer prospective readers, the ROI is less attractive for authors and publishers to commit to the resource-intensive process of creating a book. I’m grateful to have had the opportunity to write a book before the rise of the AI tools. I’m still getting positive feedback from my book which is very fulfilling. Recently I met Madison Sites who’s a reader, and they shared feedback that they’re reading the book as part of the WNB.rb book club . Madison said the book has been helpful for building vocabulary and familiarity with database concepts and implementation details, and that’s helped with database work and better comprehension for conference talks about similar topics. I appreciated hearing about that! We segued into how we’re all using AI tools like Claude in our jobs as software developers, whether it’s writing or reviewing code, doing architectural planning, or myriad other tasks. One of my favorite use cases has been generating infrastructure as code with tools like Terraform. One of the main challenges I’m experiencing is on the review side, what’s expected when it’s AI-generated code in terms of my own comprehension, and how to still provide a quality review given the ambiguity around how much comprehension is needed from me. I also find it challenging to navigate team communication dynamics with AI-generated comments. One concept that’s resonated with me is: don’t be a meat proxy , make sure to do self-reviews and add some value in the chain. I hope you can take a listen to the episode, and please let us know if you have feedback. Thank you! Listen to the Episode 👉 Andrew Atkinson on Rails Business Podcast From Christmas Outage to #1 App Store Ranking: An Aura Frames Postgres Scaling Retrospective Scaling Rails at Aura Frames: Splitting to 8 Primaries and Reaching #1 in the App Store

performance Performance ruby Ruby backend Backend database Database
0 views
Unsung 1 months ago

“Sweet, a whole website of video game menus”

The Game UI Database is a website that covers the interfaces of almost 2,000 via over 75,000 screenshots. = 2x) and (width >= 700px)" srcset="https://unsung.aresluna.org/_media/sweet-a-whole-website-of-video-game-menus/1.2096w.avif" type="image/avif"> = 3x) or (width >= 700px)" srcset="https://unsung.aresluna.org/_media/sweet-a-whole-website-of-video-game-menus/1.1600w.avif" type="image/avif"> It has meticulous information architecture, so you can jump into specific sections, for example: Even if games and productivity software seem worlds apart – the creator of the site, Edd Coates, calls himself “UI artist,” which is not a title I’ve ever seen in my line of work – I’ve long thought games do some things better, and these can be an inspiration. The database was started by Coates a few years ago. This launch article on Mashable has this interesting passage: The site is relatively short on context, with only the collections of screenshots gathered under different tags for visitors to go on. […] That’s the site working as intended. As Coates explained: “It’s useful for designers to identify recurring and pre-established patterns in successful titles when building their own interfaces.” Added context isn’t necessary because the contrasting approaches evident in each image is the whole point. Needless to say – I mean, you know this after spending 12 seconds on this blog – I disagree. But it seems the author is too, since he’s working on a book called The Game UI Bible , to be released in the first half of 2027: = 2x) and (width >= 700px)" srcset="https://unsung.aresluna.org/_media/sweet-a-whole-website-of-video-game-menus/2.2096w.avif" type="image/avif"> = 3x) or (width >= 700px)" srcset="https://unsung.aresluna.org/_media/sweet-a-whole-website-of-video-game-menus/2.1600w.avif" type="image/avif"> = 2x) and (width >= 700px)" srcset="https://unsung.aresluna.org/_media/sweet-a-whole-website-of-video-game-menus/3.2096w.avif" type="image/avif"> = 3x) or (width >= 700px)" srcset="https://unsung.aresluna.org/_media/sweet-a-whole-website-of-video-game-menus/3.1600w.avif" type="image/avif"> = 2x) and (width >= 700px)" srcset="https://unsung.aresluna.org/_media/sweet-a-whole-website-of-video-game-menus/4.2096w.avif" type="image/avif"> = 3x) or (width >= 700px)" srcset="https://unsung.aresluna.org/_media/sweet-a-whole-website-of-video-game-menus/4.1600w.avif" type="image/avif"> I recognize the volume is price’y and last time I was excited about a book , it turned out to be quite a disaster . The Game UI Bible book also touts a lot of interviews with UI artists, and I find these to be a mixed bag in practice; it’s really hard to interview people in a way that yields something interesting. But still, I am excited again – and if you’re not, the Game UI Database is worth checking out. (The title of the post comes from a different blog post about the database .) Examples of game difficulty settings Examples of game UI settings menus Modals and pop-ups Loading screens

database Database
0 views
All Things Distributed 1 months ago

DuckDB and the changing physics of analytics

In this post, Andy Warfield explains how databases like DuckDB are enabling a new way to build with data, why they matter right now, and how they complement the work we’ve been doing in S3 (e.g., S3 Files, S3 Tables, S3 Vectors). And most importantly, why DuckLabs, the team behind DuckDB, is joining AWS

ai AI database Database
0 views
Andrew Atkinson 1 months ago

PostgreSQL 18: 23x Faster Inserts With UUID v7

We recently switched to version 7 (v7) uuid primary keys and saw significantly faster inserts for some tables. The databases were running Postgres 18.4 and mostly used v1 with some v4 uuid values for primary keys. Changing the column default involved running a single alter table command, but did require an exclusive lock on the table, blocking everything including selects. To solve that, we used a short lock timeout and lots of retries. The biggest speedup was 23x faster average execution time for a multi-row insert query called 12,000 times per minute on a table with billions of rows. This September and October I'll be in Austin, TX and NYC, check my Book page for upcoming appearances. The system uses UUID primary keys throughout. I typically recommend starting with bigint and sequences over UUID v4 primary keys , although here uuid v1 was used. Insert performance is not as bad for v1 compared with v4. Still though, v7 brings better performance than both for inserts and can also result in smaller indexes with fewer page splits meaning less CPU and IO. What drives bad performance for v4 and to a lesser extent v1? Let’s do a quick refresher. As new table rows are inserted and a primary key is defined, primary key values are maintained in sorted order in a b-tree index. Just like table rows, index entries in Postgres are stored in fixed size 8kb pages. Postgres needs to know in which page to place the new index entry. For sorted order, the first bytes of new uuid values are compared. For v4 given new values are very random and not monotonically increasing (they lack “monotonicity”), values can be earlier or later, meaning they’re unlikely to be placed into the same recently accessed page. This is bad for caching! When new values are monotonically increasing, the recently accessed page is “hot” in the Postgres buffer cache (in memory copy of the on-disk page). When Postgres is not able to use the hot index page for the newly inserted value, that page could be outside the buffer cache, not in the OS cache, and ultimately result in a much slower disk read which increases latency. Besides the worse insert performance, since v4 values are scattered to more pages, this means there are more “page splits” when new inserts are attempted in full pages. Page splits cause more latency from increase WAL and IO. We experimented and benchmarked with v1, v4, and v7 uuid formats and we leveraged the research and write-ups from external sources like the ones below. Benchmarks are great, but what kind of real world results did we see? We decided to make this the new default unless v4 was needed for more randomness. After all qualified tables were changed, I began going through insert queries for each changed table. For many of the tables, there wasn’t an obvious change. However, for a handful we saw an immediate and significant improvement. I picked 5 with speedups of 6x, 8x, 9x, 20x, and 23x. The PgAnalyze graphs for the 23x, 9x, and 6x queries are shown below. Showing PgAnalyze insert query graphs for tables A, B, C: Table A - 23x reduction. 0.7ms to 0.03ms, 12000 calls/min Table B - 9x reduction. 0.6ms to 0.07ms, 2000 calls/min Table C - 6x reduction. 0.50ms to 0.08ms, 9500 calls/min Now that we’ve seen the results, let’s talk about how this was done and the challenges. The UUID values came from various sources: We replaced most of these with the function in Postgres 18. To do that, we needed to run a single statement per table. The statement ran fast, so no problem, right? One wrinkle we found was that modifying the column default while fast, required an lock. This lock type conflicts with every read and write operation including regular statements. For our highest queried tables, they’re queried constantly, so this was a problem. There was almost never a “window” to perform this operation, and we didn’t want to take downtime for this switch. While heavily queried tables were a challenge, infrequently queried tables did not pose a problem for this alter table statement at all. For those, we could use our migrations framework (Active Record in Ruby on Rails) and perform the using a regular old migration. For those, we did add some safeguards, by creating an explicit transaction and using to control timeout values. We’d set short timeouts for the to give up quickly if it didn’t work or ran too long. For the higher activity tables, we’d need some retries. We’d use a manual psql session: The would commit if it grabbed the lock within 50ms, or we’d get an error that the was reached. The benefit of the manual approach was we could retry until successful and backfill a Rails migration to keep everything in sync. A more sophisticated solution might have automated retries within Ruby. However, for our most heavily queried tables, we wanted even more control over the retries. How did we do that? Sometimes one or two retries would do the job. Great, we’d move on. However, for our most heavily queried table that didn’t work. What ended up working was using the same strategy of retries, but just adding more sophistication with looping and backoffs. Claude helped me cook up the PL/pgSQL looping retry function below, I did some testing and was ready to try it. It has these features: By using the function above, we were able to find a small window to perform the after several dozen quick retries! In cases where even many retries won’t work, and we don’t want downtime, we may be left with needing to actively monitor lock holder queries and to cancel them (assuming that’s ok). Thanks to Ants Aasma from the community PostgreSQL Slack for this idea. We didn’t end up needing to do this, but here was my prep for this. It’s still useful to review lock holder queries. First we’d inspect live queries: And identify queries holding locks: If we find them, we could cancel them to create a window to run our . That could mean bad user experience so you’d need to figure that out for your database. We’d likely want to stack up our operation to occur immediately after. Fortunately we didn’t end up needing to do this, but I’d be interested to hear the stories from others with heavily queried databases. Since uuid v7 values use a timestamp in their first bits, this timestamp can be easily decoded. This can be viewed as “leaking” or exposing the creation time of the record via that timestamp, which could be a downside for your database. You’ll have to decide that. v4 UUIDs do not expose the creation time. We found some significant speedups for insert queries after switching to primary keys, for a relatively low effort change. A nice ROI. The only wrinkle was the exclusive lock required, but we solved that with short lock related timeouts and many retries. Although this didn’t benefit 100% of our tables, the gains for some were significant and uuid v7 has become our new default choice for uuid primary keys. Thanks to the Postgres core team for creating this new capability within Postgres. The availability in core made it possible to adopt on AWS RDS which supports a limited amount of extensions. Thanks for reading, and until next time. How Sequential UUIDv7 Boosts Ingestion Performance Simplicity and power of UUID v7 PostgreSQL UUID Performance: Benchmarking Random (v4) and Time-based (v7) UUIDs The function from the module The function added in Postgres 13 that generates v4 UUIDs natively UUID v4 values sent by a client application, which meant the column default function was not used Try up to 50 times (max attempts is configurable) Add a pause in between retries, with a jittered backoff of 50-250ms

ruby Ruby database Database PostgreSQL performance Performance
0 views
Farid Zakaria 1 months ago

Your executable is a SQLite database

I have been probably obsessed with two things in the last few years: Nix as a tool to explore innovative ideas that require the capability to rebuild the world and replacing ELF with SQLite as an executable format. You might have noticed that these two ideas are well suited to each other. I explored the idea during my PhD thesis but found feedback from others unmotivating. Radical ideas are hard to sell, as you are working against the inertia of the established solution. One of the end results of that exploration was sqlelf , a tool that lets you explore an ELF file declaratively using SQL. 1 instead of fiddling with and . It was remarkably simple by leveraging virtual tables over the ELF: however I found it to be a refreshing improvement to explore the ELF file format. I knew however that there is still something much bigger to be done. I never let the idea go and with the recent improvements with LLMs, I find it compelling to revisit these ideas to explore further. Specifically, can we replace ELF with SQLite as an executable format? 🤔 Not “a database that describes an executable”, but the actual file you and run. I developed a pretty fleshed out prototype. It is called SELF , the Structured Executable & Linkable Format , because I am unoriginal. It is on GitHub if you are interested. I’m surprised about all the interesting things that fall out of this idea. Working through my PhD, I realized something that bugged me. ELF is already a database. It just implements many database primitives by hand, along with a surprising number of data structures for performance, like a bloom filter for symbol lookup. If you ever have to analyze or parse ELF, the kernel, , binutils, LIEF, goblin, , you are re-implementing the same parser over and over again. Every producer re-implements the same serializer. The format itself is incredibly terse, designed for a world where disk space and network bandwidth was at an extreme premium. Modifying the format is hard, you often have to zero out sections and add new ones since it is packed so tightly. There is also no self-describing schema. ELF itself is a very generic format that supports sections of data that by convention are interpreted in specific ways but the format does not enforce it. SQLite is the counter-example. They are a self-describing format that is extremely stable. It is designed to be extended to support new features without breaking existing consumers and supporting a wide range of queries performantly. If we were to replace ELF with SQLite, what would fall out and can all of the necessary information be represented in a SQLite database? The answer is yes, and it is surprisingly simple. A SELF file needs two tables to run: is the ELF header as key/value pairs and is the load image, one row per program header with the bytes in a : A single table for the symbol table replaces many of the ELF sections and the index. It is a single table with a single index: Our capability to include an index is equivalent to and in ELF, but it is a proper b-tree index maintained by SQLite instead of a hand-rolled bloom filter. 2 Surprisingly a lot more falls out as well: is gone, because is and SQLite already interns strings, symbol versioning is a column, not the / contraption and there is no need for a table. Other tables exist as well for metadata which exist for tooling: , , . Delete them and the program still runs, which means is a transaction: All the tools that operate on ELF files for reading, reduce to queries over the database. Any tool that modifies an ELF file, like , can operate on the database within a transaction rather than performing fragile offset surgery: is a and . is an . Any information missing from the schema can be easily exposed via a view. For example, is a query over the table, which is a join of the table with the table to find the sonames of the libraries needed by the program. SQLite reserves a 4-byte at byte offset 68 of its header, for exactly this purpose. We stamp it , so an ordinary SQLite database never matches: We can now leverage binfmt_misc , the subsystem that allows you to invoke any binary as if it were native. We need only to register the magic to trigger on and an interpreter that will invoke our new file format. On NixOS the registration is a few lines matching the SQLite magic at offset 0 and at 68: For now, I have a small tool that converts an ELF file into a SELF file. It is a simple hook you can opt into per package on NixOS. The tool reads the ELF, extracts the program headers and symbol table, and writes them into the SQLite database. We could look at extending or to emit SELF directly, but for now this is a simple way to explore the idea. is the interpreter. It is a small C program linked against . Its implementation is remarkably similar to that of but it fetches the program headers and symbol table from the database instead of reading them from the ELF file. It maps the loadable segments into memory, relocates them, and jumps to the entry point. Note has to stay an ELF file. An interpreter that also matches the registration recurses straight into . Running a static program was quick and easy but boring and unimaginative. The interesting part is dynamic linking, which is where the database shines. I explored two different ways to do dynamic linking. The first is to keep and just replace the lookup with a SQL query via rtld-audit interface, to quickly iterate on the design. The second is to replace entirely with a new dynamic linker that does the entire lookup and binding in SQL. glibc’s rtld-audit interface lets an audit library intercept every shared object lookup ( ) before any filesystem search happens, included. The audit library can then answer the question “which library satisfies this symbol?” with a SQL query instead of walking the and . Stock maps and relocates it, so the full gamut of glibc features work: lazy PLT, IFUNCs, TLS and symbol versioning, while library storage are rows and library lookups are queries. I was curious what a fully SQL dynamic linker would look like, so I prototyped one. It is called and it is a small C program that implements the dynamic linker entirely in SQL. It is a proof-of-concept, but it works. It maps every object’s segments, publishes their exports, and for each relocation patches the GOT and jumps to the start. The two things that often matter when replacing a well-established format are size and latency. How much bigger is a SELF file than an ELF file, and how much slower is it to run? Size. A SELF file carries SQLite’s b-tree overhead and lands at roughly double the ELF. Similar to ELF binaries, most of that is recoverable, because the overhead is mostly the optional tables for debugging and tooling. Stripping them and deleting them is a transaction. A stripped SELF is 1,794,048 B against the ELF’s 1,768,632 B, that is within 1% . We will see though that there are interesting ways to amortise the overhead even more which I found very unique and interesting. Latency. I benchmarked various binaries from a 15 KiB to a 42 MiB linking 47 libraries: There is a fixed ~5 ms to open SQLite and start the interpreter, plus a copy proportional to the image. That copy is worse than it looks, because the b-tree pages are not mapped into memory. Two processes running the same SELF binary do not share text pages the way a normally- ‘d ELF does, because the bytes are copied out of the b-tree rather than mapped. 3 A SQLite database though need not merely be a single executable. It can be a closure , a single file that contains a program and all of its transitive dependencies. The output of a program is ambiguous: it only lists the sonames of the libraries it needs, not the specific files that satisfy those needs. Nix improves upon this by explicitly resolving every edge to a specific store path via the use of . 4 We can do the same in SELF by storing the resolved path of each edge in the database: packs a binary and its transitive dependencies into one database with those edges filled in. Shared library resolution stops being a guess and becomes a foreign key and becomes a 🤯: This single database is a closure of the executable and its five libraries: six objects, segment bytes and all, in one 4.8 MiB file. There is no soname ambiguity inside a closure, because a closure by construction contains exactly one provider per edge. I hope you’ve been with me so far, because this is where it gets really interesting. We can go even further and pack multiple closures into a single database. I pointed at every ELF binary on this system’s : 723 executables, which pull in 400 distinct shared libraries. 1,123 objects, 346,386 symbols, 3,808 dependency edges, all as one SQLite file . Turns out when you do that, the database is much smaller than you would expect. 611.9 MiB of database against 644.4 MiB of ELF files. The whole userland, as one queryable file, is smaller than the files it came from. The b-tree cost that doubled a single amortises to nearly nothing across 1,123 objects and is roughly 6% over the actual program bytes. The libraries and closure are shared across the executables very similar to how Nix might share them across multiple closures, if the store-path was the same. If every root shipped its own private closure (i.e. the AppImage model), the same 723 programs would come to 5.53 GiB but the deduplication of libraries and symbols falls out naturally from the database schema. Many common idioms we use in ELF immediately fall out of the database. For example, is a row in a table rather than an environment variable. The table is a list of objects to map last, so their exports win. This means that turning on and off is a transaction. We were able to accomplish an atomic across a whole userland in one file, “interpose a tracing everywhere, then ” is a single transaction. 😈 The format is done and round-trips between ELF and SELF losslessly. The tooling is done and can query, modify, and pack closures. Lookup through SQL works on unmodified glibc programs perfectly and the native-SQL loader works enough to explore it as a possibility for ideas. The whole thing is at fzakaria/selfdb . boots a NixOS VM where is a SQLite database. 🙌 Nix lets us explore radical ideas like this. We can rebuild the world down to the Linux kernel if needed. We need not be constrained by the existing decisions and constraints of the past. We can explore new ideas and see what falls out. I hope you find this idea as interesting as I do. I wrote a paper, arXiv:2405.03883 , that I failed to get published and a follow-up post on querying with it .  ↩ is a bloom filter plus bucket chains, laid out so can reject a miss without touching the chain during symbol discovery.  ↩ You might notice that (274 KiB, 27 libraries) starts slower than ELF (4.6 MiB, 5 libraries). That is doing work proportional to the number of objects rather than the number of bytes, which I have complained about before .  ↩ I have written about on Nix before such as making it redundant or speeding it up .  ↩ I wrote a paper, arXiv:2405.03883 , that I failed to get published and a follow-up post on querying with it .  ↩ is a bloom filter plus bucket chains, laid out so can reject a miss without touching the chain during symbol discovery.  ↩ You might notice that (274 KiB, 27 libraries) starts slower than ELF (4.6 MiB, 5 libraries). That is doing work proportional to the number of objects rather than the number of bytes, which I have complained about before .  ↩ I have written about on Nix before such as making it redundant or speeding it up .  ↩

sql Sql database Database programming Programming SQLite
0 views
Massively Parallel Procrastination 1 months ago

I vibe-coded a C compiler that can build SQLite

A while back, Claude Opus built a pretty ambitious compiler. I didn't set out to do anything nearly as ambitious as Anthropic's. They were targeting multiple CPU architectures and they wanted to be able to compile a bootable Linux kernel. (I'd actually forgotten about that project until I started posting some screenshots of the work below to social media and someone reminded me.) Last night, as I was getting ready for bed I was reaching for some project to support and tool use in Evener, our ~new agentic harness. I popped open the mobile UI on my phone and typed "Your job is to implement a standards-compliant ARM64 C compiler for macOS in Swift." Evener asked me a couple of questions. I clarified my intent a little bit: "I'm great with radical task decomposition. You should use recursive subagents to manage context and complexity.Should structure the project in whatever sane way you want. You should work fully autonomously And do not need to ask me questions." And then I set a goal: "Implement a standards-compliant C compiler in modern Swift. It should be able to build SQLIte and have SQLite pass all tests. You may decompose the project in any way you see fit. You should use subagents, including recursive subagents, to execute effectively." I don't know what I thought was going to happen. I watched long enough to make sure that it was going to start working. At about four in the morning I rolled over and looked at my phone and it was deep into debugging a malloc issue. I woke up at a little bit before seven and it was implementing variadic functions. Today I spent most of my day in meetings and so I didn't get a lot of time to watch it, but I would occasionally pull up a session and see it making little bits of progress. Occasionally it would dump out some assembler and then dump out 's version of the assembler to hunt for differences. I saw it trying to build the 274k amalgamation and segfaulting. I saw it writing little test scripts. I saw it get to the point of trying to do an and exploding. When I came up for air at about 8 30 p.m. I popped open Evener around my phone, and was incredibly disappointed. My loop had just...stopped. It's not supposed to do that. But that's exactly the failure that I was testing for. I started reading the recent parts of the log, to figure out which obvious problem had caused the failure. I was not expecting to see this: It stopped because it had compiled SQLite and was able to do an and a . It did exactly what I asked. It took Evener + GLM 5.2 about 21 hours to build a C compiler capable of compiling SQLite and passing a basic smoke test. This is the checkpoint: https://github.com/obra/toy-c-compiler/commit/456ddfa8912b5ba47b968773ecddcdca741f0df2 (I was initially worried that it had cheated by doing the obvious web searches, but discovered late in the day that I had accidentally broken the harness's web fetch tool. And so it wasn't able to cheat that way.) Checking now, it looks like it didn't even try , which makes me happy. There are a ton of missing features. It is not yet standards compliant, but I didn't properly set the goal to force that. I've just kicked off another to get it to run through compliance suites and fill in the missing features. I'll probably let it run for another day or two just to see what happens.

programming Programming c C database Database swift Swift
0 views
Phil Eaton 2 months ago

The road to ACID transactions in Cassandra 6

This is an external post of mine. Click here if you are not redirected.

database Database open-source Open Source
0 views
Tara's Website 2 months ago

SeaweedFS, FreeBSD and a rather deep rabbit hole

SeaweedFS, FreeBSD and a rather deep rabbit hole At this point, reading my blog, you could be forgiven for thinking that I have become entirely about IBM midrange systems, COBOL and offline-first computing. That is still true, in a way. I genuinely enjoy those things. But I am still very much attracted to large, complex architectures, distributed systems, and the sort of infrastructure where several apparently innocent design decisions eventually turn into a whiteboard covered in arrows.

database Database devops DevOps
0 views
iDiallo 2 months ago

Where Did the Productivity Gains Go?

There is a machine called Productivity, and on this machine there is a knob and a button. The knob increases productivity, and the button is labeled "reset." My job was data entry at a non-profit. It was my first job in the US where I didn't have to lift heavy objects. My main task was entering collected donations into a database. The data source was email attachments, which came in various formats. Excel files, Word files, and some directly embedded in the email body. The database itself was FileMaker Pro. Every part of the process was wasteful. To receive the emails in the first place, I had to call the person who collected the donation, argue with them, and guide them through the process of emailing me the information. The donations were collected at events, and the data was often just handwritten on a note. Once I got the email, I would extract the content from the attachment and organize it into an Excel file. Then I would manually enter it into the FileMaker database, one row at a time. In a full workday, I could process no more than a dozen entries. So I decided to improve the process. I started from the end. Entering data into the database was error-prone. I had to enter the name, address, donation amount, and a plethora of other information into a single row of a database viewer, and I often put the wrong data in the wrong column. So instead, I created a form with validation. I built a standardized Excel file where the people collecting donations could enter data directly into the correct columns and send it to me. This turned out to be too much to ask. Several people didn't know how to use Excel, and it became a nightmare of technical support. Eventually, I switched to a formatted Word document. Annoying, but it worked. Lastly, I created a process. Every Monday, I would email everyone to remind them to send me their donations by Wednesday, so I would have plenty of time to enter them on schedule. The deadline was arbitrary, but it slowly hardened into policy. My system was a success. I was entering hundreds of donations every day. In terms of productivity, I had more than 10x'd my output, which I thought was a good thing, until I started getting noticed. What have you done for me lately? My productivity machine's knob had been dialed to 11. But I had also set the new standard for how much needed to get done in a day. So my manager hit the reset button. The dial went back to zero, but the expectation stayed exactly where it was. It's funny how that always happens. Increased output never turns into more free time. Instead, it becomes the new normal. Anything less looks like a drop in productivity. These days, with AI common in the workplace, every semi-technical manager has started building apps themselves instead of taking requests to the dev team. They get a rush from how much code they can produce, and they enjoy that high for a while. But they fail to see that the productivity gain only exists at the very beginning, before the app is actually in use. Once people start using it, you don't get to regenerate the app from scratch every time you want to add a feature. Instead, you have to add things slowly and carefully, without breaking what already works. All fields start green, but they brown eventually. When you get an extremely productive teammate, you start looking at the rest of the team as time-wasters. In fact, once you get used to what that teammate produces, you start treating it as the baseline and eventually you ask, "What have you done for me lately?" I was let go from that job in a little celebration, right after I'd trained my replacement. One morning, my manager simply announced that it had only ever been a summer job, and summer was ending. So my last day was set. I was too green to point out that this had never come up during hiring, and that I'd actually started well before summer began. They brought cake on my last day. What they failed to see was that my process was still very much manual. My replacement was let go when she couldn't manage more than a dozen entries a day. So much for productivity gains.

data Data database Database business Business
0 views
Xe Iaso 2 months ago

Extending immutability: deletion without losing data

Tigris has a pretty advanced replication scheme for writes. What happens when you actually need to delete things? Turns out deleting things is hard in distributed systems. Especially when you have a geo-replicated active-active database like Tigris does. We can (and do) use tombstones to mark where data once was, but how do you let people undo an accidental delete? Tigris wants to turn storage inside out , so our implementation of soft deletion is by giving users the Recycle Bin for objects and buckets. Today we're going to dig into how this works, why it works, and what this gives you in terms of using object storage today. In Windows and macOS, the Recycle Bin (or Trash can) is a form of purgatory where deleted files wait for their storage to be deallocated by the user. This allows users to hit "delete" fearlessly because if they made a mistake they can just drag it back out and go on with life. This works great in your local filesystem because there's only one writer in one region. This kinda falls apart when you have multiple regions in your database and any one of them could be writing to it. How do you name things in the recycle bin? How do you handle the conflict of an update happening in one region before the deletion was fully replicated out from another region? This is the fun of distributed systems, which is the kind of problem space that Tigris lives in. One way to think about how the Recycle Bin works is that the file metadata gets moved there when the user hits delete. No data bytes move around on the disk, but the file doesn't show up in My Documents anymore. In a distributed systems context you can't just move the metadata around, you have to leave a tombstone behind to record where that metadata once was. This prevents other regions from being confused when actions happen really close to each other in time. At a high level, a soft-delete is when a DELETE action doesn't actually remove the data. When data is soft-deleted, it's still there but just not visible in the main usage flow. This lets you get the data back when a delete is made by accident. One of the interesting side effects of designing any API is that you end up leaking the internals of how your database works to your users. Many object storage systems were designed with overwriting or deleting data as one of the primary operations, and as such have had to bolt versioning onto the side. For the most part this does work; but once you get into advanced versioning schemes everything starts to fall apart. Tigris doesn't suffer from the same problems because we built immutability into the core from day one, and in immutable systems you have to append new data on the end instead of overwriting data. At the least, actually storing the data en masse is a boring problem. You put the data somewhere, maybe name it after the checksum of its contents, and then have a daemon make sure it's copied three places. That daemon also handles cases when drives go offline and new ones are added to make sure data is shuffled around the cluster. This is largely a solved problem with projects like Ceph, Longhorn, or other distributed storage systems. Some object storage systems like S3 expose platform internals to make soft deletion work. In S3 deleting an object creates a delete marker (tombstone). A delete marker is an explicit marker that the object is deleted and should not be returned in normal operation. Here's what that looks like in practice: I don't know how I feel about this flow. Based on reading between the lines in the delete marker documentation it really feels like this is a leaked internal implementation detail of how S3's eventually consistent database works instead of a full fledged feature of the storage system. If I had to choose between leaking internal database details in the API and implementing a higher level API for something complicated like soft deletion, I'd want to implement the higher level API. Let's rethink what soft deletes really are. What if they were like the Recycle Bin in Windows? Soft-deletes are external references to buckets or objects that live in a different namespace from normal buckets or objects. We implemented them as external references instead of tombstones because this is effectively moving object metadata to the recycle bin. Tombstones mark the data as not being there, but soft-delete markers are a copy of the data that was there. This makes it easy to put the object back in place if you deleted it by mistake. One way to think about objects and buckets is that they are garbage collection roots for points in the endless sea of data. Any data in the sea without a root anchoring it down is eligible to be deleted. Uploading multiple versions of an object with a forkable bucket creates multiple metadata entries at their different timestamped version numbers. You can then fork a bucket from any one of those timestamps to see what the bucket was like at that point: This would solve the soft-delete problem, but our existing database schema using FoundationDB requires us to enable forking and snapshots at bucket creation time. In essence, we need something that's halfway between what we have (each bucket being a globally mutable namespace) and the bucket forking land of every action being appending metadata onto the end. To do that, we basically implemented most of that appending metadata on the end trick but to a different place: the soft deletion corner. When you enable soft-deletion and delete an object, its metadata gets moved to the trashcan so you can pluck it back into place: It's the same basic idea as the recycle bin on your desktop. Any buckets or objects left in the recycle bin for long enough become eligible to be deleted, which then makes the backend go and securely erase things. Effectively, any bits of metadata in the soft deletion corner are still considered garbage collection roots, they're just not shown when you do a normal call. The real fun comes into play when you remember that Tigris has a globally replicated active-active database where any region can change any object at any time. Most of the time things work out and objects are replicated without too much strife. The annoying part comes when two events are ordered weirdly. Imagine a scenario where one agent in one datacentre deletes an object after another agent in another datacentre: How would this replicate out? Well for one each change is timestamped by when it's done in terms of Unix nanoseconds, so the replication messages kinda look like this: This means that in theory, a user could DELETE an object before an update is processed by another region, and that would make the regions disagree about if the object exists or not. This is a horrible state to be in and usually requires support intervention or to recreate/re-delete the object. The root cause boils down to deleting objects actually deleting metadata from the database doesn't scale past a single region. Updates to metadata include the entire metadata object, so if you delete it locally and a new version is pushed remotely, the object will gain the remote state. We don't want users to have to deal with that, so we added the concept of anti-resurrection to Tigris. Any write to an object must prove it is newer than the deletion. In this circumstance, a user sent a DeleteObject request to the IAD datacentre at time 25, but an agent sent a new version of the object with PutObject to the ORD datacentre at time 15. The user's delete is newer than the agent's put, so the new version is rejected and the delete gets sent back to ORD. Tigris extends the S3 API by having users add headers to their requests. For example, to create a bucket with soft deletion enabled: Or to list soft-deleted objects: Or to permanently delete one soft-deleted version: When you have a soft-delete enabled bucket, you can also forcibly delete an entire bucket: Warning If you use this call on a bucket that doesn't have soft deletion enabled, you have permanently deleted your bucket. Please call this with care. Support cannot help you if you use this call wrongly. And then bring it back from the dead: Object storage entered our stacks as an unlimited FTP server we all used for backups. A distressing amount of the world's most important data lives in object storage buckets because it's the best place to put it. This is why having an "undo" button matters, it's what makes it safe to trust your backups in the cloud. To err is human, and mistakes are a "when" to plan for, not an "if" that you hopefully never have happen. The blast radius of one overly wide flag is measured in years of people's lives. One of the biggest usecases that comes to mind is ransomware prevention. Imagine a case where an attacker downloads everything in your bucket, deletes it, and asks for a ransom to send you the files back. With Tigris, soft deletes means that the ransom can be ignored, you can un-delete your data, and be on your merry way with incident response. The other big usecase is for agents, where they somehow get the idea that deleting production data is the right way to solve a problem. Both cases mean you need a quick and fast way to go back to before things went wrong. If you want true isolation instead of recovery, that's why we have bucket forking . Bucket forking needs to be enabled before a bucket is created, but you can enable soft deletion on any bucket in the dashboard whenever you want. Every storage system is going to make you choose between ones that hide how the platform works and ones that expose the gorey internals to users. I think that hiding the internals and exposing the high level operations built on top of them is the right way to go, if only because the higher level operations are much easier to make safe in our globally distributed future. Enable soft delete on any Tigris bucket, new or existing, and every delete becomes recoverable for up to 90 days. Restoring a whole bucket is one call. Read the soft delete docs .

cloud Cloud database Database
0 views
Kev Quirk 2 months ago

📝 2026-08-07 09:10: Brilliant. I'm such a "valued customer" that they couldn't even be arsed to put my...

Brilliant. I'm such a "valued customer" that they couldn't even be arsed to put my name in the email. Why is it even necessary to have a "business intelligence database" with a 3rd party? I ordered a fucking laptop. Can a tech company not create their own customer database? Full email here - https://cdn.kevquirk.com/framework-breach-email.pdf Thanks for reading this post via RSS. RSS is ace, and so are you. ❤️ You can reply to this post by email , or leave a comment .

ai AI database Database
0 views
Dangling Pointers 2 months ago

MatchBox: A Semantic Foundation for Data Plane Portability

MatchBox: A Semantic Foundation for Data Plane Portability Eric Hayden Campbell, Robert Zhang, Divyanshu Saxena, Aditya Akella, and Işıl Dillig PLDI'26 This paper identifies problems associated with subtle differences in match-action table implementations across hardware vendors and proposes some formalism to allow generic tooling to be implemented for match-action tables. A match-action table is a data structure present in some NICs and switches which specifies how packet-processing hardware should modify packets on the fly. This post on Gigaflow also discusses how a virtual switch can be implemented with match-action tables. Fig. 2 shows two match-action tables programmed in a hypothetical switch ( IP addresses in these tables are written in CIDR syntax ): Source: https://dl.acm.org/doi/10.1145/3808277 The syntax is clear but the semantics have an important footnote: switch S has a policy which causes packets that miss in the routing table to be dropped. Now imagine a switch ( ) from another vendor with a different policy: packets which cause a miss in the routing table are broadcast to all output ports. The task of porting the match-action tables from to in a way to preserves behavior is non-trivial. Fig. 3 contains a correct solution: Source: https://dl.acm.org/doi/10.1145/3808277 The difference is in the ACL table. The permissive allow entry has been replaced with two more restrictive entries that only allow packets that would not have triggered a miss in the routing table. One automated way to compute the contents of the ACL table for switch is an algorithm that looks at all pairs (i.e., the Cartesian product) of entries in the route and ACL table for switch , and then drops pairs that don’t make sense (e.g., the route and ACL addresses have no overlap). If you squint at that definition, it looks a lot like a relation join operation. In other words if you think of the ACL and routing tables as relations, then a way to compute the ACL table for switch would be to join the route and ACL tables for switch . The authors argue that relational algebra is almost the correct hammer for this nail, but not quite, and so this paper proposes Match Algebra . In Match Algebra, a match-action table can be thought of as a partial function which maps a valuation to an (action, valuation) pair. A valuation is just a set of bits (extracted from packet headers), and an action describes how a packet is to be modified. Another way to think of a match-action table is an ordered set of rules. A rule comprises: A guard, which defines the set of packets that match the rule An (action, valuation) pair Example guards could be: IPv4 packets with source address matching 10.2.1.0/24 UDP packets with a destination port of 15354 or 15355 An example (action, valuation) pairs could be: (replace the destination port, 15356) (pass the packet through unmodified, x) The set of rules is ordered such that if guards associated with multiple rules apply to a particular packet, the highest priority rule wins. The most important operators in the Match Algebra are: Sequential composition: modify the packet according to table , and use the result as input to table Preferential composition: Check to see if table has a rule matching the incoming packet, if so then apply that rule. Otherwise, use table to modify the packet. Parallel join: apply tables and to the input packet to generate two (action, valuation) pairs: , . Merge the results together. The merge operation requires that the results from and do not overlap (e.g., causes a destination port to be modified while causes destination address to be modified). The paper describes some other operators and presents formal semantics for Match Algebra. The paper describes which is a language (embedded in OCaml) which operates at the Match Algebra level of abstraction. Fig. 13 shows an example program which transforms match action tables with operators from Match Algebra: Source: https://dl.acm.org/doi/10.1145/3808277 Results The paper presents the following real-world use cases of : Transforming a set of match-action tables to an equivalent set with a different form (for porting rules between switches) Porting firewall rules between AWS, GCP, and Azure (which each have a different required structure) Translating system-wide eBPF tables to per-CPU eBPF tables Thanks for reading Dangling Pointers! Subscribe for free to receive new posts. Source: https://dl.acm.org/doi/10.1145/3808277 The syntax is clear but the semantics have an important footnote: switch S has a policy which causes packets that miss in the routing table to be dropped. Now imagine a switch ( ) from another vendor with a different policy: packets which cause a miss in the routing table are broadcast to all output ports. The task of porting the match-action tables from to in a way to preserves behavior is non-trivial. Fig. 3 contains a correct solution: Source: https://dl.acm.org/doi/10.1145/3808277 The difference is in the ACL table. The permissive allow entry has been replaced with two more restrictive entries that only allow packets that would not have triggered a miss in the routing table. One automated way to compute the contents of the ACL table for switch is an algorithm that looks at all pairs (i.e., the Cartesian product) of entries in the route and ACL table for switch , and then drops pairs that don’t make sense (e.g., the route and ACL addresses have no overlap). If you squint at that definition, it looks a lot like a relation join operation. In other words if you think of the ACL and routing tables as relations, then a way to compute the ACL table for switch would be to join the route and ACL tables for switch . The authors argue that relational algebra is almost the correct hammer for this nail, but not quite, and so this paper proposes Match Algebra . Match Algebra In Match Algebra, a match-action table can be thought of as a partial function which maps a valuation to an (action, valuation) pair. A valuation is just a set of bits (extracted from packet headers), and an action describes how a packet is to be modified. Another way to think of a match-action table is an ordered set of rules. A rule comprises: A guard, which defines the set of packets that match the rule An (action, valuation) pair IPv4 packets with source address matching 10.2.1.0/24 UDP packets with a destination port of 15354 or 15355 (replace the destination port, 15356) (pass the packet through unmodified, x) Sequential composition: modify the packet according to table , and use the result as input to table Preferential composition: Check to see if table has a rule matching the incoming packet, if so then apply that rule. Otherwise, use table to modify the packet. Parallel join: apply tables and to the input packet to generate two (action, valuation) pairs: , . Merge the results together. The merge operation requires that the results from and do not overlap (e.g., causes a destination port to be modified while causes destination address to be modified). Source: https://dl.acm.org/doi/10.1145/3808277 Results The paper presents the following real-world use cases of : Transforming a set of match-action tables to an equivalent set with a different form (for porting rules between switches) Porting firewall rules between AWS, GCP, and Azure (which each have a different required structure) Translating system-wide eBPF tables to per-CPU eBPF tables

hardware Hardware database Database programming Programming Ocaml
0 views
Ed Zitron's Where's Your Ed At 2 months ago

Premium: The Hater’s Guide To Oracle (Part 2)

Good morning premium subscribers! As ever, please ping me at [email protected] if you have any questions. Oracle has one of the strongest mythologies in the tech industry. Ask a regular person and they’ll tell you that it’s “incredibly profitable” and “growing fast,” that it’s “unstoppable,” and that Larry Ellison has the mandate of heaven with regard to the continual sales of software and hardware related to databases and AI. And those people are completely and utterly wrong. The original title of this article was “Is Oracle Dying?” because I assume, when I took a deeper look, that there’d be some sort of debate , some sort of bull case for a decades-old quasi-hyperscaler run by one of the more nakedly-evil CEOs in the history of tech. I assumed — incorrectly, I might add — that Oracle as a business was doing fine other than the ridiculous commitments it made to support the whims of Sam Altman and OpenAI via deals that I believed (and still believe) will kill Oracle . Except it turns out that Oracle has already been on a death spiral for the best part of a decade (if not longer) and has only survived this long by screwing its customers, taking on masses of debt, and — most importantly — more than $85 billion in acquisitions over the last 23 years. Pretty much every major product line outside of databases is a hodge-podge of other people’s innovation stapled together with a legendary contempt for the customer . These acquisitions (and continual price increases ) are the only thing keeping the reaper from Oracle’s door other than margin-destroying GPUs . And that’s why Oracle’s revenue looks like this : After April 2009’s $5.7 billion acquisition of Sun Microsystems , Oracle’s revenues barely kept pace with inflation until December 2021’s $28.3 billion acquisition of Cerner allowed it to create Oracle Health , adding about $6 billion in annual revenue that had 40% lower margins (about 21.7%) than Oracle’s other businesses , though Oracle immediately started closing offices and brutal layoffs to try and bring them up. And as I mentioned above, Oracle’s other plan was to sink a little over $99 billion in capital expenditures since the middle of calendar year 2020 into AI GPUs.  Anyway, let’s see what that’s done to margins - OH MY GOD ! Oracle is a decades-long mission to keep reapplying lipstick to a pig. Billions of dollars of acquisitions have, for the most part, only succeeded in keeping the company’s revenue growth from going negative, and as noted by forensic accountant Howard M. Schilit , this is one of the most well-documented cases of accounting shenanigans being used to cover up that a business is in decline. Today’s newsletter is a sequel to the Hater’s Guide To Oracle , where I told the sordid tale of how Larry Ellison grew a massive, lucrative business out of a database business that one reporter once told me was a “ law firm with a database company attached ,” an Enterprise Resource Planning (ERP) product that competes with SAP to create the most-annoying way to run a large company, and a business built around licensing Java that exists mostly to email people and say “you need to pay us for Java or we’ll sue you.”  Then, as I’ve mentioned, there’s Oracle’s cloud infrastructure business, a decade-old also-ran that was meant to compete with Microsoft Azure and Amazon Web Services, but only managed to catch up following the advent of AI GPUs and a movement where all it took to party was buying billions of GPUs and saying “gosh darn, we love AI.” I originally started drafting this as a much tamer piece where I’d ask whether Oracle was dying, but as my editor and I started digging into the research, it became obvious that not only is Oracle dying , it’s been dying for years , kept alive through decades of acquisitions and a desperate and dangerous commitment to generative AI. And AI, I believe, will be what eventually kills Oracle dead.  In the past, all Oracle had to do to survive was buy somebody else’s company and replace its flagging revenues with theirs, turning the screws on their customers and laying off as many people as necessary to balance the books. While chaotic and decaying, Oracle’s empire has kept above water by never overextending itself, always keeping a positive free cashflow , and generally avoiding buying into industry hype cycles outside of whatever SaaS vehicle might potentially plug the gap in its earnings.  Yet with AI, Oracle broke its long-standing trend of letting someone else figure out the innovation, choosing instead to build its own cloud infrastructure, spending more in capex in its last fiscal year ( $55.6 billion ) than it did in the previous nine years combined ($50.4 billion), tripling its debt from $56.91 billion in FY2017 to $167.4 billion in FY2026, a year that ended with its free cashflow sitting at negative $23.69 billion.   For comparison, Oracle has had positive free cashflow every single year since 2001, including the Great Financial Crisis and COVID. Oracle has doomed itself with its commitment to the AI bubble. It has committed to building 7.1GW of data center capacity for one company — OpenAI — as part of a $300 billion, five-year-long contract that requires it to build an impossible amount of capacity in an impossible period of time for a client that could never afford the $70 billion or more in annual costs to make any of it worth it.  Today I am going to talk trash on what I consider to be one of the single-worst companies in the tech industry that’s survived only through financial engineering and never, ever overextending itself.  With revenue plateauing and customers in revolt, Oracle’s future already looked murky, but with the power of AI — and $95 billion in FY2027 capex — it’s becoming increasingly clear that this may be Larry Ellison’s last dance with Silicon Valley. This is the Hater’s Guide To Oracle Part 2, or AIpoaclypse Now.

java Java database Database business Business
0 views
Dangling Pointers 2 months ago

Breadcrumb Filters: Fast Fully Featured Filters

Breadcrumb Filters: Fast Fully Featured Filters Andrew Krapivin, Aaditya Rangarajan, Alex Conway, Martin Farach-Colton, Rob Johnson, and Prashant Pandey SIGMOD'26 This paper presents the design of a breadcrumb filter , which is a membership testing data structure . Unlike a Bloom filter , a breadcrumb filter supports operations like deletion and merging. Breadcrumb filters also have the nice property that most operations access a single cache line (most of the time). A breadcrumb filter is a fingerprinting filter . Each item in a set is represented by its fingerprint (i.e., hash). Say a filter contains 1024 cache lines, and each cache line has storage for items. To insert an item into the filter, compute a 16-bit fingerprint of the item. Decompose that into a 10-bit integer (the cache line index) and a 6-bit integer (the remainder). Use the cache line index to determine which cache line to access. Find an empty slot in that cache line and place the remainder bits of the fingerprint into the empty slot to represent the item. A breadcrumb filter builds on top of these mechanics by cleverly dividing the filter storage into two sections: the front and back yards. The front-yard represents the fast path: each filter operation will touch one front-yard cache line. The backyard is only used to handle cases where a front-yard cache line fills up. The paper hyphenates “front-yard” but writes “backyard” as one word. The following pseudo-code illustrates how an item is inserted into a breadcrumb filter: A lookup operation follows a similar structure: To delete an item from a breadcrumb filter, it is sufficient to delete the item’s fingerprint . That wasn’t obvious to me up front. Imagine two items have the same fingerprint (hash). Inserting them both causes the same fingerprint to be inserted twice. Now, when one of them is deleted, it suffices to delete one of the copies of the fingerprint in the breadcrumb filter. The trick with deletion is that deleting a fingerprint from a front-yard cache line can require promoting an item from an associated backyard cache line. The real magic with breadcrumb filters comes in how items are moved between the front-yard and backyard. If a front-yard cache line is found to be full during insertion, then one item is moved to the backyard. That item could be the one that is currently being inserted, or it could be an item that was previously placed into the front-yard. The policy is: move the item which has the greatest value of the remainder bits . For example, if two items (A, and B) have remainder bits of 23 and 7, then item A will be moved to the backyard before B. This enables lookup operations to avoid touching backyard cache lines. For example, say the item that is being searched for has remainder bits = 23, and the (full) front-yard cache line contains items with remainder bits = [12, 5, 34, 3], there is no need to search the backyard. The “34” in the front-yard implies that no item with remainder bits value less than can be in the associated backyard cache lines. Note that this policy requires that delete operations sometimes move items from the backyard to the front-yard. The other trick is a mapping between front-yard and backyard cache lines which enables promotion of a deleted item from backyard to front-yard. Say each front-yard cache line is associated with two backyard cache lines, but those backyard cache lines are each associated with many front-yard cache lines. A front-yard cache line index is represented with 10 bits: The two backyard cache line indices associated with that front-yard cache line are: The key here is that very little information needs to be stored in the backyard in order to allow mapping from a backyard cache line index to a front-yard cache line index. To map backyard index back to , all one needs to know is the value of bit (which is stored in the backyard cache line). When an item is deleted from a front-yard cache line, the two associated backyard cache lines are searched for an item to be promoted back to the front-yard. Metadata (e.g., the value of bit ) is used to ensure that items are promoted back to the front-yard cache line from whence they came. Fig. 9 compares the throughput of the breadcrumb filter (BCF*) against other filters: Source: https://dl.acm.org/doi/10.1145/3786629 Dangling Pointers This design agrees with many others that the best one can do is read a single cache line for each lookup. I don’t have a better solution in mind, but it seems painfully slow if the filter doesn’t fit in cache. Thanks for reading Dangling Pointers! Subscribe for free to receive new posts.

database Database performance Performance programming Programming
0 views
Tenderlove Making 3 months ago

Detecting Full Table Scans With SQLite

I’m at RubyConf this week, and it’s great! I recently read that lobste.rs is now running on SQLite . One part from the post caught my attention: I wish we could say in a test, “Fail if you encounter any full table scans”. Which would have caught the perf issues we experienced during the first deploy. SQLite collects information about prepared statements and exposes those statistics though an API . The upshot of this is that we can tell whether a statement did a full table scan after executing the statement without using an . Here’s an example program that demonstrates detecting a query did a full table scan: Feels like we could integrate this in to Rails and warn or raise in test / development. I’m not sure if we’d want to check this all the time in production, but maybe it would be fine?

performance Performance database Database testing Testing ruby Ruby
0 views
Dangling Pointers 3 months ago

RABIT: Efficient Range Queries with Bitmap Indexing

RABIT: Efficient Range Queries with Bitmap Indexing Junchang Wang, Fu Xiao, and Manos Athanassoulis SIGMOD'26 This paper presents an optimization for range filtering (e.g., ). File this under: “so crazy it might actually work”. This paper builds on the concept of a bitmap index . If the cardinality of a column is (i.e., there are distinct values in the column), then a bitmap index stores additional 1-bit columns along with the original column. These additional columns are called point bitvectors . The value in additional column is 1 for a particular row if the value in that row is the th distinct value. Here is an example column with 6 rows and 4 distinct values (0, 1, 2, 6). The header indicates that this is the value column. And here is the same column with 4 additional 1-bit wide rows attached. The header indicates columns that hold point bitvectors. To find all rows which have a value of 2, simply read the 3rd (i.e., ) bitmap column. Similarly, to find all rows which have a value of 6, simply read the 4th bitmap column. To efficiently support range queries, the authors of this paper propose adding even more columns that hold 1-bit values. These columns are called cumulative bitvectors . A cumulative bitvector holds the bitwise-or of a set of point bitvectors. In the example above, let’s create a cumulative bitvector (named ) for the values 0 and 1. The value of this column for a given row will be 1 if the value contained in the row is either 0 or 1. In other words: . Similarly, we can create a cumulative bitvector for the values 2 and 6 using the equation: . Here is the full turkey: A range query can be executed by reading one or more cumulative bitvectors and possibly a few point bitvectors. For example, to find all rows that have a value < 2, all one needs to do is read the value of . Cumulative bitvectors do not add much value in this toy example because each cumulative bitvector only aggregates data for two values, but you can see how this could work well with more aggregation. This trick can even be made to work for range queries that partially overlap with a cumulative bitvector. This whole scheme relies on the fact that point and cumulative bitvectors are highly compressible. This paper assumes the use of WAH compression . The executive summary of WAH compression is to divide each bitvector into words (e.g., 32 or 64 bit). One bit of each word is metadata that determines if the word is a or a . The remaining bits of a literal word contain raw (uncompressible bits). The remaining bits of a fill word contain a value and a length (run-length encoding). Fig. 9 compares throughput of this scheme (labeled GE in the figure) to other work. means each cumulative bitvector aggregates data for 20 point bitvectors. Performance looks good even for columns holding 100K distinct elements. Source: https://dl.acm.org/doi/10.1145/3769819 Table 3 compares the storage requirements for this scheme versus other indexing schemes that support range queries, which seems too good to be true. Source: https://dl.acm.org/doi/10.1145/3769819 Dangling Pointers I wonder if this idea could be generalized to other types of filtering, such as string operations (e.g., . Thanks for reading Dangling Pointers! Subscribe for free to receive new posts.

database Database
0 views
ptrchm 3 months ago

Postgres Backups to S3 with WAL-G and Kamal

The Kamal setup guides I found online focus on S3 backups using . You don’t want that for a production database. A better solution is to set up your Postgres database for Point-In-Time Recovery (PITR) using WAL-G or pgBackRest. This means your database is continuously archiving WAL segments to an S3 bucket (roughly every 60 seconds), so you can restore to any point in time. With LLMs, it’s not that hard to set up. This quick guide focuses on WAL-G, because I’ve found it to be a lot easier to set up than pgBackRest.

devops DevOps database Database cloud Cloud
0 views
Binary Igor 3 months ago

The Order of Data: defaults, performance, determinism & paging

How does the database decide on the order, when it is not specified? What about performance? Can returned pages overlap? Meaning: might item from page 1 suddenly appear on page 2, even if the underlying data stays the same?

programming Programming database Database performance Performance
0 views
Evan Hahn 3 months ago

Prefer STRICT tables in SQLite

In short: I prefer strict tables in SQLite because they avoid some datatype problems, such as putting text in number columns. SQLite has a feature that I think is underrated: strict tables . Strict tables help enforce rigid typing, preventing mistakes like putting text into integer columns. I like them, and wrote this post to promote their use! To make a strict table, add to the end of its definition. Like this: That’s it! But what does it do? Broadly, strict tables help enforce rigid types, like other SQL engines do. Most significantly, strict tables keep you from inserting the wrong type into a column. For example, SQLite normally lets you put text into an column, but not with strict tables. Personally, I think it’s a mistake to try to put text in an integer column, or vice-versa. I don’t want SQLite to let me make this error! The same validation happens for s, too. Notably, if a value can be losslessly converted, it will still be accepted. For example, the string can be perfectly converted to an integer, so it’s allowed. These two lines are equivalent, even for a strict table: By default, you can create columns with bogus types. For example, all of these work even though they aren’t valid SQLite datatypes: I think these aren’t what the developer intended. Some of these are typos, some of them are misunderstandings of which datatypes SQLite supports , and some are egregious mistakes. Appending to any of these statements makes them error. In my opinion, that’s the correct behavior! Only , , , , , and are allowed. Strict tables also require a column type, so you can’t do . If you still need a column to be flexible, you can use the datatype. As the name suggests, it allows anything—even in a strict table. I haven’t found a use for this, but maybe you will! I prefer strict tables but I must share a few cons. Not everything is better! I think it’s best to use strictness from the start, but that’s not always possible. Unfortunately, I don’t think there’s a way to a table to make it strict. I think you have to copy the data out of the non-strict table into the strict one. Something like this: Note that this could be tricky if the non-strict table has invalid data! For example, if the old data accidentally contains text in an integer column, you’ll get errors when doing the migration. You’ll probably need to clean the data or cast it . You could make a rule for your codebase that all new tables are strict. That might be useful—at least some of your tables are valid! But it might also mean you have inconsistent validation across your tables, which might be more surprising than having weak validation on all tables. It’s up to you to decide whether this is a good fit for you. SQLite has a whole page called “The Advantages Of Flexible Typing” , where they argue that SQLite’s flexible behavior is good, actually. I hesitate to wade into the controversy of static-versus-dynamic, but I disagree in most cases. I’ve personally encountered many bugs where an unexpected data type caused subtle headaches. I’d much rather these mistakes explode loudly. But it’s worth noting that SQLite’s developers seem not to share my preference for strict tables! They point out a few good uses for flexible tables, such as “a pure key-value store” or “a place to store miscellaneous attributes” of different types. They also mention that you might want to keep the invalid data in some cases, like if you’re directly importing a messy CSV and don’t want to lose any data. I still prefer strict tables, but acknowledge there are some reasonable cases for non-strict ones. (There’s also at least one comment in the SQLite source that calls non-strict tables “legacy” , but I trust that less than the official documentation.) SQLite introduced strict tables in version 3.37.0 , released November 2021. If you’re on an older version of SQLite, you can’t use strict tables. It’s worth noting that old versions of SQLite can’t read databases with strict tables. For example, if you create a strict table in the newest version of SQLite and then try to read that database in SQLite 3.36.0 (before strict tables were added), you’ll get an error—even if the strict table is already in the database. Strict tables are theoretically slower because they have to do a little extra work. For example, they check datatypes when doing an insert or update . But in practice, I don’t think this is an issue. I wrote a hacky script that inserted millions of rows into a table with 100 columns, and there was no obvious difference on multiple machines I tried. The file size on disk was also the same. I didn’t test this thoroughly, so maybe there’s something I missed, but I don’t think strict tables present a performance problem. In fact, one might expect better performance because you won’t be accidentally mismatching SQLite’s column affinities. But again, I haven’t tested this. Personally, I think the pros of strict tables outweigh the cons. I generally prefer when types are rigidly enforced. It squashes a class of mistakes, and help enforce good data integrity. They’re not a panacea, but they’re usually easy to add and go a long way. If there’s a SQLite feature you think is underrated, please tell me .

database Database sql Sql
1 views