Latest Posts (11 found)
Anton Zhiyanov 2 weeks ago

Gist of Go: Atomics

This is a chapter from my book on Go concurrency , which teaches the topic from the ground up through interactive examples. Some concurrent operations don't require explicit synchronization. We can use these to create lock-free types and functions that are safe to use from multiple goroutines. Let's dive into the topic! Non-atomic increment • Atomic operations • Composition • Atomic vs. mutex • Keep it up Suppose multiple goroutines increment a shared counter: There are 5 goroutines, and each one increments 10,000 times, so the final result should be 50,000. But it's usually less. Let's run the code a few more times: The race detector is reporting a problem: This might seem strange — shouldn't the operation be atomic? Actually, it's not. It involves three steps (read-modify-write): If two goroutines both read the value , then each increments it and writes it back, the new will be instead of like it should be. As a result, some increments to the counter will be lost, and the final value will be less than 50,000. As we talked about in the Race conditions chapter, you can make an operation atomic by using mutexes or other synchronization tools. But for this chapter, let's agree not to use them. Here, when I say "atomic operation", I mean an operation that doesn't require the caller to use explicit locks, but is still safe to use in a concurrent environment. An operation without synchronization can only be truly atomic if it translates to a single processor instruction. Such operations don't need locks and won't cause issues when called concurrently (even the write operations). In a perfect world, every operation would be atomic, and we wouldn't have to deal with mutexes. But in reality, there are only a few atomics, and they're all found in the package. This package provides a set of atomic types: Each atomic type provides the following methods: reads the value of a variable, sets a new value: sets a new value (like ) and returns the old one: sets a new value only if the current value is still what you expect it to be: Numeric types also provide an method that increments the value by the specified amount: And the / methods for bitwise operations (Go 1.23+): All methods are translated to a single CPU instruction, so they are safe for concurrent calls. Strictly speaking, this isn't always true. Not all processors support the full set of concurrent operations, so sometimes more than one instruction is needed. But we don't have to worry about that — Go guarantees the atomicity of operations for the caller. It uses low-level mechanisms specific to each processor architecture to do this. Like other synchronization primitives, each atomic variable has its own internal state. So, you should only pass it as a pointer, not by value, to avoid accidentally copying the state. When using , all loads and stores should use the same concrete type. The following code will cause a panic: Now, let's go back to the counter program: And rewrite it to use an atomic counter: Much better! ✎ Exercise: Atomic counter +1 more Practice is crucial in turning abstract knowledge into skills, making theory alone insufficient. The full version of the book contains a lot of exercises — that's why I recommend getting it . If you are okay with just theory for now, let's continue. An atomic operation in a concurrent program is a great thing. Such operation usually transforms into a single processor instruction, and it does not require locks. You can safely call it from different goroutines and receive a predictable result. But what happens if you combine atomic operations? Let's find out. Let's look at a function that increments a counter: As you already know, isn't safe to call from multiple goroutines because causes a data race. Now I will try to fix the problem and propose several options. In each case, answer the question: if you call from 100 goroutines, is the final value of the guaranteed? Is the value guaranteed? It is guaranteed. Is the value guaranteed? It's not guaranteed. Is the value guaranteed? It's not guaranteed. People sometimes think that the composition of atomic operations also magically becomes an atomic operation. But it doesn't. For example, the second of the above examples: Call 100 times from different goroutines: Run the program with the flag — there are no races: But can we be sure what the final value of will be? Nope. and calls are interleaved from different goroutines. This causes a race condition (not to be confused with a data race) and leads to an unpredictable value. Check yourself by answering the question: in which example is an atomic operation? In none of them. In all examples, is not an atomic operation. The composition of atomics is always non-atomic. The first example, however, guarantees the final value of the in a concurrent environment: If we run 100 goroutines, the will ultimately equal 200. The reason is that is a sequence-independent operation. The runtime can perform such operations in any order, and the result will not change. The second and third examples use sequence-dependent operations. When we run 100 goroutines, the order of operations is different each time. Therefore, the result is also different. A bulletproof way to make a composite operation atomic and prevent race conditions is to use a mutex: But sometimes an atomic variable with is all you need. Let's look at an example. ✎ Exercise: Concurrent-safe stack Practice is crucial in turning abstract knowledge into skills, making theory alone insufficient. The full version of the book contains a lot of exercises — that's why I recommend getting it . If you are okay with just theory for now, let's continue. Let's say we have a gate that needs to be closed: In a concurrent environment, there are data races on the field. We can fix this with a mutex: Alternatively, we can use on an atomic instead of a mutex: The type is now more compact and simple. This isn't a very common use case — we usually want a goroutine to wait on a locked mutex and continue once it's unlocked. But for "early exit" situations, it's perfect. Atomics are a specialized but useful tool. You can use them for simple counters and flags, but be very careful when using them for more complex operations. You can also use them instead of mutexes to exit early. In the next chapter, we'll talk about testing concurrent code (coming soon). Pre-order for $10   or read online Read the current value of . Add one to it. Write the new value back to . — a boolean value; / — a 4- or 8-byte integer; / — a 4- or 8-byte unsigned integer; — a value of type; — a pointer to a value of type (generic).

0 views
Anton Zhiyanov 2 weeks ago

Go proposal: Hashers

Part of the Accepted! series, explaining the upcoming Go changes in simple terms. Provide a consistent approach to hashing and equality checks in custom data structures. Ver. 1.26 • Stdlib • Medium impact The new interface is the standard way to hash and compare elements in custom collections: The type is the default hasher implementation for comparable types, like numbers, strings, and structs with comparable fields. The package offers hash functions for byte slices and strings, but it doesn't provide any guidance on how to create custom hash-based data structures. The proposal aims to improve this by introducing hasher — a standardized interface for hashing and comparing the members of a collection, along with a default implementation. Add the hasher interface to the package: Along with the default hasher implementation for comparable types: Here's a case-insensitive string hasher: And a generic that uses a pluggable hasher for custom equality and hashing: The helper method uses the hasher to compute the hash of a value: This hash is used in the and methods. It acts as a key in the bucket map to find the right bucket for a value. checks if the value exists in the corresponding bucket: adds a value to the corresponding bucket: Now we can create a case-insensitive string set: Or a regular string set using : 𝗣 70471 • 𝗖𝗟 657296 (in progress)

2 views
Anton Zhiyanov 2 weeks ago

Write the damn code

Here's some popular programming advice these days: Learn to decompose problems into smaller chunks, be specific about what you want, pick the right AI model for the task, and iterate on your prompts . Don't do this. I mean, "learn to decompose the problem" — sure. "Iterate on your prompts" — not so much. Write the actual code instead: You probably see the pattern now. Get involved with the code, don't leave it all to AI. If, given the prompt, AI does the job perfectly on first or second iteration — fine. Otherwise, stop refining the prompt. Go write some code, then get back to the AI. You'll get much better results. Don't get me wrong: this is not anti-AI advice. Use it, by all means. Use it a lot if you want to. But don't fall into the trap of endless back-and-forth prompt refinement, trying to get the perfect result from AI by "programming in English". It's an imprecise, slow and terribly painful way to get things done. Get your hands dirty. Write the code. It's what you are good at. You are a software engineer. Don't become a prompt refiner.

1 views
Anton Zhiyanov 2 weeks ago

Go is #2 among newer languages

I checked out several programming languages rankings. If you only include newer languages (version 1.0 released after 2010), the top 6 are: ➀ TypeScript, ➁ Go, ➂ Rust, ➃ Kotlin, ➄ Dart, and ➅ Swift. Sources: IEEE , Stack Overflow , Languish . I'm not using TIOBE because their method has major flaws. TypeScript's position is very strong, of course (I guess no one likes JavaScript these days). And it's great to see that more and more developers are choosing Go for the backend. Also, Rust scores very close in all rankings except IEEE, so we'll see what happens in the coming years.

0 views
Anton Zhiyanov 3 weeks ago

Go proposal: new(expr)

Part of the Accepted! series, explaining the upcoming Go changes in simple terms. Allow the built-in to be called on expressions. Ver. 1.26 • Language • High impact Previously, you could only use the built-in with types: Now you can also use it with expressions: If the argument is an expression of type T, then allocates a variable of type T, initializes it to the value of , and returns its address, a value of type . There's an easy way to create a pointer to a composite literal: But no easy way to create a pointer to a value of simple type: The proposal aims to fix this. Update the Allocation section of the language specification as follows: The built-in function creates a new, initialized variable and returns a pointer to it. It accepts a single argument, which may be either an expression or a type. ➀ If the argument is an expression of type T, or an untyped constant expression whose default type is T, then allocates a variable of type T, initializes it to the value of , and returns its address, a value of type . ➁ If the argument is a type T, then allocates a variable initialized to the zero value of type T. For example, and each return a pointer to a new variable of type int. The value of the first variable is 123, and the value of the second is 0. ➀ is the new part, ➁ already worked as described. Pointer to a simple type: Pointer to a composite value: Pointer to the result of a function call: Passing is still not allowed: 𝗣 45624 • 𝗖𝗟 704935 , 704737 , 704955 , 705157

0 views
Anton Zhiyanov 3 weeks ago

Accepted! Go proposals distilled

I'm launching a new Go-related series named Accepted! For each accepted proposal, I'll write a one-page summary that explains the change in simple terms. This should (hopefully) be the easiest way to keep up with upcoming changes without having to read through 2,364 comments on Go's GitHub repo. Here's a sneak peak: The plan is to publish the already accepted proposals from the upcoming 1.26 release, and then publish new ones as they get accepted. I'll probably skip the minor ones, but we'll see. Stay tuned!

0 views
Anton Zhiyanov 3 weeks ago

Native threading and multiprocessing in Go

As you probably know, the only way to run tasks concurrently in Go is by using goroutines. But what if we bypass the runtime and run tasks directly on OS threads or even processes? I decided to give it a try. To safely manage threads and processes in Go, I'd normally need to modify Go's internals. But since this is just a research project, I chose to (ab)use cgo and syscalls instead. That's how I created multi — a small package that explores unconventional ways to handle concurrency in Go. Features • Goroutines • Threads • Processes • Benchmarks • Final thoughts Multi offers three types of "concurrent groups". Each one has an API similar to , but they work very differently under the hood: runs Go functions in goroutines that are locked to OS threads. Each function executes in its own goroutine. Safe to use in production, although unnecessary, because the regular non-locked goroutines work just fine. runs Go functions in separate OS threads using POSIX threads. Each function executes in its own thread. This implementation bypasses Go's runtime thread management. Calling Go code from threads not created by the Go runtime can lead to issues with garbage collection, signal handling, and the scheduler. Not meant for production use. runs Go functions in separate OS processes. Each function executes in its own process forked from the main one. This implementation uses process forking, which is not supported by the Go runtime and can cause undefined behavior, especially in programs with multiple goroutines or complex state. Not meant for production use. All groups offer an API similar to . Runs Go functions in goroutines that are locked to OS threads. starts a regular goroutine for each call, and assigns it to its own thread. Here's a simplified implementation: goro/thread.go You can use channels and other standard concurrency tools inside the functions managed by the group. Runs Go functions in separate OS threads using POSIX threads. creates a native OS thread for each call. It uses cgo to start and join threads. Here is a simplified implementation: pthread/thread.go You can use channels and other standard concurrency tools inside the functions managed by the group. Runs Go functions in separate OS processes forked from the main one. forks the main process for each call. It uses syscalls to fork processes and wait for them to finish. Here is a simplified implementation: proc/process.go You can only use to exchange data between processes, since regular Go channels and other concurrency tools don't work across process boundaries. Running some CPU-bound workload (with no allocations or I/O) on Apple M1 gives these results: And here are the results from GitHub actions: One execution here means a group of 4 workers each doing 10 million iterations of generating random numbers and adding them up. See the benchmark code for details. As you can see, the default concurrency model ( in the results, using standard goroutine scheduling without meddling with threads or processes) works just fine and doesn't add any noticeable overhead. You probably already knew that, but it's always good to double-check, right? I don't think anyone will find these concurrent groups useful in real-world situations, but it's still interesting to look at possible (even if flawed) implementations and compare them to Go's default (and only) concurrency model. Check out the nalgeon/multi repo for the implementation. P.S. Want to learn more about concurrency? Check out my interactive book

0 views
Anton Zhiyanov 2 months ago

Building blocks for idiomatic Go pipelines

I've created a Go package called chans that offers generic channel operations to make it easier to build concurrent pipelines. It aims to be flexible, unopinionated, and composable, without over-abstracting or taking control away from the developer. Here's a toy example: Now let's go over the features. Filter sends values from the input channel to the output if a predicate returns true. ■ □ ■ □ → ■ ■ Map reads values from the input channel, applies a function, and sends the result to the output. ■ ■ ■ → ● ● ● Reduce combines all values from the input channel into one using a function and returns the result. ■ ■ ■ ■ → ∑ FilterOut ignores values from the input channel if a predicate returns true, otherwise sends them to the output. ■ □ ■ □ → □ □ Drop skips the first N values from the input channel and sends the rest to the output. ➊ ➋ ➌ ➍ → ➌ ➍ DropWhile skips values from the input channel as long as a predicate returns true, then sends the rest to the output. ■ ■ ▲ ● → ▲ ● Take sends up to N values from the input channel to the output. ➊ ➋ ➌ ➍ → ➊ ➋ TakeNth sends every Nth value from the input channel to the output. ➊ ➋ ➌ ➍ → ➊ ➌ TakeWhile sends values from the input channel to the output while a predicate returns true. ■ ■ ▲ ● → ■ ■ First returns the first value from the input channel that matches a predicate. ■ ■ ▲ ● → ▲ Chunk groups values from the input channel into fixed-size slices and sends them to the output. ■ ■ ■ ■ ■ → ■ ■ │ ■ ■ │ ■ ChunkBy groups consecutive values from the input channel into slices whenever the key function's result changes. ■ ■ ● ● ▲ → ■ ■ │ ● ● │ ▲ Flatten reads slices from the input channel and sends their elements to the output in order. ■ ■ │ ■ ■ │ ■ → ■ ■ ■ ■ ■ Compact sends values from the input channel to the output, skipping consecutive duplicates. ■ ■ ● ● ■ → ■ ● ■ CompactBy sends values from the input channel to the output, skipping consecutive duplicates as determined by a custom equality function. ■ ■ ● ● ■ eq→ ■ ● ■ Distinct sends values from the input channel to the output, skipping all duplicates. ■ ■ ● ● ■ → ■ ● DistinctBy sends values from the input channel to the output, skipping duplicates as determined by a key function. ■ ■ ● ● ■ key→ ■ ● Broadcast sends every value from the input channel to all output channels. ➊ ➋ ➌ ➍ ↓ ➊ ➋ ➌ ➍ ➊ ➋ ➌ ➍ Split sends values from the input channel to output channels in round-robin fashion. ➊ ➋ ➌ ➍ ↓ ➊ ➌ ➋ ➍ Partition sends values from the input channel to one of two outputs based on a predicate. ■ □ ■ □ ↓ ■ ■ □ □ Merge concurrently sends values from multiple input channels to the output, with no guaranteed order. ■ ■ ■ ● ● ● ↓ ● ● ■ ■ ■ ● Concat sends values from multiple input channels to the output, processing each input channel in order. ■ ■ ■ ● ● ● ↓ ■ ■ ■ ● ● ● Drain consumes and discards all values from the input channel. ■ ■ ■ ■ → ∅ I think third-party concurrency packages are often too opinionated and try to hide too much complexity. As a result, they end up being inflexible and don't fit a lot of use cases. For example, here's how you use the function from the rill package: The code looks simple, but it makes pretty opinionated and not very flexible: While this approach works for many developers, I personally don't like it. With , my goal was to offer a fairly low-level set of composable channel operations and let developers decide how to use them. For comparison, here's how you use the function: only implements the core mapping logic: You decide the rest: The same principles apply to other channel operations. Let's say we want to calculate the total balance of VIP user accounts: Here's how we can do it using . First, use to get the accounts from the database: Next, use to select only the VIP accounts: Next, use to calculate the total balance: Finally, check for errors and return the result: If you're building concurrent pipelines in Go, you might find useful. See the nalgeon/chans repo if you are interested. P.S. Want to learn more about concurrency? Check out my interactive book

0 views
Anton Zhiyanov 2 months ago

Gist of Go: Signaling

This is a chapter from my book on Go concurrency , which teaches the topic from the ground up through interactive examples. The main way goroutines communicate in Go is through channels. But channels aren't the only way for goroutines to signal each other. Let's try a different approach! Signaling • One-time subscription • Broadcasting • Broadcasting w/channels • Publish/subscribe • Run once • Once-functions • Object pool • Keep it up Let's say we have a goroutine that generates a random number between 1 and 100: And the second one checks if the number is lucky or not: The second goroutine will only work correctly if the first one has already set the number. So, we need to find a way to synchronize them. For example, we can make a channel: But what if we want to be a regular number, and channels are not an option? We can make the generator goroutine signal when a number is ready, and have the checker goroutine wait for that signal. In Go, we can do this using a condition variable , which is implemented with the type. A has a mutex inside it: A has two methods — and : If there are multiple waiting goroutines when is called, only one of them will be resumed. If there are no waiting goroutines, does nothing. To see why needs to go through all this mutex trouble, check out this example: Both goroutines use the shared variable, so we need to protect it with a mutex. The checker goroutine starts by locking the mutex ➌. If the generator hasn't run yet (meaning is 0), the goroutine calls ➍ and blocks. If only blocked the goroutine, the mutex would stay locked, and the generator couldn't change ➊. That's why unlocks the mutex before blocking. The generator goroutine also starts by locking the mutex ➊. After setting the value, the generator calls ➋ to let the checker know it's ready, and then unlocks the mutex. Now, if resumed ➍ did nothing, the checker goroutine would continue running. But the mutex would stay unlocked, so working with wouldn't be safe. That's why locks the mutex again after receiving the signal. In theory, everything should work. Here's the output: Everything seems fine, but there's a subtle bug. When the checker goroutine wakes up after receiving a signal, the mutex is unlocked for a brief moment before locks it again. Theoretically, in that short time, another goroutine could sneak in and set to 0. The checker goroutine wouldn't notice this and would keep running, even though it's supposed to wait if is zero. That's why, in practice, is always called inside a for loop, not inside an if statement. Not like this: But like this (note that the condition is the same as in the if statement): In most cases, this for loop will work just like an if statement: But if another goroutine intervenes between ➊ and ➋ and sets to zero, the goroutine will notice this at ➌ and go back to waiting. This way, it will never keep running when is zero — which is exactly what we want. Here's the complete example: Like other synchronization primitives, a condition variable has its own internal state. So, you should only pass it as a pointer, not by value. Even better, don't pass it at all — wrap it inside a type instead. We'll do this in the next step. ✎ Exercise: Blocking queue Practice is crucial in turning abstract knowledge into skills, making theory alone insufficient. The full version of the book contains a lot of exercises — that's why I recommend getting it . If you are okay with just theory for now, let's continue. Let's go back to the lucky numbers example: Let's refactor the code and create a type with and methods: Here's the implementation: Example usage: Note that this is a one-time signaling, not a long-term subscription. Once a subscriber goroutine receives the generated number, it is no longer subscribed to . We'll look at an example of a long-term subscription later in the chapter. Everything works, but there's still a problem. If you call from N goroutines, you can set up N subscribers, but only notifies one of them. We'll figure out how to notify all of them in the next step. Notifying all subscribers instead of just one is called broadcasting . To do this in , we only need to change one line in the method: The method wakes up one goroutine that's waiting on , while the method wakes up all goroutines waiting on . This is exactly what we need. Here's a usage example: A typical way to use a condition variable looks like this: There is a publisher goroutine and one or more subscriber goroutines. They all use some shared state protected by a condition variable . The publisher goroutine changes the shared state and notifies either one subscriber ( ) or all subscribers ( ): Note that this is a one-time notification, not a long-term subscription. Once a subscriber goroutine receives the signal, it is no longer subscribed to the publisher. We'll look at an example of a long-term subscription later in the chapter. ✎ Exercise: Barrier using a condition variable Practice is crucial in turning abstract knowledge into skills, making theory alone insufficient. The full version of the book contains a lot of exercises — that's why I recommend getting it . If you are okay with just theory for now, let's continue. As we discussed, it's easy to implement signaling using a channel: This approach only works with one receiver. If we subscribe multiple goroutines to the channel, only one of them will get the generated number. For broadcasting, we can just close the channel: However, we can only broadcast the fact that the state has changed (the channel is closed), not the actual state (the value of ). So, we still need to protect the state with a mutex. This goes against the idea of using channels to pass data between goroutines. Also, we can't send a second broadcast notification because a channel can only be closed once. We have two problems with our "broadcast by closing a channel" approach: Let's solve both problems and create a simple publish/subscribe system: Here's the type: The method adds a subscriber and returns the channel where random numbers will be sent: Note that we use as a mutex here to protect access to the shared list of subscription channels. Without it, concurrent calls would cause a data race on . Alternatively, we can use a regular channel slice and protect is with a mutex : The number generates a number and sends it to each subscriber: Our implementation drops the message if the subscriber hasn't processed the previous one. So, always works quickly and doesn't block, but slower subscribers might miss some data. Alternatively, we can use a blocking without select to make sure everyone gets all the data, but this means the whole system will only run as fast as the slowest subscriber. The method terminates all subscriptions: Here's an example with three subscribers. Each one gets three random numbers: That's it for signals and broadcasting! Now let's look at a couple more tools from the package. Let's say we have a currency converter: Exchange rates are loaded from an external API, so we decided to fetch them lazily the first time is called: Unfortunately, this creates a data race on when used in a concurrent environment: We could protect the field with a mutex. Or we could use the type. It guarantees that a function called with runs only once: makes sure that the given function runs only once. If multiple goroutines call at the same time, only one will run the function, while the others will wait until it returns. This way, all calls to are guaranteed to proceed only after the map has been filled. is perfect for one-time initialization or cleanup in a concurrent environment. No need to worry about data races! Besides the type, the package also includes three convenience once-functions that you might find useful. Let's say we have the function that returns a random number: And the function sets the variable to a random number: It's clear that calling more than once will cause a panic (I'm keeping it simple and not using goroutines here): We can fix this by wrapping in . It returns a function that makes sure the code runs only once: wraps a function that returns a single value (like our ). The first time you call the function, it runs and calculates a value. After that, every time you call it, it just returns the same value from the first call: does the same thing for a function that returns two values: Here are the signatures of all the once-functions side by side for clarity: The functions , , and are shortcuts for common ways to use the type. You can use them if they fit your situation, or use directly if they don't. ✎ Exercise: Guess the average Practice is crucial in turning abstract knowledge into skills, making theory alone insufficient. The full version of the book contains a lot of exercises — that's why I recommend getting it . If you are okay with just theory for now, let's continue. The last tool we'll cover is . It helps reuse memory instead of allocating it every time, which reduces the load on the garbage collector. Let's say we have a program that: It looks something like this: If we run the benchmark: Here's what we'll see: Since we're allocating a new buffer on each loop iteration, we end up with 4000 memory allocations, using a total of 4 MB of memory. Even though the garbage collector eventually frees all this memory, it's quite inefficient. Ideally, we should only need 4 buffers instead of 4000 — one for each goroutine. That's where comes in handy: ➋ takes an item from the pool. If there are no available items, it creates a new one using ➊ (which we have to define ourselves, since the pool doesn't know anything about the items it creates). ➌ returns an item back to the pool. When the first goroutine calls during the first iteration, the pool is empty, so it creates a new buffer using . In the same way, each of the other goroutines create three more buffers. These four buffers are enough for the whole program. Let's benchmark: The difference in memory usage is clear. Thanks to the pool, the number of allocations has dropped by two orders of magnitude. As a result, the program uses less memory and puts minimal pressure on the garbage collector. Things to keep in mind: is a pretty niche tool that isn't used very often. However, if your program works with temporary objects that can be reused (like in our example), it might come in handy. We've covered some of the lesser-known tools in the package — condition variables ( ), one-time execution ( ), and pools ( ): Don't use these tools just because you know they exist. Rely on common sense. In the next chapter, we'll talk about atomics (coming soon). Pre-order for $10   or read online

0 views
Anton Zhiyanov 3 months ago

Expressive tests without testify/assert

Many Go programmers prefer using if-free test assertions to make their tests shorter and easier to read. So, instead of writing if statements with : They would use (or its evil twin, ): However, I don't think you need and its 40 different assertion functions to keep your tests clean. Here's an alternative approach. The testify package also provides mocks and test suite helpers. We won't talk about these — just about assertions. Equality • Errors • Other assertions • Source code • Final thoughts The most common type of test assertion is checking for equality: Let's write a basic generic assertion helper: We have to use a helper function, because the compiler doesn't allow us to compare a typed value with an untyped : Now let's use the assertion in our test: The parameter order in is (got, want), not (want, got) like it is in testify. It just feels more natural — saying "her name is Alice" instead of "Alice is her name". Also, unlike testify, our assertion doesn't support custom error messages. When a test fails, you'll end up checking the code anyway, so why bother? The default error message shows what's different, and the line number points to the rest. is already good enough for all equality checks, which probably make up to 70% of your test assertions. Not bad for a 20-line testify alternative! But we can make it a little better, so let's not miss this chance. First, types like and have an method. We should use this method to make sure the comparison is accurate: Second, we can make comparing byte slices faster by using : Finally, let's call from our function: And test it on some values: Works like a charm! Errors are everywhere in Go, so checking for them is an important part of testing: Error checks probably make up to 30% of your test assertions, so let's create a separate function for them. First we cover the basic cases — expecting no error and expecting an error: Usually we don't fail the test when an assertion fails, to see all the errors at once instead of hunting them one by one. The "unexpected error" case (want nil, got non-nil) is the only exception: the test terminates immediately because any following assertions probably won't make sense and could cause panics. Let's see how the assertion works: So far, so good. Now let's cover the rest of error checking without introducing separate functions (ErrorIs, ErrorAs, ErrorContains, etc.) like testify does. If is an error, we'll use to check if the error matches the expected value: Usage example: If is a string, we'll check that the error message contains the expected substring: Usage example: Finally, if is a type, we'll use to check if the error matches the expected type: Usage example: One last thing: doesn't make it easy to check if there was some (non-nil) error without asserting its type or value (like in testify). Let's fix this by making the parameter optional: Usage example: Now handles all the cases we need: And it's still under 40 lines of code. Not bad, right? and probably handle 85-95% of test assertions in a typical Go project. But there's still that tricky 5-15% left. We may need to check for conditions like these: Technically, we can use . But it looks a bit ugly: So let's introduce the third and final assertion function — . It's the simplest one of all: Now these assertions look better: Here's the full annotated source code for , and : Less than 120 lines of code! I don't think we need forty assertion functions to test Go apps. Three (or even two) are enough, as long as they correctly check for equality and handle different error cases. I find the "assertion trio" — Equal, Err, and True — quite useful in practice. That's why I extracted it into the github.com/nalgeon/be mini-package. If you like the approach described in this article, give it a try!

0 views
Anton Zhiyanov 3 months ago

Redka: Redis re-implemented with SQL

I'm a big fan of Redis. It's such an amazing idea to go beyond the get-set paradigm and provide a convenient API for more complex data structures: maps, sets, lists, streams, bloom filters, etc. I'm also a big fan of relational databases and their universal language, SQL. They've really stood the test of time and have proven to solve a wide range of problems from the 1970s to today. So, naturally, one day I decided to combine the two and reimplement Redis using a relational backend — first SQLite, then Postgres. That's how Redka was born. About Redka • Use cases • Usage example • Performance • Final thoughts Redka is a software written in Go. It comes in two flavors: Redka currently supports five core Redis data types: Redka can use either SQLite or PostgreSQL as its backend. It stores data in a database with a simple schema and provides views for better introspection. Here are some situations where Redka might be helpful: Embedded cache for Go applications . If your Go app already uses SQLite or just needs a built-in key-value store, Redka is a natural fit. It gives you Redis-like features without the hassle of running a separate server. You're not limited to just get/set with expiration, of course — more advanced structures like lists, maps, and sets are also available. Lightweight testing environment . Your app uses Redis in production, but setting up a Redis server for local development or integration tests can be a hassle. Redka with an in-memory database offers a fast alternative to test containers, providing full isolation for each test run. Postgres-first data structures . If you prefer to use PostgreSQL for everything but need Redis-like data structures, Redka can use your existing database as the backend. This way, you can manage both relational data and specialized data structures with the same tools and transactional guarantees. You can run the Redka server the same way you run Redis: Then use or any Redis client for your programming language, like , , , and so on: You can also use Redka as a Go package without the server: All data is stored in the database, so you can access it using SQL views: Redka is not about raw performance. You can't beat a specialized data store like Redis with a general-purpose relational backend like SQLite. However, Redka can still handle tens of thousands of operations per second, which should be more than enough for many apps. Here are the redis-benchmark results for 1,000,000 GET/SET operations on 10,000 randomized keys. Redka (SQLite): Redka (PostgreSQL): Redka for SQLite has been around for over a year, and I recently released a new version that also supports Postgres. If you like the idea of Redis with an SQL backend — feel free to try Redka in testing or (non-critical) production scenarios. See the nalgeon/redka repo for more details.

0 views