Video #240: Reliable Async Tests: 😳

Episode: Video #240 Date: Jul 3, 2023 Access: Members Only 🔒 URL: https://www.pointfree.co/episodes/ep240-reliable-async-tests

Description

We dive into Apple’s Async Algorithms package to explore some advanced usages of Swift’s concurrency runtime, including a particular tool we can leverage to bend the will of async code to our advantage in tests.

Video

Cloudflare Stream video ID: 8484fc27e12640c1948b3c284517b806 Local file: video_240_reliable-async-tests.mp4 *(download with --video 240)*

References

Transcript

— 0:05

So, that was a pretty deep overview of the current state-of-the-art when it comes to testing async code in Swift. We’re not going to mince words here: it can be incredibly frustrating to try to test features that make use of any asynchrony beyond some very simple constructs. And unfortunately we’re not really sure there is much work being done from the core Swift team to improve the situation.

— 0:24

And while what we have built is a pretty silly toy application, we promise that your application has these exact shapes somewhere. You are certainly making async network requests, and maybe you are juggling some state to let the view know when that request is in flight. And maybe you have some additional logic somewhere that determines when one should cancel that request if it is still in flight. And further, as async sequences become more and more ubiquitous in Apple’s APIs, and as Combine begins its slow march towards deprecation, you will be using async sequences a lot more.

— 0:54

As soon as you have some complex and nuanced logic involving asynchronous code you are in for a world of hurt when it comes to testing. You have no choice but to either not test all of your code, or sprinkle in some mega yields or sleeps just to get things passing, and hope it doesn’t flake on your CI server. Brandon

— 1:11

So, everything we have encountered so far is what pushed us to start a discussion on the Swift forums asking others how they are going about testing complex, async code. And unfortunately it seems that pretty much everyone is in the same boat. They either don’t test their complex async code, or at least not as deeply as they could, or they insert lots of yields and sleeps to force the concurrency runtime to push forward.

— 1:32

We wanted to find a better way, and around that time Apple had just opened sourced their new “async algorithms” package , and it had a couple of interesting advanced usages of Swift’s underlying concurrency runtime. We dug in a bit deeper and we saw that there was maybe something we could leverage to bend the will of the concurrency runtime to our advantage in tests.

— 1:55

So, let’s take a look at Apple’s code and see what clues it gives us in how we can better predict how Swift’s concurrency tools will behave at runtime. Async Algorithms’ and job enqueuing

— 2:06

I’ve got the Async Algorithms package opened up locally, and the first hint that Apple was doing some interesting with concurrency was in their test suite. They have a whole bunch of tests exercising the behavior of time-based asynchronous operators, such as debounce, throttle, buffer and more, and they used this particular syntax to do: final class TestDebounce: XCTestCase { func test_delayingValues() throws { validate { "abcd----e---f-g----|" $0.inputs[0] .debounce(for: .steps(3), clock: $0.clock) "------d----e-----g-|" } } … }

— 2:27

This is a diagrammatic verification that the debounce operator works as you expect. The first string represents an async sequence that emits some characters over time, and the dash represents a unit of time where nothing was emitted. Then that async sequence is transformed with the debounce operator, which debounced for 3 units of time. And then the second string represents what the debounced sequence emits over time.

— 3:05

In particular, if you send 4 events one after another, as shown in the first string with abcd , then the debounced async sequence will not emit at all, as shown in the second string with ---- . However, because the base sequence is debounced for 3 “steps” of the clock, if we wait for 3 more dashes in the input sequence, we will see that the output sequence finally emits the last value, d .

— 3:23

Further, if the base sequence emits e and we again wait 3 steps, then we will get the e . And on and on. We can run this test to see that it does pass, and does so nearly immediately: Test Suite 'Selected tests' passed. Executed 1 test, with 0 failures (0 unexpected) in 0.005 (0.007) seconds

— 3:54

And we can run it repeatedly to see that it seems to pass deterministically and extremely quickly: Test Suite 'Selected tests' passed. Executed 10000 tests, with 0 failures (0 unexpected) in 25.376 (27.490) seconds

— 4:07

This is super cool, but also how on earth does this work? It is somehow subscribing to an async sequence, asserting on its behavior in a completely deterministic fashion. But what we saw just a moment ago that writing tests for async sequences is incredibly difficult due to not knowing when the subscription actually starts, and not knowing how emissions will feed into the rest of our feature’s logic.

— 4:31

Well, let’s dive into the internals of this test to see what is going on. It starts with this validate helper: validate { … }

— 4:40

This is a tool that was built just for the tests in the Async Algorithms package that allows you to describe an input diagram, an async sequence to feed the input diagram into, and then a resulting output diagram that you want to assert again. It also uses @resultBuilder syntax under the hood, but that’s not important to us.

— 5:00

So, let’s dive into this validate function, and we find it just calls out to another validate function: public func validate<Test: AsyncSequenceValidationTest>( @AsyncSequenceValidationDiagram _ build: (AsyncSequenceValidationDiagram) -> Test, file: StaticString = #file, line: UInt = #line ) { validate( theme: .ascii, expectedFailures: [], build, file: file, line: line ) }

— 5:05

And so diving into that validate function shows the real meat of the helper : func validate< Test: AsyncSequenceValidationTest, Theme: AsyncSequenceValidationTheme >( theme: Theme, expectedFailures: Set<String>, @AsyncSequenceValidationDiagram _ build: (AsyncSequenceValidationDiagram) -> Test, file: StaticString = #file, line: UInt = #line ) { … }

— 5:10

After setting up a bunch of variables, the real action happens in the line that constructs the this AsyncSequenceValidationDiagram type : (result, failures) = try AsyncSequenceValidationDiagram .test(theme: theme, build)

— 5:21

Everything outside this single line is just recording failures, and so the real meat of the helper must be inside this test static method, so let’s dive into that: public static func test< Test: AsyncSequenceValidationTest, Theme: AsyncSequenceValidationTheme >( theme: Theme, @AsyncSequenceValidationDiagram _ build: (AsyncSequenceValidationDiagram) -> Test ) throws -> (ExpectationResult, [ExpectationFailure]) { … }

— 5:36

And finally we are seeing some really interesting things.

— 5:38

The first half of this method is devoted to validating and parsing the diagrams into first class data types that are easier to work with in Swift. The second half of the method is dedicated to actually running the async sequence, feeding the input diagram into the sequence, and accumulating the results that come out the other side so that it can assert against the expected diagram.

— 6:06

We can even see the exact moment the input sequence is subscribed to : let runner = Task { do { try await test.test( with: clock, activeTicks: activeTicks, output: test.output ) { event in actual.withCriticalRegion { values in values.append((clock.now, .success(event))) } } actual.withCriticalRegion { values in values.append((clock.now, .success(nil))) } } catch { actual.withCriticalRegion { values in values.append((clock.now, .failure(error))) } } }

— 6:25

And we can see it accumulating emissions into some array.

— 6:36

So far I’m not seeing anything that looks too different from what we tried to do when writing tests for async code. This code is even spinning up a new unstructured task in order to perform the async work.

— 6:46

What is different, however, is what comes just before spinning up this task. It does some really strange things with a bunch of global state : let actual = ManagedCriticalState( (Clock.Instant, Result<String?, Error>) ) Context.clock = clock Context.specificationFailures.removeAll() // This all needs to be isolated from potential Tasks // (the caller function might be async!) Context.driver = TaskDriver(queue: diagram.queue) { driver in swift_task_enqueueGlobal_hook = { job, original in Context.driver?.enqueue(job) } … }

— 6:56

This Context type has a bunch of statics on it, which are really just globals that are slightly namespaced inside a type. And a bunch of globals are being mutated, such as a clock and something called a “driver”.

— 7:13

But then there’s a global being mutated that is actually a file scope global: swift_task_enqueueGlobal_hook = { job, original in Context.driver?.enqueue(job) }

— 7:16

It’s called swift_task_enqueueGlobal_hook , and whenever it is invoked it redirects the action to the Context.driver to enqueue a job, whatever that means.

— 7:35

We can ⌘+click on that symbol to jump to its definition. And we end up in a C header file that is included with the project over in the _CAsyncSequenceValidationSupport module: typedef struct _Job* JobRef; typedef SWIFT_CC(swift) void ( *swift_task_enqueueGlobal_original )( JobRef _Nonnull job ); SWIFT_EXPORT_FROM(swift_Concurrency) SWIFT_CC(swift) void ( * _Nullable swift_task_enqueueGlobal_hook )( JobRef _Nonnull job, swift_task_enqueueGlobal_original _Nonnull original );

— 7:41

This cryptic C code is actually describing the signature of a callback function that does not return anything, void , and takes two arguments. From the name it seems to be some kind of hook that is invoked when a new task is enqueued. The first argument argument is the job being enqueued, and the second argument is the original enqueue hook, whatever that means.

— 8:33

Further, these mystical looking signatures directly correspond to actual signatures in a C header file in the Swift compiler. For example, we can open up the Swift compiler code base, search for swift_task_enqueueGlobal_hook , and we will find : SWIFT_CC(swift) void (*swift_task_enqueueGlobal_hook)( Job *job, swift_task_enqueueGlobal_original original);

— 9:00

So, what’s going on here is that the Async Algorithms package wants access to some C functions that are defined in Swift’s C++ codebase, but are not exposed with a proper Swift API. In order to do that the package defines a system library that publicly declares those signatures, and that makes it available in Swift.

— 9:19

So, when we do something like this in Swift: swift_task_enqueueGlobal_hook = { job, original in Context.driver?.enqueue(job) }

— 9:29

…we are actually overwriting the global hook in the C++ code.

— 9:34

But what is this hook? Well, according the comment right above the code in the Swift code base: /// A hook to take over global enqueuing.

— 9:41

It seems that whenever an async task is enqueued it calls this closure for processing. And in this case “task” does not refer to a literal unstructured task created with the Task type. Here “task” means any kind of asynchronous work, including every single suspension point from an await .

— 10:02

We can even see how it’s used in GlobalExecutor.cpp : void swift::swift_task_enqueueGlobal(Job *job) { _swift_tsan_release(job); concurrency::trace::job_enqueue_global(job); if (swift_task_enqueueGlobal_hook) swift_task_enqueueGlobal_hook( job, swift_task_enqueueGlobalImpl ); else swift_task_enqueueGlobalImpl(job); }

— 10:15

Here it detects if there is a global hook, and if so invokes it, and otherwise it just enqueues it directly.

— 10:27

This is what gives the Async Algorithms package its super powers when it comes to testing. It completely takes control over how async jobs are enqueued by the system in order to control the enter runtime: swift_task_enqueueGlobal_hook = { job, original in Context.driver?.enqueue(job) } Accessing the enqueue global hook

— 10:40

This is exactly what we need for testing our async code in a more deterministic fashion. We would like to forgo all of the fancy scheduling that happens in the default, global executor, and instead serialize everything so that the first job enqueued gets priority to run for as long as it wants and prevents any other job from starting.

— 10:57

Then, once the first job finishes or suspends for any reason, that will give the next job in the queue a chance to execute. And on and on and on.

— 11:06

So, how can we do that? Stephen

— 11:07

Well, we could create a new module in our project for the C header, and then use that to get access to the hook just like they do in the Async Algorithms package. But unfortunately that doesn’t work great in practice. It worked out OK for the Async Algorithms package because the C module is just used internally and isn’t exposed to the outside at all.

— 11:24

But, if we wanted to use this enqueue hook as a tool in a library that we can open source so others can make use of it, then you will quickly run into problems due to bugs in SPM and Xcode. In particular, when an SPM package includes a C module and is imported in tests, the tests must also explicitly link to the C module, which is not even possible if the package doesn’t export it, but worse, this extra linking leads to duplicate symbol issues.

— 11:48

Luckily, there’s another way, and it came to us thanks to Kabir Oberai . It’s a little less efficient than the C header style, but that doesn’t really matter since this is just test code anyway.

— 11:56

What we will do is use dlsym and dlopen to find the swift_task_enqueueGlobal_hook symbol and load it dynamically.

— 12:04

The way to dynamically open libraries in Swift is to start with dlopen , a C function, which means we can even see its documentation from the man pages in terminal: $ man dlopen

SYNOPSIS 12:26

The path is where we want to search for the dynamic library, which we can use nil if we want to search all paths, and the mode allows us to pass along various options for searching. The man pages suggest using RTLD_LAZY as a default, so that is what we will do: dlopen(nil, RTLD_LAZY)

SYNOPSIS 13:24

This returns an UnsafeMutableRawPointer? , which can be used to look up the address of a particular symbol using another C function, dlsym . Again we can refer to the man pages of this function to get some more information:

SYNOPSIS 13:42

It takes a handle as the first argument, which is the thing returned to us from dlopen , and the symbol we want to search for as the second argument, which is the swift_task_enqueueGlobal_hook callback that we saw in the C header for Async Algorithms and saw in the C++ codebase for Swift: dlsym(dlopen(nil, 0), "swift_task_enqueueGlobal_hook")

SYNOPSIS 14:18

This returns a UnsafeMutableRawPointer! , which is a pointer to the actual callback hook symbol. We can transform the raw pointer into something with actual Swift types, making it much more useful.

SYNOPSIS 14:32

Let’s first specify the actual Swift types of the hook. First, there is the “original” argument, which is the original enqueuing function that can be used to enqueue jobs. It is a function that takes an UnownedJob and returns nothing: typealias Original = @convention(thin) (UnownedJob) -> Void

SYNOPSIS 15:07

And then the signature of the hook is something that takes an UnownedJob as well as the original callback before overriding: typealias Hook = @convention(thin) (UnownedJob, Original) -> Void

SYNOPSIS 15:26

And with those defined we can now construct a typed pointer to the swift_task_enqueueGlobal_hook symbol: dlsym( dlopen(nil, 0), "swift_task_enqueueGlobal_hook" ) .assumingMemoryBound(to: Hook?.self)

SYNOPSIS 15:43

And let’s bundle this up in a variable: private let _swift_task_enqueueGlobal_hook = dlsym( dlopen(nil, 0), "swift_task_enqueueGlobal_hook" ) .assumingMemoryBound(to: Hook?.self)

SYNOPSIS 16:03

And finally we can expose an overridable version of the hook that simply mutates the pointer under the hood: var swift_task_enqueueGlobal_hook: Hook? { get { _swift_task_enqueueGlobal_hook.pointee } set { _swift_task_enqueueGlobal_hook.pointee = newValue } }

SYNOPSIS 16:38

OK, we now have access to the dynamically loaded swift_task_enqueueGlobal_hook symbol, but can we do with it?

SYNOPSIS 16:44

Well, we can completely replace it with with a whole new function to do custom enqueuing logic. Let’s play around with this by starting up a new test method: func testEnqueueHook() async { }

SYNOPSIS 17:02

And then overriding swift_task_enqueueGlobal_hook with a new closure that takes two arguments: swift_task_enqueueGlobal_hook = { job, original in } As mentioned a moment ago, the first argument represents the job being enqueued, and the second argument represents the original enqueue-r, from before we overrode the hook.

SYNOPSIS 17:15

Just to do something silly, let’s not do anything in this hook. So, when the Swift runtime asks us to enqueue a job we just won’t do anything at all. And right after overriding the hook let’s create a suspension point by yielding: func testEnqueueHook() async throws { swift_task_enqueueGlobal_hook = { job, original in } await Task.yield() }

SYNOPSIS 17:30

When we run this test we will see it hangs forever. And that makes sense. Performing an await created a suspension point, which enqueued a job, but then we never actually did anything with the job and so it never finished.

SYNOPSIS 17:52

We can even put a print statement in the hook to confirm that it is indeed being called: func testEnqueueHook() async throws { swift_task_enqueueGlobal_hook = { job, original in print("Job enqueued") } await Task.yield() }

SYNOPSIS 18:00

This does print to the console, twice in fact. The reason for this is that we actually have two versions of our code running right now. Once in the test and then also secretly in the background a simulator has been started and is actually running the entry point of our app. That is what is accounting for this extra enqueued job.

SYNOPSIS 18:18

Let’s quickly make it so that our app doesn’t execute anything at all if we are running in tests. The easiest way to do that is to simply check if it’s possible to access the XCTestCase class: @main struct ReliablyTestingAsyncApp: App { var body: some Scene { WindowGroup { if NSClassFromString("XCTestCase") == nil { ContentView(model: NumberFactModel()) } } } }

SYNOPSIS 18:49

Now when we run the test method again we see that only a single job is enqueued.

SYNOPSIS 19:01

But now let’s actually do something in the hook. If we don’t want to do any custom enqueuing logic we can just pass the buck onto the original enqueue-r: swift_task_enqueueGlobal_hook = { job, original in print("Job enqueued") original(job) }

SYNOPSIS 19:12

Let’s also put a print in here so that we can see anytime a job is enqueued: swift_task_enqueueGlobal_hook = { job, original in print("Job enqueued", job) original(job) }

SYNOPSIS 19:18

Now any time we perform an asynchronous task, whether it be spinning up a new unstructured task or simply awaiting some async function, this hook will be invoked.

SYNOPSIS 19:26

For example, the simple act of just yielding: func testEnqueueHook() async throws { swift_task_enqueueGlobal_hook = { job, original in print("Job enqueued", job) original(job) } await Task.yield() }

SYNOPSIS 19:30

…causes two jobs to be enqueued: Job enqueued UnownedJob(id: nil) Job enqueued UnownedJob(id: 2)

SYNOPSIS 19:35

The job argument is unfortunately very opaque and doesn’t give us much information. And we can’t be exactly sure why 2 jobs are enqueued. Something about Task.yield causes two jobs to be enqueued. But we are now seeing in very real terms how we can tap into the moment any async unit of work is enqueued.

SYNOPSIS 19:57

Instead of doing a simple yield, let’s spin up a new unstructured task and then await it: func testEnqueueHook() async throws { swift_task_enqueueGlobal_hook = { job, original in print("Job enqueued", job) original(job) } let task = Task { print("Hello!") } await task.value }

SYNOPSIS 20:14

Now we see two jobs enqueued: Job enqueued UnownedJob(id: 3) Hello! Job enqueued UnownedJob(id: 2)

SYNOPSIS 20:24

Most likely the first corresponds to the creation of the unstructured task and the second corresponds to awaiting the task, but also who knows? That entire process is quite opaque.

SYNOPSIS 20:38

Let’s also try the situation where we have an external async function and then we invoke it by awaiting: func testEnqueueHook() async throws { swift_task_enqueueGlobal_hook = { job, original in print("Job enqueued", job) original(job) } func hello() async {} await hello() }

SYNOPSIS 20:53

Interestingly no job is enqueued. It seems that Swift can detect that the hello function doesn’t actually do any async work and is able to elide the work of enqueuing a job. So that’s cool.

SYNOPSIS 21:11

But now let’s actually do some real async work. Let’s use the new bytes async sequence on URLSession : func testEnqueueHook() async throws { swift_task_enqueueGlobal_hook = { job, original in print("Job enqueued", job) original(job) } let (bytes, _) = try await URLSession.shared.bytes( from: URL(string: "https://www.google.com")! ) for try await _ in bytes {} }

SYNOPSIS 21:29

This produces an async sequences that emits for every single byte downloaded from the URL. It is literally every single, individual byte. So if the webpage is a mega byte, then this sequence emits 1,000,000 times.

SYNOPSIS 21:42

If we run this we see that the job prints a number of times, about 7 times. We’re not sure why 7. We can even print the thread inside the for loop to see which thread we are on while processing the byte: for try await _ in bytes { print({ Thread.current }()) }

SYNOPSIS 21:53

Even more interesting, if we make the test method @MainActor : @MainActor func testEnqueueHook() async throws { … }

SYNOPSIS 22:00

Then we will see a whole bunch of jobs created. In fact, one is created for each byte processed. We have no idea why, but that’s the way it is, and it’s pretty cool.

SYNOPSIS 22:20

It’s also worth noting that the job enqueue hook is only called when dealing with Swift’s native concurrency tools. It is not called when using other forms of concurrency, such as direct access to threads or dispatch queues.

SYNOPSIS 22:32

For example, the following test runs without crashing: func testGlobalDispatchQueue() { swift_task_enqueueGlobal_hook = { job, original in fatalError() } DispatchQueue.global().sync { } }

SYNOPSIS 22:46

Showing that enqueuing a unit of work on a dispatch queue does not involve the global enqueue hook at all. Next time: understanding the hook

SYNOPSIS 22:58

OK, so we have now seen that with a little bit of runtime hackery we can get access to a “task enqueue global hook” that is completely public and even apart of Swift’s ABI, but just isn’t easily accessible. But, once we do have a handle on the hook, then we have the opportunity to be notified anytime an async unit of work is scheduled to be performed.

SYNOPSIS 23:15

So, what can we do with this? Brandon

SYNOPSIS 23:17

Well, we can completely alter the manner in which jobs are enqueued. Right now we are just turning around and feeding the job back to the original enqueuer, which means the behavior is unchanged from the default. But we do have the ability to send those jobs to a different executor, such as a custom executor, or even serialize all the jobs onto the main executor.

SYNOPSIS 23:37

Let’s take a look…next time! References swift-async-algorithms Apple • Jan 12, 2022 A package of asynchronous sequence and advanced algorithms that involve concurrency, along with their related types. http://github.com/google/swift-benchmark Reliably testing code that adopts Swift Concurrency? Brandon Williams & Stephen Celis • May 13, 2022 A Swift Forums post from yours truly about the difficulty of testing async code in Swift. https://forums.swift.org/t/reliably-testing-code-that-adopts-swift-concurrency/57304 Concurrency Extras Brandon Williams and Stephen Celis • Jul 18, 2023 An open source Swift package that provides powerful tools for making async code easier to test. https://github.com/pointfreeco/swift-concurrency-extras Announcing Concurrency Extras: Useful, testable Swift concurrency. Brandon Williams and Stephen Celis • Jul 18, 2023 The announcement of our new Concurrency Extras library, which provides powerful tools for making async code easier to test. https://www.pointfree.co/blog/posts/109-announcing-concurrency-extras-useful-testable-swift-concurrency Downloads Sample code 0240-reliably-testing-async-pt3 Point-Free A hub for advanced Swift programming. Brought to you by Brandon Williams and Stephen Celis . Content Become a member The Point-Free Way Beta previews Gifts Videos Collections Free clips Blog More About Us Community Slack Mastodon Twitter BlueSky GitHub Contact Us Privacy Policy © 2026 Point-Free, Inc. All rights are reserved for the videos and transcripts on this site. All other content is licensed under CC BY-NC-SA 4.0 , and the underlying source code to run this site is licensed under the MIT License .