Video #241: Reliable Async Tests: 🥹

Episode: Video #241 Date: Jul 10, 2023 Access: Members Only 🔒 URL: https://www.pointfree.co/episodes/ep241-reliable-async-tests

Description

We continue our deep dive into advanced Swift concurrency by exploring the concept of “executors” to better understand its global enqueue hook. We will then put this hook to work and finally have reliable, 100% deterministic async tests.

Video

Cloudflare Stream video ID: f0efb38dbbab88b8f6b711801aa5bd25 Local file: video_241_reliable-async-tests.mp4 *(download with --video 241)*

References

Transcript

— 0:05

OK, so we have now seen that with a little bit of runtime hackery we can get access to a “task enqueue global hook” that is completely public and even apart of Swift’s ABI, but just isn’t easily accessible. But, once we do have a handle on the hook, then we have the opportunity to be notified anytime an async unit of work is scheduled to be performed.

— 0:22

So, what can we do with this? Brandon

— 0:24

Well, we can completely alter the manner in which jobs are enqueued. Right now we are just turning around and feeding the job back to the original enqueuer, which means the behavior is unchanged from the default. But we do have the ability to send those jobs to a different executor, such as a custom executor, or even serialize all the jobs onto the main executor.

— 0:45

Let’s take a look. Understanding the hook

— 0:48

There is a protocol in Swift called Executor that doesn’t get much attention in the community, and that’s because it’s not really useful right now, or even really documented at all: /// A service that can execute jobs. @available( macOS 10.15, iOS 13.0, watchOS 6.0, tvOS 13.0, * ) public protocol Executor: AnyObject, Sendable { func enqueue(_ job: UnownedJob) }

— 0:59

It is an extremely opaque protocol and there isn’t even much information about it in the various Swift evolution proposals that outlined the vision for Swift’s concurrency tools. There are some hand wavy mentions of it being a future area of development, and there is even an active proposal right now to expose a bit more of the executor underpinnings of actors, but still…they are pretty mystical as of right now.

— 1:23

There isn’t even a single known, public conformance to the protocol at the time of recording this episode. But, we can quickly pretend like we have an executor available to us by defining a quick generic function that takes some Executor conformance: swift_task_enqueueGlobal_hook = { job, original in func enqueue(executor: some Executor) { } }

— 2:10

And this at least lets us see what we could do with an executor if we had one available.

— 2:21

It looks like the only thing we can do is enqueue jobs: executor.enqueue(<#job: UnownedJob#>)

— 2:29

And it just so happens this job is the same type that is handed to our hook closure, so I can even just pass it along the executor: swift_task_enqueueGlobal_hook = { job, original in func enqueue(executor: some Executor) { executor.enqueue(job) } }

— 3:02

So, if we did have an executor, we could feed jobs to it in the hook.

— 3:08

But, like I said a moment ago, it doesn’t seem that there are any public conformances to the Executor available, so how can we get an executor?

— 3:16

Well, global actors, such as the main actor, are supposed to have an executor associated with them, which we can see by looking at the GlobalActor protocol definition: @available( macOS 10.15, iOS 13.0, watchOS 6.0, tvOS 13.0, * ) public protocol GlobalActor { associatedtype ActorType: Actor static var shared: Self.ActorType { get } static var sharedUnownedExecutor: UnownedSerialExecutor { get } }

— 3:30

There is this thing called a UnownedSerialExecutor exposed, but interestingly it doesn’t even conform to the Executor protocol despite its name: @available( macOS 10.15, iOS 13.0, watchOS 6.0, tvOS 13.0, * ) @frozen public struct UnownedSerialExecutor: Sendable { @inlinable public init<E: SerialExecutor>( ordinary executor: E ) }

— 3:50

All we are seeing here is that Swift is trying very hard to keep the concept of executor incredibly opaque. It does not want to expose those details yet, partly because some of those details haven’t gone through Swift evolution yet, and also partly because they need to wait until other tools fall into place, such as ownership of types.

— 4:08

In the future, once everything has fallen into place, you will be able to access the underlying executor for each global actor, and even regular actor, and it won’t even have these weird “unowned” prefixes thanks to the ownership machinery.

— 4:23

But, until then, there is exactly one public mechanism we get in order to enqueue something onto an executor without even referring to an executor. There is a global actor that we are probably all very familiar with, which is the main actor: MainActor

— 4:40

This is the object we use whenever we want to force an asynchronous context to execute on the main thread, which is required for making changes to most kinds of UI state.

— 4:49

The MainActor type is a singleton, and the only way to get access to a value of the type is through the shared static property: MainActor.shared

— 4:54

There is also a unownedExecutor property: MainActor.shared.unownedExecutor …but as we mentioned a moment ago this is not a real executor and there is nothing we can do with it.

— 5:03

However, there is something we can do with the shared main actor instance: MainActor.shared.enqueue(<#job: UnownedJob#>)

— 5:07

This looks very executor-like in that we can enqueue unowned jobs, but it’s not really from the executor interface. It’s just a method on MainActor to mimic the executor interface, and under the hood it does call out to an executor, but all of that is hidden from us in the C++ code.

— 5:40

So, what were to happen if we enqueued the job from the hook onto the main executor? swift_task_enqueueGlobal_hook = { job, original in MainActor.shared.enqueue(job) }

— 5:53

At this point we aren’t even using the original enqueuer so we can ignore it: swift_task_enqueueGlobal_hook = { job, _ in MainActor.shared.enqueue(job) } Deterministic async tests

— 5:56

What does this even mean?

— 5:58

We have completely hijacked the default, global manner in which async jobs are submitted to the concurrency runtime and forced all jobs to instead be enqueued on the main executor, which is backed by a single thread, and so has no choice but to serialize all jobs.

— 6:13

But what does that even mean?

— 6:15

Well, let’s try overriding the global enqueue hook to be the main executor in some of the tests we wrote in previous episodes and see how it affects things. Let’s start with the very basic tests we wrote in order to explore the order that tasks start executing and how their executions interleave. Long story short: there isn’t much we can do to predict the inner workings of Swift’s concurrency runtime.

— 6:38

Even something as simple as trying to figure out if a task’s body can start executing before the line right after the task didn’t have a deterministic answer. For a very small percentage of runs the body could run first. We had the following test to explore this: func testTaskStart() async { let values = LockIsolated<[Int]>([]) let task = Task { values.withValue { $0.append(1) } } values.withValue { $0.append(2) } await task.value XCTAssertEqual(values.value, [2, 1]) }

— 7:20

Let’s copy-and-paste this test and override the global enqueue hook before running the main logic of the test: func testTaskStart_MainExecutor() async { swift_task_enqueueGlobal_hook = { job, _ in MainActor.shared.enqueue(job) } let values = LockIsolated<[Int]>([]) let task = Task { values.withValue { $0.append(1) } } values.withValue { $0.append(2) } await task.value XCTAssertEqual(values.value, [2, 1]) }

— 7:44

This still passes, but also the other version did pass. At least for the vast majority of times.

— 7:54

But when we ran the other test many times, like say 1,000 times, we found that a handful of times it would fail. So let’s try similar with this test and run it 10,000 times: Test Suite 'Selected tests' passed. Executed 10000 tests, with 0 failures (0 unexpected) in 15.552 (20.193) seconds

— 8:05

That still passes!

— 8:13

I’m starting to think that this is now actual deterministic behavior. Note that creating an unstructured Task does not create a suspension point. The initializer of Task is just a regular, synchronous API. That means the act of creating a Task will enqueue a job, which thanks to our hook will put it at the end of a serial queue, and that means the executor has no choice but to execute the line after the Task next. It can’t possibly jump to executing lines in the task because there is no suspension point. And only once we create a suspension point is the task allowed to start executing.

— 9:06

So this is starting to look pretty great. We are already seeing a way to predict how the concurrency runtime will execute our code, at least for the purposes of tests.

— 9:17

There’s another interesting aspect that we can see if we put a breakpoint inside the Task in each of these tests and run again, we will find that in the original test the task is executing in the global executor thread pool, whereas in the new test it is being executed on the main thread. This is happening without any mention of @MainActor . The test class is not annotated with @MainActor , the test method is not @MainActor and the task itself is not annotated with @MainActor . All that we have done is overridden the global enqueue hook to execute all jobs on the main executor, and that was enough to get all async work in this test executing on the main thread.

— 9:54

Let’s move onto the next test we wrote which tried to determine the order that async tasks start executing: func testTaskStartOrder() async { let values = LockIsolated<[Int]>([]) let task1 = Task { values.withValue { $0.append(1) } } let task2 = Task { values.withValue { $0.append(2) } } _ = await (task1.value, task2.value) XCTAssertEqual(values.value, [1, 2]) }

— 10:19

This failed about 0.5% of the time, meaning that the second task created actually started executing first somehow. So we cannot depend on the order that tasks are started. However, the tasks themselves are enqueued in a very specific and well-defined order. task1 was definitely enqueued before task2 even if due to the complexities of the concurrency runtime and global executor that the second task somehow started executing before the first.

— 10:29

So, let’s see what happens when we again copy-and-paste the test and override the global enqueue hook: func testTaskStartOrder_MainExecutor() async { swift_task_enqueueGlobal_hook = { job, _ in MainActor.shared.enqueue(job) } let values = LockIsolated<[Int]>([]) let task1 = Task { values.withValue { $0.append(1) } } let task2 = Task { values.withValue { $0.append(2) } } _ = await (task1.value, task2.value) XCTAssertEqual(values.value, [1, 2]) }

— 10:47

The test seems to pass, and even more amazingly, it passes consistently even when running 10,000 times: Test Suite 'Selected tests' passed. Executed 10000 tests, with 0 failures (0 unexpected) in 11.497 (22.454) seconds

— 11:00

I’m feeling very confident that this test now just passes 100% of the time, deterministically.

— 11:07

So, why does this seem to pass consistently whereas the original did not? Well, the global executor manages an entire thread pool that it can dole work out to, and it employs a complex mechanism to figure out how and when to assign tasks to various threads. That means that although there is a well-defined enqueuing order, there is not a well-defined execution order.

— 11:40

However, by collapsing all of the enqueuing behavior to just a single executor that manages a single thread, we get to remove all of that uncertainty and complexity. The main executor has no choice but to allow a task to run on the main thread until it suspends, and at that moment the next job in the queue gets a chance to execute, and on and on.

— 12:02

So, we can now predict exactly how tasks start executing, which is pretty amazing.

— 12:07

Let’s move onto the next task, which also shows that the order tasks start executing is not predictable, but this time for a task group: func testTaskGroupStartOrder() async { let values = await withTaskGroup( of: [Int].self ) { group in for index in 1...100 { group.addTask { [index] } } return await group.reduce(into: [], +=) } XCTAssertEqual(values, [1, 2]) }

— 12:25

Let’s copy-and-paste it and override the global enqueue hook: func testTaskGroupStartOrder_MainExecutor() async { swift_task_enqueueGlobal_hook = { job, _ in MainActor.shared.enqueue(job) } let values = await withTaskGroup( of: [Int].self ) { group in for index in 1...100 { group.addTask { [index] } } return await group.reduce(into: [], +=) } XCTAssertEqual(values, Array(1...100)) }

— 12:46

And just like that the test now passes deterministically, even when run 10,000 times: Test Suite 'Selected tests' passed. Executed 10000 tests, with 0 failures (0 unexpected) in 42.211 (43.654) seconds

— 12:54

This is really amazing because we previously saw that this test failed with a roughly 80% rate. The act of spinning up so many child tasks very rarely led to a situation where tasks started up in the order they were created. So even the starting order of task groups can be made predictable if we override the global enqueue hook.

— 13:05

Let’s look at the last, and most complex test we wrote: func testYieldScheduling() async { let count = 10 let values = LockIsolated<[Int]>([]) let tasks = (0...count).map { index in Task { values.withValue { $0.append(index * 2) } await Task.yield() values.withValue { $0.append(index * 2 + 1) } } } for task in tasks { await task.value } XCTAssertEqual( values.value, [ 0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21 ] ) } This wanted to get some insight in how scheduling works in a task. For example, if you spin up multiple tasks, each performing work, and one suspends, like say due to a yield, how can we predict the next task that will start executing.

— 13:33

This test of course failed the vast majority of times because all of the tasks are running in parallel, and there is just no way to predict how suspension points will be scheduled on the global executor.

— 13:45

Well, let’s copy-and-paste it and override the global enqueue hook yet again: func testYieldScheduling_MainExecutor() async { swift_task_enqueueGlobal_hook = { job, _ in MainActor.shared.enqueue(job) } let count = 10 let values = LockIsolated<[Int]>([]) let tasks = (0...count).map { index in Task { values.withValue { $0.append(index * 2) } await Task.yield() values.withValue { $0.append(index * 2 + 1) } } } for task in tasks { await task.value } XCTAssertEqual( values.value, [ 0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21 ] ) }

— 14:06

Incredibly it passes! We can even run it 10,000 times and it passes 100% of the time: Test Suite 'Selected tests' passed. Executed 10000 tests, with 0 failures (0 unexpected) in 14.911 (25.604) seconds

— 14:09

I am starting to feel very confident that we can predict task execution order now. It’s pretty incredible. Let’s really spell out explicitly how this process is working. Assume that we only spin up 3 tasks instead of the 10 that we are doing above. When the test first starts up and all the tasks are created, we have the following job queue: /* [task0, task1, task2] */

— 14:50

That is, 3 tasks have been enqueued and are ready to start executing.

— 14:54

Then the tasks start executing, but since jobs are executed on a serial executor we can only process one at a time. So, we take task 0 and start executing it, which causes a 0 to be appended to the shared mutable array: /* [{task0}, task1, task2] => [0] */

— 15:10

Then task 0 encounters a yield, so it suspends and allows other tasks to execute. We can picture this has putting task 0 at the end of the job queue because it will only be picked up once all the other tasks are done executing: /* [task1, task2, task0] => [0] */

— 15:23

Then task 1 starts executing, causing a 2 to be appended to the shared mutable state: /* [{task1}, task2, task0] => [0, 2] */

— 15:36

Then task 1 encounters a yield, so it suspends and allows other tasks to execute. Again we can picture this as putting task 1 at the end of the job queue since it will only start again once all the other tasks are done executing: /* [task2, task0, task1] => [0, 2] */

— 15:42

And then we come to task 2, and the same story plays out again. It beings executing and a 4 is appended to the shared mutable array: /* [{task2}, task0, task1] => [0, 2, 4] */

— 15:52

Then it encounters a yield, causing it to be pushed to the end of the queue: /* [task0, task1, task2] => [0, 2, 4] */

— 15:59

Now we are essentially back to the original queue of jobs, but this time each task in the middle of executing. Task 0 is going to be resumed to start executing the work that comes after the yield, in particular it will append a 1 to the shared mutable state: /* [{task0}, task1, task2] => [0, 2, 4, 1] */

— 16:23

And then that task finishes, and so it is removed from the queue: /* [task1, task2] => [0, 2, 4, 1] */

— 16:29

Then task 1 starts up again, and it appends a 3 to the shared mutable array: /* [{task1}, task2] => [0, 2, 4, 1, 3] */

— 16:36

And then it finishes and is removed from the queue: /* [task2] => [0, 2, 4, 1, 3] */

— 16:39

And then finally task 2 starts again, appending a 5 to the array: /* [{task2}] => [0, 2, 4, 1, 3, 5] */

— 16:43

…before being removed from the queue: /* [] => [0, 2, 4, 1, 3, 5] */

— 16:45

This explains why all the even integers are appended to the array and then all the odd integers. And this form of scheduling makes perfect sense, and is fully deterministic.

— 17:03

One interesting outcome of a deep dive into this particular tricky situation is that when we use this main serial executor for our job enqueuer, then we can interpret Task.yield as putting the current job at the end of the queue. That gives all currently enqueued jobs a chance to execute before the task is resumed again. And that’s pretty cool. Deterministic feature tests

— 17:39

We have now seen that by overriding the global task enqueue hook we can take control over how async jobs are executed in Swift. By diverting all jobs to the main serial executor we get the ability to predict how jobs are enqueued, interleave and execute, and that made it possible to write 100% deterministic, passing tests. Stephen

— 18:02

However, those tests were quite simple, and more of a toy than anything. Let’s apply this tool to the feature we built in previous episodes that had some actual complex behavior to it. It performs a network request, manages some state based on the lifecycle of that request, allows cancelling the network request while it is in flight, and even manages an async sequence. Can we finally write deterministically passing tests for this feature?

— 18:29

Let’s start with the first async test that had a flakiness problem. It’s the one where we try to confirm that immediately after the “Get fact” button is tapped that the current fact is cleared out: func testFactClearsOut() async throws { let fact = AsyncStream.makeStream(of: String.self) let model = NumberFactModel( numberFact: NumberFactClient( fact: { _ in await fact.stream.first(where: { _ in true })! } ) ) model.fact = "An old fact about 0." let task = Task { await model.getFactButtonTapped() } await Task.yield() XCTAssertEqual(model.fact, nil) fact.continuation.yield("0 is a good number.") await task.value XCTAssertEqual(model.fact, "0 is a good number.") }

— 18:43

This test failed a small number of times because sometimes the Task.yield is not enough for the getFactButtonTapped method to actually start executing and so the fact state is not yet cleared out.

— 19:10

Let’s copy-and-paste the test and force it to execute on the serial main executor: func testFactClearsOut_MainSerialExecutor() async throws { swift_task_enqueueGlobal_hook = { job, _ in MainActor.shared.enqueue(job) } let fact = AsyncStream.makeStream(of: String.self) let model = withDependencies { $0.numberFact.fact = { _ in await fact.stream.first(where: { _ in true })! } } operation: { NumberFactModel() } model.fact = "An old fact about 0." let task = Task { await model.getFactButtonTapped() } await Task.yield() XCTAssertEqual(model.fact, nil) fact.continuation.yield("0 is a good number.") await task.value XCTAssertEqual(model.fact, "0 is a good number.") }

— 19:39

With no other changes applied this test passes, even when run 1,000 times. Test Suite 'Selected tests' passed. Executed 1000 tests, with 0 failures (0 unexpected) in 2.575 (2.910) seconds

— 19:52

We can even run it 10,000 times and it still passes: Test Suite 'Selected tests' passed. Executed 10000 tests, with 0 failures (0 unexpected) in 18.629 (22.745) seconds

— 20:09

This now passes deterministically, and we can even reason about its execution flow. When the unstructured task is created to invoke the getFactButtonTapped method, a new job is enqueued to the end of the main serial executor. That job will not start executing until the current job suspends. That is exactly what happens when we perform Task.yield . It pushes the current job to the end of the queue.

— 20:37

Then the getFactButtonTapped task starts executing, causing the fact state to flip to nil , and it runs until it meets a suspension point, which happens when we make the request for the fact. At that moment the test task resumes, and now our assertion that fact is nil passes just fine.

— 20:55

And to round things off we yield the stream that is holding up the fetch request over in the model, and that allows us to finally assert that the fact state updates to its final value of “0 is a good number.”

— 21:18

So, that is pretty impressive, but we can even make this better thanks to our ability to predict how jobs execute. We no longer need to construct an async stream just so that we can feed data to the model at a later time. We can just return data straight from the fact endpoint, and hopefully everything works: func testFactClearsOut_MainSerialExecutor() async throws { swift_task_enqueueGlobal_hook = { job, _ in MainActor.shared.enqueue(job) } let model = withDependencies { $0.numberFact.fact = { "\($0) is a good number." } } operation: { NumberFactModel() } model.fact = "An old fact about 0." let task = Task { await model.getFactButtonTapped() } await Task.yield() XCTAssertEqual(model.fact, nil) await task.value XCTAssertEqual(model.fact, "0 is a good number.") }

— 21:56

The test still passes, does so 100% of the time, and the code is much simpler. We no longer need to juggle a stream and continuation just to hold up the fact endpoint so that we can wiggle ourselves in between the various steps of our feature’s logic.

— 22:25

Let’s move onto the next test. It tests something similar to the previous one, except now it wants to check that while the fact request is in flight that the isLoading boolean is true, and then flips to false when the request finishes. This is an important test to have because it can help us catch bugs that may cause the loading indicator to stay on the screen forever.

— 22:48

It’s a great test to have, but unfortunately it would fail a small percentage of the time because the Task.yield was not enough for the model to update its internal state to show that it was in a loading state.

— 22:59

Let’s copy-and-paste it and install the main serial executor at the top: func testFactIsLoading_MainSerialExecutor() async { swift_task_enqueueGlobal_hook = { job, _ in MainActor.shared.enqueue(job) } let fact = AsyncStream.makeStream(of: String.self) let model = NumberFactModel( numberFact: NumberFactClient( fact: { _ in await fact.stream.first(where: { _ in true })! } ) ) let task = Task { await model.getFactButtonTapped() } await Task.yield() XCTAssertEqual(model.isLoading, true) fact.continuation.yield("0 is a great number.") await task.value XCTAssertEqual(model.fact, "0 is a great number.") XCTAssertEqual(model.isLoading, false) }

— 23:16

Without a single other change this test passes, and does so 100% of the time, deterministically: Test Suite 'Selected tests' passed. Executed 10000 tests, with 0 failures (0 unexpected) in 17.995 (21.579) seconds

— 23:34

And just as we did before, we can even simplify this test by getting rid of the extra yields and getting rid of the async stream and continuation, and providing a simpler fact endpoint for the number fact dependency: func testFactIsLoading_MainExecutor() async { swift_task_enqueueGlobal_hook = { job, _ in MainActor.shared.enqueue(job) } let model = withDependencies { $0.numberFact.fact = { await Task.yield() return "\($0) is a great number." } } operation: { NumberFactModel() } let task = Task { await model.getFactButtonTapped() } await Task.yield() XCTAssertEqual(model.isLoading, true) await task.value XCTAssertEqual(model.fact, "0 is a great number.") XCTAssertEqual(model.isLoading, false) }

— 23:57

This test still passes deterministically, but is now a lot simpler.

— 24:03

Let’s move onto the next test. It exercises the behavior of the feature when the user taps on the “Get fact” button twice really fast. We want to make sure that the first request is cancelled and the second request is allowed to resume. Previously this test helped us catch a bug in which you would get both facts back from the number fact client, and that caused weird, glitchy behavior in the application.

— 24:27

It was a really great test to have, but it also failed a small percentage of the time because sometimes we would emulate a response coming back from the fact endpoint before the model was even listening for that data.

— 24:37

Let’s copy-and-paste this test and set up the main serial executor up at the top: func testBackToBackGetFact_MainSerialExecutor() async throws { swift_task_enqueueGlobal_hook = { job, _ in MainActor.shared.enqueue(job) } let fact0 = AsyncStream.makeStream(of: String.self) let fact1 = AsyncStream.makeStream(of: String.self) var callCount = 0 let model = NumberFactModel( numberFact: NumberFactClient( fact: { number in callCount += 1 if callCount == 1 { return await fact0.stream .first(where: { _ in true }) ?? "" } else if callCount == 2 { return await fact1.stream .first(where: { _ in true }) ?? "" } else { fatalError() } } ) ) let task0 = Task { await model.getFactButtonTapped() } let task1 = Task { await model.getFactButtonTapped() } await Task.yield() fact1.continuation.yield("0 is a great number.") try await Task.sleep(for: .milliseconds(100)) fact0.continuation.yield("0 is a better number.") await task0.value await task1.value XCTAssertEqual(model.fact, "0 is a great number.") }

— 24:55

Without doing anything else it seems to be passing 100% of the time, deterministically, although a bit slowly, but that’s because we previously had to sleep for some real time to pass. We can remove that sleep: // try await Task.sleep(for: .milliseconds(100))

— 25:19

And it now passes 100% of the time, deterministically, and much more quickly.

— 25:35

We can also apply the trick we did previously to massively simplify the test and get rid of the async streams: func testBackToBackGetFact_MainExecutor() async throws { swift_task_enqueueGlobal_hook = { job, _ in MainActor.shared.enqueue(job) } var callCount = 0 let model = withDependencies { $0.numberFact.fact = { number in await Task.yield() callCount += 1 if callCount == 1 { return "0 is a good number." } else if callCount == 2 { return "0 is a great number." } else { fatalError() } } } operation: { NumberFactModel() } let task0 = Task { await model.getFactButtonTapped() } let task1 = Task { await model.getFactButtonTapped() } await Task.yield() await task0.value await task1.value XCTAssertEqual(model.fact, "0 is a great number.") }

— 26:08

And we can see it still definitely passes consistently: Test Suite 'Selected tests' passed. Executed 10000 tests, with 0 failures (0 unexpected) in 19.292 (23.098) seconds Absolutely incredible.

— 26:21

This is fun! Let’s keep going.

— 26:24

The next test we have shows that the cancellation behavior in our feature works as we expect. We emulating tapping the “Get fact” button, then tapping the “Cancel” button, and showing that no fact is loaded.

— 26:37

Again, it was a great test to have, but did not pass deterministically.

— 26:41

Let’s copy-and-paste the test, and install the main serial executor: func testCancel_MainExecutor() async throws { swift_task_enqueueGlobal_hook = { job, _ in MainActor.shared.enqueue(job) } let model = withDependencies { $0.numberFact.fact = { _ in try await Task.never() } } operation: { NumberFactModel() } let task = Task { await model.getFactButtonTapped() } await Task.megaYield() model.cancelButtonTapped() await task.value XCTAssertEqual(model.fact, nil) }

— 26:50

Now this test passes 100% of the time, deterministically: Test Suite 'Selected tests' passed. Executed 10000 tests, with 0 failures (0 unexpected) in 23.152 (27.430) seconds

— 27:12

And even better, we can now drop the megaYield and just use a regular yield: let task = Task { await model.getFactButtonTapped() } // await Task.megaYield() await Task.yield() model.cancelButtonTapped() await task.value XCTAssertEqual(model.fact, nil)

— 27:22

This still passes deterministically.

— 27:34

There’s also something fun we can explore with this example. Right now we have pretty good test coverage that the work to load the fact is indeed canceled and that the fact will stay nil . This is because we used a Task.never() for the fact endpoint, and so the mere fact that await task.value did return is proof that the task must have been cancelled, because otherwise it would have suspended forever.

— 27:57

So, it certainly does the job, but it also doesn’t really represent real life since when run in the simulator the network request would eventually return something. It doesn’t just suspend forever like Task.never() .

— 28:08

So, we can better emulate real life by checking cancellation before returning the string: let model = withDependencies { $0.numberFact.fact = { try Task.checkCancellation() return "\($0) is a good number." } } operation: { NumberFactModel() } XCTAssertEqual failed: (“Optional(“0 is a good number”)” is not equal to (“nil”))

— 28:40

It failed. Despite checking cancellation, it looks like we got the fact before cancellation took place.

— 29:07

We need to insert another suspension point before checking cancellation to give ourselves enough “time” to emulate tapping the cancel button before the fact client checks for cancellation. $0.numberFact.fact = { await Task.yield() try Task.checkCancellation() return "\($0) is a good number." }

— 29:37

And if we run the test again it does pass, and does so deterministically.

— 29:53

There’s one last test in the suite, and it emulates the user taking screenshots while the feature is running, and asserts that with each screenshot the count goes up by one.

— 30:06

Yet another great test to have, but also one that is flakey.

— 30:11

Let’s copy-and-paste it, and install the main serial executor at the top, and we can go ahead and get rid of the megaYield and loops in exchange for simple yield s: func testScreenshots_MainExecutor() async throws { swift_task_enqueueGlobal_hook = { job, _ in MainActor.shared.enqueue(job) } let model = NumberFactModel() Task { await model.onAppear() } await Task.yield() NotificationCenter.default.post( name: UIApplication.userDidTakeScreenshotNotification, object: nil ) await Task.yield() XCTAssertEqual(model.count, 1) NotificationCenter.default.post( name: UIApplication.userDidTakeScreenshotNotification, object: nil ) await Task.yield() XCTAssertEqual(model.count, 2) }

— 30:45

With no other change this test passes, and deterministically: Test Suite 'Selected tests' passed. Executed 10000 tests, with 0 failures (0 unexpected) in 247.305 (248.944) seconds It passes quite a bit slower than the other ones, but that’s probably just due to the user of Notification Center in the test.

— 31:11

So that’s pretty incredible too. Previously we had no way of knowing when the async sequence actually started up, which forced us to perform a megaYield . Now a simple yield is enough because that gets the onAppear task executing right away. And further, after posting a notification we just need one simple yield to allow the model to do its work and increment the count. There’s no guesswork for how many yields need to be performed, or if we need to sleep a small amount of time. It just works with a single yield because we can precisely reason about how these tasks are executing. A truly immediate countdown preview

— 31:52

So, this is all looking really incredible. Over, and over, and over we have seen that we can turn slow, flakey tests into fast, deterministic tests. Now that we can understand how async tasks are scheduled while testing, we can wiggle ourselves in between all the various suspension points and assert on exactly how our code is behaving each step of the way. And it’s all thanks to the global task enqueue hook.

— 32:12

Brandon : But there’s another use for the enqueue hook beyond just tests. It can also make our previews better. Recall that earlier in this series we showed a countdown feature that would count down from 10 to 0, and at 0 it would show some confetti. It was a pain to actually wait for 10 seconds to pass just to iterate on the feature, and so we controlled the time-based asynchrony with a clock, and used an immediate clock to speed up the countdown.

— 32:37

That improved things drastically, but it was still a bit of a pain because even the immediate clock is susceptible to scheduling complexities in the global executor’s thread pool. Well, our need global enqueue hook can fix this problem once and for all.

— 32:55

Let’s take a look.

— 32:59

Recall that we have this countdown file with a simple count down feature in it. We can run in the preview and see it takes 10 seconds to count down before showing some confetti. But also we can substitute in an immediate clock, and that makes it countdown faster, but it is a little glitchy.

— 33:57

And if we up the countdown time to 100 seconds instead of 10 seconds we can really start to see the wonkiness.

— 34:28

As we mentioned earlier in this series, this is because the immediate clock is forced to do some yields on the inside so that other people’s work can start executing before and after the clock sleeps. But it’s precisely those yields that is slowing this down.

— 34:56

Well, we can side step all the complexities of the global executor and just serialize everything to the main serial executor.

— 35:03

And now in the preview we can set up the global enqueue hook so that it enqueues all jobs on the main serial executor: struct ControlledContentDemo_Previews: PreviewProvider { static var previews: some View { let _ = swift_task_enqueueGlobal_hook = { job, _ in MainActor.shared.enqueue(job) } ControlledCountdownDemo(clock: ImmediateClock()) } }

— 35:31

With that one small change we now see the countdown happens much, much faster. We can even set it to 1,000 and it counts down extremely quickly. The only reason it isn’t “instantaneous” is because SwiftUI still wants to render some frames as it counts down, and so that slows things down a little bit. Next time: reliable Combine tests

— 36:16

Again we’ve seen something incredible. Although the main point of this series is that we want to reliably test async code in Swift, there is the other side benefit that surprisingly pops up. The tool we’ve cooked up for serializing asynchronous work in tests can also be used to make certain kinds of Xcode previews execute faster, and improve our ability to quickly iterate on features that make use of time-based asynchrony.

— 36:39

So, it’s all looking amazing, but I’m sure there’s a number of our viewers who are thinking that we have done some truly bizarre things in these past few episodes. We have completely altered the execution environment our tests run in, seemingly just so that we can get them passing. But doesn’t that mean there was something wrong with our features or tests in the first place? And doesn’t that mean that we aren’t actually testing reality since our apps do not operate on the main serial executor when run on our user’s devices. Stephen

— 37:09

Well, for the feature we built and the kinds of tests we wrote, neither is true. There is absolutely nothing wrong with our features or tests, and for the kinds of behavior that we are wanting to test, using the main serial executor is essentially equivalent to using default, global executor.

— 37:24

To explore this, we are going rebuild the feature from past episodes using only the tools from Combine. We aren’t going to use any fancy Swift concurrency tools. This will allow us to compare the two styles to see how they differ. In particular, we find that the Combine code is a little more verbose and annoying to work with, but overall looks roughly the same as the async version. The tests also look almost identical, however we will find that they pass deterministically, 100% of the time.

— 37:49

Let’s dig in. References Reliably testing code that adopts Swift Concurrency? Brandon Williams & Stephen Celis • May 13, 2022 A Swift Forums post from yours truly about the difficulty of testing async code in Swift. https://forums.swift.org/t/reliably-testing-code-that-adopts-swift-concurrency/57304 Concurrency Extras Brandon Williams and Stephen Celis • Jul 18, 2023 An open source Swift package that provides powerful tools for making async code easier to test. https://github.com/pointfreeco/swift-concurrency-extras Announcing Concurrency Extras: Useful, testable Swift concurrency. Brandon Williams and Stephen Celis • Jul 18, 2023 The announcement of our new Concurrency Extras library, which provides powerful tools for making async code easier to test. https://www.pointfree.co/blog/posts/109-announcing-concurrency-extras-useful-testable-swift-concurrency Downloads Sample code 0241-reliably-testing-async-pt4 Point-Free A hub for advanced Swift programming. Brought to you by Brandon Williams and Stephen Celis . Content Become a member The Point-Free Way Beta previews Gifts Videos Collections Free clips Blog More About Us Community Slack Mastodon Twitter BlueSky GitHub Contact Us Privacy Policy © 2026 Point-Free, Inc. All rights are reserved for the videos and transcripts on this site. All other content is licensed under CC BY-NC-SA 4.0 , and the underlying source code to run this site is licensed under the MIT License .