Video #242: Reliable Async Tests: The Point

Episode: Video #242 Date: Jul 17, 2023 Access: Members Only 🔒 URL: https://www.pointfree.co/episodes/ep242-reliable-async-tests-the-point

Description

What’s the point of the work we did to make async testing reliable and deterministic, and are we even testing reality anymore? We conclude our series by rewriting our feature and tests using Combine instead of async-await, and comparing both approaches.

Video

Cloudflare Stream video ID: 694d299840220f7276bfbe9a8153c244 Local file: video_242_reliable-async-tests-the-point.mp4 *(download with --video 242)*

References

Transcript

— 0:05

Again we’ve seen something incredible. Although the main point of this series is that we want to reliably test async code in Swift, there is the other side benefit that surprisingly pops up. The tool we’ve cooked up for serializing asynchronous work in tests can also be used to make certain kinds of Xcode previews execute faster, and improve our ability to quickly iterate on features that make use of time-based asynchrony.

— 0:28

So, it’s all looking amazing, but I’m sure there’s a number of our viewers who are thinking that we have done some truly bizarre things in these past few episodes. We have completely altered the execution environment our tests run in, seemingly just so that we can get them passing. But doesn’t that mean there was something wrong with our features or tests in the first place? And doesn’t that mean that we aren’t actually testing reality since our apps do not operate on the main serial executor when run on our user’s devices. Stephen

— 0:58

Well, for the feature we built and the kinds of tests we wrote, neither is true. There is absolutely nothing wrong with our features or tests, and for the kinds of behavior that we are wanting to test, using the main serial executor is essentially equivalent to using default, global executor.

— 1:13

To explore this, we are going rebuild the feature from past episodes using only the tools from Combine. We aren’t going to use any fancy Swift concurrency tools. This will allow us to compare the two styles to see how they differ. In particular, we find that the Combine code is a little more verbose and annoying to work with, but overall looks roughly the same as the async version. The tests also look almost identical, however we will find that they pass deterministically, 100% of the time.

— 1:38

Let’s dig in. Combinification of our feature

— 1:42

The Combine version of our NumberFactModel looks so similar to the async version we implemented in the past few episodes that I’m actually going to start by copying and pasting it, and giving it a new name: import Combine @MainActor class CombineNumberFactModel: ObservableObject { … }

— 2:06

We will just scan this type from top-to-bottom to see what all needs to be converted.

— 2:11

The first thing I see is the mention of the factTask , which represents the inflight network request to fetch a fact: @Published var factTask: Task<String, Error>?

— 2:18

It is used for cancelling the effect while it is inflight, and also allows us to show a progress indicator in the view thanks to this computed property: var isLoading: Bool { self.factTask != nil }

— 2:26

The Combine equivalent of this is a cancellable, and it allows you to get a hold of a running publisher and cancel it at a later date. So, let’s replace the factTask with a factCancellable : @Published var factCancellable: AnyCancellable? var isLoading: Bool { self.factCancellable != nil } func incrementButtonTapped() { self.fact = nil self.factCancellable?.cancel() self.factCancellable = nil self.count += 1 } func decrementButtonTapped() { self.fact = nil self.factCancellable?.cancel() self.factCancellable = nil self.count -= 1 }

— 2:42

That of course creates a bunch of compiler errors, and we will fix them one at a time.

— 2:48

First we have the getFactButtonTapped method, which is responsible for cancelling any inflight work, making a new request, and then cleaning up state afterwards. This method is going to change quite a bit, so I am going to comment it out so that we can easily compare the before and after: /* func getFactButtonTapped() async { self.factTask?.cancel() self.fact = nil self.factTask = Task { try await self.numberFact.fact(self.count) } defer { self.factTask = nil } do { self.fact = try await self.factTask?.value } catch { // TODO: handle error } } */

— 3:10

Now let’s start up a new getFactButtonTapped , but this time we do not need to make it async since we are only using Combine: func getFactButtonTapped() { }

— 3:19

And we can start it off similarly to the old one where we preemptively cancel any inflight work, as well as clear out the current fact since we are loading a new one: self.factCancellable?.cancel() self.fact = nil

— 3:28

Next we want to make the new API request, which requires us to interact with the number fact client: self.numberFact.<#???#>(self.count)

— 3:41

But, we don’t have an endpoint on the NumberFactClient that deals with publishers. We only have the one endpoint that performs async work, which we want to avoid right now.

— 3:48

So, let’s quickly add a new endpoint that allows us to fetch a number fact via a publisher interface: struct NumberFactClient { var fact: @Sendable (Int) async throws -> String var factPublisher: @Sendable (Int) -> AnyPublisher<String, Error> }

— 4:16

Already we can see the publisher version is a bit more complex than the async version. Instead of being able to leverage language-level features like async and throws we need to describe this generic AnyPublisher type.

— 4:31

The implementation is even more complex: extension NumberFactClient: DependencyKey { static let liveValue = Self { number in try await Task.sleep(for: .seconds(1)) return try await String( decoding: URLSession.shared.data( from: URL( string: "http://numbersapi.com/\(number)" )! ) .0, as: UTF8.self ) } factPublisher: { number in URLSession.shared .dataTaskPublisher( for: URL( string: "http://numbersapi.com/\(number)" )! ) .delay(for: 1, scheduler: DispatchQueue.main) .map { data, _ in String(decoding: data, as: UTF8.self) } .mapError { $0 as Error } .eraseToAnyPublisher() } }

— 5:48

We have to contort ourselves in all kinds of weird ways to deal with this publisher. We need to use a delay operator in order to slow down the request, whereas previously we could just do Task.sleep . We have to map on the publisher to pluck out the data from the response and convert it to a string. Then we have to cast the error so that all the types match up. And then finally we have to erase the whole publisher.

— 6:11

Compared to the async/await version this is just a lot more complicated.

— 6:16

But, now that we have that we can use it: self.numberFact.factPublisher(self.count)

— 6:26

We want to assign the result to the fact property, but we can’t do so directly since it’s wrapped in a publisher. And while the @Published property wrapper has a nice way to do this: self.numberFact .factPublisher(self.count) .assign(&self.$fact) Referencing instance method ‘assign(to:)’ on ‘Publisher’ requires the types ‘any Error’ and ‘Never’ to be equivalent

— 6:55

It unfortunately doesn’t work here since we have an error to deal with.

— 7:01

Instead, we can sink on that publisher to handle when the publisher finishes or emits a value: self.numberFact.factPublisher(self.count) .sink( receiveCompletion: <#((Subscribers.Completion<Error>) -> Void)#>, receiveValue: <#((String) -> Void)#> )

— 7:10

In the receiveValue closure we need to update the model’s state with the freshly loaded fact, but we do have to be mindful about memory management: receiveValue: { [weak self] fact in self?.fact = fact }

— 7:33

In receiveCompletion we could handle any errors if we were doing that now: receiveCompletion: { _ in // TODO: Handle error },

— 7:41

But sink returns a cancellable so we should assign it: self.factCancellable = self.numberFact .factPublisher(self.count) …

— 7:55

And then we have to make sure to clean up the factCancellable state once the request finishes since that is what drives hiding the loading indicator in the view: receiveCompletion: { [weak self] _ in // TODO: Handle error self?.factCancellable = nil },

— 8:10

And that’s about all it takes for this method, but comparing what we just came up with to what we were able to do before I think it is very clear why we might prefer async/await code over Combine code.

— 8:20

The main benefit to async/await is that it allows you to write asynchronous code in a structured manner. This means you get to use regular control flow statements, such as for loops, if/else, defers and more, right along side your async code.

— 8:34

The Combine code cannot do that. We had to give up defer statements and instead remember to clean up code in the receiveCompletion closure. And all of these escaping closures in Combine are just a reminder that we are regularly being jettisoned out of the nice, structured programming world, making it so that our code does not tell a simple, linear story from top to bottom.

— 8:56

And finally we have the onTask method, which is called when the view first appears and its purpose is to listen for screenshots being taken in the application so that we can increment the count.

— 9:09

This needs to be changed so that we use the notification center publisher to listen for notifications, and then sink on the publisher to increment the count: func onTask() { NotificationCenter.default .publisher( for: UIApplication.userDidTakeScreenshotNotification ) .sink { [weak self] _ in self?.count += 1 } }

— 9:41

However, we need to hold onto the cancellable in order for the subscription to be long living, and so we have to add yet another instance variable to the model: var notificationCancellable: AnyCancellable? func onTask() { self.notificationCancellable = NotificationCenter .default .publisher( for: UIApplication.userDidTakeScreenshotNotification ) .sink { [weak self] _ in self?.count += 1 } }

— 10:00

That’s a bit of a bummer. One of the nice things about Swift’s native concurrency tools is that thanks to structured programming, the lifecycle of a long living async unit of work can be tied to the object performing that work. Generally you do not have to do anything more.

— 10:13

But, we are now done implementing the number fact model using the tools from Combine. We can put this model in the view: struct ContentView: View { // @ObservedObject var model: NumberFactModel @ObservedObject var model: CombineNumberFactModel … }

— 10:28

And the view doesn’t even need to change at all. There are a few warnings that we are doing await in places that no longer needs it, but but we’ll just leave it for now because it’s not hurting anybody.

— 10:38

To run the app we just need to update the preview: struct ContentPreviews: PreviewProvider { static var previews: some View { ContentView(model: CombineNumberFactModel()) } }

— 10:45

And entry point of the application: @main struct ReliablyTestingAsyncApp: App { var body: some Scene { WindowGroup { if NSClassFromString("XCTestCase") == nil { ContentView(model: CombineNumberFactModel()) } } } }

— 10:49

Now when we run the app in the simulator or in a preview we see that it mostly works as it did before, but there is a bug that’s sneakily hiding from view. We inserted an arbitrary delay into our live API client code to better see loading and cancellation, but if we comment it out and run things again… // .delay(for: 1, scheduler: DispatchQueue.main)

— 11:32

We begin to get some purple warnings about updating publishers on background threads. This is happening because the URLSession publisher delivers its data on a background thread, and we have to get that back to the main thread. That was one really nice thing about the async/await version of this code. The model got to say in one place it runs on the @MainActor , and all async work automatically abided without any additional work.

— 12:05

Well, we do now have to do some additional work. The simplest thing to do would be to tack on a receive(on:) with the main dispatch queue: self.factCancellable = self.numberFact .factPublisher(self.count) .receive(on: DispatchQueue.main) .sink(…)

— 12:19

This gets the app running again without the purple warnings, but of course this live, main dispatch queue is now a dependency our feature is using, and is going to eventually wreak havoc on tests. But, we will just leave it as-is until we encounter those problems. Writing Combine tests Stephen

— 12:40

So, that was a fun little side excursion into remembering what it was like to build features with async work before we had all the wonderful concurrency tools in Swift. There was a lot more indirection where we need to perform magical incantations of publisher operators in order to massage data into the shape we needed. We also had to think about memory management more, making sure to capture self weakly in escaping closures and keeping track of more cancelation than previously needed. And we lost the benefits of structured programming along the way. Brandon

— 13:08

So, it really is plain as day that Swift’s concurrency tools have brought something amazing to the table when it comes to building features that need to perform async work. There’s no arguing that. But, what does it look like to test this new model using the tools that Combine gives us?

— 13:22

Let’s port over every test we wrote for the original model over to this new model.

— 13:30

Let’s start with the simplest test, which didn’t even involve any asynchrony whatsoever. It’s called testIncrementDecrement , and it just confirms that tapping the increment and decrement buttons does indeed make the count go up and down.

— 13:43

We can even just copy and paste it and rename the model: func testIncrementDecrement() { let model = CombineNumberFactModel() model.incrementButtonTapped() XCTAssertEqual(model.count, 1) model.decrementButtonTapped() XCTAssertEqual(model.count, 0) }

— 13:55

…and already we have a passing test.

— 13:59

That shouldn’t be too surprising because we are only asserting on synchronous behavior in this execution flow, and so it doesn’t matter whether we are using Combine or Swift’s concurrency tools.

— 14:14

Let’s go to the next test, which is a little more interesting. It’s called testGetFact and it asserts that when you tap the “Get fact” button a fact is loaded from the number fact dependency.

— 14:38

Let’s start by copying and pasting the testGetFact test, renaming it and dropping the async : func testGetFact_Combine() { … }

— 14:47

The first thing we want to do is construct a CombineNumberFactModel since that is the thing we are now testing: let model = withDependencies { … } operation: { CombineNumberFactModel() }

— 14:57

And it is no longer correct to override the fact endpoint of the dependency, and instead we should be overriding the factPublisher endpoint: let model = withDependencies { $0.numberFact.factPublisher = { number in } } operation: { … }

— 15:10

And in here we have to do work to construct a a publisher of the exact right shape, which can be annoying.

— 15:34

We would hope we could just return a publisher that synchronously and immediately returns the fact, say by using a Just publisher: $0.numberFact.factPublisher = { number in Just("\(number) is a good number.") }

— 15:43

But this doesn’t work because we need to further erase it to the AnyPublisher type: $0.numberFact.factPublisher = { number in Just("\(number) is a good number.") .eraseToAnyPublisher() }

— 16:00

And that doesn’t work either because we have to further massage the publisher to have the correct error type: $0.numberFact.factPublisher = { number in Just("\(number) is a good number.") .setFailureType(to: Error.self) .eraseToAnyPublisher() }

— 16:21

Now it finally compiles, but we’ve traded a simple one-liner for 5 lines of cryptic Combine code.

— 16:40

The last thing to do in this test is to drop the await s since getFactButtonTapped is no longer async: model.getFactButtonTapped() XCTAssertEqual(model.fact, "0 is a good number.") model.incrementButtonTapped() XCTAssertEqual(model.fact, nil) model.getFactButtonTapped() XCTAssertEqual(model.fact, "1 is a good number.")

— 16:51

Now the test is compiling, and so we would hope it passes. But it does not: (“nil”) is not equal to (“Optional(“0 is a good number.”)”) Failed: (“nil”) is not equal to (“Optional(“1 is a good number.”)”)

— 17:01

For some reason the fact is still nil .

— 17:18

This is happening because of the seemingly innocent dependency we sprinkled into our feature a moment ago. We needed to make sure the publisher emitted on the main thread, and so we crammed a live, main queue into our publisher chain: self.factCancellable = self.numberFact .factPublisher(self.count) .receive(on: DispatchQueue.main)

— 17:38

This is introducing a thread hop into our feature’s execution, which means the test is asserting before the model actually updates it’s state.

— 17:50

The fix is easy enough. We must take back control over this dependency rather than letting it control us. Luckily for us our dependencies library comes with a controllable scheduler that we can substitute for the main queue: class CombineNumberFactModel: ObservableObject { @Dependency(\.mainQueue) var mainQueue … }

— 18:13

And we can use that rather than reaching out to the global, uncontrolled main dispatch queue: self.factCancellable = self.numberFact.factPublisher(self.count) .receive(on: self.mainQueue) …

— 18:21

Now let’s run our test just to see what happens: @Dependency(\.mainQueue) - An unimplemented scheduler scheduled an action to run immediately.

— 18:27

It still fails, but for a different reason. Now it is letting us know that we are making use of a dependency in this test that we have not explicitly overridden. This is a great test failure to have because it forces you to be explicit with what dependencies are needed for exercising the user flow you are testing.

— 19:00

So, let’s override the mainQueue dependency by providing an immediate scheduler so that when one tells it to schedule work for a later time it just ignores you and executes the work right that moment: let model = withDependencies { $0.mainQueue = .immediate … } operation: { CombineNumberFactModel() }

— 19:07

Now when we run the test it passes, even if we run it 10,000 times: Test Suite 'Selected tests' passed. Executed 10000 tests, with 0 failures (0 unexpected) in 16.809 (28.412) seconds

— 19:24

So, this test is a little more annoying to write than the equivalent async/await one, but at least it works, and it seems to pass 100% of the time, deterministically.

— 19:58

But also, that’s not too surprising. After all, even the original async version of this test did pass 100% of the time. The first test we had flakiness was in the testFactClearsOut test, which wants to exercise the behavior that when the “Get fact” button is tapped it immediately clears out the fact state and the re-populates it when the API request finishes.

— 20:39

The test was flaky because it was difficult for us to wiggle ourselves into the exact moment between invoking the getFactButtonTapped method and the moment the async request finished. We fixed the flakiness by resorting to the main serial executor, and thanks to our interpretation of how async units of work are enqueued we were even able to get rid of the fact stream for instrumenting the number fact dependency.

— 21:17

Let’s copy-and-paste the test, rename it, and remove the async : func testFactClearsOut_Combine() { … }

— 21:34

For the Combine version of this test we do need something like the fact stream so that we can hold up the dependency until we decide it is ready to proceed. The way this is done in the Combine world is via a passthrough subject: let fact = PassthroughSubject<String, Error>()

— 21:58

And that can be returned directly from the factPublisher endpoint: let model = withDependencies { $0.mainQueue = .immediate $0.numberFact.factPublisher = { _ in fact.eraseToAnyPublisher() } } operation: { CombineNumberFactModel() } model.fact = "An old fact about 0."

— 22:11

Then the rest of the test is mostly the same, except we no longer need to spin up unstructured tasks or perform yields: model.getFactButtonTapped() XCTAssertEqual(model.fact, nil) fact.send("0 is a good number.") fact.send(completion: .finished) XCTAssertEqual(model.fact, "0 is a good number.")

— 22:42

That’s all it takes to convert this test. It’s basically equivalent to the original async version, maybe even a little nicer since we don’t have to spin up unstructured tasks or perform yields, and it’s only a little more verbose than the main serial executor version.

— 23:03

But, the nice thing about this test is that right out of the box, with no further work, it passes 100% of the time, deterministically. We can run it 10,000 times: Test Suite 'Selected tests' passed. Executed 10000 tests, with 0 failures (0 unexpected) in 17.794 (31.771) seconds

— 23:15

And it passes.

— 23:35

This is happening all thanks to the synchronous nature of Combine. That may sound weird to say, after all Combine seems to be positioned a tool for handling asynchronous streams of data. However, the the vast majority of APIs in the Combine framework are synchronous by nature. The only time they become asynchronous is if you start introducing things like schedulers, or construct futures that use escaping closures, or a few other things. But for the most part it really is a synchronous framework.

— 24:07

And so when the getFactButtonTapped method is invoked, it synchronously and immediately clears out the fact state, which is why we are able to assert on that behavior right away. Since getFactButtonTapped is no longer async we don’t have to worry about suspension points and thread hops and either asserting too early or too late on that state.

— 24:39

I think it’s kind of amazing how little work we have to do to wiggle ourselves in between the moment the “Get fact” button is tapped and the moment the fact request finishes so that we can assert on the fact state being nil ’d out. And because it’s so easy we can feel empowered to write tests for all of the subtle behavior in our features, or edge cases.

— 25:02

Let’s move onto the next test. It is very similar to the one we just covered, except it wants to test that the isLoading state flips from false to true while loading the fact, and then flips back to false once done. Let’s quickly re-implement this test using the tools from Combine: func testFactIsLoading_Combine() { let fact = PassthroughSubject<String, Error>() let model = withDependencies { $0.mainQueue = .immediate $0.numberFact.factPublisher = { _ in fact.eraseToAnyPublisher() } } operation: { CombineNumberFactModel() } model.fact = "An old fact about 0." model.getFactButtonTapped() XCTAssertEqual(model.isLoading, true) fact.send("0 is a good number.") fact.send(completion: .finished) XCTAssertEqual(model.fact, "0 is a good number.") XCTAssertEqual(model.isLoading, false) }

— 26:05

And we will find that this test passes 100% of the time: Test Suite 'Selected tests' passed. Executed 10000 tests, with 0 failures (0 unexpected) in 17.741 (31.599) seconds

— 26:10

So, again we are seeing a Combine test that naturally just passes 100% of the time, even though the equivalent one in async/await world does not pass deterministically. We had to resort to the main serial executor to get the test passing consistently, and once we did that it did allow us to simplify the test a bit, which at least that was nice.

— 26:38

Let’s move onto the next test. This one exercises the user behavior of what happens when the user quickly taps the “Get fact” button twice, and we emulate the first request finishing after the second. We want to make sure that that first request is cancelled so that we don’t accidentally see old data in our model.

— 27:24

Let’s quickly copy-and-paste this test and convert it to Combine: func testBackToBackGetFact_Combine() { let fact0 = PassthroughSubject<String, Error>() let fact1 = PassthroughSubject<String, Error>() let callCount = LockIsolated(0) let model = withDependencies { $0.mainQueue = .immediate $0.numberFact.factPublisher = { number in callCount.withValue { $0 += 1 } if callCount.value == 1 { return fact0.eraseToAnyPublisher() } else if callCount.value == 2 { return fact1.eraseToAnyPublisher() } else { fatalError() } } } operation: { CombineNumberFactModel() } model.getFactButtonTapped() model.getFactButtonTapped() fact1.send("0 is a great number.") fact1.send(completion: .finished) fact0.send("0 is a better number.") fact0.send(completion: .finished) XCTAssertEqual(model.fact, "0 is a great number.") }

— 28:50

Again the code is quite similar to the async/await version, but now it passes 100% of the time: Test Suite 'Selected tests' passed. Executed 1000 tests, with 0 failures (0 unexpected) in 2.092 (3.827) seconds

— 28:59

No additional tricks needed. It just works.™

— 29:08

Let’s move onto the next test, where we exercise tapping the “Get fact” button and then tapping the “Cancel” button before the fact request finishes. Let’s copy-and-paste this and convert it to Combine: func testCancel_Combine() { let model = withDependencies { $0.numberFact.factPublisher = { _ in Empty(completeImmediately: false).eraseToAnyPublisher() } } operation: { CombineNumberFactModel() } model.getFactButtonTapped() model.cancelButtonTapped() XCTAssertEqual(model.fact, nil) XCTAssertEqual(model.factCancellable, nil) }

— 30:46

This test too passes 100% of the time: Test Suite 'Selected tests' passed. Executed 1000 tests, with 0 failures (0 unexpected) in 1.923 (3.826) seconds

— 30:55

We are just writing these tests exactly in the manner we tried writing the async/await versions, but they deterministically pass right away with no additional work.

— 31:09

There’s one last test in the suite, and that’s the one that confirms that when screenshots are taken in the app, the count increments by one. Let’s copy-and-paste the test and convert to Combine: func testScreenshots_Combine() { let model = CombineNumberFactModel() model.onTask() NotificationCenter.default.post( name: UIApplication.userDidTakeScreenshotNotification, object: nil ) XCTAssertEqual(model.count, 1) NotificationCenter.default.post( name: UIApplication.userDidTakeScreenshotNotification, object: nil ) XCTAssertEqual(model.count, 2) }

— 32:15

This is quite a bit simpler than the async/await version of this test, even when using the main serial executor. And best of it all, right out of the box it passes 100% of the time: Test Suite 'Selected tests' passed. Executed 1000 tests, with 0 failures (0 unexpected) in 2.159 (3.947) seconds

— 32:29

And it’s at this moment that I think we should stop and reflect for a bit.

— 32:34

We have now implemented the NumberFactModel in two different, yet essentially equivalent, ways. One uses all the fancy new Swift concurrency tools, which means its implementation was really short and sweet, and we got to take advantage of structured programming. However, testing the model was a huge pain. If you wrote tests in the most natural and naive way, they would almost always fail. And if you took the time to sprinkle in some Task.yield s or sleeps then you could get tests to pass more often, but of course you could never be 100% sure.

— 33:37

And the other model was implemented using Apple’s Combine framework, which, let’s be honest, is basically deprecated at this point. And in using Combine the model code was a bit gnarlier and more indirect, but we could write tests in a very natural way and they passed deterministically right out of the gate. We didn’t have to do any extra work whatsoever.

— 34:27

The two models and the tests are basically equivalent, and yet I’m sure that no one would question the veracity of the Combine tests. We would never think the Combine tests don’t truly test reality. After all, they test very, very basic behavior. The user taps on a button, we assert some loading state flips to true, a moment later the fact request finishes, and the loading state flips back to false.

— 34:48

It doesn’t matter what concurrency model we are using to power this feature, this is no room for non-determinism. It is a simple state machine being executed by the user, and we want to write tests for it. Highly asynchronous testing Stephen

— 34:58

So, given that, why is it that we would now question our tests of the async/await version of this model just because we are sticking in the main serial executor. Sure, we are altering the executing context of our feature by serializing all async work to the main queue, but that’s also what we did with the Combine code. In tests we are not making a live API request, which involves hopping to a background thread to do work and then hopping back to the main thread. Instead we just ran some synchronous work right on the main thread immediately.

— 35:26

So, the two situations are not really that different. The main serial executor tool that we have created in this series merely allows us to test async/await code in Swift just we always code test Combine code. And that’s kind of amazing.

— 35:38

Now, we don’t want to leave you with the message that this main serial executor tool is appropriate for all async tests. It definitely is not. If your feature has very complex, concurrently executing behavior that really can interleave in non-deterministic ways, then you probably do want to test that with the default, global executor. Of course you can’t make absolute assertions on what happens in your feature, such as the count value is exactly such-and-such , or the fact string is exactly whatever. But, you can make weakened assertions, such as the count is at least such-and-such value, or the fact string is one of several possible values.

— 36:11

And let’s see this in really concrete terms.

— 36:15

Consider the following seemingly reasonable looking property wrapper that tries to wrap a value in a thread safe package: import Foundation @propertyWrapper struct ThreadSafe<Value: Sendable>: Sendable { private let lock = NSRecursiveLock() private var _wrappedValue: Value var wrappedValue: Value { get { self.lock.withLock { self._wrappedValue } } set { self.lock.withLock { self._wrappedValue = newValue } } } init(wrappedValue: Value) { self._wrappedValue = wrappedValue } }

— 36:23

It does so by wrapping the get and set logic in a lock.

— 36:28

This seems reasonable at first, but it is actually not safe at all. By locking the get and set independently we are allowing interleaving of multiple threads, wherein we read from a value to make a modification, but before we can make the modification some other thread has come in and mutated the value, and so then our modification overwrites the previous data incorrectly.

— 36:48

To see this, suppose we had a class that held onto a quote-un-quote “thread safe” boolean: class Toggle { @ThreadSafe var isOn = false }

— 36:52

And let’s write a test that fires up 1,000 concurrent tasks to toggle the boolean, and we will run it with the main serial executor: func testSomething() async { swift_task_enqueueGlobal_hook = { job, _ in MainActor.shared.enqueue(job) } let toggle = Toggle() await withTaskGroup(of: Void.self) { group in for _ in 1...1000 { group.addTask { toggle.isOn.toggle() } } } XCTAssertEqual(toggle.isOn, false) }

— 38:14

After the 1,000 tasks execute we expect the toggle to be back to false, and we can run the test a thousand times to see that does seem to be the case: Test Suite 'Selected tests' passed. Executed 1000 tests, with 0 failures (0 unexpected) in 20.713 (20.897) seconds

— 38:20

But this is completely misleading. The only reason this is passing is because we have serialized everything to the main thread, and so we aren’t seeing the race condition get exploited.

— 38:30

If we remove the serial executor: func testSomething() async { // swift_task_enqueueGlobal_hook = { job, _ in // MainActor.shared.enqueue(job) // } … }

— 38:38

…and run tests again, we see a lot more failures: Test Suite 'Selected tests' failed. Executed 1000 tests, with 492 failures (0 unexpected) in 5.145 (6.032) seconds

— 38:41

It fails about half the time. Conclusion

— 38:46

This goes to show that it is not appropriate to use the main serial executor in this situation, and any situation in which the thing you are testing really is a highly concurrent system. However, we feel that that is not the majority of situations we write tests for in app development. By far the most common test we write for our apps is simply exercising a state machine. A user performs an action, then another action, then another action, and we want to assert how state changes each step of the way, and sometimes even assert on state changes in between units of async work.

— 39:15

And for this reason we think it is completely fine to use the main serial executor for most of the tests we write day-to-day. It simply allows us to write tests exactly like we could in Combine since its inception, and before that, exactly as we did with RxSwift, ReactiveSwift and all those functional reactive libraries. Brandon

— 39:31

Well, we hopefully have convinced you that it is totally OK to user the main serial executor in most tests we write day-to-day, and that we aren’t in any weakening the quality of the tests. For most tests we are simply verifying that the user takes specific steps in a state machine, and the default, global, highly unpredictable executor just gets in the way. We don’t need the full power of a concurrent executor for those kinds of tests.

— 39:57

And in conjunction with this last episode of the series we are also releasing a brand new library that comes with this tool, as well as other handy concurrency tools, so you can start using it in your projects right away. And further, we are also adding support for the main serial executor in the Composable Architecture. There is now a setting in test stores that allow you to turn the main serial executor on. It is off by default for now, but come 1.0 it will be on by default.

— 40:24

That’s it for now, until next time! References Reliably testing code that adopts Swift Concurrency? Brandon Williams & Stephen Celis • May 13, 2022 A Swift Forums post from yours truly about the difficulty of testing async code in Swift. https://forums.swift.org/t/reliably-testing-code-that-adopts-swift-concurrency/57304 Concurrency Extras Brandon Williams and Stephen Celis • Jul 18, 2023 An open source Swift package that provides powerful tools for making async code easier to test. https://github.com/pointfreeco/swift-concurrency-extras Announcing Concurrency Extras: Useful, testable Swift concurrency. Brandon Williams and Stephen Celis • Jul 18, 2023 The announcement of our new Concurrency Extras library, which provides powerful tools for making async code easier to test. https://www.pointfree.co/blog/posts/109-announcing-concurrency-extras-useful-testable-swift-concurrency Downloads Sample code 0242-reliably-testing-async-pt5 Point-Free A hub for advanced Swift programming. Brought to you by Brandon Williams and Stephen Celis . Content Become a member The Point-Free Way Beta previews Gifts Videos Collections Free clips Blog More About Us Community Slack Mastodon Twitter BlueSky GitHub Contact Us Privacy Policy © 2026 Point-Free, Inc. All rights are reserved for the videos and transcripts on this site. All other content is licensed under CC BY-NC-SA 4.0 , and the underlying source code to run this site is licensed under the MIT License .