Video #31: Decodable Randomness: Part 1

Episode: Video #31 Date: Sep 24, 2018 Access: Members Only 🔒 URL: https://www.pointfree.co/episodes/ep31-decodable-randomness-part-1

Description

This week we dive deeper into randomness and composition by looking to a seemingly random place: the Decodable protocol. While we’re used to using the Codable set of protocols when working with JSON serialization and deserialization, it opens the opportunity for so much more.

Video

Cloudflare Stream video ID: 37dc7dbef2a9dce22f9006fc95b0430d Local file: video_31_decodable-randomness-part-1.mp4 *(download with --video 31)*

Transcript

— 0:06

Last week we discussed the topic of “ Composable Randomness ”, which seeks to understand how randomness can be made more composable by using function composition. We also compared this with Swift 4.2’s new randomness API, which arguably is not composable, in the sense that it is not built from units that stand on their own and combine to form new units.

— 0:24

We spent some time building up fun generators, like randomly sized arrays of random values and even random password generators, both of which are not easily expressible with Swift’s APIs.

— 0:40

Today we are going to take this a big step forward, and unlock the ability to create random values from any data type we define. We’re actually going to do this in two different ways:

— 1:02

We’ll do a bunch of upfront work with a Swift specific feature that unlocks the ability to effortlessly generate random values from any data type, but we’ll see it’s a little bit rigid.

— 1:14

Then we’ll try another approach by doing a very modest amount of upfront work to build up the tools to generate random values from any data type in a very flexible way, but it takes a lil more work to do than the first method.

— 1:31

So let’s start with the first method, and it begins by look in an unlikely place: the Decodable Protocol. An arbitrary decoder

— 1:43

The Decodable protocol is almost always considered in light of decoding Swift types from data, particularly from JSON or property lists. This makes sense because JSONDecoder and PropertyListDecoder are currently the only decoders that come with Swift. It may be surprising, then, that Decodable gives us a lot more power than the ability to deserialize values, and today we’re going to explore one example: the ability to decode a random representation of a Swift type by defining a custom decoder.

— 2:16

Let’s start by looking at the protocol itself. public protocol Decodable { public init(from decoder: Decoder) throws }

— 2:26

An interesting thing about this protocol is that there’s no Data in sight! Decodable isn’t coupled to the existence of Data being decoded. When we use a JSONDecoder , we don’t even call this initializer. Instead we call the decode method that lives on JSONDecoder itself. import Foundation JSONDecoder().decode(<#Decodable.Protocol#>, from: <#Data#>)

— 2:56

This means that the behavior of the decoding is all up to the Decoder type.

— 3:00

The Decodable protocol gives us a means of walking over a type to produce an instance. This means we can use it to walk over a type and produce a random instance as long as we have a way of producing random versions of its smallest units. So let’s create a custom decoder that can be used to generate random versions of decodable types. We’ll call it ArbitraryDecoder , as we’ll be using it to generate arbitrary values. struct ArbitraryDecoder: Decoder { }

— 3:30

And we can let Swift fill things in for us. struct ArbitraryDecoder: Decoder { var codingPath: [CodingKey] var userInfo: [CodingUserInfoKey: Any] func container<Key: CodingKey>( keyedBy type: Key.Type ) throws -> KeyedDecodingContainer<Key> { <#code#> } func unkeyedContainer() throws -> UnkeyedDecodingContainer { <#code#> } func singleValueContainer() throws -> SingleValueDecodingContainer { <#code#> } }

— 3:33

There are a bunch of things here, so let’s break it down.

— 3:38

First, codingPath is an array of CodingKey s. CodingKey s are used in “keyed” data types, like structs, which are keyed by property name, and dictionaries, which are keyed by…their keys. This can be used to produce better error messaging, but we’re not going to be worried about codingPath , so let’s set it to an empty array to satisfy the compiler. var codingPath: [CodingKey] = []

— 4:05

Next, userInfo is a dictionary of information intended for user. We’re not going to do anything with this dictionary, either, so let’s set it to an empty dictionary. var userInfo: [CodingUserInfoKey: Any] = [:]

— 4:15

Finally, we have a trio of methods that return various “decoding containers”. Decoding containers are responsible for decoding values of various shapes.

— 4:19

KeyedDecodingContainer s decode a collection of values that are associated with named keys. This includes those “keyed” data types we just mentioned, like structs and dictionaries, using a CodingKey type.

— 4:35

UnkeyedDecodingContainer s decode a collection of values that aren’t associated with named keys, like arrays and sets.

— 4:43

SingleValueDecodingContainer s decode…single values. This includes primitives like booleans, numbers, strings, but also more complicated structures that can be represented as a single value, like dates and UUIDs.

— 4:57

All three of these container types work together to decode some pretty complex, nested types!

— 5:06

Let’s fatalError() each method for now to get things building. func container<Key: CodingKey>( keyedBy type: Key.Type ) throws -> KeyedDecodingContainer<Key> { fatalError() } func unkeyedContainer() throws -> UnkeyedDecodingContainer { fatalError() } func singleValueContainer() throws -> SingleValueDecodingContainer { fatalError() } Decoding single values

— 5:15

Let’s start with the smallest, simplest container type: the single value container.

— 5:19

SingleValueDecodingContainer is a protocol, so let’s create a type that conforms to it and return an instance from the singleValueContainer method. struct ArbitraryDecoder: Decoder { … func singleValueContainer() throws -> SingleValueDecodingContainer { return SingleValueContainer() } struct SingleValueContainer: SingleValueDecodingContainer { } }

— 5:42

We can let the compiler stub things out for us. struct ArbitraryDecoder: Decoder { … struct SingleValueContainer: SingleValueDecodingContainer { var codingPath: [CodingKey] func decodeNil() -> Bool { <#code#> } func decode(_ type: Bool.Type) throws -> Bool { <#code#> } func decode(_ type: String.Type) throws -> String { <#code#> } func decode(_ type: Double.Type) throws -> Double { <#code#> } func decode(_ type: Float.Type) throws -> Float { <#code#> } func decode(_ type: Int.Type) throws -> Int { <#code#> } func decode(_ type: Int8.Type) throws -> Int8 { <#code#> } func decode(_ type: Int16.Type) throws -> Int16 { <#code#> } func decode(_ type: Int32.Type) throws -> Int32 { <#code#> } func decode(_ type: Int64.Type) throws -> Int64 { <#code#> } func decode(_ type: UInt.Type) throws -> UInt { <#code#> } func decode(_ type: UInt8.Type) throws -> UInt8 { <#code#> } func decode(_ type: UInt16.Type) throws -> UInt16 { <#code#> } func decode(_ type: UInt32.Type) throws -> UInt32 { <#code#> } func decode(_ type: UInt64.Type) throws -> UInt64 { <#code#> } func decode<T: Decodable>(_ type: T.Type) throws -> T { <#code#> } } } Whoa! That’s a lot.

— 5:52

First we have another codingPath that we can ignore, so let’s do just that. var codingPath: [CodingKey] = []

— 5:59

And then we have a whole bunch of “decode” methods. The values that are returned from these methods determine the values that are decoded, so in order to decode a random value, we want to return random values from each of these methods.

— 6:21

This is a lot to do at once, so let’s fatalError() each implementation to appease the compiler. func decodeNil() -> Bool { fatalError() } func decode(_ type: Bool.Type) throws -> Bool { fatalError() } func decode(_ type: String.Type) throws -> String { fatalError() } func decode(_ type: Double.Type) throws -> Double { fatalError() } func decode(_ type: Float.Type) throws -> Float { fatalError() } func decode(_ type: Int.Type) throws -> Int { fatalError() } func decode(_ type: Int8.Type) throws -> Int8 { fatalError() } func decode(_ type: Int16.Type) throws -> Int16 { fatalError() } func decode(_ type: Int32.Type) throws -> Int32 { fatalError() } func decode(_ type: Int64.Type) throws -> Int64 { fatalError() } func decode(_ type: UInt.Type) throws -> UInt { fatalError() } func decode(_ type: UInt8.Type) throws -> UInt8 { fatalError() } func decode(_ type: UInt16.Type) throws -> UInt16 { fatalError() } func decode(_ type: UInt32.Type) throws -> UInt32 { fatalError() } func decode(_ type: UInt64.Type) throws -> UInt64 { fatalError() } func decode<T: Decodable>(_ type: T.Type) throws -> T { fatalError() }

— 6:31

Let’s start by describing what it means to decode a random Bool . Well, we can return Bool.random() . func decode(_ type: Bool.Type) throws -> Bool { return .random() }

— 6:41

And believe it or not, we’re ready to take our decoder for a spin! try Bool(from: ArbitraryDecoder()) // true try Bool(from: ArbitraryDecoder()) // true try Bool(from: ArbitraryDecoder()) // true try Bool(from: ArbitraryDecoder()) // false try Bool(from: ArbitraryDecoder()) // false

— 7:10

Let’s keep going!

— 7:15

We can decode numbers pretty simply by using the static random functions provided. func decode(_ type: Int.Type) throws -> Int { return .random(in: .min ... .max) } Now, we needed to make a decision here, which is the range of randomness we cover. In this case we can just cover the full range.

— 7:35

And because we’re using dot-prefix syntax, we can copy-paste this implementation throughout the rest of the Int and UInt types with no changes. func decode(_ type: Int8.Type) throws -> Int8 { return .random(in: .min ... .max) } func decode(_ type: Int16.Type) throws -> Int16 { return .random(in: .min ... .max) } func decode(_ type: Int32.Type) throws -> Int32 { return .random(in: .min ... .max) } func decode(_ type: Int64.Type) throws -> Int64 { return .random(in: .min ... .max) } func decode(_ type: UInt.Type) throws -> UInt { return .random(in: .min ... .max) } func decode(_ type: UInt8.Type) throws -> UInt8 { return .random(in: .min ... .max) } func decode(_ type: UInt16.Type) throws -> UInt16 { return .random(in: .min ... .max) } func decode(_ type: UInt32.Type) throws -> UInt32 { return .random(in: .min ... .max) } func decode(_ type: UInt64.Type) throws -> UInt64 { return .random(in: .min ... .max) }

— 7:45

Now we can generate all kinds of integers using our decoder. try Int(from: ArbitraryDecoder()) // 64355822193002337 try Int(from: ArbitraryDecoder()) // 84125722372970218 try Int(from: ArbitraryDecoder()) // 204090925061536467 try Int(from: ArbitraryDecoder()) // -6152064603888612 try Int(from: ArbitraryDecoder()) // 81288851310187473 try UInt8(from: ArbitraryDecoder()) // 180 try UInt8(from: ArbitraryDecoder()) // 125 try UInt8(from: ArbitraryDecoder()) // 235 try UInt8(from: ArbitraryDecoder()) // 209 try UInt8(from: ArbitraryDecoder()) // 175

— 8:16

We can decode doubles and floats using mostly the same API, but we don’t have min and max to work with, so we need to make another decision. A random value between zero and one can be pretty useful, so let’s go with that. func decode(_ type: Double.Type) throws -> Double { return .random(in: 0...1) } func decode(_ type: Float.Type) throws -> Float { return .random(in: 0...1) }

— 8:40

And we can take them for a spin. try Double(from: ArbitraryDecoder()) // 0.1789965677483616 try Double(from: ArbitraryDecoder()) // 0.3271641345644632 try Double(from: ArbitraryDecoder()) // 0.3619418155862117 try Double(from: ArbitraryDecoder()) // 0.7558850014202971 try Double(from: ArbitraryDecoder()) // 0.3374934060055695 try Float(from: ArbitraryDecoder()) // 0.07993001 try Float(from: ArbitraryDecoder()) // 0.082057 try Float(from: ArbitraryDecoder()) // 0.7572621 try Float(from: ArbitraryDecoder()) // 0.9800898 try Float(from: ArbitraryDecoder()) // 0.07348335

— 8:53

We only have a few fatal errors left to fix on the single value container, so let’s walk through them.

— 8:55

The decodeNil method returns true or false depending on whether or not the decoded value should be nil . We can use Bool.random() again to decode nil half of the time. func decodeNil() -> Bool { return .random() }

— 9:17

Decoding a String is slightly tougher. String doesn’t come with a random function, so we’re going to have to write our own. We can generate a randomly sized array, populate it with random character codes, convert them to characters, and join them into a string. func decode(_ type: String.Type) throws -> String { return Array(repeating: (), count: .random(in: 0...280)) .map { String(UnicodeScalar(UInt8.random(in: .min ... .max))) } .joined() }

— 10:50

Finally, we have this generic decode method that works on any Decodable type. How can we implement this?

— 11:06

We need to somehow return an instance of T from this method, and the only thing we know about T is that it’s Decodable . Well, the Decodable protocol contains an initializer, init(from decoder:) , that we can use to instantiate a T . We just need a decoder! We could have passed our ArbitraryDecoder to our single value container when we instantiated it, but that’s not even really necessary here, since our decoder isn’t really stateful, so let’s just create a whole new one to get things building. func decode<T>(_ type: T.Type) throws -> T where T: Decodable { return try T(from: ArbitraryDecoder()) }

— 11:32

And now we’ve fully implemented the single value decoder!

— 11:39

Now we can decode arbitrary strings. try String(from: ArbitraryDecoder()) // "…" try String(from: ArbitraryDecoder()) // "…" try String(from: ArbitraryDecoder()) // "…" try String(from: ArbitraryDecoder()) // "…" try String(from: ArbitraryDecoder()) // "…"

— 11:48

Even optional strings! try String?(from: ArbitraryDecoder()) // nil try String?(from: ArbitraryDecoder()) // "…" try String?(from: ArbitraryDecoder()) // "…" try String?(from: ArbitraryDecoder()) // "…" try String?(from: ArbitraryDecoder()) // "…"

— 11:54

What about other single values, like dates? try Date(from: ArbitraryDecoder()) // Dec 31, 2000 at 7:00 PM try Date(from: ArbitraryDecoder()) // Dec 31, 2000 at 7:00 PM try Date(from: ArbitraryDecoder()) // Dec 31, 2000 at 7:00 PM try Date(from: ArbitraryDecoder()) // Dec 31, 2000 at 7:00 PM try Date(from: ArbitraryDecoder()) // Dec 31, 2000 at 7:00 PM

— 12:01

Okay, that’s a bit strange. It looks like they’re all the same value. Are they? Let’s check them all against a constant reference date. try Date(from: ArbitraryDecoder()).timeIntervalSinceReferenceDate // 0.2449786010351569 try Date(from: ArbitraryDecoder()).timeIntervalSinceReferenceDate // 0.9169184545284704 try Date(from: ArbitraryDecoder()).timeIntervalSinceReferenceDate // 0.8263069788210048 try Date(from: ArbitraryDecoder()).timeIntervalSinceReferenceDate // 0.7331632289872343 try Date(from: ArbitraryDecoder()).timeIntervalSinceReferenceDate // 0.8949193994447776

— 12:19

Oh wow, they’re all different, but by under a second. It turns out that Date stores itself as a TimeInterval , or Double , and that this value represents an offset to the reference date, January 1, 2001. Because we decode Double as a random number between zero and one, we’re getting very little variation here, so maybe we should change those implementations up a bit. func decode(_ type: Double.Type) throws -> Double { return .random(in: -1_000_000_000...1_000_000_000) } Now we get a lot more variation when we generate random dates try Date(from: ArbitraryDecoder()) // Dec 19, 2021 at 5:10 AM try Date(from: ArbitraryDecoder()) // Sep 17, 2015 at 12:12 AM try Date(from: ArbitraryDecoder()) // Jan 25, 2002 at 10:52 AM try Date(from: ArbitraryDecoder()) // Oct 12, 1969 at 1:27 PM try Date(from: ArbitraryDecoder()) // Feb 2, 2003 at 4:35 PM

— 13:01

It’s weird that we had to change logic upstream on Double to gain this variation, though.

— 13:16

What if we want to generate random UUIDs? try UUID(from: ArbitraryDecoder()) // An error was thrown and was not caught: // ▿ DecodingError // ▿ dataCorrupted : Context // - codingPath : 0 elements // - debugDescription : "Attempted to decode UUID from invalid UUID string." // - underlyingError : nil Uh oh, we get a runtime error! UUIDs appear to get decoded from strings, and our random strings are not likely to be in the proper format, so it seems like we’ll need to solve this problem a different way. Decoding more structure

— 14:18

Let’s keep going. While we’ve implemented the single value decoding container, which allows us to pluck single values out of thin air, the real power of Decodable is that it can walk over any type, including more complicated, nested ones. We’re going to need to implement the keyed decoding container, though, to get there.

— 15:00

What if we want to decode a random value of a more complicated data type, like this struct? struct User: Decodable { let id: Int let name: String let email: String }

— 15:11

What we want to be able to do is decode a User using our ArbitraryDecoder . try User(from: ArbitraryDecoder()) Fatal error: : file Arbitrary.playground, line 12

— 15:26

This currently hits one of our fatalError() s, so let’s fix it. struct ArbitraryDecoder: Decoder { … func container<Key: CodingKey>( keyedBy type: Key.Type ) throws -> KeyedDecodingContainer<Key> { return fatalError() } The decoder is trying to produce a keyed decoding container using this method. KeyedDecodingContainer is a type that’s generic over a coding key. Unlike the SingleValueDecodingContainer , it’s not a protocol, but a struct. Let’s see what it takes to instantiate one so that we can return it from this method.

— 15:56

It has an initializer that takes something that conforms to the KeyedDecodingContainerProtocol , which will be another container type that we can implement. struct ArbitraryDecoder: Decoder { … func container<Key: CodingKey>( keyedBy type: Key.Type ) throws -> KeyedDecodingContainer<Key> { return KeyedDecodingContainer(KeyedContainer()) } struct KeyedContainer: KeyedDecodingContainerProtocol { } … } The compiler’s not quite happy yet, so let’s have Xcode stub things in again. struct KeyedContainer: KeyedDecodingContainerProtocol { typealias Key = <#type#> } Ah, if we look at KeyedDecodingContainerProtocol , it has an associated type, Key that is constrained to CodingKey . We can implement our keyed container by making it generic over this associated key type. struct KeyedContainer<Key: CodingKey>: KeyedDecodingContainerProtocol { }

— 16:40

Alright, that satisfies one of the compiler errors, let’s try having Xcode stub things out for us again. struct KeyedContainer<Key: CodingKey>: KeyedDecodingContainerProtocol { var codingPath: [CodingKey] var allKeys: [Key] func contains(_ key: Key) -> Bool { <#code#> } func decodeNil(forKey key: Key) throws -> Bool { <#code#> } func decode(_ type: Bool.Type, forKey key: Key) throws -> Bool { <#code#> } func decode(_ type: String.Type, forKey key: Key) throws -> String { <#code#> } func decode(_ type: Double.Type, forKey key: Key) throws -> Double { <#code#> } func decode(_ type: Float.Type, forKey key: Key) throws -> Float { <#code#> } func decode(_ type: Int.Type, forKey key: Key) throws -> Int { <#code#> } func decode(_ type: Int8.Type, forKey key: Key) throws -> Int8 { <#code#> } func decode(_ type: Int16.Type, forKey key: Key) throws -> Int16 { <#code#> } func decode(_ type: Int32.Type, forKey key: Key) throws -> Int32 { <#code#> } func decode(_ type: Int64.Type, forKey key: Key) throws -> Int64 { <#code#> } func decode(_ type: UInt.Type, forKey key: Key) throws -> UInt { <#code#> } func decode(_ type: UInt8.Type, forKey key: Key) throws -> UInt8 { <#code#> } func decode(_ type: UInt16.Type, forKey key: Key) throws -> UInt16 { <#code#> } func decode(_ type: UInt32.Type, forKey key: Key) throws -> UInt32 { <#code#> } func decode(_ type: UInt64.Type, forKey key: Key) throws -> UInt64 { <#code#> } func decode<T: Decodable>( _ type: T.Type, forKey key: Key ) throws -> T { <#code#> } func nestedContainer<NestedKey: CodingKey>( keyedBy type: NestedKey.Type, forKey key: Key ) throws -> KeyedDecodingContainer<NestedKey> { <#code#> } func nestedUnkeyedContainer( forKey key: Key ) throws -> UnkeyedDecodingContainer { <#code#> } func superDecoder() throws -> Decoder { <#code#> } func superDecoder(forKey key: Key) throws -> Decoder { <#code#> } } Alright, it looks a lot like, but perhaps a little bit more than, what we saw before.

— 17:00

Let’s make the compiler happy first. We can start by ignoring codingPath again. And we can ignore allKeys , which is a list of keys the decoder can work with that we don’t need to worry about. var codingPath: [CodingKey] = [] var allKeys: [Key] = []

— 17:17

Now it looks like we have to reimplement a lot of decode logic, but luckily, some Swift magic allows us to delete most of them and they’ll be forwarded on to the single value container decoding logic, so let’s delete them and fatalError() the rest. func contains(_ key: Key) -> Bool { fatalError() } func decodeNil(forKey key: Key) throws -> Bool { fatalError() } func decode<T: Decodable>( _ type: T.Type, forKey key: Key ) throws -> T { fatalError() } func nestedContainer<NestedKey: CodingKey>( keyedBy type: NestedKey.Type, forKey key: Key ) throws -> KeyedDecodingContainer<NestedKey> { fatalError() } func nestedUnkeyedContainer( forKey key: Key ) throws -> UnkeyedDecodingContainer { fatalError() } func superDecoder() throws -> Decoder { fatalError() } func superDecoder(forKey key: Key) throws -> Decoder { fatalError() }

— 17:50

Alright, everything’s compiling, but we still have a lot more work to do.

— 17:56

For a hint of what’s next, let’s try to decode a user to see if anything blows up. try User(from: ArbitraryDecoder()) Fatal error: : file Arbitrary.playground, line 29

— 18:03

Alright, looks like we need to implement the generic decoding method, which is the same body as the one from our single value container. func decode<T: Decodable>( _ type: T.Type, forKey key: Key ) throws -> T { return try T(from: ArbitraryDecoder()) } Let’s run our playground again and see what we need to implement next.

— 18:29

And…no compiler errors. We’ve decoded a random User ! Let’s print it to see its fields. print(try User(from: ArbitraryDecoder())) // User( // id: -498815911111114036, // name: ".\u{18}±\u{7F}ïËù\u{08}âG¬Đúj©KÈLH…", // email: "ù\u{19}" // )

— 18:33

It gave us a user with an id that’s a very large, negative number, a really garbage-y name, and a very strange email. Our decoder is able to generate these random user values, though they contain some very weird information.

— 18:52

Alright, so this is pretty amazing! By merely tagging our User struct with the Decodable protocol, we were able to generate random values for free. However, there are some really strange values coming in. The name of the user is total garbage, and the emails aren’t valid at all, and the ids can come back negative!

— 19:16

On top of that, we haven’t even finished implementing Decodable , there’s more than a dozen fatalError s left to implement, and they are mostly doable but a lil tricky. Like when generating a random array you have to hard code the maximum and minimum size you are going to allow the arrays to be, which may work for some domains and not others. And what about random dictionaries? You have to generate random keys, which may collide, so you have to decide how to handle those.

— 19:51

The system is very rigid and there aren’t a lot of points of customization. Really, the only customization we can do is on our ArbitraryDecoder by providing some kind of configuration that gets injected into all of the various methods of the decoder and decoding containers, but it’ll still be very rigid. Till next time…

— 20:13

So what we are seeing is that we put in a lot of upfront work with Decodable , and got some impressive results out it because we can effortlessly generate a random value from pretty much any data type we define. However, the system is very rigid and will mostly create invalid values for your domain, like email addresses and positive user ids.

— 20:45

Maybe there’s another way. We saw last time that our Gen type was able to do things that the Swift 4.2 randomness APIs were not capable of, so maybe it can help us out again? Downloads Sample code 0031-arbitrary-pt1 Point-Free A hub for advanced Swift programming. Brought to you by Brandon Williams and Stephen Celis . Content Become a member The Point-Free Way Beta previews Gifts Videos Collections Free clips Blog More About Us Community Slack Mastodon Twitter BlueSky GitHub Contact Us Privacy Policy © 2026 Point-Free, Inc. All rights are reserved for the videos and transcripts on this site. All other content is licensed under CC BY-NC-SA 4.0 , and the underlying source code to run this site is licensed under the MIT License .