As part of a refactoring, I broke something that shouldn’t have broken (should it ever?). Fortunately, I caught this issue in my tests, but it took me on a stroll into Ktor’s and kotlinx’ serialization’s source code, which then got me thinking about expectations when related to “default” values in public API.

Context

I started refactoring parts of our codebase to make sure that all our services use the same foundations: a base HTTP server with logging, tracing and error handling configured; an HTTP client that’s properly configured with similar stuff, as well as retrying logic and so on. When I moved the last service, my integrations tests started failing for reasons unrelated to the changes. All of a sudden, most API calls from these tests were failing with HTTP 400, complaining about missing parameters.

I immediately got nervous because I find serialization in Kotlin a little confusing at times, especially coming from Swift’s Codable protocols. While not perfect, I spent enough time using them that it became a “devil you know” type of situation. Kotlin’s serialization framework is extremely powerful, but because this part of the code uses class discriminators, I started sweating profusely when I couldn’t pinpoint the exact change to blame. Fortunately, the issue was unrelated.

In the integration tests that were failing, I was sending HTTP requests to a service that I’d previously documented using an OpenAPI specification. Additionally, the model I was using to prepare the request had been previously generated, based on that spec. This is roughly what the data class looks like:

@Serializable
data class User(
	val fullname: String,
	val role: Role = Role.USER // note the default value here
)

For some reason, the requests started failing because the body couldn’t be deserialized from JSON: it was missing a value for “role”, which is a mandatory parameter.

After some digging, I realized that the issue was in the client, not in the server. By moving from Ktor’s HttpClient initialized specifically for my tests to a fancy one with all the shiny stuff (still using Ktor’s client internally though!), I had effectively changed the behaviour of the serialization to/from JSON: values with a default value where no longer encoded.

Serialization in Ktor

I like Ktor because it’s simple and easy to extend with plugins. To send a request that serializes and deserializes JSON (and many other formats), you have to install the Content Serialization plugin and configure it for JSON, like so:

HTTPClient(CIO) {
    install(ContentNegotiation) {
        json()
    }
}

This is effectively what I had in my integration test.

On the other hand, in my shiny client, the JSON serialization is configured so that unknown keys (fields in the JSON string that are not mapped to a property on your model) are ignored, so they don’t cause the serialization to fail, like so:

HTTPClient(CIO) {
    install(ContentNegotiation) {
        json(Json {
            ignoreUnknownKeys = true
        })
    }
}

You might be tempted to believe that those 2 call to json will cause the serialization to behave exactly the same, except for that one change, or you might simply be smarter than me, in which case… congrats, I guess?

Actually, they have different values. You can see in the repository what those default values are. If we compare this configuration to the one from an empty Json {}, a bunch of properties are different:

• allowSpecialFloatingPointValues is true
• allowStructuredMapKeys is true
• encodeDefaults is true
• ignoreUnknownKeys is false
• isLenient is true

In my situation, the property change that caused the tests to fail is encodeDefaults, which is set to false on Ktor’s default JSON configuration. All I had to do to fix my test was to set it back to true in the configuration.

Conclusion

I don’t think anyone is to blame (or even “wrong”) here, since I think we’re simply in a case of “opinionated defaults” on Ktor’s part. My only issue is that the documentation states that you can configure the behaviour of the JSON serialization plugin, but doesn’t mention that by choosing to set ignoreUnknownKeys to true (for example), you’re ending up with changes that you might not expect.

Note that there is another way to mitigate this, which is to make changes to the default configuration, like so:

HTTPClient(CIO) {
    install(ContentNegotiation) {
        json(Json(DefaultJson) { 
            ignoreUnknownKeys = true 
        })
    }
}

Here, the ignoreUnknownKeys is the only property that differs from the default configuration provided by Ktor. The bonus is that it becomes really (really) hard to miss that this is the part that configures the JSON serialization.