There’s always been something vulnerable about posting technical content (or, any content) online. You expose yourself to the possibility of being wrong and/or being judged. Because of that, I’ve spent a lot of time pondering if something was worth sharing at all, if that’s the kind of content people expected to see here… but I’m trying to move out of that way of thinking. With the strong possibility of coming up as naive about this project, I’ve decided to start writing my own web testing framework in Kotlin.

TL;DR: I’ve set a goal for myself of writing my own “web testing framework” in order to learn a few things. I don’t think I will release anything, I am aware that there are existing tools that I should look into, but this is something that’s been in the back of my mind for a while now.

Context

I’m going to talk about Web Apps in this blog post and I want to make it clear that I’m talking about good old server generated HTML with a bit of CSS sprinkled on top. This is not about writing tests for a Single Page Application project using React or some similar technologies. Before I saw the light and left React behind me, I remember that testing was not a real problem there, and I never felt the urge to write my own thing.

I recently broke Comics Outmash after a careless dependency update: it was a minor version update, the tests where passing so I merged the pull-request automatically created by Dependabot and like a cool guy walking away from an explosion, I pressed the “merge when green” button and went to bed. As you can expect, I woke up to a few messages on Discord from people telling me that opening the site would give you a error page. I quickly reverted the last commit and started digging.

I won’t go into details as to why this happened, because it doesn’t matter here (although I have opinions about Dependabot now). Here, writing a simple test that loads the homepage would have surfaced that error immediately. To my defence, this project used to have a few simple integration tests, but I ended up throwing it away because they were annoying to maintain.

What’s a Web Testing Framework

Testing is a complicated topic that I’m passionate about. There are many kinds of tests: unit, integration, functional, and so on. They all have their own specificity as to what they’re testing. For example, I would write a unit test for a simple bit of code that takes a string and returns a hashed version for caching. On the other hand, I would write an integration test for something more complex like making sure that when a user creates an account, the account is properly created in the database and in the case of trying to use an email address that’s already in use… the error is properly surfaced. There are many more.

So, at the risk of being wrong, here is what I what I’m calling a “Web Testing Framework”: a tool that allows me to describe a scenario mostly composed of clicks, forms submissions, and making sure that the page contains what we’re expected to see.

You may be about to tell me “but dude, those already exist” and you would be right. I can name a bunch of tools from the top of my head that would help me achieve this: Puppeteer, Selenium, Playwright… and yet I’m still going to write my own, because I’m hoping to learn something.

I’ve been thinking about this for a long time, but I never bothered to get started. For a while, I’ve had a tendency to magnify the complexity of various tools and technologies, assuming that whatever they were doing was probably super complex, until I either read the documentation or took a look at the code. Writing a web testing framework was very similar in my head. Surely, I couldn’t start until I actually had the time to fully commit to such a massive endeavour? Well screw that, I’m doing it.

Expectations

The interesting part is that I haven’t written a single line of code for this project, because I didn’t want to jump into this without having defined a few goals first.

First, I have pre-defined a few use cases that I would like to test in a simple web app made for this (probably something that allows you to post comments on a blog):

1. Create an account using an email address, a first name, a password and a confirmation password.
2. Sign in, using that password and that email address
3. Go to a blog post and post a comment
4. Sign out

Second, I want an API that is easy to read and easy to adapt, as well. One of the annoying things that comes with application testing is keeping up with the application changing under you. It’s often something simple, like a button’s label being updated. So I’m thinking of doing something pretty verbose that’s easy to return to 6 months later, such as:

Scenario("User Sign Up") {
	GoTo("/")
	
	SubmitForm("Create An Account") {
		Fill("Email Address:", with = "potato@acme.corp")
		Fill("First Name:", with = "Patrick")
		Fill("Password:", with = "123123")
		Fill("Confirm Password:", with = "123123")
	}
}

The validation part has yet to be defined, because nothing I’ve come up with has been really satisfactory, but I was picturing something simple that allows you to assert on the content of the page, like so:

Scenario {
  // ... other steps
  
	Validate {
		content shouldContain "Thank you for creating an account!"
	}
}

At this point, I’m not sure if I want to treat validation something that happens outside of the scenario or as part of it. I know that BDD traditionally defines tests in a “Given A and B”, “When I do C and D”, “Then I should see E” way, but it doesn’t seem that weird (to me) to be able to expect something to happen in the middle of a scenario and keep going.

Finally, I want it to be very easy to debug and understand why a scenario’s execution failed. I’m not sure what that means but I’m thinking having a clear HTML report, making it easy to access the state of an execution (view the content of latest response, including the headers, why a step couldn’t run successfully, etc.)

Those 3 expectations seems to be more than enough to kickstart this small project.

Conclusion

Honestly, I don’t expect much to come off this experiment except a few blog posts. I’m not planning on releasing the source of whatever I end up creating, because that would put a totally different pressure on this. That being said, I will allow myself to change my mind later. I’m nice like that.

If you have opinions on this, want to tell me how naive I’m being or suggest me tools I should look into… well I’m actually open to the idea. You can find me on Mastodon or contact me via email.

(Header photo from Christina Morillo.)