Why is front-end testing even a thing? We have LiveView, right? I can run tests on my components and everything is magical.

What is Wallaby?

You may have heard of Wallaby at the Big Elixir when Britton Broderick gave a talk about using it with LiveView, or maybe you knew of it before that. If you aren’t aware of Wallaby, it’s a testing library self-described as help to “test your web applications by simulating realistic user interactions.”

Why do we use Wallaby?

At Binary Noggin, we use Wallaby to cover numerous cases in our Phoenix applications, but we most regularly call on Wallaby for test coverage when we have Javascript and LiveView living on the same page. Wallaby lets us verify behavior in the browser in lieu of verifying the output of a function like most LiveView tests. Lest we draw the ire of the Twittersphere: we use and love LiveView tests; however, when we need some mixed interaction, our LiveView tests don’t quite get us there.

What problem did we encounter that wasn’t already solved?

The simplest way to get started with Wallaby browser testing is using ChromeDriver, a webdriver, to simulate user interactions in Chrome. In order for these tests to work, the version of the Chrome browser and the ChromeDriver server need to match.

When Chrome and the ChromeDriver server version did not match, we experienced all sorts of ambiguous errors in our test results. Usually, after 10 to 20 minutes, someone would remember they had recently updated Chrome, “Ah! I forgot to update ChromeDriver! Give me just a minute while I download the most recent version.”

** (RuntimeError) invalid session id
code: |> Module.view(module.token.id)
stacktrace:
  (wallaby 0.29.1) lib/wallaby/httpclient.ex:136: Wallaby.HTTPClient.check_for_response_errors/1
  (wallaby 0.29.1) lib/wallaby/httpclient.ex:56: Wallaby.HTTPClient.make_request/5
  (wallaby 0.29.1) lib/wallaby/webdriver_client.ex:329: Wallaby.WebdriverClient.visit/2
  (wallaby 0.29.1) lib/wallaby/driver/log_checker.ex:6: Wallaby.Driver.LogChecker.check_logs!/2
  (wallaby 0.29.1) lib/wallaby/browser.ex:1235: Wallaby.Browser.visit/2

Worse yet, we usually experienced this problem one developer after another until everyone had updated their local version of ChromeDriver. You can probably see where this becomes a bit of a time sink.

How did we solve our problem?

After a handful of run-ins with this problem, we realized it would be more convenient for the Wallaby library to tell us our dependencies were not synced instead of letting us get into actually running tests with mismatching versions of Chrome and ChromeDriver.

Initial solution?

First, we found a solution that worked for us locally. We added a check in our Mix alias commands before we executed Wallaby tests to warn us if there was a version mismatch between Chrome and ChromeDriver. This satisfied our immediate need

defmodule Mix.Tasks.Wallaby.Chromedriver do
  @moduledoc "Compares chrome version with chromedriver version and errors when major, minor and build numbers do not match."
  @shortdoc "Errors when chrome and chromedriver version do not align."

  use Mix.Task

  @impl Mix.Task
  def run(_) do
    chrome_driver =
      case System.find_executable("chromedriver") do
        nil ->
          IO.puts("chromedriver is not in PATH")
          exit({:shutdown, 1})

        path ->
          path
      end

    {chrome_driver_version_string, 0} = System.cmd(chrome_driver, ["--version"])

    chrome_driver_version =
      chrome_driver_version_string
      |> String.split(" ")
      |> Enum.at(1)
      |> String.split(".")
      |> Enum.slice(0..2)
      |> Enum.join(".")

    # TODO: figure out how to find google chrome programmaticaly
    {google_chrome_version_string, 0} =
      case :os.type() do
        {:unix, :darwin} ->
          System.cmd("/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome", [
            "--version"
          ])

        {:unix, :linux} ->
          # this assumes a ubuntu-ish install location for chrome
          System.cmd("/usr/bin/google-chrome", ["--version"])
      end

    google_chrome_version =
      google_chrome_version_string
      |> String.split(" ")
      |> Enum.at(2)
      |> String.split(".")
      |> Enum.slice(0..2)
      |> Enum.join(".")

    case Version.compare(google_chrome_version, chrome_driver_version) do
      :gt ->
        IO.puts("you need to download chromedriver #{google_chrome_version}")
        exit({:shutdown, 1})

      _ ->
        IO.puts("yay chrome and chromedriver match nothing to do here.")
    end
  end
end

Not long after, we realized we likely weren’t the only development team experiencing this time sink.

Why did we share our solution?

We could probably save someone else a little bit of time, too, right? At the very least, a human being wouldn’t have to remember what a given obscure error meant in multiple tests. They would know up front that they need to update their tools in order to get their versions of Chrome and ChromeDriver to match.

We reached out to the current maintainer of Wallaby, Mitchell Hanberg (twitter, github), and asked how he felt about our Mix alias solution. While he was interested in the end result, our method left something to be desired. Mitchell asked us if we would consider writing a pull request to add the behavior into the Wallaby library itself.

Current solution

Since then, we’ve rewritten our implementation and included a check in the initial steps of Wallaby initialization. There was already a check for the minimum version of Chrome, and that seemed like a logical place for us to add another dependency version check. We shared our pull request, and after some adjustments, the changes were accepted.

Conclusion

We know maintaining open-source tools and libraries is a challenging endeavor. We’re grateful to the engineers who give their time and expertise to helping the community. We saw this issue as an opportunity to give back. Each time we see this warning displayed in Wallaby, we’ll have a moment of nostalgia and joy that we’ve saved ourselves, and hopefully others, a few minutes of head-scratching.