Automated testing in 2.0 for .NET is something I’ve put a lot of work into. Our QE team and build team are not very big and so that often means if I want something done I will have to set it up myself. That’s exactly what I did and so I am able to speak authoritatively on how this process works.
This post does not take into account the testing that the QE team is going to do, but purely stuff that verifies a build. Come release time, QE runs an intense cycle of functional tests that takes days to complete for one build. I’m going to talk about the process that gets it to the beginning of that point. It all starts with LiteCore.
LiteCore gets tested on the following platforms before it makes its way to Couchbase Lite .NET:
- Windows Desktop x86
- Windows Desktop x64
- Ubuntu 16.04 (will become 14.04 later)
This leaves some holes, but they are filled in by testing by other teams, as well as bindings tests which are identical to one version of the native tests except they are done via bindings to another language.
After that, Couchbase Lite is built and packaged. The tests are not run before packaging because so often packaging itself causes issues that are then not caught during testing. So as soon as the build succeeds, the package gets pushed to a continuous integration Nuget feed that is internal to Couchbase. Then the next step begins!
After this is done, a master testing job is triggered which spins off 6 other jobs. This will run the unit tests on:
- .NET Core Windows
- .NET Core macOS
- .NET Core Ubuntu
- Xamarin iOS
- Xamarin Android
Each of these has its own quirks, except for .NET Core which was designed for command line usage. It makes use of the relatively new dotnet xunit command to run tests. UWP is a bit harder. An appx package is created and then tested using the MSTest command line tools. It is compiled using .NET Native to simulate how things will actually work for end user applications. Xamarin is the most annoying of the bunch.
There is no automated testing framework for Xamarin, at least not that I could find. There are UITests, but this is a database so there is no UI. So, as is often the case, I had to brew something up myself. There are test runners for xunit that I made use of, but they usually require interaction with the test runner UI. However, there is an option to automatically start it. That’s useful, but how do we get the results? A long time ago Xamarin wrote a TcpWriter class that would write all the results over TCP instead of writing them to the console and there is an old codebase from 2011 that has a program used to listen for that. With some moderate modifications, this can run for the purpose we needed. The flow will basically go like this:
- Start an instance of the listener in the background and get its PID
- Install and launch the unit test application on the target
- The unit test application is set up to kill its own process once the tests finish
- The listener is set up to end with a status code once the TCP connection is cut, or it detects a “results” line from the stream
- The job will wait for the PID it got in step 1 and check the return code to determine whether or not to pass or fail the job
This took quite a while to get right, especially since it seems that System.Environment.Exit(0) doesn’t work in Android, at least not in Genymotion. The app just stays launched. However in the end I was able to get it working.
If all of the platforms succeeds, then the overall job succeeds. On success, the job will copy the Nuget package from the CI feed to the “Internal” feed which is used by QE. This gives them a nice list of packages that have all passed automated testing so they don’t go down too many rabbit holes. As of the time of writing (DB011 release) there are 246 automated test funtions that get run on every successful build in 6 places, so that’s a nice safety net!
As a bonus fact, come release time (either DB release or actual GA) nothing is changed about the package that was tested except its metadata. This means that the actual files that get used by the end user application are bit for bit the same as those that were tested. That’s why you will see that in the logs there will be a version number like 2.0.0-b0154. That cannot be changed without altering the DLL so it is left alone, and honestly it is nice to have a unique identifier for every build that we have.