Category Archives: Software testing

Pure functions and testability

When trying to introduce unit tests into an existing project, code dependencies is often a problem. During a teeth-grinding session grappling with this problem I came to think about pure functions.

Pure functions are functions that (according to Wikipedia‘s definition):

  • always evaluates the same result value given the same argument value(s). The function result value cannot depend on any hidden information or state that may change as program execution proceeds, nor can it depend on any external input from I/O devices, and
  • evaluation of the result does not cause any semantically observable side effect or output, such as mutation of mutable objects or output to I/O devices.

Seems to me that pure functions would be a wet dream for unit testing. Imagine, you could pass in whatever the function needed to execute, and then check its return value. No mocking about with mock objects with side effect expectations needed!

But alas, we live in a world with non-pure functions. Even in functional programming, which does explicitly keep any state, you find them. In an object oriented language a primary idea is that objects should carry state. An object has a state. Period. However, if we can be inspired by the notion of pure functions, and make methods that have less dependencies, we will be much better off when it comes to unit testing.

I my experience, the ASP.NET programming model with ASPX code behind files is a model that promotes writing methods that are problematic to create unit tests for.

Take, for example, the following (theoretical, I admit) code behind snippet example:

protected void Page_Load(object sender, EventArgs e)
    {
        ShowOrHideEditButton(Page.User.Identity.Name);
    }
    void ShowOrHideEditButton(string currentUser)
    {
        if (Request.QueryString["user"].Equals(currentUser))
        {
            EditButton.Enabled = true;
        }
        else
        {
            EditButton.Enabled = false;
        }
    }

So, what is the problem here? Actually, there are two problems.

  1. The ShowOrHideMethod is not only dependant on its input parameters, but also on another member variable. When looking at the method signature, you do not see this, and might be misleaded about what the outcome of the methods is dependent on.
  2. The method does not return any value, but has a side-effect on its environment (setting the edit button visibility).

These two issues, makes it troublesome to test the method in a unit test. You have spend time to set up the methods context, and you have to measure the method’s side effects in order to make assesments about its result.

A much better approach would be something like this:

    protected void Page_Load(object sender, EventArgs e)
    {
        EditButton.Enabled = ShowOrHideEditButton(Page.User.Identity.Name, Request.QueryString["user"]);
    }
    bool ShowOrHideEditButton(string currentUserName, string profilePerson)
    {
        return (currentUserName.Equals(profilePerson));
    }

This way, we can easily test the ShowOrHideEditButton method in a unit test without having to use a great deal test code to set up the test context.

For increased testability, I would suggest the following for better method purity:

  • A method should evaluate the same result, given the same argument value(s) and object state.
  • A method execution result should be represented as, in order of preference: 1) a return value, 2) changed object state (mutation), or 3) a side-effect on an I/O device

Internet Explorer automation/Watin: catching navigation error codes

One question have troubled me for some time when automating Internet Explorer (actually, I am doing web testing with Watin): how to test for HTTP status codes. Finally, I figured out how to do this. The lies in an event that the InternetExplorer object raises when navigation is unsuccessful.

I ended up with writing a C# helper class:

using System.Net;
using WatiN.Core;
using SHDocVw;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using System.Globalization;
namespace Test
{
    public class NavigationObserver
    {
        private HttpStatusCode _statusCode;
        public NavigationObserver(IE ie)
        {
            InternetExplorer internetExplorer = (InternetExplorer)ie.InternetExplorer;
            internetExplorer.NavigateError += new DWebBrowserEvents2_NavigateErrorEventHandler(IeNavigateError);
        }
        public void ShouldHave(HttpStatusCode expectedStatusCode)
        {
            if (!_statusCode.Equals(expectedStatusCode))
            {
                Assert.Fail(string.Format(CultureInfo.InvariantCulture, "Wrong status code. Expected {0}, but was {1}",
                    expectedStatusCode, _statusCode));
            }
        }
        private void IeNavigateError(object pDisp, ref object URL, ref object Frame, ref object StatusCode, ref bool Cancel)
        {
            _statusCode = (HttpStatusCode)StatusCode;
        }
    }
}

Note that I use Visual Studio test runner to run my web tests. Then, I can use this in my test:

using (IE ie = new IE())
{
    NavigationObserver observer = new NavigationObserver(ie);
    ie.GoTo("http://some.where.com");
    observer.ShouldHave(HttpStatusCode.NotFound);
}

Sending POST requests with Watin

When doing web testing using Watin, it is not trivial to be able to do a POST request to the server. However, with the help this article on microsoft.com, I was able to figure out how. I ended up with writing this class:

public class Navigator
{
    private IE _ie;

    public Navigator(IE ie) { _ie = ie; }

    public void Post(Uri baseUri, params KeyValuePair<string, object>[] postData)
    {
        object flags = null;
        object targetFrame = null;
        object headers = "Content-Type: application/x-www-form-urlencoded" + Convert.ToChar(10) + Convert.ToChar(13);
        object postDataBytes = MakeByteStreamOf(postData);
        object resourceLocator = baseUri.ToString();
        IWebBrowser2 browser = (IWebBrowser2)_ie.InternetExplorer;
        browser.Navigate2(ref resourceLocator, ref flags, ref targetFrame, ref postDataBytes, ref headers);
        _ie.WaitForComplete();
    }

    private static byte[] MakeByteStreamOf(KeyValuePair<string, object>[] postData)
    {
        StringBuilder sb = new StringBuilder();
        if (postData.Length > 0)
        {
            foreach (KeyValuePair<string, object> postDataEntry in postData)
            {
                sb.Append(postDataEntry.Key).Append('=').Append(postDataEntry.Value).Append('&');
            }
            sb.Remove(sb.Length - 1, 1);
        }
        return ASCIIEncoding.ASCII.GetBytes(sb.ToString());
    }
}

For example, I can use it like so:

using (IE ie = new IE())
{
    Navigator navigator = new Navigator(ie);
    navigator.Post(new Uri("http://www.foo.com/"), new KeyValuePair<string, object>("p", 1));
    Assert.AreEqual("OK", ie.Text);
}

Mark of the web

Earlier, I blogged about running JavaScript tests from Visual Studio 2005 where I automate Internet Explorer to load the HTML tests files directly from disk (file:/// protocol). One hurdle in this approach is the Windows XP SP2 local machine zone lockdown which restricts running scripts in local files. (More info here).

The easiest fix to this problem seemed to be the Mark of the web. Fair enough, I went on to add the mark to my JavaScript html test files, like so:

<!-- saved from url=(0019)http://www.foo.com/ -->

Alas, no luck. I still got the annoying message that I had to tick off the warning message each time Internet Explorer started. The trick here is that the line with the code above must en with CR LF (as mentioned here). Yikes. This problem was caused by Visual Studio not using the “Microsoft/Windows/MS-DOS” standard CR LF, but Unix style line delimiters! I opened up the file in Notepad++, selected Format–>Convert to Windows format, and lo it worked like a charm!

JavaScript unit testing in Visual Studio 2005

In my current project, we see an increasing use of JavaScript in our web application. Web 2.0, ya know. Hence, we have a growing need for being able to create automated JavaScript tests. A colleague of mine, Aslak Hellesøy, suggested that we should have a look at script.aculo.us‘s unit test framework.

Fair enough, the script.aculo.us unit test framework seemed to do the trick. We use both prototype and script.aculo.us in our project, so the script.aculo.us unit test framework seemed to fit in nicely. My next concern would be to make it as easy as possible for the developers to run the tests as well as integrating
it into our continuous integration builds (we use TFS). The test results should somehow be made available in the test run report. The steps needed to achieve this would be something like
this:

  1. Write the JavaScript unit tests and place them in an HTML file
  2. From a Visual Studio Team System test method, fire up a browser that loads the HTML file and runs the test
  3. Fetch the result of the JavaScript unit tests and incorporate them into the test report

script.aculo.us provides the pieces for 1 and 2 above out of the box. It also supports POSTing the results back to a web server using Ajax. That could be an approach that would be a part of the solution for point 3 above. However, I felt that this approach is not an optimal solution. First of all because you would have to to
make the tests available thru a web server, the web server would need to receive the results, and store them somewhere. Then, the Visual Studio test would have to fetch the results and incorporate them. It all adds up to seem quite fragile. It is important that a test run has as few external dependencies as possible. Hence, I
wanted a separate approach. The browser should be able to load the JavaScript tests directly from the file system (file:///…). Then, after running the tests we should be able to traverse the DOM of
the HTML page directly to fetch the results.

As a result, we ended up with this approach:

  • Fire up Internet Explorer from .NET via its COM interface (Internet Explorer Object, SHDocVw.dll). You can find code for this here.
  • Load the file directly from the file system (file:///…) and execute the tests
  • Traverse the HMTL DOM to fetch the test results (using Microsoft HTML Object Library, mshtml.dll)
  • Report any error messages back using Assert.fail(“…message…”)

Avoiding brittle Selenium tests

When automating tests, avoiding creating brittle tests are very important. In my experience, this something that can easily happen when using Selenium if you don’t take special care.

For instance, when locating elements on the page, you typically would like to refer to them using IDs. However, can you be sure that the IDs stay the same at all times? No. This problem becomes appearent when creating ASP.NET applications in Visual Studio. Visual Studio automatically generates what is called the Client ID for a control, based on its location in the control hierarchy. This id is rendered as both the id attribute and name attribute of an HTML element. For example, an ID might look like this:

ctl00_MainArea_MainPageControl_MainPageSection2_ClearRepeater_ctl01_HyperLink1

As a consequence, if you move a control to another place in the control hierarchy, the ID will be different. If you have referred to this ID in your Selenium test, it will break! Not good.

In order to fix this situation, I created a locator that does a partial match on the element id attribute.

Using Selenium to test input validation

When testing for security, it is important to test that the application under test (AUT) handles unexpected input properly, i.e. does not enter an unsecure state. A typical example of an unsecure response to an unexpected input is to display an exception that reveals information about the system configuration that can help a malicious user breaking in to a system.

As I blogged about earlier, I am trying to use Selenium to do security testing. In order to be able to test how the AUT handles being passed a value that is not present in an HTML dropdown list, I created a Selenium extension that adds a new option to a dropdown list and selects it. This way, the test may add a new value before submitting the form to the application.

Here is the code for the Selenium action.

Quantum mechanics in software testing

The other day, a colleague of mine and I discussed our test environment. We concluded that neither
of us had used it for a while and were hence uncertain of its state. Then, I came to think of Schrödinger’s cat, the thought experiment introduced by physicist Erwin Schrödinger in which a cat is placed in a sealed box together with a container of poison and a radioactive substance. A radioactive decay would make the poison escape from the container and kill the cat. However, exactly when the decay will occur is not predicable by quantum physics, only a probability of the occurence may be given. So, given that the state of the cat, dead or alive, cannot be observed without opening the box, its state is uncertain.

According to quantum mechanics, an unobserved nucleus is said to be in a superpositioned state,
both decayed and undecayed. Only when it is observed it seizes to be superpositioned.

Now, it is quite funny to make an analogy to this regarding software development. Unless you test
(observe) your system, it is in a superpositioned state: it both works and doesn’t work. The only
way to make it work, is to test it.

Using this analogy we will arrive at the agile software development mantra of test early, test often.
For instance, using continuous integration with unit testing and automatic acceptance testing, your
system will be in a well-defined state at (almost) all times.