Tag Archives: Selenium

Avoiding brittle Selenium tests

When automating tests, avoiding creating brittle tests are very important. In my experience, this something that can easily happen when using Selenium if you don’t take special care.

For instance, when locating elements on the page, you typically would like to refer to them using IDs. However, can you be sure that the IDs stay the same at all times? No. This problem becomes appearent when creating ASP.NET applications in Visual Studio. Visual Studio automatically generates what is called the Client ID for a control, based on its location in the control hierarchy. This id is rendered as both the id attribute and name attribute of an HTML element. For example, an ID might look like this:

ctl00_MainArea_MainPageControl_MainPageSection2_ClearRepeater_ctl01_HyperLink1

As a consequence, if you move a control to another place in the control hierarchy, the ID will be different. If you have referred to this ID in your Selenium test, it will break! Not good.

In order to fix this situation, I created a locator that does a partial match on the element id attribute.

HttpOnly broke my Selenium tests

On my current project (running .NET 2.0), I have been using Selenium to test various security related aspects of the application. (Could Selenium be used for security testing?, Selenium with support for cookie-management) I have been happily using Firefox for running my tests, but today I tried to run the tests in Internet Explorer 7. Without success.

The thing is that I have been using Selenium to verify login related functionality, so for instance a test could be something like these:

  • Test that a user can successfully log in by providing correct username and password
  • Test that a user’s cookie session is ended when logging out

In order to successfully run these tests, I had to manipulate cookies in my tests:

  • To prevent tests from interfering with each other, I had to remove any session cookies in between tests
  • Test for existence of session cookies

When running my tests in Firefox, this worked well. I could perform operations on the .ASPXAUTH cookie, which is the cookie that .NET uses to identify an authenticated session. When running IE 7 it breaks. The reason for this, is that Microsoft has created a new attribute on cookies called ‘HttpOnly’ that .NET uses, and the Set-cookie HTTP header looks for instance like this:

Set-Cookie: .ASPXAUTH=bisxfb45rbiclmjmqu4aa345893763387328743238736; path=/; HttpOnly

IE 6 SP1 (and apparently also IE 7) makes such cookies inaccessible by JavaScript, as explained here: Mitigating Cross-site Scripting With HTTP-only Cookies.
Hence, my Selenium tests were unable to test for and to manipulate these cookies.

I have a mixed feeling about this. Everything that helps security makes me happy. However, everything that makes my application hard to test is baaaaaad. And I mean really bad. I think that the technical solution that Microsoft has come up with here is good – it really makes sense. Why should JavaScript be able to manipulate session cookies like these anyway? I cannot think of any good use case for that. However, this is proprietary stuff that Microsoft has come up with and is not an agreed standard. Makes my life as a developer harder. Not good.

Selenium with support for cookie-management

I blogged earlier about using Selenium for security testing. One of the shortcomings that I pointed out, was that session handling (i.e. cookie handling) was needed. I went ahead and created extensions for this. As of version 0.8.0, Selenium now supports this out of the box. This means that Selenium now is able to test scenarios such as logging in and logging out from web applications, in additional to other cookie-based functionality. Great!

Using Selenium to test input validation

When testing for security, it is important to test that the application under test (AUT) handles unexpected input properly, i.e. does not enter an unsecure state. A typical example of an unsecure response to an unexpected input is to display an exception that reveals information about the system configuration that can help a malicious user breaking in to a system.

As I blogged about earlier, I am trying to use Selenium to do security testing. In order to be able to test how the AUT handles being passed a value that is not present in an HTML dropdown list, I created a Selenium extension that adds a new option to a dropdown list and selects it. This way, the test may add a new value before submitting the form to the application.

Here is the code for the Selenium action.

Could Selenium be used for security testing?

My current project uses an agile project methodology. As such, we use Selenium to create automated acceptance tests for the project. Focusing on the security aspects of the application we are developing, I had an idea about using Selenium for security testing. Basically, there are a few points that led me to this idea:

  • Using Selenium we would perhaps not need to introduce yet another tool into the process, lowering the project’s effort to handle security.
  • Using Selenium, security testing could get attention equal to the acceptance tests.
  • Selenium tests could perhaps be created to simulate Black Hat activity.

Being a Selenium newbie, I started looking at Selenium to see of it could be used for this purpose. I quickly realised that Selenium out of the box does not have the functionality that is needed. It is, however, easily extensible.

I have found some areas where Selenium has shortcomings when it comes to security testing:

  1. Session handling: Proper session handling is important when it comes to security, for instance vulnerabilities like session fixture and rights elevation. To enable testing of session handling, Selenium needs functions for inspecting and deleting cookies. Furthermore, if one where to run test suites with tests which involves sessions, these would have to be deleted in between tests to avoid the tests interfering with each other.
  2. HTTP protocol: Testing which involves inspecting and setting HTTP headers would be an integral part of a security test suite.
  3. Input validation: Testing that the AUT does proper input validation and/or can handle input out of range would be important. For example, one test that comes to mind is whether the AUT can handle being sent a value that is not a part of an HTML drop down list.

Even though Selenium has shortcomings, I am fairly optimistic that these can be overcome, and I would like to investigate this further. For instance, I have created basic cookie handling routines for Selenium which seems to do the job. (Item 1 above) I will post them on my homepage later (www.kongsli.net). For item 3 above, I have created a method that will dynamically add options to dropdown lists on a page before the page is submitted, making it possible to send parameters out of range to the server. Furthermore, I will look into AJAX related techniques to handle item 2 above, possibly using the Dojo framework.