Vidar’s Musings

Log4Net is a popular logging framework, and if you have an existing application that you wish to move to Azure compute, you probably want to avoid rewriting your application to use another logging framework. Luckily, keeping Log4Net as your logging tool in Azure is certainly possible, but there are a few hoops you have to jump through to get there.

There are several ways to achieve this goal. I decided to rely as much as possible on a feature provided in Azure Compute that allows for automatically synchronizing certain directories on the instance’s local file system to Azure blob storage. Using this approach, there are only very few changes that need to be done in the application, and indeed none of the existing code needs to be altered.

Baseline: an existing application which uses Log4Net

In order for this example to work, we need an application that we what to move to the cloud:

1. Start off with File->New->Project... in Visual Studio and use the “ASP.NET Web Application” template
2. Add Log4Net capabilities to the application. This can be done by adding a reference to Log4Net using Nuget, and then configuring it like Phil Haack has described here.

Then, run the web application locally in Visual Studio to assert that the logging works.

Enabling the application for Azure

Starting off with the simple ASP.NET web application we created in the previous section, do the following:

1. Right-click on the solution in Visual Studio to select Add->New Project. Use the Windows Azure Project template, and do not add any roles in the dialog box initially.
2. Set the newly created cloud project as the startup project in the solution.
3. Right-click on the Roles-folder of the newly created Azure project and select Add->Web role project in solution... to add the web application project as an Azure Web role.

Now, press F5 to run the application in the local Azure development environment (DevFabric) to see that it works (functionally, not logging-wise)

So, we are done with the prerequisites. Now to the interesting parts!

Setting log directory for Azure

The first issue we will grapple with is the fact that in Azure Compute, the application effectively runs in a sandbox with limited access to the file system, which means that the “standard” approach logging to a file does not work. Basically, the Azure compute role has only access to a certain subdirectory of the file system and the exact location needs to be retrieved by the application at runtime.

In order to retain the existing logging in the application, locating the path to the role’s designated area on the disk can be solved by subclassing one of the appenders that Log4Net provides out of the box. I chose the RollingFileAppender because it provides the ability to split the log into several files. This is beneficial from an operations perspective. Here’s what the custom appender looks like:

using System.Diagnostics;
using System.IO;
using log4net.Appender;
using Microsoft.WindowsAzure.ServiceRuntime;
namespace Demo.Log4Net.Azure
{
    public class AzureAppender : RollingFileAppender
    {
        public override string File
        {
            set
            {
                base.File = RoleEnvironment.GetLocalResource("Log4Net").RootPath + @"\" 
                    + new FileInfo(value).Name + "_"
                    + Process.GetCurrentProcess().ProcessName;
            }
        }
    }
}

What happens here, is that when the configuration is read when the logging framework initializes, it calls our method to set the log file name. This corresponds to the file element in the XML configuration for the appender:

<log4net>
  <appender>
    <param name="File" value="app.log" />
    ...
  </appender>
  ...
</log4net>

What happens here, is that the application asks the role environment for the whereabouts of the local resource called “Log4Net”. This resource is a directory that we specify designated for containing our logs and needs to be specified in the ServiceDefinition.csdef file:

<ServiceDefinition>
  <WebRole name="WebRole1">
    <LocalResources>
        <LocalStorage name="Log4Net" sizeInMB="2048" cleanOnRoleRecycle="true"/> 
    </LocalResources>
  </WebRole>
</ServiceDefinition>

When we have the path of the local resource, it is used to construct an absolute path for the log file. Also note that the current process name is appended to the filename. This is done because if you run the application as a WebRole in “Full IIS” mode in Azure, the web application and the RoleEntryPoint code run in different processes. (If you look at blog entries on the Internet for Azure information, you should have in mind that the “Full IIS” mode was introduced with the Azure SDK version 1.3 in late 2010, and information that predates this might not be valid for the current Azure version.) This means that if there are log entries in the RoleEntryPoint as well as in the rest of your application, two processes would potentially try to keep a write lock on the file at the same time. Therefore, we use one log file for each process. Note that this is not a relevant topic for Worker roles. For more on the execution model, take a look here.

So, now that the new custom appender is ready, we need to change the Log4Net configuration to use it. Basically, we change the assembly type in the appender configuration section so that the configuration looks like this:

<log4net>
  <appender name="AzureRollingLogFileAppender" type="Demo.Log4Net.Azure.AzureAppender, Demo.Log4Net.Azure">
    <param name="File" value="app.log" />
    <param name="AppendToFile" value="true" />
    <param name="RollingStyle" value="Date" />
    <param name="StaticLogFileName" value="false" />
    <param name="DatePattern" value=".yyyy-MM-dd.lo\g" />
    <layout type="log4net.Layout.PatternLayout">
      <conversionPattern value="%date [%thread] %-5level %logger [%appdomain] - %message%newline" />
    </layout>
  </appender>
  <root>
    <level value="DEBUG" />
    <appender-ref ref="AzureRollingLogFileAppender" />
  </root>
</log4net>

Now it’s time to run the application to see if the logging works. First, deploy to devfabric, and then open the Windows Azure Compute Emulator. Right-click on the running instance, and click on Open local store....

Then navigate to the ‘directory\Log4Net‘ to find the log files:

Persisting logs to Azure blob storage

The next issue we need to handle, is the fact that the local file system in an Azure role instance is not persistent. Local data will be lost when the application is redeployed (and also when the Role recycles, if you have chosen to do so). Furthermore, the only way to access the local file system is using a Remote Desktop Connection. In theory, you could probably also make the directory a shared drive accessible over the Internet, but you probably would not want to do that. Besides, it will be a headache if you have a lot of instances.

So, the solution that Azure offers to this, is to have a scheduled synchronization of certain of the local resources (directories) to the Azure blob store. What we need to do, is to add the following code to the descendant of RoleEntryPoint:

public class WebRole : RoleEntryPoint
{
    public override bool OnStart()
    {
        var diagnosticsConfig = DiagnosticMonitor.GetDefaultInitialConfiguration();
        diagnosticsConfig.Directories.ScheduledTransferPeriod = TimeSpan.FromMinutes(5);
        diagnosticsConfig.Directories.DataSources.Add(
                new DirectoryConfiguration
                {
                    Path = RoleEnvironment.GetLocalResource("Log4Net").RootPath,
                    DirectoryQuotaInMB = 2048,
                    Container = "wad-log4net"
                }
        );
        DiagnosticMonitor.Start("Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString", diagnosticsConfig);
        return base.OnStart();
    }
}

…and that’s it. Now you can try to run the application and observe a container called ‘wad-log4net’ will be created in your blob service account that will contain the logs:

(I use the AzureXplorer extension for Visual Studio)

The solution shown here targeted an ASP.NET application running as a WebRole, but the setup works equally well for Worker roles.

The testability of ASP.NET code has long been a challenge; creating unit tests for your ASP.NET code has been difficult. One of the main points of the new ASP.NET MVC framework has been to make code written for it easily testable. However, not many people know that in ASP.NET 3.5, Microsoft has added a few features to make any ASP.NET applications, not only ASP.NET MVC applications, more easy to test. The System.Web.Abstractions assembly adds a few classes to the System.Web namespace that will help the situation. For instance, looking at the documentation for System.Web.HttpRequestBase, it states that

The HttpRequestBase class is an abstract class that contains the same members as the HttpRequest class. The HttpRequestBase class enables you to create derived classes that are like the HttpRequest class, but that you can customize and that work outside the ASP.NET pipeline. When you perform unit testing, you typically use a derived class to implement members that have customized behavior that fulfills the scenario that you are testing.

Very well. Looking at the documentation for HttpRequest, we see that HttpRequest is not a decendent of HttpRequestBase as one might expect from the name. The reason for this is probably that that would break backwards compatability with older versions of ASP.NET. So, how can we exploit the HttpRequestBase then? The answer is the HttpRequestWrapper class which is a decendant of HttpRequestBase and has a constructor that takes an HttpRequest object as a parameter. Then, we can take the HttpRequest object passed to our code from the framework, wrap it inside an HttpRequestWrapper object and pass it on to our code as a HttpRequestBase object. As I will show you in the examples below, this will enable us to create unit tests of our code by creating fake implementations of ASP.NET framework clases (using Rhino.Mocks).

Example #1: Testing a page codebehind file

Take, for instance, this simple page codebehind code that we would like to test:

using System;
using System.Web;

public partial class _Default : System.Web.UI.Page { protected void Page_Init(object sender, EventArgs e) { Response.Cache.SetCacheability(HttpCacheability.NoCache); } } The first step to take here, is to extract a method which take the request object as a parameter instead of fetching it from a method in a superclass. In general, this is a variation of the dependency injection pattern which in many situations will help us make our code testable (also, see my earlier related post). Like so:

using System.Web;
using System;

public partial class _Default : System.Web.UI.Page { protected void Page_Init(object sender, EventArgs e) { SetCacheablityOfResponse(Response); }

public void SetCacheablityOfResponse(HttpResponse response)
{
    response.Cache.SetCacheability(HttpCacheability.NoCache);
}

} So, then having extracted our code in a separate method, the next step is to change the parameter type of this method from HttpRequest to HttpRequestBase. Furthermore, when calling this method, we need to wrap the HttpRequest object by creating an instance of HttpRequestWrapper. The code, then, looks like this:

using System.Web;
using System;

public partial class _Default : System.Web.UI.Page { protected void Page_Init(object sender, EventArgs e) { SetCacheablityOfResponse(new HttpResponseWrapper(Response)); }

public void SetCacheablityOfResponse(HttpResponseBase response)
{
    response.Cache.SetCacheability(HttpCacheability.NoCache);
}

} Having now prepared our code for testing, we can create a unit test where we test the SetCacheabilityOfResponse method:

[TestMethod]
public void ShouldSetNoCacheabilityOnDefaultPage()
{
    _Default page = new _Default();
    MockRepository mocks = new MockRepository();
    HttpResponseBase responseStub = mocks.Stub<HttpResponseBase>();
    HttpCachePolicyBase cachePolicyMock = mocks.CreateMock<HttpCachePolicyBase>();
    With.Mocks(mocks).Expecting(delegate
    {
        SetupResult.For(responseStub.Cache).Return(cachePolicyMock);
        cachePolicyMock.SetCacheability(HttpCacheability.NoCache);
        LastCall.On(cachePolicyMock).Repeat.AtLeastOnce();
    }).Verify(delegate
    {
        page.SetCacheablityOfResponse(responseStub);
    });
}

If you are not familiar with Rhino.Mocks or any other mocking framework, there appears to be a lot going on in that test. The basic idea is that we create derivatives of the -Base classes and pass these to the code that we are going to test, mimicking the behavior of the “real” objects that the framework would pass our code at runtime. Also note that in this particular test we test the side effect of our code, namely that the code should call a the SetCacheability method with a specific parameter. This is achieved using a mock object.

Example #2: Testing an HTTP Handler

Given the following HTTP handler code:

using System;
using System.Web;

public class RedirectAuthenticatedUsersHandler : IHttpHandler { public void ProcessRequest(HttpContext context) { if (context.Request.IsAuthenticated) { context.Server.TransferRequest("/farfaraway"); } } } Again, we extract the code we want to test into a separate method, passing it a -Base object and wrap the object passed to us from the framework in a -Wrapper object:

using System;
using System.Web;

public class RedirectAuthenticatedUsersHandler : IHttpHandler { public void ProcessRequest(HttpContext context) { TransferUserIfAuthenticated(new HttpContextWrapper(context)); }

public void TransferUserIfAuthenticated(HttpContextBase context)
{
    if (context.Request.IsAuthenticated)
    {
        context.Server.TransferRequest("/farfaraway");
    }
}

} This allows us to create unit tests for the TransferUserIfAuthenticated method, for instance:

[TestMethod]
public void ShouldRedirectAuthenticatedUser()
{
    RedirectAuthenticatedUsersHandler handler = new RedirectAuthenticatedUsersHandler();
    MockRepository mocks = new MockRepository();
    HttpContextBase httpContextStub = mocks.Stub<HttpContextBase>();
    HttpRequestBase httpRequestBaseStub = mocks.Stub<HttpRequestBase>();
    HttpServerUtilityBase httpServerUtilityMock = mocks.CreateMock<HttpServerUtilityBase>();
    With.Mocks(mocks).Expecting(delegate
    {
        SetupResult.For(httpContextStub.Request).Return(httpRequestBaseStub);
        SetupResult.For(httpContextStub.Server).Return(httpServerUtilityMock);
        SetupResult.For(httpRequestBaseStub.IsAuthenticated).Return(true);
        httpServerUtilityMock.TransferRequest("/farfaraway");
        LastCall.On(httpServerUtilityMock).Repeat.AtLeastOnce();
    }).Verify(delegate
    {
        handler.TransferUserIfAuthenticated(httpContextStub);
    });
}

Summary

I have shown two very simple examples on how some of the classes in the System.Web.Abstractions assembly can help us test our ASP.NET code. I have used HttpResponseBase, HttpServerUtilityBase, HttpContextBase, HttpRequestBase, and HttpCachePolicyBase. Note that there are a number of classes available, so if you are faced with not being able to test your ASP.NET code because of dependencies to classes in the framework, take a look in the System.Web namespace and see if there are -Base classes that can help you out.

Testability is a large topic, and there is much to be said about it. I have shown a couple of very simple examples on how to improve testability. Testability has a lot to do with code design as well; in a real world I would write the test before writing the code and I would move my code out of codebehinds. Those topics are discussed a lot elsewere, hopefully this post will bring you a small step further in writing testable code.

No measurement beats WTFs/min!

When trying to introduce unit tests into an existing project, code dependencies is often a problem. During a teeth-grinding session grappling with this problem I came to think about pure functions.

Pure functions are functions that (according to Wikipedia‘s definition):

• always evaluates the same result value given the same argument value(s). The function result value cannot depend on any hidden information or state that may change as program execution proceeds, nor can it depend on any external input from I/O devices, and
• evaluation of the result does not cause any semantically observable side effect or output, such as mutation of mutable objects or output to I/O devices.

Seems to me that pure functions would be a wet dream for unit testing. Imagine, you could pass in whatever the function needed to execute, and then check its return value. No mocking about with mock objects with side effect expectations needed!

But alas, we live in a world with non-pure functions. Even in functional programming, which does explicitly keep any state, you find them. In an object oriented language a primary idea is that objects should carry state. An object has a state. Period. However, if we can be inspired by the notion of pure functions, and make methods that have less dependencies, we will be much better off when it comes to unit testing.

I my experience, the ASP.NET programming model with ASPX code behind files is a model that promotes writing methods that are problematic to create unit tests for.

Take, for example, the following (theoretical, I admit) code behind snippet example:

protected void Page_Load(object sender, EventArgs e)

    {

        ShowOrHideEditButton(Page.User.Identity.Name);

    }

    void ShowOrHideEditButton(string currentUser)

    {

        if (Request.QueryString["user"].Equals(currentUser))

        {

            EditButton.Enabled = true;

        }

        else

        {

            EditButton.Enabled = false;

        }

    }

So, what is the problem here? Actually, there are two problems.

1. The ShowOrHideMethod is not only dependant on its input parameters, but also on another member variable. When looking at the method signature, you do not see this, and might be misleaded about what the outcome of the methods is dependent on.
2. The method does not return any value, but has a side-effect on its environment (setting the edit button visibility).

These two issues, makes it troublesome to test the method in a unit test. You have spend time to set up the methods context, and you have to measure the method’s side effects in order to make assesments about its result.

A much better approach would be something like this:

    protected void Page_Load(object sender, EventArgs e)

    {

        EditButton.Enabled = ShowOrHideEditButton(Page.User.Identity.Name, Request.QueryString["user"]);

    }

    bool ShowOrHideEditButton(string currentUserName, string profilePerson)

    {

        return (currentUserName.Equals(profilePerson));

    }

This way, we can easily test the ShowOrHideEditButton method in a unit test without having to use a great deal test code to set up the test context.

For increased testability, I would suggest the following for better method purity:

• A method should evaluate the same result, given the same argument value(s) and object state.
• A method execution result should be represented as, in order of preference: 1) a return value, 2) changed object state (mutation), or 3) a side-effect on an I/O device

What are the influences of Simula 67 on today’s programming languages? Is Simula 67 the most important language in the history of programming languages? What are the next big topics in programming languages? These were some of the questions discussed in the panel debate “Celebrating 40 years of language evolution: Simula 67 to the present and beyond” today at OOPSLA.

Parallelism revisited

It seems to me that tomorrow’s most important topic in programming will be related to parallelism. Both James Gosling and Anders Hejlsberg stated that the next important thing in programming will be address multithreading better. There are several drivers for this: first of all, we see that we are getting computers with more and more CPUs, and our current programming languages and inherently programs do a poor job taking advantage of this. Furthermore, applications are becoming more and more distributed, which also drives parallelism, timing and synchronization issues. I guess the fact that the programming language Erlang, created by Ericsson Computer Science Laboratory in the eighties, is getting attention these days, can be seen to emphasise this trend. Erlang has built-in support for concurrency.

Is functional programming the real deal?

Another topic touched upon by the panel, was the role of functional programs. OOPSLA being a conference built around object orientation, the crowd is of course a bit biased towards object oriented programs, pretty much like the rest of the world is. However, the current trend is that features from functional programs are finding their ways into object oriented languages like Java and C# (closures, higher order functions, etc.). A question, then, is if functional programming will be the way of the future. Bertrand Mayer pointed out that functional programming will never be mainstream. James Gosling went on to state that the problem with functional programming is that only a small part of the programming community understands it. Furthermore, Guy Steele pointed out that functional programming avoids the notion of state, which is a notion that will get ever more important as applications become distributed.

Programming: what can we do better?

One important topic in our field today is what we can do better to cope with the ever incresing complexity in software. One thing that Anders Hejlsberg pointed out, as that today’s programming focuses too much on the ‘how’, and too little on the ‘what’. Too much focus on ‘what’ gives too much complexity. Furthermore, Ole Lehrmann Madsen pointed out that there are too many papers at this conference about ‘adding x to Java’. Instead, we should think more out of the box to find new paradigms that can bring us a leap forward in the field of computer programming. Easier said than done, I guess… Going back to the topic of concurrency, Guy Steele pointed out that the dependency on the stack is the biggest problem for concurrency in today’s programming languages. This reminds me of Gregor Hohpe‘s statement in the workshop I attended on Monday, that “SOA is like programming without a call stack”.