Vidar’s Musings

Just checked in my Azure WebRole solution in to TFS. I immediately encountered two problems. Luckily, it turned out that both where solvable.

Problem #1: CSPack Fails on TFS Build

The forum thread concludes that this is is a bug that will be fixed in the Azure SDK and possibly Visual Studio. However, I was able to work around it by combining some of the other responses in the thread, namely by editing the OutputPath element and the ServiceOutputDDirectory element in the ccproj file.

Problem #2: System.ServiceModel.CommunicationObjectFaultedException During Role Instance Start-up

This error was caused by the web.config file not being editable on the file system. Checking out the file for edit from TFS fixed the problem.

My unbridled enthusiasm was suddenly constrained when I during the Azure tutorial suddenly tripped over my first Azure hurdle. When trying to run my new “Hello Azure” application, I got an error message stating that:

Creating database DevelopmentStorageDb… Failed to create database ‘DevelopmentStorageDb…’ : CREATE DATABASE permission denied in database ‘master’. …

So, I googled the problem and found this forum entry where the following solution is suggested:

> You can try to connect “machinename\sqlexpress” via SQL Server Management Studio and grant permission to your logon user. It’s the default instance name.

Very well. There is only one problem. In my local SQL Server 2008 Express installation, which was installed with Visual Studio 2010, my Windows account does not have the sysadmin role. Furthermore, the sa account is not enabled for login. Thus, there is no user in the system which has the sysadmin role. Catch-22, it seems.

I was finally able to find this blog entry which explains how to recover a lost sa account password in SQL server 2005. Luckily, this approach also works for SQL Server 2008 Express. Voila!

I artikkelen “Vil ha helsetjenester på nett“  hos Teknologirådet heter det i ingressen:

Ni av ti nordmenn ønsker å kommunisere med fastlegen sin over internett. – Helsevesenet bør få én felles inngangsportal på nett, sier Teknologiråds-direktør Tore Tennøe.

Denne påstanden er basert på en undersøkelse foretatt av Response Analyse Oslo for Teknologirådet. Hvis man føler lenke til denne artikkelen, fremgår noe mer informasjon om metodikken bak undersøkelsen:

Et landsrepresentativt utvalg på 1098 personer fra 17 til 84 år har svart på den elektroniske spørreundersøkelsen…

Spørsmålet er da: er det et landsrepresentativt utvalg hvis man kun baserer seg på dem som svarer på elektroniske undersøkelser? En ting som er helt sikkert, er at man ikke kan si at “9 av 10 nordmenn ønsker…”. Siden undersøkelsen gjelder bruk av elektroniske hjelpemidler er det en åpenbar svakhet at (sannsynligvis) kun de som bruker elektroniske hjelpemidler har mulighet til å svare. Når det er sagt er det veldig mangelfull informasjon om metodikken brukt i undersøkelsen, så man kan ikke være bastant her. Viktige momenter er hvordan man har valgt ut respondenter, hvordan man har kontaktet dem, og ikke minst hvor mange som har unnlatt å svare.

Metodikken i undersøkelsen er det vanskelig å si noe konkret om siden veldig lite informasjon om den er tilgjengelig. En ting er dog sikkert, artikkelen som har dedusert at “9 av 10 nordmenn…” burde si noe om datagrunnlaget for påstanden. At undersøkelsen var foretatt elektronisk er essensielt. Kanskje burde det heller stå “9 av 10 nordmenn som svarer på elektroniske undersøkelser ønsker…”

Yesterday I got an email from Atlassian, the makers of applications such as Confluence and JIRA, that said that their own hosted customer site had been hacked and that my password was possibly compromised. Apparently, some passwords were stored in clear text in the database and that the hackers had gotten hand on these (See Atlassian’s blog post about the incident).

One thing is that the perpetrators could use this information to get details about my relationship with Atlassian, among other things my license keys for Atlassian products. Even worse is that they could try and use the passwords to get into my accounts at other sites. That would be successful if I used the very usual and very baaad practice of reusing the same (user name and) password on several sites and applications. Luckily I don’t.

Here is what Atlassian states about why this could happen:

During July 2008, we migrated our customer database into Atlassian Crowd, our identity management product, and all customer passwords were encrypted. However, the old database table was not taken offline or deleted, and it is this database table that we believe could have been exposed during the breach

Trying to act as a responsible company, Atlassian goes on to list what they have learned from the incident. Among other things, they state that

The legacy customer database, with passwords stored in plain text, was a liability. Even though it wasn’t active, it should have been deleted. There’s no logical explanation for why it wasn’t, other than as we moved off one project, and on to the next one, we dropped the ball and screwed up.

I am sorry, but I find it hard to believe that this is the entire truth. Yesterday afternoon (European time) I went to their site and on the login screen I used their “Forgot my password” functionality. Can you guess what happened? They sent me an email with my password in clear text! So, I would indeed say that this “legacy database” is indeed active…

Later on the day yesterday, I also got an email from the Apache Software Foundation that their JIRA instance also have been hacked. See their blog entry about the issue. According to the blog entry, the situation is a bit better than it is at Atlassian. They state that

If you are a user of the Apache hosted JIRA, Bugzilla, or Confluence, a hashed copy of your password has been compromised. JIRA and Confluence both use a SHA-512 hash, but without a random salt. We believe the risk to simple passwords based on dictionary words is quite high, and most users should rotate their passwords.

At least, the password was not stored in clear text, which is of course much better than having it in clear text. However, the compromised passwords could still be useful for an attacher because they are not salted. It allows an hacker to compare hashes of other accounts with hashes of a known password which would allow a dictionary attack (as is stated).

This is not security for the crowds (pun intended).

My application on Google App Engine recently hit a mail quota limit. Specifically, there is a quota limit on how many recipients the application can send emails to per minute. For the free version, this limit turned out to be quite low, namely eight.

The application in question has a scheduled task that once every day sends out an email which has a list of recipients. This list of recipients reached the aforementioned limit and the email sending failed. The log stated:

OverQuotaError: The API call mail.Send() required more quota than is available.

The solution to this was to change the email sending so that one email was generated per recipient, and the sending of the email was queued using the brilliant (however experimental) task queue functionality that GAE provides. The code for queuing looks like this:

queue = Queue('mail-queue')
for recipient in to:
    queue.add(Task(url='/task/mail', params= { 'to' : recipient, 'subject' : subject, 'body' : body }))

Here’s the actual task code that sends the email:

class MailSender(webapp.RequestHandler):
    def post(self):
        to = self.request.get('to')
        subject = self.request.get('subject')
        body = self.request.get('body')
        logging.info("Sending '%s' to %s" % (subject, to))
        mail.send_mail("not.the.real.sender@not.a.real.domain.com", to, subject, body)

Finally, I defined the ‘mail-queue’ in ‘queue.yaml’:

queue:
- name: mail-queue
  rate: 8/m

Works like a charm! For more information about task queuing, see http://code.google.com/intl/no/appengine/docs/python/taskqueue/overview.html

In my previous post, I talked about how we could use System.Xml.Serialization.XmlSerializer and System.Runtime.Serialization.DataContractSerializer to parse XML into an object tree. I pointed out that DataContractSerializer in my mind has some advantages in that types, properties, and members to use with the deserialization do not need to be public. On the downside, DataContractSerializer puts a quite limiting constraint on the XML format in that it cannot parse XML attributes. This is as far as I know, an absolute constraint that cannot easily be circumvented. Thus, regrettably, if we are not in control of the XML format, DataContractSerializer is sometimes useless.

In those situations, we can still use XmlSerializer. In order to achieve the same encapsulation with XmlSerializer, we have to adjust our model a bit. Here’s one suggestion on how to do this:

My approach to this is inspired by Josh Bloch‘s Builder pattern. The idea is changing the classes used for deserialization from being domain objects to being builder objects that build domain objects. This has another advantage in that our domain objects are not “polluted” with attributes and interfaces related to deserialization and are 100% plain old CLR objects (POCOs). So, lets first do a change our “deserialization” class (formerly ‘Country’) to a builder object, like so:

public class CountryBuilder
{
   [XmlElement(ElementName = "name")]
   public string Name;

[XmlElement(ElementName = "iso-3166-alpha-2-code")] public string Code;

public Country Build() { return new Country(Name, Code); } }

Note here that our builder object has a Build() method which returns the domain object ‘Country’. This is the object that we will pass on to our clients. The Country class now represents our domain object:

public class Country
{
   private readonly string _name, _code;

internal Country(string name, string code) { _name = name; _code = code; }

public string Name { get { return _name; } } public string Code { get { return _code; } } }

We have now restricted access to the creation of Country objects in that the constructor is internal, and it is also immutable (cannot change state once created) though making its fields readonly. It only exposes getters for its internal state. We can then do the same to our list of countries. The builder object for countries would look like this:

[XmlRoot("countries")]
public class Countries
{
   [XmlElement(ElementName="country")]
   public CountryBuilder[] countries;

public IEnumerable<Country> Build() { return countries.Select<CountryBuilder , Country>(x => x.Build()); } }

The code for doing the serialization will then look like this:

string xml = ...;
XmlSerializer xmlSerializer = new XmlSerializer(typeof(CountriesBuilder));
var builder = xmlSerializer.Deserialize(new StringReader(inputXml)) as CountriesBuilder;
IEnumerable<Country> cs = builder.Build();

What we have achieved now is that we now can control the accessibility and encapsulation of our domain model. The builder objects are still public to anyone, but that really does not matter much in my mind. Thus, a relatively swift parsing of XML of various formats into a well designed object model.

If you want to parse XML in .NET, you have a lot of options to choose from. You can use XmlDocument to parse the XML into a DOM tree, you can use the XmlReader to write an efficient “pull” parser, or you can leverage some of the features provided with various serialization APIs.

Given the case where you have a fairly straightforward XML document (not too deep document tree, not too complex set of attributes and elements) that maps pretty well to your domain model, the serialization options is in my mind a good choice that requires little coding. Compared with this approach, using XmlDocument seems to be a bit of an overkill if you don’t need advanced traversal of the document, and writing a parser by hand using XmlReader seems to require quite a bit of coding.

So, given the following sample XML document, I will investigate the serialization options:

<countries>
   <country>
      <iso-3166-alpha-2-code>AF</iso-3166-alpha-2-code>
      <name>Afghanistan</name>
   </country>
   <country>
      <iso-3166-alpha-2-code>AX</iso-3166-alpha-2-code>
      <name>Åland Islands</name>
   </country>
   <country>
      <iso-3166-alpha-2-code>AL</iso-3166-alpha-2-code>
      <name>Albania</name>
   </country>
</countries>

Using System.Xml.XmlSerializer

The first option that came to mind, was to use the XmlSerializer object to deserialize the XML into C# (or VB for that matter) objects. It first requires that I annotate my object model in order to tell the serializer how to deserialize the XML:

[XmlRoot("countries")]
public class Countries
{
   [XmlElement(ElementName="country")]
   public Country[] countries;
}

public class Country { [XmlElement(ElementName = "name")] public string Name;

[XmlElement(ElementName = "iso-3166-alpha-2-code")] public string Code; } Then, I can use the serializer to deserialize the code:

string xml = ...;

XmlSerializer xmlSerializer = new XmlSerializer(typeof(Countries)); Countries c = xmlSerializer.Deserialize(new StringReader(xml)) as Countries; Pretty sweet,  heh? Definitely. However, this has some drawbacks. If I want my Country class to be a well designed domain object that follows good OO design principles, I probably would like to encapsulate my data. Furthermore, I might want to restrict the creation of such objects from other parts of the code. In order for XmlSerializer to create my object, it requires that my types are public and that all properties or fields to set are public as well. What to do if I want to enforce my objects to be immutable once handed off to other parts of the code?

Using System.Runtime.Serialization.DataContractSerializer

Luckily, the serialization API that come with Windows Communication Framework has some neat features that fit like a glove. When defining my data model, it does not require that the types, neither the properties nor fields to set are public. Actually, I can restrict access to the type, its default constructor, and any of the properties or fields that I want to be deserialized! w00t!

So, this is what the Country class will looks like:

[DataContract(Name="country", Namespace="")]
internal class Country : IExtensibleDataObject
{
   private Country() { }

[DataMember(Name="name")] public string Name { get; private set; }

[DataMember(Name = "iso-3166-alpha-2-code")] public string Code { get; private set; }

public ExtensionDataObject ExtensionData { get; set; } } The XML file contains a list of countries, and luckily, we have the CollectionDataContractAttribute to denote an element that is a list of elements. It supports generics, so that we can define our class as a strongly typed list:

[CollectionDataContract(Name="countries", Namespace="")]
internal class Countries : List<Country>, IExtensibleDataObject
{
   public ExtensionDataObject ExtensionData { get; set; }
}

And that’s it. Now we can deserialize:

string xml = ...;

DataContractSerializer ser = new DataContractSerializer(typeof(Countries)); using (StringReader stringReader = new StringReader(xml)) { using (XmlReader xmlReader = XmlReader.Create(stringReader)) { Countries countries = (Countries)ser.ReadObject(xmlReader); } } Alternatively, our result could be typed as a list of countries:

IList<Country> countries = (IList<Country>)ser.ReadObject(xmlReader);

Note that there is a limitation in the latter method in that deserializing XML attributes is not supported. Thus, an XML document like the following would not work:

   <country iso-3166-alpha-2-code="AF">
      <name>Afghanistan</name>
   </country>
   <country iso-3166-alpha-2-code="AX">
      <name>Åland Islands</name>
   </country>
   <country iso-3166-alpha-2-code="AL">
      <name>Albania</name>
   </country>
</countries>

This will, however, work using the XmlSerializer.

As a follow-up to my previous post ASP.NET 3.5: improving testability with System.Web.Abstractions, I would like to show how the same testability can be achieved without using any mock framework like Rhino.Mocks. The C# 3.0 featuires ‘object initializers’ and ‘automatic properties’ makes our code sufficiently non-verbose to make it easy and readable.

So, given the same examples as in my previous post, here is what the test code will look like:

Example #1: Testing a page codebehind file

[TestMethod]
public void ShouldSetNoCacheabilityOnDefaultPage()
{
    _Default page = new _Default();
    HttpCachePolicyMock httpCachePolicyMock = new HttpCachePolicyMock();
    page.SetCacheablityOfResponse(new HttpResponseStub
    {
        TheCache = httpCachePolicyMock
    });
    httpCachePolicyMock.ShouldHaveSetCacheabilityTo(HttpCacheability.NoCache);
}

class HttpResponseStub : HttpResponseBase { public override HttpCachePolicyBase Cache { get { return TheCache; } } public HttpCachePolicyBase TheCache { get; set; } }

class HttpCachePolicyMock : HttpCachePolicyBase { private HttpCacheability _cacheability; public override void SetCacheability(HttpCacheability cacheability) { _cacheability = cacheability; } public void ShouldHaveSetCacheabilityTo(HttpCacheability expectedCacheability) { Assert.AreEqual(expectedCacheability, _cacheability); } } I have created two helper classes, one with the suffix -Stub and one with the suffix -Mock. The convention here is that a stub is a type of class used to provide a context to the class under test. Mocks also do that, but additionally a mock can make expectation about what should happen to it during the test.

Example #2: Testing an HTTP handler

[TestMethod]
public void ShouldRedirectAuthenticatedUser()
{
    HttpServerUtilityMock httpServerUtilityMock = new HttpServerUtilityMock();
    HttpContextStub httpContextStub = new HttpContextStub
    {
        TheRequest = new HttpRequestStub { IsItAuthenticated = true },
        TheServer = httpServerUtilityMock
    };
    new RedirectAuthenticatedUsersHandler().TransferUserIfAuthenticated(httpContextStub);
    httpServerUtilityMock.ShouldHaveTransferredTo("/farfaraway");
}

class HttpContextStub : HttpContextBase { public override HttpRequestBase Request { get { return TheRequest; } } public override HttpServerUtilityBase Server { get { return TheServer; } } public HttpRequestBase TheRequest { get; set; } public HttpServerUtilityBase TheServer { get; set; } }

class HttpRequestStub : HttpRequestBase { public override bool IsAuthenticated { get { return IsItAuthenticated; } } public bool IsItAuthenticated { get; set; } }

class HttpServerUtilityMock : HttpServerUtilityBase { private string _path; public override void TransferRequest(string path) { _path = path; }

public void ShouldHaveTransferredTo(string expectedPath)
{
    Assert.AreEqual(expectedPath, _path);
}

}

WTF? I am quite sure that replacing ToLower() with ToUpperInvariant() will make my test fail…

Found this nice page that summarizes how to set cache-related information in ASP.NET: ASP.NET Cache Examples and Overview