Previously I wrote about editing and validating variable-length lists with ASP.NET MVC 2, and before that with ASP.NET MVC 1. Those demos were OK, but it’s way easier with Knockout – it eliminates the most of the complex moving parts and the fiddly client-server interaction, the precise element naming requirements, and the hackery needed to make validation work.

Note: Knockout works with any server-side web development framework,
but this blog post focuses on using it with ASP.NET MVC

Live demo

Before we get into the code, see what we’re about to build. It’s simple, but nice to use. Try it out in this live IFRAME:

Note: Some RSS readers will strip out the preceding IFRAME. If you don’t see the live demo, see this post on the web instead.

Right, how’s it done?

Sending some initial state down to the client

First, define a C# model for the data items we’re editing:

public class GiftModel 
{ 
    public string Title { get; set; } 
    public double Price { get; set; } 
}

Next, create an action method that renders a view, passing some GiftModel instances to it:

public ActionResult Index() 
{ 
    var initialState = new[] { 
        new GiftModel { Title = "Tall Hat", Price = 49.95 }, 
        new GiftModel { Title = "Long Cloak", Price = 78.25 } 
    }; 
    return View(initialState); 
}

… and then, in the view, convert that C# model data into a JavaScript object:

<script type="text/javascript"> 
    var initialData = <%= new JavaScriptSerializer().Serialize(Model) %>; 
</script>

Creating a Client-side View Model

Right now our view model only needs one property – gifts – which will be an observable array of gift items. Once we’ve got that, we can tell Knockout to bind this to any HTML nodes that request binding:

var viewModel = { 
    gifts : ko.observableArray(initialData)
}; 
 
ko.applyBindings(document.body, viewModel);

To check this is all working, let’s just bind a SPAN’s text content to the length of the observable array:

<p>You have asked for <span data-bind="text: gifts().length">&nbsp;</span> gift(s)</p>

Because gifts is an observable array, the text in this SPAN will be updated automatically whenever the length of the array changes. With all this in place, you should now see the following:

Making an editable grid

Making data editable is a matter of creating HTML input controls and binding them to that data, and the easiest way to create repeated blocks of markup is to use a template. So, here’s how to make a table containing an editor row for each gift item:

<table> 
    <tbody data-bind="template: { name: 'giftRowTemplate', foreach: gifts }"></tbody> 
</table>
 
<script type="text/html" id="giftRowTemplate"> 
    <tr> 
        <td>Gift name: <input data-bind="value: Title"/></td> 
        <td>Price: \$ <input data-bind="value: Price"/></td> 
    </tr> 
</script>

This will produce the following output:

Now, if you edit the contents of the text boxes, it will actually edit the underlying data model. You can’t really see any effect of that just yet, but you will be able to when we save the data.

Adding gifts

Adding items just means adding them to the gifts observable array in the underlying view model; the UI will take care of updating itself. So, create a button that triggers an ‘add’ method on the view model:

<button data-bind="click: addGift">Add Gift</button>

… and then implement such a method, pushing a new blank item into the array:

var viewModel = {
 
    // ... leave the rest as before ...
    addGift: function () { 
        this.gifts.push({ Title: "", Price: "" }); 
    }
 
};

That does it.

Removing gifts

Again, you only need to edit the underlying array. So, edit the giftRowTemplate and put a ‘Delete’ link on each row:

<script type="text/html" id="giftRowTemplate"> 
    <tr> 
        <!-- Leave the rest as before --> 
        <td><a href="#" data-bind="click: function() { viewModel.removeGift($data) }">Delete</a></td> 
    </tr> 
</script>

… and implement a removeGift method on the view model:

var viewModel = { 
    // ... leave the rest as before ...
 
    removeGift: function (gift) { 
        this.gifts.remove(gift); 
    }
 
};

Now, when you click ‘delete’ on any item, it will vanish from both the underlying data model and the HTML UI.

Saving data (i.e., submitting it back to the server)

Since we’re following the MVVM pattern, all our really important data is in the view model – the view (the HTML UI) is merely a representation of that. So, we don’t want to submit data from the view by doing a regular form post. Instead, we want to submit the state of the view model as this may contain more data than is visible to the user.

First, we need to know when the user wants to save their data. To do this, we can wrap up our UI inside a regular HTML <form> with a submit button, and then bind the ‘submit’ event to a method on the view model. (Doing it this way preserves familiar behaviours like pressing “Enter” to submit a form)

<form class="giftListEditor" data-bind="submit: save">
    <!-- Our other UI elements, including the table and ‘add’ button, go here --> 
 
    <button data-bind="enable: gifts().length > 0" type="submit">Submit</button> 
</form>

Notice that I’ve made the “submit” button enabled only when there’s at least one item. Next, add a ‘save’ method that packages up the view model state as JSON and posts it to the server:

var viewModel = {
 
    // ... leave the rest as before ...
    save: function() { 
        ko.utils.postJson(location.href, { gifts: this.gifts }); 
    } 
};

It would be perfectly easy to submit the data via Ajax using jQuery’s $.post or $.ajax methods, but I’ve chosen to use postJson (a utility function inside Knockout) to send the data as a regular HTML form submission instead. That means the server-side code doesn’t have to know anything about Ajax and can imagine you’re posting a normal HTML form. As I said, Ajax works fine too – I’m just demonstrating another option here.

The view model posts the data to the same URL you’re already on (i.e., location.href) because that’s ASP.NET MVC’s convention. To collect the data on the server, add an action method as follows:

[HttpPost] 
public ActionResult Index([FromJson] IEnumerable<GiftModel> gifts) 
{ 
    // Can process the data any way we want here, 
    // e.g., further server-side validation, save to database, etc 
    return View("Saved", gifts); 
}

That’s pretty easy! This fits into ASP.NET MVC’s usual data-entry conventions – the server-side code is totally trivial. It doesn’t use ASP.NET MVC’s usual data binding mechanism so you don’t have to worry about element naming, field prefixes, and all that; instead, the data is packaged as JSON and deserialized using [FromJson]. I’ll omit the source code to [FromJson] for now, but it’s totally general-purpose, only about 10 lines long, and is included in the downloadable example with this blog post.

The server-side code can now do whatever it wants. For this example, I’m just rendering a different view to show what data the server received.

Adding Validation

Right now, somebody could enter a text string for “Price”, and then this will cause a deserialization error on the server. This is an abomination, and must be stopped! Let’s add some client-side validation.

Knockout works nicely with any client-side validation framework that can cope with you dynamically modifying the DOM. jQuery.Validation copes perfectly with that, so let’s use it. Having referenced jquery.validate.js, define rules by putting special CSS classes on the elements in the template. (Note that jQuery.Validation lets you define custom rule logic in this way, too.)

<script type="text/html" id="giftRowTemplate"> 
    <tr> 
        <td>Gift name: <input class="required" data-bind="value: Title, uniqueName: true"/></td> 
        <td>Price: \$ <input class="required number" data-bind="value: Price, uniqueName: true"/></td> 
        <td><a href="#" data-bind="click: function() { viewModel.removeGift($data) }">Delete</a></td> 
    </tr> 
</script>

Notice the “required” and “number” classes. One other thing to bear in mind is that, although Knockout doesn’t need your bound elements to have names or IDs, jQuery.Validation does depend on every validated element having a unique name. That’s why I’ve put uniqueName:true into the bindings. This is a trivial binding I added that just checks whether the element has a name, and if not, gives it a unique one. An easy workaround for the jQuery.Validation limitation.

Next, remove the “submit” binding from the form, and tell jQuery.Validation to catch the form’s submit event instead, and to call our view model’s save method only if the form is valid:

$("form").validate({ submitHandler: function() { viewModel.save() } });

That’s it! Now the user can’t submit blank fields, nor can they submit text for the “Price” fields. They can still add and remove items, though, and this doesn’t affect the state of the validation feedback for other rows in the table.

Of course you can still validate the data on the server, too – and you need to if blank data would actually violate your business rules in some important way. Typically I’d recommend putting such logic into your domain layer, to ensure the rules are always respected no matter what UI technology (ASP.NET MVC, Knockout, Silverlight, an iPhone app) is connected to it. To display any errors, you can use something like an ASP.NET MVC “ValidationSummary” helper. (Or you can just throw an exception and give up if it looks like the user is deliberately bypassing client-side validation – it’s up to you.)

There’s plenty more I could describe here, such as how to use bindings to add animated transitions (e.g., applying jQuery’s fadeIn or slideUp when users add or remove items), but I think this is enough for now.

Questions and support

I’ve had plenty of questions about how to do things with Knockout since my blog post last week. To capture these discussions and make them public, I’ve made a Google Group for Knockout at http://groups.google.com/group/knockoutjs – please use this for general questions about the framework. However if you have comments about the particular example shown in this post, go ahead and post them here on this blog.

If you want to play with the gift list editor code a little more, download the demo project.

So, what distinguishes this from all the other similar-sounding JavaScript libraries? (I blogged about an MVC/templating system for JavaScript back in 2007, Microsoft has suggested a data linking system for jQuery, and Google’s full of MVC-on-the-client toolkits.)

Well, Knockout combines some different ideas that open up very powerful possibilities:

• Automatic dependency tracking: At the heart of Knockout is a system of observable variables and other variables computed as functions of them. For example, if you have firstName and lastName and then define fullName by concatenating these together, the framework knows that whenever either firstName or lastName changes, it must re-evaluate fullName. You don’t have to configure this – it’s inferred from your code execution, even if you have chains of observables referencing other observables.
  This makes it easy to build sophisticated view models in which changes to a few key properties automatically ripple out to all other affected parts of the model.
• Declarative bindings: By adding a data-* attribute (part of the HTML 5 spec, and works fine in older browsers, including IE 6) you can quickly bind appearance and behaviour in your HTML to properties on a view model. For example, you can write <span data-bind="text: fullName"></span>, and then the element will display the value in your model property as text, automatically updating if any of the underlying data changes. Besides text, I’ve included a range of commonly useful declarative bindings (e.g., visible, click, and css), and you can easily plug in custom ones.
• Nested templates: When you have a rich structure of view model data to display, it makes sense to display it using a template to keep your code simple. Knockout handles this easily by letting you bind a DOM node to a template and some data. Whenever either changes, the template gets re-rendered and your UI is updated. You can use data-bind attributes from inside templates (referencing any variable in scope at that point during template execution, such as foreach loop variables) to concisely add further interactivity. Plus, you can bind to templates from within templates, creating chains of nested templates. Because of how dependency tracking works, if the underlying data changes, Knockout knows it only needs to re-render the innermost templates affected by the change, so it’s efficient. Knockout uses the jquery-tmpl template engine by default, but you can plug in another template engine if you prefer some other template syntax.

These features together, combined with following the MVVM design pattern, mean you can apply sophisticated UI behaviours in a clean, concise, and organized way. It’s also possible to combine custom bindings and templates to create reusable controls (or plugins).

Extremely Trivial Example

OK, code. You could set up a view model as follows:

<script type="text/javascript">
    var myViewModel = {
        personName : ko.observable("Bert") // Initial state
    };
</script>

Then, you could bind it to an HTML UI as follows:

Enter your name:
<input id="source" data-bind="value: personName, valueUpdate: 'keyup'" />
 
<div data-bind="visible: personName().length > 5">
    Wow, what a long name you have.
</div>

Now, the text box will take its value from personName, and will automatically update its value whenever personName changes. The binding is two-way, so when you type into the text box, this updates personName.

The DIV with the “what a long name” message will be made visible whenever the name length exceeds 5 characters, and hidden otherwise. By referencing an observable value in a binding, you’ve implicitly set up notifications.

But I can just do that with jQuery!

Yes, we all love jQuery! It’s a fantastic replacement for the hideous DOM API we had to work with before.

The point of Knockout is not to replace jQuery – I expect most people who use Knockout will use jQuery too. The point is to make it easy to have one coherent underlying data model and then bind your UI to that. Change the data model and the UI changes, and vice-versa. That way, you don’t get complicated event handlers that have to update loads of different parts of the DOM to keep the UI up-to-date; instead, when you just change the underlying data model, you know all the affected UI gets updated. This also makes client-server interactions much easier: when saving data, you don’t have to try to scrape the current state off a load of DOM elements – you’ve already got it in a clean model. Likewise when loading data from the server and wanting to update the DOM to match – just push it into the data model and the UI is updated. It’s the MVVM pattern, much beloved of Silverlight developers.

A More Detailed Tutorial

Right, getting beyond “hello world” but still keeping things simple, let’s make a simple list editor. This will demonstrate some of Knockout’s core features – observables and bindings (though we’ll ignore templating for now – you can learn about that from other examples).

To get started, create an HTML file that references knockout.js or knockout-min.js and define a simple view model class. In JavaScript, any function can act as a class constructor. The following one has a single property which represents the items in a list.

<html>
<head>
    <script type="text/javascript" src="knockout.js"></script>
</head>
<body>	
    <script type="text/javascript">
        var listEditorViewModel = function () {
            // ko.observableArray lets you track changes to a collection of things
            this.allItems = ko.observableArray(["Apple", "Banana", "Cherry"]);
        };
    </script>
</body>
</html>

Next, let’s display the list on the screen. You can do that by creating an HTML <select> element and binding its contents to a model property using Knockout’s options binding:

<div>List items:</div>
<select multiple="multiple" data-bind="options: allItems"></select>

<script type="text/javascript">
    var listEditorViewModel = function () {
        ... as before ...
    };
 
    ko.applyBindings(document.body, new listEditorViewModel());
</script>

If you open this in a browser, you’ll see the list items:

What about letting the user add some more items to the list? Update the view model class so that it has another property for tracking the new item a user is adding, and add a method that pushes the new item into the array:

var listEditorViewModel = function () {
    // ko.observableArray lets you track changes to a collection of things
    this.allItems = ko.observableArray(["Apple", "Banana", "Cherry"]);
 
    // ko.observable tracks a single item. The initial state here is an empty string.
    this.itemToAdd = ko.observable("");
 
    this.addItem = function () {
        this.allItems.push(this.itemToAdd());
        this.itemToAdd(""); // After adding an item, reset itemToAdd to an empty string 
    }
};

Next we need to add some UI elements and bind them to these new model members.

<form data-bind="submit: addItem">
    Add item:
    <input data-bind="value: itemToAdd" />
    <button type="submit">Add</button>
</form>

That will do it! Now when the user edits the text box, its value automatically goes into the itemToAdd observable because of the value binding. When they submit the form, this invokes addItem() because of the submit binding. In turn, this updates allItems in the view model, and this automatically causes the on-screen list to be updated because its collection of options is bound to that model property. With Knockout, you don’t have to remember which parts of the UI to update following an event – you just change properties on the underlying view model, and anything bound to those properties will be notified and updated.

Next thing, we might want to let the user remove items. We should let them select multiple items at once, then have a “remove” button that removes them all. First, we need to know what the user has got selected, so add a new view model property to represent selection, and bind it to the HTML element using Knockout’s selectedOptions binding.

<select multiple data-bind="options: allItems, selectedOptions: selectedItems"></select>

var listEditorViewModel = function () {
    // ... leave the rest as before ...
    this.selectedItems = ko.observableArray([]); // Initially select nothing
};

Now you can read selectedItems to see what items are selected, and if you write a value to that observable, it will change what’s selected in the UI.

To let the user remove their selected items, add a button and bind clicks on it to a method on the view model:

<button data-bind="click: removeSelected">
    Remove selected items
</button>

var listEditorViewModel = function () {
    // ... leave the rest as before ...
 
    this.removeSelected = function () {
        this.allItems.removeAll(this.selectedItems());
        this.selectedItems([]); // Reset selection to an empty set
    }
};

That’s great – now the user can remove one or more items from the list. But it doesn’t make sense for them to click “Remove selected items” if they haven’t got anything selected. Let’s make the button enabled only when at least one item is selected:

<button data-bind="click: removeSelected, enable: selectedItems().length > 0">
    Remove selected items
</button>

Easy! You can use arbitrary JavaScript expressions in bindings. If your bindings reference one or more observables, they’ll be re-evaluated automatically any time those observables change. So now the “Remove” button becomes enabled or disabled depending on whether anything is selected.

What if you wanted to let the user sort the items into alphabetical order? Again, you could do this by adding a method to the view model and binding a new button to that method. Alternatively – just to show you that you can use function literals in bindings – here’s a standalone way to do it:

<button data-bind="click: function() { allItems.sort() }, enable: allItems().length > 1">
    Sort
</button>

I’ve made this button enable only when there are at least two items (it’s pointless to “sort” a single item). Notice that selection is retained, because we didn’t change selectedItems.

That’s a fair bit of functionality in just a few lines of code. As I explained earlier, the benefit of Knockout and the MVVM pattern is that it scales very well in complexity – you don’t have to track how one part of the UI affects another; you just manipulate your view model and all affected parts of the UI change on their own. For example, in the preceding tutorial, removing items can make the ‘Sort’ button become disabled, but we didn’t have to write any explicit code in a ‘Remove’ click handler to make that happen.

To build on this knowledge, see the online demos of other examples (arranged in order of increasing sophistication) and look at their HTML source code to see how it’s done.

Requirements, Licensing, and Browser Support

Knockout is written in pure JavaScript, so it has no server-side requirements at all. Your web application can be written in ASP.NET MVC, PHP, Ruby on Rails – whatever.

The core of Knockout doesn’t depend on any other JavaScript library. However, the default template engine, jquery-tmpl, obviously depends on jQuery. If you don’t like that you can plug in another template engine, but I expect most people will be using jQuery anyway.

Regarding browser support, I’ve designed and tested it to work with IE 6 – 8, Firefox 2 – 3.6, and current versions of Chrome, Safari (verified on Windows and iPhone), and Opera. I fully expect it to work with older versions of Chrome, Safari, and Opera, but haven’t tried it – there’s only so much browser installation one person can bear… There’s a good set of automated specifications (BDD-style, written using JSSpec) so you can run these on any browser/platform to get a good sense of what features are working. Hopefully all of them

I’m releasing this project under the Ms-Pl license (an OSI-approved license). Some of my other code is licensed in the same way, so this keeps things simple for me, and it means you can use the library in any commercial or non-commercial project.

Where do I go from here?

For more details, examples, source, and downloads, see the project’s homepage at http://knockoutjs.com

The first edition went deep into the details of the MVC Framework, providing both tutorials and reference material. Judging by sales and review scores, it was a success. The second edition:

• … is thoroughly updated for ASP.NET MVC 2. It explains all the new features, including templating, metadata, validation, asynchronous controllers, areas, HTTP method overloading, strongly-typed input helpers, default parameters, etc., and many are demonstrated in the various tutorials.
• … is updated to account for .NET 4 and Visual Studio 2010. Even though .NET 4/C# 4 is the preferred technology throughout the book, all the documentation and code accounts for readers using .NET 3.5 as well.
• … is updated to reflect more recent patterns and best practices. For example, discussions of automated testing apply ideas from Behaviour Driven Development (BDD) where relevant, and tutorials and descriptions now consistently distinguish between view models and domain models.
• … is updated in light of reader feedback from the first edition. Certain explanations and terminology are overhauled, and the tutorials make use of more effective third-party libraries (e.g., Ninject for Dependency Injection).

This blog post is partly intended to build awareness of the new book, and is partly intended to deal with some of the questions I regularly get by email. So, here are some questions that people frequently ask:

Is this a new book, or an update?

It’s an update of the first edition. The following diagram should clarify what proportions of the book are new, dramatically changed, or just refreshed:

Where and exactly when can I get it?

I don’t know the exact date when it will ship; this depends on physical production and distribution schedules that are beyond my knowledge.

• For printed copies, your best bet is to pre-order with Amazon. Hopefully these will ship around the end of June, but I can’t guarantee it.
• For the eBook version, keep checking the Apress web site.

There’ll probably be a Kindle version in due course, too.

If I already have the first edition, should I buy the second edition?

If you use ASP.NET MVC regularly - especially if plan to use the new MVC 2 features but haven’t yet learned about them in depth - you may well get a lot out of reading the new edition and seeing what’s the same and what’s new. You may also have colleagues who need a deeper knowledge of the whole MVC Framework, including both v1 and v2 features.

However if you don’t use ASP.NET MVC much and don’t intend to migrate your development to v2, perhaps there isn’t such a strong case for buying an update of a book you already own.

Some readers have asked if they can get a sort of “upgrade” edition which contains only the new material. That wouldn’t really make sense for this book, as the new concepts and practices are applied throughout. I don’t present “old” material followed by “new” material – the whole manuscript is updated as a single coherent guide to ASP.NET MVC 2 from the beginning as I believe this gives the clearest understanding.

What new and updated in the second edition?

There are far too many updates and enhancements in the new edition to describe them all. Here’s a brief outline of the table of contents and roughly how each chapter has changed.

Chapter 1: What’s the Big Idea?

Updated to account for ASP.NET 4, including improvements to WebForms and how this impacts the comparison between the two frameworks. Gives an overview of what’s new in ASP.NET MVC 2. Changed the discussion general software development practices to account for more recent trends.

Chapter 2: Your First ASP.NET MVC Application

Tutorial now accounts for your choice of Visual Studio 2008 or Visual Studio 2010. Uses new ASP.NET MVC 2 features, including empty project template, automatic HTML encoding, and Data Annotations validation. Streamlined the flow of the tutorial to improve readability.

Chapter 3: Prerequisites

Describes newer architectural patterns including MVVM. Discussion of automated testing expanded to cover integration testing as well as unit testing, and demonstrates Cucumber-style BDD testing and explains the tradeoffs between this and traditional unit TDD. Various updates to terminology and explanations.

Chapter 4, 5, 6: SportsStore tutorial

Now accounts for your choice of Visual Studio 2008 or Visual Studio 2010. Improved the code: better project structure, uses Ninject instead of Castle Windsor for DI, has better unit test naming (BDD-style) with a clearer description of the limitations of such testing. Applies the viewmodel pattern and related terminology more consistently.

Adapted the tutorial to benefit from MVC 2 features including optional parameters, metadata, scaffolding, client-side validation, automatic (de)serialization, etc. Some parts of the tutorial now go in a different order to make it easier to follow.

Chapter 7: Overview of ASP.NET MVC Projects

Updated to account for ASP.NET MVC 2’s two project template options (empty and non-empty), and for changes in the core ASP.NET 4 platform regarding configuration and deployment.

Chapter 8: URLs and Routing

Now covers areas – reasons for using them, setting them up, their impact on routing, how to avoid common problems, how to unit test their routing configuration. Explains how .NET 4 changes where the routing code lives and how ASP.NET MVC 2 deals with controller namespaces. Many smaller changes.

Chapter 9: Controllers

The 1st edition’s “Controllers” chapter is now split in two – this first chapter now covers introductory topics – receiving input data with parameter binding etc, producing output with action results etc., unit testing.

Has many changes to account for ASP.NET MVC 2, such as its enhancements to TempData. Also describes .NET 4 features – using optional parameters (and how these differ from ASP.NET MVC 2’s parameter defaults), using “dynamic” as a model type, etc. Expands the coverage of unit testing, demonstrating 5 ways to make mocking controllers easier.

Chapter 10: Controller Extensibility

The 1st edition’s “Controllers” chapter is now split in two – this second chapter now covers more advanced topics – custom filters, method selectors, controller factories, etc.

Updated to reflect changes in the ASP.NET MVC 2 request-processing pipeline, new built-in filters, HTTP Method Overriding, etc. Adds (a lot of) coverage of asynchronous controllers – not just how to use them, but how to measure their impact and avoid common misconfiguration problems.

Chapter 11: Views

Goes into detail about how automatic HTML encoding works. Coverage of HTML helper methods expanded for ASP.NET MVC 2 (there’s now over 50 helpers, and that’s before you even start counting all their different overloads). Explains new ways to render partials.

Chapter 12: Models

Most of this massive chapter is totally new, and goes into great detail about metadata, templating, and validation. Covers how the built-in templates work, creating custom ones, using HTML field prefixes, implementing custom metadata sources, consuming metadata, custom validation providers, custom client-side validation, doing all this inside a multi-tier architecture, etc.

The explanation of model binding and value providers is significantly updated to account for the new architecture in ASP.NET MVC 2.

Chapter 13: User Interface Techniques

This new chapter inherits UI-related material from various parts of the 1st edition book, including wizards, CAPTCHAs, child actions, master pages, open-source view engines, custom view engines. All updated to match ASP.NET MVC 2, of course.

Chapter 14: Ajax and Client Scripting

Updated various aspects of the code and explanations to account for new framework features, and to make things work with more recent versions of IE. Expanded the coverage of JSON data services, including security issues and ways to handle cross-domain requests. Some recommendations are updated to account for client-side performance considerations(browser’s rendering pipeline, CDNs, etc).

Chapter 15: Security and Vulnerability

Mostly the same as in the first edition. Shows an alternative tamper-proofing mechanism using MVC 2 code, plus describes JavaScript string encoding and its relation to script injection. Various code changes to fit in with ASP.NET MVC 2.

Chapter 16: Deployment

Radically restructured chapter – now all organized with step-by-step guides and checklists for each targeted IIS version, so now you only have to read the material relevant to you. Covers new deployment options, including combinations of .NET 3.5 SP1, .NET 4, Server 2003, Server 2008, Server 2008 R2, Server 2008 R2 Core, shared hosting, classic/integrated pipeline mode, etc. Accounts for many changes to these deployment environments since the 1st edition, including IIS 7.5-specific issues

Clearer explanations of various IIS request-processing mechanisms. A new section describes VS2010’s improved publishing and packaging mechanisms, config file transforms, etc.

Chapter 17: Using ASP.NET Core Platform Features

Mostly the same as in the first edition. Updated to account for ASP.NET MVC 2, IIS 7.5, with tweaks to code and explanations. Information about configuration APIs moved from Deployment chapter into this chapter.

Chapter 18: Migrating Existing Applications to ASP.NET MVC 2.0

Various updates relating to .NET 4 / VS2010 / ASP.NET MVC 2, including how to upgrade Web Forms applications to support MVC, using routing when combining MVC with Web Forms (both on .NET 3.5 and .NET 4), ways you can use Web Forms server controls with postbacks in MVC 2, should you wish to.

New section describes upgrading from ASP.NET MVC 1 – using automated tooling, doing it manually, a post-upgrade checklist, workarounds for potential problems.

OK, enough details

Of course, there are other ASP.NET MVC 2 books in the pipeline too. No doubt you’ll enjoy and benefit from any of them.

• Hosting your app in a real web server and testing it through a real web browser using WatiN (plus, in this case, SpecFlow for Cucumber-style specifications)
• Hosting your app in a real web server and testing it with client-side automation using Microsoft’s Lightweight Test Automation Framework
• Hosting your app directly in the test suite process and bypassing both the web server and the browser entirely using my MvcIntegrationTestFramework experiment
• Unit testing code in isolation – best and worst practices, and why I think unit testing isn’t applicable to all types of code
• Injecting code across process boundaries to assist integration testing approaches that host your app in a real web server, using Deleporter

The reason I keep writing about this is that I still think it’s very much an unsolved problem. We all want to deliver more reliable software, we want better ways to design functionality and verify implementation… but we don’t want to get so caught up in the beaurocracy of test suite maintenance that it consumes all our time and obliterates productivity.

Yet another approach

Rails developers (and to a lesser extent Java web developers) commonly use yet another test automation technique: hosting the app in a real web server, and accessing it through a fast, invisible, simulated web browser rather than a real browser. This is known as headless browser automation.

How is this beneficial?

• It’s faster than driving a real browser. The simulated browser is just a native code library, so it’s very quick to launch and shut down, doesn’t require interprocess communication with your test suite, and doesn’t waste time physically drawing things on your screen, opening and closing pop-up windows, etc. Nonetheless, the simulated browser is full-featured: it exposes the usual HTML DOM API, runs JavaScript, evaluates CSS rules, and so on, so it’s still an effective way to specify rich client-side behaviours.
• It’s more reliable than a real browser. Real browsers are complicated. For example, if you launch and shut down Firefox many times in rapid succession, occasionally it will fail to launch because a previous instance is still locking some file. Simulated browsers are totally independent, so don’t suffer these kinds of weirdness.
• You can get more low-level control if you want it. For example, the simulated browser can easily offer an API to alter the HTTP headers it sends, or let you get or set the contents of its cache. A real browser wouldn’t usually make this easy.

And what drawbacks might you expect?

• It can be harder to debug. Because you can’t physically see the browser on your desktop, it’s not as obvious what’s happening if a test fails (or passes) unexpectedly. Your debugger’s “immediate mode” will let you call the browser’s API to figure things out, but it can be a longer investigative process.
• There’s no absolute guarantee that it faithfully replicates a real-world browser. For example, HtmlUnit can simulate multiple versions of Firefox, Internet Explorer, and Netscape, but it may not simulate every single quirk.

About HtmlUnit

HtmlUnit is a headless browser automation library for Java. It’s very well-developed and mature, as you can see from its extensive API. (Need to configure whether the user has JavaScript on or off? No problem.) It’s also the same technology that underlies Celerity, a Ruby library that exposes the Watir API but runs faster.

Unfortunately, in .NET world, we don’t have any good headless browser automation libraries like HtmlUnit that I know of. But don’t let that stop you! We do have IKVM – a near-magic way of running Java code on .NET (either by hosting a JVM on the CLR at runtime, or by a one-time conversion process that generates a native .NET assembly directly from the Java bytecode). So, why not use HtmlUnit itself?

Converting HtmlUnit to .NET

It’s surprisingly easy to get HtmlUnit, a Java library, converted into a native .NET assembly (no Java Virtual Machine needed!) using IKVM.

First, download HtmlUnit (as a compiled JAR file) from SourceForge (I’m using version 2.7), and extract all its files from the ZIP archive.

Second, download IKVM binaries from ikvm.net/SourceForge (I’m using version 0.42.0.3), and again extract all its files from the ZIP archive.

Open a command prompt, ensure you’ve added IKVM’s /bin folder to your PATH variable, change directory to HtmlUnit’s /lib folder, and then run ikvmc to convert the Java bytecode of all of the HtmlUnit JAR files into .NET bytecode. I ran this command:

ikvmc -out:htmlunit-2.7.dll *.jar

This produced a lot of warnings and a large (~10Mb) .NET assembly called htmlunit-2.7.dll. If you don’t want to bother with this process, you can download my sample project (linked below) which contains the .NET assembly I generated.

Using HtmlUnit from .NET

Once you’ve got htmlunit-2.7.dll as a .NET assembly, you can use it with any .NET unit testing library such as NUnit or XUnit. You will also need to reference a few of the IKVM runtime assemblies (I narrowed it down to six additional IKVM assemblies that appear to be needed – these are all included in IKVM’s /bin folder).

As a trivial example, here’s how you can use HtmlUnit with NUnit to load the Google homepage:

[TestFixture]
public class GoogleTests
{
    private WebClient webClient;
    [SetUp] public void Setup() { webClient = new WebClient(); }
 
    [Test]
    public void Can_Load_Google_Homepage()
    {
        var page = (HtmlPage)webClient.getPage("http://www.google.com");
        Assert.AreEqual("Google", page.getTitleText());
    }
}

Slightly more interesting, here’s how you can fill out a form (again, Google Search), click a button, and inspect the results:

[Test]
public void Google_Search_For_AspNetMvc_Yields_Link_To_Codeplex()
{
    var searchPage = (HtmlPage)webClient.getPage("http://www.google.com");
    ((HtmlInput)searchPage.getElementByName("q")).setValueAttribute("asp.net mvc");
    var resultsPage = (HtmlPage)searchPage.getElementByName("btnG").click();
 
    var linksToCodeplex = from tag in resultsPage.getElementsByTagName("a").toArray().Cast<HtmlAnchor>()
                          let href = tag.getHrefAttribute()
                          where href.StartsWith("http://")
                          let uri = new Uri(href)
                          where uri.Host.ToLower().EndsWith("codeplex.com")
                          select uri;
    CollectionAssert.IsNotEmpty(linksToCodeplex);
}

As you can see, the API is full of Java idioms and feels a bit odd to a .NET developer. It would be great if someone decided to create a .NET wrapper library to expose a nicer API, .NET-style PascalCase, better use of IEnumerable and generics so LINQ queries were simpler, etc.

The other side of it is that HtmlUnit is very powerful. You can trivially scan the DOM with XPath, search for child elements, invoke events, much more easily than with WatiN.

To show that HtmlUnit has great JavaScript and Ajax support, here’s an example of automating a jQuery AutoComplete plugin to check its suggestions:

[Test]
public void jQuery_Autocomplete_Lon_Suggests_London()
{
    // Arrange: Load the demo page
    var autocompleteDemoPage = (HtmlPage)webClient.getPage("http://jquery.bassistance.de/autocomplete/demo/");
 
    // Act: Type "lon" into the input box
    autocompleteDemoPage.getElementById("suggest1").type("lon");
    webClient.waitForBackgroundJavaScript(1000);
 
    // Assert: Suggestions should include "London"
    var suggestions = autocompleteDemoPage.getByXPath("//div[@class='ac_results']/ul/li").toArray().Cast<HtmlElement>().Select(x => x.asText());
    CollectionAssert.Contains(suggestions, "London");
}

In case you want to download all this code as a demo project you can run without needing IKVM or Java or anything weird, I’ve put it on GitHub.

Summary

This is experimental! I haven’t used this on any real project, though it was pretty effortless so far, so I’d definitely consider it. I don’t know how much faster HtmlUnit would be than a real browser, but it does bypass the trickiness of real browsers, which may be worth it alone. If I was going to use it seriously, I’d definitely make some .NET wrapper classes to hide the Java naming and idioms. HtmlUnit on .NET feels robust, but I haven’t pushed it hard.

Why would you want this? It’s mainly useful when you’re integration testing. Normally, integration tests can’t directly interact with your application’s code because it’s in a separate process (and possibly running on another machine). So, unlike unit tests, integration tests often struggle to test different configurations and have no option to use mocks.

Deleporter gets around that limitation. It lets you delve into a remote ASP.NET application’s internals without any special cooperation from the remote app (it just needs to include an extra IHttpModule in its config), and then you can do any of the following:

• Cross-process mocking, by combining it with any mocking tool. For example, you could inject a temporary mock database or simulate the passing of time (e.g., if your integration tests want to specify what happens after 30 days or whatever)
• Test different configurations, by writing to static properties in the remote ASP.NET appdomain or using the ConfigurationManager API to edit its <appSettings> entries.
• Run teardown or cleanup logic such as flushing caches. For example,  recently I needed to restore a SQL database to a known state after each test in the suite. The trouble was that ASP.NET connection pool was still holding open connections on the old database, causing connection errors. I resolved this easily by using Deleporter to issue a SqlConnection.ClearAllPools() command in the remote appdomain – the ASP.NET app under test didn’t need to know anything about it.

Rails developers can already do this either by hosting their app in-process (e.g., with WebRat) or using a cross-process mocking tool. I’ve already talked about hosting ASP.NET (MVC) apps in-process with integration tests (the starting point for this code); now I wanted to get the same benefits while hosting my app on a real web server.

This library is built on regular .NET remoting. But unlike regular remoting, it run arbitrary delegates, not just methods specifically exposed by the remoting service. If the code for your delegate isn’t already loaded into the ASP.NET appdomain, Deleporter will serialize the code and send it across. This is really essential for being useful from an integration test suite, because the test code wouldn’t normally be in the real app.

Enough talk! Code now.

To use it, first download the binary or compile the source to get Deleporter.dll. Reference it from both your integration test project and the ASP.NET MVC/WebForms app you’re testing.

Next, to get your ASP.NET app to listen for commands, edit its Web.config file to reference this extra HTTP module:

<configuration>
  <system.web>
    <httpModules>
      <!-- If you're using Visual Studio's built-in web server, or IIS 5 or 6, add this: -->
      <add name="DeleporterServerModule" type="DeleporterCore.Server.DeleporterServerModule, Deleporter" />
    </httpModules>
  </system.web>
  <system.webServer>
    <modules>
      <!-- If you're using IIS 7+, add this: -->
      <add name="DeleporterServerModule" type="DeleporterCore.Server.DeleporterServerModule, Deleporter" />
    </modules>
  </system.webServer>
</configuration>

Now, when you next start up your ASP.NET app, it will listen on TCP port 38473 - a randomly-chosen default - for commands from your integration test suite.

In your integration test code, you can now run arbitrary delegates in the remote ASP.NET app, sending both data and code into it, and optionally pulling data back. As a trivial example, you can pull back the server’s time local time:

[Test]
public void ServerClockHasCorrectYear()
{
    DateTime serverTime = Deleporter.Run(() => DateTime.Now);
    Assert.AreEqual(DateTime.Now.Year, serverTime.Year, "The server's clock is way off");
}

You can use multi-statement lambdas, too, even if they capture locals from the containing method. Deleporter will send the captured locals into the remote app (as long as they’re serializable), and if the remote app edits them, it will pull back the new values and update your locals. In other words, anonymous methods and multi-statement lambdas work with captured local variables just as if all the code was running locally. Example:

[Test]
public void SendingAndUpdatingCapturedLocals()
{
    DateTime serverTime = default(DateTime);
    string fileToLocate = "~/web.config";
    string mappedPhysicalPath = null;
 
    Deleporter.Run(() => {
        // This code, which both reads and writes locals from the outer scope,
        // runs in the remote ASP.NET application
        serverTime = DateTime.Now;
        mappedPhysicalPath = HostingEnvironment.MapPath(fileToLocate);
    });
 
    Assert.AreEqual(DateTime.Now.Year, serverTime.Year);
    Assert.IsTrue(File.Exists(mappedPhysicalPath)); // Of course, this will only be true if the ASP.NET app runs on the same machine as the integration test code
}

More realistic applications

So far I’ve only shown very artificial examples designed to illustrate the syntax. To give you a more realistic idea of how you could apply it, I can offer two sample applications.

• A very simple ASP.NET MVC app called WhatTimeIsIt whose sole behaviour varies according to the date. Like all good ASP.NET MVC applications, it uses the Dependency Injection pattern (in this example, with Ninject) to get the current date & time from an IDateProvider – the default implementation of which just returns DateTime.Now.
  It has simple integration tests that use Deleporter and Moq to inject a mock IDateProvider across the process boundary while the app is running, and then verifies some behaviour that only happens during specific dates. It also demonstrates an easy way of tidying up after the tests run (restoring the original IDateProvider) automatically without cluttering the test code with teardown logic.
  Here’s how the cross-process mocking works:

  // Inject a mock IDateProvider, setting the clock back to 1975
  var dateToSimulate = new DateTime(1975, 1, 1);
  Deleporter.Run(() => {
      var mockDateProvider = new Mock<IDateProvider>();
      mockDateProvider.Setup(x => x.CurrentDate).Returns(dateToSimulate);
      NinjectControllerFactoryUtils.TemporarilyReplaceBinding(mockDateProvider.Object);
  });

• An enhanced version of the SpecFlow BDD-style integration test suite from my previous blog post. It now uses Deleporter to apply a mock repository to verify certain specifications.
  For example, one of the Gherkin files now specifies this scenario:

  Scenario: Most recent entries are displayed first
  	Given we have the following existing entries
  		| Name      | Comment      | Posted date       |
  		| Mr. A     | I like A     | 2008-10-01 09:20  |
  		| Mrs. B    | I like B     | 2010-03-05 02:15  |
  		| Dr. C     | I like C     | 2010-02-20 12:21  |
        And I am on the guestbook page
      Then the guestbook entries includes the following, in this order
  		| Name      | Comment      | Posted date       |
  		| Mrs. B    | I like B     | 2010-03-05 02:15  |
  		| Dr. C     | I like C     | 2010-02-20 12:21  |
  		| Mr. A     | I like A     | 2008-10-01 09:20  |

Gotcha: What you must remember to avoid confusing problems

One behaviour might surprise you at first. It’s certainly caught me out plenty of times already. Each time you edit and recompile your integration test code, you must also recycle your ASP.NET appdomain (e.g., by resaving its Web.config file, or the nuclear option - running iisreset) otherwise the old code will still be loaded into it. That’s because, as far as I’m aware, there’s no way to unload an assembly from a .NET appdomain. Once your delegates have been transferred over, they’re staying there and can’t automatically be replaced by newer versions.

This isn’t a problem if you’re running an integration suite through your continuous integration (CI) system – the app code would be recompiled and hence reset between test suite runs. It’s only something to watch out for during local development. In the Deleporter.Test.Client project I’ve shown a crude but working way of automatically recycling the ASP.NET appdomain when the test suite starts running; you may be able to work out a better way of automating this if you need one.

Important security note

You should not enable DeleporterServerModule on a production web site. It’s only supposed to be used in dev and QA environments, because it’s literally an invitation to upload and execute arbitrary code.

To reduce the chance of a mistake, DeleporterServerModule will refuse to run if your web site was compiled in release mode – it will demand that you remove the IHttpModule from your Web.config file. Technically you should be OK if your firewall doesn’t accept inbound connections on port 38473 (or whatever port you configure it to listen on) anyway. Obviously, no warranties, you use it at your own risk, etc.

License: Ms-Pl

The mindset and terminology has shifted so much that some folks have started using a new name for the updated methodology: BDD. Here’s a brief overview*:

• Test Driven Development seeks to improve software quality by getting you to follow an iterative workflow often called red-green-refactor. In theory, it keeps quality high because if any change breaks earlier functionality, you should be notified by way of a failing test. Also, it should help you to make more pragmatic decisions about what code to write, because you only write the simplest code needed to make all your tests pass.
  Despite its unfortunate name, it became clear over time that TDD’s greatest potential benefit is not as a technique for testing code or detecting bugs, but in fact as a code design aid, because it provides a precise mechanism for specifying how your code should behave before you get mentally caught up in the actual implementation. However, in reality, I’ve seen many developers still struggle to break free from a test-oriented mindset and continue to amass reams of unit tests that simply exercise every possible code path without usefully describing what this proves or why it represents correct behaviour. This doesn’t aid design, very rarely detects bugs, and yet consumes a huge amount of maintenance time.
• Behaviour Driven Development retains the basic red-green workflow, but dramatically puts the emphasis on specifying behaviours that are understandable to people (often, business domain experts). It addresses questions such as “How much should I specify?” and “How should I organise and name my specifications so they are most useful?”
  To do this, BDD says you should elevate your mind to a level of behavioural abstraction above the code implementation. So don’t say “Constructor_Test” or even  “Constructor_NullParam _ThrowsArgumentNullException” but instead say “A shipment cannot be created without a recipient address”. The change of emphasis and terminology leads people to write more useful specifications.
  I won’t focus on the term BDD itself, because the term is not hugely useful: nobody has ever given it a precise definition as far as I’m aware, and there isn’t a lot of consensus about what it should mean (some say it should only be about the UI; others say not). Instead, in this blog post, I’m going to focus on the major tools that have come out of the BDD stable and the advantages they can give you.

Unit level or UI level?

You can write BDD-style specifications about individual code units, in which case you’ll  probably use a context/specification style framework like RSpec (for Ruby) or in .NET something like MSpec or SpecUnit (personally, I don’t like either). Unit-level specifications work brilliantly for most code that ultimately exposes an API for other code. But personally, I spend most of my time writing UI code, usually with ASP.NET MVC, and UI code is fundamentally different. It isn’t about producing an API – it’s about producing user experience (UX) behaviours. These behaviours typically aren’t atomic (contained within a single click or HTTP request); their essence exists only across a sequence of interactions. As such, unit-level specifications can fall short and fail to capture the UI behaviours you have in mind. I wrote before about why, in real projects, I’ve found unit tests to be of limited value for ASP.NET MVC controllers.

Alternatively, you can write BDD-style specifications about UI interactions. Assuming you’re building a web application, you’ll probably use a browser automation library like WatiR/WatiN or Selenium, and script it either using one of the frameworks I just mentioned, or a given/when/then tool such as Cucumber (for Ruby) or SpecFlow (for .NET). In this blog post, I’ll describe these tools and provide an example based on ASP.NET MVC.

* I don’t claim to be an authority on BDD, so if you think I’ve got this all wrong, I invite you to come along and tell me what I’ve misunderstood. Try to be specific, please. My existing understanding is based on having done a lot of TDD, seeing its limitations especially in large teams, and recognising that some of these BDD ideas look like they could help with that.

So, what’s Cucumber?

Cucumber is a Ruby tool that offers a natural-language way to specify behaviours in terms of “Given, When, Then” (GWT) sentence fragments. This language is called Gherkin. You create a plain-text file (called a “.feature” file) containing a set of scenario definitions written in Gherkin, for example

Scenario: Show logged in user name
  Given I am logged in as a user called "Steve"
   When I visit the homepage
   Then the page header displays the caption "Hello, Steve!"

… and also provide Ruby scripts that define what the runner should do when it hits matching steps, for example

Given /I am logged in as a user called "(.*)"/ do |name|
    create_user(name)
    sign_in_as(name)
end
 
...
 
Then /the page header displays the caption "(.*)"/ do |caption|
    page_header.should_contain(caption)
end

The syntax is extremely flexible: Cucumber only cares about the Gherkin keywords “Given”, “When”, “Then”, “And” (plus a few others); everything else is simply matched against your regular expressions. Any tokens captured by capture groups in your regexes are passed to your step definition blocks as parameters. For web applications, it’s normal for your step definitions to work by using a browser automation library such as WatiN/WatiR or Selenium.

When you invoke Cucumber, it finds, parses and runs your .feature files, reporting the results (success in green, pending in yellow, failure in red):

Because the GWT model inherently describes sequences of interactions, and because natural-language .feature files are inherently decoupled from the implementation code (they don’t mention the name of any classes or methods), this whole approach is a perfect fit for UI automation rather than unit-level testing. 

Because you can describe scenarios in the language of your domain and user interface, your resulting feature files should make perfect sense to business experts. So, unlike traditional unit test driven development, this methodology can finally deliver on the promise of executable specifications that are actually useful as documentation.

Doing it in .NET with SpecFlow

As I mentioned, Cucumber itself is a Ruby tool. At the end of this blog post I’ll discuss some options for running the original Cucumber tool in a .NET environment, but for now let’s forget about that - you can sidestep all that complexity with SpecFlow.

SpecFlow is an open-source .NET tool that lets you write specifications using 100%-Cucumber-compatible Gherkin syntax, and has a number of advantages over Cucumber itself:

• It integrates with Visual Studio, which means you get File->New templates for creating new feature files
• More crucially, it gives complete VS debugger support, so you can set breakpoints on Given/When/Then lines in your .feature files and step through their execution
• You can implement your step definitions in any .NET language
• When you compile a project containing SpecFlow feature files, the output is an NUnit test assembly, so you can use your favourite NUnit-compatible test runner or existing CI infrastructure to run the specifications with no additional configuration.
• It doesn’t have such a ridiculous name It’s sad, but at one of my recent clients, several managers refused to take Cucumber seriously and wouldn’t pay attention to Cucumber specifications purely because of the name. That’s the real world for you!

Example: Combining ASP.NET MVC, SpecFlow, and WatiN

To give you something concrete to get started with, I’ve made a very simple ASP.NET MVC 2 example application, built following the BDD workflow and with a full set of specs. To try it out,

• First download and install SpecFlow (choose either the VS2008 or VS2010 version)
• Download and open the Guestbook source code from Github
• Compile and open Guestbook.Spec.dll in NUnit GUI

You should easily be able to understand in high-level terms what the application does by reading the Gherkin specifications. Just imagine these specifications are the written result of a design meeting where you’ve got your client or boss to tell you how the user interactions should work.

Feature: Browsing
   In order to see who's been on the site
   As a user
   I want to be able to view the list of posts

Scenario: Navigation to homepage
   When I navigate to /Guestbook
   Then I should be on the guestbook page

Scenario: Viewing existing entries
   Given I am on the guestbook page
   Then I should see a list of guestbook entries
    And guestbook entries have an author
    And guestbook entries have a posted date
    And guestbook entries have a comment

Feature: Posting
   In order to express my views
   As a user
   I want to be able to post entries into the guestbook

Scenario: Navigation to posting page
   Given I am on the guestbook page
   Then I should see a button labelled "Post a New Entry"
   When I click the button labelled "Post a New Entry"
   Then I am on the posting page

Scenario: Viewing the posting page
   Given I am on the posting page
   Then I should see a field labelled "Your name"
    And I should see a field labelled "Your comment"

Scenario: Posting a valid entry
   Given I am on the posting page
     And I have filled out the form as follows
       | Label          | Value             |
       | Your name      | Jakob             |
       | Your comment   | Das ist gut!      |
   When I click the button labelled "Post"
   Then I should be on the guestbook page
    And I see the flash message "Thanks for posting!"
    And the guestbook entries includes the following
       | Name      | Comment      | Posted date          |
       | Jakob     | Das ist gut! | (within last minute) |

Whenever you hit save while editing one of these files, SpecFlow generates a C# NUnit test fixture containing a [Test] method for each scenario. You don’t have to know or care about that, though. All you have to do is create step definitions in C# to match the lines in the Gherkin files.

At first, you won’t have any matching step definitions, so the NUnit test runner will show the tests as “inconclusive”:

Notice that in the “Text Output” tab it provides C# stub code to create a matching step definition. Here’s some examples of the finished step definitions, written using WatiN to automate a browser:

[When(@"I navigate to (.*)")]
public void WhenINavigateTo(string relativeUrl)
{
    var rootUrl = new Uri(ConfigurationManager.AppSettings["RootUrl"]);
    var absoluteUrl = new Uri(rootUrl, relativeUrl);
    WebBrowser.Current.GoTo(absoluteUrl);
}
 
[Then(@"I see the flash message ""(.*)""")]
public void ThenISeeTheFlashMessage(string message)
{
    var flashElement = WebBrowser.Current.Element("flashMessage");
    Assert.That(flashElement.Text, Is.EqualTo(message));
}
 
[Given(@"I have filled out the form as follows")]
public void GivenIHaveFilledOutTheFormAsFollows(TechTalk.SpecFlow.Table table)
{
    foreach (var row in table.Rows) {
        var labelText = row["Label"] + ":";
        var value = row["Value"];
        WebBrowser.Current.TextFields.First(Find.ByLabelText(labelText)).TypeText(value);
    }
}

There’s lots more I could say about WatiN syntax and Gherkin best practices, but this is a pretty long post already so I’ll leave that to others.

In case anyone’s wondering, I’m not trying to claim with this post that Cucumber-style UI automation tests are the one true way to do BDD, or that unit testing is dead. UI automation tends to be much more difficult to do than unit-level tests, because there are many more moving parts and you have to concern yourself with test data, setup and teardown, etc. Seriously, it is hard work, and requires a lot more persistence and dedication than mere unit testing, though it can also give many extra benefits especially with regard to detecting regressions.

What about Running Cucumber Itself on .NET?

If, for some reason, you don’t want to use SpecFlow and would rather run the original Ruby-based Cucumber tool, there are two main options for .NET programmers:

• Use IronRuby. The current version of IronRuby is entirely capable of running Cucumber. It isn’t as easy to set up as SpecFlow, but only took me 30 minutes or so. The drawbacks are that (1) you have to write step definitions in Ruby, unless someone can come up with a clever way to call arbitrary .NET methods directly, (2) it runs Cucumber pretty slowly, at least in my testing, and (3) it doesn’t trivially integrate with existing CI infrastructure like an NUnit-based solution does.
• Use Cuke4Nuke. This is an open-source project that aims to bridge the gap between the original Cucumber tool running on any Ruby interpreter (not necessarily IronRuby) and step definitions written in .NET. It works by hosting your .NET step definitions in a small .NET-based “server” application, which Cucumber communicates with over a TCP/IP connection. I must admit I had a bad experience trying to make this work – it took hours. The latest Ruby interpreter didn’t include all the necessary libraries for Cuke4Nuke, and then when I’d hacked all the dependencies together, it wasn’t clear to me where to put the magic “cucumber.wire” file or what parameters you can use with the .NET server app. And then it was also pretty slow compared to keeping everything in .NET with SpecFlow. And it had a very limited API for running setup/teardown code around scenarios.

If you want to write step definitions in a .NET language, I don’t know why you wouldn’t just use SpecFlow. Since it’s 100% Gherkin-compatible, your feature files could in future be moved to a different implementation if desired. If you’ve tried others, I’d be interested to hear about your experiences. And no, I haven’t tried Spectre yet.

Injecting Mocks into a Running ASP.NET Application

In the next post, I’ll show you one possible way to simplify or speed up UI-level specifications a little. It’s a cross-process code injection library that, among other things, you can use to inject a mock database into a running ASP.NET application without it knowing about it. Or, you can override configuration, or maybe simulate the passing of time.

Sometimes, it isn’t desirable to meddle with your app’s internals, but sometimes it is. Ruby coders can already do it using Webrat or cross-stub, and I’ve found reasons to want that power over .NET apps too.

Since all the fuss about [Required] validators in ASP.NET MVC 2, the validation behaviour was changed in Release Candidate 2. Previously, the framework only validated fields that were actually posted in the form, whereas now it validates all the fields on any model object that the model binder gives you. Most people prefer this new behaviour, because they don’t like the idea that someone can avoid a [Required] rule just by using FireBug or similar to delete the HTML form field. That’s pretty understandable, and well done to the MVC team for agreeing to this change so late in the production cycle.

However, there’s a drawback to this change of behaviour. What about when you want partial validation (i.e., validating only a subset of your model’s fields)? Maybe your screen only provides input controls for a subset of fields. In fact, right now, I’m making a multi-step wizard, sharing a single model class across all steps (because it’s convenient). Obviously I don’t want to show errors on step 1 about fields that appear on step 2, especially not “A value for (some field you can’t see) is required.”

Doing Partial Validation

In this specific case, the pre-RC2 behaviour would have been more convenient. I only want to validate the values that were actually posted on each step. A very simple way of achieving that is to use this action filter:

public class ValidateOnlyIncomingValuesAttribute : ActionFilterAttribute
{
    public override void OnActionExecuting(ActionExecutingContext filterContext)
    {
        var modelState = filterContext.Controller.ViewData.ModelState;
        var valueProvider = filterContext.Controller.ValueProvider;
 
        var keysWithNoIncomingValue = modelState.Keys.Where(x => !valueProvider.ContainsPrefix(x));
        foreach (var key in keysWithNoIncomingValue)
            modelState[key].Errors.Clear();
    }
}

Now, if you use this filter on a controller or action method…

[ValidateOnlyIncomingValues]
public class SomeController : Controller
{
   // ...
}

then the filter will discard any validation errors that correspond to fields that were not even posted during this request.

But what about security?

Obviously this means that someone can bypass my [Required] rules by modifying their HTTP posts so they don’t include any key-value pair for any given field.

This is no problem at all. My real validation logic is all in the domain layer, so it doesn’t really matter what happens in the UI layer. The validation feedback from the UI layer is only for the end user’s convenience – if they choose to bypass it, they’ll just get an error message later.

But what about validation? We don’t like your blank item names or your negative prices, sonny!

As you probably know, ASP.NET MVC 2 supports DataAnnotations attributes out of the box, so you can mark up your model as follows.

public class Gift
{
    [Required]
    public string Name { get; set; }
 
    [Range(0.01, double.MaxValue, ErrorMessage = "Please enter a price above zero.")]
    public double Price { get; set; }
}

Server-side validation

Server-side validation is trivial, assuming you want to use the default model binder convention of validating each model object when it’s bound. In the action that receives the form post, just check ModelState.IsValid, and refuse to accept the data if it isn’t.

[HttpPost]
public ActionResult Index(IEnumerable<Gift> gifts)
{
    if (ModelState.IsValid)
        return View("Completed", gifts);
    else
        return View(gifts); // Redisplay the form with errors
}

You’ll also need to specify where the validation error messages should appear. Update the GiftEditorRow.ascx partial by placing a couple of Html.ValidationMessageFor() helpers somewhere inside the row.

<%= Html.ValidationMessageFor(x => x.Name) %>
<%= Html.ValidationMessageFor(x => x.Price) %>

Now, if the incoming data doesn’t satisfy your rules, the form will be re-rendered, displaying appropriate messages.

Client-side validation

It gets a bit more complicated if you also want client-side validation. You could use xVal (I’m not sure whether it would be easier or harder), but I want to get better acquainted with ASP.NET MVC 2’s built-in client-side validation feature, so I’m going to use that.

I don’t even know if there’s an officially recommended way of doing ASP.NET MVC 2 client-side validation when you’re dynamically adding and removing form elements, but I’ll show you a technique I found that will do it. Be warned: this is rather hacky. I’d be interested to hear if anyone can suggest a better way.

First, let’s set up client-side validation in the normal way. Enable client-side validation on your form by calling Html.EnableClientValidation():

<h2>Gift List</h2>
What do you want for your birthday?
 
<% Html.EnableClientValidation(); %>
<% using(Html.BeginForm()) { %>
    (rest as before)
<% } %>

Now, as long as you’ve already referenced MicrosoftAjax.js and MicrosoftMvcValidation.js, you can run the app and you’ll immediately get working client-side validation! But hang on a minute… it only works for the elements that are present when the form is first rendered, *not* the elements added when the user clicks “Add another…”.

We need a way to capture the extra validation rules added each time the user adds a new row and somehow attach them to the form. One possible approach is to create a new type of custom view result that registers the validators associated with any partial that is renders, and have it emit some JavaScript to attach those validators to the form.

Here’s a possible implementation. Don’t worry if you don’t understand it.

public class AjaxViewResult : ViewResult
{
    public string UpdateValidationForFormId { get; set; }
 
    public AjaxViewResult(string viewName, object model)
    {
        ViewName = viewName;
        ViewData = new ViewDataDictionary { Model = model };
    }
 
    public override void ExecuteResult(ControllerContext context)
    {
        var result = base.FindView(context);
        var viewContext = new ViewContext(context, result.View, ViewData, TempData, context.HttpContext.Response.Output);
 
        BeginCapturingValidation(viewContext);
        base.ExecuteResult(context);
        EndCapturingValidation(viewContext);
 
        result.ViewEngine.ReleaseView(context, result.View);
    }
 
    private void BeginCapturingValidation(ViewContext viewContext)
    {
        if (string.IsNullOrEmpty(UpdateValidationForFormId))
            return;
        viewContext.ClientValidationEnabled = true;
        viewContext.FormContext = new FormContext { FormId = UpdateValidationForFormId };
    }
 
    private void EndCapturingValidation(ViewContext viewContext)
    {
        if (!viewContext.ClientValidationEnabled)
            return;
        viewContext.OutputClientValidation();
        viewContext.Writer.WriteLine("<script type=\"text/javascript\">Sys.Mvc.FormContext._Application_Load()</script>");
    }
}

Now, we can change the BlankEditorRow() action method to render its partial using this custom view result.

public ViewResult BlankEditorRow(string formId)
{
    return new AjaxViewResult("GiftEditorRow", new Gift()) { UpdateValidationForFormId = formId };
}

You’ll notice that BlankEditorRow() now needs to be told which form it should attach the new validators to. You’ll have to update the link to BlankEditorRow to add this information to the query string.

<%= Html.ActionLink("Add another...", "BlankEditorRow", new { ViewContext.FormContext.FormId }, new { id = "addItem" }) %>

Et voila! Each time the user appends a new row, its validators will magically be associated with the form.

The problem with removing fields

If you proceed with this approach, you’ll soon discover a further flaw. Even if the user removes fields from the form, any validators associated with those fields will still be lurking.  If a field is displaying a “required” error message, and then the user deletes the containing row, it will become impossible to submit the form because the field is still required, except now there is nowhere to enter any data for it. Whoops!

Again, I don’t know if there’s a recommended way to deal with this, but one possibility is to edit the MicrosoftMvcValidation.debug.js script a little. Let’s update the logic slightly so that, if a validator is associated with fields that no longer exist, the validator should no longer apply.

Find the function called “Sys_Mvc_FieldContext$validate”, and right at the top, add the following:

// [Added] Permanently disable this validator if its associated elements are no longer in the document
for (var j = 0; j < this.elements.length; j++) {
    if (!Sys.Mvc.FormContext._isElementInHierarchy(document.body, this.elements[j])) {
        this.validations = [];
        break;
    }
}

Now, assuming you’re referencing the file you just edited (MicrosoftMvcValidation.debug.js, rather than MicrosoftMvcValidation.js), you should get the desired change in behaviour. Deleting a row in the editor now kills its associated validation rules properly, so they don’t rise from the grave and block form submissions like invisible validation zombies.

Summary

Once you’ve got the AjaxViewResult class, it doesn’t take that much work to enable client-side validation on dynamically changing forms. Maybe there’s a better and less hacky way though… Anybody got any suggestions?

• Download the completed demo project (78kb)

Download the demo project or read on for details.

Update (Feb 11, 2010):Thanks to Ryan Rivest for pointing out a bug in the original code and providing a neat fix. Code updated.

Getting Started

For this example I’m going to go with the same theme as last time and build a gift list editor. Our model object can simply be the following:

public class Gift
{
    public string Name { get; set; }
    public double Price { get; set; }
}

Displaying the Initial UI

To display the initial data entry screen, add an action method that renders a view and passes some initial collection of Gift instances.

public ActionResult Index()
{
    var initialData = new[] {
        new Gift { Name = "Tall Hat", Price = 39.95 },
        new Gift { Name = "Long Cloak", Price = 120.00 },
    };
    return View(initialData);
}

Next, add a view for this action, and make it strongly-typed with a model class of IEnumerable<Gift>. We’ll create an HTML form, and for each gift in the collection, we’ll render a partial to display an editor for that gift.

<h2>Gift List</h2>
What do you want for your birthday?
 
<% using(Html.BeginForm()) { %>
    <div id="editorRows">
        <% foreach (var item in Model)
            Html.RenderPartial("GiftEditorRow", item);
        %>
    </div>
 
    <input type="submit" value="Finished" />
<% } %>

I know it would be possible to use ASP.NET MVC 2’s Html.EditorFor() or Html.EditorForModel() helpers, but later on we’re going to need more control over the HTML field ID prefixes so in this example it turns out to be easier just to use Html.RenderPartial().

Next, to display the editor for each gift in the sequence, add a new partial view at /Views/controllerName/GiftEditorRow.ascx, strongly-typed with a model class of Gift, containing:

<div class="editorRow">
    <% using(Html.BeginCollectionItem("gifts")) { %>
        Item: <%= Html.TextBoxFor(x => x.Name) %> 
        Value: $<%= Html.TextBoxFor(x => x.Price, new { size = 4 }) %> 
    <% } %>
</div>

Here’s where we get to start using ASP.NET MVC 2 features and make things slightly easier than before. Notice that I’m using strongly-typed input helpers (Html.TextBoxFor()) to avoid the need to build element IDs manually. These helpers are smart enough to detect the “template context” in which they are being rendered, and use any field ID prefix associated with that template context.

You might also be wondering what Html.BeginCollectionItem() is. It’s a HTML helper I made that you can use when rendering a sequence of items that should later be model bound to a single collection. You give it some name for your collection, and it opens a new template context for that collection name, plus a random unique field ID prefix. It also automatically renders a hidden field, which in this case is called gifts.index, populating it with that unique ID, so when you later model bind to a list, ASP.NET MVC 2 will know that all the fields in this context should be associated with a single .NET object.

And now, if you visit the Index() action, you should see the editor  as shown below. (I’ve added some CSS styles, obviously.)

Receiving the Form Post

To receive the data posted by the user, add a new action method as follows.

[HttpPost]
public ActionResult Index(IEnumerable<Gift> gifts)
{
    // To do: do whatever you want with the data
}

How easy is that? Because Html.BeginCollectionItem() observes ASP.NET MVC 2 model binding conventions, you can receive all the items in the list without having to do anything funky.

You could also achieve the same with less code by using the built-in Html.EditorFor() or Html.EditorForModel() helpers, but because these use a different indexing convention (an ascending sequence, not a set of random unique keys), things would get more difficult when you try to add or remove items dynamically.

Dynamically Adding Items

If the user wants to add another item, they’ll need something to click to say so. Let’s add an “Add another…” link. Add the following to the main view, just before the “Finished” button.

<%= Html.ActionLink("Add another...", "BlankEditorRow", null, new { id = "addItem" }) %>

This is a link to an action called BlankEditorRow which doesn’t exist yet. The idea is that the BlankEditorRow action will return the HTML markup for a new blank row. We can fetch this markup via Ajax and dynamically append it into the page.

To make this Ajax call and append the result into the page, make sure you’ve got jQuery referenced, and create a click handler similar to this:

$("#addItem").click(function() {
    $.ajax({
        url: this.href,
        cache: false,
        success: function(html) { $("#editorRows").append(html); }
    });
    return false;
});

Note that it’s very important to tell jQuery to tell the browser not to re-use cached responses, otherwise those unique IDs won’t always be so unique… And before we forget, we’ll need to put the BlankEditorRow action in place:

public ViewResult BlankEditorRow()
{
    return View("GiftEditorRow", new Gift());
}

As you can see, it simply renders the same editor partial, passing a blank Gift object to represent the initial state. I’m very pleased that you can re-use the same editor partial in this way – it means there’s no duplication of view markup and we can stay totally DRY.

And that, in fact, is all you have to do – each time the user clicks “Add another…”, the client-side code will inject a new blank row into the editor. Because each row has its own unique ID, when the user later posts the form, all the data will be model bound into a single IEnumerable<Gift>.

Dynamically Removing Items

Removing items is easier, because all you have to do is remove the corresponding DOM elements from the HTML document. If the elements are gone, their contents won’t be posted to the server, so they won’t be present in the IEnumerable<Gift> that your action receives.

Add a “delete” link to the GiftEditorRow.ascx partial:

<a href="#" class="deleteRow">delete</a>

This needs to go inside the DIV with class “editorRow”, so you can handle clicks on it as follows:

$("a.deleteRow").live("click", function() {
    $(this).parents("div.editorRow:first").remove();
    return false;
});

Notice that this code uses jQuery’s “live” function, which tells it to apply the click handler not only to the elements that exist when the page is first loaded, but also to any matching elements that are dynamically added later.

You’ve now got a working editor with “add” and “delete” functionality.

Summary

I hope you can see that editing variable-length lists can be very easy. Other than the reusable Html.BeginCollectionItem() helper, I’ve just shown you every line of code needed for this particular strategy, and in total it’s just 36 lines (including the model, action methods, views, and JavaScript, but excluding lines that are purely whitespace or braces).

Download the full demo project

But what about validation?

Yes, I haven’t forgotten about that! You can already do your server-side validation any way you want – by default the model binder will respect any rules associated with your model.

In the next post, I’ll show how you can integrate this list editing strategy with ASP.NET MVC 2’s client-side validation feature.

• First, your controller begins one or more external I/O calls (e.g., SQL database calls or web service calls). Without waiting for them to complete, it releases the thread back into the ASP.NET worker thread pool so that it can deal with other requests.
• Later, when all of the external I/O calls have completed, the underlying ASP.NET platform grabs another free worker thread from the pool, reattaches it to your original HTTP context, and lets it complete handling the original request.

This is intended to boost your server’s capacity. Because you don’t have so many worker threads blocked while waiting for I/O, you can handle many more concurrent requests. A while back I blogged about a way of doing this with an early preview release of ASP.NET MVC 1.0, but now in ASP.NET MVC 2.0 it’s a natively supported feature.

In this blog post I’m not going to explain how to create or work with asynchronous controllers. That’s already described elsewhere on the web, and you can see the process in action in TekPub’s “Controllers: Part 2” episode, and you’ll find extremely detailed coverage and tutorials into my forthcoming ASP.NET MVC 2.0 book when it’s published. Instead, I’m going to describe one possible way to measure the performance effects of using them, and explain how I learned that under many default circumstances, you won’t get any benefit from using them unless you make some crucial configuration changes.

Measuring Response Times Under Heavy Traffic

To understand how asynchronous controllers respond to differing levels of traffic, and how this compares to a straightforward synchronous controller, I put together a little ASP.NET MVC 2.0 web application with two controllers. To simulate a long-running external I/O process, they both perform a SQL query that takes 2 seconds to complete (using the SQL command WAITFOR DELAY ‘00:00:02′) and then they return the same fixed text to the browser. One of them does it synchronously; the other asynchronously.

Next, I put together a simple C# console application that simulates heavy traffic hitting a given URL. It simply requests the same URL over and over, calculating the rolling average of the last few response times. First it does so on just one thread, but then gradually increases the number of concurrent threads to 150 over a 30-minute period. If you want to try running this tool against your own site, you can download the C# source code.

The results illustrate a number of points about how asynchronous requests perform. Check out this graph of average response times versus number of concurrent requests (lower response times are better):

To understand this, first I need to tell you that I had set my ASP.NET MVC application’s worker thread pool to an artificially low maximum limit of 50 worker threads. My server actually has a default max threadpool size of 200 – a more sensible limit – but the results are made clearer if I reduce it.

As you can see, the synchronous and asynchronous requests performed exactly the same as long as there were enough worker threads to go around. And why shouldn’t they?

But once the threadpool was exhausted (> 50 clients), the synchronous requests had to form a queue to be serviced. Basic queuing theory tells us that the average time spent waiting in a queue is given by the formula:

… and this is exactly what we see in the graph. The queuing time grows linearly with the length of the queue. (Apologies for my indulgence in using a formula – sometimes I just can’t suppress my inner mathematician. I’ll get therapy if it becomes a problem.)

The asynchronous requests didn’t need to start queuing so soon, though. They don’t need to block a worker thread while waiting, so the threadpool limit wasn’t an issue. So why did they start queuing when there were more than 100 clients? It’s because the ADO.NET connection pool is limited to 100 concurrent connections by default.

In summary, what can we learn from all this?

• Asynchronous requests are useful, but *only* if you need to handle more concurrent requests than you have worker threads. Otherwise, synchronous requests are better just because they let you write simpler code. 
  (There is an exception: ASP.NET dynamically adjusts the worker thread pool between its minimum and maximum size limits, and if you have a sudden spike in traffic, it can take several minutes for it to notice and create new worker threads. During that time your app may have only a handful of worker threads, and many requests may time out. The reason I gradually adjusted the traffic level over a 30-minute period was to give ASP.NET enough time to adapt. Asynchronous requests are better than synchronous requests at handling sudden traffic spikes, though such spikes don’t happen often in reality.)
• Even if you use asynchronous requests, your capacity is limited by the capacity of any external services you rely upon. Obvious, really, but until you measure it you might not realise how those external services are configured.
• It’s not shown on the graph, but if you have a queue of requests going into ASP.NET, then the queue delay affects *all* requests – not just the ones involving expensive I/O. This means the entire site feels slow to all users. Under the right circumstances, asynchronous requests can avoid this site-wide slowdown by not forcing the other requests to queue so much.

Gotchas with Configuring for Asynchronous Controllers

The main surprise I encountered while trying to use asynchronous controllers was that, at first, I couldn’t observe any benefit at all. In fact, it took me almost a whole day of experimentation before I discovered all the things that were preventing them from making a difference. If I hadn’t been taking measurements, I’d never have known that my asynchronous controller was entirely pointless under default configurations. For me it’s a reminder that understanding the theory isn’t enough; you have to be able to measure it.

Here are some things you might not have realised:

• Don’t even bother trying to load test your asynchronous controllers using IIS on Windows XP, Vista, or 7. Under these operating systems, IIS won’t handle more than 10 concurrent requests anyway, so you certainly won’t observe any benefits.
• If your application runs under .NET 3.5, you will almost certainly need to change its MaxConcurrentRequestsPerCPU setting. By default, you’re limited to 12 concurrent requests per CPU – and this includes asynchronous as well as synchronous ones. Under this default setting, there’s no way you can get anywhere near the default worker threadpool limit of 100 per CPU, so you might as well handle everything synchronously. For me this is the biggest surprise; it just seems like a chronic mistake. I’d be interested to know if anyone can explain this. Fortunately, if your app runs under .NET 4, then the MaxConcurrentRequestsPerCPU value is 5000 by default.
• If your external I/O operations are HTTP requests (e.g., to a web service), then you may need to turn off autoConfig and alter your maxconnections setting (either in web.config or machine.config). By default, autoconfig allows for 12 outbound TCP connections per CPU to any given address. That limit might be a little low.
• If your external I/O operations are SQL queries, then you may need to change the max ADO.NET connection pool size. By default, the limit is 100 in total. Whereas the default ASP.NET worker thread pool limit is 100 per CPU, so you’re likely to hit the connection pool size limit first (unless, I guess, you have less than one CPU…).

If you don’t have high capacity requirements – e.g., if you’re building a small intranet application – then you can probably forget all about this blog post and asynchronous controllers in general. Most ASP.NET developers have never heard of asynchronous requests and they get along just fine.

But if you are trying to boost your server’s capacity and think asynchronous controllers are the answer, please be sure to run a practical test and make sure you can observe that your configuration meets your needs.