Things, that I got during writing .Net connector for Redis.

Redis is a distributed key-value storage. Connector is a protocol-driver for storing/retrieving data from Redis.  Recently I have developed one. I taught following simple lessons from it:

SRP principle makes code better prepared for changes

During development of the connector, I were stick to "SRP" and "TDD" principle.  Later i have realized that SRP is really helps, if you face requirement of unexpected code changes.

In short SRP says that you should have only one reason to change the class.

.NET code is as fast as native implementation

Spuriously, native ANSI-C implementation of Redis benchmark is as fast, as .NET implementation of the connector with same benchmark, moreover with all those patterns and enterprise stuff.

.NET buffered stream is a good abstraction

Connector is written in a way – that avoids copying arguments (byte arrays) in one big array, for sending to Redis. Instead it saves references to all arguments and writes in in a sequence. That causes serious performance problems. I have avoided this, just by wrapping socket's write stream with buffered stream. It's gave me 2x performance 'boost' (2000 requests per second vs approx. 5000).

comand pipelininG is a good IDEA

Command pipelining, it's a feature, when you put all you requests trough single socket. For example, is you send (in a row) 3 GET's for "foo", "bar" and "baz" keys – you will receive commands results in the same order. It gave me good results over the 100MBit network:

512k Localhost
.....Normal :131.095123900879
.....Pipelined 1 conn :59.8802395209581
.....Pipelined 2 conn :79.8403193612775
.....Pipelined 5 conn :99.7804829375374
.....Pipelined 10 conn :120
.....Pipelined 50 conn :137.793707686181
512k Remote host
.....Normal :37.9241516966068
.....Pipelined 1 conn :40.7918416316737
.....Pipelined 2 conn :42.4084816963393
.....Pipelined 5 conn :42.3238171291675
.....Pipelined 10 conn :40.4080816163233
.....Pipelined 50 conn :39.8962696987832
1kb localhost
.....Normal :7330.13589128697
.....Pipelined 1 conn :6500.89802434644
.....Pipelined 2 conn :7438.32335329341
.....Pipelined 5 conn :8085.56577369052
.....Pipelined 10 conn :7821.23575284943
.....Pipelined 50 conn :6627.87442511498
1kb Remote host
.....Normal :500
.....Pipelined 1 conn :6543.7125748503
.....Pipelined 2 conn :9555.51110222044
.....Pipelined 5 conn :11052.7788884446
.....Pipelined 10 conn :10608
.....Pipelined 50 conn :520

multithreaded code is VERY hard to debug

During implementation of pipelining I faced with a racing problem. Old pipelining algorithm allows  racing during parsing response from Redis. Microsoft Chess helped me a lot to test and localize this problem. It's a special tool, that runs test code with worst-case thread switching scenario.

Free profilers exists (NProf and Slimtune)

Earlier, I had no idea how to profile application at low cost (JetBrains and Ant's Profilers are expensive). Recently I found NProf and SlimTune. They helped me a lot to find bottlenecks.

When you have a hammer, everything looks like a nail.

Once, I have discussed with my colleague architecture candidate of high-performance system. In brief, system should receive a big amount of small incoming requests and provide API for querying on this data. 

The numbers:

• number of incoming request  - 5000 per second.
• number of select - 5 per 10 seconds
• maximum latency time between registering request and it's availability trough API is 5 sec.

My colleague is a DB guy who is working with creating big and rather complex automation solutions. He proposed to take the most enterpricy  (and expensive) database, do all of the processing in db and use Table-Views as external API to the system.

This solutions introduce a lot of problems.

1. Vendor affinity

Once, you exposed a part of your db as external API, you will never change it (nor db vendor, nor db structure).

2. Performance optimization

You have absolutely no control on queries, that will run against your system. You can buy several top-end servers and still some idiot will invent a query that will bog them down. Stored-procedures is a solution, but it will refuse an argument, that exposing views will allow us not to worry about queries in external API.

3. Scalability

It's a fact that SQL server isn't scaling well, horizontally at least.  The system is not so complex, and consistency checks, joins and complex select queries is nor required here.

Better solution

System can be divided in 3 logical parts: Request Receiver, Request Store and External API.

Receiver and external API are pretty simple: protobuf and web-services. Protobuf is fast enough for thin channels of each sending-client and web-service is pretty simple for usage for API consumers.

But the most uncertainty is left to Request Store. As it's organization will be changing during load tests and API modification, it should be hidden under very strict and simple interfaces. It's handy, that system doesn't have complex objects speeded across many tables. We can use POCOs everywhere.

And the storage itself can be implemented using traditional SQL databases or distributed storages. Thanks' to interfaces, it will not affect remaining to parts.

Generalize everything pattern

Sometimes I 'am facing "generalize everything" pattern, when developer trying to invent a "racing car" for a trip for a few kilometers (or miles :) ). But as soon car is ready, it happens that car can be driven only on straight flat road, with no turns with constant speed only on 5th gear. Later, when you destinations changing you apply more and more hand-made solutions and hacks. Finally, after several month passed, car got a bit rusty and needs support, but is takes a lot of time (and money) even for clever mechanic to solve the problem. It's because car is full of "witty" and "smart" puzzles. Just take a walk, it's healthy.
A real life example: Customer required us to implement "read-only" feature for product, that we developing. After some discussion 2 possible solutions were found:

1. Automate all processing of permissions. Base Presenter will automatically determine if user is read-only, call a special method in a Base View that is responsible for hiding all buttons, that is tagged with special marked (Save buttons).
2. Move it's responsibility to concrete presenter and concrete view. View will have method, that hides edit buttons where buttons are defined explicitly. Presenter with call explicitly View.HideEditButtons() method if required.

Let's analyze pros and cons of each solution:

Maintainability

Using first approach - firstly, it seems that developers will produce less bugs during implementation of the each page. At least they need only to place properly annotated buttons on the page.  But when customer asks for a simple feature, like "please enable this button for read-only". Developer implements workaround on one page, on second, third and you have a bunch of pages with a different implementation of same feature. And finally when tester reports a bug, you need to analyze a markup (with annotation, do you remember), view implementation for hacks, base view implementation for hiding mechanism itself and presenter for an additional hacks.
In second approach, developer will modify a View.HideEditButtons method (remove or add hiding of buttons), or introduce new permission rule (if it's required). It's pretty easy now, because implementation is not coupled with "Base*" stuff and will not broke anything else in the system. Support team will need to analyze only 2 files – view and presenter, and all control hiding/showing are declared explicitly there.

Lines of code

It's nearly equal. On the left side there is a new-born framework and customizations for it. On the right side a lot of boilerplate permission checks and show/hide control code divided into methods.

Code complexity

General solution is full of polymorphism, default and implicit behavior, if branches, typeof-s. It's easy to forget a small thing, and here it is – a bug. Second solution is a set of flat methods with a rare branches like If(!Allowed){View.HideSomething();} that is easy to read even if you unfamiliar with a context.

Code duplication

Using 2nd approach, it's seems that you have written View/Hide methods and permissions check zillions of times. But that methods are completely different methods with the same pattern. Method is simple and contains only vital information that is differs from page to page. In 1st approach during simple tasks there is no code/pattern duplication at all, but later developers will invent their own hacks, that will do similar tasks in a different ways.

Robust ASP.NET navigation - Fluent interface for site map generation

ASP.Net provides classes that is suitable for holding current sitemap structure, displaying it, using default controls, searching for current selected node, but not for building it in a elegant way, at least at WCSF. The code in MSDN article is pretty simple

SiteMapNodeInfo moduleNode = new SiteMapNodeInfo("Customers", "~/Customers/ApproveCustomerView.aspx", "Approve Customer");
siteMapBuilderService.AddNode(moduleNode);
siteMapBuilderService.AddNode(moduleNode, parentNode);
siteMapBuilderService.AddNode(moduleNode, 100);
siteMapBuilderService.AddNode(moduleNode, “AllowViewModule1”);

But when you have numerous of pages, sitemap building code can turn into crap like this:

CustomSiteMapNodeInfo HomeNode = new CustomSiteMapNodeInfo("Home", "~/Default.aspx", "Home", "Home", true);
SiteMapBuilderService.AddNode(HomeNode);

SiteMapBuilderService.RootNode.Url = "~/Default.aspx";
SiteMapBuilderService.RootNode.Title = "FOO-BAR";

CustomSiteMapNodeInfo CustomersNode = new CustomSiteMapNodeInfo("Customers", "#Customers_List", "Customers", true);
SiteMapBuilderService.AddNode(CustomersNode);

SiteMapBuilderService.AddNode(SiteMapBuilderServiceHelper.CreateLocalizableNode("Customers", "CustomersList", true, true), SalesNode);
CustomSiteMapNodeInfo TransactionsNode = SiteMapBuilderServiceHelper.CreateLocalizableNode("Transactions", "TransactionsList", true, true);
SiteMapBuilderService.AddNode(TransactionsNode, CustomersNode);

CustomSiteMapNodeInfo IncomeNode = SiteMapBuilderServiceHelper.CreateLocalizableNode("Income", "IncomeList", true, true);
SiteMapBuilderService.AddNode(IncomeNode , CustomersNode);

SiteMapBuilderService.AddNode(new CustomSiteMapNodeInfo("Separator-1", String.Empty, SEPARATOR, false, true), CustomersNode);

CustomSiteMapNodeInfo OutcomeNode = SiteMapBuilderServiceHelper.CreateLocalizableNode("Outcome", "OutcomeList", true, true);
SiteMapBuilderService.AddNode(OutcomeNode , CustomersNode);

Let's try to create API that will remove all unnecessary information form the code, automatize localization stuff, and make those "true, true" arguments more descriptive.

My target would be code like this:

var builder = new SiteMapBuilder();
builder.Root(new DefaultUrl());
builder.Module("Customers")
  .Page(new CustomersUrl())
  .Page(new TransactionsUrl())
  .Page(new IncomeUrl())
  .Separator()
  .Page(new OutcomeUrl()).WithImage();
SiteMapBuilderService.AddNode(builder.GetRootNode());

Site map builder features:

• Create "Modules" - a top level menu items"
  
• Add "Pages" to modules - a sub-menu items
  
• Add "Separators" - fake menu items that will be rendered to horizontal line
  
• Set attributes to Page, like adding an image
  

First step is pretty simple, setting a root node will produce a node without child nodes.

[Test]
public void BuilderWithSetRootReturnsRootNode()
{
  var Builder = new SiteMapBuilder();
  var FooUrl = new FooUrl(String.Empty);
  Builder.Root(FooUrl);

  Assert.That(Builder.GetRootNode().Url, Is.SameAs(FooUrl));
  Assert.That(Builder.GetRootNode().ChildNodes, Is.Empty);
}

The code is very simple. Note that we have moved away a knowledge about SiteMapNode construction itself to separate factory, to keep builder responsibilities clean.

public class SiteMapBuilder
{
  private readonly SiteNodeFactory _SiteNodeFactory = new DefaultSiteNodeFactory();
 private IUrl _RootUrl;

  public SiteMapNode GetRootNode()
  {
    return _SiteNodeFactory.CreateModuleNode(_RootUrl);
  }

  public void Root(IUrl Url)
  {
    _RootUrl = Url;
  }
}

Next - module registration feature. API consumer can register several top menu items, that should produce root node with some child nodes.

The test:

[Test]
public void BuilderSeveralModulesInRoot()
{
  var Builder = new SiteMapBuilder();
  var FooUrl = new FooUrl(String.Empty);
  Builder.Root(FooUrl);

  Builder.Module(new FooUrl("a"));
  Builder.Module(new FooUrl("b"));

  Assert.That(Builder.GetRootNode().ChildNodes[0].Url, Is.EqualTo(new FooUrl("a")));
  Assert.That(Builder.GetRootNode().ChildNodes[1].Url, Is.EqualTo(new FooUrl("b")));
}

Implementetion: a list of module's urls are stored in builder and waiting to be transformed into nodes.

public class SiteMapBuilder
{
  private readonly List _Modules = new List();
  private readonly SiteNodeFactory _SiteNodeFactory = new DefaultSiteNodeFactory();
  private IUrl _RootUrl;

  public SiteMapNode GetRootNode()
  {
    var ChildNodes = _Modules.Select(Q => _SiteNodeFactory.CreateNode(Q)).ToArray();
  return _SiteNodeFactory.CreateModuleNode(ChildNodes ,_RootUrl);
  }

  public void Module(IUrl Url)
  {
    _Modules.Add(Url);
  }

  public void Root(IUrl Url)
  {
    _RootUrl = Url;
  }
}

Ok, it's good enough for this moment, you can set root url and add several top-level nodes to the site map. Only page support left, and here is "fluent" trick is. Let's add "public SiteMapBuilder Page(IUrl)" method, and return "this" reference from it. Consumer code will look like:

builder
  .Module("Customers")
  .Page(new CustomersUrl())
  .Page(new TransactionsUrl())
  .Page(new IncomeUrl())
  .Module("Second module")
  .Page(new CustomersUrl())
  .Page(new TransactionsUrl())
  .Page(new IncomeUrl());

Bad idea, auto-formatter will turn this code into flat one, without different indent, and builder class itself will be overloaded with various page construction parameters. Let's move it to ModuleBuilder class.

public class SiteMapBuilder
{
  private readonly List _Modules = new List();
  private readonly SiteNodeFactory _SiteNodeFactory = new DefaultSiteNodeFactory();
  private IUrl _RootUrl;

  public SiteMapNode GetRootNode()
  {
    var ChildNodes = _Modules.Select(Q => Q.GetNode()).ToArray();
    return _SiteNodeFactory.CreateModuleNode(ChildNodes ,_RootUrl);
  }

  public void Module(IUrl Url)
  {
    var ModuleBuilder = new ModuleBuilder(Url, _SiteNodeFactory);
    _Modules.Add(ModuleBuilder);
    return ModuleBuilder;
  }

  public void Root(IUrl Url)
  {
    _RootUrl = Url;
  }
}

public class ModuleMenuBuilder : IModuleMenuBuilder
{
  private readonly IUrl _Url;
  private readonly List _Pages = new List();

  private readonly SiteNodeFactory _SiteNodeFactory;

  public ModuleMenuBuilder(IUrl Url, SiteNodeFactory SiteNodeFactory)
  {
    _Url = Url;
    _SiteNodeFactory = SiteNodeFactory;
  }

  public SiteMapNode GetNode()
  {
    return _SiteNodeFactory.CreateModuleNode(_Pages.ToArray(), _Url);
  }

  public ModuleMenuBuilder Page(IUrl Url)
  {
    _Pages.Add(_SiteNodeFactory.CreatePageNode(Url))
    return this;
  }

  public ModuleMenuBuilder Separator()
  {
    _Pages.Add(_SiteNodeFactory.Separator());
    return this;
  }
}

It’s look like final version. Here is a usage example:

builder.Module("Customers")
  .Page(new CustomersUrl())
  .Page(new TransactionsUrl())
  .Page(new IncomeUrl());
builder.Module("Second module")
  .Page(new CustomersUrl())
  .Page(new TransactionsUrl())
  .Page(new IncomeUrl());

Robust ASP.NET navigation - Remove hardroce in query string.

Url's in application are fastly spreading across code, having different parameters and formats. This causes several problems during development and support:

• Urls mistyping
• Parameters mistyping
• No easy way to search for usage of particular page url on parameter in code.
• Hard to introduce any kind of url rewriting scheme.
• Violates DRY principle

In this article I'll describe solution that allow:

• Compile time checking of url and parameter mistyping
• Parameters type cheking
• Easy searching of url and parameter usage

The general idea - is to add for each page a "*Url" class that will contain page url and all possible arguments for the page.
Lets introduce url class for page "/Module/Example.aspx" with one mandotary parameter CustomerID (not nullable):

public ExampleUrl : IUrl
{
    [UrlParameter("CID")]
    public long CustomerID { get; set; }
    public string GetPageLocation()
    {
        return "~/Module/Example.aspx";
    }
}

From this point, all links to Example.aspx - should be composed using ExampleUrl instance, like this one:

(new ExampleUrl() {.CustomerID = 3}).MakeNavigableUrlForMe();

This will allow to search links by page or request parameters using R# - search for usages feature and to check all types of mistyping in compile time.

But to make all this stuff to work, some sort of Request assembler/disassembler is required. Lets name it UrlBuilder - a simple class whose responsibilities is to generate request from strongly typed url and populate url by parameters from request.

public class UrlBuilder
{
    public string GetUrl(IUrl PageUrl);
    public void FillInUrl(IUrl PageUrl, NameValueCollection ValueCollection);
}

That class can be easily developed using TDD manner, here are my tests:

[TestFixture]
public class UrlBuilderTest
{
    #region Public Methods

    [Test]
    public void EmptyUrlRendersToUnresolvedPageUrl()
    {
        var Builder = new UrlBuilder();
        var PageUrl = new EmptyUrl();

        var HttpUrl = Builder.GetUrl(PageUrl);

        Assert.That(HttpUrl, Is.EqualTo("~/Module/Example.aspx"));
    }

    [Test]
    public void UrlWithAttributedPropertyRendersToPageUrlWithGetParameter()
    {
        var Builder = new UrlBuilder();
        var PageUrl = new ExampleUrl();

        var HttpUrl = Builder.GetUrl(PageUrl);

        Assert.That(HttpUrl, Is.EqualTo("~/Module/Example.aspx?CID=0"));
    }

    [Test]
    public void UrlWithValuesRendersToPageUrlWithGetParameter()
    {
        var Builder = new UrlBuilder();
        var PageUrl = new AttributedUrl()
        {
            CustomerID = 123,
        };

        var HttpUrl = Builder.GetUrl(PageUrl);

        Assert.That(HttpUrl, Is.EqualTo("~/Module/Example.aspx?CID=123"));
    }

    [Test]
    public void NameValueCollectionMappedToMarkedAttributeInUrl()
    {
        var Builder = new UrlBuilder();
        var PageUrl = new AttributedUrl();
        var GetValues = new NameValueCollection();

        GetValues.Add("CID", "825");

        Builder.FillInUrl(PageUrl, GetValues);

        Assert.That(PageUrl.CustomerID, Is.EqualTo(825));
    }
    // Tests for many parameters, and different types are skipped.
}

Ok, Now IUrl is not responsible for "Making Navigable Url For Me", it's left for UrlBuilder.

And the last thing left - integrate PageUrl and UrlBuilder to WCSF. As you now, WCSF using MVP pattern. Presenter is a good integration point for both PageUrl and UrlBuilder. Let's parametrize BasePresenter generic class with IUrl and force BaseView to initialize parameters form request.

public class BasePresenterWithUrl<TView, TUrl> : BasePresenter<TView>
    where TView : class
    where TUrl : class, IUrl, new()
{
    public TUrl Url { get; set; }

    public override void InitUrlFromRequest(NameValueCollection Parameters)
    {
        Url = new TUrl();
        UrlBuilder.FillInUrl(Url, Parameters);
    }
}

public class BasePage<TPresenter, TView> : Page
    where TPresenter : BasePresenter<TView>
    where TView : class
{
    protected override void OnInit(EventArgs Args)
    {
        Presenter.InitUrlFromRequest(Request.Params);
        base.OnInit(Args);
    }
}

With this implementation, page presenter contains property Url, that is filled with values from request and ready to operate during regular presenter events like OnViewLoaded and OnViewInitialized.

Ok, ready to rumble, here is an example usage of "url pattern":

// Url setting for "Add item" link
public override void OnViewInitialized()
{
    View.SetUrlForAdd(UrlBuilder.GetUrl(new ExampleUrl(){ ClientID =  0 }));
}
// Example page presenter
public override void OnViewLoaded()
{
    if(Url.ClientID == 0)
    {
        View.SetMode(Mode.Add);
    } else {
        View.SetMode(Mode.Edit);
        View.SetCustomer(_CustomersGateway.ById(Url.ClientID);
    }
}