Strongly typed configuration library for .Net applications

The problem

During my past projects, I were always messing with hard-coded strings when doing application configuration.  

Became more mature in writing maintainable code, I have moved these hard-coded strings to separate constant class (ex. ConstAppConfig). This have helped a bit, but the code itself were filled with custom type conversions (from app.config strings to required types) and tightly coupled with .net configuration classes. 

Later, I have got an idea to create IApplicationConfig interface, which contains all configuration properties.  

The solution with configuration interface have several benefits, over constant class: 

• Interface contains properties with it's types 
• Interface make easy stubbing and mocking configuration values in test code  
• String to Type conversion are implemented in base classes, and not spread across the code  
• Makes easy to switch between config sources. like app.config, database, test source 
• Makes easy to do config validation at startup 
• Base classes provides unified way to threat non-existing or null-value scenarios
• Facilitates refactoring with automated tools 

Using the code 

Downloading library 

Up to date binaries and source code  is available at Codeplex and Nuget. 

• Codeplex 
• Nuget  

Configuring library   

Library setup and configuration is implemented via ConfigurationServiceBuilder class.

The class has RegisterConfigInterface(param IStringConfigSource[] src) methods, which allows you to register your configuration interface, and define configuration sources. 

Configuration source is a provider of configuration strings, like System.Configuration.ConfigurationManager class. You can use predefined AppConfig source or create your own implementation. 

Using library 

After configuration were finished and method Build were called, you will be provided with instance of ConfigurationService.
This instance holds reference to implementations of all interfaces, that you have registered at configuration phase. Implementations are accessible via  For() method. 

The For() method will give you an implementation of T interface, where all property accessors will do search trough provided configuration sources.

Example code  

Configuration file  

 Collapse

xml version="1.0" encoding="utf-8" ?>
<configuration>
  <appSettings>
    <add key="FooCount" value="98"/>
  </appSettings>
</configuration>  

Usage example

 Collapse

using NUnit.Framework;
using Typed.Configuration.ConfigSource;

namespace Typed.Configuration.Tests
{
    [TestFixture]
    public class ConfigServiceUsage
    {
        // Define interface with configuration properties
        public interface IConfigInterface
        {
            [FieldName("FooCount")]
            int? ItemsPerPage { get; }
            
            [FieldName("NotExistingField")]
            int? NotExistingField { get; }
        }

        [Test]
        public void FullUsageScenarioForStubSource()
        {
            var configurationServiceBuilder = new ConfigurationServiceBuilder();

            // Register configuration interface within service
            configurationServiceBuilder.RegisterConfigInterface(new AppSettingsSource(), new NullValueSource());

            // Create service instance and put it to your IoC container, or store in static vairable
            ConfigurationService configService = configurationServiceBuilder.Build();

            // Get configuration property value
            int? itemsPerPageCount = configService.For().ItemsPerPage;
            Assert.AreEqual(98, itemsPerPageCount);
            int? notExistingValue = configService.For().NotExistingField;
            Assert.IsNull(notExistingValue);
        }
    }
} 

Developer's notes on Mongo database

I like playing with new technologies and NoSQL and MongoDB is not an exception.
But, while working with Mongo, I have made few mistakes, that turned out to be painful to me.

1. Do not allow MongoDriver classes go beyond your DAL

Firstly, I neglected layered structure, planning my DAL for building fresh project on mongo. The original thought, was to create factory-class that will contain accessors for all mongo collections, of the application. Of course these collections were implemented with help of C# mongo driver (using MongoCollection class). That solution, soon stroked me in the back, since MongoCollection methods, provoke usage of hard-coded literals for complex queries.
After a while, a whole application were filled with a lot's of magic-strings with field names and mongo-specific commands.

Solution, that seems appropriate now, is usage of Repositories, for DDD's aggregation roots.
The idea is to stop "string plague" right at repository level. The rule of thumb - do not allow any Business Logic at repository, and do not allow any MongoCollection usages, on any layer, above than repository.

2. Plan yours select carefully

While working with SQL database, most of the time, complex and fast queries can be built, against any reasonable database structure.
So, typical workflow were to build lean domain model, persist it to database, in a way, your's ORM supports, and only than you can start thinking about querying data.
Only in rare cases data structure can be so in-efficient, that it will require refactoring.

Using mongo, you should always think upfront about querying data, and pre-compute a lot of things, before insertion of the document.
To make queries more efficient, we have redesigned storage structure of our data, already 3 times, in just 2 months!

3. None of yours assumptions SQL assumptions can be applied to mongo

If you'll try to bring any assumptions, about tables from SQL world to mongo, it will bite you back soon! Even most basic ones should be checked with mongo documentation.
A part of our application were build, with assumption, that no duplicated records can be returned from collection. After few "impossible" exceptions, I have realized, that sometimes records can be duplicated, if you are using cursor feature. It's not bad by itself, but thanks to my implication, that this is impossible, a lot of code should be re-tested and re-factored, to handle such cases.

4. Schema-less does't mean, that you should't care about it.

My first impression, were "Woo-hoo! No more boring alter-table scripts, just add a field, that's it". But actually, whole responsibility of management database versions is moved to your's build script or DAL code.

.Net IoC performance comparison

Finally, I have compared performance of most widely-used IoC containers.

I'am clearly understand, that for IoC performance, in most cases, is not an issue, but I were still curious enough to test.

Here are the results:

Name	Total	PerRequest
Structure Map	377ms	0.377ms
Autofac	287ms	0.287ms
Unity	351ms	0.351ms
Ninject	2638ms	2.638ms
Manual	1ms	0.001ms

Test source is available via GitHub

Assumptions, behind the test, is optimized for Web scenario, where several threads request classes in different scopes.

Declarative UI interactions using Reactive Extensions for Javascript

We are creating a page, with a lot of controls, that are talk to each other. For example "When DD1 (DropDown 1) changes, change T1 (TextBox 1), T2, reload U1 (UpdatePanel 1), but only when DD1 and DD2 has changed, update T3).
Using imperative style, your's attempt will defiantly will end with a lot of nested callbacks, and it will be very problematic to "read" original interaction scheme, from the code.

Rx will simplify a bit this problem. It will allow you to describe interactions (and data flow) between these controls, in declarative style.

Let's try to create a page, that contains 2 input text boxes, and will compute the result and write it, to the third text box. The result, will appear, after 1 second, to simulate Ajax call. If we have made many changes, than only last result will be shown to us, instead of displaying all intermediate results. And no events will be triggered, until we sat all initial values, to the controls.

Here is resulting marble diagram (red bars – are ajax calls):

 

So, the diagram shows, that T3 is calculated as T1 + T2. Calculation is made on the server, using ajax calls, and we need to recalculate T3, each time T1 or T2 changes. If Text changes, while calculation is in process (red bar), then it's result should be ignored, and new call made, to recalculate T3 with most recent data. And last thing, during initialization, no events will be fired, but when we "turn on the switch", each text box will fire the event, with it's value.

Implementation

Sample page markup:

JavaScript:

Notes:

@2 – Latch is a sort of a global "power on" switch, when we call Latch.OnComplete(), messages will start cycling around
@3 – AttachToChangeAndSetValue – it's functions that returns observable on control's change event, also, it does some "plumbering", I'll describe it later
@18, @20 – Two observables, Text1Changed and Text2Changed on marble diagram
@21 – Rx CombileLatest expression, that actually do that "value management" that is shown with arrows on a diagram
@27-28 – Actually doing calculations, and simulating Ajax call with duration of 1000ms, returns Observable, with fires, as soon, as value is ready (Ajax call returned, or 1000ms passed)
@32 – Ensures that only latest calculation result will be displayed, all earlier results will be ignored
@43-44 – Starts Computations, by releasing latch

Latch implementation

Function AttachToChangeAndSetValue creates observable from html events, but before returning this observable, it is preceded by Latch observable (@11-15 return Latch.Concat(source)). Semantic of Concat function, says that all events from source will be ignored, until Latch source is depleted. Now, we can freely change control's state (in initialization phase), without any side effects. But, as soon, as we call Latch.OnCompleted(); (@43), Concat will allow to pass change events from source.

At line @42, Latch is instructed to fire dummy event, value from this event is overwritten with control's default value (@11-14), this is for setting default values of control, and to set system to "on - tw" state (on the marble diagram).

Throwing our obsolete ajax responses

Lines @25-30 Maps update requests, to the source of sources of ajax responses. Don't be scared, it's just a list of events, and each of them, will fire when ajax response will return.
For simplification, we don't use ajax, but timer instead. Rx.Observable.Timer(1000) returns an observable, that will fire once (after specified interval). These items are represented as a red bars, on the marble diagram.
Switch function, at line @32, takes this source of sources of ajax responses, and wait until most recent event will fire, this will guarantee, that we will display data from latest recalculation.

Verify stored procedures parameters and nice side-effect of abstractions

Few week ago, I have blogged about wrapper around SPROC calling routine. At that moment I thought, about this, only like a sort of a syntax-sugar around “low level” data access classes.

Yesterday, I have faced a requirement, to verify and revalidate all my stored procedures signatures over new database. The task tended to be mundane and useless, but thanks to introduced abstraction (that builder), we can create a test, that will “fake” a stored procedure builder with a recording-mock, record all procedure calls and it’s parameters and verify it’s signatures against target database.

Here, how it works:

1. Tests replace CommandBuilderFactory, with Factory, that produces recording builders.
2. Test executes All possible methods of each Data Gateway, to record called procedures and parameters.
3. Each procedure is verified, against it’s SQL declaration.

Recording builder, in my case, allows a gateway to fill-in all required parameters. When builder is requested to do “build-up”, to produce an executor in my case, it just throws special exception. This type exception is excepted by caller.

Great, now we have a list of procedures, it’s parameters and parameter types. Next task is to check parameters of each procedure against procedure in SQL. ADO.net has method “SqlCommandBuilder.DeriveParameters”, which populates a SPROC call with it’s parameters.

Viola, the test is ready, and can be executed on continuous basis, to make shore, that nobody will commit wrong SPROC declaration anymore.

So, the lesson that I got, is, that abstraction over BCL classes has a hi chance to pay off to you :) 

Reactive extensions – WPF fold/unfold

I have just implemented WPF fold/unfold functionality using Reactive Extensions Framework (RX). Desired algorithm is to "unfold" wpf window, as soon, as mouse entered, and "fold" after 5 seconds of mouse leaving.

First, naive implementation:

This implementation contains a bug, when you move mouse in/out several times in 5sec interval, window can left collapsed, because old "collapse" events will arrive, after delay.

Here is RX implementation, that discards "fold" messages, as soon, as new "fold" messages has arrived:

Reactive extensions for javascript.

Microsoft has released Rx extensions for Javascript. It's a great tool for organizing events processing in yours UI code.

I have used it to handle following task: Call Ajax-Update every 5 seconds, after filter button clicked and on paging request, but only when previous request is finished. Tricky? Good luck debugging it later.

This can be implemented using Rx extensions. In that case all that interactions became explicit and clear, placed on few lines of code.

Here is an example:

Speed-up database insert operation using Reactive extensions

Application inserts a lot of data to mongo databse. Insertion of small records one-by-one takes a lot of time. So, obvious solution to buffer this records. Buffer feature of Rx extensions fits well for such buffering.

Code notes:

• Line 17-25 – Aggregates messages for 100ms period into an array and pass to saving function
• Line 56 – Pushes measures to Rx buffer.

This buffering gave me roughly 30% performance gain.

The code itself:

public class MeasureGateway : IMeasureGateway, IDisposable
{
	private readonly AddObserver _addObserver = new AddObserver();
	private static readonly TimeSpan BufferTimeout = TimeSpan.FromMilliseconds(100);
	private DateTime _lastMeasureAddedAt = DateTime.MinValue;
	private readonly IDisposable _bufferSubscription;

	[InjectionConstructor]
	public MeasureGateway() : this(true)
	{
		
	}
	public MeasureGateway(bool useBuffering)
	{
		if (useBuffering)
		{
			_bufferSubscription =
				_addObserver
					.Buffer(BufferTimeout).Subscribe((q) =>
														 {
															 if (q.Any())
															 {
																 AddMeasureInner(q);
															 }
														 });
		}
	}

	private void AddMeasureInner(Document q)
	{
		using (MongoScope mongo = MongoScope())
		{
			mongo.DefaultCollection.Insert(q);
		}
	}
	private void AddMeasureInner(IEnumerable<Document> q)
	{
		using (MongoScope mongo = MongoScope())
		{
			mongo.DefaultCollection.Insert(q);
		}
	}

	public void Add(DeviceMeasure measure)
	{
		Document measureDoc = GetMeasureDoc(measure);

		if (_bufferSubscription != null)
		{
			if (DateTime.Now.Subtract(_lastMeasureAddedAt) > BufferTimeout)
			{
				AddMeasureInner(measureDoc);
			}
			else
			{
				_addObserver.Add(measureDoc);
			}

			_lastMeasureAddedAt = DateTime.Now;
		}
		else
		{
			AddMeasureInner(measureDoc);
		}
	}

	private class AddObserver : IObservable<Document>, IDisposable
	{
		private IObserver<Document> _observer;

		#region IDisposable Members

		public void Dispose()
		{
		}

		#endregion

		#region IObservable<Document> Members

		public IDisposable Subscribe(IObserver<Document> observer)
		{
			_observer = observer;
			return this;
		}

		#endregion

		public void Add(Document doc)
		{
			_observer.OnNext(doc);
		}
	}

	public void Dispose()
	{
		if (_bufferSubscription != null)
		{
			_bufferSubscription.Dispose();
		}
	}

Fluent interface for hierarchical structures

Fluent interface works well in combination with builder pattern for representation of flat structures. But API gets a bit messy when you need represent some sort of hierarchical structure.

Currently I have “VeryVeryComplex” legacy entity and I need to represent it’s traversal algorithm in a friendly way. That looks easy when there is no arrays in the entity but as soon as it appears the code gets messy. The gist is to couple with hierarchy with a sort of collecting parameter pattern.

TouchEach method has 2 parameters: the collection to enumerate and “touch” delegate, that will be applied to each item in the collection. The delegate itself has also 2 arguments: t - toucher instance, that acts as a collecting parameter and i – collection element.

The consumer code would look like this:

var t = new Toucher();
var c = new VeryVeryComplexEntity();
t.Touch(c)
 .Touch(c.Child1)
 .Touch(c.Child2)
 .TouchEach(c.SubCollection1
   (t, i) => {
      t.Touch(i.SubSubChild1)
      .Touch(i.SubSubChild2);
      // The same for Subcollection of i - item
   });

Naive toucher implementation:

class Toucher
{
	List<object> _items = new List<object>();
	public Toucher Touch(object obj)
	{
		_items.Add(obj);
		return this;
	}
	public Toucher TouchEach<T>(IEnumerable<T> collection, Action<Toucher, T> subTouch)
	{
		foreach(var i in collection)
		{
			subTouch(i, this);
		}
		return this;
	}
}

Happy coding :).

Handle incoming protobuf messages with IIS Server.

The task is to receive protobuf messages that is sent from client via IIS Server. Of course, you can use native protobuf server to handle incoming request. In my case rest of the application is written using IIS, so I don’t want to cope with additional server process.

Client will send HTTP POST request with content that is protobuf encoded message. IHttpHandler on the other side will be listening for this message on the other side.

To do this, you need to modify web.config:

<configuration>
  <httpHandlers>
   <add verb="*" path="Device.proto.aspx" validate="false" type="Server.ProtoHttpHandler, Server" />
  </httpHandlers>
 <!-- 
        The system.webServer section is required for running ASP.NET AJAX under Internet
        Information Services 7.0.  It is not necessary for previous version of IIS.
    -->
 <system.webServer>
  <handlers>
            <add name="ProtoHttpHandler" path="Device.proto" verb="*" type="Server.PrototHttpHandler, Server" resourceType="Unspecified" preCondition="integratedMode" />
  </handlers>
 </system.webServer>
</configuration>

This will invoke ProtoHttpHandler that can handle incoming message.

public class ProtoHttpHandler : IHttpHandler
{
 private UnityContainer _Container;

 public ProtoHttpHandler()
 {
  _Container = new UnityContainer();
 }

 public void ProcessRequest(HttpContext context)
 {
  var handler = _Container.Resolve<RequestHandler>();
  var decoder = _Container.Resolve<ProtobufEncoder>();
  var messages = decoder.Decode(new BinaryReader(context.Request.InputStream).ReadBytes(context.Request.ContentLength));
  
  foreach (var deviceMeasureRequest in messages)
  {
   handler.OnMeasreRequest(deviceMeasureRequest);
  }
  context.Response.End();

 }

 public bool IsReusable
 {
  get { return true; }
 }
}

How to handle ASP.NET Update Panel in WatiN

Handling asp.net update panels with WatiN is a pretty annoying thing.
Click and WaitForLoad usually throws timeout every time.

I have found "just fine" solution that work for me:

static class ElementExtension
    {
        public static void ActWithToken(this Element Element, Action Action)
        {
            string Waittoken = "waittoken";
            Element.SetAttributeValue(Waittoken, Waittoken);
            Action();
            Element.WaitUntil(!Find.By(Waittoken, Waittoken));
        }
        
        public static void ActWithToken<E>(this E Element, Action<E> Action)
            where E : Element 
        {
            string Waittoken = "waittoken";
            Element.SetAttributeValue(Waittoken, Waittoken);
            Action(Element);
            Element.WaitUntil(!Find.By(Waittoken, Waittoken));
        }
    }

Usage

Page.CreditToLocation.FindItem(1).ActWithToken(e => e.ClickNoWait());

Call stored procedure with output parameter using Builder pattern

My custom DAL requirements is:

1. By given Business objects (BOs) call stored procedures
2. Handle output parameters (updating BOs fields)
3. Throw exceptions, if error code returned from stored procedure

Coding this functionality using ADO.NET is a bit annoying. The majority of code consists of communication with framework interface, and does not explicitly express developer's intensions. Example.
Such annoying code can be refactored nicely using Builder pattern. Target interface for me is interface like this:

public void Update(ObjectContext Ctx, BusinessObject Row)
{
 CommandBuilder Builder = InsertOrUpdateBuilder(Ctx, "updateBusinessObject", Row);
 Builder.ErrorCodes
  .Parameter("err_code_out")
  .Handle<ConcurrencyException>(-1);

 Builder
  .Parameter("id", Row.Id)
  .Parameter("last_update_date", Row.LastUpdateDate)
  .Parameter("last_updated_by", Row.LastUpdatedBy)
  .Output<DateTime>("last_update_date_out", (d) => Row.LastUpdateDate = d)
  .Create()
  .ExecuteUpdate();
}

public void Insert(ObjectContext Ctx, BusinessObject Row)
{
 InsertOrUpdateBuilder(Ctx, "insertBusinessObject", Row)
  .Parameter("created_by", Row.CreatedBy)
  .Output<int>("id_out", (Id) => Row.Id = Id)
  .Create()
  .ExecuteUpdate();
}

This interface is versatile enough to express required stored procedures call scenarios and explicit enough to be readable by human.

Literally Update call do this:

1. If Stored procedure returns –1 from parameter "err_code_out", Concurrency exception will be thrown.
2. Parameters "id", "last_update_date" and "last_updated_by" will be added to stored procedures
3. Parameter "last_update_date_out" will be threaded as output parameter of type DateTime and mapped to Row.LastUpdateDate.

Implementation

The idea of Builder pattern, is to hide complex construction logic behind clear and explicit API.
Builder responsibility is to collect consumer "wishes" and return something that consumer can execute, to make "wishes" come true.

Here is builder implementation:

public class CommandBuilder
{
 private readonly DbCommand _Command;
 private readonly OutputParameterMapper _ParameterMapper = new OutputParameterMapper();
 private readonly ErrorCodeHandler _ErrorCodeHandler;

 public CommandBuilder(ObjectContext Ctx, string Name)
 {
  _ErrorCodeHandler = new ErrorCodeHandler(Name);
  _Command = Ctx.CreateStoreCommand(
   Name, 
   CommandType.StoredProcedure);
 }

 public IErrorCodeHandlerBuilder ErrorCodes
 {
  get
  {
   return _ErrorCodeHandler;
  }
 }

 public CommandBuilder Parameter(string Name, object Value)
 {
  if (Value == null)
  {
   Value = DBNull.Value;
  }

  _Command.Parameters.Add(new SqlParameter(Name, Value));
  return this;
 }

 public CommandBuilder AddPagingParameters(PageSortInfo PageSortInfo)
 {
  _Command.AddPagingParameters(PageSortInfo);
  return this;
 }

 public CommandExecutor Create()
 {
  if (_ErrorCodeHandler.ParameterName != null)
  {
   Output>int<(_ErrorCodeHandler.ParameterName, (ErrorCode) => { _ErrorCodeHandler.ThrowOnError(ErrorCode); });
  }

  return new CommandExecutor(
   _Command, 
   _ParameterMapper);
 }

 public CommandBuilder Output>T<(string ParameterName, Action>T< Func)
 {
  var Parameter = new SqlParameter(ParameterName, default(T));
  Parameter.Direction = ParameterDirection.Output;

  _Command.Parameters.Add(Parameter);
  _ParameterMapper.AddMap(ParameterName, Func);
  return this;
 }
}

After class consumer calls create on Builder, it returns Command executor. Command executor is already configured to execute desired stored procedures with parameters and map output parameters after execution.

I have cheated a bit with ErrorCodeHandler, it is a builder and an executor in the same time (say hello to SRP), but it doesn't causes much damage for now. The execution interface is hidden by IErrorCodeHandlerBuilder.

public class ErrorCodeHandler : IErrorCodeHandlerBuilder
{
 private readonly IDictionary>int,type< _ErrorsDictionary = new Dictionary>int,type<();

 private readonly string _ProcedureName;
 private string _ParameterName;

 public string ParameterName
 {
  get
  {
   return _ParameterName;
  }
 }

 public ErrorCodeHandler(string ProcedureName)
 {
  _ProcedureName = ProcedureName;
 }

 public IErrorCodeHandlerBuilder Handle(int ErrorCode, Type Exception)
 {
  _ErrorsDictionary.Add(ErrorCode, Exception);
  return this;
 }

 public IErrorCodeHandlerBuilder Handle<T>(int ErrorCode)
 {
  _ErrorsDictionary.Add(ErrorCode, typeof(T));
  return this;
 }

 public IErrorCodeHandlerBuilder Parameter(string Name)
 {
  _ParameterName = Name;
  return this;
 }

 public void ThrowOnError(int ErrorCode)
 {
  Type ExceptionType;
  if (_ErrorsDictionary.TryGetValue(ErrorCode, out ExceptionType))
  {
   var Exception = (Exception)Activator.CreateInstance(
    ExceptionType,
    String.Format("Stored procedure {0} returned error code {1}", _ProcedureName, ErrorCode));
   throw Exception;
  }
 }
}

The output parameter mapper is a bit tricky. It introduces 2 classes Map<T> and Map to bypass C# generic typing systems, and make polymorphism to work.

public class OutputParameterMapper
{
 private readonly List<IMap> _Map = new List<IMap>();
 private interface IMap
 {
  string ParameterName { get; set; }
  void InvokeSetter(object Value);
 }
 public void AddMap<T>(string ParameterName, Action<T> Setter)
 {
  _Map.Add(new Map<T>()
  {
   ParameterName = ParameterName, 
   Setter = Setter
  });
 }

 public void MapParameters(DbParameterCollection Parameters)
 {
  foreach (var Map in _Map)
  {
   var Value = Parameters[Map.ParameterName].Value;
   Map.InvokeSetter(Value);
  }
 }
 private class Map<T> : IMap
 {
  public Action<T> Setter { get; set; }
  public string ParameterName { get; set; }
  public void InvokeSetter(object Value)
  {
   Setter((T)Value);
  }
 }
}

Command executor class, when called blows all that stuff (triggers exception throwing and parameters mapping)

public class CommandExecutor
{
 private readonly DbCommand _Command;
 private readonly OutputParameterMapper _Mapper;
 private readonly ErrorCodeHandler _ErrorHandler;

 public CommandExecutor(DbCommand Command, OutputParameterMapper Mapper)
 {
  _Command = Command;
  _Mapper = Mapper;
 }

 public DbCommand Command
 {
  get
  {
   return _Command;
  }
 }

 public void ExecuteUpdate()
 {
  using (_Command.Connection.CreateConnectionScope())
  {
   _Command.ExecuteNonQuery();
   _Mapper.MapParameters(_Command.Parameters);
  }
 }

 public T ExecuteScalar<T>()
 {
  using (_Command.Connection.CreateConnectionScope())
  {
   var Scalar = (T)_Command.ExecuteScalar();
   _Mapper.MapParameters(_Command.Parameters);
   return Scalar;
  }
 }
}

The idea of builder pattern was given to me by my colleague, thanks. :)

Things, that I got during writing .Net connector for Redis.

Redis is a distributed key-value storage. Connector is a protocol-driver for storing/retrieving data from Redis.  Recently I have developed one. I taught following simple lessons from it:

SRP principle makes code better prepared for changes

During development of the connector, I were stick to "SRP" and "TDD" principle.  Later i have realized that SRP is really helps, if you face requirement of unexpected code changes.

In short SRP says that you should have only one reason to change the class.

.NET code is as fast as native implementation

Spuriously, native ANSI-C implementation of Redis benchmark is as fast, as .NET implementation of the connector with same benchmark, moreover with all those patterns and enterprise stuff.

.NET buffered stream is a good abstraction

Connector is written in a way – that avoids copying arguments (byte arrays) in one big array, for sending to Redis. Instead it saves references to all arguments and writes in in a sequence. That causes serious performance problems. I have avoided this, just by wrapping socket's write stream with buffered stream. It's gave me 2x performance 'boost' (2000 requests per second vs approx. 5000).

comand pipelininG is a good IDEA

Command pipelining, it's a feature, when you put all you requests trough single socket. For example, is you send (in a row) 3 GET's for "foo", "bar" and "baz" keys – you will receive commands results in the same order. It gave me good results over the 100MBit network:

512k Localhost
.....Normal :131.095123900879
.....Pipelined 1 conn :59.8802395209581
.....Pipelined 2 conn :79.8403193612775
.....Pipelined 5 conn :99.7804829375374
.....Pipelined 10 conn :120
.....Pipelined 50 conn :137.793707686181
512k Remote host
.....Normal :37.9241516966068
.....Pipelined 1 conn :40.7918416316737
.....Pipelined 2 conn :42.4084816963393
.....Pipelined 5 conn :42.3238171291675
.....Pipelined 10 conn :40.4080816163233
.....Pipelined 50 conn :39.8962696987832
1kb localhost
.....Normal :7330.13589128697
.....Pipelined 1 conn :6500.89802434644
.....Pipelined 2 conn :7438.32335329341
.....Pipelined 5 conn :8085.56577369052
.....Pipelined 10 conn :7821.23575284943
.....Pipelined 50 conn :6627.87442511498
1kb Remote host
.....Normal :500
.....Pipelined 1 conn :6543.7125748503
.....Pipelined 2 conn :9555.51110222044
.....Pipelined 5 conn :11052.7788884446
.....Pipelined 10 conn :10608
.....Pipelined 50 conn :520

multithreaded code is VERY hard to debug

During implementation of pipelining I faced with a racing problem. Old pipelining algorithm allows  racing during parsing response from Redis. Microsoft Chess helped me a lot to test and localize this problem. It's a special tool, that runs test code with worst-case thread switching scenario.

Free profilers exists (NProf and Slimtune)

Earlier, I had no idea how to profile application at low cost (JetBrains and Ant's Profilers are expensive). Recently I found NProf and SlimTune. They helped me a lot to find bottlenecks.

When you have a hammer, everything looks like a nail.

Once, I have discussed with my colleague architecture candidate of high-performance system. In brief, system should receive a big amount of small incoming requests and provide API for querying on this data. 

The numbers:

• number of incoming request  - 5000 per second.
• number of select - 5 per 10 seconds
• maximum latency time between registering request and it's availability trough API is 5 sec.

My colleague is a DB guy who is working with creating big and rather complex automation solutions. He proposed to take the most enterpricy  (and expensive) database, do all of the processing in db and use Table-Views as external API to the system.

This solutions introduce a lot of problems.

1. Vendor affinity

Once, you exposed a part of your db as external API, you will never change it (nor db vendor, nor db structure).

2. Performance optimization

You have absolutely no control on queries, that will run against your system. You can buy several top-end servers and still some idiot will invent a query that will bog them down. Stored-procedures is a solution, but it will refuse an argument, that exposing views will allow us not to worry about queries in external API.

3. Scalability

It's a fact that SQL server isn't scaling well, horizontally at least.  The system is not so complex, and consistency checks, joins and complex select queries is nor required here.

Better solution

System can be divided in 3 logical parts: Request Receiver, Request Store and External API.

Receiver and external API are pretty simple: protobuf and web-services. Protobuf is fast enough for thin channels of each sending-client and web-service is pretty simple for usage for API consumers.

But the most uncertainty is left to Request Store. As it's organization will be changing during load tests and API modification, it should be hidden under very strict and simple interfaces. It's handy, that system doesn't have complex objects speeded across many tables. We can use POCOs everywhere.

And the storage itself can be implemented using traditional SQL databases or distributed storages. Thanks' to interfaces, it will not affect remaining to parts.

Generalize everything pattern

Sometimes I 'am facing "generalize everything" pattern, when developer trying to invent a "racing car" for a trip for a few kilometers (or miles :) ). But as soon car is ready, it happens that car can be driven only on straight flat road, with no turns with constant speed only on 5th gear. Later, when you destinations changing you apply more and more hand-made solutions and hacks. Finally, after several month passed, car got a bit rusty and needs support, but is takes a lot of time (and money) even for clever mechanic to solve the problem. It's because car is full of "witty" and "smart" puzzles. Just take a walk, it's healthy.
A real life example: Customer required us to implement "read-only" feature for product, that we developing. After some discussion 2 possible solutions were found:

1. Automate all processing of permissions. Base Presenter will automatically determine if user is read-only, call a special method in a Base View that is responsible for hiding all buttons, that is tagged with special marked (Save buttons).
2. Move it's responsibility to concrete presenter and concrete view. View will have method, that hides edit buttons where buttons are defined explicitly. Presenter with call explicitly View.HideEditButtons() method if required.

Let's analyze pros and cons of each solution:

Maintainability

Using first approach - firstly, it seems that developers will produce less bugs during implementation of the each page. At least they need only to place properly annotated buttons on the page.  But when customer asks for a simple feature, like "please enable this button for read-only". Developer implements workaround on one page, on second, third and you have a bunch of pages with a different implementation of same feature. And finally when tester reports a bug, you need to analyze a markup (with annotation, do you remember), view implementation for hacks, base view implementation for hiding mechanism itself and presenter for an additional hacks.
In second approach, developer will modify a View.HideEditButtons method (remove or add hiding of buttons), or introduce new permission rule (if it's required). It's pretty easy now, because implementation is not coupled with "Base*" stuff and will not broke anything else in the system. Support team will need to analyze only 2 files – view and presenter, and all control hiding/showing are declared explicitly there.

Lines of code

It's nearly equal. On the left side there is a new-born framework and customizations for it. On the right side a lot of boilerplate permission checks and show/hide control code divided into methods.

Code complexity

General solution is full of polymorphism, default and implicit behavior, if branches, typeof-s. It's easy to forget a small thing, and here it is – a bug. Second solution is a set of flat methods with a rare branches like If(!Allowed){View.HideSomething();} that is easy to read even if you unfamiliar with a context.

Code duplication

Using 2nd approach, it's seems that you have written View/Hide methods and permissions check zillions of times. But that methods are completely different methods with the same pattern. Method is simple and contains only vital information that is differs from page to page. In 1st approach during simple tasks there is no code/pattern duplication at all, but later developers will invent their own hacks, that will do similar tasks in a different ways.

Robust ASP.NET navigation - Fluent interface for site map generation

ASP.Net provides classes that is suitable for holding current sitemap structure, displaying it, using default controls, searching for current selected node, but not for building it in a elegant way, at least at WCSF. The code in MSDN article is pretty simple

SiteMapNodeInfo moduleNode = new SiteMapNodeInfo("Customers", "~/Customers/ApproveCustomerView.aspx", "Approve Customer");
siteMapBuilderService.AddNode(moduleNode);
siteMapBuilderService.AddNode(moduleNode, parentNode);
siteMapBuilderService.AddNode(moduleNode, 100);
siteMapBuilderService.AddNode(moduleNode, “AllowViewModule1”);

But when you have numerous of pages, sitemap building code can turn into crap like this:

CustomSiteMapNodeInfo HomeNode = new CustomSiteMapNodeInfo("Home", "~/Default.aspx", "Home", "Home", true);
SiteMapBuilderService.AddNode(HomeNode);

SiteMapBuilderService.RootNode.Url = "~/Default.aspx";
SiteMapBuilderService.RootNode.Title = "FOO-BAR";

CustomSiteMapNodeInfo CustomersNode = new CustomSiteMapNodeInfo("Customers", "#Customers_List", "Customers", true);
SiteMapBuilderService.AddNode(CustomersNode);

SiteMapBuilderService.AddNode(SiteMapBuilderServiceHelper.CreateLocalizableNode("Customers", "CustomersList", true, true), SalesNode);
CustomSiteMapNodeInfo TransactionsNode = SiteMapBuilderServiceHelper.CreateLocalizableNode("Transactions", "TransactionsList", true, true);
SiteMapBuilderService.AddNode(TransactionsNode, CustomersNode);

CustomSiteMapNodeInfo IncomeNode = SiteMapBuilderServiceHelper.CreateLocalizableNode("Income", "IncomeList", true, true);
SiteMapBuilderService.AddNode(IncomeNode , CustomersNode);

SiteMapBuilderService.AddNode(new CustomSiteMapNodeInfo("Separator-1", String.Empty, SEPARATOR, false, true), CustomersNode);

CustomSiteMapNodeInfo OutcomeNode = SiteMapBuilderServiceHelper.CreateLocalizableNode("Outcome", "OutcomeList", true, true);
SiteMapBuilderService.AddNode(OutcomeNode , CustomersNode);

Let's try to create API that will remove all unnecessary information form the code, automatize localization stuff, and make those "true, true" arguments more descriptive.

My target would be code like this:

var builder = new SiteMapBuilder();
builder.Root(new DefaultUrl());
builder.Module("Customers")
  .Page(new CustomersUrl())
  .Page(new TransactionsUrl())
  .Page(new IncomeUrl())
  .Separator()
  .Page(new OutcomeUrl()).WithImage();
SiteMapBuilderService.AddNode(builder.GetRootNode());

Site map builder features:

• Create "Modules" - a top level menu items"
  
• Add "Pages" to modules - a sub-menu items
  
• Add "Separators" - fake menu items that will be rendered to horizontal line
  
• Set attributes to Page, like adding an image
  

First step is pretty simple, setting a root node will produce a node without child nodes.

[Test]
public void BuilderWithSetRootReturnsRootNode()
{
  var Builder = new SiteMapBuilder();
  var FooUrl = new FooUrl(String.Empty);
  Builder.Root(FooUrl);

  Assert.That(Builder.GetRootNode().Url, Is.SameAs(FooUrl));
  Assert.That(Builder.GetRootNode().ChildNodes, Is.Empty);
}

The code is very simple. Note that we have moved away a knowledge about SiteMapNode construction itself to separate factory, to keep builder responsibilities clean.

public class SiteMapBuilder
{
  private readonly SiteNodeFactory _SiteNodeFactory = new DefaultSiteNodeFactory();
 private IUrl _RootUrl;

  public SiteMapNode GetRootNode()
  {
    return _SiteNodeFactory.CreateModuleNode(_RootUrl);
  }

  public void Root(IUrl Url)
  {
    _RootUrl = Url;
  }
}

Next - module registration feature. API consumer can register several top menu items, that should produce root node with some child nodes.

The test:

[Test]
public void BuilderSeveralModulesInRoot()
{
  var Builder = new SiteMapBuilder();
  var FooUrl = new FooUrl(String.Empty);
  Builder.Root(FooUrl);

  Builder.Module(new FooUrl("a"));
  Builder.Module(new FooUrl("b"));

  Assert.That(Builder.GetRootNode().ChildNodes[0].Url, Is.EqualTo(new FooUrl("a")));
  Assert.That(Builder.GetRootNode().ChildNodes[1].Url, Is.EqualTo(new FooUrl("b")));
}

Implementetion: a list of module's urls are stored in builder and waiting to be transformed into nodes.

public class SiteMapBuilder
{
  private readonly List _Modules = new List();
  private readonly SiteNodeFactory _SiteNodeFactory = new DefaultSiteNodeFactory();
  private IUrl _RootUrl;

  public SiteMapNode GetRootNode()
  {
    var ChildNodes = _Modules.Select(Q => _SiteNodeFactory.CreateNode(Q)).ToArray();
  return _SiteNodeFactory.CreateModuleNode(ChildNodes ,_RootUrl);
  }

  public void Module(IUrl Url)
  {
    _Modules.Add(Url);
  }

  public void Root(IUrl Url)
  {
    _RootUrl = Url;
  }
}

Ok, it's good enough for this moment, you can set root url and add several top-level nodes to the site map. Only page support left, and here is "fluent" trick is. Let's add "public SiteMapBuilder Page(IUrl)" method, and return "this" reference from it. Consumer code will look like:

builder
  .Module("Customers")
  .Page(new CustomersUrl())
  .Page(new TransactionsUrl())
  .Page(new IncomeUrl())
  .Module("Second module")
  .Page(new CustomersUrl())
  .Page(new TransactionsUrl())
  .Page(new IncomeUrl());

Bad idea, auto-formatter will turn this code into flat one, without different indent, and builder class itself will be overloaded with various page construction parameters. Let's move it to ModuleBuilder class.

public class SiteMapBuilder
{
  private readonly List _Modules = new List();
  private readonly SiteNodeFactory _SiteNodeFactory = new DefaultSiteNodeFactory();
  private IUrl _RootUrl;

  public SiteMapNode GetRootNode()
  {
    var ChildNodes = _Modules.Select(Q => Q.GetNode()).ToArray();
    return _SiteNodeFactory.CreateModuleNode(ChildNodes ,_RootUrl);
  }

  public void Module(IUrl Url)
  {
    var ModuleBuilder = new ModuleBuilder(Url, _SiteNodeFactory);
    _Modules.Add(ModuleBuilder);
    return ModuleBuilder;
  }

  public void Root(IUrl Url)
  {
    _RootUrl = Url;
  }
}

public class ModuleMenuBuilder : IModuleMenuBuilder
{
  private readonly IUrl _Url;
  private readonly List _Pages = new List();

  private readonly SiteNodeFactory _SiteNodeFactory;

  public ModuleMenuBuilder(IUrl Url, SiteNodeFactory SiteNodeFactory)
  {
    _Url = Url;
    _SiteNodeFactory = SiteNodeFactory;
  }

  public SiteMapNode GetNode()
  {
    return _SiteNodeFactory.CreateModuleNode(_Pages.ToArray(), _Url);
  }

  public ModuleMenuBuilder Page(IUrl Url)
  {
    _Pages.Add(_SiteNodeFactory.CreatePageNode(Url))
    return this;
  }

  public ModuleMenuBuilder Separator()
  {
    _Pages.Add(_SiteNodeFactory.Separator());
    return this;
  }
}

It’s look like final version. Here is a usage example:

builder.Module("Customers")
  .Page(new CustomersUrl())
  .Page(new TransactionsUrl())
  .Page(new IncomeUrl());
builder.Module("Second module")
  .Page(new CustomersUrl())
  .Page(new TransactionsUrl())
  .Page(new IncomeUrl());

Robust ASP.NET navigation - Remove hardroce in query string.

Url's in application are fastly spreading across code, having different parameters and formats. This causes several problems during development and support:

• Urls mistyping
• Parameters mistyping
• No easy way to search for usage of particular page url on parameter in code.
• Hard to introduce any kind of url rewriting scheme.
• Violates DRY principle

In this article I'll describe solution that allow:

• Compile time checking of url and parameter mistyping
• Parameters type cheking
• Easy searching of url and parameter usage

The general idea - is to add for each page a "*Url" class that will contain page url and all possible arguments for the page.
Lets introduce url class for page "/Module/Example.aspx" with one mandotary parameter CustomerID (not nullable):

public ExampleUrl : IUrl
{
    [UrlParameter("CID")]
    public long CustomerID { get; set; }
    public string GetPageLocation()
    {
        return "~/Module/Example.aspx";
    }
}

From this point, all links to Example.aspx - should be composed using ExampleUrl instance, like this one:

(new ExampleUrl() {.CustomerID = 3}).MakeNavigableUrlForMe();

This will allow to search links by page or request parameters using R# - search for usages feature and to check all types of mistyping in compile time.

But to make all this stuff to work, some sort of Request assembler/disassembler is required. Lets name it UrlBuilder - a simple class whose responsibilities is to generate request from strongly typed url and populate url by parameters from request.

public class UrlBuilder
{
    public string GetUrl(IUrl PageUrl);
    public void FillInUrl(IUrl PageUrl, NameValueCollection ValueCollection);
}

That class can be easily developed using TDD manner, here are my tests:

[TestFixture]
public class UrlBuilderTest
{
    #region Public Methods

    [Test]
    public void EmptyUrlRendersToUnresolvedPageUrl()
    {
        var Builder = new UrlBuilder();
        var PageUrl = new EmptyUrl();

        var HttpUrl = Builder.GetUrl(PageUrl);

        Assert.That(HttpUrl, Is.EqualTo("~/Module/Example.aspx"));
    }

    [Test]
    public void UrlWithAttributedPropertyRendersToPageUrlWithGetParameter()
    {
        var Builder = new UrlBuilder();
        var PageUrl = new ExampleUrl();

        var HttpUrl = Builder.GetUrl(PageUrl);

        Assert.That(HttpUrl, Is.EqualTo("~/Module/Example.aspx?CID=0"));
    }

    [Test]
    public void UrlWithValuesRendersToPageUrlWithGetParameter()
    {
        var Builder = new UrlBuilder();
        var PageUrl = new AttributedUrl()
        {
            CustomerID = 123,
        };

        var HttpUrl = Builder.GetUrl(PageUrl);

        Assert.That(HttpUrl, Is.EqualTo("~/Module/Example.aspx?CID=123"));
    }

    [Test]
    public void NameValueCollectionMappedToMarkedAttributeInUrl()
    {
        var Builder = new UrlBuilder();
        var PageUrl = new AttributedUrl();
        var GetValues = new NameValueCollection();

        GetValues.Add("CID", "825");

        Builder.FillInUrl(PageUrl, GetValues);

        Assert.That(PageUrl.CustomerID, Is.EqualTo(825));
    }
    // Tests for many parameters, and different types are skipped.
}

Ok, Now IUrl is not responsible for "Making Navigable Url For Me", it's left for UrlBuilder.

And the last thing left - integrate PageUrl and UrlBuilder to WCSF. As you now, WCSF using MVP pattern. Presenter is a good integration point for both PageUrl and UrlBuilder. Let's parametrize BasePresenter generic class with IUrl and force BaseView to initialize parameters form request.

public class BasePresenterWithUrl<TView, TUrl> : BasePresenter<TView>
    where TView : class
    where TUrl : class, IUrl, new()
{
    public TUrl Url { get; set; }

    public override void InitUrlFromRequest(NameValueCollection Parameters)
    {
        Url = new TUrl();
        UrlBuilder.FillInUrl(Url, Parameters);
    }
}

public class BasePage<TPresenter, TView> : Page
    where TPresenter : BasePresenter<TView>
    where TView : class
{
    protected override void OnInit(EventArgs Args)
    {
        Presenter.InitUrlFromRequest(Request.Params);
        base.OnInit(Args);
    }
}

With this implementation, page presenter contains property Url, that is filled with values from request and ready to operate during regular presenter events like OnViewLoaded and OnViewInitialized.

Ok, ready to rumble, here is an example usage of "url pattern":

// Url setting for "Add item" link
public override void OnViewInitialized()
{
    View.SetUrlForAdd(UrlBuilder.GetUrl(new ExampleUrl(){ ClientID =  0 }));
}
// Example page presenter
public override void OnViewLoaded()
{
    if(Url.ClientID == 0)
    {
        View.SetMode(Mode.Add);
    } else {
        View.SetMode(Mode.Edit);
        View.SetCustomer(_CustomersGateway.ById(Url.ClientID);
    }
}