Developer's notes on Mongo database

I like playing with new technologies and NoSQL and MongoDB is not an exception.
But, while working with Mongo, I have made few mistakes, that turned out to be painful to me.

1. Do not allow MongoDriver classes go beyond your DAL

Firstly, I neglected layered structure, planning my DAL for building fresh project on mongo. The original thought, was to create factory-class that will contain accessors for all mongo collections, of the application. Of course these collections were implemented with help of C# mongo driver (using MongoCollection class). That solution, soon stroked me in the back, since MongoCollection methods, provoke usage of hard-coded literals for complex queries.
After a while, a whole application were filled with a lot's of magic-strings with field names and mongo-specific commands.

Solution, that seems appropriate now, is usage of Repositories, for DDD's aggregation roots.
The idea is to stop "string plague" right at repository level. The rule of thumb - do not allow any Business Logic at repository, and do not allow any MongoCollection usages, on any layer, above than repository.

2. Plan yours select carefully

While working with SQL database, most of the time, complex and fast queries can be built, against any reasonable database structure.
So, typical workflow were to build lean domain model, persist it to database, in a way, your's ORM supports, and only than you can start thinking about querying data.
Only in rare cases data structure can be so in-efficient, that it will require refactoring.

Using mongo, you should always think upfront about querying data, and pre-compute a lot of things, before insertion of the document.
To make queries more efficient, we have redesigned storage structure of our data, already 3 times, in just 2 months!

3. None of yours assumptions SQL assumptions can be applied to mongo

If you'll try to bring any assumptions, about tables from SQL world to mongo, it will bite you back soon! Even most basic ones should be checked with mongo documentation.
A part of our application were build, with assumption, that no duplicated records can be returned from collection. After few "impossible" exceptions, I have realized, that sometimes records can be duplicated, if you are using cursor feature. It's not bad by itself, but thanks to my implication, that this is impossible, a lot of code should be re-tested and re-factored, to handle such cases.

4. Schema-less does't mean, that you should't care about it.

My first impression, were "Woo-hoo! No more boring alter-table scripts, just add a field, that's it". But actually, whole responsibility of management database versions is moved to your's build script or DAL code.

.Net IoC performance comparison

Finally, I have compared performance of most widely-used IoC containers.

I'am clearly understand, that for IoC performance, in most cases, is not an issue, but I were still curious enough to test.

Here are the results:

Name	Total	PerRequest
Structure Map	377ms	0.377ms
Autofac	287ms	0.287ms
Unity	351ms	0.351ms
Ninject	2638ms	2.638ms
Manual	1ms	0.001ms

Test source is available via GitHub

Assumptions, behind the test, is optimized for Web scenario, where several threads request classes in different scopes.

Declarative UI interactions using Reactive Extensions for Javascript

We are creating a page, with a lot of controls, that are talk to each other. For example "When DD1 (DropDown 1) changes, change T1 (TextBox 1), T2, reload U1 (UpdatePanel 1), but only when DD1 and DD2 has changed, update T3).
Using imperative style, your's attempt will defiantly will end with a lot of nested callbacks, and it will be very problematic to "read" original interaction scheme, from the code.

Rx will simplify a bit this problem. It will allow you to describe interactions (and data flow) between these controls, in declarative style.

Let's try to create a page, that contains 2 input text boxes, and will compute the result and write it, to the third text box. The result, will appear, after 1 second, to simulate Ajax call. If we have made many changes, than only last result will be shown to us, instead of displaying all intermediate results. And no events will be triggered, until we sat all initial values, to the controls.

Here is resulting marble diagram (red bars – are ajax calls):

 

So, the diagram shows, that T3 is calculated as T1 + T2. Calculation is made on the server, using ajax calls, and we need to recalculate T3, each time T1 or T2 changes. If Text changes, while calculation is in process (red bar), then it's result should be ignored, and new call made, to recalculate T3 with most recent data. And last thing, during initialization, no events will be fired, but when we "turn on the switch", each text box will fire the event, with it's value.

Implementation

Sample page markup:

JavaScript:

Notes:

@2 – Latch is a sort of a global "power on" switch, when we call Latch.OnComplete(), messages will start cycling around
@3 – AttachToChangeAndSetValue – it's functions that returns observable on control's change event, also, it does some "plumbering", I'll describe it later
@18, @20 – Two observables, Text1Changed and Text2Changed on marble diagram
@21 – Rx CombileLatest expression, that actually do that "value management" that is shown with arrows on a diagram
@27-28 – Actually doing calculations, and simulating Ajax call with duration of 1000ms, returns Observable, with fires, as soon, as value is ready (Ajax call returned, or 1000ms passed)
@32 – Ensures that only latest calculation result will be displayed, all earlier results will be ignored
@43-44 – Starts Computations, by releasing latch

Latch implementation

Function AttachToChangeAndSetValue creates observable from html events, but before returning this observable, it is preceded by Latch observable (@11-15 return Latch.Concat(source)). Semantic of Concat function, says that all events from source will be ignored, until Latch source is depleted. Now, we can freely change control's state (in initialization phase), without any side effects. But, as soon, as we call Latch.OnCompleted(); (@43), Concat will allow to pass change events from source.

At line @42, Latch is instructed to fire dummy event, value from this event is overwritten with control's default value (@11-14), this is for setting default values of control, and to set system to "on - tw" state (on the marble diagram).

Throwing our obsolete ajax responses

Lines @25-30 Maps update requests, to the source of sources of ajax responses. Don't be scared, it's just a list of events, and each of them, will fire when ajax response will return.
For simplification, we don't use ajax, but timer instead. Rx.Observable.Timer(1000) returns an observable, that will fire once (after specified interval). These items are represented as a red bars, on the marble diagram.
Switch function, at line @32, takes this source of sources of ajax responses, and wait until most recent event will fire, this will guarantee, that we will display data from latest recalculation.

Verify stored procedures parameters and nice side-effect of abstractions

Few week ago, I have blogged about wrapper around SPROC calling routine. At that moment I thought, about this, only like a sort of a syntax-sugar around “low level” data access classes.

Yesterday, I have faced a requirement, to verify and revalidate all my stored procedures signatures over new database. The task tended to be mundane and useless, but thanks to introduced abstraction (that builder), we can create a test, that will “fake” a stored procedure builder with a recording-mock, record all procedure calls and it’s parameters and verify it’s signatures against target database.

Here, how it works:

1. Tests replace CommandBuilderFactory, with Factory, that produces recording builders.
2. Test executes All possible methods of each Data Gateway, to record called procedures and parameters.
3. Each procedure is verified, against it’s SQL declaration.

Recording builder, in my case, allows a gateway to fill-in all required parameters. When builder is requested to do “build-up”, to produce an executor in my case, it just throws special exception. This type exception is excepted by caller.

Great, now we have a list of procedures, it’s parameters and parameter types. Next task is to check parameters of each procedure against procedure in SQL. ADO.net has method “SqlCommandBuilder.DeriveParameters”, which populates a SPROC call with it’s parameters.

Viola, the test is ready, and can be executed on continuous basis, to make shore, that nobody will commit wrong SPROC declaration anymore.

So, the lesson that I got, is, that abstraction over BCL classes has a hi chance to pay off to you :) 

Reactive extensions – WPF fold/unfold

I have just implemented WPF fold/unfold functionality using Reactive Extensions Framework (RX). Desired algorithm is to "unfold" wpf window, as soon, as mouse entered, and "fold" after 5 seconds of mouse leaving.

First, naive implementation:

This implementation contains a bug, when you move mouse in/out several times in 5sec interval, window can left collapsed, because old "collapse" events will arrive, after delay.

Here is RX implementation, that discards "fold" messages, as soon, as new "fold" messages has arrived:

Reactive extensions for javascript.

Microsoft has released Rx extensions for Javascript. It's a great tool for organizing events processing in yours UI code.

I have used it to handle following task: Call Ajax-Update every 5 seconds, after filter button clicked and on paging request, but only when previous request is finished. Tricky? Good luck debugging it later.

This can be implemented using Rx extensions. In that case all that interactions became explicit and clear, placed on few lines of code.

Here is an example:

Speed-up database insert operation using Reactive extensions

Application inserts a lot of data to mongo databse. Insertion of small records one-by-one takes a lot of time. So, obvious solution to buffer this records. Buffer feature of Rx extensions fits well for such buffering.

Code notes:

• Line 17-25 – Aggregates messages for 100ms period into an array and pass to saving function
• Line 56 – Pushes measures to Rx buffer.

This buffering gave me roughly 30% performance gain.

The code itself:

public class MeasureGateway : IMeasureGateway, IDisposable
{
	private readonly AddObserver _addObserver = new AddObserver();
	private static readonly TimeSpan BufferTimeout = TimeSpan.FromMilliseconds(100);
	private DateTime _lastMeasureAddedAt = DateTime.MinValue;
	private readonly IDisposable _bufferSubscription;

	[InjectionConstructor]
	public MeasureGateway() : this(true)
	{
		
	}
	public MeasureGateway(bool useBuffering)
	{
		if (useBuffering)
		{
			_bufferSubscription =
				_addObserver
					.Buffer(BufferTimeout).Subscribe((q) =>
														 {
															 if (q.Any())
															 {
																 AddMeasureInner(q);
															 }
														 });
		}
	}

	private void AddMeasureInner(Document q)
	{
		using (MongoScope mongo = MongoScope())
		{
			mongo.DefaultCollection.Insert(q);
		}
	}
	private void AddMeasureInner(IEnumerable<Document> q)
	{
		using (MongoScope mongo = MongoScope())
		{
			mongo.DefaultCollection.Insert(q);
		}
	}

	public void Add(DeviceMeasure measure)
	{
		Document measureDoc = GetMeasureDoc(measure);

		if (_bufferSubscription != null)
		{
			if (DateTime.Now.Subtract(_lastMeasureAddedAt) > BufferTimeout)
			{
				AddMeasureInner(measureDoc);
			}
			else
			{
				_addObserver.Add(measureDoc);
			}

			_lastMeasureAddedAt = DateTime.Now;
		}
		else
		{
			AddMeasureInner(measureDoc);
		}
	}

	private class AddObserver : IObservable<Document>, IDisposable
	{
		private IObserver<Document> _observer;

		#region IDisposable Members

		public void Dispose()
		{
		}

		#endregion

		#region IObservable<Document> Members

		public IDisposable Subscribe(IObserver<Document> observer)
		{
			_observer = observer;
			return this;
		}

		#endregion

		public void Add(Document doc)
		{
			_observer.OnNext(doc);
		}
	}

	public void Dispose()
	{
		if (_bufferSubscription != null)
		{
			_bufferSubscription.Dispose();
		}
	}