Event Store in AWS DynamoDB

In the past few weeks, I have been working on creating an event store in AWS DynamoDB and AWS S3. I use an event store for the domain driven design (DDD) concept in that system. Specifically, one of my systems uses CQRS and Event Sourcing (which is awesome, btw).

The idea for the event store started when I wanted to create my own event store for several reasons. I played with EventStore from Greg Young available at https://eventstore.org/ I encountered errors when I tested EventStore and I expected 100% functionality without issues. Besides, I did not want to babysit another persistence mechanism. I just do not have time for that.

I have created an event store before that was based entirely on Redis. That had worked great and it was super-fast. I used https://redislabs.com/ service to allow zero-maintenance of a fully clustered Redis solution. This has been working for some time now. The only problem was that it can have inconsistencies when it comes to running many applications nodes that use the Redis event store. Long story short, this has to do with complex timing, concurrency violations, etc.

So, I thought there must be a better way. I want zero administration headaches ideally, but I want to take advantage of the consistency capability of the underlying storage mechanism. I want the event store to be a service to the application itself that runs within the same process of the application. So, no need to babysit a separate event store cluster. I want the event store conceptually living side by side within the application.

If I can only take advantage of the consistency of the persistent mechanism, the application can then handle error conditions etc. accordingly based on what the use cases are.

After some experimenting and doing quiet a few load tests using http://loader.io, I can now say that the Event Store in DynamoDB has been born. My first 10,000 user / min load test has passed with flying colors and reached API response times of less than 10 ms with sustained rates of 170 clients / second. All this was running on a single cheap t2.medium instance. DynamoDB had to increase the throttling automatically to take this kind of load. This is one of those cool features, btw.

The event store I created in DynamoDB uses two concepts, Aggregates and Change Sets. Aggregates are the aggregates in your DDD system but only a handful of meta data. Change sets are one ore more domain events that are created when your DDD system processes a command.

I know I can get much higher numbers even by adding additional nodes and doing further tweaking. I would have to do much more load testing of course and tweaking but so far this has been a tremendous success and I’m super excited to take advantage of DynamoDB.

Anyways, I wanted to share this information. Maybe in the in future I can go into much more details. Time is the only problem I have really.

Advertisements

Object Persistence Reference Implementation

I’ve been updating my reference implementation in the last few days. I’m actually using this reference implement in my own projects. You can download the latest version on my GitHub repo.

This is a complete .NET C# reference implementation to help you jump start a service oriented system running in a cloud environment such as Amazon’s EC2 or on-premis clusters.

This reference implementation shows you how to build a client and the server side. The client side is a sample WPF application that communicates via http REST requests using JSON payloads to the service side. Of course, you can use any type of client as long as the client can communicate via http and REST based JSON’s.

The service side is using a Web API 2 service layer that communicates to a central domain model. The service side demonstrates how to handle exceptions and edge cases and how to communicate failure to the client.

The persistence layer demonstrates the extreamly powerful provider pattern to store the domain objects into the following databases:

  1. db4o (an object database)
  2. Redis (a NoSQL database)
  3. SimpleDB (a NoSQL database)
  4. SQL Server (comming soon)

Please note that the entire system has no knowledge on how the objects are stored. All implementation details are in the individual providers listed above. This means that you can switch the persistence provider without having to recompile and therefore switch a running system from one persistence store to another.

I will try to create a sample SQL Server provider soon.

appsworld North America 2015 at Moscone Center West, San Francisco

appsworldlogo

I’m a confirmed speaker at the appsworld North America 2015 at Moscone Center West, May 12-13 in San Francisco, CA.  Discover the future of multi-platform apps. See all confirmed speakers. This sure will be an exciting event. I’m still working on my presentation that will include Redis, Amazon AWS, C#, and more. I will see you there.

Presenting “Object Persistence in C#” in Sacramento, CA, March 25th, 2015

Wednesday, March 25th, 20015, I will be presenting “Object Persistence in C#” at the Sacramento .NET User Group (SAC.NET) at the Microsoft Office at 1415 L Street, Suite 200, Sacramento, CA 95814 starting at 6:00 pm. Maria Martinez, Co-Organizer, Sacramento .NET User Group, was kind enough in helping to get this organized. Thank you Maria. I will see you there.

Presenting “Object Persistence in C#” at Bay.NET User Group on April 9th, 2015

I will be presenting “Object Persistence in C#” at the Bay.NET user group at the Berkeley City College located at Room 451A, 2050 Center Street, Berkeley, CA from 6:15 pm – 9:00 pm.

Deborah Kurata, Co-Organizer, East Bay Chapter Leader, was kind enough in helping to get this organized. Thank you Deborah.

I will see you there.

Object Persistence, Part 5 – Video – Redis Provider

In part 4 of this series, I went through the entire Visual Studio solution and also showed the db4o object database provider.

In part 5, I’ll show you how to store your Plan Old C# Objects (POCO) into Redis using the Redis Cloud at redislabs.com service. We will build the Redis Provider and take it out for a spin storing our new domain objects.

You can download the source code at GitHub.

Object Persistence, Part 4 – Video

In part 3 of my Object Persistence series, I introduced the complete Visual Studio 2012 source code.

This is part 4 of my Object Persistence series. I created a detailed video that goes thru all the parts in the solution. Enjoy! Make sure you continue with part 5 where we build a Redis persistence provider.

Watch the video walkthrough on my YouTube channel: