Event Store in AWS DynamoDB

In the past few weeks, I have been working on creating an event store in AWS DynamoDB and AWS S3. I use an event store for the domain driven design (DDD) concept in that system. Specifically, one of my systems uses CQRS and Event Sourcing (which is awesome, btw).

The idea for the event store started when I wanted to create my own event store for several reasons. I played with EventStore from Greg Young available at https://eventstore.org/ I encountered errors when I tested EventStore and I expected 100% functionality without issues. Besides, I did not want to babysit another persistence mechanism. I just do not have time for that.

I have created an event store before that was based entirely on Redis. That had worked great and it was super-fast. I used https://redislabs.com/ service to allow zero-maintenance of a fully clustered Redis solution. This has been working for some time now. The only problem was that it can have inconsistencies when it comes to running many applications nodes that use the Redis event store. Long story short, this has to do with complex timing, concurrency violations, etc.

So, I thought there must be a better way. I want zero administration headaches ideally, but I want to take advantage of the consistency capability of the underlying storage mechanism. I want the event store to be a service to the application itself that runs within the same process of the application. So, no need to babysit a separate event store cluster. I want the event store conceptually living side by side within the application.

If I can only take advantage of the consistency of the persistent mechanism, the application can then handle error conditions etc. accordingly based on what the use cases are.

After some experimenting and doing quiet a few load tests using http://loader.io, I can now say that the Event Store in DynamoDB has been born. My first 10,000 user / min load test has passed with flying colors and reached API response times of less than 10 ms with sustained rates of 170 clients / second. All this was running on a single cheap t2.medium instance. DynamoDB had to increase the throttling automatically to take this kind of load. This is one of those cool features, btw.

The event store I created in DynamoDB uses two concepts, Aggregates and Change Sets. Aggregates are the aggregates in your DDD system but only a handful of meta data. Change sets are one ore more domain events that are created when your DDD system processes a command.

I know I can get much higher numbers even by adding additional nodes and doing further tweaking. I would have to do much more load testing of course and tweaking but so far this has been a tremendous success and I’m super excited to take advantage of DynamoDB.

Anyways, I wanted to share this information. Maybe in the in future I can go into much more details. Time is the only problem I have really.

Advertisements

Creating IDs with CQRS and Event Sourcing in Java and .NET

I’m currently working on a cloud-based system that uses microservices with domain driven design and CQRS and Event Sourcing in Java. I love C# and .NET but I decided to do this in Java for several reasons. This could have been done in C# just as well. In fact, this system is a mix of some C# microservices and Java microservices. The important thing is that we are using the same concepts that are applicable when doing DDD and CQRS and Event Sourcing.

Anyways, I wanted to put down my thoughts on what I have been thinking about since last week and I hope this might be useful to anyone who came across the same question. So, my question since last week was this:

“Who creates an entity id in DDD when doing CQRS and Event Sourcing?”

Well, the way I have been doing this was as follows:

  1. When an application service executes a use case via one of its methods, it can create a new entity via a new immutable value object. All my entities’ ids are immutable value objects.
  2. Sometimes, a factory in the domain model creates new entities. Again, the entities’ ids are created via immutable value objects.
  3. Sometimes, a domain service creates new entities with immutable value objects.

This is all fine and good. I started to look at Greg Young‘s sample C# project at his Github here. For this who do not know him, he is a fantastic speaker and mentor for CQRS and Event Sourcing. Check out his videos, papers, and blog posts. Excellent material. He has a free 6 hour video here. And, while you are at it, check out his video about: “7 Reasons Why DDD Projects Fail“.

What poked my interest was this particular line in his CreateInventoryItem command code:

public class CreateInventoryItem : Command {
    public readonly Guid InventoryItemId;
    public readonly string Name;

    public CreateInventoryItem(Guid inventoryItemId, string name)
    {
        InventoryItemId = inventoryItemId;
        Name = name;
    }
}

If you look closely on line 5, you will see that the entity id inventoryItemId is being passed as part of the CreateInventoryItem command. This is a small but important detail.

I had an interesting thread discussion with Greg Young about this here. Greg raised a few good questions:

  1. What about when you want to send 3 commands?
  2. How will you start returning ids from commands?
  3. What if a command creates 3 entities?
  4. How will the domain get those ids back to the client?
  5. My UI is creating an account then the next step is managing some details. How does the UI go from one step to the next without knowing the account id?

As you can see, there are several good reasons why a client would want to create the ids instead of the domain model. Another reason I would want to add to this is that if the domain would create the ids, it would violate the goal of keeping track of all state changes in domain events with the commands as the initiators. In my opinion:

In a CQRS based system with Event Sourcing, commands carry all attributes that are required to execute the command including the creation of identities.

So, a client can create a command with ids that are based on Guids/Uuids and then send these creation commands with those ids. This allows you to do things such as persisting the commands and being able to replay those commands as well with 100% accuracy because the domain model with arrive every time to a deterministic state when you replay the domain events.

I like this very much and it provides a lot of freedom with inductive UIs / task based UIs.