Design things to be replaceable, not reusable!

I love this quote by Andy Hunt

I find aiming to make things “replaceable” far better than making things reusable. Reusable means you’re fortune-telling, trying to divine an unknowable future.

This is what I’m aiming for as well especially when you bring in the Sacrificial Architecture. Love it.

As you create your architecture, design all components to be replaceable, not reusable. This is a whole lot easier nowadays with AWS and Lambda creating serverless masterpieces, for example.

What are Projections in an Event Sourced Microservice Architecture?

In this video I will show you the different types of projections in an event-based microservice architecture and how you can use them.

Projections can be created in code in your microservice or in read-models when domain events are received and processed, for example, you process a published domain event and then store the result of the analysis in Oracle, MySQL, SQL Server, etc.

Microservices

I just finished uploading the final video on YouTube of the little mini-series “Microservices”. Take a look here:

1. Microservices – Part 1 of 4 – What Is A Microservice?
2. Microservices – Part 2 of 4 – DDD
3. Microservices – Part 3 of 4 – Big Ball of Mud
4. Microservices – Part 4 of 4 – Event Driven

These videos are all available on my YouTube channel at: Creating Great Software

 

Why you need serverless microservices, yesterday!

I have just published my FREE course: Why You Need Serverless Microservices, Yesterday“. Enroll for FREE!

WhyYouNeedServerlessMicroservices_960x520In this course I will walk you through the many benefits of creating serverless microservices instead of the traditional node / instance approach including the use of containers. There are more than enough things to worry about when you want to create a new cloud system or transform a legacy system to operate in the cloud.

From a business point of view, there are huge benefits in going serverless rather than instance based (including containers). A very large jump in business agility can be achieved through focusing on the problems and opportunities rather than the technical jungle of traditional computing solutions.

From a technical point of view, it is almost nirvana where you can eliminate many points of failures in the architecture.

Serverless Microservices Online Courses

Update 04/03/2019: I have completed the FREE course: “Why you need serverless microservices, yesterday!“. Enroll for FREE!
At the moment, I’m working on four online courses in the following order:

 

Why You Need Serverless Microservices, Yesterday, FREE Course

WhyYouNeedServerlessMicroservices_960x520

In this course I will walk you through the many benefits of creating serverless microservices instead of the traditional node / instance approach including the use of containers. There are more than enough things to worry about when you want to create a new cloud system or transform a legacy system to operate in the cloud.

From a business point of view, there are huge benefits in going serverless rather than instance based (including containers). A very large jump in business agility can be achieved through focusing on the problems and opportunities rather than the technical jungle of traditional computing solutions.

From a technical point of view, it is almost nirvana where you can eliminate many points of failures in the architecture.

 

“How To Build An Event Store”, Paid Course

eventstore_960_520_darker

Your event store is the heart of an event sourced system. The event store is the source of truth for all business events. It needs to be able capture and replay all domain events in your system reliably and with great performance.

In this course, I will walk you through building an event store that you can re-use in your own projects. In addition, I will walk you through building a read model that allows you to query domain events from the read model.

I will also go through why building your own event store has many more advantages over using a third-party event store.

We will be building the event store in AWS DynamoDB but you can apply the design to a traditional RDMS SQL storage just as well. I will go over the pros and cons in doing so.

 

“Architecting And Designing Event Based Microservices”, Paid Course

ArchitectAndDesign

Learn the details on how to design and architect event based microservices using Domain Driven Design (DDD), Event Storming, CQRS, and EventSourcing techniques. I will show you how best combine many principles, patterns, and techniques to create an architecture with as few points of failures as possible and still deliver a great solution.

What you will learn can be applied to cloud based systems but also to traditional, on-premise systems. The benefits are great in either environments.

Whether you are designing a new system in a greenfield environment or transforming a legacy system, I will show tips & tricks that you can use depending which type of project you are in.

 

“Implementing A Serverless Microservice in AWS”, Paid Course

Code_960x520

Learn the details on how to implement a serverless microservice in AWS using Domain Driven Design (DDD), CQRS, and EventSourcing techniques.

We will build a fully functioning billing system in AWS using Visual Studio and C# .NET Core. Even if you are not familiar with C# and .NET Core, you will learn a lot of practical tips on using the different AWS services including building fully automated CI/CD pipelines.

 

I’m still working on these courses but I’m planning on publishing “Why You Need Serverless Microservices, Yesterday” and “How To Build An Event Store” first.

My New YouTube Channel

I just created my new YouTube channel “Creating Great Software”. Have a look.

I will be publishing videos about creating great software including serverless computing in AWS. In addition, I will be publishing my STRONG opinions about the state of the software industry from time to time.

I’m super excited about this new channel and I’m looking forward in seeing your feedback. See you there!

Serverless System Course

In the past 27 professional years, I have been fortunate enough to gain a lot of experience and wisdom in 10 different software industries. These experiences are not just successful accomplishments but also plenty of failures and mistakes made along the way. It is ok to make mistakes as long as you learn from them, only then you will gain true wisdom in anything you do.

Over the years, I have always thought, “If I could gather a small team, I could share my knowledge and we can create great things“. Due to many circumstances, I never had the opportunity to mentor a small team and being in full control on what we would create. By full control I mean free from politics and other constraints. The sort of freedom you need to create and explore ideas without worrying about budgets, timelines, etc. When you can foster such an environment, great things get invented. Just look at the past successes in Silicon Valley.

How can I share my experiences and wisdom most efficiently and help you achieve your goals much more rapidly rather than doing it alone? I have decided it is time to create an online school where I can share all my knowledge and wisdom from the software industry. Well, how about an online course that is very deep in all aspects. An online course that would teach you the birds eye view of concepts and then drills down into the very deep aspects of it. A course that could show you actually how to put the conceptual components together in a very cohesive way. All the different pieces put together in an architecture that makes real sense and is practical and maintainable. We’ll leave the fluff out and concentrate on the entire system starting from a users perspective and walk backwards into the technical implementation details. In short, a

Serverless System Master Class

This master class would be extensive and we will go into a lot of details until we have built a working system that you could use in your future projects. As a minimum, you would learn a lot of concepts and software architecture. Since technology is always changing, I believe that the concepts will be more valuable then the implementation details in the course. In addition, how you bring these concepts together and why are extremely important because in your day-to-day work, you will have constraints that you need to work with and make compromises. There is no perfect architecture and even in this master class, you will touch on many different options one could take and why.

Please let me know if you are interested in the Serveless System Master Class. I would love to hear your feedback as I work on the course. I plan to post regular updates on my blog here.

Leave a comment below or contact me directly at thomasjaeger at gmail dot com and let me know your level of interest, please!

Event Store in AWS DynamoDB

Update 05/27/2019: How to build an event store masterclass” is now available. Learn how to build an event store using C#. NET Core, DynamoDB, MySQL for read models, and more.

In the past few weeks, I have been working on creating an event store in AWS DynamoDB and AWS S3. I use an event store for the domain driven design (DDD) concept in that system. Specifically, one of my systems uses CQRS and Event Sourcing (which is awesome, btw).

The idea for the event store started when I wanted to create my own event store for several reasons. I played with EventStore from Greg Young available at https://eventstore.org/ I encountered errors when I tested EventStore and I expected 100% functionality without issues. Besides, I did not want to babysit another persistence mechanism. I just do not have time for that.

I have created an event store before that was based entirely on Redis. That had worked great and it was super-fast. I used https://redislabs.com/ service to allow zero-maintenance of a fully clustered Redis solution. This has been working for some time now. The only problem was that it can have inconsistencies when it comes to running many applications nodes that use the Redis event store. Long story short, this has to do with complex timing, concurrency violations, etc.

So, I thought there must be a better way. I want zero administration headaches ideally, but I want to take advantage of the consistency capability of the underlying storage mechanism. I want the event store to be a service to the application itself that runs within the same process of the application. So, no need to babysit a separate event store cluster. I want the event store conceptually living side by side within the application.

If I can only take advantage of the consistency of the persistent mechanism, the application can then handle error conditions etc. accordingly based on what the use cases are.

After some experimenting and doing quiet a few load tests using http://loader.io, I can now say that the Event Store in DynamoDB has been born. My first 10,000 user / min load test has passed with flying colors and reached API response times of less than 10 ms with sustained rates of 170 clients / second. All this was running on a single cheap t2.medium instance. DynamoDB had to increase the throttling automatically to take this kind of load. This is one of those cool features, btw.

The event store I created in DynamoDB uses two concepts, Aggregates and Change Sets. Aggregates are the aggregates in your DDD system but only a handful of meta data. Change sets are one ore more domain events that are created when your DDD system processes a command.

I know I can get much higher numbers even by adding additional nodes and doing further tweaking. I would have to do much more load testing of course and tweaking but so far this has been a tremendous success and I’m super excited to take advantage of DynamoDB.

Anyways, I wanted to share this information. Maybe in the in future I can go into much more details. Time is the only problem I have really.

World of Warcraft Classic – A technical and business analysis

WoWClassic

First of all, I’m super excited that Blizzard announced a “Classic” version of WoW. I don’t even care about the new expansion “Battle of Azeroth”. I quit WoW about 1 ½ years ago because I was very much bored in doing the mindless, dumb down, and very repetitive expansions. I normally don’t post about games here but I have always enjoyed great games and being able to immerse yourself into different worlds is just cool. I’ve always been interested in how games are made as well.

In the last few days, I have read and watched several reactions of the WoW community on WoW Classic and the topic of Vanilla WoW in general. Lot’s of people are speculating on how and especially when this could play out. Everything I have heard and read about is pretty much based from a consumers’ point of view, the players point of view. I didn’t hear anything about why and how Blizzard can do this. We all know from the announcement that “it will take some time”.

However, I wanted to share with you the possible reasons why Blizzard decided to go this route of planning to release a Classic version of WoW. In addition, and more importantly, I wanted to provide you guys into a view of how Blizzard can do this from a technical point of view and what it would take to pull this off. This would help arrive at possible timelines and a possible number of players buying WoW Classic versus how many players will buy the new expansion and even making some other predictions.

Please keep in mind that what I’m about to present to you are my own speculations based on many years of technical expertise in creating and supporting small to very large software systems. I don’t want to bore you to death but I do want to point out my qualifications at least on the technical side to be able to give you an idea of what it would take for Blizzard to do this.

My qualifications

On my technical experience, I have been programming since 1985 and still do actively today. I’m a hands-on software architect working for a very large company. I’m currently in charge for a world-wide development and deployment project that is taking 2 years in the planning and active development. I’ve been in 10 different software industries ranging from real-estate, to banking, to healthcare, to automotive, to criminal justice, to aviation, and more. I can program in multiple languages and still do today from assembly (x86 mostly, even today) to high-level languages such as C (yes, C is high level when compared to assembly), to Java, to C# and others. In short, I’m a hands-on architect who designs, comes up with solutions and can also program it to the end. I also started several companies and still have a start-up company which is a cloud-based, world-wide Software as a Service (SaaS).

On my business side of things, I have great experience on what people really want but can not really express it simply because, software is hard as heck. I’m running a successful cloud-based software start-up while working for a large company. It’s running 24/7 world-wide without having us to intervene much because it was designed like that. Simple. I know what it takes to run a software company. Programming is just one portion of it. There are many other things to worry about in order to get software into the hands of consumers. This is one of those things I wanted to share with you.

Having said all that, this of course, still does not qualify me to speak for Blizzard since I do not and never did work for Blizzard and I’m not being endorsed in any way either. I have never seen any information about the internals of keeping the lights on and on how to be able to run WoW either. These are simply my own opinions. So, take it with a grain of salt.

What is WoW Classic?

At first, this should be easy to answer, right? Not so, it depends on who you ask. Let’s ask some players who are big fans of the Vanilla version including myself. I would assume this is what Classic would mean to them (correct me if I’m wrong in the comments below and I’ll update here).

Player’s point of view:

  1. Exactly the same game as Vanilla from 2004
  2. None of the new expansion features are “added-on” like LFG, etc.
  3. Literally the same game as if you would have bought it back in 2004
  4. The same computer system requirements as back in 2004
  5. Being able to access your characters for many years from now
  6. Perhaps some patches applied to fix bugs (we’ll come back to that one)
  7. Minus the bad launch experience (we’ll come back to that one)

So, from a consumer’s point of view, these seem very reasonable requests. However, there are technical challenges in which I will go into more detail later on.

Blizzard’s point of view:

  1. Additional revenue
  2. Additional revenue
  3. Additional revenue
  4. Let players access Vanilla 2004 content
  5. Let players experience the original game play
  6. Apply critical bug fixes
  7. Smooth roll-out with today’s hardware and software infrastructure, world-wide
  8. Smooth support
  9. Prolonged marketing and name recognition

This could be a definition of what WoW Classic could mean for Blizzard. For the why and how, read on.

Why WoW Classic?

Why would Blizzard announce a Classic version of WoW? Notice that Blizzard did not announce “World of Warcraft – Vanilla”. This could be just a marketing idea because “Classic” could mean many things such as:

  1. Original vanilla
  2. Original vanilla plus BC
  3. Original vanilla plus BC plus ROTLK
  4. Original vanilla plus BC plus ROTLK plus …
  5. Heavily modified game engine with Vanilla content
  6. Heavily modified game engine with Vanilla content plus expansions
  7. New game engine with vanilla content
  8. New game engine with vanilla content plus expansions over time
  9. etc.

As you can see, Classic could leave the door open to multiple options on what to deliver and what that really means. It buys some time as well. Back to why Blizzard would create a Classic version.

There are three obvious reasons and it makes perfect sense:

  1. Additional revenue
  2. Additional revenue
  3. Additional revenue

Blizzard is a company and needs to make money. As a company, you are not here to make friends, you are here to make revenue and keep increasing revenue whenever possible. This seems obvious but it is super important to keep this mind. Almost all decisions Blizzard is making drives around this fact. So, let’s look at the financial aspect of it.

Since 2004, Blizzard has gained monthly subscription accounts and peeked at 12 million accounts in the second half of 2010. However, since 2011, Blizzard has lost players over time by no longer paying the monthly $15 subscription fee. Each expansions introduced a slight peek but then the number of accounts fell shortly afterwards. It never again reached the 12 million subscriber number. At least that is what we know of since Blizzard no longer publishes these numbers for obvious reasons. Why tell anyone that you are loosing your core customers? This is not something you want to announce. You don’t want to tell potential future customers that your existing customers are leaving by the millions. This makes perfect business sense.

It is estimated that Blizzard currently holds around 4.5 – 5.5 active and paying WoW subscribers. Say we take a wild guess and estimating an average of 7 million subscribers on a yearly basis since 2004.

Revenue (avg):

$105,000,000 / month
$1,260,000,000 / year
$16,380,000,000 total, since 2004 (that’s 16$ billion)

This does not even include any of the add-ons such as transfers, gold, pets, etc. Pretty impressive, over 16$ billion in revenues based on a game. At first, it does look great. However, let’s take a closer look on trying to gain more revenue for a company, rather lost revenue.

You gain some, you loose some. That is normal. Players stopped playing even in 2005, we just don’t know those numbers because all we heard how WoW was taking over since 2004. But, let’s just take a look at the falling numbers since 2011. That is 7 years of lost revenue for many, many reasons. If we assume that Blizzard could have held the 12 million accounts since 2010 until today, Blizzard should have generated:

$2,160,000,000 / year

but it didn’t so we assume the $1,260,000,000 / year number. That is a lost revenue of 900$ million per year. That is a lost revenue of:

$6,300,000,000 total revenue loss

That is more than 6$ billion in cash folks. This is some serious money. Like I said, there are many, many reasons of why these numbers fluctuate. Computer games in general are short-lived. There are many reasons why WoW is still the king of MMORPG’s though. I think we all know the reasons.

From a company’s point of view, I would say the question would be: “How can we recover this lost revenue?”. How could this be even increased and surpass any previous revenue records? These are the questions Blizzard should ask itself.

Another question would be, “How can we slow down or stop the decline of new and existing subscriptions?”

I’m not saying that Blizzard will ever reach 12 million subscribers again for sure. But you never know. I believe it is possible to reach 12 million subscribers again. Let me explain. Since 2011, Blizzard lost 5 million customers each year. I’m not talking about keep the same 5 million people across 7 years. That is just one part of it. That is 5 million core customers lost that includes any new players that Blizzard could have kept over the years. That is the ideal scenario.

I believe that Blizzard knows it can bring back at a minimum 5 million old customers back given the right conditions and timing. All you have to look at is the internal data of lost customers which we do not have access to.

  1. Prediction: 5 Million WoW Classic sales

My first prediction is that Blizzard will be able to sell, easily I might say, 5 Million WoW Classic copies of the game. That is roughly $100 million in revenue right there. If Blizzard requires to have active subscriptions, that would be $75 million per month or $900 million per year. That is 4.5$ billion in 5 years. That is some serious cash just from a Classic game.

I can tell you this from my experience, if I had roughly 1$ billion in almost guaranteed revenue per year, you bet I can solve all technical issues. More on that later.

Now comes the ripple effect. Even if those 5 million “returned” customers drop out and no longer play, because they have an active subscription, they might be interested in purchasing the latest expansion. Great for Blizzard. But, let’s be realistic for a moment. If Blizzard gains 5 million returned customers, how many would stick around to level 60? It’s hard to say but my guess would be that it would be the majority of those 5 million if you consider “Why” these customers are coming back.

  1. Prediction: 4 Million active WoW Classic subscribers month over month

So far, I gave you mostly the financial justifications. But there are other many other reasons why Blizzard would want to come out with a WoW Classic edition:

  1. End illegal legacy servers around the world because of the lost revenue and no longer have to spend legal costs to do so
  2. Make customers happy. This has a halo affect and drives many other sales within Blizzard’s offerings.
  3. Keep internal staff excited and stay with Blizzard. Like other companies, Blizzard needs to worry about keeping its top talent. This is an unspoken rule. Money won’t help here. People that work in the software industry want their work to be seen and excite others. That’s what it’s all about.
  4. Make WoW challenging and not dumbed down. Blizzard won’t be able to do this with the current version and would risk loosing existing customers. The Classic edition has that built in, free. You have to work your butt off and worry about a lot of aspects of the game in Vanilla. This is much more challenging than the current dumbed down WoW.
  5. Simply no longer have to deal with internal company debates of doing a Classic or not. This kind of energy can be put into the creative process of putting out a Classic edition. And also no longer have to deal with customers constantly asking for Vanilla.

I’m sure there many other reasons. Please let me know your thoughts and I’ll add them here.

What are the technical challenges?

First of all, I heard people saying that Blizzard may no longer have the source code. I can tell you right there with 100% certainty, that this is complete hogwash. Of course, Blizzard has the source code from the very beginning. As a software company, the source code is your bread and butter. You treat the source code like it is gold. It’s put away safely so the source code itself especially master source code is physically protected and multiple locations that very few people have access to. Blizzard has every bit of source code, I can guarantee this. So, let’s drop that myth right there.

Why is having the original source code important? Let me explain. Like it was announced, “We want to recreate the original Vanilla game experience but not the launch experience”. This exposes a lot of information to me. What this tells me are the following:

  1. Blizzard wants to utilize all of its experience acquired in creating and maintaining software
  2. Blizzard does NOT want to recreate bad experiences for its customers if preventable
  3. Blizzard wants to utilize modern software tools, techniques, and infrastructure
  4. Blizzard can not use the original source code the way it is if it wants to do any of the items above
  5. Blizzard must, at a minimum, modify existing server and client software if it wants to do any of the items above

I’m pretty sure Blizzard did its due diligence and analyzed the original source code and got a rough idea of how much work needs to be done to the server and client software before the announcement was made. Keep in mind there are four, large, technical components to make this work:

  1. Client software. This is the software you and I have installed on our computer
  2. Server software. This backend software we never see and is maintaining everything the the client software can not and should not do. For example, I’m not talking about just “one” physical server. This backend infrastructure has many servers working together around the world 24/7 and allows players to play the game wherever they are. It enables for players to see and talk to other players. And so many other things.
  3. Infrastructure. Back in 2004, the Internet existed but there was no cloud-computing concepts, yet. With today’s technology and the power of cloud-computing, it has changed pretty much everything we do today in software development. This is not just the software itself. This also includes networking infrastructure, security, maintenance of the servers in each region of the world, and so much more.
  4. Maintenance. Blizzard needs to backup and maintain the software and the more importantly players data such as your character information. The good news is, today’s storage capabilities in cloud-computing is dirt cheap. When compared to 2004, data storage is practically free in a cloud environment.

Here comes the interesting piece of why this will take some time for Blizzard to release a Classic edition.

  1. Blizzard will need to modify existing client software to take advantage of modern capabilities such as communicating differently to its server software than the way it did back in 2004’s version of the server software. This really depends on how the software is designed. Is it designed so co-mingled with infrastructure that it is hard to separate? Or, was it designed to easily replace certain parts of the software with other, newer parts? This is one of those difficulties of creating and maintaining software. Is it crappy software quality inside or is it high quality software. Only Blizzard knows.
  2. Blizzard will need to modify or even replace the server software. Blizzard wants to take advantage of modern infrastructure such as cloud-computing. There is a good chance that Blizzard can actually throw away old server code because cloud-computing components may already provide a lot of functionality that Blizzard had to create and maintain back in 2004. At a minimum, Blizzard will need to modify existing server software. This is probably the most important part of the entire WoW Classic edition. This piece must work really well and Blizzard will spend most of the time on this area.
  3. If the current resources working on the new expansion can not be used for this, Blizzard needs to hire additional staff under the guidance of the original, 2004 Vanilla team members. The great advantage Blizzard has here is that it knows how “not to do things”. What I mean by that is that because 13 years have passed, Blizzard must have learned an awful lot on what works and what doesn’t work. So, hiring top talent to get the job done also takes time. It’s not easy to find the best of the best. This really depends when Blizzard starting this process.
  1. Prediction: WoW Classic will be released 4th Quarter 2018.

Given the items above and given the capabilities of modern software and cloud-computing and the right number of top talent to work on this, I believe Blizzard can pull this off to release WoW Classic by the 4th Quarter of 2018.

Here we are, a few ideas and speculations from my end. Let me know your thoughts and if I’m completely off and gone crazy. I know we all want this happened by Christmas, 2017. But, that’s not going to happen, or will it? Haha.

I will see you in Classic.