Object Persistence, Part 4 – Video

In part 3 of my Object Persistence series, I introduced the complete Visual Studio 2012 source code.

This is part 4 of my Object Persistence series. I created a detailed video that goes thru all the parts in the solution. Enjoy!

Watch the video walkthrough on my YouTube channel:

Object Persistence, Part 3 – Source Code

In part 2 of my Object Persistence series, I’ve touched on the issues that still exist today.

In part 3, I’ve published a complete sample Visual Studio 2012 solution on GitHub that demonstrates object persistence using a db4o persistence provider. Over time, I will add additional sample persistence providers for Redis, SQL Server, and possibly a NoSQL provider such as SimpleDB (one of Amazon’s great NoSQL databases).

This sample solution includes complete server side and client side layers. The Server side runs as a REST based Web API 2 service. The server portion also includes a simple domain model and, of course, the persistence provider and how it is implemented. I will update the solution over time, expand the domain model, UI, etc. as required.

The client side is a WPF application that consumes the REST service. The payload to and from the REST service is via JSON objects.

ObjectPersistencePart3_WPF

I hope you like it. You can use this sample solution as a template to start simple or very complex software solutions. This solution can easily be taken and split across different nodes in a cluster of Amazon AWS EC2 instances, for example. However, for a cloud based solution, your persistence would have to support certain features. I will go into details when I add the Redis persistence provider.

Instead of writing a very long blog post, I will post a screen cast video and go through the solution. I think this will make more sense and you have a chance to go through the source code with me. So, go ahead and get the latest version from GitHub and start playing with it.

Object Persistence, Part 2

It has been several years (six years to be precise) since I published my article “What is Object Persistence”. I have received great feedback since then from the blog post and from presentations I gave about object persistence. However, the challenge of using an appropriate object persistence mechanism still exists today. Fortunately, there are even more ways to store your objects nowadays when compared to 2008. With the great opportunities that cloud computing offers, object persistence gets even more exciting.

So, I decided to publish a follow-up blog post about object persistence. In addition, I will also provide working C# code so that you can try it out yourself. Since object persistence is such an important piece of a software architecture and the depth of technical information about it can be overwhelming, I may have to spread out my thoughts and example source code over additional blog posts.

Most of the example source code I will be providing is coming straight from production systems I have built over the years with .Net and C#. Specifically, I will be providing persistence providers that you can use in your own systems or at least provide you with a huge head start. Some of the source code is changed to accommodate the example better but the provider pattern and the overall design is identical. The source code will be in C# and I will be using a .Net feature that has been available since .NET 2.0 – The Provider Pattern.

So, having said all this, I can pickup where I have left off with my first blog post. Say, you are familiar with the challenges of finding the right object persistence for your project. Let’s also assume you know “where” you want to store your objects in. If you are not familiar of what object persistence is, please take a look at my previous blog post “What is Object Persistence”.

Let’s start with a straight forward object persistence that will help you see the bigger picture and not get lost in the actual details of “how” to store the objects. At least for now. Let’s start with storing our objects in an object database named db4o. The reason why I want to start out with db4o is because it actually is the easiest way in .Net to persist your objects. I would argue that object databases can be used in at least 90% of .Net projects developed today. db4o has another advantage in that it can also run entirely in memory alone which is great for unit testing your persistence. In addition, db4o has such an extremely low learning curve that you will be up and running in no-time. Of course, the beauty of using a provider model is that you can do entirely different persistence implementations of the same domain model. So, you can use a different object database such as VelocityDB, for example.

Later on, I will show you how to store the same C# objects in different ways including the in memory version of db4o, a NoSQL solution such as Redis, which is a key-value store, and to round it out, a typical SQL storage such as SQL Server.

From an architecture point of view, it is very, very, important that our domain model has absolutely no clue about persistence. Our domain model will have no references to any persistence assemblies. Our domain model will be a lone assembly with no references to any service, interfaces, UI, and especially any persistence technologies. This is important because we want our domain model to be maintainable over time. You want your domain model to be independent from any other building blocks of your architecture because it will reflect your business domain and processes. This will make your entire system much easier to maintain and therefore much easier to react to requirements changes.

The Provider Model

The great thing about using the provider model is that the entire implementation is done inside a provider. Your entire source code on “how” to persist your objects is inside the specific provider. If you are not familiar with the provider model, please take a look at these resources to get familiar with it.

Microsoft ASP.NET 2.0 Providers: Introduction

Develop Provider-based Features of Your Application

Besides the introduction of generics in .NET 2, the provider model was in my opinion one of the most powerful features introduced. The .NET framework has been using the provider model internally ever since, all the way to the latest version of .NET 4.5. Here are some examples, where Microsoft is using the provider model:

Membership
Role management
Site map
Profile
Session state
Web events
Web Parts personalization
Protected configuration

and these are just a few of the current providers that ship with the framework. You can even build providers based on certain features of your system, for example. See the link above.

One of the great features of the provider model is that the framework will automatically load your persistence provider based on configuration information. This means that you do not even need any assembly references to your provider, the .NET framework will take care of the discovery, loading, and instanciation for you. This offers a truly decoupled implementation, a true plug-play mechanism out of the box. How cool is that?

Please keep in mind that this has nothing to do with the Repository pattern. I have used the provider model pattern for persistence for many years now and the Repository pattern does not come close to what the provider pattern can do for you. The Repository pattern violates the domain model encapsulation because the domain model is now aware of some sort of persistence idea even if the Repository is exposed via IRepository, for example. That is a big no no in my book because your goal should be to create something that is easy to maintain over a very long time.

If you are building a professional software solution, you should go with the provider pattern for abstracting persistence.

This took longer than I thought but I believe that I needed to set the stage first before we can continue. In the next blog post, we’ll get our hands dirty and start writing code and you will see how it can be done.

Continue with part 3, Object Persistence

Hash Generator Tool

There have been many times where I needed to generate hashes either from strings or from files. I usually use one of my code snippets depending what programming language I’m currently using. But, this time, I needed to generate hashes that do not change and need these hashes as constants. I actually needed these hashes in my Visual MASM IDE that I’m working on. So, I finally created a small tool called Hash Generator that can create MD5 hashes from either a string or from any file on your system. I may extend this tool to create SHA1 and other hashes but for now, it does the job for me and maybe you will find it useful, too. Let me know. 

Download it here. It’s about 2.5 MB in size and runs on Windows XP and up. It’s one exe file, no setup or any other files required. Just run the HashGenerator.exe file.

Hash Generator

Hash Generator

Visual MASM Windows Assembly IDE Sneak Peek Video

I just published a sneak peek video about my Visual MASM IDE for Windows assembly programming. Take a look and let me know what you think.


Visual MASM Windows Assembly IDE Sneak Peek

Visual MASM IDE for Microsoft Macro Assembler

Well, I decided to work on my Visual MASM for Microsoft’s Macro Assembler in my spare time. This will be a full featured IDE for assembly programmers. I published an IDE for assembly programming many years ago back in 2001 called VASM but I never finished it.

With Visual MASM you can create MS-DOS and Windows applications for 16-bit, 32-bit, and 64-bit Intel and AMD processors. Visual MASM looks & feels in many areas like the Delphi IDE or Visual Studio.

This will be an ongoing process and I will update it when I have time. Creating assembly programs for Windows is actually easier than MS-DOS applications since you no longer have to re-create the wheel(s) anymore because Windows has thousands of API’s available for you.

If you are interested in learning assembly programming and maybe find out why you would want to do that, run over over to my new website and check it out at http://www.visualmasm.com.

VisualMASM_screen01

AWS Summit

I’m planning on attending the 2014 AWS Summit on March 26th in San Francisco. Amazon has put up a great agenda. If you are doing cloud-computing and especially with the AWS services, you should attend. I will see you there.

How to turn on Long Polling on an AWS SQS Queue

The following code snippet will allow you to configure an AWS SQS queue for long polling using the ReceiveMessageWaitTimeSeconds attribute. For more information about Amazon long polling, see here:

http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html

// 5. Set Queue attributes
// The API Version 2012-11-05 of Amazon SQS provides support for long polling. (.net sdk v2)
// In the AWS console, you can verify in "Receive Message Wait Time" value of the queue
SetQueueAttributesRequest setQueueAttributesRequest = new SetQueueAttributesRequest();
List<Amazon.SQS.Model.Attribute> attributes = new List<Amazon.SQS.Model.Attribute>();
Amazon.SQS.Model.Attribute attribute = new Amazon.SQS.Model.Attribute();
attribute.Name = "ReceiveMessageWaitTimeSeconds";
attribute.Value = "20"; // 0 to 20 Seconds, default is 0
attributes.Add(attribute);
setQueueAttributesRequest.QueueUrl = queueUrl;
setQueueAttributesRequest.Attribute = attributes;
_sqs.SetQueueAttributes(setQueueAttributesRequest);

How to create an Amazon AWS SNS Topic and an SQS Queue that subscribes to it

Creating powerful cloud-based systems in C# and the .Net framework are possible for a long time now. Architecting a messages based cloud-computing solution that can handle millions of requests per day is not an easy undertaking but much easier when you use Amazon AWS services.

I recently needed to create an Amazon AWS SQS queue programmatically that subscribes to an SNS Topic that was also created programmatically. I could not find any examples in C# so I decided to post this code from one of my cloud-computing systems that I had designed and hope others find it useful. This particular code was used to test the dynamic nature of a messages based backend that can expand and contract the number of SQS queues based on how many active EC2 instances (nodes) in the cluster are available.

This code also shows how to create a policy for the SQS queue so that it will receive the messages from the SNS service. If you do not set a correct policy, the queue won’t receive any messages nor any exceptions are thrown.

Declaration code:

        private AmazonSimpleNotificationServiceClient _sns;
        private const string SNS_TOPIC = "YourTopicHere";
        private string _AWSSNSArn;

Core C# code:

            // 1. Create an Amazon SNS topic
            AmazonSimpleNotificationService sns = new AmazonSimpleNotificationServiceClient(RegionEndpoint.USWest1);
            _AWSSNSArn = sns.CreateTopic(new CreateTopicRequest
            {
                Name = SNS_TOPIC
            }).CreateTopicResult.TopicArn;

            // 2. Create the Amazon SQS In-Queue, will ignore it if it already exists
            AmazonSQS sqs = AWSClientFactory.CreateAmazonSQSClient(RegionEndpoint.USWest1);
            CreateQueueRequest sqsRequest = new CreateQueueRequest();
            sqsRequest.QueueName = General.IpAddressAWSFriendly;
            string queueUrl = sqs.CreateQueue(sqsRequest).CreateQueueResult.QueueUrl;
            GetQueueAttributesRequest getQueueAttributesRequest = new GetQueueAttributesRequest();
            List list = new List();
            list.Add("QueueArn");
            getQueueAttributesRequest.AttributeName = list;
            getQueueAttributesRequest.QueueUrl = queueUrl;
            GetQueueAttributesResponse response = sqs.GetQueueAttributes(getQueueAttributesRequest);
            string queueArn = response.GetQueueAttributesResult.QueueARN;

            // 3. Configure the Amazon SNS topic to publish to the SQS queue
            sns.Subscribe(new SubscribeRequest
            {
                TopicArn = _AWSSNSArn,
                Protocol = "sqs",
                Endpoint = queueArn
            });

            // 4. Set the queue policy to allow SNS to publish messages
            ActionIdentifier[] actions = new ActionIdentifier[2];
            actions[0] = SQSActionIdentifiers.SendMessage;
            actions[1] = SQSActionIdentifiers.ReceiveMessage;
            Policy sqsPolicy = new Policy()
                .WithStatements(new Statement(Statement.StatementEffect.Allow)
                                    .WithPrincipals(Principal.AllUsers)
                                    .WithResources(new Resource(queueArn))
                                    .WithConditions(ConditionFactory.NewSourceArnCondition(_AWSSNSArn))
                                    .WithActionIdentifiers(actions));
            SetQueueAttributesRequest setQueueAttributesRequest = new SetQueueAttributesRequest();
            List attributes = new List();
            Amazon.SQS.Model.Attribute attribute = new Amazon.SQS.Model.Attribute();
            attribute.Name = "Policy";
            attribute.Value = sqsPolicy.ToJson();
            attributes.Add(attribute);
            setQueueAttributesRequest.QueueUrl = queueUrl;
            setQueueAttributesRequest.Attribute = attributes;
            sqs.SetQueueAttributes(setQueueAttributesRequest);

Amazon AWS wins contract for CIA to host on AWS’ Cloud – Yeah!

I have been working with Amazon AWS Cloud web services since its beta introduction back in 2006 and still totally love it and use it today. When I heard about the news that Amazon won the contract to host CIA’s operations in the cloud, I was super excited and another confirmation that NO ONE knows cloud computing better than Amazon, not even IBM, the sore looser in the bidding process.

When it comes to service offerings and the rich capabilities and great SDK’s offered by Amazon AWS, Microsoft’s Azure, Google, IBM, and others do not come even close. Period!!!

http://techcrunch.com/2013/10/07/amazon-web-services-wins-again-in-battle-to-build-the-cia-and-nsa-cloud/

To get started, go to http://aws.amazon.com

Follow

Get every new post delivered to your Inbox.