Architecture of a Big Data messaging and aggregation system using Amazon Web Services part 1

Exercises in .NET with Andras Nemes


Amazon has a very wide range of cloud-based services. At the time of writing this post Amazon offered the following services:

Amazon web services offerings

It feels like Amazon is building a solution to any software architecture problem one can think of. I have been exposed to a limited number of them professionally and most of them will figure in this series: S3, DynamoDb, Data Pipeline, Elastic Beanstalk, Elastic MapReduce and RedShift. Note that we’ll not see any detailed code examples as that could fill an entire book. Here we only talk about an overall architecture where we build up the data flow step by step. I’m planning to post about how to use these components programmatically using .NET and Java during 2015 anyway.

Here’s a short description of the system we’d like to build:

  • A large amount of incoming messages needs to be analysed – by “large amount” we mean thousands per…

View original post 742 more words