Trying out non AWS options

I have been using Amazon Web Services for all of my (or companies' I work with) compute or storage needs for the last 4 years. That include AWS S3, EC2, SimpleDB and even RDS (at MobStac). For the last few weeks I have been planning the platform choices for Mag. It does include AWS DynamoDB, S3 and EC2, but the picture is a bit different.

Amazon DynamoDB guarantees a lot of performance and I personally do not want to take database headaches. DynamoDB pricing model is great to start with too. It is not cheap, but does not bite at the same time. And you continue paying as you grow. The data model is similar to other NoSQL services and you can shift out later if you want.

AWS S3 is a really reliable, cheap storage solution and there is no doubt we will use it wherever we need. EC2 is great for its scalability or its powerful solution to failure handling. Bringing up a pre-configured EC2 (many data centers around the world) is fast, easy and cost per hour is cheap.  Also extra processing as and when needed is perfect fit for EC2 Spot Instances.

While AWS EC2 provides all these great services, hosting regular websites and serving traffic is still costlier. Bandwidth is premium inside AWS. Yes they are very well connected, but you can get that from many top quality hosting providers/data centers. Also the machines themselves are not as fast as a good VPS could be. I have been searching for VPS providers with SSD storage, newer processors and RAM. Recently I tested a tiny VPS (384MB RAM) with such configuration and the Apache Benchmark shows better performance than an AWS EC2 instance (Ubuntu, nginx, PHP, APC, apc.stat=0). Although PHP on the VPS was compiled, and I will re-run the tests, but the difference was very visible.

Of course I do not intend to co-locate or manage any hardware. But I certainly want to look at good VPS or unmanaged dedicated offerings. Configuring a VPS/server is not easy too, from firewall to internal network if you run a cluster, but if the pricing differences are worth it, then it makes business sense. And EC2 is never out of the picture. They will be used in parallel but for stuff I mentioned above and not for regular traffic.

Schema Example on Amazon DynamoDB

First an apology to anyone who read my previous blog post. I had used wrong rates for throughput capacity. I noticed this since my calculations seemed a little costly :)

Throughput Capacity price as on AWS website, as of today:

* Write Throughput: $0.01 per hour for every 10 units of Write Capacity
* Read Throughput: $0.01 per hour for every 50 units of Read Capacity

Thus as per my example of 7 tables with minimum 5 Read and 5 Write throughput capacity per table, hourly pricing is:
35 * $0.001 = $0.035 (Write)
35 * $0.0002 = $0.007 (Read)
Total hourly cost = $0.042, monthly cost = $30.24

My current database design is using 3 tables. One is for authentication. We will allow users to signup/login using either email+password (MD5) or using FB Connect or similar. In both cases we will store these authentication string(s) in an "Auth" table. The Primary Key will be the auth strings themselves, either concat of email+password or OAuth tokens. The items will have another attribute which will be `UserName`.  I am yet to read FB Connect documentation, so this may need slight re-configuration.

The `UserName` is a PK in the second table which is "User". The items contain common user attributes like `FirstName`, `LastName`, `Email` and many such needed data. The last table is the "Campaign" table where we store campaign configure information for Mag. This has a Hash and Range type Primary Key. The Hash part is the `UserName`, and the Range is the `CampaignId` (unique string generated from the title of the Campaign). Each user can run multiple Campaigns. What I am not sure of is what is the performance hit if all users mostly have only 1 campaign. In that case I think it will be better to store Campaigns as a Set in the "User" table and the "Campaign" table could have just a simple Hash PK named `CampaignId`.

My doubts as of now are what values to set for throughput. I neither want to overcharge myself, nor do I want AWS to throttle the connection. I guess a real example will shine more light. Back to coding now!

Day One of Amazon DynamoDB

Its been just about a day that I have been going through the documentation of the newly released Amazon DynamoDB. Using the PHP SDK and getting a basic Table up was simple. Right now I am planning the data organization. DynamoDB is a schema less data store. There is a primary key per table which is your main query column, so as to say. I am trying to use existing MongoDB based data mappers and modify them for DynamoDB.

I am planning to use Fat-Free Framework for the back-end. This is basically the management panel's API. All user interface will be JavaScript only, within the browsers. Templates will probably be Mustache. Anyways, back to DynamoDB: once I get a data mapper done, I will release it on GitHub. I am also looking for similar stuff in Python and Node. I am sure in a couple of weeks we will see them pouring over the Internet.

Pricing
I have been doing some pricing calculations. This is important for us since we have a tiny budget to start with. The price of AWS DynamoDB depends on two main things: size of data store and the throughput needed. The size is charged by the GB/month. So a decent 5 GB data store will cost $5/month. But remember there is metadata that AWS stores and you will be charged for that. I will get a better overall idea of the numbers involved gradually but for starters here is the note from AWS site:

"Amazon DynamoDB is an indexed datastore, and the amount of disk space your data consumes will exceed the raw size of the data you have uploaded. Amazon DynamoDB measures the size of your billable data by adding up the raw byte size of the data you upload, plus a per-item storage overhead of 100 bytes to account for indexing."

The other cost involved is the throughput. This is total reads/writes per second you will need, across all your tables. The minimum throughput allowed per table is 5 each (5 read and 5 write). That means that if you initially have say 7 tables, then your minimum throughput is (7 * 5) + (7 * 5) = 70.
Price for read throughput is $0.01/hour for 10 units.
And price for write throughput is $0.01/hour for 5 units.

Thus your total hourly price comes to $0.07 + $0.035 = $0.105/hour.
Assuming 720 hours in a month, your monthly cost for throughput is $75.6.

This pricing is a little steep. Which is itching me to go through the schema design and see what our starting costs are. I had initially not taken the minimum throughput per table into account; which was yesterday. Anyway, I will need a complete table design to judge anything. But one thing is clear: table count is costly if your tables are basic and do not need even 5 writes/reads per second.

Edit: I had calculated the sum to be $0.15 / hour. I am still not out of bed I guess :)

Mag will use Amazon DynamoDB

Amazon's latest announcement (and offering) could not have come at a better time for Mag. I am talking about Amazon DynamoDB here and I feel it is a good fit for Mag.

For the last few weeks I have been looking at many available options for the data storage for Mag. Notice I mentioned data store and not RDBMS. The reason is that Mag's data is really a large collection of configurations for campaigns that our clients run. Yes they may grow in size as we add customers, but the schema is quite simple, at least for now. And I would prefer having a hosted, cost effective service that is fast. I want to concentrate on the application because that is the real power of Mag.

My current choices included the NoSQL systems like MongoDB, Couch, Redis or similar, but honestly I would not want to handle managing the storage myself. And replication, and other stuff. I had made plans to do that though, but now I will be free from them. Plus the move to SSD seems very practical and needed. I am not saying SSDs will help Mag right now, but from the industry standpoint this is a good start.

But what has really made my decision easy is that it is no hassle, no management and it scales when needed. AWS also claims really low latency, very high throughput. Although our reads/writes will be tiny to start with. I will see the latency for myself once I start but I believe their claims for now. From a startup point of view the pricing is good too. Initially free to start with, although its a tiny 100MB database, but its good to experiment stuff. I used basic assumptions to check prices and it seems good to me.

I will see what existing libraries I can re-use to make the transition smooth. Our applications are JavaScript heavy, the back-end (PHP/Python/Node mix) is just an API as such. All rendering done right on the browser. Just JSONs passing here and there. That's it then, lets code!

Free web hosting with Amazon Web Services (AWS)

I have deliberately titled this blog post such that it gets more hits when people search one of the most common issues related to hosting: free web hosting :D. Of course Amazon does not provide just free hosting, its much more that than. But the general blogger, community leader, shop owner or similar is still more interested in hosting a WordPress, Drupal, Joomla or some e-commerce application. Let me explain a bit what Amazon Web Services is all about and what this free deal means.

Amazon Web Services is a collection of products offered by Amazon (yes the original book seller on the web) which allows users to avail compute power, storage, database, content delivery and other stuff with no up front commitment. You pay for what you use. These allow you to have your application hosted on Amazon infrastructure whenever you need. You do not need to buy resources in advance since Amazon gives you access to them when needed and charges you only for what you use. Lets take WordPress as an application. It runs on PHP, uses MySQL for storage. You may also need normal disk storage if you upload audio/video content or file attachments. What Amazon gives you is building blocks. You have EC2 for running WordPress. It behaves like a complete machine which you can log in to. You buy bandwidth on a per use basis. So basically you install PHP, MySQL and necessary stuff and then throw in WordPress and hurrah! Your own blog on Amazon's servers!

Now you may be interested in hosting your blog (WordPress) or content management system (Drupal, Joomla), custom website (on PHP/Python/Ruby, MySQL etc.) or e-commerce application (Magento, Ubercart). What we are going to do is get to use Amazon's products and run our website on them for free. How free? Well Amazon is giving a free tier to every new customer till about end of this year (2011). Go and sign up for the free tier; you may also read through their product details.

Confused about how these stuff enable to host your website? Well I will agree it is not as simple as buying a hosting account from a web hosting company. Amazon gives you lower level platform on which you can build your own application. But no need to worry, you can look at services like JumpBox who provide you easy starting points and ready to start machine images for most popular software. They are even supporting the free AWS tier with ready to lunch solution for WordPress, Drupal, MediaWiki and Joomla.

Experienced PHP Developer Needed

We need a very experienced PHP developer willing to work full time on a contract basis. This is telecommute work so you can work from anywhere. You will be working on new products mostly on Amazon Web Services platform. Required:

Linux, PHP, MySQL
Amazon Web Services: EC2, S3, SDB, SQS
Using Tarzan AWS PHP library
Experience in JS is a plus (AJAX will be used)

We are looking for experienced developer who can work in a distributed team environment.
Pay is very good to say the least.
We will prefer someone from Kolkata so (s)he can work from my place if needed.

Introduction to Amazon Elastic Compute Cloud (Amazon EC2)

Continued from Dummies Introduction to Amazon Web Services...

Imagine you need a web server. The hardware and the software, all together. You have a website which is growing in users and you want to make sure that the site grows along with the user numbers, that there is no downtime, and that the website stay as fast. You have a few things to do:

• Optimize the codes that run your website
• Optimize your software stack
• Get a better hosting account or a bigger server hardware
• Get more servers

Its the last two points we are interested in right now. Getting a better hosting account is easy if you have shared hosting account. If you have a dedicated server or a vds (virtual dedicated server) this is still easy task. But you may have to wait each time you want to upgrade your web server because it takes time to setup. Thus you need to plan ahead and take extra hardware so then when there are abrupt website peaks your site does not stop working.

Now since we are discussing EC2s which are like dedicated pieces of hardware we will assume the website does not run on shared hosting. So we are comparing with dedicated hosts or vds. Now getting extra servers is easy to say, but it costs if you want to predict usage (a digg effect, slashdot) and get extra servers. Also how much usage will you get? These maybe difficult numebrs to predict. It would be a lot easier if you could get servers when you need then, no hassle of installing you full software stack, and get it ready in a few minutes. Seems like the heaven of web serving? Well that is what EC2s are all about.

To start on what EC2s do in detail and how to use them from PHP (and other languages) its good to first read on Amazon's details about them (taken from their EC2 page, I was tired :P).

To use Amazon EC2, you simply:

• Create an Amazon Machine Image (AMI) containing your applications, libraries, data and associated configuration settings. Or use pre-configured, templated images to get up and running immediately.
• Upload the AMI into Amazon S3. Amazon EC2 provides tools that make storing the AMI simple. Amazon S3 provides a safe, reliable and fast repository to store your images.
• Use Amazon EC2 web service to configure security and network access.
• Choose which instance type(s) and operating system you want, then start, terminate, and monitor as many instances of your AMI as needed, using the web service APIs or the variety of management tools provided.
• Determine whether you want to run in multiple locations, utilize static IP endpoints, or attach persistent block storage to your instances.
• Pay only for the resources that you actually consume, like instance-hours or data transfer.

You will find those and other details on this page

Dummies Introduction to Amazon Web Services

Lot of you may have heard of this. Well certainly you have heard of Amazon.com and if you are into software/IT/web there is a good chance someone in your company is thinking of that mumbo jambo called cloud computing. Well what the hell is all this and then this web services?

Note: most of what I do is related to running websites. So whatever I write here will also relate to that. Beyond that I don't know nothing :|

Cloud Computing Well there is this thing called a computer. You know it right. And you know that you need them to run your websites or web apps or what else you call them. So what choices do you have? Well here are some:

1. you can take a shared hosting account
2. can take a dedicated server
3. you can take a hosted option to run your website
4. host the website from your own home/office
5. get virtual dedicated server (its like a small slice out of a server, each slice is independent of the other at the OS layer) (short: vds)
6. or you can take a cloud service

So you can assume the cloud service is maybe not radically different from the other options. I mean it still has to run your site or app. It just differs by the way you get the service, manage it, use it. Lets understand an example:

I need to host this blog.

I can take a shared hosting account which will allow me to run a site on a server that is being shared among many other users. Each user is running from his/her own folder. Each having maybe a few MySQL or similar database server users. Some hosts will allow to use a custom .htaccess file in your folder to use custom re-write rules etc. under Apache.

Now that seems a headache for a site which is just running a blog? Well take an account from WordPress.com or Blogger, LJ, etc. These are the hosted options.

I could also take a full dedicated server or a virtual server if I need, but I will to maintain it myself. Also getting a fully dedicated server setup takes sometime (1 day or more). The last option is that I take a cloud service. What I get is an independent slice like in a virtual dedicated host. But its more. A basic cloud service should give me access to the service fast. So a new slice fitted with 1GB or RAM/X Ghz of processor etc. should be up in just a few minutes maybe. This is same in vds. The most important part: all the setup, settings should be available through APIs. What was that? Well I should have a simple protocol (mostly http based) which I or my custom code or some app can access to create a new slice, or get a bigger slice if available, check status, and maybe other stuff. That is the most important part. The ability for software itself to create its own environment to run on! Confused? Even I am :P ... well don't worry, I will explain.

Now there are many examples of cloud services, but I have (sadly) enough exposure in just one: Amazon Web Services. So lets dig into it.

Amazon Web Services

Main website

Services listing:

Amazon Elastic Compute Cloud (Amazon EC2):

This is our primary service and we will inspect this first. In brief this is the service that allows you to get a server (hardwarde + software full stack as needed) up and running in little time, assign an IP address to it, set firewall and invite users to the website. Well except for the invite part the rest are all provided by Amazon and over an API. So my PHP code coulf do that! Of course I will be billed for it.

Amazon Simple Storage Service (Amazon S3):

Just store whatever you needed, setup access rules, and forget about it. Simple!

Amazon CloudFront:

This of these as the equivalent to you local post offices. So you do not need to go to the city central post office to fetch something. Local distribution of data! Its fast way to get things to the audience.

Well there are other services but I am not as good with them. When I experiment/use them enough I will blog on them.