Frontend Driven Development

I have primarily been a backend web developer, started out with PHP around 2005 and then moving to Python in 2011. I have worked with a lot of JavaScript, mostly using jQuery and friends till 2013. One of the major JavaScript based projects I worked on in this time was an iOS (iPad in particular) based magazine product with native features like swipe, and columnar display of content. That was in 2011 and getting these done without the big JS libraries of today (2017) was a fair challenge. But since then I have focused more and more on the user side of the application but the real change came after the modern single page app frameworks went mainstream. I started out with Backbone.js, tried a bit of Angular.js for a while and finally settled with React.js in 2015.

Being a backend developer in love with Python, I was initially hesitant to move that much of the software to the frontend. I primarily focus on marketplace kind of applications, some of which are very good candidates for single page applications. To be clear, many marketplace applications will not benefit from single page applications. For example if you are just buying an selling stuff, and people stay on the app for a short time till checkout, then it probably makes less sense to build an SPA. Also, you probably need Google and other search engine bots to read your content. Then you might want to stick to backend driven applications and just print HTML.

But if your users are to stay on your application for while, to customize lots of options, to fill in many forms, get notifications, etc. then it probably makes sense to build an SPA. The main difference is this: are your users private and interactive or are your users public and mostly viewers? If your users are the first type, then SPAs make sense.

Web applications, even a few years back we not heavily customized depending on the user. But that is changing a lot. In our applications, we are trying to customize elements in the UI depending on who is seeing it, what pre-conditions they have etc. All of this is geared to make the users' flow as smooth as possible. Guide them through steps, give them important tips, store and act on their preferences and so on.

In traditional backend driven applications, all of this logic would be dealt with in the backend. But that is like expecting a piece of paper to behave like a display screen. You can surely print stuff on the paper and even erase and print again and so on, but the paper is a one-time-print medium. HTML printed from backend is kind of like that. The backend does not treat HTML as a first class citizen. But the browser does and the DOM is powerful. Sure it comes with a bag of worms, but good frameworks make that manageable. I do not have experience with many frameworks so I will stick to React.js here.

Let us take an example, a common one. A user clicks on a item, and has two options - "Add to cart", "Add to wishlist". In the backend driven application, you will make a page refresh, print a new page, with either a +1 in the cart or a +1 in the wishlist. In the frontend application, you just make an API request to update the cart or wishlist. On valid response, you just update your frontend state. The UI updates to reflect the +1 in the right place. Now when the user clicks either the cart or the wishlist icon, the UI shows the current list, which is already hydrated with correct data and actions are already bound. Then if the user wants to kick out an item, again its an API call to sync the backend data, while the frontend state track this change and UI reflects it automatically. Well the automatic part is happening due to the frontend framework.

The UI, as you may imagine, is snappy since the amount of data you are sending per request is ridiculously small compared to sending pages of HTML on every trip. Also the backend has much less work to do, since it is not longer bothered with UI state. The backend mostly becomes a DB with business logic and data validation system.

There is also an added advantage, something that big applications can use. Honestly I have not been involved in an application that has needed this recently. That advantage is load balancing! Since the frontend manages its own state, different parts of the application data could come from different backends, all behind a load balancer. Many of the backends may not need user session since it is sharing publicly available data. For example in my current project, we match students with online work. The list of online work could be considered public data, so we could have it served from a backend which does not track user session. But if the owner of a particular online work needed to change the description, the request would go through a session aware backend.

Users are getting used to quick applications. Even if you are small startup there should be no reason to not use these basic new building blocks and build applications that are more future proof. Pushing data from the backend into SPAs is also easier. Also, this is something users are very used to now - notifications even in web applications are fairly common now.

Now, all said and done, there are some things you have to understand. Building an SPA means you need people who can build rich SPAs and those who can build backends with API. These might be the same person(s) or different, depending on their skillset. Also deployment gets a bit more complicated. But the benefits really outweigh these blockers. And in the end "Least time to achieve my need" always wins. So if your application can benefit from being frontend driven, then go for it. Backend developers are a bit hesitant, but that is okay. This is big shift but has huge benefits.

Shift from mod_wsgi to uwsgi

The last couple weeks we have been thinking to shift from our current web serving platform which was powered by nginx + Apache + mod_wsgi to nginx + uwsgi. The reasons were that Apache was part of the plan for about a year and a half, but its main reason to be there was that Apache + mod_wsgi was stable. We of course wanted to keep nginx, but right now we had no reason to run the Apache + uwsgi.

I had the task to figure out what are people saying in general about nginx + uwsgi. So yesterday and day before I scouted the Internet for blogs or discussions regarding the same. And it turns out, as I hoped, that nginx + uwsgi is a leaner approach to the application serving platform. Also uwsgi seemed very well documented, managed and commercially supported as some folks mentioned. And there were quite a few articles detailing benchmarks comparing many platform choices for a Python based application. nginx + uwsgi fared better than the Apache alternative and almost everywhere I looked people mentioned nginx + uwsgi to be production grade. There are options which are better, but we preferred one which seems to have a larger following or community.

So we decided to take the plunge. Well the transition was smooth, or at least not troublesome at all. Most of the documentation is available here, here and here. Installation for us for very straightforward. We are on Ubuntu 10.04 LTS. We wanted to upgrade to the new 1.0 series. So we downloaded that, compiled and installed. Please remember you may need to change the config paths. We had to do some of that since our deploy + app reload scripts were dealing with the default Ubuntu install paths. After that just get uwsgi and again compile. That is it, you are done with the software part of it!

After that just follow the examples and docs I pointed out above. For us, we just had to change the proxy_ settings to corresponding uwsgi_ settings. Other than that, we needed to create ini files for uwsgi to run the apps properly. I could not figure out how to run a single instance of uwsgi that will read all stanzas from an ini file and run them all. Well so we are running two uwsgi daemons. Good enough for now, we will come back to the issue later on. Thus we have an updated AMI and we are actually flipping our application servers today. We do not expect any noticeable downtime. Well now lets get back to work :)

Problems with installing Yum on Fedora Core 4

"There is a problem importing one of the Python modules" : ever faced this error after installing an running yum. Well I faced that. And searched on Google for it. But in vain. Most results suggested the usual missing packages:

libxml2-python, python-sqlite, or python-elementtree

I tried installing all of them and no use. Then did lot of finding around and finally i tried to install other package managers. Finally I cam across a yum rpm package at rpmfind.net and while installing I got the dependency error for urlgrabber (http://linux.duke.edu/projects/urlgrabber/). Installed that and all was fine !!!