Wednesday, August 10, 2016

The Great Australian Census Fail of 2016

Yesterday was the online census night. The previous census night was 5 years ago and the census was a paper form. This year, the ABS (Australian Bureau of Statistics) wanted a representative of every Australian household (there are about 9 million) to log on and do a 30 minute census form. On a Tuesday night.

From my experience, Australians go to bed early during the week. Long commutes, full time work, and after school activities mean that the weekday evening lasts maybe a couple of hours. Most Australians live on the east coast, and on 9th August 2016, Brisbane, Sydney, and Melbourne had the same time. Adelaide was only 30 minutes off. Perth was 2 hours off, so let's say it does not count. Population of Western Australia is only about 2.5 million out of 24 million for the whole of Australia.

Excluding Western Australia, we had about 8 million people (2.6 people per household) trying to access a website, and do a 30 minute form, within a 2 hour window. The window could have been more likely only a 1 hour window - people tend to have similar routines.

To accomodate that traffic, I would design and test the system for the worst case scenario: 8 million submissions in about a 15 minute window. Using a 15 minute window gives us a theoretical throughput of 32 million applications per hour.

What load did the ABS plan for? 1 million applications per hour. Any wonder it failed?

Planning for traffic 32 times heavier than ABS did, might also give us some breathing room in case of DoS attacks (not much really in case of a serious one), although to fight them off I would turn before-hand to a specialist web hosting company that offers DDoS protection.

There is also the question of design. I would not go with server-heavy JSP:

I would design the website to mirror the paper form (with a smart phone friendly option): static html5 delivered by CDN with client-side validation of data, no server-side validation on submission either, just storing the document. Processing of forms would happen later, using internal servers, and could take hours or days - as much time as it needed.

15.08.2016 Update:
By a security specialist: Censusfail and the fog of war

No comments:

Post a Comment