The Great Australian Census Fail of 2016

Yesterday was the online census night. The previous census night was 5 years ago and the census was a paper form. This year, the ABS (Australian Bureau of Statistics) wanted a representative of every Australian household (there are about 9 million) to log on and do a 30 minute census form. On a Tuesday night.

From my experience, Australians go to bed early during the week. Long commutes, full time work, and after school activities mean that the weekday evening lasts maybe a couple of hours. Most Australians live on the east coast, and on 9th August 2016, Brisbane, Sydney, and Melbourne had the same time. Adelaide was only 30 minutes off. Perth was 2 hours off, so let's say it does not count. Population of Western Australia is only about 2.5 million out of 24 million for the whole of Australia.

Excluding Western Australia, we had about 8 million people (2.6 people per household) trying to access a website, and do a 30 minute form, within a 2 hour window. The window could have been more likely only a 1 hour window - people tend to have similar routines.

To accomodate that traffic, I would design and test the system for the worst case scenario: 8 million submissions in about a 15 minute window. Using a 15 minute window gives us a theoretical throughput of 32 million applications per hour.

What load did the ABS plan for? 1 million applications per hour. Any wonder it failed?

Planning for traffic 32 times heavier than ABS did, might also give us some breathing room in case of DoS attacks (not much really in case of a serious one), although to fight them off I would turn before-hand to a specialist web hosting company that offers DDoS protection.

There is also the question of design. I would not go with server-heavy JSP:

I would design the website to mirror the paper form (with a smart phone friendly option): static html5 delivered by CDN with client-side validation of data, no server-side validation on submission either, just storing the document. Processing of forms would happen later, using internal servers, and could take hours or days - as much time as it needed.

15.08.2016 Update:
By a security specialist: Censusfail and the fog of war

Siri porn query leads to a call from sex worker?

The weirdest thing happened today. I was looking for an address on my street, because I wanted to fix a delivery error of somebody's else letter. The address had my unit number, but wrong building or complex number. I asked Siri: "Show me 422 Pine Ridge Road". Siri understood that as "Show me porn", and replied, something like "I don't know how to respond to that".

A few minutes later I get a call with no caller ID. I answer, and a woman's voice starts telling me in a sweet voice "I have been waiting for you..." I disconnected. What was that? Did a sex worker or brothel (they are legal in Queensland) buy my Siri query? I never got a call like that one before, and the timing looked like it was no coincidence.

Apple, what are you doing with people's Siri conversations?

How The Mind Works by Steven Pinker

A good book. Sometimes a bit too wordy. Sometimes a bit off topic. Nevertheless, very interesting. It's not just about the mind. It is about evolution, genes, neural networks, optics, and mathematics of music. It is about reverse-engineering us. Why we act the way we act. How our senses work. 

To understand sight, we have to look to optics and computer vision systems. To understand movement, we have to look to robotics. To understand sexual and familial feeling, we have to look to Mendelian genetics. To understand cooperation and conflict, we have to look to the mathematics of games and to economic modeling.

Reverse engineering is the key to understanding how we work. Take morning sickness for example. Why is it part of pregnancy? A biologist, Margie Profet assumed that if it is there, it must be beneficial in some way, and she found it: 

1. Pregnancy sickness protects the developing baby from toxins: plant toxins in quantities that adults tolerate can cause birth defects
2. Pregnancy sickness starts when the embryo is most vulnerable to teratogens (birth defect inducing chemicals)
3. Pregnancy sickness ends when the embryo is past the critical stage of organ development and needs nutrients for further growth
4. Women with more severe pregnancy sickness are less likely to miscarry

Natural selection started with a replicator. Itself not a product of natural selection, but of chemical and physical processes. Replicators compete for resources. Copying errors that improve chances of reproduction are preserved by pushing out organisms that don't reproduce as well.

Confucius said "A common man marvels at uncommon things; a wise man marvels at the commonplace."

Steve Pinker is looking at commonplace human acts and wonders how they work, how could a robot do them.

Two related, interesting texts:
DNA seen through the eyes of a coder

The math equation for universal AI

The latter one is especially important. It may be describing how our intelligence works.

New observations allow AIXI to improve its world model, which over time gets better and better. This is the learning component. [...] 

The goal of AIXI is to maximise its reward over its lifetime – that’s the planning part. [...] 

[...] every interaction cycle consists of observation, learning, prediction, planning, decision, action and reward, followed by the next cycle.

How to Fail at Almost Everything and Still Win Big by Scott Adams

The full title includes: "Kind of the Story of My Life". Published in 2013.

Scott's recipe for happiness, condensed:
Healthy diet -> Energy -> Exercise -> More energy -> Learn new skills -> Flexible schedule

"Goals are for losers and systems are for winners"

Forcing yourself to do something, for example to eat right, or to exercise, takes a lot of willpower. The trick is to set up yourself and your environment, so the things that you want to achieve come without effort. How to do that? For eating: you could stack your fridge with healthy foods, use spices to make them taste great. For exercise: you need a routine - exercise must be done every day and must be pleasant. You should not overdo it the first day, and then be in pain for a few days.

I grew up in an environment that was setup for more healthy living:

• The pantry with the best food (sweets, sausages) was locked. My snacks were carrots and apples.
• I walked to school, shops, church. I took a train when going farther away.
• I played with other kids in the streets, in a park, and on school grounds.

Polish Proverbs - Part Three

1. Starych drzew się nie przesadza. Old trees are not to be replanted. Old people should not be uprooted. They should live out their days in a familiar environment.

2. Cicha woda brzegi rwie. Quiet water rips the banks (of a river). The quiet ones are sometimes the most adventurous.

3. Podróże kształcą. Journeys educate.

4. Cudze chwalicie, swego nie znacie. You are praising others' (things or customs), you don't know yours'. You are praising foreign things, and you are unaware of your country's achievements.

5. Potrzeba matką wynalazku. A need is a mother of invention.

6. Głodnemu chleb na myśli. A hungry person's thoughts are about bread. 

7. Lepiej późno niż wcale. Better late than never.

8. Pan Bóg nie rychliwy, ale sprawiedliwy. God is not quick, but fair/just. Similar to "You reap, what you sow".

9. Przyganiał kocioł garnkowi. The cauldron critiqued the pot. I understood it about the big one criticising the small one, while doing the same thing. It seems that it is not about the size, but about getting dirty. The full version of the saying is "Przyganiał kocioł garnkowi, a sam smoli" - the cauldron critiqued the pot, but it is sooting too.

10. W gorącej wodzie kąpany.  He who had a bath in hot (too hot) water. A quicksilver. Somebody who is quick to react, usually overreact, do something without thinking it trough.

11. Czego oczy nie widzą, tego sercu nie żal. What the eyes don't see, the heart does not regret. Sometimes it is better not to see or know about something.

See also: Polish Proverbs - Part One, Polish Proverbs - Part Two

Belgrade, Serbia

A short stopover in Belgrade, Serbia on the way to Warsaw.

Quick, random observations:

Serbia and Serbs are in some respects very similar to Poland and Poles:

• The looks: Serbs thought I was one of them - they would speak Serbian to me. I saw a few people in Belgrade who looked like doubles of people I know in Poland.
• The language: not close enough to understand when spoken, but often close enough to decipher the written form.
• History: the communism of course, although, unlike Poland, Yugoslavia was independent from the Soviet Union, but it goes further back. Serbia was a dominant force in a multicultural country that became officially known as Kingdom of Yugoslavia in 1929. The break-up of Yugoslavia in the 1990s left Serbia scarred, out of NATO, out of EU. I think Poland could face a similar fate, if after 1945 it was to keep its eastern parts with significant Ukrainian, Belorussian, and Lithuanian minorities. At some point, nationalists among these peoples could declare independence, and the clash with Polish nationalists could turn Poland into a pariah of the modern world.

Other:

• On the way from/to the airport: a big Gazprom ad with Russian and Serbian flags (same colours, just inverted.) declaring Russian-Serbian friendship.  
• Plastic bags in the forests along the road. I see that in Poland too. 
• Very few tourists. The airport to city bus driver didn't speak English. People did not know where the airport bus stop was at the central railway station.
• People in Belgrade look good. Cars look modern. Buildings, and public transport, not so much. The are modern Solaris buses visible, but most trams are ancient.
• Serbian currency is hard to convert. A currency exchange in Poland wouldn't take it.
• Big choice of tasty, cheap snacks at bakeries. 

Central Railway Station

Temple of Saint Sava

I remember such uneven asphalt from Warsaw of my youth

Looks like a WW2 era tram.

This could be in Warsaw too, although more likely 20 years ago, than now.

The NATO bombed ministry building finally being repaired.

This and many other trams are gifts from Switzerland.

Opposite the central railway station. Kvass and Adidas ads.

Panorama of Belgrade - no skyscrapers yet

Short-range Air Serbia planes at the Nikolai Tesla airport

Fields, no houses - very different from Poland. Strange shapes showing underground water?

Another look at Belgrade

Communist apartment buildings and lots of greenery. Could be in Warsaw too.

Similar to Poland, but bigger. Multi-family houses?

How to Talk So Kids Will Listen & Listen So Kids Will Talk by Adele Faber and Elaine Mazlish

by Adele Faber and Elaine Mazlish, originally published in 1980.

A great review of this book is here: http://blog.codinghorror.com/how-to-talk-to-human-beings/

Some of the techniques could also be used between adults, for example at work. To see if this works, I replaced "children" with "them". Here is the cheat sheet:

1. Helping Them Deal with Their Feelings

• Listen with full attention; don't half listen
• Acknowledge their feelings with a word "Oh", "Mmm", "I see"; don't question and don't give advice
• Give their feelings a name; don't deny the feeling
• Give them their wishes in fantasy; don't explain and argue using logic

2. Engaging Cooperation

• Describe the problem as a matter of fact, instead of: blaming, accusing, name-calling, threats, commands, lecturing, moralising, warnings, martyrdom statements, comparisons, sarcasm, prophecy
• Give information
• Say it with a single word
• Talk about your feelings
• Write a note

3. Alternatives to Punishment

• Point out a way to be helpful; don't yell, or threaten
• Express strong disapproval; don't attack character
• State your expectations
• Show how to make amends
• Offer a choice and then follow up with an action to prevent unwanted behaviour
• Allow to experience consequences, for example exclude from the activity next time

Advanced problem solving:

• Talk about their feelings and needs
• Talk about your feelings and needs
• Brainstorm solutions together
• Write down all ideas without evaluating
• Decide which ideas you both like/don't like/will follow through

4. Encourage Autonomy

• Let them make choices - as a practice
• Show respect for their struggle "it can be hard"; help, but don't do it for them
• Don't ask too many questions
• Don't rush to answer questions; encourage them to talk
• Encourage them to use external sources; so they don't need to depend on you
• Don't take away hope; let them explore and experience

5. Praise

Praise can make them doubt the praiser, lead to immediate denial, be threatening, make them focus on their weaknesses, create anxiety, be experienced as manipulation...

• Describe what you see or feel; don't evaluate "good", "great", "you are genius"
• Sum up using a single positive word: organisation, perseverance, self-control, friendship, loyalty, courage, willpower, flexibility, adaptability.

6. Free Them from Playing Roles

• Look for opportunities to show them a new picture of themselves
• Put them in situations where they can see themselves differently
• Let them overhear you say something positive about them
• Model the behaviour you'd like to see
• Be a storehouse for their special moments 
• When they behave according to the old label, state your feelings and expectations