In Defence of Randomness
In a job interview, the dialog between the interviewer and I went like this:
interviewer: What does diversity bring to a company?
me: Randomness.
interviewer: I'm afraid I'll need you to elaborate.
Elaborate I shall.
Randomness isn't usually thought of as a good thing. Most people enjoy a tidy living space. Order. Randomness is the opposite of order. Chaos. Few people enjoy that. As Uncle Roger remarked, random is not a compliment. But I hope to change your mind.
Natural Selection
Natural selection works by a process of randomness. It works something like this:
- random things happen - this can be thought of as an actor that creates choas
- sometimes things replicate and propagate - crossover and mutation happen, as is typical in sexual reproduction
- unfit things get eliminated - this can be thought of as an actor that maintains order. Things that are too weird or don't make the cut based on some "fitness criteria" are eliminated.
- repeat
Over many iterations, those that survive become more numerous relative to the rest of the field.
Natural Selection in Technology
I didn't specifically mention biology above. Natural selection is a mechanism that happens not only in our biosphere, but also in our ideas, culture and technology.
In Guns, Germs, and Steel, Jared Diamond claimed the development of agriculture is the single most important step for a society's technological advancement. Agriculture:
- is a great technological achievement in and of itself
- makes it possible to feed many people efficiently
- gives people a reason (land) to forgo nomadism and live in one place
Over time, this gives rise to cities that reach unprecedented population density. Large population centers are fertile soil for technology:
- crowdsource effect: the more people, the more likely someone will come up with good ideas/inventions
- creations outliving inventors: the more ideas/inventions, the more likely they will survive beyond the lifetime of their inventors. Use any ideas/inventions from any dead people lately?
- idea crossover: as with biology, ideas/inventions evolve by crossover and mutation. "I took your idea and I made it better." - Parks and Recreation
Like animals and plants in the realm of biology, the ubiquitous technology we use every day are here because they are the ones that survived the on-going process of evolution.
Epidemics
Large population centers also give rise to deadly epidemics. Epidemics of the kind that
- kills its host
- is extremely contagious
would not be possible within a small population because the virus would quickly reach the entire population. Everyone would be either dead, or have an immunity — which would cause the virus to die out. The only way a deadly epidemic can survive over a long period of time is if the population is so vast that:
- it takes a while for the virus to reach everyone in the population
- the rate at which babies are born is fast enough to replenish the supply of humans for the virus to infect (p. 202 Diamond)
People are fertile soil.
Connectedness and Randomness
Diamond asked the question: what makes some societies more technologically advanced than others? He looked at a society's degree of connectedness. Australia had low connectedness with its neighbors due to its geography: this hindered its technological progress. China had high connectedness within its own territory due to its own geography, and that boosted its technological advancement (p. 416 Diamond). However, this eventually backfired because it lead China towards a dictatorship model of governance. After the emperor decided to close off trade with the west, China's advancement lagged behind Europe. While China had a high degree of connectedness internally, it didn't benefit from ideas of people outside China. Fewer ideas yielded less randomness. Diamond hypothesized that the optimal level of connectedness a society should have is not too low, but not too high either. In other words, it's the Goldilocks principle.
China's closing itself off from the west is a specific instance of how a society can slow its technological progress. There's a more general rule that can be gleaned from this. I think that in general, strict rule — such as that exerted by a powerful ruler — reduces innovation. The crowdsource effect only works if people feel free to pursue ideas and communicate them freely. When a person is controlled and restricted, their behavior become more predictable and less random. Less randomness leads to fewer different ideas to choose from, which leads to fewer innovations.
Randomness in the Mind
The above rule can be applied at many different levels: families, social gatherings, classrooms, organizations, churches, companies, governments, societies. But it can also be applied to a single individual. The role of randomness can be seen in the candle problem, a study that shows when people are nervous (money on the line) in a non-trivial task, their performance worsens. What we need in order to find the solution is a relaxed mind. It's the reason George Wein of the Storyville Jazz Club wants his musicians at their "relaxed best". A relaxed mind wanders to random places and toys with strange ideas. It entertains all possibilities, not just the most plausible ones.
Kids are More Random
As a parent, I've noticed that kids are better than adults at learning how to use software. This can usually be seen on the iPad. My children are more skilled than me at using applications like YouTube, Spotify, and games. They know more features and they are more efficient because they know more shortcuts. While adults just want to get things done, kids explore. If you show a kid a weird looking icon, they will click it. An adult may never know what that icon is their whole life.
Antisocials are More Random
Malcolm Gladwell introduced me to the story of Rick Barry's underhanded freethrows. Ricky Barry was a legendary basketball player. He is one of the top 10 all-time freethrow shooters. He does this with an underhanded throw, a.k.a the granny shot. Although it's not a manly shot, it gets him great results.
Wilt Chamberlain is one of the greatest centers in the history of basketball, but he was a terrible freethrow shooter. To prevent him from taking over the game, the opposing team would use his weakness against him: he was deliberatly fouled and sent to the freethrow line towards the end of close games. When he tried switching to the underhanded shot for one season, he did much better! In one game, he made 28 out of 32 freethrows: 88%! But the next season, the unthinkable happened: he stopped using the underhanded shot and went back to being a terrible freethrow shooter.
Over the years, different basketball players have come to Rick Barry for help improving their freethrow shots. Most of them became competent at underhanded freethrows only to go home and revert back because they were too embarrassed to use it in public. What makes Barry different? He doesn't care what others think of him.
Even when there are logically good reasons to do something, it does not necessarily win against public sentiment. A person who conforms to social norms has more predictable behavior. It takes an antisocial to go against social norms.
Emma's Triumph
My daughter's name is Emma. She is a special needs child. She is 14 years of age at time of writing and is not able to speak. She's not mute, but rather she hasn't developed the necessary mental machinery to develop spoken language. One day I was out with her taking a walk. When it was time to walk back to the car, she started pulling me in another direction. I was confused. Why did she want to go there? After a brief struggle, I pulled her along with me with all my strength, overcoming her resistance. When we arrived at the parking lot, the car was nowhere to be found. That's when I realized Emma was pulling me towards where we actually parked. I apologized to Emma.
The person who on paper should know better, did not. Instead, we should have listened to the underdog. Does Emma always know better than me? Probably not. Does the higher rated tennis player always win against a lower rated opponent? No. That's why they play the game. If I had been more receptive to randomness, I would have let Emma lead, or at least considered the possibility that she was right.
Randomness in Travel
Nowadays when traveling to a new city, we easily find the best spots to visit with an online search. The search engine has already sorted the results by user ratings, so the best float to the top. But this also means everyone goes to those same places. These places become crowded, and may require scheduled reservations or you'd have to be there early if you don't want to have to wait to get in.
I feel burdened by this requirement of timeliness. After all, I am on vacation, and it is a time to relax. This is what I prefer:
- Search only within walking distance of where I happen to be. If I see something interesting on the map, walk there.
- Drop myself off at a location of interest, say the town center. Turn off the google and look around. If I see something interesting, walk there.
Some of the top destinations are usually discovered by this method, while also finding some offbeat places. I may not be able to get into some crowded destinations. However, I get a warm feeling that everything I found was due to serendipity.
Randomness in Innovation
You might have heard of Isaac Newton's story of an apple falling on his head prompting him to come up with the theory of gravity. Innovations do not come only from hard work. There also has to be an element of novelty. My favorite demonstration of eureka moments are the pivotal moments in the show House where House — a brilliant doctor — freezes and goes into deep thought mid-sentence during a conversation, often prompted by something that was said. Moments later he would come up with the diagnosis for his patient and solve the case.
When a problem is non-trival — remember the candle problem — you will fail to find the solution trying the obvious approaches. In order to find the solution, all the pieces of the puzzle have to come together. The problem is: the puzzle pieces required are rarely seen or thought of together. If only the pieces were put right in front of you, you'd find the solution in no time. That's where randomness comes in: when House has a random conversation about something unrelated to the case, he accidentally finds the final piece of the puzzle he needs.
When encountering difficult problems in programming, I used to be headstrong and power through. The solutions would work, but they weren't always the most elegant. Now that I've grown older, I've learned to take it slow and let eureka moments work for me. I'd take a break and go for a walk in the woods. Often, eureka moments would happen during a hike, and something I thought was hard would suddenly become easy.
Parting Thoughts
interviewer: What does diversity bring to a company?
me: Randomness.
Is more randomness good or bad?