Innovation – Chaos and Order

We often lament that big technological and scientific breakthroughs do not come often enough from non-western societies; India being a case in point. China has started being an exception (which is an evidence in favour of my hypothesis, as you would see). The data, no matter how you measure and slice it, is pretty damning. On all metrices that you can possibly care about; publications, patents, university rankings, Nobel Prizes, Global Product companies; we often end up bringing up the rear.

I do not propose to say that I have either the reason that explains everything, neither do I claim that I have a silver bullet, that will solve this problem once and for all. All I have is an observation that leads to a hypothesis, that still must endure its trial by hard evidence. Here it goes!

My hypothesis is that we have a sweet spot of the ratio between chaos and order, where we are most comfortable. Too much order, and we feel straitjacketed, like in a jail. Look at the welcome movement towards self-governance even in nation states that have had a benevolent dictatorship.

Too much chaos, and all hell breaks loose, and you can’t get anything done. Socially and psychologically, we need a balance between the two extremes. We have a budget for chaos. Chaos in the right place is the mother of thinking out of the box and innovation.

What seems to happen in most developing economies and societies is that we spend our budget for chaos in the mundane inefficiencies of life. We have chaotic traffic and chaotic public utilities. We seek order in work and relationships. We try to tread the beaten path. No wonder, we can’t think of new stuff. Our relationships run deep and steady, but our well of innovation runs dry.

In western civilizations, public lives are incredibly ordered, like clockwork. In fact, that is the first thing you notice when you land on those shores. All the chaos thus flourishes in private spheres, leading to lot of innovation; and broken marriages.

The obvious question now is, is this just a flight of fancy, an attempt to shove causality into merely a correlational observation? Also, what led to this chasm between the western and the non-western civilizations? Is that just an accident of history?

Posted in Uncategorized | Leave a comment

How to be employable

I had a chance to address a gathering of about 150 “soon to graduate” engineering students a few months back. Here are my speech notes:


Transcribed and lightly edited:

Long term

1. Engineering is not domain specific and cannot directly teach you to do a job, and there is a very good reason for that. You should learn how to learn and that is the principal thing that hiring managers look for. Demonstrate your

initiative in looking up and exploring new things

2. Demonstrate an ability to learn-cultivate a habit of exploring and researching and learning about the engineering marvels around us

3. Develop depth and width Look at and explore all branches of knowledge. Understand and appreciate the value of the first year subjects from other branches that are a part of your curriculum Read a lot on diverse fields Eg Medical Electronics, Natural language processing algorithms and data structures in embedded.

4. Do not forget the basics. 1st and 2nd year subjects are where the fundamentals are built and explored. Always ask the question “What is the physical implication of what we are being taught”. Do not optimize your learning only for examinations. Often, topics that do not render themselves well to examination formats are the ones that are most applicable to the real world. While you are at it do remember what you learnt in school as well. That is helpful

5. Develop a mathematical acumen. Maths is the language of the universe, and of the business world. Learn to look at numbers and draw conclusions. Brush up your statistics.

6. Develop computer skills. I advise learning at least 1 programming language for everyone Do not limit your computer skills to FB and Instagram.

7. Develop communication skills, both verbal and written Start writing a blog. Grasp opportunities to speak in public.

For a specific job application/interview

1. Spell check and grammar check your Resume.

2. Do not lie on your resume on the skills section. There might be a listed skill in which the interviewer may have an expertise, and you may be grilled on that.

3. Prepare for the interview. Never go unprepared and hope to wing it.

4 Research the organization that you are going to interview for. It is always a big “no” when the candidate asks “What does this company do? in an interview.

5. Identify a few subjects and topics that are both relevant to the organization, and interesting to you. Prepare them well and try to steer the interview towards your strengths

6 Be careful with respect to your mannerisms when on the organisation’s premises. Dress professionally. Behave in a dignified manner. Be respectful to the interviewers

7. Take individual ownership in the project that you did. We did xyz is not valid as no one is hiring the entire team. identify the specific role that YOU played in the project.

These will probably of some help to folks out there!

Posted in Information, Knowledge | Leave a comment

The identity crisis stemming from technology progress and AI

There has been a lot of discussion over the technological aspects of AI and its direct short term impact on jobs and employment. However, there is a deeper civilization change that AI may herald, which is not often explored. But first, some background to prime you to the problem space.

Effort required for Living in comfort

There is a very useful way to determine the cost of something independent of extraneous economic factors such as currency, era, country etc. This is to determine how long does an average person (with the average per capita income) need to work to earn enough to buy a certain good or service. I also happen to think that it is one of the most representative of the human state as well.

It is well known in literature that the cost of stuff that we need to do to survive and live comfortably has gone down dramatically over the eons. This includes working for money as well as household chores. From an excellent commentary on

Valerie Ramey and Neville Francis (2009)3 have studied how work and leisure have changed over the course of the 20th century.

The chart below shows how hours worked in home production have changed.

The activities included in home production are: planning, purchasing goods and services (except medical and personal care services), care of children and adults (both in the household and outside the household), general cleaning, care and repair of the house and grounds (including yard work, but excluding gardening), preparing and clearing food, making, mending, and laundering of clothing and other household textiles.

The authors find that “there were significant declines in time spent in home production by women in every age group”. You can see the data for women broken down by age group here.

Max Roser (2018) — ”Working Hours”. Published online at Retrieved from: ‘’ [Online Resource]

Max Roser (2018) — ”Working Hours”. Published online at Retrieved from: ‘’ [Online Resource]

Almost identically, the working hours required for work shows an equally steep decline, compared to the agrarian age. Availability of leisure time is one of the major perquisites of our progress. It is not for nothing that Late Hans Rosling (of Gapminder and Ted Talks fame) called the washing machine the greatest invention of all time.

AI and Automation

The decline in working hours is a consistent long-term trend, and is likely to continue. The great advances in Artificial Intelligence and increased automation also threatens to affect employment and can render a large populace to be unemployed. Even as of today, AI has created more jobs than it has eliminated, but all the jobs it has created are skilled labour jobs or technical jobs, whereas the repetitive jobs done by traditionally unskilled labour force are being automated. With the impending advent of autonomous vehicles, there is a huge question mark over the livelihood of a large number of drivers and helping staff engaged in transport of goods and people. From

Automation is increasingly proliferating in every aspect of our lives, whether it’s robots building the cars we drive or artificial intelligence systems driving the vehicles for us. With the rise of autonomous systems, the big concern for many people is how their jobs will be impacted.

A recently published report from the McKinsey Global Institute think tank attempts to tackle that question. Although robots already can replace workers who do physical labor, such as miners and factory workersas well as those who collect and process data, like bank tellers and travel agentsthe report concludes that less than 5 percent of occupations are likely to be completely wiped out by automation. But that doesn’t necessarily mean job security for workers in such industries, according to several reports.


The effect of automation on jobs really depends on the occupation. A reportby the International Institute for Sustainable Development suggests automation could replace more than half of mining jobs in the next decade. The mining industry is already using automated loaders and tunnel-boring systems, and is testing fully autonomous long-distance trains to carry materials from the mine to a port, eliminating the need for workers to do these tasks.

Truck, taxi, and delivery drivers also need to worry. “Artificial Intelligence, Automation, and the Economy,” published in December by the Executive Office of the U.S. President, states that automated vehicle technology could threaten or alter 2.2 to 3.1 million of these jobs in the United States. That means 80 percent to 100 percent of these positions will be eliminated, affecting some 1.7 million truck drivers alone. On-demand car services, like Uber, likely will rely entirely on self-driving cars in the future, the report adds.

And those looking for jobs at a factory need to have computer skills now. Yet fewer than 15 percent of the 10,000 applicants who attended a job fair at Siemens Energy, in Charlotte, N.C., were qualified for positions at the company, with ninth-grade level reading, writing, and math skills, according to The New York Times. The article goes on to say John Deere also has trouble filling its factory positions, because building and fixing tractors and grain harvesters now requires advanced math and comprehension skills.

Those working physical labor jobs are not the only ones who should be concerned. Software capable of analyzing large volumes of legal documents is expected to drastically reduce the number of paralegals, according to a article. And as such software programs advance, people with other occupations, like accountants, could become easily replaced.

The question of identity

To me, the questions of widespread unemployment and unrest is a civilizational valley that we need to get out of, but if and when we do, what happens?

When someone asks you for your introduction, one of the major things that you say about yourself is “what do you do for a living”. That is a source of our self worth and identity. And why not, when a majority of our waking hours are spent at work, and most of our social connections and relationships stem from the workplace. But does it need to be so?

Traditionally, as a financial ecosystem, the prosperity of any given civil unit is governed by the wealth creation per capita, as well as the availability of leisure time to enjoy the fruits of the collective labour. This manifests itself as the various kinds of signalling that we undertake to demonstrate our worth to others, which is largely proportional to the wealth that we have created for the society itself. Extrapolating the premise of reduced working hours and increasing leisure time for the same wealth creation potential, it is reasonable to assume that most of the members of our society may not *need* to work for a living, if sufficient wealth is being created using automation. This will tie into the already existent concept of universal full basic income (UBI). According to Wikipedia:

basic income, also called basic income guaranteeuniversal basic income(UBI), basic living stipend (BLS), or universal demogrant, is a type of program in which citizens (or permanent residents) of a country may receive a regular sum of money from a source such as the government. A pure or unconditional basic income has no means test, much like Social Security in the United States.

An unconditional income that is sufficient to meet a person’s basic needs (at or above the poverty line), is called full basic income, while if it is less than that amount, it is called partial.

Given that the wealth creation by increased automation is sufficient to ensure a UBI for all the citizens, it opens up a wealth of possibilities. How many people do you know who loath their jobs but have to persist doing the same all their lives for obvious reasons? Now, if those people are assured of living comfortably, vistas of human well being open up.

Would you be able to hold an employee to do your drudge work if they could just hop off to their life’s calling without risking their ability to put food on the table? Employee engagement takes on a new importance here. Also the need for meaningful work, with the Autonomy, Mastery and purpose (Dan Pink’s Drive, for more info on these) baked into the assignment. Lest you do that, you risk losing people to other pursuits. However, would there be people who choose to do *nothing*? I think, no, but some people think ,yes. Time will tell. To risk sounding biased towards a “no”, I think the hitherto non-lucrative creative work will come to the fore in this situation. Probably people would wish to pursue fine arts, or science or anything that elevates the human condition without risk. Would that be true freedom in the financial sense?

How does our education system address these issues and the ability to live and thrive in such a society of abundance that the search of meaning in life is divorced from the mundane considerations of putting bread on the table? Studying for the exams and the subsequent piece of paper may well be passe. The really successful and happy people will be the ones who know where their life’s passions lie and are willing to pursue those. They will be people with the right questions, even if they don’t have the right answers.

All this is very Utopian, but it is the intermediate steps to getting there that worries me!

Posted in AI | Leave a comment

Erlang’s Queuing theory applied to Capacity planning for a Development House

Any development shop has multiple functions or verticals, separately staffed and having individual capacity and service profiles. For example, a typical SW development shop will have Requirements analysts, Designers, Developers, Testers, Release and configuration management people, and support people. Each of these will have their individual service times with respect to a standard project sizing unit, and individual capacities.

Traditionally, the capacity planning of staff and equipment is done on the basis of typical gut feel and qualitative weighing of various factors such as peak loads and acceptable wait times. However, a gut feel is retrospective and there is very little opportunity for predictive and quantitative understanding of how the resources can be shuffled and optimally reallocated.

Enter a mathematical tool that is often used in various other applications such as networking and service industry design, Erlang’s Queuing Theory.

A very nice writeup on the same appears here:

The average wait time for a new customer (any job that comes from downstream to upstream)


the time between customer arrivals is random with mean time λ. The Service time of each customer is exponentially distributed with mean time 1/μ. The number of servers is s. Server utilization is ρ.

This is the standard canonical form that is a part of most modern spreadsheet programs. The assumption of a Poisson Distribution can be borne out by past data, and the expressions re-derived. This, coupled by some off the shelf optimization software can throw out capacity numbers that minimize wait times at every stage. This approach is worth exploring.

Posted in Uncategorized | Leave a comment

Enlightenment Now, Steven Pinker’s Magnum Opus- a review

One of the authors that I have taken to like a duck to water has been Steven Pinker. He has a style of writing where he breaks down complex issues and erects a data and evidence based structure to address the parts before knitting it back up. That makes for an exhilarating experience to read and ponder.

The latest book that he has come up with is “Enlightenment Now- A case for Reason, Science, Humanism and Progress”. The essential thesis is that we are much better off due to these enlightenment values irrespective of what the doom’s day prophets of the mainstream media would have us believe.

The table of contents of the book reveals the intellectual courage of Pinker to take on a expansive swath of topics with a range of impact that is almost unprecedented. You could have had a book written on each of these topics ( and many exist); but still none of the topics feel shortchanged for detail and hurried. The style of the author is that he very carefully lists all arguments that any detractor could make and addresses each one of them in great detail. This gets tiring in a (very) few places, but overall it makes the points bullet proof. Another evidence for the importance of the metaphorical Brakes in the vehicle of civilization.

Photo by Danka & Peter on Unsplash


  • LIFE

The essential take away from this is that problems are solvable. That requires that we look at the data to inform our opinions rather than preconceived notions. That does not mean that they will solve themselves, but it does mean that we can solve them if we sustain the benevolent forces of modernity that have allowed us to solve problems so far, including societal prosperity, wisely regulated markets, international governance, and investments in science and technology. That’s a solid message to keep in mind in these times of doom’s day prophets.

Posted in Uncategorized | Leave a comment

Is Human Intelligence somehow special?


The artificial Intelligence community is in the news now-a-days, mostly for the right reasons. The resurgence of research in this area has been explosive, and that is not an exaggeration.

A proxy for the uptick in research is the number of cites some pioneers of the field have gathered over the years.

Citation count for Geoffrey Hinton

Citations for Yann LeCun

With such rapid progress, there are obvious concerns over AI safety and our future as a species with these technologies, which is a valid and pertinent area of concern. There is some fabulous work going on there, and you should check out the research around the alignment problem.

However, there is another faction that is dismissive of intelligence coming out of “mere” machines.

The argument goes thus: The modern machine learning and AI is just….. something, and will never be as good as human intelligence. Let us try to understand this argument, assuming that it *IS* a valid argument, not an argument from incredulity.

Arguments from incredulity can take the form:

I cannot imagine how P could be true; therefore P must be false.

I cannot imagine how P could be false; therefore P must be true.

Arguments from incredulity happen when people make their inability to comprehend or make sense of a concept the content of their argument.

This is just a bunch of matrix multiplications, how can it ever achieve human intelligence level?

It can, and it has, in several areas. The image recognition capabilities of modern AI far outstrip human performance. So does the chess playing ability. So do many other areas. When you really get into it, your actual neurons are also a bunch of electrical impulses calculating some mathematical function, phenomenologically speaking.


Machines and AI can never do X, where X is your favorite thing that AI can’t do.

Agreed, that it can’t do X *yet*. Maybe it can, maybe it can’t. There is no way to know unless you try. Unless you believe that the robots made of meat are somehow fundamentally and irreconcilably different than robots made out of silicon and steel, this argument seems untenable. Also, funnily enough, if you define AI as something computers haven’t figured out how to do, you are of course correct. Computers can’t do what they can’t do, because if they do, it’s not AI.

Is intelligence substrate dependent? Is there something special in carbon atoms that silicon atoms can not do, even in theory? The jury is still out one way or another, and I will be surprised if the narcissistic viewpoint that human intelligence is somehow special, turns out to be true.

The reductionist viewpoint (just a matrix multiplication, just an electrical impulse, just glorified curve fitting, just… you get the point) assumes that things are more complex on the larger scale than the smaller one, whereas the physical reality is often the opposite of that. Understanding something enough to make use of it is often simpler than understanding every last detail.

Posted in Uncategorized | Leave a comment

Artificial Intelligence and Deep Learning

Submitted for publication in a magazine for restricted circulation within the employing organisation, all rights reserved. ©

In the recent years, there have been phenomenal advances in the state of the art in Deep learning and Deep neural networks, one of the most promising directions for Artificial Intelligence. Most of you would have seen the Robot “Sophia” holding her own in conversations with humans. She (it?) was recently given citizenship in Saudi Arabia as well.

Less fancy but more useful applications have transformed our lives in many subtle but important ways.

Neural networks are one of the most beautiful programming paradigms ever invented. In the conventional approach to programming, we tell the computer what to do, breaking big problems up into many small, precisely defined tasks that the computer can easily perform. By contrast, in a neural network we don’t tell the computer how to solve our problem. Instead, it learns from observational data, figuring out its own solution to the problem at hand. So, in essence, instead of writing detailed instructions for the computer, you show it the output you want and let it figure out the instructions by itself. Isn’t that how you teach a human?

The power of a deep neural network comes from the possibility to learn very complex non-linear functions. For example, the following shows the impact on revenue from consumer churn.

Automatically learning from data sounds promising. However, until 2006 we didn’t know how to train neural networks to surpass more traditional approaches, except for a few specialized problems. What changed in 2006 was the discovery of techniques for learning in so-called deep neural networks. These techniques are now known as deep learning. They’ve been developed further, and today deep neural networks and deep learning achieve outstanding performance on many important problems in computer vision, speech recognition, and natural language processing. They’re being deployed on a large scale by companies such as Google, Microsoft, and Facebook.

What is a deep neural network? It is the name we use for “stacked neural networks”; that is, networks composed of several layers. The layers are made of nodes. A node is just a place where computation happens, loosely patterned on a neuron in the human brain, which fires when it encounters sufficient stimuli.


Deep neural networks have surpassed humans in many image and language processing tasks already but they are still narrow intelligence rather than general intelligence. The frontier of research in the field is moving extremely fast. Some beautiful examples of generalization can already be seen ( and
) but there is still a wide gap. The future is indeed exciting.


Posted in Uncategorized | Leave a comment