The Mugwumpery

There’s a story on Slashdot today about “a complicated pattern that has to do with the way humans do violence in some collective way“.

Surprise.  The size and frequency of terrorist attacks follows a power law – lots of little attacks, a few big ones.

What doesn’t?  Quoting from http://en.wikipedia.org/wiki/Pareto_distribution:

The sizes of human settlements (few cities, many hamlets/villages)

File size distribution of Internet traffic which uses the TCP protocol (many smaller files, few larger ones)

Clusters of Bose-Einstein condensate near absolute zero

The values of oil reserves in oil fields (a few large fields, many small fields)

The length distribution in jobs assigned supercomputers (a few large ones, many small ones)

The standardized price returns on individual stocks

Sizes of sand particles

Sizes of meteorites

Numbers of species per genus (There is subjectivity involved: The tendency to divide a genus into two or more increases with the number of species in it)

Areas burnt in forest fires

Severity of large casualty losses for certain lines of business such as general liability, commercial auto, and workers compensation.

I could add a bunch more, but won’t bother.

Why is this considered news?  Why does it get published in Nature?  If terrorist activity didn’t follow a power law, I think that would be interesting enough to merit publication in a prestigious journal.  But this?

Is it just me, or is the quality of editorial work in science journals dropping?  I constantly see papers in Science and Nature that make the most basic scientific mistake possible – confusing correlation with causality.  And then the “quality” press such as the New York Times and the Economist pick it up and repeat the same nonsense.

See also:

• Zipf’s law
• Benford’s law

I see today that the MIT Red Balloon Challenge Team has won the DARPA Network Challenge, in a little less than 9 hours.

In case you don’t know, the challenge was for a single individual to report the locations of 10 red weather balloons moored, for a few hours yesterday, at random public places around the USA.  The idea was to see if social networks can be used to quickly coordinate the solution of difficult distributed problems.

As has become common, DARPA offered a prize of $40,000 to the winner.  This is a tremendously efficient use of government funds to do research and get challenging projects accomplished – surely it cost DARPA more than the $40k prize money just to deploy the 10 balloons and setup the website about it.  If they’d contracted the problem to a defence research organization, it’d have cost them 50x the amount or more.

The MIT team won by organizing thousands of people to report their observations, with the promise of a cleverly-allocated share of the prize money.

Goal-driven prize competitions just about disappeared for 50 years after World War II.  Prior to that, they were a common and successful way of encouraging challenging achievements – the British longitude prize that led to the development of the chronometer and the Orteig aviation prizes including the one won by Charles Lindbergh for his non-stop transatlantic flight are among the most famous.

Something about the Depression or World War II led to the decline of these prize competitions – I suspect a loss of confidence in the ability of individuals vs. large organizations to accomplish great things had much to do with it.

The Ansari X-Prize – $10M to the builder of a reusable, manned, sub-orbital spacecraft – was organized by Peter Diamandis in the 1990s,  and won by Burt Rutan (with funding from Paul Allen) in 2004.  This seems to have been the turning point for the return of such prizes.

Years ago at a business meeting in France, I said something nice about competition – some truism about how it’s a good thing and should be encouraged.  One of my French hosts mentioned to me later, “you know, in France, we are taught that competition is a bad thing”.  She was entirely serious – to her, competition was the opposite of cooperation (not of monopoly), and could only lead to a cutthroat society of winners preying on losers.

As Eric Drexler likes to say, what most people forget is that almost all competition is about who can cooperate the best.

Economic competition between firms is about who can best cooperate with customers, suppliers, and employees.  Those who cooperate best grow bigger, while the “losers” release people and and resources, making them available to other, more cooperative, organizations.

Perhaps the only significant exception to “competition-is-about-cooperation” is athletic competitions such as the Olympics – which, oddly, enjoy a much more favorable popular opinion than does economic competition.  Somehow people understand that in the field of athletics, there can be no excellence without competition.

But competition is far more important to society in the field of economics – without it, every business would resemble the Department of Motor Vehicles.  (A succinct description of Soviet communism, perhaps.  Or American public schools.)

The renaissance of prize competitions makes me optimistic about the future of American society and culture.  If both private and government organizations can use prizes to spur achievement, and be proud of it, the US may be in the process of getting it’s mojo back.

Happy Columbus Day, all – the day we celebrate the last man to discover America.

As most people know, Christopher Columbus was by no means the first person to discover America.

Aside from the early visitors and settlers of the Americas whose names are lost to history, and the well-known explorations of Erik the Red and his son Leif Erikson, there may well have been earlier visits from Europe, Africa, or China. It would be surprising if, in the 10,000 years prior to 1492, there weren’t dozens of “discoveries” of America by people from across the ocean.

But that doesn’t diminish the historical importance of Columbus one bit – on the contrary, it makes him all the more pivotal to the history of the world.  Because Columbus was the one, and only, last discoverer of America.

Before Columbus, the Americas were to the rest of the world at best a legend – a story of faraway islands.

After Columbus, the Americas were, definitively, discovered.  They were a known place, not a rumor.

And that only happened once.

My wife and I were comparing school experiences today.

I grew up in Massachusetts, she in a communist country.

My experience was horrible. I was in high school in the 1970s. The teachers were droning idiots (mostly), the students asleep most of the time. I learned, to a first approximation, nothing. (What little I know I learned outside of school; I’m lucky to be one of the few who is able do this.)

Her experience was totally different. She was challenged and, like most Europeans of my acquaintance, learned a lot in school.

Why?

I think it’s the quality of the teachers. In the US, teaching is a lousy job. It pays poorly and has low social status. Teachers are not respected.

In her country, teaching was (at least at that time) a highly respected and well-rewarded profession (as communist jobs go). So the best and brightest were attracted to the teaching profession. Both her parents were high-school teachers (biology and mathematics). Her father, retired more than 20 years now, is still called “professor” when he goes around town (with respect, not irony).

Naturally, there are good reasons why teachers aren’t respected in the US. They aren’t respectable. When I was graduating from high school, the best students were going into law, medicine, or engineering. The middling students went into business or the arts. The worst – the bottom of the barrel, the ones who could barely get accepted into any college at all – into teaching.

Because they could. There was essentially no competition in the teaching profession; anyone could get in.

It’s a vicious self-reinforcing cycle – teaching pays poorly, is low-status, so no capable person wants to be a teacher. So only the incompetent and dull become teachers. And so teaching becomes even less attractive as a profession – your colleagues are drones, the pay is poor, the administrators (drawn from the ranks of teachers) are idiots, and your friends think you’re a loser.

It gets worse. The incompetent are insecure (understandably so) and so push for unionization and bureaucratic rules to make the job less challenging, and oppose all efforts to measure or reward excellence, as they know they have little of it. (With exceptions, of course. There are always a few outliers, but early in their careers most of these either quit in frustration or are scooped up by private schools.)

So why was the situation different in communist Europe? One of my themes on this blog is the importance of competition as a necessity for excellence, yet communist primary schools, like American ones, didn’t compete with each other. (Actually they did, a little, but I think this was a minor factor.)

In less-developed countries, most people have to do manual labor – work in the fields, a factory, etc. Only a very few can hope for a profession (or even office work). These coveted slots are reserved for the best and brightest. Teaching is well-paid compared to most other jobs (and offers lots of vacation time). As a result, there is great competition to enter the teaching profession. Bright people become teachers, do a good job, and earn the respect and admiration of students and parents. They are, rightly, seen as the best, and have high social status.

A wise farmer doesn’t eat his best produce – he saves the best seeds to plant next year. We in the US have been using our worst, instead of our best, to educate the next generation, and we are reaping the rewards.

The situation in colleges and universities is completely different – US higher education is widely considered the best in the world. Why? Because universities compete with each other for professors and students. Judging from the amount and intensity of marketing materials my high-school senior son received, running a university is incredibly profitable (a racket, I suspect, but that’s for another essay). The tremendous competitive pressure forces universities to seek the best professors.

Well-paid and well-perked, being a university professor in the US is a good job that attracts highly competent people – who compete with each other for the few tenured professorship slots. Again, the self-reinforcing cycle works, but this time in the positive direction: Competent, articulate, erudite professors enjoy high social status and (reasonably) good pay.

Yet the lure of alternative professions detracts from the quality of teaching even in US universities – if 150 years ago professors were drawn from the top 0.1% of minds, today they are drawn from the top 20%. But this is a far cry from the situation in primary schools.

What is to be done?

The simplest solution is to make teaching an attractive profession again.  If teachers were paid on a scale similar to, say, lawyers, far more competent young people would be attracted into the profession.  To throw out rough numbers, say $150,000/year for starting salaries, rising to $350,000 – $400,000 for a senior teacher with an excellent reputation.

It would take a generation to make the change.  At first, only the mercenary would be attracted.  Over time, as the social status of the profession improved, others would be attracted as well.

In the early days of the change there would be understandable outrage at the idea of paying such sums to the existing incompetents.  But what alternative is there?  Any system that limited the new salaries to the competent would be bitterly fought by the powerful teaching lobbies.  And if the new salaries were offered only to new teachers entering the profession (in the unlikely event that such a scheme was politically possible), this would drive the few existing competent teachers out of the schools in protest at the unfairness.

Of course, given a fixed budget, there is always a necessary tradeoff between quality and quantity.  If teacher salaries are raised by 4x, class sizes must increase by the same amount.  So the change needs to start at the highest grades, in the high-schools, and gradually work its way toward the lower grades, as the public becomes used to the idea of larger class sizes.  (Numerous studies show that teacher quality matters far more to educational outcome than class size.)

At some point, perhaps below 5th or 6th grade, it may be better to keep the existing low-paid teachers and small class sizes.  Young children are not capable of learning very much (my mathematics professor father-in-law started school, illiterate, at the age of 14; but that’s another story), and probably need more supervision.  And it doesn’t take a tremendous intellect to teach young children.

One way to get from here to there is through competition between schools, however created.  If schools had to compete for the custom of students and parents, the successful competitors would have to improve outcomes through better teaching (and the unsuccessful ones would disappear, as they ought to).

But as the communist example shows, vast improvement is possible even without competition between schools – as long as the incentives are put in place for competent people to compete to become teachers.

I love when seemingly disparate things synchronize in unexpected ways.

According to The Legal Underground,  Nobel laureate Friedrich von Hayek (1899-1992) “was exceedingly fond of t-shirts, especially those portraying images of Mick Jagger…”

Surprising enough; doesn’t fit our image of Hayek.

But it gets better.  Sir Mick is a fan of Hayek (see 3:35 in the clip below):

Before quitting to start the Rolling Stones, Jagger attended the London School of Economics, where Hayek had taught.

Supposedly, Jagger’s adviser at the LSE “said that Mick Jagger did a careful net present value analysis of the value in attending LSE as compared to the foregone revenue from playing rock and roll. When the dollars came out higher for music, Mick came by and apologized to the adviser, but said he couldn’t afford to continue in school; LSE was just costing him too much money.”

Did I mention that Jagger also owns an Enigma machine?  The rare 4-rotor type.

Now if I could just work Salma in there somehow, it would be perfect.

Here’s another letter to The Economist, this one from 2006-11-27.  I didn’t expect them to publish it, and they didn’t.  But I had to get it off my chest (hey, that’s why I post here, too).

SIR –

Tongue-in-cheek, David Crawley suggests a defence against alien invasion [Letters, 18 November].  If it were possible to evaluate and counter the capabilities of aliens, such a plan might be wise.  Unfortunately, any hostile aliens able to bring a force to our planet are likely to be so advanced as to make any defence we might offer entirely ineffectual.  (We have no way to estimate the likelihood of such an attack, as we have no information about the distribution of intelligence in the universe. ) Yet the absence of radio signals from other stars is, if anything, ominous, as we have known since Copernicus that Earth’s situation is in no way special or unique – if our neighbor’s transmissions have been suppressed, perhaps ours will be as well.

That said, a defence against global warming [the real subject of Mr. Crawley’s letter] is not in the same category – there are remedies such as increasing the Earth’s albedo (reflectivity) – a requirement that future roads and rooftops be painted white would be an inexpensive start, or reducing the sun’s heating of the Earth, for example by placing large inflatable sunshades at the Lagrange point between the Earth and Sun.  Others will have better suggestions.  We do not need to freeze in the dark.

This guy Crawley sent a sarcastic letter criticizing governmental action on “the risk of something really catastrophic” resulting from global warming, because “only a minority of scientists perceive this as a threat and the costs of such a defence are enormous”, then comparing it to the results of an alien invasion.

I thought it was a lousy analogy, and said so above. I should have avoided getting side-tracked with a discussion of Berserkers (scary and interesting as that may be, The Economist is not ready for it).

While I’m at it, here’s another letter to The Economist, this one on 2009-01-09.  They didn’t print it, altho I got a very nice reply from the science editor.  Maybe I was a little late getting it in.

Dear Sir,

In “Darwinism”, 2008 December 20, you report that people tell
researchers “they would rather be richer than their peers even if
that means they are absolutely worse off”.

It is one thing to benefit oneself at the expense of others – but to
harm others even at one’s own expense, surely that is the very
definition of evil.

Letter to The Economist sent 2009-05-03.  I’m guessing they won’t print it – it’s too far from the mainstream.  (Also, my Hayekian slant is showing; few people get it.)  We’ll see.

Update May 25 – I was wrong.  They printed it in the May 14 issue, with some unfortunate editing.

Dear Sir:

You say (Briefing – Central Banks, 25 April) “The business cycle was supposedly subdued, yet the world is in the deepest recession since the 1930s.”

Indeed.  Perhaps there is a connection.

For many decades, forest fires were suppressed by well-meaning officials.  Fuel was allowed to build up, eventually resulting in far more devastating fires than would have occurred naturally.  Today we allow fires to serve their necessary functions, while making efforts to limit damage to people and property.

I suspect the business cycle, and attempts to subdue it, are much the same – deadwood must be cleared out, inefficient practices curbed.  The economy, like an ecology, is far more complex than we can comprehend; attempts to control it are more than likely to produce unintended results.

From The Big Bonanza: An Authentic Account of the Discovery, History, and Working of the Comstock Lode by Dan DeQuille (Hartford, American Publishing Company, 1876):

CHAPTER LIII. HOW FORTUNES ARE MADE AND LOST

Bulls and bears – Doings of the brokers – On a margin – “Pussycat Tilde” and “Bobtaile” – Going up! – Dealers and dabblers

During the prevalence of a big stock excitement, times are lively along the Comstock range. Virginia City then hums like a Brobdingnagian beehive. All who failed to make fortunes on the occasion of previous excitements in stocks are going to do better this time. They have seen how these things work and this time are going to sell when they can do so at a fair profit. They don’t want the last cent; they will give someone else a chance to make something.

This is the way they talk at the start. As soon as there is a marked advance in stocks, however, they will be heard to say: “As soon as I can double my money I am going to sell.” In three days from the time of their making this assertion, stocks have taken such a jump that they could sell and double or more than double their money. Everybody is saying, however, that they are not selling for half what they are worth; that they will sell for twice or three times present prices be­fore the end of another month.

The men who were intending to sell whenever they could double their money cannot think of doing anything of the kind as things are now looking. Instead of selling they become excited, put up their stocks (which they had probably bought and paid for “out and out”) as a “margin,” then put in all the money they can raise besides and buy as many shares of their favorite stocks as they can in any way manage to secure. Stocks still go up, and each day these dabblers will be found counting their profits. They have invested largely in the low-priced stocks of “outside mines” – mines in which nothing of value has yet been found, but mines in which, all are saying, grand developments are liable to be made at any time – mines, in short, which in dull times are generally designated as “wildcat.” The masses – the servant girls, chambermaids, cooks, hostlers, washerwomen, preachers, teachers, hackmen, and draymen – are wildly and blindly buying these low-priced stocks, and from day to day they are going up “with a rush,” and everybody is getting rich. Our men who only “went in” to make a fair profit now tell you that they made yesterday ten thousand dollars; today they have made fifteen thousand dollars, and in a week or two they will say that they are worth a quarter of a million, half a million, or a million of dollars.

But they are not going to sell yet; no indeed – the rise has only com­menced. Pretty soon stocks fall off a little. Never mind, tomorrow they will do better. Tomorrow they are still a “little off,” as is said when stocks are going down. The next day they are rather “soft,” which is the same thing as a “little off.” However, that is all right. Our dealers – amateur speculators – have some points given them by a friend who is on the inside. A development is about to be made in a favorite mine. The “bears” are trying to break the stock; but they can’t do it; no, sir! – impossible. Too much merit in the mines at this time. All will be up and “booming” in a day or two. Next time you shall see them go higher than they have yet been seen.

Our men who started in to make a fair profit might yet sell and double their money – much more than double it – but they are not going to do anything of the kind. They are going to wait till “things take a turn.” The “bulls” will soon make a grand rally, and when things go up again, our men will sell. They admit that they should have sold when their stocks were all up before, but never mind! they will go to the same figures again in less than a fortnight, when they will be sure to sell.

There does come a “spurt,” and for a day or two there is a cheering improvement in prices along the whole line. Faces brighten and everybody talks of all stocks going higher than ever.

All at once everything is again “soft,” the next day “softer,” and the next decidedly “off.” It is then said that someone in the “bear” interest has been telegraphing to the “Bay” (San Francisco) a pack of lies about the mines, and the “bears” “below” (at San Francisco) have made use of these lies to get up a “scare.” Never mind! the scare will be over in a day or two.

But stocks still go down. Then it is said that some big dealer is “unloading” and there is talk of a “crash.” Still our men who started in but to make a “fair profit” do not feel like taking thousands, when they might a short time before have taken tens of thousands of dol­lars. They still hold on, saying that even though one or two big deal­ers are unloading, the big men among the bulls will “stand in” and take all the stocks that are offered. Also, they will have some points from a friend “on the inside” and developments are about to be made in one or two of the mines that will make all who have sold “very sick,” particularly those bloodless demons who have “sold short.” The “shorts” will have a merry time of it when they come to “fill.”

Thus matters stand when suddenly there comes what looks very much like the beginning of a “crash.” The “bears” are all diligently crying: “Stand from under.” Many persons become frightened and throw their stocks upon the market. Down go prices and soon “soft” is no name for it. The masses – the tinker and the tailor, the preacher and the teacher, the hostler and the waiter – rush in to try to “save themselves” and there is seen a grand and unmistakable crash. Brokers are calling on all sides for “margins” to be “made good,” and men are rushing about trying to raise money to “put up” in order to prevent their stocks being sold at less than cost.

They perhaps raise the money required and for a few days breathe again, when there is a further decline in stocks, and the brokers are again sending notes to their customers telling them that if they do not put up more money they will be sold out. Sooner or later there comes a time when the customer can raise no more money, and his stocks are thrown into the market by the broker – in whose hands they re­main – and are sold. Thus ends the grand speculation.

Our men who at the start were resolved to be content with a fair profit are generally found among the number of those who are sold out, when they are heard to say that if they ever have another such chance to make money they will not hold on for the last cent. They have said the same thing year after year ever since the opening of the Comstock mines. But whenever there is a grand upward movement in stocks, they never fail to become excited and try to buy about ten times as much stock as they can pay for. In this way they lose all except what they may have happened to purchase at a fair price in a mine of real merit.

Today I got an email from Buy.com asking me to review a cell phone battery I’d bought.

Happy with the battery, and feeling like procrastinating for a few minutes, I decided to do it.

I clicked on the link in the email.  Buy.com immediately asked for my email address and password.

Now, which of my 5 different emails did I use for that purchase?  I guessed – wrong, apparently.

So, forget it.  I was going to do them a favor, but now I’m not.

Why do I have to authenticate myself to review a product I bought?  They know I bought it.  They know who I am – they sent me the email.  So why ask again for authentication?  They should have included a unique ID in the link, allowing me to write one (1) review for that one product.

Either some idiot at Buy.com thinks it’s necessary to re-authenticate me (likely following some corporate rule set down by God) or they’re just too lazy to bother to think about the situation.

This kind of corporate incompetence is all too common.