Philosophy, Society

On uncertainty, success and failure

The truth is, most things aren’t as predictable as we’ve come to believe them to be. There are just way too many factors at play, too many unknown variables, and too many variable we don’t even know that we would have to take into consideration.

As noted in these two great TED talks by Alain de Botton and Elizabeth Gilbert, our perception of who is responsible for our success has changed a lot. Before the modern era, poor people were called “unfortunates”, now they are simply “losers”, implying that it is mostly their own fault that they’re in this miserable situation they are now. Never before in history have we as humans believed so much in our own power to make informed decisions, to enforce our will on the world around us. Much was always believed to be in the hand of the gods or some other transcendent entity, but now it’s always our own fault when something doesn’t turn out the way it was supposed or planned to. In the tragedy, the ancient greeks had even an entire genre of drama dedicated to letting the audience empathically follow the stories of people that were simply struck by very bad luck.

It is true that the physical world at certain small scales is quite accurately modelled by sciences like physics and chemistry and that this has enabled us to achieve things our ancestor didn’t even dare to dream about. But the bigger and more complex things get, the harder to model them and be precise about them it is. Some might argue that intuition is better suited to the job then, but this is only sometimes true. Usually a combination of both, intuition and the scientific method, yields the best results. But still, usually things just don’t turn out exactly the way we predicted.

So should we stop trying just because the outcome is not as predictable as we’ve been led to believe? No, of course not. I think we should try nonetheless. And maybe we are lucky and it works. And maybe we fail and try to do better next time. And we fail again. And again. And again. And then we succeed. (Or at least we’ve helped others to come closer to the solution.)

Maybe rapid prototyping and short feedback loops help figuring out when we go into the wrong direction. And systems– and design thinking help seeing the bigger picture.

Philosophy, Society

On optimism, being wrong and truth

A very good TED Talk by Tali Sharot on The optimism bias tells us something many of us have known (or shall we say suspected) already for a long time: the brains of healthy (non-depressed) people are fundamentally tilted towards optimism. For example, even when told that 40% of all marriages break, newly married people still don’t think there is a possibility they’ll get divorced. Likewise we underestimate the likelihood that bad things will happen to us personally. Sharot suggests that when doing financial budgets or when in dangerous situations, we should remember that we have this innate bias and calculatingly adjust for it. But most of the time, the human optimism bias is a good thing. Otherwise we would never dare to venture into anything new or risky.

The flipside of all this optimism is obviously that we always think we’re right. There is a great TED Talk by Kathryn Schulz: On being wrong. We will do almost anything to avoid having that nasty feeling creep up on us when we realize we were wrong. When somebody disagrees with us, we rather think that he or she either:

  1. doesn’t have the same information as we do,
  2. is plain stupid, or if neither of those apply,
  3. that he’s plain evil.

Rather seldom the thought crosses our minds that we ourselves might be wrong. Or that different people have simply different models of the world that they think in and in which they integrate new information, and that there might be a multitude of valid worldviews. So we should pause more often and think about all the complexities of our world, and allow ourselves to recognize that we may not know, or that we may be wrong.

To the smart fellow who chips in now and says we should just abide to the scientific method and reason objectively: 1) objectivity does not exist, see intersubjectivity. 2) read the closing words of this unsettling article by Jonah Lehrer in The New Yorker about the ‘decline effect’ when trying to reproduce the results of scientific studies in fields such as medicine or psychology.

Although many scientific ideas generate conflicting results and suffer from falling effect sizes, they continue to get cited in the textbooks and drive standard medical practice. Why? Because these ideas seem true. Because they make sense. Because we can’t bear to let them go. And this is why the decline effect is so troubling. Not because it reveals the human fallibility of science, in which data are tweaked and beliefs shape perceptions. (Such shortcomings aren’t surprising, at least for scientists.) And not because it reveals that many of our most exciting theories are fleeting fads and will soon be rejected. (That idea has been around since Thomas Kuhn.) The decline effect is troubling because it reminds us how difficult it is to prove anything. We like to pretend that our experiments define the truth for us. But that’s often not the case. Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true. When the experiments are done, we still have to choose what to believe.

While I always say that it’s problematic to apply the term “prove” or “proved” to anything other than pure mathematics (in science, theories can only be substantiated or falsified), I agree with the point being made. In anything that involves living beings (as opposed to e.g. physics or chemistry), interactions pretty quickly become so complex that it is hard to tell anything with certainty. That doesn’t mean that the scientific method might not be the best (or shall we say one very good) way to approach these problems as well. But we should keep in mind that just because some theory has been substantiated by a scientific study or even two, that doesn’t mean that it’s the absolute truth.

So the bottom line is that it is good to question things every once in a while, to remember that we actually know pretty little, to remember that we’re wrong way more often than we think. And to really consider other people’s worldviews and try to learn from them. But at some point, if you want to stay sane, you have to start believing in something again. How could you ever start a venture or bring any change to the world without believing in it in the first place? That’s when our innate optimism bias is essential. So that we do something – even when all the estimates say that the odds are low – to see whether it might be possible nonetheless. And sometimes it is. Just sometimes it is.


Haskell, XQuery, JavaScript or Opa for web development

This is a follow up on a previous post of mine, XQuery, Node.js and Opa – which is the “new Java” of web development?. So here comes the table…

Haskell XQuery JavaScript Opa
Purely functional as pure as it gets not so pure but mostly functional cumbersome to write functional code not pure but functional
Statically typed yes yes no yes
integrates well with a database with RDMSs (via HaskellDB) and with a native Haskell store called acid-state with XML-DBs with MongoDB or CouchDB with MongoDB or CouchDB
Web server good good, e.g. Sausalito or eXist Node.js e.g. Node.js
Web client (browser) experimental Haskell to JavaScript compilers experimental, see XQuery in the Browser great (native) good (compiles to JavaScript)
GUI code executed seamlessly on client and server no no yes, using Meteor or Derby yes, integrated
Templating system yes, e.g. HStringTemplate not that I know of yes, e.g. Handlebars yes
Parallelism and distribution wide range of facilities for parallelism, maybe less distribution on multiple servers? current implementations spawn a new process for each request, inside that no parallelism afaik requests handled by the Node.js event loop which is non-blocking but unfair and starving. Inside WebSockets can be used uses Node.js, but automates distribution

… and now some explanations. Keep in mind that these are my research notes, I’m no expert (yet ;)).

This started when I was thinking how nice a language Haskell was, purely functional, monads, lazy evaluation and a rich static type system. What more can you ask? Going from top to bottom in the table, unfortunately Haskell doesn’t integrate that well with NoSQL databases as XQuery and JavaScript do with XML databases and MongoDB respectively. When you have to use a different language to query your database than what you use for the application tier, you have an impedance mismatch: you have to convert all the data back and forth and have to think in different ways depending on what tier you’re on as well as having to decide what should be handled in what tier. A similar impedance mismatch arises when you have to use a different language on the client-side (i.e. the browser where we’re stuck with JavaScript) and on the server side. Ideally, you write your GUI code once and it is transparently executed either on the server (upon the initial page load) or on the client (upon further navigation by the user). To pull this off, it seems to help a lot to have a templating system which also separates presentation and computation/business logic. Finally, it would be nice if you could easily (and without introducing hard to find errors) parallelize your code to run on multiple cores of a machine as well as distribute it to run on a cloud. Maybe I’ll have to look at Clojure or Erlang for that as well.

Opa seems to do a lot of things right, the question is how it fares in practice and also how open its application stack really is. For Haskell, we’ll have to wait until it compiles properly to JavaScript, and I’ll have to look at whether there is a database that performs and integrates well with Haskell, otherwise it seems to rule (except that it’s hard to learn). XQuery seems well suited for large amounts of XML data and JavaScript is pretty easy to learn, so it might become the new (concurrent) PHP. So, we’ll see… 😉