Parallel and Concurrent Programming Models

This is a simple (non-exhaustive) categorization of different approaches for parallel and concurrent programming, inspired by Parallel and Concurrent Programming in Haskell (PDF). They state that parallelism and concurrency are two solutions to two different problems (as do others).

  • Parallelism (i.e. data parallelism, is deterministic)
  • Concurrency (i.e. threads or task parallelism, is nondeterministic thus cannot be expressed in purely functional code)
    • locking shared resources
    • blocking/synchronous channels (e.g. CSP) and asynchronous message passing (e.g. Erlang)
    • cancellation: asynchronous exceptions
    • software transactional memory (STM), i.e. group code into atomic blocks

Instead of concurrency, event loops with callbacks (as in Node.js) can be used to solve the problem of multiple external agents interacting with each other.

The paper also notes:

In the world of concurrency and parallelism, there is good reason to believe that no one size fits all programming model for concurrency and parallelism exists, and so prematurely committing to one particular paradigm is likely to tilt the language towards favouring certain kinds of problem.
We might wonder whether the compiler could automatically parallelise programs for us. After all, it should be easier to do this in a pure functional language where the only dependencies between computations are data dependencies, and those are mostly perspicuous and thus readily analysed. In contrast, when effects are unrestricted, analysis of dependencies tends to be much harder, leading to greater approximation and a large degree of false
dependencies. However, even in a language with only data dependencies, automatic parallelisation still suffers from an age-old problem: managing parallel tasks requires some bookkeeping relative to sequential execution and thus has an inherent overhead, so the size of the parallel tasks must be large enough to overcome the overhead. Analysing costs at compile time is hard, so one approach is to use runtime profiling to find tasks that are costly enough and can also be run in parallel, and feed this information back into the compiler. Even this, however, has not been terribly successful in practice [Harris and Singh]. Fully automatic parallelisation is still a pipe dream. However, the parallel programming models provided by Haskell do succeed in eliminating some mundane or error-prone aspects traditionally associated with parallel programming.

Philosophy, Society

Gratitude and why you shouldn’t expect things to “just work”

I’m utterly convinced that the key to lifelong success is the regular exercise of a single emotional muscle: gratitude.

– Geoffrey James (hat tip to swissmiss)

Continuing the line of thought from my two previous posts, it feels like many of us have (or at least I have) come to expect many things that probably shouldn’t be expected. What I mean is this conception of “how things are supposed to work”, and the believe that somewhere on this planet there are those “true professionals” that do things that way. That conception, that if you adhere to a couple of smart principles, including the scientific method, sound reasoning and a lot of grit, you eventually end up with a process that is “the right way”. And the thought that if you do things that way and do them thoroughly, you will eventually come up with a “proper” product, solution, or at least a satisfactory conclusion concerning that particular problem. And if you don’t, you are just doing it wrong, have too little experience and probably aren’t (yet) one of those “true professionals”.

But what I’ve come to realize, is that those “true professionals” don’t really exist. That even the most professional experts, most of the time, have not much of a clue of what they are actually doing and are just making things up along the way. Of course, experience and formal training help, but even the best of us, most of the time, have to navigate a vast decision tree with simply too little information.

So what’s that got to do with gratitude? Well, if you expect that things can, should and are done “the right way” (that is assuming you are not working with or using a product of complete idiots, or believe you are yourself one), then you expect things to “just work” in the “expected way”. If they don’t, you are disappointed. But what’s even worse: if they do, you just ignore the fact that it just worked, without ever feeling any gratitude. Simply because that’s what you were expecting all along, that things work the way they are supposed to. So however things turn out, either you get disappointed or don’t feel much at all.

On the other hand, if you were to expect that things usually don’t “just work”, it is much easier to feel gratitude every time they actually do. And you will have a much happier life, giving you the power to tackle harder problems (not expecting to solve them on the first attempt).