In my two previous articles I solved a trivial problem in Scala and Erlang: Write a program that -given G- produces sequences of numbers x, x+1, x+2..., x+n such that G = sum(x, x+1...x+n). The challenge was not to solve this trivial problem. My quest was to compare the performance of an Erlang solution to a Scala implementation.
I stated beforehand these were my first lines of code in both Erlang and Scala. For smaller values for G, the Erlang implementation outperformed 'single-shot' execution of the Scala equivalent by a wide margin. I then modified the Scala implementation in a direction where I knew JIT would shine. And, lo and behold, in this setup Scala outperformed Erlang. For larger values for G even by a wide margin.
And then I picked up the book "Erlang and OTP in Action" by Martin Logan et al. The introduction mentions the HiPE project (High Performance Erlang), a native code compiler for Erlang. This wetted my apetite for further experiments.
So I checked my Erlang installation and was happy to find it contained HiPE support. I recompiled my little Erlang program using HiPE with maximum optimization. This dramatically improved performance.
In my article on Scala I assumed that the larger the load on a system, the more JIT would shine and the more Scala would outperform an equivalent Erlang implementation. But this was no more than an assumption. One I'm not too sure about anymore. In the Scala setup I explicitly picked a situation that would clearly benefit JIT; a small piece of code repeatedly executed to warm up JIT in order to let it shine.
Both Scala and Erlang are directed towards large scale projects containing million+ lines of code. Erlang already proved to be able to hold its ground in such situations. For Scala this remains to be seen. In a large scale system a lot of the code is executed and a lot of it is executed in parallel. But when it's different code that gets executed all the time, JIT would suffer from its warm up delay. Causing effects that resemble my first attempts with Scala where I used "one shot" execution. A situation where Erlang outperformed Scala by a margin of 1.000
My experiments so far show Erlang clearly does not suffer from any startup delay, on the contrary. So I now doubt wether my conclusion that the larger the load on a large scale system, the more JIT will shine is correct. Especially since it is probable that in such a large scale system major parts of the code will only execute incidentally. Most of the time the system will be busy executing incidental code. In such a system a small part of the code will be executed almost continually, giving JIT an opportunity to show its strength. But most of the time the system will be busy executing incidental code and will, in the Scala case, suffer from the warm up delay which severely degrades performance.
Erlang shows not to suffer from such a warm up delay and performance is quite consistent. So in the Erlang case, performance becomes predictable. Predicting performance of Scala/JIT is much more difficult because the warm up delay plays a major role. My guess is that in a large scale system most code executed, is executed some of the time, some code is executed most of the time. That spells bad for JIT because in such a situation most of the code will suffer from warm up delay.
Best case, Scala will outperform Erlang with a factor 3. But worst case, Erlang will outperform Scala with a factor of better than 1000. In my Scala post I stated my money would be on Scala. I'm glad no one took the bet.
No comments:
Post a Comment