coding, by Derek Young: Scala vs. Groovy: static typing is key to performance

Scala vs. Groovy: static typing is key to performance
Thursday September 6, 2007 by Derek Young

I was curious about the performance of two of the JVM based languages, Scala and Groovy, relative to Java itself. I suspected that by having more type information available at compile time Scala’s bytecode would look more like the output of javac and therefore have performance closer to that of straight Java.

I started with the Java version of the (somewhat notorious) Ray tracer language comparison benchmark that has been used to compare the performance of various languages. The benchmark generates a ray traced image of recursively positioned spheres. I took the minimal optimization version and ported it into both Scala and Groovy.

I kept the code as close to the Java version as possible. Neither Scala nor Groovy supports Java/C style for loops, so I substituted the recommended loops in each language. Groovy doesn’t support inner classes so I had to pull some code out to the top level. Groovy uses BigDecimal for floating point literals, so I suffixed them all with “d” to make them doubles. I left all type declarations in the Groovy version, although Groovy programs are typically not written this way.

Results

ray.java	12.89s
ray.scala	11.224s
ray.groovy	2h 31m 42s

I was expecting the Groovy code to run longer than the Scala code but was shocked at the actual difference. All three versions of the code produce identical images: (fullsize here)

Because Scala knows the types of all variables, it produces floating point operations for floats and integer operations for ints, like Java. The compiled output of Groovy treats everything dynamically, determining at runtime which functions to use for simple operations and using objects like java.lang.Double instead of double. The more the bytecode looks like Java code the better it will perform.

Unlike most software, a ray tracer executes almost entirely in its own code, not in any external libraries or waiting for I/O. This highlights the difference between the three compilers, but most software will see less of a difference.

The Scala timings demonstrate that algorithms can be coded in Scala without much loss in performance, even for critical areas of code.

I should point out it’s possible I made a mistake in the translation from Java to Groovy, but I tried to make it as fair as I could. I would be interested in hearing if there is something I should have done differently that explains the difference.

Timing Notes
All times recorded with time from the command line, which does include startup overhead. With over a 2 hour difference this doesn’t matter.

OS: Ubuntu Feisty
javac -version: build 1.6.0-b105
Groovy execution: time groovy ray.groovy 8 512
Scala compilation: scalac ray.scala
Scala execution: scala scalaRay 8 512
Java compilation: javac ray.java
Java execution: java ray 8 512

>>>groovy ray.groovy 8 512
Why you dont use groovyc unlike javac & scalac???

— Vadim Voituk Sep 6, 10:57 AM #
The only reason I didn’t use groovyc was because the difference was so great, and the compilation overhead at the beginning of the run only takes a couple seconds. I decided it wasn’t worth waiting another two and a half hours to time the compiled output. Running with groovy first compiles the code just like groovyc does, then executes that code. It doesn't interpret the source code or run any differently. I’ll make another timing tonight after work when my machine’s free and post the results though.

— Derek Young Sep 6, 01:29 PM #
Wouldn’t matter if he used groovyc. Cuts off 7 sec at the most, and that’s if it’s a Pentium II 200mhz. The real issue is that using Groovy to do ray tracing is like wearing high heel shoes to a marathon. Groovy was never meant for heavy calculations driven apps like this.

The real groovy solution would be to write the renderer in Java or Scala, and call it from Groovy. Ray tracing isn’t groovy, it’s gnarly, and requires a gnarly language to do efficiently.

— Danno Ferrin Sep 6, 01:30 PM #
Just a little stylistic note on your Scala code. You can write 1 until n instead of 1 to (n-1).

— psykotic Sep 6, 01:37 PM #
It’s not clear whether you’re measuring -server or -client

Scala while loops will probably be faster than for comprehensions over ints

— Isaac Gouy Sep 6, 01:54 PM #
There is no reason for a dynamically typed language to be slower than a statically typed language. It’s a matter of implementation. Perhaps performance is not a priority for Groovy, but I wish Groovy fans would just admit this upfront, instead of claiming that dynamic typing is inherently slow, because this has been disproven time and time again.

— Slava Pestov Sep 6, 03:36 PM #
I’ve compiled the groovy script using groovyc and ran the example. Killed the run after 4 minutes (on a 2.16ghz dualcore) nowhere near the 12 second mark.

— p3t0r Sep 6, 04:11 PM #
the biggest mistake people make is using explicit types all other. That makes Groovy slower in the current implementation, and maybe later too, because the runtime has to check and transform the types. The next thing is that many calculations in Groovy are way slower than in Java because Groovy is doing them in BigDecimal. I plan to add optimzed double and integer modes that will speed up the code very much. But that will take some more months, because before this the MetaClass system will be changed. I am not sure Groovy will ever be as fast as Java. The method calls are doing so much more in Groovy.. but I expect Groovy to become much faster. Compared to Java I think a factor below 10 is reachable.

— Jochen "blackdrag" Theodorou Sep 6, 04:20 PM #
Yes, you can speed up your Groovy version by removing static types which slow it down, using traditional for loops (Groovy 1.1 only) and not using BigDecimal (currently does calculations in BigDecimal and then converts to Double). It will currently still be much slower – but mostly nothing to do with dynamic vs static typing.

— Paul King Sep 7, 09:30 PM #
You might also use -server for about 2 times speedup and consider operator overloading for code style so that instead of:

Vec o = Vec.add(ray.orig, Vec.add(Vec.scale(i.lambda, ray.dir), Vec.scale(delta, i.normal)))

you would have:

def o = ray.orig + (i.lambda * ray.dir + delta * i.normal)

— Paul King Sep 7, 09:36 PM #
Great to get some numbers even tho they are kinda special case!
Any plans to try out Jython or/and JRuby to?

— krs Sep 9, 06:30 AM #
What is the groovy version? I get the following exception when running against groovy 1.0:

Caught: java.lang.ArrayIndexOutOfBoundsException: 1
at ray.run(ray.groovy:144) at ray.main(ray.groovy)
— Marcos Silva Pereira Sep 9, 10:57 PM #
Marcos: I used groovy V1.0. You need to pass in two arguments on the command line or it will crash like that. The first argument is the recursion level that determines how complex the scene is. The second is the resolution (the images are always square). The tests I ran were with the arguments 8 512.

— Derek Young Sep 10, 09:06 AM #
What were your runtimes for the C/C++/etc versions on your machine?

— Warren Henning Oct 2, 01:54 PM #
These are my results. I’m using a Macbook 2.0 GHz Intel Core Duo with 2 GB 667 MHz DDR2 SDRAM running MacOSX Tiger 10.4.11

$ uname -a
Darwin goku2s-computer.local 8.11.1 Darwin Kernel Version 8.11.1: Wed Oct 10 18:23:28 PDT 2007; root:xnu-792.25.20~1/RELEASE_I386 i386 i386

JAVA
$ java -version
java version “1.5.0_13”
Java™ 2 Runtime Environment, Standard Edition (build 1.5.0_13-b05-241)
Java HotSpot™ Client VM (build 1.5.0_13-121, mixed mode, sharing)

$ time javac ray.java
Note: ray.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

real 0m0.465s
user 0m0.359s
sys 0m0.063s

$ time java ray 8 512

real 0m37.126s
user 0m35.219s
sys 0m1.638s

SCALA

$ scala -version
Scala code runner version 2.6.1-final — © 2002-2007 LAMP/EPFL

$ time scalac ray.scala

real 0m6.055s
user 0m4.190s
sys 0m0.317s

$ time scala scalaRay 8 512

real 0m40.592s
user 0m39.019s
sys 0m1.406s

GROOVY

$ groovy -version
Groovy Version: 1.5.4 JVM: 1.5.0_13-121

$ time groovyc ray.groovy

real 0m1.238s
user 0m1.096s
sys 0m0.123s

$ time groovy ray.groovy 8 512

real 194m57.459s
user 192m56.086s
sys 1m0.091s

Compiled version
Obs: A changed the lines:

(new rayMain()).run(Integer.parseInt(args¹), Integer.parseInt(args⁰), 4);
to

public class Tester { public static void main(args) { (new rayMain()).run(Integer.parseInt(args¹), Integer.parseInt(args⁰), 4); }
}

To be able to call the main method os the class Tester.

$time groovyc ray.groovy

real 0m1.290s
user 0m1.114s
sys 0m0.127s

$ time java -cp /Applications/groovy-1.5.4/embeddable/groovy-all-1.5.4.jar:. Tester 8 512

real 201m29.815s
user 198m4.698s
sys 1m10.802s

We have to note that scala and groovy versions are direct translations from the java source. But that’s not a excuse to have such a diferent performance in this case. Aprox. 300 times slower than scala and java version is a huge diference.

— Goku2 Feb 9, 02:17 PM #
I’m evaluating alternatives for the SuperCollider language (http://supercollider.sf.net)

in particular i will need to process a lot arrays, lists, and dictionaries. so i wrote a simple benchmark. the disappointing result:

/////// sclang
x = Dictionary.new;
(1..1000).do({ arg key; x.put( key, \lala )});
{ a = 2000.collect({ arg key; x.includesKey( key )})}.bench;

// —> around 7ms

//////// groovy

x = new HashMap();
1000.times { x.put( it + 1, “lala” );}
t1 = System.currentTimeMillis();
a = (0..1999).collect { x.containsKey( it )};
t2 = System.currentTimeMillis();
println “time to run: “ + (t2-t1)*0.001 + “ seconds.”;

// —> around 100ms in the first execution, followed by some stable 60ms if executed repeatedly

////////// scala

var x = new java.util.HashMap[ int, String ];
for(y <- 1 to 1000) { x.put( y, “lala” )};
var t1 = System.currentTimeMillis();
var a = (0 to 1999).map( (y: int) => x.containsKey( x ));
var t2 = System.currentTimeMillis();
println( “time to run: “ + (t2-t1)*0.001 + “ seconds.” );

// —-> around 520ms

so groovy is around a magnitude slower than sclang, and scala is even a magnitude slower than groovy. (i execute both code from the scala and groovy shell respectively, since i will need to use them in a on-the-fly scripting environment).

how comes both groovy and scala are that slow and in particular, how comes that scala performs so bad here when the blog suggests that it should be the other way around?

— Sciss Jun 4, 08:22 PM #
…solved the riddle: pasting multiline code into the shell results in separate compiles per line; if i put all in one line, the result is 7ms for groovy and around 1ms for scala. so both are at least as fast as sclang, scala definitely faster, but scala has a little drawback of increased delay per compilation (around 250ms! in this case)

— Sciss Jun 4, 08:36 PM #
Sciss,

you have a mistake in your Scala code, instead of
containsKey( x )
it should be
containsKey( y )

Besides, benchmarking code on the order of 1ms is pointless as the system overheads may skew the results.

— Pavel Verevkin Jan 5, 04:04 PM #
Sciss,

One more thing. You are effectively testing Java’s HashMap class, not Groovy or Scala.

— Pavel Verevkin Jan 5, 04:08 PM #
didn’t you miss warming up the JIT? I.e. running the benchmark few times in the same JVM before taking times. It might have significant impact.
Another thing, in Scala primitives may be viewed as objects but if not needed they are compiled as primitives where in Groovy they are always objects. Since the app is very primitive heavy, Scala has a significant advantage. Would be interesting to see a comparison that plays less into Scalas advantage.

— Eishay Smith Apr 8, 07:12 PM #
Ubuntu 8.04 AMD64
Sun Java6 update 13
Scala 2.7.4.final
Groovy 1.6.2

time java ray 8 512
real 0m8.281s
user 0m7.716s
sys 0m0.444s

time scala scalaRay 8 512
real 0m7.713s
user 0m7.564s
sys 0m0.284s

time groovy ray.groovy 8 512
real 3m53.136s
user 3m47.934s
sys 0m3.292s

— goodsforyou Apr 26, 11:32 PM #
has this changed much since 2007, with the latest versions of Groovy?

— RaoB May 7, 10:08 AM #

Fixing scala-mode.el for Emacs 23 Using Eclipse Efficiently

Name		Remember
E-mail
http://
Message
	Textile Help

coding, by Derek Young

coding and design thoughts

Scala vs. Groovy: static typing is key to performance Thursday September 6, 2007 by Derek Young

Scala vs. Groovy: static typing is key to performance
Thursday September 6, 2007 by Derek Young