[revised] Benchmark: Rhino vs Chrome V8 on server side

It was a while ago, when I’ve created the post, titled ‘Benchmark: Rhino vs Chrome V8 on server side’.

Since then, it has received a lot of critics about the measurement method, so I’ve decided to rerun the measurements:

• I’ve set all the CPU cores, to the ‘performance’ CPU governor
• I’ve closed all the running applications (except for a browser which played Corrosion of Conformity‘s Albatross )
• I’ve changed the measurement plan, to execute 50 iterations of the same measurement, just to leave enough time for the JIT compiler to do its work. The running time is measured during another 50 iterations.
• I’ve added -Xcomp option, and I’ve checked that the JIT compiler is working properly, by adding -XX:+PrintCompilation and a “sysout-marker-string” which indicated the start of the measurement period.
• I’ve run the tests with Oracle’s 1.6_033 32bit JDK.
• I’ve run the tests on an Intel Core i5 – 2430M (2.4GHz).
• I’ve shared the code at github, so you can validate the results on your machine.

Here are the results:

(Note: These tests measure CPU-burning calculations written in javascript. Usually when you use Rhino, you don’t want to run CPU-burning tasks with it, instead you use it to create some low-level interface to your application (like being able to execute custom user-scripts, for example))

                          NAME  |    V8     |   Rhino     |   hino / V8
--------------------------------|-----------|-------------|------------------
                       3d-cube  | 11.49 ms  |  251.98 ms  |  21.93 x faster
                      3d-morph  | 11.82 ms  |  344.96 ms  |  29.19 x faster
                   3d-raytrace  | 11.22 ms  |  240.51 ms  |  21.43 x faster
           access-binary-trees  |  1.06 ms  |  108.04 ms  | 101.81 x faster
               access-fannkuch  | 16.08 ms  |  634.49 ms  |  39.45 x faster
                  access-nbody  | 14.84 ms  |  317.06 ms  |  21.37 x faster
                 access-nsieve  |  4.27 ms  |  232.57 ms  |  54.53 x faster
      bitops-3bit-bits-in-byte  |  5.02 ms  |  209.90 ms  |  41.81 x faster
           bitops-bits-in-byte  |  8.39 ms  |  346.20 ms  |  41.27 x faster
            bitops-bitwise-and  |  8.02 ms  |  736.00 ms  |  91.77 x faster
            bitops-nsieve-bits  | 11.02 ms  |  383.45 ms  |  34.79 x faster
         controlflow-recursive  |  2.04 ms  |  145.61 ms  |  71.35 x faster
                    crypto-aes  |  6.14 ms  |  193.49 ms  |  31.50 x faster
                    crypto-md5  |  5.22 ms  |  194.12 ms  |  37.16 x faster
                   crypto-sha1  |  6.02 ms  |  201.37 ms  |  33.45 x faster
             date-format-tofte  |  8.10 ms  |  186.33 ms  |  23.00 x faster
             date-format-xparb  |  9.24 ms  |  139.06 ms  |  15.04 x faster
                   math-cordic  | 10.39 ms  |  408.51 ms  |  39.33 x faster
             math-partial-sums  | 15.78 ms  |  303.04 ms  |  19.21 x faster
            math-spectral-norm  |  5.12 ms  |  169.76 ms  |  33.14 x faster
                    regexp-dna  |  9.12 ms  | 1268.24 ms  | 139.02 x faster
                 string-base64  |  4.00 ms  |  403.27 ms  | 100.82 x faster
                  string-fasta  |  8.33 ms  |  396.35 ms  |  47.60 x faster
               string-tagcloud  | 11.51 ms  | 1109.16 ms  |  96.36 x faster
            string-unpack-code  | 20.22 ms  | 1454.22 ms  |  71.90 x faster
         string-validate-input  |  4.39 ms  | 2311.41 ms  | 526.79 x faster

Nothing surprising: The numbers have changed, but V8 is still the better one.

I’m open to criticism, so if you think, my measurements are wrong, please show me how you measure or compare the performance of the two libraries.

(Tip, if you want to play with it: I’ve done the measurements using Oracle’s JDK 1.7.0_05 too, and with that version, Rhino runs 4-5 times slower. That could be a more interesting problem, but I want to double check the measurements, before I publish the details.)

About these ads