Language Benchmarks

Anima

Hey,

I'm currently doing up a report for college. I've to compare C, C++ and Java. I can find the easy differences alright like architecture, portability, history, advatanges/disadvantages etc but I'm having trouble finding reliable performance benchmark comparisons of each language.

A lot of them seem biased with people just stating opinions as fact. Or if I do find something, its usually well out of date.

Does anyone know where I can find relevant information?

Martyr

There are certainly plenty of articles and conflicting views about whether Java is faster than C++.

IMHO it depends alot on the programmer and/or algorithm.

I'd expect well written C/C++ to run significantly faster than Java since its closer to the metal, we could put to test

yep, just an opinion.

i found 1 comparison here amongst alot!

Anima

Cheers, yeah I've found a bit more after a bit of searching. Also theres a wiki article on Java Performance thats quite big.

It seems with JIT and other enhancements to the JVM, Java runs just as well as C in some cases and better in others, thanks to its dynamic recompilation. Fairly interesting stuff

Cantab.

I've used the Xerces XML parser on both Java and C++. I was amazed to see that the Java performance was just as good on very large XML files. Granted, Java used slightly more memory, but nonetheless.

Regarding "benchmarks": it all depends how the code is written. Code optimised for C++ is not necessarily optimised for Java.

Hobbes

Anima wrote: »

Does anyone know where I can find relevant information?

Why not just write up your own tests rather then copying previous tests?

The short answer all of them are the fastest. Depending on what it is you are doing.

How you code the apps as well is a big factor.

Anima

Well of course you'll be using a language and its features to its strengths. I mean you're hardly going to be doing GUIs in C if you had the option of doing them in Java.

What I had in mind was raw data tests such as math libraries, data I/O etc. I guess I could run my own tests for these but you know... effort.

Cantab.

Anima wrote: »

Well of course you'll be using a language and its features to its strengths. I mean you're hardly going to be doing GUIs in C if you had the option of doing them in Java.

What I had in mind was raw data tests such as math libraries, data I/O etc. I guess I could run my own tests for these but you know... effort.

Getting back to the Xerces parser issue, I'd say I was limited more by disk access speed than by any language efficiencies. If I had a not so fast processor but the same disk speeds, I might have seen a difference between the two.

There are so many variables. The Java/C++ debate is not black and white.

Martyr

Could you write an operating system kernel in Java or .NET? probably not

Can you exploit SIMD features of PowerPC/x86 cpus like Altivec/SSE2?
..yes, i think so but the big question is whether Java would generate more efficient code than C/C++ for this purpose and i'd doubt it very much.

Anima wrote:

What I had in mind was raw data tests such as math libraries, data I/O etc. I guess I could run my own tests for these but you know... effort

Cryptography would also be a good test.
Perhaps compare OpenSSL performance with Java classes.

The thing is though, Java classes are nothing but machine code for the processor of system VM is running on.

For example on windows, Java classes are stored in DLL files..much the same as crypto functions in OpenSSL DLL are accessed by C/C++ program running on windows.

Hobbes

Martyr wrote: »

Could you write an operating system kernel in Java or .NET? probably not

Wrong. You can and they exist.

I think so but the big question is whether Java would generate more efficient code than C/C++ for this purpose and i'd doubt it very much.

Again it depends very much on how and what you are doing. JIT will create java code that performs as well or better then C++ depending on what you are coding and how long the application runs. (JIT can rewrite the assembly code on the fly to optimize).

For example on windows, Java classes are stored in DLL files.

No they are not.

NeverSayDie

Personally, I have very little interest and place very little faith in "language benchmarks". They seem to be almost always generated for and used in religious wars between fanboys, or idle blog posts.

If you're worried about performance, there's better things to focus on - regardless of the languages themselves, some non-trivial code put together by an expert in language X is quite likely to outperform the equivalent built in language Y by an amateur/intermediate developer.

Then there's the big picture stuff - ease of use, tools support, third party libraries, etc. Regardless of language X being "faster" than language Y, if an engineer can work more productively in Y, then they can spend the saved development time on better design, performance optimisation, debugging and profiling, etc, which will most likely end up with a faster/better Y-based solution.

Martyr wrote: »

Could you write an operating system kernel in Java or .NET? probably not

Actually that has been done, with .NET anyway. MS released the source code a while back for a research OS they built in managed code (variant of C# I think), it's on CodePlex somewhere - it was called Singularity, far as I recall.

Martyr

hobbes wrote:

Again it depends very much on how and what you are doing. JIT will create java code that performs as well or better then C++ depending on what you are coding and how long the application runs. (JIT can rewrite the assembly code on the fly to optimize).

How would this make Java faster? Do cpu's discriminate machine code generated by C/C++ compiler?

hobbes wrote:

No they are not.

Well atleast have the courtesy to tell me where they are?

After all, your code has to run on the processor, doesn't it? so where does it come from?

Martyr

NeverSayDie wrote:

Actually that has been done, with .NET anyway. MS released the source code a while back for a research OS they built in managed code (variant of C# I think), it's on CodePlex somewhere - it was called Singularity, far as I recall.

Thanks, thats interesting.

After a quick read, it seems that most of the low-level code was written using C and x86 assembly.

But yes, kernel written in Sing#, subset of Spec# which is extension of c# :eek:

Hobbes

Martyr wrote: »

How would this make Java faster? Do cpu's discriminate machine code generated by C/C++ compiler?

A C/C++ compiler compiles once into assembly. That is then executed on the processor.

Java (and .NET for that matter) compile into Bytecode. When the JVM loads the bytecode it converts it to assembly and runs that. Over time the JIT JVM can see where sections of the application require optimization and rewrite the assembly code while the application is running to make it faster.

You can't do this with C/C++ (you can with C sharp IIRC).

Prior to JIT the java language was interpreted (bytecode instructions) and was slow. Which is where it got the bad rap from. That is many years ago.

Well atleast have the courtesy to tell me where they are?

Class files are just stored on your machine as normal files. They are not stored in DLLs. They are just btyecode representations of your source code. They are not actually compiled into something that the operating system can read.

The JVM is what is written to be OS specific. That is what reads the class files and converts it to assembly to run on the OS/processor.

Martyr

A C/C++ compiler compiles once into assembly. That is then executed on the processor.

Java (and .NET for that matter) compile into Bytecode. When the JVM loads the bytecode it converts it to assembly and runs that. Over time the JIT JVM can see where sections of the application require optimization and rewrite the assembly code while the application is running to make it faster.

You can't do this with C/C++ (you can with C sharp IIRC).

Prior to JIT the java language was interpreted (bytecode instructions) and was slow. Which is where it got the bad rap from. That is many years ago.

OK, fair enough.

hobbes wrote:

The JVM is what is written to be OS specific. That is what reads the class files and converts it to assembly to run on the OS/processor.

I was talking about JVM.dll, java classes are here by the way..should have been more specific.

Martyr

depending on your version of java runtime environment installed.

i have 1.4.2 for xp here, under the directory:

C:\Program Files\Java\j2re1.4.2_05\bin

You'll see here alot of DLL files which contain the Java classes used by Java .CLASS files executed in a JVM.

Martyr

1 other thing there is api in jvm.dll called JNI_CreateJavaVM which allows you to resolve java methods or execute other java byte code.

but the actual code from what i can tell is stored in dll files, i'll look at this later.
for example, working with jpegs, zip archives, sound, authentication..etc this isn't code generated at runtime.

Hobbes

Martyr wrote: »

You'll see here alot of DLL files which contain the Java classes used by Java .CLASS files executed in a JVM.

The DLL files are for use in the JVM.

I don't think you understand the difference between the JVM and what a Java Class is.

Java Classes are not stored in DLLs, nor are they a DLL. The actual java classes used by the JVM (java.lang.*) are stored in the lib folder in a JAR file. Normally called RT.jar. These are OS independent.

The JVM on the other hand is OS dependent so would have DLL's for windows.

Martyr

hobbes wrote:

The DLL files are for use in the JVM.

yes, I know...did you not read my response?

hobbes wrote:

I don't think you understand the difference between the JVM and what a Java Class is.

uhm, yes i do know the difference.
there was a misunderstanding by you on what i was referring to, and this was because i wasn't specific enough.

i was making point that Java classes (runtime environment) on windows were stored in DLL files, not actual Java bytecode.

hobbes wrote:

Java Classes are not stored in DLLs, nor are they a DLL. The actual java classes used by the JVM (java.lang.*) are stored in the lib folder in a JAR file. Normally called RT.jar

i'm talking about runtime environment, the classes that a JVM would invoke on behalf of bytecode to perform an operating system task...thats what i mean.

i think if you atleast take the time to look at the data contained in these runtime environment files, you'll see i'm right.

Hobbes

Martyr wrote: »

I'm talking about runtime environment, the classes that a JVM would invoke on behalf of bytecode to perform an operating system task...thats what i mean.

The runtime environment (JRE) is the java class files + JVM. As I said the JVM will convert the Java class files into assembly language which is then run on the system.

The core API for Java are stored in class files in RT.jar. The JVMs only job is converting that into Assembly.

For example if your java class is making a socket call then the JVM will write the assembly language for the OS to make that call for that system.

JNI is just a means for a java command can make a direct call into a C/C++ API.

If there is any mistake it is on what you are classifying as what a class file is. It is like saying that C++ code is hardware because the instruction set is on the chip.

i think if you atleast take the time to look at the data contained in these runtime environment files, you'll see i'm right.

SCJP1.4 + SCJP5 and I've worked with Java for years. I think I should know how it works by now.

Martyr

well i'm glad you say this, hobbes because i was hoping an experienced java programmer could produce some code to test the theory of whether java is faster than C/C++.

i'm not a programmer at all.. but i'll bet if we were to both implement a cpu intensive algorithm, you writing it in java, and me the complete noob writing it in C/C++, i'd beat your java code.

want to test the theory?

or are ya just all talk? :P

marco_polo

Martyr wrote: »

well i'm glad you say this, hobbes because i was hoping an experienced java programmer could produce some code to test the theory of whether java is faster than C/C++.

i'm not a programmer at all.. but i'll bet if we were to both implement a cpu intensive algorithm, you writing it in java, and me the complete noob writing it in C/C++, i'd beat your java code.

want to test the theory?

or are ya just all talk? :P

So you have already decided on your conclusions then?

Writing fair benchmarks for managed vs compilled languages is a difficult task (I am not sure it is 100% possible) some of the JVM specific potential pitfalls are discussed here

http://www.ibm.com/developerworks/java/library/j-benchmark1.html

Martyr

yes, marco_polo, i have.

java is slower than c/c++

if we want to put it to test, lets do that.

i'm not programmer, just noob/amatuer.
hobbes has been working with java for years, so he knows what he's talking about.

i am willing to bet that anything i write using c/c++ will result in faster code than anything hobbes writes.

Martyr

actually, not just any code, but something with cpu intensive processing..

marco_polo

Martyr wrote: »

actually, not just any code, but something with cpu intensive processing..

Including the JVM startup time or not?

Martyr

no, you can load your program and perhaps have a button which records the start time, performs its processing, records end time then displays results.

marco_polo

Ok I have selected these two pieces of code as the benchmark. I'll let you do the rest.

Be sure and post up the results.

public class Loop
{
  public static void main(String[] args)
  {
    //10 000 billion iterations
    long time = System.currentTimeMillis();
    int REPEAT1 = 1000 * 1000;
    int REPEAT2 = 1000 * 1000 * 10;
    for (int i = 0; i < REPEAT1; i++)
    {
      for (int j = 0; j < REPEAT2; j++)
      {
        //do nothing
      }
    }
    time = (System.currentTimeMillis() - time)/1000;
    System.out.println("Time taken: (in seconds) " + time);
  }
}

#include <sys/time.h>
main(int argc, char *argv[])
{
  int i, j, REPEAT1, REPEAT2;
  struct timeval before, after;
  void *tzp;
  /*10 000 billion iterations*/
  tzp = 0;
  before = (struct timeval*) malloc(sizeof(struct timeval));
  after = (struct timeval*) malloc(sizeof(struct timeval));
  gettimeofday(&before, &tzp);
  REPEAT1 = 1000 * 1000;
  REPEAT2 = 1000 * 1000 * 10;
  for (i = 0; i < REPEAT1; i++)
  {
    for (j = 0; j < REPEAT2; j++)
    {
      //do nothing
    }
  }
  gettimeofday(&after, &tzp);
  printf("Time taken (in seconds): %ld\n",
                  (after->tv_sec-before->tv_sec));  
}

Martyr

are you saying that the loop does nothing, or do you want me to pick something?

a loop that does nothing isn't very useful..

marco_polo

Martyr wrote: »

are you saying that the loop does nothing, or do you want me to pick something?

a loop that does nothing isn't very useful..

I have never seen a 'useful' benchmark test. A loop that does nothing a billion timed is not useful but very CPU intensive.

What do I get if I win BTW?

Martyr

but when the JIT sees this code, or indeed a decent C/C++ compiler, it won't generate any machine code...it will not run the loops.

marco_polo

Martyr wrote: »

but when the JIT sees this code, or indeed a decent C/C++ compiler, it won't generate any machine code...it will not run the loops.

Damn, I was hoping you wouldn't have recent C\C++ compiler.

Martyr

well marco most compilers would be smart enough to see this one, even i saw it before having to compile it

marco_polo

Martyr wrote: »

well marco most compilers would be smart enough to see this one, even i saw it before having to compile it

Must deserve some marks for effort. Good spot , Thought I might get ya.

Webmonkey

Martyr wrote: »

well marco most compilers would be smart enough to see this one, even i saw it before having to compile it

Really? which ones?

Martyr

what about a mersenne twister or Lagged Fibonacci generator psuedo random number generator, or even some crypto?

could you implement something like this?

marco_polo

Martyr wrote: »

what about a mersenne twister or Lagged Fibonacci generator psuedo random number generator, or even some crypto?

could you implement something like this?

I could, but I have neither the time or the inclination TBH.

Martyr

webmonkey wrote:

Really? which ones?

GNU C Compiler, MSVC, Pelles c compiler, watcom..most of them really.

its not exactly something you could miss.(unless you're java programmer, who thinks it would be worth running a loop that does nothing)

Martyr

I could, but I have neither the time or the inclination TBH.

yeah, i know and its because we already know that C/C++ is faster than Java.

marco_polo

Martyr wrote: »

GNU C Compiler, MSVC, Pelles c compiler, watcom..most of them really.

its not exactly something you could miss.(unless you're java programmer, who thinks it would be worth running a loop that does nothing)

Ooh, bitchy.

Martyr wrote: »

yeah, i know and its because we already know that C/C++ is faster than Java.

I am sure for the most part it still is marginally, what I don't understand is why you care so much.

Webmonkey

Martyr wrote: »

GNU C Compiler, MSVC, Pelles c compiler, watcom..most of them really.

its not exactly something you could miss.(unless you're java programmer, who thinks it would be worth running a loop that does nothing)

Where did you learn this?

Martyr

Ooh, bitchy.

i'm only kiddin, marco take it easy.

I am sure it still is for the most part, what I don't understand is why you care so much.

because it would be interesting to see where it goes.

this is a programming forum, isn't it?

its not against the rules to do a little programming now and then and show results of your work.

if you don't care, thats fine, you don't have to participate in the discussion.

Martyr

webmonkey wrote:

Where did you learn this?

learn what? are you talking about the loop? or my joke about java programmers?

the loop is obvious, webmonkey..what is there not to understand? it does nothing.
a c compiler won't generate code for that if used properly.

and the joke is just that..trying to be funny.

Webmonkey

Martyr wrote: »

learn what? are you talking about the loop? or my joke about java programmers?

the loop is obvious, webmonkey..what is there not to understand? it does nothing.
a c compiler won't generate code for that if used properly.

and the joke is just that..trying to be funny.

I don't see why a compiler would complain and prevent it from going on its way to assembly, it is syntactically correct, maybe not semantically but definitely syntactically. Have you ever tried it? - I'm still wondering where you learned it?

NeverSayDie

Webmonkey wrote: »

I don't see why a compiler would complain and prevent it from going on its way to assembly, it is syntactically correct, maybe not semantically but definitely syntactically. Have you ever tried it? - I'm still wondering where you learned it?

Modern compilers do all sorts of optimisation in the process of turning high level code into machine code. It won't complain, it'll just rearrange things to work faster. Ditching clearly redundant stuff like empty loops would be an example. See here for all sorts of crazy detail on the topic, if you're into that kind of thing;
http://en.wikipedia.org/wiki/Compiler_optimization

Re where he learnt it, possibly here;
http://www.javaperformancetuning.com/news/qotm028.shtml
http://www.javaperformancetuning.com/news/qotm029.shtml

Webmonkey

NeverSayDie wrote: »

Modern compilers do all sorts of optimisation in the process of turning high level code into machine code. It won't complain, it'll just rearrange things to work faster. Ditching clearly redundant stuff like empty loops would be an example. See here for all sorts of crazy detail on the topic, if you're into that kind of thing;
http://en.wikipedia.org/wiki/Compiler_optimization

Re where he learnt it, possibly here;
http://www.javaperformancetuning.com/news/qotm028.shtml
http://www.javaperformancetuning.com/news/qotm029.shtml

I'm not referring to Java, I'm referring to the statement of "most compilers", the GNU C compiler for example, especially since he making out C/C++ is faster than Java :pac: - Sorry I should have made it clearer.

Thanks for the links however, I am into that sort of thing :pac:

Martyr

NeverSayDie wrote:

Re where he learnt it, possibly here;

no, i didn't see that page before.

webmonkey wrote:

I'm not referring to Java, I'm referring to the statement of "most compilers", the GNU C compiler for example, especially since he making out C/C++ is faster than Java - Sorry I should have made it clearer.

yes, C/C++ is faster imho, would you like to prove me wrong? won't hold my breath

if java compensates for being a **** programmer, why can't c/c++ compilers do the same?

webmonkey wrote:

Thanks for the links however, I am into that sort of thing

hmm, i've a feeling theres alot of you expert programmers insecure about the end result of going against a noob like myself.

i'm not afraid of being proved wrong here, but i know i'm right

put your money where your mouth is, webmonkey..or any of ye for that matter :P

NeverSayDie

Martyr wrote: »

hmm, i've a feeling theres alot of you expert programmers insecure about the end result of going against a noob like myself.

i'm not afraid of being proved wrong here, but i know i'm right

put your money where your mouth is, webmonkey..or any of ye for that matter :P

Why on earth would anyone here be "insecure" about the end results, unless they'd written the compiler themselves or some such? Maybe you could discuss some of the technical reasons why you know you're right, and make things more useful for the OP?

Anima

Here is a good wikiarticle for java performance, where it is good and bad.

http://en.wikipedia.org/wiki/Java_performance#Comparison_to_other_languages

Seems to tie a lot of what I've been reading together.

Martyr

NeverSayDie wrote:

Why on earth would anyone here be "insecure" about the end results, unless they'd written the compiler themselves or some such? Maybe you could discuss some of the technical reasons why you know you're right, and make things more useful for the OP?

if you give me until sunday night or so, i should have enough time to produce some results on my own using both Java / C or C++

The reason i wanted an experienced Java programmer is so that there could be no arguement on the results, i'm not experienced writing Java but i'm sure i could cook up something..(i'm not experienced with C/C++ either, but i'm guessing it should be faster than Java anyway)

Sure, i can write some code in either and show results but chances are some expert on Java will claim to know a way of doing it better..

i'll do it, just don't have time right now, i'm busy in my call centre :pac: no doubt hobbes,webmonkey and others who disagree with me are programming some huge supercomputer using java or .net...."no time" to prove whether i'm right or wrong, just tell me i'm wrong..

Webmonkey

Martyr wrote: »

if you give me until sunday night or so, i should have enough time to produce some results on my own using both Java / C or C++

The reason i wanted an experienced Java programmer is so that there could be no arguement on the results, i'm not experienced writing Java but i'm sure i could cook up something..(i'm not experienced with C/C++ either, but i'm guessing it should be faster than Java anyway)

Sure, i can write some code in either and show results but chances are some expert on Java will claim to know a way of doing it better..

i'll do it, just don't have time right now, i'm busy in my call centre :pac: no doubt hobbes,webmonkey and others who disagree with me are programming some huge supercomputer using java or .net...."no time" to prove whether i'm right or wrong, just tell me i'm wrong..

Why do you assume I am a java expert. I am a C/C++ programmer and I hate working with Java, simply don't like it.

Where did I try to prove you wrong, I was simply saying that the code presented did not get optimized in my GNU compiler and I did try it.
However i did notice this:

for (j = 0; i < cycle1; i++)
    for(j = 0; j < cycle2; j++)

does get optimized while:

for (i = 0; i < cycle1; i++)
  for (j = 0; j < cycle2; j++)
  {
  }

does not.

Martyr

webmonkey wrote:

Why do you assume I am a java expert. I am a C/C++ programmer and I hate working with Java, simply don't like it.

ok, sorry for assuming, but i expected someone programming with C/C++ to know that most C compilers (if used correctly) would not generate machine code for a loop that does nothing.

Martyr

if compiler is used correctly, and i mean using its optimisation features..i don't think either of those loops would generate ANY machine code.

honestly, where would you see something like this used anyway?

its a bit of a bogus arguement.no offence, its not a practical example, its not a fair example to say java is better based on a loop like that ..