Re: [racket-dev] my '312' this semester, how we compare to others
On May 1, 2011, at 2:58 AM, D Herring wrote: > On 04/21/2011 01:07 PM, Matthias Felleisen wrote: >> -B- When it comes to raw computational performance (ignore loading Racket, >> start from REPL and run a single game -- 10 seconds), my implementation is >> faster than Python (but barely) and one of the Java implementations. But one >> Java implementation is 10x faster. My hunch is that our OO system is >> significantly slower than Java's because we lack types to reduce dispatch >> overhead. Then again I might be wrong. -- Another possibility is that my >> extremely heavy use of our wonderful (!) contracts may impose a large >> overhead. I use mostly ->i and ->m. > > I'd be curious what your timings are with the contract checking disabled. I have recently reported the revised timings to a small list of core Racketeers, but here is the upshot: 1. I found a serious performance bug in one of two central methods during a final code revision. 2. Now the code with contracts runs a game in just over 3s. 3. Disabling contracts lowers this to just over 2s. (These timings are averages of 30 games.) 4. The Racket implementation is now the second fastest implementation in the bunch. I still believe that the Java implementation (just under 1s without their 'Google' contract) benefits from typed dispatches. (There are no significant numeric computations in this game to affect the run time.) -- Matthias _ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev
Re: [racket-dev] my '312' this semester, how we compare to others
On 04/21/2011 01:07 PM, Matthias Felleisen wrote: -B- When it comes to raw computational performance (ignore loading Racket, start from REPL and run a single game -- 10 seconds), my implementation is faster than Python (but barely) and one of the Java implementations. But one Java implementation is 10x faster. My hunch is that our OO system is significantly slower than Java's because we lack types to reduce dispatch overhead. Then again I might be wrong. -- Another possibility is that my extremely heavy use of our wonderful (!) contracts may impose a large overhead. I use mostly ->i and ->m. I'd be curious what your timings are with the contract checking disabled. As for lacking types, types are only useful for primitive optimizations; partial evaluation and code profiling and "inlining" sequential dispatches have the potential to outperform C++ vtables, etc. C/C++ must rely on link-time optimization frameworks and Java is hampered by doing everything at runtime... I think this is a big area of opportunity for lisp systems to regain the performance lead. As to the wider issue, "glue languages" will probably win in the long run; there are huge masses of legacy code, but everybody wants to use a newer language. If you could fairly seamlessly employ a C++ or Java library in lisp... C is dominant largely because it has a simple calling convention. Lisp systems do not... They tend to be frameworks, not externally extensible tools. - Daniel _ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev
Re: [racket-dev] my '312' this semester, how we compare to others
Two points worth noting: 1. Robby pointed out that I forgot to compile my code when I ran the script. That was a critical omission on my side and it eliminates point -A- from my list of negative observations. 2. I forgot to mention the most amazing aspect of my final test run with the students. For their final, we ran their server and my clients. In addition to regular clients, I had nasty ones and bad ones. I wrote one nasty one especially for the C++ project. This client would send the following kind of XML to the server to initiate the dialog: As you can see, it's regular XML but a lot of work. The amazing point: -- the C++ project is the ONLY one (other than my own server) that 'survives' this message. (It fails to truncate the name and thus generates too much network traffic in the end. On occasion it may fail for this reason. To be precise, it times out.) -- ALL other projects (Java, Python) failed on this one. The Java and Python libraries are so clumsy that the programmers naturally make mistakes (five different mistakes in six temas). For full disclosure, the C++ hacker is a systems PhD student (uncommon for this course) and is a Google Chrome part-time developer. -- Matthias _ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev
Re: [racket-dev] my '312' this semester, how we compare to others
I had a request for line counts: -- my own project used ~5,000loc, which includes comments and blank lines -- of these, ~3,200 lines are 'real' code and some ~1,800 lines of rackunit code -- the Java projects run at about 2x to 3x the line count (the best project comes in at close to 12Kloc) -- the C++ project is close to 4Kloc, w/o any tests -- the Python project came out to 2,8000 lines, including tests Some caveats apply of course: ++ Java is extremely verbose and cumbersome, even with parametric polymorphism. ++ Only one project had a larger test ratio than my own. But the project was in Java and see preceding bullet. ++ Only the C++ project has a GUI that is comparable to my own. ++ Not one server lived up to the same quality level as mine. (I sent them a free-formatted string of 1,000,000 lines and that broke everyone's code). ++ Two of seven clients faulted all the time. One faulted on an intermittent basis. And there are other caveats. I am sure I could write more compact Java code than any of the students, though I doubt I'd get close to what I have. I will reflect some more and report on the language comparison a bit more. -- Matthias _ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev
Re: [racket-dev] my '312' this semester, how we compare to others
Matthias Felleisen wrote at 04/21/2011 01:07 PM: -A- I demanded that students deliver their functionality via Unix shell scripts, and so I did so too. My tcsh scripts check the argument number and pass the arguments on to Racket. Firing up one of my clients or servers takes several seconds. All other language implementations appear to bring up their servers or clients instantaneously. I only see slow performance like that from non-GUI Racket when the Racket code is not yet compiled, and perhaps when the files that need to be read (or ELF loaded) by the OS are not in OS caches. The "racket" REPL executable startup *did* seem to get a lot slower a while ago -- a few seconds -- compared to the old "mzscheme" REPL. But 5.1 was less than 1 second to REPL prompt when I tried it just now. Three shell script tips that shouldn't matter much, but might speed you up a little: (1) Consider using "/bin/sh" or "/bin/bash" instead of "tcsh", since the old rule is Bourne for scripting, C-shell for interactive; (2) In the "#!" line, use command line options to disable shell features that load various system and user rc files, since these are slow to read and interpret, and can be huge like completion tables, plus in a production app other people's shell rc files can introduce instability; (3) After your script does the minimal stuff it has to do in shell rather than Racket, use an "exec" to fire off "racket", so that everyone can forget sooner that shell ever happened. -- http://www.neilvandyke.org/ _ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev
[racket-dev] my '312' this semester, how we compare to others
[ In case you don't know: since the late 90s, I have been teaching a course dubbed software dev every so often. Students choose their favorite language, I choose the project, they program, I program in Racket and in DrRacket (and strictly). Over the semester the students code-walk their projects in class and the 'finals' are two-hour code walks with 'stress tests' of their servers working with my clients and vice versa. This gives me some idea of what languages contribute to the standard of programming and what's out there. ] * OBSERVATIONS from this year's '312', starting with some background: (1) this is the first year the course was open to MS students and limited enrollment only. So I ended up with 10 MS students and 6 undergraduates. (2) I saw much less variety than usual. (3) two undergraduates started with Racket but due to team rotations they ended up with Java partners. (4) all MS students fell back into their Java/Eclipse monoculture after having seen Racket/DrRacket in the Bootcamp course for MS students. (5) one pair chose Python (6) one pair chose C++, and lo and behold, this is the first time in 14 years that the C++ pair survived the course. On local basis (ten lines here, ten lines there), the C++ and Python code looks as good as Racket code. Java looks worst. Period. I stuck to Racket's OO world, and I will say that I mostly enjoyed this. Racket didn't provide me with much of an advantage anymore. I did benefit from mixins (twice), custodians (once), proper contracts (huge win) and my first large-scale use of rackunit (another large win). If I had remembered logging in Racket, I would have saved myself 40 lines of code and 2 hours of debugging. Higher-order functions and macros played very minor roles. This is possibly due to my choice of project. (Look for Ingenious board game; distributed; variations on the rules) Bottom line here: My code implements more features, is easier to read, has many fewer bugs, I am more productive than anybody in there. But it's my personal skill set (with Racket) not the language features that give me most of the advantage. ;; --- Let's get to what I learned about how others live and where we fail: * PROBLEMS WITH DRRACKET: compared to the Eclipse-Java monoculture, DrRacket suffers from three major problems as far as I am concerned as a program designer: -1- Eclipse performs 'check syntax' in an on-line fashion. If you are reading a piece of code, and you wish to jump to the definition of some method you're calling, click and you jump. -2- Eclipse autocompletes all the time and in a highly convenient manner. -3- Eclipse displays tooltips on ids with relevant docs. It makes it trivial to learn the APIs and to recall what your own methods do (assuming your types and your purpose statements contain the proper information). DrRacket/Racket is superior to Eclipse in two regards: +1+ Contracts are properly integrated into our tool chain. They are comments and or strings in the rest of the world. +2+ My discipline of using define/provide-test-suite enabled me to keep interfaces narrow and tests accurate and called properly. * PROBLEMS WITH RACKET: our performance suffers from two problems. -A- I demanded that students deliver their functionality via Unix shell scripts, and so I did so too. My tcsh scripts check the argument number and pass the arguments on to Racket. Firing up one of my clients or servers takes several seconds. All other language implementations appear to bring up their servers or clients instantaneously. -B- When it comes to raw computational performance (ignore loading Racket, start from REPL and run a single game -- 10 seconds), my implementation is faster than Python (but barely) and one of the Java implementations. But one Java implementation is 10x faster. My hunch is that our OO system is significantly slower than Java's because we lack types to reduce dispatch overhead. Then again I might be wrong. -- Another possibility is that my extremely heavy use of our wonderful (!) contracts may impose a large overhead. I use mostly ->i and ->m. ;; --- So this is the view from the frontline. I am equally delighted and frustrated this year -- Matthias _ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev
