Re: Call for masters thesis ideas (possibly related to Clojure)
(0) Create a toolkit to run multiple parallel, tightly communicating clojure apps on google-app engine, simulating a single, long-running, multithreaded JVM instance that does not appear, to the user, to be limited by the constraints of GAE's java implementation (e.g. single threading, shared refs work despite being distributed); at the same time, the toolkit would minimize resources consumed in GAE, persisting threads/continuations waiting for data, and heuristically determining lowest cost for long-term storage versus memory and runtime consumption. Note: http://elhumidor.blogspot.com/2009/04/clojure-on-google-appengine.html THE BIG CAVEAT Two unusual aspects of the Google AppEngine environment create pretty major constraints on your ability to write idiomatic Clojure. First, an AppEngine application runs in a security context that doesn't permit spawning threads, so you won't be able to use Agents, the clojure.parallel library, or Futures. Second, one of the most exciting features of AppEngine is that your application will be deployed on Google's huge infrastructure, dynamically changing its footprint depending on demand. That means you'll potentially be running on many JVMs at once. Unfortunately this is a strange fit for Clojure's concurrency features, which are most useful when you have precise control over what lives on what JVM (and simplest when everything runs on one JVM). Since shared references (Vars, Refs, and Atoms) are shared only within a single JVM, they are not suitable for many of their typical uses when running on AppEngine. You should still use Clojure's atomic references (and their associated means of modification) for any state that it makes sense to keep global per-JVM, since there may be multiple threads serving requests in one JVM. But remember JVMs will come and go during the lifetime of your application, so anything truly global should go in the Datastore or Memcache. (1) a clojure implementation of Yahoo's PNUTs, using STM's and all the cool facilities clojure provides: http://research.yahoo.com/files/pnuts.pdf (interesting to have a writeup of a real-world impl alongside comparisons to Google Bigtable and Amazon Dynamo) We describe PNUTS, a massively parallel and geographically distributed database system for Yahoo!'s web applications. The foremost requirements of web applications are scalability, consistently good response time for geographically dispersed users, and high availability. At the same time, web applications can frequently tolerate relaxed consistency guarantees. For example, if a user changes an avatar ... little harm is done if the new avatar is not initially visible to one friend It is often acceptable to read (slightly) stale data, but occasionally stronger guarantees are required by applications. PNUTS provides a consistency model that is between the two extremes of general serializability and eventual consistency ... We provide per-record timeline consistency: all replicas of a given record apply all updates to the record in the same order The application [can] indicate cases where it can do with some relaxed consistency for higher performance [such as reading] a possibly stale version of the record. Some interesting commentary from http://glinden.blogspot.com/2009/02/details-on-yahoos-distributed-database.html http://glinden.blogspot.com/2009/02/details-on-yahoos-distributed-database.html .. * * *When reading the paper, a couple things about PNUTS struck me as surprising: First, the system is layered on top of the guarantees of a reliable pub-sub message broker which acts both as our replacement for a redo log and our replication mechanism. I have to wonder if the choice to not build these pieces of the database themselves could lead to missed opportunities for improving performance and efficiency. Second, as figures 3 and 4 show, the average latency of requests to their database seems quite high, roughly 100 ms. This is high enough that web applications probably would incur too much total latency if they made a few requests serially (e.g. ask for some data, then, depending on what the data looks like, ask for some other data). That seems like a problem. Please see also my August 2006 post, Google Bigtable paperhttp://glinden.blogspot.com/2006/08/google-bigtable-paper.html, which discusses the distributed database behind many products at Google. Please see also my earlier post, Highly available distributed hash store at Amazonhttp://glinden.blogspot.com/2007/10/highly-available-distributed-hash.html, on the distributed database behind some features at Amazon.com. Please see also my earlier posts, Cassandra data store at Facebookhttp://glinden.blogspot.com/2008/08/cassandra-data-store-at-facebook.html and HBase: A Google Bigtable clonehttp://glinden.blogspot.com/2007/07/hbase-google-bigtable-clone.html . Update: One of the developers of PNUTS
Re: Call for masters thesis ideas (possibly related to Clojure)
*Re: Update: One of the developers of PNUTS commentedhttp://glinden.blogspot.com/2009/02/details-on-yahoos-distributed-database.html?showComment=123388434#c1254841206330803677 on this post, pointing out that PNUTS performance is much better in practice (1-10ms/request) when caching layers are in place and making a few comparisons to Bigtable.* Note this very interesting conversation (below) which IMHO has potential for STM implementation in clojure http://clojure.org/refs (excerpt: PNUTs supports row-level transactions too-- they are done through a get followed by test-and-set. This is nothing new: it is called optimistic concurrency control and has been around in database literature for ages, and is also used by BigTable.). ... Can and would clojure help and simplify the construction of a framework for high-volume, high-availability, distributed web apps? Has something like PNUTs been implemented for Clojure and Java Clouds? http://www.blogger.com/profile/07702915403582319413 Brian Cooper http://www.blogger.com/profile/07702915403582319413 said... Hi folks, I'm another member of the PNUTS group at Yahoo! This has been a very interesting discussion; I'm glad you folks are as interested in this stuff as we are. Just to reiterate a few points that Utkarsh brought up: - There's no free lunch for performance, and if you want consistency some writes will have to go cross-datacenter, increasing the average latency. Cross-datacenter communication is required because of our mastership protocol. Consider a user who's record is mastered in California. If she flies to Europe, or a network problem causes her request to be redirected to a datacenter on the East coast, then her write will originate in a non-master datacenter and be forwarded back to the master for that record. In practice this happens 10-20% of the time, just because of the way web requests happen. Even if you had Paxos, and managed to put the local participants in the same datacenter, occasionally a write would originate in the non-local datacenter and pay the cross-datacenter latency to find enough members of the quorum. So this cost is really unavoidable. - You could weaken the consistency, to something like eventual consistency (write anywhere and resolve conflicts later) or even best effort (write anywhere and don't worry about conflicts) and avoid ever paying the cross-datacenter cost. And in fact it is possible to turn off mastership in PNUTS. But then you need a resolution protocol, and until conflicts are resolved inconsistent copies of the data are visible to readers. So again there is no free lunch. - Anonymous is write that this system is not as optimized for scans a la MapReduce. In fact, you make different architectural decisions if you are optimizing for low-latency updates to a geographic database than if you are optimizing for scanning for MapReduce within a single datacenter. We have been able to run Hadoop jobs over PNUTS and get performance that is pretty good, just not as good as a native store (HDFS) optimized for MapReduce. So if you want to transactionally update data with very low latency and occasionally run MapReduce, you can use PNUTS; if you want to always run MapReduce but don't need a lot of high performance updates, use HDFS. - As Utkarsh has said, the hardware used for the paper is not as good as production hardware (e.g. no battery-backed write cache, and other limitations). We hope to publish some numbers from live workloads on the production system soon. .. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Image scaling libraries. Request for recommendations
In my experience, the headless suggestion works fine when running Tomcat6 where apps make use underlying graphics functionality and end up accidentally creating an X display. You have to make sure java gets the option before it attempts to open any displays -- which might happen well before the creation of any UI widgetry, solely for the purpose of picking up Xresources in preparation for any graphics operations you might later invoke (e.g. to get the display's dots-per-inch in order to select fonts and scaling). To prevent that from happening, you must give the headless directive on the Java command-line: -Djava.awt.headless=true ? For example this is what I use to launch tomcat on fedora12, where I employ underlying Java graphics functionality for image processing: JAVA_OPTS=-server -Xms160m -Xmx1024m -XX:PermSize=160m -XX:MaxPermSize=320m -XX:+DoEscapeAnalysis -XX:+UseBiasedLocking -XX:+AggressiveOpts -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintCompilation -XX:+PrintGCDetails -XX:+TraceClassLoading -XX:+TraceClassUnloading -Djavax.servlet.request.encoding=UTF-8 -Dfile.encoding=UTF-8 -DjavaEncoding=UTF-8 -Djava.awt.headless=true -Djava.library.path=/usr/lib64 http://java.sun.com/products/java-media/2D/reference/faqs/index.html Q: I have an application that reads, writes, and processes images (but does not display them) using Java2D. When I run my application on Solaris or Linux, it complains that X11 is not available. What can I do to make my application work in this environment?A: When AWT is initialized, it expects to find an Xserver, regardless of whether it is needed for actual display. Although many image operations using the Image I/O APIs or the JAI optional package might not have any obvious need for display, they often invoke code that needs an AWT resource. For example, calling getGraphics() on a BufferedImage initializes AWT and causes these error messages seen by developers. There is no way to say that a particular API does or does not have this problem; it depends on what particular operations are being invoked, and might also depend on what the application does with the images that is not strictly related to any of the APIs cited above. There are two possible solutions. As of J2SE 1.4 (and above), the preferred solution is to use the headless AWT toolkit. This feature allows you to use the Java2D API in a server-side application without the need for a display environment. To specify the headless environment, run your application with the following system property: -Djava.awt.headless=true Niels http://nielsmayer.com PS: of course, when Java is run headless, you might find some very wacky values for dots-per-inch, other unexpected initial values which can cause your image processing routines to work incorrectly. That might be an aspect of doesn't work in your situation. Ultimately, to produce correct results, one would need to get the DPI of each web-user's display and dynamically compute image scaling per connection. This is unfortunately never done, and is going to be more and more of a problem as we go from tiny portable displays to huge widescreens. Q: Why does (eg) a 10 pt font in Java applications appear to have a different size from the same font at 10pt in a native application? A: Conversion from the size in points into device pixels depends on device resolution as reported by the platform APIs. Java 2D defaults to assuming 72 dpi. Platform defaults vary. Mac OS also uses 72 dpi. Linux desktops based on GTK (Gnome) or Qt (KDE) typically default to 96 dpi and let the end-user customise what they want to use. Windows defaults to 96 dpi (VGA resolution) and also offers 120 dpi (large fonts size) and lets users further specify a custom resolution. So a couple of things can now be seen - The DPI reported by platform APIs likely has no correspondence to the true DPI of the display device - Its unlikely that Java 2D's default matches the platform default. So a typical results is that for Window's default 96 DPI that a 10 pt font in a Java application is 72/96 of the size of the native counterpart. Note that Swing's Windows and GTK LFs do scale fonts based on the system DPI to match the desktop. If you want to do the same in your application you can call java.awt.Toolkit.getScreenResolution() and use this to apply a simple scale to the size you specify for fonts. On Mon, Dec 14, 2009 at 9:25 AM, Albert Cardona sapri...@gmail.com wrote: There's a system property (since jdk 1.4) named java.awt.headless (http://java.sun.com/j2se/1.4.2/docs/guide/awt/AWTChanges.html#headless) that allows using AWT classes in server setting. In my experience, the java.awt.headless property doesn't work. I have not tried since a late 1.5; perhaps 1.6 has it fixed. I have only successfully used java.awt.headless property with java 1.4.2. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to
Re: Parallel garbage collection worthwhile?
Although the Garbage-First garbage collector would be quite useful for Clojure, here I confirming that the new G1 garbabe collector is still in beta at least for a comparative use case (using velocity/groovy/jPython in Xwiki, but not Clojure). Is anybody having success with theis GC in Clojure in a more standalone environment? The Garbage-First garbage collector (G1) is currently in beta. It is not enabled by default. The parallel collector is still the default GC and is the most efficient GC for common household usage. G1 is meant to be an alternative for the concurrent collector. It is designed to be more predictable and enable fast allocation with memory regions design. To use it: -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC For more information about GC G1, please see: - G1 Technical Session in JavaOne 2008 http://developers.sun.com/learning/javaoneonline/j1sessn.jsp?sessn=TS-5419yr=2008track=javase - Java SE 6 GC tunning http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html - Analysis of the Garbage-First Garbage Collection http://research.sun.com/jtech/pubs/04-g1-paper-ismm.pdf In my tests with Xwiki (which loads velocity, groovy, and jPython, but not yet clojure) and Roller running out of Tomcat6, you unfortunately get a Segmentation Fault at the end of the garbage collection (a regression, the early access release ran successfully for 50 minutes under active 5-way-vhosting webload in Tomcat5): # A fatal error has been detected by the Java Runtime Environment: # SIGSEGV (0xb) at pc=0x7f15ca60653a, pid=23777, tid=139729915439440 # JRE version: 6.0_14-b08 # Java VM: Java HotSpot(TM) 64-Bit Server VM (14.0-b16 mixed mode linux-amd64 ) # Problematic frame: # V [libjvm.so+0x19c53a] Caused by adding -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC flags to tomcat process below F S UIDPID PPID C PRI NI ADDR SZ WCHANRSS PSR STIME TTY TIME CMD 0 S tomcat8804 1 0 80 0 - 462172 futex_ 469116 1 Aug13 ? 9:04 /usr/java/default/bin/java -server -Xms160m -Xmx1024m -XX:PermSize=160m -XX:MaxPermSize=320m -XX:+DoEscapeAnalysis -XX:+UseBiasedLocking -XX:+AggressiveOpts -Djavax.servlet.request.encoding=UTF-8 -Dfile.encoding=UTF-8 -DjavaEncoding=UTF-8 -Djava.awt.headless=true -Djava.library.path=/usr/lib64 -classpath :/usr/share/tomcat6/bin/ bootstrap.jar:/usr/share/tomcat6/bin/tomcat-juli.jar:/ usr/share/java/commons-daemon.jar -Dcatalina.base=/usr/share/tomcat6 -Dcatalina.home=/usr/share/tomcat6 -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat6/temp -Djava.util.logging.config. file=/usr/share/tomcat6/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start. Using: XWiki 2.0M2, Sun's jdk-1.6.0_14-fcs.x86_64, Fedora's tomcat6-6.0.18-6.2.fc10.noarch, tomcat-native-1.1.16-1.fc10.x86_64, tomcat6-lib-6.0.18-6.2.fc10.noarch, tomcat6-servlet-2.5-api-6.0.18-6.2.fc10.noarch, on Linux 2.6.27.29-170.2.78.fc10.x86_64 #1 SMP Fri Jul 31 04:16:20 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux via $500 AMD Opteron 1220 2.80GHz Dual Core server http://www.http://www.tigerdirect.com/applications/SearchTools/item-details.asp?EdpNo=4312771CatId=1991 tigerdirect.com/applications/http://www.tigerdirect.com/applications/SearchTools/item-details.asp?EdpNo=4312771CatId=1991 SearchTools/item-details.asp?http://www.tigerdirect.com/applications/SearchTools/item-details.asp?EdpNo=4312771CatId=1991 EdpNo=4312771CatId=1991http://www.tigerdirect.com/applications/SearchTools/item-details.asp?EdpNo=4312771CatId=1991 -- Niels http://nielsmayer.com --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Is there something special about haskell, or could verifiability announcements happen for purely functional Clojure programs too?
Executive summary: seems like big announcements of provably correct software would be easier to achieve in Java+Clojure than for C/Haskell used in the current big news regarding a provably secure kernelhttp://news.cnet.com/8301-1009_3-10310255-83.html?tag=newsEditorsPicksArea.0. However, given the bugs I enountered in a java-based DVR contract-from-hell that was a deadlocking side-effecting nightmarehttp://reviews.cnet.com/digital-video-recorders-dvrs/directv-hr20-directv-plus/4505-6474_7-32065196.html, I think this kind of provability work would be very useful for real world java and clojure running in embedded java whether it be a DVR glitching-out due to an improbable concurrence, or hospital life-support equipment, or aircraft control, military control, etc. Seems like this could also be a good potential marketing opportunity for Clojure, were it to be used as the basis for building provably-correct apps that run in a JVM. I think that market is much bigger than the provable microkernel in C market. E.g. toolkit for building probably secure and reliable cell-phone/multimedia java apps, or toolkit for building embedded controller software on which people's lives depend, literally http://ertos.nicta.com.au/research/l4.verified/ and http://ertos.nicta.com.au/research/l4.verified/approach.pml The L4.verified project A Formally Correct Operating System Kernel In current software practice it is widely accepted that software will always have problems and that we will just have to live with the fact that it may crash at the worst possible moment: You might be on a deadline. Or, much scarier, you might be on a plane and there's a problem with the board computer. Now think what we constantly want from software: more features, better performance, cheaper prices. And we want it everywhere: in mobile phones, cars, planes, critical infrastructure, defense systems. What do we get? Mobile phones that can be hacked by SMS. Cars that have more software problems than mechanical ones. Planes where computer problems have lead to serious incidents. Computer viruses spreading through critical infrastructure control systems and defense systems. And we think See, it happens to everybody. It does not have to be that way. Imagine your company is commissioning a new vending software. Imagine you write down in a contract precisely what the software is supposed to do. And then — it does. Always. And the developers can prove it to you — with an actual mathematical machine-checked proof. Of course, the issue of software security and reliability is bigger than just the software itself and involves more than developers making implementation mistakes. In the contract, you might have said something you didn't mean (if you are in a relationship, you might have come across that problem). Or you might have meant something you didn't say and the proof is therefore based on assumptions that don't apply to your situation. Or you haven't thought of everything you need (ever went shopping?). In these cases, there will still be problems, but at least you know where the problem is not: with the developers. Eliminating the whole issue of implementation mistakes would be a huge step towards more reliable and more secure systems. Sounds like science fiction? The L4.verified project demonstrates that such contracts and proofs can be done for real-world software. Software of limited size, but real and critical. We chose an operating system kernel to demonstrate this: seL4http://ertos.nicta.com.au/research/sel4/. It is a small, 3rd generation high-performance microkernel with about 8,700 lines of C code. Such microkernels are the critical core component of modern embedded systems architectures. They are the piece of software that has the most privileged access to hardware and regulates access to that hardware for the rest of the system. If you have a modern smart-phone, your phone might be running a microkernel quite similar to seL4: OKL4 from Open Kernel Labshttp://www.ok-labs.com/ . We prove that seL4 implements its contract: an abstract, mathematical specification of what it is supposed to do. Current status: completed successfully. More information: - What we prove and what we assumehttp://ertos.nicta.com.au/research/l4.verified/proof.pml (high level, some technical background assumed) - Statistics http://ertos.nicta.com.au/research/l4.verified/numbers.pml (sizes, numbers, lines of code) - Questions and answershttp://ertos.nicta.com.au/research/l4.verified/faq.pml (high-level, some technical background assumed) - Verification approachhttp://ertos.nicta.com.au/research/l4.verified/approach.pml (for technical audience) - Scientific publicationshttp://ertos.nicta.com.au/research/l4.verified/pubs.pml (for experts) - Acknowledgements and teamhttp://ertos.nicta.com.au/research/l4.verified/acks.pml - What does a formal proof look like?
Re: binding and bundles of variables
On Thu, Aug 6, 2009 at 10:12 PM, samppi rbysam...@gmail.com wrote:
I have about six variables that are often rebound together using
binding; these variables are used to access and set data in a state
object (whose type is of the user's choice). These variables' values
(the accessors and setters) are often used together.
I'd like to be able to bundle these values into a map, and using a
function or macro called with-bundle, automatically bind the map's
vals to their respective variables:
(def my-bundle
{'*variable-1* :something1, '*variable-2* :something2})
If you don't get too religious about the curlyness of your brackets or the
inelegance of the syntax (which is basically common-lisp keywords, using
infix notation, with the entailed loss of elegance caused by that
choice)... What about using JSON, which seems to be close to the syntax
you're already using. Because JSON is ultimately a matter of calling a Java
class,all the work has already been done in terms of going back and forrth
between a map datastructure and the text-representation, both for JSON and
XML.
For example (cut/paste from http://www.javapassion.com/ajax/JSON.pdf )
JSON looks like:
var myJSONObject = {bindings: [
{ircEvent: PRIVMSG, method: newURI, regex: ^http://.*},
{ircEvent: PRIVMSG, method: deleteURI, regex: ^delete.*},
{ircEvent: PRIVMSG, method: randomURI, regex: ^random.*}
]
};
• Members can be retrieved using dot or subscript operators
myJSONObject.bindings[0].method // newURI
In Java, the same dot notation can be used for set/get, but use JSONObject
Java Class
• A JSONObject is an unordered collection of name/value pairs
• The put methods adds a name/value pair to an object
• The texts produced by the toString methods strictly conform to the JSON
syntax rules
Output:
myString = new JSONObject().put(JSON, Hello, World!).toString();
// myString is {JSON: Hello, World}
Parsing:
try {
jsonObject = new JSONObject(myString);
}
catch(ParseException pe) {
}
.
Similarly you could probably also use whatever combination of XML libraries
that'll turn XML structures into
data structures that can easily be accessed in Java, for example, here's a
little script you can throw into Xwiki to grab a RSS feed and turn it into
something that can easily be accessed in Java, Groovy, or Clojure:
Takes URL param ?zip=ZIPCODE and gets associated weather report for that
zipcode:
{{script language=groovy}}
def rss = new XmlSlurper().parseText((
http://weather.yahooapis.com/forecastrss?p=;
+
xcontext.macro.params.zip).toURL().text)
println rss.channel.title
println Sunrise: ${rss.channel.astrono...@sunrise}
println Sunset: ${rss.channel.astrono...@sunset}
println Currently:
println \t + rss.channel.item.conditi...@date
println \t + rss.channel.item.conditi...@temp
println \t + rss.channel.item.conditi...@text
{{/script}}
Nielshttp://nielsmayer.com
PS: what are people's favorite JSON or XML Java classes for doing this out
of scripting languages like Groovy or Clojure? Basically, going back and
forth between a structured textual representation and a map data structure
that can be accessed by variables like rss.channel.item.condition.text .
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
-~--~~~~--~~--~--~---
Re: JSR 223
http://www.xwiki.org/xwiki/bin/view/Main/ReleaseNotesXWikiEnterprise20M2On Fri, Jul 31, 2009 at 9:04 AM, Mike cki...@gmail.com wrote: Are there plans for Clojure to work in JSR-223 (Java Scripting Framework)? I see a lot of languages up there already. In fact, the only major JVM languages missing are Scala and Clojure. In http://groups.google.com/group/clojure/browse_thread/thread/297861670c2f5611 (orig http://lists.xwiki.org/pipermail/devs/2009-July/013763.html ) I wrote: Clojure has a JSR-223 implementation according to: http://github.com/pmf/clojure-jsr223/tree/master http://groovy.codehaus.org/JSR-223+access+to+other+JVM+languages http://sayspy.blogspot.com/2009/03/interacting-between-jvm-lang-here-and.html I've been told I'd need to Build the Clojure JSR223 engine from http://github.com/pmf/clojure-jsr223/tree/master and place it inside WEB-INF/lib in order to add Clojure as a scripting language to Xwiki (alongside existing velocity/groovyhttp://platform.xwiki.org/xwiki/bin/view/DevGuide/Scriptingand jRuby http://code.xwiki.org/xwiki/bin/view/Macros/RubyMacro scripting implementations). Any special instructions or suggestions or gotchas in creating the clojure jsr223 engine? (I'm still upgrading to Xwiki 2.0M2http://www.xwiki.org/xwiki/bin/view/Main/ReleaseNotesXWikiEnterprise20M2prior to starting my clojure experiment, so any prior advice before doing anything stupid -- e.g. running the jsr223 engine on an incompatible version of clojure -- would be helpful.) Niels http://nielsmayer.com PS: I've always seen xwiki as the emacs of webapps (and wikis)... So I'm looking forward to having a real emacsish type language -- clojure -- to extend it via a more appropriate language for scripting. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Java based DocDB
On Mon, Jul 27, 2009 at 7:46 AM, Sean Devlin francoisdev...@gmail.com
wrote:
Howdy everyone,
I've got a project that needs a Doc DB. It will need to run on
windows (and probably OS X), so I am looking for something that works
with Java. I thought I'd ask for some help here before I re-invent
the wheel.
1. Has anyone here worked with a Java based doc DB?
2. Does anyone have Clojure bindings for said doc DB?
3. Does anyone know of good guide for said doc DB?
Take a look at Xwiki http://xwiki.org, a 5-year-old open source project,
implemented in Java, which has database-backed documents, objects that can
be attached to documents, collaborative document versioning, and the ability
to write scripts in Velocity or
Groovyhttp://platform.xwiki.org/xwiki/bin/view/DevGuide/Scripting
and
jRuby http://code.xwiki.org/xwiki/bin/view/Macros/RubyMacro, and hopefully
if I have my way with it, Clojure too:
http://groups.google.com/group/clojure/browse_thread/thread/297861670c2f5611 .
This is an old Google Tech Talk on Xwiki's documents, and application
programming in Xwiki: http://www.youtube.com/watch?v=xs3LuzwqemM .. They've
gone 2.0 since this talk, but the overall flavor is the same.
For example, here's a trivial app I wrote which queries the Xwiki document
database that comprises my website, and returns a timeline of documents
published on the site:
http://nielsmayer.com/xwiki/bin/view/Main/SiteTimeline
(source code xar/jar:
http://nielsmayer.com/xwiki/bin/download/Main/SiteTimeline/SiteTimelinePkg.xar)
http://nielsmayer.com/xwiki/bin/view/Main/SiteTimelineThe wiki
codehttp://nielsmayer.com/xwiki/bin/view/Main/SiteTimeline?viewer=codeof
this app is boilerplate; the main business of this app happens inside
the
document's XWiki.JavaScriptExtension[0] object, via Hibernate Query,
processed via the Xwiki API, and scripted in Velocity. This code is used
to generate Javascript for the Simile-Widgets
Timelinehttp://simile-widgets.org/timeline/used for the display.
(The Xwiki Java API calls are prefixed with $xwiki.
$util. $request. below):
#set( $at_start_p = true )## #set( $hqlQuery = , XWikiRCSNodeInfo as ni
where doc.id=ni.id.docId and ni.id.version2=1 and doc.space'XWiki' and
doc.space'Main' and doc.space'Panels' group by doc.space, doc.name order
by max(ni.date) desc)## #set( $recentlyChangedDocuments =
$xwiki.searchDocuments($hqlQuery, $util.parseInt($request.changesNb), 0))##
var tl_events = [ #foreach( $rcName in $recentlyChangedDocuments )##{ #set(
$rcDoc = $xwiki.getDocument($rcName) )## #set( $docTitle =
$rcDoc.getDisplayTitle().replaceAll('','\\') )## #set( $docEpoch =
$rcDoc.date.getTime() )## #if($at_start_p == true) ## { #set( $at_start_p =
false )## #else ## }{ ,## #end ## } new Timeline.DefaultEventSource.Event({
start: epoch_date($docEpoch), end: null, latestStart: epoch_date($docEpoch),
latestEnd: null, instant: true, text: $docTitle, description:
TABLETBODYTRTHLink: /THTDA href='${rcDoc.getURL()}'
target='_blank' ${rcName}/A/TD/TRTRTHDate:
/THTD$xwiki.formatDate($rcDoc.date)/TD/TR #if($!rcDoc.author !=
) TRTHAuthor: /THTDA
href='${xwiki.getURL(${rcDoc.author},view)}' target='_blank'$
$xwiki.getLocalUserName(${rcDoc.author}, false)/A/TD/TR #end
#if($!rcDoc.creator != ) TRTHCreator: /THTDA
href='${xwiki.getURL(${rcDoc.creator},view)}' target='_blank'
$xwiki.getLocalUserName(${rcDoc.creator}, false)/A/TD/TR #end
/TBODY/TABLE, image: null, link: null, icon: null, color: null,
textColor: null }) #end ];
It would be much nicer to be able to use Clojure to dynamically generate
JSON on the server side. That's certainly one of the things you can do with
Xwiki, even though normally you'd be using it to dynamically generate HTML
documents or applications. For example, I'm looking at using closure to
access the Xwiki API to persist some information from my application in
Xwiki documents (everything in xwiki is a wiki-document) implemented via
Exhibit http://simile-widgets.org/exhibit/, that look like this:
http://nielsmayer.com/xwiki/bin/view/Exhibit/NPRpods3 (a work in progress).
Niels
http://nielsmayer.com
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
-~--~~~~--~~--~--~---
Re: Parallel garbage collection worthwhile?
FYI, some potentially useful info here: http://n2.nabble.com/Does-JDK6u14-%22Garbage-First-garbage-collector-%28G1%29%22-work-and-or-improve-Xwiki-performance-size-mem-locality--tp2344358p2344358.html Does JDK6u14 Garbage-First garbage collector (G1) work and/or improve Xwiki performance/size/mem-locality? *Date:* Feb 17, 2009; 05:13pm *Author:* Niels Mayer *XWiki- Dev* (http://n2.nabble.com/XWiki--Dev-f475773.html) I usually have found that the garbage collector is the hardest-working thing in a java-web app (unfortunately) so when I noticed this new GC option in *Java HotSpot 14*, I figured it might help (potentially a lot w/ high-volume sites, which are the real GC churners). Has anybody tried the Garbage-First garbage collector (G1) with Xwiki? http://platform.xwiki.org/xwiki/bin/view/AdminGuide/Performances suggests CATALINA_OPTS=-server -Xms128m -Xmx1024m -XX:MaxPermSize=128m -Dfile.encoding=utf-8 -Djava.awt.headless=true -XX:+UseParallelGC -XX:MaxGCPauseMillis=100 However these instructions don't work, as-is, since UseParallelGC is the default nowadays: https://jdk6.dev.java.net/6uNea.html says: The parallel collector is still the default GC and is the most efficient GC for common household usage. G1 is meant to be an alternative for the concurrent collector. It is designed to be more predictable and enable fast allocation with memory regions design. Here's more info Java SE 6 Update 14 Milestone Schedule b01 (1st available build) 02/11/09 FCS Q2, 2009 *Feature List:* - *Service Tag creation for Windows JRE (CR 6676849 http://bugs.sun.com/view_bug.do?bug_id=6676840 )*: For more information about Service Tag, check out this technical article http://java.sun.com/developer/technicalArticles/productregistration/ . - *Java HotSpot 14*: including the new garbage collector, G1, Big Decimal enhancements, TreeMap and HashMap enhancements, optimized compressed OOP code generation. The Garbage-First garbage collector (G1) is currently in beta. It is not enabled by default. The parallel collector is still the default GC and is the most efficient GC for common household usage. G1 is meant to be an alternative for the concurrent collector. It is designed to be more predictable and enable fast allocation with memory regions design. To use it: -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC For more information about GC G1, please see: - G1 Technical Session in JavaOne 2008 http://developers.sun.com/learning/javaoneonline/j1sessn.jsp?sessn=TS-5419yr=2008track=javase - Java SE 6 GC tunning http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html - Analysis of the Garbage-First Garbage Collection http://research.sun.com/jtech/pubs/04-g1-paper-ismm.pdf Niels http://nielsmayer.com PS: some things that cause big-growth, like importing and exporting, might not grow as large with a generation scavenging style GC as provided by Garbage-First Collection. Sometimes, just GCing a large structure being iterated-over uses a lot more memory than it needs to; because the gc is letting objects that should get collected early, like the incremental results of an iteration, build up and increase overall memory size while decreasing locality and cache-hits. This seems to cause a nearly exponential performance dropoff when very little memory churn might occur if only things got collected at the right time. http://research.sun.com/jtech/pubs/04-g1-paper-ismm.pdf suggests this new GC will help: 2.4 Generational GarbageFirst Generational garbage collection [34, 26] has several advantages, which a collection strategy ignores at its peril. Newly allocated objects are usually more likely to become garbage than older objects, and newly allocated objects are also more likely to be the target of pointer modi cations, if only because of initialization. We can take advantage of both of these properties in Garbage-First in a flexible way. We can heuristically designate a region as young when it is chosen as a mutator allocation region. This commits the region to be a member of the next collection set. In return for this loss of heuristic flexibility, we gain an important benefit: remembered set processing is not required to consider modifications in young regions. Reachable young objects will be scanned after they are evacuated as a normal part of the next evacuation pause. Note that a collection set can contain a mix of young and non-young regions. Other than the special treatment for remembered sets described above, both kinds of regions are treated uniformly. ... 2.5 Concurrent Marking Concurrent marking is an important component of the system. It provides collector completeness without imposing any order on region choice for collection sets (as, for example, the Train algorithm of Hudson and Moss [22] does). Further, it provides the live data information that allows regions to be collected in \garbage-first order. This section
Re: easy way to get Closure doing serious web apps?
easy way to get Closure doing serious web apps? Ooops. That's supposed to read Clojure in the title. Sorry about that. By the way, I previously wrote: * It could be very useful to employ massive parallelism via such Clojure* * scripts, which could achieve a xwiki-based web portal performance akin to * * Yahoo's, Google's, etc. For example, the following describes how Yahoo works* * -- and would be quite easy to implement this kind of processing for free* * in Clojure with very little code: http://research.yahoo.com/files/pnuts.pdf ...* * * Wondering whether Google App Engine would be able to host such an application, I check http://elhumidor.blogspot.com/2009/04/clojure-on-google-appengine.html which says: THE BIG CAVEAT Two unusual aspects of the Google AppEngine environment create pretty major constraints on your ability to write idiomatic Clojure. First, an AppEngine application runs in a security context that doesn't permit spawning threads, so you won't be able to use Agents, the clojure.parallel library, or Futures. Second, one of the most exciting features of AppEngine is that your application will be deployed on Google's huge infrastructure, dynamically changing its footprint depending on demand. That means you'll potentially be running on many JVMs at once. Unfortunately this is a strange fit for Clojure's concurrency features, which are most useful when you have precise control over what lives on what JVM (and simplest when everything runs on one JVM). Since shared references (Vars, Refs, and Atoms) are shared only within a single JVM, they are not suitable for many of their typical uses when running on AppEngine. You should still use Clojure's atomic references (and their associated means of modification) for any state that it makes sense to keep global per-JVM, since there may be multiple threads serving requests in one JVM. But remember JVMs will come and go during the lifetime of your application, so anything truly global should go in the Datastore or Memcache. The first point means closure isn't that interesting on GAE. Any new information to dispel the fact that GAE's Java is single threaded ? That seems pointless. Why even use Java, use something like server-side JavaScript, like http://www.freebase.com/docs/acre ? :-) Regarding the second point: why would one be using shared references in code that one wanted purely functional for the purpose of making it decompose into parallel code without needing to worry about any of the issues raised? How could one be doing functional programming if side-effects are happening to the shared references? And how could one expect that code to decompose into parallel code w/o explicitly handling locking on shared references that are being changed? As followup to GAE limitations, see http://groups.google.com/group/google-appengine-java/web/will-it-play-in-app-engine?pli=1 which points to https://sites.google.com/a/wildstartech.com/adventures-in-java/Java-Platform-Enterprise-Edition/JavaServer-Faces/sun-javaserver-faces-reference-implementation/configuring-jsf-20-to-run-on-the-google-appengine !-- Disable use of threading for single-threaded environments such as the Google AppEngine. -- ... If the com.sun.faces.enableThreading context parameter is not set to false (or omitted completely from the web.xml configuration file) an application deployed on the Google AppEngine will display a number of different errorshttps://sites.google.com/a/wildstartech.com/adventures-in-java/Java-Platform-Enterprise-Edition/JavaServer-Faces/sun-javaserver-faces-reference-implementation/configuring-jsf-20-to-run-on-the-google-appengine/samples-of-threading-issues all caused by the JSF reference implementation's attempts to construct threads. Sadly, current stable releases of Xwiki probably won't be well supported on GAE, either, due to lack of proper database support. http://groups.google.com/ goog_1248744194084group/google-appengine-java/goog_1248744194084 web/will-it-play-in-app- goog_1248744194084engine?pli=1http://groups.google.com/group/google-appengine-java/web/will-it-play-in-app-engine?pli=1 states: http://groups.google.com/group/google-appengine-java/web/will-it-play-in-app-engine?pli=1 You cannot currently use Hibernate directly. The differences between the App Engine datastore and SQL were too great to get the standard Hibernate up and running under App Engine. App Engine does support JDO and JPA, so you may be able to convert your Hibernate code to use one of these ORM interfaces. I believe, future releases of Xwiki will better support JDO/JPA through the XWQLhttp://dev.xwiki.org/xwiki/bin/view/Design/XWiki+Query+Language+Specification effort. Current work, e.g. http://xwiki1.appspot.com/ or http://xwiki1.appspot.com/bin/view/Main/ seems to be based on a non-hibernate-based datastore. -- Niels http://nielsmayer.com --~--~-~--~~~---~--~~ You received this message because
easy way to get Closure doing serious web apps?
(assuming you agree that xwiki-backed sites like http://www.idiva.com ,
http://www.curriki.org or http://www.bestventes.com constitute
serious web apps...)
On the Xwiki devs list, I inquired:
http://lists.xwiki.org/pipermail/devs/2009-July/013763.html about using
Clojure as a scripting language in Xwiki, via JSR-223. I got two very
helpful replies:
(1) http://lists.xwiki.org/pipermail/devs/2009-July/013764.html
*Asiri Rathnayake* *Sun Jul 26 10:43:25 CEST 2009*
[...]
* Clojure http://groups.google.com/group/clojure has a JSR-223 *
implementation
according to:
http://github.com/pmf/clojure-jsr223/tree/master
* http://groovy.codehaus.org/JSR-223+access+to+other+JVM+languages*
http://sayspy.blogspot.com/2009/03/interacting-between-jvm-lang-here-and.html
Are there any examples, documentation, or suggestions of writing a
script* macro to call a new jar in WEB-INF/lib ?*
1. Put the Clojure jar file
http://repo1.maven.org/maven2/org/clojure/clojure/1.0.0/clojure-1.0.0.jar
inside WEB-INF/lib directory
2. Build the Clojure JSR223 engine from
http://github.com/pmf/clojure-jsr223/tree/master and place it inside
WEB-INF/lib
3. Restart your xwiki and use the script macro as below:
{{script language=Clojure}}
// Put your clojure script here.
{{/script}}
(Note that since context is a reserved binding in JSR-223 specifications
used for Scripting Macros, the XWiki Api Context is now accessible through
xcontext.)
I haven't myself tried out clojure inside xwiki but I built a ruby
macrohttp://code.xwiki.org/xwiki/bin/view/Macros/RubyMacro(for
2.0M2) just by following the steps I described to you :)
Question: does anybody have any hints, warnings, potential gotchas
regarding this part: Build the Clojure JSR223 engine from
http://github.com/pmf/clojure-jsr223/tree/master and place it inside
WEB-INF/lib ??
(2) http://lists.xwiki.org/pipermail/devs/2009-July/013767.html
From: Vincent Massol
Date: Sun, Jul 26, 2009 at 10:21 AM
[...]
Well all you need is drop the JSR223 jar in WEB-INF/lib and then you can use
the new language using the script macro:
http://code.xwiki.org/xwiki/bin/view/Macros/ScriptMacro
Then if you want you can write a wrapping macro such as we've done for
groovy and jruby so that it can be called with {{clojure}} instead of
{{script language=clojure}}.
* PS: Here's examples of easy scripted java programming you can do in*
* Clojure (note the helpful parallelism constructs):*
*
http://travis-whitton.blogspot.com/2009/07/network-sweeping-with-clojure.html
*
***
http://travis-whitton.blogspot.com/2009/06/clojures-agents-scientists-monkeys-and_18.html
*
**
*** It could be very useful to employ massive parallelism via such Clojure*
* scripts, which could achieve a xwiki-based web portal performance akin to
*
* Yahoo's, Google's, etc. For example, the following describes how Yahoo
works*
* -- and would be quite easy to implement this kind of processing for
free*
* in Clojure with very little code:*
* http://research.yahoo.com/files/pnuts.pdf*
I'm definitely +1 to make it part of the XWiki platform (same as for jruby)
if you develop it. Similar to jruby I don't think we should
package it in the default WAR though but as an optional download to install
(will be even easier with the app manager).
Niels
http://nielsmayer.com
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
-~--~~~~--~~--~--~---
Re: Clojure as a CGI language?
On Wed, Jul 22, 2009 at 8:58 AM, Mike Hinchey hinche...@gmail.com wrote: As Daniel mentioned, Google App Engine can host java. It's very easy, just upload a war with your clj AOT-compiled. See http://elhumidor.blogspot.com/2009/04/clojure-on-google-appengine.html Very interesting! It also looks like there's also emerging support for Xwiki on Google App Engine: http://xwiki1.appspot.com/bin/view/Main/ ( http://markmail.org/thread/krlqzdzgia2j72r7 ) Support for java-based scripting is provided via xwiki's default scripting language, Velocity and JSR223. Xwiki has good support for JSR223-based scripting languages in Java: http://jerome.myxwiki.org/xwiki/bin/view/Blog/GroovyWebConsoleInXWiki Previously I asked: If Clojure has a JSR-223 implementationhttp://www.theserverside.com/news/thread.tss?thread_id=26698, you could easily leverage all the same Xwiki API's you normally access from Velocity or Groovy: VincentMassol | 2009/05/05 20:03 Niels, no there's no plan but we can now support any language using macros so feel free to provide a clojure macro if you're interested. Shouldn't be too hard to do, especially if there's a JSR-223 implementation (in which case simply dropping the jar in WEB-INF/lib should be enough - you'd then use it using the script macro). Answers own question: http://github.com/pmf/clojure-jsr223/tree/master http://groovy.codehaus.org/JSR-223+access+to+other+JVM+languages http://sayspy.blogspot.com/2009/03/interacting-between-jvm-lang-here-and.html Niels http://nielsmayer.com --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Clojure as a CGI language?
On Mon, Jul 20, 2009 at 2:01 PM, robert.e.hick...@googlemail.com robert.e.hick...@googlemail.com wrote: I was wondering what the best way of using clojure as a language for writing CGI web applications would be. I know that it can be used for web programming using java web servers, however I want to be able to develop application which can be simply uploaded to a server and `just work', like PHP. for which CGI is the only option I know of. Trying to use CGI sounds like a bad idea. It's always full of security issues http://www.w3.org/Security/Faq/wwwsf4.html. Why reinvent the wheel? App development using a database-backed web development environment like Xwiki gets rid of a lot of the issues that might give one pause programming traditional java web servers, most of which is boilerplate cruft that should just be hidden away and encapsulated by a higher level interface which is exactly what Xwiki gives in terms of app-development. I don't think it would be *that* hard (compared to the heavy lifting already done) to support Clojure as a scripting language in java-based Xwikihttp://www.xwiki.org. Xwiki already has strong support for scripting in Groovy and Velocityhttp://platform.xwiki.org/xwiki/bin/view/DevGuide/Scripting. (Although this video is old and talks about Xwiki 1.0 (2.0 is nearing release) it's still a good intro to app-development in Xwiki: http://www.youtube.com/watch?v=xs3LuzwqemM ) http://code.xwiki.org/xwiki/bin/view/Snippets/DisplayFormatedCodeAttachmentsSnippet provides a suggestion for extending Xwiki with Clojure as a scripting language. Xwiki already has extensive java plugins that create a library for web app-development that you script in either Velocityhttp://www.weather.gov/ost/SEC/AE/Site_Migration/Velocity%20User%20Guide.pdfor Groovy http://www.jcp.org/en/jsr/detail?id=241. If Clojure has a JSR-223 implementationhttp://www.theserverside.com/news/thread.tss?thread_id=26698, you could easily leverage all the same Xwiki API's you normally access from Velocity or Groovy: VincentMassol | 2009/05/05 20:03 Niels, no there's no plan but we can now support any language using macros so feel free to provide a clojure macro if you're interested. Shouldn't be too hard to do, especially if there's a JSR-223 implementation (in which case simply dropping the jar in WEB-INF/lib should be enough - you'd then use it using the script macro). I hope someone is inspired to try this out. I think it could be a surprisingly easy way to do web-programming in Lisp, by leveraging a flexible platform like Xwiki. Niels http://nielsmayer.com --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
