So I'm working on developing a Clojure api for a distributed data analysis
framework. A key part of that has to be the ability to pass functions as
parameters into various map, reduce, filter, groupBy, etc. functions.
Since this a framework developed in Scala, these function parameters
eventually become Scala Function1 or Function2. What I really want is to
be able to use this framework interactively from the Clojure repl, much as
it can already be done from the Scala repl. For example, it is possible in
the Scala repl to do something like:
aDataSet.map(n => n + 1)
which, of course, increments every element in the data set by 1. What I am
eventually working toward is being able to do the equivalent to the above
Scala interaction in Clojure:
user=> (framework-map #(+ % 1) aDataSet)
but as an intermediate step, I'm trying to get named function-like
parameters working before tackling anonymous functions. So, I'd be happy
right now with:
user=> (def f (...))
#'user/f
user=> (framework-map f aDataSet)
I can actually get this to work if I use the Java api to the framework and
use gen-class to AOT compile a class and then create an instance equivalent
to:
FrameworkFunction<Integer, Integer> f = new FrameworkFunction<Integer,
Integer>()
{public Integer call(Integer n) {return (n + 1)}}
But needing to jump through all of the AOT hoops every time the user needs
to pass a function as a parameter would put a major hitch in the
interactive giddyup, so this isn't really an acceptable solution (although
it does appear to work fine to statically create batch jobs for the
framework, or to create a library of pre-defined functions that can passed
in from the repl.) I haven't really tried to use the underlying Scala api
directly from the Clojure repl, so if someone sees a way to make that work,
then I'll gladly use less wrapper layers. Using the Java api from the
Clojure repl mostly works, but my major hangup is with not being able to
get a useful instance of the abstract FrameworkFunction class (and similar
classes.) I can use proxy to create an instance that works equivalently
directly within the repl to an instance of an AOT class, so I can do:
user=> (.call f 2)
3
regardless of whether f is an instance of a concrete AOT class extending
FrameworkFunction or f is a proxy extending FrameworkFunction. But trying
to pass the proxy as a parameter leads to trouble when the framework tries
to do things with the .class file. For example, when the following chunk
of Scala code is hit:
private def getClassReader(cls: Class[_]): ClassReader = {
logInfo("attempted class name: " + cls.getName.replaceFirst("^.*\\.",
"") + ".class")
new ClassReader(cls.getResourceAsStream(
cls.getName.replaceFirst("^.*\\.", "") + ".class"))
}
the attempt to generate an ASM class reader from Function$0.class fails
when using the proxy (IOException Class not found
org.objectweb.asm.ClassReader.a (:-1)), but works fine using the AOT
instance.
So, the upshot of all of this is that I am wondering how I should go about
creating in the repl instances of this FrameworkFunction abstract class
that can be passed into the framework HOFs. Even better would be if
someone could explain to me how in the repl to convert Clojure functions
into Scala functions so that I could then pass them directly into the
framework with less need for the Java wrapper.
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to [email protected]
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
