On Friday, 13 September 2013 12:11:50 UTC+8, tbc++ wrote:
> >> If we can do away with the interface generation, then support for
> arbitrary primitive arguments should become relatively straightforward.
>
> How would we even call a function without an interface? Remember
> everything is defined behind vars, so you simply can't assume that a
> function won't be re-defed without notice. This is the reason interfaces
> exist notice how the following works:
>
> user=> (defn foo ^long [^long x] x)
> #'user/foo
> user=> (defn baz [x] (foo x))
> #'user/baz
> user=> (baz 42)
> 42
> user=> (defn foo ^double [^double x] x)
> #'user/foo
> user=> (baz 42)
>
> ClassCastException user$foo cannot be cast to clojure.lang.IFn$LL
> user/baz (NO_SOURCE_FILE:1)
> user=>
>
Good example to discuss!
I agree there is a subtle tradeoff between performance and dynamism here.
Any interface dispatch via a var uses an extra level of indirection (in the
dynamic var case, this includes a thread-local check for any bindings) and
already adds some overhead. If you are happy using Clojure's dynamic
features to pass arbitrary (boxed / Object) arguments and redefine vars on
the fly like that, you probably don't really care about optimal performance
very much anyway and a standard IFn invoke should be OK.
For performance-sensitive code, you ideally want primitive-typed code to be
compiled to a direct method call (invokevirtual) and not go via any var
lookup / interface invoke at all. This would mean you would also need to
redefine/recompile baz if you wanted to use a changed version of foo, but
that's a relatively minor inconvenience IMHO. If you redefine baz after
redefining foo (e.g. by reloading the whole namespace), then everything
will still work. I'm not sure if it is possible to get this precise
behaviour in Clojure at present - but it would certainly be great for high
performance code.
If we want *both* maximum performance and dynamic flexibility, then I think
we need a completely different approach. Clojure vars as currently designed
won't cut it. Probably something like an on-demand re-compilation strategy
based on a var dependency graph would be needed. I personally think that's
the right long-term direction, but that's definitely Clojure 2.0 territory.
>
> Do a prototype with definterface/genclass or something and then report
> your results. My suspicions are that you'll find doing this without
> interfaces, while maintaining Clojure's semantics, is impossible.
>
You may be right. Though it also probably depends on what you consider
documented (promised not to change in public API) vs. undocumented
(implementation detail) semantics. Some of the subtleties on what happens
to existing code when you redefine vars are arguably in the latter
category. Observe:
(defn foo ^double [^double x] (inc x))
(defn baz [x] (foo x))
(baz 42)
=> 43.0
(ns-unmap *ns* 'foo)
(defn foo ^double [^double x] (dec x))
(baz 42)
=> 43.0
i.e. it's fairly easy to trick Clojure into keeping an old version of a var
around. Does that count as officially documented behaviour we are obliged
to maintain?
Either way, if Clojure's semantics prove to be a fundamental issue for
performance, then I think it is better to start work to improve Clojure's
semantics (perhaps targeting 2.0 if we think it's a really big breaking
change). This would be better, IMHO, than forever accepting semantics that
prevent idiomatic code from ever being truly fast. I'd rather see a bit of
breakage and fix my code when upgrading to Clojure 2.0 than be stuck with
poor performance forever.
>
> Timothy
>
>
> On Thu, Sep 12, 2013 at 8:41 PM, Mikera <mike.r.an...@gmail.com<javascript:>
> > wrote:
>
>> People *have* run into this problem a lot. People have just worked around
>> it. The existence of great libraries like HipHip and vectorz-clj and some
>> of Zach's stuff is IMHO to a large extent because people have needed high
>> performance operations (often involving primitives / primitive arrays /
>> custom JVM types) and have found it easier to implement these in a new
>> library than wait for the support to appear in Clojure itself.
>>
>> So we have lots of great, performance-oriented libraries. Cool.
>>
>> But not having comprehensive JVM primitive support is still a big problem
>> for Clojure. It means that:
>> a) We'll get lots of ecosystem fragmentation as people with slightly
>> different requirements implement their own solutions in slightly
>> different/incompatible ways. We're seeing this already. Lisp Curse etc.....
>> b) People with very strong performance requirements on the JVM (e.g.
>> OpenGL, big data numerics, scientific computing, heavy duty websites) are
>> more likely to avoid Clojure and do their work in Scala / Java instead. The
>> community will lose exactly the people who have both the motivation and
>> skills to fix these kind of issues.
>> c) We won't get the wider reputation for performance that I think Clojure
>> deserves and should aspire to. And this is important - rightly or wrongly,
>> many people are attracted to languages with good performance credentials.
>>
>> I'd really love to be able to write everything in pure idiomatic Clojure
>> and still guarantee great performance. Currently I can't, and in a large
>> part this is due to JVM type-related issues like the following:
>> 1) Not all primitive types are fully supported as function args and
>> return values (the issue raised in this thread)
>> 2) Lack of an efficient way to extend abstract base types (CLJ-1255 in
>> Jira, for those interested)
>> 3) Lack of support for statically compiled reference types as function
>> args (avoiding unnecessary instance checks and casting, CLJ-1256)
>> 4) The compiler still isn't very clever about using type inference to
>> eliminate boxing / typecasts (perhaps CinC can help here?)
>>
>> Personally I'm less interested in floats (when I'm doing numerical stuff,
>> double precision is usually more important), but I've certainly wanted ints
>> many many times. I'm sure other people have wanted chars or bytes for
>> various reasons (text processing? IO?). The point is that there are many
>> valid reasons to use the primitive types on the JVM. I guess somebody even
>> has a use for "short" somewhere...... ;-)
>>
>> Now let's get practical.
>>
>> What we need, I think, is some kind of "official" recognition that full
>> JVM primitive support (i.e. all primitives supported in all situations) is
>> an objective for Clojure in 1.6 or maybe 1.7. Then people can put in the
>> time / effort to implement it. I'm personally very happy to help implement
>> this myself providing that there is agreement that full primitive support
>> is the way we want to go. Without official blessing, I'm unlikely to spend
>> my weekends on speculative work that might not be accepted or appreciated.
>> I suspect this is true for others too.
>>
>> If, on the other hand, the "official" answer is "Clojure will never
>> support anything other than Objects, doubles and longs" then I'd want to
>> know that too. I think that would be a *really* poor design decision
>> because good compiled performance and good JVM interop ("embracing the host
>> platform") seem to be two key principles of Clojure that have attracted
>> many people. But hey, then I could at least save my weekends and focus on
>> writing performance-oriented code in other languages (whether that is Java,
>> Scala, pure bytecode, or some crazy new performance-oriented JVM Lisp).
>>
>>
>> On Thursday, 12 September 2013 23:11:40 UTC+8, tbc++ wrote:
>>
>>> There are a combination of issues all contributing to the lack of
>>> response on this subject. But before I continue let me state that these
>>> opinions are my own. I have worked on Clojure and core.async, but these
>>> comments are going to be personal conjecture and opinions:
>>>
>>> 1) As mentioned there is a high amount of complexity required to
>>> implement all the interfaces needed for the different variants of functions
>>> you want. Primitive hinted functions currently implement Java interfaces,
>>> implementing every combination of float, double, object, short, int, long,
>>> and byte for arities up to 5 would require the addition of over 16,000
>>> interfaces. With the current config, we only need 81 interfaces. I'm not
>>> saying this couldn't be changed to happen on-the-fly, it would just take
>>> some serious work, and would cause problems with Java Interop
>>>
>>> 2) The current system works, there are production databases (see
>>> Datomic), neural networks (see the work by Prismatic), and many high
>>> performance libraries (Zach Tellman has a reputation for bending Clojure to
>>> his will in this area), all written in Clojure.
>>>
>>> 3) Because of #2, some may even go so far as to say that we should drop
>>> everything but longs and doubles. Other dynamic languages (Python for
>>> instance) do this without problems, they simply relegate all the high
>>> performance stuff to highly optimized libraries. This is what Prismatic did
>>> with HipHop (http://blog.getprismatic.com/**blog/2013/7/10/introducing-*
>>> *hiphip-array-fast-and-**flexible-numerical-**computation-in-clojure<http://blog.getprismatic.com/blog/2013/7/10/introducing-hiphip-array-fast-and-flexible-numerical-computation-in-clojure>
>>> ).
>>>
>>> 4) Writing small functional libraries in Java that interop with Clojure
>>> is actually quite nice. So perhaps the answer is to build the inner kernels
>>> of your system in Java and then wrap them? I know that doesn't sound fun,
>>> but you won't get much faster code than that.
>>>
>>> So all the above make me sit back and say "what is it about your problem
>>> that is unique and really requires floats? Why haven't we had problems like
>>> this in the past."
>>>
>>> 5) And I think the problem really comes down to your API and target
>>> platform (GPU). See, most of us would just typehint to doubles and take the
>>> memory hit. Doubles are fast on modern CPUs and memory is cheap. Not so in
>>> your case, as you mentioned you have concerns about memory size and
>>> performance of certain primitives.
>>>
>>> My conclusion is that you'd probably be best looking at the HipHop
>>> library, and figuring out how you can adapt their ideas to your own code.
>>> That is, write a small Java library that contains all the kernel code you
>>> need, and then stitch together these primitives with Clojure.
>>>
>>> As it stands, Clojure's support for type hinting floats is poor, but
>>> that's not likely to change soon.
>>>
>>> ----
>>>
>>> TL;DR - The reason you haven't heard back is because no one has a good
>>> answer as to how to fix it, or if it even needs to be fixed. That's not the
>>> answer anyone likes to hear, but I'm afraid that's the truth. Anyone else
>>> on this list can feel free to correct me.
>>>
>>> Timothy Baldridge
>>>
>>>
>>> On Thu, Sep 12, 2013 at 8:38 AM, Alex Fowler <alex.m...@gmail.com>wrote:
>>>
>>>>
>>>> Does somebody from the development team read this user group? Or maybe
>>>> I have addressed my questions to a wrong place?
>>>>
>>>> --
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Clojure" group.
>>>> To post to this group, send email to clo...@googlegroups.com
>>>>
>>>> Note that posts from new members are moderated - please be patient with
>>>> your first post.
>>>> To unsubscribe from this group, send email to
>>>> clojure+u...@**googlegroups.com
>>>>
>>>> For more options, visit this group at
>>>> http://groups.google.com/**group/clojure?hl=en<http://groups.google.com/group/clojure?hl=en>
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Clojure" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to clojure+u...@**googlegroups.com.
>>>>
>>>> For more options, visit
>>>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>
>>>> .
>>>>
>>>
>>>
>>>
>>> --
>>> “One of the main causes of the fall of the Roman Empire was that–lacking
>>> zero–they had no way to indicate successful termination of their C
>>> programs.”
>>> (Robert Firth)
>>>
>> --
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clo...@googlegroups.com<javascript:>
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+u...@googlegroups.com <javascript:>
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+u...@googlegroups.com <javascript:>.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>
>
> --
> “One of the main causes of the fall of the Roman Empire was that–lacking
> zero–they had no way to indicate successful termination of their C
> programs.”
> (Robert Firth)
>
--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
