Re: complex number library

[Mike Anderson]

I agree that complex would be a better name.

It would be also be nice if it the 1-arg version could be idempotent (i.e. 
returns an existing complex number unchanged). The downside is that this 
would mean a slight performance hit because it would prevent the use of 
primitive arguments. Maybe we should do this but still use primitive type 
hints for the 2-arg version?

On Wednesday, 3 June 2015 01:17:49 UTC+1, Christopher Graham wrote:
>
> How about changing the name of the complex-number function to, ideally, 
> complex?
> complex-number seems irritating to (have to) read. Further, calling this 
> function is a form of type coercion. (float ...), (int ...), etc., are 
> idiomatic Clojure, whereas (float-number ...), (int-number ...), etc., were 
> not included in the language.
>
>
> On Sunday, May 31, 2015 at 6:55:46 PM UTC-4, Alan Forrester wrote:
>>
>> https://clojars.org/complex 
>>
>> https://github.com/alanforr/complex 
>> 
>>  
>>
>> Complex is a Clojure library for doing complex number calculations 
>> that wraps the Java commons-math3 Complex library. 
>>
>> complex 
>>
>> A Clojure library for doing calculations with complex numbers. Wraps 
>> the Java commons-math3 Complex library. 
>>
>> Usage 
>>
>> A complex number can be created by invoking the complex number 
>> function. With one argument the function produces a complex number in 
>> which only the real component is non-zero. With two arguments, the 
>> first argument is the real part, the second argument is the imaginary 
>> part: 
>>
>> => (complex-number 1) 
>>
>> Complex (1.0, 0.0) 
>>
>> => (complex-number 1 2) 
>>
>> Complex (1.0, 2.0). 
>>
>> The library can be used to do complex arithmetic. The + function can 
>> have any number of real or complex arguments but always produces a 
>> complex result. 
>>
>> => (+ 1 (complex-number 3 4)) 
>>
>> Complex (4.0, 4.0). 
>>
>> The same is true of the other arithmetical operations *,-,/. The 
>> arithmetical functions are fastest on a per number basis when used on 
>> only two arguments. They are also faster when their arguments are 
>> complex. 
>>
>> The library also provides other functions, such as (pow a b), which 
>> raises a to the power b, (sin a) which calculates the sine of a, and 
>> several other functions. For details, see the docs. 
>>
>> Alan 
>>
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: complex number library

[Mike Anderson]

OK, here's a basic version that I think has most of the key elements in 
place.

A lot more protocols still need implementing, but it should be a reasonable 
basis to build upon:

https://github.com/mikera/core.matrix.complex


On Tuesday, 2 June 2015 16:35:25 UTC+1, Christopher Small wrote:
>
> > The array representation could simply be a deftype which uses two 
> underlying arrays for the real and complex parts of the array respectively.
>
> Oh man; that is flipping brilliant. And simple...
>
> > The implementation should be fairly straightforward, but if anyone wants 
> I can create a repo and bang out a bare-bones implementation in an hour or 
> so that people can build upon.
>
> Please!
>
>
> Chris
>
>
>
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: complex number library

[Mike Anderson]

I think the right strategy is to make a separate complex array 
implementation library ("core.matrix.complex"?). In terms of dependencies, 
it would only need to depend depend upon "core.matrix" and "complex".

The array representation could simply be a deftype which uses two 
underlying arrays for the real and complex parts of the array respectively.

This world:
- use Complex values for array elements (mget etc.)
- allow the use of arbitrary underlying implementations (vectorz-clj, 
Clatrix etc.)
- implement all the relevant core.matrix protocols for Complex scalar values
- implement all the relevant core.matrix protocols for Complex arrays
- maybe add some special functions for complex-specific tasks
- as an added bonus, act as a proof point that the core.matrix protocols 
work for non-real valued arrays (which I think they do, but would be nice 
to confirm...)

The implementation should be fairly straightforward, but if anyone wants I 
can create a repo and bang out a bare-bones implementation in an hour or so 
that people can build upon.



On Monday, 1 June 2015 08:49:22 UTC+1, Christopher Small wrote:
>
> Are these operations (*, +, etc) interoperable with core.matrix 
> operations? That may end up being pretty key for a lot of numerical users.
>
> Chris
>
>
> On Sunday, May 31, 2015 at 3:55:46 PM UTC-7, Alan Forrester wrote:
>>
>> https://clojars.org/complex 
>>
>> https://github.com/alanforr/complex 
>>
>> Complex is a Clojure library for doing complex number calculations 
>> that wraps the Java commons-math3 Complex library. 
>>
>> complex 
>>
>> A Clojure library for doing calculations with complex numbers. Wraps 
>> the Java commons-math3 Complex library. 
>>
>> Usage 
>>
>> A complex number can be created by invoking the complex number 
>> function. With one argument the function produces a complex number in 
>> which only the real component is non-zero. With two arguments, the 
>> first argument is the real part, the second argument is the imaginary 
>> part: 
>>
>> => (complex-number 1) 
>>
>> Complex (1.0, 0.0) 
>>
>> => (complex-number 1 2) 
>>
>> Complex (1.0, 2.0). 
>>
>> The library can be used to do complex arithmetic. The + function can 
>> have any number of real or complex arguments but always produces a 
>> complex result. 
>>
>> => (+ 1 (complex-number 3 4)) 
>>
>> Complex (4.0, 4.0). 
>>
>> The same is true of the other arithmetical operations *,-,/. The 
>> arithmetical functions are fastest on a per number basis when used on 
>> only two arguments. They are also faster when their arguments are 
>> complex. 
>>
>> The library also provides other functions, such as (pow a b), which 
>> raises a to the power b, (sin a) which calculates the sine of a, and 
>> several other functions. For details, see the docs. 
>>
>> Alan 
>>
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ANN: Sparse matrix support for Clojure with vectorz-clj 0.28.0

[Mike Anderson]

Thanks Matt! I've just release Vectorz 0.45.0 including your changes.

A lot of sparse operations are much faster now!

On Monday, 29 December 2014 21:56:30 UTC+8, Matt Revelle wrote:
>
> Yes, will do.
>
> On Dec 28, 2014, at 9:58 PM, Mike Anderson  
> wrote:
>
> Looks like you have some good changes in your Vectorz branch - any chance 
> you could tidy up and make a PR?
>
> I like the idea of specialised getSlices and getColumns in particular - 
> these should be much faster than getting the slices one-by-one if the data 
> is very sparse.
>
> On Monday, 29 December 2014 09:43:54 UTC+8, Matt Revelle wrote:
>>
>>
>> On Dec 28, 2014, at 7:28 PM, Mike Anderson  
>> wrote:
>>
>> Interesting idea. The challenge is that I'm not sure how to add 
>> representation specification in an implementation independent way. It's a 
>> quirk of vectorz that it has both indexed and hashed storage, I probably 
>> wouldn't expect any other implementations to have that. Likewise row and 
>> column oriented storage are fairly obvious choices but I still wouldn't 
>> expect every implementation to support both.
>>
>> Any idea how you would specify the API?
>>
>> I guess we could simply pass an optional map argument of options, but 
>> behaviour would be completely implementation specific. 
>>
>>
>> I think the map is the way to go. You’re probably correct about few other 
>> implementations having as many options, but adding a map of “preferences” 
>> seems like a good option. Creating a sparse matrix might then look like:
>>
>> ;; preferences as a separate arg
>> (new-sparse-array [10 10] :vectorz {:order :row :indexed true})
>>
>> ;; an alternative, preferences combined with implementation selection
>> (new-sparse-array [10 10] {:impl :vectorz :order :row :indexed 
>> true})
>>
>> Implementations should throw an exception if they don’t support (or 
>> understand) the preferences.
>>
>> On Monday, 29 December 2014 02:12:05 UTC+8, Matt Revelle wrote:
>>>
>>> Glad to see the addition of new-sparse-array to core.matrix. It looks 
>>> like it defaults to SparseRowMatrix for the Vectorz implementation? Should 
>>> the API provide a way to specify which sparse matrix representation (e.g., 
>>> row- vs column-based, indexed vs hashed) should be used? I'd suggest a 
>>> 3-arity new-sparse-array which takes a keyword indicating the 
>>> representation to use as well as a new function which returns a list of 
>>> available representations for a specific implementation.
>>>
>>> I think at this point you incorporated (looks like we have some 
>>> duplication too, doh) all the changes I had made for sparse matrix support 
>>> in Vectorz, but will verify.
>>>
>>
>> I definitely haven't covered all the potential code paths - in particular 
>> a lot of things aren't yet optimised for sparse operations. So any review / 
>> patches would be appreciated!
>>
>>
>> I did some optimization of sparse ops but the code probably needs to be 
>> cleaned up before submitting (e.g., generalized and/or moved to the correct 
>> level in class hierarchy). Those changes were made hastily when I needed to 
>> quickly get a program running fast.
>>
>> A  branch containing all performance changes based on an older revision 
>> of the develop branch is available here:
>> https://github.com/mattrepl/vectorz/tree/sparse-speed
>>
>> There is a related sparse-speed branch in my forks of vectorz-clj and 
>> core.matrix.
>>
>> We should also look into other sparse array representations for Vectorz 
>> from: Matlab, MTJ (https://github.com/fommil/matrix-toolkits-java, 
>> specifically the LinkedSparseMatrix for row and column ops), etc.
>>
>> -Matt
>>
>>  
>>
>>>
>>> On Saturday, December 27, 2014 4:56:55 AM UTC-5, Mike Anderson wrote:
>>>>
>>>> Here is a little belated Christmas present for Clojure data aficionados:
>>>>
>>>> ;; setup
>>>> (use 'clojure.core.matrix)
>>>> (set-current-implementation :vectorz)
>>>>
>>>> ;; create a big sparse matrix with a trillion elements (initially zero)
>>>> (def A (new-sparse-array [100 100]))
>>>>
>>>> ;; we are hopefully smart enough to avoid printing the whole array!
>>>> A
>>>> => #
>>>>
>>>> ;; mutable setter operations supported so that you can set in

Re: ANN: boltzmann 0.1.1 - a deep-learning library

[Mike Anderson]

On Tuesday, 6 January 2015 04:27:55 UTC+8, Christian Weilbach wrote:
>
> -BEGIN PGP SIGNED MESSAGE- 
> Hash: SHA1 
>
> On 05.01.2015 03:34, Mike Anderson wrote: 
> > Very cool stuff! 
>
> Like yours! I wish nurokit was EPLed, then I could have had a look at 
> it and try to include it there. Have libraries like this high 
> commercial value? I thought the knowledge to apply them and tune them 
> to the problem is still more expensive, this is why I picked EPL. Also 
> GPL and EPL don't seem to be compatible due to my recherche (which is 
> a pity, because I like the GPL). 
>

I think there isn't much commercial value in the library itself - there are 
many free libraries for machine learning that work just fine. Nobody with 
enough skill to use the library is going to pay you for something they can 
get for free.

The commercial value is all around :
- Building a solution that solves a business problem *using* the library
- Integrating with other applications/services (Clojure shines here because 
of the JVM ecosystem)
- Professional services / consulting

 

>
> > 
> > I notice that you are specialising the RBM to a specific matrix 
> > implementation (Clatrix / JBlas) in the file "jblas.clj". Are you 
> > sure you need to do that? Part of the beauty of core.matrix is 
> > that you should be able to write your algorithms in an 
> > implementation-independent manner and still get the performance 
> > benefits of the optimised implementation when you need it. 
>
> I started with core.matrix operations and clatrix and then tried to 
> eliminate all overhead showing up in the VisualVM sampling profiler. 
> In my experiments the protocol overhead in this inner loop in 
> `cond-prob-batch` was something like 10% or so, but I am not sure 
> whether I did something wrong. In the mean time I have benchmarked my 
> cryptographic hash function, which also uses protocols, and sometimes 
> I have seen protocol overhead and sometimes not, maybe it was related 
> to tiered compilation and the JIT sometimes not optimizing it, but 
> this is only guessing. 
>

10% protocol overhead sounds like you must be doing quite a lot of protocol 
calls.

The usual trick to minimise this is to ensure that a single protocol call 
does a lot of work (i.e. work on whole arrays at a time rather than 
individual elements). If you do that, then the protocol overhead should be 
negligible.
 

>
> If you replace all the jBlas method calls with core.matrix fns in 
> `cond-prob-batch` (3), which is quick to do, do you see a performance 
> difference? 
>
> I really like core.matrix, or in general sound, light protocols and 
> then implementations. Yesterday I found an improved fork for clj-hdf5 
> for instance, which implements some of core.matrix protocols and fixed 
> that to read double matrices for me, potentially this even allows to 
> read tensors bigger than memory partially then. (1) So I didn't want 
> to inline jBlas, but really use core.matrix. This internal inlining 
> seemed to be some compromise, since it still allows to use clatrix 
> when dealing with the jblas implementation (otherwise it was just a 
> mini-batch implementation). 
>
> For deep learning most interesting was GPU support in core.matrix for 
> typical BLAS routines, e.g. with jCuBLAS or clBLAS, but I just 
> couldn't start work on this yet. You then have to be very careful not 
> to access some memory, but if this could work with core.matrix 
> protocols it was a major win. 
>

It should certainly be possible to wrap GPU matrix support in a core.matrix 
implementation, indeed I think there have been a couple of "proof of 
concept" attempts already.

I personally have in the back of my mind a GPU-accelerated extension to 
Vectorz (i.e. GPU-backed subclasses of AMatrix and AVector), using 
something like jCuBLAS. Then the full core.matrix support would come for 
free via vectorz-clj. Would possibly be the easiest way to get 
comprehensive GPU array programming support in Clojure.
 

> boltzmann's CPU version is for me 1/3 to 1/4th training speed of 
> theano (which again is 1/5 of its GPU version on my older gaming 
> laptop). Theano uses a symbolic compute graph modelled after Python's 
> numpy API and then emits that either to CPU or GPU (including some 
> numeric optimizations). I guess my jBlas backend is still slower than 
> theirs netlib-java (2) recommends building a custom version of 
> ATLAS (for Ubuntu here), have you experience with this? I probably 
> should do this for clatrix (and also for numpy). 
>

Not really - I generally do pure-JVM stuff (vectorz-clj etc.). 

Would be interested to see how vectorz-clj stacks up to Clatrix / Blas if 
you get an opportunity to benchmark thi

Re: ANN: boltzmann 0.1.1 - a deep-learning library

[Mike Anderson]

Very cool stuff!

I notice that you are specialising the RBM to a specific matrix 
implementation (Clatrix / JBlas) in the file "jblas.clj". Are you sure you 
need to do that? Part of the beauty of core.matrix is that you should be 
able to write your algorithms in an implementation-independent manner and 
still get the performance benefits of the optimised implementation when you 
need it.

For example, the core.matrix protocols (mmul, add!, add, inner-product, 
transpose etc.) should all call the right Clatrix implementation without 
any noticeable loss of performance (if they don't that's an implementation 
issue in Clatrix... would be good to unearth these!).

If the core.matrix API is insufficient to implement what you need, then I'd 
love to get issues / PRs (either for core.matrix or Clatrix).

On Monday, 5 January 2015 07:07:11 UTC+8, Christian Weilbach wrote:
>
> -BEGIN PGP SIGNED MESSAGE- 
> Hash: SHA1 
>
> Hi all, 
>
> - From the README: 
>
> This library is supposed to implement Boltzmann Machines, Autoencoders 
> and related deep learning technologies. All implementations should 
> both have a clean high-level mathematical implementation of their 
> algorithms (with core.matrix) and if possible, an optimized and 
> benchmarked version of the core routines for production use. This is 
> to facilitate learning for new users or potential contributors, to be 
> able to implement algorithms from papers/other languages and then tune 
> them for performance if needed. 
>
> This repository is supposed to cover techniques building on Restricted 
> Boltzmann Machines, like Deep Belief Networks, Deep Boltzmann Machines 
> or temporal extensions thereof as well as Autoencoders (which I am not 
> familiar enough with yet). Classical back-propagation is also often 
> used to fine-tune deep models supervisedly, so networks should support 
> it as well. 
>
>
>
> I haven't build myself deep belief networks out of it yet, but this 
> should be fairly straightforward. Also combination with the usual 
> linear classifiers (logistic regression, SVM) at the top layer can be 
> explored. If somebody has interest/experience in/with implementing 
> standard backpropagation, go ahead and open a pull-request :-). 
>
> Christian 
> -BEGIN PGP SIGNATURE- 
> Version: GnuPG v1 
>
> iQEcBAEBAgAGBQJUqceeAAoJEKel+aujRZMkJHoIAKkAbgZjvs9pzmJjzJf5Y1sg 
> EQCwf7W6Vrz0rvDtrkSiRNO+rmSEL4TpWPPlHLTYWs781Wrz9FRmkmHzR0mZ8izT 
> kWsQ3rP4TjDUDiB8S34CQxA15YLRfbvIxVv2JBfkGBWo64NHSrNUxz+Dfvu2jzbi 
> at614o/T5lZQ6qzkyputYwzOocX58AcnCtfXDVO2UJt8RU/q33FVugjtXtvsDxgM 
> AOO4WnW6mzYvLUbrhksDjuLShhs2EoCMB54cB2W5ejz+6X3oFeF/xndFqtNYdwPF 
> d13q60Ex0s/IqIo3mOwB/O1rOnsBHxiQ6nuSaphMAm7jJF9wHtDaXHWRZHa2RTg= 
> =BjnJ 
> -END PGP SIGNATURE- 
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ANN: Sparse matrix support for Clojure with vectorz-clj 0.28.0

[Mike Anderson]

Looks like you have some good changes in your Vectorz branch - any chance 
you could tidy up and make a PR?

I like the idea of specialised getSlices and getColumns in particular - 
these should be much faster than getting the slices one-by-one if the data 
is very sparse.

On Monday, 29 December 2014 09:43:54 UTC+8, Matt Revelle wrote:
>
>
> On Dec 28, 2014, at 7:28 PM, Mike Anderson  
> wrote:
>
> Interesting idea. The challenge is that I'm not sure how to add 
> representation specification in an implementation independent way. It's a 
> quirk of vectorz that it has both indexed and hashed storage, I probably 
> wouldn't expect any other implementations to have that. Likewise row and 
> column oriented storage are fairly obvious choices but I still wouldn't 
> expect every implementation to support both.
>
> Any idea how you would specify the API?
>
> I guess we could simply pass an optional map argument of options, but 
> behaviour would be completely implementation specific. 
>
>
> I think the map is the way to go. You’re probably correct about few other 
> implementations having as many options, but adding a map of “preferences” 
> seems like a good option. Creating a sparse matrix might then look like:
>
> ;; preferences as a separate arg
> (new-sparse-array [10 10] :vectorz {:order :row :indexed true})
>
> ;; an alternative, preferences combined with implementation selection
> (new-sparse-array [10 10] {:impl :vectorz :order :row :indexed 
> true})
>
> Implementations should throw an exception if they don’t support (or 
> understand) the preferences.
>
> On Monday, 29 December 2014 02:12:05 UTC+8, Matt Revelle wrote:
>>
>> Glad to see the addition of new-sparse-array to core.matrix. It looks 
>> like it defaults to SparseRowMatrix for the Vectorz implementation? Should 
>> the API provide a way to specify which sparse matrix representation (e.g., 
>> row- vs column-based, indexed vs hashed) should be used? I'd suggest a 
>> 3-arity new-sparse-array which takes a keyword indicating the 
>> representation to use as well as a new function which returns a list of 
>> available representations for a specific implementation.
>>
>> I think at this point you incorporated (looks like we have some 
>> duplication too, doh) all the changes I had made for sparse matrix support 
>> in Vectorz, but will verify.
>>
>
> I definitely haven't covered all the potential code paths - in particular 
> a lot of things aren't yet optimised for sparse operations. So any review / 
> patches would be appreciated!
>
>
> I did some optimization of sparse ops but the code probably needs to be 
> cleaned up before submitting (e.g., generalized and/or moved to the correct 
> level in class hierarchy). Those changes were made hastily when I needed to 
> quickly get a program running fast.
>
> A  branch containing all performance changes based on an older revision of 
> the develop branch is available here:
> https://github.com/mattrepl/vectorz/tree/sparse-speed
>
> There is a related sparse-speed branch in my forks of vectorz-clj and 
> core.matrix.
>
> We should also look into other sparse array representations for Vectorz 
> from: Matlab, MTJ (https://github.com/fommil/matrix-toolkits-java, 
> specifically the LinkedSparseMatrix for row and column ops), etc.
>
> -Matt
>
>  
>
>>
>> On Saturday, December 27, 2014 4:56:55 AM UTC-5, Mike Anderson wrote:
>>>
>>> Here is a little belated Christmas present for Clojure data aficionados:
>>>
>>> ;; setup
>>> (use 'clojure.core.matrix)
>>> (set-current-implementation :vectorz)
>>>
>>> ;; create a big sparse matrix with a trillion elements (initially zero)
>>> (def A (new-sparse-array [100 100]))
>>>
>>> ;; we are hopefully smart enough to avoid printing the whole array!
>>> A
>>> => #
>>>
>>> ;; mutable setter operations supported so that you can set individual 
>>> sparse elements
>>> (dotimes [i 1000]
>>>  (mset! A (rand-int 100) (rand-int 100) (rand-int 100)))
>>>
>>> ;; all standard core.matrix operations supported
>>> (esum A)
>>> => 50479.0
>>>
>>> ;; efficient addition
>>> (time (add A A))
>>> => "Elapsed time: 12.849859 msecs"
>>>
>>> ;; matrix multiplication / inner products actually complete in sensible 
>>> time
>>> ;; (i.e. much faster than than the usual O(n^3) which might take a few 
>>> thousand years)
>>> (time (mmul A (transpose A)

Re: ANN: Sparse matrix support for Clojure with vectorz-clj 0.28.0

[Mike Anderson]

Interesting idea. The challenge is that I'm not sure how to add 
representation specification in an implementation independent way. It's a 
quirk of vectorz that it has both indexed and hashed storage, I probably 
wouldn't expect any other implementations to have that. Likewise row and 
column oriented storage are fairly obvious choices but I still wouldn't 
expect every implementation to support both.

Any idea how you would specify the API?

I guess we could simply pass an optional map argument of options, but 
behaviour would be completely implementation specific. 

On Monday, 29 December 2014 02:12:05 UTC+8, Matt Revelle wrote:
>
> Glad to see the addition of new-sparse-array to core.matrix. It looks like 
> it defaults to SparseRowMatrix for the Vectorz implementation? Should the 
> API provide a way to specify which sparse matrix representation (e.g., row- 
> vs column-based, indexed vs hashed) should be used? I'd suggest a 3-arity 
> new-sparse-array which takes a keyword indicating the representation to use 
> as well as a new function which returns a list of available representations 
> for a specific implementation.
>
> I think at this point you incorporated (looks like we have some 
> duplication too, doh) all the changes I had made for sparse matrix support 
> in Vectorz, but will verify.
>

I definitely haven't covered all the potential code paths - in particular a 
lot of things aren't yet optimised for sparse operations. So any review / 
patches would be appreciated!
 

>
> On Saturday, December 27, 2014 4:56:55 AM UTC-5, Mike Anderson wrote:
>>
>> Here is a little belated Christmas present for Clojure data aficionados:
>>
>> ;; setup
>> (use 'clojure.core.matrix)
>> (set-current-implementation :vectorz)
>>
>> ;; create a big sparse matrix with a trillion elements (initially zero)
>> (def A (new-sparse-array [100 100]))
>>
>> ;; we are hopefully smart enough to avoid printing the whole array!
>> A
>> => #
>>
>> ;; mutable setter operations supported so that you can set individual 
>> sparse elements
>> (dotimes [i 1000]
>>  (mset! A (rand-int 100) (rand-int 100) (rand-int 100)))
>>
>> ;; all standard core.matrix operations supported
>> (esum A)
>> => 50479.0
>>
>> ;; efficient addition
>> (time (add A A))
>> => "Elapsed time: 12.849859 msecs"
>>
>> ;; matrix multiplication / inner products actually complete in sensible 
>> time
>> ;; (i.e. much faster than than the usual O(n^3) which might take a few 
>> thousand years)
>> (time (mmul A (transpose A)))
>> => "Elapsed time: 2673.085171 msecs"
>>
>>
>> Some nice things to note about the implementation:
>> - Everything goes through the core.matrix API, so your code won't have to 
>> change to use sparse matrices :-)
>> - Sparse matrices are 100% interoperable with non-sparse (dense) matrices
>> - Sparse arrays are fully mutable. Management of storage / indexing 
>> happens automatically
>> - It isn't just matrices - you can have sparse vectors, N-dimensional 
>> arrays etc.
>> - Code is pure JVM - no native dependencies to worry about
>>
>> This is all still very much alpha - so any comments / patches / more 
>> rigorous testing much appreciated!
>>
>>
>>
>>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

ANN: Sparse matrix support for Clojure with vectorz-clj 0.28.0

[Mike Anderson]

Here is a little belated Christmas present for Clojure data aficionados:

;; setup
(use 'clojure.core.matrix)
(set-current-implementation :vectorz)

;; create a big sparse matrix with a trillion elements (initially zero)
(def A (new-sparse-array [100 100]))

;; we are hopefully smart enough to avoid printing the whole array!
A
=> #

;; mutable setter operations supported so that you can set individual 
sparse elements
(dotimes [i 1000]
 (mset! A (rand-int 100) (rand-int 100) (rand-int 100)))

;; all standard core.matrix operations supported
(esum A)
=> 50479.0

;; efficient addition
(time (add A A))
=> "Elapsed time: 12.849859 msecs"

;; matrix multiplication / inner products actually complete in sensible time
;; (i.e. much faster than than the usual O(n^3) which might take a few 
thousand years)
(time (mmul A (transpose A)))
=> "Elapsed time: 2673.085171 msecs"


Some nice things to note about the implementation:
- Everything goes through the core.matrix API, so your code won't have to 
change to use sparse matrices :-)
- Sparse matrices are 100% interoperable with non-sparse (dense) matrices
- Sparse arrays are fully mutable. Management of storage / indexing happens 
automatically
- It isn't just matrices - you can have sparse vectors, N-dimensional 
arrays etc.
- Code is pure JVM - no native dependencies to worry about

This is all still very much alpha - so any comments / patches / more 
rigorous testing much appreciated!



-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Charting Data Format Feedback Requested

[Mike Anderson]

Lucas,

Thanks for kicking off the discussion - great to see your proposal on this. 
I think it will be really valuable if we can converge on a standard way of 
representing this kind of data in Clojure/ClojureScript. Copying the 
Incanter and main Clojure groups as well because I think there will be 
broad interest in this.

My view is that it is worth distinguishing (decomplecting?) two things:

a) The format used to convey the actual data, i.e. the labelled ":dataset" 
part of the data format suggested below
b) The format used to specify the chart (chart type, axes etc.)

I think a) Can be pretty closely linked to the standard core.matrix dataset 
/ n-dimensional array structure. 

b) is much harder and may call for something more like ggplot2, also worth 
checking out Kevin Lynagh's c2 work (https://keminglabs.com/c2/)

Therefore it may be easier to tackle a) in isolation first. b) will 
probably need more experimentation before we can settle on something 
sufficiently well designed and general.
 
On Thursday, 11 December 2014 17:45:00 UTC+8, Lucas Bradstreet wrote:
>
> Hi everyone,
>
> We are currently writing an OM based visualisation / charting library that 
> we
> intend to use extensively in combination with core.matrix and other data
> analysis libraries/tools (e.g. gorilla REPL, incanter, etc). 
>
> Some of the goals of this library:
> - Provide a clojurescript wrapper for common visualisation libraries (C3,
>   dimple, Rickshaw, NVD3) for standard charting features.
> - Provide a generic data format, with conversion functions to native 
> charting
>   library formats.
> - Provide transformation functions between core.matrix datasets, incanter
>   datasets, etc to this generic charting format.
> - Provide update functions to allow datasets to seamlessly be updated with 
> the
>   addition of data-points in map and vector formats.
> - Provide seamless transitions when a dataset is updated, ala om.
>
> We would like to hear any of your thoughts regarding the following charting
> data format below. This format maps fairly closely to core.matrix datasets
> (note, although core.matrix datasets currently do not allow labelled
> dimensions, this support is incoming).
>
> {:axes [{:label "X axis label" :type :category}
> {:label "Y axis label" :type :category}
> {:label "Z axis label" :type :measure}
> {:label "C axis label" :type :color}]
>:chart-type :area
>:dataset {:labels [; 0th dimension is labelled from the 0th dimension 
> of the 1st
>  ; 1st dimension labels (i.e. columns)
>   ["timestamp" "event-count" "magnitude" "colour"]
>   ; 2nd dimension labels (i.e. series)
>   ["series1" "series2" "series3"]] 
>  :data [[; series 1 data
>  [1 2 3 4] ; timestamp values
>  [100 200 300 400] ; event count value
>  [50 100 150 200] ; magnitude values
>  [25 50 75 100] ; colour values
>  ]
> [[1 2 3 4]
>  [1 200 3 4] 
>  [5 7 444 8] 
>  [9 10 11 12]]
> [[1 2 3 4]
>  [1 2 3 4] 
>  [5 9 7 8] 
>  [9 10 11 12]]]}}
>
> The above format is close to the native format used by C3, and can be 
> easily mapped
> to a format that is more easily consumed by dimple charts:
>
> {:axes [{:key "timestamp", :type :category, :label "X axis label"} 
> {:key "event-count", :type :category, :label "Y axis label"} 
> {:key "magnitude", :type :measure, :label "Z axis label"} 
> {:key "colour", :type :color, :label "C axis label"}], 
>  :chart-type :area, 
>  :dataset [{:name "series1", 
> :values [{"colour" 25, "magnitude" 50, "event-count" 100, 
> "timestamp" 1} 
>  {"colour" 50, "magnitude" 100, "event-count" 200, 
> "timestamp" 2} 
>  {"colour" 75, "magnitude" 150, "event-count" 300, 
> "timestamp" 3} 
>  {"colour" 100, "magnitude" 200, "event-count" 400, 
> "timestamp" 4}]} 
>{:name "series2", 
> :values [{"colour" 9, "magnitude" 5, "event-count" 1, 
> "timestamp" 1} 
>  {"colour" 10, "magnitude" 7, "event-count" 200, 
> "timestamp" 2}
>  {"colour" 11, "magnitude" 444, "event-count" 3, 
> "timestamp" 3}
>  {"colour" 12, "magnitude" 8, "event-count" 4, 
> "timestamp" 4}]}
>{:name "series3", 
> :values [{"colour" 9, "magnitude" 5, "event-count" 1, 
> "timestamp" 1}
>  {"colour" 10, "magnitude" 9, "event-count" 2, 
> "timestamp" 2}
>  {"colour" 11, "magnitude" 7, "event-count" 3, 
> "timestamp" 3}
>  {"colour" 12, "magnitude" 8, "event-count" 4, 
> "timestamp" 4}]}]}
>
>
> We would love to hear any feedback of any kind on this format.
>
> Thanks,
>
> Lucas
>

-- 
You received this mes

ANN: core.matrix "coding dojo" repo

[Mike Anderson]

Hi All,

I've made a repository for an upcoming core.matrix "coding dojo" that I'm 
organising. I think it might be a useful resource for others in the Clojure 
community, so I'm sharing the link here:
https://github.com/clojure-numerics/core.matrix.dojo

This is a basic project setup for a core.matrix project with some useful 
libraries and example code. I'll add some more examples over the next week 
or two. PRs also welcome, if anyone has other simple examples that are good 
for beginners to learn from.

  Mike.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

ANN: core.matrix 0.13.1

[Mike Anderson]

New version of core.matrix now available.

This release brings quite a lot of changes including:

- Many performance optimisations: this should be noticeably faster than 
previous releases for many operations
- The shape of a scalar is now defined to be nil. This seems to work better 
for people who want to distinguish scalars from 0-dimensional arrays (which 
continue to have a shape of [])
- Improved broadcasting support
- emin and emax functions (to find the minimum and maximum elements in an 
array respectively)
- zero-filled array constructors (zero-vector, zero-matrix and zero-array)
- Lots of fixes for various edge cases, especially around scalar and nil 
handling
- NDArray is now loaded lazily, to minimise startup time issues

As usual available from Clojars or GitHub:
- https://clojars.org/net.mikera/core.matrix
- https://github.com/mikera/core.matrix


-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

ANN: New core.matrix 0.11.0 release

[Mike Anderson]

Hi all,

New version of core.matrix now available on Clojars:

https://clojars.org/net.mikera/core.matrix/versions/0.11.0

Key items of note:
- Dmitry's GSoC NDArray project is now the default core.matrix 
implementation - this is a big milestone, congrats Dmitry!
- Everything is now AOT-compiled. This increases the size of the .jar file, 
but should improve performance, especially around startup time
- Lots of performance enhancements (e.g. matrix inverses calculated using 
LU-decomposition algorithm)
- A few minor bug fixes
- Repository on github has been renamed (old links should still work): 
https://github.com/mikera/core.matrix

As always, feedback / suggestions much appreciated.

  Mike.

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: macro woes (deftype generator)

[Mike Anderson]

Understood. vectorz-clj is going to stay JVM-only for the foreseeable
future, so it won't fit if you need ClojureScript.

On the other hand, core.matrix is pure Clojure and totally protocol-based,
so it should (I think!) be pretty easy to port to ClojureScript, and it's
easy to extend the protocols to new implementations. Also it would be nice
to have a pure Clojure small vector implementation (which we don't
currently have). Can I therefore suggest some collaboration on this front,
in the interests of saving effort and avoiding further fragmentation in the
Clojure library space?

It would mean a few things:
1) Adding core.matrix API functions / protocols that you think are
important for your use cases
2) Having your vector library implement the basic core.matrix protocols
(only a few protocols are needed, the rest are optional)
3) Extending core.matrix to support ClojureScript. Several people have
expressed a general interest in this.
4) Making sure formats are compatible. It should be possible, for example,
for someone to use vectorz-clj on the server and your library for the
ClojureScript frontend.

If we collectively do all this then your library will get full core.matrix
compatibility, core.matrix will work on ClojureScript, and people will get
a fast pure Clojure/ClojureScript small vector implementation. And we all
avoid the "Lisp Curse" for a little while longer. Does that sound like a
win/win/win goal to work towards?


On 12 September 2013 15:13, Karsten Schmidt  wrote:

> Hi Mike, thank you, I've checked out your vectorz lib in the past and
> am also aware of Zach's clj-tuple (which this concrete part is much
> closer too), but I'm looking for a pure Clojure & CLJS compatible
> solution (using CLJX). This alI is part of a longwinded (on/off 2+ yrs
> now) refactoring/total re-imagining effort of my http://toxiclibs.org/
> project, which too has a comprehensive set of vector ops (and the
> Clojure version many more), e.g:
>
> http://hg.postspectacular.com/toxiclibs/src/tip/src.core/toxi/geom/Vec3D.java
>
> K.
>

>
>
> On 12 September 2013 04:45, Mikera  wrote:
> > Hi Karsten,
> >
> > It looks to me like you're trying to define some small fixed-size numeric
> > double vectors and related operations? For graphics, games or physical
> > modelling perhaps?
> >
> > Perhaps you've already evaluated all the options and decided that a
> bunch of
> > custom deftypes is the best way go, but in case you haven't seen it
> already,
> > can I recommend considering my little library vectorz-clj?
> >
> > I designed it specifically to handle this kind of use case in Clojure,
> and
> > it includes:
> > - Fixed size 1-4D Vectors, with many specialised methods for high
> > performance
> > - Specialised 2D and 3D transformation matrices (rotations, scaling etc.)
> > - Affine transformations
> > - core.matrix  fully supported (which gives you a lot of more general
> > matrix/vector features with a nice Clojure API)
> >
> > See: https://github.com/mikera/vectorz-clj
> > And core.matrix: https://github.com/mikera/matrix-api
> >
> > If vectorz-clj and/or core.matrix doesn't quite fit your needs, could you
> > let me know why? I'm keen to address all the common vector use cases that
> > people have in Clojure, so that people can avoid reinventing the wheel.
> >
> >
> > On Thursday, 12 September 2013 07:41:34 UTC+8, Karsten Schmidt wrote:
> >>
> >> Hi, I'm (once again) despairing over some issues trying to integrate
> >> one macro with another. Both are supposed to be used together to
> >> generate a bunch of deftypes. Firstly, here's the one working
> >> correctly (stripped out error checking for simplicity):
> >>
> >> (defmacro swizzle*
> >>   "Takes a class name, source instance, keyword and metadata map,
> >>   returns new instance of type with fields populated based on
> >>   decomposed single letter key lookups. E.g.
> >>
> >>   (macroexpand-1 '(swizzle* Vec3 v :zyx))
> >>   #_=> (new Vec3 (. v z) (. v y) (. v x) {})
> >>
> >>   (macroexpand-1 '(swizzle* Vec3 v :xxz))
> >>   #_=> (new Vec3 (. v x) (. v x) (. v z) {})
> >>   "
> >>   [clz src k meta]
> >>   `(new ~clz
> >> ~@(->> k
> >>(name)
> >>(map (comp symbol str))
> >>(map (fn [f] `(. ~src ~f
> >> ~meta))
> >>
> >> Now, where things go wrong is when I try to use this swizzle macro
> >> from within the 2nd macro which generates the deftype skeleton (also
> >> stripped down):
> >>
> >> (defmacro defvec
> >>   [n]
> >>   (let [vname (symbol (str "Vec" n))
> >> syms (map (fn [s] (-> s str symbol)) (take n "xyzw"))]
> >> `(deftype ~vname [~@syms __meta#]
> >>
> >>clojure.lang.IObj
> >>(meta [this#] __meta#)
> >>(withMeta [this# mm#] (new ~vname ~@syms mm#))
> >>
> >>clojure.lang.ILookup
> >>(valAt [this# k#] (swizzle* ~vname this# k# __meta#))  ;; FIXME
> >>)))
> >>
> >> (defvec 3)
> >> CompilerException java.lang.IllegalArgumentException: No

Re: core.async - handling nils

[Mike Anderson]

On 28 August 2013 11:50, Alan Busby  wrote:

> On Wed, Aug 28, 2013 at 12:18 PM, guns  wrote:
>
>> Oh, I was confused; I was thinking about sentinel values in user code.
>> Yes, I imagine a single private (Object.) would work just fine, with
>> very little overhead.
>>
>
> First, I'd hope that sentinel values would be handled by the back-end
> implementation, as we're seeing core.sync implemented on other
> systems like ZeroMQ already.
>
> Second, as (Object.) doesn't play nicely over the wire, a random UUID
> or similar value would be much preferred.
>

Hi Alan,

Agreed on all the above. The issue is not so much what sentinel value is
used internally, but what sentinel value gets exposed to user code.


>
> Third, I'd recommend reviewing,
> http://clojure.com/blog/2013/06/28/clojure-core-async-channels.html
> to understand why core.async is not just a better queue.
>
> Fourth, if you dislike how core.async works, just wrap it in your own
> library so it works the way you'd like.
> It looks like "core.async-with-nil" is available on Clojars. ;)
>

That's precisely what I'm trying to avoid, and the reason why I've been
raising the topic here - the last thing we want in the ecosystem is more
fragmentation with incompatible subsystems and protocols. That's the Lisp
Curse all over again. I think we should aspire to better in the Clojure
community - which means working together to make the best implementation
possible and rallying around it.

Sending nil as a value over channels is clearly a significant issue if
people are willing to fork or create a new wrapper for core.async in order
to achieve it. Better, in my view, to make a breaking change to core.async
now to fix this issue rather than encouraging a free-for-all.

I'm reminded of Rich Hickey's keynote "The Language of the System", which
emphasised composing systems out of simple services that communicate via
values. core.async would IMHO be a much more useful tool for realising this
sort of vision if it is able transmit arbitrary Clojure values (i.e.
including nil).

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: core.async - handling nils

[Mike Anderson]

I still don't see why you would want to to arbitrarily limit what you can
put down a channel. FWIW, plenty of other concurrency management primitives
allow nils as values (java.util.concurrent.AtomicReference, Clojure atoms /
refs / agents to name but a few).

My motivating use case is the ability to create higher order constructs
that communicate via channels, as a way of gluing concurrent processes
together. A simplified example:

(defn printer [ch id]
  (go (while true
(let [v (! ch x)

(let [ch (chan)
  pr1 (printer ch "1")
  pr2 (printer ch "2")
  sendr (sender ch)]
  (sendr ["foo" "a"])
  (sendr ["bar"]))

Using nil as a sentinel appears to prevent such constructs from working
with arbitrary Clojure values (or alternatively forces extra wrapping /
special case handling that adds complexity and overhead). Furthermore, if
different libraries start adopting different protocols / techniques for
wrapping nils then the composability of such constructs will be severely
restricted.

I may be missing something, but this seems like a reasonable use case for
core.async to support?

Of course, if anyone has an actual technical argument why it is
necessary/better to use nil as a sentinel value, I would be delighted to
learn of it and would consider myself enlightened.


On 27 August 2013 22:58, Timothy Baldridge  wrote:

> All your arguments come down to this:
>
> "I have an arbitrary seq of things I want to send down a channel". It's
> exactly that concept I that I push against. Everything you've mentioned
> thus far is a data structure. Channels are not data structures they are
> concurrency management primitives, treat them as such and I doubt you'll
> ever have a need for nils in a channel.
>
> If we treat channels as ways of co-ordinating concurrent processes, then
> nil doesn't have a use case. In every use of channels I've had thus far,
> nil is better expressed as an empty collection, false, 0, :tick, or some
> other "ground value".
>
> It's these Rx style programming methods that make people think they need
> this feature.
>
> Timothy
>
>
>
>
> On Tue, Aug 27, 2013 at 8:51 AM, Mike Anderson <
> mike.r.anderson...@gmail.com> wrote:
>
>> On 27 August 2013 20:45, Timothy Baldridge  wrote:
>>
>>> The reason for not allowing nils isn't a complex one, and basically
>>> boils down to the following:
>>>
>>> a) to avoid race conditions, we need a single value to signal "the
>>> channel is closed". As mentioned, nil is the obvious choice for this as it
>>> matches lazy seqs and fits well with the rest of clojure:
>>>
>>>
>> Agreed that you want a single sentinel value.
>>
>> It doesn't match lazy-seqs at all though: lazy seqs can contain nils just
>> fine. There's a big difference between (next some-lazy-seq) [which could be
>> nil, indicating an empty sequence] and the actual values in the seq [which
>> could also be nil but don't indicate the end of the seq].
>>
>>
>>> (when-let [v (>>   (process v))
>>>
>>> If we chose a different value, this becomes much more ugly:
>>>
>>> (let [v (>>   (when-not (= v :async/closed)
>>> (process v)))
>>>
>>>
>> This can be solved easily by providing a macro or some other predicate
>> that knows how to check for the sentinel value correctly. e.g.
>>
>> (when-more [v (>   (process v))
>>
>>
>>> b) I question if there are any valid uses for putting nil in a channel.
>>> With all due respect to all who have written here, thus far, every
>>> complaint about nils and channels boils down to a conversion from seqs to
>>> channels. This is the wrong way to look at the problem. Channels are
>>> co-ordination primitives not data structures. Simply because a lazy seq
>>> looks like a channel, doesn't mean that they should be treated as such.
>>>
>>>
>>> In all the core.async code I've written I've never had to put a nil in a
>>> channel, so I'm left with the uncomfortable conclusion that most complaints
>>> on this subject are contrived. I could be wrong, but I just haven't seen a
>>> valid use case yet.
>>>
>>>
>> To me it's all about consistency with other Clojure constructs. You can
>> safely put nils in sequences, vectors, lists, sets etc.. nil is a valid
>> "value" just like anything else. So why can't you put them in a channel?
>>
>> Two use cases I have encountered that motivate this:
>>
&

Re: core.async - handling nils

[Mike Anderson]

On 27 August 2013 20:45, Timothy Baldridge  wrote:

> The reason for not allowing nils isn't a complex one, and basically boils
> down to the following:
>
> a) to avoid race conditions, we need a single value to signal "the channel
> is closed". As mentioned, nil is the obvious choice for this as it matches
> lazy seqs and fits well with the rest of clojure:
>
>
Agreed that you want a single sentinel value.

It doesn't match lazy-seqs at all though: lazy seqs can contain nils just
fine. There's a big difference between (next some-lazy-seq) [which could be
nil, indicating an empty sequence] and the actual values in the seq [which
could also be nil but don't indicate the end of the seq].


> (when-let [v (   (process v))
>
> If we chose a different value, this becomes much more ugly:
>
> (let [v (   (when-not (= v :async/closed)
> (process v)))
>
>
This can be solved easily by providing a macro or some other predicate that
knows how to check for the sentinel value correctly. e.g.

(when-more [v ( b) I question if there are any valid uses for putting nil in a channel.
> With all due respect to all who have written here, thus far, every
> complaint about nils and channels boils down to a conversion from seqs to
> channels. This is the wrong way to look at the problem. Channels are
> co-ordination primitives not data structures. Simply because a lazy seq
> looks like a channel, doesn't mean that they should be treated as such.
>
> In all the core.async code I've written I've never had to put a nil in a
> channel, so I'm left with the uncomfortable conclusion that most complaints
> on this subject are contrived. I could be wrong, but I just haven't seen a
> valid use case yet.
>
>
To me it's all about consistency with other Clojure constructs. You can
safely put nils in sequences, vectors, lists, sets etc.. nil is a valid
"value" just like anything else. So why can't you put them in a channel?

Two use cases I have encountered that motivate this:

a) what if you want to send a sequence through a channel? Since nil as a
value represents the empty sequence, you have to put in some extra special
case handling with the current core.async model.

b) what if you want to write generic code to send all the values in an
arbitrary collection through a channel? you would have to wrap/unwrap nils
at either end to make this work currently.

Both of these, I think, are reasonable and common enough use cases that
it's worth supporting them elegantly rather than forcing users to implement
their own nil-wrapping functionality.


> This all being said, there really isn't a technical reason to not allow
> nils, it just simplifies much of the design and that probably translates to
> better performance. So the restriction could be lifted if a rock solid
> reason could be found, but as of yet, I haven't seen it.
>

I don't believe there is any noticeable performance difference between
checking for nil and checking if a value is identical? to some sentinel
value (which would presumably be static, final, immutable and hence very
well optimised by the JVM). In addition, not allowing nils just means you
have to do extra work to wrap/unwrap nils as a user - which is almost
certainly a net loss on overall performance.

Still, I think consistency is more significant than the performance
argument in this case.


>
> Timothy Baldridge
>
>
> On Tue, Aug 27, 2013 at 2:12 AM, Max Penet  wrote:
>
>> It's a real problem for me too, I also wonder what was the intention
>> behind this. I guess there could be a very good reason for this special
>> treatement of nils, but I haven't seen it yet.
>>
>> I would love to hear about this from people involved in core.async
>> development.
>>
>> On Friday, August 16, 2013 4:44:48 AM UTC+2, Mikera wrote:
>>>
>>> Hi all,
>>>
>>> I'm experimenting with core.async. Most of it is exceptionally good, but
>>> bit I'm finding it *very* inconvenient that nil can't be sent over
>>> channels. In particular, you can't pipe arbitrary Clojure sequences through
>>> channels (since sequences can contain nils).
>>>
>>> I see this as a pretty big design flaw given the ubiquity of sequences
>>> in Clojure code - it appears to imply that you can't easily compose
>>> channels with generic sequence-handling code without some pretty ugly
>>> special-case handling.
>>>
>>> Am I missing something? Is this a real problem for others too?
>>>
>>> If it is a design flaw, can it be fixed before the API gets locked down?
>>>
>>  --
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Go

ANN: vectorz-clj 0.10.0

[Mike Anderson]

Pleased to announce the latest 0.10.0 release of vectorz-clj, a 
matrix/vector maths library for Clojure

vectorz-clj is designed so that you don't have to compromise: offering both 
high performance (about as fast as you can get on the JVM) and an idiomatic 
high-level Clojure API.

New and notable changes:
- Faster implementations for many standard mathematical functions - sqrt, 
abs, exp, log, sin etc.
- Significant performance enhancements for N-dimensional arrays where N>2
- Improved interop with regular Clojure data structures (conversions 
to/from Clojure persistent vectors)
- Specialised implementations for common functions, e.g. add-product
- Integration of core.matrix.stats for statistical functions on vectors and 
matrices (mean, variance etc.)
- Updates to latest versions of major dependencies (core.matrix 0.8.0, 
vectorz 0.15.0)

vectorz-clj is an implementation of core.matrix, and supports all features 
of the current core.matrix API. This is a big advantage: if you use  the 
core.matrix API and decide to switch to a different implementation at a 
later date (e.g. to take advantage of optimised native libraries) then most 
of your code won't need to change.

Essential links:
Clojars: https://clojars.org/net.mikera/vectorz-clj
GitHub: https://github.com/mikera/vectorz-clj

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

ANN: Task, a library for managing a set of long-running tasks

[Mike Anderson]

Hi all,

I've started writing a little library (Task) for managing tasks at the
REPL: my motivation is that I am doing a lot of analytical work with
relatively long-running tasks and I wanted to be able to fire of
multiple background tasks and then manage / monitor the state of them
interactively. Example usage:

(run {:repeat 5 :pause 1000} (.println (System/out) "bvboug"))

(ps)

:id | :status   | :source
| :options  | :result

4   | :complete | (do (println "Hello task!")) | {:repeat 5, :sleep
1000}  |
3   | :complete | (do (swap! counter inc)) | {:repeat 5, :pause
1000}  |
2   | :complete | (some-other-function)| {:repeat 5, :pause
1} |
1   | :complete | (yet-another-function)   | {:repeat 5, :pause
1000}  |


Task is available here on github:

https://github.com/mikera/task

It's still work-in-progress but I thought I'd share it early in case
it is useful for people or if the community has any good ideas on how
to improve it. I'm planning to develop this to the point where it is
broadly useful for everyone with a similar need.

Comments / suggestion / feature requests very welcome!

   Mike.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Nested identities in a value-based universe

[Mike Anderson]

On Oct 21, 9:04 pm, "Meikel Brandmeyer (kotarak)" 
wrote:
> Hi,
>
> may I question the transitivity of state information?
>
> Maybe the world's state is that player "Trantor" is at position [15 34]. Now
> Trantor eats an appel. The appel is removed from his inventory and his
> health is raised by 5 hit points. Did the state of the world change? No.
> Trantor is still at position [15 34]. Does the world have to know about the
> apple in Trantors inventory and the sad state of his hit points?
>
> Disclaimer: I have no clue about game programming, about what information is
> needed where and how this is implemented insanely fast. Just a naive
> question from the distance.

I think it generally makes sense to consider the entire world
including Trantor and his tasty apple as part of the world state. This
seems logically consistent - they are part of the world and it would
seem odd if some actions like dropping an apple on the ground at [15
34] altered the world state but eating an apple didn't.

It also has nice properties. you can then treat the world state as
a single value that you can pass to functions etc.

e.g. you could create a higher order functions to update the world
with something like:

  ((command "Trantor" :eat "apple") world-state)

The problem with identities of actors comes in when you consider code
like the following:

  (def trantor (get-actor "Trantor" world-state))

  (:hit-points trantor)
  => 10

  (def new-world-state ((command "Trantor" :eat "apple") world-state))

  (:hit-points trantor)
  => 10 (still!! because we took a snapshot of trantor..)

  (def new-trantor (get-actor "Trantor" new-world-state))

  (:hit-points new-trantor)
  => 15

Maybe this is all fine and I'm sure it is possible to successfully
write a game this way. However it does feel a little strange when you
are coming from OOP languages where you are used to simulating
everything with mutable state

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Nested identities in a value-based universe

[Mike Anderson]

On Oct 21, 4:25 pm, Ulises  wrote:
> > c) Put actor identities inside the world state - nasty! now the world
> > state is mutable
>
> Not necessarily (and I'd be interested in the replies)?
>
> I mean, here's how I view it. If actors are part of the world, then
> they are part of its state. Hence, when the state of an actor changes,
> the state of the world changes. If we accept this, then there's
> nothing wrong with keeping the state of the 'world' and that of the
> actors in a single data structure (albeit a potentially massive one)
> and call this the new world (or universe). If you have the appropriate
> functions for finding an actor inside the universe and updating it,
> then you should be good to go.

Are you arguing for my option b) then? In which case actors don't have
distinct identities, they are just part of the overall world?

I'm probably leaning towards this option myself, though it seems a
little uncomfortable that the solution boils down to "put everything
in one big ref".

Also, if you then create some kind of "multiverse" with multiple
worlds then following the same logic you need to refactor everything
to have one big ref for the multiverse with multiple worlds contained
within it. i.e. it doesn't seem to compose neatly...

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Nested identities in a value-based universe

[Mike Anderson]

I'm a big believer in Clojure / Rich Hickey's principles regarding the
separation of Identity and State (http://www.infoq.com/presentations/
Are-We-There-Yet-Rich-Hickey) and how this is generally a good idea.

However I've recently been running into what seems to be a slight
conceptual challenge with this model - I'm creating a simulation
system where there is a "world" that contains within it (among other
things) a number of "actors". Basically the actors are some form of
"nested identity" within the world.

I'd like to be able to think about the world as a single value. So the
entire "state of the world" could be considered as an identity with a
current value that is immutable and stored in a single ref. (as an
aside, making the world a persistent data structure is actually
particularly useful for things like efficiently recording a sequence
of world states, simulation of alternate histories etc.).

At the same time however, the "actors" within the world also have
their own identity. This is where it starts to get tricky. There seem
to be several options on how to handle of this, but none of them seem
very pretty..

a) separate out the actor identities but keep the value of the actors
within the world state - seems to work, but you now have to keep
multiple identities in sync - if the state of an actor changes, you
have to update *both* the world state and the state of the specific
actor. And now the entire state of the world isn't contained within
the world identity so you can't easily do snapshots etc.

b) Don't use Clojure managed references for actor identity - instead
give each actor some form of ID and look up this ID within the world
state whenever the actor is referred to. Again this seems to work, but
you now you don't get the benefits of Clojure's references, you have a
second class form of "identity" and need to wrap everything in a lot
of functions to handle all the ID lookups etc.

c) Put actor identities inside the world state - nasty! now the world
state is mutable

Am I missing something? Is there an easy way to handle this? Or is
this genuinely a complex problem with the Clojure identity/state
model?

Any thoughts appreciated!

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: New to Clojure

[Mike Anderson]

Hi Santosh,

I was in your position a little over a year ago. Some recommendations
that may help:

- If you're coming from a Java environment, you may find it easiest to
move to Clojure by using a Clojure plugin for your favourite Java IDE.
I use the Counterclockwise plugin for Eclipse which is excellent, but
I've heard great things about Enclojure for Netbeans too.
- It's worth watching the video for "Clojure for Java Programmers" by
Clojure creator Rich Hickey - 
http://blip.tv/clojure/clojure-for-java-programmers-1-of-2-989128
- I also strongly recommend this video if you want to understand
Clojure's data structures and approach to concurrency:
http://www.infoq.com/presentations/Value-Identity-State-Rich-Hickey
- I've found StackOverflow to be a great resource for Clojure tricks
and hints

Hope this helps - and good luck!

   Mike.

On Jun 7, 8:30 pm, Santosh M  wrote:
> I want to learn clojure. I already know Java. Please tell me how to
> proceed.
>
> Regards
>
> Santosh

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Problem domains for clojure

[Mike Anderson]

I'm using Clojure for some reasonably heavy computational code. It's a
great fit for the problem domain.

Some specific things I really like:
 - I use Incanter to get quick plots of outputs to test that
algorithms are working, very handy for interactive development at the
REPL
 - I can plug in Java code pretty seamlessly when needed (e.g. some of
my algorithms are implemented in Java) - Clojure makes a lovely "glue
language"
 - The concurrency features are surprisingly useful, e.g. I often kick
off long-running computations in a future

The only thing you might find tricky is getting absolutely top
performance in tightly coded algorithms. It's possible to write very
fast code in Clojure but it's not easy (i.e. you have to have an
intimate knowledge of type hinting, how to avoid boxing, exploit
primitive unchecked arithmetic, abusing mutability, how to avoid
memory allocations etc.) - so sometimes I end up writing the inner
loops of performance-sensitive algorithms in Java rather than Clojure.
Not a big deal but I look forward to the day when I can write
everything in Clojure :-)


-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Being "not Lisp" is a feature?

[Mike Anderson]

On Nov 10, 4:42 am, lprefonta...@softaddicts.ca wrote:
> Gosu -> standard athlete on performance enhancing drugs (EPO, steroids, ...)
> Clojure -> genetically modified athlete

Presumably the genetically modified athlete was also born on
Krypton :-)

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Error in 1.3 alpha 3 - "Only long and double primitives are supported"

[Mike Anderson]

Hi all,

I was testing some code under Clojure 1.3 alpha 3 that works correctly
in Clojure 1.2 and got the following error:

"CompilerException java.lang.IllegalArgumentException: Only long and
double primitives are supported"

For some reason I don't get a full stack trace saying where the error
occurred - the line that the error referred to was simply a " (java.io
File)" in the middle of a namespace import statement.

Any idea what might be going wrong here?

Also - I'm a bit worried as the message suggests that Clojure won't
support int and float primitives for some purposes - which are pretty
essential for Java interop - surely that can't be true? Or is this
just a temporary thing during the Alpha development?

  Mike.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Game development in Clojure

[Mike Anderson]

On Aug 13, 5:33 pm, Alan  wrote:
> Funny you should mention this - I was about to post a question about
> my own game when I saw your article. My issue is, I assume someone has
> written minimax and/or alpha-beta pruning in Clojure (or a java
> library that's easy to interop with). My case is slightly different in
> that the turn order is not fixes - sometimes player X goes twice in a
> row - but it ought to be pretty simple to plug into a standard AI
> library. Does anyone know where I can find such a thing?
>

I don't actually use alpha-beta: I opted to spend the effort to
develop
a decent evaluation function and then do some simple local
optimisation
on the gradient of said function.

My reasoning was that alpha-beta usually works best when the
branching
factor is low and the evaluation function pretty cheap to calculate,
sadly my situation was pretty much the reverse :-)

For your case it may be different. Two turns in a row works fine for
minimax or alpha-beta with a little tweaking (although it is likely
to
cut your search depth).

Be warned though - my experience is that it's rather hard to find an
AI library that will just "plug in" nicely to your code. Most of the
challenge in AI tends to be around special cases, embedding "expert"
knowledge and heuristics, plumbing in the right data representations
etc.
The search algorithm itself is usually the easy bit

If you're after resources, there's a decent free online book on AI in
Java that might be useful, has lots of code examples that should be
pretty easy to convert to Clojure:

http://www.scribd.com/doc/6995538/Practical-AI-in-Java

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Game development in Clojure

[Mike Anderson]

On Aug 13, 7:16 pm, Nicolas Oury  wrote:
> On Fri, Aug 13, 2010 at 2:51 PM, Mike Anderson
>
>  wrote:
> > 2. It would be great to reduce the amount of memory allocations. Yes,
> > I know memory is plentiful and GC is very cheap, but it's still not as
> > cheap as stack allocation and any noticeable GC pauses are not good
> > for the player experience in interactive games. For this reason, I
> > find myself using reduce and indexed loops a lot more than I guess
> > would normally be idiomatic, and conversely tend to avoid some of the
> > lazy constructs and functions that generate sequences. While Clojure
> > is great for a strategy game, I'd probably hesitate to use it for a
> > real-time 3D game.
>
> This can be made a bit better by turning Escape Analysis on? Have you
> tried that?
>
> The G1 collector is supposed to have lower latency. Have you tried it?
>
> http://research.sun.com/jtech/pubs/04-g1-paper-ismm.pdf

Sadly I feel that I'm going to have to target development at the
lowest common
denominator that I think users will have (hence I'm targeting Java
1.5+)

Though I am excited about the prospect for the future! I think both
escape analysis
and G1 will help Java/Clojure massively for game development.

The killer case for me is having a simple 2D or 3D point vector as a
return value. It's
such a common pattern in games/graphics, it can happen millions of
times per second,
and currently there are no nice options: it seems you either have to
put up with excessive
garbage, hack the return value into primitive arrays or manually
"inline" the inner loop function.

If escape analysis can reliably guarantee stack allocation for this
kind of return value,
it would be a massive win.


-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Game development in Clojure

[Mike Anderson]

On Aug 14, 12:38 am, BerlinBrown  wrote:
> I played it, it was pretty fun.
>
> I have UI recommendations. I couldn't tell the difference between the
> enemy and my units.
>
> I wish maybe you had some quick tips and recommended next actions so I
> could get used to how the game works.

Thanks!

Great to get the feedback, I'll work on improving this in the next
version.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Game development in Clojure

[Mike Anderson]

On Aug 14, 6:37 am, Wilson MacGyver  wrote:
> I realize that. I was pondering why I don't run into the the 2nd problem.
>
> In your code, how many files/name spaces are you creating?
> And how many lines of code are in each file? I'm curious how you
> organize your code.

Sure - I'll give a quick sketch. Would love any hints if you think
this is a good structure or if there is anything I should be doing
differently!

It's about 6,000 lines of code, 16 Clojure source files of 50-1600
lines, which typically each define their own namespace. Structure is
modular, probably reflecting my preference structuring code around
conceptually related subsystems.

Everything gets loaded in approximately this order:

protocols.clj - common protocol definition used throughout the source
(e.g. defines the PGame protocol for game state)

command.clj - message formatting functions for comms
serial.clj - serialisation functions, wrappers for JSON save games
etc.
graphics.clj - graphics functions and image resource loading
player.clj - utility functions for player management
sounds.clj - wrapper for Java sound functions and sound resource
loading

map.clj - code for handling the game map and terrain
units.clj - code for all the units in the game, including unit
definitions and unit AI
game.clj - code for managing the overall game state and updates
(implements the PGame protocol)
gamefactory.clj - game/landscape generator functions (depends heavily
on game and map)

ui.clj - common GUI functions
dialogs.clj - library of common dialogs
renderer.clj - library for drawing / animating the main game view
interface.clj - GUI interface while playing the game, click handling
etc.
frame.clj - overall game window
main.clj - main menu GUI and navigation

The main pain points are:
1. Game state is pretty integral, but all the files above game.clj
can't access it except to the extent that I expose functions in the
common PGame protocol.
2. User interface comes last, which is good in general but makes it a
royal pain to pass notifications back to the UI. In Java I would
simply have e.g. units.clj call a simple notification function in
interface.clj, in Clojure I can't do that because I can't mutually
require the namespaces so I end up either passing a callback
function or polling some atom, neither of which are particularly
elegant.
3. Although it works, I don't really like declaring all the common
protocols up front because it seems to violate modularity - it would
seem nicer if the protocols could be bundled with their implementation
in a library file and included with just one :require

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Game development in Clojure

[Mike Anderson]

On Aug 13, 7:06 pm, Brian Carper  wrote:

> Looks great.  Thanks for sharing your experiences.
>
> Do you plan to share the source code?  Any reason you went with Swing
> instead of OpenGL?
>

Main reason I went with Swing was wanting to get something up and
running quickly (this is my first Clojure project after all!) and I
didn't really see the need for OpenGL for a graphically quite simple
game like this. It would always be possible to switch the renderer
later if needed.

I wasn't planning on making the code open source, but if you're just
after a quick peek you can always just take a look at the .clj files
inside the jar :-)



-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Game development in Clojure

[Mike Anderson]

Hello all,

I've recently been working on a game development project in Clojure
which is now starting to bear fruit. I thought people here might be
interested, and that it would be worthwhile to share some experiences
and perspectives.

The project is a steampunk-themed strategy game, and a playable in-
development version is available here:

   http://mikera.net/ironclad/


Overall, I think Clojure is a fantastic language for game development.
Some thought on what makes it really compelling:

1. Immutable data structures work surprisingly well for games. I
represent the entire game state with a big defrecord, and the main
engine loop is basically:
  a) Get commands from player / AI (e.g. "Move unit from A to B")
  b) Convert each command into set of atomic updates ("Remove unit
from A", "Add unit to B")
  c) Apply updates sequentially to the game state
  d) Trigger any feedback to player e.g. animations, sounds

2. Concurrency support has been very helpful. With immutable game
state, it has been trivial to separate the game engine from the
renderer from the AI calculations - each effectively gets to operate
on its own "snapshot" of the entire game state. This can be a big
headache in many game engines that have to mediate access to a single,
mutable game state. I can also see even greater benefits when I
finally start adding some multi-player features.

3. High level, functional programming works great for rapid, iterative
and dynamic development. Of all the languages I've used, Clojure has
probably been quickest in terms of time required to add a given new
piece of functionality. For example, the map generator code probably
only took about 30 minutes to write, pretty good for a fractal
landscape generation system!

4. The Java interoperability support is fantastic - definitely counts
as one of the "killer features" of Clojure. I've been able to reuse a
lot of Java code libraries, both my own and standard libraries such as
Swing. As an example, I have a well optimised immutable persistent 2D
grid data structure in Java (basically a very efficient, spatially
partitioned map from (int,int) to Object) that I have been able to use
pretty much seamlessly in Clojure thanks to extend-protocol and
similar.


Some interesting observations / discoveries / surprises in the
process:

1. The game worked first time on Mac and Linux, despite having been
tested exclusively on Windows. Great kudos to to both the Java
platform and the Clojure libraries!

2. Reflection is *really slow*. After getting 20-100x speedups from
eliminating reflection warnings in the AI code, I now treat any
reflection warning in code other than one-time setup code as a bug to
be fixed immediately.


Finally, here are some things I think could be great improvements to
Clojure in the future from a game developer's perspective:

1. Better support for primitive parameters / return values / unchecked
arithmetic - in performance sensitive code, these are pretty much
essential. In a few cases in the graphics and AI code, I've had to
drop back to Java to get the performance I need.

2. It would be great to reduce the amount of memory allocations. Yes,
I know memory is plentiful and GC is very cheap, but it's still not as
cheap as stack allocation and any noticeable GC pauses are not good
for the player experience in interactive games. For this reason, I
find myself using reduce and indexed loops a lot more than I guess
would normally be idiomatic, and conversely tend to avoid some of the
lazy constructs and functions that generate sequences. While Clojure
is great for a strategy game, I'd probably hesitate to use it for a
real-time 3D game.

3. I think it would be great to have better support for circular
references - perhaps a two-pass compile? The reason this is
particularly acute in game development is that different subsystems
have quite a lot of inter-dependencies. AI evaluation system needs to
understand game state/engine so it can evaluate a position. Game state/
engine needs to understand units so it can manipulate them. Units need
to understand AI evaluation system so they can decide which actions to
take.. obviously it's possible to work around all this, but it's a
major pain, adds complexity and means that you need to structure code
to manage dependencies rather than in logical modules (which would be
easier to manage and maintain!)


Would love to hear thoughts, and particularly any other experiences
people have had in using Clojure for game development!

  Mike.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Building mixed clojure and java code

[Mike Anderson]

Hi Martin,

Not sure how it fits with the rest of your environment but I've had
good success with just the following:
- Eclipse Helios (3.6)
- CounterClockwise plugin

CounterClockwise integrates well with the Eclipse build system, so
I've been able to do most of the stuff I need (e.g. exporting to a
runnable .jar file) pretty much automatically. I have multiple
projects (some Java, some Clojure, some mixed) on the build path.

Mike

On Jul 14, 2:15 pm, Martin DeMello  wrote:
> What are people using to build mixed clojure/java code? Currently just
> using lein {uber,}jar to build and distribute.
>
> martin

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Let's respect CLOJURE_HOME

[Mike Anderson]

On Jun 30, 6:45 pm, Greg  wrote:
> It seems like a lot of n00b (and non-n00b) related problems have to do with 
> the location of clojure.jar and clojure-contrib.jar. People generally don't 
> like having to keep track of all the clojure.jars, and it would be nice if it 
> was easy to switch versions for scripts like clj and such.
>
> May I propose as a possible remedy CLOJURE_HOME. CLOJURE_HOME is the absolute 
> path of a directory containing clojure.jar and possibly clojure-contrib.jar. 
> Scripts should check if it's defined and use it instead of hard-coded paths, 
> as an example, here's my clj script (in newLISP):

Sounds sensible in principle, though I think the issue for n00bs is
that configuring *anything* is a barrier because even the slightest
mistake in interpreting the documentation or configuring your
environment is pretty painful.

For n00bs, if it is much more complicated than unzipping a Clojure
distribution or navigating to the right Eclipse update site then
you're already going to lose a lot of people.

For those of us n00bs who primarily use IDEs like myself, I'd vote for
just improving the integration with the IDE's automatic management of
classpaths / build paths etc. To give it credit, Counterclockwise does
a decent job to get people started quickly in terms of adding the
Clojure jars automatically to an Eclipse project.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Duplicate key bug in hash-maps

[Mike Anderson]

I agree that duplicate keys in literals are probably a coder error but
IMO this deserves some kind of compiler warning rather than an error.

You're going to get into lots of sticky situations otherwise that only
confuse people if the semantics are different between literals and
other usage. Simple is good.

On Jun 25, 3:31 pm, Stuart Halloway  wrote:
> Duplicate keys in maps/sets are disallowed in literals and factory functions, 
> where data is generally literal & inline and therefore likely represents 
> coder error:
>
> ; all disallowed
> #{:a :a}
> {:a 1 :a 2}
> (hash-map :a 1 :a 2)
> (hash-set :a :a)
>
> They are allowed in other contexts, where the data could come from anywhere:
>
> ; dumb, but these forms not generally called with a literal
> (set [:a :a])
> (into #{} [:a :a])
>
> I find this behavior consistent and easy to explain, but I was involved in 
> the design conversation so maybe I have participant blindness. :-)
>
> Stu
>
>
>
> > On Fri, 25 Jun 2010 15:36:31 +0200
> > Michael Wood  wrote:
>
> >> On 25 June 2010 12:27, Tim Robinson  wrote:
> >>> I tried Clojure via Githhub today.
>
> >>> Anyone notice this bug that hadn't existed in Version 1.1
>
> >>> user=> #{:item1 {:a "A" :b "B"} :item2 {:a "A" :b "B"}}
> >>> java.lang.IllegalArgumentException: Duplicate key: {:a "A", :b "B"}
>
> >> You're trying to put duplicate values into a set.
>
> > So? Most places, putting a value that's already in a set into the set
> > is a nop. Even in clojure that exhibits the above behavior:
>
> > user=> #{:a :a}
> > java.lang.IllegalArgumentException: Duplicate key: :a
> > user=> (set [:a :a])
> > #{:a}
> > user=> (conj #{:a} :a)
> > #{:a}
> > user=>
>
> > Apparently, duplicate keys in sets are only disallowed in set
> > literals. Arguably, that must be a mistake on the users part, but
> > it sure seems to clash with the behavior of sets elsewhere.
>
> >      > --
> > Mike Meyer               
> > http://www.mired.org/consulting.html
> > Independent Network/Unix/Perforce consultant, email for more information.
>
> > O< ascii ribbon campaign - stop html mail -www.asciiribbon.org
>
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Clojure" group.
> > To post to this group, send email to clojure@googlegroups.com
> > Note that posts from new members are moderated - please be patient with 
> > your first post.
> > To unsubscribe from this group, send email to
> > clojure+unsubscr...@googlegroups.com
> > For more options, visit this group at
> >http://groups.google.com/group/clojure?hl=en

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Enhanced Primitive Support

[Mike Anderson]

On Jun 22, 1:27 pm, David Nolen  wrote:
> On Tue, Jun 22, 2010 at 6:04 AM, Heinz N. Gies  wrote:
>
>
>
> > > Yes. With Rich's primitive work we can get to *1 billion arithmetic
> > operations* in < 2/3 of a second on commodity hardware.
>
> > Which is absolutely great since I always wanted to do that :P ,
> > meaning the example is kind of far fetched even compared to fact (which is
> > working code with useful results).
>
> Perhaps you aren't interested in using Clojure for graphics or audio work.
>
> David

Most of my code does fairly intensive graphics work. I'd love to be
able to write
it all in Clojure but currently (sadly) it is often easier for me to
write the processing
code in Java and use on the Java interop to call it.

Also if anyone is interested in real-time performance (games,
animation etc.) then
lots of small boxing allocations can be unhelpful in terms of
increasing the number
and severity of GC pauses - to some extent that's even worse than the
overall
performance hit.

Aside from my personal utility and preferences (which are compelling
for me!)
I'm in the "primitive by default" camp for a few reasons:

1. The "pay for what you use" argument is very convincing
2. It's more conceptually close to what Java/C# developers are used to
so it helps
 with the learning curve
3. Primitive by default is going to work much better for non-
mathematical uses
of numbers (array indexes, simple loops, counts of collection sizes
etc. ) which
I believe is the common case rather than heavy-duty mathematics with
huge integer ranges
4. It minimises the risk of a "Clojure is slow" reputation developing.
Which you are
likely to get if people start comparing micro-benchmarks of non-hinted
code against
languages with static / primitive support.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: RFC: clj-ds Clojure data structure for Java (et al.)

[Mike Anderson]

On Jun 8, 1:34 pm, Krukow  wrote:
> I would like to hear the groups opinion before (and if) I release this
> to the general public.
>
> http://github.com/krukow/clj-ds

I really like this approach.

Not sure if it's any use, but I created a data structure library of my
own in Java which may have some useful code you can borrow (you are
free to use anything you like to include in this project).

http://code.google.com/p/mikeralib/source/browse/#svn/trunk/Mikera/src/mikera/persistent

In particular, there is a persistent String class
(mikera.persistent.Text) which I always thought would be a good fit
with Clojure.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en