Thanks for the helpful information Christopher. I'll have to look at Powderkeg.
The AOT issue is a big one. Being able to launch things from the REPL is huge. That's actually one of the many advantages of Onyx over Storm (if you're looking at the streaming side of things). Towards the end of my using Storm I became increasingly frustrated with the project. At the time, it was an Apache Incubator project, and development had slowed to a grind. The Clojure API became woefully incompatible with more recent Clojures, preventing us from upgrading for some time. They also began shifting focus away from the Clojure API, and in turn the documentation became woefully out of date. I've heard that some of these issues got a bit better as the project came out of Incubator status, but others remain. In contrast, Onyx has been very well maintained, has excellent documentation, and doesn't suffer any of the AOT issues. Chris On Thu, Oct 19, 2017 at 1:02 PM, Christopher Penrose <cpenr...@steelcase.com > wrote: > > The #bigdata channel over on Clojurians slack is also suspiciously quiet, >> as are many of the Google groups. >> >> Ray. >> > > I worked with Sparkling and Flambo about a year ago, while Mr. Macbeth is > a fellow Portlander and has a solid API, I found Sparkling to be somewhat > more direct and compact. For ETL via Hadoop I wouldn't hesitate to try > either of these libraries. I found them to be stable and preferable to > using Spark in Scala. However, I used Powderkeg (https://github.com/ > HCADatalab/powderkeg) a bit and found it the most intriguing. Christophe > Grand last updated PowderKeg three hours ago (from time of my posting > obviously). Powderkeg relies heavily on Clojure transducers and is the > only Clojure Spark library I am aware of which doesn't require AOT > compilation -- you can use a Clojure repl to directly spawn jobs on a Spark > cluster. If you are interested in Clojure interoperability with Spark, I > would look at Powderkeg first. > > If you require Spark Streaming, you might be better off writing Scala, or > considering another streaming solution such as Storm. The closest I have > come to getting Spark Streaming to work in Clojure was with Powderkeg. It > might be worth seeing if Powderkeg has made progress in this area. > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to a topic in the > Google Groups "Clojure" group. > To unsubscribe from this topic, visit https://groups.google.com/d/ > topic/clojure/ESkUu0Tmqmg/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
