>From Spark perspective, I would really advise to use Dataframe API as much as possible, including the Spark Structured Streaming instead of Spark Streaming - the main reason is more optimized execution of the code because of all optimizations that Catalyst is able to make. But I really don't see libraries that wrap dataframe API
On Thu, Jul 9, 2020 at 11:36 PM Tim Clemons <tclem...@gmail.com> wrote: > I'm putting together a big data system centered around using Spark > Streaming for data ingest and Spark SQL for querying the stored data. I've > been investigating what options there are for implementing Spark > applications using Clojure. It's been close to a decade since sparkling or > flambo have received any updates and it doesn't look like either will > accommodate recent distributions of Spark. I've found powderkeg an > interesting option, and I like how it supports remote REPLs and the use of > tranducers rather than wrapped Scala fns. However, it looks like it's also > seen a few years without commits and I've heard loose talk that the > developers have moved on to other pursuits. > > Part of the problem seems to be Spark. The project seem unapologetic > about breaking interfaces and seems willing to sacrifice third-party code > that tries to track Spark's development. > > So my options seem to be the following: > > 1. Deploy an older version of Spark that's compatible with one of the > above mentioned libraries. While we don't need to be bleeding edge, > deploying a three year old version just to accommodate my preferred > language is hard to justify. > > 2. Create a merge to update one of those libraries to more recent versions > of Spark and be prepared to maintain it internally for the lifespan of this > project. This may be vastly overestimating my personal heroics. > > 3. Code my own solution from scratch using Java/Scala interop, sketching > out just enough of a Clojure wrapper to suit my ends. > > 4. Learn Scala. > > I realize that Spark isn't the only game in town (Onyx, for example). > However, I'm working with a team of developers who are not familiar with > Clojure (though I'm working to be an advocate). I choose Spark as an > established solution that supports multiple languages and handles both > streaming and batch processing. > > Any insights? Any solutions I'm overlooking? > > > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/clojure/259f5ff6-dd66-4688-aa80-439fed88ab39o%40googlegroups.com > <https://groups.google.com/d/msgid/clojure/259f5ff6-dd66-4688-aa80-439fed88ab39o%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- With best wishes, Alex Ott http://alexott.net/ Twitter: alexott_en (English), alexott (Russian) -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/clojure/CALV1_%3DJtBC02CchwoCT3%3DgHbdMBfaACRA_T6yRnZo0KCr9tACg%40mail.gmail.com.
