Re: Tools comparable to Luigi for Clojure or the JVM?

[Miguel Ping]

If it's hadoop, I would take a look at either cascalog 
 or pigpen 

On Thursday, September 17, 2015 at 4:17:13 AM UTC+1, Christopher Small 
wrote:
>
> Check out prismatic's plumbing. Its a reall y lovely tool that among other 
> things let's you create directed graph structures representing 
> computational pipelines, and then compile them into callable functions. 
> It's has different compilers for profiling, lazy evaluation, parallelized 
> computation and eager seqiential processing. I don't know that anyone has 
> done it, but it should be possible to compile down to something distributed 
> if needed. I've been interested in compiling to onyx actually. 
>
> Best
>
> Chris
>
> Disclaimer: Written on cell phone
>
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Tools comparable to Luigi for Clojure or the JVM?

[Christopher Small]

Check out prismatic's plumbing. Its a reall y lovely tool that among other 
things let's you create directed graph structures representing computational 
pipelines, and then compile them into callable functions. It's has different 
compilers for profiling, lazy evaluation, parallelized computation and eager 
seqiential processing. I don't know that anyone has done it, but it should be 
possible to compile down to something distributed if needed. I've been 
interested in compiling to onyx actually. 

Best

Chris

Disclaimer: Written on cell phone

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Tools comparable to Luigi for Clojure or the JVM?

[Ben Mabey]

Drake is similar to what you want and it is written in Clojure: 
https://github.com/Factual/drake


-Ben

On 9/16/15 12:12 PM, Max Countryman wrote:

Hi,

I’m working on a project at work for which we’re currently considering a Python 
tool built by Spotify called Luigi[1]. Luigi is a tool for building data 
processing pipelines (hence the name). It’s capable of connecting to various 
processors such as Hadoop and is limited in scope to building and managing 
these pipelines as a directed graph. I’m curious if there are similar tools for 
the JVM we should be taking into consideration? (Certainly a big plus if it’s 
Clojure, from my perspective.) Note that we’re processing a relatively small 
amount of data: at this point we’re just taking in some server logs and doing 
post-processing over them to build aggregates of certain statistics.

Thanks,


Max

[1] https://github.com/spotify/luigi



--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Tools comparable to Luigi for Clojure or the JVM?

[Max Countryman]

Hi,

I’m working on a project at work for which we’re currently considering a Python 
tool built by Spotify called Luigi[1]. Luigi is a tool for building data 
processing pipelines (hence the name). It’s capable of connecting to various 
processors such as Hadoop and is limited in scope to building and managing 
these pipelines as a directed graph. I’m curious if there are similar tools for 
the JVM we should be taking into consideration? (Certainly a big plus if it’s 
Clojure, from my perspective.) Note that we’re processing a relatively small 
amount of data: at this point we’re just taking in some server logs and doing 
post-processing over them to build aggregates of certain statistics.

Thanks,


Max

[1] https://github.com/spotify/luigi

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.