Today we (Netflix) released PigPen; Map-Reduce for Clojure!
PigPen allows you to write what looks like regular Clojure code and compile
it to an Apache Pig script that can be used in a Hadoop map-reduce cluster.
Check out the blog post & other links below for more info:
Blog post:
http://techb
I would recommend the Pig setup guide
here: http://pig.apache.org/docs/r0.11.0/start.html
Then you can run it like this:
$ pig -x local -f my-script.pig
That said, you really only have to install Pig if you want to run it on a
cluster. To run/test/develop queries locally, you can use the dump
I'm excited to announce the release of PigPen v0.3.0, which now includes
support for Cascading.
PigPen is Map-Reduce for Clojure - you write idiomatic Clojure code, we
compile it into an Apache Pig script or a Cascading flow that runs on
Hadoop.
https://github.com/Netflix/PigPen
An RC build i
/speakers/matt-bossenbroek
Questions & Complaints: pigpen-supp...@googlegroups.com
PigPen does use Apache Pig, but it uses it as a host language, similar to
how Clojure uses the JVM. It's not a Clojure wrapper for writing Pig
scripts.
Enjoy!
-Matt
--
You received this message becaus
It is my pleasure to announce simple-time to the
world: https://github.com/mbossenbroek/simple-time
simple-time is a dead simple datetime & timespan library for Clojure. It's
an opinionated alternative for clj-time that takes a more functional twist
on the object-heavy Joda time library.
Full
datetime doesn't have a timezone, so more akin
> to LocalDateTime, right?
>
> There's also https://github.com/dm3/clojure.joda-time, in case you haven't
> seen it - more of a complete Joda API wrapper.
>
> On Thursday, 20 March 2014 20:15:55 UTC+2, Matt Bossenb
Thanks for the tip - I was not aware of that!
Thanks,
Matt
On Monday, March 24, 2014 at 4:48 AM, Zoka wrote:
> Hi Matt,
>
> Looks very good - the little speed up you can make is to use :const
> attribute for conversion constants such as
> milliseconds-per-XXX , so there is no var lookup
No complaints, so PigPen 0.3.0 is now officially released.
Enjoy!
-Matt
On Monday, May 11, 2015 at 8:40 AM, Matt Bossenbroek wrote:
> I'm excited to announce the release of PigPen v0.3.0, which now includes
> support for Cascading.
>
> PigPen is Map-Reduce for Clojure - yo
FWIW, We use edn (serialized with nippy [1]) in hadoop & it works very well for
us:
https://github.com/Netflix/PigPen
In some places we use maps for the expressiveness and in some we use vectors
for more performance.
Whatever I lose in raw performance I can trivially throw a few more boxes a
Just saw this response - disregard the questions I asked you on the pigpen
support DL.
I'll pull in the new instaparse & get a new PigPen build out soonish (within a
day or two).
-Matt
On Thursday, September 11, 2014 at 6:28 PM, Mark Engelberg wrote:
> You're probably using Clojure 1.7.0 a
e meantime.
-Matt
On Thursday, September 11, 2014 at 7:52 PM, Sunil S Nandihalli wrote:
> Thanks Mark and Matt, changing the version back to clojure version 1.6.0
> fixed it.
> Sunil
>
>
> On Fri, Sep 12, 2014 at 7:05 AM, 'Matt Bossenbroek' via Clojure
> mailto:c
Ran into an interesting problem today. In short, this works:
(count (repeat 1e8 "stuff"))
But this doesn't:
(map count [(repeat 1e8 "stuff")])
To be fair, given sufficient memory, it would eventually complete. (If the
second example does work for you, change it to 1e10 or something higher).
s)) (map f (rest s))
>
>
> holds onto the head via needing to keep the value of "s" around throughout
> the entire call to (f (first s)) in order to later make the call (map f (rest
> s)). In map2, the value of s is no longer needed by the time f is called.
>
&g
I'm using test.check to test a live service. Occasionally it gets a 503
from the service and spends hours trying to shrink the input & reproduce
the error.
Is there a way to limit the shrinking process to n iterations? Or disable
it entirely for some tests?
Is there a better approach for using
, Daniel Compton (
daniel.compton.li...@gmail.com) wrote:
If the 503 is only returned by failures not relating to what you are
testing (e.g. load), then one option might be to catch the exception and
retry that request?
On Thu, Feb 9, 2017 at 6:48 AM 'Matt Bossenbroek' via Clojure
17 at 11:55:38 AM UTC-6, Matt Bossenbroek wrote:
>
> I considered that, but that only partially fixes the issue. If it does
> actually find a real problem, it’ll never complete because the shrinking
> takes too long.
>
> In the end I’d rather have something not fully shrunk tha
We just released PigPen 0.3.1 with a couple of minor improvements:
Update cascading version to 2.7.0
Update nippy (for serialization) to 2.10.0 & tune performance
PigPen is map-reduce for Clojure, or distributed Clojure. You write idiomatic
Clojure code, we run it on thousands of machines usin
How about adding a state of Datomic survey? :)
-Matt
On Sunday, December 6, 2015 at 4:12 PM, Mars0i wrote:
>
>
> On Sunday, December 6, 2015 at 3:26:55 PM UTC-6, Lee wrote:
> >
> > On Dec 6, 2015, at 3:00 PM, Alex Miller > (javascript:)> wrote:
> >
> > > Almost all of the questions are o
18 matches
Mail list logo