Re: In memory shuffle phase

2013-06-20 Thread 牛兆捷
*import* *org**.apache.tez.dag.api.records*.DAGProtos.DAGPlan; why this import errors in tez trunk ? 2013/6/14 Arun C Murthy > https://issues.apache.org/jira/browse/TEZ-65 > > Arun > > On Jun 12, 2013, at 12:28 AM, 牛兆捷 wrote: > > > Is there any design document about tez? > > > > 2013/6/12 Arun C

Re: In memory shuffle phase

2013-06-13 Thread Arun C Murthy
https://issues.apache.org/jira/browse/TEZ-65 Arun On Jun 12, 2013, at 12:28 AM, 牛兆捷 wrote: > Is there any design document about tez? > > 2013/6/12 Arun C Murthy > >> Take a look at Apache Tez (http://incubator.apache.org/projects/tez.html). >> >> I'll warn you that it's a little rough, but w

Re: In memory shuffle phase

2013-06-13 Thread Zhijie Shen
You may want to have a look at the following webpage: http://hortonworks.com/blog/introducing-tez-faster-hadoop-processing/ On Wed, Jun 12, 2013 at 12:28 AM, 牛兆捷 wrote: > Is there any design document about tez? > > 2013/6/12 Arun C Murthy > > > Take a look at Apache Tez (http://incubator.apac

Re: In memory shuffle phase

2013-06-12 Thread 牛兆捷
Is there any design document about tez? 2013/6/12 Arun C Murthy > Take a look at Apache Tez (http://incubator.apache.org/projects/tez.html). > > I'll warn you that it's a little rough, but works. > > Arun > > On Jun 10, 2013, at 5:46 PM, Grandl Robert wrote: > > > Hi, > > > > > > It is possible

Re: In memory shuffle phase

2013-06-11 Thread Arun C Murthy
Take a look at Apache Tez (http://incubator.apache.org/projects/tez.html). I'll warn you that it's a little rough, but works. Arun On Jun 10, 2013, at 5:46 PM, Grandl Robert wrote: > Hi, > > > It is possible to run shuffle phase in memory ? I mean, the mapper to write > the intermediate outp

Re: In memory shuffle phase

2013-06-11 Thread Radim Kolar
Dne 11.6.2013 2:46, Grandl Robert napsal(a): Hi, It is possible to run shuffle phase in memory ? I mean, the mapper to write the intermediate output in memory only and the reducer to read it from there. So no disk involved for intermediate output. I do not think that this is possible in hadoo

In memory shuffle phase

2013-06-10 Thread Grandl Robert
Hi, It is possible to run shuffle phase in memory ? I mean, the mapper to write the intermediate output in memory only and the reducer to read it from there. So no disk involved for intermediate output. If so, what are the configuration parameters I need to tune ? Thanks, Robert