Re: [DISCUSS] Replacing MapReduce with Tez

2020-12-21 Thread Lewis John McGibbney
Hi dev@, I've documented my Tez journey so far at https://cwiki.apache.org/confluence/display/NUTCH/Running+Nutch+on+Tez Things are getting quite interesting. Please share any experiences using Nutch on Tez or improvements to the documentation especially any experiments you can document. Thank y

Re: RE: [DISCUSS] Replacing MapReduce with Tez

2020-12-21 Thread Lewis John McGibbney
Hi Markus, Thanks for chiming in :) My responses below On 2020/12/21 21:32:08, Markus Jelsma wrote: > Hello Lewis, > > 1. counters, for me they are a requirement to have as they are key to regular > inspections of ongoing crawls, finding errors and debugging. I hope you can > find a work arou

RE: [DISCUSS] Replacing MapReduce with Tez

2020-12-21 Thread Markus Jelsma
ranch. Have a nice evening! Markus -Original message- > From:Lewis John McGibbney > Sent: Monday 21st December 2020 21:40 > To: dev@nutch.apache.org > Subject: Re: [DISCUSS] Replacing MapReduce with Tez > > Hi dev@, > Short update here. I've documented my ini

Re: [DISCUSS] Replacing MapReduce with Tez

2020-12-21 Thread Lewis John McGibbney
Hi dev@, Short update here. I've documented my initial observations running Nutch on Tez at https://s.apache.org/viee3 Specific early finding are as follows 1. Counters don't appear to work... which makes sense as all existing counters are manifested using the MapReduce framework. I'm not sure if

Re: [DISCUSS] Replacing MapReduce with Tez

2020-12-10 Thread BlackIce
Sounds interesting On Thu, Dec 10, 2020, 10:29 wrote: > Hi Lewis, > > I have some background using and developing on tez for sometime. It might > be good improvement if we can fit our model to Apache Tez. If dev@ agrees > and it is feasible I will be happy to check and work on this. > > Thanks,

Re: [DISCUSS] Replacing MapReduce with Tez

2020-12-10 Thread shbalakuntala
Hi Lewis, I have some background using and developing on tez for sometime. It might be good improvement if we can fit our model to Apache Tez. If dev@ agrees and it is feasible I will be happy to check and work on this. Thanks, Shashanka Sent from my iPhone > On 10-Dec-2020, at 1:16 PM, lewi