Hi László, Responses inline below On 2020/12/22 13:59:58, László Bodor <[email protected]> wrote: > Hi lewismc! > > This is very cool, thanks! > Please let us know if nutch jira project has an umbrella about tez > integration tasks. I think further adaptation steps will be needed for full > integration (like counters as you mentioned).
Yes I entirely agree. https://issues.apache.org/jira/browse/NUTCH-2838 I've already discovered that the Generator job does not work... I suspect that this has to do with counters as well but I will find out soon as I continue my investigation. > > Regarding initial performance improvements: I guess for shorter tasks you > can already find a perf improvement because of default > *tez.am.container.reuse.enabled=true*. This especially applies for shorter > runtimes, where e.g. JVM startup time/warmup really counts + your runtimes > look like a cold -> warm pattern to me in case of tez, I hope it's accurate. > Yes thanks for that commentary. I had also thought that container reuse would be beneficial. More to come! lewismc
