RE: Having plugin as a separate project
Hi, Here are examples using Maven: https://github.com/ATLANTBH/nutch-plugins/tree/master/nutch-plugins Regards, Markus -Original message- > From:Yash Thenuan Thenuan <rit2014...@iiita.ac.in> > Sent: Monday 7th May 2018 11:51 > To: user@nutch.apache.org > Subject: Re: Having plugin as a separate project > > Hey, > Thanks for the answer, But my question was can't we write plugin by > downloading the nutch jar and using it as a dependency, rather than adding > the code in nutch source code? > > On Fri, May 4, 2018 at 8:08 PM, Jorge Betancourt <betancourt.jo...@gmail.com > > wrote: > > > Usually we tend to develop everything inside the Nutch file structure, > > specially useful if you need to deploy to a Hadoop cluster later on > > (because you need to bundle everything in a job file). > > > > But, if you really want to develop the plugin in isolation you only need to > > create a new project in your preferred IDE/maven/ant/gradle and add the > > dependencies that you need from the lib/ directory (or the global > > dependencies with the same version). > > > > Then just compile everything to a jar and place it in the proper plugin > > structure in the Nutch installation. Although this should work is not > > really a smooth development experience. > > You need to be careful and not bundle all libs inside your jar, etc. > > > > The path suggested by Sebastian is much better, in the end while developing > > you want to have everything, perhaps just compile/test your plugin and > > later on you can copy the final jar of your plugin to the desired Nutch > > installation. > > > > Best Regards, > > Jorge > > > > On Fri, May 4, 2018 at 4:02 PM narendra singh arya <nsary...@gmail.com> > > wrote: > > > > > Can we have nutch plugin as a separate project? > > > > > > On Fri, 4 May 2018, 19:26 Sebastian Nagel, <wastl.na...@googlemail.com> > > > wrote: > > > > > > > That's trivial. Just run ant in the plugin's source folder: > > > > > > > > cd src/plugin/urlnormalizer-basic/ > > > > ant > > > > > > > > or to run also the tests > > > > > > > > cd src/plugin/urlnormalizer-basic/ > > > > ant test > > > > > > > > Note: you have to compile the core test classes first by running > > > > > > > > ant compile-core-test > > > > > > > > in the Nutch "root" folder. > > > > > > > > A little bit slower but guarantees that everything is compiled: > > > > > > > > ant -Dplugin=urlnormalizer-basic test-plugin > > > > > > > > Or sometimes it's enough to skip some of the long running tests: > > > > > > > > ant -Dtest.exclude='TestSegmentMerger*' clean runtime test > > > > > > > > > > > > Best, > > > > Sebastian > > > > > > > > On 05/04/2018 01:13 PM, Yash Thenuan Thenuan wrote: > > > > > Hi all, > > > > > I want to compile my plugins separately so that I need not compile > > > > > the whole project again when I make a change in some plugin. How can > > I > > > > > achieve that? > > > > > Thanks > > > > > > > > > > > > > > > > > > >
Re: Having plugin as a separate project
Hey, Thanks for the answer, But my question was can't we write plugin by downloading the nutch jar and using it as a dependency, rather than adding the code in nutch source code? On Fri, May 4, 2018 at 8:08 PM, Jorge Betancourtwrote: > Usually we tend to develop everything inside the Nutch file structure, > specially useful if you need to deploy to a Hadoop cluster later on > (because you need to bundle everything in a job file). > > But, if you really want to develop the plugin in isolation you only need to > create a new project in your preferred IDE/maven/ant/gradle and add the > dependencies that you need from the lib/ directory (or the global > dependencies with the same version). > > Then just compile everything to a jar and place it in the proper plugin > structure in the Nutch installation. Although this should work is not > really a smooth development experience. > You need to be careful and not bundle all libs inside your jar, etc. > > The path suggested by Sebastian is much better, in the end while developing > you want to have everything, perhaps just compile/test your plugin and > later on you can copy the final jar of your plugin to the desired Nutch > installation. > > Best Regards, > Jorge > > On Fri, May 4, 2018 at 4:02 PM narendra singh arya > wrote: > > > Can we have nutch plugin as a separate project? > > > > On Fri, 4 May 2018, 19:26 Sebastian Nagel, > > wrote: > > > > > That's trivial. Just run ant in the plugin's source folder: > > > > > > cd src/plugin/urlnormalizer-basic/ > > > ant > > > > > > or to run also the tests > > > > > > cd src/plugin/urlnormalizer-basic/ > > > ant test > > > > > > Note: you have to compile the core test classes first by running > > > > > > ant compile-core-test > > > > > > in the Nutch "root" folder. > > > > > > A little bit slower but guarantees that everything is compiled: > > > > > > ant -Dplugin=urlnormalizer-basic test-plugin > > > > > > Or sometimes it's enough to skip some of the long running tests: > > > > > > ant -Dtest.exclude='TestSegmentMerger*' clean runtime test > > > > > > > > > Best, > > > Sebastian > > > > > > On 05/04/2018 01:13 PM, Yash Thenuan Thenuan wrote: > > > > Hi all, > > > > I want to compile my plugins separately so that I need not compile > > > > the whole project again when I make a change in some plugin. How can > I > > > > achieve that? > > > > Thanks > > > > > > > > > > > > >
Re: Having plugin as a separate project
Usually we tend to develop everything inside the Nutch file structure, specially useful if you need to deploy to a Hadoop cluster later on (because you need to bundle everything in a job file). But, if you really want to develop the plugin in isolation you only need to create a new project in your preferred IDE/maven/ant/gradle and add the dependencies that you need from the lib/ directory (or the global dependencies with the same version). Then just compile everything to a jar and place it in the proper plugin structure in the Nutch installation. Although this should work is not really a smooth development experience. You need to be careful and not bundle all libs inside your jar, etc. The path suggested by Sebastian is much better, in the end while developing you want to have everything, perhaps just compile/test your plugin and later on you can copy the final jar of your plugin to the desired Nutch installation. Best Regards, Jorge On Fri, May 4, 2018 at 4:02 PM narendra singh aryawrote: > Can we have nutch plugin as a separate project? > > On Fri, 4 May 2018, 19:26 Sebastian Nagel, > wrote: > > > That's trivial. Just run ant in the plugin's source folder: > > > > cd src/plugin/urlnormalizer-basic/ > > ant > > > > or to run also the tests > > > > cd src/plugin/urlnormalizer-basic/ > > ant test > > > > Note: you have to compile the core test classes first by running > > > > ant compile-core-test > > > > in the Nutch "root" folder. > > > > A little bit slower but guarantees that everything is compiled: > > > > ant -Dplugin=urlnormalizer-basic test-plugin > > > > Or sometimes it's enough to skip some of the long running tests: > > > > ant -Dtest.exclude='TestSegmentMerger*' clean runtime test > > > > > > Best, > > Sebastian > > > > On 05/04/2018 01:13 PM, Yash Thenuan Thenuan wrote: > > > Hi all, > > > I want to compile my plugins separately so that I need not compile > > > the whole project again when I make a change in some plugin. How can I > > > achieve that? > > > Thanks > > > > > > > >
Re: Having plugin as a separate project
Can we have nutch plugin as a separate project? On Fri, 4 May 2018, 19:26 Sebastian Nagel,wrote: > That's trivial. Just run ant in the plugin's source folder: > > cd src/plugin/urlnormalizer-basic/ > ant > > or to run also the tests > > cd src/plugin/urlnormalizer-basic/ > ant test > > Note: you have to compile the core test classes first by running > > ant compile-core-test > > in the Nutch "root" folder. > > A little bit slower but guarantees that everything is compiled: > > ant -Dplugin=urlnormalizer-basic test-plugin > > Or sometimes it's enough to skip some of the long running tests: > > ant -Dtest.exclude='TestSegmentMerger*' clean runtime test > > > Best, > Sebastian > > On 05/04/2018 01:13 PM, Yash Thenuan Thenuan wrote: > > Hi all, > > I want to compile my plugins separately so that I need not compile > > the whole project again when I make a change in some plugin. How can I > > achieve that? > > Thanks > > > >
Re: Having plugin as a separate project
That's trivial. Just run ant in the plugin's source folder: cd src/plugin/urlnormalizer-basic/ ant or to run also the tests cd src/plugin/urlnormalizer-basic/ ant test Note: you have to compile the core test classes first by running ant compile-core-test in the Nutch "root" folder. A little bit slower but guarantees that everything is compiled: ant -Dplugin=urlnormalizer-basic test-plugin Or sometimes it's enough to skip some of the long running tests: ant -Dtest.exclude='TestSegmentMerger*' clean runtime test Best, Sebastian On 05/04/2018 01:13 PM, Yash Thenuan Thenuan wrote: > Hi all, > I want to compile my plugins separately so that I need not compile > the whole project again when I make a change in some plugin. How can I > achieve that? > Thanks >