Re: Problems running some ant targets on recent trunk

[Julien Nioche]

Hi

This is probably due to some of the recent changes I made e.g.
https://issues.apache.org/jira/browse/NUTCH-1804

I'll have a look at this.

Thanks

Julien


On 16 July 2014 23:10, Sebastian Nagel  wrote:

> Hi,
>
> I have some problems running ant targets on recent trunk:
>
> % ant runtime
> fails if run from scratch (after "ant clean")
> but it succeeds after "ant test" or "ant nightly".
>
> in a plugin folder, e.g., src/plugin/parse-metatags
> % ant test
>
>
> The error causing the failure is always:
>  .../trunk/src/plugin/build-plugin.xml:190: .../trunk/build/test/lib does
> not exist.
> e.g. within the chain of calls:
> BUILD FAILED
> .../trunk/build.xml:112: The following error occurred while executing this
> line:
> .../trunk/src/plugin/build.xml:63: The following error occurred while
> executing this line:
> .../trunk/src/plugin/urlfilter-automaton/build.xml:25: The following error
> occurred while executing
> this line:
> .../trunk/src/plugin/build-plugin.xml:190: .../trunk/build/test/lib does
> not exist.
>
> Indeed the directory does not exist because it's removed by target
> "clean-lib".
> In this case it is the target "compile-test" of lib-regex-filter which
> fails.
> Should it be really called for target "runtime"?
>
>   
> 
>  dir="../lib-regex-filter"/>
>   
>
>
> Thanks,
> Sebastian
>



-- 

Open Source Solutions for Text Engineering

http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
http://twitter.com/digitalpebble

Re: Problems running some ant targets on recent trunk

[Julien Nioche]

>
> In this case it is the target "compile-test" of lib-regex-filter which
> fails.
> Should it be really called for target "runtime"?
>   
> 
>  dir="../lib-regex-filter"/>
>   


This is the source of the problem indeed, the second line should not be
there : the test classes are not required at that stage.
Both urlfilter-automaton and urlfilter-regex have the same problem, which
was not apparent until I introduced a cleaner separation between the
compilation and test deps.

I've fixed that for trunk in revision 1611303.

As for the issue with calling 'ant test' from a plugin dir, this is not a
new issue : we get the same thing with a fresh copy of Nutch1.8. It's just
that the test task for the plugins assumes that the core classes and ivy
jars have already been resolved.

Thanks

Julien


On 16 July 2014 23:10, Sebastian Nagel  wrote:

> Hi,
>
> I have some problems running ant targets on recent trunk:
>
> % ant runtime
> fails if run from scratch (after "ant clean")
> but it succeeds after "ant test" or "ant nightly".
>
> in a plugin folder, e.g., src/plugin/parse-metatags
> % ant test
>
>
> The error causing the failure is always:
>  .../trunk/src/plugin/build-plugin.xml:190: .../trunk/build/test/lib does
> not exist.
> e.g. within the chain of calls:
> BUILD FAILED
> .../trunk/build.xml:112: The following error occurred while executing this
> line:
> .../trunk/src/plugin/build.xml:63: The following error occurred while
> executing this line:
> .../trunk/src/plugin/urlfilter-automaton/build.xml:25: The following error
> occurred while executing
> this line:
> .../trunk/src/plugin/build-plugin.xml:190: .../trunk/build/test/lib does
> not exist.
>
> Indeed the directory does not exist because it's removed by target
> "clean-lib".
> In this case it is the target "compile-test" of lib-regex-filter which
> fails.
> Should it be really called for target "runtime"?
>
>   
> 
>  dir="../lib-regex-filter"/>
>   
>
>
> Thanks,
> Sebastian
>



-- 

Open Source Solutions for Text Engineering

http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
http://twitter.com/digitalpebble

Build failed in Jenkins: Nutch-trunk #2710

[Apache Jenkins Server]

See 

Changes:

[jnioche] Wrong task called in deps-jar for urlfilter-* plugins prevents ant 
runtime from working

--
[...truncated 4985 lines...]
init:

init-plugin:

deps-jar:

clean-lib:

resolve-default:
[ivy:resolve] :: loading settings :: file = 


compile:
 [echo] Compiling plugin: urlfilter-validator

compile-test:
[javac] Compiling 1 source file to 

[javac] warning: [options] bootstrap class path not set in conjunction with 
-source 1.6
[javac] 1 warning

jar:

deps-test:

deploy:

copy-generated-lib:

test:
 [echo] Testing plugin: urlfilter-validator
[junit] WARNING: multiple versions of ant detected in path for junit 
[junit]  
jar:file:/home/hudson/tools/ant/latest/lib/ant.jar!/org/apache/tools/ant/Project.class
[junit]  and 
jar:
[junit] Running org.apache.nutch.urlfilter.validator.TestUrlValidator
[junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 
0.027 sec

init:

init-plugin:

deps-jar:

clean-lib:

resolve-default:
[ivy:resolve] :: loading settings :: file = 


compile:
 [echo] Compiling plugin: urlnormalizer-basic

compile-test:
[javac] Compiling 1 source file to 

[javac] warning: [options] bootstrap class path not set in conjunction with 
-source 1.6
[javac] 1 warning

jar:

deps-test:

deploy:

copy-generated-lib:

test:
 [echo] Testing plugin: urlnormalizer-basic
[junit] WARNING: multiple versions of ant detected in path for junit 
[junit]  
jar:file:/home/hudson/tools/ant/latest/lib/ant.jar!/org/apache/tools/ant/Project.class
[junit]  and 
jar:
[junit] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 
1.868 sec
[junit] Running 
org.apache.nutch.net.urlnormalizer.basic.TestBasicURLNormalizer
[junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 
0.268 sec

init:

init-plugin:

deps-jar:

clean-lib:

resolve-default:
[ivy:resolve] :: loading settings :: file = 


compile:
 [echo] Compiling plugin: urlnormalizer-host

compile-test:
[javac] Compiling 1 source file to 

[javac] warning: [options] bootstrap class path not set in conjunction with 
-source 1.6
[javac] 1 warning

jar:

deps-test:

deploy:

copy-generated-lib:

test:
 [echo] Testing plugin: urlnormalizer-host
[junit] Running org.apache.nutch.tika.TestOOParser
[junit] WARNING: multiple versions of ant detected in path for junit 
[junit]  
jar:file:/home/hudson/tools/ant/latest/lib/ant.jar!/org/apache/tools/ant/Project.class
[junit]  and 
jar:
[junit] Running 
org.apache.nutch.net.urlnormalizer.host.TestHostURLNormalizer
[junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 
0.474 sec

init:

init-plugin:

deps-jar:

clean-lib:

resolve-default:
[ivy:resolve] :: loading settings :: file = 


compile:
 [echo] Compiling plugin: urlnormalizer-pass

compile-test:
[javac] Compiling 1 source file to 

[javac] warning: [options] bootstrap class path not set in conjunction with 
-source 1.6
[javac] 1 warning

jar:

deps-test:

deploy:

copy-generated-lib:

test:
 [echo] Testing plugin: urlnormalizer-pass
[junit] WARNING: multiple versions of ant detected in path for junit 
[junit]  
jar:file:/home/hudson/tools/ant/latest/lib/ant.jar!/org/apache/tools/ant/Project.class
[junit]  and 
jar:
[junit] Running 
org.apache.nutch.net.urlnormalizer.pass.TestPassURLNormalizer
[junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 
0.195 sec

init:

init-plugin:

deps-jar:

clean-lib:

resolve-default:
[ivy:resolve] :: loading settings :: file = 


compile:
 [echo] Compiling plugin: urlnormalizer-querystring

comp

[jira] [Created] (NUTCH-1818) deps-test task for plugins used at deployment time

[Julien Nioche (JIRA)]

Julien Nioche created NUTCH-1818:


 Summary: deps-test task for plugins used at deployment time
 Key: NUTCH-1818
 URL: https://issues.apache.org/jira/browse/NUTCH-1818
 Project: Nutch
  Issue Type: Bug
  Components: build
Reporter: Julien Nioche
Assignee: Julien Nioche
 Fix For: 1.9


>From email thread : 

[After ant clean] the target "compile-test" of lib-regex-filter which fails.
Should it be really called for target "runtime"?
  


  

This is the source of the problem indeed, the second line should not be there : 
the test classes are not required at that stage. Both urlfilter-automaton and 
urlfilter-regex have the same problem, which was not apparent until I 
introduced a cleaner separation between the compilation and test deps.

I've fixed that for trunk in revision 1611303.



This revealed another hidden issue which is that the test classes for the 
dependency plugins (e.g. lib-regex-filter) are not compiled anymore in the 
deps-jar step and we can't refer to them in the classpath.

There actually is a task deps-test which gets overriden by the plugins but this 
is not used by compile-test and despite its name has nothing to do with the 
tests but with the deployment of the dependency plugins. 

The simplest solution is to add : 



to the

and make so that compile-test in plugins depends on deploy. This way  deps-test 
will be called prior to compiling the test. 






--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (NUTCH-1818) deps-test task for plugins used at deployment time

[Julien Nioche (JIRA)]

[ 
https://issues.apache.org/jira/browse/NUTCH-1818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14064851#comment-14064851
 ] 

Julien Nioche commented on NUTCH-1818:
--

In the future we should probably separate deps-test and deps-deploy

> deps-test task for plugins used at deployment time
> --
>
> Key: NUTCH-1818
> URL: https://issues.apache.org/jira/browse/NUTCH-1818
> Project: Nutch
>  Issue Type: Bug
>  Components: build
>Reporter: Julien Nioche
>Assignee: Julien Nioche
> Fix For: 1.9
>
>
> From email thread : 
> [After ant clean] the target "compile-test" of lib-regex-filter which fails.
> Should it be really called for target "runtime"?
>   
> 
> 
>   
> This is the source of the problem indeed, the second line should not be there 
> : the test classes are not required at that stage. Both urlfilter-automaton 
> and urlfilter-regex have the same problem, which was not apparent until I 
> introduced a cleaner separation between the compilation and test deps.
> I've fixed that for trunk in revision 1611303.
> 
> This revealed another hidden issue which is that the test classes for the 
> dependency plugins (e.g. lib-regex-filter) are not compiled anymore in the 
> deps-jar step and we can't refer to them in the classpath.
> There actually is a task deps-test which gets overriden by the plugins but 
> this is not used by compile-test and despite its name has nothing to do with 
> the tests but with the deployment of the dependency plugins. 
> The simplest solution is to add : 
> 
> to the
> and make so that compile-test in plugins depends on deploy. This way  
> deps-test will be called prior to compiling the test. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (NUTCH-1818) deps-test task for plugins used at deployment time

[Julien Nioche (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NUTCH-1818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julien Nioche updated NUTCH-1818:
-

Description: 
>From email thread : 

[After ant clean] the target "compile-test" of lib-regex-filter which fails.
Should it be really called for target "runtime"?
  


  

This is the source of the problem indeed, the second line should not be there : 
the test classes are not required at that stage. Both urlfilter-automaton and 
urlfilter-regex have the same problem, which was not apparent until I 
introduced a cleaner separation between the compilation and test deps.

I've fixed that for trunk in revision 1611303.



This revealed another hidden issue which is that the test classes for the 
dependency plugins (e.g. lib-regex-filter) are not compiled anymore in the 
deps-jar step and we can't refer to them in the classpath.

There actually is a task deps-test which gets overriden by the plugins but this 
is not used by compile-test and despite its name has nothing to do with the 
tests but with the deployment of the dependency plugins. 

The best approach is to add a new overridable tasks for the plugins 
'deps-test-compile' and trigger the compilation of the test classes for the dep 
plugins from there. 



  was:
>From email thread : 

[After ant clean] the target "compile-test" of lib-regex-filter which fails.
Should it be really called for target "runtime"?
  


  

This is the source of the problem indeed, the second line should not be there : 
the test classes are not required at that stage. Both urlfilter-automaton and 
urlfilter-regex have the same problem, which was not apparent until I 
introduced a cleaner separation between the compilation and test deps.

I've fixed that for trunk in revision 1611303.



This revealed another hidden issue which is that the test classes for the 
dependency plugins (e.g. lib-regex-filter) are not compiled anymore in the 
deps-jar step and we can't refer to them in the classpath.

There actually is a task deps-test which gets overriden by the plugins but this 
is not used by compile-test and despite its name has nothing to do with the 
tests but with the deployment of the dependency plugins. 

The simplest solution is to add : 



to the

and make so that compile-test in plugins depends on deploy. This way  deps-test 
will be called prior to compiling the test. 





> deps-test task for plugins used at deployment time
> --
>
> Key: NUTCH-1818
> URL: https://issues.apache.org/jira/browse/NUTCH-1818
> Project: Nutch
>  Issue Type: Bug
>  Components: build
>Reporter: Julien Nioche
>Assignee: Julien Nioche
> Fix For: 1.9
>
>
> From email thread : 
> [After ant clean] the target "compile-test" of lib-regex-filter which fails.
> Should it be really called for target "runtime"?
>   
> 
> 
>   
> This is the source of the problem indeed, the second line should not be there 
> : the test classes are not required at that stage. Both urlfilter-automaton 
> and urlfilter-regex have the same problem, which was not apparent until I 
> introduced a cleaner separation between the compilation and test deps.
> I've fixed that for trunk in revision 1611303.
> 
> This revealed another hidden issue which is that the test classes for the 
> dependency plugins (e.g. lib-regex-filter) are not compiled anymore in the 
> deps-jar step and we can't refer to them in the classpath.
> There actually is a task deps-test which gets overriden by the plugins but 
> this is not used by compile-test and despite its name has nothing to do with 
> the tests but with the deployment of the dependency plugins. 
> The best approach is to add a new overridable tasks for the plugins 
> 'deps-test-compile' and trigger the compilation of the test classes for the 
> dep plugins from there. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Comment Edited] (NUTCH-1818) deps-test task for plugins used at deployment time

[Julien Nioche (JIRA)]

[ 
https://issues.apache.org/jira/browse/NUTCH-1818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14064851#comment-14064851
 ] 

Julien Nioche edited comment on NUTCH-1818 at 7/17/14 12:21 PM:


In the future we should probably rename deps-test into deps-deploy


was (Author: jnioche):
In the future we should probably separate deps-test and deps-deploy

> deps-test task for plugins used at deployment time
> --
>
> Key: NUTCH-1818
> URL: https://issues.apache.org/jira/browse/NUTCH-1818
> Project: Nutch
>  Issue Type: Bug
>  Components: build
>Reporter: Julien Nioche
>Assignee: Julien Nioche
> Fix For: 1.9
>
>
> From email thread : 
> [After ant clean] the target "compile-test" of lib-regex-filter which fails.
> Should it be really called for target "runtime"?
>   
> 
> 
>   
> This is the source of the problem indeed, the second line should not be there 
> : the test classes are not required at that stage. Both urlfilter-automaton 
> and urlfilter-regex have the same problem, which was not apparent until I 
> introduced a cleaner separation between the compilation and test deps.
> I've fixed that for trunk in revision 1611303.
> 
> This revealed another hidden issue which is that the test classes for the 
> dependency plugins (e.g. lib-regex-filter) are not compiled anymore in the 
> deps-jar step and we can't refer to them in the classpath.
> There actually is a task deps-test which gets overriden by the plugins but 
> this is not used by compile-test and despite its name has nothing to do with 
> the tests but with the deployment of the dependency plugins. 
> The best approach is to add a new overridable tasks for the plugins 
> 'deps-test-compile' and trigger the compilation of the test classes for the 
> dep plugins from there. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (NUTCH-1818) deps-test task for plugins used at deployment time

[Julien Nioche (JIRA)]

[ 
https://issues.apache.org/jira/browse/NUTCH-1818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14064862#comment-14064862
 ] 

Julien Nioche commented on NUTCH-1818:
--

and have a separate overridable path for   

> deps-test task for plugins used at deployment time
> --
>
> Key: NUTCH-1818
> URL: https://issues.apache.org/jira/browse/NUTCH-1818
> Project: Nutch
>  Issue Type: Bug
>  Components: build
>Reporter: Julien Nioche
>Assignee: Julien Nioche
> Fix For: 1.9
>
>
> From email thread : 
> [After ant clean] the target "compile-test" of lib-regex-filter which fails.
> Should it be really called for target "runtime"?
>   
> 
> 
>   
> This is the source of the problem indeed, the second line should not be there 
> : the test classes are not required at that stage. Both urlfilter-automaton 
> and urlfilter-regex have the same problem, which was not apparent until I 
> introduced a cleaner separation between the compilation and test deps.
> I've fixed that for trunk in revision 1611303.
> 
> This revealed another hidden issue which is that the test classes for the 
> dependency plugins (e.g. lib-regex-filter) are not compiled anymore in the 
> deps-jar step and we can't refer to them in the classpath.
> There actually is a task deps-test which gets overriden by the plugins but 
> this is not used by compile-test and despite its name has nothing to do with 
> the tests but with the deployment of the dependency plugins. 
> The best approach is to add a new overridable tasks for the plugins 
> 'deps-test-compile' and trigger the compilation of the test classes for the 
> dep plugins from there. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (NUTCH-1818) Add deps-test-compile task for building plugins

[Julien Nioche (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NUTCH-1818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julien Nioche resolved NUTCH-1818.
--

Resolution: Fixed

Committed revision 1611343.


> Add deps-test-compile task for building plugins
> ---
>
> Key: NUTCH-1818
> URL: https://issues.apache.org/jira/browse/NUTCH-1818
> Project: Nutch
>  Issue Type: Bug
>  Components: build
>Reporter: Julien Nioche
>Assignee: Julien Nioche
> Fix For: 1.9
>
>
> From email thread : 
> [After ant clean] the target "compile-test" of lib-regex-filter which fails.
> Should it be really called for target "runtime"?
>   
> 
> 
>   
> This is the source of the problem indeed, the second line should not be there 
> : the test classes are not required at that stage. Both urlfilter-automaton 
> and urlfilter-regex have the same problem, which was not apparent until I 
> introduced a cleaner separation between the compilation and test deps.
> I've fixed that for trunk in revision 1611303.
> 
> This revealed another hidden issue which is that the test classes for the 
> dependency plugins (e.g. lib-regex-filter) are not compiled anymore in the 
> deps-jar step and we can't refer to them in the classpath.
> There actually is a task deps-test which gets overriden by the plugins but 
> this is not used by compile-test and despite its name has nothing to do with 
> the tests but with the deployment of the dependency plugins. 
> The best approach is to add a new overridable tasks for the plugins 
> 'deps-test-compile' and trigger the compilation of the test classes for the 
> dep plugins from there. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (NUTCH-1818) Add deps-test-compile task for building plugins

[Julien Nioche (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NUTCH-1818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julien Nioche updated NUTCH-1818:
-

Summary: Add deps-test-compile task for building plugins  (was: deps-test 
task for plugins used at deployment time)

> Add deps-test-compile task for building plugins
> ---
>
> Key: NUTCH-1818
> URL: https://issues.apache.org/jira/browse/NUTCH-1818
> Project: Nutch
>  Issue Type: Bug
>  Components: build
>Reporter: Julien Nioche
>Assignee: Julien Nioche
> Fix For: 1.9
>
>
> From email thread : 
> [After ant clean] the target "compile-test" of lib-regex-filter which fails.
> Should it be really called for target "runtime"?
>   
> 
> 
>   
> This is the source of the problem indeed, the second line should not be there 
> : the test classes are not required at that stage. Both urlfilter-automaton 
> and urlfilter-regex have the same problem, which was not apparent until I 
> introduced a cleaner separation between the compilation and test deps.
> I've fixed that for trunk in revision 1611303.
> 
> This revealed another hidden issue which is that the test classes for the 
> dependency plugins (e.g. lib-regex-filter) are not compiled anymore in the 
> deps-jar step and we can't refer to them in the classpath.
> There actually is a task deps-test which gets overriden by the plugins but 
> this is not used by compile-test and despite its name has nothing to do with 
> the tests but with the deployment of the dependency plugins. 
> The best approach is to add a new overridable tasks for the plugins 
> 'deps-test-compile' and trigger the compilation of the test classes for the 
> dep plugins from there. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Jenkins build is back to normal : Nutch-trunk #2711

[Apache Jenkins Server]

See 

[jira] [Commented] (NUTCH-1818) Add deps-test-compile task for building plugins

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/NUTCH-1818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14064879#comment-14064879
 ] 

Hudson commented on NUTCH-1818:
---

SUCCESS: Integrated in Nutch-trunk #2711 (See 
[https://builds.apache.org/job/Nutch-trunk/2711/])
NUTCH-1818 Add deps-test-compile task for building plugins (jnioche: 
http://svn.apache.org/viewvc/nutch/trunk/?view=rev&rev=1611343)
* /nutch/trunk/CHANGES.txt
* /nutch/trunk/src/plugin/build-plugin.xml
* /nutch/trunk/src/plugin/urlfilter-automaton/build.xml
* /nutch/trunk/src/plugin/urlfilter-regex/build.xml


> Add deps-test-compile task for building plugins
> ---
>
> Key: NUTCH-1818
> URL: https://issues.apache.org/jira/browse/NUTCH-1818
> Project: Nutch
>  Issue Type: Bug
>  Components: build
>Reporter: Julien Nioche
>Assignee: Julien Nioche
> Fix For: 1.9
>
>
> From email thread : 
> [After ant clean] the target "compile-test" of lib-regex-filter which fails.
> Should it be really called for target "runtime"?
>   
> 
> 
>   
> This is the source of the problem indeed, the second line should not be there 
> : the test classes are not required at that stage. Both urlfilter-automaton 
> and urlfilter-regex have the same problem, which was not apparent until I 
> introduced a cleaner separation between the compilation and test deps.
> I've fixed that for trunk in revision 1611303.
> 
> This revealed another hidden issue which is that the test classes for the 
> dependency plugins (e.g. lib-regex-filter) are not compiled anymore in the 
> deps-jar step and we can't refer to them in the classpath.
> There actually is a task deps-test which gets overriden by the plugins but 
> this is not used by compile-test and despite its name has nothing to do with 
> the tests but with the deployment of the dependency plugins. 
> The best approach is to add a new overridable tasks for the plugins 
> 'deps-test-compile' and trigger the compilation of the test classes for the 
> dep plugins from there. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)