Re: Question on hadoop dependencies.
Hi Steve, I'll definitely take a look, although I'm not sure when exactly :( Currently busy with the Maven AppAssembler. 2013/11/13 Steve Loughran > Petar, > > We've just been through some of 2.3, on the branch > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2 > > Most of that was update, apart from moving grizzly and junit to test scope, > we've not done much. > > why do you have a look and help clean things out. I'd particularly like to > see "lean" clients for HDFS, YARN and mapreduce > > > On 30 October 2013 22:25, Petar Tahchiev wrote: > > > Hi Roman, > > > > looks like they have already upgraded to 2.2 > > > > https://issues.apache.org/jira/browse/SOLR-5382 > > > > and will be shipping it SOLR 4.6. I just hope you guys release cleaned > 2.3 > > first :) > > > > > > 2013/10/30 Roman Shaposhnik > > > > > On Wed, Oct 30, 2013 at 1:07 PM, Steve Loughran < > ste...@hortonworks.com> > > > wrote: > > > > On 30 October 2013 13:07, Petar Tahchiev > > wrote: > > > >> So spring-data-solr (1.1.SNAPSHOT) uses solr 4.5.1 (just came out a > > few > > > >> days ago), which uses Hadoop 2.0.5-alpha. > > > >> I would be glad if we can clean up the poms a bit and leave only the > > > >> dependencies > > > >> that hadoop really depend on. > > > > > > To pile on top of what Steve has said -- do you happen to know if > there's > > > a JIRA to re target Solr to depend on Hadoop 2.2.0? > > > > > > Thanks, > > > Roman. > > > > > > > > > > > -- > > Regards, Petar! > > Karlovo, Bulgaria. > > --- > > Public PGP Key at: > > > https://keyserver1.pgp.com/vkd/DownloadKey.event?keyid=0x19658550C3110611 > > Key Fingerprint: A369 A7EE 61BC 93A3 CDFF 55A5 1965 8550 C311 0611 > > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. > -- Regards, Petar! Karlovo, Bulgaria. --- Public PGP Key at: https://keyserver1.pgp.com/vkd/DownloadKey.event?keyid=0x19658550C3110611 Key Fingerprint: A369 A7EE 61BC 93A3 CDFF 55A5 1965 8550 C311 0611
Re: Question on hadoop dependencies.
Petar, We've just been through some of 2.3, on the branch https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2 Most of that was update, apart from moving grizzly and junit to test scope, we've not done much. why do you have a look and help clean things out. I'd particularly like to see "lean" clients for HDFS, YARN and mapreduce On 30 October 2013 22:25, Petar Tahchiev wrote: > Hi Roman, > > looks like they have already upgraded to 2.2 > > https://issues.apache.org/jira/browse/SOLR-5382 > > and will be shipping it SOLR 4.6. I just hope you guys release cleaned 2.3 > first :) > > > 2013/10/30 Roman Shaposhnik > > > On Wed, Oct 30, 2013 at 1:07 PM, Steve Loughran > > wrote: > > > On 30 October 2013 13:07, Petar Tahchiev > wrote: > > >> So spring-data-solr (1.1.SNAPSHOT) uses solr 4.5.1 (just came out a > few > > >> days ago), which uses Hadoop 2.0.5-alpha. > > >> I would be glad if we can clean up the poms a bit and leave only the > > >> dependencies > > >> that hadoop really depend on. > > > > To pile on top of what Steve has said -- do you happen to know if there's > > a JIRA to re target Solr to depend on Hadoop 2.2.0? > > > > Thanks, > > Roman. > > > > > > -- > Regards, Petar! > Karlovo, Bulgaria. > --- > Public PGP Key at: > https://keyserver1.pgp.com/vkd/DownloadKey.event?keyid=0x19658550C3110611 > Key Fingerprint: A369 A7EE 61BC 93A3 CDFF 55A5 1965 8550 C311 0611 > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Question on hadoop dependencies.
Hi Roman, looks like they have already upgraded to 2.2 https://issues.apache.org/jira/browse/SOLR-5382 and will be shipping it SOLR 4.6. I just hope you guys release cleaned 2.3 first :) 2013/10/30 Roman Shaposhnik > On Wed, Oct 30, 2013 at 1:07 PM, Steve Loughran > wrote: > > On 30 October 2013 13:07, Petar Tahchiev wrote: > >> So spring-data-solr (1.1.SNAPSHOT) uses solr 4.5.1 (just came out a few > >> days ago), which uses Hadoop 2.0.5-alpha. > >> I would be glad if we can clean up the poms a bit and leave only the > >> dependencies > >> that hadoop really depend on. > > To pile on top of what Steve has said -- do you happen to know if there's > a JIRA to re target Solr to depend on Hadoop 2.2.0? > > Thanks, > Roman. > -- Regards, Petar! Karlovo, Bulgaria. --- Public PGP Key at: https://keyserver1.pgp.com/vkd/DownloadKey.event?keyid=0x19658550C3110611 Key Fingerprint: A369 A7EE 61BC 93A3 CDFF 55A5 1965 8550 C311 0611
Re: Question on hadoop dependencies.
On Wed, Oct 30, 2013 at 1:07 PM, Steve Loughran wrote: > On 30 October 2013 13:07, Petar Tahchiev wrote: >> So spring-data-solr (1.1.SNAPSHOT) uses solr 4.5.1 (just came out a few >> days ago), which uses Hadoop 2.0.5-alpha. >> I would be glad if we can clean up the poms a bit and leave only the >> dependencies >> that hadoop really depend on. To pile on top of what Steve has said -- do you happen to know if there's a JIRA to re target Solr to depend on Hadoop 2.2.0? Thanks, Roman.
Re: Question on hadoop dependencies.
On 30 October 2013 13:07, Petar Tahchiev wrote: > Oh, hi Steve, > > didn't know you were on this list :) ... > Well I didn' t know you were doing Hadoop stuff > > So spring-data-solr (1.1.SNAPSHOT) uses solr 4.5.1 (just came out a few > days ago), which uses Hadoop 2.0.5-alpha. > I would be glad if we can clean up the poms a bit and leave only the > dependencies > that hadoop really depend on. > I'd like to take this opportunity to assign a JIRA to you https://issues.apache.org/jira/browse/HADOOP-9991 spring-data needs to move on to Hadoop 2.2 -by the time they are ready we can make sure that the 2.3 poms are better -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Question on hadoop dependencies.
Oh, hi Steve,
didn't know you were on this list :) ...
So spring-data-solr (1.1.SNAPSHOT) uses solr 4.5.1 (just came out a few
days ago), which uses Hadoop 2.0.5-alpha.
I would be glad if we can clean up the poms a bit and leave only the
dependencies
that hadoop really depend on.
I'll drop an email to the solr guys and see what exactly are they using
hadoop for.
Cheers, Petar.
2013/10/30 Steve Loughran
> Why hello Peter,
>
> Which version are you using?
>
> The reason those dependencies are declared are because things like Jetty
> use them -and the classpath for the server side Hadoop is "things needed to
> run Hadoop".
>
> Client-side I think there's too much in the maven dependency tree
> (servlets, jetty, ...)
>
>
>
> On 29 October 2013 19:22, Petar Tahchiev wrote:
>
> > Hi guys,
> >
> > I'm using Spring-data-solr in my project. SDS declares Solr-core as a
> > dependency. Solr declares hadoop-auth, hadoop-common, hadoop-hdfs
> > dependencies. Each and every one of those dependencies declares log4j and
> > slf4j-log4j12 as runtime dependencies, and also hadoop-common declares
> > servlet-api version 2.5 as runtime dependency. So in the end I also get
> > servlet-api 2.5, slf4 and log4j in my classpath.
> > This normally shouldn't be a problem, but in my case I'm using Servlet
> 3.0,
> > log4j2, and SL4j for log4j2. This completely messes up my classpath, so I
> > have to manually exclude those dependencies, like this:
> > ---
> >
> > org.apache.solr
> > solr-core
> > ${solr.version}
> >
> >
> > org.slf4j
> > slf4j-jdk14
> >
> >
> > org.apache.solr
> > solr-core
> >
> >
> >
> >
> > org.apache.solr
> > solr-core
> > ${solr.version}
> >
> >
> > org.apache.hadoop
> > hadoop-auth
> >
> >
> > org.apache.hadoop
> > hadoop-common
> >
> >
> > org.apache.hadoop
> > hadoop-hdfs
> >
> >
> >
> >
> > org.apache.hadoop
> > hadoop-auth
> > ${hadoop.version}
> >
> >
> > log4j
> > log4j
> >
> >
> > org.slf4j
> > slf4j-log4j12
> >
> >
> >
> >
> > org.apache.hadoop
> > hadoop-common
> > ${hadoop.version}
> >
> >
> > log4j
> > log4j
> >
> >
> > org.slf4j
> > slf4j-log4j12
> >
> >
> > javax.servlet
> > servlet-api
> >
> >
> >
> >
> > org.apache.hadoop
> > hadoop-hdfs
> > ${hadoop.version}
> >
> >
> > log4j
> > log4j
> >
> >
> > org.slf4j
> > slf4j-log4j12
> >
> >
> > javax.servlet
> > servlet-api
> >
> >
> >
> > ---
> > Just out of curiosity here - what is the reason to include log4j,
> > slf4j-log4j12 and servlet-api as runtime dependencies? I think they
> should
> > be with scope provided. It's good to program your API against slf4j, but
> > then the connector should be specified by the user (Log4J2 in my case).
> > Also are there any plans to migrate to Log4J2 - it seems pretty solid (9
> > betas so far) and It is supposed to be released soon.
> >
> > Thanks for you time and keep up the good work.
> >
> > --
> > Regards, Petar!
> > Karlovo, Bulgaria.
> > ---
> > Public PGP Key at:
> >
> https://keyserver1.pgp.com/vkd/DownloadKey.event?keyid=0x19658550C3110611
> > Key Fingerprint: A369 A7EE 61BC 93A3 CDFF 55A5 1965 8550 C311 0611
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibite
Re: Question on hadoop dependencies.
Why hello Peter,
Which version are you using?
The reason those dependencies are declared are because things like Jetty
use them -and the classpath for the server side Hadoop is "things needed to
run Hadoop".
Client-side I think there's too much in the maven dependency tree
(servlets, jetty, ...)
On 29 October 2013 19:22, Petar Tahchiev wrote:
> Hi guys,
>
> I'm using Spring-data-solr in my project. SDS declares Solr-core as a
> dependency. Solr declares hadoop-auth, hadoop-common, hadoop-hdfs
> dependencies. Each and every one of those dependencies declares log4j and
> slf4j-log4j12 as runtime dependencies, and also hadoop-common declares
> servlet-api version 2.5 as runtime dependency. So in the end I also get
> servlet-api 2.5, slf4 and log4j in my classpath.
> This normally shouldn't be a problem, but in my case I'm using Servlet 3.0,
> log4j2, and SL4j for log4j2. This completely messes up my classpath, so I
> have to manually exclude those dependencies, like this:
> ---
>
> org.apache.solr
> solr-core
> ${solr.version}
>
>
> org.slf4j
> slf4j-jdk14
>
>
> org.apache.solr
> solr-core
>
>
>
>
> org.apache.solr
> solr-core
> ${solr.version}
>
>
> org.apache.hadoop
> hadoop-auth
>
>
> org.apache.hadoop
> hadoop-common
>
>
> org.apache.hadoop
> hadoop-hdfs
>
>
>
>
> org.apache.hadoop
> hadoop-auth
> ${hadoop.version}
>
>
> log4j
> log4j
>
>
> org.slf4j
> slf4j-log4j12
>
>
>
>
> org.apache.hadoop
> hadoop-common
> ${hadoop.version}
>
>
> log4j
> log4j
>
>
> org.slf4j
> slf4j-log4j12
>
>
> javax.servlet
> servlet-api
>
>
>
>
> org.apache.hadoop
> hadoop-hdfs
> ${hadoop.version}
>
>
> log4j
> log4j
>
>
> org.slf4j
> slf4j-log4j12
>
>
> javax.servlet
> servlet-api
>
>
>
> ---
> Just out of curiosity here - what is the reason to include log4j,
> slf4j-log4j12 and servlet-api as runtime dependencies? I think they should
> be with scope provided. It's good to program your API against slf4j, but
> then the connector should be specified by the user (Log4J2 in my case).
> Also are there any plans to migrate to Log4J2 - it seems pretty solid (9
> betas so far) and It is supposed to be released soon.
>
> Thanks for you time and keep up the good work.
>
> --
> Regards, Petar!
> Karlovo, Bulgaria.
> ---
> Public PGP Key at:
> https://keyserver1.pgp.com/vkd/DownloadKey.event?keyid=0x19658550C3110611
> Key Fingerprint: A369 A7EE 61BC 93A3 CDFF 55A5 1965 8550 C311 0611
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Question on hadoop dependencies.
Hi guys,
I'm using Spring-data-solr in my project. SDS declares Solr-core as a
dependency. Solr declares hadoop-auth, hadoop-common, hadoop-hdfs
dependencies. Each and every one of those dependencies declares log4j and
slf4j-log4j12 as runtime dependencies, and also hadoop-common declares
servlet-api version 2.5 as runtime dependency. So in the end I also get
servlet-api 2.5, slf4 and log4j in my classpath.
This normally shouldn't be a problem, but in my case I'm using Servlet 3.0,
log4j2, and SL4j for log4j2. This completely messes up my classpath, so I
have to manually exclude those dependencies, like this:
---
org.apache.solr
solr-core
${solr.version}
org.slf4j
slf4j-jdk14
org.apache.solr
solr-core
org.apache.solr
solr-core
${solr.version}
org.apache.hadoop
hadoop-auth
org.apache.hadoop
hadoop-common
org.apache.hadoop
hadoop-hdfs
org.apache.hadoop
hadoop-auth
${hadoop.version}
log4j
log4j
org.slf4j
slf4j-log4j12
org.apache.hadoop
hadoop-common
${hadoop.version}
log4j
log4j
org.slf4j
slf4j-log4j12
javax.servlet
servlet-api
org.apache.hadoop
hadoop-hdfs
${hadoop.version}
log4j
log4j
org.slf4j
slf4j-log4j12
javax.servlet
servlet-api
---
Just out of curiosity here - what is the reason to include log4j,
slf4j-log4j12 and servlet-api as runtime dependencies? I think they should
be with scope provided. It's good to program your API against slf4j, but
then the connector should be specified by the user (Log4J2 in my case).
Also are there any plans to migrate to Log4J2 - it seems pretty solid (9
betas so far) and It is supposed to be released soon.
Thanks for you time and keep up the good work.
--
Regards, Petar!
Karlovo, Bulgaria.
---
Public PGP Key at:
https://keyserver1.pgp.com/vkd/DownloadKey.event?keyid=0x19658550C3110611
Key Fingerprint: A369 A7EE 61BC 93A3 CDFF 55A5 1965 8550 C311 0611
