Re: Plans of moving towards JDK7 in trunk

[Alejandro Abdelnur]

newer jetties have non backwards compat APIs, we would break any user app
using jetty (consumed via hadoop classpath)



On Fri, Apr 11, 2014 at 2:16 PM, Steve Loughran wrote:

> that doesn't actually stop is from switching in our own code to alternate
> web servers,  only that jetty can remain a published artifact in the
> hadoop/lib dir
>
>
> On 11 April 2014 21:16, Alejandro Abdelnur  wrote:
>
> > because it is exposed as classpath dependency, changing it breaks
> backward
> > compatibility.
> >
> >
> > On Fri, Apr 11, 2014 at 1:02 PM, Steve Loughran  > >wrote:
> >
> > > Jetty's a big change, it's fairly intimately involved in bits of the
> code
> > >
> > > but: it's a source of grief, currently webhdfs is an example
> > > https://issues.apache.org/jira/browse/HDFS-6221
> > >
> > > all YARN apps seem to get hosted by it too
> > >
> > >
> > > On 11 April 2014 20:56, Robert Rati  wrote:
> > >
> > > > I don't mean to be dense, but can you expand on why jetty 8 can't go
> > into
> > > > branch2?  What is the concern?
> > > >
> > > > Rob
> > > >
> > > >
> > > > On 04/11/2014 10:55 AM, Alejandro Abdelnur wrote:
> > > >
> > > >> if you mean updating jetty on branch2, we cannot do that. it has to
> be
> > > >> done in trunk.
> > > >>
> > > >> thx
> > > >>
> > > >> Alejandro
> > > >> (phone typing)
> > > >>
> > > >>  On Apr 11, 2014, at 4:46, Robert Rati  wrote:
> > > >>>
> > > >>> Just an FYI, but I'm working on updating that jetty patch for the
> > > >>> current 2.4.0 release.  The one that is there won't cleanly apply
> > > because
> > > >>> so much has changed since it was posted.  I'll post a new patch
> when
> > > it's
> > > >>> done.
> > > >>>
> > > >>> Rob
> > > >>>
> > > >>>  On 04/11/2014 04:24 AM, Steve Loughran wrote:
> > > 
> > > > On 10 April 2014 18:12, Eli Collins  wrote:
> > > >
> > > > Let's speak less abstractly, are there particular features or new
> > > > dependencies that you would like to contribute (or see
> contributed)
> > > > that
> > > > require using the Java 1.7 APIs?  Breaking compat in v2 or
> rolling
> > a
> > > v3
> > > > release are both non-trivial, not something I suspect we'd want
> to
> > do
> > > > just
> > > > because it would be, for example, nicer to have a newer version
> of
> > > > Jetty.
> > > >
> > > 
> > >  Oddly enough, rolling the web framework is something I'd like to
> see
> > > in
> > >  a
> > >  v3. the shuffle may be off jetty, but webhdfs isn't. Moving up
> also
> > >  lets is
> > >  reliably switch to servlet API v3
> > > 
> > >  But.. I think we may be able to increment Jetty more without going
> > to
> > >  java7, see https://issues.apache.org/jira/browse/HADOOP-9650 .
> > > 
> > > 
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
> >
> >
> > --
> > Alejandro
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>



-- 
Alejandro

Re: Plans of moving towards JDK7 in trunk

[Steve Loughran]

that doesn't actually stop is from switching in our own code to alternate
web servers,  only that jetty can remain a published artifact in the
hadoop/lib dir


On 11 April 2014 21:16, Alejandro Abdelnur  wrote:

> because it is exposed as classpath dependency, changing it breaks backward
> compatibility.
>
>
> On Fri, Apr 11, 2014 at 1:02 PM, Steve Loughran  >wrote:
>
> > Jetty's a big change, it's fairly intimately involved in bits of the code
> >
> > but: it's a source of grief, currently webhdfs is an example
> > https://issues.apache.org/jira/browse/HDFS-6221
> >
> > all YARN apps seem to get hosted by it too
> >
> >
> > On 11 April 2014 20:56, Robert Rati  wrote:
> >
> > > I don't mean to be dense, but can you expand on why jetty 8 can't go
> into
> > > branch2?  What is the concern?
> > >
> > > Rob
> > >
> > >
> > > On 04/11/2014 10:55 AM, Alejandro Abdelnur wrote:
> > >
> > >> if you mean updating jetty on branch2, we cannot do that. it has to be
> > >> done in trunk.
> > >>
> > >> thx
> > >>
> > >> Alejandro
> > >> (phone typing)
> > >>
> > >>  On Apr 11, 2014, at 4:46, Robert Rati  wrote:
> > >>>
> > >>> Just an FYI, but I'm working on updating that jetty patch for the
> > >>> current 2.4.0 release.  The one that is there won't cleanly apply
> > because
> > >>> so much has changed since it was posted.  I'll post a new patch when
> > it's
> > >>> done.
> > >>>
> > >>> Rob
> > >>>
> > >>>  On 04/11/2014 04:24 AM, Steve Loughran wrote:
> > 
> > > On 10 April 2014 18:12, Eli Collins  wrote:
> > >
> > > Let's speak less abstractly, are there particular features or new
> > > dependencies that you would like to contribute (or see contributed)
> > > that
> > > require using the Java 1.7 APIs?  Breaking compat in v2 or rolling
> a
> > v3
> > > release are both non-trivial, not something I suspect we'd want to
> do
> > > just
> > > because it would be, for example, nicer to have a newer version of
> > > Jetty.
> > >
> > 
> >  Oddly enough, rolling the web framework is something I'd like to see
> > in
> >  a
> >  v3. the shuffle may be off jetty, but webhdfs isn't. Moving up also
> >  lets is
> >  reliably switch to servlet API v3
> > 
> >  But.. I think we may be able to increment Jetty more without going
> to
> >  java7, see https://issues.apache.org/jira/browse/HADOOP-9650 .
> > 
> > 
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
>
>
> --
> Alejandro
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Plans of moving towards JDK7 in trunk

[Alejandro Abdelnur]

because it is exposed as classpath dependency, changing it breaks backward
compatibility.


On Fri, Apr 11, 2014 at 1:02 PM, Steve Loughran wrote:

> Jetty's a big change, it's fairly intimately involved in bits of the code
>
> but: it's a source of grief, currently webhdfs is an example
> https://issues.apache.org/jira/browse/HDFS-6221
>
> all YARN apps seem to get hosted by it too
>
>
> On 11 April 2014 20:56, Robert Rati  wrote:
>
> > I don't mean to be dense, but can you expand on why jetty 8 can't go into
> > branch2?  What is the concern?
> >
> > Rob
> >
> >
> > On 04/11/2014 10:55 AM, Alejandro Abdelnur wrote:
> >
> >> if you mean updating jetty on branch2, we cannot do that. it has to be
> >> done in trunk.
> >>
> >> thx
> >>
> >> Alejandro
> >> (phone typing)
> >>
> >>  On Apr 11, 2014, at 4:46, Robert Rati  wrote:
> >>>
> >>> Just an FYI, but I'm working on updating that jetty patch for the
> >>> current 2.4.0 release.  The one that is there won't cleanly apply
> because
> >>> so much has changed since it was posted.  I'll post a new patch when
> it's
> >>> done.
> >>>
> >>> Rob
> >>>
> >>>  On 04/11/2014 04:24 AM, Steve Loughran wrote:
> 
> > On 10 April 2014 18:12, Eli Collins  wrote:
> >
> > Let's speak less abstractly, are there particular features or new
> > dependencies that you would like to contribute (or see contributed)
> > that
> > require using the Java 1.7 APIs?  Breaking compat in v2 or rolling a
> v3
> > release are both non-trivial, not something I suspect we'd want to do
> > just
> > because it would be, for example, nicer to have a newer version of
> > Jetty.
> >
> 
>  Oddly enough, rolling the web framework is something I'd like to see
> in
>  a
>  v3. the shuffle may be off jetty, but webhdfs isn't. Moving up also
>  lets is
>  reliably switch to servlet API v3
> 
>  But.. I think we may be able to increment Jetty more without going to
>  java7, see https://issues.apache.org/jira/browse/HADOOP-9650 .
> 
> 
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>



-- 
Alejandro

Re: Plans of moving towards JDK7 in trunk

[Steve Loughran]

Jetty's a big change, it's fairly intimately involved in bits of the code

but: it's a source of grief, currently webhdfs is an example
https://issues.apache.org/jira/browse/HDFS-6221

all YARN apps seem to get hosted by it too


On 11 April 2014 20:56, Robert Rati  wrote:

> I don't mean to be dense, but can you expand on why jetty 8 can't go into
> branch2?  What is the concern?
>
> Rob
>
>
> On 04/11/2014 10:55 AM, Alejandro Abdelnur wrote:
>
>> if you mean updating jetty on branch2, we cannot do that. it has to be
>> done in trunk.
>>
>> thx
>>
>> Alejandro
>> (phone typing)
>>
>>  On Apr 11, 2014, at 4:46, Robert Rati  wrote:
>>>
>>> Just an FYI, but I'm working on updating that jetty patch for the
>>> current 2.4.0 release.  The one that is there won't cleanly apply because
>>> so much has changed since it was posted.  I'll post a new patch when it's
>>> done.
>>>
>>> Rob
>>>
>>>  On 04/11/2014 04:24 AM, Steve Loughran wrote:

> On 10 April 2014 18:12, Eli Collins  wrote:
>
> Let's speak less abstractly, are there particular features or new
> dependencies that you would like to contribute (or see contributed)
> that
> require using the Java 1.7 APIs?  Breaking compat in v2 or rolling a v3
> release are both non-trivial, not something I suspect we'd want to do
> just
> because it would be, for example, nicer to have a newer version of
> Jetty.
>

 Oddly enough, rolling the web framework is something I'd like to see in
 a
 v3. the shuffle may be off jetty, but webhdfs isn't. Moving up also
 lets is
 reliably switch to servlet API v3

 But.. I think we may be able to increment Jetty more without going to
 java7, see https://issues.apache.org/jira/browse/HADOOP-9650 .



-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Plans of moving towards JDK7 in trunk

[Robert Rati]

I don't mean to be dense, but can you expand on why jetty 8 can't go 
into branch2?  What is the concern?


Rob

On 04/11/2014 10:55 AM, Alejandro Abdelnur wrote:

if you mean updating jetty on branch2, we cannot do that. it has to be done in 
trunk.

thx

Alejandro
(phone typing)


On Apr 11, 2014, at 4:46, Robert Rati  wrote:

Just an FYI, but I'm working on updating that jetty patch for the current 2.4.0 
release.  The one that is there won't cleanly apply because so much has changed 
since it was posted.  I'll post a new patch when it's done.

Rob


On 04/11/2014 04:24 AM, Steve Loughran wrote:

On 10 April 2014 18:12, Eli Collins  wrote:

Let's speak less abstractly, are there particular features or new
dependencies that you would like to contribute (or see contributed) that
require using the Java 1.7 APIs?  Breaking compat in v2 or rolling a v3
release are both non-trivial, not something I suspect we'd want to do just
because it would be, for example, nicer to have a newer version of Jetty.


Oddly enough, rolling the web framework is something I'd like to see in a
v3. the shuffle may be off jetty, but webhdfs isn't. Moving up also lets is
reliably switch to servlet API v3

But.. I think we may be able to increment Jetty more without going to
java7, see https://issues.apache.org/jira/browse/HADOOP-9650 .

Re: [Important] Confirmation related to OpenSsl security issue

[Colin McCabe]

I took a quick glance at the build output, and I don't think openssl
is getting linked statically into libhadooppipes.a.

I see the following lines:

Linking CXX static library libhadooppipes.a
/usr/bin/cmake -P CMakeFiles/hadooppipes.dir/cmake_clean_target.cmake
/usr/bin/cmake -E cmake_link_script
CMakeFiles/hadooppipes.dir/link.txt --verbose=1
/usr/bin/ar cr libhadooppipes.a
CMakeFiles/hadooppipes.dir/main/native/pipes/impl/HadoopPipes.cc.o
/usr/bin/ranlib libhadooppipes.a

later on there are lines like this:

/usr/bin/c++-g -Wall -O2 -D_REENTRANT -D_GNU_SOURCE
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
CMakeFiles/pipes-sort.dir/main/native/examples/impl/sort.cc.o  -o
examples/pipes-sort -rdynamic libhadooppipes.a libhadooputils.a -lssl
-lcrypto -lpthread

So when using libhadooppipes.a, you must supply your own copy of
libssl.so.  If you supply a vulnerable copy, you will be vulnerable.
If you supply a non-vulnerable copy, you won't be.  So I don't think
there is an impact on our build (unless I missed something here).

Just to make sure, it would be good if someone who uses
libhadooppipes.a to look at one of the versions in our release tarball
and verify that it works with the fixed openssl.

Colin

On Fri, Apr 11, 2014 at 2:27 AM, Vinayakumar B  wrote:
> Hi,
>
> Recently one security issue has been found in OpenSSL which has impacted so 
> many customers of different vendors.
>
> http://www.kb.cert.org/vuls/byvendor?searchview&Query=FIELD+Reference=720951&SearchOrder=4
>
> I want to ask, whether is there in impact of this on the Hadoop Release?
>
> Currently Hadoop-pipes are uses openssl-devel packages for building native 
> support.
>
> Can someone familiar with Hadoop-pipes can confirm whether is there any 
> impact of this security issue on builds of Hadoop built with defective 
> openssl?
>
> Regards,
>Vinay
>
> 
> This e-mail and attachments contain confidential information from HUAWEI,
> which is intended only for the person or entity whose address is listed
> above. Any use of the information contained herein in any way (including,
> but not limited to, total or partial disclosure, reproduction, or
> dissemination) by persons other than the intended recipient's) is
> prohibited. If you receive this e-mail in error, please notify the sender by
> phone or email immediately and delete it!
>

[jira] [Resolved] (HADOOP-10431) Change visibility of KeyStore.Options getter methods to public

[Alejandro Abdelnur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-10431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alejandro Abdelnur resolved HADOOP-10431.
-

   Resolution: Fixed
Fix Version/s: 3.0.0
 Hadoop Flags: Reviewed

committed to trunk.

> Change visibility of KeyStore.Options getter methods to public
> --
>
> Key: HADOOP-10431
> URL: https://issues.apache.org/jira/browse/HADOOP-10431
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: security
>Affects Versions: 3.0.0
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Fix For: 3.0.0
>
> Attachments: HADOOP-10431.patch, HADOOP-10431.patch, 
> HADOOP-10431.patch
>
>
> Making Options getter methods public will enable {{KeyProvider}} 
> implementations to use those classes.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: DISCUSS: use SLF4J APIs in new modules?

[Alejandro Abdelnur]

if you dont convert mgs to templates the dont remove the guards, else you 
create str mgs obj even when not logging. 

thx

Alejandro
(phone typing)

> On Apr 11, 2014, at 10:06, Karthik Kambatla  wrote:
> 
> On Fri, Apr 11, 2014 at 1:37 AM, Steve Loughran wrote:
> 
>> On 10 April 2014 16:38, Karthik Kambatla  wrote:
>> 
>>> +1 to use slf4j. I would actually vote for (1) new modules must-use, (2)
>>> new classes in existing modules are strongly recommended to use, (3)
>>> existing classes can switch to. That would take us closer to using slf4j
>>> everywhere faster.
>> 
>> 
>> #1  appeals to me.
>> 
>> #2 & #3, we'd have a mixed phase but ultimately it'd be good. Maybe have a
>> policy for a class switch process of
>> a) switch the LOG declaration, change the imports
>> b) clean up all log statements, dropping the ifDebug and moving to {}
>> strings instead of "text"+ "value
> 
> I feel more the requirements we add to switching, the less likely people
> will make the time for it. I think it is reasonable to switch LOG
> declaration and drop ifDebug. Changing all log messages to use {} instead
> of " " +  " " could be really time-taking especially for classes with tons
> of log messages. On the other hand, asking people to do that if they are
> editing an existing log message anyway, it might fly.
> 
> 
>> 
>> or do both at the same time, one class or package at time.
>> 
>> 
>> Having a consistent log scheme across all classes makes copying and moving
>> code easier, especially copy+paste
>> 
>> I think there's some bits of code that takes a commons-log argument as a
>> parameter. If these are public they'd need to be retained, and we'd have to
>> add an SLF4J logger equivalent.
>> 
>> -Steve
>> 
>> 
>>> 
>>> On Thu, Apr 10, 2014 at 8:17 AM, Alejandro Abdelnur >>> wrote:
>>> 
 +1 pn slf4j.
 
 one thing Jay, the issues with log4j will still be there as log4j will
 still be under the hood.
 
 thx
 
 Alejandro
 (phone typing)
 
> On Apr 10, 2014, at 7:35, Andrew Wang 
>>> wrote:
> 
> +1 from me, it'd be lovely to get rid of all those isDebugEnabled
>>> checks.
> 
> 
>> On Thu, Apr 10, 2014 at 4:13 AM, Jay Vyas 
>>> wrote:
>> 
>> Slf4j is definetly a great step forward.  Log4j is restrictive for
 complex
>> and multi tenant apps like hadoop.
>> 
>> Also the fact that slf4j doesn't use any magic when binding to its
>> log
>> provider makes it way easier to swap out its implementation then
>> tools
 of
>> the past.
>> 
 On Apr 10, 2014, at 2:16 AM, Steve Loughran <
>> ste...@hortonworks.com
 
>>> wrote:
>>> 
>>> If we're thinking of future progress, here's a little low-level
>> one:
>> adopt
>>> SLF4J as the API for logging
>>> 
>>> 
>>> 1. its the new defacto standard of logging APIs
>>> 2. its a lot better than commons-logging with on demand Inline
>>> string
>>> expansion of varags arguments.
>>> 3. we already ship it, as jetty uses it
>>> 4. we already depend on it, client-side and server-side in the
>>> hadoop-auth package
>>> 5. it lets people log via logback if they want to. That's
>>> client-side,
>>> even if the server stays on log4j
>>> 6. It's way faster than using String.format()
>>> 
>>> 
>>> The best initial thing about SL4FJ is how it only expands its
>>> arguments
>>> string values if needed
>>> 
>>>LOG.debug("Initialized, principal [{}] from keytab [{}]",
 principal,
>>> keytab);
>>> 
>>> not logging at debug? No need to test first. That alone saves code
>>> and
>>> improves readability.
>>> 
>>> The slf4 expansion code handles null values as well as calling
 toString()
>>> on non-null arguments. Oh and it does arrays too.
>>> 
>>> int array = [1, 2, 3];
>>> String undef = null;
>>> 
>>> LOG.info("a = {}, u = {}", array, undef)  -> "a = [1, 2, 3],  u =
>>> null"
>>> 
>>> Switching to SLF4J from commons-logging is as trivial as changing
>> the
>> type
>>> of the logger created, but with one logger per class that does get
>>> expensive in terms of change. Moving to SLF4J across the board
>> would
 be a
>>> big piece of work -but doable.
>>> 
>>> Rather than push for a dramatic change why not adopt a policy of
>> demanding
>>> it in new maven subprojects? hadoop-auth shows we permit it, so why
>>> not
>> say
>>> "you MUST"?
>>> 
>>> Once people have experience in using it, and are happy, then we
>> could
>> think
>>> about switching to the new APIs in the core modules. The only
 troublespot
>>> there is where code calls getLogger() on the commons log to get at
>>> the
>>> log4j appender -there's ~3 places in production code that does
>> this,
 200+
>>> in tests -tests that do it to turn back log levels. Those tests can
 stay
>>> with commons-log

Re: DISCUSS: use SLF4J APIs in new modules?

[Abdelrahman Shettia]

In addition, we need to consider limiting any printing messages to the
stdout in some cases. This can impact other running some apache products in
silent mode such as hive in 'hive -S' option.

Thanks
-Rahman


On Fri, Apr 11, 2014 at 10:06 AM, Karthik Kambatla wrote:

> On Fri, Apr 11, 2014 at 1:37 AM, Steve Loughran  >wrote:
>
> > On 10 April 2014 16:38, Karthik Kambatla  wrote:
> >
> > > +1 to use slf4j. I would actually vote for (1) new modules must-use,
> (2)
> > > new classes in existing modules are strongly recommended to use, (3)
> > > existing classes can switch to. That would take us closer to using
> slf4j
> > > everywhere faster.
> > >
> >
> >
> > #1  appeals to me.
> >
> > #2 & #3, we'd have a mixed phase but ultimately it'd be good. Maybe have
> a
> > policy for a class switch process of
> > a) switch the LOG declaration, change the imports
> > b) clean up all log statements, dropping the ifDebug and moving to {}
> > strings instead of "text"+ "value
> >
>
> I feel more the requirements we add to switching, the less likely people
> will make the time for it. I think it is reasonable to switch LOG
> declaration and drop ifDebug. Changing all log messages to use {} instead
> of " " +  " " could be really time-taking especially for classes with tons
> of log messages. On the other hand, asking people to do that if they are
> editing an existing log message anyway, it might fly.
>
>
> >
> > or do both at the same time, one class or package at time.
> >
> >
> > Having a consistent log scheme across all classes makes copying and
> moving
> > code easier, especially copy+paste
> >
> > I think there's some bits of code that takes a commons-log argument as a
> > parameter. If these are public they'd need to be retained, and we'd have
> to
> > add an SLF4J logger equivalent.
> >
> > -Steve
> >
> >
> > >
> > > On Thu, Apr 10, 2014 at 8:17 AM, Alejandro Abdelnur  > > >wrote:
> > >
> > > > +1 pn slf4j.
> > > >
> > > > one thing Jay, the issues with log4j will still be there as log4j
> will
> > > > still be under the hood.
> > > >
> > > > thx
> > > >
> > > > Alejandro
> > > > (phone typing)
> > > >
> > > > > On Apr 10, 2014, at 7:35, Andrew Wang 
> > > wrote:
> > > > >
> > > > > +1 from me, it'd be lovely to get rid of all those isDebugEnabled
> > > checks.
> > > > >
> > > > >
> > > > >> On Thu, Apr 10, 2014 at 4:13 AM, Jay Vyas 
> > > wrote:
> > > > >>
> > > > >> Slf4j is definetly a great step forward.  Log4j is restrictive for
> > > > complex
> > > > >> and multi tenant apps like hadoop.
> > > > >>
> > > > >> Also the fact that slf4j doesn't use any magic when binding to its
> > log
> > > > >> provider makes it way easier to swap out its implementation then
> > tools
> > > > of
> > > > >> the past.
> > > > >>
> > > >  On Apr 10, 2014, at 2:16 AM, Steve Loughran <
> > ste...@hortonworks.com
> > > >
> > > > >>> wrote:
> > > > >>>
> > > > >>> If we're thinking of future progress, here's a little low-level
> > one:
> > > > >> adopt
> > > > >>> SLF4J as the API for logging
> > > > >>>
> > > > >>>
> > > > >>>  1. its the new defacto standard of logging APIs
> > > > >>>  2. its a lot better than commons-logging with on demand Inline
> > > string
> > > > >>>  expansion of varags arguments.
> > > > >>>  3. we already ship it, as jetty uses it
> > > > >>>  4. we already depend on it, client-side and server-side in the
> > > > >>>  hadoop-auth package
> > > > >>>  5. it lets people log via logback if they want to. That's
> > > client-side,
> > > > >>>  even if the server stays on log4j
> > > > >>>  6. It's way faster than using String.format()
> > > > >>>
> > > > >>>
> > > > >>> The best initial thing about SL4FJ is how it only expands its
> > > arguments
> > > > >>> string values if needed
> > > > >>>
> > > > >>> LOG.debug("Initialized, principal [{}] from keytab [{}]",
> > > > principal,
> > > > >>> keytab);
> > > > >>>
> > > > >>> not logging at debug? No need to test first. That alone saves
> code
> > > and
> > > > >>> improves readability.
> > > > >>>
> > > > >>> The slf4 expansion code handles null values as well as calling
> > > > toString()
> > > > >>> on non-null arguments. Oh and it does arrays too.
> > > > >>>
> > > > >>> int array = [1, 2, 3];
> > > > >>> String undef = null;
> > > > >>>
> > > > >>> LOG.info("a = {}, u = {}", array, undef)  -> "a = [1, 2, 3],  u =
> > > null"
> > > > >>>
> > > > >>> Switching to SLF4J from commons-logging is as trivial as changing
> > the
> > > > >> type
> > > > >>> of the logger created, but with one logger per class that does
> get
> > > > >>> expensive in terms of change. Moving to SLF4J across the board
> > would
> > > > be a
> > > > >>> big piece of work -but doable.
> > > > >>>
> > > > >>> Rather than push for a dramatic change why not adopt a policy of
> > > > >> demanding
> > > > >>> it in new maven subprojects? hadoop-auth shows we permit it, so
> why
> > > not
> > > > >> say
> > > > >>> "you MUST"?
> > > > >>>
> > > > >>> Once people have experience i

Re: DISCUSS: use SLF4J APIs in new modules?

[Karthik Kambatla]

On Fri, Apr 11, 2014 at 1:37 AM, Steve Loughran wrote:

> On 10 April 2014 16:38, Karthik Kambatla  wrote:
>
> > +1 to use slf4j. I would actually vote for (1) new modules must-use, (2)
> > new classes in existing modules are strongly recommended to use, (3)
> > existing classes can switch to. That would take us closer to using slf4j
> > everywhere faster.
> >
>
>
> #1  appeals to me.
>
> #2 & #3, we'd have a mixed phase but ultimately it'd be good. Maybe have a
> policy for a class switch process of
> a) switch the LOG declaration, change the imports
> b) clean up all log statements, dropping the ifDebug and moving to {}
> strings instead of "text"+ "value
>

I feel more the requirements we add to switching, the less likely people
will make the time for it. I think it is reasonable to switch LOG
declaration and drop ifDebug. Changing all log messages to use {} instead
of " " +  " " could be really time-taking especially for classes with tons
of log messages. On the other hand, asking people to do that if they are
editing an existing log message anyway, it might fly.


>
> or do both at the same time, one class or package at time.
>
>
> Having a consistent log scheme across all classes makes copying and moving
> code easier, especially copy+paste
>
> I think there's some bits of code that takes a commons-log argument as a
> parameter. If these are public they'd need to be retained, and we'd have to
> add an SLF4J logger equivalent.
>
> -Steve
>
>
> >
> > On Thu, Apr 10, 2014 at 8:17 AM, Alejandro Abdelnur  > >wrote:
> >
> > > +1 pn slf4j.
> > >
> > > one thing Jay, the issues with log4j will still be there as log4j will
> > > still be under the hood.
> > >
> > > thx
> > >
> > > Alejandro
> > > (phone typing)
> > >
> > > > On Apr 10, 2014, at 7:35, Andrew Wang 
> > wrote:
> > > >
> > > > +1 from me, it'd be lovely to get rid of all those isDebugEnabled
> > checks.
> > > >
> > > >
> > > >> On Thu, Apr 10, 2014 at 4:13 AM, Jay Vyas 
> > wrote:
> > > >>
> > > >> Slf4j is definetly a great step forward.  Log4j is restrictive for
> > > complex
> > > >> and multi tenant apps like hadoop.
> > > >>
> > > >> Also the fact that slf4j doesn't use any magic when binding to its
> log
> > > >> provider makes it way easier to swap out its implementation then
> tools
> > > of
> > > >> the past.
> > > >>
> > >  On Apr 10, 2014, at 2:16 AM, Steve Loughran <
> ste...@hortonworks.com
> > >
> > > >>> wrote:
> > > >>>
> > > >>> If we're thinking of future progress, here's a little low-level
> one:
> > > >> adopt
> > > >>> SLF4J as the API for logging
> > > >>>
> > > >>>
> > > >>>  1. its the new defacto standard of logging APIs
> > > >>>  2. its a lot better than commons-logging with on demand Inline
> > string
> > > >>>  expansion of varags arguments.
> > > >>>  3. we already ship it, as jetty uses it
> > > >>>  4. we already depend on it, client-side and server-side in the
> > > >>>  hadoop-auth package
> > > >>>  5. it lets people log via logback if they want to. That's
> > client-side,
> > > >>>  even if the server stays on log4j
> > > >>>  6. It's way faster than using String.format()
> > > >>>
> > > >>>
> > > >>> The best initial thing about SL4FJ is how it only expands its
> > arguments
> > > >>> string values if needed
> > > >>>
> > > >>> LOG.debug("Initialized, principal [{}] from keytab [{}]",
> > > principal,
> > > >>> keytab);
> > > >>>
> > > >>> not logging at debug? No need to test first. That alone saves code
> > and
> > > >>> improves readability.
> > > >>>
> > > >>> The slf4 expansion code handles null values as well as calling
> > > toString()
> > > >>> on non-null arguments. Oh and it does arrays too.
> > > >>>
> > > >>> int array = [1, 2, 3];
> > > >>> String undef = null;
> > > >>>
> > > >>> LOG.info("a = {}, u = {}", array, undef)  -> "a = [1, 2, 3],  u =
> > null"
> > > >>>
> > > >>> Switching to SLF4J from commons-logging is as trivial as changing
> the
> > > >> type
> > > >>> of the logger created, but with one logger per class that does get
> > > >>> expensive in terms of change. Moving to SLF4J across the board
> would
> > > be a
> > > >>> big piece of work -but doable.
> > > >>>
> > > >>> Rather than push for a dramatic change why not adopt a policy of
> > > >> demanding
> > > >>> it in new maven subprojects? hadoop-auth shows we permit it, so why
> > not
> > > >> say
> > > >>> "you MUST"?
> > > >>>
> > > >>> Once people have experience in using it, and are happy, then we
> could
> > > >> think
> > > >>> about switching to the new APIs in the core modules. The only
> > > troublespot
> > > >>> there is where code calls getLogger() on the commons log to get at
> > the
> > > >>> log4j appender -there's ~3 places in production code that does
> this,
> > > 200+
> > > >>> in tests -tests that do it to turn back log levels. Those tests can
> > > stay
> > > >>> with commons-logging, same for the production uses. Mixing
> > > >> commons-logging
> > > >>> and slf4j isn't drastic -they both route to log4

[jira] [Resolved] (HADOOP-9219) coverage fixing for org.apache.hadoop.tools.rumen

[Jonathan Eagles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Eagles resolved HADOOP-9219.
-

Resolution: Fixed

Duping this issue to MAPREDUCE-3860 as per Andrey's comment.

> coverage fixing for org.apache.hadoop.tools.rumen
> -
>
> Key: HADOOP-9219
> URL: https://issues.apache.org/jira/browse/HADOOP-9219
> Project: Hadoop Common
>  Issue Type: Test
>  Components: tools
>Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6
>Reporter: Aleksey Gorshkov
>Assignee: Aleksey Gorshkov
> Attachments: HADOOP-9219-trunk-a.patch, HADOOP-9219-trunk-b.patch, 
> HADOOP-9219-trunk.patch
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> coverage fixing for org.apache.hadoop.tools.rumen 
> HADOOP-9219-trunk.patch for trunk, brunch-2 and branch-0.23



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Plans of moving towards JDK7 in trunk

[Alejandro Abdelnur]

if you mean updating jetty on branch2, we cannot do that. it has to be done in 
trunk. 

thx

Alejandro
(phone typing)

> On Apr 11, 2014, at 4:46, Robert Rati  wrote:
> 
> Just an FYI, but I'm working on updating that jetty patch for the current 
> 2.4.0 release.  The one that is there won't cleanly apply because so much has 
> changed since it was posted.  I'll post a new patch when it's done.
> 
> Rob
> 
>> On 04/11/2014 04:24 AM, Steve Loughran wrote:
>>> On 10 April 2014 18:12, Eli Collins  wrote:
>>> 
>>> Let's speak less abstractly, are there particular features or new
>>> dependencies that you would like to contribute (or see contributed) that
>>> require using the Java 1.7 APIs?  Breaking compat in v2 or rolling a v3
>>> release are both non-trivial, not something I suspect we'd want to do just
>>> because it would be, for example, nicer to have a newer version of Jetty.
>> 
>> Oddly enough, rolling the web framework is something I'd like to see in a
>> v3. the shuffle may be off jetty, but webhdfs isn't. Moving up also lets is
>> reliably switch to servlet API v3
>> 
>> But.. I think we may be able to increment Jetty more without going to
>> java7, see https://issues.apache.org/jira/browse/HADOOP-9650 .
>> 

[jira] [Resolved] (HADOOP-8746) TestNativeIO fails when run with jdk7

[Mit Desai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-8746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mit Desai resolved HADOOP-8746.
---

  Resolution: Not a Problem
Target Version/s: 2.0.2-alpha, 0.23.3, 3.0.0  (was: 0.23.3, 3.0.0, 
2.0.2-alpha)

> TestNativeIO fails when run with jdk7
> -
>
> Key: HADOOP-8746
> URL: https://issues.apache.org/jira/browse/HADOOP-8746
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 0.23.3, 2.0.2-alpha
>Reporter: Thomas Graves
>Assignee: Thomas Graves
>  Labels: java7
>
> TestNativeIo fails when run with jdk7.
> Test set: org.apache.hadoop.io.nativeio.TestNativeIO
> ---
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.232 sec <<< 
> FAILURE!
> testSyncFileRange(org.apache.hadoop.io.nativeio.TestNativeIO)  Time elapsed: 
> 0.166 sec  <<< ERROR!
> EINVAL: Invalid argument
> at org.apache.hadoop.io.nativeio.NativeIO.sync_file_range(Native 
> Method)
> at 
> org.apache.hadoop.io.nativeio.TestNativeIO.testSyncFileRange(TestNativeIO.java:254)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: [Important] Confirmation related to OpenSsl security issue

[Steve Loughran]

I don't know anything about that, but I do know the apache infrastructure
related changes

-apache.org was vulnerable
-a new *.apache.org certificate is being obtained
-once issued, committers and anyone with JIRA admin access are going to
have to /should change passwords
-JIRA login passwords are best rolled too.
-github was also vulnerable; it's upgraded its cert an its time to update
passwords: https://lastpass.com/heartbleed/?h=github.com



On 11 April 2014 10:27, Vinayakumar B  wrote:

> Hi,
>
> Recently one security issue has been found in OpenSSL which has impacted
> so many customers of different vendors.
>
> http://www.kb.cert.org/vuls/byvendor?searchview&Query=FIELD+Reference=720951&SearchOrder=4
>
> I want to ask, whether is there in impact of this on the Hadoop Release?
>
> Currently Hadoop-pipes are uses openssl-devel packages for building native
> support.
>
> Can someone familiar with Hadoop-pipes can confirm whether is there any
> impact of this security issue on builds of Hadoop built with defective
> openssl?
>
> Regards,
>Vinay
>
>
> 
> This e-mail and attachments contain confidential information from HUAWEI,
> which is intended only for the person or entity whose address is listed
> above. Any use of the information contained herein in any way (including,
> but not limited to, total or partial disclosure, reproduction, or
> dissemination) by persons other than the intended recipient's) is
> prohibited. If you receive this e-mail in error, please notify the sender
> by
> phone or email immediately and delete it!
>
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Created] (HADOOP-10492) Help Commands needs change after deprecation

[Raja Nagendra Kumar (JIRA)]

Raja Nagendra Kumar created HADOOP-10492:


 Summary: Help Commands needs change after deprecation
 Key: HADOOP-10492
 URL: https://issues.apache.org/jira/browse/HADOOP-10492
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Raja Nagendra Kumar


As hadoop fs is deprecated, the help should show usage with HDFS

e.g in the following command it still refers to 
Usage: hadoop fs [generic options]

D:\Apps\java\BI\hadoop\hw\hdp\hadoop-2.2.0.2.0.6.0-0009>hdfs dfs
Usage: hadoop fs [generic options]
[-appendToFile  ... ]
[-cat [-ignoreCrc]  ...]
[-checksum  ...]
[-chgrp [-R] GROUP PATH...]
[-chmod [-R]  PATH...]
[-chown [-R] [OWNER][:[GROUP]] PATH...]
[-copyFromLocal [-f] [-p]  ... ]
[-copyToLocal [-p] [-ignoreCrc] [-crc]  ... ]
[-count [-q]  ...]
[-cp [-f] [-p]  ... ]
[-createSnapshot  []]
[-deleteSnapshot  ]
[-df [-h] [ ...]]
[-du [-s] [-h]  ...]



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HADOOP-10491) Add Collection of Labels to KeyProvider API

[Larry McCay (JIRA)]

Larry McCay created HADOOP-10491:


 Summary: Add Collection of Labels to KeyProvider API
 Key: HADOOP-10491
 URL: https://issues.apache.org/jira/browse/HADOOP-10491
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Reporter: Larry McCay
Assignee: Larry McCay


A set of arbitrary labels would provide opportunity for interesting access 
policy decisions based on things like classification, etc.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Plans of moving towards JDK7 in trunk

[Robert Rati]

Just an FYI, but I'm working on updating that jetty patch for the 
current 2.4.0 release.  The one that is there won't cleanly apply 
because so much has changed since it was posted.  I'll post a new patch 
when it's done.


Rob

On 04/11/2014 04:24 AM, Steve Loughran wrote:

On 10 April 2014 18:12, Eli Collins  wrote:


Let's speak less abstractly, are there particular features or new
dependencies that you would like to contribute (or see contributed) that
require using the Java 1.7 APIs?  Breaking compat in v2 or rolling a v3
release are both non-trivial, not something I suspect we'd want to do just
because it would be, for example, nicer to have a newer version of Jetty.



Oddly enough, rolling the web framework is something I'd like to see in a
v3. the shuffle may be off jetty, but webhdfs isn't. Moving up also lets is
reliably switch to servlet API v3

But.. I think we may be able to increment Jetty more without going to
java7, see https://issues.apache.org/jira/browse/HADOOP-9650 .

Build failed in Jenkins: Hadoop-Common-trunk #1096

[Apache Jenkins Server]

See 

Changes:

[cnauroth] HADOOP-10490. TestMapFile and TestBloomMapFile leak file 
descriptors. Contributed by Chris Nauroth.

[vinodkv] MAPREDUCE-5826. Fixed HistoryServerFileSystemStore to use right 
permissions on Windows for temporary files and thus also fix the test-issue with
TestHistoryServerFileSystemStateStoreService. Contributed by Varun Vasudev.

[vinodkv] YARN-1926. Changed DistributedShell to use appIDs as unique 
identifiers for HDFS paths and thus avoid test failures on Windows. Contributed 
by Varun Vasudev.

[vinodkv] MAPREDUCE-5815. Fixed test-failure of TestMRAppMaster by making 
MRAppMaster gracefully handle empty-queue names. Contributed by Akira Ajisaka.

[cnauroth] HDFS-6231. DFSClient hangs infinitely if using hedged reads and all 
eligible datanodes die. Contributed by Chris Nauroth.

[zjshen] YARN-1924. Made ZKRMStateStore updateApplication(Attempt)StateInternal 
work when Application(Attempt) state hasn't been stored before. Contributed by 
Jian He.

[jianhe] YARN-1903. Set exit code and diagnostics when container is killed at 
NEW/LOCALIZING state. Contributed by Zhijie Shen

[wang] Undo accidental FSNamesystem change introduced in HDFS-6224 commit.

[wang] HDFS-6224. Add a unit test to TestAuditLogger for file permissions 
passed to logAuditEvent. Contributed by Charles Lamb.

[jlowe] MAPREDUCE-5825. Provide diagnostics for reducers killed during ramp 
down. Contributed by Gera Shegalov

[vinodkv] YARN-1914. Fixed resource-download on NodeManagers to skip permission 
verification of public cache files in Windows+local file-system environment. 
Contribued by Varun Vasudev.

[vinayakumarb] HADOOP-10350. BUILDING.txt should mention openssl dependency 
required for hadoop-pipes (Vinayakumar B)

[vinayakumarb] Reverse merged revision(s) 1586425 from hadoop/common/trunk:
HADOOP-10350. BUILDING.txt should mention openssl dependency required for 
hadoop-pipes (Vinayakumar B)


[vinayakumarb] HADOOP-10350. BUILDING.txt should mention openssl dependency 
required for hadoop-pipes (Vinayakumar B)

[zjshen] YARN-1920. Fixed TestFileSystemApplicationHistoryStore failure on 
windows. Contributed by Vinod Kumar Vavilapalli.

[umamahesh] HDFS-5669. Storage#tryLock() should check for null before logging 
successfull message. Contributed by Vinayakumar B

[tucu] HADOOP-10488. TestKeyProviderFactory fails randomly. (tucu)

[vinodkv] MAPREDUCE-5824. Fixed test-failure of TestPipesNonJavaInputFormat in 
Windows. Contributed by Xuan Gong.

[umamahesh] HDFS-6228. comments typo fix for FsDatasetImpl.java Contributed by 
zhaoyunjiong.

--
[...truncated 60552 lines...]
Adding reference: maven.local.repository
[DEBUG] Initialize Maven Ant Tasks
parsing buildfile 
jar:file:/home/jenkins/.m2/repository/org/apache/maven/plugins/maven-antrun-plugin/1.7/maven-antrun-plugin-1.7.jar!/org/apache/maven/ant/tasks/antlib.xml
 with URI = 
jar:file:/home/jenkins/.m2/repository/org/apache/maven/plugins/maven-antrun-plugin/1.7/maven-antrun-plugin-1.7.jar!/org/apache/maven/ant/tasks/antlib.xml
 from a zip file
parsing buildfile 
jar:file:/home/jenkins/.m2/repository/org/apache/ant/ant/1.8.2/ant-1.8.2.jar!/org/apache/tools/ant/antlib.xml
 with URI = 
jar:file:/home/jenkins/.m2/repository/org/apache/ant/ant/1.8.2/ant-1.8.2.jar!/org/apache/tools/ant/antlib.xml
 from a zip file
Class org.apache.maven.ant.tasks.AttachArtifactTask loaded from parent loader 
(parentFirst)
 +Datatype attachartifact org.apache.maven.ant.tasks.AttachArtifactTask
Class org.apache.maven.ant.tasks.DependencyFilesetsTask loaded from parent 
loader (parentFirst)
 +Datatype dependencyfilesets org.apache.maven.ant.tasks.DependencyFilesetsTask
Setting project property: test.build.dir -> 

Setting project property: test.exclude.pattern -> _
Setting project property: hadoop.assemblies.version -> 3.0.0-SNAPSHOT
Setting project property: test.exclude -> _
Setting project property: distMgmtSnapshotsId -> apache.snapshots.https
Setting project property: project.build.sourceEncoding -> UTF-8
Setting project property: java.security.egd -> file:///dev/urandom
Setting project property: distMgmtSnapshotsUrl -> 
https://repository.apache.org/content/repositories/snapshots
Setting project property: distMgmtStagingUrl -> 
https://repository.apache.org/service/local/staging/deploy/maven2
Setting project property: avro.version -> 1.7.4
Setting project property: test.build.data -> 

Setting project property: commons-daemon.version -> 1.0.13
Setting project property: hadoop.common.build.dir -> 

Setting project property: testsThreadCount -

[Important] Confirmation related to OpenSsl security issue

[Vinayakumar B]

Hi,

Recently one security issue has been found in OpenSSL which has impacted so 
many customers of different vendors.
   
http://www.kb.cert.org/vuls/byvendor?searchview&Query=FIELD+Reference=720951&SearchOrder=4

I want to ask, whether is there in impact of this on the Hadoop Release?

Currently Hadoop-pipes are uses openssl-devel packages for building native 
support.

Can someone familiar with Hadoop-pipes can confirm whether is there any impact 
of this security issue on builds of Hadoop built with defective openssl?

Regards,
   Vinay


This e-mail and attachments contain confidential information from HUAWEI,
which is intended only for the person or entity whose address is listed
above. Any use of the information contained herein in any way (including,
but not limited to, total or partial disclosure, reproduction, or
dissemination) by persons other than the intended recipient's) is
prohibited. If you receive this e-mail in error, please notify the sender by
phone or email immediately and delete it!

Re: DISCUSS: use SLF4J APIs in new modules?

[Steve Loughran]

On 10 April 2014 16:38, Karthik Kambatla  wrote:

> +1 to use slf4j. I would actually vote for (1) new modules must-use, (2)
> new classes in existing modules are strongly recommended to use, (3)
> existing classes can switch to. That would take us closer to using slf4j
> everywhere faster.
>


#1  appeals to me.

#2 & #3, we'd have a mixed phase but ultimately it'd be good. Maybe have a
policy for a class switch process of
a) switch the LOG declaration, change the imports
b) clean up all log statements, dropping the ifDebug and moving to {}
strings instead of "text"+ "value

or do both at the same time, one class or package at time.


Having a consistent log scheme across all classes makes copying and moving
code easier, especially copy+paste

I think there's some bits of code that takes a commons-log argument as a
parameter. If these are public they'd need to be retained, and we'd have to
add an SLF4J logger equivalent.

-Steve


>
> On Thu, Apr 10, 2014 at 8:17 AM, Alejandro Abdelnur  >wrote:
>
> > +1 pn slf4j.
> >
> > one thing Jay, the issues with log4j will still be there as log4j will
> > still be under the hood.
> >
> > thx
> >
> > Alejandro
> > (phone typing)
> >
> > > On Apr 10, 2014, at 7:35, Andrew Wang 
> wrote:
> > >
> > > +1 from me, it'd be lovely to get rid of all those isDebugEnabled
> checks.
> > >
> > >
> > >> On Thu, Apr 10, 2014 at 4:13 AM, Jay Vyas 
> wrote:
> > >>
> > >> Slf4j is definetly a great step forward.  Log4j is restrictive for
> > complex
> > >> and multi tenant apps like hadoop.
> > >>
> > >> Also the fact that slf4j doesn't use any magic when binding to its log
> > >> provider makes it way easier to swap out its implementation then tools
> > of
> > >> the past.
> > >>
> >  On Apr 10, 2014, at 2:16 AM, Steve Loughran  >
> > >>> wrote:
> > >>>
> > >>> If we're thinking of future progress, here's a little low-level one:
> > >> adopt
> > >>> SLF4J as the API for logging
> > >>>
> > >>>
> > >>>  1. its the new defacto standard of logging APIs
> > >>>  2. its a lot better than commons-logging with on demand Inline
> string
> > >>>  expansion of varags arguments.
> > >>>  3. we already ship it, as jetty uses it
> > >>>  4. we already depend on it, client-side and server-side in the
> > >>>  hadoop-auth package
> > >>>  5. it lets people log via logback if they want to. That's
> client-side,
> > >>>  even if the server stays on log4j
> > >>>  6. It's way faster than using String.format()
> > >>>
> > >>>
> > >>> The best initial thing about SL4FJ is how it only expands its
> arguments
> > >>> string values if needed
> > >>>
> > >>> LOG.debug("Initialized, principal [{}] from keytab [{}]",
> > principal,
> > >>> keytab);
> > >>>
> > >>> not logging at debug? No need to test first. That alone saves code
> and
> > >>> improves readability.
> > >>>
> > >>> The slf4 expansion code handles null values as well as calling
> > toString()
> > >>> on non-null arguments. Oh and it does arrays too.
> > >>>
> > >>> int array = [1, 2, 3];
> > >>> String undef = null;
> > >>>
> > >>> LOG.info("a = {}, u = {}", array, undef)  -> "a = [1, 2, 3],  u =
> null"
> > >>>
> > >>> Switching to SLF4J from commons-logging is as trivial as changing the
> > >> type
> > >>> of the logger created, but with one logger per class that does get
> > >>> expensive in terms of change. Moving to SLF4J across the board would
> > be a
> > >>> big piece of work -but doable.
> > >>>
> > >>> Rather than push for a dramatic change why not adopt a policy of
> > >> demanding
> > >>> it in new maven subprojects? hadoop-auth shows we permit it, so why
> not
> > >> say
> > >>> "you MUST"?
> > >>>
> > >>> Once people have experience in using it, and are happy, then we could
> > >> think
> > >>> about switching to the new APIs in the core modules. The only
> > troublespot
> > >>> there is where code calls getLogger() on the commons log to get at
> the
> > >>> log4j appender -there's ~3 places in production code that does this,
> > 200+
> > >>> in tests -tests that do it to turn back log levels. Those tests can
> > stay
> > >>> with commons-logging, same for the production uses. Mixing
> > >> commons-logging
> > >>> and slf4j isn't drastic -they both route to log4j or a.n.other back
> > end.
> > >>>
> > >>> -Stevve
> > >>>
> > >>> --
> > >>> CONFIDENTIALITY NOTICE
> > >>> NOTICE: This message is intended for the use of the individual or
> > entity
> > >> to
> > >>> which it is addressed and may contain information that is
> confidential,
> > >>> privileged and exempt from disclosure under applicable law. If the
> > reader
> > >>> of this message is not the intended recipient, you are hereby
> notified
> > >> that
> > >>> any printing, copying, dissemination, distribution, disclosure or
> > >>> forwarding of this communication is strictly prohibited. If you have
> > >>> received this communication in error, please contact the sender
> > >> immediately
> > >>> and delete it from your system. Thank You.
> > >>
> >
>

-- 
CONFI

Re: DISCUSS: use SLF4J APIs in new modules?

[Steve Loughran]

On 10 April 2014 16:17, Alejandro Abdelnur  wrote:

> +1 pn slf4j.
>
> one thing Jay, the issues with log4j will still be there as log4j will
> still be under the hood.
>
>

*may* still be under the hood. Even with commons-logging you can swap out
log4j, and indeed, I did exactly that for hadoop 0.17 at one point.

what we get with slf4j is easier coding

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Plans of moving towards JDK7 in trunk

[Steve Loughran]

On 10 April 2014 18:12, Eli Collins  wrote:

> Let's speak less abstractly, are there particular features or new
> dependencies that you would like to contribute (or see contributed) that
> require using the Java 1.7 APIs?  Breaking compat in v2 or rolling a v3
> release are both non-trivial, not something I suspect we'd want to do just
> because it would be, for example, nicer to have a newer version of Jetty.
>

Oddly enough, rolling the web framework is something I'd like to see in a
v3. the shuffle may be off jetty, but webhdfs isn't. Moving up also lets is
reliably switch to servlet API v3

But.. I think we may be able to increment Jetty more without going to
java7, see https://issues.apache.org/jira/browse/HADOOP-9650 .

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.