Re: Plans of moving towards JDK7 in trunk
On 10 April 2014 18:12, Eli Collins e...@cloudera.com wrote: Let's speak less abstractly, are there particular features or new dependencies that you would like to contribute (or see contributed) that require using the Java 1.7 APIs? Breaking compat in v2 or rolling a v3 release are both non-trivial, not something I suspect we'd want to do just because it would be, for example, nicer to have a newer version of Jetty. Oddly enough, rolling the web framework is something I'd like to see in a v3. the shuffle may be off jetty, but webhdfs isn't. Moving up also lets is reliably switch to servlet API v3 But.. I think we may be able to increment Jetty more without going to java7, see https://issues.apache.org/jira/browse/HADOOP-9650 . -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[Important] Confirmation related to OpenSsl security issue
Hi, Recently one security issue has been found in OpenSSL which has impacted so many customers of different vendors. http://www.kb.cert.org/vuls/byvendor?searchviewQuery=FIELD+Reference=720951SearchOrder=4 I want to ask, whether is there in impact of this on the Hadoop Release? Currently Hadoop-pipes are uses openssl-devel packages for building native support. Can someone familiar with Hadoop-pipes can confirm whether is there any impact of this security issue on builds of Hadoop built with defective openssl? Regards, Vinay This e-mail and attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient's) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!
Build failed in Jenkins: Hadoop-Common-trunk #1096
See https://builds.apache.org/job/Hadoop-Common-trunk/1096/changes Changes: [cnauroth] HADOOP-10490. TestMapFile and TestBloomMapFile leak file descriptors. Contributed by Chris Nauroth. [vinodkv] MAPREDUCE-5826. Fixed HistoryServerFileSystemStore to use right permissions on Windows for temporary files and thus also fix the test-issue with TestHistoryServerFileSystemStateStoreService. Contributed by Varun Vasudev. [vinodkv] YARN-1926. Changed DistributedShell to use appIDs as unique identifiers for HDFS paths and thus avoid test failures on Windows. Contributed by Varun Vasudev. [vinodkv] MAPREDUCE-5815. Fixed test-failure of TestMRAppMaster by making MRAppMaster gracefully handle empty-queue names. Contributed by Akira Ajisaka. [cnauroth] HDFS-6231. DFSClient hangs infinitely if using hedged reads and all eligible datanodes die. Contributed by Chris Nauroth. [zjshen] YARN-1924. Made ZKRMStateStore updateApplication(Attempt)StateInternal work when Application(Attempt) state hasn't been stored before. Contributed by Jian He. [jianhe] YARN-1903. Set exit code and diagnostics when container is killed at NEW/LOCALIZING state. Contributed by Zhijie Shen [wang] Undo accidental FSNamesystem change introduced in HDFS-6224 commit. [wang] HDFS-6224. Add a unit test to TestAuditLogger for file permissions passed to logAuditEvent. Contributed by Charles Lamb. [jlowe] MAPREDUCE-5825. Provide diagnostics for reducers killed during ramp down. Contributed by Gera Shegalov [vinodkv] YARN-1914. Fixed resource-download on NodeManagers to skip permission verification of public cache files in Windows+local file-system environment. Contribued by Varun Vasudev. [vinayakumarb] HADOOP-10350. BUILDING.txt should mention openssl dependency required for hadoop-pipes (Vinayakumar B) [vinayakumarb] Reverse merged revision(s) 1586425 from hadoop/common/trunk: HADOOP-10350. BUILDING.txt should mention openssl dependency required for hadoop-pipes (Vinayakumar B) [vinayakumarb] HADOOP-10350. BUILDING.txt should mention openssl dependency required for hadoop-pipes (Vinayakumar B) [zjshen] YARN-1920. Fixed TestFileSystemApplicationHistoryStore failure on windows. Contributed by Vinod Kumar Vavilapalli. [umamahesh] HDFS-5669. Storage#tryLock() should check for null before logging successfull message. Contributed by Vinayakumar B [tucu] HADOOP-10488. TestKeyProviderFactory fails randomly. (tucu) [vinodkv] MAPREDUCE-5824. Fixed test-failure of TestPipesNonJavaInputFormat in Windows. Contributed by Xuan Gong. [umamahesh] HDFS-6228. comments typo fix for FsDatasetImpl.java Contributed by zhaoyunjiong. -- [...truncated 60552 lines...] Adding reference: maven.local.repository [DEBUG] Initialize Maven Ant Tasks parsing buildfile jar:file:/home/jenkins/.m2/repository/org/apache/maven/plugins/maven-antrun-plugin/1.7/maven-antrun-plugin-1.7.jar!/org/apache/maven/ant/tasks/antlib.xml with URI = jar:file:/home/jenkins/.m2/repository/org/apache/maven/plugins/maven-antrun-plugin/1.7/maven-antrun-plugin-1.7.jar!/org/apache/maven/ant/tasks/antlib.xml from a zip file parsing buildfile jar:file:/home/jenkins/.m2/repository/org/apache/ant/ant/1.8.2/ant-1.8.2.jar!/org/apache/tools/ant/antlib.xml with URI = jar:file:/home/jenkins/.m2/repository/org/apache/ant/ant/1.8.2/ant-1.8.2.jar!/org/apache/tools/ant/antlib.xml from a zip file Class org.apache.maven.ant.tasks.AttachArtifactTask loaded from parent loader (parentFirst) +Datatype attachartifact org.apache.maven.ant.tasks.AttachArtifactTask Class org.apache.maven.ant.tasks.DependencyFilesetsTask loaded from parent loader (parentFirst) +Datatype dependencyfilesets org.apache.maven.ant.tasks.DependencyFilesetsTask Setting project property: test.build.dir - https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/target/test-dir Setting project property: test.exclude.pattern - _ Setting project property: hadoop.assemblies.version - 3.0.0-SNAPSHOT Setting project property: test.exclude - _ Setting project property: distMgmtSnapshotsId - apache.snapshots.https Setting project property: project.build.sourceEncoding - UTF-8 Setting project property: java.security.egd - file:///dev/urandom Setting project property: distMgmtSnapshotsUrl - https://repository.apache.org/content/repositories/snapshots Setting project property: distMgmtStagingUrl - https://repository.apache.org/service/local/staging/deploy/maven2 Setting project property: avro.version - 1.7.4 Setting project property: test.build.data - https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/target/test-dir Setting project property: commons-daemon.version - 1.0.13 Setting project property: hadoop.common.build.dir - https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/../../hadoop-common-project/hadoop-common/target Setting project property: testsThreadCount - 4 Setting project
Re: Plans of moving towards JDK7 in trunk
Just an FYI, but I'm working on updating that jetty patch for the current 2.4.0 release. The one that is there won't cleanly apply because so much has changed since it was posted. I'll post a new patch when it's done. Rob On 04/11/2014 04:24 AM, Steve Loughran wrote: On 10 April 2014 18:12, Eli Collins e...@cloudera.com wrote: Let's speak less abstractly, are there particular features or new dependencies that you would like to contribute (or see contributed) that require using the Java 1.7 APIs? Breaking compat in v2 or rolling a v3 release are both non-trivial, not something I suspect we'd want to do just because it would be, for example, nicer to have a newer version of Jetty. Oddly enough, rolling the web framework is something I'd like to see in a v3. the shuffle may be off jetty, but webhdfs isn't. Moving up also lets is reliably switch to servlet API v3 But.. I think we may be able to increment Jetty more without going to java7, see https://issues.apache.org/jira/browse/HADOOP-9650 .
[jira] [Created] (HADOOP-10491) Add Collection of Labels to KeyProvider API
Larry McCay created HADOOP-10491: Summary: Add Collection of Labels to KeyProvider API Key: HADOOP-10491 URL: https://issues.apache.org/jira/browse/HADOOP-10491 Project: Hadoop Common Issue Type: Improvement Components: security Reporter: Larry McCay Assignee: Larry McCay A set of arbitrary labels would provide opportunity for interesting access policy decisions based on things like classification, etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10492) Help Commands needs change after deprecation
Raja Nagendra Kumar created HADOOP-10492: Summary: Help Commands needs change after deprecation Key: HADOOP-10492 URL: https://issues.apache.org/jira/browse/HADOOP-10492 Project: Hadoop Common Issue Type: Bug Reporter: Raja Nagendra Kumar As hadoop fs is deprecated, the help should show usage with HDFS e.g in the following command it still refers to Usage: hadoop fs [generic options] D:\Apps\java\BI\hadoop\hw\hdp\hadoop-2.2.0.2.0.6.0-0009hdfs dfs Usage: hadoop fs [generic options] [-appendToFile localsrc ... dst] [-cat [-ignoreCrc] src ...] [-checksum src ...] [-chgrp [-R] GROUP PATH...] [-chmod [-R] MODE[,MODE]... | OCTALMODE PATH...] [-chown [-R] [OWNER][:[GROUP]] PATH...] [-copyFromLocal [-f] [-p] localsrc ... dst] [-copyToLocal [-p] [-ignoreCrc] [-crc] src ... localdst] [-count [-q] path ...] [-cp [-f] [-p] src ... dst] [-createSnapshot snapshotDir [snapshotName]] [-deleteSnapshot snapshotDir snapshotName] [-df [-h] [path ...]] [-du [-s] [-h] path ...] -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: [Important] Confirmation related to OpenSsl security issue
I don't know anything about that, but I do know the apache infrastructure related changes -apache.org was vulnerable -a new *.apache.org certificate is being obtained -once issued, committers and anyone with JIRA admin access are going to have to /should change passwords -JIRA login passwords are best rolled too. -github was also vulnerable; it's upgraded its cert an its time to update passwords: https://lastpass.com/heartbleed/?h=github.com On 11 April 2014 10:27, Vinayakumar B vinayakuma...@huawei.com wrote: Hi, Recently one security issue has been found in OpenSSL which has impacted so many customers of different vendors. http://www.kb.cert.org/vuls/byvendor?searchviewQuery=FIELD+Reference=720951SearchOrder=4 I want to ask, whether is there in impact of this on the Hadoop Release? Currently Hadoop-pipes are uses openssl-devel packages for building native support. Can someone familiar with Hadoop-pipes can confirm whether is there any impact of this security issue on builds of Hadoop built with defective openssl? Regards, Vinay This e-mail and attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient's) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it! -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Resolved] (HADOOP-8746) TestNativeIO fails when run with jdk7
[ https://issues.apache.org/jira/browse/HADOOP-8746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai resolved HADOOP-8746. --- Resolution: Not a Problem Target Version/s: 2.0.2-alpha, 0.23.3, 3.0.0 (was: 0.23.3, 3.0.0, 2.0.2-alpha) TestNativeIO fails when run with jdk7 - Key: HADOOP-8746 URL: https://issues.apache.org/jira/browse/HADOOP-8746 Project: Hadoop Common Issue Type: Bug Affects Versions: 0.23.3, 2.0.2-alpha Reporter: Thomas Graves Assignee: Thomas Graves Labels: java7 TestNativeIo fails when run with jdk7. Test set: org.apache.hadoop.io.nativeio.TestNativeIO --- Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.232 sec FAILURE! testSyncFileRange(org.apache.hadoop.io.nativeio.TestNativeIO) Time elapsed: 0.166 sec ERROR! EINVAL: Invalid argument at org.apache.hadoop.io.nativeio.NativeIO.sync_file_range(Native Method) at org.apache.hadoop.io.nativeio.TestNativeIO.testSyncFileRange(TestNativeIO.java:254) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Plans of moving towards JDK7 in trunk
if you mean updating jetty on branch2, we cannot do that. it has to be done in trunk. thx Alejandro (phone typing) On Apr 11, 2014, at 4:46, Robert Rati rr...@redhat.com wrote: Just an FYI, but I'm working on updating that jetty patch for the current 2.4.0 release. The one that is there won't cleanly apply because so much has changed since it was posted. I'll post a new patch when it's done. Rob On 04/11/2014 04:24 AM, Steve Loughran wrote: On 10 April 2014 18:12, Eli Collins e...@cloudera.com wrote: Let's speak less abstractly, are there particular features or new dependencies that you would like to contribute (or see contributed) that require using the Java 1.7 APIs? Breaking compat in v2 or rolling a v3 release are both non-trivial, not something I suspect we'd want to do just because it would be, for example, nicer to have a newer version of Jetty. Oddly enough, rolling the web framework is something I'd like to see in a v3. the shuffle may be off jetty, but webhdfs isn't. Moving up also lets is reliably switch to servlet API v3 But.. I think we may be able to increment Jetty more without going to java7, see https://issues.apache.org/jira/browse/HADOOP-9650 .
[jira] [Resolved] (HADOOP-9219) coverage fixing for org.apache.hadoop.tools.rumen
[ https://issues.apache.org/jira/browse/HADOOP-9219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles resolved HADOOP-9219. - Resolution: Fixed Duping this issue to MAPREDUCE-3860 as per Andrey's comment. coverage fixing for org.apache.hadoop.tools.rumen - Key: HADOOP-9219 URL: https://issues.apache.org/jira/browse/HADOOP-9219 Project: Hadoop Common Issue Type: Test Components: tools Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 Reporter: Aleksey Gorshkov Assignee: Aleksey Gorshkov Attachments: HADOOP-9219-trunk-a.patch, HADOOP-9219-trunk-b.patch, HADOOP-9219-trunk.patch Original Estimate: 168h Remaining Estimate: 168h coverage fixing for org.apache.hadoop.tools.rumen HADOOP-9219-trunk.patch for trunk, brunch-2 and branch-0.23 -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: DISCUSS: use SLF4J APIs in new modules?
On Fri, Apr 11, 2014 at 1:37 AM, Steve Loughran ste...@hortonworks.comwrote:
On 10 April 2014 16:38, Karthik Kambatla ka...@cloudera.com wrote:
+1 to use slf4j. I would actually vote for (1) new modules must-use, (2)
new classes in existing modules are strongly recommended to use, (3)
existing classes can switch to. That would take us closer to using slf4j
everywhere faster.
#1 appeals to me.
#2 #3, we'd have a mixed phase but ultimately it'd be good. Maybe have a
policy for a class switch process of
a) switch the LOG declaration, change the imports
b) clean up all log statements, dropping the ifDebug and moving to {}
strings instead of text+ value
I feel more the requirements we add to switching, the less likely people
will make the time for it. I think it is reasonable to switch LOG
declaration and drop ifDebug. Changing all log messages to use {} instead
of +could be really time-taking especially for classes with tons
of log messages. On the other hand, asking people to do that if they are
editing an existing log message anyway, it might fly.
or do both at the same time, one class or package at time.
Having a consistent log scheme across all classes makes copying and moving
code easier, especially copy+paste
I think there's some bits of code that takes a commons-log argument as a
parameter. If these are public they'd need to be retained, and we'd have to
add an SLF4J logger equivalent.
-Steve
On Thu, Apr 10, 2014 at 8:17 AM, Alejandro Abdelnur t...@cloudera.com
wrote:
+1 pn slf4j.
one thing Jay, the issues with log4j will still be there as log4j will
still be under the hood.
thx
Alejandro
(phone typing)
On Apr 10, 2014, at 7:35, Andrew Wang andrew.w...@cloudera.com
wrote:
+1 from me, it'd be lovely to get rid of all those isDebugEnabled
checks.
On Thu, Apr 10, 2014 at 4:13 AM, Jay Vyas jayunit...@gmail.com
wrote:
Slf4j is definetly a great step forward. Log4j is restrictive for
complex
and multi tenant apps like hadoop.
Also the fact that slf4j doesn't use any magic when binding to its
log
provider makes it way easier to swap out its implementation then
tools
of
the past.
On Apr 10, 2014, at 2:16 AM, Steve Loughran
ste...@hortonworks.com
wrote:
If we're thinking of future progress, here's a little low-level
one:
adopt
SLF4J as the API for logging
1. its the new defacto standard of logging APIs
2. its a lot better than commons-logging with on demand Inline
string
expansion of varags arguments.
3. we already ship it, as jetty uses it
4. we already depend on it, client-side and server-side in the
hadoop-auth package
5. it lets people log via logback if they want to. That's
client-side,
even if the server stays on log4j
6. It's way faster than using String.format()
The best initial thing about SL4FJ is how it only expands its
arguments
string values if needed
LOG.debug(Initialized, principal [{}] from keytab [{}],
principal,
keytab);
not logging at debug? No need to test first. That alone saves code
and
improves readability.
The slf4 expansion code handles null values as well as calling
toString()
on non-null arguments. Oh and it does arrays too.
int array = [1, 2, 3];
String undef = null;
LOG.info(a = {}, u = {}, array, undef) - a = [1, 2, 3], u =
null
Switching to SLF4J from commons-logging is as trivial as changing
the
type
of the logger created, but with one logger per class that does get
expensive in terms of change. Moving to SLF4J across the board
would
be a
big piece of work -but doable.
Rather than push for a dramatic change why not adopt a policy of
demanding
it in new maven subprojects? hadoop-auth shows we permit it, so why
not
say
you MUST?
Once people have experience in using it, and are happy, then we
could
think
about switching to the new APIs in the core modules. The only
troublespot
there is where code calls getLogger() on the commons log to get at
the
log4j appender -there's ~3 places in production code that does
this,
200+
in tests -tests that do it to turn back log levels. Those tests can
stay
with commons-logging, same for the production uses. Mixing
commons-logging
and slf4j isn't drastic -they both route to log4j or a.n.other back
end.
-Stevve
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or
entity
to
which it is addressed and may contain information that is
confidential,
privileged and exempt from disclosure under applicable law. If the
reader
of this message is not the intended recipient, you are hereby
notified
that
any printing, copying,
Re: DISCUSS: use SLF4J APIs in new modules?
In addition, we need to consider limiting any printing messages to the
stdout in some cases. This can impact other running some apache products in
silent mode such as hive in 'hive -S' option.
Thanks
-Rahman
On Fri, Apr 11, 2014 at 10:06 AM, Karthik Kambatla ka...@cloudera.comwrote:
On Fri, Apr 11, 2014 at 1:37 AM, Steve Loughran ste...@hortonworks.com
wrote:
On 10 April 2014 16:38, Karthik Kambatla ka...@cloudera.com wrote:
+1 to use slf4j. I would actually vote for (1) new modules must-use,
(2)
new classes in existing modules are strongly recommended to use, (3)
existing classes can switch to. That would take us closer to using
slf4j
everywhere faster.
#1 appeals to me.
#2 #3, we'd have a mixed phase but ultimately it'd be good. Maybe have
a
policy for a class switch process of
a) switch the LOG declaration, change the imports
b) clean up all log statements, dropping the ifDebug and moving to {}
strings instead of text+ value
I feel more the requirements we add to switching, the less likely people
will make the time for it. I think it is reasonable to switch LOG
declaration and drop ifDebug. Changing all log messages to use {} instead
of +could be really time-taking especially for classes with tons
of log messages. On the other hand, asking people to do that if they are
editing an existing log message anyway, it might fly.
or do both at the same time, one class or package at time.
Having a consistent log scheme across all classes makes copying and
moving
code easier, especially copy+paste
I think there's some bits of code that takes a commons-log argument as a
parameter. If these are public they'd need to be retained, and we'd have
to
add an SLF4J logger equivalent.
-Steve
On Thu, Apr 10, 2014 at 8:17 AM, Alejandro Abdelnur t...@cloudera.com
wrote:
+1 pn slf4j.
one thing Jay, the issues with log4j will still be there as log4j
will
still be under the hood.
thx
Alejandro
(phone typing)
On Apr 10, 2014, at 7:35, Andrew Wang andrew.w...@cloudera.com
wrote:
+1 from me, it'd be lovely to get rid of all those isDebugEnabled
checks.
On Thu, Apr 10, 2014 at 4:13 AM, Jay Vyas jayunit...@gmail.com
wrote:
Slf4j is definetly a great step forward. Log4j is restrictive for
complex
and multi tenant apps like hadoop.
Also the fact that slf4j doesn't use any magic when binding to its
log
provider makes it way easier to swap out its implementation then
tools
of
the past.
On Apr 10, 2014, at 2:16 AM, Steve Loughran
ste...@hortonworks.com
wrote:
If we're thinking of future progress, here's a little low-level
one:
adopt
SLF4J as the API for logging
1. its the new defacto standard of logging APIs
2. its a lot better than commons-logging with on demand Inline
string
expansion of varags arguments.
3. we already ship it, as jetty uses it
4. we already depend on it, client-side and server-side in the
hadoop-auth package
5. it lets people log via logback if they want to. That's
client-side,
even if the server stays on log4j
6. It's way faster than using String.format()
The best initial thing about SL4FJ is how it only expands its
arguments
string values if needed
LOG.debug(Initialized, principal [{}] from keytab [{}],
principal,
keytab);
not logging at debug? No need to test first. That alone saves
code
and
improves readability.
The slf4 expansion code handles null values as well as calling
toString()
on non-null arguments. Oh and it does arrays too.
int array = [1, 2, 3];
String undef = null;
LOG.info(a = {}, u = {}, array, undef) - a = [1, 2, 3], u =
null
Switching to SLF4J from commons-logging is as trivial as changing
the
type
of the logger created, but with one logger per class that does
get
expensive in terms of change. Moving to SLF4J across the board
would
be a
big piece of work -but doable.
Rather than push for a dramatic change why not adopt a policy of
demanding
it in new maven subprojects? hadoop-auth shows we permit it, so
why
not
say
you MUST?
Once people have experience in using it, and are happy, then we
could
think
about switching to the new APIs in the core modules. The only
troublespot
there is where code calls getLogger() on the commons log to get
at
the
log4j appender -there's ~3 places in production code that does
this,
200+
in tests -tests that do it to turn back log levels. Those tests
can
stay
with commons-logging, same for the production uses. Mixing
commons-logging
and slf4j isn't drastic -they both route to
Re: DISCUSS: use SLF4J APIs in new modules?
if you dont convert mgs to templates the dont remove the guards, else you
create str mgs obj even when not logging.
thx
Alejandro
(phone typing)
On Apr 11, 2014, at 10:06, Karthik Kambatla ka...@cloudera.com wrote:
On Fri, Apr 11, 2014 at 1:37 AM, Steve Loughran ste...@hortonworks.comwrote:
On 10 April 2014 16:38, Karthik Kambatla ka...@cloudera.com wrote:
+1 to use slf4j. I would actually vote for (1) new modules must-use, (2)
new classes in existing modules are strongly recommended to use, (3)
existing classes can switch to. That would take us closer to using slf4j
everywhere faster.
#1 appeals to me.
#2 #3, we'd have a mixed phase but ultimately it'd be good. Maybe have a
policy for a class switch process of
a) switch the LOG declaration, change the imports
b) clean up all log statements, dropping the ifDebug and moving to {}
strings instead of text+ value
I feel more the requirements we add to switching, the less likely people
will make the time for it. I think it is reasonable to switch LOG
declaration and drop ifDebug. Changing all log messages to use {} instead
of +could be really time-taking especially for classes with tons
of log messages. On the other hand, asking people to do that if they are
editing an existing log message anyway, it might fly.
or do both at the same time, one class or package at time.
Having a consistent log scheme across all classes makes copying and moving
code easier, especially copy+paste
I think there's some bits of code that takes a commons-log argument as a
parameter. If these are public they'd need to be retained, and we'd have to
add an SLF4J logger equivalent.
-Steve
On Thu, Apr 10, 2014 at 8:17 AM, Alejandro Abdelnur t...@cloudera.com
wrote:
+1 pn slf4j.
one thing Jay, the issues with log4j will still be there as log4j will
still be under the hood.
thx
Alejandro
(phone typing)
On Apr 10, 2014, at 7:35, Andrew Wang andrew.w...@cloudera.com
wrote:
+1 from me, it'd be lovely to get rid of all those isDebugEnabled
checks.
On Thu, Apr 10, 2014 at 4:13 AM, Jay Vyas jayunit...@gmail.com
wrote:
Slf4j is definetly a great step forward. Log4j is restrictive for
complex
and multi tenant apps like hadoop.
Also the fact that slf4j doesn't use any magic when binding to its
log
provider makes it way easier to swap out its implementation then
tools
of
the past.
On Apr 10, 2014, at 2:16 AM, Steve Loughran
ste...@hortonworks.com
wrote:
If we're thinking of future progress, here's a little low-level
one:
adopt
SLF4J as the API for logging
1. its the new defacto standard of logging APIs
2. its a lot better than commons-logging with on demand Inline
string
expansion of varags arguments.
3. we already ship it, as jetty uses it
4. we already depend on it, client-side and server-side in the
hadoop-auth package
5. it lets people log via logback if they want to. That's
client-side,
even if the server stays on log4j
6. It's way faster than using String.format()
The best initial thing about SL4FJ is how it only expands its
arguments
string values if needed
LOG.debug(Initialized, principal [{}] from keytab [{}],
principal,
keytab);
not logging at debug? No need to test first. That alone saves code
and
improves readability.
The slf4 expansion code handles null values as well as calling
toString()
on non-null arguments. Oh and it does arrays too.
int array = [1, 2, 3];
String undef = null;
LOG.info(a = {}, u = {}, array, undef) - a = [1, 2, 3], u =
null
Switching to SLF4J from commons-logging is as trivial as changing
the
type
of the logger created, but with one logger per class that does get
expensive in terms of change. Moving to SLF4J across the board
would
be a
big piece of work -but doable.
Rather than push for a dramatic change why not adopt a policy of
demanding
it in new maven subprojects? hadoop-auth shows we permit it, so why
not
say
you MUST?
Once people have experience in using it, and are happy, then we
could
think
about switching to the new APIs in the core modules. The only
troublespot
there is where code calls getLogger() on the commons log to get at
the
log4j appender -there's ~3 places in production code that does
this,
200+
in tests -tests that do it to turn back log levels. Those tests can
stay
with commons-logging, same for the production uses. Mixing
commons-logging
and slf4j isn't drastic -they both route to log4j or a.n.other back
end.
-Stevve
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or
entity
to
which it is addressed and may contain information that is
confidential,
privileged and exempt from disclosure under applicable law. If the
reader
of this message is not the intended recipient, you are hereby
notified
that
any printing, copying, dissemination, distribution, disclosure or
forwarding of
[jira] [Resolved] (HADOOP-10431) Change visibility of KeyStore.Options getter methods to public
[
https://issues.apache.org/jira/browse/HADOOP-10431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur resolved HADOOP-10431.
-
Resolution: Fixed
Fix Version/s: 3.0.0
Hadoop Flags: Reviewed
committed to trunk.
Change visibility of KeyStore.Options getter methods to public
--
Key: HADOOP-10431
URL: https://issues.apache.org/jira/browse/HADOOP-10431
Project: Hadoop Common
Issue Type: Improvement
Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Fix For: 3.0.0
Attachments: HADOOP-10431.patch, HADOOP-10431.patch,
HADOOP-10431.patch
Making Options getter methods public will enable {{KeyProvider}}
implementations to use those classes.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Re: [Important] Confirmation related to OpenSsl security issue
I took a quick glance at the build output, and I don't think openssl is getting linked statically into libhadooppipes.a. I see the following lines: Linking CXX static library libhadooppipes.a /usr/bin/cmake -P CMakeFiles/hadooppipes.dir/cmake_clean_target.cmake /usr/bin/cmake -E cmake_link_script CMakeFiles/hadooppipes.dir/link.txt --verbose=1 /usr/bin/ar cr libhadooppipes.a CMakeFiles/hadooppipes.dir/main/native/pipes/impl/HadoopPipes.cc.o /usr/bin/ranlib libhadooppipes.a later on there are lines like this: /usr/bin/c++-g -Wall -O2 -D_REENTRANT -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 CMakeFiles/pipes-sort.dir/main/native/examples/impl/sort.cc.o -o examples/pipes-sort -rdynamic libhadooppipes.a libhadooputils.a -lssl -lcrypto -lpthread So when using libhadooppipes.a, you must supply your own copy of libssl.so. If you supply a vulnerable copy, you will be vulnerable. If you supply a non-vulnerable copy, you won't be. So I don't think there is an impact on our build (unless I missed something here). Just to make sure, it would be good if someone who uses libhadooppipes.a to look at one of the versions in our release tarball and verify that it works with the fixed openssl. Colin On Fri, Apr 11, 2014 at 2:27 AM, Vinayakumar B vinayakuma...@huawei.com wrote: Hi, Recently one security issue has been found in OpenSSL which has impacted so many customers of different vendors. http://www.kb.cert.org/vuls/byvendor?searchviewQuery=FIELD+Reference=720951SearchOrder=4 I want to ask, whether is there in impact of this on the Hadoop Release? Currently Hadoop-pipes are uses openssl-devel packages for building native support. Can someone familiar with Hadoop-pipes can confirm whether is there any impact of this security issue on builds of Hadoop built with defective openssl? Regards, Vinay This e-mail and attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient's) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!
Re: Plans of moving towards JDK7 in trunk
I don't mean to be dense, but can you expand on why jetty 8 can't go into branch2? What is the concern? Rob On 04/11/2014 10:55 AM, Alejandro Abdelnur wrote: if you mean updating jetty on branch2, we cannot do that. it has to be done in trunk. thx Alejandro (phone typing) On Apr 11, 2014, at 4:46, Robert Rati rr...@redhat.com wrote: Just an FYI, but I'm working on updating that jetty patch for the current 2.4.0 release. The one that is there won't cleanly apply because so much has changed since it was posted. I'll post a new patch when it's done. Rob On 04/11/2014 04:24 AM, Steve Loughran wrote: On 10 April 2014 18:12, Eli Collins e...@cloudera.com wrote: Let's speak less abstractly, are there particular features or new dependencies that you would like to contribute (or see contributed) that require using the Java 1.7 APIs? Breaking compat in v2 or rolling a v3 release are both non-trivial, not something I suspect we'd want to do just because it would be, for example, nicer to have a newer version of Jetty. Oddly enough, rolling the web framework is something I'd like to see in a v3. the shuffle may be off jetty, but webhdfs isn't. Moving up also lets is reliably switch to servlet API v3 But.. I think we may be able to increment Jetty more without going to java7, see https://issues.apache.org/jira/browse/HADOOP-9650 .
Re: Plans of moving towards JDK7 in trunk
because it is exposed as classpath dependency, changing it breaks backward compatibility. On Fri, Apr 11, 2014 at 1:02 PM, Steve Loughran ste...@hortonworks.comwrote: Jetty's a big change, it's fairly intimately involved in bits of the code but: it's a source of grief, currently webhdfs is an example https://issues.apache.org/jira/browse/HDFS-6221 all YARN apps seem to get hosted by it too On 11 April 2014 20:56, Robert Rati rr...@redhat.com wrote: I don't mean to be dense, but can you expand on why jetty 8 can't go into branch2? What is the concern? Rob On 04/11/2014 10:55 AM, Alejandro Abdelnur wrote: if you mean updating jetty on branch2, we cannot do that. it has to be done in trunk. thx Alejandro (phone typing) On Apr 11, 2014, at 4:46, Robert Rati rr...@redhat.com wrote: Just an FYI, but I'm working on updating that jetty patch for the current 2.4.0 release. The one that is there won't cleanly apply because so much has changed since it was posted. I'll post a new patch when it's done. Rob On 04/11/2014 04:24 AM, Steve Loughran wrote: On 10 April 2014 18:12, Eli Collins e...@cloudera.com wrote: Let's speak less abstractly, are there particular features or new dependencies that you would like to contribute (or see contributed) that require using the Java 1.7 APIs? Breaking compat in v2 or rolling a v3 release are both non-trivial, not something I suspect we'd want to do just because it would be, for example, nicer to have a newer version of Jetty. Oddly enough, rolling the web framework is something I'd like to see in a v3. the shuffle may be off jetty, but webhdfs isn't. Moving up also lets is reliably switch to servlet API v3 But.. I think we may be able to increment Jetty more without going to java7, see https://issues.apache.org/jira/browse/HADOOP-9650 . -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Alejandro
Re: Plans of moving towards JDK7 in trunk
that doesn't actually stop is from switching in our own code to alternate web servers, only that jetty can remain a published artifact in the hadoop/lib dir On 11 April 2014 21:16, Alejandro Abdelnur t...@cloudera.com wrote: because it is exposed as classpath dependency, changing it breaks backward compatibility. On Fri, Apr 11, 2014 at 1:02 PM, Steve Loughran ste...@hortonworks.com wrote: Jetty's a big change, it's fairly intimately involved in bits of the code but: it's a source of grief, currently webhdfs is an example https://issues.apache.org/jira/browse/HDFS-6221 all YARN apps seem to get hosted by it too On 11 April 2014 20:56, Robert Rati rr...@redhat.com wrote: I don't mean to be dense, but can you expand on why jetty 8 can't go into branch2? What is the concern? Rob On 04/11/2014 10:55 AM, Alejandro Abdelnur wrote: if you mean updating jetty on branch2, we cannot do that. it has to be done in trunk. thx Alejandro (phone typing) On Apr 11, 2014, at 4:46, Robert Rati rr...@redhat.com wrote: Just an FYI, but I'm working on updating that jetty patch for the current 2.4.0 release. The one that is there won't cleanly apply because so much has changed since it was posted. I'll post a new patch when it's done. Rob On 04/11/2014 04:24 AM, Steve Loughran wrote: On 10 April 2014 18:12, Eli Collins e...@cloudera.com wrote: Let's speak less abstractly, are there particular features or new dependencies that you would like to contribute (or see contributed) that require using the Java 1.7 APIs? Breaking compat in v2 or rolling a v3 release are both non-trivial, not something I suspect we'd want to do just because it would be, for example, nicer to have a newer version of Jetty. Oddly enough, rolling the web framework is something I'd like to see in a v3. the shuffle may be off jetty, but webhdfs isn't. Moving up also lets is reliably switch to servlet API v3 But.. I think we may be able to increment Jetty more without going to java7, see https://issues.apache.org/jira/browse/HADOOP-9650 . -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Alejandro -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Plans of moving towards JDK7 in trunk
newer jetties have non backwards compat APIs, we would break any user app using jetty (consumed via hadoop classpath) On Fri, Apr 11, 2014 at 2:16 PM, Steve Loughran ste...@hortonworks.comwrote: that doesn't actually stop is from switching in our own code to alternate web servers, only that jetty can remain a published artifact in the hadoop/lib dir On 11 April 2014 21:16, Alejandro Abdelnur t...@cloudera.com wrote: because it is exposed as classpath dependency, changing it breaks backward compatibility. On Fri, Apr 11, 2014 at 1:02 PM, Steve Loughran ste...@hortonworks.com wrote: Jetty's a big change, it's fairly intimately involved in bits of the code but: it's a source of grief, currently webhdfs is an example https://issues.apache.org/jira/browse/HDFS-6221 all YARN apps seem to get hosted by it too On 11 April 2014 20:56, Robert Rati rr...@redhat.com wrote: I don't mean to be dense, but can you expand on why jetty 8 can't go into branch2? What is the concern? Rob On 04/11/2014 10:55 AM, Alejandro Abdelnur wrote: if you mean updating jetty on branch2, we cannot do that. it has to be done in trunk. thx Alejandro (phone typing) On Apr 11, 2014, at 4:46, Robert Rati rr...@redhat.com wrote: Just an FYI, but I'm working on updating that jetty patch for the current 2.4.0 release. The one that is there won't cleanly apply because so much has changed since it was posted. I'll post a new patch when it's done. Rob On 04/11/2014 04:24 AM, Steve Loughran wrote: On 10 April 2014 18:12, Eli Collins e...@cloudera.com wrote: Let's speak less abstractly, are there particular features or new dependencies that you would like to contribute (or see contributed) that require using the Java 1.7 APIs? Breaking compat in v2 or rolling a v3 release are both non-trivial, not something I suspect we'd want to do just because it would be, for example, nicer to have a newer version of Jetty. Oddly enough, rolling the web framework is something I'd like to see in a v3. the shuffle may be off jetty, but webhdfs isn't. Moving up also lets is reliably switch to servlet API v3 But.. I think we may be able to increment Jetty more without going to java7, see https://issues.apache.org/jira/browse/HADOOP-9650 . -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Alejandro -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Alejandro
