[jira] [Updated] (HADOOP-9776) HarFileSystem.listStatus() returns "har://-localhost:/..." if port number is empty
[ https://issues.apache.org/jira/browse/HADOOP-9776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated HADOOP-9776: --- Status: Patch Available (was: Open) > HarFileSystem.listStatus() returns "har://-localhost:/..." if port > number is empty > -- > > Key: HADOOP-9776 > URL: https://issues.apache.org/jira/browse/HADOOP-9776 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: shanyu zhao > Attachments: HADOOP-9776.patch > > > If the given har URI is "har://-localhost/usr/my.har/a", the result > of HarFileSystem.listStatus() will have a ":" appended after localhost, like > this: "har://-localhost:/usr/my.har/a". it should return > "har://-localhost/usr/my.bar/a" instead. > This creates problem when running a hive unit test TestCliDriver > (archive_excludeHadoop20.q), generating the following error: > java.io.IOException: cannot find dir = > har://pfile-localhost:/GitHub/hive-monarch/build/ql/test/data/warehouse/tstsrcpart/ds=2008-04-08/hr=12/data.har/00_0 > in pathToPartitionInfo: > [pfile:/GitHub/hive-monarch/build/ql/test/data/warehouse/tstsrcpart/ds=2008-04-08/hr=11, > > har://pfile-localhost/GitHub/hive-monarch/build/ql/test/data/warehouse/tstsrcpart/ds=2008-04-08/hr=12/data.har] > [junit] at > org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:298) > [junit] at > org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:260) > [junit] at > org.apache.hadoop.hive.ql.io.CombineHiveInputFormat$CombineHiveInputSplit.(CombineHiveInputFormat.java:104) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9698) RPCv9 client must honor server's SASL negotiate response
[ https://issues.apache.org/jira/browse/HADOOP-9698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13720498#comment-13720498 ] Sanjay Radia commented on HADOOP-9698: -- Please change comment on Server#authorizeConnection // If auth method is DIGEST, the token was obtained by the Change DIGEST to TOKEN since we these patches have changed the terminilogy. > RPCv9 client must honor server's SASL negotiate response > > > Key: HADOOP-9698 > URL: https://issues.apache.org/jira/browse/HADOOP-9698 > Project: Hadoop Common > Issue Type: Sub-task > Components: ipc >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Blocker > Attachments: HADOOP-9698.patch > > > As of HADOOP-9421, a RPCv9 server will advertise its authentication methods. > This is meant to support features such as IP failover, better token > selection, and interoperability in a heterogenous security environment. > Currently the client ignores the negotiate response and just blindly attempts > to authenticate instead of choosing a mutually agreeable auth method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9698) RPCv9 client must honor server's SASL negotiate response
[ https://issues.apache.org/jira/browse/HADOOP-9698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13720501#comment-13720501 ] Sanjay Radia commented on HADOOP-9698: -- +1 most of my comments can be done in another Jira if desired. BTW the patch needs to be updated, it fails. > RPCv9 client must honor server's SASL negotiate response > > > Key: HADOOP-9698 > URL: https://issues.apache.org/jira/browse/HADOOP-9698 > Project: Hadoop Common > Issue Type: Sub-task > Components: ipc >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Blocker > Attachments: HADOOP-9698.patch > > > As of HADOOP-9421, a RPCv9 server will advertise its authentication methods. > This is meant to support features such as IP failover, better token > selection, and interoperability in a heterogenous security environment. > Currently the client ignores the negotiate response and just blindly attempts > to authenticate instead of choosing a mutually agreeable auth method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9776) HarFileSystem.listStatus() returns "har://-localhost:/..." if port number is empty
[
https://issues.apache.org/jira/browse/HADOOP-9776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13720522#comment-13720522
]
Hadoop QA commented on HADOOP-9776:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12594311/HADOOP-9776.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2853//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2853//console
This message is automatically generated.
> HarFileSystem.listStatus() returns "har://-localhost:/..." if port
> number is empty
> --
>
> Key: HADOOP-9776
> URL: https://issues.apache.org/jira/browse/HADOOP-9776
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: shanyu zhao
> Attachments: HADOOP-9776.patch
>
>
> If the given har URI is "har://-localhost/usr/my.har/a", the result
> of HarFileSystem.listStatus() will have a ":" appended after localhost, like
> this: "har://-localhost:/usr/my.har/a". it should return
> "har://-localhost/usr/my.bar/a" instead.
> This creates problem when running a hive unit test TestCliDriver
> (archive_excludeHadoop20.q), generating the following error:
> java.io.IOException: cannot find dir =
> har://pfile-localhost:/GitHub/hive-monarch/build/ql/test/data/warehouse/tstsrcpart/ds=2008-04-08/hr=12/data.har/00_0
> in pathToPartitionInfo:
> [pfile:/GitHub/hive-monarch/build/ql/test/data/warehouse/tstsrcpart/ds=2008-04-08/hr=11,
>
> har://pfile-localhost/GitHub/hive-monarch/build/ql/test/data/warehouse/tstsrcpart/ds=2008-04-08/hr=12/data.har]
> [junit] at
> org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:298)
> [junit] at
> org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:260)
> [junit] at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat$CombineHiveInputSplit.(CombineHiveInputFormat.java:104)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9777) RPM should not claim ownership of paths owned by the platform
Stevo Slavic created HADOOP-9777: Summary: RPM should not claim ownership of paths owned by the platform Key: HADOOP-9777 URL: https://issues.apache.org/jira/browse/HADOOP-9777 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 1.1.2 Environment: Fedora 19 x64 Reporter: Stevo Slavic Installing Apache Hadoop rpm ( hadoop-1.1.2-1.x86_64.rpm ) on Fedora 19 x64 fails with: [root@laptop hadoop]# rpm -i /home/sslavic/Downloads/hadoop-1.1.2-1.x86_64.rpm file /usr/bin from install of hadoop-1.1.2-1.x86_64 conflicts with file from package filesystem-3.2-12.fc19.x86_64 file /usr/lib from install of hadoop-1.1.2-1.x86_64 conflicts with file from package filesystem-3.2-12.fc19.x86_64 file /usr/lib64 from install of hadoop-1.1.2-1.x86_64 conflicts with file from package filesystem-3.2-12.fc19.x86_64 file /usr/sbin from install of hadoop-1.1.2-1.x86_64 conflicts with file from package filesystem-3.2-12.fc19.x86_64 Same issue occurs if one tries to install as non-root user: [sslavic@laptop ~]$ sudo rpm -i Downloads/hadoop-1.1.2-1.x86_64.rpm file /usr/bin from install of hadoop-1.1.2-1.x86_64 conflicts with file from package filesystem-3.2-12.fc19.x86_64 file /usr/lib from install of hadoop-1.1.2-1.x86_64 conflicts with file from package filesystem-3.2-12.fc19.x86_64 file /usr/lib64 from install of hadoop-1.1.2-1.x86_64 conflicts with file from package filesystem-3.2-12.fc19.x86_64 file /usr/sbin from install of hadoop-1.1.2-1.x86_64 conflicts with file from package filesystem-3.2-12.fc19.x86_64 It seems these 4 directories in Hadoop rpm have wrong permissions (+w for owner). This is violation of packaging rules. Hadoop rpm spec and/or build scripts need to be fixed, so that rpm on installation doesn't try to claim ownership of paths owned by the platform, in this case, filesystem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9756) Additional cleanup RPC code
[ https://issues.apache.org/jira/browse/HADOOP-9756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HADOOP-9756: --- Hadoop Flags: Reviewed +1 patch looks good. > Additional cleanup RPC code > --- > > Key: HADOOP-9756 > URL: https://issues.apache.org/jira/browse/HADOOP-9756 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Junping Du >Assignee: Junping Du >Priority: Minor > Attachments: HADOOP-9756.patch, HADOOP-9756-v2.patch > > > HADOOP-9754 already did good job to address most of work for cleanup RPC > code. Here is some additional work, include: > - Remove some unused deprecated code. > - Narrow "throws Exception" to throw some specific exception and remove some > unnecessary exceptions > - Fix a generic warning and correct spell issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9756) Additional cleanup RPC code
[ https://issues.apache.org/jira/browse/HADOOP-9756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13720668#comment-13720668 ] Tsz Wo (Nicholas), SZE commented on HADOOP-9756: Hi Junping, I committed the patch to trunk and tried to merge it to branch-2 but it failed since there were some compilation errors. Could you take a look and post a patch for branch-2? > Additional cleanup RPC code > --- > > Key: HADOOP-9756 > URL: https://issues.apache.org/jira/browse/HADOOP-9756 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Junping Du >Assignee: Junping Du >Priority: Minor > Attachments: HADOOP-9756.patch, HADOOP-9756-v2.patch > > > HADOOP-9754 already did good job to address most of work for cleanup RPC > code. Here is some additional work, include: > - Remove some unused deprecated code. > - Narrow "throws Exception" to throw some specific exception and remove some > unnecessary exceptions > - Fix a generic warning and correct spell issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9756) Additional cleanup RPC code
[ https://issues.apache.org/jira/browse/HADOOP-9756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13720671#comment-13720671 ] Tsz Wo (Nicholas), SZE commented on HADOOP-9756: > ... btw, shall we mark this JIRA with incompatible? We shall not since RPC is annotated as LimitedPrivate. It is not a public API. > Additional cleanup RPC code > --- > > Key: HADOOP-9756 > URL: https://issues.apache.org/jira/browse/HADOOP-9756 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Junping Du >Assignee: Junping Du >Priority: Minor > Attachments: HADOOP-9756.patch, HADOOP-9756-v2.patch > > > HADOOP-9754 already did good job to address most of work for cleanup RPC > code. Here is some additional work, include: > - Remove some unused deprecated code. > - Narrow "throws Exception" to throw some specific exception and remove some > unnecessary exceptions > - Fix a generic warning and correct spell issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9756) Additional cleanup RPC code
[ https://issues.apache.org/jira/browse/HADOOP-9756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13720688#comment-13720688 ] Junping Du commented on HADOOP-9756: bq. Could you take a look and post a patch for branch-2? Sure. Will upload a patch soon. bq. We shall not since RPC is annotated as LimitedPrivate. It is not a public API. Ok. As saw discussion in HADOOP-8754, todd said HAMA use RPC.getServer, so asking here. > Additional cleanup RPC code > --- > > Key: HADOOP-9756 > URL: https://issues.apache.org/jira/browse/HADOOP-9756 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Junping Du >Assignee: Junping Du >Priority: Minor > Attachments: HADOOP-9756.patch, HADOOP-9756-v2.patch > > > HADOOP-9754 already did good job to address most of work for cleanup RPC > code. Here is some additional work, include: > - Remove some unused deprecated code. > - Narrow "throws Exception" to throw some specific exception and remove some > unnecessary exceptions > - Fix a generic warning and correct spell issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-9160) Adopt Jolokia as the JMX HTTP/JSON bridge.
[ https://issues.apache.org/jira/browse/HADOOP-9160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du reassigned HADOOP-9160: -- Assignee: Junping Du > Adopt Jolokia as the JMX HTTP/JSON bridge. > -- > > Key: HADOOP-9160 > URL: https://issues.apache.org/jira/browse/HADOOP-9160 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Luke Lu >Assignee: Junping Du > Attachments: hadoop-9160-demo-branch-1.txt > > > The current JMX HTTP bridge has served its purpose, while a more complete > solution: Jolokia (formerly Jmx4Perl) has been developed/matured over the > years. > Jolokia provides comprehensive JMX features over HTTP/JSON including search > and list of JMX attributes and operations metadata, which helps to support > inter framework/platform compatibility. It has first class language bindings > for Perl, Python, Javascript, Java. > It's trivial (see demo patch) to incorporate Jolokia servlet into Hadoop HTTP > servers and use the same security mechanisms. > Adopting Jolokia will substantially improve the manageability of Hadoop and > its ecosystem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9160) Adopt Jolokia as the JMX HTTP/JSON bridge.
[ https://issues.apache.org/jira/browse/HADOOP-9160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13720730#comment-13720730 ] Junping Du commented on HADOOP-9160: Jolokia sounds good to me. If nobody against this, I will go ahead to work on this and deliver a patch later. > Adopt Jolokia as the JMX HTTP/JSON bridge. > -- > > Key: HADOOP-9160 > URL: https://issues.apache.org/jira/browse/HADOOP-9160 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Luke Lu > Attachments: hadoop-9160-demo-branch-1.txt > > > The current JMX HTTP bridge has served its purpose, while a more complete > solution: Jolokia (formerly Jmx4Perl) has been developed/matured over the > years. > Jolokia provides comprehensive JMX features over HTTP/JSON including search > and list of JMX attributes and operations metadata, which helps to support > inter framework/platform compatibility. It has first class language bindings > for Perl, Python, Javascript, Java. > It's trivial (see demo patch) to incorporate Jolokia servlet into Hadoop HTTP > servers and use the same security mechanisms. > Adopting Jolokia will substantially improve the manageability of Hadoop and > its ecosystem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9756) Additional cleanup RPC code
[ https://issues.apache.org/jira/browse/HADOOP-9756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13720773#comment-13720773 ] Junping Du commented on HADOOP-9756: Hi Nicholas, the compilation error for branch-2 is because several jiras (HDFS-3880 and MAPREDUCE-4628) relates to HADOOP-8736 are not committed to branch-2. I have tried to backport that 2 jiras to branch-2 and applied with patch on this jira and build successful. Shall I integrate 3 patches together and deliver it on this JIRA or deliver the other 2 patches on counterpart jiras? > Additional cleanup RPC code > --- > > Key: HADOOP-9756 > URL: https://issues.apache.org/jira/browse/HADOOP-9756 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Junping Du >Assignee: Junping Du >Priority: Minor > Attachments: HADOOP-9756.patch, HADOOP-9756-v2.patch > > > HADOOP-9754 already did good job to address most of work for cleanup RPC > code. Here is some additional work, include: > - Remove some unused deprecated code. > - Narrow "throws Exception" to throw some specific exception and remove some > unnecessary exceptions > - Fix a generic warning and correct spell issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9776) HarFileSystem.listStatus() returns "har://-localhost:/..." if port number is empty
[ https://issues.apache.org/jira/browse/HADOOP-9776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13720841#comment-13720841 ] Chuan Liu commented on HADOOP-9776: --- The change looks good to me. Probably worth adding a unit test case to Hadoop to cover the scenario. > HarFileSystem.listStatus() returns "har://-localhost:/..." if port > number is empty > -- > > Key: HADOOP-9776 > URL: https://issues.apache.org/jira/browse/HADOOP-9776 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: shanyu zhao > Attachments: HADOOP-9776.patch > > > If the given har URI is "har://-localhost/usr/my.har/a", the result > of HarFileSystem.listStatus() will have a ":" appended after localhost, like > this: "har://-localhost:/usr/my.har/a". it should return > "har://-localhost/usr/my.bar/a" instead. > This creates problem when running a hive unit test TestCliDriver > (archive_excludeHadoop20.q), generating the following error: > java.io.IOException: cannot find dir = > har://pfile-localhost:/GitHub/hive-monarch/build/ql/test/data/warehouse/tstsrcpart/ds=2008-04-08/hr=12/data.har/00_0 > in pathToPartitionInfo: > [pfile:/GitHub/hive-monarch/build/ql/test/data/warehouse/tstsrcpart/ds=2008-04-08/hr=11, > > har://pfile-localhost/GitHub/hive-monarch/build/ql/test/data/warehouse/tstsrcpart/ds=2008-04-08/hr=12/data.har] > [junit] at > org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:298) > [junit] at > org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:260) > [junit] at > org.apache.hadoop.hive.ql.io.CombineHiveInputFormat$CombineHiveInputSplit.(CombineHiveInputFormat.java:104) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9774) RawLocalFileSystem.listStatus() return absolution paths when input path is relative on Windows
[
https://issues.apache.org/jira/browse/HADOOP-9774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13720849#comment-13720849
]
Chuan Liu commented on HADOOP-9774:
---
I think the behavior of {{RawLocalFileSystem.listStatus()}} will diverge with
this patch, which is not a very good practice. If the intended behavior is the
following, we should make it consistent on Linux and Windows in my opinion.
bq. When the caller use a relative path (without drive spec) to do listStatus()
the resulting path should be relative.
It will be good to add a unit test case to cover this scenario as well.
> RawLocalFileSystem.listStatus() return absolution paths when input path is
> relative on Windows
> --
>
> Key: HADOOP-9774
> URL: https://issues.apache.org/jira/browse/HADOOP-9774
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: shanyu zhao
> Attachments: HADOOP-9774.patch
>
>
> On Windows, when using RawLocalFileSystem.listStatus() to enumerate a
> relative path (without drive spec), e.g., "file:///mydata", the resulting
> paths become absolute paths, e.g., ["file://E:/mydata/t1.txt",
> "file://E:/mydata/t2.txt"...].
> Note that if we use it to enumerate an absolute path, e.g.,
> "file://E:/mydata" then the we get the same results as above.
> This breaks some hive unit tests which uses local file system to simulate
> HDFS when testing, therefore the drive spec is removed. Then after
> listStatus() the path is changed to absolute path, hive failed to find the
> path in its map reduce job.
> You'll see the following exception:
> [junit] java.io.IOException: cannot find dir =
> pfile:/E:/GitHub/hive-monarch/build/ql/test/data/warehouse/src/kv1.txt in
> pathToPartitionInfo:
> [pfile:/GitHub/hive-monarch/build/ql/test/data/warehouse/src]
> [junit] at
> org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:298)
> This problem is introduced by this JIRA:
> HADOOP-8962
> Prior to the fix for HADOOP-8962 (merged in 0.23.5), the resulting paths are
> relative paths if the parent paths are relative, e.g.,
> ["file:///mydata/t1.txt", "file:///mydata/t2.txt"...]
> This behavior change is a side effect of the fix in HADOOP-8962, not an
> intended change. The resulting behavior, even though is legitimate from a
> function point of view, break consistency from the caller's point of view.
> When the caller use a relative path (without drive spec) to do listStatus()
> the resulting path should be relative. Therefore, I think this should be
> fixed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9776) HarFileSystem.listStatus() returns "har://-localhost:/..." if port number is empty
[ https://issues.apache.org/jira/browse/HADOOP-9776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13720854#comment-13720854 ] Jason Lowe commented on HADOOP-9776: Agreed, change looks good but would like to see a unit test. It should be fairly straightforward to create one given the infrastructure already in TestHarFilesystemBasics.java. > HarFileSystem.listStatus() returns "har://-localhost:/..." if port > number is empty > -- > > Key: HADOOP-9776 > URL: https://issues.apache.org/jira/browse/HADOOP-9776 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: shanyu zhao > Attachments: HADOOP-9776.patch > > > If the given har URI is "har://-localhost/usr/my.har/a", the result > of HarFileSystem.listStatus() will have a ":" appended after localhost, like > this: "har://-localhost:/usr/my.har/a". it should return > "har://-localhost/usr/my.bar/a" instead. > This creates problem when running a hive unit test TestCliDriver > (archive_excludeHadoop20.q), generating the following error: > java.io.IOException: cannot find dir = > har://pfile-localhost:/GitHub/hive-monarch/build/ql/test/data/warehouse/tstsrcpart/ds=2008-04-08/hr=12/data.har/00_0 > in pathToPartitionInfo: > [pfile:/GitHub/hive-monarch/build/ql/test/data/warehouse/tstsrcpart/ds=2008-04-08/hr=11, > > har://pfile-localhost/GitHub/hive-monarch/build/ql/test/data/warehouse/tstsrcpart/ds=2008-04-08/hr=12/data.har] > [junit] at > org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:298) > [junit] at > org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:260) > [junit] at > org.apache.hadoop.hive.ql.io.CombineHiveInputFormat$CombineHiveInputSplit.(CombineHiveInputFormat.java:104) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9777) RPM should not claim ownership of paths owned by the platform
[ https://issues.apache.org/jira/browse/HADOOP-9777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9777: - Priority: Critical (was: Major) > RPM should not claim ownership of paths owned by the platform > - > > Key: HADOOP-9777 > URL: https://issues.apache.org/jira/browse/HADOOP-9777 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 1.1.2 > Environment: Fedora 19 x64 >Reporter: Stevo Slavic >Priority: Critical > > Installing Apache Hadoop rpm ( hadoop-1.1.2-1.x86_64.rpm ) on Fedora 19 x64 > fails with: > [root@laptop hadoop]# rpm -i /home/sslavic/Downloads/hadoop-1.1.2-1.x86_64.rpm > file /usr/bin from install of hadoop-1.1.2-1.x86_64 conflicts with file from > package filesystem-3.2-12.fc19.x86_64 > file /usr/lib from install of hadoop-1.1.2-1.x86_64 conflicts with file from > package filesystem-3.2-12.fc19.x86_64 > file /usr/lib64 from install of hadoop-1.1.2-1.x86_64 conflicts with file > from package filesystem-3.2-12.fc19.x86_64 > file /usr/sbin from install of hadoop-1.1.2-1.x86_64 conflicts with file from > package filesystem-3.2-12.fc19.x86_64 > Same issue occurs if one tries to install as non-root user: > [sslavic@laptop ~]$ sudo rpm -i Downloads/hadoop-1.1.2-1.x86_64.rpm > file /usr/bin from install of hadoop-1.1.2-1.x86_64 conflicts with file from > package filesystem-3.2-12.fc19.x86_64 > file /usr/lib from install of hadoop-1.1.2-1.x86_64 conflicts with file from > package filesystem-3.2-12.fc19.x86_64 > file /usr/lib64 from install of hadoop-1.1.2-1.x86_64 conflicts with file > from package filesystem-3.2-12.fc19.x86_64 > file /usr/sbin from install of hadoop-1.1.2-1.x86_64 conflicts with file from > package filesystem-3.2-12.fc19.x86_64 > It seems these 4 directories in Hadoop rpm have wrong permissions (+w for > owner). > This is violation of packaging rules. Hadoop rpm spec and/or build scripts > need to be fixed, so that rpm on installation doesn't try to claim ownership > of paths owned by the platform, in this case, filesystem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9768) chown and chgrp reject users and groups with spaces on platforms where spaces are otherwise acceptable
[
https://issues.apache.org/jira/browse/HADOOP-9768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13720901#comment-13720901
]
Ravi Prakash commented on HADOOP-9768:
--
Hi Chris!
Thanks for updating the patches. They look good to me. +1.
I agree, we should not be duplicating anything which the OS does anyway. I
agree we can handle that in a separate JIRA.
> chown and chgrp reject users and groups with spaces on platforms where spaces
> are otherwise acceptable
> --
>
> Key: HADOOP-9768
> URL: https://issues.apache.org/jira/browse/HADOOP-9768
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HADOOP-9768-branch-1.1.patch,
> HADOOP-9768-branch-1.2.patch, HADOOP-9768-branch-1.3.patch,
> HADOOP-9768-branch-1-win.3.patch, HADOOP-9768-trunk.1.patch,
> HADOOP-9768-trunk.2.patch, HADOOP-9768-trunk.3.patch
>
>
> The chown and chgrp commands enforce a check on a valid set of characters for
> user and group. The set of valid characters does not include space. On some
> platforms (notably Windows), the space character is acceptable. We've seen
> this cause test failures in {{TestFsShellReturnCode}} when running on Windows
> if the logged-in user is a member of a group like Remote Desktop Users.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9778) Remove user name and group name validation logic from chown and chgrp shell commands.
Chris Nauroth created HADOOP-9778: - Summary: Remove user name and group name validation logic from chown and chgrp shell commands. Key: HADOOP-9778 URL: https://issues.apache.org/jira/browse/HADOOP-9778 Project: Hadoop Common Issue Type: Improvement Components: fs Affects Versions: 3.0.0, 2.3.0 Reporter: Chris Nauroth The chown and chgrp shell commands enforce a valid set of characters for user names and group names. The validation rules for user names and group names vary across different OSes, so this is a portability problem and a maintenance burden. It requires us to change the code every time we discover a difference in the rules. I propose that we remove this validation logic and instead pass through input user names and group names to the OS. We can handle error reporting by catching errors returned from the OS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9778) Remove user name and group name validation logic from chown and chgrp shell commands.
[ https://issues.apache.org/jira/browse/HADOOP-9778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-9778: -- Priority: Minor (was: Major) > Remove user name and group name validation logic from chown and chgrp shell > commands. > - > > Key: HADOOP-9778 > URL: https://issues.apache.org/jira/browse/HADOOP-9778 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 3.0.0, 2.3.0 >Reporter: Chris Nauroth >Priority: Minor > > The chown and chgrp shell commands enforce a valid set of characters for user > names and group names. The validation rules for user names and group names > vary across different OSes, so this is a portability problem and a > maintenance burden. It requires us to change the code every time we discover > a difference in the rules. I propose that we remove this validation logic > and instead pass through input user names and group names to the OS. We can > handle error reporting by catching errors returned from the OS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9768) chown and chgrp reject users and groups with spaces on platforms where spaces are otherwise acceptable
[
https://issues.apache.org/jira/browse/HADOOP-9768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13720953#comment-13720953
]
Chris Nauroth commented on HADOOP-9768:
---
Thanks for the reviews, Arpit and Ravi!
{quote}
I agree, we should not be duplicating anything which the OS does anyway. I
agree we can handle that in a separate JIRA.
{quote}
Thanks, Ravi. I filed HADOOP-9778 to track this.
[~jingzhao], are you still +1 for the new patch? If so, then I would like to
commit it today.
> chown and chgrp reject users and groups with spaces on platforms where spaces
> are otherwise acceptable
> --
>
> Key: HADOOP-9768
> URL: https://issues.apache.org/jira/browse/HADOOP-9768
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HADOOP-9768-branch-1.1.patch,
> HADOOP-9768-branch-1.2.patch, HADOOP-9768-branch-1.3.patch,
> HADOOP-9768-branch-1-win.3.patch, HADOOP-9768-trunk.1.patch,
> HADOOP-9768-trunk.2.patch, HADOOP-9768-trunk.3.patch
>
>
> The chown and chgrp commands enforce a check on a valid set of characters for
> user and group. The set of valid characters does not include space. On some
> platforms (notably Windows), the space character is acceptable. We've seen
> this cause test failures in {{TestFsShellReturnCode}} when running on Windows
> if the logged-in user is a member of a group like Remote Desktop Users.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9507) LocalFileSystem rename() is broken in some cases when destination exists
[
https://issues.apache.org/jira/browse/HADOOP-9507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13720955#comment-13720955
]
Chris Nauroth commented on HADOOP-9507:
---
Thank you, everyone, for the great code reviews on this patch.
[~szetszwo], are you still +1 for the patch with the change suggested by Arpit
to use non-recursive delete? If so, then I'd like to commit today. It's a
one-line change since last time (see incremental diff below). Thanks!
{code}
diff --git
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
hadoop-common-projec
index 2553880..dbe29a2 100644
---
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
+++
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
@@ -338,7 +338,7 @@ public boolean rename(Path src, Path dst) throws
IOException {
LOG.debug("Deleting empty destination and renaming " + src + " to " +
dst);
}
-if (this.delete(dst, true) && srcFile.renameTo(dstFile)) {
+if (this.delete(dst, false) && srcFile.renameTo(dstFile)) {
return true;
}
}
{code}
> LocalFileSystem rename() is broken in some cases when destination exists
>
>
> Key: HADOOP-9507
> URL: https://issues.apache.org/jira/browse/HADOOP-9507
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
>Reporter: Mostafa Elhemali
>Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HADOOP-9507-branch-1.1.patch,
> HADOOP-9507-branch-1.3.patch, HADOOP-9507-branch-1.4.patch,
> HADOOP-9507-branch-1.5.patch, HADOOP-9507-branch-1-win.1.patch,
> HADOOP-9507-branch-1-win.3.patch, HADOOP-9507-branch-1-win.4.patch,
> HADOOP-9507-branch-1-win.5.patch, HADOOP-9507.branch-1-win.patch,
> HADOOP-9507-trunk.1.patch, HADOOP-9507-trunk.2.patch,
> HADOOP-9507-trunk.3.patch, HADOOP-9507-trunk.4.patch,
> HADOOP-9507-trunk.5.patch
>
>
> The rename() method in RawLocalFileSystem uses FileUtil.copy() without
> realizing that FileUtil.copy() has a special behavior that if you're copying
> /foo to /bar and /bar exists and is a directory, it'll copy /foo inside /bar
> instead of overwriting it, which is not what rename() wants. So you end up
> with weird behaviors like in this repro:
> {code}
> c:
> cd \
> md Foo
> md Bar
> md Foo\X
> md Bar\X
> hadoop fs -mv file:///c:/Foo file:///c:/Bar
> {code}
> At the end of this, you would expect to find only Bar\X, but you instead find
> Bar\X\X.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9761) ViewFileSystem#rename fails when using DistributedFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-9761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-9761: - Status: Patch Available (was: Open) > ViewFileSystem#rename fails when using DistributedFileSystem > > > Key: HADOOP-9761 > URL: https://issues.apache.org/jira/browse/HADOOP-9761 > Project: Hadoop Common > Issue Type: Bug > Components: viewfs >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: hadoop-9761-1.patch > > > ViewFileSystem currently passes unqualified paths (no scheme or authority) to > underlying FileSystems when doing a rename. DistributedFileSystem symlink > support added in HADOOP-9418 needs to qualify and check rename sources and > destinations since cross-filesystem renames aren't supported, so this breaks > in the following way > - Default FS URI is configured to viewfs:// > - When doing a rename, ViewFileSystem checks to make sure both src and dst > FileSystems are the same (which they are, both in same DFS), and then calls > DistributedFileSystem#rename with unqualified "remainder" paths > - Since these paths are unqualified, DFS qualifies them with the default FS > to check that it can do the rename. This turns it into > viewfs:/// > - Since viewfs:// is not the DFS's URI, DFS errors out the rename. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9761) ViewFileSystem#rename fails when using DistributedFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-9761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13720969#comment-13720969 ] Colin Patrick McCabe commented on HADOOP-9761: -- I'm not sure why Jenkins didn't run on this, but I re-submitted it. > ViewFileSystem#rename fails when using DistributedFileSystem > > > Key: HADOOP-9761 > URL: https://issues.apache.org/jira/browse/HADOOP-9761 > Project: Hadoop Common > Issue Type: Bug > Components: viewfs >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: hadoop-9761-1.patch > > > ViewFileSystem currently passes unqualified paths (no scheme or authority) to > underlying FileSystems when doing a rename. DistributedFileSystem symlink > support added in HADOOP-9418 needs to qualify and check rename sources and > destinations since cross-filesystem renames aren't supported, so this breaks > in the following way > - Default FS URI is configured to viewfs:// > - When doing a rename, ViewFileSystem checks to make sure both src and dst > FileSystems are the same (which they are, both in same DFS), and then calls > DistributedFileSystem#rename with unqualified "remainder" paths > - Since these paths are unqualified, DFS qualifies them with the default FS > to check that it can do the rename. This turns it into > viewfs:/// > - Since viewfs:// is not the DFS's URI, DFS errors out the rename. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9761) ViewFileSystem#rename fails when using DistributedFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-9761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-9761: - Status: Open (was: Patch Available) > ViewFileSystem#rename fails when using DistributedFileSystem > > > Key: HADOOP-9761 > URL: https://issues.apache.org/jira/browse/HADOOP-9761 > Project: Hadoop Common > Issue Type: Bug > Components: viewfs >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: hadoop-9761-1.patch > > > ViewFileSystem currently passes unqualified paths (no scheme or authority) to > underlying FileSystems when doing a rename. DistributedFileSystem symlink > support added in HADOOP-9418 needs to qualify and check rename sources and > destinations since cross-filesystem renames aren't supported, so this breaks > in the following way > - Default FS URI is configured to viewfs:// > - When doing a rename, ViewFileSystem checks to make sure both src and dst > FileSystems are the same (which they are, both in same DFS), and then calls > DistributedFileSystem#rename with unqualified "remainder" paths > - Since these paths are unqualified, DFS qualifies them with the default FS > to check that it can do the rename. This turns it into > viewfs:/// > - Since viewfs:// is not the DFS's URI, DFS errors out the rename. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9507) LocalFileSystem rename() is broken in some cases when destination exists
[
https://issues.apache.org/jira/browse/HADOOP-9507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13720980#comment-13720980
]
Mostafa Elhemali commented on HADOOP-9507:
--
Thanks [~cnauroth], +1 from my side.
> LocalFileSystem rename() is broken in some cases when destination exists
>
>
> Key: HADOOP-9507
> URL: https://issues.apache.org/jira/browse/HADOOP-9507
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
>Reporter: Mostafa Elhemali
>Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HADOOP-9507-branch-1.1.patch,
> HADOOP-9507-branch-1.3.patch, HADOOP-9507-branch-1.4.patch,
> HADOOP-9507-branch-1.5.patch, HADOOP-9507-branch-1-win.1.patch,
> HADOOP-9507-branch-1-win.3.patch, HADOOP-9507-branch-1-win.4.patch,
> HADOOP-9507-branch-1-win.5.patch, HADOOP-9507.branch-1-win.patch,
> HADOOP-9507-trunk.1.patch, HADOOP-9507-trunk.2.patch,
> HADOOP-9507-trunk.3.patch, HADOOP-9507-trunk.4.patch,
> HADOOP-9507-trunk.5.patch
>
>
> The rename() method in RawLocalFileSystem uses FileUtil.copy() without
> realizing that FileUtil.copy() has a special behavior that if you're copying
> /foo to /bar and /bar exists and is a directory, it'll copy /foo inside /bar
> instead of overwriting it, which is not what rename() wants. So you end up
> with weird behaviors like in this repro:
> {code}
> c:
> cd \
> md Foo
> md Bar
> md Foo\X
> md Bar\X
> hadoop fs -mv file:///c:/Foo file:///c:/Bar
> {code}
> At the end of this, you would expect to find only Bar\X, but you instead find
> Bar\X\X.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9778) Remove user name and group name validation logic from chown and chgrp shell commands.
[ https://issues.apache.org/jira/browse/HADOOP-9778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13720981#comment-13720981 ] Colin Patrick McCabe commented on HADOOP-9778: -- Here's an idea: why not make the validation configurable? I don't think there is a great demand for zany user names, and if we shell-expand them in even one place, we could have problems. Probably the default validation pattern for windows should be different and include spaces. I haven't actually found a place where we shell expand user names (despite its name, ShellBasedUnixGroupsMapping doesn't actually create a shell). > Remove user name and group name validation logic from chown and chgrp shell > commands. > - > > Key: HADOOP-9778 > URL: https://issues.apache.org/jira/browse/HADOOP-9778 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 3.0.0, 2.3.0 >Reporter: Chris Nauroth >Priority: Minor > > The chown and chgrp shell commands enforce a valid set of characters for user > names and group names. The validation rules for user names and group names > vary across different OSes, so this is a portability problem and a > maintenance burden. It requires us to change the code every time we discover > a difference in the rules. I propose that we remove this validation logic > and instead pass through input user names and group names to the OS. We can > handle error reporting by catching errors returned from the OS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9778) Remove user name and group name validation logic from chown and chgrp shell commands.
[
https://issues.apache.org/jira/browse/HADOOP-9778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721002#comment-13721002
]
Chris Nauroth commented on HADOOP-9778:
---
{quote}
Here's an idea: why not make the validation configurable?
{quote}
This could be a good compromise. Something that just occurred to me is that we
don't have a way to express different default property values per platform in
*-default.xml. That might be a handy thing to build. Even without that, I
suppose we could document this in core-default.xml with commented-out XML.
{quote}
I don't think there is a great demand for zany user names, and if we
shell-expand them in even one place, we could have problems.
{quote}
Thanks, I hadn't considered this risk. Configurability with a default would
maintain the current behavior and protect against this.
> Remove user name and group name validation logic from chown and chgrp shell
> commands.
> -
>
> Key: HADOOP-9778
> URL: https://issues.apache.org/jira/browse/HADOOP-9778
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Chris Nauroth
>Priority: Minor
>
> The chown and chgrp shell commands enforce a valid set of characters for user
> names and group names. The validation rules for user names and group names
> vary across different OSes, so this is a portability problem and a
> maintenance burden. It requires us to change the code every time we discover
> a difference in the rules. I propose that we remove this validation logic
> and instead pass through input user names and group names to the OS. We can
> handle error reporting by catching errors returned from the OS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9778) Support configurable user name and group name validation patterns in chown and chgrp shell commands.
[ https://issues.apache.org/jira/browse/HADOOP-9778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-9778: -- Summary: Support configurable user name and group name validation patterns in chown and chgrp shell commands. (was: Remove user name and group name validation logic from chown and chgrp shell commands.) > Support configurable user name and group name validation patterns in chown > and chgrp shell commands. > > > Key: HADOOP-9778 > URL: https://issues.apache.org/jira/browse/HADOOP-9778 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 3.0.0, 2.3.0 >Reporter: Chris Nauroth >Priority: Minor > > The chown and chgrp shell commands enforce a valid set of characters for user > names and group names. The validation rules for user names and group names > vary across different OSes, so this is a portability problem and a > maintenance burden. It requires us to change the code every time we discover > a difference in the rules. I propose that we remove this validation logic > and instead pass through input user names and group names to the OS. We can > handle error reporting by catching errors returned from the OS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9609) Remove sh dependency of bin-package target
[ https://issues.apache.org/jira/browse/HADOOP-9609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth resolved HADOOP-9609. --- Resolution: Fixed Fix Version/s: 1-win I've committed this to branch-1-win. Thank you to Chuan for contributing the patch, and thank you to Ivan for the code reviews. > Remove sh dependency of bin-package target > -- > > Key: HADOOP-9609 > URL: https://issues.apache.org/jira/browse/HADOOP-9609 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 1-win >Reporter: Chuan Liu >Assignee: Chuan Liu >Priority: Minor > Fix For: 1-win > > Attachments: HADOOP-9609-branch-1-win-2.patch, > HADOOP-9609-branch-1-win-3.patch, HADOOP-9609-branch-1-win.patch > > > In Ant package target, we no longer use packageBinNativeHadoop.sh to place > native library binaries. However, the same script is still present in the > bin-package target. We should remove bin-package target's dependency on sh to > keep it consistent with package target. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9698) RPCv9 client must honor server's SASL negotiate response
[ https://issues.apache.org/jira/browse/HADOOP-9698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HADOOP-9698: Attachment: HADOOP-9698.patch Patch didn't apply due to a deleted line changing, and casting that was removed on lines of context in the patch. Will wait for pre-commit anyway. I will make all discussed changes in a subsequent patch for IP failover. > RPCv9 client must honor server's SASL negotiate response > > > Key: HADOOP-9698 > URL: https://issues.apache.org/jira/browse/HADOOP-9698 > Project: Hadoop Common > Issue Type: Sub-task > Components: ipc >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Blocker > Attachments: HADOOP-9698.patch, HADOOP-9698.patch > > > As of HADOOP-9421, a RPCv9 server will advertise its authentication methods. > This is meant to support features such as IP failover, better token > selection, and interoperability in a heterogenous security environment. > Currently the client ignores the negotiate response and just blindly attempts > to authenticate instead of choosing a mutually agreeable auth method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9698) RPCv9 client must honor server's SASL negotiate response
[ https://issues.apache.org/jira/browse/HADOOP-9698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HADOOP-9698: Attachment: RPCv9 Authentication.pdf Draft documentation of the protocol for those interested! > RPCv9 client must honor server's SASL negotiate response > > > Key: HADOOP-9698 > URL: https://issues.apache.org/jira/browse/HADOOP-9698 > Project: Hadoop Common > Issue Type: Sub-task > Components: ipc >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Blocker > Attachments: HADOOP-9698.patch, HADOOP-9698.patch, RPCv9 > Authentication.pdf > > > As of HADOOP-9421, a RPCv9 server will advertise its authentication methods. > This is meant to support features such as IP failover, better token > selection, and interoperability in a heterogenous security environment. > Currently the client ignores the negotiate response and just blindly attempts > to authenticate instead of choosing a mutually agreeable auth method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9778) Support configurable user name and group name validation patterns in chown and chgrp shell commands.
[ https://issues.apache.org/jira/browse/HADOOP-9778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721039#comment-13721039 ] Daryn Sharp commented on HADOOP-9778: - If we are *nix shell expanding usernames, that definitely needs to be fixed. I personally agree with the original premise of let the OS decide but I suppose you could make the pattern ".*" if that's what you want. > Support configurable user name and group name validation patterns in chown > and chgrp shell commands. > > > Key: HADOOP-9778 > URL: https://issues.apache.org/jira/browse/HADOOP-9778 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 3.0.0, 2.3.0 >Reporter: Chris Nauroth >Priority: Minor > > The chown and chgrp shell commands enforce a valid set of characters for user > names and group names. The validation rules for user names and group names > vary across different OSes, so this is a portability problem and a > maintenance burden. It requires us to change the code every time we discover > a difference in the rules. I propose that we remove this validation logic > and instead pass through input user names and group names to the OS. We can > handle error reporting by catching errors returned from the OS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9770) RetryCache#state need not be volatile
[ https://issues.apache.org/jira/browse/HADOOP-9770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721041#comment-13721041 ] Suresh Srinivas commented on HADOOP-9770: - [~cmccabe] If there are no further comments, I would like to commit this patch soon. > RetryCache#state need not be volatile > - > > Key: HADOOP-9770 > URL: https://issues.apache.org/jira/browse/HADOOP-9770 > Project: Hadoop Common > Issue Type: Improvement > Components: util >Affects Versions: 2.1.0-beta >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Minor > Attachments: HADOOP-9770.1.patch, HADOOP-9770.patch > > > See the comment - > https://issues.apache.org/jira/browse/HDFS-4979?focusedCommentId=13719111&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13719111 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9698) RPCv9 client must honor server's SASL negotiate response
[
https://issues.apache.org/jira/browse/HADOOP-9698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721040#comment-13721040
]
Hadoop QA commented on HADOOP-9698:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12594431/RPCv9%20Authentication.pdf
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2854//console
This message is automatically generated.
> RPCv9 client must honor server's SASL negotiate response
>
>
> Key: HADOOP-9698
> URL: https://issues.apache.org/jira/browse/HADOOP-9698
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ipc
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Blocker
> Attachments: HADOOP-9698.patch, HADOOP-9698.patch, RPCv9
> Authentication.pdf
>
>
> As of HADOOP-9421, a RPCv9 server will advertise its authentication methods.
> This is meant to support features such as IP failover, better token
> selection, and interoperability in a heterogenous security environment.
> Currently the client ignores the negotiate response and just blindly attempts
> to authenticate instead of choosing a mutually agreeable auth method.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-6424) Port FsShell to FileContext
[ https://issues.apache.org/jira/browse/HADOOP-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reassigned HADOOP-6424: --- Assignee: (was: Andrew Wang) > Port FsShell to FileContext > > > Key: HADOOP-6424 > URL: https://issues.apache.org/jira/browse/HADOOP-6424 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 2.0.0-alpha >Reporter: Eli Collins > Attachments: HADOOP-6424.patch, HADOOP-6424.patch > > > FsShell currently uses FileSystem, needs to be moved over to FileContext. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9770) RetryCache#state need not be volatile
[ https://issues.apache.org/jira/browse/HADOOP-9770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721100#comment-13721100 ] Colin Patrick McCabe commented on HADOOP-9770: -- +1 > RetryCache#state need not be volatile > - > > Key: HADOOP-9770 > URL: https://issues.apache.org/jira/browse/HADOOP-9770 > Project: Hadoop Common > Issue Type: Improvement > Components: util >Affects Versions: 2.1.0-beta >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Minor > Attachments: HADOOP-9770.1.patch, HADOOP-9770.patch > > > See the comment - > https://issues.apache.org/jira/browse/HDFS-4979?focusedCommentId=13719111&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13719111 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9770) RetryCache#state need not be volatile
[ https://issues.apache.org/jira/browse/HADOOP-9770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721105#comment-13721105 ] Jing Zhao commented on HADOOP-9770: --- +1 also. > RetryCache#state need not be volatile > - > > Key: HADOOP-9770 > URL: https://issues.apache.org/jira/browse/HADOOP-9770 > Project: Hadoop Common > Issue Type: Improvement > Components: util >Affects Versions: 2.1.0-beta >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Minor > Attachments: HADOOP-9770.1.patch, HADOOP-9770.patch > > > See the comment - > https://issues.apache.org/jira/browse/HDFS-4979?focusedCommentId=13719111&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13719111 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9588) unit test failure: org.apache.hadoop.io.compress.TestCodec.testGzipLongOverflow
[
https://issues.apache.org/jira/browse/HADOOP-9588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721108#comment-13721108
]
Chris Nauroth commented on HADOOP-9588:
---
This failure only repros for me when the native code is loaded. The problem is
this assertion:
{code}
public void testGzipLongOverflow() throws IOException {
LOG.info("testGzipLongOverflow");
// Don't use native libs for this test.
Configuration conf = new Configuration();
conf.setBoolean("io.native.lib.available", false);
assertFalse("ZlibFactory is using native libs against request",
ZlibFactory.isNativeZlibLoaded(conf));
{code}
The test is trying to disable native library loading by setting configuration,
but the code in {{ZlibFactory}} isn't inspecting the correct configuration
property:
{code}
public static boolean isNativeZlibLoaded(Configuration conf) {
return nativeZlibLoaded && conf.getBoolean("hadoop.native.lib", true);
}
{code}
The trunk version of the code doesn't have this problem.
> unit test failure:
> org.apache.hadoop.io.compress.TestCodec.testGzipLongOverflow
> ---
>
> Key: HADOOP-9588
> URL: https://issues.apache.org/jira/browse/HADOOP-9588
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Affects Versions: 1.3.0
> Environment:
> https://builds.apache.org/job/Hadoop-branch1/lastCompletedBuild/testReport/org.apache.hadoop.io.compress/TestCodec/testGzipLongOverflow/
>Reporter: Giridharan Kesavan
>
> Error Message
> ZlibFactory is using native libs against request
> Stacktrace
> junit.framework.AssertionFailedError: ZlibFactory is using native libs
> against request
> at
> org.apache.hadoop.io.compress.TestCodec.testGzipLongOverflow(TestCodec.java:822)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9698) RPCv9 client must honor server's SASL negotiate response
[ https://issues.apache.org/jira/browse/HADOOP-9698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HADOOP-9698: Attachment: HADOOP-9698.patch Same patch as before. Dumb pre-commit tried to use my design doc as a patch... > RPCv9 client must honor server's SASL negotiate response > > > Key: HADOOP-9698 > URL: https://issues.apache.org/jira/browse/HADOOP-9698 > Project: Hadoop Common > Issue Type: Sub-task > Components: ipc >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Blocker > Attachments: HADOOP-9698.patch, HADOOP-9698.patch, HADOOP-9698.patch, > RPCv9 Authentication.pdf > > > As of HADOOP-9421, a RPCv9 server will advertise its authentication methods. > This is meant to support features such as IP failover, better token > selection, and interoperability in a heterogenous security environment. > Currently the client ignores the negotiate response and just blindly attempts > to authenticate instead of choosing a mutually agreeable auth method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9588) unit test failure: org.apache.hadoop.io.compress.TestCodec.testGzipLongOverflow
[ https://issues.apache.org/jira/browse/HADOOP-9588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-9588: -- Target Version/s: 1.3.0 Labels: newbie (was: ) > unit test failure: > org.apache.hadoop.io.compress.TestCodec.testGzipLongOverflow > --- > > Key: HADOOP-9588 > URL: https://issues.apache.org/jira/browse/HADOOP-9588 > Project: Hadoop Common > Issue Type: Bug > Components: test >Affects Versions: 1.3.0 > Environment: > https://builds.apache.org/job/Hadoop-branch1/lastCompletedBuild/testReport/org.apache.hadoop.io.compress/TestCodec/testGzipLongOverflow/ >Reporter: Giridharan Kesavan > Labels: newbie > > Error Message > ZlibFactory is using native libs against request > Stacktrace > junit.framework.AssertionFailedError: ZlibFactory is using native libs > against request > at > org.apache.hadoop.io.compress.TestCodec.testGzipLongOverflow(TestCodec.java:822) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9676) make maximum RPC buffer size configurable
[ https://issues.apache.org/jira/browse/HADOOP-9676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721116#comment-13721116 ] Vinod Kumar Vavilapalli commented on HADOOP-9676: - [~cmccabe], can you please set the fix-version for this one? 2.1.0-beta? (Haven't seen your other commits, but saying out just in case - we set target-version pre-commit and fix-version at commit time.) > make maximum RPC buffer size configurable > - > > Key: HADOOP-9676 > URL: https://issues.apache.org/jira/browse/HADOOP-9676 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.1.0-beta >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: HADOOP-9676.001.patch, HADOOP-9676.003.patch > > > Currently the RPC server just allocates however much memory the client asks > for, without validating. It would be nice to make the maximum RPC buffer > size configurable. This would prevent a rogue client from bringing down the > NameNode (or other Hadoop daemon) with a few requests for 2 GB buffers. It > would also make it easier to debug issues with super-large RPCs or malformed > headers, since OOMs can be difficult for developers to reproduce. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9770) Make RetryCache#state non volatile
[ https://issues.apache.org/jira/browse/HADOOP-9770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-9770: Summary: Make RetryCache#state non volatile (was: RetryCache#state need not be volatile) > Make RetryCache#state non volatile > -- > > Key: HADOOP-9770 > URL: https://issues.apache.org/jira/browse/HADOOP-9770 > Project: Hadoop Common > Issue Type: Improvement > Components: util >Affects Versions: 2.1.0-beta >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Minor > Attachments: HADOOP-9770.1.patch, HADOOP-9770.patch > > > See the comment - > https://issues.apache.org/jira/browse/HDFS-4979?focusedCommentId=13719111&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13719111 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9770) Make RetryCache#state non volatile
[ https://issues.apache.org/jira/browse/HADOOP-9770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-9770: Resolution: Fixed Fix Version/s: 2.1.0-beta Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I committed this patch to trunk, branch-2 and branch-2.1. [~jingzhao] and [~cmccabe], thank you for the review. > Make RetryCache#state non volatile > -- > > Key: HADOOP-9770 > URL: https://issues.apache.org/jira/browse/HADOOP-9770 > Project: Hadoop Common > Issue Type: Improvement > Components: util >Affects Versions: 2.1.0-beta >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Minor > Fix For: 2.1.0-beta > > Attachments: HADOOP-9770.1.patch, HADOOP-9770.patch > > > See the comment - > https://issues.apache.org/jira/browse/HDFS-4979?focusedCommentId=13719111&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13719111 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9164) Print paths of loaded native libraries in NativeLibraryChecker
[ https://issues.apache.org/jira/browse/HADOOP-9164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721151#comment-13721151 ] German Florez-Larrahondo commented on HADOOP-9164: -- Regards I've been using this command and it is quite useful. However, shouldn't we also be checking for LZO being loaded? Thanks ./g > Print paths of loaded native libraries in NativeLibraryChecker > -- > > Key: HADOOP-9164 > URL: https://issues.apache.org/jira/browse/HADOOP-9164 > Project: Hadoop Common > Issue Type: Improvement > Components: native >Affects Versions: 2.0.2-alpha >Reporter: Binglin Chang >Assignee: Binglin Chang >Priority: Minor > Fix For: 2.1.0-beta > > Attachments: HADOOP-9164.v1.patch, HADOOP-9164.v2.patch, > HADOOP-9164.v3.patch, HADOOP-9164.v4.2.patch, HADOOP-9164.v4.patch, > HADOOP-9164.v4.patch, HADOOP-9164.v5.patch, HADOOP-9164.v6.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9698) RPCv9 client must honor server's SASL negotiate response
[
https://issues.apache.org/jira/browse/HADOOP-9698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721155#comment-13721155
]
Hadoop QA commented on HADOOP-9698:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12594437/HADOOP-9698.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2855//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2855//console
This message is automatically generated.
> RPCv9 client must honor server's SASL negotiate response
>
>
> Key: HADOOP-9698
> URL: https://issues.apache.org/jira/browse/HADOOP-9698
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ipc
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Blocker
> Attachments: HADOOP-9698.patch, HADOOP-9698.patch, HADOOP-9698.patch,
> RPCv9 Authentication.pdf
>
>
> As of HADOOP-9421, a RPCv9 server will advertise its authentication methods.
> This is meant to support features such as IP failover, better token
> selection, and interoperability in a heterogenous security environment.
> Currently the client ignores the negotiate response and just blindly attempts
> to authenticate instead of choosing a mutually agreeable auth method.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9507) LocalFileSystem rename() is broken in some cases when destination exists
[
https://issues.apache.org/jira/browse/HADOOP-9507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721165#comment-13721165
]
Tsz Wo (Nicholas), SZE commented on HADOOP-9507:
+1 the latest patch looks good. Thanks.
> LocalFileSystem rename() is broken in some cases when destination exists
>
>
> Key: HADOOP-9507
> URL: https://issues.apache.org/jira/browse/HADOOP-9507
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
>Reporter: Mostafa Elhemali
>Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HADOOP-9507-branch-1.1.patch,
> HADOOP-9507-branch-1.3.patch, HADOOP-9507-branch-1.4.patch,
> HADOOP-9507-branch-1.5.patch, HADOOP-9507-branch-1-win.1.patch,
> HADOOP-9507-branch-1-win.3.patch, HADOOP-9507-branch-1-win.4.patch,
> HADOOP-9507-branch-1-win.5.patch, HADOOP-9507.branch-1-win.patch,
> HADOOP-9507-trunk.1.patch, HADOOP-9507-trunk.2.patch,
> HADOOP-9507-trunk.3.patch, HADOOP-9507-trunk.4.patch,
> HADOOP-9507-trunk.5.patch
>
>
> The rename() method in RawLocalFileSystem uses FileUtil.copy() without
> realizing that FileUtil.copy() has a special behavior that if you're copying
> /foo to /bar and /bar exists and is a directory, it'll copy /foo inside /bar
> instead of overwriting it, which is not what rename() wants. So you end up
> with weird behaviors like in this repro:
> {code}
> c:
> cd \
> md Foo
> md Bar
> md Foo\X
> md Bar\X
> hadoop fs -mv file:///c:/Foo file:///c:/Bar
> {code}
> At the end of this, you would expect to find only Bar\X, but you instead find
> Bar\X\X.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9665) BlockDecompressorStream#decompress will throw EOFException instead of return -1 when EOF
[
https://issues.apache.org/jira/browse/HADOOP-9665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721176#comment-13721176
]
Chris Nauroth commented on HADOOP-9665:
---
I intend to merge this patch to branch-1-win later today. The branch-1 patch
applies cleanly to branch-1-win.
> BlockDecompressorStream#decompress will throw EOFException instead of return
> -1 when EOF
>
>
> Key: HADOOP-9665
> URL: https://issues.apache.org/jira/browse/HADOOP-9665
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 1.1.2, 2.1.0-beta, 2.3.0
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
>Priority: Critical
> Fix For: 2.1.0-beta, 1.2.1
>
> Attachments: HADOOP-9665.1.patch, HADOOP-9665.2.patch,
> HADOOP-9665-branch-1.1.patch
>
>
> BlockDecompressorStream#decompress ultimately calls rawReadInt, which will
> throw EOFException instead of return -1 when encountering end of a stream.
> Then, decompress will be called by read. However, InputStream#read is
> supposed to return -1 instead of throwing EOFException to indicate the end of
> a stream. This explains why in LineReader,
> {code}
> if (bufferPosn >= bufferLength) {
> startPosn = bufferPosn = 0;
> if (prevCharCR)
> ++bytesConsumed; //account for CR from previous read
> bufferLength = in.read(buffer);
> if (bufferLength <= 0)
> break; // EOF
> }
> {code}
> -1 is checked instead of catching EOFException.
> Now the problem will occur with SnappyCodec. If an input file is compressed
> with SnappyCodec, it needs to be decompressed through BlockDecompressorStream
> when it is read. Then, if it empty, EOFException will been thrown from
> rawReadInt and break LineReader.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9507) LocalFileSystem rename() is broken in some cases when destination exists
[
https://issues.apache.org/jira/browse/HADOOP-9507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-9507:
--
Resolution: Fixed
Fix Version/s: 1.3.0
2.1.0-beta
1-win
3.0.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I've committed this to trunk, branch-2, branch-2.1-beta, branch-1, and
branch-1-win. Thank you to Chuan, Mostafa, Arpit, and Nicholas for the
excellent code review feedback!
> LocalFileSystem rename() is broken in some cases when destination exists
>
>
> Key: HADOOP-9507
> URL: https://issues.apache.org/jira/browse/HADOOP-9507
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
>Reporter: Mostafa Elhemali
>Assignee: Chris Nauroth
>Priority: Minor
> Fix For: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
>
> Attachments: HADOOP-9507-branch-1.1.patch,
> HADOOP-9507-branch-1.3.patch, HADOOP-9507-branch-1.4.patch,
> HADOOP-9507-branch-1.5.patch, HADOOP-9507-branch-1-win.1.patch,
> HADOOP-9507-branch-1-win.3.patch, HADOOP-9507-branch-1-win.4.patch,
> HADOOP-9507-branch-1-win.5.patch, HADOOP-9507.branch-1-win.patch,
> HADOOP-9507-trunk.1.patch, HADOOP-9507-trunk.2.patch,
> HADOOP-9507-trunk.3.patch, HADOOP-9507-trunk.4.patch,
> HADOOP-9507-trunk.5.patch
>
>
> The rename() method in RawLocalFileSystem uses FileUtil.copy() without
> realizing that FileUtil.copy() has a special behavior that if you're copying
> /foo to /bar and /bar exists and is a directory, it'll copy /foo inside /bar
> instead of overwriting it, which is not what rename() wants. So you end up
> with weird behaviors like in this repro:
> {code}
> c:
> cd \
> md Foo
> md Bar
> md Foo\X
> md Bar\X
> hadoop fs -mv file:///c:/Foo file:///c:/Bar
> {code}
> At the end of this, you would expect to find only Bar\X, but you instead find
> Bar\X\X.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9665) BlockDecompressorStream#decompress will throw EOFException instead of return -1 when EOF
[
https://issues.apache.org/jira/browse/HADOOP-9665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-9665:
--
Fix Version/s: 1-win
I've committed this to branch-1-win. Thanks again for this patch, Zhijie!
> BlockDecompressorStream#decompress will throw EOFException instead of return
> -1 when EOF
>
>
> Key: HADOOP-9665
> URL: https://issues.apache.org/jira/browse/HADOOP-9665
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 1.1.2, 2.1.0-beta, 2.3.0
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
>Priority: Critical
> Fix For: 1-win, 2.1.0-beta, 1.2.1
>
> Attachments: HADOOP-9665.1.patch, HADOOP-9665.2.patch,
> HADOOP-9665-branch-1.1.patch
>
>
> BlockDecompressorStream#decompress ultimately calls rawReadInt, which will
> throw EOFException instead of return -1 when encountering end of a stream.
> Then, decompress will be called by read. However, InputStream#read is
> supposed to return -1 instead of throwing EOFException to indicate the end of
> a stream. This explains why in LineReader,
> {code}
> if (bufferPosn >= bufferLength) {
> startPosn = bufferPosn = 0;
> if (prevCharCR)
> ++bytesConsumed; //account for CR from previous read
> bufferLength = in.read(buffer);
> if (bufferLength <= 0)
> break; // EOF
> }
> {code}
> -1 is checked instead of catching EOFException.
> Now the problem will occur with SnappyCodec. If an input file is compressed
> with SnappyCodec, it needs to be decompressed through BlockDecompressorStream
> when it is read. Then, if it empty, EOFException will been thrown from
> rawReadInt and break LineReader.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9779) create constant RpcConstants.CLIENT_ID_LENGTH rather than hard-coding 16 everywhere
Colin Patrick McCabe created HADOOP-9779: Summary: create constant RpcConstants.CLIENT_ID_LENGTH rather than hard-coding 16 everywhere Key: HADOOP-9779 URL: https://issues.apache.org/jira/browse/HADOOP-9779 Project: Hadoop Common Issue Type: Bug Reporter: Colin Patrick McCabe Priority: Minor We should create a constant RpcConstants.CLIENT_ID_LENGTH rather than hard-coding the number 16 everywhere. Sun Java coding style convention 10.3: bq. Numerical constants (literals) should not be coded directly, except for -1, 0, and 1, which can appear in a for loop as counter values. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9676) make maximum RPC buffer size configurable
[ https://issues.apache.org/jira/browse/HADOOP-9676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-9676: - Fix Version/s: 2.1.0-beta > make maximum RPC buffer size configurable > - > > Key: HADOOP-9676 > URL: https://issues.apache.org/jira/browse/HADOOP-9676 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.1.0-beta >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Fix For: 2.1.0-beta > > Attachments: HADOOP-9676.001.patch, HADOOP-9676.003.patch > > > Currently the RPC server just allocates however much memory the client asks > for, without validating. It would be nice to make the maximum RPC buffer > size configurable. This would prevent a rogue client from bringing down the > NameNode (or other Hadoop daemon) with a few requests for 2 GB buffers. It > would also make it easier to debug issues with super-large RPCs or malformed > headers, since OOMs can be difficult for developers to reproduce. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9779) create constant RpcConstants.CLIENT_ID_LENGTH rather than hard-coding 16 everywhere
[ https://issues.apache.org/jira/browse/HADOOP-9779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721335#comment-13721335 ] Suresh Srinivas commented on HADOOP-9779: - +1 for the idea. > create constant RpcConstants.CLIENT_ID_LENGTH rather than hard-coding 16 > everywhere > --- > > Key: HADOOP-9779 > URL: https://issues.apache.org/jira/browse/HADOOP-9779 > Project: Hadoop Common > Issue Type: Bug >Reporter: Colin Patrick McCabe >Priority: Minor > > We should create a constant RpcConstants.CLIENT_ID_LENGTH rather than > hard-coding the number 16 everywhere. > Sun Java coding style convention 10.3: > bq. Numerical constants (literals) should not be coded directly, except for > -1, 0, and 1, which can appear in a for loop as counter values. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HADOOP-9652:
-
Resolution: Fixed
Fix Version/s: 2.3.0
Target Version/s: 2.3.0 (was: 2.1.0-beta)
Status: Resolved (was: Patch Available)
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9780) Consider providing original symlink path in getFileLinkStatus rather than resolved path
Colin Patrick McCabe created HADOOP-9780:
Summary: Consider providing original symlink path in
getFileLinkStatus rather than resolved path
Key: HADOOP-9780
URL: https://issues.apache.org/jira/browse/HADOOP-9780
Project: Hadoop Common
Issue Type: Bug
Reporter: Colin Patrick McCabe
Priority: Minor
We should consider providing the original symlink path in getFileLinkStatus
rather than the resolved path. Users who want the resolved path can use
{{FileSystem#resolvePath}} to get that; users who just want information about
the symlink itself (probably most users) don't have to pay the cost of
resolution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721360#comment-13721360
]
Colin Patrick McCabe commented on HADOOP-9652:
--
I filed https://issues.apache.org/jira/browse/HADOOP-9780 to discuss the
{{FileSystem#getLinkTarget}} issue, which is unrelated to this stuff
> RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
> -
>
> Key: HADOOP-9652
> URL: https://issues.apache.org/jira/browse/HADOOP-9652
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Andrew Wang
> Fix For: 2.3.0
>
> Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
> hadoop-9652-3.patch
>
>
> {{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
> the symlink, but instead uses the owner and mode of the symlink target. If
> the target can't be found, it fills in bogus values (the empty string and
> FsPermission.getDefault) for these.
> Symlinks have an owner distinct from the owner of the target they point to,
> and getFileLinkStatus ought to expose this.
> In some operating systems, symlinks can have a permission other than 0777.
> We ought to expose this in RawLocalFilesystem and other places, although we
> don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9758) Provide configuration option for FileSystem/FileContext symlink resolution
[
https://issues.apache.org/jira/browse/HADOOP-9758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721391#comment-13721391
]
Colin Patrick McCabe commented on HADOOP-9758:
--
bq. btw, I'm also going to move this to HADOOP rather than HDFS, since it's not
just disabling DFS resolution.
OK.
bq. The provided test case also tests both FileContext and FileSystem. I could
separate these if you wish.
Nah, I think they're fine. Thanks.
I don't think we should change {{FileContext}} to extend {{Configurable}}. The
JavaDoc for FileContext clearly states that if you want a non-default
configuration, you should call {{getFileContext}}.
{code}
* Example 4: Use a specific config, ignoring $HADOOP_CONFIG
* Generally you should not need use a config unless you are doing
*
*configX = someConfigSomeOnePassedToYou;
*myFContext = getFileContext(configX); // configX is not changed,
* // is passed down
*myFContext.create(path, ...);
* ...
*
{code}
Let's cache the {{resolveSymlinks}} boolean, consistent with how we cache the
other client configuration information in final variables in
{{DFSClient#conf}}. Doing a lookup every time is slow and definitely not
necessary.
{code}
} catch (UnresolvedLinkException e) {
Configuration conf = filesys.getConf();
boolean resolveSymlinks = conf.getBoolean(
CommonConfigurationKeys.FS_SYMLINKS_RESOLVE_KEY,
CommonConfigurationKeys.FS_SYMLINKS_RESOLVE_DEFAULT);
if (!resolveSymlinks) {
throw new IOException("Path " + path + " contains a symlink"
+ " and symlink resolution is disabled ("
+ CommonConfigurationKeys.FS_SYMLINKS_RESOLVE_KEY + ").", e);
}
{code}
It's dangerous to change the Configuration object inside a {{FileSystem}},
since you don't know how many other threads are sharing that same object (due
to the caching). Users who want a {{FileSystem}} with a specific configuration
can use {{FileSystem#newInstance(Configuration conf)}}.
> Provide configuration option for FileSystem/FileContext symlink resolution
> --
>
> Key: HADOOP-9758
> URL: https://issues.apache.org/jira/browse/HADOOP-9758
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: hdfs-4968-1.patch, hdfs-4968-2.patch, hdfs-4968-3.patch
>
>
> With FileSystem symlink support incoming in HADOOP-8040, some clients will
> wish to not transparently resolve symlinks. This is somewhat similar to
> O_NOFOLLOW in open(2).
> Rationale for is for a security model where a user can invoke a third-party
> service running as a service user to operate on the user's data. For
> instance, users might want to use Hive to query data in their homedirs, where
> Hive runs as the Hive user and the data is readable by the Hive user. This
> leads to a security issue with symlinks:
> # User Mallory invokes Hive to process data files in {{/user/mallory/hive/}}
> # Hive checks permissions on the files in {{/user/mallory/hive/}} and allows
> the query to proceed.
> # RACE: Mallory replaces the files in {{/user/mallory/hive}} with symlinks
> that point to user Ann's Hive files in {{/user/ann/hive}}. These files aren't
> readable by Mallory, but she can create whatever symlinks she wants in her
> own scratch directory.
> # Hive's MR jobs happily resolve the symlinks and accesses Ann's private data.
> This is also potentially useful for clients using FileContext, so let's add
> it there too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9776) HarFileSystem.listStatus() returns "har://-localhost:/..." if port number is empty
[ https://issues.apache.org/jira/browse/HADOOP-9776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shanyu zhao updated HADOOP-9776: Attachment: HADOOP-9776-2.patch Adding unit test case > HarFileSystem.listStatus() returns "har://-localhost:/..." if port > number is empty > -- > > Key: HADOOP-9776 > URL: https://issues.apache.org/jira/browse/HADOOP-9776 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: shanyu zhao > Attachments: HADOOP-9776-2.patch, HADOOP-9776.patch > > > If the given har URI is "har://-localhost/usr/my.har/a", the result > of HarFileSystem.listStatus() will have a ":" appended after localhost, like > this: "har://-localhost:/usr/my.har/a". it should return > "har://-localhost/usr/my.bar/a" instead. > This creates problem when running a hive unit test TestCliDriver > (archive_excludeHadoop20.q), generating the following error: > java.io.IOException: cannot find dir = > har://pfile-localhost:/GitHub/hive-monarch/build/ql/test/data/warehouse/tstsrcpart/ds=2008-04-08/hr=12/data.har/00_0 > in pathToPartitionInfo: > [pfile:/GitHub/hive-monarch/build/ql/test/data/warehouse/tstsrcpart/ds=2008-04-08/hr=11, > > har://pfile-localhost/GitHub/hive-monarch/build/ql/test/data/warehouse/tstsrcpart/ds=2008-04-08/hr=12/data.har] > [junit] at > org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:298) > [junit] at > org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:260) > [junit] at > org.apache.hadoop.hive.ql.io.CombineHiveInputFormat$CombineHiveInputSplit.(CombineHiveInputFormat.java:104) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9776) HarFileSystem.listStatus() returns "har://-localhost:/..." if port number is empty
[ https://issues.apache.org/jira/browse/HADOOP-9776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721399#comment-13721399 ] Chuan Liu commented on HADOOP-9776: --- +1 nitpick: indentation seems wrong in the beginning of the test case. > HarFileSystem.listStatus() returns "har://-localhost:/..." if port > number is empty > -- > > Key: HADOOP-9776 > URL: https://issues.apache.org/jira/browse/HADOOP-9776 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: shanyu zhao > Attachments: HADOOP-9776-2.patch, HADOOP-9776.patch > > > If the given har URI is "har://-localhost/usr/my.har/a", the result > of HarFileSystem.listStatus() will have a ":" appended after localhost, like > this: "har://-localhost:/usr/my.har/a". it should return > "har://-localhost/usr/my.bar/a" instead. > This creates problem when running a hive unit test TestCliDriver > (archive_excludeHadoop20.q), generating the following error: > java.io.IOException: cannot find dir = > har://pfile-localhost:/GitHub/hive-monarch/build/ql/test/data/warehouse/tstsrcpart/ds=2008-04-08/hr=12/data.har/00_0 > in pathToPartitionInfo: > [pfile:/GitHub/hive-monarch/build/ql/test/data/warehouse/tstsrcpart/ds=2008-04-08/hr=11, > > har://pfile-localhost/GitHub/hive-monarch/build/ql/test/data/warehouse/tstsrcpart/ds=2008-04-08/hr=12/data.har] > [junit] at > org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:298) > [junit] at > org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:260) > [junit] at > org.apache.hadoop.hive.ql.io.CombineHiveInputFormat$CombineHiveInputSplit.(CombineHiveInputFormat.java:104) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9776) HarFileSystem.listStatus() returns "har://-localhost:/..." if port number is empty
[
https://issues.apache.org/jira/browse/HADOOP-9776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721426#comment-13721426
]
Hadoop QA commented on HADOOP-9776:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12594483/HADOOP-9776-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2856//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2856//console
This message is automatically generated.
> HarFileSystem.listStatus() returns "har://-localhost:/..." if port
> number is empty
> --
>
> Key: HADOOP-9776
> URL: https://issues.apache.org/jira/browse/HADOOP-9776
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: shanyu zhao
> Attachments: HADOOP-9776-2.patch, HADOOP-9776.patch
>
>
> If the given har URI is "har://-localhost/usr/my.har/a", the result
> of HarFileSystem.listStatus() will have a ":" appended after localhost, like
> this: "har://-localhost:/usr/my.har/a". it should return
> "har://-localhost/usr/my.bar/a" instead.
> This creates problem when running a hive unit test TestCliDriver
> (archive_excludeHadoop20.q), generating the following error:
> java.io.IOException: cannot find dir =
> har://pfile-localhost:/GitHub/hive-monarch/build/ql/test/data/warehouse/tstsrcpart/ds=2008-04-08/hr=12/data.har/00_0
> in pathToPartitionInfo:
> [pfile:/GitHub/hive-monarch/build/ql/test/data/warehouse/tstsrcpart/ds=2008-04-08/hr=11,
>
> har://pfile-localhost/GitHub/hive-monarch/build/ql/test/data/warehouse/tstsrcpart/ds=2008-04-08/hr=12/data.har]
> [junit] at
> org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:298)
> [junit] at
> org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:260)
> [junit] at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat$CombineHiveInputSplit.(CombineHiveInputFormat.java:104)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9534) Credential Management Framework (CMF)
[
https://issues.apache.org/jira/browse/HADOOP-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Larry McCay updated HADOOP-9534:
Attachment: 0001-HADOOP-9534-Credential-Management-Framework-initial-.patch
Initial Patch
> Credential Management Framework (CMF)
> -
>
> Key: HADOOP-9534
> URL: https://issues.apache.org/jira/browse/HADOOP-9534
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: security
>Affects Versions: 3.0.0
>Reporter: Larry McCay
> Labels: patch
> Attachments:
> 0001-HADOOP-9534-Credential-Management-Framework-initial-.patch
>
> Original Estimate: 504h
> Remaining Estimate: 504h
>
> The credential management framework consists of library for securing,
> acquiring and rolling credentials for a given Hadoop service.
> Specifically the library will provide:
> 1. Password Indirection or Aliasing
> 2. Management of identity and trust keystores
> 3. Rolling of key pairs and credentials
> 4. Discovery of externally provisioned credentials
> 5. Service specific CMF secret protection
> 6. Syntax for Aliases within configuration files
> Password Indirection or Aliasing:
> By providing alias based access to actual secrets stored within a service
> specific JCEKS keystore, we are able to eliminate the need for any secret to
> be stored in clear text on the filesystem. This is a current redflag during
> security reviews for many customers.
> Management of Identity and Trust Keystores:
> Service specific identity and trust keystores will be managed by a
> combination of the HSSO service and CMF.
> Upon registration with the HSSO service a dependent service will be able
> discover externally provisioned keystores or have them created by the HSSO
> service on its behalf. The public key of the HSSO service will be provided to
> the service to be imported into its service specific trust store.
> Service specific keystores and credential stores will be protected with the
> service specific CMF secret.
> Rolling of Keypairs and Credentials:
> The ability to automate the rolling of PKI keypairs and credentials provide
> the services a common facility for discovering new HSSO public keys and the
> need and means to roll their own credentials while being able to retain a
> number of previous values (as needed).
> Discovery of Externally Provisioned Credentials:
> For environments that want control over the certificate generation and
> provisioning, CMF provides the ability to discover preprovisioned artifacts
> based on naming conventions of the artifacts and the use of the service
> specific CMF secret to access the credentials within the keystores.
> Service Specific CMF Secret Protection:
> By providing a common facility to prompt for and optionally persist a service
> specific CMF secret at service installation/startup, we enable the ability to
> protect all the service specific security artifacts with this protected
> secret. It is protected with a combination of AES 128 bit encryption and file
> permissions set for only the service specific OS user.
> Syntax for Aliases within configuration files:
> In order to facilitate the use of aliases but also preserve backward
> compatibility of config files, we will introduce a syntax for marking a value
> in a configuration file as an alias. A getSecret(String value) type utility
> method will encapsulate the recognition and parsing of an alias and the
> retrieval from CMF or return the provided value as the password.
> For instance, if a properties file were to require a password to be provided
> instead of:
> passwd=supersecret
> we would provide an alias as such:
> passwd=${ALIAS=supersecret}
> At runtime, the value from the properties file is provided to the
> CMF.getSecret(value) method and it either resolves the alias (where it finds
> the alias syntax) or returns the value (when there is no alias syntax).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9534) Credential Management Framework (CMF)
[
https://issues.apache.org/jira/browse/HADOOP-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Larry McCay updated HADOOP-9534:
Labels: patch (was: )
Target Version/s: 3.0.0
Affects Version/s: 3.0.0
Release Note: Initial patch for CMF functionality
Status: Patch Available (was: Open)
This is an initial patch for the credential management framework.
> Credential Management Framework (CMF)
> -
>
> Key: HADOOP-9534
> URL: https://issues.apache.org/jira/browse/HADOOP-9534
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: security
>Affects Versions: 3.0.0
>Reporter: Larry McCay
> Labels: patch
> Attachments:
> 0001-HADOOP-9534-Credential-Management-Framework-initial-.patch
>
> Original Estimate: 504h
> Remaining Estimate: 504h
>
> The credential management framework consists of library for securing,
> acquiring and rolling credentials for a given Hadoop service.
> Specifically the library will provide:
> 1. Password Indirection or Aliasing
> 2. Management of identity and trust keystores
> 3. Rolling of key pairs and credentials
> 4. Discovery of externally provisioned credentials
> 5. Service specific CMF secret protection
> 6. Syntax for Aliases within configuration files
> Password Indirection or Aliasing:
> By providing alias based access to actual secrets stored within a service
> specific JCEKS keystore, we are able to eliminate the need for any secret to
> be stored in clear text on the filesystem. This is a current redflag during
> security reviews for many customers.
> Management of Identity and Trust Keystores:
> Service specific identity and trust keystores will be managed by a
> combination of the HSSO service and CMF.
> Upon registration with the HSSO service a dependent service will be able
> discover externally provisioned keystores or have them created by the HSSO
> service on its behalf. The public key of the HSSO service will be provided to
> the service to be imported into its service specific trust store.
> Service specific keystores and credential stores will be protected with the
> service specific CMF secret.
> Rolling of Keypairs and Credentials:
> The ability to automate the rolling of PKI keypairs and credentials provide
> the services a common facility for discovering new HSSO public keys and the
> need and means to roll their own credentials while being able to retain a
> number of previous values (as needed).
> Discovery of Externally Provisioned Credentials:
> For environments that want control over the certificate generation and
> provisioning, CMF provides the ability to discover preprovisioned artifacts
> based on naming conventions of the artifacts and the use of the service
> specific CMF secret to access the credentials within the keystores.
> Service Specific CMF Secret Protection:
> By providing a common facility to prompt for and optionally persist a service
> specific CMF secret at service installation/startup, we enable the ability to
> protect all the service specific security artifacts with this protected
> secret. It is protected with a combination of AES 128 bit encryption and file
> permissions set for only the service specific OS user.
> Syntax for Aliases within configuration files:
> In order to facilitate the use of aliases but also preserve backward
> compatibility of config files, we will introduce a syntax for marking a value
> in a configuration file as an alias. A getSecret(String value) type utility
> method will encapsulate the recognition and parsing of an alias and the
> retrieval from CMF or return the provided value as the password.
> For instance, if a properties file were to require a password to be provided
> instead of:
> passwd=supersecret
> we would provide an alias as such:
> passwd=${ALIAS=supersecret}
> At runtime, the value from the properties file is provided to the
> CMF.getSecret(value) method and it either resolves the alias (where it finds
> the alias syntax) or returns the value (when there is no alias syntax).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9774) RawLocalFileSystem.listStatus() return absolution paths when input path is relative on Windows
[ https://issues.apache.org/jira/browse/HADOOP-9774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721463#comment-13721463 ] shanyu zhao commented on HADOOP-9774: - The reason why I didn't add a unit test is because this behavior cannot be tested on Linux. I'm reusing HADOOP-8962's unit test case to verify that this change doesn't break on Linux. In this specific context, by "relative path" I mean a path without a drive spec on Windows, e.g. "file:///mydata" (instead of "file:///E:/mydata"). I do not refer to a relative path in the sense of "../mypath/". Maybe I should say "Windows relative path". > RawLocalFileSystem.listStatus() return absolution paths when input path is > relative on Windows > -- > > Key: HADOOP-9774 > URL: https://issues.apache.org/jira/browse/HADOOP-9774 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: shanyu zhao > Attachments: HADOOP-9774.patch > > > On Windows, when using RawLocalFileSystem.listStatus() to enumerate a > relative path (without drive spec), e.g., "file:///mydata", the resulting > paths become absolute paths, e.g., ["file://E:/mydata/t1.txt", > "file://E:/mydata/t2.txt"...]. > Note that if we use it to enumerate an absolute path, e.g., > "file://E:/mydata" then the we get the same results as above. > This breaks some hive unit tests which uses local file system to simulate > HDFS when testing, therefore the drive spec is removed. Then after > listStatus() the path is changed to absolute path, hive failed to find the > path in its map reduce job. > You'll see the following exception: > [junit] java.io.IOException: cannot find dir = > pfile:/E:/GitHub/hive-monarch/build/ql/test/data/warehouse/src/kv1.txt in > pathToPartitionInfo: > [pfile:/GitHub/hive-monarch/build/ql/test/data/warehouse/src] > [junit] at > org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:298) > This problem is introduced by this JIRA: > HADOOP-8962 > Prior to the fix for HADOOP-8962 (merged in 0.23.5), the resulting paths are > relative paths if the parent paths are relative, e.g., > ["file:///mydata/t1.txt", "file:///mydata/t2.txt"...] > This behavior change is a side effect of the fix in HADOOP-8962, not an > intended change. The resulting behavior, even though is legitimate from a > function point of view, break consistency from the caller's point of view. > When the caller use a relative path (without drive spec) to do listStatus() > the resulting path should be relative. Therefore, I think this should be > fixed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9781) JWT SSO Token and Authority
Larry McCay created HADOOP-9781: --- Summary: JWT SSO Token and Authority Key: HADOOP-9781 URL: https://issues.apache.org/jira/browse/HADOOP-9781 Project: Hadoop Common Issue Type: Sub-task Components: security Affects Versions: 3.0.0 Reporter: Larry McCay Various token formats exist for enterprise SSO solutions. In order to limit the number of token formats a SSO system for Hadoop needs to process we need to define an initial out of the box format. By transforming an incoming token into a canonical format the Hadoop authentication mechanisms need only be able to understand the SSO token formats. This JIRA is intended to introduce one such format. It is based on the JSON Web Token format - numerous jwt profiles are emerging in the industry - see: http://tools.ietf.org/html/draft-ietf-oauth-jwt-bearer-04 as an example. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9781) JWT SSO Token and Authority
[ https://issues.apache.org/jira/browse/HADOOP-9781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry McCay updated HADOOP-9781: Attachment: 0001-Initial-SSO-module-contribution.patch This is an initial patch for a JWT SSO format and basic authority implementation. > JWT SSO Token and Authority > --- > > Key: HADOOP-9781 > URL: https://issues.apache.org/jira/browse/HADOOP-9781 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: 3.0.0 >Reporter: Larry McCay > Attachments: 0001-Initial-SSO-module-contribution.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > Various token formats exist for enterprise SSO solutions. In order to limit > the number of token formats a SSO system for Hadoop needs to process we need > to define an initial out of the box format. > By transforming an incoming token into a canonical format the Hadoop > authentication mechanisms need only be able to understand the SSO token > formats. > This JIRA is intended to introduce one such format. > It is based on the JSON Web Token format - numerous jwt profiles are emerging > in the industry - see: > http://tools.ietf.org/html/draft-ietf-oauth-jwt-bearer-04 as an example. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9534) Credential Management Framework (CMF)
[
https://issues.apache.org/jira/browse/HADOOP-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721472#comment-13721472
]
Hadoop QA commented on HADOOP-9534:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12594495/0001-HADOOP-9534-Credential-Management-Framework-initial-.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1192 javac
compiler warnings (more than the trunk's current 1151 warnings).
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
24 warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 11 new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-cmf.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2857//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2857//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2857//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-cmf.html
Javac warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2857//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2857//console
This message is automatically generated.
> Credential Management Framework (CMF)
> -
>
> Key: HADOOP-9534
> URL: https://issues.apache.org/jira/browse/HADOOP-9534
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: security
>Affects Versions: 3.0.0
>Reporter: Larry McCay
> Labels: patch
> Attachments:
> 0001-HADOOP-9534-Credential-Management-Framework-initial-.patch
>
> Original Estimate: 504h
> Remaining Estimate: 504h
>
> The credential management framework consists of library for securing,
> acquiring and rolling credentials for a given Hadoop service.
> Specifically the library will provide:
> 1. Password Indirection or Aliasing
> 2. Management of identity and trust keystores
> 3. Rolling of key pairs and credentials
> 4. Discovery of externally provisioned credentials
> 5. Service specific CMF secret protection
> 6. Syntax for Aliases within configuration files
> Password Indirection or Aliasing:
> By providing alias based access to actual secrets stored within a service
> specific JCEKS keystore, we are able to eliminate the need for any secret to
> be stored in clear text on the filesystem. This is a current redflag during
> security reviews for many customers.
> Management of Identity and Trust Keystores:
> Service specific identity and trust keystores will be managed by a
> combination of the HSSO service and CMF.
> Upon registration with the HSSO service a dependent service will be able
> discover externally provisioned keystores or have them created by the HSSO
> service on its behalf. The public key of the HSSO service will be provided to
> the service to be imported into its service specific trust store.
> Service specific keystores and credential stores will be protected with the
> service specific CMF secret.
> Rolling of Keypairs and Credentials:
> The ability to automate the rolling of PKI keypairs and credentials provide
> the services a common facility for discovering new HSSO public keys and the
> need and means to roll their own credentials while being able to retain a
> number of previous values (as needed).
> Discovery of Externally Provisioned Credentials:
> For environments that want control over the certificate generation and
> provisioning, CMF provides the ability to discover preprovisioned artifacts
> based on naming conventions of the artifacts and the use of the service
> specific CMF secret to access the credentials within the keystores.
> Service Specific CMF Secret Protection:
> By providing a common facility to prompt for and optionally persist a service
> specific CMF secret at service installation/startup, we enable the ability to
> protect all the service specific security artifacts with this protected
> secret. It is protected with a combination of AES 128 bit encryption and file
> permissions set for only the service specific OS user
[jira] [Commented] (HADOOP-9534) Credential Management Framework (CMF)
[
https://issues.apache.org/jira/browse/HADOOP-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721483#comment-13721483
]
Larry McCay commented on HADOOP-9534:
-
I will investigate the QA related issues.
> Credential Management Framework (CMF)
> -
>
> Key: HADOOP-9534
> URL: https://issues.apache.org/jira/browse/HADOOP-9534
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: security
>Affects Versions: 3.0.0
>Reporter: Larry McCay
> Labels: patch
> Attachments:
> 0001-HADOOP-9534-Credential-Management-Framework-initial-.patch
>
> Original Estimate: 504h
> Remaining Estimate: 504h
>
> The credential management framework consists of library for securing,
> acquiring and rolling credentials for a given Hadoop service.
> Specifically the library will provide:
> 1. Password Indirection or Aliasing
> 2. Management of identity and trust keystores
> 3. Rolling of key pairs and credentials
> 4. Discovery of externally provisioned credentials
> 5. Service specific CMF secret protection
> 6. Syntax for Aliases within configuration files
> Password Indirection or Aliasing:
> By providing alias based access to actual secrets stored within a service
> specific JCEKS keystore, we are able to eliminate the need for any secret to
> be stored in clear text on the filesystem. This is a current redflag during
> security reviews for many customers.
> Management of Identity and Trust Keystores:
> Service specific identity and trust keystores will be managed by a
> combination of the HSSO service and CMF.
> Upon registration with the HSSO service a dependent service will be able
> discover externally provisioned keystores or have them created by the HSSO
> service on its behalf. The public key of the HSSO service will be provided to
> the service to be imported into its service specific trust store.
> Service specific keystores and credential stores will be protected with the
> service specific CMF secret.
> Rolling of Keypairs and Credentials:
> The ability to automate the rolling of PKI keypairs and credentials provide
> the services a common facility for discovering new HSSO public keys and the
> need and means to roll their own credentials while being able to retain a
> number of previous values (as needed).
> Discovery of Externally Provisioned Credentials:
> For environments that want control over the certificate generation and
> provisioning, CMF provides the ability to discover preprovisioned artifacts
> based on naming conventions of the artifacts and the use of the service
> specific CMF secret to access the credentials within the keystores.
> Service Specific CMF Secret Protection:
> By providing a common facility to prompt for and optionally persist a service
> specific CMF secret at service installation/startup, we enable the ability to
> protect all the service specific security artifacts with this protected
> secret. It is protected with a combination of AES 128 bit encryption and file
> permissions set for only the service specific OS user.
> Syntax for Aliases within configuration files:
> In order to facilitate the use of aliases but also preserve backward
> compatibility of config files, we will introduce a syntax for marking a value
> in a configuration file as an alias. A getSecret(String value) type utility
> method will encapsulate the recognition and parsing of an alias and the
> retrieval from CMF or return the provided value as the password.
> For instance, if a properties file were to require a password to be provided
> instead of:
> passwd=supersecret
> we would provide an alias as such:
> passwd=${ALIAS=supersecret}
> At runtime, the value from the properties file is provided to the
> CMF.getSecret(value) method and it either resolves the alias (where it finds
> the alias syntax) or returns the value (when there is no alias syntax).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9756) Additional cleanup RPC code
[ https://issues.apache.org/jira/browse/HADOOP-9756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HADOOP-9756: --- Attachment: HADOOP-9756-branch2.patch Hi Nicholas, I put prerequisite patches on HDFS-3880 and YARN-84 (previously as MAPREDUCE-4628). After that two patches go in, we can apply this branch2 patch. > Additional cleanup RPC code > --- > > Key: HADOOP-9756 > URL: https://issues.apache.org/jira/browse/HADOOP-9756 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Junping Du >Assignee: Junping Du >Priority: Minor > Attachments: HADOOP-9756-branch2.patch, HADOOP-9756.patch, > HADOOP-9756-v2.patch > > > HADOOP-9754 already did good job to address most of work for cleanup RPC > code. Here is some additional work, include: > - Remove some unused deprecated code. > - Narrow "throws Exception" to throw some specific exception and remove some > unnecessary exceptions > - Fix a generic warning and correct spell issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9756) Additional cleanup RPC code
[
https://issues.apache.org/jira/browse/HADOOP-9756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721492#comment-13721492
]
Hadoop QA commented on HADOOP-9756:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12594503/HADOOP-9756-branch2.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2858//console
This message is automatically generated.
> Additional cleanup RPC code
> ---
>
> Key: HADOOP-9756
> URL: https://issues.apache.org/jira/browse/HADOOP-9756
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
>Reporter: Junping Du
>Assignee: Junping Du
>Priority: Minor
> Attachments: HADOOP-9756-branch2.patch, HADOOP-9756.patch,
> HADOOP-9756-v2.patch
>
>
> HADOOP-9754 already did good job to address most of work for cleanup RPC
> code. Here is some additional work, include:
> - Remove some unused deprecated code.
> - Narrow "throws Exception" to throw some specific exception and remove some
> unnecessary exceptions
> - Fix a generic warning and correct spell issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9164) Print paths of loaded native libraries in NativeLibraryChecker
[ https://issues.apache.org/jira/browse/HADOOP-9164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721504#comment-13721504 ] Binglin Chang commented on HADOOP-9164: --- Sorry, LZO is not in trunk code base so I can't add related code to LZO, I think it's because LZO is GPL. > Print paths of loaded native libraries in NativeLibraryChecker > -- > > Key: HADOOP-9164 > URL: https://issues.apache.org/jira/browse/HADOOP-9164 > Project: Hadoop Common > Issue Type: Improvement > Components: native >Affects Versions: 2.0.2-alpha >Reporter: Binglin Chang >Assignee: Binglin Chang >Priority: Minor > Fix For: 2.1.0-beta > > Attachments: HADOOP-9164.v1.patch, HADOOP-9164.v2.patch, > HADOOP-9164.v3.patch, HADOOP-9164.v4.2.patch, HADOOP-9164.v4.patch, > HADOOP-9164.v4.patch, HADOOP-9164.v5.patch, HADOOP-9164.v6.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
