[jira] [Commented] (HADOOP-15338) Support Java 11 LTS in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-15338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902490#comment-16902490 ] Matt Foley commented on HADOOP-15338: - MCOMPILER-342 shows fixed for the compiler plugin in version 3.8.0. But the jira notes other parts of the maven system (specifically maven-plugin-plugin) are not there yet. > Support Java 11 LTS in Hadoop > - > > Key: HADOOP-15338 > URL: https://issues.apache.org/jira/browse/HADOOP-15338 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Akira Ajisaka >Priority: Major > > Oracle JDK 8 will be EoL during January 2019, and RedHat will end support for > OpenJDK 8 in June 2023 ([https://access.redhat.com/articles/1299013]), so we > need to support Java 11 LTS at least before June 2023. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16166) TestRawLocalFileSystemContract fails with build Docker container running on Mac
[ https://issues.apache.org/jira/browse/HADOOP-16166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley resolved HADOOP-16166. - Resolution: Fixed > TestRawLocalFileSystemContract fails with build Docker container running on > Mac > --- > > Key: HADOOP-16166 > URL: https://issues.apache.org/jira/browse/HADOOP-16166 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.3.0 >Reporter: Matt Foley >Assignee: Matt Foley >Priority: Minor > > The Mac has a case-insensitive filesystem. When using the recommended build > Docker container via `start-build-env.sh`, the container attaches to the Mac > FS to share the local git repository for Hadoop. Which is very nice and > convenient. > This means the TestRawLocalFileSystemContract#testFilesystemIsCaseSensitive() > test case (which is inherited from FileSystemContractBaseTest) should be > skipped. It fails to be skipped, and therefore throws a Unit Test failure, > because @Override TestRawLocalFileSystemContract#filesystemIsCaseSensitive() > does not take into account the possibility of a Linux OS mounting a MacOS > filesystem. > The fix would extend > TestRawLocalFileSystemContract#filesystemIsCaseSensitive() to recognize this > case. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16109) Parquet reading S3AFileSystem causes EOF
[ https://issues.apache.org/jira/browse/HADOOP-16109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791901#comment-16791901 ] Matt Foley commented on HADOOP-16109: - Thanks very much [~ste...@apache.org]! > Parquet reading S3AFileSystem causes EOF > > > Key: HADOOP-16109 > URL: https://issues.apache.org/jira/browse/HADOOP-16109 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.2, 2.8.5, 3.3.0, 3.1.2 >Reporter: Dave Christianson >Assignee: Steve Loughran >Priority: Blocker > Fix For: 2.10.0, 3.0.4, 3.3.0, 2.8.6, 3.2.1, 2.9.3, 3.1.3 > > Attachments: HADOOP-16109-branch-3.1-003.patch > > > When using S3AFileSystem to read Parquet files a specific set of > circumstances causes an EOFException that is not thrown when reading the > same file from local disk > Note this has only been observed under specific circumstances: > - when the reader is doing a projection (will cause it to do a seek > backwards and put the filesystem into random mode) > - when the file is larger than the readahead buffer size > - when the seek behavior of the Parquet reader causes the reader to seek > towards the end of the current input stream without reopening, such that the > next read on the currently open stream will read past the end of the > currently open stream. > Exception from Parquet reader is as follows: > {code} > Caused by: java.io.EOFException: Reached the end of stream with 51 bytes left > to read > at > org.apache.parquet.io.DelegatingSeekableInputStream.readFully(DelegatingSeekableInputStream.java:104) > at > org.apache.parquet.io.DelegatingSeekableInputStream.readFullyHeapBuffer(DelegatingSeekableInputStream.java:127) > at > org.apache.parquet.io.DelegatingSeekableInputStream.readFully(DelegatingSeekableInputStream.java:91) > at > org.apache.parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:1174) > at > org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:805) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:127) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:222) > at > org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:207) > at > org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.fetchNext(HadoopInputFormatBase.java:206) > at > org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.reachedEnd(HadoopInputFormatBase.java:199) > at > org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:190) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711) > at java.lang.Thread.run(Thread.java:748) > {code} > The following example program generate the same root behavior (sans finding a > Parquet file that happens to trigger this condition) by purposely reading > past the already active readahead range on any file >= 1029 bytes in size.. > {code:java} > final Configuration conf = new Configuration(); > conf.set("fs.s3a.readahead.range", "1K"); > conf.set("fs.s3a.experimental.input.fadvise", "random"); > final FileSystem fs = FileSystem.get(path.toUri(), conf); > // forward seek reading across readahead boundary > try (FSDataInputStream in = fs.open(path)) { > final byte[] temp = new byte[5]; > in.readByte(); > in.readFully(1023, temp); // <-- works > } > // forward seek reading from end of readahead boundary > try (FSDataInputStream in = fs.open(path)) { > final byte[] temp = new byte[5]; > in.readByte(); > in.readFully(1024, temp); // <-- throws EOFException > } > {code} > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16166) TestRawLocalFileSystemContract fails with build Docker container running on Mac
[ https://issues.apache.org/jira/browse/HADOOP-16166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16788223#comment-16788223 ] Matt Foley edited comment on HADOOP-16166 at 3/8/19 8:17 PM: - Submitted fix as PR# 580 Included similar fix for Windows. was (Author: mattf): Submitted fix as PR# 580 > TestRawLocalFileSystemContract fails with build Docker container running on > Mac > --- > > Key: HADOOP-16166 > URL: https://issues.apache.org/jira/browse/HADOOP-16166 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.3.0 >Reporter: Matt Foley >Assignee: Matt Foley >Priority: Minor > > The Mac has a case-insensitive filesystem. When using the recommended build > Docker container via `start-build-env.sh`, the container attaches to the Mac > FS to share the local git repository for Hadoop. Which is very nice and > convenient. > This means the TestRawLocalFileSystemContract#testFilesystemIsCaseSensitive() > test case (which is inherited from FileSystemContractBaseTest) should be > skipped. It fails to be skipped, and therefore throws a Unit Test failure, > because @Override TestRawLocalFileSystemContract#filesystemIsCaseSensitive() > does not take into account the possibility of a Linux OS mounting a MacOS > filesystem. > The fix would extend > TestRawLocalFileSystemContract#filesystemIsCaseSensitive() to recognize this > case. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16166) TestRawLocalFileSystemContract fails with build Docker container running on Mac
[ https://issues.apache.org/jira/browse/HADOOP-16166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley reassigned HADOOP-16166: --- Assignee: Matt Foley > TestRawLocalFileSystemContract fails with build Docker container running on > Mac > --- > > Key: HADOOP-16166 > URL: https://issues.apache.org/jira/browse/HADOOP-16166 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.3.0 >Reporter: Matt Foley >Assignee: Matt Foley >Priority: Minor > > The Mac has a case-insensitive filesystem. When using the recommended build > Docker container via `start-build-env.sh`, the container attaches to the Mac > FS to share the local git repository for Hadoop. Which is very nice and > convenient. > This means the TestRawLocalFileSystemContract#testFilesystemIsCaseSensitive() > test case (which is inherited from FileSystemContractBaseTest) should be > skipped. It fails to be skipped, and therefore throws a Unit Test failure, > because @Override TestRawLocalFileSystemContract#filesystemIsCaseSensitive() > does not take into account the possibility of a Linux OS mounting a MacOS > filesystem. > The fix would extend > TestRawLocalFileSystemContract#filesystemIsCaseSensitive() to recognize this > case. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16166) TestRawLocalFileSystemContract fails with build Docker container running on Mac
[ https://issues.apache.org/jira/browse/HADOOP-16166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16788223#comment-16788223 ] Matt Foley commented on HADOOP-16166: - Submitted fix as PR# 580 > TestRawLocalFileSystemContract fails with build Docker container running on > Mac > --- > > Key: HADOOP-16166 > URL: https://issues.apache.org/jira/browse/HADOOP-16166 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.3.0 >Reporter: Matt Foley >Priority: Minor > > The Mac has a case-insensitive filesystem. When using the recommended build > Docker container via `start-build-env.sh`, the container attaches to the Mac > FS to share the local git repository for Hadoop. Which is very nice and > convenient. > This means the TestRawLocalFileSystemContract#testFilesystemIsCaseSensitive() > test case (which is inherited from FileSystemContractBaseTest) should be > skipped. It fails to be skipped, and therefore throws a Unit Test failure, > because @Override TestRawLocalFileSystemContract#filesystemIsCaseSensitive() > does not take into account the possibility of a Linux OS mounting a MacOS > filesystem. > The fix would extend > TestRawLocalFileSystemContract#filesystemIsCaseSensitive() to recognize this > case. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16109) Parquet reading S3AFileSystem causes EOF
[ https://issues.apache.org/jira/browse/HADOOP-16109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16780826#comment-16780826 ] Matt Foley edited comment on HADOOP-16109 at 3/7/19 12:53 AM: -- Hi [~ste...@apache.org], one of my colleagues, Shruti Gumma, has a proposed fix which I'll help him post here today. Edit: Shruti is unable to test on S3, so requests you proceed with your fix. was (Author: mattf): Hi [~ste...@apache.org], one of my colleagues, Shruti Gumma, has a proposed fix which I'll help him post here today. > Parquet reading S3AFileSystem causes EOF > > > Key: HADOOP-16109 > URL: https://issues.apache.org/jira/browse/HADOOP-16109 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.2, 2.8.5, 3.3.0, 3.1.2 >Reporter: Dave Christianson >Assignee: Steve Loughran >Priority: Blocker > > When using S3AFileSystem to read Parquet files a specific set of > circumstances causes an EOFException that is not thrown when reading the > same file from local disk > Note this has only been observed under specific circumstances: > - when the reader is doing a projection (will cause it to do a seek > backwards and put the filesystem into random mode) > - when the file is larger than the readahead buffer size > - when the seek behavior of the Parquet reader causes the reader to seek > towards the end of the current input stream without reopening, such that the > next read on the currently open stream will read past the end of the > currently open stream. > Exception from Parquet reader is as follows: > {code} > Caused by: java.io.EOFException: Reached the end of stream with 51 bytes left > to read > at > org.apache.parquet.io.DelegatingSeekableInputStream.readFully(DelegatingSeekableInputStream.java:104) > at > org.apache.parquet.io.DelegatingSeekableInputStream.readFullyHeapBuffer(DelegatingSeekableInputStream.java:127) > at > org.apache.parquet.io.DelegatingSeekableInputStream.readFully(DelegatingSeekableInputStream.java:91) > at > org.apache.parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:1174) > at > org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:805) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:127) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:222) > at > org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:207) > at > org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.fetchNext(HadoopInputFormatBase.java:206) > at > org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.reachedEnd(HadoopInputFormatBase.java:199) > at > org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:190) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711) > at java.lang.Thread.run(Thread.java:748) > {code} > The following example program generate the same root behavior (sans finding a > Parquet file that happens to trigger this condition) by purposely reading > past the already active readahead range on any file >= 1029 bytes in size.. > {code:java} > final Configuration conf = new Configuration(); > conf.set("fs.s3a.readahead.range", "1K"); > conf.set("fs.s3a.experimental.input.fadvise", "random"); > final FileSystem fs = FileSystem.get(path.toUri(), conf); > // forward seek reading across readahead boundary > try (FSDataInputStream in = fs.open(path)) { > final byte[] temp = new byte[5]; > in.readByte(); > in.readFully(1023, temp); // <-- works > } > // forward seek reading from end of readahead boundary > try (FSDataInputStream in = fs.open(path)) { > final byte[] temp = new byte[5]; > in.readByte(); > in.readFully(1024, temp); // <-- throws EOFException > } > {code} > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16166) TestRawLocalFileSystemContract fails with build Docker container running on Mac
[ https://issues.apache.org/jira/browse/HADOOP-16166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783920#comment-16783920 ] Matt Foley commented on HADOOP-16166: - Yes, which is exactly what testFilesystemIsCaseSensitive() tests :D That would get a little tautological... Probably best to stick with current model of identifying which systems we expect to be case-(in)sensitive, or not. I'm putting together a patch using `df .` which returns "osxfs" in the first field, for the case of interest. > TestRawLocalFileSystemContract fails with build Docker container running on > Mac > --- > > Key: HADOOP-16166 > URL: https://issues.apache.org/jira/browse/HADOOP-16166 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.3.0 >Reporter: Matt Foley >Priority: Minor > > The Mac has a case-insensitive filesystem. When using the recommended build > Docker container via `start-build-env.sh`, the container attaches to the Mac > FS to share the local git repository for Hadoop. Which is very nice and > convenient. > This means the TestRawLocalFileSystemContract#testFilesystemIsCaseSensitive() > test case (which is inherited from FileSystemContractBaseTest) should be > skipped. It fails to be skipped, and therefore throws a Unit Test failure, > because @Override TestRawLocalFileSystemContract#filesystemIsCaseSensitive() > does not take into account the possibility of a Linux OS mounting a MacOS > filesystem. > The fix would extend > TestRawLocalFileSystemContract#filesystemIsCaseSensitive() to recognize this > case. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16166) TestRawLocalFileSystemContract fails with build Docker container running on Mac
[ https://issues.apache.org/jira/browse/HADOOP-16166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-16166: Description: The Mac has a case-insensitive filesystem. When using the recommended build Docker container via `start-build-env.sh`, the container attaches to the Mac FS to share the local git repository for Hadoop. Which is very nice and convenient. This means the TestRawLocalFileSystemContract#testFilesystemIsCaseSensitive() test case (which is inherited from FileSystemContractBaseTest) should be skipped. It fails to be skipped, and therefore throws a Unit Test failure, because @Override TestRawLocalFileSystemContract#filesystemIsCaseSensitive() does not take into account the possibility of a Linux OS mounting a MacOS filesystem. The fix would extend TestRawLocalFileSystemContract#filesystemIsCaseSensitive() to recognize this case. was: The Mac has a case-insensitive filesystem. When using the recommended build Docker container via `start-build-env.sh`, the container attaches to the Mac FS to share the local git repository for Hadoop. Which is very nice and convenient. This means the TestRawLocalFileSystemContract::testFilesystemIsCaseSensitive() test case (which is inherited from FileSystemContractBaseTest) should be skipped. It fails to be skipped, and therefore throws a Unit Test failure, because @Override TestRawLocalFileSystemContract::filesystemIsCaseSensitive() does not take into account the possibility of a Linux OS mounting a MacOS filesystem. The fix would extend TestRawLocalFileSystemContract::filesystemIsCaseSensitive() to recognize this case. > TestRawLocalFileSystemContract fails with build Docker container running on > Mac > --- > > Key: HADOOP-16166 > URL: https://issues.apache.org/jira/browse/HADOOP-16166 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.3.0 >Reporter: Matt Foley >Priority: Minor > > The Mac has a case-insensitive filesystem. When using the recommended build > Docker container via `start-build-env.sh`, the container attaches to the Mac > FS to share the local git repository for Hadoop. Which is very nice and > convenient. > This means the TestRawLocalFileSystemContract#testFilesystemIsCaseSensitive() > test case (which is inherited from FileSystemContractBaseTest) should be > skipped. It fails to be skipped, and therefore throws a Unit Test failure, > because @Override TestRawLocalFileSystemContract#filesystemIsCaseSensitive() > does not take into account the possibility of a Linux OS mounting a MacOS > filesystem. > The fix would extend > TestRawLocalFileSystemContract#filesystemIsCaseSensitive() to recognize this > case. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16166) TestRawLocalFileSystemContract fails with build Docker container running on Mac
Matt Foley created HADOOP-16166: --- Summary: TestRawLocalFileSystemContract fails with build Docker container running on Mac Key: HADOOP-16166 URL: https://issues.apache.org/jira/browse/HADOOP-16166 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 3.3.0 Reporter: Matt Foley The Mac has a case-insensitive filesystem. When using the recommended build Docker container via `start-build-env.sh`, the container attaches to the Mac FS to share the local git repository for Hadoop. Which is very nice and convenient. This means the TestRawLocalFileSystemContract::testFilesystemIsCaseSensitive() test case (which is inherited from FileSystemContractBaseTest) should be skipped. It fails to be skipped, and therefore throws a Unit Test failure, because @Override TestRawLocalFileSystemContract::filesystemIsCaseSensitive() does not take into account the possibility of a Linux OS mounting a MacOS filesystem. The fix would extend TestRawLocalFileSystemContract::filesystemIsCaseSensitive() to recognize this case. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16109) Parquet reading S3AFileSystem causes EOF
[ https://issues.apache.org/jira/browse/HADOOP-16109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781137#comment-16781137 ] Matt Foley edited comment on HADOOP-16109 at 3/1/19 12:49 AM: -- Yes, I'm thinking at [https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L264] we need {{&& diff < forwardSeekLimit;}} instead of {{<=}} What do you think? The big question I have is, some of the text description talks about "reading past the already active readahead range", i.e., past {{remainingInCurrentRequest}}, as being a problem, but it seems to me that should be okay; the problem documented so far is *seeking* past {{remainingInCurrentRequest}} (specifically to exactly the end of CurrentRequest, which is incorrectly guarded by the above L264 inequality) and then not doing a stream close, which causes the problem. Do you know if, say, seeking to a few bytes before the end of CurrentRequest, then reading past it (when the S3 file does indeed have more to read), also causes an EOF, or does the stream machinery handle that case correctly? I'm putting together a test platform so I can answer such questions myself, but it will take me a few hours; I haven't worked in s3a before. was (Author: mattf): Yes, I'm thinking at [https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L264] we need {{ && diff < forwardSeekLimit; }} instead of {{ <= }} What do you think? The big question I have is, some of the text description talks about "reading past the already active readahead range", i.e., past {{remainingInCurrentRequest}}, as being a problem, but it seems to me that should be okay; the problem documented so far is *seeking* past {{remainingInCurrentRequest}} (specifically to exactly the end of CurrentRequest, which is incorrectly guarded by the above L264 inequality) and then not doing a stream close, which causes the problem. Do you know if, say, seeking to a few bytes before the end of CurrentRequest, then reading past it (when the S3 file does indeed have more to read), also causes an EOF, or does the stream machinery handle that case correctly? I'm putting together a test platform so I can answer such questions myself, but it will take me a few hours; I haven't worked in s3a before. > Parquet reading S3AFileSystem causes EOF > > > Key: HADOOP-16109 > URL: https://issues.apache.org/jira/browse/HADOOP-16109 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.2, 2.8.5, 3.3.0, 3.1.2 >Reporter: Dave Christianson >Assignee: Steve Loughran >Priority: Blocker > > When using S3AFileSystem to read Parquet files a specific set of > circumstances causes an EOFException that is not thrown when reading the > same file from local disk > Note this has only been observed under specific circumstances: > - when the reader is doing a projection (will cause it to do a seek > backwards and put the filesystem into random mode) > - when the file is larger than the readahead buffer size > - when the seek behavior of the Parquet reader causes the reader to seek > towards the end of the current input stream without reopening, such that the > next read on the currently open stream will read past the end of the > currently open stream. > Exception from Parquet reader is as follows: > {code} > Caused by: java.io.EOFException: Reached the end of stream with 51 bytes left > to read > at > org.apache.parquet.io.DelegatingSeekableInputStream.readFully(DelegatingSeekableInputStream.java:104) > at > org.apache.parquet.io.DelegatingSeekableInputStream.readFullyHeapBuffer(DelegatingSeekableInputStream.java:127) > at > org.apache.parquet.io.DelegatingSeekableInputStream.readFully(DelegatingSeekableInputStream.java:91) > at > org.apache.parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:1174) > at > org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:805) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:127) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:222) > at > org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:207) > at > org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.fetchNext(HadoopInputFormatBase.java:206) > at > org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.reachedEnd(HadoopInputFormatBase.java:199) > at > org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:190) > at
[jira] [Comment Edited] (HADOOP-16109) Parquet reading S3AFileSystem causes EOF
[ https://issues.apache.org/jira/browse/HADOOP-16109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781137#comment-16781137 ] Matt Foley edited comment on HADOOP-16109 at 3/1/19 12:50 AM: -- Yes, I'm thinking at [https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L264] we need {{&& diff < forwardSeekLimit; // instead of <=}} What do you think? The big question I have is, some of the text description talks about "reading past the already active readahead range", i.e., past {{remainingInCurrentRequest}}, as being a problem, but it seems to me that should be okay; the problem documented so far is *seeking* past {{remainingInCurrentRequest}} (specifically to exactly the end of CurrentRequest, which is incorrectly guarded by the above L264 inequality) and then not doing a stream close, which causes the problem. Do you know if, say, seeking to a few bytes before the end of CurrentRequest, then reading past it (when the S3 file does indeed have more to read), also causes an EOF, or does the stream machinery handle that case correctly? I'm putting together a test platform so I can answer such questions myself, but it will take me a few hours; I haven't worked in s3a before. was (Author: mattf): Yes, I'm thinking at [https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L264] we need {{&& diff < forwardSeekLimit;}} instead of {{<=}} What do you think? The big question I have is, some of the text description talks about "reading past the already active readahead range", i.e., past {{remainingInCurrentRequest}}, as being a problem, but it seems to me that should be okay; the problem documented so far is *seeking* past {{remainingInCurrentRequest}} (specifically to exactly the end of CurrentRequest, which is incorrectly guarded by the above L264 inequality) and then not doing a stream close, which causes the problem. Do you know if, say, seeking to a few bytes before the end of CurrentRequest, then reading past it (when the S3 file does indeed have more to read), also causes an EOF, or does the stream machinery handle that case correctly? I'm putting together a test platform so I can answer such questions myself, but it will take me a few hours; I haven't worked in s3a before. > Parquet reading S3AFileSystem causes EOF > > > Key: HADOOP-16109 > URL: https://issues.apache.org/jira/browse/HADOOP-16109 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.2, 2.8.5, 3.3.0, 3.1.2 >Reporter: Dave Christianson >Assignee: Steve Loughran >Priority: Blocker > > When using S3AFileSystem to read Parquet files a specific set of > circumstances causes an EOFException that is not thrown when reading the > same file from local disk > Note this has only been observed under specific circumstances: > - when the reader is doing a projection (will cause it to do a seek > backwards and put the filesystem into random mode) > - when the file is larger than the readahead buffer size > - when the seek behavior of the Parquet reader causes the reader to seek > towards the end of the current input stream without reopening, such that the > next read on the currently open stream will read past the end of the > currently open stream. > Exception from Parquet reader is as follows: > {code} > Caused by: java.io.EOFException: Reached the end of stream with 51 bytes left > to read > at > org.apache.parquet.io.DelegatingSeekableInputStream.readFully(DelegatingSeekableInputStream.java:104) > at > org.apache.parquet.io.DelegatingSeekableInputStream.readFullyHeapBuffer(DelegatingSeekableInputStream.java:127) > at > org.apache.parquet.io.DelegatingSeekableInputStream.readFully(DelegatingSeekableInputStream.java:91) > at > org.apache.parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:1174) > at > org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:805) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:127) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:222) > at > org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:207) > at > org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.fetchNext(HadoopInputFormatBase.java:206) > at > org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.reachedEnd(HadoopInputFormatBase.java:199) > at > org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:190) > at
[jira] [Comment Edited] (HADOOP-16109) Parquet reading S3AFileSystem causes EOF
[ https://issues.apache.org/jira/browse/HADOOP-16109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781137#comment-16781137 ] Matt Foley edited comment on HADOOP-16109 at 3/1/19 12:48 AM: -- Yes, I'm thinking at [https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L264] we need {{ && diff < forwardSeekLimit; }} instead of {{ <= }} What do you think? The big question I have is, some of the text description talks about "reading past the already active readahead range", i.e., past {{remainingInCurrentRequest}}, as being a problem, but it seems to me that should be okay; the problem documented so far is *seeking* past {{remainingInCurrentRequest}} (specifically to exactly the end of CurrentRequest, which is incorrectly guarded by the above L264 inequality) and then not doing a stream close, which causes the problem. Do you know if, say, seeking to a few bytes before the end of CurrentRequest, then reading past it (when the S3 file does indeed have more to read), also causes an EOF, or does the stream machinery handle that case correctly? I'm putting together a test platform so I can answer such questions myself, but it will take me a few hours; I haven't worked in s3a before. was (Author: mattf): Yes, I'm thinking at [https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L264] we need `&& diff < forwardSeekLimit;` instead of `<=` What do you think? The big question I have is, some of the text description talks about "reading past the already active readahead range", i.e., past `remainingInCurrentRequest`, as being a problem, but it seems to me that should be okay; the problem documented so far is *seeking* past `remainingInCurrentRequest` (specifically to exactly the end of CurrentRequest, which is incorrectly guarded by the above L264 inequality) and then not doing a stream close, which causes the problem. Do you know if, say, seeking to a few bytes before the end of CurrentRequest, then reading past it (when the S3 file does indeed have more to read), also causes an EOF, or does the stream machinery handle that case correctly? I'm putting together a test platform so I can answer such questions myself, but it will take me a few hours; I haven't worked in s3a before. > Parquet reading S3AFileSystem causes EOF > > > Key: HADOOP-16109 > URL: https://issues.apache.org/jira/browse/HADOOP-16109 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.2, 2.8.5, 3.3.0, 3.1.2 >Reporter: Dave Christianson >Assignee: Steve Loughran >Priority: Blocker > > When using S3AFileSystem to read Parquet files a specific set of > circumstances causes an EOFException that is not thrown when reading the > same file from local disk > Note this has only been observed under specific circumstances: > - when the reader is doing a projection (will cause it to do a seek > backwards and put the filesystem into random mode) > - when the file is larger than the readahead buffer size > - when the seek behavior of the Parquet reader causes the reader to seek > towards the end of the current input stream without reopening, such that the > next read on the currently open stream will read past the end of the > currently open stream. > Exception from Parquet reader is as follows: > {code} > Caused by: java.io.EOFException: Reached the end of stream with 51 bytes left > to read > at > org.apache.parquet.io.DelegatingSeekableInputStream.readFully(DelegatingSeekableInputStream.java:104) > at > org.apache.parquet.io.DelegatingSeekableInputStream.readFullyHeapBuffer(DelegatingSeekableInputStream.java:127) > at > org.apache.parquet.io.DelegatingSeekableInputStream.readFully(DelegatingSeekableInputStream.java:91) > at > org.apache.parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:1174) > at > org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:805) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:127) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:222) > at > org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:207) > at > org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.fetchNext(HadoopInputFormatBase.java:206) > at > org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.reachedEnd(HadoopInputFormatBase.java:199) > at > org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:190) > at
[jira] [Commented] (HADOOP-16109) Parquet reading S3AFileSystem causes EOF
[ https://issues.apache.org/jira/browse/HADOOP-16109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781137#comment-16781137 ] Matt Foley commented on HADOOP-16109: - Yes, I'm thinking at [https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L264] we need `&& diff < forwardSeekLimit;` instead of `<=` What do you think? The big question I have is, some of the text description talks about "reading past the already active readahead range", i.e., past `remainingInCurrentRequest`, as being a problem, but it seems to me that should be okay; the problem documented so far is *seeking* past `remainingInCurrentRequest` (specifically to exactly the end of CurrentRequest, which is incorrectly guarded by the above L264 inequality) and then not doing a stream close, which causes the problem. Do you know if, say, seeking to a few bytes before the end of CurrentRequest, then reading past it (when the S3 file does indeed have more to read), also causes an EOF, or does the stream machinery handle that case correctly? I'm putting together a test platform so I can answer such questions myself, but it will take me a few hours; I haven't worked in s3a before. > Parquet reading S3AFileSystem causes EOF > > > Key: HADOOP-16109 > URL: https://issues.apache.org/jira/browse/HADOOP-16109 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.2, 2.8.5, 3.3.0, 3.1.2 >Reporter: Dave Christianson >Assignee: Steve Loughran >Priority: Blocker > > When using S3AFileSystem to read Parquet files a specific set of > circumstances causes an EOFException that is not thrown when reading the > same file from local disk > Note this has only been observed under specific circumstances: > - when the reader is doing a projection (will cause it to do a seek > backwards and put the filesystem into random mode) > - when the file is larger than the readahead buffer size > - when the seek behavior of the Parquet reader causes the reader to seek > towards the end of the current input stream without reopening, such that the > next read on the currently open stream will read past the end of the > currently open stream. > Exception from Parquet reader is as follows: > {code} > Caused by: java.io.EOFException: Reached the end of stream with 51 bytes left > to read > at > org.apache.parquet.io.DelegatingSeekableInputStream.readFully(DelegatingSeekableInputStream.java:104) > at > org.apache.parquet.io.DelegatingSeekableInputStream.readFullyHeapBuffer(DelegatingSeekableInputStream.java:127) > at > org.apache.parquet.io.DelegatingSeekableInputStream.readFully(DelegatingSeekableInputStream.java:91) > at > org.apache.parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:1174) > at > org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:805) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:127) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:222) > at > org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:207) > at > org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.fetchNext(HadoopInputFormatBase.java:206) > at > org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.reachedEnd(HadoopInputFormatBase.java:199) > at > org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:190) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711) > at java.lang.Thread.run(Thread.java:748) > {code} > The following example program generate the same root behavior (sans finding a > Parquet file that happens to trigger this condition) by purposely reading > past the already active readahead range on any file >= 1029 bytes in size.. > {code:java} > final Configuration conf = new Configuration(); > conf.set("fs.s3a.readahead.range", "1K"); > conf.set("fs.s3a.experimental.input.fadvise", "random"); > final FileSystem fs = FileSystem.get(path.toUri(), conf); > // forward seek reading across readahead boundary > try (FSDataInputStream in = fs.open(path)) { > final byte[] temp = new byte[5]; > in.readByte(); > in.readFully(1023, temp); // <-- works > } > // forward seek reading from end of readahead boundary > try (FSDataInputStream in = fs.open(path)) { > final byte[] temp = new byte[5]; > in.readByte(); > in.readFully(1024, temp); // <-- throws EOFException > } > {code} > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail:
[jira] [Commented] (HADOOP-16109) Parquet reading S3AFileSystem causes EOF
[ https://issues.apache.org/jira/browse/HADOOP-16109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16780826#comment-16780826 ] Matt Foley commented on HADOOP-16109: - Hi [~ste...@apache.org], one of my colleagues, Shruti Gumma, has a proposed fix which I'll help him post here today. > Parquet reading S3AFileSystem causes EOF > > > Key: HADOOP-16109 > URL: https://issues.apache.org/jira/browse/HADOOP-16109 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Dave Christianson >Assignee: Steve Loughran >Priority: Blocker > > When using S3AFileSystem to read Parquet files a specific set of > circumstances causes an EOFException that is not thrown when reading the > same file from local disk > Note this has only been observed under specific circumstances: > - when the reader is doing a projection (will cause it to do a seek > backwards and put the filesystem into random mode) > - when the file is larger than the readahead buffer size > - when the seek behavior of the Parquet reader causes the reader to seek > towards the end of the current input stream without reopening, such that the > next read on the currently open stream will read past the end of the > currently open stream. > Exception from Parquet reader is as follows: > {code} > Caused by: java.io.EOFException: Reached the end of stream with 51 bytes left > to read > at > org.apache.parquet.io.DelegatingSeekableInputStream.readFully(DelegatingSeekableInputStream.java:104) > at > org.apache.parquet.io.DelegatingSeekableInputStream.readFullyHeapBuffer(DelegatingSeekableInputStream.java:127) > at > org.apache.parquet.io.DelegatingSeekableInputStream.readFully(DelegatingSeekableInputStream.java:91) > at > org.apache.parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:1174) > at > org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:805) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:127) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:222) > at > org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:207) > at > org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.fetchNext(HadoopInputFormatBase.java:206) > at > org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.reachedEnd(HadoopInputFormatBase.java:199) > at > org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:190) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711) > at java.lang.Thread.run(Thread.java:748) > {code} > The following example program generate the same root behavior (sans finding a > Parquet file that happens to trigger this condition) by purposely reading > past the already active readahead range on any file >= 1029 bytes in size.. > {code:java} > final Configuration conf = new Configuration(); > conf.set("fs.s3a.readahead.range", "1K"); > conf.set("fs.s3a.experimental.input.fadvise", "random"); > final FileSystem fs = FileSystem.get(path.toUri(), conf); > // forward seek reading across readahead boundary > try (FSDataInputStream in = fs.open(path)) { > final byte[] temp = new byte[5]; > in.readByte(); > in.readFully(1023, temp); // <-- works > } > // forward seek reading from end of readahead boundary > try (FSDataInputStream in = fs.open(path)) { > final byte[] temp = new byte[5]; > in.readByte(); > in.readFully(1024, temp); // <-- throws EOFException > } > {code} > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-9359) Add Windows build and unit test to test-patch pre-commit testing
[ https://issues.apache.org/jira/browse/HADOOP-9359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley reassigned HADOOP-9359: -- Assignee: (was: Matt Foley) > Add Windows build and unit test to test-patch pre-commit testing > > > Key: HADOOP-9359 > URL: https://issues.apache.org/jira/browse/HADOOP-9359 > Project: Hadoop Common > Issue Type: Sub-task > Components: native >Reporter: Matt Foley >Priority: Major > > The "test-patch" utility is triggered by "Patch Available" state in Jira, and > runs nine different sets of builds, tests, and static analysis tools. > Currently only the Linux environment is tested. Need to add tests for Java > build under Windows, and unit test execution under Windows. > At this time, the community has decided that "-1" on these new additional > tests shall not block commits to the code base. However, contributors and > code reviewers are encouraged to utilize the information provided by these > tests to help keep Hadoop cross-platform compatible. Modify > http://wiki.apache.org/hadoop/HowToContribute to document this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-9359) Add Windows build and unit test to test-patch pre-commit testing
[ https://issues.apache.org/jira/browse/HADOOP-9359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley resolved HADOOP-9359. Resolution: Won't Fix Closed due to evident insufficient interest. > Add Windows build and unit test to test-patch pre-commit testing > > > Key: HADOOP-9359 > URL: https://issues.apache.org/jira/browse/HADOOP-9359 > Project: Hadoop Common > Issue Type: Sub-task > Components: native >Reporter: Matt Foley >Priority: Major > > The "test-patch" utility is triggered by "Patch Available" state in Jira, and > runs nine different sets of builds, tests, and static analysis tools. > Currently only the Linux environment is tested. Need to add tests for Java > build under Windows, and unit test execution under Windows. > At this time, the community has decided that "-1" on these new additional > tests shall not block commits to the code base. However, contributors and > code reviewers are encouraged to utilize the information provided by these > tests to help keep Hadoop cross-platform compatible. Modify > http://wiki.apache.org/hadoop/HowToContribute to document this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-7158) Reduce RPC packet size for homogeneous arrays, such as the array responses to listStatus() and getBlockLocations()
[ https://issues.apache.org/jira/browse/HADOOP-7158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley reassigned HADOOP-7158: -- Assignee: (was: Matt Foley) > Reduce RPC packet size for homogeneous arrays, such as the array responses to > listStatus() and getBlockLocations() > -- > > Key: HADOOP-7158 > URL: https://issues.apache.org/jira/browse/HADOOP-7158 > Project: Hadoop Common > Issue Type: Improvement > Components: io >Affects Versions: 0.22.0 >Reporter: Matt Foley >Priority: Major > > While commenting on HADOOP-6949, which proposes a big improvement in the RPC > wire format for arrays of primitives, Konstantin Shvachko said: > "Can/should we extend this to arrays of non-primitive types? This should > benefit return types for calls like listStatus() and getBlockLocations() on a > large directory." > The improvement for primitive arrays is based on not type-labeling every > element in the array, so the array in question must be strictly homogenous; > it cannot have subtypes of the assignable type. For instance, it could not > be applied to heartbeat responses of DatanodeCommand[], whose array elements > carry subtypes of DatanodeCommand, each of which must be type-labeled > independently. However, as Konstantin points out, it could really help > lengthy response arrays for things like listStatus() and getBlockLocations(). > I will attach a prototype implementation to this Jira, for discussion. > However, since it can't be automatically applied to all arrays passing > through RPC, I'll just providing the wrapper type. By using it, a caller is > asserting that the array is strictly homogeneous in the above sense. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-7809) Backport HADOOP-5839 to 0.20-security - fixes to ec2 scripts to allow remote job submission
[ https://issues.apache.org/jira/browse/HADOOP-7809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley resolved HADOOP-7809. Resolution: Won't Do Aged out. > Backport HADOOP-5839 to 0.20-security - fixes to ec2 scripts to allow remote > job submission > --- > > Key: HADOOP-7809 > URL: https://issues.apache.org/jira/browse/HADOOP-7809 > Project: Hadoop Common > Issue Type: Improvement > Components: contrib/cloud >Reporter: Joydeep Sen Sarma >Priority: Major > Attachments: ASF.LICENSE.NOT.GRANTED--hadoop-5839.2.patch > > > The fix for HADOOP-5839 was committed to 0.21 more than a year ago. This bug > is to backport the change (which is only 14 lines) to branch-0.20-security. > === > Original description: > i would very much like the option of submitting jobs from a workstation > outside ec2 to a hadoop cluster in ec2. This has been explored here: > http://www.nabble.com/public-IP-for-datanode-on-EC2-tt19336240.html > the net result of this is that we can make this work (along with using a > socks proxy) with a couple of changes in the ec2 scripts: > a) use public 'hostname' for fs.default.name setting (instead of the private > hostname being used currently) > b) mark hadoop.rpc.socket.factory.class.default as final variable in the > generated hadoop-site.xml (that applies to server side) > #a has no downside as far as i can tell since public hostnames resolve to > internal/private IP addresses within ec2 (so traffic is optimally routed). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-7809) Backport HADOOP-5839 to 0.20-security - fixes to ec2 scripts to allow remote job submission
[ https://issues.apache.org/jira/browse/HADOOP-7809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley reassigned HADOOP-7809: -- Assignee: (was: Matt Foley) > Backport HADOOP-5839 to 0.20-security - fixes to ec2 scripts to allow remote > job submission > --- > > Key: HADOOP-7809 > URL: https://issues.apache.org/jira/browse/HADOOP-7809 > Project: Hadoop Common > Issue Type: Improvement > Components: contrib/cloud >Reporter: Joydeep Sen Sarma >Priority: Major > Attachments: ASF.LICENSE.NOT.GRANTED--hadoop-5839.2.patch > > > The fix for HADOOP-5839 was committed to 0.21 more than a year ago. This bug > is to backport the change (which is only 14 lines) to branch-0.20-security. > === > Original description: > i would very much like the option of submitting jobs from a workstation > outside ec2 to a hadoop cluster in ec2. This has been explored here: > http://www.nabble.com/public-IP-for-datanode-on-EC2-tt19336240.html > the net result of this is that we can make this work (along with using a > socks proxy) with a couple of changes in the ec2 scripts: > a) use public 'hostname' for fs.default.name setting (instead of the private > hostname being used currently) > b) mark hadoop.rpc.socket.factory.class.default as final variable in the > generated hadoop-site.xml (that applies to server side) > #a has no downside as far as i can tell since public hostnames resolve to > internal/private IP addresses within ec2 (so traffic is optimally routed). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects
[ https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-13382: Description: In branch-2.8 and later, the patches for various child and related bugs listed in HADOOP-10105, most recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, HADOOP-12552, and HDFS-10623, eliminate all use of "commons-httpclient" from Hadoop and its sub-projects (except for hadoop-tools/hadoop-openstack; see HADOOP-11614). However, after incorporating these patches, "commons-httpclient" is still listed as a dependency in these POM files: * hadoop-project/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml We wish to remove these, but since commons-httpclient is still used in many files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to * hadoop-tools/hadoop-openstack/pom.xml (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is removed from hadoop-openstack.) In 2.8, this was mostly done by HADOOP-12552, but the version info formerly inherited from hadoop-project/pom.xml also needs to be added, so that is in the branch-2.8 version of the patch. Other projects with undeclared transitive dependencies on commons-httpclient, previously provided via hadoop-common or hadoop-client, may find this to be an incompatible change. Of course that also means such project is exposed to the commons-httpclient CVE, and needs to be fixed for that reason as well. was: In branch-2.8 and later, the patches for various child and related bugs listed in HADOOP-10105, most recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, HADOOP-12552, and HDFS-10623, eliminate all use of "commons-httpclient" from Hadoop and its sub-projects (except for hadoop-tools/hadoop-openstack; see HADOOP-11614). However, after incorporating these patches, "commons-httpclient" is still listed as a dependency in these POM files: * hadoop-project/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml We wish to remove these, but since commons-httpclient is still used in many files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to * hadoop-tools/hadoop-openstack/pom.xml (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is removed from hadoop-openstack.) In 2.8, this was mostly done by HADOOP-12552, but the version info formerly inherited from hadoop-project/pom.xml also needs to be added, so that is in the branch-2.8 version of the patch. > remove unneeded commons-httpclient dependencies from POM files in Hadoop and > sub-projects > - > > Key: HADOOP-13382 > URL: https://issues.apache.org/jira/browse/HADOOP-13382 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 2.8.0 >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: 2.8.0 > > Attachments: HADOOP-13382-branch-2.000.patch, > HADOOP-13382-branch-2.8.000.patch, HADOOP-13382.000.patch > > > In branch-2.8 and later, the patches for various child and related bugs > listed in HADOOP-10105, most recently including HADOOP-11613, HADOOP-12710, > HADOOP-12711, HADOOP-12552, and HDFS-10623, eliminate all use of > "commons-httpclient" from Hadoop and its sub-projects (except for > hadoop-tools/hadoop-openstack; see HADOOP-11614). > However, after incorporating these patches, "commons-httpclient" is still > listed as a dependency in these POM files: > * hadoop-project/pom.xml > * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml > We wish to remove these, but since commons-httpclient is still used in many > files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to > * hadoop-tools/hadoop-openstack/pom.xml > (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is > removed from hadoop-openstack.) > In 2.8, this was mostly done by HADOOP-12552, but the version info formerly > inherited from hadoop-project/pom.xml also needs to be added, so that is in > the branch-2.8 version of the patch. > Other projects with undeclared transitive dependencies on commons-httpclient, > previously provided via hadoop-common or hadoop-client, may find this to be > an incompatible change. Of course that also means such project is exposed to > the commons-httpclient CVE, and needs to be fixed for that reason as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects
[ https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412097#comment-15412097 ] Matt Foley commented on HADOOP-13382: - [~steve_l]: sorry. Fixed in 2.8.0. [~gsaha]: It is true that other projects with undeclared transitive dependencies on commons-httpclient, previously provided via hadoop-common or hadoop-client, may find this to be an incompatible change. Of course that also means such project is exposed to the commons-httpclient CVE, and needs to be fixed for that reason as well. Will update the Description to note this. Thanks for setting the appropriate flags in the jira. > remove unneeded commons-httpclient dependencies from POM files in Hadoop and > sub-projects > - > > Key: HADOOP-13382 > URL: https://issues.apache.org/jira/browse/HADOOP-13382 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 2.8.0 >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: 2.8.0 > > Attachments: HADOOP-13382-branch-2.000.patch, > HADOOP-13382-branch-2.8.000.patch, HADOOP-13382.000.patch > > > In branch-2.8 and later, the patches for various child and related bugs > listed in HADOOP-10105, most recently including HADOOP-11613, HADOOP-12710, > HADOOP-12711, HADOOP-12552, and HDFS-10623, eliminate all use of > "commons-httpclient" from Hadoop and its sub-projects (except for > hadoop-tools/hadoop-openstack; see HADOOP-11614). > However, after incorporating these patches, "commons-httpclient" is still > listed as a dependency in these POM files: > * hadoop-project/pom.xml > * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml > We wish to remove these, but since commons-httpclient is still used in many > files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to > * hadoop-tools/hadoop-openstack/pom.xml > (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is > removed from hadoop-openstack.) > In 2.8, this was mostly done by HADOOP-12552, but the version info formerly > inherited from hadoop-project/pom.xml also needs to be added, so that is in > the branch-2.8 version of the patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects
[ https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-13382: Fix Version/s: 2.8.0 > remove unneeded commons-httpclient dependencies from POM files in Hadoop and > sub-projects > - > > Key: HADOOP-13382 > URL: https://issues.apache.org/jira/browse/HADOOP-13382 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 2.8.0 >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: 2.8.0 > > Attachments: HADOOP-13382-branch-2.000.patch, > HADOOP-13382-branch-2.8.000.patch, HADOOP-13382.000.patch > > > In branch-2.8 and later, the patches for various child and related bugs > listed in HADOOP-10105, most recently including HADOOP-11613, HADOOP-12710, > HADOOP-12711, HADOOP-12552, and HDFS-10623, eliminate all use of > "commons-httpclient" from Hadoop and its sub-projects (except for > hadoop-tools/hadoop-openstack; see HADOOP-11614). > However, after incorporating these patches, "commons-httpclient" is still > listed as a dependency in these POM files: > * hadoop-project/pom.xml > * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml > We wish to remove these, but since commons-httpclient is still used in many > files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to > * hadoop-tools/hadoop-openstack/pom.xml > (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is > removed from hadoop-openstack.) > In 2.8, this was mostly done by HADOOP-12552, but the version info formerly > inherited from hadoop-project/pom.xml also needs to be added, so that is in > the branch-2.8 version of the patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects
[ https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-13382: Resolution: Fixed Status: Resolved (was: Patch Available) Thanks, [~cnauroth] and [~steve_l]. Committed as: * trunk - 12aa184479675d6c9bd * branch-2 - ea10e1384ff65e27521 * branch-2.8 - c96cb3fd48925b3eb2c > remove unneeded commons-httpclient dependencies from POM files in Hadoop and > sub-projects > - > > Key: HADOOP-13382 > URL: https://issues.apache.org/jira/browse/HADOOP-13382 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 2.8.0 >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-13382-branch-2.000.patch, > HADOOP-13382-branch-2.8.000.patch, HADOOP-13382.000.patch > > > In branch-2.8 and later, the patches for various child and related bugs > listed in HADOOP-10105, most recently including HADOOP-11613, HADOOP-12710, > HADOOP-12711, HADOOP-12552, and HDFS-10623, eliminate all use of > "commons-httpclient" from Hadoop and its sub-projects (except for > hadoop-tools/hadoop-openstack; see HADOOP-11614). > However, after incorporating these patches, "commons-httpclient" is still > listed as a dependency in these POM files: > * hadoop-project/pom.xml > * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml > We wish to remove these, but since commons-httpclient is still used in many > files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to > * hadoop-tools/hadoop-openstack/pom.xml > (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is > removed from hadoop-openstack.) > In 2.8, this was mostly done by HADOOP-12552, but the version info formerly > inherited from hadoop-project/pom.xml also needs to be added, so that is in > the branch-2.8 version of the patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects
[ https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384907#comment-15384907 ] Matt Foley edited comment on HADOOP-13382 at 7/19/16 10:15 PM: --- Response to the Hadoop QA robot complaint about no new unit tests: This patch seeks to produce no functional change in the behavior of the code, therefore there are no new unit tests needed. There are no new negative tests needed either, because if the patch breaks anything, it will be a gross breakage of hadoop-openstack. Since all existing unit tests continue to work correctly, that's sufficient. was (Author: mattf): Response to the Hadoop QA complaint: This patch seeks to produce no functional change in the behavior of the code, therefore there are no new unit tests needed. There are no new negative tests needed either, because if the patch breaks anything, it will be a gross breakage of hadoop-openstack. Since all existing unit tests continue to work correctly, that's sufficient. > remove unneeded commons-httpclient dependencies from POM files in Hadoop and > sub-projects > - > > Key: HADOOP-13382 > URL: https://issues.apache.org/jira/browse/HADOOP-13382 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 2.8.0 >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-13382-branch-2.000.patch, > HADOOP-13382-branch-2.8.000.patch, HADOOP-13382.000.patch > > > In branch-2.8 and later, the patches for various child and related bugs > listed in HADOOP-10105, most recently including HADOOP-11613, HADOOP-12710, > HADOOP-12711, HADOOP-12552, and HDFS-10623, eliminate all use of > "commons-httpclient" from Hadoop and its sub-projects (except for > hadoop-tools/hadoop-openstack; see HADOOP-11614). > However, after incorporating these patches, "commons-httpclient" is still > listed as a dependency in these POM files: > * hadoop-project/pom.xml > * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml > We wish to remove these, but since commons-httpclient is still used in many > files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to > * hadoop-tools/hadoop-openstack/pom.xml > (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is > removed from hadoop-openstack.) > In 2.8, this was mostly done by HADOOP-12552, but the version info formerly > inherited from hadoop-project/pom.xml also needs to be added, so that is in > the branch-2.8 version of the patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects
[ https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384907#comment-15384907 ] Matt Foley commented on HADOOP-13382: - Response to the Hadoop QA complaint: This patch seeks to produce no functional change in the behavior of the code, therefore there are no new unit tests needed. There are no new negative tests needed either, because if the patch breaks anything, it will be a gross breakage of hadoop-openstack. Since all existing unit tests continue to work correctly, that's sufficient. > remove unneeded commons-httpclient dependencies from POM files in Hadoop and > sub-projects > - > > Key: HADOOP-13382 > URL: https://issues.apache.org/jira/browse/HADOOP-13382 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 2.8.0 >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-13382-branch-2.000.patch, > HADOOP-13382-branch-2.8.000.patch, HADOOP-13382.000.patch > > > In branch-2.8 and later, the patches for various child and related bugs > listed in HADOOP-10105, most recently including HADOOP-11613, HADOOP-12710, > HADOOP-12711, HADOOP-12552, and HDFS-10623, eliminate all use of > "commons-httpclient" from Hadoop and its sub-projects (except for > hadoop-tools/hadoop-openstack; see HADOOP-11614). > However, after incorporating these patches, "commons-httpclient" is still > listed as a dependency in these POM files: > * hadoop-project/pom.xml > * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml > We wish to remove these, but since commons-httpclient is still used in many > files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to > * hadoop-tools/hadoop-openstack/pom.xml > (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is > removed from hadoop-openstack.) > In 2.8, this was mostly done by HADOOP-12552, but the version info formerly > inherited from hadoop-project/pom.xml also needs to be added, so that is in > the branch-2.8 version of the patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects
[ https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-13382: Description: In branch-2.8 and later, the patches for various child and related bugs listed in HADOOP-10105, most recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, HADOOP-12552, and HDFS-10623, eliminate all use of "commons-httpclient" from Hadoop and its sub-projects (except for hadoop-tools/hadoop-openstack; see HADOOP-11614). However, after incorporating these patches, "commons-httpclient" is still listed as a dependency in these POM files: * hadoop-project/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml We wish to remove these, but since commons-httpclient is still used in many files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to * hadoop-tools/hadoop-openstack/pom.xml (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is removed from hadoop-openstack.) In 2.8, this was mostly done by HADOOP-12552, but the version info formerly inherited from hadoop-project/pom.xml also needs to be added, so that is in the branch-2.8 version of the patch. was: The patches for various child and related bugs listed in HADOOP-10105, most recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, HADOOP-12552, and HDFS-10623, eliminate all use of "commons-httpclient" from Hadoop and its sub-projects (except for hadoop-tools/hadoop-openstack; see HADOOP-11614). However, after incorporating these patches, "commons-httpclient" is still listed as a dependency in these POM files: * hadoop-project/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml * (in branch-2.7.3 only) hadoop-common-project/hadoop-common/pom.xml We wish to remove these, but since commons-httpclient is still used in many files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to * hadoop-tools/hadoop-openstack/pom.xml (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is removed from hadoop-openstack.) In 2.8, this was mostly done by HADOOP-12552, but the version info formerly inherited from hadoop-project/pom.xml also needs to be added, so that is in the branch-2.8 version of the patch. > remove unneeded commons-httpclient dependencies from POM files in Hadoop and > sub-projects > - > > Key: HADOOP-13382 > URL: https://issues.apache.org/jira/browse/HADOOP-13382 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 2.8.0 >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: 2.8.0 > > Attachments: HADOOP-13382-branch-2.000.patch, > HADOOP-13382-branch-2.8.000.patch, HADOOP-13382.000.patch > > > In branch-2.8 and later, the patches for various child and related bugs > listed in HADOOP-10105, most recently including HADOOP-11613, HADOOP-12710, > HADOOP-12711, HADOOP-12552, and HDFS-10623, eliminate all use of > "commons-httpclient" from Hadoop and its sub-projects (except for > hadoop-tools/hadoop-openstack; see HADOOP-11614). > However, after incorporating these patches, "commons-httpclient" is still > listed as a dependency in these POM files: > * hadoop-project/pom.xml > * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml > We wish to remove these, but since commons-httpclient is still used in many > files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to > * hadoop-tools/hadoop-openstack/pom.xml > (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is > removed from hadoop-openstack.) > In 2.8, this was mostly done by HADOOP-12552, but the version info formerly > inherited from hadoop-project/pom.xml also needs to be added, so that is in > the branch-2.8 version of the patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects
[ https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-13382: Fix Version/s: 2.8.0 > remove unneeded commons-httpclient dependencies from POM files in Hadoop and > sub-projects > - > > Key: HADOOP-13382 > URL: https://issues.apache.org/jira/browse/HADOOP-13382 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 2.8.0 >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: 2.8.0 > > Attachments: HADOOP-13382-branch-2.000.patch, > HADOOP-13382-branch-2.8.000.patch, HADOOP-13382.000.patch > > > In branch-2.8 and later, the patches for various child and related bugs > listed in HADOOP-10105, most recently including HADOOP-11613, HADOOP-12710, > HADOOP-12711, HADOOP-12552, and HDFS-10623, eliminate all use of > "commons-httpclient" from Hadoop and its sub-projects (except for > hadoop-tools/hadoop-openstack; see HADOOP-11614). > However, after incorporating these patches, "commons-httpclient" is still > listed as a dependency in these POM files: > * hadoop-project/pom.xml > * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml > We wish to remove these, but since commons-httpclient is still used in many > files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to > * hadoop-tools/hadoop-openstack/pom.xml > (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is > removed from hadoop-openstack.) > In 2.8, this was mostly done by HADOOP-12552, but the version info formerly > inherited from hadoop-project/pom.xml also needs to be added, so that is in > the branch-2.8 version of the patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11614) Remove httpclient dependency from hadoop-openstack
[ https://issues.apache.org/jira/browse/HADOOP-11614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-11614: Description: Remove httpclient dependency from hadoop-openstack and its pom.xml file. (was: Remove httpclient dependency from hadoop-openstack.) > Remove httpclient dependency from hadoop-openstack > -- > > Key: HADOOP-11614 > URL: https://issues.apache.org/jira/browse/HADOOP-11614 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Akira Ajisaka >Assignee: Brahma Reddy Battula > Attachments: HADOOP-11614-002.patch, HADOOP-11614.patch > > > Remove httpclient dependency from hadoop-openstack and its pom.xml file. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects
[ https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-13382: Affects Version/s: 2.8.0 Target Version/s: 2.8.0 Status: Patch Available (was: In Progress) > remove unneeded commons-httpclient dependencies from POM files in Hadoop and > sub-projects > - > > Key: HADOOP-13382 > URL: https://issues.apache.org/jira/browse/HADOOP-13382 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 2.8.0 >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-13382-branch-2.000.patch, > HADOOP-13382-branch-2.8.000.patch, HADOOP-13382.000.patch > > > The patches for various child and related bugs listed in HADOOP-10105, most > recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, HADOOP-12552, > and HDFS-10623, eliminate all use of "commons-httpclient" from Hadoop and its > sub-projects (except for hadoop-tools/hadoop-openstack; see HADOOP-11614). > However, after incorporating these patches, "commons-httpclient" is still > listed as a dependency in these POM files: > * hadoop-project/pom.xml > * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml > * (in branch-2.7.3 only) hadoop-common-project/hadoop-common/pom.xml > We wish to remove these, but since commons-httpclient is still used in many > files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to > * hadoop-tools/hadoop-openstack/pom.xml > (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is > removed from hadoop-openstack.) > In 2.8, this was mostly done by HADOOP-12552, but the version info formerly > inherited from hadoop-project/pom.xml also needs to be added, so that is in > the branch-2.8 version of the patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects
[ https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-13382: Attachment: HADOOP-13382.000.patch HADOOP-13382-branch-2.000.patch > remove unneeded commons-httpclient dependencies from POM files in Hadoop and > sub-projects > - > > Key: HADOOP-13382 > URL: https://issues.apache.org/jira/browse/HADOOP-13382 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-13382-branch-2.000.patch, > HADOOP-13382-branch-2.8.000.patch, HADOOP-13382.000.patch > > > The patches for various child and related bugs listed in HADOOP-10105, most > recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, HADOOP-12552, > and HDFS-10623, eliminate all use of "commons-httpclient" from Hadoop and its > sub-projects (except for hadoop-tools/hadoop-openstack; see HADOOP-11614). > However, after incorporating these patches, "commons-httpclient" is still > listed as a dependency in these POM files: > * hadoop-project/pom.xml > * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml > * (in branch-2.7.3 only) hadoop-common-project/hadoop-common/pom.xml > We wish to remove these, but since commons-httpclient is still used in many > files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to > * hadoop-tools/hadoop-openstack/pom.xml > (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is > removed from hadoop-openstack.) > In 2.8, this was mostly done by HADOOP-12552, but the version info formerly > inherited from hadoop-project/pom.xml also needs to be added, so that is in > the branch-2.8 version of the patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects
[ https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380340#comment-15380340 ] Matt Foley edited comment on HADOOP-13382 at 7/16/16 1:21 AM: -- Proposed for branch-2.8, to go along with HADOOP-11613, HADOOP-12711, and HADOOP-12552. Branch-2 and trunk need exactly the same patch. was (Author: mattf): Proposed for branch-2.8, to go along with HADOOP-11613, HADOOP-12711, and HADOOP-12552. > remove unneeded commons-httpclient dependencies from POM files in Hadoop and > sub-projects > - > > Key: HADOOP-13382 > URL: https://issues.apache.org/jira/browse/HADOOP-13382 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-13382-branch-2.8.000.patch > > > The patches for various child and related bugs listed in HADOOP-10105, most > recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, HADOOP-12552, > and HDFS-10623, eliminate all use of "commons-httpclient" from Hadoop and its > sub-projects (except for hadoop-tools/hadoop-openstack; see HADOOP-11614). > However, after incorporating these patches, "commons-httpclient" is still > listed as a dependency in these POM files: > * hadoop-project/pom.xml > * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml > * (in branch-2.7.3 only) hadoop-common-project/hadoop-common/pom.xml > We wish to remove these, but since commons-httpclient is still used in many > files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to > * hadoop-tools/hadoop-openstack/pom.xml > (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is > removed from hadoop-openstack.) > In 2.8, this was mostly done by HADOOP-12552, but the version info formerly > inherited from hadoop-project/pom.xml also needs to be added, so that is in > the branch-2.8 version of the patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects
[ https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-13382: Attachment: HADOOP-13382-branch-2.8.000.patch > remove unneeded commons-httpclient dependencies from POM files in Hadoop and > sub-projects > - > > Key: HADOOP-13382 > URL: https://issues.apache.org/jira/browse/HADOOP-13382 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-13382-branch-2.8.000.patch > > > The patches for various child and related bugs listed in HADOOP-10105, most > recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, HADOOP-12552, > and HDFS-10623, eliminate all use of "commons-httpclient" from Hadoop and its > sub-projects (except for hadoop-tools/hadoop-openstack; see HADOOP-11614). > However, after incorporating these patches, "commons-httpclient" is still > listed as a dependency in these POM files: > * hadoop-project/pom.xml > * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml > * (in branch-2.7.3 only) hadoop-common-project/hadoop-common/pom.xml > We wish to remove these, but since commons-httpclient is still used in many > files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to > * hadoop-tools/hadoop-openstack/pom.xml > (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is > removed from hadoop-openstack.) > In 2.8, this was mostly done by HADOOP-12552, but the version info formerly > inherited from hadoop-project/pom.xml also needs to be added, so that is in > the branch-2.8 version of the patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects
[ https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380340#comment-15380340 ] Matt Foley edited comment on HADOOP-13382 at 7/16/16 12:57 AM: --- Proposed for branch-2.8, to go along with HADOOP-11613, HADOOP-12711, and HADOOP-12552. was (Author: mattf): Proposed for branch-2.8, to go along with HADOOP-11613 and HADOOP-12711. > remove unneeded commons-httpclient dependencies from POM files in Hadoop and > sub-projects > - > > Key: HADOOP-13382 > URL: https://issues.apache.org/jira/browse/HADOOP-13382 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Matt Foley >Assignee: Matt Foley > > The patches for various child and related bugs listed in HADOOP-10105, most > recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, HADOOP-12552, > and HDFS-10623, eliminate all use of "commons-httpclient" from Hadoop and its > sub-projects (except for hadoop-tools/hadoop-openstack; see HADOOP-11614). > However, after incorporating these patches, "commons-httpclient" is still > listed as a dependency in these POM files: > * hadoop-project/pom.xml > * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml > * (in branch-2.7.3 only) hadoop-common-project/hadoop-common/pom.xml > We wish to remove these, but since commons-httpclient is still used in many > files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to > * hadoop-tools/hadoop-openstack/pom.xml > (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is > removed from hadoop-openstack.) > In 2.8, this was mostly done by HADOOP-12552, but the version info formerly > inherited from hadoop-project/pom.xml also needs to be added, so that is in > the branch-2.8 version of the patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects
[ https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-13382: Description: The patches for various child and related bugs listed in HADOOP-10105, most recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, HADOOP-12552, and HDFS-10623, eliminate all use of "commons-httpclient" from Hadoop and its sub-projects (except for hadoop-tools/hadoop-openstack; see HADOOP-11614). However, after incorporating these patches, "commons-httpclient" is still listed as a dependency in these POM files: * hadoop-project/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml * (in branch-2.7.3 only) hadoop-common-project/hadoop-common/pom.xml We wish to remove these, but since commons-httpclient is still used in many files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to * hadoop-tools/hadoop-openstack/pom.xml (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is removed from hadoop-openstack.) In 2.8, this was mostly done by HADOOP-12552, but the version info formerly inherited from hadoop-project/pom.xml also needs to be added, so that is in the branch-2.8 version of the patch. was: The patches for various child and related bugs listed in HADOOP-10105, most recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, and HDFS-10623, eliminate all use of "commons-httpclient" from Hadoop and its sub-projects (except for hadoop-tools/hadoop-openstack; see HADOOP-11614). However, after incorporating these patches, "commons-httpclient" is still listed as a dependency in these POM files: * hadoop-project/pom.xml * hadoop-common-project/hadoop-common/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml We wish to remove these, but since commons-httpclient is still used in many files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to * hadoop-tools/hadoop-openstack/pom.xml (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is removed from hadoop-openstack.) > remove unneeded commons-httpclient dependencies from POM files in Hadoop and > sub-projects > - > > Key: HADOOP-13382 > URL: https://issues.apache.org/jira/browse/HADOOP-13382 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Matt Foley >Assignee: Matt Foley > > The patches for various child and related bugs listed in HADOOP-10105, most > recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, HADOOP-12552, > and HDFS-10623, eliminate all use of "commons-httpclient" from Hadoop and its > sub-projects (except for hadoop-tools/hadoop-openstack; see HADOOP-11614). > However, after incorporating these patches, "commons-httpclient" is still > listed as a dependency in these POM files: > * hadoop-project/pom.xml > * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml > * (in branch-2.7.3 only) hadoop-common-project/hadoop-common/pom.xml > We wish to remove these, but since commons-httpclient is still used in many > files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to > * hadoop-tools/hadoop-openstack/pom.xml > (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is > removed from hadoop-openstack.) > In 2.8, this was mostly done by HADOOP-12552, but the version info formerly > inherited from hadoop-project/pom.xml also needs to be added, so that is in > the branch-2.8 version of the patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects
[ https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380340#comment-15380340 ] Matt Foley edited comment on HADOOP-13382 at 7/16/16 12:27 AM: --- Proposed for branch-2.8, to go along with HADOOP-11613 and HADOOP-12711. was (Author: mattf): Proposed for branch-2.8 > remove unneeded commons-httpclient dependencies from POM files in Hadoop and > sub-projects > - > > Key: HADOOP-13382 > URL: https://issues.apache.org/jira/browse/HADOOP-13382 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Matt Foley >Assignee: Matt Foley > > The patches for various child and related bugs listed in HADOOP-10105, most > recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, and HDFS-10623, > eliminate all use of "commons-httpclient" from Hadoop and its sub-projects > (except for hadoop-tools/hadoop-openstack; see HADOOP-11614). > However, after incorporating these patches, "commons-httpclient" is still > listed as a dependency in these POM files: > * hadoop-project/pom.xml > * hadoop-common-project/hadoop-common/pom.xml > * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml > We wish to remove these, but since commons-httpclient is still used in many > files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to > * hadoop-tools/hadoop-openstack/pom.xml > (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is > removed from hadoop-openstack.) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects
[ https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-13382: Attachment: (was: HADOOP-13382-branch-2.8.000.patch) > remove unneeded commons-httpclient dependencies from POM files in Hadoop and > sub-projects > - > > Key: HADOOP-13382 > URL: https://issues.apache.org/jira/browse/HADOOP-13382 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Matt Foley >Assignee: Matt Foley > > The patches for various child and related bugs listed in HADOOP-10105, most > recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, and HDFS-10623, > eliminate all use of "commons-httpclient" from Hadoop and its sub-projects > (except for hadoop-tools/hadoop-openstack; see HADOOP-11614). > However, after incorporating these patches, "commons-httpclient" is still > listed as a dependency in these POM files: > * hadoop-project/pom.xml > * hadoop-common-project/hadoop-common/pom.xml > * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml > We wish to remove these, but since commons-httpclient is still used in many > files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to > * hadoop-tools/hadoop-openstack/pom.xml > (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is > removed from hadoop-openstack.) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects
[ https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-13382: Attachment: HADOOP-13382-branch-2.8.000.patch > remove unneeded commons-httpclient dependencies from POM files in Hadoop and > sub-projects > - > > Key: HADOOP-13382 > URL: https://issues.apache.org/jira/browse/HADOOP-13382 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-13382-branch-2.8.000.patch > > > The patches for various child and related bugs listed in HADOOP-10105, most > recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, and HDFS-10623, > eliminate all use of "commons-httpclient" from Hadoop and its sub-projects > (except for hadoop-tools/hadoop-openstack; see HADOOP-11614). > However, after incorporating these patches, "commons-httpclient" is still > listed as a dependency in these POM files: > * hadoop-project/pom.xml > * hadoop-common-project/hadoop-common/pom.xml > * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml > We wish to remove these, but since commons-httpclient is still used in many > files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to > * hadoop-tools/hadoop-openstack/pom.xml > (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is > removed from hadoop-openstack.) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects
[ https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-13382: Attachment: (was: HADOOP-13382.patch) > remove unneeded commons-httpclient dependencies from POM files in Hadoop and > sub-projects > - > > Key: HADOOP-13382 > URL: https://issues.apache.org/jira/browse/HADOOP-13382 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-13382-branch-2.8.000.patch > > > The patches for various child and related bugs listed in HADOOP-10105, most > recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, and HDFS-10623, > eliminate all use of "commons-httpclient" from Hadoop and its sub-projects > (except for hadoop-tools/hadoop-openstack; see HADOOP-11614). > However, after incorporating these patches, "commons-httpclient" is still > listed as a dependency in these POM files: > * hadoop-project/pom.xml > * hadoop-common-project/hadoop-common/pom.xml > * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml > We wish to remove these, but since commons-httpclient is still used in many > files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to > * hadoop-tools/hadoop-openstack/pom.xml > (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is > removed from hadoop-openstack.) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects
[ https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-13382: Attachment: HADOOP-13382.patch Proposed for branch-2.8 > remove unneeded commons-httpclient dependencies from POM files in Hadoop and > sub-projects > - > > Key: HADOOP-13382 > URL: https://issues.apache.org/jira/browse/HADOOP-13382 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-13382.patch > > > The patches for various child and related bugs listed in HADOOP-10105, most > recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, and HDFS-10623, > eliminate all use of "commons-httpclient" from Hadoop and its sub-projects > (except for hadoop-tools/hadoop-openstack; see HADOOP-11614). > However, after incorporating these patches, "commons-httpclient" is still > listed as a dependency in these POM files: > * hadoop-project/pom.xml > * hadoop-common-project/hadoop-common/pom.xml > * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml > We wish to remove these, but since commons-httpclient is still used in many > files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to > * hadoop-tools/hadoop-openstack/pom.xml > (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is > removed from hadoop-openstack.) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects
[ https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-13382 started by Matt Foley. --- > remove unneeded commons-httpclient dependencies from POM files in Hadoop and > sub-projects > - > > Key: HADOOP-13382 > URL: https://issues.apache.org/jira/browse/HADOOP-13382 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Matt Foley >Assignee: Matt Foley > > The patches for various child and related bugs listed in HADOOP-10105, most > recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, and HDFS-10623, > eliminate all use of "commons-httpclient" from Hadoop and its sub-projects > (except for hadoop-tools/hadoop-openstack; see HADOOP-11614). > However, after incorporating these patches, "commons-httpclient" is still > listed as a dependency in these POM files: > * hadoop-project/pom.xml > * hadoop-common-project/hadoop-common/pom.xml > * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml > We wish to remove these, but since commons-httpclient is still used in many > files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to > * hadoop-tools/hadoop-openstack/pom.xml > (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is > removed from hadoop-openstack.) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects
[ https://issues.apache.org/jira/browse/HADOOP-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley reassigned HADOOP-13382: --- Assignee: Matt Foley > remove unneeded commons-httpclient dependencies from POM files in Hadoop and > sub-projects > - > > Key: HADOOP-13382 > URL: https://issues.apache.org/jira/browse/HADOOP-13382 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Matt Foley >Assignee: Matt Foley > > The patches for various child and related bugs listed in HADOOP-10105, most > recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, and HDFS-10623, > eliminate all use of "commons-httpclient" from Hadoop and its sub-projects > (except for hadoop-tools/hadoop-openstack; see HADOOP-11614). > However, after incorporating these patches, "commons-httpclient" is still > listed as a dependency in these POM files: > * hadoop-project/pom.xml > * hadoop-common-project/hadoop-common/pom.xml > * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml > We wish to remove these, but since commons-httpclient is still used in many > files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to > * hadoop-tools/hadoop-openstack/pom.xml > (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is > removed from hadoop-openstack.) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12710) Remove dependency on commons-httpclient for TestHttpServerLogs
[ https://issues.apache.org/jira/browse/HADOOP-12710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380332#comment-15380332 ] Matt Foley commented on HADOOP-12710: - This jira is marked fixed in 2.9.0, which is true. However, it is worth noting that the affected file, hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServerLogs.java, is new in 2.9.0. That is, it was not introduced to branch-2 until after branch-2.8 was forked off, so the issue does not exist in 2.8.0 or earlier. > Remove dependency on commons-httpclient for TestHttpServerLogs > -- > > Key: HADOOP-12710 > URL: https://issues.apache.org/jira/browse/HADOOP-12710 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha1 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang > Fix For: 2.9.0 > > Attachments: HADOOP-12710.001.patch > > > Commons-httpclient has long been EOL. Critically, it has several security > vulnerabilities: CVE-2012-5783 > http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-5783. > I saw a recent commit that depends on commons-httpclient for > TestHttpServerLogs (HADOOP-12625) This JIRA intends to replace the dependency > with httpclient APIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HADOOP-12710) Remove dependency on commons-httpclient for TestHttpServerLogs
[ https://issues.apache.org/jira/browse/HADOOP-12710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-12710: Comment: was deleted (was: -Fixed Fix_Version from 2.9.0 to 2.8.0, since it went in branch 2.8 which is not yet released.- Apologies, my mistake. Only went into trunk.) > Remove dependency on commons-httpclient for TestHttpServerLogs > -- > > Key: HADOOP-12710 > URL: https://issues.apache.org/jira/browse/HADOOP-12710 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha1 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang > Fix For: 2.9.0 > > Attachments: HADOOP-12710.001.patch > > > Commons-httpclient has long been EOL. Critically, it has several security > vulnerabilities: CVE-2012-5783 > http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-5783. > I saw a recent commit that depends on commons-httpclient for > TestHttpServerLogs (HADOOP-12625) This JIRA intends to replace the dependency > with httpclient APIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-12710) Remove dependency on commons-httpclient for TestHttpServerLogs
[ https://issues.apache.org/jira/browse/HADOOP-12710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380304#comment-15380304 ] Matt Foley edited comment on HADOOP-12710 at 7/15/16 11:44 PM: --- -Fixed Fix_Version from 2.9.0 to 2.8.0, since it went in branch 2.8 which is not yet released.- Apologies, my mistake. Only went into trunk. was (Author: mattf): Fixed Fix_Version from 2.9.0 to 2.8.0, since it went in branch 2.8 which is not yet released. > Remove dependency on commons-httpclient for TestHttpServerLogs > -- > > Key: HADOOP-12710 > URL: https://issues.apache.org/jira/browse/HADOOP-12710 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha1 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang > Fix For: 2.9.0 > > Attachments: HADOOP-12710.001.patch > > > Commons-httpclient has long been EOL. Critically, it has several security > vulnerabilities: CVE-2012-5783 > http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-5783. > I saw a recent commit that depends on commons-httpclient for > TestHttpServerLogs (HADOOP-12625) This JIRA intends to replace the dependency > with httpclient APIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12710) Remove dependency on commons-httpclient for TestHttpServerLogs
[ https://issues.apache.org/jira/browse/HADOOP-12710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-12710: Fix Version/s: (was: 2.8.0) 2.9.0 > Remove dependency on commons-httpclient for TestHttpServerLogs > -- > > Key: HADOOP-12710 > URL: https://issues.apache.org/jira/browse/HADOOP-12710 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha1 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang > Fix For: 2.9.0 > > Attachments: HADOOP-12710.001.patch > > > Commons-httpclient has long been EOL. Critically, it has several security > vulnerabilities: CVE-2012-5783 > http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-5783. > I saw a recent commit that depends on commons-httpclient for > TestHttpServerLogs (HADOOP-12625) This JIRA intends to replace the dependency > with httpclient APIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12710) Remove dependency on commons-httpclient for TestHttpServerLogs
[ https://issues.apache.org/jira/browse/HADOOP-12710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-12710: Fix Version/s: (was: 2.9.0) 2.8.0 > Remove dependency on commons-httpclient for TestHttpServerLogs > -- > > Key: HADOOP-12710 > URL: https://issues.apache.org/jira/browse/HADOOP-12710 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha1 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang > Fix For: 2.8.0 > > Attachments: HADOOP-12710.001.patch > > > Commons-httpclient has long been EOL. Critically, it has several security > vulnerabilities: CVE-2012-5783 > http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-5783. > I saw a recent commit that depends on commons-httpclient for > TestHttpServerLogs (HADOOP-12625) This JIRA intends to replace the dependency > with httpclient APIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12710) Remove dependency on commons-httpclient for TestHttpServerLogs
[ https://issues.apache.org/jira/browse/HADOOP-12710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380304#comment-15380304 ] Matt Foley commented on HADOOP-12710: - Fixed Fix_Version from 2.9.0 to 2.8.0, since it went in branch 2.8 which is not yet released. > Remove dependency on commons-httpclient for TestHttpServerLogs > -- > > Key: HADOOP-12710 > URL: https://issues.apache.org/jira/browse/HADOOP-12710 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha1 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang > Fix For: 2.8.0 > > Attachments: HADOOP-12710.001.patch > > > Commons-httpclient has long been EOL. Critically, it has several security > vulnerabilities: CVE-2012-5783 > http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-5783. > I saw a recent commit that depends on commons-httpclient for > TestHttpServerLogs (HADOOP-12625) This JIRA intends to replace the dependency > with httpclient APIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13382) remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects
Matt Foley created HADOOP-13382: --- Summary: remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects Key: HADOOP-13382 URL: https://issues.apache.org/jira/browse/HADOOP-13382 Project: Hadoop Common Issue Type: Improvement Components: build Reporter: Matt Foley The patches for various child and related bugs listed in HADOOP-10105, most recently including HADOOP-11613, HADOOP-12710, HADOOP-12711, and HDFS-10623, eliminate all use of "commons-httpclient" from Hadoop and its sub-projects (except for hadoop-tools/hadoop-openstack; see HADOOP-11614). However, after incorporating these patches, "commons-httpclient" is still listed as a dependency in these POM files: * hadoop-project/pom.xml * hadoop-common-project/hadoop-common/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/pom.xml We wish to remove these, but since commons-httpclient is still used in many files in hadoop-tools/hadoop-openstack, we'll need to _add_ the dependency to * hadoop-tools/hadoop-openstack/pom.xml (We'll add a note to HADOOP-11614 to undo this when commons-httpclient is removed from hadoop-openstack.) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12665) Document hadoop.security.token.service.use_ip
[ https://issues.apache.org/jira/browse/HADOOP-12665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15088151#comment-15088151 ] Matt Foley commented on HADOOP-12665: - In branch-1 it was documented in [core-default.xml|https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1/src/core/core-default.xml] as: {code} hadoop.security.token.service.use_ip true Controls whether tokens always use IP addresses. DNS changes will not be detected if this option is enabled. Existing client connections that break will always reconnect to the IP of the original host. New clients will connect to the host's new IP but fail to locate a token. Disabling this option will allow existing and new clients to detect an IP change and continue to locate the new host's token. {code} This resulted in a corresponding entry in https://hadoop.apache.org/docs/r1.2.1/core-default.html Apparently in branch-2 it was removed from core-default.xml, presumably because it is a rarely used parameter. However, it still needs to be documented somewhere because it is required for *multi-homed servers* if kerberos security is enabled, as seen in certain customer complaints (that have not been reported as Apache Jiras since they were resolved as misconfigurations rather than code bugs). I have documented it thus: bq. Parameters for Security Token service host resolution bq. In secure multi-homed environments, the following parameter will need to be set to false (it is true by default) on both cluster servers and clients (see HADOOP-7733), in core-site.xml. If it is not set correctly, the symptom will be inability to submit an application to YARN from an external client (with error "client host not a member of the Hadoop cluster"), or even from an in-cluster client if server failover occurs. I'm including this as part of a white paper I'm writing on the whole topic of multi-homed support. I was planning to integrate that into Apache Hadoop docs when it is done in a couple weeks. So [~anu], if you like, you can reassign this docs jira to me. > Document hadoop.security.token.service.use_ip > - > > Key: HADOOP-12665 > URL: https://issues.apache.org/jira/browse/HADOOP-12665 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 2.8.0 >Reporter: Arpit Agarwal >Assignee: Anu Engineer > > {{hadoop.security.token.service.use_ip}} is not documented in 2.x/trunk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal
[ https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-12617: Attachment: HADOOP-12617.006.patch > SPNEGO authentication request to non-default realm gets default realm name > inserted in target server principal > -- > > Key: HADOOP-12617 > URL: https://issues.apache.org/jira/browse/HADOOP-12617 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.1 > Environment: Java client talking to two secure clusters in different > Kerberos realms, > or talking to any secure cluster in non-default realm >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-12617-branch-2.7.001.patch, > HADOOP-12617.003.patch, HADOOP-12617.005.patch, HADOOP-12617.006.patch > > > Note: This is NOT a vulnerability. > In order for a single Java client to communicate with two different secure > clusters in different realms (only one of which can be the "default_realm"), > the client's krb5.conf file must specify both realms, and provide a > \[domain_realm\] section that maps cluster servers' domains to the correct > realms. With other appropriate behaviors (such as using the config from each > cluster to talk to the respective clusters, and a user principal from each > realm to talk to the respective realms), this is sufficient for most Hadoop > ecosystem clients. > But our SPNEGO using clients, such as Oozie, have a bug when it comes to > talking to a non-default realm. The default realm name gets incorrectly > inserted into the construction of the target server principal for the > non-default-realm cluster. Details and proposed solution are given in the > first comments below. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal
[ https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-12617: Attachment: HADOOP-12617-branch-2.7.002.patch > SPNEGO authentication request to non-default realm gets default realm name > inserted in target server principal > -- > > Key: HADOOP-12617 > URL: https://issues.apache.org/jira/browse/HADOOP-12617 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.1 > Environment: Java client talking to two secure clusters in different > Kerberos realms, > or talking to any secure cluster in non-default realm >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-12617-branch-2.7.001.patch, > HADOOP-12617-branch-2.7.002.patch, HADOOP-12617.003.patch, > HADOOP-12617.005.patch, HADOOP-12617.006.patch > > > Note: This is NOT a vulnerability. > In order for a single Java client to communicate with two different secure > clusters in different realms (only one of which can be the "default_realm"), > the client's krb5.conf file must specify both realms, and provide a > \[domain_realm\] section that maps cluster servers' domains to the correct > realms. With other appropriate behaviors (such as using the config from each > cluster to talk to the respective clusters, and a user principal from each > realm to talk to the respective realms), this is sufficient for most Hadoop > ecosystem clients. > But our SPNEGO using clients, such as Oozie, have a bug when it comes to > talking to a non-default realm. The default realm name gets incorrectly > inserted into the construction of the target server principal for the > non-default-realm cluster. Details and proposed solution are given in the > first comments below. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal
[ https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-12617: Attachment: HADOOP-12617.007.patch > SPNEGO authentication request to non-default realm gets default realm name > inserted in target server principal > -- > > Key: HADOOP-12617 > URL: https://issues.apache.org/jira/browse/HADOOP-12617 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.1 > Environment: Java client talking to two secure clusters in different > Kerberos realms, > or talking to any secure cluster in non-default realm >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-12617-branch-2.7.001.patch, > HADOOP-12617-branch-2.7.002.patch, HADOOP-12617.003.patch, > HADOOP-12617.005.patch, HADOOP-12617.006.patch, HADOOP-12617.007.patch > > > Note: This is NOT a vulnerability. > In order for a single Java client to communicate with two different secure > clusters in different realms (only one of which can be the "default_realm"), > the client's krb5.conf file must specify both realms, and provide a > \[domain_realm\] section that maps cluster servers' domains to the correct > realms. With other appropriate behaviors (such as using the config from each > cluster to talk to the respective clusters, and a user principal from each > realm to talk to the respective realms), this is sufficient for most Hadoop > ecosystem clients. > But our SPNEGO using clients, such as Oozie, have a bug when it comes to > talking to a non-default realm. The default realm name gets incorrectly > inserted into the construction of the target server principal for the > non-default-realm cluster. Details and proposed solution are given in the > first comments below. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal
[ https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15047453#comment-15047453 ] Matt Foley commented on HADOOP-12617: - [~vinayrpet], thanks for the review. Agree re IBM_JAVA, have changed it. Rather than mix usages, I also corrected two existing instances of 'System.getProperty("java.vendor").contains("IBM")', so now KerberosUtil.java uses solely the 'IBM_JAVA' usage. I only did this in the trunk patch, not for maintenance branches, since it is a cleanup not a bug fix. Will run it thru the Hadoop QA robot once more, then I will commit it per your +1. > SPNEGO authentication request to non-default realm gets default realm name > inserted in target server principal > -- > > Key: HADOOP-12617 > URL: https://issues.apache.org/jira/browse/HADOOP-12617 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.1 > Environment: Java client talking to two secure clusters in different > Kerberos realms, > or talking to any secure cluster in non-default realm >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-12617-branch-2.7.001.patch, > HADOOP-12617-branch-2.7.002.patch, HADOOP-12617.003.patch, > HADOOP-12617.005.patch, HADOOP-12617.006.patch > > > Note: This is NOT a vulnerability. > In order for a single Java client to communicate with two different secure > clusters in different realms (only one of which can be the "default_realm"), > the client's krb5.conf file must specify both realms, and provide a > \[domain_realm\] section that maps cluster servers' domains to the correct > realms. With other appropriate behaviors (such as using the config from each > cluster to talk to the respective clusters, and a user principal from each > realm to talk to the respective realms), this is sufficient for most Hadoop > ecosystem clients. > But our SPNEGO using clients, such as Oozie, have a bug when it comes to > talking to a non-default realm. The default realm name gets incorrectly > inserted into the construction of the target server principal for the > non-default-realm cluster. Details and proposed solution are given in the > first comments below. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal
[ https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-12617: Attachment: HADOOP-12617.008.patch > SPNEGO authentication request to non-default realm gets default realm name > inserted in target server principal > -- > > Key: HADOOP-12617 > URL: https://issues.apache.org/jira/browse/HADOOP-12617 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.1 > Environment: Java client talking to two secure clusters in different > Kerberos realms, > or talking to any secure cluster in non-default realm >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-12617-branch-2.7.001.patch, > HADOOP-12617-branch-2.7.002.patch, HADOOP-12617.003.patch, > HADOOP-12617.005.patch, HADOOP-12617.006.patch, HADOOP-12617.007.patch, > HADOOP-12617.008.patch > > > Note: This is NOT a vulnerability. > In order for a single Java client to communicate with two different secure > clusters in different realms (only one of which can be the "default_realm"), > the client's krb5.conf file must specify both realms, and provide a > \[domain_realm\] section that maps cluster servers' domains to the correct > realms. With other appropriate behaviors (such as using the config from each > cluster to talk to the respective clusters, and a user principal from each > realm to talk to the respective realms), this is sufficient for most Hadoop > ecosystem clients. > But our SPNEGO using clients, such as Oozie, have a bug when it comes to > talking to a non-default realm. The default realm name gets incorrectly > inserted into the construction of the target server principal for the > non-default-realm cluster. Details and proposed solution are given in the > first comments below. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal
[ https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-12617: Status: Open (was: Patch Available) > SPNEGO authentication request to non-default realm gets default realm name > inserted in target server principal > -- > > Key: HADOOP-12617 > URL: https://issues.apache.org/jira/browse/HADOOP-12617 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.1 > Environment: Java client talking to two secure clusters in different > Kerberos realms, > or talking to any secure cluster in non-default realm >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-12617-branch-2.7.001.patch, > HADOOP-12617-branch-2.7.002.patch, HADOOP-12617.003.patch, > HADOOP-12617.005.patch, HADOOP-12617.006.patch, HADOOP-12617.007.patch, > HADOOP-12617.008.patch > > > Note: This is NOT a vulnerability. > In order for a single Java client to communicate with two different secure > clusters in different realms (only one of which can be the "default_realm"), > the client's krb5.conf file must specify both realms, and provide a > \[domain_realm\] section that maps cluster servers' domains to the correct > realms. With other appropriate behaviors (such as using the config from each > cluster to talk to the respective clusters, and a user principal from each > realm to talk to the respective realms), this is sufficient for most Hadoop > ecosystem clients. > But our SPNEGO using clients, such as Oozie, have a bug when it comes to > talking to a non-default realm. The default realm name gets incorrectly > inserted into the construction of the target server principal for the > non-default-realm cluster. Details and proposed solution are given in the > first comments below. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal
[ https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley resolved HADOOP-12617. - Resolution: Fixed Fix Version/s: 2.8.0 > SPNEGO authentication request to non-default realm gets default realm name > inserted in target server principal > -- > > Key: HADOOP-12617 > URL: https://issues.apache.org/jira/browse/HADOOP-12617 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.1 > Environment: Java client talking to two secure clusters in different > Kerberos realms, > or talking to any secure cluster in non-default realm >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: 2.8.0 > > Attachments: HADOOP-12617-branch-2.7.001.patch, > HADOOP-12617-branch-2.7.003.patch, HADOOP-12617.003.patch, > HADOOP-12617.005.patch, HADOOP-12617.006.patch, HADOOP-12617.007.patch, > HADOOP-12617.008.patch > > > Note: This is NOT a vulnerability. > In order for a single Java client to communicate with two different secure > clusters in different realms (only one of which can be the "default_realm"), > the client's krb5.conf file must specify both realms, and provide a > \[domain_realm\] section that maps cluster servers' domains to the correct > realms. With other appropriate behaviors (such as using the config from each > cluster to talk to the respective clusters, and a user principal from each > realm to talk to the respective realms), this is sufficient for most Hadoop > ecosystem clients. > But our SPNEGO using clients, such as Oozie, have a bug when it comes to > talking to a non-default realm. The default realm name gets incorrectly > inserted into the construction of the target server principal for the > non-default-realm cluster. Details and proposed solution are given in the > first comments below. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal
[ https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-12617: Target Version/s: 2.8.0, 3.0.0 (was: 3.0.0, 2.7.3, 2.9.0) > SPNEGO authentication request to non-default realm gets default realm name > inserted in target server principal > -- > > Key: HADOOP-12617 > URL: https://issues.apache.org/jira/browse/HADOOP-12617 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.1 > Environment: Java client talking to two secure clusters in different > Kerberos realms, > or talking to any secure cluster in non-default realm >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-12617-branch-2.7.001.patch, > HADOOP-12617-branch-2.7.003.patch, HADOOP-12617.003.patch, > HADOOP-12617.005.patch, HADOOP-12617.006.patch, HADOOP-12617.007.patch, > HADOOP-12617.008.patch > > > Note: This is NOT a vulnerability. > In order for a single Java client to communicate with two different secure > clusters in different realms (only one of which can be the "default_realm"), > the client's krb5.conf file must specify both realms, and provide a > \[domain_realm\] section that maps cluster servers' domains to the correct > realms. With other appropriate behaviors (such as using the config from each > cluster to talk to the respective clusters, and a user principal from each > realm to talk to the respective realms), this is sufficient for most Hadoop > ecosystem clients. > But our SPNEGO using clients, such as Oozie, have a bug when it comes to > talking to a non-default realm. The default realm name gets incorrectly > inserted into the construction of the target server principal for the > non-default-realm cluster. Details and proposed solution are given in the > first comments below. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal
[ https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15047768#comment-15047768 ] Matt Foley commented on HADOOP-12617: - Happily, the spurious asflicense check went away. Also, I ran an IBM Java SDK (version 8.0) on a Linux CentOS 6 VM, and established that the correct fully qualified class name for PrincipalName is "com.ibm.security.krb5.PrincipalName", not "com.ibm.security.krb5.internal.PrincipalName". With the corrected path, it works as expected. So I'm uploading one more version of the patchfile, HADOOP-12617.008.patch, with that correction, but there's no need to run the robot again. Committing underway. > SPNEGO authentication request to non-default realm gets default realm name > inserted in target server principal > -- > > Key: HADOOP-12617 > URL: https://issues.apache.org/jira/browse/HADOOP-12617 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.1 > Environment: Java client talking to two secure clusters in different > Kerberos realms, > or talking to any secure cluster in non-default realm >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-12617-branch-2.7.001.patch, > HADOOP-12617-branch-2.7.002.patch, HADOOP-12617.003.patch, > HADOOP-12617.005.patch, HADOOP-12617.006.patch, HADOOP-12617.007.patch > > > Note: This is NOT a vulnerability. > In order for a single Java client to communicate with two different secure > clusters in different realms (only one of which can be the "default_realm"), > the client's krb5.conf file must specify both realms, and provide a > \[domain_realm\] section that maps cluster servers' domains to the correct > realms. With other appropriate behaviors (such as using the config from each > cluster to talk to the respective clusters, and a user principal from each > realm to talk to the respective realms), this is sufficient for most Hadoop > ecosystem clients. > But our SPNEGO using clients, such as Oozie, have a bug when it comes to > talking to a non-default realm. The default realm name gets incorrectly > inserted into the construction of the target server principal for the > non-default-realm cluster. Details and proposed solution are given in the > first comments below. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal
[ https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15047854#comment-15047854 ] Matt Foley commented on HADOOP-12617: - Committed to trunk as ada9c2c410c15e95, branch-2 as 472541291bf868c, branch-2.8 as 05a7fb3a9a02326cb46. Should therefore be expected in 2.8.0 and all later releases. > SPNEGO authentication request to non-default realm gets default realm name > inserted in target server principal > -- > > Key: HADOOP-12617 > URL: https://issues.apache.org/jira/browse/HADOOP-12617 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.1 > Environment: Java client talking to two secure clusters in different > Kerberos realms, > or talking to any secure cluster in non-default realm >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-12617-branch-2.7.001.patch, > HADOOP-12617-branch-2.7.003.patch, HADOOP-12617.003.patch, > HADOOP-12617.005.patch, HADOOP-12617.006.patch, HADOOP-12617.007.patch, > HADOOP-12617.008.patch > > > Note: This is NOT a vulnerability. > In order for a single Java client to communicate with two different secure > clusters in different realms (only one of which can be the "default_realm"), > the client's krb5.conf file must specify both realms, and provide a > \[domain_realm\] section that maps cluster servers' domains to the correct > realms. With other appropriate behaviors (such as using the config from each > cluster to talk to the respective clusters, and a user principal from each > realm to talk to the respective realms), this is sufficient for most Hadoop > ecosystem clients. > But our SPNEGO using clients, such as Oozie, have a bug when it comes to > talking to a non-default realm. The default realm name gets incorrectly > inserted into the construction of the target server principal for the > non-default-realm cluster. Details and proposed solution are given in the > first comments below. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal
[ https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-12617: Attachment: HADOOP-12617-branch-2.7.003.patch > SPNEGO authentication request to non-default realm gets default realm name > inserted in target server principal > -- > > Key: HADOOP-12617 > URL: https://issues.apache.org/jira/browse/HADOOP-12617 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.1 > Environment: Java client talking to two secure clusters in different > Kerberos realms, > or talking to any secure cluster in non-default realm >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-12617-branch-2.7.001.patch, > HADOOP-12617-branch-2.7.002.patch, HADOOP-12617-branch-2.7.003.patch, > HADOOP-12617.003.patch, HADOOP-12617.005.patch, HADOOP-12617.006.patch, > HADOOP-12617.007.patch, HADOOP-12617.008.patch > > > Note: This is NOT a vulnerability. > In order for a single Java client to communicate with two different secure > clusters in different realms (only one of which can be the "default_realm"), > the client's krb5.conf file must specify both realms, and provide a > \[domain_realm\] section that maps cluster servers' domains to the correct > realms. With other appropriate behaviors (such as using the config from each > cluster to talk to the respective clusters, and a user principal from each > realm to talk to the respective realms), this is sufficient for most Hadoop > ecosystem clients. > But our SPNEGO using clients, such as Oozie, have a bug when it comes to > talking to a non-default realm. The default realm name gets incorrectly > inserted into the construction of the target server principal for the > non-default-realm cluster. Details and proposed solution are given in the > first comments below. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal
[ https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-12617: Attachment: (was: HADOOP-12617-branch-2.7.002.patch) > SPNEGO authentication request to non-default realm gets default realm name > inserted in target server principal > -- > > Key: HADOOP-12617 > URL: https://issues.apache.org/jira/browse/HADOOP-12617 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.1 > Environment: Java client talking to two secure clusters in different > Kerberos realms, > or talking to any secure cluster in non-default realm >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-12617-branch-2.7.001.patch, > HADOOP-12617-branch-2.7.003.patch, HADOOP-12617.003.patch, > HADOOP-12617.005.patch, HADOOP-12617.006.patch, HADOOP-12617.007.patch, > HADOOP-12617.008.patch > > > Note: This is NOT a vulnerability. > In order for a single Java client to communicate with two different secure > clusters in different realms (only one of which can be the "default_realm"), > the client's krb5.conf file must specify both realms, and provide a > \[domain_realm\] section that maps cluster servers' domains to the correct > realms. With other appropriate behaviors (such as using the config from each > cluster to talk to the respective clusters, and a user principal from each > realm to talk to the respective realms), this is sufficient for most Hadoop > ecosystem clients. > But our SPNEGO using clients, such as Oozie, have a bug when it comes to > talking to a non-default realm. The default realm name gets incorrectly > inserted into the construction of the target server principal for the > non-default-realm cluster. Details and proposed solution are given in the > first comments below. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal
[ https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-12617: Attachment: HADOOP-12617.002.patch > SPNEGO authentication request to non-default realm gets default realm name > inserted in target server principal > -- > > Key: HADOOP-12617 > URL: https://issues.apache.org/jira/browse/HADOOP-12617 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.1 > Environment: Java client talking to two secure clusters in different > Kerberos realms, > or talking to any secure cluster in non-default realm >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-12617-branch-2.7.001.patch, > HADOOP-12617.001.patch, HADOOP-12617.002.patch > > > Note: This is NOT a vulnerability. > In order for a single Java client to communicate with two different secure > clusters in different realms (only one of which can be the "default_realm"), > the client's krb5.conf file must specify both realms, and provide a > \[domain_realm\] section that maps cluster servers' domains to the correct > realms. With other appropriate behaviors (such as using the config from each > cluster to talk to the respective clusters, and a user principal from each > realm to talk to the respective realms), this is sufficient for most Hadoop > ecosystem clients. > But our SPNEGO using clients, such as Oozie, have a bug when it comes to > talking to a non-default realm. The default realm name gets incorrectly > inserted into the construction of the target server principal for the > non-default-realm cluster. Details and proposed solution are given in the > first comments below. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal
[ https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045271#comment-15045271 ] Matt Foley commented on HADOOP-12617: - [~ste...@apache.org], not in a properly configured system. There is some evidence that if run on a system with Kerberos configured on but default realm not defined (either via System Property "java.security.krb5.realm" or krb5.conf parameter "default_realm"), that getDefaultRealm() may throw an exception. But that's a fairly gross misconfiguration. Do you think the error should be suppressed without logging? I had wondered why there was no logging in the KerberosUtils class. Also, does anyone listening have access to the documentation for IBM Java package "com.ibm.security.krb5"? getDomainRealm() follows the model of getDefaultRealm(), but I was unable to confirm that com.ibm.security.krb5.internal.PrincipalName exists and respects the semantics of principal string type "KRB_NT_SRV_HST". Thanks. > SPNEGO authentication request to non-default realm gets default realm name > inserted in target server principal > -- > > Key: HADOOP-12617 > URL: https://issues.apache.org/jira/browse/HADOOP-12617 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.1 > Environment: Java client talking to two secure clusters in different > Kerberos realms, > or talking to any secure cluster in non-default realm >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-12617-branch-2.7.001.patch, HADOOP-12617.001.patch > > > Note: This is NOT a vulnerability. > In order for a single Java client to communicate with two different secure > clusters in different realms (only one of which can be the "default_realm"), > the client's krb5.conf file must specify both realms, and provide a > \[domain_realm\] section that maps cluster servers' domains to the correct > realms. With other appropriate behaviors (such as using the config from each > cluster to talk to the respective clusters, and a user principal from each > realm to talk to the respective realms), this is sufficient for most Hadoop > ecosystem clients. > But our SPNEGO using clients, such as Oozie, have a bug when it comes to > talking to a non-default realm. The default realm name gets incorrectly > inserted into the construction of the target server principal for the > non-default-realm cluster. Details and proposed solution are given in the > first comments below. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal
[ https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045271#comment-15045271 ] Matt Foley edited comment on HADOOP-12617 at 12/7/15 5:43 PM: -- [~ste...@apache.org], not in a properly configured system. There is some evidence that if run on a system with Kerberos configured on but default realm not defined (either via System Property "java.security.krb5.realm" or krb5.conf parameter "default_realm"), that getDefaultRealm() or getDomainRealm() may throw an exception. But that's a fairly gross misconfiguration. Do you think the error should be suppressed without logging? I had wondered why there was no logging in the KerberosUtils class. Also, does anyone listening have access to the documentation for IBM Java package "com.ibm.security.krb5"? getDomainRealm() follows the model of getDefaultRealm(), but I was unable to confirm that com.ibm.security.krb5.internal.PrincipalName exists and respects the semantics of principal string type "KRB_NT_SRV_HST". Thanks. was (Author: mattf): [~ste...@apache.org], not in a properly configured system. There is some evidence that if run on a system with Kerberos configured on but default realm not defined (either via System Property "java.security.krb5.realm" or krb5.conf parameter "default_realm"), that getDefaultRealm() may throw an exception. But that's a fairly gross misconfiguration. Do you think the error should be suppressed without logging? I had wondered why there was no logging in the KerberosUtils class. Also, does anyone listening have access to the documentation for IBM Java package "com.ibm.security.krb5"? getDomainRealm() follows the model of getDefaultRealm(), but I was unable to confirm that com.ibm.security.krb5.internal.PrincipalName exists and respects the semantics of principal string type "KRB_NT_SRV_HST". Thanks. > SPNEGO authentication request to non-default realm gets default realm name > inserted in target server principal > -- > > Key: HADOOP-12617 > URL: https://issues.apache.org/jira/browse/HADOOP-12617 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.1 > Environment: Java client talking to two secure clusters in different > Kerberos realms, > or talking to any secure cluster in non-default realm >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-12617-branch-2.7.001.patch, > HADOOP-12617.001.patch, HADOOP-12617.002.patch > > > Note: This is NOT a vulnerability. > In order for a single Java client to communicate with two different secure > clusters in different realms (only one of which can be the "default_realm"), > the client's krb5.conf file must specify both realms, and provide a > \[domain_realm\] section that maps cluster servers' domains to the correct > realms. With other appropriate behaviors (such as using the config from each > cluster to talk to the respective clusters, and a user principal from each > realm to talk to the respective realms), this is sufficient for most Hadoop > ecosystem clients. > But our SPNEGO using clients, such as Oozie, have a bug when it comes to > talking to a non-default realm. The default realm name gets incorrectly > inserted into the construction of the target server principal for the > non-default-realm cluster. Details and proposed solution are given in the > first comments below. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal
[ https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-12617: Attachment: HADOOP-12617.003.patch > SPNEGO authentication request to non-default realm gets default realm name > inserted in target server principal > -- > > Key: HADOOP-12617 > URL: https://issues.apache.org/jira/browse/HADOOP-12617 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.1 > Environment: Java client talking to two secure clusters in different > Kerberos realms, > or talking to any secure cluster in non-default realm >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-12617-branch-2.7.001.patch, > HADOOP-12617.001.patch, HADOOP-12617.002.patch, HADOOP-12617.003.patch > > > Note: This is NOT a vulnerability. > In order for a single Java client to communicate with two different secure > clusters in different realms (only one of which can be the "default_realm"), > the client's krb5.conf file must specify both realms, and provide a > \[domain_realm\] section that maps cluster servers' domains to the correct > realms. With other appropriate behaviors (such as using the config from each > cluster to talk to the respective clusters, and a user principal from each > realm to talk to the respective realms), this is sufficient for most Hadoop > ecosystem clients. > But our SPNEGO using clients, such as Oozie, have a bug when it comes to > talking to a non-default realm. The default realm name gets incorrectly > inserted into the construction of the target server principal for the > non-default-realm cluster. Details and proposed solution are given in the > first comments below. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal
[ https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045271#comment-15045271 ] Matt Foley edited comment on HADOOP-12617 at 12/7/15 8:32 PM: -- [~ste...@apache.org], not in a properly configured system. There is some evidence that if run on a system with Kerberos configured on but default realm not defined (either via System Property "java.security.krb5.realm" or krb5.conf parameter "default_realm"), that getDefaultRealm() or getDomainRealm() may throw an exception. But that's a fairly gross misconfiguration. Also, I put a call to both getDefaultRealm() and getDomainRealm() in the revised unit test, and they did not throw an exception on the Jenkins test host, which may or may not be well configured. Do you think the error should be suppressed without logging? I had wondered why there was no logging in the KerberosUtils class. Also, does anyone listening have access to the documentation for IBM Java package "com.ibm.security.krb5"? getDomainRealm() follows the model of getDefaultRealm(), but I was unable to confirm that com.ibm.security.krb5.internal.PrincipalName exists and respects the semantics of principal string type "KRB_NT_SRV_HST". Thanks. was (Author: mattf): [~ste...@apache.org], not in a properly configured system. There is some evidence that if run on a system with Kerberos configured on but default realm not defined (either via System Property "java.security.krb5.realm" or krb5.conf parameter "default_realm"), that getDefaultRealm() or getDomainRealm() may throw an exception. But that's a fairly gross misconfiguration. Do you think the error should be suppressed without logging? I had wondered why there was no logging in the KerberosUtils class. Also, does anyone listening have access to the documentation for IBM Java package "com.ibm.security.krb5"? getDomainRealm() follows the model of getDefaultRealm(), but I was unable to confirm that com.ibm.security.krb5.internal.PrincipalName exists and respects the semantics of principal string type "KRB_NT_SRV_HST". Thanks. > SPNEGO authentication request to non-default realm gets default realm name > inserted in target server principal > -- > > Key: HADOOP-12617 > URL: https://issues.apache.org/jira/browse/HADOOP-12617 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.1 > Environment: Java client talking to two secure clusters in different > Kerberos realms, > or talking to any secure cluster in non-default realm >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-12617-branch-2.7.001.patch, > HADOOP-12617.001.patch, HADOOP-12617.002.patch, HADOOP-12617.003.patch > > > Note: This is NOT a vulnerability. > In order for a single Java client to communicate with two different secure > clusters in different realms (only one of which can be the "default_realm"), > the client's krb5.conf file must specify both realms, and provide a > \[domain_realm\] section that maps cluster servers' domains to the correct > realms. With other appropriate behaviors (such as using the config from each > cluster to talk to the respective clusters, and a user principal from each > realm to talk to the respective realms), this is sufficient for most Hadoop > ecosystem clients. > But our SPNEGO using clients, such as Oozie, have a bug when it comes to > talking to a non-default realm. The default realm name gets incorrectly > inserted into the construction of the target server principal for the > non-default-realm cluster. Details and proposed solution are given in the > first comments below. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal
[ https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-12617: Attachment: (was: HADOOP-12617.004.patch) > SPNEGO authentication request to non-default realm gets default realm name > inserted in target server principal > -- > > Key: HADOOP-12617 > URL: https://issues.apache.org/jira/browse/HADOOP-12617 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.1 > Environment: Java client talking to two secure clusters in different > Kerberos realms, > or talking to any secure cluster in non-default realm >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-12617-branch-2.7.001.patch, > HADOOP-12617.003.patch, HADOOP-12617.005.patch > > > Note: This is NOT a vulnerability. > In order for a single Java client to communicate with two different secure > clusters in different realms (only one of which can be the "default_realm"), > the client's krb5.conf file must specify both realms, and provide a > \[domain_realm\] section that maps cluster servers' domains to the correct > realms. With other appropriate behaviors (such as using the config from each > cluster to talk to the respective clusters, and a user principal from each > realm to talk to the respective realms), this is sufficient for most Hadoop > ecosystem clients. > But our SPNEGO using clients, such as Oozie, have a bug when it comes to > talking to a non-default realm. The default realm name gets incorrectly > inserted into the construction of the target server principal for the > non-default-realm cluster. Details and proposed solution are given in the > first comments below. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal
[ https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-12617: Attachment: HADOOP-12617.005.patch > SPNEGO authentication request to non-default realm gets default realm name > inserted in target server principal > -- > > Key: HADOOP-12617 > URL: https://issues.apache.org/jira/browse/HADOOP-12617 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.1 > Environment: Java client talking to two secure clusters in different > Kerberos realms, > or talking to any secure cluster in non-default realm >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-12617-branch-2.7.001.patch, > HADOOP-12617.003.patch, HADOOP-12617.005.patch > > > Note: This is NOT a vulnerability. > In order for a single Java client to communicate with two different secure > clusters in different realms (only one of which can be the "default_realm"), > the client's krb5.conf file must specify both realms, and provide a > \[domain_realm\] section that maps cluster servers' domains to the correct > realms. With other appropriate behaviors (such as using the config from each > cluster to talk to the respective clusters, and a user principal from each > realm to talk to the respective realms), this is sufficient for most Hadoop > ecosystem clients. > But our SPNEGO using clients, such as Oozie, have a bug when it comes to > talking to a non-default realm. The default realm name gets incorrectly > inserted into the construction of the target server principal for the > non-default-realm cluster. Details and proposed solution are given in the > first comments below. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal
[ https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-12617: Attachment: (was: HADOOP-12617.001.patch) > SPNEGO authentication request to non-default realm gets default realm name > inserted in target server principal > -- > > Key: HADOOP-12617 > URL: https://issues.apache.org/jira/browse/HADOOP-12617 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.1 > Environment: Java client talking to two secure clusters in different > Kerberos realms, > or talking to any secure cluster in non-default realm >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-12617-branch-2.7.001.patch, > HADOOP-12617.003.patch, HADOOP-12617.004.patch > > > Note: This is NOT a vulnerability. > In order for a single Java client to communicate with two different secure > clusters in different realms (only one of which can be the "default_realm"), > the client's krb5.conf file must specify both realms, and provide a > \[domain_realm\] section that maps cluster servers' domains to the correct > realms. With other appropriate behaviors (such as using the config from each > cluster to talk to the respective clusters, and a user principal from each > realm to talk to the respective realms), this is sufficient for most Hadoop > ecosystem clients. > But our SPNEGO using clients, such as Oozie, have a bug when it comes to > talking to a non-default realm. The default realm name gets incorrectly > inserted into the construction of the target server principal for the > non-default-realm cluster. Details and proposed solution are given in the > first comments below. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal
[ https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-12617: Attachment: (was: HADOOP-12617.002.patch) > SPNEGO authentication request to non-default realm gets default realm name > inserted in target server principal > -- > > Key: HADOOP-12617 > URL: https://issues.apache.org/jira/browse/HADOOP-12617 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.1 > Environment: Java client talking to two secure clusters in different > Kerberos realms, > or talking to any secure cluster in non-default realm >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-12617-branch-2.7.001.patch, > HADOOP-12617.003.patch, HADOOP-12617.004.patch > > > Note: This is NOT a vulnerability. > In order for a single Java client to communicate with two different secure > clusters in different realms (only one of which can be the "default_realm"), > the client's krb5.conf file must specify both realms, and provide a > \[domain_realm\] section that maps cluster servers' domains to the correct > realms. With other appropriate behaviors (such as using the config from each > cluster to talk to the respective clusters, and a user principal from each > realm to talk to the respective realms), this is sufficient for most Hadoop > ecosystem clients. > But our SPNEGO using clients, such as Oozie, have a bug when it comes to > talking to a non-default realm. The default realm name gets incorrectly > inserted into the construction of the target server principal for the > non-default-realm cluster. Details and proposed solution are given in the > first comments below. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal
[ https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-12617: Attachment: (was: HADOOP-12617.004.patch) > SPNEGO authentication request to non-default realm gets default realm name > inserted in target server principal > -- > > Key: HADOOP-12617 > URL: https://issues.apache.org/jira/browse/HADOOP-12617 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.1 > Environment: Java client talking to two secure clusters in different > Kerberos realms, > or talking to any secure cluster in non-default realm >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-12617-branch-2.7.001.patch, HADOOP-12617.003.patch > > > Note: This is NOT a vulnerability. > In order for a single Java client to communicate with two different secure > clusters in different realms (only one of which can be the "default_realm"), > the client's krb5.conf file must specify both realms, and provide a > \[domain_realm\] section that maps cluster servers' domains to the correct > realms. With other appropriate behaviors (such as using the config from each > cluster to talk to the respective clusters, and a user principal from each > realm to talk to the respective realms), this is sufficient for most Hadoop > ecosystem clients. > But our SPNEGO using clients, such as Oozie, have a bug when it comes to > talking to a non-default realm. The default realm name gets incorrectly > inserted into the construction of the target server principal for the > non-default-realm cluster. Details and proposed solution are given in the > first comments below. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal
[ https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-12617: Attachment: HADOOP-12617.004.patch > SPNEGO authentication request to non-default realm gets default realm name > inserted in target server principal > -- > > Key: HADOOP-12617 > URL: https://issues.apache.org/jira/browse/HADOOP-12617 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.1 > Environment: Java client talking to two secure clusters in different > Kerberos realms, > or talking to any secure cluster in non-default realm >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-12617-branch-2.7.001.patch, > HADOOP-12617.001.patch, HADOOP-12617.002.patch, HADOOP-12617.003.patch, > HADOOP-12617.004.patch > > > Note: This is NOT a vulnerability. > In order for a single Java client to communicate with two different secure > clusters in different realms (only one of which can be the "default_realm"), > the client's krb5.conf file must specify both realms, and provide a > \[domain_realm\] section that maps cluster servers' domains to the correct > realms. With other appropriate behaviors (such as using the config from each > cluster to talk to the respective clusters, and a user principal from each > realm to talk to the respective realms), this is sufficient for most Hadoop > ecosystem clients. > But our SPNEGO using clients, such as Oozie, have a bug when it comes to > talking to a non-default realm. The default realm name gets incorrectly > inserted into the construction of the target server principal for the > non-default-realm cluster. Details and proposed solution are given in the > first comments below. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal
[ https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-12617: Attachment: HADOOP-12617.004.patch > SPNEGO authentication request to non-default realm gets default realm name > inserted in target server principal > -- > > Key: HADOOP-12617 > URL: https://issues.apache.org/jira/browse/HADOOP-12617 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.1 > Environment: Java client talking to two secure clusters in different > Kerberos realms, > or talking to any secure cluster in non-default realm >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-12617-branch-2.7.001.patch, > HADOOP-12617.003.patch, HADOOP-12617.004.patch > > > Note: This is NOT a vulnerability. > In order for a single Java client to communicate with two different secure > clusters in different realms (only one of which can be the "default_realm"), > the client's krb5.conf file must specify both realms, and provide a > \[domain_realm\] section that maps cluster servers' domains to the correct > realms. With other appropriate behaviors (such as using the config from each > cluster to talk to the respective clusters, and a user principal from each > realm to talk to the respective realms), this is sufficient for most Hadoop > ecosystem clients. > But our SPNEGO using clients, such as Oozie, have a bug when it comes to > talking to a non-default realm. The default realm name gets incorrectly > inserted into the construction of the target server principal for the > non-default-realm cluster. Details and proposed solution are given in the > first comments below. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal
[ https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-12617: Status: Patch Available (was: Open) > SPNEGO authentication request to non-default realm gets default realm name > inserted in target server principal > -- > > Key: HADOOP-12617 > URL: https://issues.apache.org/jira/browse/HADOOP-12617 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.1 > Environment: Java client talking to two secure clusters in different > Kerberos realms, > or talking to any secure cluster in non-default realm >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-12617-branch-2.7.001.patch, HADOOP-12617.001.patch > > > Note: This is NOT a vulnerability. > In order for a single Java client to communicate with two different secure > clusters in different realms (only one of which can be the "default_realm"), > the client's krb5.conf file must specify both realms, and provide a > \[domain_realm\] section that maps cluster servers' domains to the correct > realms. With other appropriate behaviors (such as using the config from each > cluster to talk to the respective clusters, and a user principal from each > realm to talk to the respective realms), this is sufficient for most Hadoop > ecosystem clients. > But our SPNEGO using clients, such as Oozie, have a bug when it comes to > talking to a non-default realm. The default realm name gets incorrectly > inserted into the construction of the target server principal for the > non-default-realm cluster. Details and proposed solution are given in the > first comments below. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal
[ https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044177#comment-15044177 ] Matt Foley commented on HADOOP-12617: - Teradata engineering team provided debugger information leading to the following analysis: A Java client desiring to initiate communication via HTTP/SPNEGO does the following: In the context of a UGI.doAs(), it calls org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(), which in turn calls doSpnegoSequence(). Inside doSpnegoSequence(), is this code at https://github.com/apache/hadoop/blob/branch-2.7.2/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java#L287-L298 : {code} Subject.doAs(subject, new PrivilegedExceptionAction() { public Void run() throws Exception { GSSContext gssContext = null; try { GSSManager gssManager = GSSManager.getInstance(); String servicePrincipal = KerberosUtil.getServicePrincipal("HTTP", KerberosAuthenticator.this.url.getHost()); Oid oid = KerberosUtil.getOidInstance("NT_GSS_KRB5_PRINCIPAL"); GSSName serviceName = gssManager.createName(servicePrincipal, oid); {code} Subject is set with the correct user principal and target server name and realm. After executing the line: {code} String servicePrincipal = KerberosUtil.getServicePrincipal("HTTP", KerberosAuthenticator.this.url.getHost()); {code} The ServicePrincipal is returned as: "HTTP/hostname" (with an actual value for "hostname"), but no realm. Then, when it gets the serviceName {code} GSSName serviceName = gssManager.createName(servicePrincipal, oid); {code} The resulting data structure includes a subfield Krb5PrincipalName with value "HTTP/hostname@DEFAULTREALM", where "hostname" and "DEFAULTREALM" are of course substituted with real values. The default realm should not be here. It is required that the correct non-default realm should be derived from the domain of "hostname". However, the second component of a principal is, strictly speaking, the "instance", not the "server". Thus, the Kerb libraries are not inferring the realm from the domain portion of the server name, because the formal semantics don't actually match; and of course there is nowhere else the realm could be correctly inferred from. KerberosUtil.getServicePrincipal() has to return a full principal with correct realm, rather than the short principal with inferred default realm. > SPNEGO authentication request to non-default realm gets default realm name > inserted in target server principal > -- > > Key: HADOOP-12617 > URL: https://issues.apache.org/jira/browse/HADOOP-12617 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.1 > Environment: Java client talking to two secure clusters in different > Kerberos realms, > or talking to any secure cluster in non-default realm >Reporter: Matt Foley >Assignee: Matt Foley > > Note: This is NOT a vulnerability. > In order for a single Java client to communicate with two different secure > clusters in different realms (only one of which can be the "default_realm"), > the client's krb5.conf file must specify both realms, and provide a > \[domain_realm\] section that maps cluster servers' domains to the correct > realms. With other appropriate behaviors (such as using the config from each > cluster to talk to the respective clusters, and a user principal from each > realm to talk to the respective realms), this is sufficient for most Hadoop > ecosystem clients. > But our SPNEGO using clients, such as Oozie, have a bug when it comes to > talking to a non-default realm. The default realm name gets incorrectly > inserted into the construction of the target server principal for the > non-default-realm cluster. Details and proposed solution are given in the > first comments below. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal
[ https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-12617: Attachment: HADOOP-12617.001.patch HADOOP-12617-branch-2.7.001.patch > SPNEGO authentication request to non-default realm gets default realm name > inserted in target server principal > -- > > Key: HADOOP-12617 > URL: https://issues.apache.org/jira/browse/HADOOP-12617 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.1 > Environment: Java client talking to two secure clusters in different > Kerberos realms, > or talking to any secure cluster in non-default realm >Reporter: Matt Foley >Assignee: Matt Foley > Attachments: HADOOP-12617-branch-2.7.001.patch, HADOOP-12617.001.patch > > > Note: This is NOT a vulnerability. > In order for a single Java client to communicate with two different secure > clusters in different realms (only one of which can be the "default_realm"), > the client's krb5.conf file must specify both realms, and provide a > \[domain_realm\] section that maps cluster servers' domains to the correct > realms. With other appropriate behaviors (such as using the config from each > cluster to talk to the respective clusters, and a user principal from each > realm to talk to the respective realms), this is sufficient for most Hadoop > ecosystem clients. > But our SPNEGO using clients, such as Oozie, have a bug when it comes to > talking to a non-default realm. The default realm name gets incorrectly > inserted into the construction of the target server principal for the > non-default-realm cluster. Details and proposed solution are given in the > first comments below. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal
Matt Foley created HADOOP-12617: --- Summary: SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal Key: HADOOP-12617 URL: https://issues.apache.org/jira/browse/HADOOP-12617 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 2.7.1 Environment: Java client talking to two secure clusters in different Kerberos realms, or talking to any secure cluster in non-default realm Reporter: Matt Foley Assignee: Matt Foley Note: This is NOT a vulnerability. In order for a single Java client to communicate with two different secure clusters in different realms (only one of which can be the "default_realm"), the client's krb5.conf file must specify both realms, and provide a \[domain_realm\] section that maps cluster servers' domains to the correct realms. With other appropriate behaviors (such as using the config from each cluster to talk to the respective clusters, and a user principal from each realm to talk to the respective realms), this is sufficient for most Hadoop ecosystem clients. But our SPNEGO using clients, such as Oozie, have a bug when it comes to talking to a non-default realm. The default realm name gets incorrectly inserted into the construction of the target server principal for the non-default-realm cluster. Details and proposed solution are given in the first comments below. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal
[ https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044180#comment-15044180 ] Matt Foley commented on HADOOP-12617: - The proposed patch provides the required functionality with no changes to API signatures or exceptions, and following the model of other code in KerberosUtils.java. Furthermore, I grepped all java files in a broad set of Hadoop ecosystem components: * accumulo, atlas, calcite, datafu, falcon, flume, hadoop, hbase, hive, hue, kafka, knox, mahout, oozie, phoenix, pig, ranger, slider, spark, sqoop, storm, tez, zookeeper and found that no code calls KerberosUtils.getServicePrincipal() directly except that in the Hadoop auth package. Both instances there use the result ONLY as an argument for an immediate call to gssManager.createName(), which accepts the fully qualified principal and does the right thing with it. So I believe this patch is safe. > SPNEGO authentication request to non-default realm gets default realm name > inserted in target server principal > -- > > Key: HADOOP-12617 > URL: https://issues.apache.org/jira/browse/HADOOP-12617 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.1 > Environment: Java client talking to two secure clusters in different > Kerberos realms, > or talking to any secure cluster in non-default realm >Reporter: Matt Foley >Assignee: Matt Foley > > Note: This is NOT a vulnerability. > In order for a single Java client to communicate with two different secure > clusters in different realms (only one of which can be the "default_realm"), > the client's krb5.conf file must specify both realms, and provide a > \[domain_realm\] section that maps cluster servers' domains to the correct > realms. With other appropriate behaviors (such as using the config from each > cluster to talk to the respective clusters, and a user principal from each > realm to talk to the respective realms), this is sufficient for most Hadoop > ecosystem clients. > But our SPNEGO using clients, such as Oozie, have a bug when it comes to > talking to a non-default realm. The default realm name gets incorrectly > inserted into the construction of the target server principal for the > non-default-realm cluster. Details and proposed solution are given in the > first comments below. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12617) SPNEGO authentication request to non-default realm gets default realm name inserted in target server principal
[ https://issues.apache.org/jira/browse/HADOOP-12617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-12617: Target Version/s: 3.0.0, 2.7.3, 2.9.0 > SPNEGO authentication request to non-default realm gets default realm name > inserted in target server principal > -- > > Key: HADOOP-12617 > URL: https://issues.apache.org/jira/browse/HADOOP-12617 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.1 > Environment: Java client talking to two secure clusters in different > Kerberos realms, > or talking to any secure cluster in non-default realm >Reporter: Matt Foley >Assignee: Matt Foley > > Note: This is NOT a vulnerability. > In order for a single Java client to communicate with two different secure > clusters in different realms (only one of which can be the "default_realm"), > the client's krb5.conf file must specify both realms, and provide a > \[domain_realm\] section that maps cluster servers' domains to the correct > realms. With other appropriate behaviors (such as using the config from each > cluster to talk to the respective clusters, and a user principal from each > realm to talk to the respective realms), this is sufficient for most Hadoop > ecosystem clients. > But our SPNEGO using clients, such as Oozie, have a bug when it comes to > talking to a non-default realm. The default realm name gets incorrectly > inserted into the construction of the target server principal for the > non-default-realm cluster. Details and proposed solution are given in the > first comments below. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12437) Allow SecurityUtil to lookup alternate hostnames
[ https://issues.apache.org/jira/browse/HADOOP-12437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-12437: Environment: multi-homed > Allow SecurityUtil to lookup alternate hostnames > - > > Key: HADOOP-12437 > URL: https://issues.apache.org/jira/browse/HADOOP-12437 > Project: Hadoop Common > Issue Type: Bug > Components: net, security > Environment: multi-homed >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Fix For: 2.8.0 > > Attachments: HADOOP-12437.04.patch, HADOOP-12437.05.patch, > HDFS-9109.01.patch, HDFS-9109.02.patch, HDFS-9109.03.patch > > > The configuration setting {{dfs.datanode.dns.interface}} lets the DataNode > select its hostname by doing a reverse lookup of IP addresses on the specific > network interface. This does not work {{when /etc/hosts}} is used to setup > alternate hostnames, since {{DNS#reverseDns}} only queries the DNS servers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-9730) fix hadoop.spec to add task-log4j.properties
[ https://issues.apache.org/jira/browse/HADOOP-9730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13728989#comment-13728989 ] Matt Foley commented on HADOOP-9730: Committed to branch-1 on 8/4/2013 while checking for consistency between 1.2.1 release and branch-1. fix hadoop.spec to add task-log4j.properties - Key: HADOOP-9730 URL: https://issues.apache.org/jira/browse/HADOOP-9730 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 1.2.1 Reporter: Giridharan Kesavan Assignee: Giridharan Kesavan Fix For: 1.2.1 Attachments: HADOOP-9730.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9730) fix hadoop.spec to add task-log4j.properties
[ https://issues.apache.org/jira/browse/HADOOP-9730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley resolved HADOOP-9730. Resolution: Fixed Fix Version/s: 1.2.1 +1. Thanks, Giri, that fixed the problem. fix hadoop.spec to add task-log4j.properties - Key: HADOOP-9730 URL: https://issues.apache.org/jira/browse/HADOOP-9730 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 1.2.1 Reporter: Giridharan Kesavan Assignee: Giridharan Kesavan Fix For: 1.2.1 Attachments: HADOOP-9730.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9668) Configuration#iterator method should interpret variables in a property as well as Configuration#get method
[ https://issues.apache.org/jira/browse/HADOOP-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-9668: --- Target Version/s: 2.1.0-beta, 1.3.0 (was: 2.1.0-beta, 1.2.1) Configuration#iterator method should interpret variables in a property as well as Configuration#get method -- Key: HADOOP-9668 URL: https://issues.apache.org/jira/browse/HADOOP-9668 Project: Hadoop Common Issue Type: Bug Components: conf Affects Versions: 2.1.0-beta, 1.2.1 Reporter: Kousuke Saruta Priority: Minor Labels: newbie o.a.h.conf.Configuration#get method interpret variables in a property in *-site.xml but Configuration#iterator method doesn't. For example, when a property user.name is set to hadoop and another property hadoop.tmp.dir is set to /tmp/${user.name}, Configuration#get interpret hadoop.tmp.dir as /tmp/hadoop but Configuration#iterator doesn't interpret. I think Configuration#iterator should interpret variables or if there are some reasons, it should be documented that Configuration#iterator doesn't interpret variables. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9668) Configuration#iterator method should interpret variables in a property as well as Configuration#get method
[ https://issues.apache.org/jira/browse/HADOOP-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708162#comment-13708162 ] Matt Foley commented on HADOOP-9668: Since work has not yet started, and we are about to produce 1.2.1-rc, deferring this to 1.3.0. Configuration#iterator method should interpret variables in a property as well as Configuration#get method -- Key: HADOOP-9668 URL: https://issues.apache.org/jira/browse/HADOOP-9668 Project: Hadoop Common Issue Type: Bug Components: conf Affects Versions: 2.1.0-beta, 1.2.1 Reporter: Kousuke Saruta Priority: Minor Labels: newbie o.a.h.conf.Configuration#get method interpret variables in a property in *-site.xml but Configuration#iterator method doesn't. For example, when a property user.name is set to hadoop and another property hadoop.tmp.dir is set to /tmp/${user.name}, Configuration#get interpret hadoop.tmp.dir as /tmp/hadoop but Configuration#iterator doesn't interpret. I think Configuration#iterator should interpret variables or if there are some reasons, it should be documented that Configuration#iterator doesn't interpret variables. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9504) MetricsDynamicMBeanBase has concurrency issues in createMBeanInfo
[ https://issues.apache.org/jira/browse/HADOOP-9504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley resolved HADOOP-9504. Resolution: Fixed Target Version/s: 0.23.8, 2.1.0-beta, 1.2.1 (was: 2.1.0-beta, 0.23.8, 1.2.1) Committed to branch-1 and branch-1.2. Thanks, Liang and Jason! MetricsDynamicMBeanBase has concurrency issues in createMBeanInfo - Key: HADOOP-9504 URL: https://issues.apache.org/jira/browse/HADOOP-9504 Project: Hadoop Common Issue Type: Bug Components: metrics Affects Versions: 3.0.0, 2.0.3-alpha Reporter: Liang Xie Assignee: Liang Xie Priority: Critical Fix For: 2.1.0-beta, 1.2.1, 0.23.8 Attachments: HADOOP-9504-branch-1.txt, HADOOP-9504.txt, HADOOP-9504-v2.txt Please see HBASE-8416 for detail information. we need to take care of the synchronization for HashMap put(), otherwise it may lead to spin loop. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9504) MetricsDynamicMBeanBase has concurrency issues in createMBeanInfo
[ https://issues.apache.org/jira/browse/HADOOP-9504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-9504: --- Fix Version/s: 1.2.1 MetricsDynamicMBeanBase has concurrency issues in createMBeanInfo - Key: HADOOP-9504 URL: https://issues.apache.org/jira/browse/HADOOP-9504 Project: Hadoop Common Issue Type: Bug Components: metrics Affects Versions: 3.0.0, 2.0.3-alpha Reporter: Liang Xie Assignee: Liang Xie Priority: Critical Fix For: 2.1.0-beta, 0.23.8, 1.2.1 Attachments: HADOOP-9504-branch-1.txt, HADOOP-9504.txt, HADOOP-9504-v2.txt Please see HBASE-8416 for detail information. we need to take care of the synchronization for HashMap put(), otherwise it may lead to spin loop. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9573) Fix test-patch script to work with the enhanced PreCommit-Admin script.
[ https://issues.apache.org/jira/browse/HADOOP-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666428#comment-13666428 ] Matt Foley commented on HADOOP-9573: +1, Thanks, Giri! Fix test-patch script to work with the enhanced PreCommit-Admin script. --- Key: HADOOP-9573 URL: https://issues.apache.org/jira/browse/HADOOP-9573 Project: Hadoop Common Issue Type: Sub-task Components: build Affects Versions: 1.0.3 Reporter: Giridharan Kesavan Assignee: Giridharan Kesavan Attachments: 0001-Fix-test-patch-scrit-to-work-with-the-enhanced-PreCo.patch, hadoop-9573.patch test-patch script currently take the latest available patch for a given jira and performs the test. This jira is to enhance test-patch script to take attachment-id of a patch as an input and perform the tests using that attachment-id -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9573) Fix test-patch script to work with the enhanced PreCommit-Admin script.
[ https://issues.apache.org/jira/browse/HADOOP-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662371#comment-13662371 ] Matt Foley commented on HADOOP-9573: Exchanged comments with Giri: .bq What happened to CheckStyle? It looks like it got commented out some time ago, but why? Or was it entirely replaced by FindBugs? [~gkesavan]: Nope, we never had checkstyle in test-patch working. Right from the early days checkstyle function was always commented out, so this time around I removed it instead of leaving it as a comment. It would be very painful to implement checkstyle at this point in time, as making sure hadoop code base is complaint with checkstyle won't be that easy. Fix test-patch script to work with the enhanced PreCommit-Admin script. --- Key: HADOOP-9573 URL: https://issues.apache.org/jira/browse/HADOOP-9573 Project: Hadoop Common Issue Type: Sub-task Components: build Affects Versions: 1.0.3 Reporter: Giridharan Kesavan Assignee: Giridharan Kesavan Attachments: 0001-Fix-test-patch-scrit-to-work-with-the-enhanced-PreCo.patch test-patch script currently take the latest available patch for a given jira and performs the test. This jira is to enhance test-patch script to take attachment-id of a patch as an input and perform the tests using that attachment-id -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9573) Fix test-patch script to work with the enhanced PreCommit-Admin script.
[ https://issues.apache.org/jira/browse/HADOOP-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662390#comment-13662390 ] Matt Foley commented on HADOOP-9573: Review posted in reviewboard. Lot of good work, a few things need cleanup. I assume you've tested this with test Jiras. Please describe tests done. Thanks, Giri. Fix test-patch script to work with the enhanced PreCommit-Admin script. --- Key: HADOOP-9573 URL: https://issues.apache.org/jira/browse/HADOOP-9573 Project: Hadoop Common Issue Type: Sub-task Components: build Affects Versions: 1.0.3 Reporter: Giridharan Kesavan Assignee: Giridharan Kesavan Attachments: 0001-Fix-test-patch-scrit-to-work-with-the-enhanced-PreCo.patch test-patch script currently take the latest available patch for a given jira and performs the test. This jira is to enhance test-patch script to take attachment-id of a patch as an input and perform the tests using that attachment-id -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HADOOP-9573) Fix test-patch script to work with the enhanced PreCommit-Admin script.
[ https://issues.apache.org/jira/browse/HADOOP-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662371#comment-13662371 ] Matt Foley edited comment on HADOOP-9573 at 5/20/13 10:01 PM: -- Exchanged comments with Giri: bq. What happened to CheckStyle? It looks like it got commented out some time ago, but why? Or was it entirely replaced by FindBugs? [~gkesavan]: Nope, we never had checkstyle in test-patch working. Right from the early days checkstyle function was always commented out, so this time around I removed it instead of leaving it as a comment. It would be very painful to implement checkstyle at this point in time, as making sure hadoop code base is complaint with checkstyle won't be that easy. was (Author: mattf): Exchanged comments with Giri: .bq What happened to CheckStyle? It looks like it got commented out some time ago, but why? Or was it entirely replaced by FindBugs? [~gkesavan]: Nope, we never had checkstyle in test-patch working. Right from the early days checkstyle function was always commented out, so this time around I removed it instead of leaving it as a comment. It would be very painful to implement checkstyle at this point in time, as making sure hadoop code base is complaint with checkstyle won't be that easy. Fix test-patch script to work with the enhanced PreCommit-Admin script. --- Key: HADOOP-9573 URL: https://issues.apache.org/jira/browse/HADOOP-9573 Project: Hadoop Common Issue Type: Sub-task Components: build Affects Versions: 1.0.3 Reporter: Giridharan Kesavan Assignee: Giridharan Kesavan Attachments: 0001-Fix-test-patch-scrit-to-work-with-the-enhanced-PreCo.patch test-patch script currently take the latest available patch for a given jira and performs the test. This jira is to enhance test-patch script to take attachment-id of a patch as an input and perform the tests using that attachment-id -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HADOOP-9573) Fix test-patch script to work with the enhanced PreCommit-Admin script.
[ https://issues.apache.org/jira/browse/HADOOP-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662390#comment-13662390 ] Matt Foley edited comment on HADOOP-9573 at 5/20/13 10:02 PM: -- Review posted in reviewboard. Lot of good work, a few things need cleanup. I assume you've tested this with test Jiras. Please describe tests done. Thanks Giri! was (Author: mattf): Review posted in reviewboard. Lot of good work, a few things need cleanup. I assume you've tested this with test Jiras. Please describe tests done. Thanks, Giri. Fix test-patch script to work with the enhanced PreCommit-Admin script. --- Key: HADOOP-9573 URL: https://issues.apache.org/jira/browse/HADOOP-9573 Project: Hadoop Common Issue Type: Sub-task Components: build Affects Versions: 1.0.3 Reporter: Giridharan Kesavan Assignee: Giridharan Kesavan Attachments: 0001-Fix-test-patch-scrit-to-work-with-the-enhanced-PreCo.patch test-patch script currently take the latest available patch for a given jira and performs the test. This jira is to enhance test-patch script to take attachment-id of a patch as an input and perform the tests using that attachment-id -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8515) Upgrade Jetty to the current Jetty 7 release
[ https://issues.apache.org/jira/browse/HADOOP-8515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-8515: --- Target Version/s: 3.0.0, 1.3.0 (was: 1.2.0, 3.0.0) Upgrade Jetty to the current Jetty 7 release Key: HADOOP-8515 URL: https://issues.apache.org/jira/browse/HADOOP-8515 Project: Hadoop Common Issue Type: Improvement Reporter: Luke Lu Labels: jetty Attachments: hadoop_jetty.patch.v2 According to http://dev.eclipse.org/mhonarc/lists/jetty-announce/msg00026.html, jetty-6 has been effectively EOL since January. Let's bump jetty to the 7 series. The current jetty 6.1.26 contains at least one known vulnerability: CVE-2011-4461. Note this can be an incompatible change if you reference jetty-6 packages (org.mortbay.*). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8515) Upgrade Jetty to the current Jetty 7 release
[ https://issues.apache.org/jira/browse/HADOOP-8515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656735#comment-13656735 ] Matt Foley commented on HADOOP-8515: Changed Target Version to 1.3.0 upon release of 1.2.0. Please change to 1.2.1 if you intend to submit a fix for branch-1.2. Upgrade Jetty to the current Jetty 7 release Key: HADOOP-8515 URL: https://issues.apache.org/jira/browse/HADOOP-8515 Project: Hadoop Common Issue Type: Improvement Reporter: Luke Lu Labels: jetty Attachments: hadoop_jetty.patch.v2 According to http://dev.eclipse.org/mhonarc/lists/jetty-announce/msg00026.html, jetty-6 has been effectively EOL since January. Let's bump jetty to the 7 series. The current jetty 6.1.26 contains at least one known vulnerability: CVE-2011-4461. Note this can be an incompatible change if you reference jetty-6 packages (org.mortbay.*). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8477) Pull in Yahoo! Hadoop Tutorial and update it accordingly.
[ https://issues.apache.org/jira/browse/HADOOP-8477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-8477: --- Target Version/s: 3.0.0, 1.3.0 (was: 1.2.0, 3.0.0) Pull in Yahoo! Hadoop Tutorial and update it accordingly. - Key: HADOOP-8477 URL: https://issues.apache.org/jira/browse/HADOOP-8477 Project: Hadoop Common Issue Type: Improvement Components: documentation Affects Versions: 1.1.0, 2.0.0-alpha Reporter: Robert Joseph Evans Attachments: tutorial.tgz I was able to get the Yahoo! Hadoop tutorial released under an Apache 2.0 license. This allows us to make it a official part of the Hadoop Project. This ticket is to pull the tutorial and update it as needed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8477) Pull in Yahoo! Hadoop Tutorial and update it accordingly.
[ https://issues.apache.org/jira/browse/HADOOP-8477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656736#comment-13656736 ] Matt Foley commented on HADOOP-8477: Changed Target Version to 1.3.0 upon release of 1.2.0. Please change to 1.2.1 if you intend to submit a fix for branch-1.2. Pull in Yahoo! Hadoop Tutorial and update it accordingly. - Key: HADOOP-8477 URL: https://issues.apache.org/jira/browse/HADOOP-8477 Project: Hadoop Common Issue Type: Improvement Components: documentation Affects Versions: 1.1.0, 2.0.0-alpha Reporter: Robert Joseph Evans Attachments: tutorial.tgz I was able to get the Yahoo! Hadoop tutorial released under an Apache 2.0 license. This allows us to make it a official part of the Hadoop Project. This ticket is to pull the tutorial and update it as needed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8161) Support for Azure Storage
[ https://issues.apache.org/jira/browse/HADOOP-8161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-8161: --- Target Version/s: 0.24.0, 1.3.0 (was: 1.2.0, 0.24.0) Support for Azure Storage - Key: HADOOP-8161 URL: https://issues.apache.org/jira/browse/HADOOP-8161 Project: Hadoop Common Issue Type: Sub-task Components: native Reporter: Sanjay Radia -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8161) Support for Azure Storage
[ https://issues.apache.org/jira/browse/HADOOP-8161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656737#comment-13656737 ] Matt Foley commented on HADOOP-8161: Changed Target Version to 1.3.0 upon release of 1.2.0. Please change to 1.2.1 if you intend to submit a fix for branch-1.2. Support for Azure Storage - Key: HADOOP-8161 URL: https://issues.apache.org/jira/browse/HADOOP-8161 Project: Hadoop Common Issue Type: Sub-task Components: native Reporter: Sanjay Radia -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8103) Hadoop-bin commands for windows
[ https://issues.apache.org/jira/browse/HADOOP-8103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-8103: --- Target Version/s: 0.24.0, 1.3.0 (was: 1.2.0, 0.24.0) Hadoop-bin commands for windows --- Key: HADOOP-8103 URL: https://issues.apache.org/jira/browse/HADOOP-8103 Project: Hadoop Common Issue Type: Sub-task Components: native Affects Versions: 1.1.0, 0.24.0 Reporter: Sanjay Radia Attachments: windows-cmd-scripts.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8103) Hadoop-bin commands for windows
[ https://issues.apache.org/jira/browse/HADOOP-8103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656738#comment-13656738 ] Matt Foley commented on HADOOP-8103: Changed Target Version to 1.3.0 upon release of 1.2.0. Please change to 1.2.1 if you intend to submit a fix for branch-1.2. Hadoop-bin commands for windows --- Key: HADOOP-8103 URL: https://issues.apache.org/jira/browse/HADOOP-8103 Project: Hadoop Common Issue Type: Sub-task Components: native Affects Versions: 1.1.0, 0.24.0 Reporter: Sanjay Radia Attachments: windows-cmd-scripts.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira