[jira] [Commented] (HADOOP-10641) Introduce Coordination Engine
[ https://issues.apache.org/jira/browse/HADOOP-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14021117#comment-14021117 ] Lohit Vijayarenu commented on HADOOP-10641: --- Minor comments. - It looks like checkQuorum is kind of noop for submitProposal in ZK based implementation, since zooKeeper.create would fail if there is no quorum anyways? - In ZK based Coordination Engine implementation, how are ZNodes cleaned up? Looking at patch each proposal creates PERSISTENT_SEQUENTIAL, but no mention of cleanup. Introduce Coordination Engine - Key: HADOOP-10641 URL: https://issues.apache.org/jira/browse/HADOOP-10641 Project: Hadoop Common Issue Type: New Feature Affects Versions: 3.0.0 Reporter: Konstantin Shvachko Assignee: Plamen Jeliazkov Attachments: HADOOP-10641.patch, HADOOP-10641.patch, HADOOP-10641.patch Coordination Engine (CE) is a system, which allows to agree on a sequence of events in a distributed system. In order to be reliable CE should be distributed by itself. Coordination Engine can be based on different algorithms (paxos, raft, 2PC, zab) and have different implementations, depending on use cases, reliability, availability, and performance requirements. CE should have a common API, so that it could serve as a pluggable component in different projects. The immediate beneficiaries are HDFS (HDFS-6469) and HBase (HBASE-10909). First implementation is proposed to be based on ZooKeeper. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9361) Strictly define the expected behavior of filesystem APIs and write tests to verify compliance
[ https://issues.apache.org/jira/browse/HADOOP-9361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-9361: --- Status: Open (was: Patch Available) Strictly define the expected behavior of filesystem APIs and write tests to verify compliance - Key: HADOOP-9361 URL: https://issues.apache.org/jira/browse/HADOOP-9361 Project: Hadoop Common Issue Type: Improvement Components: fs, test Affects Versions: 2.4.0, 3.0.0 Reporter: Steve Loughran Assignee: Steve Loughran Attachments: HADOOP-9361-001.patch, HADOOP-9361-002.patch, HADOOP-9361-003.patch, HADOOP-9361-004.patch, HADOOP-9361-005.patch, HADOOP-9361-006.patch, HADOOP-9361-007.patch, HADOOP-9361-008.patch, HADOOP-9361-009.patch, HADOOP-9361-011.patch, HADOOP-9361-012.patch, HADOOP-9361-013.patch, HADOOP-9361-014.patch, HADOOP-9361.awang-addendum.patch {{FileSystem}} and {{FileContract}} aren't tested rigorously enough -while HDFS gets tested downstream, other filesystems, such as blobstore bindings, don't. The only tests that are common are those of {{FileSystemContractTestBase}}, which HADOOP-9258 shows is incomplete. I propose # writing more tests which clarify expected behavior # testing operations in the interface being in their own JUnit4 test classes, instead of one big test suite. # Having each FS declare via a properties file what behaviors they offer, such as atomic-rename, atomic-delete, umask, immediate-consistency -test methods can downgrade to skipped test cases if a feature is missing. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9361) Strictly define the expected behavior of filesystem APIs and write tests to verify compliance
[ https://issues.apache.org/jira/browse/HADOOP-9361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-9361: --- Attachment: HADOOP-9361-015.patch This is revision -015 of the patch # incorporates all of Andrew's modifications. Andrew - thanks for putting in the effort! # added a section on object stores in the introduction, to clarify how they are different. Once we add a `Blobstore` marker to object stores, we can expand that a bit more. Strictly define the expected behavior of filesystem APIs and write tests to verify compliance - Key: HADOOP-9361 URL: https://issues.apache.org/jira/browse/HADOOP-9361 Project: Hadoop Common Issue Type: Improvement Components: fs, test Affects Versions: 3.0.0, 2.4.0 Reporter: Steve Loughran Assignee: Steve Loughran Attachments: HADOOP-9361-001.patch, HADOOP-9361-002.patch, HADOOP-9361-003.patch, HADOOP-9361-004.patch, HADOOP-9361-005.patch, HADOOP-9361-006.patch, HADOOP-9361-007.patch, HADOOP-9361-008.patch, HADOOP-9361-009.patch, HADOOP-9361-011.patch, HADOOP-9361-012.patch, HADOOP-9361-013.patch, HADOOP-9361-014.patch, HADOOP-9361-015.patch, HADOOP-9361.awang-addendum.patch {{FileSystem}} and {{FileContract}} aren't tested rigorously enough -while HDFS gets tested downstream, other filesystems, such as blobstore bindings, don't. The only tests that are common are those of {{FileSystemContractTestBase}}, which HADOOP-9258 shows is incomplete. I propose # writing more tests which clarify expected behavior # testing operations in the interface being in their own JUnit4 test classes, instead of one big test suite. # Having each FS declare via a properties file what behaviors they offer, such as atomic-rename, atomic-delete, umask, immediate-consistency -test methods can downgrade to skipped test cases if a feature is missing. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10648) Service Authorization Improvements
[ https://issues.apache.org/jira/browse/HADOOP-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HADOOP-10648: -- Description: Umbrella jira for a set of improvements on service Authorization (was: Umbrella jira for set of improvements on service Authorization) Service Authorization Improvements -- Key: HADOOP-10648 URL: https://issues.apache.org/jira/browse/HADOOP-10648 Project: Hadoop Common Issue Type: Improvement Components: security Reporter: Benoy Antony Assignee: Benoy Antony Umbrella jira for a set of improvements on service Authorization -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9361) Strictly define the expected behavior of filesystem APIs and write tests to verify compliance
[ https://issues.apache.org/jira/browse/HADOOP-9361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-9361: --- Status: Patch Available (was: Open) Strictly define the expected behavior of filesystem APIs and write tests to verify compliance - Key: HADOOP-9361 URL: https://issues.apache.org/jira/browse/HADOOP-9361 Project: Hadoop Common Issue Type: Improvement Components: fs, test Affects Versions: 2.4.0, 3.0.0 Reporter: Steve Loughran Assignee: Steve Loughran Attachments: HADOOP-9361-001.patch, HADOOP-9361-002.patch, HADOOP-9361-003.patch, HADOOP-9361-004.patch, HADOOP-9361-005.patch, HADOOP-9361-006.patch, HADOOP-9361-007.patch, HADOOP-9361-008.patch, HADOOP-9361-009.patch, HADOOP-9361-011.patch, HADOOP-9361-012.patch, HADOOP-9361-013.patch, HADOOP-9361-014.patch, HADOOP-9361-015.patch, HADOOP-9361.awang-addendum.patch {{FileSystem}} and {{FileContract}} aren't tested rigorously enough -while HDFS gets tested downstream, other filesystems, such as blobstore bindings, don't. The only tests that are common are those of {{FileSystemContractTestBase}}, which HADOOP-9258 shows is incomplete. I propose # writing more tests which clarify expected behavior # testing operations in the interface being in their own JUnit4 test classes, instead of one big test suite. # Having each FS declare via a properties file what behaviors they offer, such as atomic-rename, atomic-delete, umask, immediate-consistency -test methods can downgrade to skipped test cases if a feature is missing. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10561) Copy command with preserve option should handle Xattrs
[ https://issues.apache.org/jira/browse/HADOOP-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14021221#comment-14021221 ] Uma Maheswara Rao G commented on HADOOP-10561: -- Overall latest patch looks good to me. Tiny nit: input attibute -- input attribute +1 from me on addressing this comment. Copy command with preserve option should handle Xattrs -- Key: HADOOP-10561 URL: https://issues.apache.org/jira/browse/HADOOP-10561 Project: Hadoop Common Issue Type: Improvement Components: fs Affects Versions: 3.0.0 Reporter: Uma Maheswara Rao G Assignee: Yi Liu Attachments: HADOOP-10561.1.patch, HADOOP-10561.2.patch, HADOOP-10561.3.patch, HADOOP-10561.patch The design docs for Xattrs stated that we handle preserve options with copy commands From doc: Preserve option of commands like “cp -p” shell command and “distcp -p” should work on XAttrs. In the case of source fs supports XAttrs but target fs does not support, XAttrs will be ignored with warning message -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9361) Strictly define the expected behavior of filesystem APIs and write tests to verify compliance
[ https://issues.apache.org/jira/browse/HADOOP-9361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14021294#comment-14021294 ] Hadoop QA commented on HADOOP-9361: --- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12648875/HADOOP-9361-015.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 77 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-openstack. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4024//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4024//console This message is automatically generated. Strictly define the expected behavior of filesystem APIs and write tests to verify compliance - Key: HADOOP-9361 URL: https://issues.apache.org/jira/browse/HADOOP-9361 Project: Hadoop Common Issue Type: Improvement Components: fs, test Affects Versions: 3.0.0, 2.4.0 Reporter: Steve Loughran Assignee: Steve Loughran Attachments: HADOOP-9361-001.patch, HADOOP-9361-002.patch, HADOOP-9361-003.patch, HADOOP-9361-004.patch, HADOOP-9361-005.patch, HADOOP-9361-006.patch, HADOOP-9361-007.patch, HADOOP-9361-008.patch, HADOOP-9361-009.patch, HADOOP-9361-011.patch, HADOOP-9361-012.patch, HADOOP-9361-013.patch, HADOOP-9361-014.patch, HADOOP-9361-015.patch, HADOOP-9361.awang-addendum.patch {{FileSystem}} and {{FileContract}} aren't tested rigorously enough -while HDFS gets tested downstream, other filesystems, such as blobstore bindings, don't. The only tests that are common are those of {{FileSystemContractTestBase}}, which HADOOP-9258 shows is incomplete. I propose # writing more tests which clarify expected behavior # testing operations in the interface being in their own JUnit4 test classes, instead of one big test suite. # Having each FS declare via a properties file what behaviors they offer, such as atomic-rename, atomic-delete, umask, immediate-consistency -test methods can downgrade to skipped test cases if a feature is missing. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10641) Introduce Coordination Engine
[ https://issues.apache.org/jira/browse/HADOOP-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14021473#comment-14021473 ] Plamen Jeliazkov commented on HADOOP-10641: --- Hi Lohit, thanks for your comments! # checkQuorum is an optimization some coordination engines may choose to implement in order to fail-fast to client requests. In the NameNode case, if quorum loss was suspected, that NameNode could start issuing StandbyExceptions. # You are correct that the ZKCoordinationEngine does not implement ZNode clean-up currently. That is because it was made as a proof of concept for the CoordinationEngine API. Nonetheless, proper clean-up can be implemented. All one has to do is delete the ZNodes that everyone else has already learned about. ## Suppose you have Node A, B, and C, and Agreements 1, 2, 3, 4, and 5. ## Node A and B learn Agreement 1 first. Node C is a lagging node. A B contain 1. C contains nothing. ## Node A and B continue onwards, learning up to Agreement 4. A B contain 1, 2, 3, and 4 now. C contains nothing. ## Node C finally learns Agreement 1. A B contain 1, 2, 3, and 4 now. C contains 1. ## We can now discard Agreement 1 from persistence because we know that all the Nodes, A, B, and C, have safely learned about and applied Agreement 1. ## We can apply this process for all other Agreements. Introduce Coordination Engine - Key: HADOOP-10641 URL: https://issues.apache.org/jira/browse/HADOOP-10641 Project: Hadoop Common Issue Type: New Feature Affects Versions: 3.0.0 Reporter: Konstantin Shvachko Assignee: Plamen Jeliazkov Attachments: HADOOP-10641.patch, HADOOP-10641.patch, HADOOP-10641.patch Coordination Engine (CE) is a system, which allows to agree on a sequence of events in a distributed system. In order to be reliable CE should be distributed by itself. Coordination Engine can be based on different algorithms (paxos, raft, 2PC, zab) and have different implementations, depending on use cases, reliability, availability, and performance requirements. CE should have a common API, so that it could serve as a pluggable component in different projects. The immediate beneficiaries are HDFS (HDFS-6469) and HBase (HBASE-10909). First implementation is proposed to be based on ZooKeeper. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10669) Avro serialization does not flush buffered serialized values causing data lost
[ https://issues.apache.org/jira/browse/HADOOP-10669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Bernadsky updated HADOOP-10669: --- Attachment: HADOOP-10669_alt.patch Avro serialization does not flush buffered serialized values causing data lost -- Key: HADOOP-10669 URL: https://issues.apache.org/jira/browse/HADOOP-10669 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 2.4.0 Reporter: Mikhail Bernadsky Attachments: HADOOP-10669.patch, HADOOP-10669_alt.patch Found this debugging Nutch. MapTask serializes keys and values to the same stream, in pairs: keySerializer.serialize(key); . valSerializer.serialize(value); . bb.write(b0, 0, 0); AvroSerializer does not flush its buffer after each serialization. So if it is used for valSerializer, the values are only partially written or not written at all to the output stream before the record is marked as complete (the last line above). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10669) Avro serialization does not flush buffered serialized values causing data lost
[ https://issues.apache.org/jira/browse/HADOOP-10669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Bernadsky updated HADOOP-10669: --- Description: Found this debugging Nutch. MapTask serializes keys and values to the same stream, in pairs: keySerializer.serialize(key); . valSerializer.serialize(value); . bb.write(b0, 0, 0); AvroSerializer does not flush its buffer after each serialization. So if it is used for valSerializer, the values are only partially written or not written at all to the output stream before the record is marked as complete (the last line above). EDIT Added HADOOP-10699_all.patch. This is a less intrusive fix, as it does not try to flush MapTask stream. Instead, we write serialized values directly to MapTask stream and avoid using a buffer on avro side. was: Found this debugging Nutch. MapTask serializes keys and values to the same stream, in pairs: keySerializer.serialize(key); . valSerializer.serialize(value); . bb.write(b0, 0, 0); AvroSerializer does not flush its buffer after each serialization. So if it is used for valSerializer, the values are only partially written or not written at all to the output stream before the record is marked as complete (the last line above). Avro serialization does not flush buffered serialized values causing data lost -- Key: HADOOP-10669 URL: https://issues.apache.org/jira/browse/HADOOP-10669 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 2.4.0 Reporter: Mikhail Bernadsky Attachments: HADOOP-10669.patch, HADOOP-10669_alt.patch Found this debugging Nutch. MapTask serializes keys and values to the same stream, in pairs: keySerializer.serialize(key); . valSerializer.serialize(value); . bb.write(b0, 0, 0); AvroSerializer does not flush its buffer after each serialization. So if it is used for valSerializer, the values are only partially written or not written at all to the output stream before the record is marked as complete (the last line above). EDIT Added HADOOP-10699_all.patch. This is a less intrusive fix, as it does not try to flush MapTask stream. Instead, we write serialized values directly to MapTask stream and avoid using a buffer on avro side. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9629) Support Windows Azure Storage - Blob as a file system in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Liddell updated HADOOP-9629: - Attachment: HADOOP-9629.trunk.3.patch Support Windows Azure Storage - Blob as a file system in Hadoop --- Key: HADOOP-9629 URL: https://issues.apache.org/jira/browse/HADOOP-9629 Project: Hadoop Common Issue Type: Improvement Reporter: Mostafa Elhemali Assignee: Mike Liddell Attachments: HADOOP-9629 - Azure Filesystem - Information for developers.docx, HADOOP-9629 - Azure Filesystem - Information for developers.pdf, HADOOP-9629.2.patch, HADOOP-9629.3.patch, HADOOP-9629.patch, HADOOP-9629.trunk.1.patch, HADOOP-9629.trunk.2.patch, HADOOP-9629.trunk.3.patch h2. Description This JIRA incorporates adding a new file system implementation for accessing Windows Azure Storage - Blob from within Hadoop, such as using blobs as input to MR jobs or configuring MR jobs to put their output directly into blob storage. h2. High level design At a high level, the code here extends the FileSystem class to provide an implementation for accessing blob storage; the scheme wasb is used for accessing it over HTTP, and wasbs for accessing over HTTPS. We use the URI scheme: {code}wasb[s]://container@account/path/to/file{code} to address individual blobs. We use the standard Azure Java SDK (com.microsoft.windowsazure) to do most of the work. In order to map a hierarchical file system over the flat name-value pair nature of blob storage, we create a specially tagged blob named path/to/dir whenever we create a directory called path/to/dir, then files under that are stored as normal blobs path/to/dir/file. We have many metrics implemented for it using the Metrics2 interface. Tests are implemented mostly using a mock implementation for the Azure SDK functionality, with an option to test against a real blob storage if configured (instructions provided inside in README.txt). h2. Credits and history This has been ongoing work for a while, and the early version of this work can be seen in HADOOP-8079. This JIRA is a significant revision of that and we'll post the patch here for Hadoop trunk first, then post a patch for branch-1 as well for backporting the functionality if accepted. Credit for this work goes to the early team: [~minwei], [~davidlao], [~lengningliu] and [~stojanovic] as well as multiple people who have taken over this work since then (hope I don't forget anyone): [~dexterb], Johannes Klein, [~ivanmi], Michael Rys, [~mostafae], [~brian_swan], [~mikelid], [~xifang], and [~chuanliu]. h2. Test Besides unit tests, we have used WASB as the default file system in our service product. (HDFS is also used but not as default file system.) Various different customer and test workloads have been run against clusters with such configurations for quite some time. The current version reflects to the version of the code tested and used in our production environment. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9629) Support Windows Azure Storage - Blob as a file system in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14021497#comment-14021497 ] Mike Liddell commented on HADOOP-9629: -- The annotations and suggested usages sound good. The only changes that I suggest are: - AzureException: Public + Evolving - WasbFsck: Public + Evolving. sound good? Support Windows Azure Storage - Blob as a file system in Hadoop --- Key: HADOOP-9629 URL: https://issues.apache.org/jira/browse/HADOOP-9629 Project: Hadoop Common Issue Type: Improvement Reporter: Mostafa Elhemali Assignee: Mike Liddell Attachments: HADOOP-9629 - Azure Filesystem - Information for developers.docx, HADOOP-9629 - Azure Filesystem - Information for developers.pdf, HADOOP-9629.2.patch, HADOOP-9629.3.patch, HADOOP-9629.patch, HADOOP-9629.trunk.1.patch, HADOOP-9629.trunk.2.patch, HADOOP-9629.trunk.3.patch h2. Description This JIRA incorporates adding a new file system implementation for accessing Windows Azure Storage - Blob from within Hadoop, such as using blobs as input to MR jobs or configuring MR jobs to put their output directly into blob storage. h2. High level design At a high level, the code here extends the FileSystem class to provide an implementation for accessing blob storage; the scheme wasb is used for accessing it over HTTP, and wasbs for accessing over HTTPS. We use the URI scheme: {code}wasb[s]://container@account/path/to/file{code} to address individual blobs. We use the standard Azure Java SDK (com.microsoft.windowsazure) to do most of the work. In order to map a hierarchical file system over the flat name-value pair nature of blob storage, we create a specially tagged blob named path/to/dir whenever we create a directory called path/to/dir, then files under that are stored as normal blobs path/to/dir/file. We have many metrics implemented for it using the Metrics2 interface. Tests are implemented mostly using a mock implementation for the Azure SDK functionality, with an option to test against a real blob storage if configured (instructions provided inside in README.txt). h2. Credits and history This has been ongoing work for a while, and the early version of this work can be seen in HADOOP-8079. This JIRA is a significant revision of that and we'll post the patch here for Hadoop trunk first, then post a patch for branch-1 as well for backporting the functionality if accepted. Credit for this work goes to the early team: [~minwei], [~davidlao], [~lengningliu] and [~stojanovic] as well as multiple people who have taken over this work since then (hope I don't forget anyone): [~dexterb], Johannes Klein, [~ivanmi], Michael Rys, [~mostafae], [~brian_swan], [~mikelid], [~xifang], and [~chuanliu]. h2. Test Besides unit tests, we have used WASB as the default file system in our service product. (HDFS is also used but not as default file system.) Various different customer and test workloads have been run against clusters with such configurations for quite some time. The current version reflects to the version of the code tested and used in our production environment. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10561) Copy command with preserve option should handle Xattrs
[ https://issues.apache.org/jira/browse/HADOOP-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu updated HADOOP-10561: Attachment: HADOOP-10561.4.patch Thanks Uma for review. The new patch includes update for your comments. Copy command with preserve option should handle Xattrs -- Key: HADOOP-10561 URL: https://issues.apache.org/jira/browse/HADOOP-10561 Project: Hadoop Common Issue Type: Improvement Components: fs Affects Versions: 3.0.0 Reporter: Uma Maheswara Rao G Assignee: Yi Liu Attachments: HADOOP-10561.1.patch, HADOOP-10561.2.patch, HADOOP-10561.3.patch, HADOOP-10561.4.patch, HADOOP-10561.patch The design docs for Xattrs stated that we handle preserve options with copy commands From doc: Preserve option of commands like “cp -p” shell command and “distcp -p” should work on XAttrs. In the case of source fs supports XAttrs but target fs does not support, XAttrs will be ignored with warning message -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9629) Support Windows Azure Storage - Blob as a file system in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Liddell updated HADOOP-9629: - Attachment: (was: HADOOP-9629.trunk.3.patch) Support Windows Azure Storage - Blob as a file system in Hadoop --- Key: HADOOP-9629 URL: https://issues.apache.org/jira/browse/HADOOP-9629 Project: Hadoop Common Issue Type: Improvement Reporter: Mostafa Elhemali Assignee: Mike Liddell Attachments: HADOOP-9629 - Azure Filesystem - Information for developers.docx, HADOOP-9629 - Azure Filesystem - Information for developers.pdf, HADOOP-9629.2.patch, HADOOP-9629.3.patch, HADOOP-9629.patch, HADOOP-9629.trunk.1.patch, HADOOP-9629.trunk.2.patch h2. Description This JIRA incorporates adding a new file system implementation for accessing Windows Azure Storage - Blob from within Hadoop, such as using blobs as input to MR jobs or configuring MR jobs to put their output directly into blob storage. h2. High level design At a high level, the code here extends the FileSystem class to provide an implementation for accessing blob storage; the scheme wasb is used for accessing it over HTTP, and wasbs for accessing over HTTPS. We use the URI scheme: {code}wasb[s]://container@account/path/to/file{code} to address individual blobs. We use the standard Azure Java SDK (com.microsoft.windowsazure) to do most of the work. In order to map a hierarchical file system over the flat name-value pair nature of blob storage, we create a specially tagged blob named path/to/dir whenever we create a directory called path/to/dir, then files under that are stored as normal blobs path/to/dir/file. We have many metrics implemented for it using the Metrics2 interface. Tests are implemented mostly using a mock implementation for the Azure SDK functionality, with an option to test against a real blob storage if configured (instructions provided inside in README.txt). h2. Credits and history This has been ongoing work for a while, and the early version of this work can be seen in HADOOP-8079. This JIRA is a significant revision of that and we'll post the patch here for Hadoop trunk first, then post a patch for branch-1 as well for backporting the functionality if accepted. Credit for this work goes to the early team: [~minwei], [~davidlao], [~lengningliu] and [~stojanovic] as well as multiple people who have taken over this work since then (hope I don't forget anyone): [~dexterb], Johannes Klein, [~ivanmi], Michael Rys, [~mostafae], [~brian_swan], [~mikelid], [~xifang], and [~chuanliu]. h2. Test Besides unit tests, we have used WASB as the default file system in our service product. (HDFS is also used but not as default file system.) Various different customer and test workloads have been run against clusters with such configurations for quite some time. The current version reflects to the version of the code tested and used in our production environment. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9629) Support Windows Azure Storage - Blob as a file system in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Liddell updated HADOOP-9629: - Attachment: HADOOP-9629.trunk.3.patch Support Windows Azure Storage - Blob as a file system in Hadoop --- Key: HADOOP-9629 URL: https://issues.apache.org/jira/browse/HADOOP-9629 Project: Hadoop Common Issue Type: Improvement Reporter: Mostafa Elhemali Assignee: Mike Liddell Attachments: HADOOP-9629 - Azure Filesystem - Information for developers.docx, HADOOP-9629 - Azure Filesystem - Information for developers.pdf, HADOOP-9629.2.patch, HADOOP-9629.3.patch, HADOOP-9629.patch, HADOOP-9629.trunk.1.patch, HADOOP-9629.trunk.2.patch, HADOOP-9629.trunk.3.patch h2. Description This JIRA incorporates adding a new file system implementation for accessing Windows Azure Storage - Blob from within Hadoop, such as using blobs as input to MR jobs or configuring MR jobs to put their output directly into blob storage. h2. High level design At a high level, the code here extends the FileSystem class to provide an implementation for accessing blob storage; the scheme wasb is used for accessing it over HTTP, and wasbs for accessing over HTTPS. We use the URI scheme: {code}wasb[s]://container@account/path/to/file{code} to address individual blobs. We use the standard Azure Java SDK (com.microsoft.windowsazure) to do most of the work. In order to map a hierarchical file system over the flat name-value pair nature of blob storage, we create a specially tagged blob named path/to/dir whenever we create a directory called path/to/dir, then files under that are stored as normal blobs path/to/dir/file. We have many metrics implemented for it using the Metrics2 interface. Tests are implemented mostly using a mock implementation for the Azure SDK functionality, with an option to test against a real blob storage if configured (instructions provided inside in README.txt). h2. Credits and history This has been ongoing work for a while, and the early version of this work can be seen in HADOOP-8079. This JIRA is a significant revision of that and we'll post the patch here for Hadoop trunk first, then post a patch for branch-1 as well for backporting the functionality if accepted. Credit for this work goes to the early team: [~minwei], [~davidlao], [~lengningliu] and [~stojanovic] as well as multiple people who have taken over this work since then (hope I don't forget anyone): [~dexterb], Johannes Klein, [~ivanmi], Michael Rys, [~mostafae], [~brian_swan], [~mikelid], [~xifang], and [~chuanliu]. h2. Test Besides unit tests, we have used WASB as the default file system in our service product. (HDFS is also used but not as default file system.) Various different customer and test workloads have been run against clusters with such configurations for quite some time. The current version reflects to the version of the code tested and used in our production environment. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9629) Support Windows Azure Storage - Blob as a file system in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14021499#comment-14021499 ] Mike Liddell commented on HADOOP-9629: -- new patch: HADOOP-9629.trunk.4.patch - addresses code-review comments from [~cnauroth], see https://reviews.apache.org/r/22096/ - adds InterfaceAudience and InterfaceStability annotations to the main classes. Support Windows Azure Storage - Blob as a file system in Hadoop --- Key: HADOOP-9629 URL: https://issues.apache.org/jira/browse/HADOOP-9629 Project: Hadoop Common Issue Type: Improvement Reporter: Mostafa Elhemali Assignee: Mike Liddell Attachments: HADOOP-9629 - Azure Filesystem - Information for developers.docx, HADOOP-9629 - Azure Filesystem - Information for developers.pdf, HADOOP-9629.2.patch, HADOOP-9629.3.patch, HADOOP-9629.patch, HADOOP-9629.trunk.1.patch, HADOOP-9629.trunk.2.patch, HADOOP-9629.trunk.3.patch h2. Description This JIRA incorporates adding a new file system implementation for accessing Windows Azure Storage - Blob from within Hadoop, such as using blobs as input to MR jobs or configuring MR jobs to put their output directly into blob storage. h2. High level design At a high level, the code here extends the FileSystem class to provide an implementation for accessing blob storage; the scheme wasb is used for accessing it over HTTP, and wasbs for accessing over HTTPS. We use the URI scheme: {code}wasb[s]://container@account/path/to/file{code} to address individual blobs. We use the standard Azure Java SDK (com.microsoft.windowsazure) to do most of the work. In order to map a hierarchical file system over the flat name-value pair nature of blob storage, we create a specially tagged blob named path/to/dir whenever we create a directory called path/to/dir, then files under that are stored as normal blobs path/to/dir/file. We have many metrics implemented for it using the Metrics2 interface. Tests are implemented mostly using a mock implementation for the Azure SDK functionality, with an option to test against a real blob storage if configured (instructions provided inside in README.txt). h2. Credits and history This has been ongoing work for a while, and the early version of this work can be seen in HADOOP-8079. This JIRA is a significant revision of that and we'll post the patch here for Hadoop trunk first, then post a patch for branch-1 as well for backporting the functionality if accepted. Credit for this work goes to the early team: [~minwei], [~davidlao], [~lengningliu] and [~stojanovic] as well as multiple people who have taken over this work since then (hope I don't forget anyone): [~dexterb], Johannes Klein, [~ivanmi], Michael Rys, [~mostafae], [~brian_swan], [~mikelid], [~xifang], and [~chuanliu]. h2. Test Besides unit tests, we have used WASB as the default file system in our service product. (HDFS is also used but not as default file system.) Various different customer and test workloads have been run against clusters with such configurations for quite some time. The current version reflects to the version of the code tested and used in our production environment. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9629) Support Windows Azure Storage - Blob as a file system in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14021502#comment-14021502 ] Hadoop QA commented on HADOOP-9629: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12648900/HADOOP-9629.trunk.3.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 25 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:red}-1 javadoc{color}. The javadoc tool appears to have generated 100 warning messages. See https://builds.apache.org/job/PreCommit-HADOOP-Build/4025//artifact/trunk/patchprocess/diffJavadocWarnings.txt for details. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-tools/hadoop-azure hadoop-tools/hadoop-tools-dist. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4025//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4025//console This message is automatically generated. Support Windows Azure Storage - Blob as a file system in Hadoop --- Key: HADOOP-9629 URL: https://issues.apache.org/jira/browse/HADOOP-9629 Project: Hadoop Common Issue Type: Improvement Reporter: Mostafa Elhemali Assignee: Mike Liddell Attachments: HADOOP-9629 - Azure Filesystem - Information for developers.docx, HADOOP-9629 - Azure Filesystem - Information for developers.pdf, HADOOP-9629.2.patch, HADOOP-9629.3.patch, HADOOP-9629.patch, HADOOP-9629.trunk.1.patch, HADOOP-9629.trunk.2.patch, HADOOP-9629.trunk.3.patch h2. Description This JIRA incorporates adding a new file system implementation for accessing Windows Azure Storage - Blob from within Hadoop, such as using blobs as input to MR jobs or configuring MR jobs to put their output directly into blob storage. h2. High level design At a high level, the code here extends the FileSystem class to provide an implementation for accessing blob storage; the scheme wasb is used for accessing it over HTTP, and wasbs for accessing over HTTPS. We use the URI scheme: {code}wasb[s]://container@account/path/to/file{code} to address individual blobs. We use the standard Azure Java SDK (com.microsoft.windowsazure) to do most of the work. In order to map a hierarchical file system over the flat name-value pair nature of blob storage, we create a specially tagged blob named path/to/dir whenever we create a directory called path/to/dir, then files under that are stored as normal blobs path/to/dir/file. We have many metrics implemented for it using the Metrics2 interface. Tests are implemented mostly using a mock implementation for the Azure SDK functionality, with an option to test against a real blob storage if configured (instructions provided inside in README.txt). h2. Credits and history This has been ongoing work for a while, and the early version of this work can be seen in HADOOP-8079. This JIRA is a significant revision of that and we'll post the patch here for Hadoop trunk first, then post a patch for branch-1 as well for backporting the functionality if accepted. Credit for this work goes to the early team: [~minwei], [~davidlao], [~lengningliu] and [~stojanovic] as well as multiple people who have taken over this work since then (hope I don't forget anyone): [~dexterb], Johannes Klein, [~ivanmi], Michael Rys, [~mostafae], [~brian_swan], [~mikelid], [~xifang], and [~chuanliu]. h2. Test Besides unit tests, we have used WASB as the default file system in our service product. (HDFS is also used but not as default file system.) Various different customer and test workloads have been run against clusters with such configurations for quite some time. The current version reflects to the version of the code tested and used in our production environment. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9629) Support Windows Azure Storage - Blob as a file system in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14021506#comment-14021506 ] Mike Liddell commented on HADOOP-9629: -- Previous comment about new patch file had name wrong: The new patch is HADOOP-9629.trunk.3.patch Support Windows Azure Storage - Blob as a file system in Hadoop --- Key: HADOOP-9629 URL: https://issues.apache.org/jira/browse/HADOOP-9629 Project: Hadoop Common Issue Type: Improvement Reporter: Mostafa Elhemali Assignee: Mike Liddell Attachments: HADOOP-9629 - Azure Filesystem - Information for developers.docx, HADOOP-9629 - Azure Filesystem - Information for developers.pdf, HADOOP-9629.2.patch, HADOOP-9629.3.patch, HADOOP-9629.patch, HADOOP-9629.trunk.1.patch, HADOOP-9629.trunk.2.patch, HADOOP-9629.trunk.3.patch h2. Description This JIRA incorporates adding a new file system implementation for accessing Windows Azure Storage - Blob from within Hadoop, such as using blobs as input to MR jobs or configuring MR jobs to put their output directly into blob storage. h2. High level design At a high level, the code here extends the FileSystem class to provide an implementation for accessing blob storage; the scheme wasb is used for accessing it over HTTP, and wasbs for accessing over HTTPS. We use the URI scheme: {code}wasb[s]://container@account/path/to/file{code} to address individual blobs. We use the standard Azure Java SDK (com.microsoft.windowsazure) to do most of the work. In order to map a hierarchical file system over the flat name-value pair nature of blob storage, we create a specially tagged blob named path/to/dir whenever we create a directory called path/to/dir, then files under that are stored as normal blobs path/to/dir/file. We have many metrics implemented for it using the Metrics2 interface. Tests are implemented mostly using a mock implementation for the Azure SDK functionality, with an option to test against a real blob storage if configured (instructions provided inside in README.txt). h2. Credits and history This has been ongoing work for a while, and the early version of this work can be seen in HADOOP-8079. This JIRA is a significant revision of that and we'll post the patch here for Hadoop trunk first, then post a patch for branch-1 as well for backporting the functionality if accepted. Credit for this work goes to the early team: [~minwei], [~davidlao], [~lengningliu] and [~stojanovic] as well as multiple people who have taken over this work since then (hope I don't forget anyone): [~dexterb], Johannes Klein, [~ivanmi], Michael Rys, [~mostafae], [~brian_swan], [~mikelid], [~xifang], and [~chuanliu]. h2. Test Besides unit tests, we have used WASB as the default file system in our service product. (HDFS is also used but not as default file system.) Various different customer and test workloads have been run against clusters with such configurations for quite some time. The current version reflects to the version of the code tested and used in our production environment. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9629) Support Windows Azure Storage - Blob as a file system in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14021511#comment-14021511 ] Hadoop QA commented on HADOOP-9629: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12648902/HADOOP-9629.trunk.3.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 25 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:red}-1 javadoc{color}. The javadoc tool appears to have generated 100 warning messages. See https://builds.apache.org/job/PreCommit-HADOOP-Build/4027//artifact/trunk/patchprocess/diffJavadocWarnings.txt for details. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-tools/hadoop-azure hadoop-tools/hadoop-tools-dist. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4027//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4027//console This message is automatically generated. Support Windows Azure Storage - Blob as a file system in Hadoop --- Key: HADOOP-9629 URL: https://issues.apache.org/jira/browse/HADOOP-9629 Project: Hadoop Common Issue Type: Improvement Reporter: Mostafa Elhemali Assignee: Mike Liddell Attachments: HADOOP-9629 - Azure Filesystem - Information for developers.docx, HADOOP-9629 - Azure Filesystem - Information for developers.pdf, HADOOP-9629.2.patch, HADOOP-9629.3.patch, HADOOP-9629.patch, HADOOP-9629.trunk.1.patch, HADOOP-9629.trunk.2.patch, HADOOP-9629.trunk.3.patch h2. Description This JIRA incorporates adding a new file system implementation for accessing Windows Azure Storage - Blob from within Hadoop, such as using blobs as input to MR jobs or configuring MR jobs to put their output directly into blob storage. h2. High level design At a high level, the code here extends the FileSystem class to provide an implementation for accessing blob storage; the scheme wasb is used for accessing it over HTTP, and wasbs for accessing over HTTPS. We use the URI scheme: {code}wasb[s]://container@account/path/to/file{code} to address individual blobs. We use the standard Azure Java SDK (com.microsoft.windowsazure) to do most of the work. In order to map a hierarchical file system over the flat name-value pair nature of blob storage, we create a specially tagged blob named path/to/dir whenever we create a directory called path/to/dir, then files under that are stored as normal blobs path/to/dir/file. We have many metrics implemented for it using the Metrics2 interface. Tests are implemented mostly using a mock implementation for the Azure SDK functionality, with an option to test against a real blob storage if configured (instructions provided inside in README.txt). h2. Credits and history This has been ongoing work for a while, and the early version of this work can be seen in HADOOP-8079. This JIRA is a significant revision of that and we'll post the patch here for Hadoop trunk first, then post a patch for branch-1 as well for backporting the functionality if accepted. Credit for this work goes to the early team: [~minwei], [~davidlao], [~lengningliu] and [~stojanovic] as well as multiple people who have taken over this work since then (hope I don't forget anyone): [~dexterb], Johannes Klein, [~ivanmi], Michael Rys, [~mostafae], [~brian_swan], [~mikelid], [~xifang], and [~chuanliu]. h2. Test Besides unit tests, we have used WASB as the default file system in our service product. (HDFS is also used but not as default file system.) Various different customer and test workloads have been run against clusters with such configurations for quite some time. The current version reflects to the version of the code tested and used in our production environment. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10640) Implement Namenode RPCs in HDFS native client
[ https://issues.apache.org/jira/browse/HADOOP-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14021541#comment-14021541 ] Wenwu Peng commented on HADOOP-10640: - run command NAMENODE_URI=hdfs://localhost:8020 ./test_libhdfs_meta_ops and hit this error hdfsBuilderConnect: ndfs failed to connect: org.apache.hadoop.native.HadoopCore.OutOfMemoryException: cnn_get_server_defaults: failed to allocate sync_ctx (error 12) org.apache.hadoop.native.HadoopCore.OutOfMemoryException: cnn_get_server_defaults: failed to allocate sync_ctxerror: did not expect '0': '/root/hadoop-common/hadoop-native-core/src/main/native/test/fs/test_libhdfs_meta_ops.c at line 60' Implement Namenode RPCs in HDFS native client - Key: HADOOP-10640 URL: https://issues.apache.org/jira/browse/HADOOP-10640 Project: Hadoop Common Issue Type: Sub-task Components: native Affects Versions: HADOOP-10388 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HADOOP-10640-pnative.001.patch, HADOOP-10640-pnative.002.patch, HADOOP-10640-pnative.003.patch Implement the parts of libhdfs that just involve making RPCs to the Namenode, such as mkdir, rename, etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10561) Copy command with preserve option should handle Xattrs
[ https://issues.apache.org/jira/browse/HADOOP-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14021558#comment-14021558 ] Hadoop QA commented on HADOOP-10561: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12648901/HADOOP-10561.4.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.datanode.TestBPOfferService {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4026//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4026//console This message is automatically generated. Copy command with preserve option should handle Xattrs -- Key: HADOOP-10561 URL: https://issues.apache.org/jira/browse/HADOOP-10561 Project: Hadoop Common Issue Type: Improvement Components: fs Affects Versions: 3.0.0 Reporter: Uma Maheswara Rao G Assignee: Yi Liu Attachments: HADOOP-10561.1.patch, HADOOP-10561.2.patch, HADOOP-10561.3.patch, HADOOP-10561.4.patch, HADOOP-10561.patch The design docs for Xattrs stated that we handle preserve options with copy commands From doc: Preserve option of commands like “cp -p” shell command and “distcp -p” should work on XAttrs. In the case of source fs supports XAttrs but target fs does not support, XAttrs will be ignored with warning message -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10670) Allow AuthenticationFilter to respect signature secret file even without AuthenticationFilterInitializer
Kai Zheng created HADOOP-10670: -- Summary: Allow AuthenticationFilter to respect signature secret file even without AuthenticationFilterInitializer Key: HADOOP-10670 URL: https://issues.apache.org/jira/browse/HADOOP-10670 Project: Hadoop Common Issue Type: Improvement Reporter: Kai Zheng Assignee: Kai Zheng Priority: Minor In Hadoop web console, by using AuthenticationFilterInitializer, it's allowed to configure AuthenticationFilter for the required signature secret by specifying signature.secret.file property. This improvement would also allow this when AuthenticationFilterInitializer isn't used in situations like webhdfs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10670) Allow AuthenticationFilter to respect signature secret file even without AuthenticationFilterInitializer
[ https://issues.apache.org/jira/browse/HADOOP-10670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-10670: --- Attachment: hadoop-10670.patch Allow AuthenticationFilter to respect signature secret file even without AuthenticationFilterInitializer Key: HADOOP-10670 URL: https://issues.apache.org/jira/browse/HADOOP-10670 Project: Hadoop Common Issue Type: Improvement Reporter: Kai Zheng Assignee: Kai Zheng Priority: Minor Attachments: hadoop-10670.patch In Hadoop web console, by using AuthenticationFilterInitializer, it's allowed to configure AuthenticationFilter for the required signature secret by specifying signature.secret.file property. This improvement would also allow this when AuthenticationFilterInitializer isn't used in situations like webhdfs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10670) Allow AuthenticationFilter to respect signature secret file even without AuthenticationFilterInitializer
[ https://issues.apache.org/jira/browse/HADOOP-10670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-10670: --- Status: Patch Available (was: Open) Attached a patch. Changes summary: 1. Moving signature file reading from AuthenticationFilterInitializer to AuthenticationFilter: 2. And in AuthenticationFilter, if SIGNATURE_SECRET is configured, then use it; otherwise if SIGNATURE_SECRET_FILE is configured, then use it; otherwise generate a secret as before. Allow AuthenticationFilter to respect signature secret file even without AuthenticationFilterInitializer Key: HADOOP-10670 URL: https://issues.apache.org/jira/browse/HADOOP-10670 Project: Hadoop Common Issue Type: Improvement Reporter: Kai Zheng Assignee: Kai Zheng Priority: Minor Attachments: hadoop-10670.patch In Hadoop web console, by using AuthenticationFilterInitializer, it's allowed to configure AuthenticationFilter for the required signature secret by specifying signature.secret.file property. This improvement would also allow this when AuthenticationFilterInitializer isn't used in situations like webhdfs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10640) Implement Namenode RPCs in HDFS native client
[ https://issues.apache.org/jira/browse/HADOOP-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14021649#comment-14021649 ] Binglin Chang commented on HADOOP-10640: sizeof(struct hrpc_proxy) always larger than RPC_PROXY_USERDATA_MAX? {code} void *hrpc_proxy_alloc_userdata(struct hrpc_proxy *proxy, size_t size) { if (size RPC_PROXY_USERDATA_MAX) { return NULL; } return proxy-userdata; } struct hrpc_sync_ctx *hrpc_proxy_alloc_sync_ctx(struct hrpc_proxy *proxy) { struct hrpc_sync_ctx *ctx = hrpc_proxy_alloc_userdata(proxy, sizeof(struct hrpc_proxy)); if (!ctx) { return NULL; } if (uv_sem_init(ctx-sem, 0)) { return NULL; } memset(ctx, 0, sizeof(ctx)); return ctx; } {code} Implement Namenode RPCs in HDFS native client - Key: HADOOP-10640 URL: https://issues.apache.org/jira/browse/HADOOP-10640 Project: Hadoop Common Issue Type: Sub-task Components: native Affects Versions: HADOOP-10388 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HADOOP-10640-pnative.001.patch, HADOOP-10640-pnative.002.patch, HADOOP-10640-pnative.003.patch Implement the parts of libhdfs that just involve making RPCs to the Namenode, such as mkdir, rename, etc. -- This message was sent by Atlassian JIRA (v6.2#6252)