[jira] [Updated] (HDFS-9195) TestDelegationTokenForProxyUser.testWebHdfsDoAs fails on trunk
[
https://issues.apache.org/jira/browse/HDFS-9195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramtin updated HDFS-9195:
-
Assignee: ramtin
Status: Patch Available (was: Open)
> TestDelegationTokenForProxyUser.testWebHdfsDoAs fails on trunk
> --
>
> Key: HDFS-9195
> URL: https://issues.apache.org/jira/browse/HDFS-9195
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Tsuyoshi Ozawa
>Assignee: ramtin
> Attachments: HDFS-9195.001.patch
>
>
> {quote}
> testWebHdfsDoAs(org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser)
> Time elapsed: 1.299 sec <<< FAILURE!
> org.junit.ComparisonFailure: expected:<...ocalhost:44528/user/[Proxy]User>
> but was:<...ocalhost:44528/user/[Real]User>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser.testWebHdfsDoAs(TestDelegationTokenForProxyUser.java:163)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9195) TestDelegationTokenForProxyUser.testWebHdfsDoAs fails on trunk
[
https://issues.apache.org/jira/browse/HDFS-9195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramtin updated HDFS-9195:
-
Attachment: HDFS-9195.001.patch
cachedHomeDirectory generated for RealUser and webhdfs.getHomeDirectory() won't
provide a new path for ProxyUser.
> TestDelegationTokenForProxyUser.testWebHdfsDoAs fails on trunk
> --
>
> Key: HDFS-9195
> URL: https://issues.apache.org/jira/browse/HDFS-9195
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Tsuyoshi Ozawa
> Attachments: HDFS-9195.001.patch
>
>
> {quote}
> testWebHdfsDoAs(org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser)
> Time elapsed: 1.299 sec <<< FAILURE!
> org.junit.ComparisonFailure: expected:<...ocalhost:44528/user/[Proxy]User>
> but was:<...ocalhost:44528/user/[Real]User>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser.testWebHdfsDoAs(TestDelegationTokenForProxyUser.java:163)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-11110) Hardcoded BLOCK SIZE value of 4096 is not appropriate for PowerPC
ramtin created HDFS-0: - Summary: Hardcoded BLOCK SIZE value of 4096 is not appropriate for PowerPC Key: HDFS-0 URL: https://issues.apache.org/jira/browse/HDFS-0 Project: Hadoop HDFS Issue Type: Bug Reporter: ramtin Assignee: ramtin -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-11110) Hardcoded BLOCK SIZE value of 4096 is not appropriate for PowerPC
[ https://issues.apache.org/jira/browse/HDFS-0?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramtin reassigned HDFS-0: - Assignee: ramtin > Hardcoded BLOCK SIZE value of 4096 is not appropriate for PowerPC > - > > Key: HDFS-0 > URL: https://issues.apache.org/jira/browse/HDFS-0 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: ramtin >Assignee: ramtin > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11110) Hardcoded BLOCK SIZE value of 4096 is not appropriate for PowerPC
[ https://issues.apache.org/jira/browse/HDFS-0?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramtin updated HDFS-0: -- Assignee: (was: ramtin) > Hardcoded BLOCK SIZE value of 4096 is not appropriate for PowerPC > - > > Key: HDFS-0 > URL: https://issues.apache.org/jira/browse/HDFS-0 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: ramtin > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11110) Hardcoded BLOCK SIZE value of 4096 is not appropriate for PowerPC
[ https://issues.apache.org/jira/browse/HDFS-0?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramtin updated HDFS-0: -- Status: Patch Available (was: Open) > Hardcoded BLOCK SIZE value of 4096 is not appropriate for PowerPC > - > > Key: HDFS-0 > URL: https://issues.apache.org/jira/browse/HDFS-0 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: ramtin >Assignee: ramtin > Attachments: HDFS-0.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11110) Hardcoded BLOCK SIZE value of 4096 is not appropriate for PowerPC
[ https://issues.apache.org/jira/browse/HDFS-0?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramtin updated HDFS-0: -- Attachment: HDFS-0.001.patch > Hardcoded BLOCK SIZE value of 4096 is not appropriate for PowerPC > - > > Key: HDFS-0 > URL: https://issues.apache.org/jira/browse/HDFS-0 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: ramtin >Assignee: ramtin > Attachments: HDFS-0.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11110) Hardcoded BLOCK SIZE value of 4096 is not appropriate for PowerPC
[ https://issues.apache.org/jira/browse/HDFS-0?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramtin updated HDFS-0: -- Description: Using NativeIO.POSIX.getCacheManipulator().getOperatingSystemPageSize() function rather than hard coded block size > Hardcoded BLOCK SIZE value of 4096 is not appropriate for PowerPC > - > > Key: HDFS-0 > URL: https://issues.apache.org/jira/browse/HDFS-0 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: ramtin >Assignee: ramtin > Attachments: HDFS-0.001.patch > > > Using NativeIO.POSIX.getCacheManipulator().getOperatingSystemPageSize() > function rather than hard coded block size -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10382) In WebHDFS numeric usernames do not work with DataNode
ramtin created HDFS-10382:
-
Summary: In WebHDFS numeric usernames do not work with DataNode
Key: HDFS-10382
URL: https://issues.apache.org/jira/browse/HDFS-10382
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Reporter: ramtin
Assignee: ramtin
Operations like {code:java}curl -i
-L"http://:/webhdfs/v1/?user.name=0123&op=OPEN"{code} that
directed to DataNode fail because of not reading the suggested domain pattern
from the configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10382) In WebHDFS numeric usernames do not work with DataNode
[
https://issues.apache.org/jira/browse/HDFS-10382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramtin updated HDFS-10382:
--
Status: Patch Available (was: Open)
> In WebHDFS numeric usernames do not work with DataNode
> --
>
> Key: HDFS-10382
> URL: https://issues.apache.org/jira/browse/HDFS-10382
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Reporter: ramtin
>Assignee: ramtin
>
> Operations like {code:java}curl -i
> -L"http://:/webhdfs/v1/?user.name=0123&op=OPEN"{code} that
> directed to DataNode fail because of not reading the suggested domain pattern
> from the configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10382) In WebHDFS numeric usernames do not work with DataNode
[
https://issues.apache.org/jira/browse/HDFS-10382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramtin updated HDFS-10382:
--
Attachment: HADOOP-10382.patch
> In WebHDFS numeric usernames do not work with DataNode
> --
>
> Key: HDFS-10382
> URL: https://issues.apache.org/jira/browse/HDFS-10382
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Reporter: ramtin
>Assignee: ramtin
> Attachments: HADOOP-10382.patch
>
>
> Operations like {code:java}curl -i
> -L"http://:/webhdfs/v1/?user.name=0123&op=OPEN"{code} that
> directed to DataNode fail because of not reading the suggested domain pattern
> from the configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10382) In WebHDFS numeric usernames do not work with DataNode
[
https://issues.apache.org/jira/browse/HDFS-10382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289536#comment-15289536
]
ramtin commented on HDFS-10382:
---
Thanks [~aw] for your comment. I think you are right but this patch just tries
to fix HDFS-4983 problem in reading domain pattern from the configuration and
even this problem can happen for non-numeric usernames.
> In WebHDFS numeric usernames do not work with DataNode
> --
>
> Key: HDFS-10382
> URL: https://issues.apache.org/jira/browse/HDFS-10382
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Reporter: ramtin
>Assignee: ramtin
> Attachments: HADOOP-10382.patch
>
>
> Operations like {code:java}curl -i
> -L"http://:/webhdfs/v1/?user.name=0123&op=OPEN"{code} that
> directed to DataNode fail because of not reading the suggested domain pattern
> from the configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-1950) Blocks that are under construction are not getting read if the blocks are more than 10. Only complete blocks are read properly.
[ https://issues.apache.org/jira/browse/HDFS-1950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramtin reassigned HDFS-1950: Assignee: ramtin (was: Uma Maheswara Rao G) > Blocks that are under construction are not getting read if the blocks are > more than 10. Only complete blocks are read properly. > > > Key: HDFS-1950 > URL: https://issues.apache.org/jira/browse/HDFS-1950 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client, namenode >Affects Versions: 0.20.205.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramtin >Priority: Blocker > Attachments: HDFS-1950-2.patch, HDFS-1950.1.patch, > hdfs-1950-0.20-append-tests.txt, hdfs-1950-trunk-test.txt, > hdfs-1950-trunk-test.txt > > > Before going to the root cause lets see the read behavior for a file having > more than 10 blocks in append case.. > Logic: > > There is prefetch size dfs.read.prefetch.size for the DFSInputStream which > has default value of 10 > This prefetch size is the number of blocks that the client will fetch from > the namenode for reading a file.. > For example lets assume that a file X having 22 blocks is residing in HDFS > The reader first fetches first 10 blocks from the namenode and start reading > After the above step , the reader fetches the next 10 blocks from NN and > continue reading > Then the reader fetches the remaining 2 blocks from NN and complete the write > Cause: > === > Lets see the cause for this issue now... > Scenario that will fail is "Writer wrote 10+ blocks and a partial block and > called sync. Reader trying to read the file will not get the last partial > block" . > Client first gets the 10 block locations from the NN. Now it checks whether > the file is under construction and if so it gets the size of the last partial > block from datanode and reads the full file > However when the number of blocks is more than 10, the last block will not be > in the first fetch. It will be in the second or other blocks(last block will > be in (num of blocks / 10)th fetch) > The problem now is, in DFSClient there is no logic to get the size of the > last partial block(as in case of point 1), for the rest of the fetches other > than first fetch, the reader will not be able to read the complete data > synced...!! > also the InputStream.available api uses the first fetched block size to > iterate. Ideally this size has to be increased -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-1950) Blocks that are under construction are not getting read if the blocks are more than 10. Only complete blocks are read properly.
[ https://issues.apache.org/jira/browse/HDFS-1950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramtin updated HDFS-1950: - Assignee: (was: ramtin) > Blocks that are under construction are not getting read if the blocks are > more than 10. Only complete blocks are read properly. > > > Key: HDFS-1950 > URL: https://issues.apache.org/jira/browse/HDFS-1950 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client, namenode >Affects Versions: 0.20.205.0 >Reporter: ramkrishna.s.vasudevan >Priority: Blocker > Attachments: HDFS-1950-2.patch, HDFS-1950.1.patch, > hdfs-1950-0.20-append-tests.txt, hdfs-1950-trunk-test.txt, > hdfs-1950-trunk-test.txt > > > Before going to the root cause lets see the read behavior for a file having > more than 10 blocks in append case.. > Logic: > > There is prefetch size dfs.read.prefetch.size for the DFSInputStream which > has default value of 10 > This prefetch size is the number of blocks that the client will fetch from > the namenode for reading a file.. > For example lets assume that a file X having 22 blocks is residing in HDFS > The reader first fetches first 10 blocks from the namenode and start reading > After the above step , the reader fetches the next 10 blocks from NN and > continue reading > Then the reader fetches the remaining 2 blocks from NN and complete the write > Cause: > === > Lets see the cause for this issue now... > Scenario that will fail is "Writer wrote 10+ blocks and a partial block and > called sync. Reader trying to read the file will not get the last partial > block" . > Client first gets the 10 block locations from the NN. Now it checks whether > the file is under construction and if so it gets the size of the last partial > block from datanode and reads the full file > However when the number of blocks is more than 10, the last block will not be > in the first fetch. It will be in the second or other blocks(last block will > be in (num of blocks / 10)th fetch) > The problem now is, in DFSClient there is no logic to get the size of the > last partial block(as in case of point 1), for the rest of the fetches other > than first fetch, the reader will not be able to read the complete data > synced...!! > also the InputStream.available api uses the first fetched block size to > iterate. Ideally this size has to be increased -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-1950) Blocks that are under construction are not getting read if the blocks are more than 10. Only complete blocks are read properly.
[ https://issues.apache.org/jira/browse/HDFS-1950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramtin updated HDFS-1950: - Assignee: Uma Maheswara Rao G > Blocks that are under construction are not getting read if the blocks are > more than 10. Only complete blocks are read properly. > > > Key: HDFS-1950 > URL: https://issues.apache.org/jira/browse/HDFS-1950 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client, namenode >Affects Versions: 0.20.205.0 >Reporter: ramkrishna.s.vasudevan >Assignee: Uma Maheswara Rao G >Priority: Blocker > Attachments: HDFS-1950-2.patch, HDFS-1950.1.patch, > hdfs-1950-0.20-append-tests.txt, hdfs-1950-trunk-test.txt, > hdfs-1950-trunk-test.txt > > > Before going to the root cause lets see the read behavior for a file having > more than 10 blocks in append case.. > Logic: > > There is prefetch size dfs.read.prefetch.size for the DFSInputStream which > has default value of 10 > This prefetch size is the number of blocks that the client will fetch from > the namenode for reading a file.. > For example lets assume that a file X having 22 blocks is residing in HDFS > The reader first fetches first 10 blocks from the namenode and start reading > After the above step , the reader fetches the next 10 blocks from NN and > continue reading > Then the reader fetches the remaining 2 blocks from NN and complete the write > Cause: > === > Lets see the cause for this issue now... > Scenario that will fail is "Writer wrote 10+ blocks and a partial block and > called sync. Reader trying to read the file will not get the last partial > block" . > Client first gets the 10 block locations from the NN. Now it checks whether > the file is under construction and if so it gets the size of the last partial > block from datanode and reads the full file > However when the number of blocks is more than 10, the last block will not be > in the first fetch. It will be in the second or other blocks(last block will > be in (num of blocks / 10)th fetch) > The problem now is, in DFSClient there is no logic to get the size of the > last partial block(as in case of point 1), for the rest of the fetches other > than first fetch, the reader will not be able to read the complete data > synced...!! > also the InputStream.available api uses the first fetched block size to > iterate. Ideally this size has to be increased -- This message was sent by Atlassian JIRA (v6.3.4#6332)
