[jira] [Updated] (HDFS-9100) HDFS Balancer does not respect dfs.client.use.datanode.hostname
[
https://issues.apache.org/jira/browse/HDFS-9100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Casey Brotherton updated HDFS-9100:
---
Status: Open (was: Patch Available)
> HDFS Balancer does not respect dfs.client.use.datanode.hostname
> ---
>
> Key: HDFS-9100
> URL: https://issues.apache.org/jira/browse/HDFS-9100
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, HDFS
>Reporter: Yongjun Zhang
>Assignee: Casey Brotherton
> Attachments: HDFS-9100.000.patch, HDFS-9100.001.patch,
> HDFS-9100.002.patch
>
>
> In Balancer Dispatch.java:
> {code}
>private void dispatch() {
> LOG.info("Start moving " + this);
> Socket sock = new Socket();
> DataOutputStream out = null;
> DataInputStream in = null;
> try {
> sock.connect(
> NetUtils.createSocketAddr(target.getDatanodeInfo().getXferAddr()),
> HdfsConstants.READ_TIMEOUT);
> {code}
> getXferAddr() is called without taking into consideration of
> dfs.client.use.datanode.hostname setting, this would possibly fail balancer
> run issued from outside a cluster.
> Thanks [~caseyjbrotherton] for reporting the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9100) HDFS Balancer does not respect dfs.client.use.datanode.hostname
[
https://issues.apache.org/jira/browse/HDFS-9100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Casey Brotherton updated HDFS-9100:
---
Attachment: HDFS-9100.003.patch
> HDFS Balancer does not respect dfs.client.use.datanode.hostname
> ---
>
> Key: HDFS-9100
> URL: https://issues.apache.org/jira/browse/HDFS-9100
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, HDFS
>Reporter: Yongjun Zhang
>Assignee: Casey Brotherton
> Attachments: HDFS-9100.000.patch, HDFS-9100.001.patch,
> HDFS-9100.002.patch, HDFS-9100.003.patch
>
>
> In Balancer Dispatch.java:
> {code}
>private void dispatch() {
> LOG.info("Start moving " + this);
> Socket sock = new Socket();
> DataOutputStream out = null;
> DataInputStream in = null;
> try {
> sock.connect(
> NetUtils.createSocketAddr(target.getDatanodeInfo().getXferAddr()),
> HdfsConstants.READ_TIMEOUT);
> {code}
> getXferAddr() is called without taking into consideration of
> dfs.client.use.datanode.hostname setting, this would possibly fail balancer
> run issued from outside a cluster.
> Thanks [~caseyjbrotherton] for reporting the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9100) HDFS Balancer does not respect dfs.client.use.datanode.hostname
[
https://issues.apache.org/jira/browse/HDFS-9100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Casey Brotherton updated HDFS-9100:
---
Status: Patch Available (was: Open)
> HDFS Balancer does not respect dfs.client.use.datanode.hostname
> ---
>
> Key: HDFS-9100
> URL: https://issues.apache.org/jira/browse/HDFS-9100
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, HDFS
>Reporter: Yongjun Zhang
>Assignee: Casey Brotherton
> Attachments: HDFS-9100.000.patch, HDFS-9100.001.patch,
> HDFS-9100.002.patch, HDFS-9100.003.patch
>
>
> In Balancer Dispatch.java:
> {code}
>private void dispatch() {
> LOG.info("Start moving " + this);
> Socket sock = new Socket();
> DataOutputStream out = null;
> DataInputStream in = null;
> try {
> sock.connect(
> NetUtils.createSocketAddr(target.getDatanodeInfo().getXferAddr()),
> HdfsConstants.READ_TIMEOUT);
> {code}
> getXferAddr() is called without taking into consideration of
> dfs.client.use.datanode.hostname setting, this would possibly fail balancer
> run issued from outside a cluster.
> Thanks [~caseyjbrotherton] for reporting the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9100) HDFS Balancer does not respect dfs.client.use.datanode.hostname
[
https://issues.apache.org/jira/browse/HDFS-9100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14935653#comment-14935653
]
Casey Brotherton commented on HDFS-9100:
Patch is limited to Dispatcher.java
pre-patch concerns shows something with:
org.apache.hadoop.hdfs.server.datanode.DirectoryScanner
Test log failure was with:
org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner
Not sure what the issue is.
> HDFS Balancer does not respect dfs.client.use.datanode.hostname
> ---
>
> Key: HDFS-9100
> URL: https://issues.apache.org/jira/browse/HDFS-9100
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, HDFS
>Reporter: Yongjun Zhang
>Assignee: Casey Brotherton
> Attachments: HDFS-9100.000.patch, HDFS-9100.001.patch,
> HDFS-9100.002.patch, HDFS-9100.003.patch
>
>
> In Balancer Dispatch.java:
> {code}
>private void dispatch() {
> LOG.info("Start moving " + this);
> Socket sock = new Socket();
> DataOutputStream out = null;
> DataInputStream in = null;
> try {
> sock.connect(
> NetUtils.createSocketAddr(target.getDatanodeInfo().getXferAddr()),
> HdfsConstants.READ_TIMEOUT);
> {code}
> getXferAddr() is called without taking into consideration of
> dfs.client.use.datanode.hostname setting, this would possibly fail balancer
> run issued from outside a cluster.
> Thanks [~caseyjbrotherton] for reporting the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9100) HDFS Balancer does not respect dfs.client.use.datanode.hostname
[
https://issues.apache.org/jira/browse/HDFS-9100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14934593#comment-14934593
]
Casey Brotherton commented on HDFS-9100:
I am sorry, the failure appears to be due to changes with HDFS-8053.
Will reapply the patch, and run tests tonight.
> HDFS Balancer does not respect dfs.client.use.datanode.hostname
> ---
>
> Key: HDFS-9100
> URL: https://issues.apache.org/jira/browse/HDFS-9100
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, HDFS
>Reporter: Yongjun Zhang
>Assignee: Casey Brotherton
> Attachments: HDFS-9100.000.patch, HDFS-9100.001.patch,
> HDFS-9100.002.patch
>
>
> In Balancer Dispatch.java:
> {code}
>private void dispatch() {
> LOG.info("Start moving " + this);
> Socket sock = new Socket();
> DataOutputStream out = null;
> DataInputStream in = null;
> try {
> sock.connect(
> NetUtils.createSocketAddr(target.getDatanodeInfo().getXferAddr()),
> HdfsConstants.READ_TIMEOUT);
> {code}
> getXferAddr() is called without taking into consideration of
> dfs.client.use.datanode.hostname setting, this would possibly fail balancer
> run issued from outside a cluster.
> Thanks [~caseyjbrotherton] for reporting the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9100) HDFS Balancer does not respect dfs.client.use.datanode.hostname
[
https://issues.apache.org/jira/browse/HDFS-9100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Casey Brotherton updated HDFS-9100:
---
Status: Open (was: Patch Available)
> HDFS Balancer does not respect dfs.client.use.datanode.hostname
> ---
>
> Key: HDFS-9100
> URL: https://issues.apache.org/jira/browse/HDFS-9100
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, HDFS
>Reporter: Yongjun Zhang
>Assignee: Casey Brotherton
> Attachments: HDFS-9100.000.patch, HDFS-9100.001.patch
>
>
> In Balancer Dispatch.java:
> {code}
>private void dispatch() {
> LOG.info("Start moving " + this);
> Socket sock = new Socket();
> DataOutputStream out = null;
> DataInputStream in = null;
> try {
> sock.connect(
> NetUtils.createSocketAddr(target.getDatanodeInfo().getXferAddr()),
> HdfsConstants.READ_TIMEOUT);
> {code}
> getXferAddr() is called without taking into consideration of
> dfs.client.use.datanode.hostname setting, this would possibly fail balancer
> run issued from outside a cluster.
> Thanks [~caseyjbrotherton] for reporting the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9100) HDFS Balancer does not respect dfs.client.use.datanode.hostname
[
https://issues.apache.org/jira/browse/HDFS-9100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Casey Brotherton updated HDFS-9100:
---
Attachment: HDFS-9100.002.patch
Thank you for the review, [~yzhangal]
I have made the changes and uploaded a new patch file.
> HDFS Balancer does not respect dfs.client.use.datanode.hostname
> ---
>
> Key: HDFS-9100
> URL: https://issues.apache.org/jira/browse/HDFS-9100
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, HDFS
>Reporter: Yongjun Zhang
>Assignee: Casey Brotherton
> Attachments: HDFS-9100.000.patch, HDFS-9100.001.patch,
> HDFS-9100.002.patch
>
>
> In Balancer Dispatch.java:
> {code}
>private void dispatch() {
> LOG.info("Start moving " + this);
> Socket sock = new Socket();
> DataOutputStream out = null;
> DataInputStream in = null;
> try {
> sock.connect(
> NetUtils.createSocketAddr(target.getDatanodeInfo().getXferAddr()),
> HdfsConstants.READ_TIMEOUT);
> {code}
> getXferAddr() is called without taking into consideration of
> dfs.client.use.datanode.hostname setting, this would possibly fail balancer
> run issued from outside a cluster.
> Thanks [~caseyjbrotherton] for reporting the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9100) HDFS Balancer does not respect dfs.client.use.datanode.hostname
[
https://issues.apache.org/jira/browse/HDFS-9100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Casey Brotherton updated HDFS-9100:
---
Status: Patch Available (was: Open)
> HDFS Balancer does not respect dfs.client.use.datanode.hostname
> ---
>
> Key: HDFS-9100
> URL: https://issues.apache.org/jira/browse/HDFS-9100
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, HDFS
>Reporter: Yongjun Zhang
>Assignee: Casey Brotherton
> Attachments: HDFS-9100.000.patch, HDFS-9100.001.patch,
> HDFS-9100.002.patch
>
>
> In Balancer Dispatch.java:
> {code}
>private void dispatch() {
> LOG.info("Start moving " + this);
> Socket sock = new Socket();
> DataOutputStream out = null;
> DataInputStream in = null;
> try {
> sock.connect(
> NetUtils.createSocketAddr(target.getDatanodeInfo().getXferAddr()),
> HdfsConstants.READ_TIMEOUT);
> {code}
> getXferAddr() is called without taking into consideration of
> dfs.client.use.datanode.hostname setting, this would possibly fail balancer
> run issued from outside a cluster.
> Thanks [~caseyjbrotherton] for reporting the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9100) HDFS Balancer does not respect dfs.client.use.datanode.hostname
[
https://issues.apache.org/jira/browse/HDFS-9100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14908125#comment-14908125
]
Casey Brotherton commented on HDFS-9100:
Hello
hdfs tests were related to hflush, and secondary namenode webui. Both of which
I didn't change.
Should I re-publish the patch to see if those tests resolve?
Thanks,
Casey
> HDFS Balancer does not respect dfs.client.use.datanode.hostname
> ---
>
> Key: HDFS-9100
> URL: https://issues.apache.org/jira/browse/HDFS-9100
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, HDFS
>Reporter: Yongjun Zhang
>Assignee: Casey Brotherton
> Attachments: HDFS-9100.000.patch, HDFS-9100.001.patch
>
>
> In Balancer Dispatch.java:
> {code}
>private void dispatch() {
> LOG.info("Start moving " + this);
> Socket sock = new Socket();
> DataOutputStream out = null;
> DataInputStream in = null;
> try {
> sock.connect(
> NetUtils.createSocketAddr(target.getDatanodeInfo().getXferAddr()),
> HdfsConstants.READ_TIMEOUT);
> {code}
> getXferAddr() is called without taking into consideration of
> dfs.client.use.datanode.hostname setting, this would possibly fail balancer
> run issued from outside a cluster.
> Thanks [~caseyjbrotherton] for reporting the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9100) HDFS Balancer does not respect dfs.client.use.datanode.hostname
[
https://issues.apache.org/jira/browse/HDFS-9100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Casey Brotherton updated HDFS-9100:
---
Status: Patch Available (was: Open)
> HDFS Balancer does not respect dfs.client.use.datanode.hostname
> ---
>
> Key: HDFS-9100
> URL: https://issues.apache.org/jira/browse/HDFS-9100
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, HDFS
>Reporter: Yongjun Zhang
>Assignee: Casey Brotherton
> Attachments: HDFS-9100.000.patch
>
>
> In Balancer Dispatch.java:
> {code}
>private void dispatch() {
> LOG.info("Start moving " + this);
> Socket sock = new Socket();
> DataOutputStream out = null;
> DataInputStream in = null;
> try {
> sock.connect(
> NetUtils.createSocketAddr(target.getDatanodeInfo().getXferAddr()),
> HdfsConstants.READ_TIMEOUT);
> {code}
> getXferAddr() is called without taking into consideration of
> dfs.client.use.datanode.hostname setting, this would possibly fail balancer
> run issued from outside a cluster.
> Thanks [~caseyjbrotherton] for reporting the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9100) HDFS Balancer does not respect dfs.client.use.datanode.hostname
[
https://issues.apache.org/jira/browse/HDFS-9100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Casey Brotherton updated HDFS-9100:
---
Attachment: HDFS-9100.000.patch
Still testing in my lab. A unit test with multiple networks is difficult.
Will change to patch-available once I finished testing
> HDFS Balancer does not respect dfs.client.use.datanode.hostname
> ---
>
> Key: HDFS-9100
> URL: https://issues.apache.org/jira/browse/HDFS-9100
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, HDFS
>Reporter: Yongjun Zhang
>Assignee: Casey Brotherton
> Attachments: HDFS-9100.000.patch
>
>
> In Balancer Dispatch.java:
> {code}
>private void dispatch() {
> LOG.info("Start moving " + this);
> Socket sock = new Socket();
> DataOutputStream out = null;
> DataInputStream in = null;
> try {
> sock.connect(
> NetUtils.createSocketAddr(target.getDatanodeInfo().getXferAddr()),
> HdfsConstants.READ_TIMEOUT);
> {code}
> getXferAddr() is called without taking into consideration of
> dfs.client.use.datanode.hostname setting, this would possibly fail balancer
> run issued from outside a cluster.
> Thanks [~caseyjbrotherton] for reporting the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9100) HDFS Balancer does not respect dfs.client.use.datanode.hostname
[
https://issues.apache.org/jira/browse/HDFS-9100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904479#comment-14904479
]
Casey Brotherton commented on HDFS-9100:
Created a multi-node cluster with multi-homed networks.
Created one of the nodes with only one ip address, on the subnet
that was not the primary network of the other hosts in the cluster.
Verified that without appropriate configuration, the lone node
could not send data to datanodes.
Confirmed that changing dfs.client.use.datanode.hostname allowed the
lone node to put and read from the datanodes.
Placed data within the HDFS cluster in an unbalanced way. ( dfs.replication=1
on one datanode,
with minimal data spread across two datanodes with dfs.replication=2)
Running the balancer from the lone node causes these errors:
15/09/23 06:07:04 WARN balancer.Dispatcher: Failed to move blk_1073742600_1776
with size=4128368 from 10.17.74.156:50010:DISK to 10.17.74.158:50010:DISK
through 10.17.74.156:50010: Network is unreachable
After the changes, the balancer worked. The logging still showed the IP
address.
15/09/23 06:05:38 INFO balancer.Dispatcher: Successfully moved
blk_1073742139_1315 with size=4128368 from 10.17.74.156:50010:DISK to
10.17.74.158:50010:DISK through 10.17.74.156:50010
This did not test a large environment, where blocks were moved by proxy,
although I don't believe that would cause any problems.
> HDFS Balancer does not respect dfs.client.use.datanode.hostname
> ---
>
> Key: HDFS-9100
> URL: https://issues.apache.org/jira/browse/HDFS-9100
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, HDFS
>Reporter: Yongjun Zhang
>Assignee: Casey Brotherton
> Attachments: HDFS-9100.000.patch
>
>
> In Balancer Dispatch.java:
> {code}
>private void dispatch() {
> LOG.info("Start moving " + this);
> Socket sock = new Socket();
> DataOutputStream out = null;
> DataInputStream in = null;
> try {
> sock.connect(
> NetUtils.createSocketAddr(target.getDatanodeInfo().getXferAddr()),
> HdfsConstants.READ_TIMEOUT);
> {code}
> getXferAddr() is called without taking into consideration of
> dfs.client.use.datanode.hostname setting, this would possibly fail balancer
> run issued from outside a cluster.
> Thanks [~caseyjbrotherton] for reporting the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9100) HDFS Balancer does not respect dfs.client.use.datanode.hostname
[
https://issues.apache.org/jira/browse/HDFS-9100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Casey Brotherton updated HDFS-9100:
---
Attachment: HDFS-9100.001.patch
Fixed the whitespace, and line length, and deprecation change
> HDFS Balancer does not respect dfs.client.use.datanode.hostname
> ---
>
> Key: HDFS-9100
> URL: https://issues.apache.org/jira/browse/HDFS-9100
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, HDFS
>Reporter: Yongjun Zhang
>Assignee: Casey Brotherton
> Attachments: HDFS-9100.000.patch, HDFS-9100.001.patch
>
>
> In Balancer Dispatch.java:
> {code}
>private void dispatch() {
> LOG.info("Start moving " + this);
> Socket sock = new Socket();
> DataOutputStream out = null;
> DataInputStream in = null;
> try {
> sock.connect(
> NetUtils.createSocketAddr(target.getDatanodeInfo().getXferAddr()),
> HdfsConstants.READ_TIMEOUT);
> {code}
> getXferAddr() is called without taking into consideration of
> dfs.client.use.datanode.hostname setting, this would possibly fail balancer
> run issued from outside a cluster.
> Thanks [~caseyjbrotherton] for reporting the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9100) HDFS Balancer does not respect dfs.client.use.datanode.hostname
[
https://issues.apache.org/jira/browse/HDFS-9100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Casey Brotherton updated HDFS-9100:
---
Status: Patch Available (was: Open)
> HDFS Balancer does not respect dfs.client.use.datanode.hostname
> ---
>
> Key: HDFS-9100
> URL: https://issues.apache.org/jira/browse/HDFS-9100
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, HDFS
>Reporter: Yongjun Zhang
>Assignee: Casey Brotherton
> Attachments: HDFS-9100.000.patch, HDFS-9100.001.patch
>
>
> In Balancer Dispatch.java:
> {code}
>private void dispatch() {
> LOG.info("Start moving " + this);
> Socket sock = new Socket();
> DataOutputStream out = null;
> DataInputStream in = null;
> try {
> sock.connect(
> NetUtils.createSocketAddr(target.getDatanodeInfo().getXferAddr()),
> HdfsConstants.READ_TIMEOUT);
> {code}
> getXferAddr() is called without taking into consideration of
> dfs.client.use.datanode.hostname setting, this would possibly fail balancer
> run issued from outside a cluster.
> Thanks [~caseyjbrotherton] for reporting the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9100) HDFS Balancer does not respect dfs.client.use.datanode.hostname
[
https://issues.apache.org/jira/browse/HDFS-9100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Casey Brotherton updated HDFS-9100:
---
Status: Open (was: Patch Available)
> HDFS Balancer does not respect dfs.client.use.datanode.hostname
> ---
>
> Key: HDFS-9100
> URL: https://issues.apache.org/jira/browse/HDFS-9100
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, HDFS
>Reporter: Yongjun Zhang
>Assignee: Casey Brotherton
> Attachments: HDFS-9100.000.patch
>
>
> In Balancer Dispatch.java:
> {code}
>private void dispatch() {
> LOG.info("Start moving " + this);
> Socket sock = new Socket();
> DataOutputStream out = null;
> DataInputStream in = null;
> try {
> sock.connect(
> NetUtils.createSocketAddr(target.getDatanodeInfo().getXferAddr()),
> HdfsConstants.READ_TIMEOUT);
> {code}
> getXferAddr() is called without taking into consideration of
> dfs.client.use.datanode.hostname setting, this would possibly fail balancer
> run issued from outside a cluster.
> Thanks [~caseyjbrotherton] for reporting the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8118) Delay in checkpointing Trash can leave trash for 2 intervals before deleting
[ https://issues.apache.org/jira/browse/HDFS-8118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14709816#comment-14709816 ] Casey Brotherton commented on HDFS-8118: Issue in the common tests is a bind issue in the unit test: TestWebDelegationToken was a bind issue: java.net.BindException: Address already in use Is there a way to trigger the tests again to see if they clear independently? That would provide some indication whether there is a timing issue with the TestWebDelegationToken test. Thanks, Casey Delay in checkpointing Trash can leave trash for 2 intervals before deleting Key: HDFS-8118 URL: https://issues.apache.org/jira/browse/HDFS-8118 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.7.1 Reporter: Casey Brotherton Assignee: Casey Brotherton Priority: Trivial Attachments: HDFS-8118.001.patch, HDFS-8118.patch When the fs.trash.checkpoint.interval and the fs.trash.interval are set non-zero and the same, it is possible for trash to be left for two intervals. The TrashPolicyDefault will use a floor and ceiling function to ensure that the Trash will be checkpointed every interval of minutes. Each user's trash is checkpointed individually. The time resolution of the checkpoint timestamp is to the second. If the seconds switch while one user is checkpointing, then the next user's timestamp will be later. This will cause the next user's checkpoint to not be deleted at the next interval. I have recreated this in a lab cluster I also have a suggestion for a patch that I can upload later tonight after testing it further. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8118) Delay in checkpointing Trash can leave trash for 2 intervals before deleting
[ https://issues.apache.org/jira/browse/HDFS-8118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Brotherton updated HDFS-8118: --- Attachment: HDFS-8118.001.patch This is a simplified patch addressing only the defect, and not the testcases. Delay in checkpointing Trash can leave trash for 2 intervals before deleting Key: HDFS-8118 URL: https://issues.apache.org/jira/browse/HDFS-8118 Project: Hadoop HDFS Issue Type: Bug Reporter: Casey Brotherton Assignee: Casey Brotherton Priority: Trivial Attachments: HDFS-8118.001.patch, HDFS-8118.patch When the fs.trash.checkpoint.interval and the fs.trash.interval are set non-zero and the same, it is possible for trash to be left for two intervals. The TrashPolicyDefault will use a floor and ceiling function to ensure that the Trash will be checkpointed every interval of minutes. Each user's trash is checkpointed individually. The time resolution of the checkpoint timestamp is to the second. If the seconds switch while one user is checkpointing, then the next user's timestamp will be later. This will cause the next user's checkpoint to not be deleted at the next interval. I have recreated this in a lab cluster I also have a suggestion for a patch that I can upload later tonight after testing it further. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8118) Delay in checkpointing Trash can leave trash for 2 intervals before deleting
[ https://issues.apache.org/jira/browse/HDFS-8118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660213#comment-14660213 ] Casey Brotherton commented on HDFS-8118: Talked with Harsh about this offline. Still working through a testcase. We have talked about a method for delaying between the different checkpoints. There are still difficulties, though , as the problem occurs when two different users delete files, and there is a delay between checkpointing each user's deletes. However, TestTrash operates under the assumption that there is only one user that is creating files, and removing them for the trash. ( For example, the shell is used to getCurrentTrashDirectory, and that will only return one location. For a test, I will either need to break Object Oriented walls, and create a path with users/d/.Trash/Current, or create a test that doesn't really test for the issue. ) Still trying to work on this. Delay in checkpointing Trash can leave trash for 2 intervals before deleting Key: HDFS-8118 URL: https://issues.apache.org/jira/browse/HDFS-8118 Project: Hadoop HDFS Issue Type: Bug Reporter: Casey Brotherton Assignee: Casey Brotherton Priority: Trivial Attachments: HDFS-8118.patch When the fs.trash.checkpoint.interval and the fs.trash.interval are set non-zero and the same, it is possible for trash to be left for two intervals. The TrashPolicyDefault will use a floor and ceiling function to ensure that the Trash will be checkpointed every interval of minutes. Each user's trash is checkpointed individually. The time resolution of the checkpoint timestamp is to the second. If the seconds switch while one user is checkpointing, then the next user's timestamp will be later. This will cause the next user's checkpoint to not be deleted at the next interval. I have recreated this in a lab cluster I also have a suggestion for a patch that I can upload later tonight after testing it further. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-5696) Examples for httpfs REST API incorrect on apache.org
Casey Brotherton created HDFS-5696: -- Summary: Examples for httpfs REST API incorrect on apache.org Key: HDFS-5696 URL: https://issues.apache.org/jira/browse/HDFS-5696 Project: Hadoop HDFS Issue Type: Bug Components: documentation Affects Versions: 2.2.0 Environment: NA Reporter: Casey Brotherton Priority: Minor The examples provided for the httpfs REST API are incorrect. http://hadoop.apache.org/docs/r2.2.0/hadoop-hdfs-httpfs/index.html http://hadoop.apache.org/docs/r2.0.5-alpha/hadoop-hdfs-httpfs/index.html From the documentation: * HttpFS is a separate service from Hadoop NameNode. HttpFS itself is Java web-application and it runs using a preconfigured Tomcat bundled with HttpFS binary distribution. HttpFS HTTP web-service API calls are HTTP REST calls that map to a HDFS file system operation. For example, using the curl Unix command: $ curl http://httpfs-host:14000/webhdfs/v1/user/foo/README.txt returns the contents of the HDFS /user/foo/README.txt file. $ curl http://httpfs-host:14000/webhdfs/v1/user/foo?op=list returns the contents of the HDFS /user/foo directory in JSON format. $ curl -X POST http://httpfs-host:14000/webhdfs/v1/user/foo/bar?op=mkdirs creates the HDFS /user/foo.bar directory. *** The commands have incorrect operations. ( Verified through source code in HttpFSFileSystem.java ) In addition, although the webhdfs documentation specifies user.name as optional, on my cluster, each action required a user.name It should be included in the short examples to allow for the greatest chance of success. Three examples rewritten: curl -i -L http://httpfs-host:14000/webhdfs/v1/user/foo/README.txt?op=openuser.name=hdfsuser; curl -i http://httpfs-host:14000/webhdfs/v1/user/foo/?op=liststatususer.name=hdfsuser; curl -i -X PUT http://httpfs-host:14000/webhdfs/v1/user/foo/bar?op=mkdirsuser.name=hdfsuser; Not sure what the convention should be for specifying the user.name. Use hdfs? or a name that is obviously an example? It would also be beneficial if the HTTPfs page linked to the webhdfs documentation page in the text instead of just on the menu sidebar. http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5696) Examples for httpfs REST API incorrect on apache.org
[ https://issues.apache.org/jira/browse/HDFS-5696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Brotherton updated HDFS-5696: --- Priority: Trivial (was: Minor) Examples for httpfs REST API incorrect on apache.org Key: HDFS-5696 URL: https://issues.apache.org/jira/browse/HDFS-5696 Project: Hadoop HDFS Issue Type: Bug Components: documentation Affects Versions: 2.2.0 Environment: NA Reporter: Casey Brotherton Priority: Trivial The examples provided for the httpfs REST API are incorrect. http://hadoop.apache.org/docs/r2.2.0/hadoop-hdfs-httpfs/index.html http://hadoop.apache.org/docs/r2.0.5-alpha/hadoop-hdfs-httpfs/index.html From the documentation: * HttpFS is a separate service from Hadoop NameNode. HttpFS itself is Java web-application and it runs using a preconfigured Tomcat bundled with HttpFS binary distribution. HttpFS HTTP web-service API calls are HTTP REST calls that map to a HDFS file system operation. For example, using the curl Unix command: $ curl http://httpfs-host:14000/webhdfs/v1/user/foo/README.txt returns the contents of the HDFS /user/foo/README.txt file. $ curl http://httpfs-host:14000/webhdfs/v1/user/foo?op=list returns the contents of the HDFS /user/foo directory in JSON format. $ curl -X POST http://httpfs-host:14000/webhdfs/v1/user/foo/bar?op=mkdirs creates the HDFS /user/foo.bar directory. *** The commands have incorrect operations. ( Verified through source code in HttpFSFileSystem.java ) In addition, although the webhdfs documentation specifies user.name as optional, on my cluster, each action required a user.name It should be included in the short examples to allow for the greatest chance of success. Three examples rewritten: curl -i -L http://httpfs-host:14000/webhdfs/v1/user/foo/README.txt?op=openuser.name=hdfsuser; curl -i http://httpfs-host:14000/webhdfs/v1/user/foo/?op=liststatususer.name=hdfsuser; curl -i -X PUT http://httpfs-host:14000/webhdfs/v1/user/foo/bar?op=mkdirsuser.name=hdfsuser; Not sure what the convention should be for specifying the user.name. Use hdfs? or a name that is obviously an example? It would also be beneficial if the HTTPfs page linked to the webhdfs documentation page in the text instead of just on the menu sidebar. http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html -- This message was sent by Atlassian JIRA (v6.1.4#6159)
