[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13637072#comment-13637072
]
Varun Sharma commented on HDFS-3703:
Thanks, Jing..
This holds for UNDER_RECOVERY bloc
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13637028#comment-13637028
]
Jing Zhao commented on HDFS-3703:
-
If the block is still under construction, the namenode w
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13636998#comment-13636998
]
Varun Sharma commented on HDFS-3703:
Do you know if for a block which is not FINALIZED
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13636959#comment-13636959
]
Varun Sharma commented on HDFS-3703:
I actually am seeing an interesting race condition
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13636948#comment-13636948
]
Jing Zhao commented on HDFS-3703:
-
For reading NN still returns the stale nodes to client.
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13636941#comment-13636941
]
Varun Sharma commented on HDFS-3703:
I have a question.
What happens if a client tries
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455749#comment-13455749
]
Tsz Wo (Nicholas), SZE commented on HDFS-3703:
--
That's correct. The MiniDFSCl
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455568#comment-13455568
]
Tsz Wo (Nicholas), SZE commented on HDFS-3703:
--
Hi Jing, I tried to commit the
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455505#comment-13455505
]
Hadoop QA commented on HDFS-3703:
-
-1 overall. Here are the results of testing the latest
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455310#comment-13455310
]
Suresh Srinivas commented on HDFS-3703:
---
We are planning to finish this patch and mak
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455176#comment-13455176
]
Ted Yu commented on HDFS-3703:
--
@N:
My patch needs some more work in the unit test.
I think Ji
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455165#comment-13455165
]
nkeywal commented on HDFS-3703:
---
Thanks a lot, all. I can't wait to have hardware issues on p
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455006#comment-13455006
]
Suresh Srinivas commented on HDFS-3703:
---
bq. I think disabling the data node heart be
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454904#comment-13454904
]
Hudson commented on HDFS-3703:
--
Integrated in Hadoop-Mapreduce-trunk #1195 (See
[https://buil
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454865#comment-13454865
]
Hudson commented on HDFS-3703:
--
Integrated in Hadoop-Hdfs-trunk #1164 (See
[https://builds.ap
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454722#comment-13454722
]
Ted Yu commented on HDFS-3703:
--
I think disabling the data node heart beat simulates GC pause,
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454687#comment-13454687
]
Hudson commented on HDFS-3703:
--
Integrated in Hadoop-Mapreduce-trunk-Commit #2752 (See
[https
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454677#comment-13454677
]
Suresh Srinivas commented on HDFS-3703:
---
Merged the change to branch-2 as well.
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454674#comment-13454674
]
Hudson commented on HDFS-3703:
--
Integrated in Hadoop-Common-trunk-Commit #2728 (See
[https://
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454672#comment-13454672
]
Hudson commented on HDFS-3703:
--
Integrated in Hadoop-Hdfs-trunk-Commit #2791 (See
[https://bu
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454668#comment-13454668
]
Suresh Srinivas commented on HDFS-3703:
---
@Ted, alternatively we could just shutdown t
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454667#comment-13454667
]
Suresh Srinivas commented on HDFS-3703:
---
+1 for the trunk patch. I committed the trun
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454630#comment-13454630
]
Ted Yu commented on HDFS-3703:
--
Looking at how DataNode heart beat is disabled in trunk, I thi
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454529#comment-13454529
]
Hadoop QA commented on HDFS-3703:
-
-1 overall. Here are the results of testing the latest
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454523#comment-13454523
]
Jing Zhao commented on HDFS-3703:
-
Ted: maybe we can also set the heartbeat interval of dat
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454505#comment-13454505
]
Hadoop QA commented on HDFS-3703:
-
-1 overall. Here are the results of testing the latest
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454473#comment-13454473
]
Ted Yu commented on HDFS-3703:
--
What about disabling DataNode heart beat ?
I guess that should
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454469#comment-13454469
]
Jing Zhao commented on HDFS-3703:
-
Ted: in the current patch for trunk I removed the append
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454466#comment-13454466
]
Ted Yu commented on HDFS-3703:
--
I noticed the following in FSNamesystem of hadoop 1.0:
{code}
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454463#comment-13454463
]
Ted Yu commented on HDFS-3703:
--
Should 1.0.4 be put back as Fix Version ?
> D
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454409#comment-13454409
]
Suresh Srinivas commented on HDFS-3703:
---
bq. If I understand your comment correctly,
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454339#comment-13454339
]
Ted Yu commented on HDFS-3703:
--
@Suresh:
If I understand your comment correctly, I should rena
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454301#comment-13454301
]
Suresh Srinivas commented on HDFS-3703:
---
+1 for the trunk patch.
>
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454294#comment-13454294
]
Suresh Srinivas commented on HDFS-3703:
---
bq. Should the backport bring in the above a
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454210#comment-13454210
]
Ted Yu commented on HDFS-3703:
--
In hadoop 1.0, I don't see the following code in
FSNamesystem
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454112#comment-13454112
]
Jing Zhao commented on HDFS-3703:
-
Ted: Thanks very much for volunteering! And please go ah
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454076#comment-13454076
]
nkeywal commented on HDFS-3703:
---
Well, I think it would be quite useful on v1 as well, as HDF
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454073#comment-13454073
]
Ted Yu commented on HDFS-3703:
--
@Jing:
If backport to 1.0 is needed and you're busy, I can giv
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453653#comment-13453653
]
Jing Zhao commented on HDFS-3703:
-
The two test failures are mentioned in HDFS-3811 and HDF
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453631#comment-13453631
]
Hadoop QA commented on HDFS-3703:
-
-1 overall. Here are the results of testing the latest
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453521#comment-13453521
]
Jing Zhao commented on HDFS-3703:
-
Also combine the two test cases in TestGetBlocks.
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453382#comment-13453382
]
Suresh Srinivas commented on HDFS-3703:
---
Some of the failures I checked were related
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453367#comment-13453367
]
Hadoop QA commented on HDFS-3703:
-
-1 overall. Here are the results of testing the latest
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453200#comment-13453200
]
Suresh Srinivas commented on HDFS-3703:
---
bq. I think we can lift the comparator const
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453069#comment-13453069
]
Ted Yu commented on HDFS-3703:
--
{code}
Arrays.sort(b.getLocations(), new DFSUtil.Decom
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452452#comment-13452452
]
Suresh Srinivas commented on HDFS-3703:
---
Some comments on the tests in the patch - te
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452439#comment-13452439
]
nkeywal commented on HDFS-3703:
---
@Jing
Sure. The scenario is:
- open a file for writing, wri
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452420#comment-13452420
]
Jing Zhao commented on HDFS-3703:
-
Nicolas: I'm also looking at the lastLocatedBlocks part.
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452417#comment-13452417
]
Suresh Srinivas commented on HDFS-3703:
---
bq. So this is a cluster-wide setting. I thi
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452408#comment-13452408
]
Ted Yu commented on HDFS-3703:
--
bq. based on the number of nodes marked stale in the cluster
S
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452380#comment-13452380
]
Jing Zhao commented on HDFS-3703:
-
So after the datanodemanager loads the initial stale int
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452324#comment-13452324
]
Ted Yu commented on HDFS-3703:
--
The retrieval of value for staleInterval is in DatanodeManager
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452303#comment-13452303
]
nkeywal commented on HDFS-3703:
---
Hi,
For the DFSInputStream#readBlockLength, I got it I thin
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452277#comment-13452277
]
Ted Yu commented on HDFS-3703:
--
{code}
++ ", which is smaller than the minimal val
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452160#comment-13452160
]
Suresh Srinivas commented on HDFS-3703:
---
Nicolas, lets open a separate jira for the i
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452005#comment-13452005
]
nkeywal commented on HDFS-3703:
---
fyi, I've got an issue when trying with HBase on a real clus
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451123#comment-13451123
]
Ted Yu commented on HDFS-3703:
--
{code}
+ public static final String DFS_DATANODE_STALE_STATE_
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448997#comment-13448997
]
nkeywal commented on HDFS-3703:
---
Hi,
I've done a test on HBase, with the minicluster. Basica
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13447878#comment-13447878
]
nkeywal commented on HDFS-3703:
---
Hi Jing, Suresh,
I'm currently testing the patch with HBase
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13425629#comment-13425629
]
nkeywal commented on HDFS-3703:
---
For HBase, it would be good to have an option to cap this va
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13425305#comment-13425305
]
Jing Zhao commented on HDFS-3703:
-
With respect to Suresh's proposal:
If the last heartbeat
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424910#comment-13424910
]
nkeywal commented on HDFS-3703:
---
The 3 approaches all have advantages. I don't think they are
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423428#comment-13423428
]
Sanjay Radia commented on HDFS-3703:
HDFS-3705 and HDFS-3706 are client based. Suresh
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1340#comment-1340
]
stack commented on HDFS-3703:
-
bq. I think there are differences in the definition of failures
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13421436#comment-13421436
]
Kihwal Lee commented on HDFS-3703:
--
I think there are differences in the definition of fai
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13421018#comment-13421018
]
Kihwal Lee commented on HDFS-3703:
--
bq. It seems like the assumption behind having a "thir
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13421005#comment-13421005
]
nkeywal commented on HDFS-3703:
---
bq. If the last heartbeat time for datanode is more than cer
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420989#comment-13420989
]
Colin Patrick McCabe commented on HDFS-3703:
bq. Can you describe why [inbound
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420935#comment-13420935
]
Suresh Srinivas commented on HDFS-3703:
---
bq. whereas inbound client traffic won't cau
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420926#comment-13420926
]
Colin Patrick McCabe commented on HDFS-3703:
It seems like the assumption behin
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420919#comment-13420919
]
Eli Collins commented on HDFS-3703:
---
Btw the formula for determining whether a DN is dead
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420908#comment-13420908
]
Suresh Srinivas commented on HDFS-3703:
---
Thanks for answering my questions. Here is s
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420851#comment-13420851
]
nkeywal commented on HDFS-3703:
---
bq. Can you describe this better?
If we see this in layers,
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420795#comment-13420795
]
Suresh Srinivas commented on HDFS-3703:
---
bq. Globally, it would be ideal if HDFS sett
[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420783#comment-13420783
]
stack commented on HDFS-3703:
-
...and writing the deadnodes to a file and kicking namenode to r
75 matches
Mail list logo
