[jira] [Updated] (HDFS-16493) [SBN Read]When fast path tail enabled, standby or observer namenode may read uncommitted data
[
https://issues.apache.org/jira/browse/HDFS-16493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liutongwei updated HDFS-16493:
--
Description:
Although fast path tail use quorum read to pull edit log, it seem like can read
uncommitted data in some corner case.
Here is an example. Suppose we have three JN, their init state is:
{code:java}
epoch 1
JN1 [1-3](in-progress)
JN2 [1-3](in-progress)
JN3 [1-4](in-progress)
Note that, in epoch 1 txid 1-3 was committed, and txid 4 not.
{code}
When a failover occur, if a new writer cannot contact to JN3 for network
partition, and finish the recovery stage, and write a new txid 4 in epoch 2,
which value not equal to JN3's.
{code:java}
epcho 2
JN1 [1-3](finalized) [4-4](inprogress)
JN2 [1-3](finalized) [4-4](inprogress)
JN3 [1-4](inprogress)
Note that, in JN3 txid4's value not equal to other JN.
{code}
Now there is a read namenode to pull edits, and it contact to JN3 and JN2, it
got majority response. But it got logs of same length but different content.And
no more information to choose which log is right. If we choose JN3, we got meta
data corruption.
There is a test example patch [^example.patch] for running and debug.
For fix it i think we should add finalized state to
{{{}GetJournaledEditsResponseProto{}}}, so we can discard the fault log.
was:
Although fast path tail use quorum read to pull edit log, it seem like is can
read uncommitted data in some corner case.
Here is an example. Suppose we have three JN, their init state is:
{code:java}
epoch 1
JN1 [1-3](in-progress)
JN2 [1-3](in-progress)
JN3 [1-4](in-progress)
Note that, in epoch 1 txid 1-3 was committed, and txid 4 not.
{code}
When a failover occur, if a new writer cannot contact to JN3 for network
partition, and finish the recovery stage, and write a new txid 4 in epoch 2,
which value not equal to JN3's.
{code:java}
epcho 2
JN1 [1-3](finalized) [4-4](inprogress)
JN2 [1-3](finalized) [4-4](inprogress)
JN3 [1-4](inprogress)
Note that, in JN3 txid4's value not equal to other JN.
{code}
Now there is a read namenode to pull edits, and it contact to JN3 and JN2, it
got majority response. But it got logs of same length but different content.And
no more information to choose which log is right. If we choose JN3, we got meta
data corruption.
There is a test example patch [^example.patch] for running and debug.
For fix it i think we should add finalized state to
{{{}GetJournaledEditsResponseProto{}}}, so we can discard the fault log.
> [SBN Read]When fast path tail enabled, standby or observer namenode may read
> uncommitted data
> -
>
> Key: HDFS-16493
> URL: https://issues.apache.org/jira/browse/HDFS-16493
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: journal-node, namanode
>Reporter: liutongwei
>Priority: Critical
> Attachments: example.patch
>
>
> Although fast path tail use quorum read to pull edit log, it seem like can
> read uncommitted data in some corner case.
> Here is an example. Suppose we have three JN, their init state is:
>
> {code:java}
> epoch 1
> JN1 [1-3](in-progress)
> JN2 [1-3](in-progress)
> JN3 [1-4](in-progress)
> Note that, in epoch 1 txid 1-3 was committed, and txid 4 not.
> {code}
> When a failover occur, if a new writer cannot contact to JN3 for network
> partition, and finish the recovery stage, and write a new txid 4 in epoch 2,
> which value not equal to JN3's.
>
> {code:java}
> epcho 2
> JN1 [1-3](finalized) [4-4](inprogress)
> JN2 [1-3](finalized) [4-4](inprogress)
> JN3 [1-4](inprogress)
> Note that, in JN3 txid4's value not equal to other JN.
> {code}
>
> Now there is a read namenode to pull edits, and it contact to JN3 and JN2, it
> got majority response. But it got logs of same length but different
> content.And no more information to choose which log is right. If we choose
> JN3, we got meta data corruption.
> There is a test example patch [^example.patch] for running and debug.
> For fix it i think we should add finalized state to
> {{{}GetJournaledEditsResponseProto{}}}, so we can discard the fault log.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-16493) [SBN Read]When fast path tail enabled, standby or observer namenode may read uncommitted data
[
https://issues.apache.org/jira/browse/HDFS-16493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17501172#comment-17501172
]
liutongwei edited comment on HDFS-16493 at 3/4/22, 6:14 AM:
[~shv] [~xkrogen] , I have add a test case for the concern mentioned in
[here|https://issues.apache.org/jira/browse/HDFS-13150?focusedCommentId=17408479=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17408479].
was (Author: liutongwei):
[~shv] [~xkrogen] , I have add a test case for the concern mentioned in
https://issues.apache.org/jira/browse/HDFS-13150?focusedCommentId=17408479=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17408479.
> [SBN Read]When fast path tail enabled, standby or observer namenode may read
> uncommitted data
> -
>
> Key: HDFS-16493
> URL: https://issues.apache.org/jira/browse/HDFS-16493
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: journal-node, namanode
>Reporter: liutongwei
>Priority: Critical
> Attachments: example.patch
>
>
> Although fast path tail use quorum read to pull edit log, it seem like is can
> read uncommitted data in some corner case.
> Here is an example. Suppose we have three JN, their init state is:
>
> {code:java}
> epoch 1
> JN1 [1-3](in-progress)
> JN2 [1-3](in-progress)
> JN3 [1-4](in-progress)
> Note that, in epoch 1 txid 1-3 was committed, and txid 4 not.
> {code}
> When a failover occur, if a new writer cannot contact to JN3 for network
> partition, and finish the recovery stage, and write a new txid 4 in epoch 2,
> which value not equal to JN3's.
>
> {code:java}
> epcho 2
> JN1 [1-3](finalized) [4-4](inprogress)
> JN2 [1-3](finalized) [4-4](inprogress)
> JN3 [1-4](inprogress)
> Note that, in JN3 txid4's value not equal to other JN.
> {code}
>
> Now there is a read namenode to pull edits, and it contact to JN3 and JN2, it
> got majority response. But it got logs of same length but different
> content.And no more information to choose which log is right. If we choose
> JN3, we got meta data corruption.
> There is a test example patch [^example.patch] for running and debug.
> For fix it i think we should add finalized state to
> {{{}GetJournaledEditsResponseProto{}}}, so we can discard the fault log.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16493) [SBN Read]When fast path tail enabled, standby or observer namenode may read uncommitted data
[
https://issues.apache.org/jira/browse/HDFS-16493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17501172#comment-17501172
]
liutongwei commented on HDFS-16493:
---
[~shv] [~xkrogen] , I have add a test case for the concern mentioned in
https://issues.apache.org/jira/browse/HDFS-13150?focusedCommentId=17408479=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17408479.
> [SBN Read]When fast path tail enabled, standby or observer namenode may read
> uncommitted data
> -
>
> Key: HDFS-16493
> URL: https://issues.apache.org/jira/browse/HDFS-16493
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: journal-node, namanode
>Reporter: liutongwei
>Priority: Critical
> Attachments: example.patch
>
>
> Although fast path tail use quorum read to pull edit log, it seem like is can
> read uncommitted data in some corner case.
> Here is an example. Suppose we have three JN, their init state is:
>
> {code:java}
> epoch 1
> JN1 [1-3](in-progress)
> JN2 [1-3](in-progress)
> JN3 [1-4](in-progress)
> Note that, in epoch 1 txid 1-3 was committed, and txid 4 not.
> {code}
> When a failover occur, if a new writer cannot contact to JN3 for network
> partition, and finish the recovery stage, and write a new txid 4 in epoch 2,
> which value not equal to JN3's.
>
> {code:java}
> epcho 2
> JN1 [1-3](finalized) [4-4](inprogress)
> JN2 [1-3](finalized) [4-4](inprogress)
> JN3 [1-4](inprogress)
> Note that, in JN3 txid4's value not equal to other JN.
> {code}
>
> Now there is a read namenode to pull edits, and it contact to JN3 and JN2, it
> got majority response. But it got logs of same length but different
> content.And no more information to choose which log is right. If we choose
> JN3, we got meta data corruption.
> There is a test example patch [^example.patch] for running and debug.
> For fix it i think we should add finalized state to
> {{{}GetJournaledEditsResponseProto{}}}, so we can discard the fault log.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16493) [SBN Read]When fast path tail enabled, standby or observer namenode may read uncommitted data
liutongwei created HDFS-16493:
-
Summary: [SBN Read]When fast path tail enabled, standby or
observer namenode may read uncommitted data
Key: HDFS-16493
URL: https://issues.apache.org/jira/browse/HDFS-16493
Project: Hadoop HDFS
Issue Type: Bug
Components: journal-node, namanode
Reporter: liutongwei
Attachments: example.patch
Although fast path tail use quorum read to pull edit log, it seem like is can
read uncommitted data in some corner case.
Here is an example. Suppose we have three JN, their init state is:
{code:java}
epoch 1
JN1 [1-3](in-progress)
JN2 [1-3](in-progress)
JN3 [1-4](in-progress)
Note that, in epoch 1 txid 1-3 was committed, and txid 4 not.
{code}
When a failover occur, if a new writer cannot contact to JN3 for network
partition, and finish the recovery stage, and write a new txid 4 in epoch 2,
which value not equal to JN3's.
{code:java}
epcho 2
JN1 [1-3](finalized) [4-4](inprogress)
JN2 [1-3](finalized) [4-4](inprogress)
JN3 [1-4](inprogress)
Note that, in JN3 txid4's value not equal to other JN.
{code}
Now there is a read namenode to pull edits, and it contact to JN3 and JN2, it
got majority response. But it got logs of same length but different content.And
no more information to choose which log is right. If we choose JN3, we got meta
data corruption.
There is a test example patch [^example.patch] for running and debug.
For fix it i think we should add finalized state to
{{{}GetJournaledEditsResponseProto{}}}, so we can discard the fault log.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16477) [SPS]: Add metric PendingSPSPaths for getting the number of paths to be processed by SPS
[ https://issues.apache.org/jira/browse/HDFS-16477?focusedWorklogId=736485=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-736485 ] ASF GitHub Bot logged work on HDFS-16477: - Author: ASF GitHub Bot Created on: 04/Mar/22 05:42 Start Date: 04/Mar/22 05:42 Worklog Time Spent: 10m Work Description: tomscut edited a comment on pull request #4009: URL: https://github.com/apache/hadoop/pull/4009#issuecomment-1058847572 Hi @tasanuma @ayushtkn @virajith @virajjasani , could you please also take a look at this? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 736485) Time Spent: 1h (was: 50m) > [SPS]: Add metric PendingSPSPaths for getting the number of paths to be > processed by SPS > > > Key: HDFS-16477 > URL: https://issues.apache.org/jira/browse/HDFS-16477 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > Currently we have no idea how many paths are waiting to be processed when > using the SPS feature. We should add metric PendingSPSPaths for getting the > number of paths to be processed by SPS in NameNode. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16477) [SPS]: Add metric PendingSPSPaths for getting the number of paths to be processed by SPS
[ https://issues.apache.org/jira/browse/HDFS-16477?focusedWorklogId=736484=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-736484 ] ASF GitHub Bot logged work on HDFS-16477: - Author: ASF GitHub Bot Created on: 04/Mar/22 05:41 Start Date: 04/Mar/22 05:41 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #4009: URL: https://github.com/apache/hadoop/pull/4009#issuecomment-1058847572 Hi @tasanuma @ayushtkn @virajith , could you please also take a look at this? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 736484) Time Spent: 50m (was: 40m) > [SPS]: Add metric PendingSPSPaths for getting the number of paths to be > processed by SPS > > > Key: HDFS-16477 > URL: https://issues.apache.org/jira/browse/HDFS-16477 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Currently we have no idea how many paths are waiting to be processed when > using the SPS feature. We should add metric PendingSPSPaths for getting the > number of paths to be processed by SPS in NameNode. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16481) Provide support to set Http and Rpc ports in MiniJournalCluster
[ https://issues.apache.org/jira/browse/HDFS-16481?focusedWorklogId=736483=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-736483 ] ASF GitHub Bot logged work on HDFS-16481: - Author: ASF GitHub Bot Created on: 04/Mar/22 05:39 Start Date: 04/Mar/22 05:39 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #4028: URL: https://github.com/apache/hadoop/pull/4028#issuecomment-1058846529 `NetUtils. GetFreeSocketPort()` can greatly reduce the probability of port occupy. The latest version looks good to me. Thanks @ayushtkn and @virajjasani . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 736483) Time Spent: 5h 50m (was: 5h 40m) > Provide support to set Http and Rpc ports in MiniJournalCluster > --- > > Key: HDFS-16481 > URL: https://issues.apache.org/jira/browse/HDFS-16481 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Time Spent: 5h 50m > Remaining Estimate: 0h > > We should provide support for clients to set Http and Rpc ports of > JournalNodes in MiniJournalCluster. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16488) [SPS]: Expose metrics to JMX for external SPS
[ https://issues.apache.org/jira/browse/HDFS-16488?focusedWorklogId=736479=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-736479 ] ASF GitHub Bot logged work on HDFS-16488: - Author: ASF GitHub Bot Created on: 04/Mar/22 05:22 Start Date: 04/Mar/22 05:22 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #4035: URL: https://github.com/apache/hadoop/pull/4035#issuecomment-1058839156 Hi @tasanuma @jojochuang @ferhui , could you please take a look? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 736479) Time Spent: 40m (was: 0.5h) > [SPS]: Expose metrics to JMX for external SPS > - > > Key: HDFS-16488 > URL: https://issues.apache.org/jira/browse/HDFS-16488 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Attachments: image-2022-02-26-22-15-25-543.png > > Time Spent: 40m > Remaining Estimate: 0h > > Currently, external SPS has no monitoring metrics. We do not know how many > blocks are waiting to be processed, how many blocks are waiting to be > retried, and how many blocks have been migrated. > We can expose these metrics in JMX for easy collection and display by > monitoring systems. > !image-2022-02-26-22-15-25-543.png|width=631,height=170! > For example, in our cluster, we exposed these metrics to JMX, collected by > JMX-Exporter and combined with Prometheus, and finally display by Grafana. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16492) Add metrics and operation names for dataset lock
Yuanbo Liu created HDFS-16492: - Summary: Add metrics and operation names for dataset lock Key: HDFS-16492 URL: https://issues.apache.org/jira/browse/HDFS-16492 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Yuanbo Liu -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16481) Provide support to set Http and Rpc ports in MiniJournalCluster
[
https://issues.apache.org/jira/browse/HDFS-16481?focusedWorklogId=736425=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-736425
]
ASF GitHub Bot logged work on HDFS-16481:
-
Author: ASF GitHub Bot
Created on: 04/Mar/22 00:36
Start Date: 04/Mar/22 00:36
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#issuecomment-1058708707
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 35s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 3s | | trunk passed |
| +1 :green_heart: | compile | 24m 27s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 20m 48s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 48s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 13s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 16s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 17s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 5m 58s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 9s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 18s | | the patch passed |
| +1 :green_heart: | compile | 23m 39s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 23m 39s | | the patch passed |
| +1 :green_heart: | compile | 20m 49s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 49s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 45s | | root: The patch generated
0 new + 23 unchanged - 1 fixed = 23 total (was 24) |
| +1 :green_heart: | mvnsite | 3m 10s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 15s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 19s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 17s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 15s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 29s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 322m 47s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 58s | | The patch does not
generate ASF License warnings. |
| | | 560m 8s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4028 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 337880d17ba8 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18
11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 84e430e4cc436387f66b860d49a6d6d14b7378e3 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/8/testReport/ |
| Max. process+thread count | 1910 (vs. ulimit of 5500) |
| modules | C:
[jira] [Work logged] (HDFS-16476) Increase the number of metrics used to record PendingRecoveryBlocks
[
https://issues.apache.org/jira/browse/HDFS-16476?focusedWorklogId=736418=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-736418
]
ASF GitHub Bot logged work on HDFS-16476:
-
Author: ASF GitHub Bot
Created on: 03/Mar/22 23:53
Start Date: 03/Mar/22 23:53
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #4010:
URL: https://github.com/apache/hadoop/pull/4010#issuecomment-1058677665
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 40s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 53s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 48s | | trunk passed |
| +1 :green_heart: | compile | 22m 23s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 19m 40s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 20s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 23s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 4m 44s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 7m 24s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 23s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 53s | | the patch passed |
| +1 :green_heart: | compile | 21m 43s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | cc | 21m 43s |
[/results-compile-cc-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4010/2/artifact/out/results-compile-cc-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt)
| root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 33 new + 290 unchanged - 33
fixed = 323 total (was 323) |
| +1 :green_heart: | javac | 21m 43s | | the patch passed |
| +1 :green_heart: | compile | 19m 41s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | cc | 19m 41s |
[/results-compile-cc-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4010/2/artifact/out/results-compile-cc-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 10 new + 313
unchanged - 10 fixed = 323 total (was 323) |
| +1 :green_heart: | javac | 19m 41s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 40s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4010/2/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 2 new + 255 unchanged - 1 fixed = 257 total (was
256) |
| +1 :green_heart: | mvnsite | 4m 15s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 21s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 4m 41s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 7m 57s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 38s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 50s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 227m 3s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | unit | 21m 18s | | hadoop-hdfs-rbf in the patch
passed. |
[jira] [Work logged] (HDFS-16462) Make HDFS get tool cross platform
[ https://issues.apache.org/jira/browse/HDFS-16462?focusedWorklogId=736146=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-736146 ] ASF GitHub Bot logged work on HDFS-16462: - Author: ASF GitHub Bot Created on: 03/Mar/22 16:45 Start Date: 03/Mar/22 16:45 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #4003: URL: https://github.com/apache/hadoop/pull/4003#issuecomment-1058248724 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | docker | 0m 30s | | Docker failed to build yetus/hadoop:c42441a1ec4. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/4003 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4003/10/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 736146) Time Spent: 1h 20m (was: 1h 10m) > Make HDFS get tool cross platform > - > > Key: HDFS-16462 > URL: https://issues.apache.org/jira/browse/HDFS-16462 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, libhdfs++, tools >Affects Versions: 3.4.0 > Environment: Centos 7, Centos 8, Debian 10, Ubuntu Focal >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: libhdfscpp, pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > The source files for *hdfs_get* uses *getopt* for parsing the command line > arguments. getopt is available only on Linux and thus, isn't cross platform. > We need to replace getopt with *boost::program_options* to make these tools > cross platform. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16462) Make HDFS get tool cross platform
[
https://issues.apache.org/jira/browse/HDFS-16462?focusedWorklogId=736141=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-736141
]
ASF GitHub Bot logged work on HDFS-16462:
-
Author: ASF GitHub Bot
Created on: 03/Mar/22 16:43
Start Date: 03/Mar/22 16:43
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #4003:
URL: https://github.com/apache/hadoop/pull/4003#issuecomment-1058246686
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 51m 38s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 6s | | trunk passed |
| +1 :green_heart: | compile | 4m 19s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 31s | | trunk passed |
| +1 :green_heart: | shadedclient | 68m 48s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 16s | | the patch passed |
| +1 :green_heart: | compile | 3m 49s | | the patch passed |
| +1 :green_heart: | cc | 3m 49s | | the patch passed |
| +1 :green_heart: | golang | 3m 49s | | the patch passed |
| +1 :green_heart: | javac | 3m 49s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 18s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 20s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 117m 7s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | | The patch does not
generate ASF License warnings. |
| | | 270m 21s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4003/10/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4003 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell golang |
| uname | Linux 480d780e8b9d 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29
06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a5437123151c595b6f4095b04ea9b115881ceb2f |
| Default Java | Red Hat, Inc.-1.8.0_322-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4003/10/testReport/ |
| Max. process+thread count | 523 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4003/10/console |
| versions | git=2.9.5 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 736141)
Time Spent: 1h 10m (was: 1h)
> Make HDFS get tool cross platform
> -
>
> Key: HDFS-16462
> URL: https://issues.apache.org/jira/browse/HDFS-16462
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client, libhdfs++, tools
>Affects Versions: 3.4.0
> Environment: Centos 7, Centos 8, Debian 10, Ubuntu Focal
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: libhdfscpp, pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The source files for *hdfs_get* uses *getopt* for parsing the command line
> arguments. getopt is available only on Linux and thus, isn't cross platform.
> We need to replace getopt with
[jira] [Work logged] (HDFS-16481) Provide support to set Http and Rpc ports in MiniJournalCluster
[
https://issues.apache.org/jira/browse/HDFS-16481?focusedWorklogId=736129=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-736129
]
ASF GitHub Bot logged work on HDFS-16481:
-
Author: ASF GitHub Bot
Created on: 03/Mar/22 16:23
Start Date: 03/Mar/22 16:23
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#issuecomment-1058227393
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 58s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 36s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 8s | | trunk passed |
| +1 :green_heart: | compile | 24m 37s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 21m 31s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 54s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 16s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 23s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 14s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 10s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 28s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 25s | | the patch passed |
| +1 :green_heart: | compile | 25m 42s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 25m 42s | | the patch passed |
| +1 :green_heart: | compile | 22m 5s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 22m 5s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 49s | | root: The patch generated
0 new + 23 unchanged - 1 fixed = 23 total (was 24) |
| +1 :green_heart: | mvnsite | 3m 22s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 16s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 25s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 25s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 25s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 36s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 370m 20s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 24s | | The patch does not
generate ASF License warnings. |
| | | 613m 57s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4028 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 0691f6cfd6d2 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29
06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 68e7e2eb60f236b7b59a383ab2b8da06e724648d |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4028/7/testReport/ |
| Max. process+thread count | 3046 (vs. ulimit of 5500) |
| modules | C:
[jira] [Work logged] (HDFS-16481) Provide support to set Http and Rpc ports in MiniJournalCluster
[
https://issues.apache.org/jira/browse/HDFS-16481?focusedWorklogId=736062=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-736062
]
ASF GitHub Bot logged work on HDFS-16481:
-
Author: ASF GitHub Bot
Created on: 03/Mar/22 15:15
Start Date: 03/Mar/22 15:15
Worklog Time Spent: 10m
Work Description: virajjasani commented on a change in pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#discussion_r818756871
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestMiniJournalCluster.java
##
@@ -52,4 +60,92 @@ public void testStartStop() throws IOException {
c.shutdown();
}
}
+
+ @Test
+ public void testStartStopWithPorts() throws Exception {
+Configuration conf = new Configuration();
+
+LambdaTestUtils.intercept(
+IllegalArgumentException.class,
+"Num of http ports (1) should match num of JournalNodes (3)",
+"MiniJournalCluster port validation failed",
+() -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(8481).build();
+});
+
+LambdaTestUtils.intercept(
+IllegalArgumentException.class,
+"Num of rpc ports (2) should match num of JournalNodes (3)",
+"MiniJournalCluster port validation failed",
+() -> {
+ new MiniJournalCluster.Builder(conf).setRpcPorts(8481, 8482).build();
+});
+
+LambdaTestUtils.intercept(
+IllegalArgumentException.class,
+"Num of rpc ports (1) should match num of JournalNodes (3)",
+"MiniJournalCluster port validation failed",
+() -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000,
1).setRpcPorts(8481)
+ .build();
+});
+
+LambdaTestUtils.intercept(
+IllegalArgumentException.class,
+"Num of http ports (4) should match num of JournalNodes (3)",
+"MiniJournalCluster port validation failed",
+() -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 1000,
2000)
+ .setRpcPorts(8481, 8482, 8483).build();
+});
+
+final Set httpAndRpcPorts = NetUtils.getFreeSocketPorts(6);
+LOG.info("Free socket ports: {}", httpAndRpcPorts);
+
+final int[] httpPorts = new int[3];
+final int[] rpcPorts = new int[3];
+int httpPortIdx = 0;
+int rpcPortIdx = 0;
+for (Integer httpAndRpcPort : httpAndRpcPorts) {
+ if (httpPortIdx < 3) {
+httpPorts[httpPortIdx++] = httpAndRpcPort;
+ } else {
+rpcPorts[rpcPortIdx++] = httpAndRpcPort;
+ }
+}
+
+LOG.info("Http ports selected: {}", httpPorts);
+LOG.info("Rpc ports selected: {}", rpcPorts);
+
+for (int i = 0; i < 3; i++) {
+ assertNotEquals(0, rpcPorts[i]);
+ assertNotEquals(0, httpPorts[i]);
+}
Review comment:
Done, Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 736062)
Time Spent: 5h 20m (was: 5h 10m)
> Provide support to set Http and Rpc ports in MiniJournalCluster
> ---
>
> Key: HDFS-16481
> URL: https://issues.apache.org/jira/browse/HDFS-16481
> Project: Hadoop HDFS
> Issue Type: Task
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> We should provide support for clients to set Http and Rpc ports of
> JournalNodes in MiniJournalCluster.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16481) Provide support to set Http and Rpc ports in MiniJournalCluster
[
https://issues.apache.org/jira/browse/HDFS-16481?focusedWorklogId=735894=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-735894
]
ASF GitHub Bot logged work on HDFS-16481:
-
Author: ASF GitHub Bot
Created on: 03/Mar/22 12:01
Start Date: 03/Mar/22 12:01
Worklog Time Spent: 10m
Work Description: ayushtkn commented on a change in pull request #4028:
URL: https://github.com/apache/hadoop/pull/4028#discussion_r818587730
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestMiniJournalCluster.java
##
@@ -52,4 +60,92 @@ public void testStartStop() throws IOException {
c.shutdown();
}
}
+
+ @Test
+ public void testStartStopWithPorts() throws Exception {
+Configuration conf = new Configuration();
+
+LambdaTestUtils.intercept(
+IllegalArgumentException.class,
+"Num of http ports (1) should match num of JournalNodes (3)",
+"MiniJournalCluster port validation failed",
+() -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(8481).build();
+});
+
+LambdaTestUtils.intercept(
+IllegalArgumentException.class,
+"Num of rpc ports (2) should match num of JournalNodes (3)",
+"MiniJournalCluster port validation failed",
+() -> {
+ new MiniJournalCluster.Builder(conf).setRpcPorts(8481, 8482).build();
+});
+
+LambdaTestUtils.intercept(
+IllegalArgumentException.class,
+"Num of rpc ports (1) should match num of JournalNodes (3)",
+"MiniJournalCluster port validation failed",
+() -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000,
1).setRpcPorts(8481)
+ .build();
+});
+
+LambdaTestUtils.intercept(
+IllegalArgumentException.class,
+"Num of http ports (4) should match num of JournalNodes (3)",
+"MiniJournalCluster port validation failed",
+() -> {
+ new MiniJournalCluster.Builder(conf).setHttpPorts(800, 9000, 1000,
2000)
+ .setRpcPorts(8481, 8482, 8483).build();
+});
+
+final Set httpAndRpcPorts = NetUtils.getFreeSocketPorts(6);
+LOG.info("Free socket ports: {}", httpAndRpcPorts);
+
+final int[] httpPorts = new int[3];
+final int[] rpcPorts = new int[3];
+int httpPortIdx = 0;
+int rpcPortIdx = 0;
+for (Integer httpAndRpcPort : httpAndRpcPorts) {
+ if (httpPortIdx < 3) {
+httpPorts[httpPortIdx++] = httpAndRpcPort;
+ } else {
+rpcPorts[rpcPortIdx++] = httpAndRpcPort;
+ }
+}
+
+LOG.info("Http ports selected: {}", httpPorts);
+LOG.info("Rpc ports selected: {}", rpcPorts);
+
+for (int i = 0; i < 3; i++) {
+ assertNotEquals(0, rpcPorts[i]);
+ assertNotEquals(0, httpPorts[i]);
+}
Review comment:
you could have asserted directly on httpAndRpcPorts. :-)
```
assertFalse(httpAndRpcPorts.contains(0));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 735894)
Time Spent: 5h 10m (was: 5h)
> Provide support to set Http and Rpc ports in MiniJournalCluster
> ---
>
> Key: HDFS-16481
> URL: https://issues.apache.org/jira/browse/HDFS-16481
> Project: Hadoop HDFS
> Issue Type: Task
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> We should provide support for clients to set Http and Rpc ports of
> JournalNodes in MiniJournalCluster.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16462) Make HDFS get tool cross platform
[
https://issues.apache.org/jira/browse/HDFS-16462?focusedWorklogId=735865=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-735865
]
ASF GitHub Bot logged work on HDFS-16462:
-
Author: ASF GitHub Bot
Created on: 03/Mar/22 11:06
Start Date: 03/Mar/22 11:06
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #4003:
URL: https://github.com/apache/hadoop/pull/4003#issuecomment-1057932045
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 39m 16s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 4m 59s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4003/9/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| +1 :green_heart: | compile | 3m 36s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 21s | | trunk passed |
| +1 :green_heart: | shadedclient | 30m 1s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 16s | | the patch passed |
| +1 :green_heart: | compile | 3m 13s | | the patch passed |
| +1 :green_heart: | cc | 3m 13s | | the patch passed |
| +1 :green_heart: | golang | 3m 13s | | the patch passed |
| +1 :green_heart: | javac | 3m 13s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 18s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 35s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 85m 56s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 180m 36s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4003/9/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4003 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell golang |
| uname | Linux 9f07cb6071e9 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a5437123151c595b6f4095b04ea9b115881ceb2f |
| Default Java | Red Hat, Inc.-1.8.0_322-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4003/9/testReport/ |
| Max. process+thread count | 722 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4003/9/console |
| versions | git=2.9.5 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 735865)
Time Spent: 1h (was: 50m)
> Make HDFS get tool cross platform
> -
>
> Key: HDFS-16462
> URL: https://issues.apache.org/jira/browse/HDFS-16462
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client, libhdfs++, tools
>Affects Versions: 3.4.0
> Environment: Centos 7, Centos 8, Debian 10, Ubuntu Focal
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: libhdfscpp, pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The source files for *hdfs_get* uses *getopt* for parsing the command line
