[jira] [Created] (HDFS-17130) Blocks on IN_MAINTENANCE DNs should be sorted properly in LocatedBlocks.

[huangzhaobo99 (Jira)]

huangzhaobo99 created HDFS-17130:


 Summary: Blocks on IN_MAINTENANCE DNs should be sorted properly in 
LocatedBlocks.
 Key: HDFS-17130
 URL: https://issues.apache.org/jira/browse/HDFS-17130
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: dfs
Reporter: huangzhaobo99


This PR supplements the in_maintenance status of DN when sorting LocatedBlocks 
in ClientProtocol#getBlockLocations.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-17117) Print reconstructionQueuesInitProgress periodically when BlockManager processMisReplicatesAsync.

[Xiaoqiao He (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-17117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaoqiao He resolved HDFS-17117.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Print reconstructionQueuesInitProgress periodically when BlockManager 
> processMisReplicatesAsync.
> 
>
> Key: HDFS-17117
> URL: https://issues.apache.org/jira/browse/HDFS-17117
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> BlockManager#processMisReplicatesAsync can periodically print 
> reconstructionQueuesInitProgress,  so that the admin can get the progress of 
> the replication queues initialisation.
> Ready to add logs and metrics.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-17016) Cleanup method calls to static Assert and Assume methods.

[WangYuanben (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-17016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

WangYuanben resolved HDFS-17016.

Resolution: Not A Problem

> Cleanup method calls to static Assert and Assume methods.
> -
>
> Key: HDFS-17016
> URL: https://issues.apache.org/jira/browse/HDFS-17016
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Reporter: WangYuanben
>Assignee: WangYuanben
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Cleanup method calls to static Assert and Assume methods.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-17129) mis-order of ibr and fbr on datanode

[liuguanghua (Jira)]

liuguanghua created HDFS-17129:
--

 Summary: mis-order of ibr and fbr on datanode 
 Key: HDFS-17129
 URL: https://issues.apache.org/jira/browse/HDFS-17129
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.4.0
 Environment: hdfs3.4.0
Reporter: liuguanghua


HDFS-16016 , provide new thread to handler IBR. That is a greate improvement. 
But it maybe casue the mis-order of ibr and fbr



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

[Apache Jenkins Server]

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1100/

No changes


ERROR: File 'out/email-report.txt' does not exist

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

HADOOP-18207 hadoop-logging module about to land

[Wei-Chiu Chuang]

Hi,

I am preparing to resolve HADOOP-18207
 (
https://github.com/apache/hadoop/pull/5717).

This change affects all modules. With this change, it will eliminate almost
all the direct log4j usage.

As always, landing such a big piece is tricky. I am sorry for the mishaps
last time and am doing more due diligence to make it a smoother transition.
I am triggering one last precommit check. Once the change is merged, Viraj
and I will pay attention to any potential problems.

Weichiu

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

[Apache Jenkins Server]

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1299/

No changes




-1 overall


The following subsystems voted -1:
blanks hadolint pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 

Failed junit tests :

   hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 
   hadoop.mapreduce.v2.TestUberAM 
   hadoop.mapreduce.v2.TestMRJobsWithProfiler 
   hadoop.mapreduce.v2.TestMRJobs 
  

   cc:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1299/artifact/out/results-compile-cc-root.txt
 [96K]

   javac:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1299/artifact/out/results-compile-javac-root.txt
 [12K]

   blanks:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1299/artifact/out/blanks-eol.txt
 [15M]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1299/artifact/out/blanks-tabs.txt
 [2.0M]

   checkstyle:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1299/artifact/out/results-checkstyle-root.txt
 [13M]

   hadolint:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1299/artifact/out/results-hadolint.txt
 [20K]

   pathlen:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1299/artifact/out/results-pathlen.txt
 [16K]

   pylint:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1299/artifact/out/results-pylint.txt
 [20K]

   shellcheck:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1299/artifact/out/results-shellcheck.txt
 [24K]

   xml:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1299/artifact/out/xml.txt
 [24K]

   javadoc:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1299/artifact/out/results-javadoc-javadoc-root.txt
 [244K]

   unit:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1299/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
 [24K]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1299/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
 [72K]

Powered by Apache Yetus 0.14.0-SNAPSHOT   https://yetus.apache.org

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-17128) SQLDelegationTokenSecretManager should use version of tokens updated by other routers

[Hector Sandoval Chaverri (Jira)]

Hector Sandoval Chaverri created HDFS-17128:
---

 Summary: SQLDelegationTokenSecretManager should use version of 
tokens updated by other routers
 Key: HDFS-17128
 URL: https://issues.apache.org/jira/browse/HDFS-17128
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: rbf
Reporter: Hector Sandoval Chaverri


The SQLDelegationTokenSecretManager keeps tokens that it has interacted with in 
a memory cache. This prevents routers from connecting to the SQL server for 
each token operation.

We've noticed issues with some tokens being loaded in one router's cache and 
later renewed on a different one. If clients try to use the token in the 
outdated router, it will throw an "Auth failed" error when the cached token's 
expiration has passed.

This can also affect cancelation scenarios since a token can be removed from 
one router's cache and still exist in another one.

A possible solution is already implemented on the 
ZKDelegationTokenSecretManager, which consists of having an executor refreshing 
each router's cache on a periodic basis. We should evaluate whether this will 
work with the volume of tokens expected to be handled by the 
SQLDelegationTokenSecretManager.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-17127) GCP is not mentioned as a load balancer option

[C.J. Collier (Jira)]

C.J. Collier created HDFS-17127:
---

 Summary: GCP is not mentioned as a load balancer option
 Key: HDFS-17127
 URL: https://issues.apache.org/jira/browse/HDFS-17127
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: documentation
Affects Versions: 3.3.6
Reporter: C.J. Collier
 Attachments: HDFS-17127.patch

In the HDFSHighAvailabilityWithQJM introduction documentation, Azure and AWS 
load balancer options are mentioned but not GCP.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-17126) FsDataSetImpl#checkAndUpdate should delete duplicated block meta file.

[farmmamba (Jira)]

farmmamba created HDFS-17126:


 Summary: FsDataSetImpl#checkAndUpdate should delete duplicated 
block meta file.
 Key: HDFS-17126
 URL: https://issues.apache.org/jira/browse/HDFS-17126
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: datanode
Affects Versions: 3.4.0
Reporter: farmmamba


Think about below case :

We have one datanode called dn1,   it has two storages call ds1 and ds2 
respectively.

Suppose we have blk_123 and blk_123_1001.meta in ds1 and blk_123_1001.meta in 
ds2.

The current logic will not handle the file blk_123_1001.meta in ds2 and only 
prints logs to tell us DirectoryScanner scan missing block files.

I think we should do something here.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-17115) HttpFS Add Support getErasureCodeCodecs API

[Ayush Saxena (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-17115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ayush Saxena resolved HDFS-17115.
-
Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> HttpFS Add Support getErasureCodeCodecs API
> ---
>
> Key: HDFS-17115
> URL: https://issues.apache.org/jira/browse/HDFS-17115
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: httpfs
>Affects Versions: 3.4.0
>Reporter: Hualong Zhang
>Assignee: Hualong Zhang
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> We should ensure that *WebHDFS* remains synchronized with {*}HttpFS{*}, as 
> the former has already implemented the *getErasureCodeCodecs* interface.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-17111) RBF: Optimize msync to only call nameservices that have observer reads enabled.

[Ayush Saxena (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-17111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ayush Saxena resolved HDFS-17111.
-
Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> RBF: Optimize msync to only call nameservices that have observer reads 
> enabled.
> ---
>
> Key: HDFS-17111
> URL: https://issues.apache.org/jira/browse/HDFS-17111
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Right now when a client MSYNCs to the router, the call is fanned out to all 
> nameservices. We only need to proxy the msync to nameservices that have 
> observer reads configured.
> We can do this either by adding a new config for the admin to specify which 
> nameservices have CRS configured, or we can try to automatically detect these.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org