Re: [DISCUSS] Hadoop 3.3.6 release planning

2023-05-04 Thread Xiaoqiao He 
Thanks Wei-Chiu for driving this release.
Cherry-pick YARN-11482 to branch-3.3 and mark 3.3.6 as the fixed version.

so far only 8 jiras were resolved in the branch-3.3 line.


If we should consider both 3.3.6 and 3.3.9 (which is from release-3.3.5
discuss)[1] for this release line?
I try to query with `project in (HDFS, YARN, HADOOP, MAPREDUCE) AND
fixVersion in (3.3.6, 3.3.9)`[2],
there are more than hundred jiras now.

Best Regards,
- He Xiaoqiao

[1] https://lists.apache.org/thread/kln96frt2tcg93x6ht99yck9m7r9qwxp
[2]
https://issues.apache.org/jira/browse/YARN-11482?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20fixVersion%20in%20(3.3.6%2C%203.3.9)


On Fri, May 5, 2023 at 1:19 AM Wei-Chiu Chuang  wrote:

> Hi community,
>
> I'd like to kick off the discussion around Hadoop 3.3.6 release plan.
>
> I'm being selfish but my intent for 3.3.6 is to have the new APIs in
> HADOOP-18671  added so
> we can have HBase to adopt this new API. Other than that, perhaps
> thirdparty dependency updates.
>
> If you have open items to be added in the coming weeks, please add 3.3.6 to
> the target release version. Right now I am only seeing three open jiras
> targeting 3.3.6.
>
> I imagine this is going to be a small release as 3.3.5 (hat tip to Steve)
> was only made two months back, and so far only 8 jiras were resolved in the
> branch-3.3 line.
>
> Best,
> Weichiu
>

[jira] [Created] (HDFS-16999) Fix wrong use of processFirstBlockReport()

2023-05-04 Thread Shuyan Zhang (Jira) 
Shuyan Zhang created HDFS-16999:
---

 Summary: Fix wrong use of processFirstBlockReport()
 Key: HDFS-16999
 URL: https://issues.apache.org/jira/browse/HDFS-16999
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Shuyan Zhang


`processFirstBlockReport()` is used to process first block report from 
datanode. It does not calculating `toRemove` list because it believes that 
there is no metadata about the datanode in the namenode. However, If a datanode 
is re registered after restarting, its `blockReportCount` will be updated to 0. 
That is to say, the first block report after a datanode restarts will be 
processed by `processFirstBlockReport()`.  This is unreasonable because the 
metadata of the datanode already exists in namenode at this time, and if 
redundant replica metadata is not removed in time, the blocks with insufficient 
replicas cannot be reconstruct in time, which increases the risk of missing 
block. In summary, `processFirstBlockReport()` should only be used when the 
namenode restarts, not when the datanode restarts. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

2023-05-04 Thread Apache Jenkins Server 
For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1017/

No changes


ERROR: File 'out/email-report.txt' does not exist

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

2023-05-04 Thread Apache Jenkins Server 
For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1216/

[May 3, 2023, 10:05:55 AM] (github) HADOOP-18671. Add recoverLease(), 
setSafeMode(), isFileClosed() as interfaces to hadoop-common (#5553)
[May 3, 2023, 4:19:54 PM] (github) YARN-9049. Add application submit data to 
state store. (#5606)
[May 3, 2023, 4:21:56 PM] (github) YARN-10144. Federation: Add missing 
FederationClientInterceptor APIs. (#5587)
[May 3, 2023, 5:14:54 PM] (github) HADOOP-18134. Setup Jenkins nightly CI for 
Windows 10 (#5062)




-1 overall


The following subsystems voted -1:
blanks hadolint pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 

Failed junit tests :

   hadoop.hdfs.server.datanode.TestDirectoryScanner 
   hadoop.mapreduce.v2.TestUberAM 
   hadoop.mapreduce.v2.TestMRJobsWithProfiler 
   hadoop.mapreduce.v2.TestMRJobs 
  

   cc:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1216/artifact/out/results-compile-cc-root.txt
 [96K]

   javac:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1216/artifact/out/results-compile-javac-root.txt
 [12K]

   blanks:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1216/artifact/out/blanks-eol.txt
 [14M]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1216/artifact/out/blanks-tabs.txt
 [2.0M]

   checkstyle:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1216/artifact/out/results-checkstyle-root.txt
 [13M]

   hadolint:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1216/artifact/out/results-hadolint.txt
 [20K]

   pathlen:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1216/artifact/out/results-pathlen.txt
 [16K]

   pylint:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1216/artifact/out/results-pylint.txt
 [20K]

   shellcheck:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1216/artifact/out/results-shellcheck.txt
 [24K]

   xml:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1216/artifact/out/xml.txt
 [24K]

   javadoc:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1216/artifact/out/results-javadoc-javadoc-root.txt
 [244K]

   unit:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1216/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 [228K]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1216/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
 [72K]

Powered by Apache Yetus 0.14.0-SNAPSHOT   https://yetus.apache.org

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[DISCUSS] Hadoop 3.3.6 release planning

2023-05-04 Thread Wei-Chiu Chuang 
Hi community,

I'd like to kick off the discussion around Hadoop 3.3.6 release plan.

I'm being selfish but my intent for 3.3.6 is to have the new APIs in
HADOOP-18671  added so
we can have HBase to adopt this new API. Other than that, perhaps
thirdparty dependency updates.

If you have open items to be added in the coming weeks, please add 3.3.6 to
the target release version. Right now I am only seeing three open jiras
targeting 3.3.6.

I imagine this is going to be a small release as 3.3.5 (hat tip to Steve)
was only made two months back, and so far only 8 jiras were resolved in the
branch-3.3 line.

Best,
Weichiu

[jira] [Resolved] (HDFS-16998) RBF: Add ops metrics for getSlowDatanodeReport in RouterClientActivity

2023-05-04 Thread Jira 


 [ 
https://issues.apache.org/jira/browse/HDFS-16998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-16998.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> RBF: Add ops metrics for getSlowDatanodeReport in RouterClientActivity
> --
>
> Key: HDFS-16998
> URL: https://issues.apache.org/jira/browse/HDFS-16998
> Project: Hadoop HDFS
>  Issue Type: Task
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org