[jira] [Created] (HDFS-15469) Dynamically configure the size of PacketReceiver#MAX_PACKET_SIZE

[jianghua zhu (Jira)]

jianghua zhu created HDFS-15469:
---

 Summary: Dynamically configure the size of 
PacketReceiver#MAX_PACKET_SIZE
 Key: HDFS-15469
 URL: https://issues.apache.org/jira/browse/HDFS-15469
 Project: Hadoop HDFS
  Issue Type: Improvement
Affects Versions: 3.0.3
Reporter: jianghua zhu


Now the value of PacketReceiver#MAX_PACKET_SIZE is fixed and the size is 16M. 
This value should be configurable to facilitate better performance in different 
environments. For example, when the network environment is poor, or the machine 
quality is not good, and the hard disk quality is not good, this value should 
be set below 16M, such as 8M, which will be more conducive to the stability of 
the cluster.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[RESULT][VOTE] Rlease Apache Hadoop-3.3.0

[Brahma Reddy Battula]

Hi All,

With 8 binding and 11 non-binding +1s and no -1s the vote for Apache
hadoop-3.3.0 Release
passes.

Thank you everybody for contributing to the release, testing, and voting.

Special thanks whoever verified the ARM Binary as this is the first release
to support the ARM in hadoop.


Binding +1s

=
Akira Ajisaka
Vinayakumar B
Inigo Goiri
Surendra Singh Lilhore
Masatake Iwasaki
Rakesh Radhakrishnan
Eric Badger
Brahma Reddy Battula

Non-binding +1s

=
Zhenyu Zheng
Sheng Liu
Yikun Jiang
Tianhua huang
Ayush Saxena
Hemanth Boyina
Bilwa S T
Takanobu Asanuma
Xiaoqiao He
CR Hota
Gergely Pollak

I'm going to work on staging the release.


The voting thread is:

 https://s.apache.org/hadoop-3.3.0-Release-vote-thread



--Brahma Reddy Battula

Re: [VOTE] Release Apache Hadoop 3.1.4 (RC3)

[Masatake Iwasaki]

Hi Gabor Bota,

> The RC is available at: 
http://people.apache.org/~gabota/hadoop-3.1.4-RC3/


I could not find .sha512 for src and bin tarballs.
Could you upload the files too?

I'm +1 (binding), pending on them.

* verified the signature of the source tarball.
* built from source tarball with native profile enabled on CentOS 7 and 
OpenJDK 8.

* built documentation and skimmed the contents.
* ran example jobs on 3 nodes docker cluster with NN-HA and RM-HA enblaed.
* launched pseudo-distributed cluster with Kerberos and SSL enabled, ran 
basic EZ operation, ran example MR jobs.


Thanks,
Masatake Iwasaki

On 2020/07/13 19:36, Gabor Bota wrote:

Hi folks,

I have put together a release candidate (RC3) for Hadoop 3.1.4.

*
The RC includes in addition to the previous ones:
* fix of YARN-10347. Fix double locking in
CapacityScheduler#reinitialize in branch-3.1
(https://issues.apache.org/jira/browse/YARN-10347)
* the revert of HDFS-14941, as it caused
HDFS-15421. IBR leak causes standby NN to be stuck in safe mode.
(https://issues.apache.org/jira/browse/HDFS-15421)
* HDFS-15323, as requested.
(https://issues.apache.org/jira/browse/HDFS-15323)
*

The RC is available at: http://people.apache.org/~gabota/hadoop-3.1.4-RC3/
The RC tag in git is here:
https://github.com/apache/hadoop/releases/tag/release-3.1.4-RC3
The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1274/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
and http://keys.gnupg.net/pks/lookup?op=get&search=0xB86249D83539B38C

Please try the release and vote. The vote will run for 7 weekdays,
until July 22. 2020. 23:00 CET.


Thanks,
Gabor

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org



-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: branch2.10+JDK7 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86/747/

[Jul 13, 2020 6:45:09 AM] (hexiaoqiao) HDFS-14498 LeaseManager can loop forever 
on the file for which create
[Jul 13, 2020 3:57:11 PM] (hexiaoqiao) Revert "HDFS-14498 LeaseManager can loop 
forever on the file for which
[Jul 13, 2020 9:51:32 PM] (ericp) YARN-10297.




-1 overall


The following subsystems voted -1:
asflicense findbugs hadolint jshint pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/empty-configuration.xml
 
   hadoop-tools/hadoop-azure/src/config/checkstyle-suppressions.xml 
   hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/public/crossdomain.xml 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml
 

findbugs :

   module:hadoop-yarn-project/hadoop-yarn 
   Useless object stored in variable removedNullContainers of method 
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List)
 At NodeStatusUpdaterImpl.java:removedNullContainers of method 
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List)
 At NodeStatusUpdaterImpl.java:[line 664] 
   
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeVeryOldStoppedContainersFromCache()
 makes inefficient use of keySet iterator instead of entrySet iterator At 
NodeStatusUpdaterImpl.java:keySet iterator instead of entrySet iterator At 
NodeStatusUpdaterImpl.java:[line 741] 
   
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.createStatus()
 makes inefficient use of keySet iterator instead of entrySet iterator At 
ContainerLocalizer.java:keySet iterator instead of entrySet iterator At 
ContainerLocalizer.java:[line 359] 
   
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainerMetrics.usageMetrics
 is a mutable collection which should be package protected At 
ContainerMetrics.java:which should be package protected At 
ContainerMetrics.java:[line 134] 
   Boxed value is unboxed and then immediately reboxed in 
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result,
 byte[], byte[], KeyConverter, ValueConverter, boolean) At 
ColumnRWHelper.java:then immediately reboxed in 
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result,
 byte[], byte[], KeyConverter, ValueConverter, boolean) At 
ColumnRWHelper.java:[line 335] 
   
org.apache.hadoop.yarn.state.StateMachineFactory.generateStateGraph(String) 
makes inefficient use of keySet iterator instead of entrySet iterator At 
StateMachineFactory.java:keySet iterator instead of entrySet iterator At 
StateMachineFactory.java:[line 505] 

findbugs :

   module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common 
   
org.apache.hadoop.yarn.state.StateMachineFactory.generateStateGraph(String) 
makes inefficient use of keySet iterator instead of entrySet iterator At 
StateMachineFactory.java:keySet iterator instead of entrySet iterator At 
StateMachineFactory.java:[line 505] 

findbugs :

   module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server 
   Useless object stored in variable removedNullContainers of method 
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List)
 At NodeStatusUpdaterImpl.java:removedNullContainers of method 
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List)
 At NodeStatusUpdaterImpl.java:[line 664] 
   
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeVeryOldStoppedContainersFromCache()
 makes inefficient use of keySet iterator instead of entrySet iterator At 
NodeStatusUpdaterImpl.java:keySet iterator instead of entrySet iterator At 
NodeStatusUpdaterImpl.java:[line 741] 
   
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.createStatus()
 makes inefficient use of keySet iterator instead of entrySet iterator At 
ContainerLocalizer.java:keySet iterator instead of entrySet iterator At 
ContainerLocalizer.java:[line 359] 
   
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainerMetrics.usageMetrics
 is a mutable collection which should be package protected At 
ContainerMetrics.java:which should be package protected At 
ContainerMetrics.java:[line 134] 
   Boxed value is unboxed and then immediately reboxed in 
org.apach

[jira] [Resolved] (HDFS-15371) Nonstandard characters exist in NameNode.java

[Steve Loughran (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-15371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HDFS-15371.
---
Fix Version/s: 3.4.0
   Resolution: Fixed

fixed in trunk, thanks

> Nonstandard characters exist in NameNode.java
> -
>
> Key: HDFS-15371
> URL: https://issues.apache.org/jira/browse/HDFS-15371
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Affects Versions: 3.1.0
>Reporter: jianghua zhu
>Assignee: Zhao Yi Ming
>Priority: Minor
> Fix For: 3.4.0
>
>
> In NameNode.Java, DFS_HA_ZKFC_PORT_KEY has non-standard characters behind it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-15470) Added more unit tests to validate rename behaviour across snapshots

[Shashikant Banerjee (Jira)]

Shashikant Banerjee created HDFS-15470:
--

 Summary: Added more unit tests to validate rename behaviour across 
snapshots
 Key: HDFS-15470
 URL: https://issues.apache.org/jira/browse/HDFS-15470
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: snapshots
Reporter: Shashikant Banerjee
Assignee: Shashikant Banerjee
 Fix For: 3.0.4


HDFS-15313 fixes a critical issue which will avoid deletion of data in active 
fs with a sequence of snapshot deletes. The idea is to add more tests to verify 
the behaviour.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-15471) TestHDFSContractMultipartUploader fails on trunk

[Ahmed Hussein (Jira)]

Ahmed Hussein created HDFS-15471:


 Summary: TestHDFSContractMultipartUploader fails on trunk
 Key: HDFS-15471
 URL: https://issues.apache.org/jira/browse/HDFS-15471
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ahmed Hussein


{{TestHDFSContractMultipartUploader}} fails on trunk with 
{{IllegalArgumentException}}


{code:bash}
[ERROR] 
testConcurrentUploads(org.apache.hadoop.fs.contract.hdfs.TestHDFSContractMultipartUploader)
  Time elapsed: 0.127 s  <<< ERROR!
java.lang.IllegalArgumentException
at 
com.google.common.base.Preconditions.checkArgument(Preconditions.java:127)
at 
org.apache.hadoop.test.LambdaTestUtils$ProportionalRetryInterval.(LambdaTestUtils.java:907)
at 
org.apache.hadoop.fs.contract.AbstractContractMultipartUploaderTest.testConcurrentUploads(AbstractContractMultipartUploaderTest.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
{code}






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

hdfs pure c++ sdk support write or not

[kevin liu]

I am trying to use hdfs pure c++ sdk，refer to  
https://issues.apache.org/jira/browse/HDFS-8707 
 .

The read API works well, but I can’t find write related API in 
hdfs::FileHandle. 

Is write functionality implemented in current hdfs++ code? If not, what's the 
plan for adding this functionality?

Thanks!

[jira] [Resolved] (HDFS-15385) Upgrade boost library to 1.72

[Akira Ajisaka (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-15385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka resolved HDFS-15385.
--
Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

Merged the PR into trunk. Thank you [~gautham]!

> Upgrade boost library to 1.72
> -
>
> Key: HDFS-15385
> URL: https://issues.apache.org/jira/browse/HDFS-15385
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: libhdfs++
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Critical
> Fix For: 3.4.0
>
> Attachments: image-2020-06-03-21-41-49-397.png
>
>
> The version of the boost library that's currently being used in HDFS is 
> 1.10.2. It is VERY old. It's from a time when it perhaps the name "boost" 
> wasn't even conceived. Going by the name of the library, it was probably just 
> called as "asio".
> From [https://www.boost.org/users/history/] website, the earliest available 
> version,1.10.3 is more than 2 decades old as it was released in 1999 -
> !image-2020-06-03-21-41-49-397.png!
> This really poses a big hurdle when it comes to upgrading to newer compiler 
> versions as, some the programming constructs that are used in asio-1.10.2 
> which were mere warnings back then, get flagged as outright errors with 
> modern compilers. (I tried to compile Hadoop with Visual Studio 2019 and saw 
> plenty of such errors).
> In the interest of keeping the Hadoop project modern and live, I would like 
> to propose the idea of upgrading the boost library with the latest.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org