[jira] [Resolved] (HADOOP-18989) Use thread pool to improve the speed of creating control files in TestDFSIO

2023-12-08 Thread Shuyan Zhang (Jira) 


 [ 
https://issues.apache.org/jira/browse/HADOOP-18989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shuyan Zhang resolved HADOOP-18989.
---
Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Use thread pool to improve the speed of creating control files in TestDFSIO
> ---
>
> Key: HADOOP-18989
> URL: https://issues.apache.org/jira/browse/HADOOP-18989
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: benchmarks, common
>Affects Versions: 3.3.6
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> When we use TestDFSIO tool to test the throughouts of HDFS clusters, we found 
> it is so slow in the creating controll files stage. 
> After refering to the source code, we found that method createControlFile try 
> to create control files serially. It can be improved by using thread pool.
> After optimizing, the TestDFSIO tool runs quicker than before.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18982) Fix doc about loading native libraries

2023-11-19 Thread Shuyan Zhang (Jira) 
Shuyan Zhang created HADOOP-18982:
-

 Summary: Fix doc about loading native libraries
 Key: HADOOP-18982
 URL: https://issues.apache.org/jira/browse/HADOOP-18982
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Shuyan Zhang


When we want load a native library libmyexample.so, the right way is to call 
System.loadLibrary("myexample") rather than 
System.loadLibrary("libmyexample.so").



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18726) Set the locale to avoid printing useless logs.

2023-05-01 Thread Shuyan Zhang (Jira) 
Shuyan Zhang created HADOOP-18726:
-

 Summary: Set the locale to avoid printing useless logs.
 Key: HADOOP-18726
 URL: https://issues.apache.org/jira/browse/HADOOP-18726
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Shuyan Zhang


In our production environment, if the hadoop process is started in a 
non-English environment, many unexpected error logs will be printed. The 
following is the error message printed by datanode.
```
2023-05-01 09:10:50,299 ERROR 
org.apache.hadoop.hdfs.server.datanode.FileIoProvider: error in op 
transferToSocketFully : 断开的管道
2023-05-01 09:10:50,299 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: 
BlockSender.sendChunks() exception: 
java.io.IOException: 断开的管道
at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
at 
sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428)
at 
sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493)
at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:608)
at 
org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:242)
at 
org.apache.hadoop.hdfs.server.datanode.FileIoProvider.transferToSocketFully(FileIoProvider.java:260)
at 
org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:559)
at 
org.apache.hadoop.hdfs.server.datanode.BlockSender.doSendBlock(BlockSender.java:801)
at 
org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:755)
at 
org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:580)
at 
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:116)
at 
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
at 
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:258)
at java.lang.Thread.run(Thread.java:745)
2023-05-01 09:10:50,298 ERROR 
org.apache.hadoop.hdfs.server.datanode.FileIoProvider: error in op 
transferToSocketFully : 断开的管道
2023-05-01 09:10:50,298 ERROR 
org.apache.hadoop.hdfs.server.datanode.FileIoProvider: error in op 
transferToSocketFully : 断开的管道
2023-05-01 09:10:50,298 ERROR 
org.apache.hadoop.hdfs.server.datanode.FileIoProvider: error in op 
transferToSocketFully : 断开的管道
2023-05-01 09:10:50,298 ERROR 
org.apache.hadoop.hdfs.server.datanode.FileIoProvider: error in op 
transferToSocketFully : 断开的管道
2023-05-01 09:10:50,302 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: 
BlockSender.sendChunks() exception: 
java.io.IOException: 断开的管道
at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
at 
sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428)
at 
sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493)
at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:608)
at 
org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:242)
at 
org.apache.hadoop.hdfs.server.datanode.FileIoProvider.transferToSocketFully(FileIoProvider.java:260)
at 
org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:559)
at 
org.apache.hadoop.hdfs.server.datanode.BlockSender.doSendBlock(BlockSender.java:801)
at 
org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:755)
at 
org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:580)
at 
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:116)
at 
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
at 
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:258)
at java.lang.Thread.run(Thread.java:745)
2023-05-01 09:10:50,303 ERROR 
org.apache.hadoop.hdfs.server.datanode.FileIoProvider: error in op 
transferToSocketFully : 断开的管道
2023-05-01 09:10:50,303 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: 
BlockSender.sendChunks() exception: 
java.io.IOException: 断开的管道
at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
at 
sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428)
at 
sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493)
at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:608)
at 
org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:242)
at 
org.apache.hadoop.hdfs.server.datanode.FileIoProvider.transferToSocketFully(FileIoProvider.java:260)
at 
org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:568)
at 
org.apache.hadoop.hdfs.server.datanode.BlockSender.doSendBlock(BlockSender.java:801)
at

[jira] [Created] (HADOOP-18426) Improve the accuracy of MutableStat mean

2022-08-26 Thread Shuyan Zhang (Jira) 
Shuyan Zhang created HADOOP-18426:
-

 Summary: Improve the accuracy of MutableStat mean
 Key: HADOOP-18426
 URL: https://issues.apache.org/jira/browse/HADOOP-18426
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Shuyan Zhang


The current MutableStat mean calculation method is more prone to loss accuracy 
because the sum of samples is too large. We can process each sample on its own 
to improve mean accuracy.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org