[jira] [Updated] (HDFS-6524) Choosing datanode retries times considering with block replica number

2019-09-17 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-6524: -- Attachment: HDFS-6524.005.patch > Choosing datanode retries times considering with block replica

[jira] [Commented] (HDFS-14795) Add Throttler for writing block

2019-09-17 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931339#comment-16931339 ] Lisheng Sun commented on HDFS-14795: hi [~elgoiri], should we commit this patch to trunk? Thank you.

[jira] [Commented] (HDFS-14844) Make buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream configurable

2019-09-17 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931336#comment-16931336 ] Lisheng Sun commented on HDFS-14844: hi [~elgoiri], could you help continue to take a review? Thank

[jira] [Updated] (HDFS-14850) Optimize FileSystemAccessService#getFileSystemConfiguration

2019-09-17 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14850: --- Attachment: HDFS-14850.001.patch Status: Patch Available (was: Open) > Optimize

[jira] [Commented] (HDFS-6524) Choosing datanode retries times considering with block replica number

2019-09-16 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930997#comment-16930997 ] Lisheng Sun commented on HDFS-6524: --- hi [~elgoiri] , i agree your option. There are same problems in

[jira] [Updated] (HDFS-6524) Choosing datanode retries times considering with block replica number

2019-09-16 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-6524: -- Attachment: HDFS-6524.005.patch > Choosing datanode retries times considering with block replica

[jira] [Created] (HDFS-14850) Optimize FileSystemAccessService#getFileSystemConfiguration

2019-09-15 Thread Lisheng Sun (Jira)
Lisheng Sun created HDFS-14850: -- Summary: Optimize FileSystemAccessService#getFileSystemConfiguration Key: HDFS-14850 URL: https://issues.apache.org/jira/browse/HDFS-14850 Project: Hadoop HDFS

[jira] [Updated] (HDFS-14850) Optimize FileSystemAccessService#getFileSystemConfiguration

2019-09-15 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14850: --- Environment: (was: *strong text*) > Optimize FileSystemAccessService#getFileSystemConfiguration

[jira] [Updated] (HDFS-14283) DFSInputStream to prefer cached replica

2019-09-15 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14283: --- Status: Patch Available (was: Open) > DFSInputStream to prefer cached replica >

[jira] [Commented] (HDFS-14283) DFSInputStream to prefer cached replica

2019-09-15 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929932#comment-16929932 ] Lisheng Sun commented on HDFS-14283: hi [~jojochuang] I just changed

[jira] [Updated] (HDFS-14283) DFSInputStream to prefer cached replica

2019-09-15 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14283: --- Attachment: HDFS-14283.001.patch > DFSInputStream to prefer cached replica >

[jira] [Updated] (HDFS-6524) Choosing datanode retries times considering with block replica number

2019-09-15 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-6524: -- Attachment: HDFS-6524.004.patch > Choosing datanode retries times considering with block replica

[jira] [Updated] (HDFS-6524) Choosing datanode retries times considering with block replica number

2019-09-15 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-6524: -- Attachment: (was: HDFS-6524.004.patch) > Choosing datanode retries times considering with block

[jira] [Updated] (HDFS-6524) Choosing datanode retries times considering with block replica number

2019-09-15 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-6524: -- Attachment: HDFS-6524.004.patch > Choosing datanode retries times considering with block replica

[jira] [Updated] (HDFS-6524) Choosing datanode retries times considering with block replica number

2019-09-14 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-6524: -- Attachment: HDFS-6524.003.patch > Choosing datanode retries times considering with block replica

[jira] [Commented] (HDFS-14844) Make buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream configurable

2019-09-13 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929638#comment-16929638 ] Lisheng Sun commented on HDFS-14844: Thanks [~elgoiri] for your deeply review. I updated the patch

[jira] [Updated] (HDFS-14844) Make buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream configurable

2019-09-13 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14844: --- Attachment: HDFS-14844.004.patch > Make buffer of

[jira] [Commented] (HDFS-14795) Add Throttler for writing block

2019-09-13 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929634#comment-16929634 ] Lisheng Sun commented on HDFS-14795: I confirmed all failed UTs are ok in local. So they are

[jira] [Commented] (HDFS-14844) Make buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream configurable

2019-09-13 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929318#comment-16929318 ] Lisheng Sun commented on HDFS-14844: uploaded the v003 patch. > Make buffer of

[jira] [Updated] (HDFS-14844) Make buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream configurable

2019-09-13 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14844: --- Attachment: HDFS-14844.003.patch > Make buffer of

[jira] [Commented] (HDFS-14844) Make buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream configurable

2019-09-13 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929301#comment-16929301 ] Lisheng Sun commented on HDFS-14844: hi [~elgoiri] {quote} This is kind of a readahead buffer that

[jira] [Commented] (HDFS-14795) Add Throttler for writing block

2019-09-13 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929144#comment-16929144 ] Lisheng Sun commented on HDFS-14795: Thanks [~elgoiri] for your good suggestion.  I added to check

[jira] [Updated] (HDFS-14795) Add Throttler for writing block

2019-09-13 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14795: --- Attachment: HDFS-14795.012.patch > Add Throttler for writing block > ---

[jira] [Updated] (HDFS-14844) Make buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream configurable

2019-09-11 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14844: --- Attachment: HDFS-14844.002.patch > Make buffer of

[jira] [Updated] (HDFS-14795) Add Throttler for writing block

2019-09-11 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14795: --- Attachment: HDFS-14795.011.patch > Add Throttler for writing block > ---

[jira] [Commented] (HDFS-14795) Add Throttler for writing block

2019-09-11 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927786#comment-16927786 ] Lisheng Sun commented on HDFS-14795: Fixed the the problem in hdfs-default.xml and uploaded the v010

[jira] [Updated] (HDFS-14795) Add Throttler for writing block

2019-09-11 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14795: --- Attachment: HDFS-14795.010.patch > Add Throttler for writing block > ---

[jira] [Updated] (HDFS-14844) Make buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream configurable

2019-09-11 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14844: --- Attachment: HDFS-14844.001.patch > Make buffer of

[jira] [Updated] (HDFS-14844) Make buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream configurable

2019-09-10 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14844: --- Description: details for HDFS-14820 > Make buffer of

[jira] [Created] (HDFS-14844) Make buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream configurable

2019-09-10 Thread Lisheng Sun (Jira)
Lisheng Sun created HDFS-14844: -- Summary: Make buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream configurable Key: HDFS-14844 URL: https://issues.apache.org/jira/browse/HDFS-14844

[jira] [Commented] (HDFS-14795) Add Throttler for writing block

2019-09-10 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927208#comment-16927208 ] Lisheng Sun commented on HDFS-14795: Thanks [~elgoiri] for your suggestion.  I fixed the checkstyle

[jira] [Updated] (HDFS-14795) Add Throttler for writing block

2019-09-10 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14795: --- Attachment: HDFS-14795.009.patch > Add Throttler for writing block > ---

[jira] [Updated] (HDFS-14795) Add Throttler for writing block

2019-09-10 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14795: --- Attachment: (was: HDFS-14795.009.patch) > Add Throttler for writing block >

[jira] [Updated] (HDFS-14795) Add Throttler for writing block

2019-09-10 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14795: --- Attachment: HDFS-14795.009.patch > Add Throttler for writing block > ---

[jira] [Updated] (HDFS-4916) DataTransfer may mask the IOException during block transfering

2019-09-10 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-4916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-4916: -- Resolution: Implemented Status: Resolved (was: Patch Available) > DataTransfer may mask the

[jira] [Commented] (HDFS-6524) Choosing datanode retries times considering with block replica number

2019-09-10 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926744#comment-16926744 ] Lisheng Sun commented on HDFS-6524: --- hi [~elgoiri], [~ayushtkn] [~xkrogen] Could you have time to help

[jira] [Updated] (HDFS-14795) Add Throttler for writing block

2019-09-10 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14795: --- Attachment: HDFS-14795.008.patch > Add Throttler for writing block > ---

[jira] [Commented] (HDFS-14795) Add Throttler for writing block

2019-09-10 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926671#comment-16926671 ] Lisheng Sun commented on HDFS-14795: fixed checkstyle and uploaded the v008 patch. > Add Throttler

[jira] [Updated] (HDFS-14795) Add Throttler for writing block

2019-09-10 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14795: --- Attachment: HDFS-14795.007.patch > Add Throttler for writing block > ---

[jira] [Updated] (HDFS-14795) Add Throttler for writing block

2019-09-10 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14795: --- Attachment: HDFS-14795.006.patch > Add Throttler for writing block > ---

[jira] [Comment Edited] (HDFS-14820) The default 8KB buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream is too big

2019-09-09 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926304#comment-16926304 ] Lisheng Sun edited comment on HDFS-14820 at 9/10/19 3:38 AM: - hi [~elgoiri]

[jira] [Commented] (HDFS-14820) The default 8KB buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream is too big

2019-09-09 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926304#comment-16926304 ] Lisheng Sun commented on HDFS-14820: hi [~elgoiri] {quote}What is the current default value? 8KB?

[jira] [Commented] (HDFS-14283) DFSInputStream to prefer cached replica

2019-09-09 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926296#comment-16926296 ] Lisheng Sun commented on HDFS-14283: [~smeng] i are working on this jira.  upload this patch later.

[jira] [Updated] (HDFS-14795) Add Throttler for writing block

2019-09-09 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14795: --- Attachment: HDFS-14795.005.patch > Add Throttler for writing block > ---

[jira] [Updated] (HDFS-14795) Add Throttler for writing block

2019-09-08 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14795: --- Attachment: HDFS-14795.004.patch > Add Throttler for writing block > ---

[jira] [Commented] (HDFS-14795) Add Throttler for writing block

2019-09-08 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925199#comment-16925199 ] Lisheng Sun commented on HDFS-14795: Fixed checkstyle and uploaded the v004 patch. > Add Throttler

[jira] [Commented] (HDFS-14795) Add Throttler for writing block

2019-09-08 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925126#comment-16925126 ] Lisheng Sun commented on HDFS-14795: hi [~elgoiri] I updated this patch and uploaded v003 patch.

[jira] [Updated] (HDFS-14795) Add Throttler for writing block

2019-09-08 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14795: --- Attachment: HDFS-14795.003.patch > Add Throttler for writing block > ---

[jira] [Commented] (HDFS-14820) The default 8KB buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream is too big

2019-09-08 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925114#comment-16925114 ] Lisheng Sun commented on HDFS-14820: hi [~elgoiri], [~ayushtkn] [~xkrogen] Could you have time to

[jira] [Commented] (HDFS-14820) The default 8KB buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream is too big

2019-09-07 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924924#comment-16924924 ] Lisheng Sun commented on HDFS-14820: hi [~tlipcon], this Jira is similar to HDFS-14535.  Could you

[jira] [Commented] (HDFS-14795) Add Throttler for writing block

2019-09-06 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924697#comment-16924697 ] Lisheng Sun commented on HDFS-14795: Thanks [~elgoiri] for good suggestions. I will update this patch

[jira] [Updated] (HDFS-14795) Add Throttler for writing block

2019-09-06 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14795: --- Attachment: HDFS-14795.002.patch > Add Throttler for writing block > ---

[jira] [Updated] (HDFS-14795) Add Throttler for writing block

2019-09-05 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14795: --- Status: Patch Available (was: Open) > Add Throttler for writing block >

[jira] [Updated] (HDFS-14795) Add Throttler for writing block

2019-09-05 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14795: --- Attachment: HDFS-14795.001.patch > Add Throttler for writing block > ---

[jira] [Commented] (HDFS-12904) Add DataTransferThrottler to the Datanode transfers

2019-09-05 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-12904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923300#comment-16923300 ] Lisheng Sun commented on HDFS-12904: [~elgoiri] Could you have time to take another review? Thank

[jira] [Updated] (HDFS-14823) BlockReaderRemote#readNextPacket is jittering

2019-09-04 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14823: --- Description: When use the ycsb for pressure test of remote read, I found continuous jitter in

[jira] [Updated] (HDFS-14823) BlockReaderRemote#readNextPacket jitter

2019-09-04 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14823: --- Description: When use the ycsb for pressure test of remote read, I found continuous jitter in

[jira] [Updated] (HDFS-14823) BlockReaderRemote#readNextPacket is jittering

2019-09-04 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14823: --- Summary: BlockReaderRemote#readNextPacket is jittering (was: BlockReaderRemote#readNextPacket

[jira] [Updated] (HDFS-14823) BlockReaderRemote#readNextPacket jitter

2019-09-04 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14823: --- Description: When use the ycsb for pressure test of remote read, I found continuous jitter in

[jira] [Updated] (HDFS-14823) BlockReaderRemote#readNextPacket jitter

2019-09-04 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14823: --- Attachment: remote_read_qps.png regionserver_gc.png >

[jira] [Updated] (HDFS-14823) BlockReaderRemote#readNextPacket jitter

2019-09-04 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14823: --- Attachment: 99-percentile_readNextPacket.png 99-percentile_readAll.png

[jira] [Updated] (HDFS-14823) BlockReaderRemote#readNextPacket jitter

2019-09-04 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14823: --- Attachment: CPS-1-min_readNextPacket.png CPS-1-min_readAll.png >

[jira] [Created] (HDFS-14823) BlockReaderRemote#readNextPacket jitter

2019-09-04 Thread Lisheng Sun (Jira)
Lisheng Sun created HDFS-14823: -- Summary: BlockReaderRemote#readNextPacket jitter Key: HDFS-14823 URL: https://issues.apache.org/jira/browse/HDFS-14823 Project: Hadoop HDFS Issue Type: Bug

[jira] [Updated] (HDFS-14820) The default 8KB buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream is too big

2019-09-04 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14820: --- Attachment: HDFS-14820.001.patch Assignee: Lisheng Sun Status: Patch Available (was:

[jira] [Updated] (HDFS-14820) The default 8KB buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream is too big

2019-09-04 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14820: --- Description: this issue is similar to HDFS-14535. {code:java} public static BlockReader

[jira] [Created] (HDFS-14820) The default 8KB buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream is too big

2019-09-04 Thread Lisheng Sun (Jira)
Lisheng Sun created HDFS-14820: -- Summary: The default 8KB buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream is too big Key: HDFS-14820 URL: https://issues.apache.org/jira/browse/HDFS-14820

[jira] [Commented] (HDFS-14648) DeadNodeDetector basic model

2019-09-04 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922317#comment-16922317 ] Lisheng Sun commented on HDFS-14648: Detailed DeadNodeDetector desgin documents attached HDFS-13571.

[jira] [Commented] (HDFS-13571) Dead DataNode Detector

2019-09-04 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-13571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922314#comment-16922314 ] Lisheng Sun commented on HDFS-13571: uploaded DeadNodeDetectorDesign document. > Dead DataNode

[jira] [Updated] (HDFS-13571) Dead DataNode Detector

2019-09-04 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-13571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-13571: --- Attachment: DeadNodeDetectorDesign.pdf > Dead DataNode Detector > -- > >

[jira] [Commented] (HDFS-6524) Choosing datanode retries times considering with block replica number

2019-09-02 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920925#comment-16920925 ] Lisheng Sun commented on HDFS-6524: --- hi [~jojochuang] [~ayushtkn]  Could you help take a review for this

[jira] [Commented] (HDFS-14751) TestNameNodeMetadataConsistency#testGenerationStampInFuture fail in trunk

2019-09-02 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920915#comment-16920915 ] Lisheng Sun commented on HDFS-14751: hi [~seanlook] i agree your idea.   we can synchronize

[jira] [Commented] (HDFS-12904) Add DataTransferThrottler to the Datanode transfers

2019-09-02 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-12904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920879#comment-16920879 ] Lisheng Sun commented on HDFS-12904: hi [~elgoiri] Should we commit this patch to trunk? Thank you.

[jira] [Commented] (HDFS-14795) Add Throttler for writing block

2019-08-30 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920032#comment-16920032 ] Lisheng Sun commented on HDFS-14795: hi [~elgoiri] I have finished this patch,but this patch depends

[jira] [Updated] (HDFS-14648) DeadNodeDetector basic model

2019-08-30 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14648: --- Attachment: HDFS-14648.006.patch > DeadNodeDetector basic model > > >

[jira] [Updated] (HDFS-14797) Define LOG instead of BlockManager.LOG in ReplicationWork

2019-08-29 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14797: --- Attachment: HDFS-14797.001.patch Status: Patch Available (was: Open) > Define LOG instead

[jira] [Commented] (HDFS-14796) Define LOG instead of BlockManager.LOG in ErasureCodingWork

2019-08-29 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918721#comment-16918721 ] Lisheng Sun commented on HDFS-14796: hi [~surendrasingh] I also found this log problem in

[jira] [Created] (HDFS-14797) Define LOG instead of BlockManager.LOG in ReplicationWork

2019-08-29 Thread Lisheng Sun (Jira)
Lisheng Sun created HDFS-14797: -- Summary: Define LOG instead of BlockManager.LOG in ReplicationWork Key: HDFS-14797 URL: https://issues.apache.org/jira/browse/HDFS-14797 Project: Hadoop HDFS

[jira] [Commented] (HDFS-14648) DeadNodeDetector basic model

2019-08-29 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918707#comment-16918707 ] Lisheng Sun commented on HDFS-14648: Thank [~hexiaoqiao] for deep reivew and good suggestion. {quote}

[jira] [Updated] (HDFS-14648) DeadNodeDetector basic model

2019-08-29 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14648: --- Attachment: HDFS-14648.005.patch > DeadNodeDetector basic model > > >

[jira] [Commented] (HDFS-14748) Make DataNodePeerMetrics#minOutlierDetectionSamples configurable

2019-08-28 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918216#comment-16918216 ] Lisheng Sun commented on HDFS-14748:  [~xkrogen] [~jojochuang] Could you have time to review this

[jira] [Commented] (HDFS-12904) Add DataTransferThrottler to the Datanode transfers

2019-08-28 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-12904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918212#comment-16918212 ] Lisheng Sun commented on HDFS-12904: [~elgoiri]   HDFS-14795  is for adding throttle for writing

[jira] [Updated] (HDFS-14795) Add Throttler for writing block

2019-08-28 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14795: --- Description: DataXceiver#writeBlock {code:java} blockReceiver.receiveBlock(mirrorOut, mirrorIn,

[jira] [Updated] (HDFS-14795) Add Throttler for writing block

2019-08-28 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14795: --- Description: DataXceiver#writeBlock {code:java} blockReceiver.receiveBlock(mirrorOut, mirrorIn,

[jira] [Created] (HDFS-14795) Add Throttler for writing block

2019-08-28 Thread Lisheng Sun (Jira)
Lisheng Sun created HDFS-14795: -- Summary: Add Throttler for writing block Key: HDFS-14795 URL: https://issues.apache.org/jira/browse/HDFS-14795 Project: Hadoop HDFS Issue Type: Improvement

[jira] [Commented] (HDFS-12904) Add DataTransferThrottler to the Datanode transfers

2019-08-28 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-12904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16917514#comment-16917514 ] Lisheng Sun commented on HDFS-12904: [~elgoiri] DataXceiver#writeBlock {code:java}

[jira] [Updated] (HDFS-12904) Add DataTransferThrottler to the Datanode transfers

2019-08-28 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-12904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-12904: --- Attachment: HDFS-12904.006.patch > Add DataTransferThrottler to the Datanode transfers >

[jira] [Updated] (HDFS-12904) Add DataTransferThrottler to the Datanode transfers

2019-08-26 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-12904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-12904: --- Attachment: HDFS-12904.005.patch > Add DataTransferThrottler to the Datanode transfers >

[jira] [Updated] (HDFS-6524) Choosing datanode retries times considering with block replica number

2019-08-26 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-6524: -- Attachment: HDFS-6524.002.patch > Choosing datanode retries times considering with block replica

[jira] [Updated] (HDFS-6524) Choosing datanode retries times considering with block replica number

2019-08-26 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-6524: -- Summary: Choosing datanode retries times considering with block replica number (was: Choose datanode

[jira] [Updated] (HDFS-6524) Choose datanode decides retry times considering with block replica number

2019-08-26 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-6524: -- Summary: Choose datanode decides retry times considering with block replica number (was: Choose

[jira] [Updated] (HDFS-6524) Choose decides retry times considering with block replica number

2019-08-26 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-6524: -- Summary: Choose decides retry times considering with block replica number (was: Cdecides retry times

[jira] [Commented] (HDFS-12904) Add DataTransferThrottler to the Datanode transfers

2019-08-25 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-12904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915466#comment-16915466 ] Lisheng Sun commented on HDFS-12904: I confirmed the two UT are ok in my local. So the UT failures

[jira] [Commented] (HDFS-14748) Make DataNodePeerMetrics#minOutlierDetectionSamples configurable

2019-08-25 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915464#comment-16915464 ] Lisheng Sun commented on HDFS-14748: hi [~xkrogen] [~jojochuang] Could you help take a reiview for

[jira] [Commented] (HDFS-14648) DeadNodeDetector basic model

2019-08-25 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915281#comment-16915281 ] Lisheng Sun commented on HDFS-14648: Thank [~zhangchen] for good suggestion. I will upload the a

[jira] [Commented] (HDFS-12904) Add DataTransferThrottler to the Datanode transfers

2019-08-25 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-12904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915199#comment-16915199 ] Lisheng Sun commented on HDFS-12904: Uploaded the v003 patch. > Add DataTransferThrottler to the

[jira] [Updated] (HDFS-12904) Add DataTransferThrottler to the Datanode transfers

2019-08-25 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-12904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-12904: --- Attachment: HDFS-12904.003.patch > Add DataTransferThrottler to the Datanode transfers >

[jira] [Commented] (HDFS-14648) DeadNodeDetector basic model

2019-08-24 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915143#comment-16915143 ] Lisheng Sun commented on HDFS-14648: Thank [~jojochuang] for your good suggetions. I updated the

[jira] [Updated] (HDFS-14648) DeadNodeDetector basic model

2019-08-24 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14648: --- Description: This Jira constructs DeadNodeDetector state machine model. The function it implements

[jira] [Updated] (HDFS-14648) DeadNodeDetector basic model

2019-08-24 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14648: --- Description: This Jira constructs DeadNodeDetector state machine model. The function it implements

[jira] [Updated] (HDFS-14648) DeadNodeDetector basic model

2019-08-24 Thread Lisheng Sun (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14648: --- Description: This Jira constructs DeadNodeDetector state machine model. The function it implements

<    1   2   3   4   5   6   7   8   9   10   >