[jira] [Updated] (HDFS-9521) TransferFsImage.receiveFile should account and log separate times for image download and fsync to disk
[
https://issues.apache.org/jira/browse/HDFS-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated HDFS-9521:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.9.0
Status: Resolved (was: Patch Available)
> TransferFsImage.receiveFile should account and log separate times for image
> download and fsync to disk
> ---
>
> Key: HDFS-9521
> URL: https://issues.apache.org/jira/browse/HDFS-9521
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Fix For: 2.9.0
>
> Attachments: HDFS-9521-2.patch, HDFS-9521-3.patch,
> HDFS-9521.004.patch, HDFS-9521.patch, HDFS-9521.patch.1
>
>
> Currently, TransferFsImage.receiveFile is logging total transfer time as
> below:
> {noformat}
> double xferSec = Math.max(
>((float)(Time.monotonicNow() - startTime)) / 1000.0, 0.001);
> long xferKb = received / 1024;
> LOG.info(String.format("Transfer took %.2fs at %.2f KB/s",xferSec, xferKb /
> xferSec))
> {noformat}
> This is really useful, but it just measures the total method execution time,
> which includes time taken to download the image and do an fsync to all the
> namenode metadata directories.
> Sometime when troubleshooting these imager transfer problems, it's
> interesting to know which part of the process is being the bottleneck
> (whether network or disk write).
> This patch accounts time for image download and fsync to each disk
> separately, logging how much time did it take on each operation.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9521) TransferFsImage.receiveFile should account and log separate times for image download and fsync to disk
[
https://issues.apache.org/jira/browse/HDFS-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated HDFS-9521:
--
Attachment: HDFS-9521.004.patch
LGTM. Just had two checkstyle nits I've corrected in this variant, aside of
some spacing logic. Will commit once jenkins returns +1.
The previously failed tests don't appear related.
> TransferFsImage.receiveFile should account and log separate times for image
> download and fsync to disk
> ---
>
> Key: HDFS-9521
> URL: https://issues.apache.org/jira/browse/HDFS-9521
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-9521-2.patch, HDFS-9521-3.patch,
> HDFS-9521.004.patch, HDFS-9521.patch, HDFS-9521.patch.1
>
>
> Currently, TransferFsImage.receiveFile is logging total transfer time as
> below:
> {noformat}
> double xferSec = Math.max(
>((float)(Time.monotonicNow() - startTime)) / 1000.0, 0.001);
> long xferKb = received / 1024;
> LOG.info(String.format("Transfer took %.2fs at %.2f KB/s",xferSec, xferKb /
> xferSec))
> {noformat}
> This is really useful, but it just measures the total method execution time,
> which includes time taken to download the image and do an fsync to all the
> namenode metadata directories.
> Sometime when troubleshooting these imager transfer problems, it's
> interesting to know which part of the process is being the bottleneck
> (whether network or disk write).
> This patch accounts time for image download and fsync to each disk
> separately, logging how much time did it take on each operation.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9521) TransferFsImage.receiveFile should account and log separate times for image download and fsync to disk
[
https://issues.apache.org/jira/browse/HDFS-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HDFS-9521:
---
Status: Patch Available (was: In Progress)
> TransferFsImage.receiveFile should account and log separate times for image
> download and fsync to disk
> ---
>
> Key: HDFS-9521
> URL: https://issues.apache.org/jira/browse/HDFS-9521
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-9521-2.patch, HDFS-9521-3.patch, HDFS-9521.patch,
> HDFS-9521.patch.1
>
>
> Currently, TransferFsImage.receiveFile is logging total transfer time as
> below:
> {noformat}
> double xferSec = Math.max(
>((float)(Time.monotonicNow() - startTime)) / 1000.0, 0.001);
> long xferKb = received / 1024;
> LOG.info(String.format("Transfer took %.2fs at %.2f KB/s",xferSec, xferKb /
> xferSec))
> {noformat}
> This is really useful, but it just measures the total method execution time,
> which includes time taken to download the image and do an fsync to all the
> namenode metadata directories.
> Sometime when troubleshooting these imager transfer problems, it's
> interesting to know which part of the process is being the bottleneck
> (whether network or disk write).
> This patch accounts time for image download and fsync to each disk
> separately, logging how much time did it take on each operation.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9521) TransferFsImage.receiveFile should account and log separate times for image download and fsync to disk
[
https://issues.apache.org/jira/browse/HDFS-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HDFS-9521:
---
Attachment: HDFS-9521-3.patch
Removed the new line on the log message. Inserting the combined time before the
separate fsync for each disk.
> TransferFsImage.receiveFile should account and log separate times for image
> download and fsync to disk
> ---
>
> Key: HDFS-9521
> URL: https://issues.apache.org/jira/browse/HDFS-9521
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-9521-2.patch, HDFS-9521-3.patch, HDFS-9521.patch,
> HDFS-9521.patch.1
>
>
> Currently, TransferFsImage.receiveFile is logging total transfer time as
> below:
> {noformat}
> double xferSec = Math.max(
>((float)(Time.monotonicNow() - startTime)) / 1000.0, 0.001);
> long xferKb = received / 1024;
> LOG.info(String.format("Transfer took %.2fs at %.2f KB/s",xferSec, xferKb /
> xferSec))
> {noformat}
> This is really useful, but it just measures the total method execution time,
> which includes time taken to download the image and do an fsync to all the
> namenode metadata directories.
> Sometime when troubleshooting these imager transfer problems, it's
> interesting to know which part of the process is being the bottleneck
> (whether network or disk write).
> This patch accounts time for image download and fsync to each disk
> separately, logging how much time did it take on each operation.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9521) TransferFsImage.receiveFile should account and log separate times for image download and fsync to disk
[
https://issues.apache.org/jira/browse/HDFS-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HDFS-9521:
---
Status: In Progress (was: Patch Available)
> TransferFsImage.receiveFile should account and log separate times for image
> download and fsync to disk
> ---
>
> Key: HDFS-9521
> URL: https://issues.apache.org/jira/browse/HDFS-9521
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-9521-2.patch, HDFS-9521-3.patch, HDFS-9521.patch,
> HDFS-9521.patch.1
>
>
> Currently, TransferFsImage.receiveFile is logging total transfer time as
> below:
> {noformat}
> double xferSec = Math.max(
>((float)(Time.monotonicNow() - startTime)) / 1000.0, 0.001);
> long xferKb = received / 1024;
> LOG.info(String.format("Transfer took %.2fs at %.2f KB/s",xferSec, xferKb /
> xferSec))
> {noformat}
> This is really useful, but it just measures the total method execution time,
> which includes time taken to download the image and do an fsync to all the
> namenode metadata directories.
> Sometime when troubleshooting these imager transfer problems, it's
> interesting to know which part of the process is being the bottleneck
> (whether network or disk write).
> This patch accounts time for image download and fsync to each disk
> separately, logging how much time did it take on each operation.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9521) TransferFsImage.receiveFile should account and log separate times for image download and fsync to disk
[
https://issues.apache.org/jira/browse/HDFS-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HDFS-9521:
---
Attachment: HDFS-9521-2.patch
New patch version. Combined all logs in a single log version.
> TransferFsImage.receiveFile should account and log separate times for image
> download and fsync to disk
> ---
>
> Key: HDFS-9521
> URL: https://issues.apache.org/jira/browse/HDFS-9521
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-9521-2.patch, HDFS-9521.patch, HDFS-9521.patch.1
>
>
> Currently, TransferFsImage.receiveFile is logging total transfer time as
> below:
> {noformat}
> double xferSec = Math.max(
>((float)(Time.monotonicNow() - startTime)) / 1000.0, 0.001);
> long xferKb = received / 1024;
> LOG.info(String.format("Transfer took %.2fs at %.2f KB/s",xferSec, xferKb /
> xferSec))
> {noformat}
> This is really useful, but it just measures the total method execution time,
> which includes time taken to download the image and do an fsync to all the
> namenode metadata directories.
> Sometime when troubleshooting these imager transfer problems, it's
> interesting to know which part of the process is being the bottleneck
> (whether network or disk write).
> This patch accounts time for image download and fsync to each disk
> separately, logging how much time did it take on each operation.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9521) TransferFsImage.receiveFile should account and log separate times for image download and fsync to disk
[
https://issues.apache.org/jira/browse/HDFS-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HDFS-9521:
---
Status: Patch Available (was: In Progress)
> TransferFsImage.receiveFile should account and log separate times for image
> download and fsync to disk
> ---
>
> Key: HDFS-9521
> URL: https://issues.apache.org/jira/browse/HDFS-9521
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-9521-2.patch, HDFS-9521.patch, HDFS-9521.patch.1
>
>
> Currently, TransferFsImage.receiveFile is logging total transfer time as
> below:
> {noformat}
> double xferSec = Math.max(
>((float)(Time.monotonicNow() - startTime)) / 1000.0, 0.001);
> long xferKb = received / 1024;
> LOG.info(String.format("Transfer took %.2fs at %.2f KB/s",xferSec, xferKb /
> xferSec))
> {noformat}
> This is really useful, but it just measures the total method execution time,
> which includes time taken to download the image and do an fsync to all the
> namenode metadata directories.
> Sometime when troubleshooting these imager transfer problems, it's
> interesting to know which part of the process is being the bottleneck
> (whether network or disk write).
> This patch accounts time for image download and fsync to each disk
> separately, logging how much time did it take on each operation.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9521) TransferFsImage.receiveFile should account and log separate times for image download and fsync to disk
[
https://issues.apache.org/jira/browse/HDFS-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HDFS-9521:
---
Status: In Progress (was: Patch Available)
> TransferFsImage.receiveFile should account and log separate times for image
> download and fsync to disk
> ---
>
> Key: HDFS-9521
> URL: https://issues.apache.org/jira/browse/HDFS-9521
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-9521-2.patch, HDFS-9521.patch, HDFS-9521.patch.1
>
>
> Currently, TransferFsImage.receiveFile is logging total transfer time as
> below:
> {noformat}
> double xferSec = Math.max(
>((float)(Time.monotonicNow() - startTime)) / 1000.0, 0.001);
> long xferKb = received / 1024;
> LOG.info(String.format("Transfer took %.2fs at %.2f KB/s",xferSec, xferKb /
> xferSec))
> {noformat}
> This is really useful, but it just measures the total method execution time,
> which includes time taken to download the image and do an fsync to all the
> namenode metadata directories.
> Sometime when troubleshooting these imager transfer problems, it's
> interesting to know which part of the process is being the bottleneck
> (whether network or disk write).
> This patch accounts time for image download and fsync to each disk
> separately, logging how much time did it take on each operation.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9521) TransferFsImage.receiveFile should account and log separate times for image download and fsync to disk
[
https://issues.apache.org/jira/browse/HDFS-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HDFS-9521:
---
Attachment: HDFS-9521.patch.1
New patch version including logging for combined time for download + fsyncs to
all disks.
Also fixed minor checkstyles issues.
> TransferFsImage.receiveFile should account and log separate times for image
> download and fsync to disk
> ---
>
> Key: HDFS-9521
> URL: https://issues.apache.org/jira/browse/HDFS-9521
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-9521.patch, HDFS-9521.patch.1
>
>
> Currently, TransferFsImage.receiveFile is logging total transfer time as
> below:
> {noformat}
> double xferSec = Math.max(
>((float)(Time.monotonicNow() - startTime)) / 1000.0, 0.001);
> long xferKb = received / 1024;
> LOG.info(String.format("Transfer took %.2fs at %.2f KB/s",xferSec, xferKb /
> xferSec))
> {noformat}
> This is really useful, but it just measures the total method execution time,
> which includes time taken to download the image and do an fsync to all the
> namenode metadata directories.
> Sometime when troubleshooting these imager transfer problems, it's
> interesting to know which part of the process is being the bottleneck
> (whether network or disk write).
> This patch accounts time for image download and fsync to each disk
> separately, logging how much time did it take on each operation.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9521) TransferFsImage.receiveFile should account and log separate times for image download and fsync to disk
[
https://issues.apache.org/jira/browse/HDFS-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HDFS-9521:
---
Status: Patch Available (was: In Progress)
> TransferFsImage.receiveFile should account and log separate times for image
> download and fsync to disk
> ---
>
> Key: HDFS-9521
> URL: https://issues.apache.org/jira/browse/HDFS-9521
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-9521.patch, HDFS-9521.patch.1
>
>
> Currently, TransferFsImage.receiveFile is logging total transfer time as
> below:
> {noformat}
> double xferSec = Math.max(
>((float)(Time.monotonicNow() - startTime)) / 1000.0, 0.001);
> long xferKb = received / 1024;
> LOG.info(String.format("Transfer took %.2fs at %.2f KB/s",xferSec, xferKb /
> xferSec))
> {noformat}
> This is really useful, but it just measures the total method execution time,
> which includes time taken to download the image and do an fsync to all the
> namenode metadata directories.
> Sometime when troubleshooting these imager transfer problems, it's
> interesting to know which part of the process is being the bottleneck
> (whether network or disk write).
> This patch accounts time for image download and fsync to each disk
> separately, logging how much time did it take on each operation.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9521) TransferFsImage.receiveFile should account and log separate times for image download and fsync to disk
[
https://issues.apache.org/jira/browse/HDFS-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HDFS-9521:
---
Status: In Progress (was: Patch Available)
New patch version available
> TransferFsImage.receiveFile should account and log separate times for image
> download and fsync to disk
> ---
>
> Key: HDFS-9521
> URL: https://issues.apache.org/jira/browse/HDFS-9521
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-9521.patch, HDFS-9521.patch.1
>
>
> Currently, TransferFsImage.receiveFile is logging total transfer time as
> below:
> {noformat}
> double xferSec = Math.max(
>((float)(Time.monotonicNow() - startTime)) / 1000.0, 0.001);
> long xferKb = received / 1024;
> LOG.info(String.format("Transfer took %.2fs at %.2f KB/s",xferSec, xferKb /
> xferSec))
> {noformat}
> This is really useful, but it just measures the total method execution time,
> which includes time taken to download the image and do an fsync to all the
> namenode metadata directories.
> Sometime when troubleshooting these imager transfer problems, it's
> interesting to know which part of the process is being the bottleneck
> (whether network or disk write).
> This patch accounts time for image download and fsync to each disk
> separately, logging how much time did it take on each operation.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9521) TransferFsImage.receiveFile should account and log separate times for image download and fsync to disk
[
https://issues.apache.org/jira/browse/HDFS-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HDFS-9521:
---
Status: Patch Available (was: In Progress)
Initial patch version has been submitted.
> TransferFsImage.receiveFile should account and log separate times for image
> download and fsync to disk
> ---
>
> Key: HDFS-9521
> URL: https://issues.apache.org/jira/browse/HDFS-9521
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-9521.patch
>
>
> Currently, TransferFsImage.receiveFile is logging total transfer time as
> below:
> {noformat}
> double xferSec = Math.max(
>((float)(Time.monotonicNow() - startTime)) / 1000.0, 0.001);
> long xferKb = received / 1024;
> LOG.info(String.format("Transfer took %.2fs at %.2f KB/s",xferSec, xferKb /
> xferSec))
> {noformat}
> This is really useful, but it just measures the total method execution time,
> which includes time taken to download the image and do an fsync to all the
> namenode metadata directories.
> Sometime when troubleshooting these imager transfer problems, it's
> interesting to know which part of the process is being the bottleneck
> (whether network or disk write).
> This patch accounts time for image download and fsync to each disk
> separately, logging how much time did it take on each operation.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9521) TransferFsImage.receiveFile should account and log separate times for image download and fsync to disk
[
https://issues.apache.org/jira/browse/HDFS-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HDFS-9521:
---
Attachment: HDFS-9521.patch
> TransferFsImage.receiveFile should account and log separate times for image
> download and fsync to disk
> ---
>
> Key: HDFS-9521
> URL: https://issues.apache.org/jira/browse/HDFS-9521
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-9521.patch
>
>
> Currently, TransferFsImage.receiveFile is logging total transfer time as
> below:
> {noformat}
> double xferSec = Math.max(
>((float)(Time.monotonicNow() - startTime)) / 1000.0, 0.001);
> long xferKb = received / 1024;
> LOG.info(String.format("Transfer took %.2fs at %.2f KB/s",xferSec, xferKb /
> xferSec))
> {noformat}
> This is really useful, but it just measures the total method execution time,
> which includes time taken to download the image and do an fsync to all the
> namenode metadata directories.
> Sometime when troubleshooting these imager transfer problems, it's
> interesting to know which part of the process is being the bottleneck
> (whether network or disk write).
> This patch accounts time for image download and fsync to each disk
> separately, logging how much time did it take on each operation.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9521) TransferFsImage.receiveFile should account and log separate times for image download and fsync to disk
[
https://issues.apache.org/jira/browse/HDFS-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HDFS-9521:
---
Priority: Minor (was: Major)
> TransferFsImage.receiveFile should account and log separate times for image
> download and fsync to disk
> ---
>
> Key: HDFS-9521
> URL: https://issues.apache.org/jira/browse/HDFS-9521
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
>
> Currently, TransferFsImage.receiveFile is logging total transfer time as
> below:
> {noformat}
> double xferSec = Math.max(
>((float)(Time.monotonicNow() - startTime)) / 1000.0, 0.001);
> long xferKb = received / 1024;
> LOG.info(String.format("Transfer took %.2fs at %.2f KB/s",xferSec, xferKb /
> xferSec))
> {noformat}
> This is really useful, but it just measures the total method execution time,
> which includes time taken to download the image and do an fsync to all the
> namenode metadata directories.
> Sometime when troubleshooting these imager transfer problems, it's
> interesting to know which part of the process is being the bottleneck
> (whether network or disk write).
> This patch accounts time for image download and fsync to each disk
> separately, logging how much time did it take on each operation.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
