[jira] [Resolved] (HADOOP-15182) Support to change back to signature version 2 of AWS SDK
[
https://issues.apache.org/jira/browse/HADOOP-15182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yonger resolved HADOOP-15182.
-
Resolution: Not A Problem
> Support to change back to signature version 2 of AWS SDK
>
>
> Key: HADOOP-15182
> URL: https://issues.apache.org/jira/browse/HADOOP-15182
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.9.0
> Environment:
>
>
>Reporter: Yonger
>Priority: Minor
>
> Current s3a depend on aws-java-sdk-bundle-1.11.199 which use signature v4.
> So for some s3-compatible system(Ceph) which still using v2, Hadoop can't
> work on them.
> s3cmd can use v2 via specify option like :
> {code:java}
> s3cmd --signature-v2 ls s3://xxx/{code}
>
> maybe we can add a parameter to allow back to use signature v2 in s3a.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15182) Support to change back to signature version 2 of AWS SDK
[
https://issues.apache.org/jira/browse/HADOOP-15182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333778#comment-16333778
]
Yonger commented on HADOOP-15182:
-
Just know how to roll back to signature v2, it's a bit hard to find out this
approach for me.
{code:java}
fs.s3a.signing-algorithm
S3SignerType
{code}
Sorry to wast your time, I will close this ticket.
> Support to change back to signature version 2 of AWS SDK
>
>
> Key: HADOOP-15182
> URL: https://issues.apache.org/jira/browse/HADOOP-15182
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.9.0
> Environment:
>
>
>Reporter: Yonger
>Priority: Minor
>
> Current s3a depend on aws-java-sdk-bundle-1.11.199 which use signature v4.
> So for some s3-compatible system(Ceph) which still using v2, Hadoop can't
> work on them.
> s3cmd can use v2 via specify option like :
> {code:java}
> s3cmd --signature-v2 ls s3://xxx/{code}
>
> maybe we can add a parameter to allow back to use signature v2 in s3a.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15182) Support to change back to signature version 2 of AWS SDK
[
https://issues.apache.org/jira/browse/HADOOP-15182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16331554#comment-16331554
]
Yonger commented on HADOOP-15182:
-
You means "fs.s3a.signing-algorithm"?, but I can't find an algorithm to match
v2:
{code:java}
static {
// Register the standard signer types.
SIGNERS.put(QUERY_STRING_SIGNER, QueryStringSigner.class);
SIGNERS.put(VERSION_THREE_SIGNER, AWS3Signer.class);
SIGNERS.put(VERSION_FOUR_SIGNER, AWS4Signer.class);
SIGNERS.put(VERSION_FOUR_UNSIGNED_PAYLOAD_SIGNER,
AWS4UnsignedPayloadSigner.class);
SIGNERS.put(NO_OP_SIGNER, NoOpSigner.class);
}{code}
> Support to change back to signature version 2 of AWS SDK
>
>
> Key: HADOOP-15182
> URL: https://issues.apache.org/jira/browse/HADOOP-15182
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.9.0
> Environment:
>
>
>Reporter: Yonger
>Priority: Minor
>
> Current s3a depend on aws-java-sdk-bundle-1.11.199 which use signature v4.
> So for some s3-compatible system(Ceph) which still using v2, Hadoop can't
> work on them.
> s3cmd can use v2 via specify option like :
> {code:java}
> s3cmd --signature-v2 ls s3://xxx/{code}
>
> maybe we can add a parameter to allow back to use signature v2 in s3a.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14943) Add common getFileBlockLocations() emulation for object stores, including S3A
[
https://issues.apache.org/jira/browse/HADOOP-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16303790#comment-16303790
]
Yonger commented on HADOOP-14943:
-
[~ste...@apache.org] As Hadoop running on s3-compatible storage, compute and
storage separately, so I called the host that running MR/Spark/Hive application
"compute host".
> Add common getFileBlockLocations() emulation for object stores, including S3A
> -
>
> Key: HADOOP-14943
> URL: https://issues.apache.org/jira/browse/HADOOP-14943
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14943-001.patch, HADOOP-14943-002.patch,
> HADOOP-14943-002.patch, HADOOP-14943-003.patch
>
>
> It looks suspiciously like S3A isn't providing the partitioning data needed
> in {{listLocatedStatus}} and {{getFileBlockLocations()}} needed to break up a
> file by the blocksize. This will stop tools using the MRv1 APIS doing the
> partitioning properly if the input format isn't doing it own split logic.
> FileInputFormat in MRv2 is a bit more configurable about input split
> calculation & will split up large files. but otherwise, the partitioning is
> being done more by the default values of the executing engine, rather than
> any config data from the filesystem about what its "block size" is,
> NativeAzureFS does a better job; maybe that could be factored out to
> hadoop-common and reused?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15087) Write directly without creating temp directory to avoid rename
Yonger created HADOOP-15087: --- Summary: Write directly without creating temp directory to avoid rename Key: HADOOP-15087 URL: https://issues.apache.org/jira/browse/HADOOP-15087 Project: Hadoop Common Issue Type: Improvement Components: fs/s3 Reporter: Yonger Rename in workloads like Teragen/Terasort who use Hadoop default outputcommitters really hurt performance a lot. Stocator announce it doesn't create the temporary directories any all, and still preserves Hadoop's fault tolerance. I add a switch when creating file via integrating it's code into s3a, I got 5x performance gain in Teragen and 15% performance improvement in Terasort. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[
https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16268148#comment-16268148
]
Yonger commented on HADOOP-14475:
-
[~mackrorysd] Thank you. I start to understand your logic, and I think you are
right.
{code:java}
There's no guarantee that metrics source names would even be consistent among
all JVMs for a given bucket, since they're assigned numbers in the order that
they're created
{code}
I can get the info what i want to by aggregating based on bucket instead of
metric source name which is not still unique in multiple JVM processes.
> Metrics of S3A don't print out when enable it in Hadoop metrics property file
> --
>
> Key: HADOOP-14475
> URL: https://issues.apache.org/jira/browse/HADOOP-14475
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
> Environment: uname -a
> Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017
> x86_64 x86_64 x86_64 GNU/Linux
> cat /etc/issue
> Ubuntu 16.04.2 LTS \n \l
>Reporter: Yonger
>Assignee: Yonger
> Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch,
> HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch,
> HADOOP-14475.009.patch, HADOOP-14475.010.patch, HADOOP-14475.011.patch,
> HADOOP-14475.012.patch, HADOOP-14475.013.patch, HADOOP-14475.014.patch,
> HADOOP-14475.015.patch, HADOOP-14775.007.patch, failsafe-report-s3a-it.html,
> failsafe-report-s3a-scale.html, failsafe-report-scale.html,
> failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip
>
>
> *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink
> #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #*.sink.influxdb.url=http:/xx
> #*.sink.influxdb.influxdb_port=8086
> #*.sink.influxdb.database=hadoop
> #*.sink.influxdb.influxdb_username=hadoop
> #*.sink.influxdb.influxdb_password=hadoop
> #*.sink.ingluxdb.cluster=c1
> *.period=10
> #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out
> I can't find the out put file even i run a MR job which should be used s3.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16268051#comment-16268051 ] Yonger commented on HADOOP-14475: - [~mackrorysd] So our final output format of record doesn't include any metric source info? if yes, I think it is not friendly to make a statistic chart like via InfluxDB+Grafana, because only fsId is unique in a record, especially there are multiple metric sources registered with the same bucket(I don't know why, but they were exist in my test), whose output records can't be easy to distinguish except fsid, then your chart in Grafana is hard readable. > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch, > HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch, > HADOOP-14475.009.patch, HADOOP-14475.010.patch, HADOOP-14475.011.patch, > HADOOP-14475.012.patch, HADOOP-14475.013.patch, HADOOP-14475.014.patch, > HADOOP-14475.015.patch, HADOOP-14775.007.patch, failsafe-report-s3a-it.html, > failsafe-report-s3a-scale.html, failsafe-report-scale.html, > failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16266866#comment-16266866 ] Yonger commented on HADOOP-14475: - azure just give the registry name to the record, which is confused. We can't distinguish the different records if there are more than one relative metric source registered within a process, what I given the example in previous comments. > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch, > HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch, > HADOOP-14475.009.patch, HADOOP-14475.010.patch, HADOOP-14475.011.patch, > HADOOP-14475.012.patch, HADOOP-14475.013.patch, HADOOP-14775.007.patch, > failsafe-report-s3a-it.html, failsafe-report-s3a-scale.html, > failsafe-report-scale.html, failsafe-report-scale.zip, s3a-metrics.patch1, > stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14943) Add common getFileBlockLocations() emulation for object stores, including S3A
[
https://issues.apache.org/jira/browse/HADOOP-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16266856#comment-16266856
]
Yonger commented on HADOOP-14943:
-
[~ste...@apache.org]I remember there are some discussion about how to configure
the fake host list, such as returning endpoint, compute hosts and a star, is
this right? I am not sure whether i understand these points totally.
I just test these four cases with 1TB dataset on query42 of TPC-DS, results are
below(seconds):
||default localhost||endpoint||star||compute host list||
|16|16l 16|28|
>From this result, performance are equal in these cases except returning
>compute host list.
> Add common getFileBlockLocations() emulation for object stores, including S3A
> -
>
> Key: HADOOP-14943
> URL: https://issues.apache.org/jira/browse/HADOOP-14943
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14943-001.patch, HADOOP-14943-002.patch,
> HADOOP-14943-002.patch, HADOOP-14943-003.patch
>
>
> It looks suspiciously like S3A isn't providing the partitioning data needed
> in {{listLocatedStatus}} and {{getFileBlockLocations()}} needed to break up a
> file by the blocksize. This will stop tools using the MRv1 APIS doing the
> partitioning properly if the input format isn't doing it own split logic.
> FileInputFormat in MRv2 is a bit more configurable about input split
> calculation & will split up large files. but otherwise, the partitioning is
> being done more by the default values of the executing engine, rather than
> any config data from the filesystem about what its "block size" is,
> NativeAzureFS does a better job; maybe that could be factored out to
> hadoop-common and reused?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14943) Add common getFileBlockLocations() emulation for object stores, including S3A
[
https://issues.apache.org/jira/browse/HADOOP-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16266856#comment-16266856
]
Yonger edited comment on HADOOP-14943 at 11/27/17 2:15 PM:
---
[~ste...@apache.org]I remember there are some discussion about how to configure
the fake host list, such as returning endpoint, compute hosts and a star, is
this right? I am not sure whether i understand these points totally.
I just test these four cases with 1TB dataset on query42 of TPC-DS, results are
below(seconds):
||default localhost||endpoint||star||compute host list||
|16|16|16|28|
>From this result, performance are equal in these cases except returning
>compute host list.
was (Author: iyonger):
[~ste...@apache.org]I remember there are some discussion about how to configure
the fake host list, such as returning endpoint, compute hosts and a star, is
this right? I am not sure whether i understand these points totally.
I just test these four cases with 1TB dataset on query42 of TPC-DS, results are
below(seconds):
||default localhost||endpoint||star||compute host list||
|16|16l 16|28|
>From this result, performance are equal in these cases except returning
>compute host list.
> Add common getFileBlockLocations() emulation for object stores, including S3A
> -
>
> Key: HADOOP-14943
> URL: https://issues.apache.org/jira/browse/HADOOP-14943
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14943-001.patch, HADOOP-14943-002.patch,
> HADOOP-14943-002.patch, HADOOP-14943-003.patch
>
>
> It looks suspiciously like S3A isn't providing the partitioning data needed
> in {{listLocatedStatus}} and {{getFileBlockLocations()}} needed to break up a
> file by the blocksize. This will stop tools using the MRv1 APIS doing the
> partitioning properly if the input format isn't doing it own split logic.
> FileInputFormat in MRv2 is a bit more configurable about input split
> calculation & will split up large files. but otherwise, the partitioning is
> being done more by the default values of the executing engine, rather than
> any config data from the filesystem about what its "block size" is,
> NativeAzureFS does a better job; maybe that could be factored out to
> hadoop-common and reused?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[
https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16260436#comment-16260436
]
Yonger commented on HADOOP-14475:
-
[~mackrorysd]Thanks for refining the code, that looks pretty good for me.
The only thing want to discuss with you:
>From the latest code,
{code:java}
+String msName = METRICS_SOURCE_BASENAME + number;
+metricsSourceName = msName + "-" + name.getHost();
+this.recordName = metricsSourceName;
{code}
I think we don't need to add name.getHost to record name, because each record
has a field "fsURI" that include the host name/bucket name.
> Metrics of S3A don't print out when enable it in Hadoop metrics property file
> --
>
> Key: HADOOP-14475
> URL: https://issues.apache.org/jira/browse/HADOOP-14475
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.8.0
> Environment: uname -a
> Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017
> x86_64 x86_64 x86_64 GNU/Linux
> cat /etc/issue
> Ubuntu 16.04.2 LTS \n \l
>Reporter: Yonger
>Assignee: Yonger
> Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch,
> HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch,
> HADOOP-14475.009.patch, HADOOP-14475.010.patch, HADOOP-14475.011.patch,
> HADOOP-14475.012.patch, HADOOP-14475.013.patch, HADOOP-14775.007.patch,
> failsafe-report-s3a-it.html, failsafe-report-s3a-scale.html,
> failsafe-report-scale.html, failsafe-report-scale.zip, s3a-metrics.patch1,
> stdout.zip
>
>
> *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink
> #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #*.sink.influxdb.url=http:/xx
> #*.sink.influxdb.influxdb_port=8086
> #*.sink.influxdb.database=hadoop
> #*.sink.influxdb.influxdb_username=hadoop
> #*.sink.influxdb.influxdb_password=hadoop
> #*.sink.ingluxdb.cluster=c1
> *.period=10
> #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out
> I can't find the out put file even i run a MR job which should be used s3.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16252976#comment-16252976 ] Yonger commented on HADOOP-14475: - [~mackrorysd]Thank you. > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch, > HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch, > HADOOP-14775.007.patch, failsafe-report-s3a-it.html, > failsafe-report-s3a-scale.html, failsafe-report-scale.html, > failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[
https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16252804#comment-16252804
]
Yonger edited comment on HADOOP-14475 at 11/15/17 1:44 AM:
---
For the # 2 point of my update
--
without this update,the metrics be flush into file like this(each record with
same URI mark the same "context+registry name" as begin):
{code:java}
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
Note, these three records written at the same time instead of different
intervals, but they should be consider three different kinds of
metrics(different fsid,different values in real logs).
(Additional, I don't know why the metrics be registered many times within a
process,like why we need to method "newMetricsSourceName")
with this update, each record will be easy to distinguish:
{code:java}
1510631110244 s3afs.S3aFileSystemMetrics: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics2: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics3: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
was (Author: iyonger):
For the # 2 point of my update
--
without this update,the metrics be flush into file like this(each record with
same URI mark the same "context+registry name" as begin):
{code:java}
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
Note, these three records written at the same time instead of different
intervals, so they should be consider three different kinds of metrics.
(Additional, I don't know why the metrics be registered many times within a
process,like why we need to method "newMetricsSourceName")
with this update, each record will be easy to distinguish:
{code:java}
1510631110244 s3afs.S3aFileSystemMetrics: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics2: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics3: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
> Metrics of S3A don't print out when enable it in Hadoop metrics property file
> --
>
> Key: HADOOP-14475
> URL: https://issues.apache.org/jira/browse/HADOOP-14475
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.8.0
> Environment: uname -a
> Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017
> x86_64 x86_64 x86_64 GNU/Linux
> cat /etc/issue
> Ubuntu 16.04.2 LTS \n \l
>Reporter: Yonger
>Assignee: Yonger
> Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch,
> HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch,
> HADOOP-14775.007.patch, failsafe-report-s3a-it.html,
> failsafe-report-s3a-scale.html, failsafe-report-scale.html,
> failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip
>
>
> *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink
> #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #*.sink.influxdb.url=http:/xx
> #*.sink.influxdb.influxdb_port=8086
> #*.sink.influxdb.database=hadoop
> #*.sink.influxdb.influxdb_username=hadoop
> #*.sink.influxdb.influxdb_password=hadoop
> #*.sink.ingluxdb.cluster=c1
> *.period=10
> #namenode.si
[jira] [Comment Edited] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[
https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16252804#comment-16252804
]
Yonger edited comment on HADOOP-14475 at 11/15/17 1:42 AM:
---
For the # 2 point of my update
--
without this update,the metrics be flush into file like this(each record with
same URI mark the same "context+registry name" as begin):
{code:java}
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
Note, these three records written at the same time instead of different
intervals, so they should be consider three different kinds of metrics.
(Additional, I don't know why the metrics be registered many times within a
process,like why we need to method "newMetricsSourceName")
with this update, each record will be easy to distinguish:
{code:java}
1510631110244 s3afs.S3aFileSystemMetrics: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics2: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics3: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
was (Author: iyonger):
For the # 2 point of my update
--
without this update,the metrics be flush into file like this(each record with
same URI mark the same "context+registry name" as begin):
{code:java}
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
with this update, each record will be easy to distinguish:
{code:java}
1510631110244 s3afs.S3aFileSystemMetrics: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics2: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics3: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
> Metrics of S3A don't print out when enable it in Hadoop metrics property file
> --
>
> Key: HADOOP-14475
> URL: https://issues.apache.org/jira/browse/HADOOP-14475
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.8.0
> Environment: uname -a
> Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017
> x86_64 x86_64 x86_64 GNU/Linux
> cat /etc/issue
> Ubuntu 16.04.2 LTS \n \l
>Reporter: Yonger
>Assignee: Yonger
> Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch,
> HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch,
> HADOOP-14775.007.patch, failsafe-report-s3a-it.html,
> failsafe-report-s3a-scale.html, failsafe-report-scale.html,
> failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip
>
>
> *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink
> #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #*.sink.influxdb.url=http:/xx
> #*.sink.influxdb.influxdb_port=8086
> #*.sink.influxdb.database=hadoop
> #*.sink.influxdb.influxdb_username=hadoop
> #*.sink.influxdb.influxdb_password=hadoop
> #*.sink.ingluxdb.cluster=c1
> *.period=10
> #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out
> I can't find the out put file even i run a MR job which should be used s3.
--
This message was sent
[jira] [Comment Edited] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[
https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16252804#comment-16252804
]
Yonger edited comment on HADOOP-14475 at 11/15/17 1:31 AM:
---
For the # 2 point of my update
--
without this update,the metrics be flush into file like this(each record with
same URI mark the same "context+registry name" as begin):
{code:java}
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
with this update, each record will be easy to distinguish:
{code:java}
1510631110244 s3afs.S3aFileSystemMetrics: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics2: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics3: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
was (Author: iyonger):
For the # 2 point of my update
--
without this update,the metrics be flush into file like this(each record mark
the same "context+registry name" as begin):
{code:java}
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
with this update, each record will be easy to distinguish:
{code:java}
1510631110244 s3afs.S3aFileSystemMetrics: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics2: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics3: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
> Metrics of S3A don't print out when enable it in Hadoop metrics property file
> --
>
> Key: HADOOP-14475
> URL: https://issues.apache.org/jira/browse/HADOOP-14475
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.8.0
> Environment: uname -a
> Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017
> x86_64 x86_64 x86_64 GNU/Linux
> cat /etc/issue
> Ubuntu 16.04.2 LTS \n \l
>Reporter: Yonger
>Assignee: Yonger
> Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch,
> HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch,
> HADOOP-14775.007.patch, failsafe-report-s3a-it.html,
> failsafe-report-s3a-scale.html, failsafe-report-scale.html,
> failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip
>
>
> *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink
> #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #*.sink.influxdb.url=http:/xx
> #*.sink.influxdb.influxdb_port=8086
> #*.sink.influxdb.database=hadoop
> #*.sink.influxdb.influxdb_username=hadoop
> #*.sink.influxdb.influxdb_password=hadoop
> #*.sink.ingluxdb.cluster=c1
> *.period=10
> #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out
> I can't find the out put file even i run a MR job which should be used s3.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[
https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16252804#comment-16252804
]
Yonger commented on HADOOP-14475:
-
For the # 2 point of my update
--
without this update,the metrics be flush into file like this(each record mark
the same "context+registry name" as begin):
{code:java}
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
with this update, each record will be easy to distinguish:
{code:java}
1510631110244 s3afs.S3aFileSystemMetrics: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics2: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics3: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
> Metrics of S3A don't print out when enable it in Hadoop metrics property file
> --
>
> Key: HADOOP-14475
> URL: https://issues.apache.org/jira/browse/HADOOP-14475
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.8.0
> Environment: uname -a
> Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017
> x86_64 x86_64 x86_64 GNU/Linux
> cat /etc/issue
> Ubuntu 16.04.2 LTS \n \l
>Reporter: Yonger
>Assignee: Yonger
> Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch,
> HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch,
> HADOOP-14775.007.patch, failsafe-report-s3a-it.html,
> failsafe-report-s3a-scale.html, failsafe-report-scale.html,
> failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip
>
>
> *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink
> #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #*.sink.influxdb.url=http:/xx
> #*.sink.influxdb.influxdb_port=8086
> #*.sink.influxdb.database=hadoop
> #*.sink.influxdb.influxdb_username=hadoop
> #*.sink.influxdb.influxdb_password=hadoop
> #*.sink.ingluxdb.cluster=c1
> *.period=10
> #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out
> I can't find the out put file even i run a MR job which should be used s3.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16252798#comment-16252798 ] Yonger commented on HADOOP-14475: - [~macdonsp]the .008 patch is newer than .007 patch, I think it's the correct code that I want to give you guys. > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch, > HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch, > HADOOP-14775.007.patch, failsafe-report-s3a-it.html, > failsafe-report-s3a-scale.html, failsafe-report-scale.html, > failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16251020#comment-16251020 ] Yonger commented on HADOOP-14475: - [~mackrorysd] Thank you very much. For the test case, I removed the assertion for file status counter while it's not necessary. For hard cord, that I copy from some other place(maybe old azure fs implementation, forgot), now I don't use fix length string to append, so this hard code also removed. At last, I just only put this setting into metrics property file: s3afilesystem.sink.file.filename=s3afilesystem-metrics.out which would print all metrics info into the specify file. > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch, > HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch, > HADOOP-14775.007.patch, failsafe-report-s3a-it.html, > failsafe-report-s3a-scale.html, failsafe-report-scale.html, > failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yonger updated HADOOP-14475: Attachment: HADOOP-14475.008.patch update: 1.chang prefix and context name to dfs style 2.give each record a different name by the generated metric source name 3.add s3a metric system destroy logic in file system close method > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch, > HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch, > HADOOP-14775.007.patch, failsafe-report-s3a-it.html, > failsafe-report-s3a-scale.html, failsafe-report-scale.html, > failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15000) s3a new getdefaultblocksize be called in getFileStatus which has not been implemented in s3afilesystem yet
[
https://issues.apache.org/jira/browse/HADOOP-15000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16235037#comment-16235037
]
Yonger commented on HADOOP-15000:
-
Thanks for your detail interpretation.
But s3a has it's own configuration for block size, if we don't implement the
newer, callers will use default dfs block size instead.
So we still need to add the newer implementation into s3a code, right ?
> s3a new getdefaultblocksize be called in getFileStatus which has not been
> implemented in s3afilesystem yet
> --
>
> Key: HADOOP-15000
> URL: https://issues.apache.org/jira/browse/HADOOP-15000
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Affects Versions: 2.9.0
>Reporter: Yonger
>Priority: Minor
>
> new implementation of getting block size has been called in getFileStatus
> method:
> {code:java}
> return new S3AFileStatus(meta.getContentLength(),
> dateToLong(meta.getLastModified()),
> path,
> getDefaultBlockSize(path),
> username);
> }
> {code}
> while we don't implement it in our s3afilesystem currently, also we need to
> implement this new method as the old one deprecated.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15000) s3a new getdefaultblocksize be called in getFileStatus which has not been implemented in s3afilesystem yet
Yonger created HADOOP-15000:
---
Summary: s3a new getdefaultblocksize be called in getFileStatus
which has not been implemented in s3afilesystem yet
Key: HADOOP-15000
URL: https://issues.apache.org/jira/browse/HADOOP-15000
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Reporter: Yonger
Priority: Normal
new implementation of getting block size has been called in getFileStatus
method:
{code:java}
return new S3AFileStatus(meta.getContentLength(),
dateToLong(meta.getLastModified()),
path,
getDefaultBlockSize(path),
username);
}
{code}
while we don't implement it in our s3afilesystem currently, also we need to
implement this new method as the old one deprecated.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yonger updated HADOOP-14475: Status: Patch Available (was: Open) > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch, > HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14775.007.patch, > failsafe-report-s3a-it.html, failsafe-report-s3a-scale.html, > failsafe-report-scale.html, failsafe-report-scale.zip, s3a-metrics.patch1, > stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yonger updated HADOOP-14475: Attachment: HADOOP-14775.007.patch based trunk. > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch, > HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14775.007.patch, > failsafe-report-s3a-it.html, failsafe-report-s3a-scale.html, > failsafe-report-scale.html, failsafe-report-scale.zip, s3a-metrics.patch1, > stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yonger updated HADOOP-14475: Status: Open (was: Patch Available) > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch, > HADOOP-14475.005.patch, HADOOP-14475.006.patch, failsafe-report-s3a-it.html, > failsafe-report-s3a-scale.html, failsafe-report-scale.html, > failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14770) S3A http connection in s3a driver not reuse in Spark application
[
https://issues.apache.org/jira/browse/HADOOP-14770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yonger resolved HADOOP-14770.
-
Resolution: Duplicate
Thanks Steve. when we apply the read input policy with random in our workload
after upgrade to Hadoop 2.8.1, it works as my expect, connections not destroyed
every time but reused.
> S3A http connection in s3a driver not reuse in Spark application
>
>
> Key: HADOOP-14770
> URL: https://issues.apache.org/jira/browse/HADOOP-14770
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.7.3
>Reporter: Yonger
>Assignee: Yonger
>Priority: Minor
>
> I print out connection stats every 2 s when running Spark application against
> s3-compatible storage:
> {code}
> ESTAB 0 0 :::10.0.2.36:6
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44454
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44374
> :::10.0.2.254:80
> ESTAB 159724 0 :::10.0.2.36:44436
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:8
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44338
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44438
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44414
> :::10.0.2.254:80
> ESTAB 0 480 :::10.0.2.36:44450
> :::10.0.2.254:80 timer:(on,170ms,0)
> ESTAB 0 0 :::10.0.2.36:2
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44390
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44326
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44452
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44394
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:4
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44456
> :::10.0.2.254:80
> ==
> ESTAB 0 0 :::10.0.2.36:44508
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44476
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44524
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44374
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44500
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44504
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44512
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44506
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44464
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44518
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44510
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:2
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44526
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44472
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44466
> :::10.0.2.254:80
> {code}
> the connection in the above of "=" and below were changed all the time. But
> this haven't seen in MR application.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14774) S3A case "testRandomReadOverBuffer" failed due to improper range parameter
[
https://issues.apache.org/jira/browse/HADOOP-14774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16156959#comment-16156959
]
Yonger commented on HADOOP-14774:
-
[~ste...@apache.org] Is the latest patch OK?
> S3A case "testRandomReadOverBuffer" failed due to improper range parameter
> --
>
> Key: HADOOP-14774
> URL: https://issues.apache.org/jira/browse/HADOOP-14774
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
> Environment: Hadoop 2.8.0
> s3-compatible storage
>Reporter: Yonger
>Assignee: Yonger
>Priority: Minor
> Attachments: failsafe-report.html, HADOOP-14774.001.patch,
> HADOOP-14774.002.patch, HADOOP-14774.003.patch
>
>
> {code:java}
> Running org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.821 sec <<<
> FAILURE! - in org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
> testRandomReadOverBuffer(org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance)
> Time elapsed: 2.605 sec <<< FAILURE!
> java.lang.AssertionError: expected:<8192> but was:<8193>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance.testRandomReadOverBuffer(ITestS3AInputStreamPerformance.java:533)
> {code}
> From log, the length of content is exceed than our expect:
> {code:java}
> 2017-08-15 16:53:16,464 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(280)) - >> GET
> /test-aws-s3a/test/testReadOverBuffer.bin HTTP/1.1
> 2017-08-15 16:53:16,464 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Host: 10.0.2.254
> 2017-08-15 16:53:16,465 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >>
> x-amz-content-sha256:
> e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
> 2017-08-15 16:53:16,465 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Authorization:
> AWS4-HMAC-SHA256
> Credential=JFDAM9KF9IY8S5P0JIV6/20170815/us-east-1/s3/aws4_request,
> SignedHeaders=content-type;host;range;user-agent;x-amz-content-sha256;x-amz-date,
> Signature=42bce4a43d2b1bf6e6d599613c60812e6716514da4ef5b3839ef0566c31279ee
> 2017-08-15 16:53:16,466 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> X-Amz-Date:
> 20170815T085316Z
> 2017-08-15 16:53:16,466 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> User-Agent: Hadoop
> 2.8.0, aws-sdk-java/1.10.6 Linux/3.10.0-514.21.2.el7.x86_64
> Java_HotSpot(TM)_64-Bit_Server_VM/25.131-b11/1.8.0_131
> 2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Range: bytes=0-8192
> 2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Content-Type:
> application/x-www-form-urlencoded; charset=utf-8
> 2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Connection:
> Keep-Alive
> 2017-08-15 16:53:16,473 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "HTTP/1.1 206 Partial Content[\r][\n]"
> 2017-08-15 16:53:16,475 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Content-Range: bytes 0-8192/32768[\r][\n]"
> 2017-08-15 16:53:16,476 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Content-Length: 8193[\r][\n]"
> 2017-08-15 16:53:16,477 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Accept-Ranges: bytes[\r][\n]"
> 2017-08-15 16:53:16,478 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Last-Modified: Tue, 15 Aug 2017 08:51:39
> GMT[\r][\n]"
> 2017-08-15 16:53:16,479 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "ETag: "e7191764798ba504d6671d4c434d2f4d"[\r][\n]"
> 2017-08-15 16:53:16,480 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "x-amz-request-id:
> tx0001e-005992b67e-27a45-default[\r][\n]"
> 2017-08-15 16:53:16,481 [JUnit-testRandomRe
[jira] [Updated] (HADOOP-14774) S3A case "testRandomReadOverBuffer" failed due to improper range parameter
[
https://issues.apache.org/jira/browse/HADOOP-14774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yonger updated HADOOP-14774:
Attachment: HADOOP-14774.003.patch
> S3A case "testRandomReadOverBuffer" failed due to improper range parameter
> --
>
> Key: HADOOP-14774
> URL: https://issues.apache.org/jira/browse/HADOOP-14774
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
> Environment: Hadoop 2.8.0
> s3-compatible storage
>Reporter: Yonger
>Assignee: Yonger
>Priority: Minor
> Attachments: failsafe-report.html, HADOOP-14774.001.patch,
> HADOOP-14774.002.patch, HADOOP-14774.003.patch
>
>
> {code:java}
> Running org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.821 sec <<<
> FAILURE! - in org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
> testRandomReadOverBuffer(org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance)
> Time elapsed: 2.605 sec <<< FAILURE!
> java.lang.AssertionError: expected:<8192> but was:<8193>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance.testRandomReadOverBuffer(ITestS3AInputStreamPerformance.java:533)
> {code}
> From log, the length of content is exceed than our expect:
> {code:java}
> 2017-08-15 16:53:16,464 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(280)) - >> GET
> /test-aws-s3a/test/testReadOverBuffer.bin HTTP/1.1
> 2017-08-15 16:53:16,464 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Host: 10.0.2.254
> 2017-08-15 16:53:16,465 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >>
> x-amz-content-sha256:
> e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
> 2017-08-15 16:53:16,465 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Authorization:
> AWS4-HMAC-SHA256
> Credential=JFDAM9KF9IY8S5P0JIV6/20170815/us-east-1/s3/aws4_request,
> SignedHeaders=content-type;host;range;user-agent;x-amz-content-sha256;x-amz-date,
> Signature=42bce4a43d2b1bf6e6d599613c60812e6716514da4ef5b3839ef0566c31279ee
> 2017-08-15 16:53:16,466 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> X-Amz-Date:
> 20170815T085316Z
> 2017-08-15 16:53:16,466 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> User-Agent: Hadoop
> 2.8.0, aws-sdk-java/1.10.6 Linux/3.10.0-514.21.2.el7.x86_64
> Java_HotSpot(TM)_64-Bit_Server_VM/25.131-b11/1.8.0_131
> 2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Range: bytes=0-8192
> 2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Content-Type:
> application/x-www-form-urlencoded; charset=utf-8
> 2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Connection:
> Keep-Alive
> 2017-08-15 16:53:16,473 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "HTTP/1.1 206 Partial Content[\r][\n]"
> 2017-08-15 16:53:16,475 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Content-Range: bytes 0-8192/32768[\r][\n]"
> 2017-08-15 16:53:16,476 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Content-Length: 8193[\r][\n]"
> 2017-08-15 16:53:16,477 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Accept-Ranges: bytes[\r][\n]"
> 2017-08-15 16:53:16,478 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Last-Modified: Tue, 15 Aug 2017 08:51:39
> GMT[\r][\n]"
> 2017-08-15 16:53:16,479 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "ETag: "e7191764798ba504d6671d4c434d2f4d"[\r][\n]"
> 2017-08-15 16:53:16,480 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "x-amz-request-id:
> tx0001e-005992b67e-27a45-default[\r][\n]"
> 2017-08-15 16:53:16,481 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Con
[jira] [Updated] (HADOOP-14774) S3A case "testRandomReadOverBuffer" failed due to improper range parameter
[
https://issues.apache.org/jira/browse/HADOOP-14774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yonger updated HADOOP-14774:
Attachment: HADOOP-14774.002.patch
> S3A case "testRandomReadOverBuffer" failed due to improper range parameter
> --
>
> Key: HADOOP-14774
> URL: https://issues.apache.org/jira/browse/HADOOP-14774
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
> Environment: Hadoop 2.8.0
> s3-compatible storage
>Reporter: Yonger
>Assignee: Yonger
>Priority: Minor
> Attachments: failsafe-report.html, HADOOP-14774.001.patch,
> HADOOP-14774.002.patch
>
>
> {code:java}
> Running org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.821 sec <<<
> FAILURE! - in org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
> testRandomReadOverBuffer(org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance)
> Time elapsed: 2.605 sec <<< FAILURE!
> java.lang.AssertionError: expected:<8192> but was:<8193>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance.testRandomReadOverBuffer(ITestS3AInputStreamPerformance.java:533)
> {code}
> From log, the length of content is exceed than our expect:
> {code:java}
> 2017-08-15 16:53:16,464 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(280)) - >> GET
> /test-aws-s3a/test/testReadOverBuffer.bin HTTP/1.1
> 2017-08-15 16:53:16,464 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Host: 10.0.2.254
> 2017-08-15 16:53:16,465 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >>
> x-amz-content-sha256:
> e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
> 2017-08-15 16:53:16,465 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Authorization:
> AWS4-HMAC-SHA256
> Credential=JFDAM9KF9IY8S5P0JIV6/20170815/us-east-1/s3/aws4_request,
> SignedHeaders=content-type;host;range;user-agent;x-amz-content-sha256;x-amz-date,
> Signature=42bce4a43d2b1bf6e6d599613c60812e6716514da4ef5b3839ef0566c31279ee
> 2017-08-15 16:53:16,466 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> X-Amz-Date:
> 20170815T085316Z
> 2017-08-15 16:53:16,466 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> User-Agent: Hadoop
> 2.8.0, aws-sdk-java/1.10.6 Linux/3.10.0-514.21.2.el7.x86_64
> Java_HotSpot(TM)_64-Bit_Server_VM/25.131-b11/1.8.0_131
> 2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Range: bytes=0-8192
> 2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Content-Type:
> application/x-www-form-urlencoded; charset=utf-8
> 2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Connection:
> Keep-Alive
> 2017-08-15 16:53:16,473 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "HTTP/1.1 206 Partial Content[\r][\n]"
> 2017-08-15 16:53:16,475 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Content-Range: bytes 0-8192/32768[\r][\n]"
> 2017-08-15 16:53:16,476 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Content-Length: 8193[\r][\n]"
> 2017-08-15 16:53:16,477 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Accept-Ranges: bytes[\r][\n]"
> 2017-08-15 16:53:16,478 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Last-Modified: Tue, 15 Aug 2017 08:51:39
> GMT[\r][\n]"
> 2017-08-15 16:53:16,479 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "ETag: "e7191764798ba504d6671d4c434d2f4d"[\r][\n]"
> 2017-08-15 16:53:16,480 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "x-amz-request-id:
> tx0001e-005992b67e-27a45-default[\r][\n]"
> 2017-08-15 16:53:16,481 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Content-Type: application/o
[jira] [Updated] (HADOOP-14774) S3A case "testRandomReadOverBuffer" failed due to improper range parameter
[
https://issues.apache.org/jira/browse/HADOOP-14774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yonger updated HADOOP-14774:
Attachment: failsafe-report.html
test against on Ceph object store over s3a.
> S3A case "testRandomReadOverBuffer" failed due to improper range parameter
> --
>
> Key: HADOOP-14774
> URL: https://issues.apache.org/jira/browse/HADOOP-14774
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
> Environment: Hadoop 2.8.0
> s3-compatible storage
>Reporter: Yonger
>Assignee: Yonger
>Priority: Minor
> Attachments: failsafe-report.html, HADOOP-14774.001.patch
>
>
> {code:java}
> Running org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.821 sec <<<
> FAILURE! - in org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
> testRandomReadOverBuffer(org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance)
> Time elapsed: 2.605 sec <<< FAILURE!
> java.lang.AssertionError: expected:<8192> but was:<8193>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance.testRandomReadOverBuffer(ITestS3AInputStreamPerformance.java:533)
> {code}
> From log, the length of content is exceed than our expect:
> {code:java}
> 2017-08-15 16:53:16,464 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(280)) - >> GET
> /test-aws-s3a/test/testReadOverBuffer.bin HTTP/1.1
> 2017-08-15 16:53:16,464 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Host: 10.0.2.254
> 2017-08-15 16:53:16,465 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >>
> x-amz-content-sha256:
> e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
> 2017-08-15 16:53:16,465 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Authorization:
> AWS4-HMAC-SHA256
> Credential=JFDAM9KF9IY8S5P0JIV6/20170815/us-east-1/s3/aws4_request,
> SignedHeaders=content-type;host;range;user-agent;x-amz-content-sha256;x-amz-date,
> Signature=42bce4a43d2b1bf6e6d599613c60812e6716514da4ef5b3839ef0566c31279ee
> 2017-08-15 16:53:16,466 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> X-Amz-Date:
> 20170815T085316Z
> 2017-08-15 16:53:16,466 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> User-Agent: Hadoop
> 2.8.0, aws-sdk-java/1.10.6 Linux/3.10.0-514.21.2.el7.x86_64
> Java_HotSpot(TM)_64-Bit_Server_VM/25.131-b11/1.8.0_131
> 2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Range: bytes=0-8192
> 2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Content-Type:
> application/x-www-form-urlencoded; charset=utf-8
> 2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Connection:
> Keep-Alive
> 2017-08-15 16:53:16,473 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "HTTP/1.1 206 Partial Content[\r][\n]"
> 2017-08-15 16:53:16,475 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Content-Range: bytes 0-8192/32768[\r][\n]"
> 2017-08-15 16:53:16,476 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Content-Length: 8193[\r][\n]"
> 2017-08-15 16:53:16,477 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Accept-Ranges: bytes[\r][\n]"
> 2017-08-15 16:53:16,478 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Last-Modified: Tue, 15 Aug 2017 08:51:39
> GMT[\r][\n]"
> 2017-08-15 16:53:16,479 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "ETag: "e7191764798ba504d6671d4c434d2f4d"[\r][\n]"
> 2017-08-15 16:53:16,480 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "x-amz-request-id:
> tx0001e-005992b67e-27a45-default[\r][\n]"
> 2017-08-15 16:53:16,481 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Content-Typ
[jira] [Updated] (HADOOP-14774) S3A case "testRandomReadOverBuffer" failed due to improper range parameter
[
https://issues.apache.org/jira/browse/HADOOP-14774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yonger updated HADOOP-14774:
Attachment: HADOOP-14774.001.patch
upload patch first since my s3-compatible storage backend not works well, then
i will upload my test results if the storage come back.
> S3A case "testRandomReadOverBuffer" failed due to improper range parameter
> --
>
> Key: HADOOP-14774
> URL: https://issues.apache.org/jira/browse/HADOOP-14774
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
> Environment: Hadoop 2.8.0
> s3-compatible storage
>Reporter: Yonger
>Assignee: Yonger
>Priority: Minor
> Attachments: HADOOP-14774.001.patch
>
>
> {code:java}
> Running org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.821 sec <<<
> FAILURE! - in org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
> testRandomReadOverBuffer(org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance)
> Time elapsed: 2.605 sec <<< FAILURE!
> java.lang.AssertionError: expected:<8192> but was:<8193>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance.testRandomReadOverBuffer(ITestS3AInputStreamPerformance.java:533)
> {code}
> From log, the length of content is exceed than our expect:
> {code:java}
> 2017-08-15 16:53:16,464 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(280)) - >> GET
> /test-aws-s3a/test/testReadOverBuffer.bin HTTP/1.1
> 2017-08-15 16:53:16,464 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Host: 10.0.2.254
> 2017-08-15 16:53:16,465 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >>
> x-amz-content-sha256:
> e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
> 2017-08-15 16:53:16,465 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Authorization:
> AWS4-HMAC-SHA256
> Credential=JFDAM9KF9IY8S5P0JIV6/20170815/us-east-1/s3/aws4_request,
> SignedHeaders=content-type;host;range;user-agent;x-amz-content-sha256;x-amz-date,
> Signature=42bce4a43d2b1bf6e6d599613c60812e6716514da4ef5b3839ef0566c31279ee
> 2017-08-15 16:53:16,466 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> X-Amz-Date:
> 20170815T085316Z
> 2017-08-15 16:53:16,466 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> User-Agent: Hadoop
> 2.8.0, aws-sdk-java/1.10.6 Linux/3.10.0-514.21.2.el7.x86_64
> Java_HotSpot(TM)_64-Bit_Server_VM/25.131-b11/1.8.0_131
> 2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Range: bytes=0-8192
> 2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Content-Type:
> application/x-www-form-urlencoded; charset=utf-8
> 2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Connection:
> Keep-Alive
> 2017-08-15 16:53:16,473 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "HTTP/1.1 206 Partial Content[\r][\n]"
> 2017-08-15 16:53:16,475 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Content-Range: bytes 0-8192/32768[\r][\n]"
> 2017-08-15 16:53:16,476 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Content-Length: 8193[\r][\n]"
> 2017-08-15 16:53:16,477 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Accept-Ranges: bytes[\r][\n]"
> 2017-08-15 16:53:16,478 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Last-Modified: Tue, 15 Aug 2017 08:51:39
> GMT[\r][\n]"
> 2017-08-15 16:53:16,479 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "ETag: "e7191764798ba504d6671d4c434d2f4d"[\r][\n]"
> 2017-08-15 16:53:16,480 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "x-amz-request-id:
> tx0001e-005992b67e-27a45-default[\r][\n]"
> 2017-08-15 16:53:16,481 [JUnit-testRandomRe
[jira] [Issue Comment Deleted] (HADOOP-14774) S3A case "testRandomReadOverBuffer" failed due to improper range parameter
[
https://issues.apache.org/jira/browse/HADOOP-14774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yonger updated HADOOP-14774:
Comment: was deleted
(was: I saw the doc of withRange said :
* The first byte in an object has
* position 0; as an example, the first ten bytes of an object can be
* downloaded by specifying a range of 0 to 9.
So, s3 and s3-compatible store should follow this semantics, doesn't it? and
this is the issue?)
> S3A case "testRandomReadOverBuffer" failed due to improper range parameter
> --
>
> Key: HADOOP-14774
> URL: https://issues.apache.org/jira/browse/HADOOP-14774
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
> Environment: Hadoop 2.8.0
> s3-compatible storage
>Reporter: Yonger
>Assignee: Yonger
>Priority: Minor
>
> {code:java}
> Running org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.821 sec <<<
> FAILURE! - in org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
> testRandomReadOverBuffer(org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance)
> Time elapsed: 2.605 sec <<< FAILURE!
> java.lang.AssertionError: expected:<8192> but was:<8193>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance.testRandomReadOverBuffer(ITestS3AInputStreamPerformance.java:533)
> {code}
> From log, the length of content is exceed than our expect:
> {code:java}
> 2017-08-15 16:53:16,464 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(280)) - >> GET
> /test-aws-s3a/test/testReadOverBuffer.bin HTTP/1.1
> 2017-08-15 16:53:16,464 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Host: 10.0.2.254
> 2017-08-15 16:53:16,465 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >>
> x-amz-content-sha256:
> e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
> 2017-08-15 16:53:16,465 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Authorization:
> AWS4-HMAC-SHA256
> Credential=JFDAM9KF9IY8S5P0JIV6/20170815/us-east-1/s3/aws4_request,
> SignedHeaders=content-type;host;range;user-agent;x-amz-content-sha256;x-amz-date,
> Signature=42bce4a43d2b1bf6e6d599613c60812e6716514da4ef5b3839ef0566c31279ee
> 2017-08-15 16:53:16,466 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> X-Amz-Date:
> 20170815T085316Z
> 2017-08-15 16:53:16,466 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> User-Agent: Hadoop
> 2.8.0, aws-sdk-java/1.10.6 Linux/3.10.0-514.21.2.el7.x86_64
> Java_HotSpot(TM)_64-Bit_Server_VM/25.131-b11/1.8.0_131
> 2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Range: bytes=0-8192
> 2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Content-Type:
> application/x-www-form-urlencoded; charset=utf-8
> 2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Connection:
> Keep-Alive
> 2017-08-15 16:53:16,473 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "HTTP/1.1 206 Partial Content[\r][\n]"
> 2017-08-15 16:53:16,475 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Content-Range: bytes 0-8192/32768[\r][\n]"
> 2017-08-15 16:53:16,476 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Content-Length: 8193[\r][\n]"
> 2017-08-15 16:53:16,477 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Accept-Ranges: bytes[\r][\n]"
> 2017-08-15 16:53:16,478 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Last-Modified: Tue, 15 Aug 2017 08:51:39
> GMT[\r][\n]"
> 2017-08-15 16:53:16,479 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "ETag: "e7191764798ba504d6671d4c434d2f4d"[\r][\n]"
> 2017-08-15 16:53:16,480 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "x-amz-request-id:
>
[jira] [Commented] (HADOOP-14774) S3A case "testRandomReadOverBuffer" failed due to improper range parameter
[
https://issues.apache.org/jira/browse/HADOOP-14774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16128838#comment-16128838
]
Yonger commented on HADOOP-14774:
-
I saw the doc of withRange said :
* The first byte in an object has
* position 0; as an example, the first ten bytes of an object can be
* downloaded by specifying a range of 0 to 9.
So, s3 and s3-compatible store should follow this semantics, doesn't it? and
this is the issue?
> S3A case "testRandomReadOverBuffer" failed due to improper range parameter
> --
>
> Key: HADOOP-14774
> URL: https://issues.apache.org/jira/browse/HADOOP-14774
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
> Environment: Hadoop 2.8.0
> s3-compatible storage
>Reporter: Yonger
>Assignee: Yonger
>Priority: Minor
>
> {code:java}
> Running org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.821 sec <<<
> FAILURE! - in org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
> testRandomReadOverBuffer(org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance)
> Time elapsed: 2.605 sec <<< FAILURE!
> java.lang.AssertionError: expected:<8192> but was:<8193>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance.testRandomReadOverBuffer(ITestS3AInputStreamPerformance.java:533)
> {code}
> From log, the length of content is exceed than our expect:
> {code:java}
> 2017-08-15 16:53:16,464 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(280)) - >> GET
> /test-aws-s3a/test/testReadOverBuffer.bin HTTP/1.1
> 2017-08-15 16:53:16,464 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Host: 10.0.2.254
> 2017-08-15 16:53:16,465 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >>
> x-amz-content-sha256:
> e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
> 2017-08-15 16:53:16,465 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Authorization:
> AWS4-HMAC-SHA256
> Credential=JFDAM9KF9IY8S5P0JIV6/20170815/us-east-1/s3/aws4_request,
> SignedHeaders=content-type;host;range;user-agent;x-amz-content-sha256;x-amz-date,
> Signature=42bce4a43d2b1bf6e6d599613c60812e6716514da4ef5b3839ef0566c31279ee
> 2017-08-15 16:53:16,466 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> X-Amz-Date:
> 20170815T085316Z
> 2017-08-15 16:53:16,466 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> User-Agent: Hadoop
> 2.8.0, aws-sdk-java/1.10.6 Linux/3.10.0-514.21.2.el7.x86_64
> Java_HotSpot(TM)_64-Bit_Server_VM/25.131-b11/1.8.0_131
> 2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Range: bytes=0-8192
> 2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Content-Type:
> application/x-www-form-urlencoded; charset=utf-8
> 2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Connection:
> Keep-Alive
> 2017-08-15 16:53:16,473 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "HTTP/1.1 206 Partial Content[\r][\n]"
> 2017-08-15 16:53:16,475 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Content-Range: bytes 0-8192/32768[\r][\n]"
> 2017-08-15 16:53:16,476 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Content-Length: 8193[\r][\n]"
> 2017-08-15 16:53:16,477 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Accept-Ranges: bytes[\r][\n]"
> 2017-08-15 16:53:16,478 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Last-Modified: Tue, 15 Aug 2017 08:51:39
> GMT[\r][\n]"
> 2017-08-15 16:53:16,479 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "ETag: "e7191764798ba504d6671d4c434d2f4d"[\r][\n]"
> 2017-08-15 16:53:16,480 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) -
[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16128548#comment-16128548 ] Yonger commented on HADOOP-14475: - [~ste...@apache.org] Can you help to verify and review this patch? > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: failsafe-report-s3a-it.html, > failsafe-report-s3a-scale.html, failsafe-report-scale.html, > failsafe-report-scale.zip, HADOOP-14475.002.patch, HADOOP-14475-003.patch, > HADOOP-14475.005.patch, HADOOP-14475.006.patch, s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14770) S3A http connection in s3a driver not reuse in Spark application
[
https://issues.apache.org/jira/browse/HADOOP-14770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16127218#comment-16127218
]
Yonger commented on HADOOP-14770:
-
Sorry, not yet. I am working with multiple partners on our big data cluster, so
it's not easy to move to 2.8. But I will complete it ASAP.
> S3A http connection in s3a driver not reuse in Spark application
>
>
> Key: HADOOP-14770
> URL: https://issues.apache.org/jira/browse/HADOOP-14770
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.7.3
>Reporter: Yonger
>Assignee: Yonger
>Priority: Minor
>
> I print out connection stats every 2 s when running Spark application against
> s3-compatible storage:
> {code}
> ESTAB 0 0 :::10.0.2.36:6
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44454
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44374
> :::10.0.2.254:80
> ESTAB 159724 0 :::10.0.2.36:44436
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:8
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44338
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44438
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44414
> :::10.0.2.254:80
> ESTAB 0 480 :::10.0.2.36:44450
> :::10.0.2.254:80 timer:(on,170ms,0)
> ESTAB 0 0 :::10.0.2.36:2
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44390
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44326
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44452
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44394
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:4
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44456
> :::10.0.2.254:80
> ==
> ESTAB 0 0 :::10.0.2.36:44508
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44476
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44524
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44374
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44500
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44504
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44512
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44506
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44464
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44518
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44510
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:2
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44526
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44472
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44466
> :::10.0.2.254:80
> {code}
> the connection in the above of "=" and below were changed all the time. But
> this haven't seen in MR application.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14774) S3A case "testRandomReadOverBuffer" failed due to improper range parameter
[
https://issues.apache.org/jira/browse/HADOOP-14774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16126990#comment-16126990
]
Yonger commented on HADOOP-14774:
-
{code:java}
GetObjectRequest request = new GetObjectRequest(bucket, key)
.withRange(targetPos, contentRangeFinish);
{code}
we should pass contentRangeFinish-1 instead of contentRangeFinish into
withRange method
> S3A case "testRandomReadOverBuffer" failed due to improper range parameter
> --
>
> Key: HADOOP-14774
> URL: https://issues.apache.org/jira/browse/HADOOP-14774
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.8.0
> Environment: Hadoop 2.8.0
> s3-compatible storage
>Reporter: Yonger
>Assignee: Yonger
>
> {code:java}
> Running org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.821 sec <<<
> FAILURE! - in org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
> testRandomReadOverBuffer(org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance)
> Time elapsed: 2.605 sec <<< FAILURE!
> java.lang.AssertionError: expected:<8192> but was:<8193>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance.testRandomReadOverBuffer(ITestS3AInputStreamPerformance.java:533)
> {code}
> From log, the length of content is exceed than our expect:
> {code:java}
> 2017-08-15 16:53:16,464 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(280)) - >> GET
> /test-aws-s3a/test/testReadOverBuffer.bin HTTP/1.1
> 2017-08-15 16:53:16,464 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Host: 10.0.2.254
> 2017-08-15 16:53:16,465 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >>
> x-amz-content-sha256:
> e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
> 2017-08-15 16:53:16,465 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Authorization:
> AWS4-HMAC-SHA256
> Credential=JFDAM9KF9IY8S5P0JIV6/20170815/us-east-1/s3/aws4_request,
> SignedHeaders=content-type;host;range;user-agent;x-amz-content-sha256;x-amz-date,
> Signature=42bce4a43d2b1bf6e6d599613c60812e6716514da4ef5b3839ef0566c31279ee
> 2017-08-15 16:53:16,466 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> X-Amz-Date:
> 20170815T085316Z
> 2017-08-15 16:53:16,466 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> User-Agent: Hadoop
> 2.8.0, aws-sdk-java/1.10.6 Linux/3.10.0-514.21.2.el7.x86_64
> Java_HotSpot(TM)_64-Bit_Server_VM/25.131-b11/1.8.0_131
> 2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Range: bytes=0-8192
> 2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Content-Type:
> application/x-www-form-urlencoded; charset=utf-8
> 2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
> (DefaultClientConnection.java:sendRequestHeader(283)) - >> Connection:
> Keep-Alive
> 2017-08-15 16:53:16,473 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "HTTP/1.1 206 Partial Content[\r][\n]"
> 2017-08-15 16:53:16,475 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Content-Range: bytes 0-8192/32768[\r][\n]"
> 2017-08-15 16:53:16,476 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Content-Length: 8193[\r][\n]"
> 2017-08-15 16:53:16,477 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Accept-Ranges: bytes[\r][\n]"
> 2017-08-15 16:53:16,478 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "Last-Modified: Tue, 15 Aug 2017 08:51:39
> GMT[\r][\n]"
> 2017-08-15 16:53:16,479 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "ETag: "e7191764798ba504d6671d4c434d2f4d"[\r][\n]"
> 2017-08-15 16:53:16,480 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
> (Wire.java:wire(72)) - << "x-amz-request-id:
> tx0001e-005992b67e-27a45-default[\r][\n]"
> 2017-08-15 16:
[jira] [Updated] (HADOOP-14774) S3A case "testRandomReadOverBuffer" failed due to improper range parameter
[
https://issues.apache.org/jira/browse/HADOOP-14774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yonger updated HADOOP-14774:
Description:
{code:java}
Running org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.821 sec <<<
FAILURE! - in org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
testRandomReadOverBuffer(org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance)
Time elapsed: 2.605 sec <<< FAILURE!
java.lang.AssertionError: expected:<8192> but was:<8193>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance.testRandomReadOverBuffer(ITestS3AInputStreamPerformance.java:533)
{code}
>From log, the length of content is exceed than our expect:
{code:java}
2017-08-15 16:53:16,464 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
(DefaultClientConnection.java:sendRequestHeader(280)) - >> GET
/test-aws-s3a/test/testReadOverBuffer.bin HTTP/1.1
2017-08-15 16:53:16,464 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
(DefaultClientConnection.java:sendRequestHeader(283)) - >> Host: 10.0.2.254
2017-08-15 16:53:16,465 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
(DefaultClientConnection.java:sendRequestHeader(283)) - >>
x-amz-content-sha256:
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2017-08-15 16:53:16,465 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
(DefaultClientConnection.java:sendRequestHeader(283)) - >> Authorization:
AWS4-HMAC-SHA256
Credential=JFDAM9KF9IY8S5P0JIV6/20170815/us-east-1/s3/aws4_request,
SignedHeaders=content-type;host;range;user-agent;x-amz-content-sha256;x-amz-date,
Signature=42bce4a43d2b1bf6e6d599613c60812e6716514da4ef5b3839ef0566c31279ee
2017-08-15 16:53:16,466 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
(DefaultClientConnection.java:sendRequestHeader(283)) - >> X-Amz-Date:
20170815T085316Z
2017-08-15 16:53:16,466 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
(DefaultClientConnection.java:sendRequestHeader(283)) - >> User-Agent: Hadoop
2.8.0, aws-sdk-java/1.10.6 Linux/3.10.0-514.21.2.el7.x86_64
Java_HotSpot(TM)_64-Bit_Server_VM/25.131-b11/1.8.0_131
2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
(DefaultClientConnection.java:sendRequestHeader(283)) - >> Range: bytes=0-8192
2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
(DefaultClientConnection.java:sendRequestHeader(283)) - >> Content-Type:
application/x-www-form-urlencoded; charset=utf-8
2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers
(DefaultClientConnection.java:sendRequestHeader(283)) - >> Connection:
Keep-Alive
2017-08-15 16:53:16,473 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
(Wire.java:wire(72)) - << "HTTP/1.1 206 Partial Content[\r][\n]"
2017-08-15 16:53:16,475 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
(Wire.java:wire(72)) - << "Content-Range: bytes 0-8192/32768[\r][\n]"
2017-08-15 16:53:16,476 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
(Wire.java:wire(72)) - << "Content-Length: 8193[\r][\n]"
2017-08-15 16:53:16,477 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
(Wire.java:wire(72)) - << "Accept-Ranges: bytes[\r][\n]"
2017-08-15 16:53:16,478 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
(Wire.java:wire(72)) - << "Last-Modified: Tue, 15 Aug 2017 08:51:39
GMT[\r][\n]"
2017-08-15 16:53:16,479 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
(Wire.java:wire(72)) - << "ETag: "e7191764798ba504d6671d4c434d2f4d"[\r][\n]"
2017-08-15 16:53:16,480 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
(Wire.java:wire(72)) - << "x-amz-request-id:
tx0001e-005992b67e-27a45-default[\r][\n]"
2017-08-15 16:53:16,481 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
(Wire.java:wire(72)) - << "Content-Type: application/octet-stream[\r][\n]"
2017-08-15 16:53:16,482 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
(Wire.java:wire(72)) - << "Date: Tue, 15 Aug 2017 08:53:18 GMT[\r][\n]"
2017-08-15 16:53:16,483 [JUnit-testRandomReadOverBuffer] DEBUG http.wire
(Wire.java:wire(72)) - << "[\r][\n]"
{code}
was:
Running org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.821 sec <<<
FAILURE! - in org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
testRandomReadOverBuffer(org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance)
Time elapsed: 2.605 sec <<< FAILURE!
java.lang.AssertionError: expected:<8192> but was:<8193>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.A
[jira] [Created] (HADOOP-14774) S3A case "testRandomReadOverBuffer" failed due to improper range parameter
Yonger created HADOOP-14774: --- Summary: S3A case "testRandomReadOverBuffer" failed due to improper range parameter Key: HADOOP-14774 URL: https://issues.apache.org/jira/browse/HADOOP-14774 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Affects Versions: 2.8.0 Environment: Hadoop 2.8.0 s3-compatible storage Reporter: Yonger Running org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.821 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance testRandomReadOverBuffer(org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance) Time elapsed: 2.605 sec <<< FAILURE! java.lang.AssertionError: expected:<8192> but was:<8193> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance.testRandomReadOverBuffer(ITestS3AInputStreamPerformance.java:533) >From log, the length of content is exceed than our expect: 2017-08-15 16:53:16,464 [JUnit-testRandomReadOverBuffer] DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(280)) - >> GET /test-aws-s3a/test/testReadOverBuffer.bin HTTP/1.1 2017-08-15 16:53:16,464 [JUnit-testRandomReadOverBuffer] DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> Host: 10.0.2.254 2017-08-15 16:53:16,465 [JUnit-testRandomReadOverBuffer] DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2017-08-15 16:53:16,465 [JUnit-testRandomReadOverBuffer] DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> Authorization: AWS4-HMAC-SHA256 Credential=JFDAM9KF9IY8S5P0JIV6/20170815/us-east-1/s3/aws4_request, SignedHeaders=content-type;host;range;user-agent;x-amz-content-sha256;x-amz-date, Signature=42bce4a43d2b1bf6e6d599613c60812e6716514da4ef5b3839ef0566c31279ee 2017-08-15 16:53:16,466 [JUnit-testRandomReadOverBuffer] DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> X-Amz-Date: 20170815T085316Z 2017-08-15 16:53:16,466 [JUnit-testRandomReadOverBuffer] DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> User-Agent: Hadoop 2.8.0, aws-sdk-java/1.10.6 Linux/3.10.0-514.21.2.el7.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.131-b11/1.8.0_131 2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> Range: bytes=0-8192 2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> Content-Type: application/x-www-form-urlencoded; charset=utf-8 2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> Connection: Keep-Alive 2017-08-15 16:53:16,473 [JUnit-testRandomReadOverBuffer] DEBUG http.wire (Wire.java:wire(72)) - << "HTTP/1.1 206 Partial Content[\r][\n]" 2017-08-15 16:53:16,475 [JUnit-testRandomReadOverBuffer] DEBUG http.wire (Wire.java:wire(72)) - << "Content-Range: bytes 0-8192/32768[\r][\n]" 2017-08-15 16:53:16,476 [JUnit-testRandomReadOverBuffer] DEBUG http.wire (Wire.java:wire(72)) - << "Content-Length: 8193[\r][\n]" 2017-08-15 16:53:16,477 [JUnit-testRandomReadOverBuffer] DEBUG http.wire (Wire.java:wire(72)) - << "Accept-Ranges: bytes[\r][\n]" 2017-08-15 16:53:16,478 [JUnit-testRandomReadOverBuffer] DEBUG http.wire (Wire.java:wire(72)) - << "Last-Modified: Tue, 15 Aug 2017 08:51:39 GMT[\r][\n]" 2017-08-15 16:53:16,479 [JUnit-testRandomReadOverBuffer] DEBUG http.wire (Wire.java:wire(72)) - << "ETag: "e7191764798ba504d6671d4c434d2f4d"[\r][\n]" 2017-08-15 16:53:16,480 [JUnit-testRandomReadOverBuffer] DEBUG http.wire (Wire.java:wire(72)) - << "x-amz-request-id: tx0001e-005992b67e-27a45-default[\r][\n]" 2017-08-15 16:53:16,481 [JUnit-testRandomReadOverBuffer] DEBUG http.wire (Wire.java:wire(72)) - << "Content-Type: application/octet-stream[\r][\n]" 2017-08-15 16:53:16,482 [JUnit-testRandomReadOverBuffer] DEBUG http.wire (Wire.java:wire(72)) - << "Date: Tue, 15 Aug 2017 08:53:18 GMT[\r][\n]" 2017-08-15 16:53:16,483 [JUnit-testRandomReadOverBuffer] DEBUG http.wire (Wire.java:wire(72)) - << "[\r][\n]" -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.
[jira] [Assigned] (HADOOP-14774) S3A case "testRandomReadOverBuffer" failed due to improper range parameter
[ https://issues.apache.org/jira/browse/HADOOP-14774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yonger reassigned HADOOP-14774: --- Assignee: Yonger > S3A case "testRandomReadOverBuffer" failed due to improper range parameter > -- > > Key: HADOOP-14774 > URL: https://issues.apache.org/jira/browse/HADOOP-14774 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: Hadoop 2.8.0 > s3-compatible storage >Reporter: Yonger >Assignee: Yonger > > Running org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance > Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.821 sec <<< > FAILURE! - in org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance > testRandomReadOverBuffer(org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance) > Time elapsed: 2.605 sec <<< FAILURE! > java.lang.AssertionError: expected:<8192> but was:<8193> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.junit.Assert.assertEquals(Assert.java:555) > at org.junit.Assert.assertEquals(Assert.java:542) > at > org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance.testRandomReadOverBuffer(ITestS3AInputStreamPerformance.java:533) > From log, the length of content is exceed than our expect: > 2017-08-15 16:53:16,464 [JUnit-testRandomReadOverBuffer] DEBUG http.headers > (DefaultClientConnection.java:sendRequestHeader(280)) - >> GET > /test-aws-s3a/test/testReadOverBuffer.bin HTTP/1.1 > 2017-08-15 16:53:16,464 [JUnit-testRandomReadOverBuffer] DEBUG http.headers > (DefaultClientConnection.java:sendRequestHeader(283)) - >> Host: 10.0.2.254 > 2017-08-15 16:53:16,465 [JUnit-testRandomReadOverBuffer] DEBUG http.headers > (DefaultClientConnection.java:sendRequestHeader(283)) - >> > x-amz-content-sha256: > e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 > 2017-08-15 16:53:16,465 [JUnit-testRandomReadOverBuffer] DEBUG http.headers > (DefaultClientConnection.java:sendRequestHeader(283)) - >> Authorization: > AWS4-HMAC-SHA256 > Credential=JFDAM9KF9IY8S5P0JIV6/20170815/us-east-1/s3/aws4_request, > SignedHeaders=content-type;host;range;user-agent;x-amz-content-sha256;x-amz-date, > Signature=42bce4a43d2b1bf6e6d599613c60812e6716514da4ef5b3839ef0566c31279ee > 2017-08-15 16:53:16,466 [JUnit-testRandomReadOverBuffer] DEBUG http.headers > (DefaultClientConnection.java:sendRequestHeader(283)) - >> X-Amz-Date: > 20170815T085316Z > 2017-08-15 16:53:16,466 [JUnit-testRandomReadOverBuffer] DEBUG http.headers > (DefaultClientConnection.java:sendRequestHeader(283)) - >> User-Agent: Hadoop > 2.8.0, aws-sdk-java/1.10.6 Linux/3.10.0-514.21.2.el7.x86_64 > Java_HotSpot(TM)_64-Bit_Server_VM/25.131-b11/1.8.0_131 > 2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers > (DefaultClientConnection.java:sendRequestHeader(283)) - >> Range: bytes=0-8192 > 2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers > (DefaultClientConnection.java:sendRequestHeader(283)) - >> Content-Type: > application/x-www-form-urlencoded; charset=utf-8 > 2017-08-15 16:53:16,467 [JUnit-testRandomReadOverBuffer] DEBUG http.headers > (DefaultClientConnection.java:sendRequestHeader(283)) - >> Connection: > Keep-Alive > 2017-08-15 16:53:16,473 [JUnit-testRandomReadOverBuffer] DEBUG http.wire > (Wire.java:wire(72)) - << "HTTP/1.1 206 Partial Content[\r][\n]" > 2017-08-15 16:53:16,475 [JUnit-testRandomReadOverBuffer] DEBUG http.wire > (Wire.java:wire(72)) - << "Content-Range: bytes 0-8192/32768[\r][\n]" > 2017-08-15 16:53:16,476 [JUnit-testRandomReadOverBuffer] DEBUG http.wire > (Wire.java:wire(72)) - << "Content-Length: 8193[\r][\n]" > 2017-08-15 16:53:16,477 [JUnit-testRandomReadOverBuffer] DEBUG http.wire > (Wire.java:wire(72)) - << "Accept-Ranges: bytes[\r][\n]" > 2017-08-15 16:53:16,478 [JUnit-testRandomReadOverBuffer] DEBUG http.wire > (Wire.java:wire(72)) - << "Last-Modified: Tue, 15 Aug 2017 08:51:39 > GMT[\r][\n]" > 2017-08-15 16:53:16,479 [JUnit-testRandomReadOverBuffer] DEBUG http.wire > (Wire.java:wire(72)) - << "ETag: "e7191764798ba504d6671d4c434d2f4d"[\r][\n]" > 2017-08-15 16:53:16,480 [JUnit-testRandomReadOverBuffer] DEBUG http.wire > (Wire.java:wire(72)) - << "x-amz-request-id: > tx0001e-005992b67e-27a45-default[\r][\n]" > 2017-08-15 16:53:16,481 [JUnit-testRandomReadOverBuffer] DEBUG http.wire > (Wire.java:wire(72)) - << "Content-Type: application/octet-stream[\r][\n]" > 2017-08-15 16:53:16,482 [JUnit-testRandomReadOverBuffer] DEBUG http.wire > (Wire.java:wire(72)) - << "Date: Tue, 15 Aug 2017 08:53:18 GMT[\r][\n]" > 2017-
[jira] [Commented] (HADOOP-14770) S3A http connection in s3a driver not reuse in Spark application
[
https://issues.apache.org/jira/browse/HADOOP-14770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16125536#comment-16125536
]
Yonger commented on HADOOP-14770:
-
Thanks Steve, the application running on Hadoop 2.7.3 and against ORC file
format. I will upgrade to Hadoop 2.8.0 to verify.
> S3A http connection in s3a driver not reuse in Spark application
>
>
> Key: HADOOP-14770
> URL: https://issues.apache.org/jira/browse/HADOOP-14770
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.7.3
>Reporter: Yonger
>Assignee: Yonger
>
> I print out connection stats every 2 s when running Spark application against
> s3-compatible storage:
> {code}
> ESTAB 0 0 :::10.0.2.36:6
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44454
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44374
> :::10.0.2.254:80
> ESTAB 159724 0 :::10.0.2.36:44436
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:8
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44338
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44438
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44414
> :::10.0.2.254:80
> ESTAB 0 480 :::10.0.2.36:44450
> :::10.0.2.254:80 timer:(on,170ms,0)
> ESTAB 0 0 :::10.0.2.36:2
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44390
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44326
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44452
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44394
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:4
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44456
> :::10.0.2.254:80
> ==
> ESTAB 0 0 :::10.0.2.36:44508
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44476
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44524
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44374
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44500
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44504
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44512
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44506
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44464
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44518
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44510
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:2
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44526
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44472
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44466
> :::10.0.2.254:80
> {code}
> the connection in the above of "=" and below were changed all the time. But
> this haven't seen in MR application.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14770) S3A http connection in s3a driver not reuse in Spark application
[
https://issues.apache.org/jira/browse/HADOOP-14770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yonger updated HADOOP-14770:
Affects Version/s: 2.7.3
> S3A http connection in s3a driver not reuse in Spark application
>
>
> Key: HADOOP-14770
> URL: https://issues.apache.org/jira/browse/HADOOP-14770
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.7.3
>Reporter: Yonger
>Assignee: Yonger
>
> I print out connection stats every 2 s when running Spark application against
> s3-compatible storage:
> {code}
> ESTAB 0 0 :::10.0.2.36:6
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44454
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44374
> :::10.0.2.254:80
> ESTAB 159724 0 :::10.0.2.36:44436
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:8
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44338
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44438
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44414
> :::10.0.2.254:80
> ESTAB 0 480 :::10.0.2.36:44450
> :::10.0.2.254:80 timer:(on,170ms,0)
> ESTAB 0 0 :::10.0.2.36:2
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44390
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44326
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44452
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44394
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:4
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44456
> :::10.0.2.254:80
> ==
> ESTAB 0 0 :::10.0.2.36:44508
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44476
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44524
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44374
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44500
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44504
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44512
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44506
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44464
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44518
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44510
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:2
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44526
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44472
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44466
> :::10.0.2.254:80
> {code}
> the connection in the above of "=" and below were changed all the time. But
> this haven't seen in MR application.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14770) S3A http connection in s3a driver not reuse in Spark application
[
https://issues.apache.org/jira/browse/HADOOP-14770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yonger updated HADOOP-14770:
Component/s: fs/s3
> S3A http connection in s3a driver not reuse in Spark application
>
>
> Key: HADOOP-14770
> URL: https://issues.apache.org/jira/browse/HADOOP-14770
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.7.3
>Reporter: Yonger
>Assignee: Yonger
>
> I print out connection stats every 2 s when running Spark application against
> s3-compatible storage:
> {code}
> ESTAB 0 0 :::10.0.2.36:6
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44454
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44374
> :::10.0.2.254:80
> ESTAB 159724 0 :::10.0.2.36:44436
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:8
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44338
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44438
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44414
> :::10.0.2.254:80
> ESTAB 0 480 :::10.0.2.36:44450
> :::10.0.2.254:80 timer:(on,170ms,0)
> ESTAB 0 0 :::10.0.2.36:2
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44390
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44326
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44452
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44394
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:4
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44456
> :::10.0.2.254:80
> ==
> ESTAB 0 0 :::10.0.2.36:44508
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44476
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44524
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44374
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44500
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44504
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44512
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44506
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44464
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44518
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44510
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:2
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44526
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44472
> :::10.0.2.254:80
> ESTAB 0 0 :::10.0.2.36:44466
> :::10.0.2.254:80
> {code}
> the connection in the above of "=" and below were changed all the time. But
> this haven't seen in MR application.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14770) S3A http connection in s3a driver not reuse in Spark application
[ https://issues.apache.org/jira/browse/HADOOP-14770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yonger reassigned HADOOP-14770: --- Assignee: Yonger > S3A http connection in s3a driver not reuse in Spark application > > > Key: HADOOP-14770 > URL: https://issues.apache.org/jira/browse/HADOOP-14770 > Project: Hadoop Common > Issue Type: Bug >Reporter: Yonger >Assignee: Yonger > > I print out connection stats every 2 s when running Spark application against > s3-compatible storage: > ESTAB 0 0 :::10.0.2.36:6 > :::10.0.2.254:80 > ESTAB 0 0 :::10.0.2.36:44454 > :::10.0.2.254:80 > ESTAB 0 0 :::10.0.2.36:44374 > :::10.0.2.254:80 > ESTAB 159724 0 :::10.0.2.36:44436 > :::10.0.2.254:80 > ESTAB 0 0 :::10.0.2.36:8 > :::10.0.2.254:80 > ESTAB 0 0 :::10.0.2.36:44338 > :::10.0.2.254:80 > ESTAB 0 0 :::10.0.2.36:44438 > :::10.0.2.254:80 > ESTAB 0 0 :::10.0.2.36:44414 > :::10.0.2.254:80 > ESTAB 0 480 :::10.0.2.36:44450 > :::10.0.2.254:80 timer:(on,170ms,0) > ESTAB 0 0 :::10.0.2.36:2 > :::10.0.2.254:80 > ESTAB 0 0 :::10.0.2.36:44390 > :::10.0.2.254:80 > ESTAB 0 0 :::10.0.2.36:44326 > :::10.0.2.254:80 > ESTAB 0 0 :::10.0.2.36:44452 > :::10.0.2.254:80 > ESTAB 0 0 :::10.0.2.36:44394 > :::10.0.2.254:80 > ESTAB 0 0 :::10.0.2.36:4 > :::10.0.2.254:80 > ESTAB 0 0 :::10.0.2.36:44456 > :::10.0.2.254:80 > == > ESTAB 0 0 :::10.0.2.36:44508 > :::10.0.2.254:80 > ESTAB 0 0 :::10.0.2.36:44476 > :::10.0.2.254:80 > ESTAB 0 0 :::10.0.2.36:44524 > :::10.0.2.254:80 > ESTAB 0 0 :::10.0.2.36:44374 > :::10.0.2.254:80 > ESTAB 0 0 :::10.0.2.36:44500 > :::10.0.2.254:80 > ESTAB 0 0 :::10.0.2.36:44504 > :::10.0.2.254:80 > ESTAB 0 0 :::10.0.2.36:44512 > :::10.0.2.254:80 > ESTAB 0 0 :::10.0.2.36:44506 > :::10.0.2.254:80 > ESTAB 0 0 :::10.0.2.36:44464 > :::10.0.2.254:80 > ESTAB 0 0 :::10.0.2.36:44518 > :::10.0.2.254:80 > ESTAB 0 0 :::10.0.2.36:44510 > :::10.0.2.254:80 > ESTAB 0 0 :::10.0.2.36:2 > :::10.0.2.254:80 > ESTAB 0 0 :::10.0.2.36:44526 > :::10.0.2.254:80 > ESTAB 0 0 :::10.0.2.36:44472 > :::10.0.2.254:80 > ESTAB 0 0 :::10.0.2.36:44466 > :::10.0.2.254:80 > the connection in the above of "=" and below were changed all the time. But > this haven't seen in MR application. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14770) S3A http connection in s3a driver not reuse in Spark application
Yonger created HADOOP-14770: --- Summary: S3A http connection in s3a driver not reuse in Spark application Key: HADOOP-14770 URL: https://issues.apache.org/jira/browse/HADOOP-14770 Project: Hadoop Common Issue Type: Bug Reporter: Yonger I print out connection stats every 2 s when running Spark application against s3-compatible storage: ESTAB 0 0 :::10.0.2.36:6 :::10.0.2.254:80 ESTAB 0 0 :::10.0.2.36:44454 :::10.0.2.254:80 ESTAB 0 0 :::10.0.2.36:44374 :::10.0.2.254:80 ESTAB 159724 0 :::10.0.2.36:44436 :::10.0.2.254:80 ESTAB 0 0 :::10.0.2.36:8 :::10.0.2.254:80 ESTAB 0 0 :::10.0.2.36:44338 :::10.0.2.254:80 ESTAB 0 0 :::10.0.2.36:44438 :::10.0.2.254:80 ESTAB 0 0 :::10.0.2.36:44414 :::10.0.2.254:80 ESTAB 0 480 :::10.0.2.36:44450 :::10.0.2.254:80 timer:(on,170ms,0) ESTAB 0 0 :::10.0.2.36:2 :::10.0.2.254:80 ESTAB 0 0 :::10.0.2.36:44390 :::10.0.2.254:80 ESTAB 0 0 :::10.0.2.36:44326 :::10.0.2.254:80 ESTAB 0 0 :::10.0.2.36:44452 :::10.0.2.254:80 ESTAB 0 0 :::10.0.2.36:44394 :::10.0.2.254:80 ESTAB 0 0 :::10.0.2.36:4 :::10.0.2.254:80 ESTAB 0 0 :::10.0.2.36:44456 :::10.0.2.254:80 == ESTAB 0 0 :::10.0.2.36:44508 :::10.0.2.254:80 ESTAB 0 0 :::10.0.2.36:44476 :::10.0.2.254:80 ESTAB 0 0 :::10.0.2.36:44524 :::10.0.2.254:80 ESTAB 0 0 :::10.0.2.36:44374 :::10.0.2.254:80 ESTAB 0 0 :::10.0.2.36:44500 :::10.0.2.254:80 ESTAB 0 0 :::10.0.2.36:44504 :::10.0.2.254:80 ESTAB 0 0 :::10.0.2.36:44512 :::10.0.2.254:80 ESTAB 0 0 :::10.0.2.36:44506 :::10.0.2.254:80 ESTAB 0 0 :::10.0.2.36:44464 :::10.0.2.254:80 ESTAB 0 0 :::10.0.2.36:44518 :::10.0.2.254:80 ESTAB 0 0 :::10.0.2.36:44510 :::10.0.2.254:80 ESTAB 0 0 :::10.0.2.36:2 :::10.0.2.254:80 ESTAB 0 0 :::10.0.2.36:44526 :::10.0.2.254:80 ESTAB 0 0 :::10.0.2.36:44472 :::10.0.2.254:80 ESTAB 0 0 :::10.0.2.36:44466 :::10.0.2.254:80 the connection in the above of "=" and below were changed all the time. But this haven't seen in MR application. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14745) s3a getFileStatus can't return expect result when existing a file and directory with the same name
[
https://issues.apache.org/jira/browse/HADOOP-14745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16118177#comment-16118177
]
Yonger commented on HADOOP-14745:
-
I create that path structure through s3cmd, and yes , we can't do this by s3a
itself.
But, s3 or s3-compatible storage allow them existed within the same folder, and
the common workload running on Hadoop on Ceph is data analysis, which means the
data in these storage should be store by other way instead of s3a, and read
data from storage through s3a, which can't stop this issue happen again.
> s3a getFileStatus can't return expect result when existing a file and
> directory with the same name
> --
>
> Key: HADOOP-14745
> URL: https://issues.apache.org/jira/browse/HADOOP-14745
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Yonger
>Assignee: Yonger
>
> {code}
> [ hadoop-aws]# /root/hadoop/s3cmd/s3cmd ls s3://test-aws-s3a/user/root/
>DIR s3://test-aws-s3a/user/root/ccc/
> 2017-08-08 07:04 0 s3://test-aws-s3a/user/root/ccc
> {code}
> if we expect to ccc is a directory by code :
> {code}
> Path test=new Path("ccc");
> fs.getFileStatus(test);
> {code}
> actually, it will tell us it is a file:
> {code}
> 2017-08-08 15:08:40,566 [JUnit-case1] DEBUG s3a.S3AFileSystem
> (S3AFileSystem.java:getFileStatus(1576)) - Getting path status for
> s3a://test-aws-s3a/user/root/ccc (user/root/ccc)
> 2017-08-08 15:08:40,566 [JUnit-case1] DEBUG s3a.S3AFileSystem
> (S3AStorageStatistics.java:incrementCounter(60)) - object_metadata_requests
> += 1 -> 3
> 2017-08-08 15:08:40,580 [JUnit-case1] DEBUG s3a.S3AFileSystem
> (S3AFileSystem.java:getFileStatus(1585)) - Found exact file: normal file
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14745) s3a getFileStatus can't return expect result when existing a file and directory with the same name
[
https://issues.apache.org/jira/browse/HADOOP-14745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16118026#comment-16118026
]
Yonger commented on HADOOP-14745:
-
On the other hand, I saw hdfs implementation not allow existing any file with
the same name of directory under it's parent path.
> s3a getFileStatus can't return expect result when existing a file and
> directory with the same name
> --
>
> Key: HADOOP-14745
> URL: https://issues.apache.org/jira/browse/HADOOP-14745
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Yonger
>Assignee: Yonger
>
> [ hadoop-aws]# /root/hadoop/s3cmd/s3cmd ls s3://test-aws-s3a/user/root/
>DIR s3://test-aws-s3a/user/root/ccc/
> 2017-08-08 07:04 0 s3://test-aws-s3a/user/root/ccc
> if we expect to ccc is a directory by code :
> Path test=new Path("ccc");
> fs.getFileStatus(test);
> actually, it will tell us it is a file:
> 2017-08-08 15:08:40,566 [JUnit-case1] DEBUG s3a.S3AFileSystem
> (S3AFileSystem.java:getFileStatus(1576)) - Getting path status for
> s3a://test-aws-s3a/user/root/ccc (user/root/ccc)
> 2017-08-08 15:08:40,566 [JUnit-case1] DEBUG s3a.S3AFileSystem
> (S3AStorageStatistics.java:incrementCounter(60)) - object_metadata_requests
> += 1 -> 3
> 2017-08-08 15:08:40,580 [JUnit-case1] DEBUG s3a.S3AFileSystem
> (S3AFileSystem.java:getFileStatus(1585)) - Found exact file: normal file
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14745) s3a getFileStatus can't return expect result when existing a file and directory with the same name
[
https://issues.apache.org/jira/browse/HADOOP-14745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16118009#comment-16118009
]
Yonger commented on HADOOP-14745:
-
So i think we should call getFileStatus with explicitly expect, e.g.
getFileStatus(path,true)
true means we think the path we input is a directory
In internal of getFileStatus, i think the call like above will skip the first
two getObjectMetadata call, and only list the object under this given path,
which also benefit the performance of this network consuming function.
> s3a getFileStatus can't return expect result when existing a file and
> directory with the same name
> --
>
> Key: HADOOP-14745
> URL: https://issues.apache.org/jira/browse/HADOOP-14745
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Yonger
>Assignee: Yonger
>
> [ hadoop-aws]# /root/hadoop/s3cmd/s3cmd ls s3://test-aws-s3a/user/root/
>DIR s3://test-aws-s3a/user/root/ccc/
> 2017-08-08 07:04 0 s3://test-aws-s3a/user/root/ccc
> if we expect to ccc is a directory by code :
> Path test=new Path("ccc");
> fs.getFileStatus(test);
> actually, it will tell us it is a file:
> 2017-08-08 15:08:40,566 [JUnit-case1] DEBUG s3a.S3AFileSystem
> (S3AFileSystem.java:getFileStatus(1576)) - Getting path status for
> s3a://test-aws-s3a/user/root/ccc (user/root/ccc)
> 2017-08-08 15:08:40,566 [JUnit-case1] DEBUG s3a.S3AFileSystem
> (S3AStorageStatistics.java:incrementCounter(60)) - object_metadata_requests
> += 1 -> 3
> 2017-08-08 15:08:40,580 [JUnit-case1] DEBUG s3a.S3AFileSystem
> (S3AFileSystem.java:getFileStatus(1585)) - Found exact file: normal file
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14745) s3a getFileStatus can't return expect result when existing a file and directory with the same name
Yonger created HADOOP-14745:
---
Summary: s3a getFileStatus can't return expect result when
existing a file and directory with the same name
Key: HADOOP-14745
URL: https://issues.apache.org/jira/browse/HADOOP-14745
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 2.8.0
Reporter: Yonger
Assignee: Yonger
[ hadoop-aws]# /root/hadoop/s3cmd/s3cmd ls s3://test-aws-s3a/user/root/
DIR s3://test-aws-s3a/user/root/ccc/
2017-08-08 07:04 0 s3://test-aws-s3a/user/root/ccc
if we expect to ccc is a directory by code :
Path test=new Path("ccc");
fs.getFileStatus(test);
actually, it will tell us it is a file:
2017-08-08 15:08:40,566 [JUnit-case1] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:getFileStatus(1576)) - Getting path status for
s3a://test-aws-s3a/user/root/ccc (user/root/ccc)
2017-08-08 15:08:40,566 [JUnit-case1] DEBUG s3a.S3AFileSystem
(S3AStorageStatistics.java:incrementCounter(60)) - object_metadata_requests +=
1 -> 3
2017-08-08 15:08:40,580 [JUnit-case1] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:getFileStatus(1585)) - Found exact file: normal file
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13430) Optimize and fix getFileStatus in S3A
[ https://issues.apache.org/jira/browse/HADOOP-13430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116700#comment-16116700 ] Yonger commented on HADOOP-13430: - [~ste...@apache.org] I really can't see any optimization for getFileStatus in ticket [https://issues.apache.org/jira/browse/HADOOP-13208] , with my understand, there is still exist 2 head +1 list operations in this function. And I scan the code where call getFileStatus, and found that in some place, we can know the path that input is a file or a directory, e.g. in create and open function, we don't need to check the path is a directory with calling getfileStatus, just consider the path is a file(according the implementation of Presto), thus when calling getFileStatus, we know the input path is a file so that it don't need to call getmetadata method again with suffix "/". Totally, can we reduce the s3 call over network as possible by tell getFileStatus the path is a file or directory explicitly? > Optimize and fix getFileStatus in S3A > - > > Key: HADOOP-13430 > URL: https://issues.apache.org/jira/browse/HADOOP-13430 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steven K. Wong >Assignee: Steven K. Wong >Priority: Minor > Attachments: HADOOP-13430.001.WIP.patch > > > Currently, S3AFileSystem.getFileStatus(Path f) sends up to 3 requests to S3 > when pathToKey(f) = key = "foo/bar" is a directory: > 1. HEAD key=foo/bar \[continue if not found] > 2. HEAD key=foo/bar/ \[continue if not found] > 3. LIST prefix=foo/bar/ delimiter=/ max-keys=1 > My experience (and generally true, I reckon) is that almost all directories > are nonempty directories without a "fake directory" file (e.g. "foo/bar/"). > Under this condition, request #2 is mostly unhelpful; it only slows down > getFileStatus. Therefore, I propose swapping the order of requests #2 and #3. > The swapped HEAD request will be skipped in practically all cases. > Furthermore, when key = "foo/bar" is a nonempty directory that contains a > "fake directory" file (in addition to actual files), getFileStatus currently > returns an S3AFileStatus with isEmptyDirectory=true, which is wrong. Swapping > will fix this. The swapped LIST request will use max-keys=2 to determine > isEmptyDirectory correctly. (Removing the delimiter from the LIST request > should make the logic a little simpler than otherwise.) > Note that key = "foo/bar/" has the same problem with isEmptyDirectory. To fix > it, I propose skipping request #1 when key ends with "/". The price is this > will, for an empty directory, replace a HEAD request with a LIST request > that's generally more taxing on S3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yonger updated HADOOP-14475: Attachment: HADOOP-14475.006.patch update: 1.add ASF license for new test class > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: failsafe-report-s3a-it.html, > failsafe-report-s3a-scale.html, failsafe-report-scale.html, > failsafe-report-scale.zip, HADOOP-14475.002.patch, HADOOP-14475-003.patch, > HADOOP-14475.005.patch, HADOOP-14475.006.patch, s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yonger updated HADOOP-14475: Attachment: HADOOP-14475.005.patch update: 1.remove streamMetrics hashmap, register all metrics into registry 2.add missing metric STREAM_WRITE_QUEUE_DURATION 3.add test case in ITestS3AMetrics to verify metrics resource registration > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: failsafe-report-s3a-it.html, > failsafe-report-s3a-scale.html, failsafe-report-scale.html, > failsafe-report-scale.zip, HADOOP-14475.002.patch, HADOOP-14475-003.patch, > HADOOP-14475.005.patch, s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100142#comment-16100142 ] Yonger commented on HADOOP-14475: - [~ste...@apache.org] I found the output of metrics missing part of metrics in streamMetrics HashMap, because they didn't register into MetricsRegistry. With my understand, i will remove streamMetrics HashMap, and register stream related metrics directly like method "counter". But i note there is function "dump" will read from this HashMap, and few other class call this dump method to print stream metrics. I don't think it's a good idea to print any metrics to log file, we should put all performance metrics into other place(e.g. a separate file or database) . Any suggestion? > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: failsafe-report-s3a-it.html, > failsafe-report-s3a-scale.html, failsafe-report-scale.html, > failsafe-report-scale.zip, HADOOP-14475.002.patch, HADOOP-14475-003.patch, > s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13845) s3a to instrument duration of HTTP calls
[
https://issues.apache.org/jira/browse/HADOOP-13845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098416#comment-16098416
]
Yonger commented on HADOOP-13845:
-
Does this make sense?
{code:java}
Duration duration = new Duration();
ObjectListing listing= s3.listObjects(request);
duration.finished();
durationStats.add(method.getName()+" " + reason, duration, success);
return listing;
{code}
> s3a to instrument duration of HTTP calls
>
>
> Key: HADOOP-13845
> URL: https://issues.apache.org/jira/browse/HADOOP-13845
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Priority: Minor
>
> HADOOP-13844 proposes pulling out the swift duration classes for reuse; this
> patch proposes instrumenting s3a with it.
> One interesting question: what to do with the values. For now, they could
> just be printed, but it might be interesting to include in FS stats collected
> at the end of a run. However, those are all assumed to be simple counters
> where merging is a matter of addition. These are more metrics
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13845) s3a to instrument duration of HTTP calls
[ https://issues.apache.org/jira/browse/HADOOP-13845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095706#comment-16095706 ] Yonger commented on HADOOP-13845: - Another usage is push http duration into database(such as influxdb), and get a realtime show via grafana. that you can know the duration of one kind(path) of request when is longest when suffering some bottleneck, and then you can do trouble shooting work in the range of higher duration time. which can not be done if you just only output the final result in FS stats. And what's going on for HADOOP-13844? i expect to put http duration into s3a metrics. > s3a to instrument duration of HTTP calls > > > Key: HADOOP-13845 > URL: https://issues.apache.org/jira/browse/HADOOP-13845 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Priority: Minor > > HADOOP-13844 proposes pulling out the swift duration classes for reuse; this > patch proposes instrumenting s3a with it. > One interesting question: what to do with the values. For now, they could > just be printed, but it might be interesting to include in FS stats collected > at the end of a run. However, those are all assumed to be simple counters > where merging is a matter of addition. These are more metrics -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13762) S3A: Set thread names with more specific information about the call.
[
https://issues.apache.org/jira/browse/HADOOP-13762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095684#comment-16095684
]
Yonger commented on HADOOP-13762:
-
I agree a thread name with meaningful information is a great idea, but I also
think logging the information you mentioned(userid, timestamp, and path) is
better than to insert into thread name from a comprehensive perspective. You
just need to put information into log component and without additional
complexity and performance consideration. And with these meaningful information
in logs, we also start a trouble shooting for performance problem, also this
approach applied in big company with high throughput computer system.
> S3A: Set thread names with more specific information about the call.
>
>
> Key: HADOOP-13762
> URL: https://issues.apache.org/jira/browse/HADOOP-13762
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Chris Nauroth
>
> Running {{jstack}} on a hung process and reading the stack traces is a
> helpful way to determine exactly what code in the process is stuck. This
> would be even more helpful if we included more descriptive information about
> the specific file system method call.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13637) improve setting of max connections in AWS client
[ https://issues.apache.org/jira/browse/HADOOP-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094592#comment-16094592 ] Yonger edited comment on HADOOP-13637 at 7/20/17 12:13 PM: --- [~ste...@apache.org] I don't think we should create a connection pool bigger than a work thread pool. Assume the application run on Hadoop kick of request with X rate, each request cost T1 time on thread and T1 time on connection, obviously, T1 will bigger than T2 due to a request always get response to release a connection and then release a thread later . That means XT1>XT2, and according little's law, N1(threads are needed concurrently) >N2 (connections are needed concurrently). So the result is opposite of yours, we should set minimum thread pool size base on max connection + n. was (Author: iyonger): [~steve_l] I don't think we should create a connection pool bigger than a work thread pool. Assume the application run on Hadoop kick of request with X rate, each request cost T1 time on thread and T1 time on connection, obviously, T1 will bigger than T2 due to a request always get response to release a connection and then release a thread later . That means XT1>XT2, and according little's law, N1(threads are needed concurrently) >N2 (connections are needed concurrently). So the result is opposite of yours, we should set minimum thread pool size base on max connection + n. > improve setting of max connections in AWS client > > > Key: HADOOP-13637 > URL: https://issues.apache.org/jira/browse/HADOOP-13637 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Priority: Minor > > things can go badly wrong if the S3A FS creates a thread pool for IO > than > the number of pooled AWS http connections (set by property > MAXIMUM_CONNECTIONS); you also need some for any other IO requests coming in. > The max connections property is currently independent of thread pool size, > and has a default value of 1. > this is why there is a troubleshooting section in the docs showing the stack > trace and instructions to fix". > Better: have a dynamic minimum like thread pool size + n, for a value of n to > be chosen. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13757) Remove verifyBuckets overhead in S3AFileSystem::initialize()
[
https://issues.apache.org/jira/browse/HADOOP-13757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094598#comment-16094598
]
Yonger commented on HADOOP-13757:
-
[~ste...@apache.org]Can you give more specific suggestion on this issue, then I
can try to work on it in my spare time.
> Remove verifyBuckets overhead in S3AFileSystem::initialize()
>
>
> Key: HADOOP-13757
> URL: https://issues.apache.org/jira/browse/HADOOP-13757
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Rajesh Balamohan
>Priority: Minor
>
> {{S3AFileSystem.initialize()}} invokes verifyBuckets, but in case the bucket
> does not exist and gets a 403 error message, it ends up returning {{true}}
> for {{s3.doesBucketExists(bucketName}}. In that aspect, verifyBuckets() is
> an unnecessary call during initialization.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13637) improve setting of max connections in AWS client
[ https://issues.apache.org/jira/browse/HADOOP-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094592#comment-16094592 ] Yonger commented on HADOOP-13637: - [~steve_l] I don't think we should create a connection pool bigger than a work thread pool. Assume the application run on Hadoop kick of request with X rate, each request cost T1 time on thread and T1 time on connection, obviously, T1 will bigger than T2 due to a request always get response to release a connection and then release a thread later . That means XT1>XT2, and according little's law, N1(threads are needed concurrently) >N2 (connections are needed concurrently). So the result is opposite of yours, we should set minimum thread pool size base on max connection + n. > improve setting of max connections in AWS client > > > Key: HADOOP-13637 > URL: https://issues.apache.org/jira/browse/HADOOP-13637 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Priority: Minor > > things can go badly wrong if the S3A FS creates a thread pool for IO > than > the number of pooled AWS http connections (set by property > MAXIMUM_CONNECTIONS); you also need some for any other IO requests coming in. > The max connections property is currently independent of thread pool size, > and has a default value of 1. > this is why there is a troubleshooting section in the docs showing the stack > trace and instructions to fix". > Better: have a dynamic minimum like thread pool size + n, for a value of n to > be chosen. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yonger updated HADOOP-14475: Attachment: HADOOP-14475-003.patch compliant with raw code format and update the logic of metrics register name > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: failsafe-report-s3a-it.html, > failsafe-report-s3a-scale.html, failsafe-report-scale.html, > failsafe-report-scale.zip, HADOOP-14475.002.patch, HADOOP-14475-003.patch, > s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yonger updated HADOOP-14475: Attachment: failsafe-report-scale.html failsafe-report-scale.zip Only one case failed. mvn -Dscale clean verify and skip s3 and s3n cases > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: failsafe-report-s3a-it.html, > failsafe-report-s3a-scale.html, failsafe-report-scale.html, > failsafe-report-scale.zip, HADOOP-14475.002.patch, s3a-metrics.patch1, > stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076511#comment-16076511 ] Yonger edited comment on HADOOP-14475 at 7/6/17 1:42 PM: - @steve the method you mentioned give an empty url to skip the landsat-pds tests is not work, also I upload the gz file into my bucket according to the guide, but it failed too. when giving the empty string, error message: Tests run: 9, Failures: 0, Errors: 6, Skipped: 0, Time elapsed: 0.325 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider testInstantiationChain(org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider) Time elapsed: 0.018 sec <<< ERROR! java.lang.IllegalArgumentException: Can not create a Path from an empty string at org.apache.hadoop.fs.Path.checkPathArg(Path.java:163) at org.apache.hadoop.fs.Path.(Path.java:175) at org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider.testInstantiationChain(TestS3AAWSCredentialsProvider.java:92) and if i use default value and upload the gz file, which give me a error message with code 403. was (Author: iyonger): [~stevea] the method you mentioned give an empty url to skip the landsat-pds tests is not work, also I upload the gz file into my bucket according to the guide, but it failed too. when giving the empty string, error message: Tests run: 9, Failures: 0, Errors: 6, Skipped: 0, Time elapsed: 0.325 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider testInstantiationChain(org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider) Time elapsed: 0.018 sec <<< ERROR! java.lang.IllegalArgumentException: Can not create a Path from an empty string at org.apache.hadoop.fs.Path.checkPathArg(Path.java:163) at org.apache.hadoop.fs.Path.(Path.java:175) at org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider.testInstantiationChain(TestS3AAWSCredentialsProvider.java:92) and if i use default value and upload the gz file, which give me a error message with code 403. > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: failsafe-report-s3a-it.html, > failsafe-report-s3a-scale.html, HADOOP-14475.002.patch, s3a-metrics.patch1, > stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076511#comment-16076511 ] Yonger commented on HADOOP-14475: - [~stevea] the method you mentioned give an empty url to skip the landsat-pds tests is not work, also I upload the gz file into my bucket according to the guide, but it failed too. when giving the empty string, error message: Tests run: 9, Failures: 0, Errors: 6, Skipped: 0, Time elapsed: 0.325 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider testInstantiationChain(org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider) Time elapsed: 0.018 sec <<< ERROR! java.lang.IllegalArgumentException: Can not create a Path from an empty string at org.apache.hadoop.fs.Path.checkPathArg(Path.java:163) at org.apache.hadoop.fs.Path.(Path.java:175) at org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider.testInstantiationChain(TestS3AAWSCredentialsProvider.java:92) and if i use default value and upload the gz file, which give me a error message with code 403. > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: failsafe-report-s3a-it.html, > failsafe-report-s3a-scale.html, HADOOP-14475.002.patch, s3a-metrics.patch1, > stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yonger updated HADOOP-14475: Attachment: failsafe-report-s3a-it.html > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: failsafe-report-s3a-it.html, > failsafe-report-s3a-scale.html, HADOOP-14475.002.patch, s3a-metrics.patch1, > stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yonger updated HADOOP-14475: Attachment: (was: failsafe-report-it.html) > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: failsafe-report-s3a-it.html, > failsafe-report-s3a-scale.html, HADOOP-14475.002.patch, s3a-metrics.patch1, > stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yonger updated HADOOP-14475: Attachment: failsafe-report-it.html failsafe-report-s3a-scale.html I run cases on Ceph storage, so part of them failed due to can't find something in s3 storage. > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: failsafe-report-it.html, failsafe-report-s3a-scale.html, > HADOOP-14475.002.patch, s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16058815#comment-16058815 ] Yonger edited comment on HADOOP-14475 at 6/22/17 6:18 AM: -- Fixed this issue by initializing metrics system and registering s3a metrics source implementation was (Author: iyonger): Fixed this issue by initializing metrics system and registering s3a metrcis source implementation > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: HADOOP-14475.002.patch, s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yonger updated HADOOP-14475: Attachment: HADOOP-14475.002.patch > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: HADOOP-14475.002.patch, s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yonger updated HADOOP-14475: Release Note: (was: fix S3AInstrumentation registration issue ) Status: Patch Available (was: Open) Fixed this issue by initializing metrics system and registering s3a metrcis source implementation > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yonger updated HADOOP-14475: Status: Open (was: Patch Available) > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16056807#comment-16056807 ] Yonger commented on HADOOP-14475: - @Deng, each task should have an individual s3a file system that i agree with you. But i seen the s3a file system had been initialized multiple times with the same path in driver application, you can also check in my last uploaded logs. BTW, the new patch that fix this issue totally is in the process of verifying, i don't know when it available to upload since i have no aws s3 test environment, i am running on Ceph now. > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger > Attachments: s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16040018#comment-16040018 ] Yonger commented on HADOOP-14475: - In my case, i just run DFSIO with 20 map/reduce tasks in a 4-nodes cluster. And i enable debug log to console, if you grep by "Metrics system initialized" the attached log file, you will find the s3a file system has been initialized multiple times under one bucket(s3a://test-bucket). > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger > Attachments: s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yonger updated HADOOP-14475: Attachment: stdout.zip > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger > Attachments: s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16038864#comment-16038864 ] Yonger commented on HADOOP-14475: - Thanks for your suggestion. I will take case for code style and do test by myself before submitting. And answer your questions: 1. the name change of context just for distinguish with other attributes, such as MetricsRegistry and Metrics name. From the following log, it shows using different names is better than ones with the same name: 17/06/05 20:32:54 DEBUG impl.MetricsSinkAdapter: Pushing record S3AFileSystemMetrics.s3a.s3afilesystem to file is it? 2.after i make a collection the relationship of those classes, i also think the functions of class S3AFileSystemMetricsSystem can be merge into some existed class, maybe S3AFileSystem. 3.that is the issue confused me. I still don't know why the filesystem(S3AFileSystem) be initialized multiple times in a MR job. From AzureFileSystem and DataNodeMetric, their filesystem and MetricSystem should be only initialized once. > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger > Attachments: s3a-metrics.patch1 > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16037032#comment-16037032 ] Yonger edited comment on HADOOP-14475 at 6/5/17 2:28 PM: - This patch can fix the issue that s3a metrics don't print out to sink(file).But there also introduce an issue that the metrics info only print out once to file instead of printing out periodically. was (Author: iyonger): This patch can fix the issue that s3a metrics don't print out to sink(file).But there also introduce an issue that the metrics info only print out once in file instead of printing our periodically. > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger > Attachments: s3a-metrics.patch1 > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yonger updated HADOOP-14475: Attachment: s3a-metrics.patch1 This patch can fix the issue that s3a metrics don't print out to sink(file).But there also introduce an issue that the metrics info only print out once in file instead of printing our periodically. > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger > Attachments: s3a-metrics.patch1 > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yonger updated HADOOP-14475: Release Note: fix S3AInstrumentation registration issue Status: Patch Available (was: Open) > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yonger updated HADOOP-14475: Component/s: (was: s3) fs/s3 > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yonger updated HADOOP-14475: Summary: Metrics of S3A don't print out when enable it in Hadoop metrics property file (was: Metrics of S3A doesn't print out when enable it in Hadoop metrics property file) > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14475) Metrics of S3A doesn't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yonger updated HADOOP-14475: Description: *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink #*.sink.influxdb.url=http:/xx #*.sink.influxdb.influxdb_port=8086 #*.sink.influxdb.database=hadoop #*.sink.influxdb.influxdb_username=hadoop #*.sink.influxdb.influxdb_password=hadoop #*.sink.ingluxdb.cluster=c1 ## default sampling period, in seconds *.period=10 #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out I can't find the out put file even i run a MR job which should be used s3. was: *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink #*.sink.influxdb.url=http:/xx #*.sink.influxdb.influxdb_port=8086 #*.sink.influxdb.database=hadoop #*.sink.influxdb.influxdb_username=hadoop #*.sink.influxdb.influxdb_password=hadoop #*.sink.ingluxdb.cluster=c1 # default sampling period, in seconds *.period=10 #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out I can't find the out put file even i run a MR job which should be used s3. > Metrics of S3A doesn't print out when enable it in Hadoop metrics property > file > > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > ## default sampling period, in seconds > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14475) Metrics of S3A doesn't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yonger updated HADOOP-14475: Description: *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink #*.sink.influxdb.url=http:/xx #*.sink.influxdb.influxdb_port=8086 #*.sink.influxdb.database=hadoop #*.sink.influxdb.influxdb_username=hadoop #*.sink.influxdb.influxdb_password=hadoop #*.sink.ingluxdb.cluster=c1 *.period=10 #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out I can't find the out put file even i run a MR job which should be used s3. was: *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink #*.sink.influxdb.url=http:/xx #*.sink.influxdb.influxdb_port=8086 #*.sink.influxdb.database=hadoop #*.sink.influxdb.influxdb_username=hadoop #*.sink.influxdb.influxdb_password=hadoop #*.sink.ingluxdb.cluster=c1 ## default sampling period, in seconds *.period=10 #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out I can't find the out put file even i run a MR job which should be used s3. > Metrics of S3A doesn't print out when enable it in Hadoop metrics property > file > > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14475) Metrics of S3A doesn't print out when enable it in Hadoop metrics property file
Yonger created HADOOP-14475: --- Summary: Metrics of S3A doesn't print out when enable it in Hadoop metrics property file Key: HADOOP-14475 URL: https://issues.apache.org/jira/browse/HADOOP-14475 Project: Hadoop Common Issue Type: Bug Components: s3 Affects Versions: 2.8.0 Environment: uname -a Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux cat /etc/issue Ubuntu 16.04.2 LTS \n \l Reporter: Yonger *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink #*.sink.influxdb.url=http:/xx #*.sink.influxdb.influxdb_port=8086 #*.sink.influxdb.database=hadoop #*.sink.influxdb.influxdb_username=hadoop #*.sink.influxdb.influxdb_password=hadoop #*.sink.ingluxdb.cluster=c1 # default sampling period, in seconds *.period=10 #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
