[jira] [Commented] (HADOOP-16954) Add -S option in "Count" command to show only Snapshot Counts
[
https://issues.apache.org/jira/browse/HADOOP-16954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17074315#comment-17074315
]
Hadoop QA commented on HADOOP-16954:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
40s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 17s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} hadoop-common-project/hadoop-common: The patch
generated 0 new + 36 unchanged - 1 fixed = 36 total (was 37) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 19s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
17s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}102m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 Image:yetus/hadoop:4454c6d14b7 |
| JIRA Issue | HADOOP-16954 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12998685/HADOOP-16954.003.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 409fa29de66d 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f8e598f |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16817/testReport/ |
| Max. process+thread count | 3234 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16817/console |
| Powe
[jira] [Commented] (HADOOP-16954) Add -S option in "Count" command to show only Snapshot Counts
[
https://issues.apache.org/jira/browse/HADOOP-16954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17074307#comment-17074307
]
Ayush Saxena commented on HADOOP-16954:
---
Is the full stop wrongly placed or am I understanding the sentence wrong?
{code:java}
448* Return the string representation of the snapshot counts in the
output.
449* format
{code}
> Add -S option in "Count" command to show only Snapshot Counts
> -
>
> Key: HADOOP-16954
> URL: https://issues.apache.org/jira/browse/HADOOP-16954
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: hemanthboyina
>Assignee: hemanthboyina
>Priority: Major
> Attachments: HADOOP-16954.003.after.JPG, HADOOP-16954.003.patch,
> HDFS-15233.001.patch, HDFS-15233.002.output.png, HDFS-15233.002.patch,
> quota.output.png
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16952) ADD .diff to gitignore
[ https://issues.apache.org/jira/browse/HADOOP-16952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17074305#comment-17074305 ] Vinayakumar B commented on HADOOP-16952: +1 > ADD .diff to gitignore > -- > > Key: HADOOP-16952 > URL: https://issues.apache.org/jira/browse/HADOOP-16952 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Minor > Attachments: HADOOP-16952-01.patch > > > Add .diff to gitignore. > Else on git add . > it even stages the .diff files too. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16954) Add -S option in "Count" command to show only Snapshot Counts
[ https://issues.apache.org/jira/browse/HADOOP-16954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17074293#comment-17074293 ] Takanobu Asanuma commented on HADOOP-16954: --- +1 on [^ HADOOP-16954.003.patch], pending jenkins. > Add -S option in "Count" command to show only Snapshot Counts > - > > Key: HADOOP-16954 > URL: https://issues.apache.org/jira/browse/HADOOP-16954 > Project: Hadoop Common > Issue Type: Improvement >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: HADOOP-16954.003.after.JPG, HADOOP-16954.003.patch, > HDFS-15233.001.patch, HDFS-15233.002.output.png, HDFS-15233.002.patch, > quota.output.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[
https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17074290#comment-17074290
]
Masatake Iwasaki commented on HADOOP-16647:
---
[~rakeshr] I got no issue with the patch applied on Debian 9 with openssl
1.1.0l-1~deb9u1.
I'm +1 if formatting nits are addressed. {{#if}} and {{#endif}} should not have
leading whitespaces. The code inside the conditional should have same
indentation level with the outside.
> Support OpenSSL 1.1.1 LTS
> -
>
> Key: HADOOP-16647
> URL: https://issues.apache.org/jira/browse/HADOOP-16647
> Project: Hadoop Common
> Issue Type: Task
> Components: security
>Reporter: Wei-Chiu Chuang
>Assignee: Rakesh Radhakrishnan
>Priority: Critical
> Attachments: HADOOP-16647-00.patch, HADOOP-16647-01.patch
>
>
> See Hadoop user mailing list
> http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E
> Hadoop 2 supports OpenSSL 1.0.2.
> Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too.
> Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html
> * 1.1.0 is EOL 2019/09/11
> * 1.0.2 EOL 2019/12/31
> * 1.1.1 is EOL 2023/09/11 (LTS)
> Many Hadoop installation relies on the OpenSSL package provided by Linux
> distros, but it's not clear to me if Linux distros are going support
> 1.1.0/1.0.2 beyond this date.
> We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the
> openssl version supported. File this jira to test/document/fix bugs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16954) Add -S option in "Count" command to show only Snapshot Counts
[ https://issues.apache.org/jira/browse/HADOOP-16954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HADOOP-16954: -- Attachment: HADOOP-16954.003.patch > Add -S option in "Count" command to show only Snapshot Counts > - > > Key: HADOOP-16954 > URL: https://issues.apache.org/jira/browse/HADOOP-16954 > Project: Hadoop Common > Issue Type: Improvement >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: HADOOP-16954.003.after.JPG, HADOOP-16954.003.patch, > HDFS-15233.001.patch, HDFS-15233.002.output.png, HDFS-15233.002.patch, > quota.output.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16954) Add -S option in "Count" command to show only Snapshot Counts
[ https://issues.apache.org/jira/browse/HADOOP-16954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HADOOP-16954: -- Attachment: (was: HADOOP-16954.003.patch) > Add -S option in "Count" command to show only Snapshot Counts > - > > Key: HADOOP-16954 > URL: https://issues.apache.org/jira/browse/HADOOP-16954 > Project: Hadoop Common > Issue Type: Improvement >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: HADOOP-16954.003.after.JPG, HADOOP-16954.003.patch, > HDFS-15233.001.patch, HDFS-15233.002.output.png, HDFS-15233.002.patch, > quota.output.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16954) Add -S option in "Count" command to show only Snapshot Counts
[ https://issues.apache.org/jira/browse/HADOOP-16954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17074267#comment-17074267 ] Takanobu Asanuma commented on HADOOP-16954: --- Thanks for updating the patch. I'd like to resubmit HADOOP-16954.003.patch to run jenkins. > Add -S option in "Count" command to show only Snapshot Counts > - > > Key: HADOOP-16954 > URL: https://issues.apache.org/jira/browse/HADOOP-16954 > Project: Hadoop Common > Issue Type: Improvement >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: HADOOP-16954.003.after.JPG, HADOOP-16954.003.patch, > HDFS-15233.001.patch, HDFS-15233.002.output.png, HDFS-15233.002.patch, > quota.output.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[ https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17074212#comment-17074212 ] Wei-Chiu Chuang commented on HADOOP-16647: -- The cc warning doesn't look related. We can file a jira to remove that warning. Re: OpenSSLSecureRandom Hadoop 3.3.0 makes OpenSSLSecureRandom the default (HADOOP-16011) previously openssl crypto uses OsSecureRandom. Maybe that's we only see the runtime error now. > Support OpenSSL 1.1.1 LTS > - > > Key: HADOOP-16647 > URL: https://issues.apache.org/jira/browse/HADOOP-16647 > Project: Hadoop Common > Issue Type: Task > Components: security >Reporter: Wei-Chiu Chuang >Assignee: Rakesh Radhakrishnan >Priority: Critical > Attachments: HADOOP-16647-00.patch, HADOOP-16647-01.patch > > > See Hadoop user mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E > Hadoop 2 supports OpenSSL 1.0.2. > Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too. > Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html > * 1.1.0 is EOL 2019/09/11 > * 1.0.2 EOL 2019/12/31 > * 1.1.1 is EOL 2023/09/11 (LTS) > Many Hadoop installation relies on the OpenSSL package provided by Linux > distros, but it's not clear to me if Linux distros are going support > 1.1.0/1.0.2 beyond this date. > We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the > openssl version supported. File this jira to test/document/fix bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jiwq commented on issue #1935: MAPREDUCE-7268. Fix TestMapreduceConfigFields
jiwq commented on issue #1935: MAPREDUCE-7268. Fix TestMapreduceConfigFields URL: https://github.com/apache/hadoop/pull/1935#issuecomment-608191178 Thanks @aajisaka for the work. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka merged pull request #1935: MAPREDUCE-7268. Fix TestMapreduceConfigFields
aajisaka merged pull request #1935: MAPREDUCE-7268. Fix TestMapreduceConfigFields URL: https://github.com/apache/hadoop/pull/1935 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16947) Expired SampleStat should ignore when generating SlowPeersReport
[
https://issues.apache.org/jira/browse/HADOOP-16947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17074078#comment-17074078
]
Hadoop QA commented on HADOOP-16947:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
38s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
20m 9s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 31s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 2s{color}
| {color:red} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}105m
34s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
47s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}227m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ha.TestZKFailoverController |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 Image:yetus/hadoop:4454c6d14b7 |
| JIRA Issue | HADOOP-16947 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12998641/HADOOP-16947-005.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 58bb4b40cc71 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 25361b0 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16814/a
[GitHub] [hadoop] hadoop-yetus commented on issue #1861: HADOOP-13230. Optionally retain directory markers
hadoop-yetus commented on issue #1861: HADOOP-13230. Optionally retain directory markers URL: https://github.com/apache/hadoop/pull/1861#issuecomment-608048937 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 11s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 10 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 21m 36s | trunk passed | | +1 :green_heart: | compile | 18m 6s | trunk passed | | +1 :green_heart: | checkstyle | 2m 47s | trunk passed | | +1 :green_heart: | mvnsite | 2m 8s | trunk passed | | +1 :green_heart: | shadedclient | 22m 19s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 28s | trunk passed | | +0 :ok: | spotbugs | 1m 13s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 24s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 1m 25s | the patch passed | | +1 :green_heart: | compile | 25m 7s | the patch passed | | +1 :green_heart: | javac | 25m 7s | the patch passed | | -0 :warning: | checkstyle | 4m 6s | root: The patch generated 40 new + 42 unchanged - 0 fixed = 82 total (was 42) | | +1 :green_heart: | mvnsite | 3m 23s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 22m 31s | patch has no errors when building and testing our client artifacts. | | -1 :x: | javadoc | 0m 52s | hadoop-tools_hadoop-aws generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4) | | +1 :green_heart: | findbugs | 5m 21s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 13m 25s | hadoop-common in the patch passed. | | -1 :x: | unit | 2m 25s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 59s | The patch does not generate ASF License warnings. | | | | 155m 17s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.fs.s3a.TestS3AGetFileStatus | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/7/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1861 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux 258a331795e4 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 25361b0 | | Default Java | 1.8.0_242 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/7/artifact/out/diff-checkstyle-root.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/7/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/7/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/7/testReport/ | | Max. process+thread count | 2831 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/7/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issu
[jira] [Commented] (HADOOP-16954) Add -S option in "Count" command to show only Snapshot Counts
[
https://issues.apache.org/jira/browse/HADOOP-16954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073975#comment-17073975
]
Hadoop QA commented on HADOOP-16954:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HADOOP-16954 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-16954 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16816/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add -S option in "Count" command to show only Snapshot Counts
> -
>
> Key: HADOOP-16954
> URL: https://issues.apache.org/jira/browse/HADOOP-16954
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: hemanthboyina
>Assignee: hemanthboyina
>Priority: Major
> Attachments: HADOOP-16954.003.after.JPG, HADOOP-16954.003.patch,
> HDFS-15233.001.patch, HDFS-15233.002.output.png, HDFS-15233.002.patch,
> quota.output.png
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16954) Add -S option in "Count" command to show only Snapshot Counts
[ https://issues.apache.org/jira/browse/HADOOP-16954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073973#comment-17073973 ] hemanthboyina commented on HADOOP-16954: thanks for the review [~tasanuma] have updated the patch , please review snapshot output after 003.patch: !HADOOP-16954.003.after.JPG! > Add -S option in "Count" command to show only Snapshot Counts > - > > Key: HADOOP-16954 > URL: https://issues.apache.org/jira/browse/HADOOP-16954 > Project: Hadoop Common > Issue Type: Improvement >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: HADOOP-16954.003.after.JPG, HADOOP-16954.003.patch, > HDFS-15233.001.patch, HDFS-15233.002.output.png, HDFS-15233.002.patch, > quota.output.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16954) Add -S option in "Count" command to show only Snapshot Counts
[
https://issues.apache.org/jira/browse/HADOOP-16954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073971#comment-17073971
]
Hadoop QA commented on HADOOP-16954:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} HADOOP-16954 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-16954 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16815/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add -S option in "Count" command to show only Snapshot Counts
> -
>
> Key: HADOOP-16954
> URL: https://issues.apache.org/jira/browse/HADOOP-16954
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: hemanthboyina
>Assignee: hemanthboyina
>Priority: Major
> Attachments: HADOOP-16954.003.after.JPG, HADOOP-16954.003.patch,
> HDFS-15233.001.patch, HDFS-15233.002.output.png, HDFS-15233.002.patch,
> quota.output.png
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16954) Add -S option in "Count" command to show only Snapshot Counts
[ https://issues.apache.org/jira/browse/HADOOP-16954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hemanthboyina updated HADOOP-16954: --- Attachment: HADOOP-16954.003.after.JPG > Add -S option in "Count" command to show only Snapshot Counts > - > > Key: HADOOP-16954 > URL: https://issues.apache.org/jira/browse/HADOOP-16954 > Project: Hadoop Common > Issue Type: Improvement >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: HADOOP-16954.003.after.JPG, HADOOP-16954.003.patch, > HDFS-15233.001.patch, HDFS-15233.002.output.png, HDFS-15233.002.patch, > quota.output.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16954) Add -S option in "Count" command to show only Snapshot Counts
[ https://issues.apache.org/jira/browse/HADOOP-16954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hemanthboyina updated HADOOP-16954: --- Attachment: HADOOP-16954.003.patch > Add -S option in "Count" command to show only Snapshot Counts > - > > Key: HADOOP-16954 > URL: https://issues.apache.org/jira/browse/HADOOP-16954 > Project: Hadoop Common > Issue Type: Improvement >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: HADOOP-16954.003.patch, HDFS-15233.001.patch, > HDFS-15233.002.output.png, HDFS-15233.002.patch, quota.output.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3
hadoop-yetus commented on issue #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3 URL: https://github.com/apache/hadoop/pull/1936#issuecomment-608020742 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 19m 25s | trunk passed | | +1 :green_heart: | compile | 0m 29s | trunk passed | | +1 :green_heart: | checkstyle | 0m 28s | trunk passed | | +1 :green_heart: | mvnsite | 0m 31s | trunk passed | | +1 :green_heart: | shadedclient | 15m 21s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 23s | trunk passed | | +0 :ok: | spotbugs | 0m 47s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 0m 45s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 26s | the patch passed | | +1 :green_heart: | compile | 0m 21s | the patch passed | | +1 :green_heart: | javac | 0m 21s | the patch passed | | +1 :green_heart: | checkstyle | 0m 18s | the patch passed | | +1 :green_heart: | mvnsite | 0m 23s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 13m 50s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 18s | the patch passed | | +1 :green_heart: | findbugs | 0m 46s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 11m 0s | hadoop-distcp in the patch passed. | | +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. | | | | 67m 13s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.tools.util.TestDistCpUtils | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1936/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1936 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux b0dd8d2ac766 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 25361b0 | | Default Java | 1.8.0_242 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1936/1/artifact/out/patch-unit-hadoop-tools_hadoop-distcp.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1936/1/testReport/ | | Max. process+thread count | 482 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1936/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-8143) Change distcp to have -pb on by default
[
https://issues.apache.org/jira/browse/HADOOP-8143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073938#comment-17073938
]
Mithun Radhakrishnan commented on HADOOP-8143:
--
Thank you for pointing to HDFS-13056. This should address the crux of the
problem, i.e. decoupling checksums from block-size. I have only perused it
briefly, but the following section from the HDFS-13056's description is
promising:
{quote}This option can be enabled or disabled at the granularity of individual
client calls by setting the new configuration option
`dfs.checksum.combine.mode` to `COMPOSITE_CRC`
{quote}
It appears that this doesn't require opt-in on HDFS/Name-node, and that
querying for a file's checksum with
{{`dfs.checksum.combine.mode=COMPOSITE_CRC`}} should return a CRC independent
of block-size.
If this holds, perhaps DistCp should be changed to fetch CRCs thus, freeing us
of requiring to preserve block-size for the sake of correctness. (It'll only
hold on Hadoop 3.1.1+.)
{quote}At the very least, we need a way to turn this new default off.
Especially when -skipCrcCheck is true.
{quote}
I'm a little rusty, but it surprises me that block-size preservation isn't
turned off when {{`-skipCrcCheck && (!-pb)`}}. If this isn't so, then that's an
oversight and needs fixing. As a workaround, specifying `-pu`, for instance,
should disable block-size preservation.
> Change distcp to have -pb on by default
> ---
>
> Key: HADOOP-8143
> URL: https://issues.apache.org/jira/browse/HADOOP-8143
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Dave Thompson
>Assignee: Mithun Radhakrishnan
>Priority: Minor
> Fix For: 3.0.0-alpha4
>
> Attachments: HADOOP-8143.1.patch, HADOOP-8143.2.patch,
> HADOOP-8143.3.patch
>
>
> We should have the preserve blocksize (-pb) on in distcp by default.
> checksum which is on by default will always fail if blocksize is not the same.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16955) Tencent Cloud COS: Optimize the upload buffer
[ https://issues.apache.org/jira/browse/HADOOP-16955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YangY updated HADOOP-16955: --- Description: Provides three types of upload buffers: mapped_disk (MappedByteBuffer), on-heap and off-heap memory. In addition, fixed the issue that the buffer may fail to release when some exceptions occur. was: Provides three types of upload buffers: mapped_disk (MappedByteBuffer), on-heap and off-heap memory. In addition, fixed the issue that the buffer may fail to return when some exceptions occur. > Tencent Cloud COS: Optimize the upload buffer > - > > Key: HADOOP-16955 > URL: https://issues.apache.org/jira/browse/HADOOP-16955 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/cos >Affects Versions: 3.3.0 >Reporter: YangY >Assignee: YangY >Priority: Major > Original Estimate: 48h > Remaining Estimate: 48h > > Provides three types of upload buffers: mapped_disk (MappedByteBuffer), > on-heap and off-heap memory. > > In addition, fixed the issue that the buffer may fail to release when some > exceptions occur. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16955) Tencent Cloud COS: Optimize the upload buffer
YangY created HADOOP-16955: -- Summary: Tencent Cloud COS: Optimize the upload buffer Key: HADOOP-16955 URL: https://issues.apache.org/jira/browse/HADOOP-16955 Project: Hadoop Common Issue Type: Improvement Components: fs/cos Affects Versions: 3.3.0 Reporter: YangY Assignee: YangY Provides three types of upload buffers: mapped_disk (MappedByteBuffer), on-heap and off-heap memory. In addition, fixed the issue that the buffer may fail to return when some exceptions occur. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-16955) Tencent Cloud COS: Optimize the upload buffer
[ https://issues.apache.org/jira/browse/HADOOP-16955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-16955 started by YangY. -- > Tencent Cloud COS: Optimize the upload buffer > - > > Key: HADOOP-16955 > URL: https://issues.apache.org/jira/browse/HADOOP-16955 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/cos >Affects Versions: 3.3.0 >Reporter: YangY >Assignee: YangY >Priority: Major > Original Estimate: 48h > Remaining Estimate: 48h > > Provides three types of upload buffers: mapped_disk (MappedByteBuffer), > on-heap and off-heap memory. > > In addition, fixed the issue that the buffer may fail to return when some > exceptions occur. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] liuml07 edited a comment on issue #1875: HADOOP-16794. S3A reverts KMS encryption to the bucket's default KMS …
liuml07 edited a comment on issue #1875: HADOOP-16794. S3A reverts KMS encryption to the bucket's default KMS … URL: https://github.com/apache/hadoop/pull/1875#issuecomment-607979327 @steveloughran Thanks. I think bulk backporting 3.3 stuff to 3.2 for most if not all hadoop-aws module changes makes sense. I see [HADOOP-15619 ](https://issues.apache.org/jira/browse/HADOOP-15619) is already fixed. Is this a good time to do the bulk backport? Specially, I think we need to backport this security fix to 2.10 branch. We locally backported (without test coding due to conflicts) and it works well on that branch so far. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on issue #1875: HADOOP-16794. S3A reverts KMS encryption to the bucket's default KMS …
liuml07 commented on issue #1875: HADOOP-16794. S3A reverts KMS encryption to the bucket's default KMS … URL: https://github.com/apache/hadoop/pull/1875#issuecomment-607979327 @steveloughran Thanks. I think bulk backporting 3.3 stuff to 3.2 for most if not all hadoop-aws module makes sense. I see [HADOOP-15619 ](https://issues.apache.org/jira/browse/HADOOP-15619) is already fixed. Is this a good time to do the bulk backport? Specially, I think we need to backport this security fix to 2.10 branch. We locally backported (without test coding due to conflicts) and it works well on that branch so far. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3
steveloughran commented on issue #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3 URL: https://github.com/apache/hadoop/pull/1936#issuecomment-607972418 adds a quick fix and tests; verified that without the changes in production code the test fails ``` java.io.FileNotFoundException: File does not exist: /tmp/testSkipsNeedlessAttributes/dest at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1729) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1722) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1737) at org.apache.hadoop.tools.util.DistCpUtils.preserve(DistCpUtils.java:210) at org.apache.hadoop.tools.util.TestDistCpUtils.testSkipsNeedlessAttributes(TestDistCpUtils.java:234) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[ https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073903#comment-17073903 ] Rakesh Radhakrishnan commented on HADOOP-16647: --- Thanks a lot [~elukey] for the detailed reviews and suggestions. > Support OpenSSL 1.1.1 LTS > - > > Key: HADOOP-16647 > URL: https://issues.apache.org/jira/browse/HADOOP-16647 > Project: Hadoop Common > Issue Type: Task > Components: security >Reporter: Wei-Chiu Chuang >Assignee: Rakesh Radhakrishnan >Priority: Critical > Attachments: HADOOP-16647-00.patch, HADOOP-16647-01.patch > > > See Hadoop user mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E > Hadoop 2 supports OpenSSL 1.0.2. > Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too. > Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html > * 1.1.0 is EOL 2019/09/11 > * 1.0.2 EOL 2019/12/31 > * 1.1.1 is EOL 2023/09/11 (LTS) > Many Hadoop installation relies on the OpenSSL package provided by Linux > distros, but it's not clear to me if Linux distros are going support > 1.1.0/1.0.2 beyond this date. > We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the > openssl version supported. File this jira to test/document/fix bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran opened a new pull request #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3
steveloughran opened a new pull request #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3 URL: https://github.com/apache/hadoop/pull/1936 Contributed by Steve Loughran. Strips out all the attributes which have already been processed, before deciding whether or not to query the far end for the status of the file. This will avoid 404 caching-related issues in S3, and save time and a bit of IO everywhere. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[ https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073900#comment-17073900 ] Rakesh Radhakrishnan commented on HADOOP-16647: --- Thanks a lot [~iwasakims] for testing in different env and its really a useful feedback. > Support OpenSSL 1.1.1 LTS > - > > Key: HADOOP-16647 > URL: https://issues.apache.org/jira/browse/HADOOP-16647 > Project: Hadoop Common > Issue Type: Task > Components: security >Reporter: Wei-Chiu Chuang >Assignee: Rakesh Radhakrishnan >Priority: Critical > Attachments: HADOOP-16647-00.patch, HADOOP-16647-01.patch > > > See Hadoop user mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E > Hadoop 2 supports OpenSSL 1.0.2. > Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too. > Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html > * 1.1.0 is EOL 2019/09/11 > * 1.0.2 EOL 2019/12/31 > * 1.1.1 is EOL 2023/09/11 (LTS) > Many Hadoop installation relies on the OpenSSL package provided by Linux > distros, but it's not clear to me if Linux distros are going support > 1.1.0/1.0.2 beyond this date. > We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the > openssl version supported. File this jira to test/document/fix bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16947) Expired SampleStat should ignore when generating SlowPeersReport
[
https://issues.apache.org/jira/browse/HADOOP-16947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073894#comment-17073894
]
Haibin Huang commented on HADOOP-16947:
---
update patch v005
> Expired SampleStat should ignore when generating SlowPeersReport
>
>
> Key: HADOOP-16947
> URL: https://issues.apache.org/jira/browse/HADOOP-16947
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Haibin Huang
>Assignee: Haibin Huang
>Priority: Major
> Attachments: HADOOP-16947-001.patch, HADOOP-16947-002.patch,
> HADOOP-16947-003.patch, HADOOP-16947-004.patch, HADOOP-16947-005.patch,
> HDFS-14783, HDFS-14783-001.patch, HDFS-14783-002.patch, HDFS-14783-003.patch,
> HDFS-14783-004.patch, HDFS-14783-005.patch
>
>
> SlowPeersReport is generated by the SampleStat between tow dn, so it can
> present on nn's jmx like this:
> {code:java}
> "SlowPeersReport" :[{"SlowNode":"dn2","ReportingNodes":["dn1"]}]
> {code}
> In each period, MutableRollingAverages will do a rollOverAvgs(), it will
> generate a SumAndCount object which is based on SampleStat, and store it in a
> LinkedBlockingDeque, the deque will be used to generate
> SlowPeersReport. And the old member of deque won't be removed until the queue
> is full. However, if dn1 don't send any packet to dn2 in the last of
> 36*300_000 ms, the deque will be filled with an old member, because the
> number of last SampleStat never change.I think these old SampleStats should
> be considered as expired message and ignore them when generating a new
> SlowPeersReport.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16947) Expired SampleStat should ignore when generating SlowPeersReport
[
https://issues.apache.org/jira/browse/HADOOP-16947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibin Huang updated HADOOP-16947:
--
Attachment: HADOOP-16947-005.patch
> Expired SampleStat should ignore when generating SlowPeersReport
>
>
> Key: HADOOP-16947
> URL: https://issues.apache.org/jira/browse/HADOOP-16947
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Haibin Huang
>Assignee: Haibin Huang
>Priority: Major
> Attachments: HADOOP-16947-001.patch, HADOOP-16947-002.patch,
> HADOOP-16947-003.patch, HADOOP-16947-004.patch, HADOOP-16947-005.patch,
> HDFS-14783, HDFS-14783-001.patch, HDFS-14783-002.patch, HDFS-14783-003.patch,
> HDFS-14783-004.patch, HDFS-14783-005.patch
>
>
> SlowPeersReport is generated by the SampleStat between tow dn, so it can
> present on nn's jmx like this:
> {code:java}
> "SlowPeersReport" :[{"SlowNode":"dn2","ReportingNodes":["dn1"]}]
> {code}
> In each period, MutableRollingAverages will do a rollOverAvgs(), it will
> generate a SumAndCount object which is based on SampleStat, and store it in a
> LinkedBlockingDeque, the deque will be used to generate
> SlowPeersReport. And the old member of deque won't be removed until the queue
> is full. However, if dn1 don't send any packet to dn2 in the last of
> 36*300_000 ms, the deque will be filled with an old member, because the
> number of last SampleStat never change.I think these old SampleStats should
> be considered as expired message and ignore them when generating a new
> SlowPeersReport.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1861: HADOOP-13230. Optionally retain directory markers
hadoop-yetus removed a comment on issue #1861: HADOOP-13230. Optionally retain directory markers URL: https://github.com/apache/hadoop/pull/1861#issuecomment-598392416 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 20s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 22m 24s | trunk passed | | +1 :green_heart: | compile | 0m 31s | trunk passed | | +1 :green_heart: | checkstyle | 0m 24s | trunk passed | | +1 :green_heart: | mvnsite | 0m 37s | trunk passed | | +1 :green_heart: | shadedclient | 16m 30s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 25s | trunk passed | | +0 :ok: | spotbugs | 0m 58s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 0m 56s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 32s | the patch passed | | +1 :green_heart: | compile | 0m 25s | the patch passed | | +1 :green_heart: | javac | 0m 25s | the patch passed | | -0 :warning: | checkstyle | 0m 17s | hadoop-tools/hadoop-aws: The patch generated 1 new + 12 unchanged - 0 fixed = 13 total (was 12) | | +1 :green_heart: | mvnsite | 0m 31s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 15m 24s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 22s | the patch passed | | +1 :green_heart: | findbugs | 1m 1s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 1m 29s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 27s | The patch does not generate ASF License warnings. | | | | 64m 57s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.fs.s3a.TestS3AGetFileStatus | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1861 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 28818861734a 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 0a9b3c9 | | Default Java | 1.8.0_242 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/3/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/3/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/3/testReport/ | | Max. process+thread count | 342 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/3/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1861: HADOOP-13230. Optionally retain directory markers
hadoop-yetus removed a comment on issue #1861: HADOOP-13230. Optionally retain directory markers URL: https://github.com/apache/hadoop/pull/1861#issuecomment-604011125 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 11s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 22m 4s | trunk passed | | +1 :green_heart: | compile | 0m 34s | trunk passed | | +1 :green_heart: | checkstyle | 0m 23s | trunk passed | | +1 :green_heart: | mvnsite | 0m 45s | trunk passed | | +1 :green_heart: | shadedclient | 16m 49s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 24s | trunk passed | | +0 :ok: | spotbugs | 1m 3s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 1s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 34s | the patch passed | | +1 :green_heart: | compile | 0m 28s | the patch passed | | +1 :green_heart: | javac | 0m 28s | the patch passed | | -0 :warning: | checkstyle | 0m 19s | hadoop-tools/hadoop-aws: The patch generated 16 new + 32 unchanged - 0 fixed = 48 total (was 32) | | +1 :green_heart: | mvnsite | 0m 32s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 15m 22s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 23s | the patch passed | | +1 :green_heart: | findbugs | 1m 10s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 1m 31s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. | | | | 65m 21s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.fs.s3a.TestS3AGetFileStatus | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/6/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1861 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle | | uname | Linux a4b7a71bc7d1 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / e3fbdcb | | Default Java | 1.8.0_242 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/6/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/6/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/6/testReport/ | | Max. process+thread count | 338 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/6/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1861: HADOOP-13230. Optionally retain directory markers
hadoop-yetus removed a comment on issue #1861: HADOOP-13230. Optionally retain directory markers URL: https://github.com/apache/hadoop/pull/1861#issuecomment-592115890 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 18s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 23m 5s | trunk passed | | +1 :green_heart: | compile | 0m 32s | trunk passed | | +1 :green_heart: | checkstyle | 0m 24s | trunk passed | | +1 :green_heart: | mvnsite | 0m 37s | trunk passed | | +1 :green_heart: | shadedclient | 17m 52s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 30s | trunk passed | | +0 :ok: | spotbugs | 1m 13s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 11s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 47s | the patch passed | | +1 :green_heart: | compile | 0m 33s | the patch passed | | +1 :green_heart: | javac | 0m 33s | the patch passed | | -0 :warning: | checkstyle | 0m 20s | hadoop-tools/hadoop-aws: The patch generated 1 new + 12 unchanged - 0 fixed = 13 total (was 12) | | +1 :green_heart: | mvnsite | 0m 36s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 17m 57s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 28s | the patch passed | | +1 :green_heart: | findbugs | 1m 16s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 1m 42s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. | | | | 71m 16s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.fs.s3a.TestS3AGetFileStatus | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1861 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 9d7d4b8e62a5 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / cd2c6b1 | | Default Java | 1.8.0_242 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/2/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/2/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/2/testReport/ | | Max. process+thread count | 341 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1861/2/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-8143) Change distcp to have -pb on by default
[ https://issues.apache.org/jira/browse/HADOOP-8143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073870#comment-17073870 ] Steve Loughran commented on HADOOP-8143: This turns out to have caused a couple of regressions when working with object storage. # HADOOP-16932: S3A 404 caching can break copy, because HADOOP-13145 only skips the probe if the attribute set is empty (WiP fix: strip out blocksize, replication and checksum options) # HADOOP-16756 : incremental backups to to s3a, abfs or any other store where the blocksize is just come client-side config options *will now always back up every single file*. Always. Issue number one is straightforward to fix and I am doing so as we speak. Issue number two is different, *because there is now no way to say "I don't want blocksize preserved" * if we do not want checksum validation (-skipCrcCheck), we don't need to preserve block size. * even if we do want checksums, if HDFS-13056 is enabled, checksums are now independent of block size So what to do here? I don't think we need this and for cloud storage it is a major regression. At the very least, we need a way to turn this new default off. Especially when -skipCrcCheck is true. > Change distcp to have -pb on by default > --- > > Key: HADOOP-8143 > URL: https://issues.apache.org/jira/browse/HADOOP-8143 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Dave Thompson >Assignee: Mithun Radhakrishnan >Priority: Minor > Fix For: 3.0.0-alpha4 > > Attachments: HADOOP-8143.1.patch, HADOOP-8143.2.patch, > HADOOP-8143.3.patch > > > We should have the preserve blocksize (-pb) on in distcp by default. > checksum which is on by default will always fail if blocksize is not the same. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16947) Expired SampleStat should ignore when generating SlowPeersReport
[
https://issues.apache.org/jira/browse/HADOOP-16947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073866#comment-17073866
]
Haibin Huang commented on HADOOP-16947:
---
Thanks [~elgoiri], there is a sentence that I don't quite understand.
{quote}Anyway, let's leave it in commons.
{quote}
Do you mean leave the boolean *skipExpired* in hadoop common project?
> Expired SampleStat should ignore when generating SlowPeersReport
>
>
> Key: HADOOP-16947
> URL: https://issues.apache.org/jira/browse/HADOOP-16947
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Haibin Huang
>Assignee: Haibin Huang
>Priority: Major
> Attachments: HADOOP-16947-001.patch, HADOOP-16947-002.patch,
> HADOOP-16947-003.patch, HADOOP-16947-004.patch, HDFS-14783,
> HDFS-14783-001.patch, HDFS-14783-002.patch, HDFS-14783-003.patch,
> HDFS-14783-004.patch, HDFS-14783-005.patch
>
>
> SlowPeersReport is generated by the SampleStat between tow dn, so it can
> present on nn's jmx like this:
> {code:java}
> "SlowPeersReport" :[{"SlowNode":"dn2","ReportingNodes":["dn1"]}]
> {code}
> In each period, MutableRollingAverages will do a rollOverAvgs(), it will
> generate a SumAndCount object which is based on SampleStat, and store it in a
> LinkedBlockingDeque, the deque will be used to generate
> SlowPeersReport. And the old member of deque won't be removed until the queue
> is full. However, if dn1 don't send any packet to dn2 in the last of
> 36*300_000 ms, the deque will be filled with an old member, because the
> number of last SampleStat never change.I think these old SampleStats should
> be considered as expired message and ignore them when generating a new
> SlowPeersReport.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16582) LocalFileSystem's mkdirs() does not work as expected under viewfs.
[
https://issues.apache.org/jira/browse/HADOOP-16582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073843#comment-17073843
]
Zoltan Haindrich commented on HADOOP-16582:
---
[~ste...@apache.org], [~kihwal] this commit fundamentally changes how
{{mkdirs()}} work:
earlier:
* all plain {{mkdirs(somePath)}} were fast-tracked to {{FileSystem.mkdirs}}
which have rerouted them to {{mkdirs(somePath, somePerm)}} method with some
defaults (which were static)
* an implementation of {{FileSystem}} have only needed implement
"mkdirs(somePath, somePerm)" - because the other was not neccessarily called if
it was always in a {{FilterFileSystem}} or something like that
now:
* especially {{FilterFileSystem}} forwards the call of {{mkdirs(p)}} to the
actual fs implementation...which may skip overriden
{{mkdirs(somPath,somePerm)}} methods
* ...and could cause issues for existing FileSystem implementations
> LocalFileSystem's mkdirs() does not work as expected under viewfs.
> --
>
> Key: HADOOP-16582
> URL: https://issues.apache.org/jira/browse/HADOOP-16582
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>Priority: Major
> Fix For: 2.10.0, 3.3.0, 2.8.6, 2.9.3, 3.1.3, 3.2.2
>
> Attachments: HADOOP-16582.1.patch, HADOOP-16582.patch
>
>
> When {{mkdirs(Path)}} is called against {{LocalFileSystem}}, the
> implementation in {{RawLocalFileSystem}} is called and the directory
> permission is determined by the umask. However, if it is under
> {{ViewFileSystem}}, the default implementation in {{FileSystem}} is called
> and this causes explicit {{chmod()}} to 0777.
> The {{mkdirs(Path)}} method needs to be overriden in
> - ViewFileSystem to avoid calling the default implementation
> - ChRootedFileSystem for proper resolution of viewfs mount table
> - FilterFileSystem to avoid calling the default implementation
> Only then the same method in the target ({{LocalFileSystem}} in this case)
> will be called. Hdfs does not suffer from the same flaw since it applies
> umask in all cases, regardless of what version of {{mkdirs()}} was called.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1935: MAPREDUCE-7268. Fix TestMapreduceConfigFields
hadoop-yetus commented on issue #1935: MAPREDUCE-7268. Fix TestMapreduceConfigFields URL: https://github.com/apache/hadoop/pull/1935#issuecomment-607926199 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 18m 58s | trunk passed | | +1 :green_heart: | compile | 0m 26s | trunk passed | | +1 :green_heart: | checkstyle | 0m 22s | trunk passed | | +1 :green_heart: | mvnsite | 0m 28s | trunk passed | | +1 :green_heart: | shadedclient | 15m 55s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 25s | trunk passed | | +0 :ok: | spotbugs | 0m 42s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 0m 40s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 28s | the patch passed | | +1 :green_heart: | compile | 0m 21s | the patch passed | | +1 :green_heart: | javac | 0m 21s | the patch passed | | +1 :green_heart: | checkstyle | 0m 15s | the patch passed | | +1 :green_heart: | mvnsite | 0m 22s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 16m 5s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 20s | the patch passed | | +1 :green_heart: | findbugs | 0m 42s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 30s | hadoop-mapreduce-client-shuffle in the patch passed. | | +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. | | | | 59m 3s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1935/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1935 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 4ee17ab0df6c 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 25361b0 | | Default Java | 1.8.0_242 | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1935/1/testReport/ | | Max. process+thread count | 400 (vs. ulimit of 5500) | | modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1935/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16932) Distcp - Error: java.io.FileNotFoundException: No such file or directory
[ https://issues.apache.org/jira/browse/HADOOP-16932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073825#comment-17073825 ] Steve Loughran commented on HADOOP-16932: - Caused by HADOOP-8143. Default -p option went from "none" to "blocksize", so distcp always does the getFileStatus call. Because, on incremental backup a HEAD call was made earlier, if there is only an interval of a few seconds between the two operations (i.e. small file) the 404 is still cached, and the second getFileStatus fails. Fix will be to strip out blocksize and checksum options from the preservation flag list before deciding to issue the getFileStatus call based on the list not being empty. > Distcp - Error: java.io.FileNotFoundException: No such file or directory > > > Key: HADOOP-16932 > URL: https://issues.apache.org/jira/browse/HADOOP-16932 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3, tools/distcp >Affects Versions: 3.0.0, 3.1.2, 3.2.1 > Environment: Hadoop CDH 6.3 >Reporter: Thangamani Murugasamy >Priority: Minor > > Distcp to AWS s3 was working fine on CDH 5.16 with distcp 2.6.0. but after > upgrade to CDH 6.3 which comes with distcp-3.0 JAR which is through error as > below. > The same error with repeats on Hadoop-distcp-3.2.1.jar as well. Tried with > -direct option in 3.2.1, still same error. > > Error: java.io.FileNotFoundException: No such file or directory: > s3a://X/part-00012-baa6a706-3816-4dfa-ba07-0fb56fd38178-c000.snappy.parquet > at > org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2255) > at > org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2149) > at > org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2088) > at org.apache.hadoop.tools.util.DistCpUtils.preserve(DistCpUtils.java:203) > at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:220) > at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:48) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16756) distcp -update to S3A always overwrites due to block size mismatch
[
https://issues.apache.org/jira/browse/HADOOP-16756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073818#comment-17073818
]
Steve Loughran commented on HADOOP-16756:
-
this is a regression caused by HADOOP-8143. it defaults to preserving block
size, so for all stores where we make the block size up (all the object
stores), it is forcing the copy
> distcp -update to S3A always overwrites due to block size mismatch
> --
>
> Key: HADOOP-16756
> URL: https://issues.apache.org/jira/browse/HADOOP-16756
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Daisuke Kobayashi
>Priority: Major
>
> Distcp over S3A always copies all source files no matter the files are
> changed or not. This is opposite to the statement in the doc below.
> [http://hadoop.apache.org/docs/current/hadoop-distcp/DistCp.html]
> {noformat}
> And to use -update to only copy changed files.
> {noformat}
> CopyMapper compares file length as well as block size before copying. While
> the file length should match, the block size does not. This is apparently
> because the returned block size from S3A is always 32MB.
> [https://github.com/apache/hadoop/blob/release-3.2.0-RC1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java#L348]
> I'd suppose we should update the documentation or make code change.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16947) Expired SampleStat should ignore when generating SlowPeersReport
[
https://issues.apache.org/jira/browse/HADOOP-16947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073815#comment-17073815
]
Íñigo Goiri commented on HADOOP-16947:
--
[~huanghaibin], it makes sense but now we are going to the HDFS way :)
Anyway, let's leave it in commons.
Let's add a one-line javadoc comment to {{skipExpired}} and explain a little
better what setSkipExpired() does.
BTW, let's start the javadocs with capitals.
> Expired SampleStat should ignore when generating SlowPeersReport
>
>
> Key: HADOOP-16947
> URL: https://issues.apache.org/jira/browse/HADOOP-16947
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Haibin Huang
>Assignee: Haibin Huang
>Priority: Major
> Attachments: HADOOP-16947-001.patch, HADOOP-16947-002.patch,
> HADOOP-16947-003.patch, HADOOP-16947-004.patch, HDFS-14783,
> HDFS-14783-001.patch, HDFS-14783-002.patch, HDFS-14783-003.patch,
> HDFS-14783-004.patch, HDFS-14783-005.patch
>
>
> SlowPeersReport is generated by the SampleStat between tow dn, so it can
> present on nn's jmx like this:
> {code:java}
> "SlowPeersReport" :[{"SlowNode":"dn2","ReportingNodes":["dn1"]}]
> {code}
> In each period, MutableRollingAverages will do a rollOverAvgs(), it will
> generate a SumAndCount object which is based on SampleStat, and store it in a
> LinkedBlockingDeque, the deque will be used to generate
> SlowPeersReport. And the old member of deque won't be removed until the queue
> is full. However, if dn1 don't send any packet to dn2 in the last of
> 36*300_000 ms, the deque will be filled with an old member, because the
> number of last SampleStat never change.I think these old SampleStats should
> be considered as expired message and ignore them when generating a new
> SlowPeersReport.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jiwq commented on issue #1935: MAPREDUCE-7268. Fix TestMapreduceConfigFields
jiwq commented on issue #1935: MAPREDUCE-7268. Fix TestMapreduceConfigFields URL: https://github.com/apache/hadoop/pull/1935#issuecomment-607906089 Thanks @ayushtkn for the reporting and reviewing, sorry for the trouble. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[ https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073783#comment-17073783 ] Masatake Iwasaki commented on HADOOP-16647: --- Thanks for working on this, [~rakeshr]. I could reproduce the error "java.lang.UnsatisfiedLinkError: CRYPTO_num_locks" on CentOS 8 with openssl-1.1.1-8.el8.x86_64. I got the error when I tried to put a file into Encryption Zone. There was no error on compilation and checknative. The HADOOP-16647-01.patch fixed the error. I also tried the HADOOP-16647-01.patch on CentOS 7 with openssl-1.0.2k-16.el7_6.1.x86_64. No issue on compilation and runtime. > Support OpenSSL 1.1.1 LTS > - > > Key: HADOOP-16647 > URL: https://issues.apache.org/jira/browse/HADOOP-16647 > Project: Hadoop Common > Issue Type: Task > Components: security >Reporter: Wei-Chiu Chuang >Assignee: Rakesh Radhakrishnan >Priority: Critical > Attachments: HADOOP-16647-00.patch, HADOOP-16647-01.patch > > > See Hadoop user mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E > Hadoop 2 supports OpenSSL 1.0.2. > Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too. > Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html > * 1.1.0 is EOL 2019/09/11 > * 1.0.2 EOL 2019/12/31 > * 1.1.1 is EOL 2023/09/11 (LTS) > Many Hadoop installation relies on the OpenSSL package provided by Linux > distros, but it's not clear to me if Linux distros are going support > 1.1.0/1.0.2 beyond this date. > We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the > openssl version supported. File this jira to test/document/fix bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jiwq opened a new pull request #1935: MAPREDUCE-7268. Fix TestMapreduceConfigFields
jiwq opened a new pull request #1935: MAPREDUCE-7268. Fix TestMapreduceConfigFields URL: https://github.com/apache/hadoop/pull/1935 https://issues.apache.org/jira/browse/MAPREDUCE-7268 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1934: HDFS-15258. RBF: Mark Router FSCK unstable.
hadoop-yetus commented on issue #1934: HDFS-15258. RBF: Mark Router FSCK unstable. URL: https://github.com/apache/hadoop/pull/1934#issuecomment-607862342 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 29m 12s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 22m 4s | trunk passed | | +1 :green_heart: | compile | 0m 31s | trunk passed | | +1 :green_heart: | checkstyle | 0m 20s | trunk passed | | +1 :green_heart: | mvnsite | 0m 35s | trunk passed | | +1 :green_heart: | shadedclient | 16m 25s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 30s | trunk passed | | +0 :ok: | spotbugs | 1m 8s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 6s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 32s | the patch passed | | +1 :green_heart: | compile | 0m 25s | the patch passed | | +1 :green_heart: | javac | 0m 25s | the patch passed | | +1 :green_heart: | checkstyle | 0m 15s | the patch passed | | +1 :green_heart: | mvnsite | 0m 29s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 15m 13s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 26s | the patch passed | | +1 :green_heart: | findbugs | 1m 12s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 9m 45s | hadoop-hdfs-rbf in the patch passed. | | +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. | | | | 101m 43s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1934/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1934 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 9317c9e8b4e8 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 736659e | | Default Java | 1.8.0_242 | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1934/1/testReport/ | | Max. process+thread count | 2913 (vs. ulimit of 5500) | | modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1934/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16954) Add -S option in "Count" command to show only Snapshot Counts
[
https://issues.apache.org/jira/browse/HADOOP-16954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073733#comment-17073733
]
Hadoop QA commented on HADOOP-16954:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} HADOOP-16954 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-16954 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16813/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add -S option in "Count" command to show only Snapshot Counts
> -
>
> Key: HADOOP-16954
> URL: https://issues.apache.org/jira/browse/HADOOP-16954
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: hemanthboyina
>Assignee: hemanthboyina
>Priority: Major
> Attachments: HDFS-15233.001.patch, HDFS-15233.002.output.png,
> HDFS-15233.002.patch, quota.output.png
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16954) Add -S option in "Count" command to show only Snapshot Counts
[ https://issues.apache.org/jira/browse/HADOOP-16954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073714#comment-17073714 ] Ayush Saxena commented on HADOOP-16954: --- moved to hadoop-common since the changes aren't in HDFS > Add -S option in "Count" command to show only Snapshot Counts > - > > Key: HADOOP-16954 > URL: https://issues.apache.org/jira/browse/HADOOP-16954 > Project: Hadoop Common > Issue Type: Improvement >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: HDFS-15233.001.patch, HDFS-15233.002.output.png, > HDFS-15233.002.patch, quota.output.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Moved] (HADOOP-16954) Add -S option in "Count" command to show only Snapshot Counts
[ https://issues.apache.org/jira/browse/HADOOP-16954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena moved HDFS-15233 to HADOOP-16954: -- Key: HADOOP-16954 (was: HDFS-15233) Project: Hadoop Common (was: Hadoop HDFS) > Add -S option in "Count" command to show only Snapshot Counts > - > > Key: HADOOP-16954 > URL: https://issues.apache.org/jira/browse/HADOOP-16954 > Project: Hadoop Common > Issue Type: Improvement >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: HDFS-15233.001.patch, HDFS-15233.002.output.png, > HDFS-15233.002.patch, quota.output.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka opened a new pull request #1934: HDFS-15258. RBF: Mark Router FSCK unstable.
aajisaka opened a new pull request #1934: HDFS-15258. RBF: Mark Router FSCK unstable. URL: https://github.com/apache/hadoop/pull/1934 JIRA: https://issues.apache.org/jira/browse/HDFS-15258 Add a message to show that the feature is unstable. The APIs are already marked as `@Private` so I didn't touch them. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[
https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073647#comment-17073647
]
Hadoop QA commented on HADOOP-16647:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
44s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
51m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 15m 15s{color} |
{color:red} root generated 1 new + 25 unchanged - 1 fixed = 26 total (was 26)
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 9s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
3s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
44s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 94m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 Image:yetus/hadoop:4454c6d14b7 |
| JIRA Issue | HADOOP-16647 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12998587/HADOOP-16647-01.patch
|
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux 336834ee478d 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 4a3eb10 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| cc |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16812/artifact/out/diff-compile-cc-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16812/testReport/ |
| Max. process+thread count | 1347 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16812/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Support OpenSSL 1.1.1 LTS
> -
>
> Key: HADOOP-16647
> URL: https://issues.apache.org/jira/browse/HADOOP-16647
> Project: Hadoop Common
> Issue Type: Task
> Components: security
>Reporter: Wei-Chiu Chuang
>Assignee: Rakesh Radhakrishnan
>Priority: Critical
> Attachments: HADOOP-16647-00.patch, HADOOP-16647-01.patch
>
>
> See Hadoop user mailing list
> http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%4
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[ https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073640#comment-17073640 ] Luca Toscano commented on HADOOP-16647: --- [~rakeshr] thanks for the clarification, now it looks really better and sound. I am not an expert in Openssl internals so the link provided looks reasonable, maybe given the importance of the patch we could reach our to openssl-user/dev@ to double check. > Support OpenSSL 1.1.1 LTS > - > > Key: HADOOP-16647 > URL: https://issues.apache.org/jira/browse/HADOOP-16647 > Project: Hadoop Common > Issue Type: Task > Components: security >Reporter: Wei-Chiu Chuang >Assignee: Rakesh Radhakrishnan >Priority: Critical > Attachments: HADOOP-16647-00.patch, HADOOP-16647-01.patch > > > See Hadoop user mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E > Hadoop 2 supports OpenSSL 1.0.2. > Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too. > Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html > * 1.1.0 is EOL 2019/09/11 > * 1.0.2 EOL 2019/12/31 > * 1.1.1 is EOL 2023/09/11 (LTS) > Many Hadoop installation relies on the OpenSSL package provided by Linux > distros, but it's not clear to me if Linux distros are going support > 1.1.0/1.0.2 beyond this date. > We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the > openssl version supported. File this jira to test/document/fix bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16947) Expired SampleStat should ignore when generating SlowPeersReport
[
https://issues.apache.org/jira/browse/HADOOP-16947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073612#comment-17073612
]
Haibin Huang commented on HADOOP-16947:
---
Thansk [~elgoiri] comment, i think we can use boolean *skipExpired* to make
sure this feature can only use in DataNodePeerMetrics, can you take a look at
[^HADOOP-16947-004.patch]
> Expired SampleStat should ignore when generating SlowPeersReport
>
>
> Key: HADOOP-16947
> URL: https://issues.apache.org/jira/browse/HADOOP-16947
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Haibin Huang
>Assignee: Haibin Huang
>Priority: Major
> Attachments: HADOOP-16947-001.patch, HADOOP-16947-002.patch,
> HADOOP-16947-003.patch, HADOOP-16947-004.patch, HDFS-14783,
> HDFS-14783-001.patch, HDFS-14783-002.patch, HDFS-14783-003.patch,
> HDFS-14783-004.patch, HDFS-14783-005.patch
>
>
> SlowPeersReport is generated by the SampleStat between tow dn, so it can
> present on nn's jmx like this:
> {code:java}
> "SlowPeersReport" :[{"SlowNode":"dn2","ReportingNodes":["dn1"]}]
> {code}
> In each period, MutableRollingAverages will do a rollOverAvgs(), it will
> generate a SumAndCount object which is based on SampleStat, and store it in a
> LinkedBlockingDeque, the deque will be used to generate
> SlowPeersReport. And the old member of deque won't be removed until the queue
> is full. However, if dn1 don't send any packet to dn2 in the last of
> 36*300_000 ms, the deque will be filled with an old member, because the
> number of last SampleStat never change.I think these old SampleStats should
> be considered as expired message and ignore them when generating a new
> SlowPeersReport.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on issue #1929: HADOOP-16855. Changing wildfly dependacy scope to compile
bilaharith commented on issue #1929: HADOOP-16855. Changing wildfly dependacy scope to compile URL: https://github.com/apache/hadoop/pull/1929#issuecomment-607757522 **Driver test results using accounts in Central India** mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify **Account with HNS Support** [INFO] Tests run: 52, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 416, Failures: 0, Errors: 0, Skipped: 66 [WARNING] Tests run: 206, Failures: 0, Errors: 0, Skipped: 24 **Account without HNS support** [INFO] Tests run: 52, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 416, Failures: 0, Errors: 0, Skipped: 240 [WARNING] Tests run: 206, Failures: 0, Errors: 0, Skipped: 24 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16855) ABFS: hadoop-dist fails to add wildfly in class path for hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-16855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bilahari T H updated HADOOP-16855: -- Attachment: (was: image-2020-04-02-15-47-30-198.png) > ABFS: hadoop-dist fails to add wildfly in class path for hadoop-azure > - > > Key: HADOOP-16855 > URL: https://issues.apache.org/jira/browse/HADOOP-16855 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sneha Vijayarajan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16855) ABFS: hadoop-dist fails to add wildfly in class path for hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-16855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bilahari T H updated HADOOP-16855: -- Attachment: (was: image-2020-04-02-15-47-39-278.png) > ABFS: hadoop-dist fails to add wildfly in class path for hadoop-azure > - > > Key: HADOOP-16855 > URL: https://issues.apache.org/jira/browse/HADOOP-16855 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sneha Vijayarajan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16855) ABFS: hadoop-dist fails to add wildfly in class path for hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-16855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bilahari T H updated HADOOP-16855: -- Attachment: (was: image-2020-04-02-15-47-01-077.png) > ABFS: hadoop-dist fails to add wildfly in class path for hadoop-azure > - > > Key: HADOOP-16855 > URL: https://issues.apache.org/jira/browse/HADOOP-16855 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sneha Vijayarajan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16855) ABFS: hadoop-dist fails to add wildfly in class path for hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-16855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073579#comment-17073579 ] Bilahari T H edited comment on HADOOP-16855 at 4/2/20, 10:18 AM: - *Driver test results using accounts in Central India* mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify *Account with HNS Support* [INFO] Tests run: 52, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 416, Failures: 0, Errors: 0, Skipped: 66 [WARNING] Tests run: 206, Failures: 0, Errors: 0, Skipped: 24 *Account without HNS support* [INFO] Tests run: 52, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 416, Failures: 0, Errors: 0, Skipped: 240 [WARNING] Tests run: 206, Failures: 0, Errors: 0, Skipped: 24 was (Author: bilahari.th): *Driver test results using accounts in Central India* mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify *Account with HNS Support* [INFO] Tests run: 52, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 416, Failures: 0, Errors: 0, Skipped: 66 [WARNING] Tests run: 206, Failures: 0, Errors: 0, Skipped: 24 *Account without HNS support* [INFO] Tests run: 52, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 416, Failures: 0, Errors: 0, Skipped: 240 [WARNING] Tests run: 206, Failures: 0, Errors: 0, Skipped: 24 > ABFS: hadoop-dist fails to add wildfly in class path for hadoop-azure > - > > Key: HADOOP-16855 > URL: https://issues.apache.org/jira/browse/HADOOP-16855 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sneha Vijayarajan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16855) ABFS: hadoop-dist fails to add wildfly in class path for hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-16855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bilahari T H reassigned HADOOP-16855: - Assignee: Bilahari T H > ABFS: hadoop-dist fails to add wildfly in class path for hadoop-azure > - > > Key: HADOOP-16855 > URL: https://issues.apache.org/jira/browse/HADOOP-16855 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sneha Vijayarajan >Assignee: Bilahari T H >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16855) ABFS: hadoop-dist fails to add wildfly in class path for hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-16855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073579#comment-17073579 ] Bilahari T H commented on HADOOP-16855: --- *Driver test results using accounts in Central India* mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify *Account with HNS Support* [INFO] Tests run: 52, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 416, Failures: 0, Errors: 0, Skipped: 66 [WARNING] Tests run: 206, Failures: 0, Errors: 0, Skipped: 24 *Account without HNS support* [INFO] Tests run: 52, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 416, Failures: 0, Errors: 0, Skipped: 240 [WARNING] Tests run: 206, Failures: 0, Errors: 0, Skipped: 24 > ABFS: hadoop-dist fails to add wildfly in class path for hadoop-azure > - > > Key: HADOOP-16855 > URL: https://issues.apache.org/jira/browse/HADOOP-16855 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sneha Vijayarajan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[ https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rakesh Radhakrishnan updated HADOOP-16647: -- Attachment: HADOOP-16647-01.patch > Support OpenSSL 1.1.1 LTS > - > > Key: HADOOP-16647 > URL: https://issues.apache.org/jira/browse/HADOOP-16647 > Project: Hadoop Common > Issue Type: Task > Components: security >Reporter: Wei-Chiu Chuang >Assignee: Rakesh Radhakrishnan >Priority: Critical > Attachments: HADOOP-16647-00.patch, HADOOP-16647-01.patch > > > See Hadoop user mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E > Hadoop 2 supports OpenSSL 1.0.2. > Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too. > Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html > * 1.1.0 is EOL 2019/09/11 > * 1.0.2 EOL 2019/12/31 > * 1.1.1 is EOL 2023/09/11 (LTS) > Many Hadoop installation relies on the OpenSSL package provided by Linux > distros, but it's not clear to me if Linux distros are going support > 1.1.0/1.0.2 beyond this date. > We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the > openssl version supported. File this jira to test/document/fix bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16647) Support OpenSSL 1.1.1 LTS
[
https://issues.apache.org/jira/browse/HADOOP-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073570#comment-17073570
]
Rakesh Radhakrishnan commented on HADOOP-16647:
---
{quote}Rakesh, can you also check it can compile against openssl 1.0.2 as well
as 1.1.0/1.1.1?
{quote}
[~weichiu] Thanks for providing verification points, will try setup env and
verify in all these versions.
Hi [~elukey],
{quote}What versions of openssl/hadoop/os are you using to test? For example,
if you are on Debian, what version of libcrypto.so you have if you do ls -l
/usr/lib/x86_64-linux-gnu/libcrypto.so ?
{quote}
I am using libcrypto.so.1.1 and OpenSSL 1.1.1 11 Sep 2018
{quote}when I asked to run hadoop checknative as test, the error reported was
EVP_CIPHER_CTX_cleanup
{quote}
Yes, your observation is correct. HADOOP-14597 was not applied in my build and
caused EVP_CIPHER_CTX_cleanup error while exec {{hadoop checknative -a}}. Sorry
for the confusion due to my mistake. Now, I have upgraded the code and hits
{{java.lang.UnsatisfiedLinkError: CRYPTO_num_locks}} error during {{hdfs put
command}}.
Also, am attaching another patch covering only {{ifdefs}} in
OpensslSecureRandom.c
{quote}I am wondering if the locking code needs to happen also for 1.1.1, maybe
using something different than num_lock.
{quote}
Good point. I have referred following links and it says "OpenSSL 1.1.0+ "can be
safely used in multi-threaded applications provided that support for the
underlying OS threading API is built-in"
References:
[link-1|https://stackoverflow.com/questions/58224138/do-i-need-to-use-crypto-locking-functions-for-thread-safety-in-openssl-1-1-0]
and [link-2|https://curl.haxx.se/libcurl/c/threadsafe.html]
> Support OpenSSL 1.1.1 LTS
> -
>
> Key: HADOOP-16647
> URL: https://issues.apache.org/jira/browse/HADOOP-16647
> Project: Hadoop Common
> Issue Type: Task
> Components: security
>Reporter: Wei-Chiu Chuang
>Assignee: Rakesh Radhakrishnan
>Priority: Critical
> Attachments: HADOOP-16647-00.patch
>
>
> See Hadoop user mailing list
> http://mail-archives.apache.org/mod_mbox/hadoop-user/201910.mbox/%3CCADiq6%3DweDFxHTL_7eGwDNnxVCza39y2QYQTSggfLn7mXhMLOdg%40mail.gmail.com%3E
> Hadoop 2 supports OpenSSL 1.0.2.
> Hadoop 3 supports OpenSSL 1.1.0 (HADOOP-14597) and I believe 1.0.2 too.
> Per OpenSSL blog https://www.openssl.org/policies/releasestrat.html
> * 1.1.0 is EOL 2019/09/11
> * 1.0.2 EOL 2019/12/31
> * 1.1.1 is EOL 2023/09/11 (LTS)
> Many Hadoop installation relies on the OpenSSL package provided by Linux
> distros, but it's not clear to me if Linux distros are going support
> 1.1.0/1.0.2 beyond this date.
> We should make sure Hadoop works with OpenSSL 1.1.1, as well as document the
> openssl version supported. File this jira to test/document/fix bugs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16953) remove org.apache.hadoop off s3guard off warning config name
Steve Loughran created HADOOP-16953: --- Summary: remove org.apache.hadoop off s3guard off warning config name Key: HADOOP-16953 URL: https://issues.apache.org/jira/browse/HADOOP-16953 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Reporter: Steve Loughran config option org.apache.hadoop.fs.s3a.s3guard.disabled.warn.level should be fs.s3a.s3guard.disabled.warn.level need to fix that and add the existing one as deprecated -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16953) remove org.apache.hadoop off s3guard off warning config name
[ https://issues.apache.org/jira/browse/HADOOP-16953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16953: Target Version/s: 3.3.1 Affects Version/s: 3.3.0 > remove org.apache.hadoop off s3guard off warning config name > > > Key: HADOOP-16953 > URL: https://issues.apache.org/jira/browse/HADOOP-16953 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Priority: Minor > > config option org.apache.hadoop.fs.s3a.s3guard.disabled.warn.level should be > fs.s3a.s3guard.disabled.warn.level > need to fix that and add the existing one as deprecated -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16947) Expired SampleStat should ignore when generating SlowPeersReport
[
https://issues.apache.org/jira/browse/HADOOP-16947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073446#comment-17073446
]
Hadoop QA commented on HADOOP-16947:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
39s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
20m 19s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 17s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
21s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}108m 24s{color}
| {color:red} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
53s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}229m 11s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 Image:yetus/hadoop:4454c6d14b7 |
| JIRA Issue | HADOOP-16947 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12998541/HADOOP-16947-004.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 1d29880adb1a 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d3b5951 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/Pr
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding Output Stream Counters in ABFS
mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding
Output Stream Counters in ABFS
URL: https://github.com/apache/hadoop/pull/1899#discussion_r402073163
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStreamStatistics.java
##
@@ -0,0 +1,78 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import org.apache.hadoop.classification.InterfaceStability;
+
+/**
+ * Interface for {@link AbfsOutputStream} statistics.
+ */
+@InterfaceStability.Unstable
+public interface AbfsOutputStreamStatistics {
+
+ /**
+ * Number of bytes to be uploaded.
+ *
+ * @param bytes number of bytes to upload
Review comment:
Add trailing . in the end for all java docs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding Output Stream Counters in ABFS
mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding
Output Stream Counters in ABFS
URL: https://github.com/apache/hadoop/pull/1899#discussion_r402075785
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStreamStatisticsImpl.java
##
@@ -0,0 +1,148 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+/**
+ * OutputStream Statistics Implementation for Abfs.
+ * timeSpendOnTaskWait - Time spend on waiting for tasks to be complete on
+ * Blocking Queue in AbfsOutputStream.
+ *
+ * queueShrink - Number of times Blocking Queue was shrunk after writing
+ * data.
+ *
+ * WriteCurrentBufferOperations - Number of times
+ * {@code writeCurrentBufferToService()} calls were made.
+ */
+public class AbfsOutputStreamStatisticsImpl
+implements AbfsOutputStreamStatistics {
+ private volatile long bytesToUpload;
+ private volatile long bytesUploadSuccessful;
+ private volatile long bytesUploadFailed;
+ private volatile long timeSpendOnTaskWait;
+ private volatile long queueShrink;
+ private volatile long writeCurrentBufferOperations;
+
+ /**
+ * Number of bytes uploaded.
+ *
+ * @param bytes negative values are ignored
+ */
+ @Override
+ public void bytesToUpload(long bytes) {
+if (bytes > 0) {
+ bytesToUpload += bytes;
+}
+ }
+
+ /**
+ * Upload successful with the number of bytes.
+ * @param bytes number of bytes that were successfully uploaded
+ */
+ @Override
+ public void uploadSuccessful(long bytes) {
+if (bytes > 0) {
+ bytesUploadSuccessful += bytes;
+}
+ }
+
+ /**
+ * Upload failed and the number of bytes.
+ *
+ * @param bytes negative values are ignored
+ */
+ @Override
+ public void uploadFailed(long bytes) {
+if (bytes > 0) {
+ bytesUploadFailed += bytes;
+}
+ }
+
+ /**
+ * Time spent for waiting a task to be completed.
+ *
+ * @param startTime on calling {@code waitForTaskToComplete()}
Review comment:
Write full method in javadoc like AbfsOutputStream#waitForTaskToComplete()
otherwise people will have to figure out where this method actually is.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding Output Stream Counters in ABFS
mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding Output Stream Counters in ABFS URL: https://github.com/apache/hadoop/pull/1899#discussion_r402070920 ## File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java ## @@ -80,6 +82,7 @@ = new ElasticByteBufferPool(); private final Statistics statistics; + private final AbfsOutputStreamStatisticsImpl outputStreamStatistics; Review comment: We should be defining here using the interface not concrete class and initialising it in constructor with concrete class implementation. That whole point of using interface is we can change add the new implementation in future and change the the implementation to be used in constructor using some configuration. Right @steveloughran? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding Output Stream Counters in ABFS
mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding
Output Stream Counters in ABFS
URL: https://github.com/apache/hadoop/pull/1899#discussion_r402076899
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStreamStatisticsImpl.java
##
@@ -0,0 +1,148 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+/**
+ * OutputStream Statistics Implementation for Abfs.
+ * timeSpendOnTaskWait - Time spend on waiting for tasks to be complete on
+ * Blocking Queue in AbfsOutputStream.
+ *
+ * queueShrink - Number of times Blocking Queue was shrunk after writing
+ * data.
+ *
+ * WriteCurrentBufferOperations - Number of times
+ * {@code writeCurrentBufferToService()} calls were made.
+ */
+public class AbfsOutputStreamStatisticsImpl
+implements AbfsOutputStreamStatistics {
+ private volatile long bytesToUpload;
+ private volatile long bytesUploadSuccessful;
+ private volatile long bytesUploadFailed;
+ private volatile long timeSpendOnTaskWait;
+ private volatile long queueShrink;
+ private volatile long writeCurrentBufferOperations;
+
+ /**
+ * Number of bytes uploaded.
+ *
+ * @param bytes negative values are ignored
+ */
+ @Override
+ public void bytesToUpload(long bytes) {
+if (bytes > 0) {
+ bytesToUpload += bytes;
+}
+ }
+
+ /**
+ * Upload successful with the number of bytes.
+ * @param bytes number of bytes that were successfully uploaded
+ */
+ @Override
+ public void uploadSuccessful(long bytes) {
+if (bytes > 0) {
+ bytesUploadSuccessful += bytes;
+}
+ }
+
+ /**
+ * Upload failed and the number of bytes.
+ *
+ * @param bytes negative values are ignored
+ */
+ @Override
+ public void uploadFailed(long bytes) {
+if (bytes > 0) {
+ bytesUploadFailed += bytes;
+}
+ }
+
+ /**
+ * Time spent for waiting a task to be completed.
+ *
+ * @param startTime on calling {@code waitForTaskToComplete()}
+ * @param endTime on method completing
+ */
+ @Override
+ public void timeSpentTaskWait(long startTime, long endTime) {
+timeSpendOnTaskWait += endTime - startTime;
+ }
+
+ /**
+ * Number of calls to {@code shrinkWriteOperationQueue()}.
+ */
+ @Override
+ public void queueShrinked() {
Review comment:
Once again which queue. How is this metrics important??
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding Output Stream Counters in ABFS
mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding
Output Stream Counters in ABFS
URL: https://github.com/apache/hadoop/pull/1899#discussion_r402076584
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStreamStatisticsImpl.java
##
@@ -0,0 +1,148 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+/**
+ * OutputStream Statistics Implementation for Abfs.
+ * timeSpendOnTaskWait - Time spend on waiting for tasks to be complete on
+ * Blocking Queue in AbfsOutputStream.
+ *
+ * queueShrink - Number of times Blocking Queue was shrunk after writing
+ * data.
+ *
+ * WriteCurrentBufferOperations - Number of times
+ * {@code writeCurrentBufferToService()} calls were made.
+ */
+public class AbfsOutputStreamStatisticsImpl
+implements AbfsOutputStreamStatistics {
+ private volatile long bytesToUpload;
+ private volatile long bytesUploadSuccessful;
+ private volatile long bytesUploadFailed;
+ private volatile long timeSpendOnTaskWait;
+ private volatile long queueShrink;
+ private volatile long writeCurrentBufferOperations;
+
+ /**
+ * Number of bytes uploaded.
+ *
+ * @param bytes negative values are ignored
+ */
+ @Override
+ public void bytesToUpload(long bytes) {
+if (bytes > 0) {
+ bytesToUpload += bytes;
+}
+ }
+
+ /**
+ * Upload successful with the number of bytes.
+ * @param bytes number of bytes that were successfully uploaded
+ */
+ @Override
+ public void uploadSuccessful(long bytes) {
+if (bytes > 0) {
+ bytesUploadSuccessful += bytes;
+}
+ }
+
+ /**
+ * Upload failed and the number of bytes.
+ *
+ * @param bytes negative values are ignored
+ */
+ @Override
+ public void uploadFailed(long bytes) {
+if (bytes > 0) {
+ bytesUploadFailed += bytes;
+}
+ }
+
+ /**
+ * Time spent for waiting a task to be completed.
Review comment:
What is the task here? Please explain. This is for all javadocs. There is no
harm in writing more lines :P
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on a change in pull request #1899: HADOOP-16914 Adding Output Stream Counters in ABFS
mehakmeet commented on a change in pull request #1899: HADOOP-16914 Adding
Output Stream Counters in ABFS
URL: https://github.com/apache/hadoop/pull/1899#discussion_r402095417
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStreamStatisticsImpl.java
##
@@ -0,0 +1,148 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+/**
+ * OutputStream Statistics Implementation for Abfs.
+ * timeSpendOnTaskWait - Time spend on waiting for tasks to be complete on
+ * Blocking Queue in AbfsOutputStream.
+ *
+ * queueShrink - Number of times Blocking Queue was shrunk after writing
+ * data.
+ *
+ * WriteCurrentBufferOperations - Number of times
+ * {@code writeCurrentBufferToService()} calls were made.
+ */
Review comment:
this java doc above variable name
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on a change in pull request #1899: HADOOP-16914 Adding Output Stream Counters in ABFS
mehakmeet commented on a change in pull request #1899: HADOOP-16914 Adding
Output Stream Counters in ABFS
URL: https://github.com/apache/hadoop/pull/1899#discussion_r402095177
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStreamStatistics.java
##
@@ -0,0 +1,78 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import org.apache.hadoop.classification.InterfaceStability;
+
+/**
+ * Interface for {@link AbfsOutputStream} statistics.
+ */
+@InterfaceStability.Unstable
+public interface AbfsOutputStreamStatistics {
+
+ /**
+ * Number of bytes to be uploaded.
+ *
+ * @param bytes number of bytes to upload
+ */
+ void bytesToUpload(long bytes);
+
+ /**
+ * Records a successful upload and the number of bytes uploaded.
+ *
+ * @param bytes number of bytes that were successfully uploaded
+ */
+ void uploadSuccessful(long bytes);
+
+ /**
+ * Records that upload is failed and the number of bytes.
+ *
+ * @param bytes number of bytes that failed to upload
+ */
+ void uploadFailed(long bytes);
+
+ /**
+ * Time spent in waiting for tasks to be completed in the blocking Queue.
Review comment:
explain more
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
