[jira] [Updated] (HADOOP-17370) Upgrade commons-compress to 1.21
[ https://issues.apache.org/jira/browse/HADOOP-17370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-17370: --- Target Version/s: 3.4.0, 2.10.2, 3.2.3, 3.3.2 > Upgrade commons-compress to 1.21 > > > Key: HADOOP-17370 > URL: https://issues.apache.org/jira/browse/HADOOP-17370 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.3.0, 3.2.1 >Reporter: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17370) Upgrade commons-compress to 1.21
[ https://issues.apache.org/jira/browse/HADOOP-17370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394537#comment-17394537 ] Akira Ajisaka commented on HADOOP-17370: Now the latest version is 1.21 and several CVEs are fixed. Let's upgrade. - https://nvd.nist.gov/vuln/detail/CVE-2021-35515 - https://nvd.nist.gov/vuln/detail/CVE-2021-35516 - https://nvd.nist.gov/vuln/detail/CVE-2021-36090 > Upgrade commons-compress to 1.21 > > > Key: HADOOP-17370 > URL: https://issues.apache.org/jira/browse/HADOOP-17370 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.3.0, 3.2.1 >Reporter: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17370) Upgrade commons-compress to 1.21
[ https://issues.apache.org/jira/browse/HADOOP-17370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-17370: --- Summary: Upgrade commons-compress to 1.21 (was: Upgrade commons-compress to 1.20) > Upgrade commons-compress to 1.21 > > > Key: HADOOP-17370 > URL: https://issues.apache.org/jira/browse/HADOOP-17370 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.3.0, 3.2.1 >Reporter: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on pull request #3256: HDFS-16151.Improve the parameter comments related to ProtobufRpcEngine2#Server().
jojochuang commented on pull request #3256: URL: https://github.com/apache/hadoop/pull/3256#issuecomment-893973569 triggering the build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17825) Add BuiltInGzipCompressor
[
https://issues.apache.org/jira/browse/HADOOP-17825?focusedWorklogId=634936=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634936
]
ASF GitHub Bot logged work on HADOOP-17825:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 03:16
Start Date: 06/Aug/21 03:16
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#issuecomment-893970892
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 40s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 58s | | trunk passed |
| +1 :green_heart: | compile | 21m 8s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 18m 30s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 9s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 39s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 23s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 3s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 55s | | the patch passed |
| +1 :green_heart: | compile | 22m 22s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 22m 22s | | the patch passed |
| +1 :green_heart: | compile | 20m 48s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 20m 48s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 9s |
[/results-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3250/19/artifact/out/results-checkstyle-hadoop-common-project_hadoop-common.txt)
| hadoop-common-project/hadoop-common: The patch generated 7 new + 332
unchanged - 0 fixed = 339 total (was 332) |
| +1 :green_heart: | mvnsite | 1m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 7s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 35s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 43s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 51s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 34s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 54s | | The patch does not
generate ASF License warnings. |
| | | 182m 50s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3250/19/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3250 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 05889202c053 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d5ac82e295d0dacc890c74786d15758c4b8e51e3 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3250/19/testReport/ |
| Max. process+thread count | 3149 (vs. ulimit of 5500) |
| modules | C:
[GitHub] [hadoop] hadoop-yetus commented on pull request #3250: HADOOP-17825. Add BuiltInGzipCompressor
hadoop-yetus commented on pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#issuecomment-893970892
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 40s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 58s | | trunk passed |
| +1 :green_heart: | compile | 21m 8s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 18m 30s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 9s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 39s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 23s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 3s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 55s | | the patch passed |
| +1 :green_heart: | compile | 22m 22s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 22m 22s | | the patch passed |
| +1 :green_heart: | compile | 20m 48s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 20m 48s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 9s |
[/results-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3250/19/artifact/out/results-checkstyle-hadoop-common-project_hadoop-common.txt)
| hadoop-common-project/hadoop-common: The patch generated 7 new + 332
unchanged - 0 fixed = 339 total (was 332) |
| +1 :green_heart: | mvnsite | 1m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 7s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 35s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 43s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 51s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 34s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 54s | | The patch does not
generate ASF License warnings. |
| | | 182m 50s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3250/19/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3250 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 05889202c053 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d5ac82e295d0dacc890c74786d15758c4b8e51e3 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3250/19/testReport/ |
| Max. process+thread count | 3149 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3250/19/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond
[jira] [Work logged] (HADOOP-17825) Add BuiltInGzipCompressor
[
https://issues.apache.org/jira/browse/HADOOP-17825?focusedWorklogId=634933=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634933
]
ASF GitHub Bot logged work on HADOOP-17825:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 02:52
Start Date: 06/Aug/21 02:52
Worklog Time Spent: 10m
Work Description: sunchao commented on a change in pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#discussion_r683910228
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress.zlib;
+
+import java.io.IOException;
+import java.util.zip.Checksum;
+import java.util.zip.Deflater;
+import java.util.zip.GZIPOutputStream;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.compress.Compressor;
+import org.apache.hadoop.io.compress.DoNotPool;
+import org.apache.hadoop.util.DataChecksum;
+
+/**
+ * A {@link Compressor} based on the popular gzip compressed file format.
+ * http://www.gzip.org/
+ */
+@DoNotPool
+public class BuiltInGzipCompressor implements Compressor {
+
+ /**
+ * Fixed ten-byte gzip header. See {@link GZIPOutputStream}'s source for
+ * details.
+ */
+ private static final byte[] GZIP_HEADER = new byte[]{
+ 0x1f, (byte) 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
+
+ // The trailer will be overwritten based on crc and output size.
+ private byte[] GZIP_TRAILER = new byte[]{0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00};
+
+ private final int GZIP_HEADER_LEN = GZIP_HEADER.length;
+ private final int GZIP_TRAILER_LEN = GZIP_TRAILER.length;
+
+ private Deflater deflater;
+
+ private int headerOff = 0;
+ private int trailerOff = 0;
+
+ private int numBytesWritten = 0;
+
+ private int currentBufLen = 0;
+
+ private final Checksum crc = DataChecksum.newCrc32();
+
+ private BuiltInGzipDecompressor.GzipStateLabel state;
+
+ public BuiltInGzipCompressor(Configuration conf) { init(conf); }
+
+ @Override
+ public boolean finished() {
+// Only if the trailer is also written, it is thought as finished.
+return deflater.finished() && state ==
BuiltInGzipDecompressor.GzipStateLabel.FINISHED;
+ }
+
+ @Override
+ public boolean needsInput() {
+if (state == BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM) {
+ return deflater.needsInput();
+}
+
+return (state != BuiltInGzipDecompressor.GzipStateLabel.FINISHED);
+ }
+
+ @Override
+ public int compress(byte[] b, int off, int len) throws IOException {
+int compressedBytesWritten = 0;
+
+// If we are not within uncompressed data yet, output the header.
+if (state != BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM &&
Review comment:
I think calling finish will set a flag to tell it flush the data when
finished.
Yea the current approach also looks good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 634933)
Time Spent: 14.5h (was: 14h 20m)
> Add BuiltInGzipCompressor
> -
>
> Key: HADOOP-17825
> URL: https://issues.apache.org/jira/browse/HADOOP-17825
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 14.5h
> Remaining Estimate: 0h
>
> Currently, GzipCodec only supports BuiltInGzipDecompressor, if native zlib is
> not loaded. So, without Hadoop native codec installed, saving SequenceFile
> using GzipCodec will throw exception like "SequenceFile doesn't work with
> GzipCodec without
[GitHub] [hadoop] sunchao commented on a change in pull request #3250: HADOOP-17825. Add BuiltInGzipCompressor
sunchao commented on a change in pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#discussion_r683910228
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress.zlib;
+
+import java.io.IOException;
+import java.util.zip.Checksum;
+import java.util.zip.Deflater;
+import java.util.zip.GZIPOutputStream;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.compress.Compressor;
+import org.apache.hadoop.io.compress.DoNotPool;
+import org.apache.hadoop.util.DataChecksum;
+
+/**
+ * A {@link Compressor} based on the popular gzip compressed file format.
+ * http://www.gzip.org/
+ */
+@DoNotPool
+public class BuiltInGzipCompressor implements Compressor {
+
+ /**
+ * Fixed ten-byte gzip header. See {@link GZIPOutputStream}'s source for
+ * details.
+ */
+ private static final byte[] GZIP_HEADER = new byte[]{
+ 0x1f, (byte) 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
+
+ // The trailer will be overwritten based on crc and output size.
+ private byte[] GZIP_TRAILER = new byte[]{0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00};
+
+ private final int GZIP_HEADER_LEN = GZIP_HEADER.length;
+ private final int GZIP_TRAILER_LEN = GZIP_TRAILER.length;
+
+ private Deflater deflater;
+
+ private int headerOff = 0;
+ private int trailerOff = 0;
+
+ private int numBytesWritten = 0;
+
+ private int currentBufLen = 0;
+
+ private final Checksum crc = DataChecksum.newCrc32();
+
+ private BuiltInGzipDecompressor.GzipStateLabel state;
+
+ public BuiltInGzipCompressor(Configuration conf) { init(conf); }
+
+ @Override
+ public boolean finished() {
+// Only if the trailer is also written, it is thought as finished.
+return deflater.finished() && state ==
BuiltInGzipDecompressor.GzipStateLabel.FINISHED;
+ }
+
+ @Override
+ public boolean needsInput() {
+if (state == BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM) {
+ return deflater.needsInput();
+}
+
+return (state != BuiltInGzipDecompressor.GzipStateLabel.FINISHED);
+ }
+
+ @Override
+ public int compress(byte[] b, int off, int len) throws IOException {
+int compressedBytesWritten = 0;
+
+// If we are not within uncompressed data yet, output the header.
+if (state != BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM &&
Review comment:
I think calling finish will set a flag to tell it flush the data when
finished.
Yea the current approach also looks good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17808) ipc.Client not setting interrupt flag after catching InterruptedException
[ https://issues.apache.org/jira/browse/HADOOP-17808?focusedWorklogId=634932=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634932 ] ASF GitHub Bot logged work on HADOOP-17808: --- Author: ASF GitHub Bot Created on: 06/Aug/21 02:47 Start Date: 06/Aug/21 02:47 Worklog Time Spent: 10m Work Description: virajjasani commented on pull request #3267: URL: https://github.com/apache/hadoop/pull/3267#issuecomment-893961654 Thank you -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 634932) Time Spent: 5h 20m (was: 5h 10m) > ipc.Client not setting interrupt flag after catching InterruptedException > - > > Key: HADOOP-17808 > URL: https://issues.apache.org/jira/browse/HADOOP-17808 > Project: Hadoop Common > Issue Type: Task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.2.3, 3.3.2 > > Time Spent: 5h 20m > Remaining Estimate: 0h > > ipc.Client is swallowing InterruptedException at a couple of places: > # While waiting on all connections to be closed > # While waiting to retrieve some RPC response > We should at least set the interrupt signal and also log the > InterruptedException caught. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #3267: HADOOP-17808. Avoid excessive logging for interruption (ADDENDUM)
virajjasani commented on pull request #3267: URL: https://github.com/apache/hadoop/pull/3267#issuecomment-893961654 Thank you -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ferhui commented on pull request #3267: HADOOP-17808. Avoid excessive logging for interruption (ADDENDUM)
ferhui commented on pull request #3267: URL: https://github.com/apache/hadoop/pull/3267#issuecomment-893961313 @virajjasani Yes, have pushed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17808) ipc.Client not setting interrupt flag after catching InterruptedException
[ https://issues.apache.org/jira/browse/HADOOP-17808?focusedWorklogId=634930=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634930 ] ASF GitHub Bot logged work on HADOOP-17808: --- Author: ASF GitHub Bot Created on: 06/Aug/21 02:45 Start Date: 06/Aug/21 02:45 Worklog Time Spent: 10m Work Description: ferhui commented on pull request #3267: URL: https://github.com/apache/hadoop/pull/3267#issuecomment-893961313 @virajjasani Yes, have pushed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 634930) Time Spent: 5h 10m (was: 5h) > ipc.Client not setting interrupt flag after catching InterruptedException > - > > Key: HADOOP-17808 > URL: https://issues.apache.org/jira/browse/HADOOP-17808 > Project: Hadoop Common > Issue Type: Task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.2.3, 3.3.2 > > Time Spent: 5h 10m > Remaining Estimate: 0h > > ipc.Client is swallowing InterruptedException at a couple of places: > # While waiting on all connections to be closed > # While waiting to retrieve some RPC response > We should at least set the interrupt signal and also log the > InterruptedException caught. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17808) ipc.Client not setting interrupt flag after catching InterruptedException
[ https://issues.apache.org/jira/browse/HADOOP-17808?focusedWorklogId=634929=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634929 ] ASF GitHub Bot logged work on HADOOP-17808: --- Author: ASF GitHub Bot Created on: 06/Aug/21 02:44 Start Date: 06/Aug/21 02:44 Worklog Time Spent: 10m Work Description: virajjasani commented on pull request #3267: URL: https://github.com/apache/hadoop/pull/3267#issuecomment-893961046 Thanks everyone for your reviews. @ferhui Could you please also help backport this to branch-3.2 as the previous patch is already present there? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 634929) Time Spent: 5h (was: 4h 50m) > ipc.Client not setting interrupt flag after catching InterruptedException > - > > Key: HADOOP-17808 > URL: https://issues.apache.org/jira/browse/HADOOP-17808 > Project: Hadoop Common > Issue Type: Task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.2.3, 3.3.2 > > Time Spent: 5h > Remaining Estimate: 0h > > ipc.Client is swallowing InterruptedException at a couple of places: > # While waiting on all connections to be closed > # While waiting to retrieve some RPC response > We should at least set the interrupt signal and also log the > InterruptedException caught. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17808) ipc.Client not setting interrupt flag after catching InterruptedException
[ https://issues.apache.org/jira/browse/HADOOP-17808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Fei updated HADOOP-17808: - Resolution: Fixed Status: Resolved (was: Patch Available) > ipc.Client not setting interrupt flag after catching InterruptedException > - > > Key: HADOOP-17808 > URL: https://issues.apache.org/jira/browse/HADOOP-17808 > Project: Hadoop Common > Issue Type: Task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.2.3, 3.3.2 > > Time Spent: 4h 50m > Remaining Estimate: 0h > > ipc.Client is swallowing InterruptedException at a couple of places: > # While waiting on all connections to be closed > # While waiting to retrieve some RPC response > We should at least set the interrupt signal and also log the > InterruptedException caught. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #3267: HADOOP-17808. Avoid excessive logging for interruption (ADDENDUM)
virajjasani commented on pull request #3267: URL: https://github.com/apache/hadoop/pull/3267#issuecomment-893961046 Thanks everyone for your reviews. @ferhui Could you please also help backport this to branch-3.2 as the previous patch is already present there? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17808) ipc.Client not setting interrupt flag after catching InterruptedException
[ https://issues.apache.org/jira/browse/HADOOP-17808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394451#comment-17394451 ] Hui Fei commented on HADOOP-17808: -- h1. ADDENDUM patch pushed to branch-3.2 & branch-3.3 > ipc.Client not setting interrupt flag after catching InterruptedException > - > > Key: HADOOP-17808 > URL: https://issues.apache.org/jira/browse/HADOOP-17808 > Project: Hadoop Common > Issue Type: Task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.2.3, 3.3.2 > > Time Spent: 4h 50m > Remaining Estimate: 0h > > ipc.Client is swallowing InterruptedException at a couple of places: > # While waiting on all connections to be closed > # While waiting to retrieve some RPC response > We should at least set the interrupt signal and also log the > InterruptedException caught. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] wzhallright commented on a change in pull request #3269: HDFS-16153. Avoid evaluation of LOG.debug statement in QuorumJournalM…
wzhallright commented on a change in pull request #3269:
URL: https://github.com/apache/hadoop/pull/3269#discussion_r683907289
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java
##
@@ -618,8 +618,7 @@ private void selectStreamingInputStreams(
Map resps =
loggers.waitForWriteQuorum(q, selectInputStreamsTimeoutMs,
"selectStreamingInputStreams");
-
-LOG.debug("selectStreamingInputStream manifests:\n" +
+LOG.debug("selectStreamingInputStream manifests:\n {}",
Review comment:
Thanks for review.I hava fixed it in the latest commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17825) Add BuiltInGzipCompressor
[
https://issues.apache.org/jira/browse/HADOOP-17825?focusedWorklogId=634926=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634926
]
ASF GitHub Bot logged work on HADOOP-17825:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 02:29
Start Date: 06/Aug/21 02:29
Worklog Time Spent: 10m
Work Description: viirya commented on a change in pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#discussion_r683902983
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress.zlib;
+
+import java.io.IOException;
+import java.util.zip.Checksum;
+import java.util.zip.Deflater;
+import java.util.zip.GZIPOutputStream;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.compress.Compressor;
+import org.apache.hadoop.io.compress.DoNotPool;
+import org.apache.hadoop.util.DataChecksum;
+
+/**
+ * A {@link Compressor} based on the popular gzip compressed file format.
+ * http://www.gzip.org/
+ */
+@DoNotPool
+public class BuiltInGzipCompressor implements Compressor {
+
+ /**
+ * Fixed ten-byte gzip header. See {@link GZIPOutputStream}'s source for
+ * details.
+ */
+ private static final byte[] GZIP_HEADER = new byte[]{
+ 0x1f, (byte) 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
+
+ // The trailer will be overwritten based on crc and output size.
+ private byte[] GZIP_TRAILER = new byte[]{0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00};
+
+ private final int GZIP_HEADER_LEN = GZIP_HEADER.length;
+ private final int GZIP_TRAILER_LEN = GZIP_TRAILER.length;
+
+ private Deflater deflater;
+
+ private int headerOff = 0;
+ private int trailerOff = 0;
+
+ private int numBytesWritten = 0;
+
+ private int currentBufLen = 0;
+
+ private final Checksum crc = DataChecksum.newCrc32();
+
+ private BuiltInGzipDecompressor.GzipStateLabel state;
+
+ public BuiltInGzipCompressor(Configuration conf) { init(conf); }
+
+ @Override
+ public boolean finished() {
+// Only if the trailer is also written, it is thought as finished.
+return deflater.finished() && state ==
BuiltInGzipDecompressor.GzipStateLabel.FINISHED;
+ }
+
+ @Override
+ public boolean needsInput() {
+if (state == BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM) {
+ return deflater.needsInput();
+}
+
+return (state != BuiltInGzipDecompressor.GzipStateLabel.FINISHED);
+ }
+
+ @Override
+ public int compress(byte[] b, int off, int len) throws IOException {
+int compressedBytesWritten = 0;
+
+// If we are not within uncompressed data yet, output the header.
+if (state != BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM &&
Review comment:
Hmm, I'm not sure. :)
I looked at `finish()` at `Deflater`. It just sets a `finish` variable to
true. But the variable is not used at all. So technically, I guess you still
can call its `setInput` to set new input and `deflate` again?
Because of that, I take more conservative approach here in case of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 634926)
Time Spent: 14h 20m (was: 14h 10m)
> Add BuiltInGzipCompressor
> -
>
> Key: HADOOP-17825
> URL: https://issues.apache.org/jira/browse/HADOOP-17825
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 14h 20m
> Remaining Estimate: 0h
>
> Currently, GzipCodec only supports BuiltInGzipDecompressor, if
[GitHub] [hadoop] viirya commented on a change in pull request #3250: HADOOP-17825. Add BuiltInGzipCompressor
viirya commented on a change in pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#discussion_r683902983
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress.zlib;
+
+import java.io.IOException;
+import java.util.zip.Checksum;
+import java.util.zip.Deflater;
+import java.util.zip.GZIPOutputStream;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.compress.Compressor;
+import org.apache.hadoop.io.compress.DoNotPool;
+import org.apache.hadoop.util.DataChecksum;
+
+/**
+ * A {@link Compressor} based on the popular gzip compressed file format.
+ * http://www.gzip.org/
+ */
+@DoNotPool
+public class BuiltInGzipCompressor implements Compressor {
+
+ /**
+ * Fixed ten-byte gzip header. See {@link GZIPOutputStream}'s source for
+ * details.
+ */
+ private static final byte[] GZIP_HEADER = new byte[]{
+ 0x1f, (byte) 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
+
+ // The trailer will be overwritten based on crc and output size.
+ private byte[] GZIP_TRAILER = new byte[]{0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00};
+
+ private final int GZIP_HEADER_LEN = GZIP_HEADER.length;
+ private final int GZIP_TRAILER_LEN = GZIP_TRAILER.length;
+
+ private Deflater deflater;
+
+ private int headerOff = 0;
+ private int trailerOff = 0;
+
+ private int numBytesWritten = 0;
+
+ private int currentBufLen = 0;
+
+ private final Checksum crc = DataChecksum.newCrc32();
+
+ private BuiltInGzipDecompressor.GzipStateLabel state;
+
+ public BuiltInGzipCompressor(Configuration conf) { init(conf); }
+
+ @Override
+ public boolean finished() {
+// Only if the trailer is also written, it is thought as finished.
+return deflater.finished() && state ==
BuiltInGzipDecompressor.GzipStateLabel.FINISHED;
+ }
+
+ @Override
+ public boolean needsInput() {
+if (state == BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM) {
+ return deflater.needsInput();
+}
+
+return (state != BuiltInGzipDecompressor.GzipStateLabel.FINISHED);
+ }
+
+ @Override
+ public int compress(byte[] b, int off, int len) throws IOException {
+int compressedBytesWritten = 0;
+
+// If we are not within uncompressed data yet, output the header.
+if (state != BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM &&
Review comment:
Hmm, I'm not sure. :)
I looked at `finish()` at `Deflater`. It just sets a `finish` variable to
true. But the variable is not used at all. So technically, I guess you still
can call its `setInput` to set new input and `deflate` again?
Because of that, I take more conservative approach here in case of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17825) Add BuiltInGzipCompressor
[
https://issues.apache.org/jira/browse/HADOOP-17825?focusedWorklogId=634924=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634924
]
ASF GitHub Bot logged work on HADOOP-17825:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 02:17
Start Date: 06/Aug/21 02:17
Worklog Time Spent: 10m
Work Description: sunchao commented on a change in pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#discussion_r683899349
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress.zlib;
+
+import java.io.IOException;
+import java.util.zip.Checksum;
+import java.util.zip.Deflater;
+import java.util.zip.GZIPOutputStream;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.compress.Compressor;
+import org.apache.hadoop.io.compress.DoNotPool;
+import org.apache.hadoop.util.DataChecksum;
+
+/**
+ * A {@link Compressor} based on the popular gzip compressed file format.
+ * http://www.gzip.org/
+ */
+@DoNotPool
+public class BuiltInGzipCompressor implements Compressor {
+
+ /**
+ * Fixed ten-byte gzip header. See {@link GZIPOutputStream}'s source for
+ * details.
+ */
+ private static final byte[] GZIP_HEADER = new byte[]{
+ 0x1f, (byte) 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
+
+ // The trailer will be overwritten based on crc and output size.
+ private byte[] GZIP_TRAILER = new byte[]{0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00};
+
+ private final int GZIP_HEADER_LEN = GZIP_HEADER.length;
+ private final int GZIP_TRAILER_LEN = GZIP_TRAILER.length;
+
+ private Deflater deflater;
+
+ private int headerOff = 0;
+ private int trailerOff = 0;
+
+ private int numBytesWritten = 0;
+
+ private int currentBufLen = 0;
+
+ private final Checksum crc = DataChecksum.newCrc32();
+
+ private BuiltInGzipDecompressor.GzipStateLabel state;
+
+ public BuiltInGzipCompressor(Configuration conf) { init(conf); }
+
+ @Override
+ public boolean finished() {
+// Only if the trailer is also written, it is thought as finished.
+return deflater.finished() && state ==
BuiltInGzipDecompressor.GzipStateLabel.FINISHED;
+ }
+
+ @Override
+ public boolean needsInput() {
+if (state == BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM) {
+ return deflater.needsInput();
+}
+
+return (state != BuiltInGzipDecompressor.GzipStateLabel.FINISHED);
+ }
+
+ @Override
+ public int compress(byte[] b, int off, int len) throws IOException {
+int compressedBytesWritten = 0;
+
+// If we are not within uncompressed data yet, output the header.
+if (state != BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM &&
Review comment:
I see. Yea I meant `CompressorStream` calls its own `compress` method
which calls `compressor.compress` indirectly.
> Calling finish() on this compressor won't set state to FINISHED.
Oh sorry I was looking at an old version of this PR which still set it to
`FINISHED` in `finish()`. Never mind.
So this looks good then :) Although I think the following:
> What I thought is, the caller might set input and compress until it
doesn't need input. The state is in FINISHED and the caller might call set
input and compress again. At the moment this check isn't effective to write the
header.
Will also not happen? since when the state is finished, the caller should
not call `setInput` before it calls `reset` on the compressor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 634924)
Time Spent: 14h 10m (was: 14h)
> Add BuiltInGzipCompressor
>
[GitHub] [hadoop] sunchao commented on a change in pull request #3250: HADOOP-17825. Add BuiltInGzipCompressor
sunchao commented on a change in pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#discussion_r683899349
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress.zlib;
+
+import java.io.IOException;
+import java.util.zip.Checksum;
+import java.util.zip.Deflater;
+import java.util.zip.GZIPOutputStream;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.compress.Compressor;
+import org.apache.hadoop.io.compress.DoNotPool;
+import org.apache.hadoop.util.DataChecksum;
+
+/**
+ * A {@link Compressor} based on the popular gzip compressed file format.
+ * http://www.gzip.org/
+ */
+@DoNotPool
+public class BuiltInGzipCompressor implements Compressor {
+
+ /**
+ * Fixed ten-byte gzip header. See {@link GZIPOutputStream}'s source for
+ * details.
+ */
+ private static final byte[] GZIP_HEADER = new byte[]{
+ 0x1f, (byte) 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
+
+ // The trailer will be overwritten based on crc and output size.
+ private byte[] GZIP_TRAILER = new byte[]{0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00};
+
+ private final int GZIP_HEADER_LEN = GZIP_HEADER.length;
+ private final int GZIP_TRAILER_LEN = GZIP_TRAILER.length;
+
+ private Deflater deflater;
+
+ private int headerOff = 0;
+ private int trailerOff = 0;
+
+ private int numBytesWritten = 0;
+
+ private int currentBufLen = 0;
+
+ private final Checksum crc = DataChecksum.newCrc32();
+
+ private BuiltInGzipDecompressor.GzipStateLabel state;
+
+ public BuiltInGzipCompressor(Configuration conf) { init(conf); }
+
+ @Override
+ public boolean finished() {
+// Only if the trailer is also written, it is thought as finished.
+return deflater.finished() && state ==
BuiltInGzipDecompressor.GzipStateLabel.FINISHED;
+ }
+
+ @Override
+ public boolean needsInput() {
+if (state == BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM) {
+ return deflater.needsInput();
+}
+
+return (state != BuiltInGzipDecompressor.GzipStateLabel.FINISHED);
+ }
+
+ @Override
+ public int compress(byte[] b, int off, int len) throws IOException {
+int compressedBytesWritten = 0;
+
+// If we are not within uncompressed data yet, output the header.
+if (state != BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM &&
Review comment:
I see. Yea I meant `CompressorStream` calls its own `compress` method
which calls `compressor.compress` indirectly.
> Calling finish() on this compressor won't set state to FINISHED.
Oh sorry I was looking at an old version of this PR which still set it to
`FINISHED` in `finish()`. Never mind.
So this looks good then :) Although I think the following:
> What I thought is, the caller might set input and compress until it
doesn't need input. The state is in FINISHED and the caller might call set
input and compress again. At the moment this check isn't effective to write the
header.
Will also not happen? since when the state is finished, the caller should
not call `setInput` before it calls `reset` on the compressor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17825) Add BuiltInGzipCompressor
[
https://issues.apache.org/jira/browse/HADOOP-17825?focusedWorklogId=634919=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634919
]
ASF GitHub Bot logged work on HADOOP-17825:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 01:54
Start Date: 06/Aug/21 01:54
Worklog Time Spent: 10m
Work Description: viirya commented on a change in pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#discussion_r683892398
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress.zlib;
+
+import java.io.IOException;
+import java.util.zip.Checksum;
+import java.util.zip.Deflater;
+import java.util.zip.GZIPOutputStream;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.compress.Compressor;
+import org.apache.hadoop.io.compress.DoNotPool;
+import org.apache.hadoop.util.DataChecksum;
+
+/**
+ * A {@link Compressor} based on the popular gzip compressed file format.
+ * http://www.gzip.org/
+ */
+@DoNotPool
+public class BuiltInGzipCompressor implements Compressor {
+
+ /**
+ * Fixed ten-byte gzip header. See {@link GZIPOutputStream}'s source for
+ * details.
+ */
+ private static final byte[] GZIP_HEADER = new byte[]{
+ 0x1f, (byte) 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
+
+ // The trailer will be overwritten based on crc and output size.
+ private byte[] GZIP_TRAILER = new byte[]{0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00};
+
+ private final int GZIP_HEADER_LEN = GZIP_HEADER.length;
+ private final int GZIP_TRAILER_LEN = GZIP_TRAILER.length;
+
+ private Deflater deflater;
+
+ private int headerOff = 0;
+ private int trailerOff = 0;
+
+ private int numBytesWritten = 0;
+
+ private int currentBufLen = 0;
+
+ private final Checksum crc = DataChecksum.newCrc32();
+
+ private BuiltInGzipDecompressor.GzipStateLabel state;
+
+ public BuiltInGzipCompressor(Configuration conf) { init(conf); }
+
+ @Override
+ public boolean finished() {
+// Only if the trailer is also written, it is thought as finished.
+return deflater.finished() && state ==
BuiltInGzipDecompressor.GzipStateLabel.FINISHED;
+ }
+
+ @Override
+ public boolean needsInput() {
+if (state == BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM) {
+ return deflater.needsInput();
+}
+
+return (state != BuiltInGzipDecompressor.GzipStateLabel.FINISHED);
+ }
+
+ @Override
+ public int compress(byte[] b, int off, int len) throws IOException {
+int compressedBytesWritten = 0;
+
+// If we are not within uncompressed data yet, output the header.
+if (state != BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM &&
Review comment:
Oh, I mean `Compressor` doesn't have a `compress()` method. Once calling
`compress`, the caller must provide a buffer for compressed output.
Calling `finish()` on this compressor won't set state to `FINISHED`. Only if
we output the trailer, we set the state to `FINISHED`. In this state, the
caller can set new input and call `compress(buf, off, len)` again. We will
output new header and enter into another compressed output (section).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 634919)
Time Spent: 14h (was: 13h 50m)
> Add BuiltInGzipCompressor
> -
>
> Key: HADOOP-17825
> URL: https://issues.apache.org/jira/browse/HADOOP-17825
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
>
[GitHub] [hadoop] viirya commented on a change in pull request #3250: HADOOP-17825. Add BuiltInGzipCompressor
viirya commented on a change in pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#discussion_r683892398
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress.zlib;
+
+import java.io.IOException;
+import java.util.zip.Checksum;
+import java.util.zip.Deflater;
+import java.util.zip.GZIPOutputStream;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.compress.Compressor;
+import org.apache.hadoop.io.compress.DoNotPool;
+import org.apache.hadoop.util.DataChecksum;
+
+/**
+ * A {@link Compressor} based on the popular gzip compressed file format.
+ * http://www.gzip.org/
+ */
+@DoNotPool
+public class BuiltInGzipCompressor implements Compressor {
+
+ /**
+ * Fixed ten-byte gzip header. See {@link GZIPOutputStream}'s source for
+ * details.
+ */
+ private static final byte[] GZIP_HEADER = new byte[]{
+ 0x1f, (byte) 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
+
+ // The trailer will be overwritten based on crc and output size.
+ private byte[] GZIP_TRAILER = new byte[]{0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00};
+
+ private final int GZIP_HEADER_LEN = GZIP_HEADER.length;
+ private final int GZIP_TRAILER_LEN = GZIP_TRAILER.length;
+
+ private Deflater deflater;
+
+ private int headerOff = 0;
+ private int trailerOff = 0;
+
+ private int numBytesWritten = 0;
+
+ private int currentBufLen = 0;
+
+ private final Checksum crc = DataChecksum.newCrc32();
+
+ private BuiltInGzipDecompressor.GzipStateLabel state;
+
+ public BuiltInGzipCompressor(Configuration conf) { init(conf); }
+
+ @Override
+ public boolean finished() {
+// Only if the trailer is also written, it is thought as finished.
+return deflater.finished() && state ==
BuiltInGzipDecompressor.GzipStateLabel.FINISHED;
+ }
+
+ @Override
+ public boolean needsInput() {
+if (state == BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM) {
+ return deflater.needsInput();
+}
+
+return (state != BuiltInGzipDecompressor.GzipStateLabel.FINISHED);
+ }
+
+ @Override
+ public int compress(byte[] b, int off, int len) throws IOException {
+int compressedBytesWritten = 0;
+
+// If we are not within uncompressed data yet, output the header.
+if (state != BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM &&
Review comment:
Oh, I mean `Compressor` doesn't have a `compress()` method. Once calling
`compress`, the caller must provide a buffer for compressed output.
Calling `finish()` on this compressor won't set state to `FINISHED`. Only if
we output the trailer, we set the state to `FINISHED`. In this state, the
caller can set new input and call `compress(buf, off, len)` again. We will
output new header and enter into another compressed output (section).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17825) Add BuiltInGzipCompressor
[
https://issues.apache.org/jira/browse/HADOOP-17825?focusedWorklogId=634918=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634918
]
ASF GitHub Bot logged work on HADOOP-17825:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 01:49
Start Date: 06/Aug/21 01:49
Worklog Time Spent: 10m
Work Description: sunchao commented on a change in pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#discussion_r683890692
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress.zlib;
+
+import java.io.IOException;
+import java.util.zip.Checksum;
+import java.util.zip.Deflater;
+import java.util.zip.GZIPOutputStream;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.compress.Compressor;
+import org.apache.hadoop.io.compress.DoNotPool;
+import org.apache.hadoop.util.DataChecksum;
+
+/**
+ * A {@link Compressor} based on the popular gzip compressed file format.
+ * http://www.gzip.org/
+ */
+@DoNotPool
+public class BuiltInGzipCompressor implements Compressor {
+
+ /**
+ * Fixed ten-byte gzip header. See {@link GZIPOutputStream}'s source for
+ * details.
+ */
+ private static final byte[] GZIP_HEADER = new byte[]{
+ 0x1f, (byte) 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
+
+ // The trailer will be overwritten based on crc and output size.
+ private byte[] GZIP_TRAILER = new byte[]{0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00};
+
+ private final int GZIP_HEADER_LEN = GZIP_HEADER.length;
+ private final int GZIP_TRAILER_LEN = GZIP_TRAILER.length;
+
+ private Deflater deflater;
+
+ private int headerOff = 0;
+ private int trailerOff = 0;
+
+ private int numBytesWritten = 0;
+
+ private int currentBufLen = 0;
+
+ private final Checksum crc = DataChecksum.newCrc32();
+
+ private BuiltInGzipDecompressor.GzipStateLabel state;
+
+ public BuiltInGzipCompressor(Configuration conf) { init(conf); }
+
+ @Override
+ public boolean finished() {
+// Only if the trailer is also written, it is thought as finished.
+return deflater.finished() && state ==
BuiltInGzipDecompressor.GzipStateLabel.FINISHED;
+ }
+
+ @Override
+ public boolean needsInput() {
+if (state == BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM) {
+ return deflater.needsInput();
+}
+
+return (state != BuiltInGzipDecompressor.GzipStateLabel.FINISHED);
+ }
+
+ @Override
+ public int compress(byte[] b, int off, int len) throws IOException {
+int compressedBytesWritten = 0;
+
+// If we are not within uncompressed data yet, output the header.
+if (state != BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM &&
Review comment:
> So seems we cannot do like CompressorStream.
Hmm what do you mean here? sorry don't quite understand it.
> Once the caller calls finish on this Compressor, we only call finish on
the deflator. The caller then will call finished to verify if it reaches
finished state. If not, it should call compress with buffer to get more
compressed output.
Yes. So if the input is large but the buffer in a `CompressorStream` is
small, potentially it will need to call `compress` multiple times after the
`finish()` is invoked, before it can reach to the `finished` status (i.e.,
`deflater.finished()` returns true).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 634918)
Time Spent: 13h 50m (was: 13h 40m)
> Add BuiltInGzipCompressor
> -
>
> Key: HADOOP-17825
> URL: https://issues.apache.org/jira/browse/HADOOP-17825
>
[GitHub] [hadoop] sunchao commented on a change in pull request #3250: HADOOP-17825. Add BuiltInGzipCompressor
sunchao commented on a change in pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#discussion_r683890692
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress.zlib;
+
+import java.io.IOException;
+import java.util.zip.Checksum;
+import java.util.zip.Deflater;
+import java.util.zip.GZIPOutputStream;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.compress.Compressor;
+import org.apache.hadoop.io.compress.DoNotPool;
+import org.apache.hadoop.util.DataChecksum;
+
+/**
+ * A {@link Compressor} based on the popular gzip compressed file format.
+ * http://www.gzip.org/
+ */
+@DoNotPool
+public class BuiltInGzipCompressor implements Compressor {
+
+ /**
+ * Fixed ten-byte gzip header. See {@link GZIPOutputStream}'s source for
+ * details.
+ */
+ private static final byte[] GZIP_HEADER = new byte[]{
+ 0x1f, (byte) 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
+
+ // The trailer will be overwritten based on crc and output size.
+ private byte[] GZIP_TRAILER = new byte[]{0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00};
+
+ private final int GZIP_HEADER_LEN = GZIP_HEADER.length;
+ private final int GZIP_TRAILER_LEN = GZIP_TRAILER.length;
+
+ private Deflater deflater;
+
+ private int headerOff = 0;
+ private int trailerOff = 0;
+
+ private int numBytesWritten = 0;
+
+ private int currentBufLen = 0;
+
+ private final Checksum crc = DataChecksum.newCrc32();
+
+ private BuiltInGzipDecompressor.GzipStateLabel state;
+
+ public BuiltInGzipCompressor(Configuration conf) { init(conf); }
+
+ @Override
+ public boolean finished() {
+// Only if the trailer is also written, it is thought as finished.
+return deflater.finished() && state ==
BuiltInGzipDecompressor.GzipStateLabel.FINISHED;
+ }
+
+ @Override
+ public boolean needsInput() {
+if (state == BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM) {
+ return deflater.needsInput();
+}
+
+return (state != BuiltInGzipDecompressor.GzipStateLabel.FINISHED);
+ }
+
+ @Override
+ public int compress(byte[] b, int off, int len) throws IOException {
+int compressedBytesWritten = 0;
+
+// If we are not within uncompressed data yet, output the header.
+if (state != BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM &&
Review comment:
> So seems we cannot do like CompressorStream.
Hmm what do you mean here? sorry don't quite understand it.
> Once the caller calls finish on this Compressor, we only call finish on
the deflator. The caller then will call finished to verify if it reaches
finished state. If not, it should call compress with buffer to get more
compressed output.
Yes. So if the input is large but the buffer in a `CompressorStream` is
small, potentially it will need to call `compress` multiple times after the
`finish()` is invoked, before it can reach to the `finished` status (i.e.,
`deflater.finished()` returns true).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17825) Add BuiltInGzipCompressor
[
https://issues.apache.org/jira/browse/HADOOP-17825?focusedWorklogId=634913=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634913
]
ASF GitHub Bot logged work on HADOOP-17825:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 01:37
Start Date: 06/Aug/21 01:37
Worklog Time Spent: 10m
Work Description: viirya commented on a change in pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#discussion_r683887010
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress.zlib;
+
+import java.io.IOException;
+import java.util.zip.Checksum;
+import java.util.zip.Deflater;
+import java.util.zip.GZIPOutputStream;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.compress.Compressor;
+import org.apache.hadoop.io.compress.DoNotPool;
+import org.apache.hadoop.util.DataChecksum;
+
+/**
+ * A {@link Compressor} based on the popular gzip compressed file format.
+ * http://www.gzip.org/
+ */
+@DoNotPool
+public class BuiltInGzipCompressor implements Compressor {
+
+ /**
+ * Fixed ten-byte gzip header. See {@link GZIPOutputStream}'s source for
+ * details.
+ */
+ private static final byte[] GZIP_HEADER = new byte[]{
+ 0x1f, (byte) 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
+
+ // The trailer will be overwritten based on crc and output size.
+ private byte[] GZIP_TRAILER = new byte[]{0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00};
+
+ private final int GZIP_HEADER_LEN = GZIP_HEADER.length;
+ private final int GZIP_TRAILER_LEN = GZIP_TRAILER.length;
+
+ private Deflater deflater;
+
+ private int headerOff = 0;
+ private int trailerOff = 0;
+
+ private int numBytesWritten = 0;
+
+ private int currentBufLen = 0;
+
+ private final Checksum crc = DataChecksum.newCrc32();
+
+ private BuiltInGzipDecompressor.GzipStateLabel state;
+
+ public BuiltInGzipCompressor(Configuration conf) { init(conf); }
+
+ @Override
+ public boolean finished() {
+// Only if the trailer is also written, it is thought as finished.
+return deflater.finished() && state ==
BuiltInGzipDecompressor.GzipStateLabel.FINISHED;
+ }
+
+ @Override
+ public boolean needsInput() {
+if (state == BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM) {
+ return deflater.needsInput();
+}
+
+return (state != BuiltInGzipDecompressor.GzipStateLabel.FINISHED);
+ }
+
+ @Override
+ public int compress(byte[] b, int off, int len) throws IOException {
+int compressedBytesWritten = 0;
+
+// If we are not within uncompressed data yet, output the header.
+if (state != BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM &&
Review comment:
For `Compressor` here, `compress` requires a buffer to write compressed
output to. So seems we cannot do like `CompressorStream`.
Once the caller calls `finish` on this `Compressor`, we only call `finish`
on the deflator`. The caller then will call `finished` to verify if it reaches
finished state. If not, it should call `compress` with buffer to get more
compressed output.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 634913)
Time Spent: 13h 40m (was: 13.5h)
> Add BuiltInGzipCompressor
> -
>
> Key: HADOOP-17825
> URL: https://issues.apache.org/jira/browse/HADOOP-17825
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 13h 40m
> Remaining Estimate: 0h
>
>
[GitHub] [hadoop] viirya commented on a change in pull request #3250: HADOOP-17825. Add BuiltInGzipCompressor
viirya commented on a change in pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#discussion_r683887010
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress.zlib;
+
+import java.io.IOException;
+import java.util.zip.Checksum;
+import java.util.zip.Deflater;
+import java.util.zip.GZIPOutputStream;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.compress.Compressor;
+import org.apache.hadoop.io.compress.DoNotPool;
+import org.apache.hadoop.util.DataChecksum;
+
+/**
+ * A {@link Compressor} based on the popular gzip compressed file format.
+ * http://www.gzip.org/
+ */
+@DoNotPool
+public class BuiltInGzipCompressor implements Compressor {
+
+ /**
+ * Fixed ten-byte gzip header. See {@link GZIPOutputStream}'s source for
+ * details.
+ */
+ private static final byte[] GZIP_HEADER = new byte[]{
+ 0x1f, (byte) 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
+
+ // The trailer will be overwritten based on crc and output size.
+ private byte[] GZIP_TRAILER = new byte[]{0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00};
+
+ private final int GZIP_HEADER_LEN = GZIP_HEADER.length;
+ private final int GZIP_TRAILER_LEN = GZIP_TRAILER.length;
+
+ private Deflater deflater;
+
+ private int headerOff = 0;
+ private int trailerOff = 0;
+
+ private int numBytesWritten = 0;
+
+ private int currentBufLen = 0;
+
+ private final Checksum crc = DataChecksum.newCrc32();
+
+ private BuiltInGzipDecompressor.GzipStateLabel state;
+
+ public BuiltInGzipCompressor(Configuration conf) { init(conf); }
+
+ @Override
+ public boolean finished() {
+// Only if the trailer is also written, it is thought as finished.
+return deflater.finished() && state ==
BuiltInGzipDecompressor.GzipStateLabel.FINISHED;
+ }
+
+ @Override
+ public boolean needsInput() {
+if (state == BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM) {
+ return deflater.needsInput();
+}
+
+return (state != BuiltInGzipDecompressor.GzipStateLabel.FINISHED);
+ }
+
+ @Override
+ public int compress(byte[] b, int off, int len) throws IOException {
+int compressedBytesWritten = 0;
+
+// If we are not within uncompressed data yet, output the header.
+if (state != BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM &&
Review comment:
For `Compressor` here, `compress` requires a buffer to write compressed
output to. So seems we cannot do like `CompressorStream`.
Once the caller calls `finish` on this `Compressor`, we only call `finish`
on the deflator`. The caller then will call `finished` to verify if it reaches
finished state. If not, it should call `compress` with buffer to get more
compressed output.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17808) ipc.Client not setting interrupt flag after catching InterruptedException
[ https://issues.apache.org/jira/browse/HADOOP-17808?focusedWorklogId=634910=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634910 ] ASF GitHub Bot logged work on HADOOP-17808: --- Author: ASF GitHub Bot Created on: 06/Aug/21 01:27 Start Date: 06/Aug/21 01:27 Worklog Time Spent: 10m Work Description: ferhui commented on pull request #3267: URL: https://github.com/apache/hadoop/pull/3267#issuecomment-893935057 Thanks you all for report and review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 634910) Time Spent: 4h 50m (was: 4h 40m) > ipc.Client not setting interrupt flag after catching InterruptedException > - > > Key: HADOOP-17808 > URL: https://issues.apache.org/jira/browse/HADOOP-17808 > Project: Hadoop Common > Issue Type: Task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.2.3, 3.3.2 > > Time Spent: 4h 50m > Remaining Estimate: 0h > > ipc.Client is swallowing InterruptedException at a couple of places: > # While waiting on all connections to be closed > # While waiting to retrieve some RPC response > We should at least set the interrupt signal and also log the > InterruptedException caught. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17808) ipc.Client not setting interrupt flag after catching InterruptedException
[ https://issues.apache.org/jira/browse/HADOOP-17808?focusedWorklogId=634909=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634909 ] ASF GitHub Bot logged work on HADOOP-17808: --- Author: ASF GitHub Bot Created on: 06/Aug/21 01:27 Start Date: 06/Aug/21 01:27 Worklog Time Spent: 10m Work Description: ferhui merged pull request #3267: URL: https://github.com/apache/hadoop/pull/3267 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 634909) Time Spent: 4h 40m (was: 4.5h) > ipc.Client not setting interrupt flag after catching InterruptedException > - > > Key: HADOOP-17808 > URL: https://issues.apache.org/jira/browse/HADOOP-17808 > Project: Hadoop Common > Issue Type: Task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.2.3, 3.3.2 > > Time Spent: 4h 40m > Remaining Estimate: 0h > > ipc.Client is swallowing InterruptedException at a couple of places: > # While waiting on all connections to be closed > # While waiting to retrieve some RPC response > We should at least set the interrupt signal and also log the > InterruptedException caught. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ferhui commented on pull request #3267: HADOOP-17808. Avoid excessive logging for interruption (ADDENDUM)
ferhui commented on pull request #3267: URL: https://github.com/apache/hadoop/pull/3267#issuecomment-893935057 Thanks you all for report and review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ferhui merged pull request #3267: HADOOP-17808. Avoid excessive logging for interruption (ADDENDUM)
ferhui merged pull request #3267: URL: https://github.com/apache/hadoop/pull/3267 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17825) Add BuiltInGzipCompressor
[
https://issues.apache.org/jira/browse/HADOOP-17825?focusedWorklogId=634899=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634899
]
ASF GitHub Bot logged work on HADOOP-17825:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 00:42
Start Date: 06/Aug/21 00:42
Worklog Time Spent: 10m
Work Description: sunchao commented on a change in pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#discussion_r683870968
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress.zlib;
+
+import java.io.IOException;
+import java.util.zip.Checksum;
+import java.util.zip.Deflater;
+import java.util.zip.GZIPOutputStream;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.compress.Compressor;
+import org.apache.hadoop.io.compress.DoNotPool;
+import org.apache.hadoop.util.DataChecksum;
+
+/**
+ * A {@link Compressor} based on the popular gzip compressed file format.
+ * http://www.gzip.org/
+ */
+@DoNotPool
+public class BuiltInGzipCompressor implements Compressor {
+
+ /**
+ * Fixed ten-byte gzip header. See {@link GZIPOutputStream}'s source for
+ * details.
+ */
+ private static final byte[] GZIP_HEADER = new byte[]{
+ 0x1f, (byte) 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
+
+ // The trailer will be overwritten based on crc and output size.
+ private byte[] GZIP_TRAILER = new byte[]{0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00};
+
+ private final int GZIP_HEADER_LEN = GZIP_HEADER.length;
+ private final int GZIP_TRAILER_LEN = GZIP_TRAILER.length;
+
+ private Deflater deflater;
+
+ private int headerOff = 0;
+ private int trailerOff = 0;
+
+ private int numBytesWritten = 0;
+
+ private int currentBufLen = 0;
+
+ private final Checksum crc = DataChecksum.newCrc32();
+
+ private BuiltInGzipDecompressor.GzipStateLabel state;
+
+ public BuiltInGzipCompressor(Configuration conf) { init(conf); }
+
+ @Override
+ public boolean finished() {
+// Only if the trailer is also written, it is thought as finished.
+return deflater.finished() && state ==
BuiltInGzipDecompressor.GzipStateLabel.FINISHED;
+ }
+
+ @Override
+ public boolean needsInput() {
+if (state == BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM) {
+ return deflater.needsInput();
+}
+
+return (state != BuiltInGzipDecompressor.GzipStateLabel.FINISHED);
+ }
+
+ @Override
+ public int compress(byte[] b, int off, int len) throws IOException {
+int compressedBytesWritten = 0;
+
+// If we are not within uncompressed data yet, output the header.
+if (state != BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM &&
Review comment:
Ah OK, so the compression can happen on multiple inputs. Then, I'm
curious whether we should handle the `FINISHED` state too in this `if` clause.
For instance, in the following situation:
```java
@Override
public void finish() throws IOException {
if (!compressor.finished()) {
compressor.finish();
while (!compressor.finished()) {
compress();
}
}
}
```
the `CompressorStream` will first set the state to be `FINISHED` and then
keep calling `compress` until it is finished.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 634899)
Time Spent: 13.5h (was: 13h 20m)
> Add BuiltInGzipCompressor
> -
>
> Key: HADOOP-17825
> URL: https://issues.apache.org/jira/browse/HADOOP-17825
> Project: Hadoop Common
> Issue Type: Improvement
>
[GitHub] [hadoop] sunchao commented on a change in pull request #3250: HADOOP-17825. Add BuiltInGzipCompressor
sunchao commented on a change in pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#discussion_r683870968
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress.zlib;
+
+import java.io.IOException;
+import java.util.zip.Checksum;
+import java.util.zip.Deflater;
+import java.util.zip.GZIPOutputStream;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.compress.Compressor;
+import org.apache.hadoop.io.compress.DoNotPool;
+import org.apache.hadoop.util.DataChecksum;
+
+/**
+ * A {@link Compressor} based on the popular gzip compressed file format.
+ * http://www.gzip.org/
+ */
+@DoNotPool
+public class BuiltInGzipCompressor implements Compressor {
+
+ /**
+ * Fixed ten-byte gzip header. See {@link GZIPOutputStream}'s source for
+ * details.
+ */
+ private static final byte[] GZIP_HEADER = new byte[]{
+ 0x1f, (byte) 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
+
+ // The trailer will be overwritten based on crc and output size.
+ private byte[] GZIP_TRAILER = new byte[]{0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00};
+
+ private final int GZIP_HEADER_LEN = GZIP_HEADER.length;
+ private final int GZIP_TRAILER_LEN = GZIP_TRAILER.length;
+
+ private Deflater deflater;
+
+ private int headerOff = 0;
+ private int trailerOff = 0;
+
+ private int numBytesWritten = 0;
+
+ private int currentBufLen = 0;
+
+ private final Checksum crc = DataChecksum.newCrc32();
+
+ private BuiltInGzipDecompressor.GzipStateLabel state;
+
+ public BuiltInGzipCompressor(Configuration conf) { init(conf); }
+
+ @Override
+ public boolean finished() {
+// Only if the trailer is also written, it is thought as finished.
+return deflater.finished() && state ==
BuiltInGzipDecompressor.GzipStateLabel.FINISHED;
+ }
+
+ @Override
+ public boolean needsInput() {
+if (state == BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM) {
+ return deflater.needsInput();
+}
+
+return (state != BuiltInGzipDecompressor.GzipStateLabel.FINISHED);
+ }
+
+ @Override
+ public int compress(byte[] b, int off, int len) throws IOException {
+int compressedBytesWritten = 0;
+
+// If we are not within uncompressed data yet, output the header.
+if (state != BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM &&
Review comment:
Ah OK, so the compression can happen on multiple inputs. Then, I'm
curious whether we should handle the `FINISHED` state too in this `if` clause.
For instance, in the following situation:
```java
@Override
public void finish() throws IOException {
if (!compressor.finished()) {
compressor.finish();
while (!compressor.finished()) {
compress();
}
}
}
```
the `CompressorStream` will first set the state to be `FINISHED` and then
keep calling `compress` until it is finished.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17825) Add BuiltInGzipCompressor
[
https://issues.apache.org/jira/browse/HADOOP-17825?focusedWorklogId=634887=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634887
]
ASF GitHub Bot logged work on HADOOP-17825:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 00:11
Start Date: 06/Aug/21 00:11
Worklog Time Spent: 10m
Work Description: viirya commented on a change in pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#discussion_r683862198
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress.zlib;
+
+import java.io.IOException;
+import java.util.zip.Checksum;
+import java.util.zip.Deflater;
+import java.util.zip.GZIPOutputStream;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.compress.Compressor;
+import org.apache.hadoop.io.compress.DoNotPool;
+import org.apache.hadoop.util.DataChecksum;
+
+/**
+ * A {@link Compressor} based on the popular gzip compressed file format.
+ * http://www.gzip.org/
+ */
+@DoNotPool
+public class BuiltInGzipCompressor implements Compressor {
+
+ /**
+ * Fixed ten-byte gzip header. See {@link GZIPOutputStream}'s source for
+ * details.
+ */
+ private static final byte[] GZIP_HEADER = new byte[]{
+ 0x1f, (byte) 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
+
+ // The trailer will be overwritten based on crc and output size.
+ private byte[] GZIP_TRAILER = new byte[]{0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00};
+
+ private final int GZIP_HEADER_LEN = GZIP_HEADER.length;
+ private final int GZIP_TRAILER_LEN = GZIP_TRAILER.length;
+
+ private Deflater deflater;
+
+ private int headerOff = 0;
+ private int trailerOff = 0;
+
+ private int numBytesWritten = 0;
+
+ private int currentBufLen = 0;
+
+ private final Checksum crc = DataChecksum.newCrc32();
+
+ private BuiltInGzipDecompressor.GzipStateLabel state;
+
+ public BuiltInGzipCompressor(Configuration conf) { init(conf); }
+
+ @Override
+ public boolean finished() {
+// Only if the trailer is also written, it is thought as finished.
+return deflater.finished() && state ==
BuiltInGzipDecompressor.GzipStateLabel.FINISHED;
+ }
+
+ @Override
+ public boolean needsInput() {
+if (state == BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM) {
+ return deflater.needsInput();
+}
+
+return (state != BuiltInGzipDecompressor.GzipStateLabel.FINISHED);
+ }
+
+ @Override
+ public int compress(byte[] b, int off, int len) throws IOException {
+int compressedBytesWritten = 0;
+
+// If we are not within uncompressed data yet, output the header.
+if (state != BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM &&
Review comment:
Let me revert to original condition `INFLATE_STREAM` and `TRAILER_CRC`.
It looks more reliable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 634887)
Time Spent: 13h 20m (was: 13h 10m)
> Add BuiltInGzipCompressor
> -
>
> Key: HADOOP-17825
> URL: https://issues.apache.org/jira/browse/HADOOP-17825
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 13h 20m
> Remaining Estimate: 0h
>
> Currently, GzipCodec only supports BuiltInGzipDecompressor, if native zlib is
> not loaded. So, without Hadoop native codec installed, saving SequenceFile
> using GzipCodec will throw exception like "SequenceFile doesn't work with
> GzipCodec without native-hadoop code!"
> Same as other codecs
[GitHub] [hadoop] viirya commented on a change in pull request #3250: HADOOP-17825. Add BuiltInGzipCompressor
viirya commented on a change in pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#discussion_r683862198
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress.zlib;
+
+import java.io.IOException;
+import java.util.zip.Checksum;
+import java.util.zip.Deflater;
+import java.util.zip.GZIPOutputStream;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.compress.Compressor;
+import org.apache.hadoop.io.compress.DoNotPool;
+import org.apache.hadoop.util.DataChecksum;
+
+/**
+ * A {@link Compressor} based on the popular gzip compressed file format.
+ * http://www.gzip.org/
+ */
+@DoNotPool
+public class BuiltInGzipCompressor implements Compressor {
+
+ /**
+ * Fixed ten-byte gzip header. See {@link GZIPOutputStream}'s source for
+ * details.
+ */
+ private static final byte[] GZIP_HEADER = new byte[]{
+ 0x1f, (byte) 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
+
+ // The trailer will be overwritten based on crc and output size.
+ private byte[] GZIP_TRAILER = new byte[]{0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00};
+
+ private final int GZIP_HEADER_LEN = GZIP_HEADER.length;
+ private final int GZIP_TRAILER_LEN = GZIP_TRAILER.length;
+
+ private Deflater deflater;
+
+ private int headerOff = 0;
+ private int trailerOff = 0;
+
+ private int numBytesWritten = 0;
+
+ private int currentBufLen = 0;
+
+ private final Checksum crc = DataChecksum.newCrc32();
+
+ private BuiltInGzipDecompressor.GzipStateLabel state;
+
+ public BuiltInGzipCompressor(Configuration conf) { init(conf); }
+
+ @Override
+ public boolean finished() {
+// Only if the trailer is also written, it is thought as finished.
+return deflater.finished() && state ==
BuiltInGzipDecompressor.GzipStateLabel.FINISHED;
+ }
+
+ @Override
+ public boolean needsInput() {
+if (state == BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM) {
+ return deflater.needsInput();
+}
+
+return (state != BuiltInGzipDecompressor.GzipStateLabel.FINISHED);
+ }
+
+ @Override
+ public int compress(byte[] b, int off, int len) throws IOException {
+int compressedBytesWritten = 0;
+
+// If we are not within uncompressed data yet, output the header.
+if (state != BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM &&
Review comment:
Let me revert to original condition `INFLATE_STREAM` and `TRAILER_CRC`.
It looks more reliable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17825) Add BuiltInGzipCompressor
[
https://issues.apache.org/jira/browse/HADOOP-17825?focusedWorklogId=634877=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634877
]
ASF GitHub Bot logged work on HADOOP-17825:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 00:03
Start Date: 06/Aug/21 00:03
Worklog Time Spent: 10m
Work Description: viirya commented on a change in pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#discussion_r683859384
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress.zlib;
+
+import java.io.IOException;
+import java.util.zip.Checksum;
+import java.util.zip.Deflater;
+import java.util.zip.GZIPOutputStream;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.compress.Compressor;
+import org.apache.hadoop.io.compress.DoNotPool;
+import org.apache.hadoop.util.DataChecksum;
+
+/**
+ * A {@link Compressor} based on the popular gzip compressed file format.
+ * http://www.gzip.org/
+ */
+@DoNotPool
+public class BuiltInGzipCompressor implements Compressor {
+
+ /**
+ * Fixed ten-byte gzip header. See {@link GZIPOutputStream}'s source for
+ * details.
+ */
+ private static final byte[] GZIP_HEADER = new byte[]{
+ 0x1f, (byte) 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
+
+ // The trailer will be overwritten based on crc and output size.
+ private byte[] GZIP_TRAILER = new byte[]{0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00};
+
+ private final int GZIP_HEADER_LEN = GZIP_HEADER.length;
+ private final int GZIP_TRAILER_LEN = GZIP_TRAILER.length;
+
+ private Deflater deflater;
+
+ private int headerOff = 0;
+ private int trailerOff = 0;
+
+ private int numBytesWritten = 0;
+
+ private int currentBufLen = 0;
+
+ private final Checksum crc = DataChecksum.newCrc32();
+
+ private BuiltInGzipDecompressor.GzipStateLabel state;
+
+ public BuiltInGzipCompressor(Configuration conf) { init(conf); }
+
+ @Override
+ public boolean finished() {
+// Only if the trailer is also written, it is thought as finished.
+return deflater.finished() && state ==
BuiltInGzipDecompressor.GzipStateLabel.FINISHED;
+ }
+
+ @Override
+ public boolean needsInput() {
+if (state == BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM) {
+ return deflater.needsInput();
+}
+
+return (state != BuiltInGzipDecompressor.GzipStateLabel.FINISHED);
+ }
+
+ @Override
+ public int compress(byte[] b, int off, int len) throws IOException {
+int compressedBytesWritten = 0;
+
+// If we are not within uncompressed data yet, output the header.
+if (state != BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM &&
Review comment:
Hmm, oh, no, I think we cannot do it. `setInput` can be called multiple
times before we reach `FINISHED` status. If we set the state to `HEADER_BASIC`,
it will re-output the header, but the current trailer is not output yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 634877)
Time Spent: 13h 10m (was: 13h)
> Add BuiltInGzipCompressor
> -
>
> Key: HADOOP-17825
> URL: https://issues.apache.org/jira/browse/HADOOP-17825
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 13h 10m
> Remaining Estimate: 0h
>
> Currently, GzipCodec only supports BuiltInGzipDecompressor, if native zlib is
> not loaded. So, without Hadoop native codec installed, saving SequenceFile
> using
[GitHub] [hadoop] viirya commented on a change in pull request #3250: HADOOP-17825. Add BuiltInGzipCompressor
viirya commented on a change in pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#discussion_r683859384
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress.zlib;
+
+import java.io.IOException;
+import java.util.zip.Checksum;
+import java.util.zip.Deflater;
+import java.util.zip.GZIPOutputStream;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.compress.Compressor;
+import org.apache.hadoop.io.compress.DoNotPool;
+import org.apache.hadoop.util.DataChecksum;
+
+/**
+ * A {@link Compressor} based on the popular gzip compressed file format.
+ * http://www.gzip.org/
+ */
+@DoNotPool
+public class BuiltInGzipCompressor implements Compressor {
+
+ /**
+ * Fixed ten-byte gzip header. See {@link GZIPOutputStream}'s source for
+ * details.
+ */
+ private static final byte[] GZIP_HEADER = new byte[]{
+ 0x1f, (byte) 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
+
+ // The trailer will be overwritten based on crc and output size.
+ private byte[] GZIP_TRAILER = new byte[]{0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00};
+
+ private final int GZIP_HEADER_LEN = GZIP_HEADER.length;
+ private final int GZIP_TRAILER_LEN = GZIP_TRAILER.length;
+
+ private Deflater deflater;
+
+ private int headerOff = 0;
+ private int trailerOff = 0;
+
+ private int numBytesWritten = 0;
+
+ private int currentBufLen = 0;
+
+ private final Checksum crc = DataChecksum.newCrc32();
+
+ private BuiltInGzipDecompressor.GzipStateLabel state;
+
+ public BuiltInGzipCompressor(Configuration conf) { init(conf); }
+
+ @Override
+ public boolean finished() {
+// Only if the trailer is also written, it is thought as finished.
+return deflater.finished() && state ==
BuiltInGzipDecompressor.GzipStateLabel.FINISHED;
+ }
+
+ @Override
+ public boolean needsInput() {
+if (state == BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM) {
+ return deflater.needsInput();
+}
+
+return (state != BuiltInGzipDecompressor.GzipStateLabel.FINISHED);
+ }
+
+ @Override
+ public int compress(byte[] b, int off, int len) throws IOException {
+int compressedBytesWritten = 0;
+
+// If we are not within uncompressed data yet, output the header.
+if (state != BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM &&
Review comment:
Hmm, oh, no, I think we cannot do it. `setInput` can be called multiple
times before we reach `FINISHED` status. If we set the state to `HEADER_BASIC`,
it will re-output the header, but the current trailer is not output yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17825) Add BuiltInGzipCompressor
[
https://issues.apache.org/jira/browse/HADOOP-17825?focusedWorklogId=634875=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634875
]
ASF GitHub Bot logged work on HADOOP-17825:
---
Author: ASF GitHub Bot
Created on: 06/Aug/21 00:02
Start Date: 06/Aug/21 00:02
Worklog Time Spent: 10m
Work Description: viirya commented on a change in pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#discussion_r683859384
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress.zlib;
+
+import java.io.IOException;
+import java.util.zip.Checksum;
+import java.util.zip.Deflater;
+import java.util.zip.GZIPOutputStream;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.compress.Compressor;
+import org.apache.hadoop.io.compress.DoNotPool;
+import org.apache.hadoop.util.DataChecksum;
+
+/**
+ * A {@link Compressor} based on the popular gzip compressed file format.
+ * http://www.gzip.org/
+ */
+@DoNotPool
+public class BuiltInGzipCompressor implements Compressor {
+
+ /**
+ * Fixed ten-byte gzip header. See {@link GZIPOutputStream}'s source for
+ * details.
+ */
+ private static final byte[] GZIP_HEADER = new byte[]{
+ 0x1f, (byte) 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
+
+ // The trailer will be overwritten based on crc and output size.
+ private byte[] GZIP_TRAILER = new byte[]{0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00};
+
+ private final int GZIP_HEADER_LEN = GZIP_HEADER.length;
+ private final int GZIP_TRAILER_LEN = GZIP_TRAILER.length;
+
+ private Deflater deflater;
+
+ private int headerOff = 0;
+ private int trailerOff = 0;
+
+ private int numBytesWritten = 0;
+
+ private int currentBufLen = 0;
+
+ private final Checksum crc = DataChecksum.newCrc32();
+
+ private BuiltInGzipDecompressor.GzipStateLabel state;
+
+ public BuiltInGzipCompressor(Configuration conf) { init(conf); }
+
+ @Override
+ public boolean finished() {
+// Only if the trailer is also written, it is thought as finished.
+return deflater.finished() && state ==
BuiltInGzipDecompressor.GzipStateLabel.FINISHED;
+ }
+
+ @Override
+ public boolean needsInput() {
+if (state == BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM) {
+ return deflater.needsInput();
+}
+
+return (state != BuiltInGzipDecompressor.GzipStateLabel.FINISHED);
+ }
+
+ @Override
+ public int compress(byte[] b, int off, int len) throws IOException {
+int compressedBytesWritten = 0;
+
+// If we are not within uncompressed data yet, output the header.
+if (state != BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM &&
Review comment:
Hmm, oh, no, I think we cannot do it. `setInput` can be called multiple
times before we reach finished status. If we set the state to `HEADER_BASIC`,
it will re-output the header, but the current trailer is not output yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 634875)
Time Spent: 13h (was: 12h 50m)
> Add BuiltInGzipCompressor
> -
>
> Key: HADOOP-17825
> URL: https://issues.apache.org/jira/browse/HADOOP-17825
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 13h
> Remaining Estimate: 0h
>
> Currently, GzipCodec only supports BuiltInGzipDecompressor, if native zlib is
> not loaded. So, without Hadoop native codec installed, saving SequenceFile
> using GzipCodec
[GitHub] [hadoop] viirya commented on a change in pull request #3250: HADOOP-17825. Add BuiltInGzipCompressor
viirya commented on a change in pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#discussion_r683859384
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress.zlib;
+
+import java.io.IOException;
+import java.util.zip.Checksum;
+import java.util.zip.Deflater;
+import java.util.zip.GZIPOutputStream;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.compress.Compressor;
+import org.apache.hadoop.io.compress.DoNotPool;
+import org.apache.hadoop.util.DataChecksum;
+
+/**
+ * A {@link Compressor} based on the popular gzip compressed file format.
+ * http://www.gzip.org/
+ */
+@DoNotPool
+public class BuiltInGzipCompressor implements Compressor {
+
+ /**
+ * Fixed ten-byte gzip header. See {@link GZIPOutputStream}'s source for
+ * details.
+ */
+ private static final byte[] GZIP_HEADER = new byte[]{
+ 0x1f, (byte) 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
+
+ // The trailer will be overwritten based on crc and output size.
+ private byte[] GZIP_TRAILER = new byte[]{0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00};
+
+ private final int GZIP_HEADER_LEN = GZIP_HEADER.length;
+ private final int GZIP_TRAILER_LEN = GZIP_TRAILER.length;
+
+ private Deflater deflater;
+
+ private int headerOff = 0;
+ private int trailerOff = 0;
+
+ private int numBytesWritten = 0;
+
+ private int currentBufLen = 0;
+
+ private final Checksum crc = DataChecksum.newCrc32();
+
+ private BuiltInGzipDecompressor.GzipStateLabel state;
+
+ public BuiltInGzipCompressor(Configuration conf) { init(conf); }
+
+ @Override
+ public boolean finished() {
+// Only if the trailer is also written, it is thought as finished.
+return deflater.finished() && state ==
BuiltInGzipDecompressor.GzipStateLabel.FINISHED;
+ }
+
+ @Override
+ public boolean needsInput() {
+if (state == BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM) {
+ return deflater.needsInput();
+}
+
+return (state != BuiltInGzipDecompressor.GzipStateLabel.FINISHED);
+ }
+
+ @Override
+ public int compress(byte[] b, int off, int len) throws IOException {
+int compressedBytesWritten = 0;
+
+// If we are not within uncompressed data yet, output the header.
+if (state != BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM &&
Review comment:
Hmm, oh, no, I think we cannot do it. `setInput` can be called multiple
times before we reach finished status. If we set the state to `HEADER_BASIC`,
it will re-output the header, but the current trailer is not output yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3271: HDFS-16155: Allow configurable exponential backoff in DFSInputStream refetchLocations
hadoop-yetus commented on pull request #3271:
URL: https://github.com/apache/hadoop/pull/3271#issuecomment-893900236
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 44s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 30s | | trunk passed |
| +1 :green_heart: | compile | 4m 58s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 4m 36s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 19s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 39s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 10s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 5m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 8s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 1s | | the patch passed |
| +1 :green_heart: | compile | 4m 43s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 4m 43s | | the patch passed |
| +1 :green_heart: | compile | 4m 27s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 4m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 6s |
[/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3271/1/artifact/out/results-checkstyle-hadoop-hdfs-project.txt)
| hadoop-hdfs-project: The patch generated 9 new + 333 unchanged - 0 fixed =
342 total (was 333) |
| +1 :green_heart: | mvnsite | 2m 5s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 23s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 48s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 5m 37s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 57s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 21s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 235m 47s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3271/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 349m 21s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes |
| | hadoop.tools.TestHdfsConfigFields |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3271/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3271 |
| JIRA Issue | HDFS-16155 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 9e6a49c0255e 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 36ae933e9b812891a13427b3ef69d0c2cb0b79ce |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
[jira] [Work logged] (HADOOP-15327) Upgrade MR ShuffleHandler to use Netty4
[
https://issues.apache.org/jira/browse/HADOOP-15327?focusedWorklogId=634865=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634865
]
ASF GitHub Bot logged work on HADOOP-15327:
---
Author: ASF GitHub Bot
Created on: 05/Aug/21 23:37
Start Date: 05/Aug/21 23:37
Worklog Time Spent: 10m
Work Description: shuzirra commented on a change in pull request #3259:
URL: https://github.com/apache/hadoop/pull/3259#discussion_r683795468
##
File path:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
##
@@ -106,10 +129,584 @@
LoggerFactory.getLogger(TestShuffleHandler.class);
private static final File ABS_LOG_DIR = GenericTestUtils.getTestDir(
TestShuffleHandler.class.getSimpleName() + "LocDir");
+ private static final long ATTEMPT_ID = 12345L;
+ private static final long ATTEMPT_ID_2 = 12346L;
+
+
+ //Control test execution properties with these flags
+ private static final boolean DEBUG_MODE = false;
+ //WARNING: If this is set to true and proxy server is not running, tests
will fail!
+ private static final boolean USE_PROXY = false;
+ private static final int HEADER_WRITE_COUNT = 10;
+ private static TestExecution TEST_EXECUTION;
+
+ private static class TestExecution {
+private static final int DEFAULT_KEEP_ALIVE_TIMEOUT = -100;
Review comment:
Interesting choice, why is it -100? Is it a special value, or we just
want a non-positive? If it has a special meaning please add a comment here.
##
File path:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
##
@@ -106,10 +129,584 @@
LoggerFactory.getLogger(TestShuffleHandler.class);
private static final File ABS_LOG_DIR = GenericTestUtils.getTestDir(
TestShuffleHandler.class.getSimpleName() + "LocDir");
+ private static final long ATTEMPT_ID = 12345L;
+ private static final long ATTEMPT_ID_2 = 12346L;
+
+
+ //Control test execution properties with these flags
+ private static final boolean DEBUG_MODE = false;
+ //WARNING: If this is set to true and proxy server is not running, tests
will fail!
+ private static final boolean USE_PROXY = false;
+ private static final int HEADER_WRITE_COUNT = 10;
+ private static TestExecution TEST_EXECUTION;
+
+ private static class TestExecution {
+private static final int DEFAULT_KEEP_ALIVE_TIMEOUT = -100;
+private static final int DEBUG_FRIENDLY_KEEP_ALIVE = 1000;
+private static final int DEFAULT_PORT = 0; //random port
+private static final int FIXED_PORT = 8088;
+private static final String PROXY_HOST = "127.0.0.1";
+private static final int PROXY_PORT = ;
+private final boolean debugMode;
+private final boolean useProxy;
+
+public TestExecution(boolean debugMode, boolean useProxy) {
+ this.debugMode = debugMode;
+ this.useProxy = useProxy;
+}
+
+int getKeepAliveTimeout() {
+ if (debugMode) {
+return DEBUG_FRIENDLY_KEEP_ALIVE;
+ }
+ return DEFAULT_KEEP_ALIVE_TIMEOUT;
+}
+
+HttpURLConnection openConnection(URL url) throws IOException {
+ HttpURLConnection conn;
+ if (useProxy) {
+Proxy proxy
+= new Proxy(Proxy.Type.HTTP, new InetSocketAddress(PROXY_HOST,
PROXY_PORT));
+conn = (HttpURLConnection) url.openConnection(proxy);
+ } else {
+conn = (HttpURLConnection) url.openConnection();
+ }
+ return conn;
+}
+
+int shuffleHandlerPort() {
+ if (debugMode) {
+return FIXED_PORT;
+ } else {
+return DEFAULT_PORT;
+ }
+}
+
+void parameterizeConnection(URLConnection conn) {
+ if (DEBUG_MODE) {
+conn.setReadTimeout(100);
+conn.setConnectTimeout(100);
+ }
+}
+ }
+
+ private static class ResponseConfig {
+private static final int ONE_HEADER_DISPLACEMENT = 1;
+
+private final int headerWriteCount;
+private final long actualHeaderWriteCount;
+private final int mapOutputCount;
+private final int contentLengthOfOneMapOutput;
+private long headerSize;
+public long contentLengthOfResponse;
+
+public ResponseConfig(int headerWriteCount, int mapOutputCount, int
contentLengthOfOneMapOutput) {
+ if (mapOutputCount <= 0 && contentLengthOfOneMapOutput > 0) {
+throw new IllegalStateException("mapOutputCount should be at least 1");
+ }
+ this.headerWriteCount = headerWriteCount;
+ this.mapOutputCount = mapOutputCount;
+ this.contentLengthOfOneMapOutput = contentLengthOfOneMapOutput;
+ //MapOutputSender#send will send header N + 1 times
+ //So, (N + 1) * headerSize should be the Content-length header + the
expected
[GitHub] [hadoop] shuzirra commented on a change in pull request #3259: HADOOP-15327. Upgrade MR ShuffleHandler to use Netty4
shuzirra commented on a change in pull request #3259:
URL: https://github.com/apache/hadoop/pull/3259#discussion_r683795468
##
File path:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
##
@@ -106,10 +129,584 @@
LoggerFactory.getLogger(TestShuffleHandler.class);
private static final File ABS_LOG_DIR = GenericTestUtils.getTestDir(
TestShuffleHandler.class.getSimpleName() + "LocDir");
+ private static final long ATTEMPT_ID = 12345L;
+ private static final long ATTEMPT_ID_2 = 12346L;
+
+
+ //Control test execution properties with these flags
+ private static final boolean DEBUG_MODE = false;
+ //WARNING: If this is set to true and proxy server is not running, tests
will fail!
+ private static final boolean USE_PROXY = false;
+ private static final int HEADER_WRITE_COUNT = 10;
+ private static TestExecution TEST_EXECUTION;
+
+ private static class TestExecution {
+private static final int DEFAULT_KEEP_ALIVE_TIMEOUT = -100;
Review comment:
Interesting choice, why is it -100? Is it a special value, or we just
want a non-positive? If it has a special meaning please add a comment here.
##
File path:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
##
@@ -106,10 +129,584 @@
LoggerFactory.getLogger(TestShuffleHandler.class);
private static final File ABS_LOG_DIR = GenericTestUtils.getTestDir(
TestShuffleHandler.class.getSimpleName() + "LocDir");
+ private static final long ATTEMPT_ID = 12345L;
+ private static final long ATTEMPT_ID_2 = 12346L;
+
+
+ //Control test execution properties with these flags
+ private static final boolean DEBUG_MODE = false;
+ //WARNING: If this is set to true and proxy server is not running, tests
will fail!
+ private static final boolean USE_PROXY = false;
+ private static final int HEADER_WRITE_COUNT = 10;
+ private static TestExecution TEST_EXECUTION;
+
+ private static class TestExecution {
+private static final int DEFAULT_KEEP_ALIVE_TIMEOUT = -100;
+private static final int DEBUG_FRIENDLY_KEEP_ALIVE = 1000;
+private static final int DEFAULT_PORT = 0; //random port
+private static final int FIXED_PORT = 8088;
+private static final String PROXY_HOST = "127.0.0.1";
+private static final int PROXY_PORT = ;
+private final boolean debugMode;
+private final boolean useProxy;
+
+public TestExecution(boolean debugMode, boolean useProxy) {
+ this.debugMode = debugMode;
+ this.useProxy = useProxy;
+}
+
+int getKeepAliveTimeout() {
+ if (debugMode) {
+return DEBUG_FRIENDLY_KEEP_ALIVE;
+ }
+ return DEFAULT_KEEP_ALIVE_TIMEOUT;
+}
+
+HttpURLConnection openConnection(URL url) throws IOException {
+ HttpURLConnection conn;
+ if (useProxy) {
+Proxy proxy
+= new Proxy(Proxy.Type.HTTP, new InetSocketAddress(PROXY_HOST,
PROXY_PORT));
+conn = (HttpURLConnection) url.openConnection(proxy);
+ } else {
+conn = (HttpURLConnection) url.openConnection();
+ }
+ return conn;
+}
+
+int shuffleHandlerPort() {
+ if (debugMode) {
+return FIXED_PORT;
+ } else {
+return DEFAULT_PORT;
+ }
+}
+
+void parameterizeConnection(URLConnection conn) {
+ if (DEBUG_MODE) {
+conn.setReadTimeout(100);
+conn.setConnectTimeout(100);
+ }
+}
+ }
+
+ private static class ResponseConfig {
+private static final int ONE_HEADER_DISPLACEMENT = 1;
+
+private final int headerWriteCount;
+private final long actualHeaderWriteCount;
+private final int mapOutputCount;
+private final int contentLengthOfOneMapOutput;
+private long headerSize;
+public long contentLengthOfResponse;
+
+public ResponseConfig(int headerWriteCount, int mapOutputCount, int
contentLengthOfOneMapOutput) {
+ if (mapOutputCount <= 0 && contentLengthOfOneMapOutput > 0) {
+throw new IllegalStateException("mapOutputCount should be at least 1");
+ }
+ this.headerWriteCount = headerWriteCount;
+ this.mapOutputCount = mapOutputCount;
+ this.contentLengthOfOneMapOutput = contentLengthOfOneMapOutput;
+ //MapOutputSender#send will send header N + 1 times
+ //So, (N + 1) * headerSize should be the Content-length header + the
expected Content-length as well
+ this.actualHeaderWriteCount = headerWriteCount + ONE_HEADER_DISPLACEMENT;
+}
+
+private void setHeaderSize(long headerSize) {
+ this.headerSize = headerSize;
+ long contentLengthOfAllHeaders = actualHeaderWriteCount * headerSize;
+ this.contentLengthOfResponse =
computeContentLengthOfResponse(contentLengthOfAllHeaders);
+ LOG.debug("Content-length of all headers:
[jira] [Work logged] (HADOOP-17825) Add BuiltInGzipCompressor
[
https://issues.apache.org/jira/browse/HADOOP-17825?focusedWorklogId=634858=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634858
]
ASF GitHub Bot logged work on HADOOP-17825:
---
Author: ASF GitHub Bot
Created on: 05/Aug/21 23:07
Start Date: 05/Aug/21 23:07
Worklog Time Spent: 10m
Work Description: viirya commented on a change in pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#discussion_r683840945
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress.zlib;
+
+import java.io.IOException;
+import java.util.zip.Checksum;
+import java.util.zip.Deflater;
+import java.util.zip.GZIPOutputStream;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.compress.Compressor;
+import org.apache.hadoop.io.compress.DoNotPool;
+import org.apache.hadoop.util.DataChecksum;
+
+/**
+ * A {@link Compressor} based on the popular gzip compressed file format.
+ * http://www.gzip.org/
+ */
+@DoNotPool
+public class BuiltInGzipCompressor implements Compressor {
+
+ /**
+ * Fixed ten-byte gzip header. See {@link GZIPOutputStream}'s source for
+ * details.
+ */
+ private static final byte[] GZIP_HEADER = new byte[]{
+ 0x1f, (byte) 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
+
+ // The trailer will be overwritten based on crc and output size.
+ private byte[] GZIP_TRAILER = new byte[]{0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00};
+
+ private final int GZIP_HEADER_LEN = GZIP_HEADER.length;
+ private final int GZIP_TRAILER_LEN = GZIP_TRAILER.length;
+
+ private Deflater deflater;
+
+ private int headerOff = 0;
+ private int trailerOff = 0;
+
+ private int numBytesWritten = 0;
+
+ private int currentBufLen = 0;
+
+ private final Checksum crc = DataChecksum.newCrc32();
+
+ private BuiltInGzipDecompressor.GzipStateLabel state;
+
+ public BuiltInGzipCompressor(Configuration conf) { init(conf); }
+
+ @Override
+ public boolean finished() {
+// Only if the trailer is also written, it is thought as finished.
+return deflater.finished() && state ==
BuiltInGzipDecompressor.GzipStateLabel.FINISHED;
+ }
+
+ @Override
+ public boolean needsInput() {
+if (state == BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM) {
+ return deflater.needsInput();
+}
+
+return (state != BuiltInGzipDecompressor.GzipStateLabel.FINISHED);
+ }
+
+ @Override
+ public int compress(byte[] b, int off, int len) throws IOException {
+int compressedBytesWritten = 0;
+
+// If we are not within uncompressed data yet, output the header.
+if (state != BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM &&
Review comment:
Seems so. Let me update it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 634858)
Time Spent: 12h 50m (was: 12h 40m)
> Add BuiltInGzipCompressor
> -
>
> Key: HADOOP-17825
> URL: https://issues.apache.org/jira/browse/HADOOP-17825
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 12h 50m
> Remaining Estimate: 0h
>
> Currently, GzipCodec only supports BuiltInGzipDecompressor, if native zlib is
> not loaded. So, without Hadoop native codec installed, saving SequenceFile
> using GzipCodec will throw exception like "SequenceFile doesn't work with
> GzipCodec without native-hadoop code!"
> Same as other codecs which we migrated to using prepared packages (lz4,
> snappy), it
[GitHub] [hadoop] viirya commented on a change in pull request #3250: HADOOP-17825. Add BuiltInGzipCompressor
viirya commented on a change in pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#discussion_r683840945
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress.zlib;
+
+import java.io.IOException;
+import java.util.zip.Checksum;
+import java.util.zip.Deflater;
+import java.util.zip.GZIPOutputStream;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.compress.Compressor;
+import org.apache.hadoop.io.compress.DoNotPool;
+import org.apache.hadoop.util.DataChecksum;
+
+/**
+ * A {@link Compressor} based on the popular gzip compressed file format.
+ * http://www.gzip.org/
+ */
+@DoNotPool
+public class BuiltInGzipCompressor implements Compressor {
+
+ /**
+ * Fixed ten-byte gzip header. See {@link GZIPOutputStream}'s source for
+ * details.
+ */
+ private static final byte[] GZIP_HEADER = new byte[]{
+ 0x1f, (byte) 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
+
+ // The trailer will be overwritten based on crc and output size.
+ private byte[] GZIP_TRAILER = new byte[]{0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00};
+
+ private final int GZIP_HEADER_LEN = GZIP_HEADER.length;
+ private final int GZIP_TRAILER_LEN = GZIP_TRAILER.length;
+
+ private Deflater deflater;
+
+ private int headerOff = 0;
+ private int trailerOff = 0;
+
+ private int numBytesWritten = 0;
+
+ private int currentBufLen = 0;
+
+ private final Checksum crc = DataChecksum.newCrc32();
+
+ private BuiltInGzipDecompressor.GzipStateLabel state;
+
+ public BuiltInGzipCompressor(Configuration conf) { init(conf); }
+
+ @Override
+ public boolean finished() {
+// Only if the trailer is also written, it is thought as finished.
+return deflater.finished() && state ==
BuiltInGzipDecompressor.GzipStateLabel.FINISHED;
+ }
+
+ @Override
+ public boolean needsInput() {
+if (state == BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM) {
+ return deflater.needsInput();
+}
+
+return (state != BuiltInGzipDecompressor.GzipStateLabel.FINISHED);
+ }
+
+ @Override
+ public int compress(byte[] b, int off, int len) throws IOException {
+int compressedBytesWritten = 0;
+
+// If we are not within uncompressed data yet, output the header.
+if (state != BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM &&
Review comment:
Seems so. Let me update it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17825) Add BuiltInGzipCompressor
[
https://issues.apache.org/jira/browse/HADOOP-17825?focusedWorklogId=634855=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634855
]
ASF GitHub Bot logged work on HADOOP-17825:
---
Author: ASF GitHub Bot
Created on: 05/Aug/21 22:58
Start Date: 05/Aug/21 22:58
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#issuecomment-893873105
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 1s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 50s | | trunk passed |
| +1 :green_heart: | compile | 21m 15s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 18m 34s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 10s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 43s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 23s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 47s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 56s | | the patch passed |
| +1 :green_heart: | compile | 20m 29s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 20m 29s | | the patch passed |
| +1 :green_heart: | compile | 18m 36s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 18m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 8s |
[/results-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3250/18/artifact/out/results-checkstyle-hadoop-common-project_hadoop-common.txt)
| hadoop-common-project/hadoop-common: The patch generated 7 new + 332
unchanged - 0 fixed = 339 total (was 332) |
| +1 :green_heart: | mvnsite | 1m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 8s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 38s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 33s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 58s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 13s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 58s | | The patch does not
generate ASF License warnings. |
| | | 178m 1s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3250/18/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3250 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 53a5467acfb9 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 204709d1bc0ebb41521219367c39d1badc698ab1 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3250/18/testReport/ |
| Max. process+thread count | 1266 (vs. ulimit of 5500) |
| modules | C:
[GitHub] [hadoop] hadoop-yetus commented on pull request #3250: HADOOP-17825. Add BuiltInGzipCompressor
hadoop-yetus commented on pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#issuecomment-893873105
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 1s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 50s | | trunk passed |
| +1 :green_heart: | compile | 21m 15s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 18m 34s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 10s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 43s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 23s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 47s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 56s | | the patch passed |
| +1 :green_heart: | compile | 20m 29s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 20m 29s | | the patch passed |
| +1 :green_heart: | compile | 18m 36s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 18m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 8s |
[/results-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3250/18/artifact/out/results-checkstyle-hadoop-common-project_hadoop-common.txt)
| hadoop-common-project/hadoop-common: The patch generated 7 new + 332
unchanged - 0 fixed = 339 total (was 332) |
| +1 :green_heart: | mvnsite | 1m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 8s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 38s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 33s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 58s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 13s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 58s | | The patch does not
generate ASF License warnings. |
| | | 178m 1s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3250/18/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3250 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 53a5467acfb9 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 204709d1bc0ebb41521219367c39d1badc698ab1 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3250/18/testReport/ |
| Max. process+thread count | 1266 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3250/18/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond
[jira] [Commented] (HADOOP-17837) Make it easier to debug UnknownHostExceptions from NetUtils.connect
[
https://issues.apache.org/jira/browse/HADOOP-17837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394347#comment-17394347
]
Hadoop QA commented on HADOOP-17837:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
51s{color} | | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} codespell {color} | {color:blue} 0m
0s{color} | | {color:blue} codespell was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | | {color:green} The patch does not contain any @author tags.
{color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | | {color:green} The patch appears to include 1 new or modified
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 30m
42s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
13s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
26s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
38s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
41s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 2m
26s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 52s{color} | | {color:green} branch has no errors when building and
testing our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
25s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 20m
25s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
25s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 18m
25s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} blanks {color} | {color:green} 0m
0s{color} | | {color:green} The patch has no blanks issues. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
36s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
41s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 2m
34s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 5s{color} | | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} || ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 17m
11s{color} | | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
59s{color} | | {color:green} The patch
[jira] [Work logged] (HADOOP-17837) Make it easier to debug UnknownHostExceptions from NetUtils.connect
[
https://issues.apache.org/jira/browse/HADOOP-17837?focusedWorklogId=634843=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634843
]
ASF GitHub Bot logged work on HADOOP-17837:
---
Author: ASF GitHub Bot
Created on: 05/Aug/21 22:38
Start Date: 05/Aug/21 22:38
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3272:
URL: https://github.com/apache/hadoop/pull/3272#issuecomment-893865705
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 42s | | trunk passed |
| +1 :green_heart: | compile | 21m 13s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 18m 26s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 8s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 38s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 41s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 52s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 54s | | the patch passed |
| +1 :green_heart: | compile | 20m 25s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 20m 25s | | the patch passed |
| +1 :green_heart: | compile | 18m 25s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 18m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 6s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 36s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 8s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 41s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 34s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 5s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 11s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not
generate ASF License warnings. |
| | | 177m 48s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3272/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3272 |
| JIRA Issue | HADOOP-17837 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 696c85f030a5 4.15.0-151-generic #157-Ubuntu SMP Fri Jul 9
23:07:57 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 571fc27f6c0e17af6b04fc79a7c14b2d40938da3 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3272/1/testReport/ |
| Max. process+thread count | 1534 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3272/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT
[GitHub] [hadoop] hadoop-yetus commented on pull request #3272: HADOOP-17837: Add unresolved endpoint value to UnknownHostException
hadoop-yetus commented on pull request #3272:
URL: https://github.com/apache/hadoop/pull/3272#issuecomment-893865705
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 42s | | trunk passed |
| +1 :green_heart: | compile | 21m 13s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 18m 26s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 8s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 38s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 41s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 52s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 54s | | the patch passed |
| +1 :green_heart: | compile | 20m 25s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 20m 25s | | the patch passed |
| +1 :green_heart: | compile | 18m 25s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 18m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 6s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 36s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 8s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 41s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 34s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 5s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 11s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not
generate ASF License warnings. |
| | | 177m 48s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3272/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3272 |
| JIRA Issue | HADOOP-17837 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 696c85f030a5 4.15.0-151-generic #157-Ubuntu SMP Fri Jul 9
23:07:57 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 571fc27f6c0e17af6b04fc79a7c14b2d40938da3 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3272/1/testReport/ |
| Max. process+thread count | 1534 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3272/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Work logged] (HADOOP-17825) Add BuiltInGzipCompressor
[
https://issues.apache.org/jira/browse/HADOOP-17825?focusedWorklogId=634831=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634831
]
ASF GitHub Bot logged work on HADOOP-17825:
---
Author: ASF GitHub Bot
Created on: 05/Aug/21 22:04
Start Date: 05/Aug/21 22:04
Worklog Time Spent: 10m
Work Description: sunchao commented on a change in pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#discussion_r683816220
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress.zlib;
+
+import java.io.IOException;
+import java.util.zip.Checksum;
+import java.util.zip.Deflater;
+import java.util.zip.GZIPOutputStream;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.compress.Compressor;
+import org.apache.hadoop.io.compress.DoNotPool;
+import org.apache.hadoop.util.DataChecksum;
+
+/**
+ * A {@link Compressor} based on the popular gzip compressed file format.
+ * http://www.gzip.org/
+ */
+@DoNotPool
+public class BuiltInGzipCompressor implements Compressor {
+
+ /**
+ * Fixed ten-byte gzip header. See {@link GZIPOutputStream}'s source for
+ * details.
+ */
+ private static final byte[] GZIP_HEADER = new byte[]{
+ 0x1f, (byte) 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
+
+ // The trailer will be overwritten based on crc and output size.
+ private byte[] GZIP_TRAILER = new byte[]{0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00};
+
+ private final int GZIP_HEADER_LEN = GZIP_HEADER.length;
+ private final int GZIP_TRAILER_LEN = GZIP_TRAILER.length;
+
+ private Deflater deflater;
+
+ private int headerOff = 0;
+ private int trailerOff = 0;
+
+ private int numBytesWritten = 0;
+
+ private int currentBufLen = 0;
+
+ private final Checksum crc = DataChecksum.newCrc32();
+
+ private BuiltInGzipDecompressor.GzipStateLabel state;
+
+ public BuiltInGzipCompressor(Configuration conf) { init(conf); }
+
+ @Override
+ public boolean finished() {
+// Only if the trailer is also written, it is thought as finished.
+return deflater.finished() && state ==
BuiltInGzipDecompressor.GzipStateLabel.FINISHED;
+ }
+
+ @Override
+ public boolean needsInput() {
+if (state == BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM) {
+ return deflater.needsInput();
+}
+
+return (state != BuiltInGzipDecompressor.GzipStateLabel.FINISHED);
+ }
+
+ @Override
+ public int compress(byte[] b, int off, int len) throws IOException {
+int compressedBytesWritten = 0;
+
+// If we are not within uncompressed data yet, output the header.
+if (state != BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM &&
Review comment:
Hmm you are right. Should we change the state to `HEADER_BASIC` in
`setInput`? it seems we should do so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 634831)
Time Spent: 12.5h (was: 12h 20m)
> Add BuiltInGzipCompressor
> -
>
> Key: HADOOP-17825
> URL: https://issues.apache.org/jira/browse/HADOOP-17825
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 12.5h
> Remaining Estimate: 0h
>
> Currently, GzipCodec only supports BuiltInGzipDecompressor, if native zlib is
> not loaded. So, without Hadoop native codec installed, saving SequenceFile
> using GzipCodec will throw exception like "SequenceFile doesn't work with
> GzipCodec without native-hadoop code!"
> Same as other
[GitHub] [hadoop] sunchao commented on a change in pull request #3250: HADOOP-17825. Add BuiltInGzipCompressor
sunchao commented on a change in pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#discussion_r683816220
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress.zlib;
+
+import java.io.IOException;
+import java.util.zip.Checksum;
+import java.util.zip.Deflater;
+import java.util.zip.GZIPOutputStream;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.compress.Compressor;
+import org.apache.hadoop.io.compress.DoNotPool;
+import org.apache.hadoop.util.DataChecksum;
+
+/**
+ * A {@link Compressor} based on the popular gzip compressed file format.
+ * http://www.gzip.org/
+ */
+@DoNotPool
+public class BuiltInGzipCompressor implements Compressor {
+
+ /**
+ * Fixed ten-byte gzip header. See {@link GZIPOutputStream}'s source for
+ * details.
+ */
+ private static final byte[] GZIP_HEADER = new byte[]{
+ 0x1f, (byte) 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
+
+ // The trailer will be overwritten based on crc and output size.
+ private byte[] GZIP_TRAILER = new byte[]{0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00};
+
+ private final int GZIP_HEADER_LEN = GZIP_HEADER.length;
+ private final int GZIP_TRAILER_LEN = GZIP_TRAILER.length;
+
+ private Deflater deflater;
+
+ private int headerOff = 0;
+ private int trailerOff = 0;
+
+ private int numBytesWritten = 0;
+
+ private int currentBufLen = 0;
+
+ private final Checksum crc = DataChecksum.newCrc32();
+
+ private BuiltInGzipDecompressor.GzipStateLabel state;
+
+ public BuiltInGzipCompressor(Configuration conf) { init(conf); }
+
+ @Override
+ public boolean finished() {
+// Only if the trailer is also written, it is thought as finished.
+return deflater.finished() && state ==
BuiltInGzipDecompressor.GzipStateLabel.FINISHED;
+ }
+
+ @Override
+ public boolean needsInput() {
+if (state == BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM) {
+ return deflater.needsInput();
+}
+
+return (state != BuiltInGzipDecompressor.GzipStateLabel.FINISHED);
+ }
+
+ @Override
+ public int compress(byte[] b, int off, int len) throws IOException {
+int compressedBytesWritten = 0;
+
+// If we are not within uncompressed data yet, output the header.
+if (state != BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM &&
Review comment:
Hmm you are right. Should we change the state to `HEADER_BASIC` in
`setInput`? it seems we should do so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17825) Add BuiltInGzipCompressor
[
https://issues.apache.org/jira/browse/HADOOP-17825?focusedWorklogId=634786=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634786
]
ASF GitHub Bot logged work on HADOOP-17825:
---
Author: ASF GitHub Bot
Created on: 05/Aug/21 20:08
Start Date: 05/Aug/21 20:08
Worklog Time Spent: 10m
Work Description: viirya commented on a change in pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#discussion_r683753560
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress.zlib;
+
+import java.io.IOException;
+import java.util.zip.Checksum;
+import java.util.zip.Deflater;
+import java.util.zip.GZIPOutputStream;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.compress.Compressor;
+import org.apache.hadoop.io.compress.DoNotPool;
+import org.apache.hadoop.util.DataChecksum;
+
+/**
+ * A {@link Compressor} based on the popular gzip compressed file format.
+ * http://www.gzip.org/
+ */
+@DoNotPool
+public class BuiltInGzipCompressor implements Compressor {
+
+ /**
+ * Fixed ten-byte gzip header. See {@link GZIPOutputStream}'s source for
+ * details.
+ */
+ private static final byte[] GZIP_HEADER = new byte[]{
+ 0x1f, (byte) 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
+
+ // The trailer will be overwritten based on crc and output size.
+ private byte[] GZIP_TRAILER = new byte[]{0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00};
+
+ private final int GZIP_HEADER_LEN = GZIP_HEADER.length;
+ private final int GZIP_TRAILER_LEN = GZIP_TRAILER.length;
+
+ private Deflater deflater;
+
+ private int headerOff = 0;
+ private int trailerOff = 0;
+
+ private int numBytesWritten = 0;
+
+ private int currentBufLen = 0;
+
+ private final Checksum crc = DataChecksum.newCrc32();
+
+ private BuiltInGzipDecompressor.GzipStateLabel state;
+
+ public BuiltInGzipCompressor(Configuration conf) { init(conf); }
+
+ @Override
+ public boolean finished() {
+// Only if the trailer is also written, it is thought as finished.
+return deflater.finished() && state ==
BuiltInGzipDecompressor.GzipStateLabel.FINISHED;
+ }
+
+ @Override
+ public boolean needsInput() {
+if (state == BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM) {
+ return deflater.needsInput();
+}
+
+return (state != BuiltInGzipDecompressor.GzipStateLabel.FINISHED);
+ }
+
+ @Override
+ public int compress(byte[] b, int off, int len) throws IOException {
+int compressedBytesWritten = 0;
+
+// If we are not within uncompressed data yet, output the header.
+if (state != BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM &&
Review comment:
What I thought is, the caller might set input and compress until it
doesn't need input. The state is in `FINISHED` and the caller might call set
input and compress again. At the moment this check isn't effective to write the
header.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 634786)
Time Spent: 12h 20m (was: 12h 10m)
> Add BuiltInGzipCompressor
> -
>
> Key: HADOOP-17825
> URL: https://issues.apache.org/jira/browse/HADOOP-17825
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 12h 20m
> Remaining Estimate: 0h
>
> Currently, GzipCodec only supports BuiltInGzipDecompressor, if native zlib is
> not loaded. So, without Hadoop native codec installed, saving SequenceFile
>
[GitHub] [hadoop] viirya commented on a change in pull request #3250: HADOOP-17825. Add BuiltInGzipCompressor
viirya commented on a change in pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#discussion_r683753560
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress.zlib;
+
+import java.io.IOException;
+import java.util.zip.Checksum;
+import java.util.zip.Deflater;
+import java.util.zip.GZIPOutputStream;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.compress.Compressor;
+import org.apache.hadoop.io.compress.DoNotPool;
+import org.apache.hadoop.util.DataChecksum;
+
+/**
+ * A {@link Compressor} based on the popular gzip compressed file format.
+ * http://www.gzip.org/
+ */
+@DoNotPool
+public class BuiltInGzipCompressor implements Compressor {
+
+ /**
+ * Fixed ten-byte gzip header. See {@link GZIPOutputStream}'s source for
+ * details.
+ */
+ private static final byte[] GZIP_HEADER = new byte[]{
+ 0x1f, (byte) 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
+
+ // The trailer will be overwritten based on crc and output size.
+ private byte[] GZIP_TRAILER = new byte[]{0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00};
+
+ private final int GZIP_HEADER_LEN = GZIP_HEADER.length;
+ private final int GZIP_TRAILER_LEN = GZIP_TRAILER.length;
+
+ private Deflater deflater;
+
+ private int headerOff = 0;
+ private int trailerOff = 0;
+
+ private int numBytesWritten = 0;
+
+ private int currentBufLen = 0;
+
+ private final Checksum crc = DataChecksum.newCrc32();
+
+ private BuiltInGzipDecompressor.GzipStateLabel state;
+
+ public BuiltInGzipCompressor(Configuration conf) { init(conf); }
+
+ @Override
+ public boolean finished() {
+// Only if the trailer is also written, it is thought as finished.
+return deflater.finished() && state ==
BuiltInGzipDecompressor.GzipStateLabel.FINISHED;
+ }
+
+ @Override
+ public boolean needsInput() {
+if (state == BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM) {
+ return deflater.needsInput();
+}
+
+return (state != BuiltInGzipDecompressor.GzipStateLabel.FINISHED);
+ }
+
+ @Override
+ public int compress(byte[] b, int off, int len) throws IOException {
+int compressedBytesWritten = 0;
+
+// If we are not within uncompressed data yet, output the header.
+if (state != BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM &&
Review comment:
What I thought is, the caller might set input and compress until it
doesn't need input. The state is in `FINISHED` and the caller might call set
input and compress again. At the moment this check isn't effective to write the
header.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17825) Add BuiltInGzipCompressor
[
https://issues.apache.org/jira/browse/HADOOP-17825?focusedWorklogId=634779=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634779
]
ASF GitHub Bot logged work on HADOOP-17825:
---
Author: ASF GitHub Bot
Created on: 05/Aug/21 20:01
Start Date: 05/Aug/21 20:01
Worklog Time Spent: 10m
Work Description: viirya commented on a change in pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#discussion_r683749490
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress.zlib;
+
+import java.io.IOException;
+import java.util.zip.Checksum;
+import java.util.zip.Deflater;
+import java.util.zip.GZIPOutputStream;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.compress.Compressor;
+import org.apache.hadoop.io.compress.DoNotPool;
+import org.apache.hadoop.util.DataChecksum;
+
+/**
+ * A {@link Compressor} based on the popular gzip compressed file format.
+ * http://www.gzip.org/
+ */
+@DoNotPool
+public class BuiltInGzipCompressor implements Compressor {
+
+ /**
+ * Fixed ten-byte gzip header. See {@link GZIPOutputStream}'s source for
+ * details.
+ */
+ private static final byte[] GZIP_HEADER = new byte[]{
+ 0x1f, (byte) 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
+
+ // The trailer will be overwritten based on crc and output size.
+ private byte[] GZIP_TRAILER = new byte[]{0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00};
+
+ private final int GZIP_HEADER_LEN = GZIP_HEADER.length;
+ private final int GZIP_TRAILER_LEN = GZIP_TRAILER.length;
+
+ private Deflater deflater;
+
+ private int headerOff = 0;
+ private int trailerOff = 0;
+
+ private int numBytesWritten = 0;
+
+ private int currentBufLen = 0;
+
+ private final Checksum crc = DataChecksum.newCrc32();
+
+ private BuiltInGzipDecompressor.GzipStateLabel state;
+
+ public BuiltInGzipCompressor(Configuration conf) { init(conf); }
+
+ @Override
+ public boolean finished() {
+// Only if the trailer is also written, it is thought as finished.
+return deflater.finished() && state ==
BuiltInGzipDecompressor.GzipStateLabel.FINISHED;
+ }
+
+ @Override
+ public boolean needsInput() {
+if (state == BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM) {
+ return deflater.needsInput();
+}
+
+return (state != BuiltInGzipDecompressor.GzipStateLabel.FINISHED);
+ }
+
+ @Override
+ public int compress(byte[] b, int off, int len) throws IOException {
+int compressedBytesWritten = 0;
+
+// If we are not within uncompressed data yet, output the header.
+if (state != BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM &&
Review comment:
Hmm, okay. I set it to `HEADER_BASIC` and see if CI can pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 634779)
Time Spent: 12h 10m (was: 12h)
> Add BuiltInGzipCompressor
> -
>
> Key: HADOOP-17825
> URL: https://issues.apache.org/jira/browse/HADOOP-17825
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 12h 10m
> Remaining Estimate: 0h
>
> Currently, GzipCodec only supports BuiltInGzipDecompressor, if native zlib is
> not loaded. So, without Hadoop native codec installed, saving SequenceFile
> using GzipCodec will throw exception like "SequenceFile doesn't work with
> GzipCodec without native-hadoop code!"
> Same as other codecs which we migrated to using prepared
[GitHub] [hadoop] viirya commented on a change in pull request #3250: HADOOP-17825. Add BuiltInGzipCompressor
viirya commented on a change in pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#discussion_r683749490
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress.zlib;
+
+import java.io.IOException;
+import java.util.zip.Checksum;
+import java.util.zip.Deflater;
+import java.util.zip.GZIPOutputStream;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.compress.Compressor;
+import org.apache.hadoop.io.compress.DoNotPool;
+import org.apache.hadoop.util.DataChecksum;
+
+/**
+ * A {@link Compressor} based on the popular gzip compressed file format.
+ * http://www.gzip.org/
+ */
+@DoNotPool
+public class BuiltInGzipCompressor implements Compressor {
+
+ /**
+ * Fixed ten-byte gzip header. See {@link GZIPOutputStream}'s source for
+ * details.
+ */
+ private static final byte[] GZIP_HEADER = new byte[]{
+ 0x1f, (byte) 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
+
+ // The trailer will be overwritten based on crc and output size.
+ private byte[] GZIP_TRAILER = new byte[]{0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00};
+
+ private final int GZIP_HEADER_LEN = GZIP_HEADER.length;
+ private final int GZIP_TRAILER_LEN = GZIP_TRAILER.length;
+
+ private Deflater deflater;
+
+ private int headerOff = 0;
+ private int trailerOff = 0;
+
+ private int numBytesWritten = 0;
+
+ private int currentBufLen = 0;
+
+ private final Checksum crc = DataChecksum.newCrc32();
+
+ private BuiltInGzipDecompressor.GzipStateLabel state;
+
+ public BuiltInGzipCompressor(Configuration conf) { init(conf); }
+
+ @Override
+ public boolean finished() {
+// Only if the trailer is also written, it is thought as finished.
+return deflater.finished() && state ==
BuiltInGzipDecompressor.GzipStateLabel.FINISHED;
+ }
+
+ @Override
+ public boolean needsInput() {
+if (state == BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM) {
+ return deflater.needsInput();
+}
+
+return (state != BuiltInGzipDecompressor.GzipStateLabel.FINISHED);
+ }
+
+ @Override
+ public int compress(byte[] b, int off, int len) throws IOException {
+int compressedBytesWritten = 0;
+
+// If we are not within uncompressed data yet, output the header.
+if (state != BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM &&
Review comment:
Hmm, okay. I set it to `HEADER_BASIC` and see if CI can pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3269: HDFS-16153. Avoid evaluation of LOG.debug statement in QuorumJournalM…
hadoop-yetus commented on pull request #3269:
URL: https://github.com/apache/hadoop/pull/3269#issuecomment-893751402
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 39s | | trunk passed |
| +1 :green_heart: | compile | 1m 26s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 14s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 58s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 56s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 25s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 17s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 8s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 12s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 12s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 16s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 49s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 22s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 20s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 33s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 351m 56s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3269/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 444m 11s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3269/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3269 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux a13402af33a4 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18
19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 29cb5eabcc6a5f6831195c761a6c085cfbb4443a |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3269/2/testReport/ |
| Max. process+thread count | 2482 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Work logged] (HADOOP-17837) Make it easier to debug UnknownHostExceptions from NetUtils.connect
[ https://issues.apache.org/jira/browse/HADOOP-17837?focusedWorklogId=634764=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634764 ] ASF GitHub Bot logged work on HADOOP-17837: --- Author: ASF GitHub Bot Created on: 05/Aug/21 19:39 Start Date: 05/Aug/21 19:39 Worklog Time Spent: 10m Work Description: bbeaudreault opened a new pull request #3272: URL: https://github.com/apache/hadoop/pull/3272 Per https://issues.apache.org/jira/browse/HADOOP-17837, adds `endpoint.toString() to the UnknownHostException. UnresolvedAddressException cannot have a message, so the current behavior is a UHE with no message as well. Adding the endpoint to the exception will make it easier to debug when these arise. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 634764) Remaining Estimate: 0h Time Spent: 10m > Make it easier to debug UnknownHostExceptions from NetUtils.connect > --- > > Key: HADOOP-17837 > URL: https://issues.apache.org/jira/browse/HADOOP-17837 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Bryan Beaudreault >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > Most UnknownHostExceptions thrown throughout hadoop include a useful message, > either the hostname that was not found or some other descriptor of the > problem. The UnknownHostException thrown from NetUtils.connect only includes > the [message of the underlying > UnresolvedAddressException|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java#L592]. > If you take a look at the source for UnresolvedAddressException, [it only > has a no-args > constructor|https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/nio/channels/UnresolvedAddressException.html] > (java11, but same is true in other versions). So it never has a message, > meaning the UnknownHostException message is empty. > We should include the endpoint.toString() in the UnknownHostException thrown > by NetUtils.connect -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17837) Make it easier to debug UnknownHostExceptions from NetUtils.connect
[ https://issues.apache.org/jira/browse/HADOOP-17837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-17837: Labels: pull-request-available (was: ) > Make it easier to debug UnknownHostExceptions from NetUtils.connect > --- > > Key: HADOOP-17837 > URL: https://issues.apache.org/jira/browse/HADOOP-17837 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Bryan Beaudreault >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Most UnknownHostExceptions thrown throughout hadoop include a useful message, > either the hostname that was not found or some other descriptor of the > problem. The UnknownHostException thrown from NetUtils.connect only includes > the [message of the underlying > UnresolvedAddressException|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java#L592]. > If you take a look at the source for UnresolvedAddressException, [it only > has a no-args > constructor|https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/nio/channels/UnresolvedAddressException.html] > (java11, but same is true in other versions). So it never has a message, > meaning the UnknownHostException message is empty. > We should include the endpoint.toString() in the UnknownHostException thrown > by NetUtils.connect -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bbeaudreault opened a new pull request #3272: HADOOP-17837: Add unresolved endpoint value to UnknownHostException
bbeaudreault opened a new pull request #3272: URL: https://github.com/apache/hadoop/pull/3272 Per https://issues.apache.org/jira/browse/HADOOP-17837, adds `endpoint.toString() to the UnknownHostException. UnresolvedAddressException cannot have a message, so the current behavior is a UHE with no message as well. Adding the endpoint to the exception will make it easier to debug when these arise. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17825) Add BuiltInGzipCompressor
[
https://issues.apache.org/jira/browse/HADOOP-17825?focusedWorklogId=634763=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634763
]
ASF GitHub Bot logged work on HADOOP-17825:
---
Author: ASF GitHub Bot
Created on: 05/Aug/21 19:31
Start Date: 05/Aug/21 19:31
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#issuecomment-893728072
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 43s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 9s | | trunk passed |
| +1 :green_heart: | compile | 21m 24s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 18m 42s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 8s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 38s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 10s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 44s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 24s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 52s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 56s | | the patch passed |
| +1 :green_heart: | compile | 20m 36s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 20m 36s | | the patch passed |
| +1 :green_heart: | compile | 18m 34s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 18m 34s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 9s |
[/results-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3250/17/artifact/out/results-checkstyle-hadoop-common-project_hadoop-common.txt)
| hadoop-common-project/hadoop-common: The patch generated 7 new + 332
unchanged - 0 fixed = 339 total (was 332) |
| +1 :green_heart: | mvnsite | 1m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 8s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 44s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 34s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 52s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 17s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not
generate ASF License warnings. |
| | | 178m 55s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3250/17/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3250 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux e238c79df9e4 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f1db328eb900b4ffe96f4150ecb4359d389c67de |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3250/17/testReport/ |
| Max. process+thread count | 3158 (vs. ulimit of 5500) |
| modules | C:
[GitHub] [hadoop] hadoop-yetus commented on pull request #3250: HADOOP-17825. Add BuiltInGzipCompressor
hadoop-yetus commented on pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#issuecomment-893728072
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 43s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 9s | | trunk passed |
| +1 :green_heart: | compile | 21m 24s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 18m 42s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 8s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 38s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 10s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 44s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 24s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 52s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 56s | | the patch passed |
| +1 :green_heart: | compile | 20m 36s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 20m 36s | | the patch passed |
| +1 :green_heart: | compile | 18m 34s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 18m 34s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 9s |
[/results-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3250/17/artifact/out/results-checkstyle-hadoop-common-project_hadoop-common.txt)
| hadoop-common-project/hadoop-common: The patch generated 7 new + 332
unchanged - 0 fixed = 339 total (was 332) |
| +1 :green_heart: | mvnsite | 1m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 8s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 44s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 34s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 52s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 17s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not
generate ASF License warnings. |
| | | 178m 55s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3250/17/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3250 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux e238c79df9e4 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f1db328eb900b4ffe96f4150ecb4359d389c67de |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3250/17/testReport/ |
| Max. process+thread count | 3158 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3250/17/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond
[jira] [Created] (HADOOP-17837) Make it easier to debug UnknownHostExceptions from NetUtils.connect
Bryan Beaudreault created HADOOP-17837: -- Summary: Make it easier to debug UnknownHostExceptions from NetUtils.connect Key: HADOOP-17837 URL: https://issues.apache.org/jira/browse/HADOOP-17837 Project: Hadoop Common Issue Type: Improvement Reporter: Bryan Beaudreault Most UnknownHostExceptions thrown throughout hadoop include a useful message, either the hostname that was not found or some other descriptor of the problem. The UnknownHostException thrown from NetUtils.connect only includes the [message of the underlying UnresolvedAddressException|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java#L592]. If you take a look at the source for UnresolvedAddressException, [it only has a no-args constructor|https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/nio/channels/UnresolvedAddressException.html] (java11, but same is true in other versions). So it never has a message, meaning the UnknownHostException message is empty. We should include the endpoint.toString() in the UnknownHostException thrown by NetUtils.connect -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bbeaudreault opened a new pull request #3271: HDFS-16155: Allow configurable exponential backoff in DFSInputStream refetchLocations
bbeaudreault opened a new pull request #3271: URL: https://github.com/apache/hadoop/pull/3271 Per https://issues.apache.org/jira/browse/HDFS-16155, we would like the ability to customize the backoff strategy when BlockMissingException occurs. This can happen when the balancer moves blocks, and in low latency clusters the existing backoff is too conservative. Drastically reducing the existing window base config would help but expose the namenode to a potential DDOS if many blocks became missing, because the current backoff would grow slowly. Adding a configurable exponential component allows for aggressive early retries that back off quickly enough to mitigate stampeding herds. We make the backoff configurable by adding two new configs: - `dfs.client.retry.window.multiplier`: defaults to 1 to preserve existing behavior. Increasing this can result in a steeper backoff curve when desired - `dfs.client.retry.window.max`: defaults to Int.MAX to preserve existing behavior. Decreasing this can help put a ceiling on exponential backoffs that could quickly grow to effectively unlimited levels. As described, the default behavior is maintained and I've added a test case to verify that. Someone looking for a more aggressive initial retry that backs off quickly in case of continuous failure could try setting `window.base` to 10, `window.multiplier` to 5, and `window.max` to 1. This would result in a quick initial retry of max 50ms, but quickly backoff to a few seconds within 3 retries. In order to improve the testability of this feature, I pulled out the existing refetchLocations retry configs into a FetchBlockLocationsRetryer class. I also improved the readability of the comment describing the backoff strategy, and fully tested the new retryer in TestFetchBlockLocationsRetryer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17835) Use CuratorCache implementation instead of PathChildrenCache / TreeCache
[
https://issues.apache.org/jira/browse/HADOOP-17835?focusedWorklogId=634668=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634668
]
ASF GitHub Bot logged work on HADOOP-17835:
---
Author: ASF GitHub Bot
Created on: 05/Aug/21 16:39
Start Date: 05/Aug/21 16:39
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3266:
URL: https://github.com/apache/hadoop/pull/3266#issuecomment-893603031
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 2s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 30s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 35s | | trunk passed |
| +1 :green_heart: | compile | 29m 54s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 24m 8s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 20s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 30s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 1s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 28s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 7s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 32s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 38s | | the patch passed |
| +1 :green_heart: | compile | 27m 27s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 27m 27s | |
root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 0 new + 1921 unchanged - 10
fixed = 1921 total (was 1931) |
| +1 :green_heart: | compile | 23m 49s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 23m 49s | |
root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 0 new + 1797 unchanged -
10 fixed = 1797 total (was 1807) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 19s | | hadoop-common-project:
The patch generated 0 new + 21 unchanged - 1 fixed = 21 total (was 22) |
| +1 :green_heart: | mvnsite | 2m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 42s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 30s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 42s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 19m 58s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 1m 16s | | hadoop-registry in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 48s | | The patch does not
generate ASF License warnings. |
| | | 237m 55s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3266/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3266 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux e743ffa71b4a 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9
06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ed4b013c5a91fe6b3cc983cbb75017e529f52b75 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
[GitHub] [hadoop] hadoop-yetus commented on pull request #3266: HADOOP-17835. Use CuratorCache implementation instead of PathChildrenCache / TreeCache
hadoop-yetus commented on pull request #3266:
URL: https://github.com/apache/hadoop/pull/3266#issuecomment-893603031
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 2s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 30s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 35s | | trunk passed |
| +1 :green_heart: | compile | 29m 54s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 24m 8s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 20s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 30s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 1s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 28s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 7s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 32s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 38s | | the patch passed |
| +1 :green_heart: | compile | 27m 27s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 27m 27s | |
root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 0 new + 1921 unchanged - 10
fixed = 1921 total (was 1931) |
| +1 :green_heart: | compile | 23m 49s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 23m 49s | |
root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 0 new + 1797 unchanged -
10 fixed = 1797 total (was 1807) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 19s | | hadoop-common-project:
The patch generated 0 new + 21 unchanged - 1 fixed = 21 total (was 22) |
| +1 :green_heart: | mvnsite | 2m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 42s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 30s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 42s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 19m 58s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 1m 16s | | hadoop-registry in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 48s | | The patch does not
generate ASF License warnings. |
| | | 237m 55s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3266/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3266 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux e743ffa71b4a 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9
06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ed4b013c5a91fe6b3cc983cbb75017e529f52b75 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3266/2/testReport/ |
| Max. process+thread count | 2901 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common
hadoop-common-project/hadoop-registry U:
[GitHub] [hadoop] hadoop-yetus commented on pull request #3270: HDFS-16154. TestMiniJournalCluster failing intermittently because of …
hadoop-yetus commented on pull request #3270:
URL: https://github.com/apache/hadoop/pull/3270#issuecomment-893598141
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 52s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 17s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 58s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 30s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 13s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 19s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 11s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 10s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 10s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 51s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 46s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 25s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 9s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 13s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 231m 29s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 315m 58s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3270/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3270 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux a3264e739081 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 193b21b7e2376b68f6fa151a2c6078f1dbde8e9a |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3270/1/testReport/ |
| Max. process+thread count | 3339 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3270/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Work logged] (HADOOP-17825) Add BuiltInGzipCompressor
[ https://issues.apache.org/jira/browse/HADOOP-17825?focusedWorklogId=634658=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634658 ] ASF GitHub Bot logged work on HADOOP-17825: --- Author: ASF GitHub Bot Created on: 05/Aug/21 16:31 Start Date: 05/Aug/21 16:31 Worklog Time Spent: 10m Work Description: sunchao commented on pull request #3250: URL: https://github.com/apache/hadoop/pull/3250#issuecomment-893597879 Re-triggered the CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 634658) Time Spent: 11h 50m (was: 11h 40m) > Add BuiltInGzipCompressor > - > > Key: HADOOP-17825 > URL: https://issues.apache.org/jira/browse/HADOOP-17825 > Project: Hadoop Common > Issue Type: Improvement >Reporter: L. C. Hsieh >Priority: Major > Labels: pull-request-available > Time Spent: 11h 50m > Remaining Estimate: 0h > > Currently, GzipCodec only supports BuiltInGzipDecompressor, if native zlib is > not loaded. So, without Hadoop native codec installed, saving SequenceFile > using GzipCodec will throw exception like "SequenceFile doesn't work with > GzipCodec without native-hadoop code!" > Same as other codecs which we migrated to using prepared packages (lz4, > snappy), it will be better if we support GzipCodec generally without Hadoop > native codec installed. Similar to BuiltInGzipDecompressor, we can use Java > Deflater to support BuiltInGzipCompressor. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on pull request #3250: HADOOP-17825. Add BuiltInGzipCompressor
sunchao commented on pull request #3250: URL: https://github.com/apache/hadoop/pull/3250#issuecomment-893597879 Re-triggered the CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17825) Add BuiltInGzipCompressor
[
https://issues.apache.org/jira/browse/HADOOP-17825?focusedWorklogId=634656=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634656
]
ASF GitHub Bot logged work on HADOOP-17825:
---
Author: ASF GitHub Bot
Created on: 05/Aug/21 16:30
Start Date: 05/Aug/21 16:30
Worklog Time Spent: 10m
Work Description: sunchao commented on a change in pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#discussion_r683610965
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress.zlib;
+
+import java.io.IOException;
+import java.util.zip.Checksum;
+import java.util.zip.Deflater;
+import java.util.zip.GZIPOutputStream;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.compress.Compressor;
+import org.apache.hadoop.io.compress.DoNotPool;
+import org.apache.hadoop.util.DataChecksum;
+
+/**
+ * A {@link Compressor} based on the popular gzip compressed file format.
+ * http://www.gzip.org/
+ */
+@DoNotPool
+public class BuiltInGzipCompressor implements Compressor {
+
+ /**
+ * Fixed ten-byte gzip header. See {@link GZIPOutputStream}'s source for
+ * details.
+ */
+ private static final byte[] GZIP_HEADER = new byte[]{
+ 0x1f, (byte) 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
+
+ // The trailer will be overwritten based on crc and output size.
+ private byte[] GZIP_TRAILER = new byte[]{0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00};
+
+ private final int GZIP_HEADER_LEN = GZIP_HEADER.length;
+ private final int GZIP_TRAILER_LEN = GZIP_TRAILER.length;
+
+ private Deflater deflater;
+
+ private int headerOff = 0;
+ private int trailerOff = 0;
+
+ private int numBytesWritten = 0;
+
+ private int currentBufLen = 0;
+
+ private final Checksum crc = DataChecksum.newCrc32();
+
+ private BuiltInGzipDecompressor.GzipStateLabel state;
+
+ public BuiltInGzipCompressor(Configuration conf) { init(conf); }
+
+ @Override
+ public boolean finished() {
+// Only if the trailer is also written, it is thought as finished.
+return deflater.finished() && state ==
BuiltInGzipDecompressor.GzipStateLabel.FINISHED;
+ }
+
+ @Override
+ public boolean needsInput() {
+if (state == BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM) {
+ return deflater.needsInput();
+}
+
+return (state != BuiltInGzipDecompressor.GzipStateLabel.FINISHED);
+ }
+
+ @Override
+ public int compress(byte[] b, int off, int len) throws IOException {
+int compressedBytesWritten = 0;
+
+// If we are not within uncompressed data yet, output the header.
+if (state != BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM &&
Review comment:
I think `compress` is always used in loops like this:
```java
while (!compresser.needsInput()) {
compresser.compress(..)
}
```
so if the state transitioned to `FINISHED`, we'd come out of the loop and
ask for more input to compress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 634656)
Time Spent: 11h 40m (was: 11.5h)
> Add BuiltInGzipCompressor
> -
>
> Key: HADOOP-17825
> URL: https://issues.apache.org/jira/browse/HADOOP-17825
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 11h 40m
> Remaining Estimate: 0h
>
> Currently, GzipCodec only supports BuiltInGzipDecompressor, if native zlib is
> not loaded. So, without Hadoop native codec installed,
[GitHub] [hadoop] sunchao commented on a change in pull request #3250: HADOOP-17825. Add BuiltInGzipCompressor
sunchao commented on a change in pull request #3250:
URL: https://github.com/apache/hadoop/pull/3250#discussion_r683610965
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.io.compress.zlib;
+
+import java.io.IOException;
+import java.util.zip.Checksum;
+import java.util.zip.Deflater;
+import java.util.zip.GZIPOutputStream;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.compress.Compressor;
+import org.apache.hadoop.io.compress.DoNotPool;
+import org.apache.hadoop.util.DataChecksum;
+
+/**
+ * A {@link Compressor} based on the popular gzip compressed file format.
+ * http://www.gzip.org/
+ */
+@DoNotPool
+public class BuiltInGzipCompressor implements Compressor {
+
+ /**
+ * Fixed ten-byte gzip header. See {@link GZIPOutputStream}'s source for
+ * details.
+ */
+ private static final byte[] GZIP_HEADER = new byte[]{
+ 0x1f, (byte) 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
+
+ // The trailer will be overwritten based on crc and output size.
+ private byte[] GZIP_TRAILER = new byte[]{0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00};
+
+ private final int GZIP_HEADER_LEN = GZIP_HEADER.length;
+ private final int GZIP_TRAILER_LEN = GZIP_TRAILER.length;
+
+ private Deflater deflater;
+
+ private int headerOff = 0;
+ private int trailerOff = 0;
+
+ private int numBytesWritten = 0;
+
+ private int currentBufLen = 0;
+
+ private final Checksum crc = DataChecksum.newCrc32();
+
+ private BuiltInGzipDecompressor.GzipStateLabel state;
+
+ public BuiltInGzipCompressor(Configuration conf) { init(conf); }
+
+ @Override
+ public boolean finished() {
+// Only if the trailer is also written, it is thought as finished.
+return deflater.finished() && state ==
BuiltInGzipDecompressor.GzipStateLabel.FINISHED;
+ }
+
+ @Override
+ public boolean needsInput() {
+if (state == BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM) {
+ return deflater.needsInput();
+}
+
+return (state != BuiltInGzipDecompressor.GzipStateLabel.FINISHED);
+ }
+
+ @Override
+ public int compress(byte[] b, int off, int len) throws IOException {
+int compressedBytesWritten = 0;
+
+// If we are not within uncompressed data yet, output the header.
+if (state != BuiltInGzipDecompressor.GzipStateLabel.INFLATE_STREAM &&
Review comment:
I think `compress` is always used in loops like this:
```java
while (!compresser.needsInput()) {
compresser.compress(..)
}
```
so if the state transitioned to `FINISHED`, we'd come out of the loop and
ask for more input to compress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17825) Add BuiltInGzipCompressor
[ https://issues.apache.org/jira/browse/HADOOP-17825?focusedWorklogId=634652=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634652 ] ASF GitHub Bot logged work on HADOOP-17825: --- Author: ASF GitHub Bot Created on: 05/Aug/21 16:25 Start Date: 05/Aug/21 16:25 Worklog Time Spent: 10m Work Description: viirya commented on pull request #3250: URL: https://github.com/apache/hadoop/pull/3250#issuecomment-893593265 @sunchao Could you trigger the CI again? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 634652) Time Spent: 11.5h (was: 11h 20m) > Add BuiltInGzipCompressor > - > > Key: HADOOP-17825 > URL: https://issues.apache.org/jira/browse/HADOOP-17825 > Project: Hadoop Common > Issue Type: Improvement >Reporter: L. C. Hsieh >Priority: Major > Labels: pull-request-available > Time Spent: 11.5h > Remaining Estimate: 0h > > Currently, GzipCodec only supports BuiltInGzipDecompressor, if native zlib is > not loaded. So, without Hadoop native codec installed, saving SequenceFile > using GzipCodec will throw exception like "SequenceFile doesn't work with > GzipCodec without native-hadoop code!" > Same as other codecs which we migrated to using prepared packages (lz4, > snappy), it will be better if we support GzipCodec generally without Hadoop > native codec installed. Similar to BuiltInGzipDecompressor, we can use Java > Deflater to support BuiltInGzipCompressor. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17825) Add BuiltInGzipCompressor
[ https://issues.apache.org/jira/browse/HADOOP-17825?focusedWorklogId=634650=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634650 ] ASF GitHub Bot logged work on HADOOP-17825: --- Author: ASF GitHub Bot Created on: 05/Aug/21 16:24 Start Date: 05/Aug/21 16:24 Worklog Time Spent: 10m Work Description: viirya commented on pull request #3250: URL: https://github.com/apache/hadoop/pull/3250#issuecomment-893592758 Last CI failure looks unrelated: ``` [ERROR] Failed to execute goal on project hadoop-yarn-common: Could not resolve dependencies for project org.apache.hadoop:hadoop-yarn-common:jar:3.4.0-SNAPSHOT: Failed to collect dependencies at com.sun.jersey:jersey-client:jar:1.19: Failed to read artifact descriptor for com.sun.jersey:jersey-client:jar:1.19: Could not transfer artifact com.sun.jersey:jersey-client:pom:1.19 from/to central (https://repo.maven.apache.org/maven2): Transfer failed for https://repo.maven.apache.org/maven2/com/sun/jersey/jersey-client/1.19/jersey-client-1.19.pom: Connection reset -> [Help 1] ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 634650) Time Spent: 11h 20m (was: 11h 10m) > Add BuiltInGzipCompressor > - > > Key: HADOOP-17825 > URL: https://issues.apache.org/jira/browse/HADOOP-17825 > Project: Hadoop Common > Issue Type: Improvement >Reporter: L. C. Hsieh >Priority: Major > Labels: pull-request-available > Time Spent: 11h 20m > Remaining Estimate: 0h > > Currently, GzipCodec only supports BuiltInGzipDecompressor, if native zlib is > not loaded. So, without Hadoop native codec installed, saving SequenceFile > using GzipCodec will throw exception like "SequenceFile doesn't work with > GzipCodec without native-hadoop code!" > Same as other codecs which we migrated to using prepared packages (lz4, > snappy), it will be better if we support GzipCodec generally without Hadoop > native codec installed. Similar to BuiltInGzipDecompressor, we can use Java > Deflater to support BuiltInGzipCompressor. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] viirya commented on pull request #3250: HADOOP-17825. Add BuiltInGzipCompressor
viirya commented on pull request #3250: URL: https://github.com/apache/hadoop/pull/3250#issuecomment-893593265 @sunchao Could you trigger the CI again? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] viirya commented on pull request #3250: HADOOP-17825. Add BuiltInGzipCompressor
viirya commented on pull request #3250: URL: https://github.com/apache/hadoop/pull/3250#issuecomment-893592758 Last CI failure looks unrelated: ``` [ERROR] Failed to execute goal on project hadoop-yarn-common: Could not resolve dependencies for project org.apache.hadoop:hadoop-yarn-common:jar:3.4.0-SNAPSHOT: Failed to collect dependencies at com.sun.jersey:jersey-client:jar:1.19: Failed to read artifact descriptor for com.sun.jersey:jersey-client:jar:1.19: Could not transfer artifact com.sun.jersey:jersey-client:pom:1.19 from/to central (https://repo.maven.apache.org/maven2): Transfer failed for https://repo.maven.apache.org/maven2/com/sun/jersey/jersey-client/1.19/jersey-client-1.19.pom: Connection reset -> [Help 1] ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3209: HDFS-16129. Fixing the signature secret file misusage in HttpFS.
hadoop-yetus commented on pull request #3209:
URL: https://github.com/apache/hadoop/pull/3209#issuecomment-893585180
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 43s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 7 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 27s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 25s | | trunk passed |
| +1 :green_heart: | compile | 23m 18s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 21m 4s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 4m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 16s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 31s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 51s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 23s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 34s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 52s | | the patch passed |
| +1 :green_heart: | compile | 23m 27s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 23m 27s | | the patch passed |
| +1 :green_heart: | compile | 20m 45s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 20m 45s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 4s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3209/12/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 95 unchanged - 0 fixed = 96 total (was
95) |
| +1 :green_heart: | mvnsite | 3m 9s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 2m 30s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 59s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 5m 5s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 25s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 3s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 43s | | hadoop-kms in the patch passed.
|
| +1 :green_heart: | unit | 6m 30s | | hadoop-hdfs-httpfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 56s | | The patch does not
generate ASF License warnings. |
| | | 226m 55s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3209/12/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3209 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux b4349a829922 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d2b5b76fe3bf242d8031489dbced91d7fa17307e |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3209/12/testReport/ |
| Max. process+thread count | 1251 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common
hadoop-common-project/hadoop-kms hadoop-hdfs-project/hadoop-hdfs-httpfs U:
[jira] [Updated] (HADOOP-17826) ABFS: Transient failure of TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting
[ https://issues.apache.org/jira/browse/HADOOP-17826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17826: Component/s: test > ABFS: Transient failure of > TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting > -- > > Key: HADOOP-17826 > URL: https://issues.apache.org/jira/browse/HADOOP-17826 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, test >Affects Versions: 3.4.0 >Reporter: Sumangala Patki >Priority: Major > > Transient failure of the below test observed for HNS OAuth, AppendBlob HNS > OAuth and Non-HNS SharedKey combinations. The value denoted by "actual value" > below varies across failures, and exceeds the upper limit of the expected > range. > _TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:171->fuzzyValidate:49 > The actual value 10 is not within the expected range: [5.60, 8.40]._ > Verified failure with client and server in the same region to rule out > network issues. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17826) ABFS: Transient failure of TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting
[ https://issues.apache.org/jira/browse/HADOOP-17826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394065#comment-17394065 ] Steve Loughran commented on HADOOP-17826: - is this a parallel run or sequential? some of the tests which measure time ranges fail when the scheduling of the test cases puts too big an interval for their assertions > ABFS: Transient failure of > TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting > -- > > Key: HADOOP-17826 > URL: https://issues.apache.org/jira/browse/HADOOP-17826 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, test >Affects Versions: 3.4.0 >Reporter: Sumangala Patki >Priority: Major > > Transient failure of the below test observed for HNS OAuth, AppendBlob HNS > OAuth and Non-HNS SharedKey combinations. The value denoted by "actual value" > below varies across failures, and exceeds the upper limit of the expected > range. > _TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:171->fuzzyValidate:49 > The actual value 10 is not within the expected range: [5.60, 8.40]._ > Verified failure with client and server in the same region to rule out > network issues. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sodonnel commented on a change in pull request #3269: HDFS-16153. Avoid evaluation of LOG.debug statement in QuorumJournalM…
sodonnel commented on a change in pull request #3269:
URL: https://github.com/apache/hadoop/pull/3269#discussion_r683499117
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java
##
@@ -618,8 +618,7 @@ private void selectStreamingInputStreams(
Map resps =
loggers.waitForWriteQuorum(q, selectInputStreamsTimeoutMs,
"selectStreamingInputStreams");
-
-LOG.debug("selectStreamingInputStream manifests:\n" +
+LOG.debug("selectStreamingInputStream manifests:\n {}",
Review comment:
This one still needs wrapped in `if (LOG.isDebugEnable)` otherwise the
`Joiner` will always be evaluated before the parameters are passed to the
logger.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17198) Support S3 Access Points
[
https://issues.apache.org/jira/browse/HADOOP-17198?focusedWorklogId=634590=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634590
]
ASF GitHub Bot logged work on HADOOP-17198:
---
Author: ASF GitHub Bot
Created on: 05/Aug/21 14:08
Start Date: 05/Aug/21 14:08
Worklog Time Spent: 10m
Work Description: mukund-thakur commented on a change in pull request
#3260:
URL: https://github.com/apache/hadoop/pull/3260#discussion_r683387906
##
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
##
@@ -1576,6 +1576,81 @@ Why explicitly declare a bucket bound to the central
endpoint? It ensures
that if the default endpoint is changed to a new region, data store in
US-east is still reachable.
+## Configuring S3 AccessPoints usage with S3a
Review comment:
nit: using S3A.
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/AbstractS3ACostTest.java
##
@@ -125,6 +126,13 @@ public AbstractS3ACostTest(
public Configuration createConfiguration() {
Configuration conf = super.createConfiguration();
String bucketName = getTestBucketName(conf);
+// If AccessPoint ARN is set guarded tests are skipped
+String apArn = conf.get(ACCESS_POINT_ARN, "");
+if (isGuarded() && !apArn.isEmpty()) {
+ LOG.warn("Skipping test since AccessPoint ARN is set and is incompatible
with S3Guard.");
+ Assume.assumeFalse(true);
Review comment:
nit: maybe put the warning in assumeFalse only.
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -400,6 +410,14 @@ public void initialize(URI name, Configuration
originalConf)
LOG.debug("Initializing S3AFileSystem for {}", bucket);
// clone the configuration into one with propagated bucket options
Configuration conf = propagateBucketOptions(originalConf, bucket);
+
+ String apArn = conf.getTrimmed(ACCESS_POINT_ARN, "");
Review comment:
Yes use ACCESS_POINT_ARN_DEFAULT as suggested by steve above as well.
##
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
##
@@ -1576,6 +1576,81 @@ Why explicitly declare a bucket bound to the central
endpoint? It ensures
that if the default endpoint is changed to a new region, data store in
US-east is still reachable.
+## Configuring S3 AccessPoints usage with S3a
+S3a now supports [S3 Access
Point](https://aws.amazon.com/s3/features/access-points/) usage which
+improves VPC integration with S3 and simplifies your data's permission model
because different
+policies can be applied now on the Access Point level. For more information
about why to use them
+make sure to read the official documentation.
+
+Accessing data through an access point, is done by using its ARN, as opposed
to just the bucket name.
+You can set the Access Point ARN property using the following configuration
property:
+```xml
+
+fs.s3a.accesspoint.arn
+ {ACCESSPOINT_ARN_HERE}
+Configure S3a traffic to use this AccessPoint
+
+```
+
+Be mindful that this configures **all access** to S3a, and in turn S3, to go
through that ARN.
+So for example `s3a://yourbucket/key` will now use your configured ARN when
getting data from S3
+instead of your bucket. The flip side to this is that if you're working with
multiple buckets
+`s3a://yourbucket` and `s3a://yourotherbucket` both of their requests will go
through the same
+Access Point ARN. To configure different Access Point ARNs, per bucket
overrides can be used with
+access point names instead of bucket names as such:
+
+- Let's assume you have an existing workflow with the following paths
`s3a://data-bucket`,
+`s3a://output-bucket` and you want to work with a new Access Point called
`finance-accesspoint`. All
+you would then need to add is the following per bucket configuration change:
+```xml
+
+fs.s3a.bucket.finance-accesspoint.accesspoint.arn
+ arn:aws:s3:eu-west-1:123456789101:accesspoint/finance-accesspoint
+
+```
+
+While keeping the global `accesspoint.arn` property set to empty `" "` which
is the default.
Review comment:
I think we discussed it to be "" above so the docs should be consistent
as well.
##
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
##
@@ -1576,6 +1576,81 @@ Why explicitly declare a bucket bound to the central
endpoint? It ensures
that if the default endpoint is changed to a new region, data store in
US-east is still reachable.
+## Configuring S3 AccessPoints usage with S3a
+S3a now supports [S3 Access
Point](https://aws.amazon.com/s3/features/access-points/) usage which
+improves VPC integration with S3 and simplifies your data's permission model
because different
+policies can be applied now on the Access Point
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #3260: HADOOP-17198 Support S3 AccessPoint
mukund-thakur commented on a change in pull request #3260:
URL: https://github.com/apache/hadoop/pull/3260#discussion_r683387906
##
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
##
@@ -1576,6 +1576,81 @@ Why explicitly declare a bucket bound to the central
endpoint? It ensures
that if the default endpoint is changed to a new region, data store in
US-east is still reachable.
+## Configuring S3 AccessPoints usage with S3a
Review comment:
nit: using S3A.
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/AbstractS3ACostTest.java
##
@@ -125,6 +126,13 @@ public AbstractS3ACostTest(
public Configuration createConfiguration() {
Configuration conf = super.createConfiguration();
String bucketName = getTestBucketName(conf);
+// If AccessPoint ARN is set guarded tests are skipped
+String apArn = conf.get(ACCESS_POINT_ARN, "");
+if (isGuarded() && !apArn.isEmpty()) {
+ LOG.warn("Skipping test since AccessPoint ARN is set and is incompatible
with S3Guard.");
+ Assume.assumeFalse(true);
Review comment:
nit: maybe put the warning in assumeFalse only.
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -400,6 +410,14 @@ public void initialize(URI name, Configuration
originalConf)
LOG.debug("Initializing S3AFileSystem for {}", bucket);
// clone the configuration into one with propagated bucket options
Configuration conf = propagateBucketOptions(originalConf, bucket);
+
+ String apArn = conf.getTrimmed(ACCESS_POINT_ARN, "");
Review comment:
Yes use ACCESS_POINT_ARN_DEFAULT as suggested by steve above as well.
##
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
##
@@ -1576,6 +1576,81 @@ Why explicitly declare a bucket bound to the central
endpoint? It ensures
that if the default endpoint is changed to a new region, data store in
US-east is still reachable.
+## Configuring S3 AccessPoints usage with S3a
+S3a now supports [S3 Access
Point](https://aws.amazon.com/s3/features/access-points/) usage which
+improves VPC integration with S3 and simplifies your data's permission model
because different
+policies can be applied now on the Access Point level. For more information
about why to use them
+make sure to read the official documentation.
+
+Accessing data through an access point, is done by using its ARN, as opposed
to just the bucket name.
+You can set the Access Point ARN property using the following configuration
property:
+```xml
+
+fs.s3a.accesspoint.arn
+ {ACCESSPOINT_ARN_HERE}
+Configure S3a traffic to use this AccessPoint
+
+```
+
+Be mindful that this configures **all access** to S3a, and in turn S3, to go
through that ARN.
+So for example `s3a://yourbucket/key` will now use your configured ARN when
getting data from S3
+instead of your bucket. The flip side to this is that if you're working with
multiple buckets
+`s3a://yourbucket` and `s3a://yourotherbucket` both of their requests will go
through the same
+Access Point ARN. To configure different Access Point ARNs, per bucket
overrides can be used with
+access point names instead of bucket names as such:
+
+- Let's assume you have an existing workflow with the following paths
`s3a://data-bucket`,
+`s3a://output-bucket` and you want to work with a new Access Point called
`finance-accesspoint`. All
+you would then need to add is the following per bucket configuration change:
+```xml
+
+fs.s3a.bucket.finance-accesspoint.accesspoint.arn
+ arn:aws:s3:eu-west-1:123456789101:accesspoint/finance-accesspoint
+
+```
+
+While keeping the global `accesspoint.arn` property set to empty `" "` which
is the default.
Review comment:
I think we discussed it to be "" above so the docs should be consistent
as well.
##
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
##
@@ -1576,6 +1576,81 @@ Why explicitly declare a bucket bound to the central
endpoint? It ensures
that if the default endpoint is changed to a new region, data store in
US-east is still reachable.
+## Configuring S3 AccessPoints usage with S3a
+S3a now supports [S3 Access
Point](https://aws.amazon.com/s3/features/access-points/) usage which
+improves VPC integration with S3 and simplifies your data's permission model
because different
+policies can be applied now on the Access Point level. For more information
about why to use them
+make sure to read the official documentation.
+
+Accessing data through an access point, is done by using its ARN, as opposed
to just the bucket name.
+You can set the Access Point ARN property using the following configuration
property:
+```xml
+
+fs.s3a.accesspoint.arn
+ {ACCESSPOINT_ARN_HERE}
+Configure S3a traffic to use this AccessPoint
+
+```
+
+Be
[GitHub] [hadoop] hadoop-yetus commented on pull request #3269: HDFS-16153. Avoid evaluation of LOG.debug statement in QuorumJournalM…
hadoop-yetus commented on pull request #3269:
URL: https://github.com/apache/hadoop/pull/3269#issuecomment-893465532
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 59s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 10s | | trunk passed |
| +1 :green_heart: | compile | 1m 32s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 21s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 6s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 53s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 27s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 21s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 30s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 11s | | the patch passed |
| +1 :green_heart: | compile | 1m 16s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 16s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 51s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 16s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 47s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 18s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 8s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 13s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 211m 9s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3269/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 40s | | The patch does not
generate ASF License warnings. |
| | | 299m 33s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestPendingDataNodeMessages |
| | hadoop.hdfs.server.balancer.TestBalancerRPCDelay |
| | hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer |
| | hadoop.hdfs.TestViewDistributedFileSystemContract |
| |
hadoop.hdfs.server.blockmanagement.TestAvailableSpaceRackFaultTolerantBPP |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3269/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3269 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux b0c43234b0c1 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 99be40c1a221c831891b7d9539adf4a08840672e |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
[jira] [Work logged] (HADOOP-17835) Use CuratorCache implementation instead of PathChildrenCache / TreeCache
[ https://issues.apache.org/jira/browse/HADOOP-17835?focusedWorklogId=634542=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634542 ] ASF GitHub Bot logged work on HADOOP-17835: --- Author: ASF GitHub Bot Created on: 05/Aug/21 12:45 Start Date: 05/Aug/21 12:45 Worklog Time Spent: 10m Work Description: virajjasani commented on pull request #3266: URL: https://github.com/apache/hadoop/pull/3266#issuecomment-893429450 @Randgalt Thanks for the review. Addressed all comments. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 634542) Time Spent: 2.5h (was: 2h 20m) > Use CuratorCache implementation instead of PathChildrenCache / TreeCache > > > Key: HADOOP-17835 > URL: https://issues.apache.org/jira/browse/HADOOP-17835 > Project: Hadoop Common > Issue Type: Task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > As we have moved to Curator 5.2.0 for Hadoop 3.4.0, we should start using new > CuratorCache service implementation in place of deprecated PathChildrenCache > and TreeCache usecases. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17835) Use CuratorCache implementation instead of PathChildrenCache / TreeCache
[
https://issues.apache.org/jira/browse/HADOOP-17835?focusedWorklogId=634540=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634540
]
ASF GitHub Bot logged work on HADOOP-17835:
---
Author: ASF GitHub Bot
Created on: 05/Aug/21 12:45
Start Date: 05/Aug/21 12:45
Worklog Time Spent: 10m
Work Description: virajjasani commented on a change in pull request #3266:
URL: https://github.com/apache/hadoop/pull/3266#discussion_r683417575
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java
##
@@ -363,105 +357,122 @@ public void startThreads() throws IOException {
throw new RuntimeException("Could not create ZK paths");
}
try {
- keyCache = new PathChildrenCache(zkClient, ZK_DTSM_MASTER_KEY_ROOT,
true);
+ keyCache = CuratorCache.bridgeBuilder(zkClient, ZK_DTSM_MASTER_KEY_ROOT)
+ .build();
if (keyCache != null) {
-keyCache.start(StartMode.BUILD_INITIAL_CACHE);
-keyCache.getListenable().addListener(new PathChildrenCacheListener() {
- @Override
- public void childEvent(CuratorFramework client,
- PathChildrenCacheEvent event)
- throws Exception {
-switch (event.getType()) {
-case CHILD_ADDED:
- processKeyAddOrUpdate(event.getData().getData());
- break;
-case CHILD_UPDATED:
- processKeyAddOrUpdate(event.getData().getData());
- break;
-case CHILD_REMOVED:
- processKeyRemoved(event.getData().getPath());
- break;
-default:
- break;
-}
- }
-}, listenerThreadPool);
+CuratorCacheListener keyCacheListener = CuratorCacheListener.builder()
+.forCreates(childData -> {
+ try {
+processKeyAddOrUpdate(childData.getData());
+ } catch (IOException e) {
+LOG.error("Error while processing Curator keyCacheListener "
++ "NODE_CREATED event");
+throw new UncheckedIOException(e);
+ }
+})
+.forChanges((oldNode, node) -> {
+ try {
+processKeyAddOrUpdate(node.getData());
+ } catch (IOException e) {
+LOG.error("Error while processing Curator keyCacheListener "
++ "NODE_CHANGED event");
+throw new UncheckedIOException(e);
+ }
+})
+.forDeletes(childData -> processKeyRemoved(childData.getPath()))
+.build();
+keyCache.listenable().addListener(keyCacheListener);
+keyCache.start();
loadFromZKCache(false);
}
} catch (Exception e) {
- throw new IOException("Could not start PathChildrenCache for keys", e);
+ throw new IOException("Could not start Curator keyCacheListener for
keys",
+ e);
}
if (isTokenWatcherEnabled) {
LOG.info("TokenCache is enabled");
try {
-tokenCache = new PathChildrenCache(zkClient, ZK_DTSM_TOKENS_ROOT,
true);
+tokenCache = CuratorCache.bridgeBuilder(zkClient, ZK_DTSM_TOKENS_ROOT)
+.build();
if (tokenCache != null) {
- tokenCache.start(StartMode.BUILD_INITIAL_CACHE);
- tokenCache.getListenable().addListener(new
PathChildrenCacheListener() {
-
-@Override
-public void childEvent(CuratorFramework client,
- PathChildrenCacheEvent event) throws
Exception {
- switch (event.getType()) {
-case CHILD_ADDED:
- processTokenAddOrUpdate(event.getData().getData());
- break;
-case CHILD_UPDATED:
- processTokenAddOrUpdate(event.getData().getData());
- break;
-case CHILD_REMOVED:
- processTokenRemoved(event.getData());
- break;
-default:
- break;
- }
-}
- }, listenerThreadPool);
+ CuratorCacheListener tokenCacheListener =
CuratorCacheListener.builder()
+ .forCreates(childData -> {
+try {
+ processTokenAddOrUpdate(childData.getData());
+} catch (IOException e) {
+ LOG.error("Error while processing Curator tokenCacheListener
"
+ + "NODE_CREATED event");
+ throw new UncheckedIOException(e);
+}
+ })
+ .forChanges((oldNode, node) -> {
+try {
+ processTokenAddOrUpdate(node.getData());
+} catch (IOException e) {
+
[GitHub] [hadoop] virajjasani commented on pull request #3266: HADOOP-17835. Use CuratorCache implementation instead of PathChildrenCache / TreeCache
virajjasani commented on pull request #3266: URL: https://github.com/apache/hadoop/pull/3266#issuecomment-893429450 @Randgalt Thanks for the review. Addressed all comments. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a change in pull request #3266: HADOOP-17835. Use CuratorCache implementation instead of PathChildrenCache / TreeCache
virajjasani commented on a change in pull request #3266:
URL: https://github.com/apache/hadoop/pull/3266#discussion_r683417575
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java
##
@@ -363,105 +357,122 @@ public void startThreads() throws IOException {
throw new RuntimeException("Could not create ZK paths");
}
try {
- keyCache = new PathChildrenCache(zkClient, ZK_DTSM_MASTER_KEY_ROOT,
true);
+ keyCache = CuratorCache.bridgeBuilder(zkClient, ZK_DTSM_MASTER_KEY_ROOT)
+ .build();
if (keyCache != null) {
-keyCache.start(StartMode.BUILD_INITIAL_CACHE);
-keyCache.getListenable().addListener(new PathChildrenCacheListener() {
- @Override
- public void childEvent(CuratorFramework client,
- PathChildrenCacheEvent event)
- throws Exception {
-switch (event.getType()) {
-case CHILD_ADDED:
- processKeyAddOrUpdate(event.getData().getData());
- break;
-case CHILD_UPDATED:
- processKeyAddOrUpdate(event.getData().getData());
- break;
-case CHILD_REMOVED:
- processKeyRemoved(event.getData().getPath());
- break;
-default:
- break;
-}
- }
-}, listenerThreadPool);
+CuratorCacheListener keyCacheListener = CuratorCacheListener.builder()
+.forCreates(childData -> {
+ try {
+processKeyAddOrUpdate(childData.getData());
+ } catch (IOException e) {
+LOG.error("Error while processing Curator keyCacheListener "
++ "NODE_CREATED event");
+throw new UncheckedIOException(e);
+ }
+})
+.forChanges((oldNode, node) -> {
+ try {
+processKeyAddOrUpdate(node.getData());
+ } catch (IOException e) {
+LOG.error("Error while processing Curator keyCacheListener "
++ "NODE_CHANGED event");
+throw new UncheckedIOException(e);
+ }
+})
+.forDeletes(childData -> processKeyRemoved(childData.getPath()))
+.build();
+keyCache.listenable().addListener(keyCacheListener);
+keyCache.start();
loadFromZKCache(false);
}
} catch (Exception e) {
- throw new IOException("Could not start PathChildrenCache for keys", e);
+ throw new IOException("Could not start Curator keyCacheListener for
keys",
+ e);
}
if (isTokenWatcherEnabled) {
LOG.info("TokenCache is enabled");
try {
-tokenCache = new PathChildrenCache(zkClient, ZK_DTSM_TOKENS_ROOT,
true);
+tokenCache = CuratorCache.bridgeBuilder(zkClient, ZK_DTSM_TOKENS_ROOT)
+.build();
if (tokenCache != null) {
- tokenCache.start(StartMode.BUILD_INITIAL_CACHE);
- tokenCache.getListenable().addListener(new
PathChildrenCacheListener() {
-
-@Override
-public void childEvent(CuratorFramework client,
- PathChildrenCacheEvent event) throws
Exception {
- switch (event.getType()) {
-case CHILD_ADDED:
- processTokenAddOrUpdate(event.getData().getData());
- break;
-case CHILD_UPDATED:
- processTokenAddOrUpdate(event.getData().getData());
- break;
-case CHILD_REMOVED:
- processTokenRemoved(event.getData());
- break;
-default:
- break;
- }
-}
- }, listenerThreadPool);
+ CuratorCacheListener tokenCacheListener =
CuratorCacheListener.builder()
+ .forCreates(childData -> {
+try {
+ processTokenAddOrUpdate(childData.getData());
+} catch (IOException e) {
+ LOG.error("Error while processing Curator tokenCacheListener
"
+ + "NODE_CREATED event");
+ throw new UncheckedIOException(e);
+}
+ })
+ .forChanges((oldNode, node) -> {
+try {
+ processTokenAddOrUpdate(node.getData());
+} catch (IOException e) {
+ LOG.error("Error while processing Curator tokenCacheListener
"
+ + "NODE_CHANGED event");
+ throw new UncheckedIOException(e);
+}
+ })
+ .forDeletes(childData -> {
+try {
+ processTokenRemoved(childData);
+} catch (IOException e) {
+ LOG.error("Error while processing Curator
[jira] [Work logged] (HADOOP-17835) Use CuratorCache implementation instead of PathChildrenCache / TreeCache
[
https://issues.apache.org/jira/browse/HADOOP-17835?focusedWorklogId=634535=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634535
]
ASF GitHub Bot logged work on HADOOP-17835:
---
Author: ASF GitHub Bot
Created on: 05/Aug/21 12:41
Start Date: 05/Aug/21 12:41
Worklog Time Spent: 10m
Work Description: virajjasani commented on a change in pull request #3266:
URL: https://github.com/apache/hadoop/pull/3266#discussion_r683414348
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java
##
@@ -363,105 +357,122 @@ public void startThreads() throws IOException {
throw new RuntimeException("Could not create ZK paths");
}
try {
- keyCache = new PathChildrenCache(zkClient, ZK_DTSM_MASTER_KEY_ROOT,
true);
+ keyCache = CuratorCache.bridgeBuilder(zkClient, ZK_DTSM_MASTER_KEY_ROOT)
+ .build();
if (keyCache != null) {
Review comment:
Yeah this was not applicable even before with PathChildrenCache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 634535)
Time Spent: 2h 10m (was: 2h)
> Use CuratorCache implementation instead of PathChildrenCache / TreeCache
>
>
> Key: HADOOP-17835
> URL: https://issues.apache.org/jira/browse/HADOOP-17835
> Project: Hadoop Common
> Issue Type: Task
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> As we have moved to Curator 5.2.0 for Hadoop 3.4.0, we should start using new
> CuratorCache service implementation in place of deprecated PathChildrenCache
> and TreeCache usecases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a change in pull request #3266: HADOOP-17835. Use CuratorCache implementation instead of PathChildrenCache / TreeCache
virajjasani commented on a change in pull request #3266:
URL: https://github.com/apache/hadoop/pull/3266#discussion_r683414348
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java
##
@@ -363,105 +357,122 @@ public void startThreads() throws IOException {
throw new RuntimeException("Could not create ZK paths");
}
try {
- keyCache = new PathChildrenCache(zkClient, ZK_DTSM_MASTER_KEY_ROOT,
true);
+ keyCache = CuratorCache.bridgeBuilder(zkClient, ZK_DTSM_MASTER_KEY_ROOT)
+ .build();
if (keyCache != null) {
Review comment:
Yeah this was not applicable even before with PathChildrenCache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17816) Run optional CI for changes in C
[ https://issues.apache.org/jira/browse/HADOOP-17816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph resolved HADOOP-17816. Fix Version/s: 3.4.0 Resolution: Fixed > Run optional CI for changes in C > > > Key: HADOOP-17816 > URL: https://issues.apache.org/jira/browse/HADOOP-17816 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.4.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Critical > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > We need to ensure that we run the CI for all the platforms when there are > changes in C files. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] wzhallright commented on a change in pull request #3269: HDFS-16153. Avoid evaluation of LOG.debug statement in QuorumJournalM…
wzhallright commented on a change in pull request #3269:
URL: https://github.com/apache/hadoop/pull/3269#discussion_r683403944
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java
##
@@ -585,8 +585,10 @@ private void
selectRpcInputStreams(Collection streams,
int maxAllowedTxns = !onlyDurableTxns ? highestTxnCount :
responseCounts.get(responseCounts.size() - loggers.getMajoritySize());
if (maxAllowedTxns == 0) {
- LOG.debug("No new edits available in logs; requested starting from " +
- "ID " + fromTxnId);
+ if (LOG.isDebugEnabled()) {
+LOG.debug("No new edits available in logs; requested starting from " +
+"ID " + fromTxnId);
+ }
Review comment:
Thanks for review @sodonnel ,I will change it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17835) Use CuratorCache implementation instead of PathChildrenCache / TreeCache
[
https://issues.apache.org/jira/browse/HADOOP-17835?focusedWorklogId=634526=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634526
]
ASF GitHub Bot logged work on HADOOP-17835:
---
Author: ASF GitHub Bot
Created on: 05/Aug/21 12:25
Start Date: 05/Aug/21 12:25
Worklog Time Spent: 10m
Work Description: Randgalt commented on a change in pull request #3266:
URL: https://github.com/apache/hadoop/pull/3266#discussion_r683403623
##
File path:
hadoop-common-project/hadoop-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/CuratorService.java
##
@@ -824,73 +825,65 @@ protected String dumpRegistryRobustly(boolean verbose) {
*
* @param listener the listener.
* @return a handle allowing for the management of the listener.
- * @throws Exception if registration fails due to error.
*/
- public ListenerHandle registerPathListener(final PathListener listener)
- throws Exception {
-
-final TreeCacheListener pathChildrenCacheListener =
-new TreeCacheListener() {
-
- public void childEvent(CuratorFramework curatorFramework,
- TreeCacheEvent event)
- throws Exception {
-String path = null;
-if (event != null && event.getData() != null) {
- path = event.getData().getPath();
-}
-assert event != null;
-switch (event.getType()) {
-case NODE_ADDED:
- LOG.info("Informing listener of added node {}", path);
- listener.nodeAdded(path);
-
- break;
-
-case NODE_REMOVED:
- LOG.info("Informing listener of removed node {}", path);
- listener.nodeRemoved(path);
-
- break;
-
-case NODE_UPDATED:
- LOG.info("Informing listener of updated node {}", path);
- listener.nodeAdded(path);
-
- break;
-
-default:
- // do nothing
- break;
-
-}
+ public ListenerHandle registerPathListener(final PathListener listener) {
+
+CuratorCacheListener cacheListener = CuratorCacheListener.builder()
+.forCreates(childData -> {
Review comment:
Note: there is a `forCreatesAndChanges()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 634526)
Time Spent: 2h (was: 1h 50m)
> Use CuratorCache implementation instead of PathChildrenCache / TreeCache
>
>
> Key: HADOOP-17835
> URL: https://issues.apache.org/jira/browse/HADOOP-17835
> Project: Hadoop Common
> Issue Type: Task
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> As we have moved to Curator 5.2.0 for Hadoop 3.4.0, we should start using new
> CuratorCache service implementation in place of deprecated PathChildrenCache
> and TreeCache usecases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Randgalt commented on a change in pull request #3266: HADOOP-17835. Use CuratorCache implementation instead of PathChildrenCache / TreeCache
Randgalt commented on a change in pull request #3266:
URL: https://github.com/apache/hadoop/pull/3266#discussion_r683403623
##
File path:
hadoop-common-project/hadoop-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/CuratorService.java
##
@@ -824,73 +825,65 @@ protected String dumpRegistryRobustly(boolean verbose) {
*
* @param listener the listener.
* @return a handle allowing for the management of the listener.
- * @throws Exception if registration fails due to error.
*/
- public ListenerHandle registerPathListener(final PathListener listener)
- throws Exception {
-
-final TreeCacheListener pathChildrenCacheListener =
-new TreeCacheListener() {
-
- public void childEvent(CuratorFramework curatorFramework,
- TreeCacheEvent event)
- throws Exception {
-String path = null;
-if (event != null && event.getData() != null) {
- path = event.getData().getPath();
-}
-assert event != null;
-switch (event.getType()) {
-case NODE_ADDED:
- LOG.info("Informing listener of added node {}", path);
- listener.nodeAdded(path);
-
- break;
-
-case NODE_REMOVED:
- LOG.info("Informing listener of removed node {}", path);
- listener.nodeRemoved(path);
-
- break;
-
-case NODE_UPDATED:
- LOG.info("Informing listener of updated node {}", path);
- listener.nodeAdded(path);
-
- break;
-
-default:
- // do nothing
- break;
-
-}
+ public ListenerHandle registerPathListener(final PathListener listener) {
+
+CuratorCacheListener cacheListener = CuratorCacheListener.builder()
+.forCreates(childData -> {
Review comment:
Note: there is a `forCreatesAndChanges()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17835) Use CuratorCache implementation instead of PathChildrenCache / TreeCache
[
https://issues.apache.org/jira/browse/HADOOP-17835?focusedWorklogId=634523=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634523
]
ASF GitHub Bot logged work on HADOOP-17835:
---
Author: ASF GitHub Bot
Created on: 05/Aug/21 12:24
Start Date: 05/Aug/21 12:24
Worklog Time Spent: 10m
Work Description: Randgalt commented on a change in pull request #3266:
URL: https://github.com/apache/hadoop/pull/3266#discussion_r683402550
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java
##
@@ -363,105 +357,122 @@ public void startThreads() throws IOException {
throw new RuntimeException("Could not create ZK paths");
}
try {
- keyCache = new PathChildrenCache(zkClient, ZK_DTSM_MASTER_KEY_ROOT,
true);
+ keyCache = CuratorCache.bridgeBuilder(zkClient, ZK_DTSM_MASTER_KEY_ROOT)
+ .build();
if (keyCache != null) {
-keyCache.start(StartMode.BUILD_INITIAL_CACHE);
-keyCache.getListenable().addListener(new PathChildrenCacheListener() {
- @Override
- public void childEvent(CuratorFramework client,
- PathChildrenCacheEvent event)
- throws Exception {
-switch (event.getType()) {
-case CHILD_ADDED:
- processKeyAddOrUpdate(event.getData().getData());
- break;
-case CHILD_UPDATED:
- processKeyAddOrUpdate(event.getData().getData());
- break;
-case CHILD_REMOVED:
- processKeyRemoved(event.getData().getPath());
- break;
-default:
- break;
-}
- }
-}, listenerThreadPool);
+CuratorCacheListener keyCacheListener = CuratorCacheListener.builder()
+.forCreates(childData -> {
+ try {
+processKeyAddOrUpdate(childData.getData());
+ } catch (IOException e) {
+LOG.error("Error while processing Curator keyCacheListener "
++ "NODE_CREATED event");
+throw new UncheckedIOException(e);
+ }
+})
+.forChanges((oldNode, node) -> {
+ try {
+processKeyAddOrUpdate(node.getData());
+ } catch (IOException e) {
+LOG.error("Error while processing Curator keyCacheListener "
++ "NODE_CHANGED event");
+throw new UncheckedIOException(e);
+ }
+})
+.forDeletes(childData -> processKeyRemoved(childData.getPath()))
+.build();
+keyCache.listenable().addListener(keyCacheListener);
+keyCache.start();
loadFromZKCache(false);
}
} catch (Exception e) {
- throw new IOException("Could not start PathChildrenCache for keys", e);
+ throw new IOException("Could not start Curator keyCacheListener for
keys",
+ e);
}
if (isTokenWatcherEnabled) {
LOG.info("TokenCache is enabled");
try {
-tokenCache = new PathChildrenCache(zkClient, ZK_DTSM_TOKENS_ROOT,
true);
+tokenCache = CuratorCache.bridgeBuilder(zkClient, ZK_DTSM_TOKENS_ROOT)
+.build();
if (tokenCache != null) {
- tokenCache.start(StartMode.BUILD_INITIAL_CACHE);
- tokenCache.getListenable().addListener(new
PathChildrenCacheListener() {
-
-@Override
-public void childEvent(CuratorFramework client,
- PathChildrenCacheEvent event) throws
Exception {
- switch (event.getType()) {
-case CHILD_ADDED:
- processTokenAddOrUpdate(event.getData().getData());
- break;
-case CHILD_UPDATED:
- processTokenAddOrUpdate(event.getData().getData());
- break;
-case CHILD_REMOVED:
- processTokenRemoved(event.getData());
- break;
-default:
- break;
- }
-}
- }, listenerThreadPool);
+ CuratorCacheListener tokenCacheListener =
CuratorCacheListener.builder()
+ .forCreates(childData -> {
Review comment:
Note: there is a `forCreatesAndChanges()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 634523)
Time Spent: 1h 50m (was: 1h 40m)
> Use CuratorCache
[GitHub] [hadoop] jianghuazhu commented on pull request #3256: HDFS-16151.Improve the parameter comments related to ProtobufRpcEngine2#Server().
jianghuazhu commented on pull request #3256: URL: https://github.com/apache/hadoop/pull/3256#issuecomment-893416848 @jojochuang , can you help re-execute CI/CD? I don’t have permission at the moment. thank you very much. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Randgalt commented on a change in pull request #3266: HADOOP-17835. Use CuratorCache implementation instead of PathChildrenCache / TreeCache
Randgalt commented on a change in pull request #3266:
URL: https://github.com/apache/hadoop/pull/3266#discussion_r683402550
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java
##
@@ -363,105 +357,122 @@ public void startThreads() throws IOException {
throw new RuntimeException("Could not create ZK paths");
}
try {
- keyCache = new PathChildrenCache(zkClient, ZK_DTSM_MASTER_KEY_ROOT,
true);
+ keyCache = CuratorCache.bridgeBuilder(zkClient, ZK_DTSM_MASTER_KEY_ROOT)
+ .build();
if (keyCache != null) {
-keyCache.start(StartMode.BUILD_INITIAL_CACHE);
-keyCache.getListenable().addListener(new PathChildrenCacheListener() {
- @Override
- public void childEvent(CuratorFramework client,
- PathChildrenCacheEvent event)
- throws Exception {
-switch (event.getType()) {
-case CHILD_ADDED:
- processKeyAddOrUpdate(event.getData().getData());
- break;
-case CHILD_UPDATED:
- processKeyAddOrUpdate(event.getData().getData());
- break;
-case CHILD_REMOVED:
- processKeyRemoved(event.getData().getPath());
- break;
-default:
- break;
-}
- }
-}, listenerThreadPool);
+CuratorCacheListener keyCacheListener = CuratorCacheListener.builder()
+.forCreates(childData -> {
+ try {
+processKeyAddOrUpdate(childData.getData());
+ } catch (IOException e) {
+LOG.error("Error while processing Curator keyCacheListener "
++ "NODE_CREATED event");
+throw new UncheckedIOException(e);
+ }
+})
+.forChanges((oldNode, node) -> {
+ try {
+processKeyAddOrUpdate(node.getData());
+ } catch (IOException e) {
+LOG.error("Error while processing Curator keyCacheListener "
++ "NODE_CHANGED event");
+throw new UncheckedIOException(e);
+ }
+})
+.forDeletes(childData -> processKeyRemoved(childData.getPath()))
+.build();
+keyCache.listenable().addListener(keyCacheListener);
+keyCache.start();
loadFromZKCache(false);
}
} catch (Exception e) {
- throw new IOException("Could not start PathChildrenCache for keys", e);
+ throw new IOException("Could not start Curator keyCacheListener for
keys",
+ e);
}
if (isTokenWatcherEnabled) {
LOG.info("TokenCache is enabled");
try {
-tokenCache = new PathChildrenCache(zkClient, ZK_DTSM_TOKENS_ROOT,
true);
+tokenCache = CuratorCache.bridgeBuilder(zkClient, ZK_DTSM_TOKENS_ROOT)
+.build();
if (tokenCache != null) {
- tokenCache.start(StartMode.BUILD_INITIAL_CACHE);
- tokenCache.getListenable().addListener(new
PathChildrenCacheListener() {
-
-@Override
-public void childEvent(CuratorFramework client,
- PathChildrenCacheEvent event) throws
Exception {
- switch (event.getType()) {
-case CHILD_ADDED:
- processTokenAddOrUpdate(event.getData().getData());
- break;
-case CHILD_UPDATED:
- processTokenAddOrUpdate(event.getData().getData());
- break;
-case CHILD_REMOVED:
- processTokenRemoved(event.getData());
- break;
-default:
- break;
- }
-}
- }, listenerThreadPool);
+ CuratorCacheListener tokenCacheListener =
CuratorCacheListener.builder()
+ .forCreates(childData -> {
Review comment:
Note: there is a `forCreatesAndChanges()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17835) Use CuratorCache implementation instead of PathChildrenCache / TreeCache
[
https://issues.apache.org/jira/browse/HADOOP-17835?focusedWorklogId=634521=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634521
]
ASF GitHub Bot logged work on HADOOP-17835:
---
Author: ASF GitHub Bot
Created on: 05/Aug/21 12:21
Start Date: 05/Aug/21 12:21
Worklog Time Spent: 10m
Work Description: Randgalt commented on a change in pull request #3266:
URL: https://github.com/apache/hadoop/pull/3266#discussion_r683400351
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java
##
@@ -363,105 +357,122 @@ public void startThreads() throws IOException {
throw new RuntimeException("Could not create ZK paths");
}
try {
- keyCache = new PathChildrenCache(zkClient, ZK_DTSM_MASTER_KEY_ROOT,
true);
+ keyCache = CuratorCache.bridgeBuilder(zkClient, ZK_DTSM_MASTER_KEY_ROOT)
+ .build();
if (keyCache != null) {
-keyCache.start(StartMode.BUILD_INITIAL_CACHE);
-keyCache.getListenable().addListener(new PathChildrenCacheListener() {
- @Override
- public void childEvent(CuratorFramework client,
- PathChildrenCacheEvent event)
- throws Exception {
-switch (event.getType()) {
-case CHILD_ADDED:
- processKeyAddOrUpdate(event.getData().getData());
- break;
-case CHILD_UPDATED:
- processKeyAddOrUpdate(event.getData().getData());
- break;
-case CHILD_REMOVED:
- processKeyRemoved(event.getData().getPath());
- break;
-default:
- break;
-}
- }
-}, listenerThreadPool);
+CuratorCacheListener keyCacheListener = CuratorCacheListener.builder()
+.forCreates(childData -> {
Review comment:
Note: there is a `forCreatesAndChanges()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 634521)
Time Spent: 1h 40m (was: 1.5h)
> Use CuratorCache implementation instead of PathChildrenCache / TreeCache
>
>
> Key: HADOOP-17835
> URL: https://issues.apache.org/jira/browse/HADOOP-17835
> Project: Hadoop Common
> Issue Type: Task
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> As we have moved to Curator 5.2.0 for Hadoop 3.4.0, we should start using new
> CuratorCache service implementation in place of deprecated PathChildrenCache
> and TreeCache usecases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Randgalt commented on a change in pull request #3266: HADOOP-17835. Use CuratorCache implementation instead of PathChildrenCache / TreeCache
Randgalt commented on a change in pull request #3266:
URL: https://github.com/apache/hadoop/pull/3266#discussion_r683400351
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java
##
@@ -363,105 +357,122 @@ public void startThreads() throws IOException {
throw new RuntimeException("Could not create ZK paths");
}
try {
- keyCache = new PathChildrenCache(zkClient, ZK_DTSM_MASTER_KEY_ROOT,
true);
+ keyCache = CuratorCache.bridgeBuilder(zkClient, ZK_DTSM_MASTER_KEY_ROOT)
+ .build();
if (keyCache != null) {
-keyCache.start(StartMode.BUILD_INITIAL_CACHE);
-keyCache.getListenable().addListener(new PathChildrenCacheListener() {
- @Override
- public void childEvent(CuratorFramework client,
- PathChildrenCacheEvent event)
- throws Exception {
-switch (event.getType()) {
-case CHILD_ADDED:
- processKeyAddOrUpdate(event.getData().getData());
- break;
-case CHILD_UPDATED:
- processKeyAddOrUpdate(event.getData().getData());
- break;
-case CHILD_REMOVED:
- processKeyRemoved(event.getData().getPath());
- break;
-default:
- break;
-}
- }
-}, listenerThreadPool);
+CuratorCacheListener keyCacheListener = CuratorCacheListener.builder()
+.forCreates(childData -> {
Review comment:
Note: there is a `forCreatesAndChanges()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17835) Use CuratorCache implementation instead of PathChildrenCache / TreeCache
[
https://issues.apache.org/jira/browse/HADOOP-17835?focusedWorklogId=634520=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634520
]
ASF GitHub Bot logged work on HADOOP-17835:
---
Author: ASF GitHub Bot
Created on: 05/Aug/21 12:19
Start Date: 05/Aug/21 12:19
Worklog Time Spent: 10m
Work Description: Randgalt commented on a change in pull request #3266:
URL: https://github.com/apache/hadoop/pull/3266#discussion_r683399151
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java
##
@@ -363,105 +357,122 @@ public void startThreads() throws IOException {
throw new RuntimeException("Could not create ZK paths");
}
try {
- keyCache = new PathChildrenCache(zkClient, ZK_DTSM_MASTER_KEY_ROOT,
true);
+ keyCache = CuratorCache.bridgeBuilder(zkClient, ZK_DTSM_MASTER_KEY_ROOT)
+ .build();
if (keyCache != null) {
Review comment:
`keyCache` will never be `null`
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java
##
@@ -363,105 +357,122 @@ public void startThreads() throws IOException {
throw new RuntimeException("Could not create ZK paths");
}
try {
- keyCache = new PathChildrenCache(zkClient, ZK_DTSM_MASTER_KEY_ROOT,
true);
+ keyCache = CuratorCache.bridgeBuilder(zkClient, ZK_DTSM_MASTER_KEY_ROOT)
+ .build();
if (keyCache != null) {
-keyCache.start(StartMode.BUILD_INITIAL_CACHE);
-keyCache.getListenable().addListener(new PathChildrenCacheListener() {
- @Override
- public void childEvent(CuratorFramework client,
- PathChildrenCacheEvent event)
- throws Exception {
-switch (event.getType()) {
-case CHILD_ADDED:
- processKeyAddOrUpdate(event.getData().getData());
- break;
-case CHILD_UPDATED:
- processKeyAddOrUpdate(event.getData().getData());
- break;
-case CHILD_REMOVED:
- processKeyRemoved(event.getData().getPath());
- break;
-default:
- break;
-}
- }
-}, listenerThreadPool);
+CuratorCacheListener keyCacheListener = CuratorCacheListener.builder()
+.forCreates(childData -> {
+ try {
+processKeyAddOrUpdate(childData.getData());
+ } catch (IOException e) {
+LOG.error("Error while processing Curator keyCacheListener "
++ "NODE_CREATED event");
+throw new UncheckedIOException(e);
+ }
+})
+.forChanges((oldNode, node) -> {
+ try {
+processKeyAddOrUpdate(node.getData());
+ } catch (IOException e) {
+LOG.error("Error while processing Curator keyCacheListener "
++ "NODE_CHANGED event");
+throw new UncheckedIOException(e);
+ }
+})
+.forDeletes(childData -> processKeyRemoved(childData.getPath()))
+.build();
+keyCache.listenable().addListener(keyCacheListener);
+keyCache.start();
loadFromZKCache(false);
}
} catch (Exception e) {
- throw new IOException("Could not start PathChildrenCache for keys", e);
+ throw new IOException("Could not start Curator keyCacheListener for
keys",
+ e);
}
if (isTokenWatcherEnabled) {
LOG.info("TokenCache is enabled");
try {
-tokenCache = new PathChildrenCache(zkClient, ZK_DTSM_TOKENS_ROOT,
true);
+tokenCache = CuratorCache.bridgeBuilder(zkClient, ZK_DTSM_TOKENS_ROOT)
+.build();
if (tokenCache != null) {
Review comment:
`tokenCache ` will never be `null`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 634520)
Time Spent: 1.5h (was: 1h 20m)
> Use CuratorCache implementation instead of PathChildrenCache / TreeCache
>
>
> Key: HADOOP-17835
> URL: https://issues.apache.org/jira/browse/HADOOP-17835
> Project: Hadoop Common
> Issue Type: Task
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
>
[GitHub] [hadoop] Randgalt commented on a change in pull request #3266: HADOOP-17835. Use CuratorCache implementation instead of PathChildrenCache / TreeCache
Randgalt commented on a change in pull request #3266:
URL: https://github.com/apache/hadoop/pull/3266#discussion_r683399151
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java
##
@@ -363,105 +357,122 @@ public void startThreads() throws IOException {
throw new RuntimeException("Could not create ZK paths");
}
try {
- keyCache = new PathChildrenCache(zkClient, ZK_DTSM_MASTER_KEY_ROOT,
true);
+ keyCache = CuratorCache.bridgeBuilder(zkClient, ZK_DTSM_MASTER_KEY_ROOT)
+ .build();
if (keyCache != null) {
Review comment:
`keyCache` will never be `null`
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java
##
@@ -363,105 +357,122 @@ public void startThreads() throws IOException {
throw new RuntimeException("Could not create ZK paths");
}
try {
- keyCache = new PathChildrenCache(zkClient, ZK_DTSM_MASTER_KEY_ROOT,
true);
+ keyCache = CuratorCache.bridgeBuilder(zkClient, ZK_DTSM_MASTER_KEY_ROOT)
+ .build();
if (keyCache != null) {
-keyCache.start(StartMode.BUILD_INITIAL_CACHE);
-keyCache.getListenable().addListener(new PathChildrenCacheListener() {
- @Override
- public void childEvent(CuratorFramework client,
- PathChildrenCacheEvent event)
- throws Exception {
-switch (event.getType()) {
-case CHILD_ADDED:
- processKeyAddOrUpdate(event.getData().getData());
- break;
-case CHILD_UPDATED:
- processKeyAddOrUpdate(event.getData().getData());
- break;
-case CHILD_REMOVED:
- processKeyRemoved(event.getData().getPath());
- break;
-default:
- break;
-}
- }
-}, listenerThreadPool);
+CuratorCacheListener keyCacheListener = CuratorCacheListener.builder()
+.forCreates(childData -> {
+ try {
+processKeyAddOrUpdate(childData.getData());
+ } catch (IOException e) {
+LOG.error("Error while processing Curator keyCacheListener "
++ "NODE_CREATED event");
+throw new UncheckedIOException(e);
+ }
+})
+.forChanges((oldNode, node) -> {
+ try {
+processKeyAddOrUpdate(node.getData());
+ } catch (IOException e) {
+LOG.error("Error while processing Curator keyCacheListener "
++ "NODE_CHANGED event");
+throw new UncheckedIOException(e);
+ }
+})
+.forDeletes(childData -> processKeyRemoved(childData.getPath()))
+.build();
+keyCache.listenable().addListener(keyCacheListener);
+keyCache.start();
loadFromZKCache(false);
}
} catch (Exception e) {
- throw new IOException("Could not start PathChildrenCache for keys", e);
+ throw new IOException("Could not start Curator keyCacheListener for
keys",
+ e);
}
if (isTokenWatcherEnabled) {
LOG.info("TokenCache is enabled");
try {
-tokenCache = new PathChildrenCache(zkClient, ZK_DTSM_TOKENS_ROOT,
true);
+tokenCache = CuratorCache.bridgeBuilder(zkClient, ZK_DTSM_TOKENS_ROOT)
+.build();
if (tokenCache != null) {
Review comment:
`tokenCache ` will never be `null`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17816) Run optional CI for changes in C
[ https://issues.apache.org/jira/browse/HADOOP-17816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17393946#comment-17393946 ] Prabhu Joseph commented on HADOOP-17816: Thanks [~gautham] for the patch. Have committed it to trunk. > Run optional CI for changes in C > > > Key: HADOOP-17816 > URL: https://issues.apache.org/jira/browse/HADOOP-17816 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.4.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Critical > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > We need to ensure that we run the CI for all the platforms when there are > changes in C files. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17816) Run optional CI for changes in C
[ https://issues.apache.org/jira/browse/HADOOP-17816?focusedWorklogId=634517=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634517 ] ASF GitHub Bot logged work on HADOOP-17816: --- Author: ASF GitHub Bot Created on: 05/Aug/21 12:13 Start Date: 05/Aug/21 12:13 Worklog Time Spent: 10m Work Description: PrabhuJoseph merged pull request #3238: URL: https://github.com/apache/hadoop/pull/3238 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 634517) Time Spent: 2h 50m (was: 2h 40m) > Run optional CI for changes in C > > > Key: HADOOP-17816 > URL: https://issues.apache.org/jira/browse/HADOOP-17816 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.4.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Critical > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > We need to ensure that we run the CI for all the platforms when there are > changes in C files. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
