[jira] [Commented] (HADOOP-16763) Make Curator 4 run in soft-compatibility mode with ZooKeeper 3.4
[
https://issues.apache.org/jira/browse/HADOOP-16763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16996075#comment-16996075
]
Íñigo Goiri commented on HADOOP-16763:
--
I don't quite get how this works but apparently is as easy as adding an
exclusion in the curator-recipes:
{code}
org.apache.curator
curator-recipes
${curator-version}
org.apache.zookeeper
zookeeper
{code}
> Make Curator 4 run in soft-compatibility mode with ZooKeeper 3.4
>
>
> Key: HADOOP-16763
> URL: https://issues.apache.org/jira/browse/HADOOP-16763
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Íñigo Goiri
>Priority: Major
>
> HADOOP-16579 changed Curator to 4.2 and ZooKeeper to 3.5.
> This change relate to the client libraries used by the components.
> However, the ensemble in most deployments is 3.4 (default in Ubuntu for
> example).
> To allow this mode, there is a soft-compatibility mode described in
> http://curator.apache.org/zk-compatibility.html
> We should enable this soft-compatibility mode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16763) Make Curator 4 run in soft-compatibility mode with ZooKeeper 3.4
Íñigo Goiri created HADOOP-16763: Summary: Make Curator 4 run in soft-compatibility mode with ZooKeeper 3.4 Key: HADOOP-16763 URL: https://issues.apache.org/jira/browse/HADOOP-16763 Project: Hadoop Common Issue Type: Improvement Reporter: Íñigo Goiri HADOOP-16579 changed Curator to 4.2 and ZooKeeper to 3.5. This change relate to the client libraries used by the components. However, the ensemble in most deployments is 3.4 (default in Ubuntu for example). To allow this mode, there is a soft-compatibility mode described in http://curator.apache.org/zk-compatibility.html We should enable this soft-compatibility mode. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 and ZooKeeper 3.5.6 in Hadoop
[
https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16996070#comment-16996070
]
Íñigo Goiri commented on HADOOP-16579:
--
I am currently testing trunk and I'm having issues with this.
My ZooKeeper servers use 3.4.10 (the one in Ubuntu).
As we know have Curato 4.X, this is not compatible so it fails.
They talk about in: http://curator.apache.org/zk-compatibility.html
Apparently, the solution is to tweak the dependencies and do:
{code}
org.apache.curator
curator-recipes
${curator-version}
org.apache.zookeeper
zookeeper
{code}
[~weichiu], should we do that?
> Upgrade to Apache Curator 4.2.0 and ZooKeeper 3.5.6 in Hadoop
> -
>
> Key: HADOOP-16579
> URL: https://issues.apache.org/jira/browse/HADOOP-16579
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Mate Szalay-Beko
>Assignee: Norbert Kalmár
>Priority: Major
> Fix For: 3.3.0
>
>
> *Update:* the original idea was to only update Curator but keep the old
> ZooKeeper version in Hadoop. However, we encountered some run-time
> backward-incompatibility during unit tests with Curator 4.2.0 and ZooKeeper
> 3.5.5. We haven't really investigated deeply these issues, but upgraded to
> ZooKeeper 3.5.5 (and later to 3.5.6). We had to do some minor fixes in the
> unit tests (and also had to change some deprecated Curator API calls), but
> [the latest PR|https://github.com/apache/hadoop/pull/1656] seems to be stable.
> ZooKeeper 3.5.6 just got released during our work. (I think the official
> announcement will get out maybe tomorrow, but it is already available in
> maven central or on the [Apache ZooKeeper ftp
> site|https://www-eu.apache.org/dist/zookeeper/]). It is considered to be a
> stable version, contains some minor fixes and improvements, plus some CVE
> fixes. See the [release
> notes|https://github.com/apache/zookeeper/blob/branch-3.5.6/zookeeper-docs/src/main/resources/markdown/releasenotes.md].
>
>
> Currently in Hadoop we are using [ZooKeeper version
> 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90].
> ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains
> many new features (including SSL related improvements which can be very
> important for production use; see [the release
> notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]).
> Apache Curator is a high level ZooKeeper client library, that makes it easier
> to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator
> 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91]
> and [in Ozone we use Curator
> 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146].
> Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator
> 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately,
> the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and
> 3.5.x. (see [the relevant Curator
> page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects
> have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other
> components are doing it right now (e.g. Hive).
> *The aims of this task are* to:
> - change Curator version in Hadoop to the latest stable 4.x version
> (currently 4.2.0)
> - also make sure we don't have multiple ZooKeeper versions in the classpath
> to avoid runtime problems (it is
> [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the
> ZooKeeper which come with Curator, so that there will be only a single
> ZooKeeper version used runtime in Hadoop)
> In this ticket we still don't want to change the default ZooKeeper version in
> Hadoop, we only want to make it possible for the community to be able to
> build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the
> ZooKeeper communication with SSL, what is only supported in the new ZooKeeper
> version). Upgrading to Curator 4.x should keep Hadoop to be compatible with
> both ZooKeeper 3.4 and 3.5.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16762) Add support for Filesystem#getFileChecksum in ABFS driver
[ https://issues.apache.org/jira/browse/HADOOP-16762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16995969#comment-16995969 ] Virajith Jalaparti commented on HADOOP-16762: - Hi [~bilahari.th] and [~snvijaya], do you have any plans to add this API to the ABFS driver? > Add support for Filesystem#getFileChecksum in ABFS driver > - > > Key: HADOOP-16762 > URL: https://issues.apache.org/jira/browse/HADOOP-16762 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Virajith Jalaparti >Priority: Major > > Currently, ABFS driver does not support Filesystem#getFileChecksum even > though the underlying ADLS REST API does. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16762) Add support for Filesystem#getFileChecksum in ABFS driver
Virajith Jalaparti created HADOOP-16762: --- Summary: Add support for Filesystem#getFileChecksum in ABFS driver Key: HADOOP-16762 URL: https://issues.apache.org/jira/browse/HADOOP-16762 Project: Hadoop Common Issue Type: Sub-task Reporter: Virajith Jalaparti Currently, ABFS driver does not support Filesystem#getFileChecksum even though the underlying ADLS REST API does. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Moved] (HADOOP-16761) KMSClientProvider does not work with client using ticket logged in externally
[ https://issues.apache.org/jira/browse/HADOOP-16761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao moved HDFS-15061 to HADOOP-16761: Key: HADOOP-16761 (was: HDFS-15061) Project: Hadoop Common (was: Hadoop HDFS) > KMSClientProvider does not work with client using ticket logged in externally > -- > > Key: HADOOP-16761 > URL: https://issues.apache.org/jira/browse/HADOOP-16761 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > > This is a regression from HDFS-13682 that checks not only the kerberos > credential but also enforce the login is non-external. This breaks client > applications that need to access HDFS encrypted file using kerberos ticket > that logged in external in ticket cache. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1752: HADOOP-16732. S3Guard to support encrypted DynamoDB table
hadoop-yetus commented on issue #1752: HADOOP-16732. S3Guard to support encrypted DynamoDB table URL: https://github.com/apache/hadoop/pull/1752#issuecomment-565581811 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 18m 6s | trunk passed | | +1 :green_heart: | compile | 17m 13s | trunk passed | | +1 :green_heart: | checkstyle | 2m 42s | trunk passed | | +1 :green_heart: | mvnsite | 2m 15s | trunk passed | | +1 :green_heart: | shadedclient | 18m 43s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 9s | trunk passed | | +0 :ok: | spotbugs | 1m 12s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 13s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 1m 19s | the patch passed | | +1 :green_heart: | compile | 16m 34s | the patch passed | | +1 :green_heart: | javac | 16m 34s | the patch passed | | -0 :warning: | checkstyle | 2m 36s | root: The patch generated 3 new + 20 unchanged - 0 fixed = 23 total (was 20) | | +1 :green_heart: | mvnsite | 2m 15s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 12m 47s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 13s | the patch passed | | +1 :green_heart: | findbugs | 3m 30s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 9m 10s | hadoop-common in the patch passed. | | +1 :green_heart: | unit | 1m 36s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 53s | The patch does not generate ASF License warnings. | | | | 119m 3s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1752/5/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1752 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint | | uname | Linux 4a31fc63641e 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / d12ad9e | | Default Java | 1.8.0_222 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1752/5/artifact/out/diff-checkstyle-root.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1752/5/testReport/ | | Max. process+thread count | 1550 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1752/5/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on a change in pull request #1761: HADOOP-16759. Filesystem openFile() builder to take a FileStatus param
liuml07 commented on a change in pull request #1761: HADOOP-16759. Filesystem
openFile() builder to take a FileStatus param
URL: https://github.com/apache/hadoop/pull/1761#discussion_r357767186
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/OpenFileParameters.java
##
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.impl;
+
+import java.util.Set;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+
+/**
+ * All the parameters from the openFile builder for the {@code
openFileWithOptions} commands.
+ *
+ * If/when new attributes added to the builder, this class will be extended.
+ */
+public class OpenFileParameters {
Review comment:
I'm thinking does it need a builder itself for easier construction and being
immutable?
```
OpenFileParameters parameters = new OpenFileParameters();
parameters.setMandatoryKeys(getMandatoryKeys());
parameters.setOptions(getOptions());
parameters.setBufferSize(getBufferSize());
parameters.setStatus(getStatus());
```
to
```
OpenFileParameters parameters = OpenFileParameters.builder()
.mandatoryKeys(getMandatoryKeys())
.ptions(getOptions())
.bufferSize(getBufferSize())
.status(getStatus())
.build();
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on a change in pull request #1761: HADOOP-16759. Filesystem openFile() builder to take a FileStatus param
liuml07 commented on a change in pull request #1761: HADOOP-16759. Filesystem
openFile() builder to take a FileStatus param
URL: https://github.com/apache/hadoop/pull/1761#discussion_r357769429
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileContext.java
##
@@ -2924,16 +2924,19 @@ protected FSDataInputStreamBuilder(
@Override
public CompletableFuture build() throws IOException {
final Path absF = fixRelativePart(getPath());
+ OpenFileParameters parameters = new OpenFileParameters();
+ parameters.setMandatoryKeys(getMandatoryKeys());
+ parameters.setOptions(getOptions());
+ parameters.setBufferSize(getBufferSize());
+ parameters.setStatus(getStatus());
return new FSLinkResolver>() {
@Override
public CompletableFuture next(
final AbstractFileSystem fs,
final Path p)
throws IOException {
return fs.openFileWithOptions(p,
- getMandatoryKeys(),
- getOptions(),
- getBufferSize());
+ parameters);
Review comment:
nit: merge with previous line seems no longer than 80
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on a change in pull request #1761: HADOOP-16759. Filesystem openFile() builder to take a FileStatus param
liuml07 commented on a change in pull request #1761: HADOOP-16759. Filesystem openFile() builder to take a FileStatus param URL: https://github.com/apache/hadoop/pull/1761#discussion_r357790680 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java ## @@ -4366,15 +4378,38 @@ private void requireSelectSupport(final Path source) throws InternalConstants.STANDARD_OPENFILE_KEYS, "for " + path + " in non-select file I/O"); } +FileStatus status = parameters.getStatus(); Review comment: nit: This `status` can be named `providedStatus` or something. Clearer as it's referred multiple times following. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on a change in pull request #1761: HADOOP-16759. Filesystem openFile() builder to take a FileStatus param
liuml07 commented on a change in pull request #1761: HADOOP-16759. Filesystem
openFile() builder to take a FileStatus param
URL: https://github.com/apache/hadoop/pull/1761#discussion_r357794184
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -955,23 +956,33 @@ protected URI canonicalizeUri(URI rawUri) {
@Retries.RetryTranslated
public FSDataInputStream open(Path f, int bufferSize)
throws IOException {
-return open(f, Optional.empty());
+return open(f, Optional.empty(), null);
}
/**
* Opens an FSDataInputStream at the indicated Path.
- * @param path the file to open
+ * if status contains an S3AFileStatus reference, it is used
+ * and so a HEAD request to the store is avoided.
+ *
+ * @param file the file to open
* @param options configuration options if opened with the builder API.
+ * @param status optional file status.
* @throws IOException IO failure.
*/
@Retries.RetryTranslated
private FSDataInputStream open(
- final Path path,
- final Optional options)
+ final Path file,
+ final Optional options,
+ final S3AFileStatus status)
Review comment:
I'm not sure, but status can be Optional?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on a change in pull request #1761: HADOOP-16759. Filesystem openFile() builder to take a FileStatus param
liuml07 commented on a change in pull request #1761: HADOOP-16759. Filesystem
openFile() builder to take a FileStatus param
URL: https://github.com/apache/hadoop/pull/1761#discussion_r357792630
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ARemoteFileChanged.java
##
@@ -432,6 +433,42 @@ public void testReadFileChangedOutOfSyncMetadata() throws
Throwable {
}
}
+ /**
+ * Verifies that when the openFile builder is passed in a status,
+ * then that is used to eliminate the getFileStatus call in open();
+ * thus the version and etag passed down are still used.
+ */
+ @Test
+ public void testOpenFileWithStatus() throws Throwable {
+final Path testpath = path("testOpenFileWithStatus.dat");
+final byte[] dataset = TEST_DATA_BYTES;
+S3AFileStatus originalStatus =
+writeFile(testpath, dataset, dataset.length, true);
+
+// forge a file status with a different tag
+S3AFileStatus forgedStatus =
+S3AFileStatus.fromFileStatus(originalStatus, Tristate.FALSE,
+originalStatus.getETag() + "-fake",
+originalStatus.getVersionId() + "");
+fs.getMetadataStore().put(
+new PathMetadata(forgedStatus, Tristate.FALSE, false));
+
+// By passing in the status open() doesn't need to check s3guard
+// And hence the existing file is opened
+try (FSDataInputStream instream = fs.openFile(testpath)
+.withFileStatus(originalStatus)
+.build().get()) {
+ instream.read();
+}
+
+// and this holds for S3A Located Status
+try (FSDataInputStream instream = fs.openFile(testpath)
+.withFileStatus(new S3ALocatedFileStatus(originalStatus, null))
+.build().get()) {
+ instream.read();
+}
+ }
Review comment:
What about vanilla `fs.openFile(testPath)` without file status? It will
simply fail right? Do we need a test for that?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13144) Enhancing IPC client throughput via multiple connections per user
[
https://issues.apache.org/jira/browse/HADOOP-13144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16995816#comment-16995816
]
Íñigo Goiri commented on HADOOP-13144:
--
Thanks [~Symious] for the update.
Now that we have proof that this is actually helpful, we should just do the RPC
change and then do the Router changes in HDFS-13274.
The only part I'm not sure about which interfaces in RpcEngine and RPC we
should extend.
Maybe just stick with the ones here.
The other question is what to do with 3.3 and 2.10.
> Enhancing IPC client throughput via multiple connections per user
> -
>
> Key: HADOOP-13144
> URL: https://issues.apache.org/jira/browse/HADOOP-13144
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
>Reporter: Jason Kace
>Assignee: Íñigo Goiri
>Priority: Minor
> Attachments: HADOOP-13144-branch-2.9.001.patch,
> HADOOP-13144-branch-2.9.002.patch, HADOOP-13144-branch-2.9.003.patch,
> HADOOP-13144-branch-2.9.004.patch, HADOOP-13144-performance.patch,
> HADOOP-13144.000.patch, HADOOP-13144.001.patch, HADOOP-13144.002.patch,
> HADOOP-13144.003.patch
>
>
> The generic IPC client ({{org.apache.hadoop.ipc.Client}}) utilizes a single
> connection thread for each {{ConnectionId}}. The {{ConnectionId}} is unique
> to the connection's remote address, ticket and protocol. Each ConnectionId
> is 1:1 mapped to a connection thread by the client via a map cache.
> The result is to serialize all IPC read/write activity through a single
> thread for a each user/ticket + address. If a single user makes repeated
> calls (1k-100k/sec) to the same destination, the IPC client becomes a
> bottleneck.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on issue #1760: HDFS-15058. Enlarge the sleep time of Fsck tests for checking corrupt files
goiri commented on issue #1760: HDFS-15058. Enlarge the sleep time of Fsck tests for checking corrupt files URL: https://github.com/apache/hadoop/pull/1760#issuecomment-565540742 HDFS-15038 is already fixing this with a wait instead of sleeping blindly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1752: HADOOP-16732. S3Guard to support encrypted DynamoDB table
hadoop-yetus removed a comment on issue #1752: HADOOP-16732. S3Guard to support encrypted DynamoDB table URL: https://github.com/apache/hadoop/pull/1752#issuecomment-564431241 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 0s | Docker mode activated. | | -1 :x: | docker | 0m 7s | Docker failed to build yetus/hadoop:104ccca9169. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/1752 | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1752/2/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on issue #1738: HADOOP-16748. Support Python 3 in dev-support scripts.
liuml07 commented on issue #1738: HADOOP-16748. Support Python 3 in dev-support scripts. URL: https://github.com/apache/hadoop/pull/1738#issuecomment-565539869 Looks good to me overall, but I'm not Python expert to be +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16757) Increase timeout unit test rule for MetadataStoreTestBase
[
https://issues.apache.org/jira/browse/HADOOP-16757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu resolved HADOOP-16757.
Fix Version/s: 3.3.0
Hadoop Flags: Reviewed
Resolution: Fixed
Committed to {{trunk}} branch. Not sure about backporting practice here, and
left a message in the uber JIRA. Thanks [~ste...@apache.org] for reviewing!
> Increase timeout unit test rule for MetadataStoreTestBase
> -
>
> Key: HADOOP-16757
> URL: https://issues.apache.org/jira/browse/HADOOP-16757
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Major
> Fix For: 3.3.0
>
>
> Not sure if this is a good proposal, but I saw a few cases where some
> integration test methods in {{ITestDynamoDBMetadataStore}} timed out simply.
> Specially, the one keeps failing me today is
> {{testAncestorOverwriteConflict}}. I increase the timeout to 200s and it
> works for me happily. Am I using VPN and a slow home network, I'm afraid so.
> The time out rule, as inherited from base class {{HadoopTestBase}}, is 100s
> by default. Though that 100s time out default value can be overridden in base
> class via system property {{test.default.timeout}}, that's system wide
> affecting all other tests. Changing that time out value for one test is no
> better than overriding in this test {{ITestDynamoDBMetadataStore}}
> explicitly. I think doubling it to 200s would not be very crazy considering
> we are testing against a remote web service, create and destroy tables
> sometimes.
> Thoughts?
> [EDIT]: I meant 100s when I said 10s. Sorry for confusion.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15620) Über-jira: S3A phase VI: Hadoop 3.3 features
[ https://issues.apache.org/jira/browse/HADOOP-15620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16995780#comment-16995780 ] Mingliang Liu commented on HADOOP-15620: Hi [~ste...@apache.org], I see we are not backporting good stuff to 3.2- branches any more. So what's the deal here? These changes will only target for 3.3 (aka trunk) branch, so we should use 3.3 hadoop-aws module with all 3.x Hadoop deployment? For branch-2.10 I will think about that as well...our HBase cluster is not yet built against Hadoop 3. Thanks, > Über-jira: S3A phase VI: Hadoop 3.3 features > > > Key: HADOOP-15620 > URL: https://issues.apache.org/jira/browse/HADOOP-15620 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16757) Increase timeout unit test rule for MetadataStoreTestBase
[
https://issues.apache.org/jira/browse/HADOOP-16757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16995756#comment-16995756
]
Hudson commented on HADOOP-16757:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17759 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17759/])
HADOOP-16757. Increase timeout unit test rule for MetadataStoreTestBase
(github: rev d12ad9e8adbce5ee4c9a5883f25c1635cd122878)
* (edit)
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/HadoopTestBase.java
* (edit)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/MetadataStoreTestBase.java
> Increase timeout unit test rule for MetadataStoreTestBase
> -
>
> Key: HADOOP-16757
> URL: https://issues.apache.org/jira/browse/HADOOP-16757
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Major
>
> Not sure if this is a good proposal, but I saw a few cases where some
> integration test methods in {{ITestDynamoDBMetadataStore}} timed out simply.
> Specially, the one keeps failing me today is
> {{testAncestorOverwriteConflict}}. I increase the timeout to 200s and it
> works for me happily. Am I using VPN and a slow home network, I'm afraid so.
> The time out rule, as inherited from base class {{HadoopTestBase}}, is 100s
> by default. Though that 100s time out default value can be overridden in base
> class via system property {{test.default.timeout}}, that's system wide
> affecting all other tests. Changing that time out value for one test is no
> better than overriding in this test {{ITestDynamoDBMetadataStore}}
> explicitly. I think doubling it to 200s would not be very crazy considering
> we are testing against a remote web service, create and destroy tables
> sometimes.
> Thoughts?
> [EDIT]: I meant 100s when I said 10s. Sorry for confusion.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] liuml07 merged pull request #1757: HADOOP-16757. Increase timeout unit test rule for MetadataStoreTestBase
liuml07 merged pull request #1757: HADOOP-16757. Increase timeout unit test rule for MetadataStoreTestBase URL: https://github.com/apache/hadoop/pull/1757 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1761: HADOOP-16759. Filesystem openFile() builder to take a FileStatus param
hadoop-yetus commented on issue #1761: HADOOP-16759. Filesystem openFile() builder to take a FileStatus param URL: https://github.com/apache/hadoop/pull/1761#issuecomment-565468853 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 35s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 20m 27s | trunk passed | | +1 :green_heart: | compile | 17m 41s | trunk passed | | +1 :green_heart: | checkstyle | 2m 45s | trunk passed | | +1 :green_heart: | mvnsite | 2m 18s | trunk passed | | +1 :green_heart: | shadedclient | 19m 1s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 16s | trunk passed | | +0 :ok: | spotbugs | 1m 12s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 14s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 1m 21s | the patch passed | | +1 :green_heart: | compile | 16m 29s | the patch passed | | +1 :green_heart: | javac | 16m 29s | the patch passed | | -0 :warning: | checkstyle | 2m 46s | root: The patch generated 9 new + 289 unchanged - 2 fixed = 298 total (was 291) | | +1 :green_heart: | mvnsite | 2m 17s | the patch passed | | -1 :x: | whitespace | 0m 0s | The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | shadedclient | 12m 57s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 14s | the patch passed | | +1 :green_heart: | findbugs | 3m 28s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 8m 51s | hadoop-common in the patch failed. | | +1 :green_heart: | unit | 1m 36s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 56s | The patch does not generate ASF License warnings. | | | | 122m 16s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.fs.TestHarFileSystem | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1761/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1761 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint | | uname | Linux 9e65e4d29757 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 65c4660 | | Default Java | 1.8.0_222 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1761/1/artifact/out/diff-checkstyle-root.txt | | whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-1761/1/artifact/out/whitespace-eol.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1761/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1761/1/testReport/ | | Max. process+thread count | 1475 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1761/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional
[jira] [Commented] (HADOOP-13144) Enhancing IPC client throughput via multiple connections per user
[
https://issues.apache.org/jira/browse/HADOOP-13144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16995633#comment-16995633
]
Janus Chow commented on HADOOP-13144:
-
Finally uploaded.
> Enhancing IPC client throughput via multiple connections per user
> -
>
> Key: HADOOP-13144
> URL: https://issues.apache.org/jira/browse/HADOOP-13144
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
>Reporter: Jason Kace
>Assignee: Íñigo Goiri
>Priority: Minor
> Attachments: HADOOP-13144-branch-2.9.001.patch,
> HADOOP-13144-branch-2.9.002.patch, HADOOP-13144-branch-2.9.003.patch,
> HADOOP-13144-branch-2.9.004.patch, HADOOP-13144-performance.patch,
> HADOOP-13144.000.patch, HADOOP-13144.001.patch, HADOOP-13144.002.patch,
> HADOOP-13144.003.patch
>
>
> The generic IPC client ({{org.apache.hadoop.ipc.Client}}) utilizes a single
> connection thread for each {{ConnectionId}}. The {{ConnectionId}} is unique
> to the connection's remote address, ticket and protocol. Each ConnectionId
> is 1:1 mapped to a connection thread by the client via a map cache.
> The result is to serialize all IPC read/write activity through a single
> thread for a each user/ticket + address. If a single user makes repeated
> calls (1k-100k/sec) to the same destination, the IPC client becomes a
> bottleneck.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16756) Inconsistent Behavior on distcp -update over S3
[
https://issues.apache.org/jira/browse/HADOOP-16756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16995628#comment-16995628
]
Steve Loughran commented on HADOOP-16756:
-
not as simple as looking at FS schema as webhdfs and hdfs interop with
checksums and we don't want to break that
> Inconsistent Behavior on distcp -update over S3
> ---
>
> Key: HADOOP-16756
> URL: https://issues.apache.org/jira/browse/HADOOP-16756
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Daisuke Kobayashi
>Priority: Major
>
> Distcp over S3A always copies all source files no matter the files are
> changed or not. This is opposite to the statement in the doc below.
> [http://hadoop.apache.org/docs/current/hadoop-distcp/DistCp.html]
> {noformat}
> And to use -update to only copy changed files.
> {noformat}
> CopyMapper compares file length as well as block size before copying. While
> the file length should match, the block size does not. This is apparently
> because the returned block size from S3A is always 32MB.
> [https://github.com/apache/hadoop/blob/release-3.2.0-RC1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java#L348]
> I'd suppose we should update the documentation or make code change.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13144) Enhancing IPC client throughput via multiple connections per user
[
https://issues.apache.org/jira/browse/HADOOP-13144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16995599#comment-16995599
]
Hadoop QA commented on HADOOP-13144:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
42s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.9 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
37s{color} | {color:green} branch-2.9 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
49s{color} | {color:green} branch-2.9 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
54s{color} | {color:green} branch-2.9 passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
49s{color} | {color:green} branch-2.9 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
44s{color} | {color:green} branch-2.9 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
32s{color} | {color:green} branch-2.9 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
54s{color} | {color:green} branch-2.9 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} branch-2.9 passed with JDK v1.8.0_222 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
4s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
58s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
58s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 53s{color} | {color:orange} root: The patch generated 219 new + 376
unchanged - 0 fixed = 595 total (was 376) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
52s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
23s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 15m
4s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}106m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:8ebaf02db31 |
| JIRA Issue | HADOOP-13144 |
| JIRA Patch URL |
[GitHub] [hadoop] steveloughran commented on issue #1757: HADOOP-16757. Increase timeout unit test rule for MetadataStoreTestBase
steveloughran commented on issue #1757: HADOOP-16757. Increase timeout unit test rule for MetadataStoreTestBase URL: https://github.com/apache/hadoop/pull/1757#issuecomment-565430468 LGTM +1 thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1761: HADOOP-16759. Filesystem openFile() builder to take a FileStatus param
steveloughran commented on issue #1761: HADOOP-16759. Filesystem openFile() builder to take a FileStatus param URL: https://github.com/apache/hadoop/pull/1761#issuecomment-565430013 tested -s3a ireland. Not tested the other stores to make sure they don't break (as they don't read the status, it's hard to see how). Could also add more failure tests (path mismatch, ...) and use s3a metrics to verify HEAD doesn't count This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran opened a new pull request #1761: HADOOP-16759. Filesystem openFile() builder to take a FileStatus param
steveloughran opened a new pull request #1761: HADOOP-16759. Filesystem openFile() builder to take a FileStatus param URL: https://github.com/apache/hadoop/pull/1761 * Enhanced builder + FS spec * s3a FS to use this to skip HEAD on open * and to use version/etag when opening the file works with S3AFileStatus FS and S3ALocatedFileStatus Change-Id: If80f73137643fd50a969a92ad5794d0d09e3aee6 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13144) Enhancing IPC client throughput via multiple connections per user
[
https://issues.apache.org/jira/browse/HADOOP-13144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Janus Chow updated HADOOP-13144:
Attachment: HADOOP-13144-branch-2.9.004.patch
> Enhancing IPC client throughput via multiple connections per user
> -
>
> Key: HADOOP-13144
> URL: https://issues.apache.org/jira/browse/HADOOP-13144
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
>Reporter: Jason Kace
>Assignee: Íñigo Goiri
>Priority: Minor
> Attachments: HADOOP-13144-branch-2.9.001.patch,
> HADOOP-13144-branch-2.9.002.patch, HADOOP-13144-branch-2.9.003.patch,
> HADOOP-13144-branch-2.9.004.patch, HADOOP-13144-performance.patch,
> HADOOP-13144.000.patch, HADOOP-13144.001.patch, HADOOP-13144.002.patch,
> HADOOP-13144.003.patch
>
>
> The generic IPC client ({{org.apache.hadoop.ipc.Client}}) utilizes a single
> connection thread for each {{ConnectionId}}. The {{ConnectionId}} is unique
> to the connection's remote address, ticket and protocol. Each ConnectionId
> is 1:1 mapped to a connection thread by the client via a map cache.
> The result is to serialize all IPC read/write activity through a single
> thread for a each user/ticket + address. If a single user makes repeated
> calls (1k-100k/sec) to the same destination, the IPC client becomes a
> bottleneck.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13144) Enhancing IPC client throughput via multiple connections per user
[
https://issues.apache.org/jira/browse/HADOOP-13144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Janus Chow updated HADOOP-13144:
Attachment: (was: HADOOP-13144-branch-2.9.004.patch)
> Enhancing IPC client throughput via multiple connections per user
> -
>
> Key: HADOOP-13144
> URL: https://issues.apache.org/jira/browse/HADOOP-13144
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
>Reporter: Jason Kace
>Assignee: Íñigo Goiri
>Priority: Minor
> Attachments: HADOOP-13144-branch-2.9.001.patch,
> HADOOP-13144-branch-2.9.002.patch, HADOOP-13144-branch-2.9.003.patch,
> HADOOP-13144-branch-2.9.004.patch, HADOOP-13144-performance.patch,
> HADOOP-13144.000.patch, HADOOP-13144.001.patch, HADOOP-13144.002.patch,
> HADOOP-13144.003.patch
>
>
> The generic IPC client ({{org.apache.hadoop.ipc.Client}}) utilizes a single
> connection thread for each {{ConnectionId}}. The {{ConnectionId}} is unique
> to the connection's remote address, ticket and protocol. Each ConnectionId
> is 1:1 mapped to a connection thread by the client via a map cache.
> The result is to serialize all IPC read/write activity through a single
> thread for a each user/ticket + address. If a single user makes repeated
> calls (1k-100k/sec) to the same destination, the IPC client becomes a
> bottleneck.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13144) Enhancing IPC client throughput via multiple connections per user
[
https://issues.apache.org/jira/browse/HADOOP-13144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16995542#comment-16995542
]
Hadoop QA commented on HADOOP-13144:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m
46s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.9 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
22s{color} | {color:green} branch-2.9 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
0s{color} | {color:green} branch-2.9 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
59s{color} | {color:green} branch-2.9 passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
52s{color} | {color:green} branch-2.9 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
46s{color} | {color:green} branch-2.9 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
37s{color} | {color:green} branch-2.9 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
56s{color} | {color:green} branch-2.9 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} branch-2.9 passed with JDK v1.8.0_222 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
17s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
57s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
57s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 54s{color} | {color:orange} root: The patch generated 219 new + 376
unchanged - 0 fixed = 595 total (was 376) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
52s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 12s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 16m
17s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}125m 34s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.net.TestClusterTopology |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:8ebaf02db31 |
| JIRA Issue |
[GitHub] [hadoop] hadoop-yetus commented on issue #1760: HDFS-15058. Enlarge the sleep time of Fsck tests for checking corrupt files
hadoop-yetus commented on issue #1760: HDFS-15058. Enlarge the sleep time of Fsck tests for checking corrupt files URL: https://github.com/apache/hadoop/pull/1760#issuecomment-565391678 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 21m 9s | trunk passed | | +1 :green_heart: | compile | 1m 6s | trunk passed | | +1 :green_heart: | checkstyle | 0m 52s | trunk passed | | +1 :green_heart: | mvnsite | 1m 14s | trunk passed | | +1 :green_heart: | shadedclient | 14m 31s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 25s | trunk passed | | +0 :ok: | spotbugs | 3m 34s | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 :x: | findbugs | 3m 31s | hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant findbugs warnings. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 1m 12s | the patch passed | | +1 :green_heart: | compile | 1m 5s | the patch passed | | +1 :green_heart: | javac | 1m 5s | the patch passed | | +1 :green_heart: | checkstyle | 0m 42s | the patch passed | | +1 :green_heart: | mvnsite | 1m 14s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 12m 59s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 18s | the patch passed | | +1 :green_heart: | findbugs | 2m 53s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 94m 3s | hadoop-hdfs in the patch failed. | | +1 :green_heart: | asflicense | 0m 41s | The patch does not generate ASF License warnings. | | | | 161m 24s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks | | | hadoop.hdfs.server.balancer.TestBalancerRPCDelay | | | hadoop.hdfs.server.namenode.TestFsck | | | hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA | | | hadoop.hdfs.server.datanode.TestDataNodeMetrics | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1760/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1760 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 04ca6a9ab4b2 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 65c4660 | | Default Java | 1.8.0_222 | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1760/1/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1760/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1760/1/testReport/ | | Max. process+thread count | 4455 (vs. ulimit of 5500) | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1760/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1758: HDFS-15052. WebHDFS getTrashRoot leads to OOM due to FileSystem objec…
hadoop-yetus commented on issue #1758: HDFS-15052. WebHDFS getTrashRoot leads to OOM due to FileSystem objec… URL: https://github.com/apache/hadoop/pull/1758#issuecomment-565389566 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 16s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 22m 32s | trunk passed | | +1 :green_heart: | compile | 1m 8s | trunk passed | | +1 :green_heart: | checkstyle | 0m 49s | trunk passed | | +1 :green_heart: | mvnsite | 1m 18s | trunk passed | | +1 :green_heart: | shadedclient | 16m 53s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 24s | trunk passed | | +0 :ok: | spotbugs | 3m 6s | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 :x: | findbugs | 3m 2s | hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant findbugs warnings. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 1m 9s | the patch passed | | +1 :green_heart: | compile | 1m 5s | the patch passed | | +1 :green_heart: | javac | 1m 5s | the patch passed | | +1 :green_heart: | checkstyle | 0m 46s | the patch passed | | +1 :green_heart: | mvnsite | 1m 15s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 14m 44s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 13s | the patch passed | | +1 :green_heart: | findbugs | 3m 6s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 105m 58s | hadoop-hdfs in the patch failed. | | +1 :green_heart: | asflicense | 0m 38s | The patch does not generate ASF License warnings. | | | | 180m 8s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.server.balancer.TestBalancerRPCDelay | | | hadoop.hdfs.server.namenode.TestFsck | | | hadoop.hdfs.TestDeadNodeDetection | | | hadoop.hdfs.server.datanode.TestDataNodeUUID | | | hadoop.hdfs.server.datanode.TestDataNodeLifeline | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1758/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1758 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux b0e788b40a5d 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 65c4660 | | Default Java | 1.8.0_222 | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1758/3/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1758/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1758/3/testReport/ | | Max. process+thread count | 2847 (vs. ulimit of 5500) | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1758/3/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16756) Inconsistent Behavior on distcp -update over S3
[
https://issues.apache.org/jira/browse/HADOOP-16756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16995514#comment-16995514
]
Srinivasu Majeti commented on HADOOP-16756:
---
Hi [~ste...@apache.org], [~daisuke.kobayashi], Should we need another option
like -skipblocklengthcheck for a copy from on-prem to the cloud? Or skip it
always if target file system is different from the source ?
> Inconsistent Behavior on distcp -update over S3
> ---
>
> Key: HADOOP-16756
> URL: https://issues.apache.org/jira/browse/HADOOP-16756
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3, tools/distcp
>Affects Versions: 3.3.0
>Reporter: Daisuke Kobayashi
>Priority: Major
>
> Distcp over S3A always copies all source files no matter the files are
> changed or not. This is opposite to the statement in the doc below.
> [http://hadoop.apache.org/docs/current/hadoop-distcp/DistCp.html]
> {noformat}
> And to use -update to only copy changed files.
> {noformat}
> CopyMapper compares file length as well as block size before copying. While
> the file length should match, the block size does not. This is apparently
> because the returned block size from S3A is always 32MB.
> [https://github.com/apache/hadoop/blob/release-3.2.0-RC1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java#L348]
> I'd suppose we should update the documentation or make code change.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13144) Enhancing IPC client throughput via multiple connections per user
[
https://issues.apache.org/jira/browse/HADOOP-13144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Janus Chow updated HADOOP-13144:
Attachment: HADOOP-13144-branch-2.9.004.patch
> Enhancing IPC client throughput via multiple connections per user
> -
>
> Key: HADOOP-13144
> URL: https://issues.apache.org/jira/browse/HADOOP-13144
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
>Reporter: Jason Kace
>Assignee: Íñigo Goiri
>Priority: Minor
> Attachments: HADOOP-13144-branch-2.9.001.patch,
> HADOOP-13144-branch-2.9.002.patch, HADOOP-13144-branch-2.9.003.patch,
> HADOOP-13144-branch-2.9.004.patch, HADOOP-13144-performance.patch,
> HADOOP-13144.000.patch, HADOOP-13144.001.patch, HADOOP-13144.002.patch,
> HADOOP-13144.003.patch
>
>
> The generic IPC client ({{org.apache.hadoop.ipc.Client}}) utilizes a single
> connection thread for each {{ConnectionId}}. The {{ConnectionId}} is unique
> to the connection's remote address, ticket and protocol. Each ConnectionId
> is 1:1 mapped to a connection thread by the client via a map cache.
> The result is to serialize all IPC read/write activity through a single
> thread for a each user/ticket + address. If a single user makes repeated
> calls (1k-100k/sec) to the same destination, the IPC client becomes a
> bottleneck.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1752: HADOOP-16732. S3Guard to support encrypted DynamoDB table
hadoop-yetus commented on issue #1752: HADOOP-16732. S3Guard to support encrypted DynamoDB table URL: https://github.com/apache/hadoop/pull/1752#issuecomment-565352545 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 50s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 1m 6s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 18m 19s | trunk passed | | +1 :green_heart: | compile | 17m 1s | trunk passed | | +1 :green_heart: | checkstyle | 2m 41s | trunk passed | | +1 :green_heart: | mvnsite | 2m 17s | trunk passed | | +1 :green_heart: | shadedclient | 18m 54s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 11s | trunk passed | | +0 :ok: | spotbugs | 1m 10s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 9s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 1m 20s | the patch passed | | +1 :green_heart: | compile | 16m 25s | the patch passed | | +1 :green_heart: | javac | 16m 25s | the patch passed | | -0 :warning: | checkstyle | 2m 38s | root: The patch generated 3 new + 20 unchanged - 0 fixed = 23 total (was 20) | | +1 :green_heart: | mvnsite | 2m 15s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 12m 51s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 15s | the patch passed | | +1 :green_heart: | findbugs | 3m 33s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 9m 7s | hadoop-common in the patch passed. | | +1 :green_heart: | unit | 1m 34s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 55s | The patch does not generate ASF License warnings. | | | | 122m 8s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1752/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1752 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint | | uname | Linux 0ab4c7b0435b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 65c4660 | | Default Java | 1.8.0_222 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1752/4/artifact/out/diff-checkstyle-root.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1752/4/testReport/ | | Max. process+thread count | 1593 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1752/4/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1752: HADOOP-16732. S3Guard to support encrypted DynamoDB table
hadoop-yetus commented on issue #1752: HADOOP-16732. S3Guard to support encrypted DynamoDB table URL: https://github.com/apache/hadoop/pull/1752#issuecomment-565348816 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 1s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 1m 10s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 18m 36s | trunk passed | | +1 :green_heart: | compile | 17m 8s | trunk passed | | +1 :green_heart: | checkstyle | 2m 41s | trunk passed | | +1 :green_heart: | mvnsite | 2m 15s | trunk passed | | +1 :green_heart: | shadedclient | 18m 30s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 13s | trunk passed | | +0 :ok: | spotbugs | 1m 11s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 14s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 1m 19s | the patch passed | | +1 :green_heart: | compile | 16m 26s | the patch passed | | +1 :green_heart: | javac | 16m 26s | the patch passed | | -0 :warning: | checkstyle | 2m 38s | root: The patch generated 3 new + 20 unchanged - 0 fixed = 23 total (was 20) | | +1 :green_heart: | mvnsite | 2m 18s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 12m 56s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 13s | the patch passed | | +1 :green_heart: | findbugs | 3m 27s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 9m 9s | hadoop-common in the patch passed. | | +1 :green_heart: | unit | 1m 35s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 56s | The patch does not generate ASF License warnings. | | | | 131m 25s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1752/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1752 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint | | uname | Linux 523ff6a3bf80 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 65c4660 | | Default Java | 1.8.0_222 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1752/3/artifact/out/diff-checkstyle-root.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1752/3/testReport/ | | Max. process+thread count | 1347 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1752/3/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13144) Enhancing IPC client throughput via multiple connections per user
[
https://issues.apache.org/jira/browse/HADOOP-13144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16995458#comment-16995458
]
Hadoop QA commented on HADOOP-13144:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
50s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.9 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
1s{color} | {color:green} branch-2.9 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
51s{color} | {color:green} branch-2.9 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
53s{color} | {color:green} branch-2.9 passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
49s{color} | {color:green} branch-2.9 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
41s{color} | {color:green} branch-2.9 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
36s{color} | {color:green} branch-2.9 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
54s{color} | {color:green} branch-2.9 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} branch-2.9 passed with JDK v1.8.0_222 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
59s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
41s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 51s{color} | {color:orange} root: The patch generated 219 new + 376
unchanged - 0 fixed = 595 total (was 376) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
1s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
34s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 16m
15s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
37s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}107m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:8ebaf02db31 |
| JIRA Issue | HADOOP-13144 |
| JIRA Patch URL |
