[GitHub] [hadoop] sunchao commented on pull request #2297: HADOOP-17125. Using snappy-java in SnappyCodec
sunchao commented on pull request #2297: URL: https://github.com/apache/hadoop/pull/2297#issuecomment-691875174 Just re-triggered the job let's see what happens This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17125) Using snappy-java in SnappyCodec
[ https://issues.apache.org/jira/browse/HADOOP-17125?focusedWorklogId=483830&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-483830 ] ASF GitHub Bot logged work on HADOOP-17125: --- Author: ASF GitHub Bot Created on: 14/Sep/20 07:31 Start Date: 14/Sep/20 07:31 Worklog Time Spent: 10m Work Description: sunchao commented on pull request #2297: URL: https://github.com/apache/hadoop/pull/2297#issuecomment-691875174 Just re-triggered the job let's see what happens This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 483830) Time Spent: 8h 10m (was: 8h) > Using snappy-java in SnappyCodec > > > Key: HADOOP-17125 > URL: https://issues.apache.org/jira/browse/HADOOP-17125 > Project: Hadoop Common > Issue Type: New Feature > Components: common >Affects Versions: 3.3.0 >Reporter: DB Tsai >Priority: Major > Labels: pull-request-available > Time Spent: 8h 10m > Remaining Estimate: 0h > > In Hadoop, we use native libs for snappy codec which has several > disadvantages: > * It requires native *libhadoop* and *libsnappy* to be installed in system > *LD_LIBRARY_PATH*, and they have to be installed separately on each node of > the clusters, container images, or local test environments which adds huge > complexities from deployment point of view. In some environments, it requires > compiling the natives from sources which is non-trivial. Also, this approach > is platform dependent; the binary may not work in different platform, so it > requires recompilation. > * It requires extra configuration of *java.library.path* to load the > natives, and it results higher application deployment and maintenance cost > for users. > Projects such as *Spark* and *Parquet* use > [snappy-java|[https://github.com/xerial/snappy-java]] which is JNI-based > implementation. It contains native binaries for Linux, Mac, and IBM in jar > file, and it can automatically load the native binaries into JVM from jar > without any setup. If a native implementation can not be found for a > platform, it can fallback to pure-java implementation of snappy based on > [aircompressor|[https://github.com/airlift/aircompressor/tree/master/src/main/java/io/airlift/compress/snappy]]. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17203) Test failures in ITestAzureBlobFileSystemCheckAccess in ABFS
[ https://issues.apache.org/jira/browse/HADOOP-17203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195261#comment-17195261 ] Mukund Thakur commented on HADOOP-17203: Even I am seeing this failure and I am not even setting any authtype in my conf. Shouldn't we skip the test if authtype is not set ? > Test failures in ITestAzureBlobFileSystemCheckAccess in ABFS > > > Key: HADOOP-17203 > URL: https://issues.apache.org/jira/browse/HADOOP-17203 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Mehakmeet Singh >Assignee: Bilahari T H >Priority: Major > Labels: abfsactive > > ITestAzureBlobFileSystemCheckAccess is giving test failures while running > both in parallel as well as in stand-alone(in IDE). > Tested by: mvn -T 1C -Dparallel-tests=abfs clean verify > Region: East US -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16254) Add proxy address in IPC connection
[ https://issues.apache.org/jira/browse/HADOOP-16254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195299#comment-17195299 ] zhengchenyu commented on HADOOP-16254: -- long time so that forget it. I repair HADOOP-16254.004.patch, success in our test cluster for long time. Here I submit HADOOP-16254.005.patch. Note: HADOOP-16254.005.patch contain some code which solve the problem HDFS-13248 mentioned. Maybe will remove in later version. > Add proxy address in IPC connection > --- > > Key: HADOOP-16254 > URL: https://issues.apache.org/jira/browse/HADOOP-16254 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HADOOP-16254.001.patch, HADOOP-16254.002.patch, > HADOOP-16254.004.patch > > > In order to support data locality of RBF, we need to add new field about > client hostname in the RPC headers of Router protocol calls. > clientHostname represents hostname of client and forward by Router to > Namenode to support data locality friendly. See more [RBF Data Locality > Design|https://issues.apache.org/jira/secure/attachment/12965092/RBF%20Data%20Locality%20Design.pdf] > in HDFS-13248 and [maillist > vote|http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201904.mbox/%3CCAF3Ajax7hGxvowg4K_HVTZeDqC5H=3bfb7mv5sz5mgvadhv...@mail.gmail.com%3E]. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HADOOP-16254) Add proxy address in IPC connection
[ https://issues.apache.org/jira/browse/HADOOP-16254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengchenyu updated HADOOP-16254: - Comment: was deleted (was: long time so that forget it. I repair HADOOP-16254.004.patch, success in our test cluster for long time. Here I submit HADOOP-16254.005.patch. Note: HADOOP-16254.005.patch contain some code which solve the problem HDFS-13248 mentioned. Maybe will remove in later version. ) > Add proxy address in IPC connection > --- > > Key: HADOOP-16254 > URL: https://issues.apache.org/jira/browse/HADOOP-16254 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HADOOP-16254.001.patch, HADOOP-16254.002.patch, > HADOOP-16254.004.patch > > > In order to support data locality of RBF, we need to add new field about > client hostname in the RPC headers of Router protocol calls. > clientHostname represents hostname of client and forward by Router to > Namenode to support data locality friendly. See more [RBF Data Locality > Design|https://issues.apache.org/jira/secure/attachment/12965092/RBF%20Data%20Locality%20Design.pdf] > in HDFS-13248 and [maillist > vote|http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201904.mbox/%3CCAF3Ajax7hGxvowg4K_HVTZeDqC5H=3bfb7mv5sz5mgvadhv...@mail.gmail.com%3E]. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #2246: Hadoop-17215. ABFS: Disable default create overwrite
snvijaya commented on a change in pull request #2246:
URL: https://github.com/apache/hadoop/pull/2246#discussion_r480968643
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##
@@ -271,10 +272,67 @@ public AbfsRestOperation deleteFilesystem() throws
AzureBlobFileSystemException
return op;
}
- public AbfsRestOperation createPath(final String path, final boolean isFile,
final boolean overwrite,
- final String permission, final String
umask,
- final boolean isAppendBlob) throws
AzureBlobFileSystemException {
+ public AbfsRestOperation createPath(final String path,
+ final boolean isFile,
+ final boolean overwrite,
+ final String permission,
+ final String umask,
+ final boolean isAppendBlob) throws AzureBlobFileSystemException {
+String operation = isFile
+? SASTokenProvider.CREATE_FILE_OPERATION
+: SASTokenProvider.CREATE_DIRECTORY_OPERATION;
+
+// HDFS FS defaults overwrite behaviour to true for create file which leads
+// to majority create API traffic with overwrite=true. In some cases, this
+// will end in race conditions at backend with parallel operations issued
to
+// same path either by means of the customer workload or ABFS driver retry.
+// Disabling the create overwrite default setting to false should
+// significantly reduce the chances for such race conditions.
+boolean isFirstAttemptToCreateWithoutOverwrite = false;
+if (isFile && overwrite
+&& abfsConfiguration.isDefaultCreateOverwriteDisabled()) {
+ isFirstAttemptToCreateWithoutOverwrite = true;
+}
+
+AbfsRestOperation op = null;
+// Query builder
+final AbfsUriQueryBuilder abfsUriQueryBuilder =
createDefaultUriQueryBuilder();
+abfsUriQueryBuilder.addQuery(QUERY_PARAM_RESOURCE,
+operation.equals(SASTokenProvider.CREATE_FILE_OPERATION)
+? FILE
+: DIRECTORY);
+if (isAppendBlob) {
+ abfsUriQueryBuilder.addQuery(QUERY_PARAM_BLOBTYPE, APPEND_BLOB_TYPE);
+}
+
+appendSASTokenToQuery(path, operation, abfsUriQueryBuilder);
+
+try {
+ op = createPathImpl(path, abfsUriQueryBuilder,
+ (isFirstAttemptToCreateWithoutOverwrite ? false : overwrite),
+ permission, umask);
+} catch (AbfsRestOperationException e) {
+ if ((e.getStatusCode() == HttpURLConnection.HTTP_CONFLICT)
+ && isFirstAttemptToCreateWithoutOverwrite) {
+isFirstAttemptToCreateWithoutOverwrite = false;
+// was a first attempt made to create without overwrite. Now try again
+// with overwrite now.
+op = createPathImpl(path, abfsUriQueryBuilder, true, permission,
umask);
Review comment:
PR has been modified to include the step to fetch eTag to ensure that
the create overwrite=true retry case is handled too.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17244) HADOOP-17244. S3A directory delete tombstones dir markers prematurely.
[
https://issues.apache.org/jira/browse/HADOOP-17244?focusedWorklogId=483879&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-483879
]
ASF GitHub Bot logged work on HADOOP-17244:
---
Author: ASF GitHub Bot
Created on: 14/Sep/20 09:16
Start Date: 14/Sep/20 09:16
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #2280:
URL: https://github.com/apache/hadoop/pull/2280#issuecomment-691930699
This patch is incomplete. Reverted and will create a new one.
Issues
* cleanup on a multipart fail will tombstone everything; no information at
that point as to what is/isn't a dir
* delete needs more coverage
I think I may want to push this delete/no-delete logic into s3guard -it can
do the check rather than relying on the layer above. But: we will need to pass
down for each path whether a path is a dir or a file, so that s3guard doesn't
do a DDB query on every single path entry to work that out for itself
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 483879)
Time Spent: 1.5h (was: 1h 20m)
> HADOOP-17244. S3A directory delete tombstones dir markers prematurely.
> --
>
> Key: HADOOP-17244
> URL: https://issues.apache.org/jira/browse/HADOOP-17244
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 3.3.1
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Test failure:
> {{ITestS3AFileContextMainOperations#testRenameDirectoryAsNonExistentDirectory}}
> This is repeatable on -Dauth runs (we haven't been running them, have we?)
> Either its from the recent dir marker changes (initial hypothesis) or its
> been lurking a while and not been picked up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2280: HADOOP-17244. S3A directory delete tombstones dir markers prematurely.
steveloughran commented on pull request #2280: URL: https://github.com/apache/hadoop/pull/2280#issuecomment-691930699 This patch is incomplete. Reverted and will create a new one. Issues * cleanup on a multipart fail will tombstone everything; no information at that point as to what is/isn't a dir * delete needs more coverage I think I may want to push this delete/no-delete logic into s3guard -it can do the check rather than relying on the layer above. But: we will need to pass down for each path whether a path is a dir or a file, so that s3guard doesn't do a DDB query on every single path entry to work that out for itself This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tasanuma opened a new pull request #2302: HDFS-15577. Refactor TestTracing.
tasanuma opened a new pull request #2302: URL: https://github.com/apache/hadoop/pull/2302 ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.) For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16211) Update guava to 27.0-jre in hadoop-project branch-3.2
[ https://issues.apache.org/jira/browse/HADOOP-16211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195408#comment-17195408 ] Ayush Saxena commented on HADOOP-16211: --- Yahh, Its already done in 3.2.1, so I think we don't have anything that we can do with 3.2.1, Reverting this in 3.2.2 would worsen things only. [~gabor.bota] [~mackrorysd] Why wasn't this marked incompatible? > Update guava to 27.0-jre in hadoop-project branch-3.2 > - > > Key: HADOOP-16211 > URL: https://issues.apache.org/jira/browse/HADOOP-16211 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.2.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > Fix For: 3.2.1 > > Attachments: HADOOP-16211-branch-3.2.001.patch, > HADOOP-16211-branch-3.2.002.patch, HADOOP-16211-branch-3.2.003.patch, > HADOOP-16211-branch-3.2.004.patch, HADOOP-16211-branch-3.2.005.patch, > HADOOP-16211-branch-3.2.006.patch > > > com.google.guava:guava should be upgraded to 27.0-jre due to new CVE's found > CVE-2018-10237. > This is a sub-task for branch-3.2 from HADOOP-15960 to track issues on that > particular branch. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16281) ABFS: Rename operation, GetFileStatus before rename operation and throw exception on the driver side
[ https://issues.apache.org/jira/browse/HADOOP-16281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16281: Target Version/s: 3.4.0 (was: 3.2.2) > ABFS: Rename operation, GetFileStatus before rename operation and throw > exception on the driver side > - > > Key: HADOOP-16281 > URL: https://issues.apache.org/jira/browse/HADOOP-16281 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Da Zhou >Assignee: Steve Loughran >Priority: Major > Labels: abfsactive > > ABFS should add the rename with options: > [https://github.com/apache/hadoop/pull/743] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16281) ABFS: Rename operation, GetFileStatus before rename operation and throw exception on the driver side
[ https://issues.apache.org/jira/browse/HADOOP-16281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195418#comment-17195418 ] Steve Loughran commented on HADOOP-16281: - moved to 3.4.x > ABFS: Rename operation, GetFileStatus before rename operation and throw > exception on the driver side > - > > Key: HADOOP-16281 > URL: https://issues.apache.org/jira/browse/HADOOP-16281 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Da Zhou >Assignee: Steve Loughran >Priority: Major > Labels: abfsactive > > ABFS should add the rename with options: > [https://github.com/apache/hadoop/pull/743] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-11452) Make FileSystem.rename(path, path, options) public, specified, tested
[
https://issues.apache.org/jira/browse/HADOOP-11452?focusedWorklogId=483957&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-483957
]
ASF GitHub Bot logged work on HADOOP-11452:
---
Author: ASF GitHub Bot
Created on: 14/Sep/20 12:08
Start Date: 14/Sep/20 12:08
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #743:
URL: https://github.com/apache/hadoop/pull/743#issuecomment-692010404
as per Daryn's comments in :
https://issues.apache.org/jira/browse/HADOOP-16281?focusedCommentId=16832628&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16832628
No changes to HDFS error messages. Review this
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 483957)
Time Spent: 2h 20m (was: 2h 10m)
> Make FileSystem.rename(path, path, options) public, specified, tested
> -
>
> Key: HADOOP-11452
> URL: https://issues.apache.org/jira/browse/HADOOP-11452
> Project: Hadoop Common
> Issue Type: Task
> Components: fs
>Affects Versions: 2.7.3
>Reporter: Yi Liu
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-11452-001.patch, HADOOP-11452-002.patch,
> HADOOP-14452-004.patch, HADOOP-14452-branch-2-003.patch
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Currently in {{FileSystem}}, {{rename}} with _Rename options_ is protected
> and with _deprecated_ annotation. And the default implementation is not
> atomic.
> So this method is not able to be used outside. On the other hand, HDFS has a
> good and atomic implementation. (Also an interesting thing in {{DFSClient}},
> the _deprecated_ annotations for these two methods are opposite).
> It makes sense to make public for {{rename}} with _Rename options_, since
> it's atomic for rename+overwrite, also it saves RPC calls if user desires
> rename+overwrite.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #743: HADOOP-11452 make rename/3 public
steveloughran commented on pull request #743: URL: https://github.com/apache/hadoop/pull/743#issuecomment-692010404 as per Daryn's comments in : https://issues.apache.org/jira/browse/HADOOP-16281?focusedCommentId=16832628&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16832628 No changes to HDFS error messages. Review this This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16083) DistCp shouldn't always overwrite the target file when checksums match
[
https://issues.apache.org/jira/browse/HADOOP-16083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-16083:
Target Version/s: 3.3.1 (was: 3.2.3)
> DistCp shouldn't always overwrite the target file when checksums match
> --
>
> Key: HADOOP-16083
> URL: https://issues.apache.org/jira/browse/HADOOP-16083
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Affects Versions: 3.2.0, 3.1.1, 3.3.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HADOOP-16083.001.patch
>
>
> {code:java|title=CopyMapper#setup}
> ...
> try {
> overWrite = overWrite ||
> targetFS.getFileStatus(targetFinalPath).isFile();
> } catch (FileNotFoundException ignored) {
> }
> ...
> {code}
> The above code overrides config key "overWrite" to "true" when the target
> path is a file. Therefore, unnecessary transfer happens when the source and
> target file have the same checksums.
> My suggestion is: remove the code above. If the user insists to overwrite,
> just add -overwrite in the options:
> {code:bash|title=DistCp command with -overwrite option}
> hadoop distcp -overwrite hdfs://localhost:64464/source/5/6.txt
> hdfs://localhost:64464/target/5/6.txt
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16259) Distcp to set S3 Storage Class
[ https://issues.apache.org/jira/browse/HADOOP-16259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16259: Target Version/s: (was: 3.2.3) > Distcp to set S3 Storage Class > -- > > Key: HADOOP-16259 > URL: https://issues.apache.org/jira/browse/HADOOP-16259 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3, tools/distcp >Affects Versions: 2.8.4 >Reporter: Prakash Gopalsamy >Priority: Minor > Attachments: ENHANCE_HADOOP_DISTCP_FOR_CUSTOM_S3_STORAGE_CLASS.docx > > Original Estimate: 168h > Remaining Estimate: 168h > > Hadoop distcp implementation doesn’t have properties to override Storage > class while transferring data to Amazon S3 storage. Hadoop distcp doesn’t set > any storage class while transferring data to Amazon S3 storage. Due to this > all the objects moved from cluster to S3 using Hadoop Distcp are been stored > in the default storage class “STANDARD”. By providing a new feature to > override the default S3 storage class through configuration properties will > be helpful to upload objects in other storage classes. I have come up with a > design to implement this feature in a design document and uploaded the same > in the JIRA. Kindly review and let me know for your suggestions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16259) Distcp to set S3 Storage Class
[ https://issues.apache.org/jira/browse/HADOOP-16259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16259: Parent: HADOOP-16829 Issue Type: Sub-task (was: New Feature) > Distcp to set S3 Storage Class > -- > > Key: HADOOP-16259 > URL: https://issues.apache.org/jira/browse/HADOOP-16259 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, tools/distcp >Affects Versions: 2.8.4 >Reporter: Prakash Gopalsamy >Priority: Minor > Attachments: ENHANCE_HADOOP_DISTCP_FOR_CUSTOM_S3_STORAGE_CLASS.docx > > Original Estimate: 168h > Remaining Estimate: 168h > > Hadoop distcp implementation doesn’t have properties to override Storage > class while transferring data to Amazon S3 storage. Hadoop distcp doesn’t set > any storage class while transferring data to Amazon S3 storage. Due to this > all the objects moved from cluster to S3 using Hadoop Distcp are been stored > in the default storage class “STANDARD”. By providing a new feature to > override the default S3 storage class through configuration properties will > be helpful to upload objects in other storage classes. I have come up with a > design to implement this feature in a design document and uploaded the same > in the JIRA. Kindly review and let me know for your suggestions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16211) Update guava to 27.0-jre in hadoop-project branch-3.2
[ https://issues.apache.org/jira/browse/HADOOP-16211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195425#comment-17195425 ] Steve Loughran commented on HADOOP-16211: - we don't have a choice about this update, security reasons force us > Update guava to 27.0-jre in hadoop-project branch-3.2 > - > > Key: HADOOP-16211 > URL: https://issues.apache.org/jira/browse/HADOOP-16211 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.2.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > Fix For: 3.2.1 > > Attachments: HADOOP-16211-branch-3.2.001.patch, > HADOOP-16211-branch-3.2.002.patch, HADOOP-16211-branch-3.2.003.patch, > HADOOP-16211-branch-3.2.004.patch, HADOOP-16211-branch-3.2.005.patch, > HADOOP-16211-branch-3.2.006.patch > > > com.google.guava:guava should be upgraded to 27.0-jre due to new CVE's found > CVE-2018-10237. > This is a sub-task for branch-3.2 from HADOOP-15960 to track issues on that > particular branch. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16317) ABFS: improve random read performance
[ https://issues.apache.org/jira/browse/HADOOP-16317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16317: Target Version/s: (was: 3.2.3) > ABFS: improve random read performance > - > > Key: HADOOP-16317 > URL: https://issues.apache.org/jira/browse/HADOOP-16317 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Da Zhou >Priority: Major > Labels: abfsactive > > Improving random read performance is an interesting topic. ABFS doesn't > perform well when reading column format files as the process involves with > many seek operations which make the readAhead no use, and if readAhead is > used unwisely it would lead to unnecessary data request. > Hence creating this Jira as a reminder to track the investigation and > progress of the work. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17258) MagicS3GuardCommitter fails with `pendingset` already exists
[
https://issues.apache.org/jira/browse/HADOOP-17258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195429#comment-17195429
]
Steve Loughran commented on HADOOP-17258:
-
That failure implies that attempt 1 succeeded & created the file, so TA 2
couldn't. Maybe the thing to do here is on that second write, say "the file is
there, so let's gracefully abort our commit, relying on attempt #1 to be
correct. Then it should abort its uploads and return success
Makes sense?
What happens to the whole job? Does it fail?
(FWIW, failures during task commit is something the s3a committers MUST be
resilient to. Spark expects commits to be atomic and exactly one attempt to
succeed, but it doesn't care which (more specifically: your code mustn't care)
> MagicS3GuardCommitter fails with `pendingset` already exists
>
>
> Key: HADOOP-17258
> URL: https://issues.apache.org/jira/browse/HADOOP-17258
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Dongjoon Hyun
>Priority: Major
>
> In `trunk/branch-3.3/branch-3.2`, `MagicS3GuardCommitter.innerCommitTask` has
> `false` at `pendingSet.save`.
> {code}
> try {
> pendingSet.save(getDestFS(), taskOutcomePath, false);
> } catch (IOException e) {
> LOG.warn("Failed to save task commit data to {} ",
> taskOutcomePath, e);
> abortPendingUploads(context, pendingSet.getCommits(), true);
> throw e;
> }
> {code}
> And, it can cause a job failure like the following.
> {code}
> WARN TaskSetManager: Lost task 1562.1 in stage 1.0 (TID 1788, 100.92.11.63,
> executor 26): org.apache.spark.SparkException: Task failed while writing rows.
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:257)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
> at org.apache.spark.scheduler.Task.run(Task.scala:123)
> at
> org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> at java.base/java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.hadoop.fs.FileAlreadyExistsException:
> s3a://xxx/__magic/app-attempt-/task_20200911063607_0001_m_001562.pendingset
> already exists
> at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:761)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1118)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1098)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:987)
> at
> org.apache.hadoop.util.JsonSerialization.save(JsonSerialization.java:269)
> at
> org.apache.hadoop.fs.s3a.commit.files.PendingSet.save(PendingSet.java:170)
> at
> org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter.innerCommitTask(MagicS3GuardCommitter.java:220)
> at
> org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter.commitTask(MagicS3GuardCommitter.java:165)
> at
> org.apache.spark.mapred.SparkHadoopMapRedUtil$.performCommit$1(SparkHadoopMapRedUtil.scala:50)
> at
> org.apache.spark.mapred.SparkHadoopMapRedUtil$.commitTask(SparkHadoopMapRedUtil.scala:77)
> at
> org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitTask(HadoopMapReduceCommitProtocol.scala:244)
> at
> org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:78)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:247)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:242)
> {code}
> {code}
> 20/09/11 07:44:38 ERROR TaskSetManager: Task 957.1 in stage 1.0 (TID 1412)
> can not write to output file:
> org.apache.hadoop.fs.FileAlreadyExistsException:
> s3a://xxx/t/__magic/app-attempt-/task_20200911073922_0001_m_000957.pendingset
> already exists; not retrying
> {code}
--
This message was sent by Atlassian Jira
[GitHub] [hadoop] snvijaya commented on pull request #2246: Hadoop-17215. ABFS: Disable default create overwrite
snvijaya commented on pull request #2246: URL: https://github.com/apache/hadoop/pull/2246#issuecomment-692028113 Non-HNS account with SharedKey `[INFO] Results: [INFO] [INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 246 [WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 24` Non-HNS account with OAuth `[INFO] Results: [INFO] [INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 250 WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 140` HNS account with SharedKey `[INFO] Results: [INFO] [INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 41 [WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 24` HNS account with OAuth `[INFO] Results: [INFO] [INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 74 [WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 140` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya edited a comment on pull request #2246: Hadoop-17215. ABFS: Disable default create overwrite
snvijaya edited a comment on pull request #2246: URL: https://github.com/apache/hadoop/pull/2246#issuecomment-692028113 **Non-HNS account with SharedKey** `[INFO] Results: [INFO] [INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 246 [WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 24` **Non-HNS account with OAuth** `[INFO] Results: [INFO] [INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 250 WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 140` **HNS account with SharedKey** `[INFO] Results: [INFO] [INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 41 [WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 24` **HNS account with OAuth** `[INFO] Results: [INFO] [INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 74 [WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 140` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya edited a comment on pull request #2246: Hadoop-17215. ABFS: Disable default create overwrite
snvijaya edited a comment on pull request #2246: URL: https://github.com/apache/hadoop/pull/2246#issuecomment-692028113 Non-HNS account with SharedKey `[INFO] Results: [INFO] [INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 246 [WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 24` Non-HNS account with OAuth `[INFO] Results: [INFO] [INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 250 WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 140` HNS account with SharedKey `[INFO] Results: [INFO] [INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 41 [WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 24` HNS account with OAuth `[INFO] Results: [INFO] [INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 74 [WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 140` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16211) Update guava to 27.0-jre in hadoop-project branch-3.2
[ https://issues.apache.org/jira/browse/HADOOP-16211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195447#comment-17195447 ] Gabor Bota commented on HADOOP-16211: - Sorry [~ayushtkn], I hope that did not do any damage, and it can be marked as incompatible now > Update guava to 27.0-jre in hadoop-project branch-3.2 > - > > Key: HADOOP-16211 > URL: https://issues.apache.org/jira/browse/HADOOP-16211 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.2.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > Fix For: 3.2.1 > > Attachments: HADOOP-16211-branch-3.2.001.patch, > HADOOP-16211-branch-3.2.002.patch, HADOOP-16211-branch-3.2.003.patch, > HADOOP-16211-branch-3.2.004.patch, HADOOP-16211-branch-3.2.005.patch, > HADOOP-16211-branch-3.2.006.patch > > > com.google.guava:guava should be upgraded to 27.0-jre due to new CVE's found > CVE-2018-10237. > This is a sub-task for branch-3.2 from HADOOP-15960 to track issues on that > particular branch. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16211) Update guava to 27.0-jre in hadoop-project branch-3.2
[ https://issues.apache.org/jira/browse/HADOOP-16211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-16211: Labels: incompatible (was: ) > Update guava to 27.0-jre in hadoop-project branch-3.2 > - > > Key: HADOOP-16211 > URL: https://issues.apache.org/jira/browse/HADOOP-16211 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.2.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > Labels: incompatible > Fix For: 3.2.1 > > Attachments: HADOOP-16211-branch-3.2.001.patch, > HADOOP-16211-branch-3.2.002.patch, HADOOP-16211-branch-3.2.003.patch, > HADOOP-16211-branch-3.2.004.patch, HADOOP-16211-branch-3.2.005.patch, > HADOOP-16211-branch-3.2.006.patch > > > com.google.guava:guava should be upgraded to 27.0-jre due to new CVE's found > CVE-2018-10237. > This is a sub-task for branch-3.2 from HADOOP-15960 to track issues on that > particular branch. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15538) Possible RPC deadlock in Client
[
https://issues.apache.org/jira/browse/HADOOP-15538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195549#comment-17195549
]
Wei-Chiu Chuang commented on HADOOP-15538:
--
Thanks [~abaldocchi]! good to know that piece of information. We are mostly on
JDK8 so it'll be interested to see if it reproduces after JDK8u251.
> Possible RPC deadlock in Client
> ---
>
> Key: HADOOP-15538
> URL: https://issues.apache.org/jira/browse/HADOOP-15538
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Major
> Attachments: t1+13min.jstack, t1.jstack
>
>
> We have a jstack collection that spans 13 minutes. One frame per ~1.5
> minutes. And for each of the frame, I observed the following:
> {code:java}
> Found one Java-level deadlock:
> =
> "IPC Parameter Sending Thread #294":
> waiting to lock monitor 0x7f68f21f3188 (object 0x000621745390, a
> java.lang.Object),
> which is held by UNKNOWN_owner_addr=0x7f68332e2800
> Java stack information for the threads listed above:
> ===
> "IPC Parameter Sending Thread #294":
> at
> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:268)
> - waiting to lock <0x000621745390> (a java.lang.Object)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:461)
> - locked <0x000621745380> (a java.lang.Object)
> at
> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
> at
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
> at
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> - locked <0x000621749850> (a java.io.BufferedOutputStream)
> at java.io.DataOutputStream.flush(DataOutputStream.java:123)
> at org.apache.hadoop.ipc.Client$Connection$3.run(Client.java:1072)
> - locked <0x00062174b878> (a java.io.DataOutputStream)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Found one Java-level deadlock:
> =
> "IPC Client (297602875) connection to x.y.z.p:8020 from impala":
> waiting to lock monitor 0x7f68f21f3188 (object 0x000621745390, a
> java.lang.Object),
> which is held by UNKNOWN_owner_addr=0x7f68332e2800
> Java stack information for the threads listed above:
> ===
> "IPC Client (297602875) connection to x.y.z.p:8020 from impala":
> at
> sun.nio.ch.SocketChannelImpl.readerCleanup(SocketChannelImpl.java:279)
> - waiting to lock <0x000621745390> (a java.lang.Object)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:390)
> - locked <0x000621745370> (a java.lang.Object)
> at
> org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
> at
> org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:553)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
> - locked <0x0006217476f0> (a java.io.BufferedInputStream)
> at java.io.DataInputStream.readInt(DataInputStream.java:387)
> at
> org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1113)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1006)
> Found 2 deadlocks.
> {code}
> This happens with jdk1.8.0_162 on 2.6.32-696.18.7.el6.x86_64.
> The code appears to match
> [https://github.com/tuxjdk/jdk8u/blob/master/jdk/src/share/classes/sun/nio/ch/SocketChannelImpl.java].
>
[jira] [Commented] (HADOOP-15538) Possible RPC deadlock in Client
[
https://issues.apache.org/jira/browse/HADOOP-15538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195548#comment-17195548
]
Wei-Chiu Chuang commented on HADOOP-15538:
--
bq. Hi Wei-Chiu Chuang, I met the same bug in Impala 3.2.0. An exception
"hdfsOpenFile() for hdfs://xxx failed to finish before the 300 second timeout"
is thrown when querying. You say after bumped up the heap setings of the Impala
daemons on the cluster, it will not happen. What I want to know is that which
parameter in the configuration you bumped up. Thank you.
I believe it is configured by JAVA_TOOL_OPTIONS="-Xmx16g"
You can increase it to 32g, or even more.
If you use CM, it's in the "Impala Daemon Environment Advanced Configuration
Snippet (Safety Valve)"
> Possible RPC deadlock in Client
> ---
>
> Key: HADOOP-15538
> URL: https://issues.apache.org/jira/browse/HADOOP-15538
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Major
> Attachments: t1+13min.jstack, t1.jstack
>
>
> We have a jstack collection that spans 13 minutes. One frame per ~1.5
> minutes. And for each of the frame, I observed the following:
> {code:java}
> Found one Java-level deadlock:
> =
> "IPC Parameter Sending Thread #294":
> waiting to lock monitor 0x7f68f21f3188 (object 0x000621745390, a
> java.lang.Object),
> which is held by UNKNOWN_owner_addr=0x7f68332e2800
> Java stack information for the threads listed above:
> ===
> "IPC Parameter Sending Thread #294":
> at
> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:268)
> - waiting to lock <0x000621745390> (a java.lang.Object)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:461)
> - locked <0x000621745380> (a java.lang.Object)
> at
> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
> at
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
> at
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> - locked <0x000621749850> (a java.io.BufferedOutputStream)
> at java.io.DataOutputStream.flush(DataOutputStream.java:123)
> at org.apache.hadoop.ipc.Client$Connection$3.run(Client.java:1072)
> - locked <0x00062174b878> (a java.io.DataOutputStream)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Found one Java-level deadlock:
> =
> "IPC Client (297602875) connection to x.y.z.p:8020 from impala":
> waiting to lock monitor 0x7f68f21f3188 (object 0x000621745390, a
> java.lang.Object),
> which is held by UNKNOWN_owner_addr=0x7f68332e2800
> Java stack information for the threads listed above:
> ===
> "IPC Client (297602875) connection to x.y.z.p:8020 from impala":
> at
> sun.nio.ch.SocketChannelImpl.readerCleanup(SocketChannelImpl.java:279)
> - waiting to lock <0x000621745390> (a java.lang.Object)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:390)
> - locked <0x000621745370> (a java.lang.Object)
> at
> org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
> at
> org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:553)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
> - locked <0x0006217476f0> (a java.io.BufferedInputStream)
> at java.io.DataInputStream.rea
[GitHub] [hadoop] viirya commented on pull request #2297: HADOOP-17125. Using snappy-java in SnappyCodec
viirya commented on pull request #2297: URL: https://github.com/apache/hadoop/pull/2297#issuecomment-692157442 It seems still failed to fetch and install yetus, and not just this PR, other PRs also encountered it... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17125) Using snappy-java in SnappyCodec
[ https://issues.apache.org/jira/browse/HADOOP-17125?focusedWorklogId=484081&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-484081 ] ASF GitHub Bot logged work on HADOOP-17125: --- Author: ASF GitHub Bot Created on: 14/Sep/20 16:08 Start Date: 14/Sep/20 16:08 Worklog Time Spent: 10m Work Description: viirya commented on pull request #2297: URL: https://github.com/apache/hadoop/pull/2297#issuecomment-692157442 It seems still failed to fetch and install yetus, and not just this PR, other PRs also encountered it... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 484081) Time Spent: 8h 20m (was: 8h 10m) > Using snappy-java in SnappyCodec > > > Key: HADOOP-17125 > URL: https://issues.apache.org/jira/browse/HADOOP-17125 > Project: Hadoop Common > Issue Type: New Feature > Components: common >Affects Versions: 3.3.0 >Reporter: DB Tsai >Priority: Major > Labels: pull-request-available > Time Spent: 8h 20m > Remaining Estimate: 0h > > In Hadoop, we use native libs for snappy codec which has several > disadvantages: > * It requires native *libhadoop* and *libsnappy* to be installed in system > *LD_LIBRARY_PATH*, and they have to be installed separately on each node of > the clusters, container images, or local test environments which adds huge > complexities from deployment point of view. In some environments, it requires > compiling the natives from sources which is non-trivial. Also, this approach > is platform dependent; the binary may not work in different platform, so it > requires recompilation. > * It requires extra configuration of *java.library.path* to load the > natives, and it results higher application deployment and maintenance cost > for users. > Projects such as *Spark* and *Parquet* use > [snappy-java|[https://github.com/xerial/snappy-java]] which is JNI-based > implementation. It contains native binaries for Linux, Mac, and IBM in jar > file, and it can automatically load the native binaries into JVM from jar > without any setup. If a native implementation can not be found for a > platform, it can fallback to pure-java implementation of snappy based on > [aircompressor|[https://github.com/airlift/aircompressor/tree/master/src/main/java/io/airlift/compress/snappy]]. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17125) Using snappy-java in SnappyCodec
[ https://issues.apache.org/jira/browse/HADOOP-17125?focusedWorklogId=484088&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-484088 ] ASF GitHub Bot logged work on HADOOP-17125: --- Author: ASF GitHub Bot Created on: 14/Sep/20 16:25 Start Date: 14/Sep/20 16:25 Worklog Time Spent: 10m Work Description: viirya commented on pull request #2297: URL: https://github.com/apache/hadoop/pull/2297#issuecomment-692167279 @sunchao Who we should let them know about the CI issue? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 484088) Time Spent: 8.5h (was: 8h 20m) > Using snappy-java in SnappyCodec > > > Key: HADOOP-17125 > URL: https://issues.apache.org/jira/browse/HADOOP-17125 > Project: Hadoop Common > Issue Type: New Feature > Components: common >Affects Versions: 3.3.0 >Reporter: DB Tsai >Priority: Major > Labels: pull-request-available > Time Spent: 8.5h > Remaining Estimate: 0h > > In Hadoop, we use native libs for snappy codec which has several > disadvantages: > * It requires native *libhadoop* and *libsnappy* to be installed in system > *LD_LIBRARY_PATH*, and they have to be installed separately on each node of > the clusters, container images, or local test environments which adds huge > complexities from deployment point of view. In some environments, it requires > compiling the natives from sources which is non-trivial. Also, this approach > is platform dependent; the binary may not work in different platform, so it > requires recompilation. > * It requires extra configuration of *java.library.path* to load the > natives, and it results higher application deployment and maintenance cost > for users. > Projects such as *Spark* and *Parquet* use > [snappy-java|[https://github.com/xerial/snappy-java]] which is JNI-based > implementation. It contains native binaries for Linux, Mac, and IBM in jar > file, and it can automatically load the native binaries into JVM from jar > without any setup. If a native implementation can not be found for a > platform, it can fallback to pure-java implementation of snappy based on > [aircompressor|[https://github.com/airlift/aircompressor/tree/master/src/main/java/io/airlift/compress/snappy]]. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] viirya commented on pull request #2297: HADOOP-17125. Using snappy-java in SnappyCodec
viirya commented on pull request #2297: URL: https://github.com/apache/hadoop/pull/2297#issuecomment-692167279 @sunchao Who we should let them know about the CI issue? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17125) Using snappy-java in SnappyCodec
[ https://issues.apache.org/jira/browse/HADOOP-17125?focusedWorklogId=484100&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-484100 ] ASF GitHub Bot logged work on HADOOP-17125: --- Author: ASF GitHub Bot Created on: 14/Sep/20 16:39 Start Date: 14/Sep/20 16:39 Worklog Time Spent: 10m Work Description: sunchao commented on pull request #2297: URL: https://github.com/apache/hadoop/pull/2297#issuecomment-692174719 @viirya interesting ... I think you can send an email to the Hadoop dev list (`common-...@hadoop.apache.org`, you may need to subscribe first). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 484100) Time Spent: 8h 40m (was: 8.5h) > Using snappy-java in SnappyCodec > > > Key: HADOOP-17125 > URL: https://issues.apache.org/jira/browse/HADOOP-17125 > Project: Hadoop Common > Issue Type: New Feature > Components: common >Affects Versions: 3.3.0 >Reporter: DB Tsai >Priority: Major > Labels: pull-request-available > Time Spent: 8h 40m > Remaining Estimate: 0h > > In Hadoop, we use native libs for snappy codec which has several > disadvantages: > * It requires native *libhadoop* and *libsnappy* to be installed in system > *LD_LIBRARY_PATH*, and they have to be installed separately on each node of > the clusters, container images, or local test environments which adds huge > complexities from deployment point of view. In some environments, it requires > compiling the natives from sources which is non-trivial. Also, this approach > is platform dependent; the binary may not work in different platform, so it > requires recompilation. > * It requires extra configuration of *java.library.path* to load the > natives, and it results higher application deployment and maintenance cost > for users. > Projects such as *Spark* and *Parquet* use > [snappy-java|[https://github.com/xerial/snappy-java]] which is JNI-based > implementation. It contains native binaries for Linux, Mac, and IBM in jar > file, and it can automatically load the native binaries into JVM from jar > without any setup. If a native implementation can not be found for a > platform, it can fallback to pure-java implementation of snappy based on > [aircompressor|[https://github.com/airlift/aircompressor/tree/master/src/main/java/io/airlift/compress/snappy]]. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on pull request #2297: HADOOP-17125. Using snappy-java in SnappyCodec
sunchao commented on pull request #2297: URL: https://github.com/apache/hadoop/pull/2297#issuecomment-692174719 @viirya interesting ... I think you can send an email to the Hadoop dev list (`common-...@hadoop.apache.org`, you may need to subscribe first). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #2296: HDFS-15568. namenode start failed to start when dfs.namenode.max.snapshot.limit set.
goiri commented on a change in pull request #2296:
URL: https://github.com/apache/hadoop/pull/2296#discussion_r488109432
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotManager.java
##
@@ -133,4 +138,57 @@ public void testValidateSnapshotIDWidth() throws Exception
{
getMaxSnapshotID() < Snapshot.CURRENT_STATE_ID);
}
+ @Test
+ public void testSnapshotLimitOnRestart() throws Exception {
+final Configuration conf = new Configuration();
+final Path snapshottableDir
+= new Path("/" + getClass().getSimpleName());
+int numSnapshots = 5;
+conf.setInt(DFSConfigKeys.
+DFS_NAMENODE_SNAPSHOT_MAX_LIMIT, numSnapshots);
+conf.setInt(DFSConfigKeys.DFS_NAMENODE_SNAPSHOT_FILESYSTEM_LIMIT,
+numSnapshots * 2);
+MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).
+numDataNodes(0).build();
+cluster.waitActive();
+DistributedFileSystem hdfs = cluster.getFileSystem();
+hdfs.mkdirs(snapshottableDir);
+hdfs.allowSnapshot(snapshottableDir);
+for (int i = 0; i < numSnapshots; i++) {
+ hdfs.createSnapshot(snapshottableDir, "s" + i);
+}
+LambdaTestUtils.intercept(SnapshotException.class,
+"snapshot limit",
+() -> hdfs.createSnapshot(snapshottableDir, "s5"));
+
+// now change max snapshot directory limit to 2 and restart namenode
+cluster.getNameNode().getConf().setInt(DFSConfigKeys.
+DFS_NAMENODE_SNAPSHOT_MAX_LIMIT, 2);
+cluster.restartNameNodes();
+SnapshotManager snapshotManager = cluster.getNamesystem().
+getSnapshotManager();
+
+// make sure edits of all previous 5 create snapshots are replayed
+Assert.assertEquals(numSnapshots, snapshotManager.getNumSnapshots());
+
+// make sure namenode has the new snapshot limit configured as 2
+Assert.assertEquals(2, snapshotManager.getMaxSnapshotLimit());
+
+// Any new snapshot creation should still fail
+LambdaTestUtils.intercept(SnapshotException.class,
+"snapshot limit", () -> hdfs.createSnapshot(snapshottableDir, "s5"));
+// now change max snapshot FS limit to 2 and restart namenode
+cluster.getNameNode().getConf().setInt(DFSConfigKeys.
+DFS_NAMENODE_SNAPSHOT_FILESYSTEM_LIMIT, 2);
+cluster.restartNameNodes();
+snapshotManager = cluster.getNamesystem().
+getSnapshotManager();
+// make sure edits of all previous 5 create snapshots are replayed
+Assert.assertEquals(numSnapshots, snapshotManager.getNumSnapshots());
+
+// make sure namenode has the new snapshot limit configured as 2
+Assert.assertEquals(2, snapshotManager.getMaxSnapshotLimit());
+cluster.shutdown();
Review comment:
We may need to do it in a finally to make sure we always clean it. Also
check for null.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17259) Allow SSLFactory fallback to input config if ssl-*.xml fail to load from classpath
[ https://issues.apache.org/jira/browse/HADOOP-17259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195663#comment-17195663 ] Xiaoyu Yao commented on HADOOP-17259: - cc: [~weichiu] > Allow SSLFactory fallback to input config if ssl-*.xml fail to load from > classpath > -- > > Key: HADOOP-17259 > URL: https://issues.apache.org/jira/browse/HADOOP-17259 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.8.5 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > Some applications like Tez does not have ssl-client.xml and ssl-server.xml in > classpath. Instead, it directly pass the parsed SSL configuration as the > input configuration object. This ticket is opened to allow this case. > TEZ-4096 attempts to solve this issue but but take a different approach which > may not work in existing Hadoop clients that use SSLFactory from > hadoop-common. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17255) JavaKeyStoreProvider fails to create a new key if the keystore is HDFS
[ https://issues.apache.org/jira/browse/HADOOP-17255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195688#comment-17195688 ] Wei-Chiu Chuang commented on HADOOP-17255: -- So.. my impression came from this jira HADOOP-15412 where at the time, KMS wouldn't even start. Maybe things have changed so that it now starts fine but only fails after. Aside from that, there's security implication. Quoting myself in the jira: br. Even if you use a shared file system (like NFS?) you still need to make sure the network communication is authentication and encrypted. It wouldn't work if the keystore is in a HDFS encryption zone (it would end up in a recursive loop). Storing key store in unencrypted HDFS could in theory work, but transmitting unencrypted key store compromises security. > JavaKeyStoreProvider fails to create a new key if the keystore is HDFS > -- > > Key: HADOOP-17255 > URL: https://issues.apache.org/jira/browse/HADOOP-17255 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Critical > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > The caller of JavaKeyStoreProvider#renameOrFail assumes that it throws > FileNotFoundException if the src does not exist. However, > JavaKeyStoreProvider#renameOrFail calls the old rename API. In > DistributedFileSystem, the old API returns false if the src does not exist. > That way JavaKeyStoreProvider fails to create a new key if the keystore is > HDFS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17261) s3a rename() now requires s3:deleteObjectVersion permission
Steve Loughran created HADOOP-17261: --- Summary: s3a rename() now requires s3:deleteObjectVersion permission Key: HADOOP-17261 URL: https://issues.apache.org/jira/browse/HADOOP-17261 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.4.0 Reporter: Steve Loughran Assignee: Steve Loughran With the directory marker change (HADOOP-13230) you need the s3:deleteObjectVersion permission in your role, else the operation will fail in the bulk delete, *if S3Guard is in use* Root cause -if fileStatus has a versionId, we pass that in to the delete KeyVersion pair -an unguarded listing doesn't get that versionId, so this is not an issue -but if files in a directory were previously created such that S3Guard has their versionId in its tables, that is used in the request -which then fails if the caller doesn't have the permission Although we say "you need s3:delete*", this is a regression as any IAM role without the permission will have rename fail during delete -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17261) s3a rename() now requires s3:deleteObjectVersion permission
[
https://issues.apache.org/jira/browse/HADOOP-17261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195722#comment-17195722
]
Steve Loughran commented on HADOOP-17261:
-
{code}
2020-09-14 20:45:22,207 [main] DEBUG shell.Command
(Command.java:displayError(461)) - mv failure
java.nio.file.AccessDeniedException: rename s3a://stevel-london/src to
s3a://stevel-london/dest on s3a://stevel-london/src:
com.amazonaws.services.s3.model.MultiObjectDeleteException: One or more objects
could not be deleted (Service: null; Status Code: 200; Error Code: null;
Request ID: 005E717D632BA6AF; S3 Extended Request ID:
o9P5GsHwIedAdCsXvpDN6JSmHi5DvV02tW234Es2eIcrGItcEMW+su2Qcy9aIEJ2VdlmtlBLUKo=),
S3 Extended Request ID:
o9P5GsHwIedAdCsXvpDN6JSmHi5DvV02tW234Es2eIcrGItcEMW+su2Qcy9aIEJ2VdlmtlBLUKo=:null:
AccessDenied: src/file2: Access Denied
AccessDenied: src/file1: Access Denied
AccessDenied: src/file4: Access Denied
AccessDenied: src/file10: Access Denied
AccessDenied: src/file3: Access Denied
at
org.apache.hadoop.fs.s3a.impl.MultiObjectDeleteSupport.translateDeleteException(MultiObjectDeleteSupport.java:101)
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:304)
at
org.apache.hadoop.fs.s3a.s3guard.RenameTracker.convertToIOException(RenameTracker.java:267)
at
org.apache.hadoop.fs.s3a.s3guard.RenameTracker.deleteFailed(RenameTracker.java:198)
at
org.apache.hadoop.fs.s3a.impl.RenameOperation.removeSourceObjects(RenameOperation.java:695)
at
org.apache.hadoop.fs.s3a.impl.RenameOperation.completeActiveCopiesAndDeleteSources(RenameOperation.java:265)
at
org.apache.hadoop.fs.s3a.impl.RenameOperation.endOfLoopActions(RenameOperation.java:490)
at
org.apache.hadoop.fs.s3a.impl.RenameOperation.recursiveDirectoryRename(RenameOperation.java:465)
at
org.apache.hadoop.fs.s3a.impl.RenameOperation.execute(RenameOperation.java:303)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:1526)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:1376)
at
org.apache.hadoop.fs.shell.MoveCommands$Rename.processPath(MoveCommands.java:124)
at
org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:271)
at
org.apache.hadoop.fs.shell.Command.processPathInternal(Command.java:367)
at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:331)
at
org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:304)
at
org.apache.hadoop.fs.shell.CommandWithDestination.processPathArgument(CommandWithDestination.java:266)
at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:286)
at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:270)
at
org.apache.hadoop.fs.shell.CommandWithDestination.processArguments(CommandWithDestination.java:237)
at
org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:120)
at org.apache.hadoop.fs.shell.Command.run(Command.java:177)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:327)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:390)
Caused by: com.amazonaws.services.s3.model.MultiObjectDeleteException: One or
more objects could not be deleted (Service: null; Status Code: 200; Error Code:
null; Request ID: 005E717D632BA6AF; S3 Extended Request ID:
o9P5GsHwIedAdCsXvpDN6JSmHi5DvV02tW234Es2eIcrGItcEMW+su2Qcy9aIEJ2VdlmtlBLUKo=),
S3 Extended Request ID:
o9P5GsHwIedAdCsXvpDN6JSmHi5DvV02tW234Es2eIcrGItcEMW+su2Qcy9aIEJ2VdlmtlBLUKo=
at
com.amazonaws.services.s3.AmazonS3Client.deleteObjects(AmazonS3Client.java:2262)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$deleteObjects$11(S3AFileSystem.java:2162)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:407)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.deleteObjects(S3AFileSystem.java:2154)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeysS3(S3AFileSystem.java:2440)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeys(S3AFileSystem.java:2541)
at
org.apache.hadoop.fs.s3a.S3AFileSystem$OperationCallbacksImpl.removeKeys(S3AFileSystem.java:1610)
at
org.apache.hadoop.fs.s3a.impl.RenameOperation.removeSourceObjects(RenameOperation.java:680)
... 21 more
{code}
> s3a rename() now requires s3:deleteObjectVersion permission
> ---
>
> Key: HADOOP-17261
> URL: https://issues.apache.org/jira/browse/HADOOP-17261
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.
[jira] [Updated] (HADOOP-17261) s3a rename() now requires s3:deleteObjectVersion permission
[ https://issues.apache.org/jira/browse/HADOOP-17261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-17261: Labels: pull-request-available (was: ) > s3a rename() now requires s3:deleteObjectVersion permission > --- > > Key: HADOOP-17261 > URL: https://issues.apache.org/jira/browse/HADOOP-17261 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > With the directory marker change (HADOOP-13230) you need the > s3:deleteObjectVersion permission in your role, else the operation will fail > in the bulk delete, *if S3Guard is in use* > Root cause > -if fileStatus has a versionId, we pass that in to the delete KeyVersion pair > -an unguarded listing doesn't get that versionId, so this is not an issue > -but if files in a directory were previously created such that S3Guard has > their versionId in its tables, that is used in the request > -which then fails if the caller doesn't have the permission > Although we say "you need s3:delete*", this is a regression as any IAM role > without the permission will have rename fail during delete -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17261) s3a rename() now requires s3:deleteObjectVersion permission
[ https://issues.apache.org/jira/browse/HADOOP-17261?focusedWorklogId=484193&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-484193 ] ASF GitHub Bot logged work on HADOOP-17261: --- Author: ASF GitHub Bot Created on: 14/Sep/20 19:59 Start Date: 14/Sep/20 19:59 Worklog Time Spent: 10m Work Description: steveloughran opened a new pull request #2303: URL: https://github.com/apache/hadoop/pull/2303 Stop using the versionId when building the list of files to delete as the rename progresses Testing: manual. For anyone wishing to replicate the problem on an unpatched release 1. Have a versioned bucket. 1. Latest version of cloudstore now allows you to create a set of session keys from an IAM role and json file, printing them as env vars and hadoop config options 1. You need S3Guard enabled and the files to have been created individually by a client through S3Guard (i.e. the entries not discovered through listing). Then S3Guard caches the version ID in the DB, which is then used when deleting the object. 1. mkdir src/; touchz src/file1 file2 2. mv src dst if that works, try renaming back. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 484193) Remaining Estimate: 0h Time Spent: 10m > s3a rename() now requires s3:deleteObjectVersion permission > --- > > Key: HADOOP-17261 > URL: https://issues.apache.org/jira/browse/HADOOP-17261 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > With the directory marker change (HADOOP-13230) you need the > s3:deleteObjectVersion permission in your role, else the operation will fail > in the bulk delete, *if S3Guard is in use* > Root cause > -if fileStatus has a versionId, we pass that in to the delete KeyVersion pair > -an unguarded listing doesn't get that versionId, so this is not an issue > -but if files in a directory were previously created such that S3Guard has > their versionId in its tables, that is used in the request > -which then fails if the caller doesn't have the permission > Although we say "you need s3:delete*", this is a regression as any IAM role > without the permission will have rename fail during delete -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran opened a new pull request #2303: HADOOP-17261. s3a rename() needs s3:deleteObjectVersion permission
steveloughran opened a new pull request #2303: URL: https://github.com/apache/hadoop/pull/2303 Stop using the versionId when building the list of files to delete as the rename progresses Testing: manual. For anyone wishing to replicate the problem on an unpatched release 1. Have a versioned bucket. 1. Latest version of cloudstore now allows you to create a set of session keys from an IAM role and json file, printing them as env vars and hadoop config options 1. You need S3Guard enabled and the files to have been created individually by a client through S3Guard (i.e. the entries not discovered through listing). Then S3Guard caches the version ID in the DB, which is then used when deleting the object. 1. mkdir src/; touchz src/file1 file2 2. mv src dst if that works, try renaming back. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] smengcl merged pull request #2258: HDFS-15539. When disallowing snapshot on a dir, throw exception if its trash root is not empty
smengcl merged pull request #2258: URL: https://github.com/apache/hadoop/pull/2258 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17255) JavaKeyStoreProvider fails to create a new key if the keystore is HDFS
[ https://issues.apache.org/jira/browse/HADOOP-17255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195743#comment-17195743 ] Wei-Chiu Chuang commented on HADOOP-17255: -- That said, the PR itself looks good to me. > JavaKeyStoreProvider fails to create a new key if the keystore is HDFS > -- > > Key: HADOOP-17255 > URL: https://issues.apache.org/jira/browse/HADOOP-17255 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Critical > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > The caller of JavaKeyStoreProvider#renameOrFail assumes that it throws > FileNotFoundException if the src does not exist. However, > JavaKeyStoreProvider#renameOrFail calls the old rename API. In > DistributedFileSystem, the old API returns false if the src does not exist. > That way JavaKeyStoreProvider fails to create a new key if the keystore is > HDFS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] smengcl commented on pull request #2176: HDFS-15492. Make trash root inside each snapshottable directory
smengcl commented on pull request #2176: URL: https://github.com/apache/hadoop/pull/2176#issuecomment-692330196 > I'm late to this. But we should verify to make sure httpfs also behaves the sames. Otherwise it'll break Hue. hey @jojochuang, thanks for checking in! I have written a simple test for verifying HttpFS (over WebHDFS). HttpFS works as expected. But I haven't figured out how to add a config to the HDFS cluster that `BaseTestHttpFSWith` launches. Currently the config change is a bit hacky. Ergo, I haven't submitted a patch for new jira to add this test. I would file a new jira to add it once this little problem is solved. [HDFS-15492.TestHttpFS.001.patch.txt](https://github.com/apache/hadoop/files/5221189/HDFS-15492.TestHttpFS.001.patch.txt) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] smengcl edited a comment on pull request #2176: HDFS-15492. Make trash root inside each snapshottable directory
smengcl edited a comment on pull request #2176: URL: https://github.com/apache/hadoop/pull/2176#issuecomment-692330196 > I'm late to this. But we should verify to make sure httpfs also behaves the sames. Otherwise it'll break Hue. hey @jojochuang, thanks for checking in! I have written a simple test for verifying HttpFS (over WebHDFS). HttpFS works as expected. But I haven't figured out how to add a config to the HDFS cluster that `BaseTestHttpFSWith` launches. Currently the config change for this test is a bit hacky. Ergo, I haven't submitted a patch for new jira to add this test. I would file a new jira to add it once this little problem is solved. [HDFS-15492.TestHttpFS.001.patch.txt](https://github.com/apache/hadoop/files/5221189/HDFS-15492.TestHttpFS.001.patch.txt) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jiwq commented on pull request #2277: HADOOP-17246. Fix build the hadoop-build Docker image failed
jiwq commented on pull request #2277: URL: https://github.com/apache/hadoop/pull/2277#issuecomment-692389614 > > I get, should we specific the version of astroid? > > Yes. Could you try "isort==4.3.21" and "astroid==1.6.6"? They are the latest versions that support Python 2.7. Okay, I had fix it and test passed in my local environment. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17246) Fix build the hadoop-build Docker image failed
[
https://issues.apache.org/jira/browse/HADOOP-17246?focusedWorklogId=484250&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-484250
]
ASF GitHub Bot logged work on HADOOP-17246:
---
Author: ASF GitHub Bot
Created on: 15/Sep/20 00:32
Start Date: 15/Sep/20 00:32
Worklog Time Spent: 10m
Work Description: jiwq commented on pull request #2277:
URL: https://github.com/apache/hadoop/pull/2277#issuecomment-692389614
> > I get, should we specific the version of astroid?
>
> Yes. Could you try "isort==4.3.21" and "astroid==1.6.6"? They are the
latest versions that support Python 2.7.
Okay, I had fix it and test passed in my local environment.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 484250)
Time Spent: 2h 40m (was: 2.5h)
> Fix build the hadoop-build Docker image failed
> --
>
> Key: HADOOP-17246
> URL: https://issues.apache.org/jira/browse/HADOOP-17246
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Reporter: Wanqiang Ji
>Assignee: Wanqiang Ji
>Priority: Major
> Labels: dockerfile, pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> When I build the docker-build image under macOS, it failed caused by:
> {code:java}
>
> Command "/usr/bin/python -u -c "import setuptools,
> tokenize;__file__='/tmp/pip-build-vKHcWu/isort/setup.py';exec(compile(getattr(tokenize,
> 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))"
> install --record /tmp/pip-odL0bY-record/install-record.txt
> --single-version-externally-managed --compile" failed with error code 1 in
> /tmp/pip-build-vKHcWu/isort/
> You are using pip version 8.1.1, however version 20.2.2 is available.
> You should consider upgrading via the 'pip install --upgrade pip' command.
> The command '/bin/bash -o pipefail -c pip2 install configparser==4.0.2
> pylint==1.9.2' returned a non-zero code: 1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17262) Switch to Yetus main branch
Akira Ajisaka created HADOOP-17262: -- Summary: Switch to Yetus main branch Key: HADOOP-17262 URL: https://issues.apache.org/jira/browse/HADOOP-17262 Project: Hadoop Common Issue Type: Bug Components: build Reporter: Akira Ajisaka Assignee: Akira Ajisaka YETUS-986 switched the development branch from master to main. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka opened a new pull request #2304: HADOOP-17262. Switch to Yetus main branch.
aajisaka opened a new pull request #2304: URL: https://github.com/apache/hadoop/pull/2304 JIRA: https://issues.apache.org/jira/browse/HADOOP-17262 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17262) Switch to Yetus main branch
[ https://issues.apache.org/jira/browse/HADOOP-17262?focusedWorklogId=484263&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-484263 ] ASF GitHub Bot logged work on HADOOP-17262: --- Author: ASF GitHub Bot Created on: 15/Sep/20 01:27 Start Date: 15/Sep/20 01:27 Worklog Time Spent: 10m Work Description: aajisaka opened a new pull request #2304: URL: https://github.com/apache/hadoop/pull/2304 JIRA: https://issues.apache.org/jira/browse/HADOOP-17262 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 484263) Remaining Estimate: 0h Time Spent: 10m > Switch to Yetus main branch > --- > > Key: HADOOP-17262 > URL: https://issues.apache.org/jira/browse/HADOOP-17262 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > YETUS-986 switched the development branch from master to main. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17262) Switch to Yetus main branch
[ https://issues.apache.org/jira/browse/HADOOP-17262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-17262: Labels: pull-request-available (was: ) > Switch to Yetus main branch > --- > > Key: HADOOP-17262 > URL: https://issues.apache.org/jira/browse/HADOOP-17262 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > YETUS-986 switched the development branch from master to main. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17262) Switch to Yetus main branch
[ https://issues.apache.org/jira/browse/HADOOP-17262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195801#comment-17195801 ] Akira Ajisaka commented on HADOOP-17262: Note: * This PR is for hadoop-multibranch job (i.e. PreCommit jobs at GitHub). * I've already updated the configs of PreCommit-(HADOOP|HDFS|MAPREDUCE|YARN)-Build jobs (i.e. PreCommit jobs at JIRA). > Switch to Yetus main branch > --- > > Key: HADOOP-17262 > URL: https://issues.apache.org/jira/browse/HADOOP-17262 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > YETUS-986 switched the development branch from master to main. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2304: HADOOP-17262. Switch to Yetus main branch.
hadoop-yetus commented on pull request #2304: URL: https://github.com/apache/hadoop/pull/2304#issuecomment-692418359 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 3s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | shadedclient | 22m 36s | branch has no errors when building and testing our client artifacts. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | shellcheck | 0m 1s | There were no new shellcheck issues. | | +1 :green_heart: | shelldocs | 0m 13s | There were no new shelldocs issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 15m 46s | patch has no errors when building and testing our client artifacts. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 33s | The patch does not generate ASF License warnings. | | | | 41m 54s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2304/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2304 | | Optional Tests | dupname asflicense shellcheck shelldocs | | uname | Linux 600add54ddb5 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 90894ea641f | | Max. process+thread count | 311 (vs. ulimit of 5500) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2304/1/console | | versions | git=2.17.1 maven=3.6.0 shellcheck=0.4.6 | | Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on pull request #2304: HADOOP-17262. Switch to Yetus main branch.
aajisaka commented on pull request #2304: URL: https://github.com/apache/hadoop/pull/2304#issuecomment-692418756 @jojochuang @sunchao @tasanuma Thank you for your reviews! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka merged pull request #2304: HADOOP-17262. Switch to Yetus main branch.
aajisaka merged pull request #2304: URL: https://github.com/apache/hadoop/pull/2304 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17262) Switch to Yetus main branch
[ https://issues.apache.org/jira/browse/HADOOP-17262?focusedWorklogId=484265&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-484265 ] ASF GitHub Bot logged work on HADOOP-17262: --- Author: ASF GitHub Bot Created on: 15/Sep/20 02:11 Start Date: 15/Sep/20 02:11 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2304: URL: https://github.com/apache/hadoop/pull/2304#issuecomment-692418359 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 3s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | shadedclient | 22m 36s | branch has no errors when building and testing our client artifacts. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | shellcheck | 0m 1s | There were no new shellcheck issues. | | +1 :green_heart: | shelldocs | 0m 13s | There were no new shelldocs issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 15m 46s | patch has no errors when building and testing our client artifacts. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 33s | The patch does not generate ASF License warnings. | | | | 41m 54s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2304/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2304 | | Optional Tests | dupname asflicense shellcheck shelldocs | | uname | Linux 600add54ddb5 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 90894ea641f | | Max. process+thread count | 311 (vs. ulimit of 5500) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2304/1/console | | versions | git=2.17.1 maven=3.6.0 shellcheck=0.4.6 | | Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 484265) Time Spent: 20m (was: 10m) > Switch to Yetus main branch > --- > > Key: HADOOP-17262 > URL: https://issues.apache.org/jira/browse/HADOOP-17262 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > YETUS-986 switched the development branch from master to main. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17262) Switch to Yetus main branch
[ https://issues.apache.org/jira/browse/HADOOP-17262?focusedWorklogId=484267&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-484267 ] ASF GitHub Bot logged work on HADOOP-17262: --- Author: ASF GitHub Bot Created on: 15/Sep/20 02:12 Start Date: 15/Sep/20 02:12 Worklog Time Spent: 10m Work Description: aajisaka merged pull request #2304: URL: https://github.com/apache/hadoop/pull/2304 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 484267) Time Spent: 0.5h (was: 20m) > Switch to Yetus main branch > --- > > Key: HADOOP-17262 > URL: https://issues.apache.org/jira/browse/HADOOP-17262 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > YETUS-986 switched the development branch from master to main. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17262) Switch to Yetus main branch
[ https://issues.apache.org/jira/browse/HADOOP-17262?focusedWorklogId=484269&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-484269 ] ASF GitHub Bot logged work on HADOOP-17262: --- Author: ASF GitHub Bot Created on: 15/Sep/20 02:12 Start Date: 15/Sep/20 02:12 Worklog Time Spent: 10m Work Description: aajisaka commented on pull request #2304: URL: https://github.com/apache/hadoop/pull/2304#issuecomment-692418756 @jojochuang @sunchao @tasanuma Thank you for your reviews! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 484269) Time Spent: 40m (was: 0.5h) > Switch to Yetus main branch > --- > > Key: HADOOP-17262 > URL: https://issues.apache.org/jira/browse/HADOOP-17262 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > YETUS-986 switched the development branch from master to main. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17262) Switch to Yetus main branch
[ https://issues.apache.org/jira/browse/HADOOP-17262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-17262: --- Target Version/s: 3.4.0 (was: 3.3.1, 3.4.0, 3.1.5, 2.10.2, 3.2.3) > Switch to Yetus main branch > --- > > Key: HADOOP-17262 > URL: https://issues.apache.org/jira/browse/HADOOP-17262 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > YETUS-986 switched the development branch from master to main. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17262) Switch to Yetus main branch
[ https://issues.apache.org/jira/browse/HADOOP-17262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka resolved HADOOP-17262. Fix Version/s: 3.4.0 Resolution: Fixed Committed to trunk. > Switch to Yetus main branch > --- > > Key: HADOOP-17262 > URL: https://issues.apache.org/jira/browse/HADOOP-17262 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > YETUS-986 switched the development branch from master to main. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on a change in pull request #2259: HADOOP-17208. LoadBalanceKMSClientProvider#deleteKey should invalidat…
Hexiaoqiao commented on a change in pull request #2259:
URL: https://github.com/apache/hadoop/pull/2259#discussion_r488362922
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/ValueQueue.java
##
@@ -299,19 +299,18 @@ public E getNext(String keyName)
* @param keyName the key to drain the Queue for
*/
public void drain(String keyName) {
+Runnable e;
+while ((e = queue.deleteByName(keyName)) != null) {
+ executor.remove(e);
+}
+writeLock(keyName);
try {
- Runnable e;
- while ((e = queue.deleteByName(keyName)) != null) {
-executor.remove(e);
- }
- writeLock(keyName);
- try {
-keyQueues.get(keyName).clear();
- } finally {
-writeUnlock(keyName);
+ LinkedBlockingQueue kq = keyQueues.getIfPresent(keyName);
+ if (kq != null) {
+kq.clear();
}
-} catch (ExecutionException ex) {
Review comment:
Hi @xiaoyuyao , Not sure if it is compatible to throw out this
exception. Other are fair enough to me. Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17208) LoadBalanceKMSClientProvider#deleteKey should invalidateCache via all KMSClientProvider instances
[
https://issues.apache.org/jira/browse/HADOOP-17208?focusedWorklogId=484281&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-484281
]
ASF GitHub Bot logged work on HADOOP-17208:
---
Author: ASF GitHub Bot
Created on: 15/Sep/20 03:37
Start Date: 15/Sep/20 03:37
Worklog Time Spent: 10m
Work Description: Hexiaoqiao commented on a change in pull request #2259:
URL: https://github.com/apache/hadoop/pull/2259#discussion_r488362922
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/ValueQueue.java
##
@@ -299,19 +299,18 @@ public E getNext(String keyName)
* @param keyName the key to drain the Queue for
*/
public void drain(String keyName) {
+Runnable e;
+while ((e = queue.deleteByName(keyName)) != null) {
+ executor.remove(e);
+}
+writeLock(keyName);
try {
- Runnable e;
- while ((e = queue.deleteByName(keyName)) != null) {
-executor.remove(e);
- }
- writeLock(keyName);
- try {
-keyQueues.get(keyName).clear();
- } finally {
-writeUnlock(keyName);
+ LinkedBlockingQueue kq = keyQueues.getIfPresent(keyName);
+ if (kq != null) {
+kq.clear();
}
-} catch (ExecutionException ex) {
Review comment:
Hi @xiaoyuyao , Not sure if it is compatible to throw out this
exception. Other are fair enough to me. Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 484281)
Time Spent: 2h (was: 1h 50m)
> LoadBalanceKMSClientProvider#deleteKey should invalidateCache via all
> KMSClientProvider instances
> -
>
> Key: HADOOP-17208
> URL: https://issues.apache.org/jira/browse/HADOOP-17208
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.8.4
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Without invalidateCache, the deleted key may still exists in the servers' key
> cache (CachingKeyProvider in KMSWebApp.java) where the delete key was not
> hit. Client may still be able to access encrypted files by specifying to
> connect to KMS instances with a cached version of the deleted key before the
> cache entry (10 min by default) expired.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17258) MagicS3GuardCommitter fails with `pendingset` already exists
[
https://issues.apache.org/jira/browse/HADOOP-17258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195850#comment-17195850
]
Dongjoon Hyun commented on HADOOP-17258:
Thank you for commenting, [~ste...@apache.org]. Yes, it makes sense. Currently,
this causes the job failure at the almost end of execution. I used DynamoDB and
S3Guard and S3Magic committer and have been monitoring the target directory.
> MagicS3GuardCommitter fails with `pendingset` already exists
>
>
> Key: HADOOP-17258
> URL: https://issues.apache.org/jira/browse/HADOOP-17258
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Dongjoon Hyun
>Priority: Major
>
> In `trunk/branch-3.3/branch-3.2`, `MagicS3GuardCommitter.innerCommitTask` has
> `false` at `pendingSet.save`.
> {code}
> try {
> pendingSet.save(getDestFS(), taskOutcomePath, false);
> } catch (IOException e) {
> LOG.warn("Failed to save task commit data to {} ",
> taskOutcomePath, e);
> abortPendingUploads(context, pendingSet.getCommits(), true);
> throw e;
> }
> {code}
> And, it can cause a job failure like the following.
> {code}
> WARN TaskSetManager: Lost task 1562.1 in stage 1.0 (TID 1788, 100.92.11.63,
> executor 26): org.apache.spark.SparkException: Task failed while writing rows.
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:257)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
> at org.apache.spark.scheduler.Task.run(Task.scala:123)
> at
> org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> at java.base/java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.hadoop.fs.FileAlreadyExistsException:
> s3a://xxx/__magic/app-attempt-/task_20200911063607_0001_m_001562.pendingset
> already exists
> at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:761)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1118)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1098)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:987)
> at
> org.apache.hadoop.util.JsonSerialization.save(JsonSerialization.java:269)
> at
> org.apache.hadoop.fs.s3a.commit.files.PendingSet.save(PendingSet.java:170)
> at
> org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter.innerCommitTask(MagicS3GuardCommitter.java:220)
> at
> org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter.commitTask(MagicS3GuardCommitter.java:165)
> at
> org.apache.spark.mapred.SparkHadoopMapRedUtil$.performCommit$1(SparkHadoopMapRedUtil.scala:50)
> at
> org.apache.spark.mapred.SparkHadoopMapRedUtil$.commitTask(SparkHadoopMapRedUtil.scala:77)
> at
> org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitTask(HadoopMapReduceCommitProtocol.scala:244)
> at
> org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:78)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:247)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:242)
> {code}
> {code}
> 20/09/11 07:44:38 ERROR TaskSetManager: Task 957.1 in stage 1.0 (TID 1412)
> can not write to output file:
> org.apache.hadoop.fs.FileAlreadyExistsException:
> s3a://xxx/t/__magic/app-attempt-/task_20200911073922_0001_m_000957.pendingset
> already exists; not retrying
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17258) MagicS3GuardCommitter fails with `pendingset` already exists
[
https://issues.apache.org/jira/browse/HADOOP-17258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195853#comment-17195853
]
Dongjoon Hyun commented on HADOOP-17258:
Also, cc [~chaosun]
> MagicS3GuardCommitter fails with `pendingset` already exists
>
>
> Key: HADOOP-17258
> URL: https://issues.apache.org/jira/browse/HADOOP-17258
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Dongjoon Hyun
>Priority: Major
>
> In `trunk/branch-3.3/branch-3.2`, `MagicS3GuardCommitter.innerCommitTask` has
> `false` at `pendingSet.save`.
> {code}
> try {
> pendingSet.save(getDestFS(), taskOutcomePath, false);
> } catch (IOException e) {
> LOG.warn("Failed to save task commit data to {} ",
> taskOutcomePath, e);
> abortPendingUploads(context, pendingSet.getCommits(), true);
> throw e;
> }
> {code}
> And, it can cause a job failure like the following.
> {code}
> WARN TaskSetManager: Lost task 1562.1 in stage 1.0 (TID 1788, 100.92.11.63,
> executor 26): org.apache.spark.SparkException: Task failed while writing rows.
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:257)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
> at org.apache.spark.scheduler.Task.run(Task.scala:123)
> at
> org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> at java.base/java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.hadoop.fs.FileAlreadyExistsException:
> s3a://xxx/__magic/app-attempt-/task_20200911063607_0001_m_001562.pendingset
> already exists
> at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:761)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1118)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1098)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:987)
> at
> org.apache.hadoop.util.JsonSerialization.save(JsonSerialization.java:269)
> at
> org.apache.hadoop.fs.s3a.commit.files.PendingSet.save(PendingSet.java:170)
> at
> org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter.innerCommitTask(MagicS3GuardCommitter.java:220)
> at
> org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter.commitTask(MagicS3GuardCommitter.java:165)
> at
> org.apache.spark.mapred.SparkHadoopMapRedUtil$.performCommit$1(SparkHadoopMapRedUtil.scala:50)
> at
> org.apache.spark.mapred.SparkHadoopMapRedUtil$.commitTask(SparkHadoopMapRedUtil.scala:77)
> at
> org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitTask(HadoopMapReduceCommitProtocol.scala:244)
> at
> org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:78)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:247)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:242)
> {code}
> {code}
> 20/09/11 07:44:38 ERROR TaskSetManager: Task 957.1 in stage 1.0 (TID 1412)
> can not write to output file:
> org.apache.hadoop.fs.FileAlreadyExistsException:
> s3a://xxx/t/__magic/app-attempt-/task_20200911073922_0001_m_000957.pendingset
> already exists; not retrying
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on pull request #2277: HADOOP-17246. Fix build the hadoop-build Docker image failed
Hexiaoqiao commented on pull request #2277: URL: https://github.com/apache/hadoop/pull/2277#issuecomment-692447959 It seems that CI not trigger auto, Try to replay manually, please ref to https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2277/4/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17246) Fix build the hadoop-build Docker image failed
[
https://issues.apache.org/jira/browse/HADOOP-17246?focusedWorklogId=484284&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-484284
]
ASF GitHub Bot logged work on HADOOP-17246:
---
Author: ASF GitHub Bot
Created on: 15/Sep/20 03:58
Start Date: 15/Sep/20 03:58
Worklog Time Spent: 10m
Work Description: Hexiaoqiao commented on pull request #2277:
URL: https://github.com/apache/hadoop/pull/2277#issuecomment-692447959
It seems that CI not trigger auto, Try to replay manually, please ref to
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2277/4/
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 484284)
Time Spent: 2h 50m (was: 2h 40m)
> Fix build the hadoop-build Docker image failed
> --
>
> Key: HADOOP-17246
> URL: https://issues.apache.org/jira/browse/HADOOP-17246
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Reporter: Wanqiang Ji
>Assignee: Wanqiang Ji
>Priority: Major
> Labels: dockerfile, pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> When I build the docker-build image under macOS, it failed caused by:
> {code:java}
>
> Command "/usr/bin/python -u -c "import setuptools,
> tokenize;__file__='/tmp/pip-build-vKHcWu/isort/setup.py';exec(compile(getattr(tokenize,
> 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))"
> install --record /tmp/pip-odL0bY-record/install-record.txt
> --single-version-externally-managed --compile" failed with error code 1 in
> /tmp/pip-build-vKHcWu/isort/
> You are using pip version 8.1.1, however version 20.2.2 is available.
> You should consider upgrading via the 'pip install --upgrade pip' command.
> The command '/bin/bash -o pipefail -c pip2 install configparser==4.0.2
> pylint==1.9.2' returned a non-zero code: 1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16254) Add proxy address in IPC connection
[ https://issues.apache.org/jira/browse/HADOOP-16254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195867#comment-17195867 ] Xiaoqiao He commented on HADOOP-16254: -- Thanks [~zhengchenyu] for your comments. Please ref to [^HADOOP-16254.002.patch]. If any issue, please feel free to submit fixed patch. We have to solve the security issue as [~daryn] mentioned above if want to push forward. Thanks. > Add proxy address in IPC connection > --- > > Key: HADOOP-16254 > URL: https://issues.apache.org/jira/browse/HADOOP-16254 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Attachments: HADOOP-16254.001.patch, HADOOP-16254.002.patch, > HADOOP-16254.004.patch > > > In order to support data locality of RBF, we need to add new field about > client hostname in the RPC headers of Router protocol calls. > clientHostname represents hostname of client and forward by Router to > Namenode to support data locality friendly. See more [RBF Data Locality > Design|https://issues.apache.org/jira/secure/attachment/12965092/RBF%20Data%20Locality%20Design.pdf] > in HDFS-13248 and [maillist > vote|http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201904.mbox/%3CCAF3Ajax7hGxvowg4K_HVTZeDqC5H=3bfb7mv5sz5mgvadhv...@mail.gmail.com%3E]. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16254) Add proxy address in IPC connection
[
https://issues.apache.org/jira/browse/HADOOP-16254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195870#comment-17195870
]
Hadoop QA commented on HADOOP-16254:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 33m
0s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} buf {color} | {color:blue} 0m 0s{color}
| {color:blue} buf was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
45s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
31s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 52s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
15s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
6s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 20m 6s{color} |
{color:red} root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 21 new + 142 unchanged -
21 fixed = 163 total (was 163) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 20m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
35s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 17m 35s{color} |
{color:red} root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with
JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 15 new + 148
unchanged - 15 fixed = 163 total (was 163) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m
35s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 49s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 9 new + 342 unchanged - 1 fixed = 351 total (was 343) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 30s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color
[GitHub] [hadoop] Hexiaoqiao commented on pull request #2277: HADOOP-17246. Fix build the hadoop-build Docker image failed
Hexiaoqiao commented on pull request #2277: URL: https://github.com/apache/hadoop/pull/2277#issuecomment-692479933 @aajisaka it still seems building failed, would you mind have another review? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17246) Fix build the hadoop-build Docker image failed
[
https://issues.apache.org/jira/browse/HADOOP-17246?focusedWorklogId=484308&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-484308
]
ASF GitHub Bot logged work on HADOOP-17246:
---
Author: ASF GitHub Bot
Created on: 15/Sep/20 05:48
Start Date: 15/Sep/20 05:48
Worklog Time Spent: 10m
Work Description: Hexiaoqiao commented on pull request #2277:
URL: https://github.com/apache/hadoop/pull/2277#issuecomment-692479933
@aajisaka it still seems building failed, would you mind have another review?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 484308)
Time Spent: 3h (was: 2h 50m)
> Fix build the hadoop-build Docker image failed
> --
>
> Key: HADOOP-17246
> URL: https://issues.apache.org/jira/browse/HADOOP-17246
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Reporter: Wanqiang Ji
>Assignee: Wanqiang Ji
>Priority: Major
> Labels: dockerfile, pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
> When I build the docker-build image under macOS, it failed caused by:
> {code:java}
>
> Command "/usr/bin/python -u -c "import setuptools,
> tokenize;__file__='/tmp/pip-build-vKHcWu/isort/setup.py';exec(compile(getattr(tokenize,
> 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))"
> install --record /tmp/pip-odL0bY-record/install-record.txt
> --single-version-externally-managed --compile" failed with error code 1 in
> /tmp/pip-build-vKHcWu/isort/
> You are using pip version 8.1.1, however version 20.2.2 is available.
> You should consider upgrading via the 'pip install --upgrade pip' command.
> The command '/bin/bash -o pipefail -c pip2 install configparser==4.0.2
> pylint==1.9.2' returned a non-zero code: 1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2302: HDFS-15577. Refactor TestTracing.
hadoop-yetus commented on pull request #2302: URL: https://github.com/apache/hadoop/pull/2302#issuecomment-692485519 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 28m 42s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 29m 7s | trunk passed | | +1 :green_heart: | compile | 1m 17s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 1m 10s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 51s | trunk passed | | +1 :green_heart: | mvnsite | 1m 16s | trunk passed | | +1 :green_heart: | shadedclient | 16m 15s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 52s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 1m 22s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 3m 2s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 2m 59s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 1m 11s | the patch passed | | +1 :green_heart: | compile | 1m 10s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 1m 10s | the patch passed | | +1 :green_heart: | compile | 1m 7s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 1m 7s | the patch passed | | +1 :green_heart: | checkstyle | 0m 39s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 0 unchanged - 32 fixed = 0 total (was 32) | | +1 :green_heart: | mvnsite | 1m 10s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 13m 57s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 45s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 1m 19s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 2m 58s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 95m 16s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | asflicense | 0m 44s | The patch does not generate ASF License warnings. | | | | 206m 4s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.TestFileChecksumCompositeCrc | | | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped | | | hadoop.hdfs.TestFileChecksum | | | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier | | | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2302/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2302 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 740d6a63e43f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 65df30bad63 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2302/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2302/4/testReport/ | | Max. process+thread count | 4548 (vs. ulimit of 5500) | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2302/4/console | | versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
[GitHub] [hadoop] JohnZZGithub commented on pull request #2223: MAPREDUCE-7294. Fix the bug to make sure only application master upload resource to Yarn Shared Cache
JohnZZGithub commented on pull request #2223: URL: https://github.com/apache/hadoop/pull/2223#issuecomment-692507296 @liuml07 Thanks a lot for the detailed review, updated the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
