[jira] [Commented] (HADOOP-16260) Allow Distcp to create a new tempTarget file per File
[
https://issues.apache.org/jira/browse/HADOOP-16260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820456#comment-16820456
]
Arun Suresh commented on HADOOP-16260:
--
Thanks Gabor.. yeah just saw that. We are experimenting with it. Looks like it
was checked in recently - so it's not in any release yet.
> Allow Distcp to create a new tempTarget file per File
> -
>
> Key: HADOOP-16260
> URL: https://issues.apache.org/jira/browse/HADOOP-16260
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.9.2
>Reporter: Arun Suresh
>Priority: Major
>
> We use distcp to copy entire HDFS clusters to GCS.
> In the process, we hit the following error:
> {noformat}
> INFO: Encountered status code 410 when accessing URL
> https://www.googleapis.com/upload/storage/v1/b/app/o?ifGenerationMatch=0=analytics/.distcp.tmp.attempt_local1083459072_0001_m_00_0=resumable_id=AEnB2Uq4mZeZxXgs2Mhx0uskNpZ4Cka8pT4aCcd7v6UC4TDQx-h0uEFWoPpdOO4pWEdmaKnhTjxVva5Ow4vXbTe6_JScIU5fsQSaIwNkF3D84DHjtuhKSCU.
> Delegating to response handler for possible retry.
> Apr 14, 2019 5:53:17 AM
> com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation
> call
> SEVERE: Exception not convertible into handled response
> com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.json.GoogleJsonResponseException:
> 410 Gone
> {
> "code" : 429,
> "errors" : [ {
> "domain" : "usageLimits",
> "message" : "The total number of changes to the object
> app/folder/.distcp.tmp.attempt_local1083459072_0001_m_00_0 exceeds the
> rate limit. Please reduce the rate of create, update, and delete requests.",
> "reason" : "rateLimitExceeded"
> } ],
> "message" : "The total number of changes to the object
> app/folder/.distcp.tmp.attempt_local1083459072_0001_m_00_0 exceeds the
> rate limit. Please reduce the rate of create, update, and delete requests."
> }
>at
> com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:150)
> at
> com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
> at
> com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
> at
> com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
> at
> com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
> at
> com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
> at
> com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:301)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> {noformat}
> Looking at the code, it looks like a distCp mapper gets a list of files to
> copy from src to target filesystem. The mapper handles each file in its list
> sequentially: It first creates/overwrites a temp file
> (*.distcp.tmp.attempt_local1083459072_0001_m_00_0*), then it copies the
> src file to the temp file, and finally renames the temp file to the actual
> target file.
> The temp file name (which contains the task ID) is reused for all the files
> in the mapper's batch. It looks like GCP enforces a rate-limit on the number
> of operations per sec on any object (even though we are actually creating a
> new file and renaming it to the final target, gcp assumes we are making
> changes to the same object)
> Even though it is possible to play around with the number of Maps / split
> size etc. It is hard to arrive at one of those values based on any rate-limit.
> Thus, we propose we add a flag to allow the DistCp mapper to use a different
> temp file PER file.
> Thoughts ? (cc/[~steve_l], [~benoyantony])
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands,
[jira] [Commented] (HADOOP-15281) Distcp to add no-rename copy option
[
https://issues.apache.org/jira/browse/HADOOP-15281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820197#comment-16820197
]
Arun Suresh commented on HADOOP-15281:
--
Thanks for posting the patch for branch-2 [~ste...@apache.org]. Found a minor
nit while testing though:
{{OptionsParser:165}} we need *{{option.setDirectWrite(false);}}* to be
*{{option.setDirectWrite(true);}}*
> Distcp to add no-rename copy option
> ---
>
> Key: HADOOP-15281
> URL: https://issues.apache.org/jira/browse/HADOOP-15281
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Andrew Olson
>Priority: Major
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HADOOP-15281-001.patch, HADOOP-15281-002.patch,
> HADOOP-15281-003.patch, HADOOP-15281-004.patch,
> HADOOP-15281-branch-2-001.patch
>
>
> Currently Distcp uploads a file by two strategies
> # append parts
> # copy to temp then rename
> option 2 executes the following sequence in {{promoteTmpToTarget}}
> {code}
> if ((fs.exists(target) && !fs.delete(target, false))
> || (!fs.exists(target.getParent()) && !fs.mkdirs(target.getParent()))
> || !fs.rename(tmpTarget, target)) {
> throw new IOException("Failed to promote tmp-file:" + tmpTarget
> + " to: " + target);
> }
> {code}
> For any object store, that's a lot of HTTP requests; for S3A you are looking
> at 12+ requests and an O(data) copy call.
> This is not a good upload strategy for any store which manifests its output
> atomically at the end of the write().
> Proposed: add a switch to write directly to the dest path, which can be
> supplied as either a conf option (distcp.direct.write = true) or a CLI option
> (-direct).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16260) Allow Distcp to create a new tempTarget file per File
[
https://issues.apache.org/jira/browse/HADOOP-16260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-16260:
-
Description:
We use distcp to copy entire HDFS clusters to GCS.
In the process, we hit the following error:
{noformat}
INFO: Encountered status code 410 when accessing URL
https://www.googleapis.com/upload/storage/v1/b/app/o?ifGenerationMatch=0=analytics/.distcp.tmp.attempt_local1083459072_0001_m_00_0=resumable_id=AEnB2Uq4mZeZxXgs2Mhx0uskNpZ4Cka8pT4aCcd7v6UC4TDQx-h0uEFWoPpdOO4pWEdmaKnhTjxVva5Ow4vXbTe6_JScIU5fsQSaIwNkF3D84DHjtuhKSCU.
Delegating to response handler for possible retry.
Apr 14, 2019 5:53:17 AM
com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation
call
SEVERE: Exception not convertible into handled response
com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.json.GoogleJsonResponseException:
410 Gone
{
"code" : 429,
"errors" : [ {
"domain" : "usageLimits",
"message" : "The total number of changes to the object
app/folder/.distcp.tmp.attempt_local1083459072_0001_m_00_0 exceeds the rate
limit. Please reduce the rate of create, update, and delete requests.",
"reason" : "rateLimitExceeded"
} ],
"message" : "The total number of changes to the object
app/folder/.distcp.tmp.attempt_local1083459072_0001_m_00_0 exceeds the rate
limit. Please reduce the rate of create, update, and delete requests."
}
at
com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:150)
at
com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at
com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at
com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at
com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at
com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at
com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:301)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
Looking at the code, it looks like a distCp mapper gets a list of files to copy
from src to target filesystem. The mapper handles each file in its list
sequentially: It first creates/overwrites a temp file
(*.distcp.tmp.attempt_local1083459072_0001_m_00_0*), then it copies the src
file to the temp file, and finally renames the temp file to the actual target
file.
The temp file name (which contains the task ID) is reused for all the files in
the mapper's batch. It looks like GCP enforces a rate-limit on the number of
operations per sec on any object (even though we are actually creating a new
file and renaming it to the final target, gcp assumes we are making changes to
the same object)
Even though it is possible to play around with the number of Maps / split size
etc. It is hard to arrive at one of those values based on any rate-limit.
Thus, we propose we add a flag to allow the DistCp mapper to use a different
temp file PER file.
Thoughts ? (cc/[~steve_l], [~benoyantony])
was:
We use distcp to copy entire HDFS clusters to GCS.
In the process, we hit the following error:
{noformat}
INFO: Encountered status code 410 when accessing URL
https://www.googleapis.com/upload/storage/v1/b/ap10data1/o?ifGenerationMatch=0=analytics/.distcp.tmp.attempt_local1083459072_0001_m_00_0=resumable_id=AEnB2Uq4mZeZxXgs2Mhx0uskNpZ4Cka8pT4aCcd7v6UC4TDQx-h0uEFWoPpdOO4pWEdmaKnhTjxVva5Ow4vXbTe6_JScIU5fsQSaIwNkF3D84DHjtuhKSCU.
Delegating to response handler for possible retry.
Apr 14, 2019 5:53:17 AM
com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation
call
SEVERE: Exception not convertible into handled response
com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.json.GoogleJsonResponseException:
410 Gone
{
"code" : 429,
"errors" : [ {
"domain" : "usageLimits",
"message"
[jira] [Created] (HADOOP-16260) Allow Distcp to create a new tempTarget file per File
Arun Suresh created HADOOP-16260:
Summary: Allow Distcp to create a new tempTarget file per File
Key: HADOOP-16260
URL: https://issues.apache.org/jira/browse/HADOOP-16260
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.9.2
Reporter: Arun Suresh
We use distcp to copy entire HDFS clusters to GCS.
In the process, we hit the following error:
{noformat}
INFO: Encountered status code 410 when accessing URL
https://www.googleapis.com/upload/storage/v1/b/ap10data1/o?ifGenerationMatch=0=analytics/.distcp.tmp.attempt_local1083459072_0001_m_00_0=resumable_id=AEnB2Uq4mZeZxXgs2Mhx0uskNpZ4Cka8pT4aCcd7v6UC4TDQx-h0uEFWoPpdOO4pWEdmaKnhTjxVva5Ow4vXbTe6_JScIU5fsQSaIwNkF3D84DHjtuhKSCU.
Delegating to response handler for possible retry.
Apr 14, 2019 5:53:17 AM
com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation
call
SEVERE: Exception not convertible into handled response
com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.json.GoogleJsonResponseException:
410 Gone
{
"code" : 429,
"errors" : [ {
"domain" : "usageLimits",
"message" : "The total number of changes to the object
ap10data1/analytics/.distcp.tmp.attempt_local1083459072_0001_m_00_0 exceeds
the rate limit. Please reduce the rate of create, update, and delete requests.",
"reason" : "rateLimitExceeded"
} ],
"message" : "The total number of changes to the object
ap10data1/analytics/.distcp.tmp.attempt_local1083459072_0001_m_00_0 exceeds
the rate limit. Please reduce the rate of create, update, and delete requests."
}
at
com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:150)
at
com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at
com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at
com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at
com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at
com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at
com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:301)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
Looking at the code, it looks like a distCp mapper gets a list of files to copy
from src to target filesystem. The mapper handles each file in its list
sequentially: It first creates/overwrites a temp file
(*.distcp.tmp.attempt_local1083459072_0001_m_00_0*), then it copies the src
file to the temp file, and finally renames the temp file to the actual target
file.
The temp file name (which contains the task ID) is reused for all the files in
the mapper's batch. It looks like GCP enforces a rate-limit on the number of
operations per sec on any object (even though we are actually creating a new
file and renaming it to the final target, gcp assumes we are making changes to
the same object)
Even though it is possible to play around with the number of Maps / split size
etc. It is hard to arrive at one of those values based on any rate-limit.
Thus, we propose we add a flag to allow the DistCp mapper to use a different
temp file PER file.
Thoughts ? (cc/[~steve_l], [~benoyantony])
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15711) Move branch-2 precommit/nightly test builds to java 8
[
https://issues.apache.org/jira/browse/HADOOP-15711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763916#comment-16763916
]
Arun Suresh commented on HADOOP-15711:
--
Given the discussion in the vote
[thread|https://lists.apache.org/thread.html/734c73111c17663675a76f0355f4f4bce630ef3b9084d588e04006d2@%3Cyarn-dev.hadoop.apache.org%3E]
+1, Thanks [~jhung]
> Move branch-2 precommit/nightly test builds to java 8
> -
>
> Key: HADOOP-15711
> URL: https://issues.apache.org/jira/browse/HADOOP-15711
> Project: Hadoop Common
> Issue Type: Task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Critical
> Attachments: HADOOP-15711-branch-2.002.patch,
> HADOOP-15711.001.branch-2.patch
>
>
> Branch-2 builds have been disabled for a while:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/
> A test run here causes hdfs tests to hang:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86-jhung/4/
> Running hadoop-hdfs tests locally reveal some errors such
> as:{noformat}[ERROR]
> testComplexAppend2(org.apache.hadoop.hdfs.TestFileAppend2) Time elapsed:
> 0.059 s <<< ERROR!
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1164)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1128)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:174)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1172)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:403)
> at
> org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:234)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1080)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:883)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:514)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:473)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend(TestFileAppend2.java:489)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend2(TestFileAppend2.java:543)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){noformat}
> I was able to get more tests passing locally by increasing the max user
> process count on my machine. But the error suggests that there's an issue in
> the tests themselves. Not sure if the error seen locally is the same reason
> as why jenkins builds are failing, I wasn't able to confirm based on the
> jenkins builds' lack of output.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15711) Move branch-2 precommit/nightly test builds to java 8
[
https://issues.apache.org/jira/browse/HADOOP-15711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-15711:
-
Summary: Move branch-2 precommit/nightly test builds to java 8 (was: Fix
branch-2 builds)
> Move branch-2 precommit/nightly test builds to java 8
> -
>
> Key: HADOOP-15711
> URL: https://issues.apache.org/jira/browse/HADOOP-15711
> Project: Hadoop Common
> Issue Type: Task
>Reporter: Jonathan Hung
>Priority: Critical
> Attachments: HADOOP-15711-branch-2.002.patch,
> HADOOP-15711.001.branch-2.patch
>
>
> Branch-2 builds have been disabled for a while:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/
> A test run here causes hdfs tests to hang:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86-jhung/4/
> Running hadoop-hdfs tests locally reveal some errors such
> as:{noformat}[ERROR]
> testComplexAppend2(org.apache.hadoop.hdfs.TestFileAppend2) Time elapsed:
> 0.059 s <<< ERROR!
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1164)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1128)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:174)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1172)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:403)
> at
> org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:234)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1080)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:883)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:514)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:473)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend(TestFileAppend2.java:489)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend2(TestFileAppend2.java:543)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){noformat}
> I was able to get more tests passing locally by increasing the max user
> process count on my machine. But the error suggests that there's an issue in
> the tests themselves. Not sure if the error seen locally is the same reason
> as why jenkins builds are failing, I wasn't able to confirm based on the
> jenkins builds' lack of output.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15711) Fix branch-2 builds
[
https://issues.apache.org/jira/browse/HADOOP-15711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623202#comment-16623202
]
Arun Suresh commented on HADOOP-15711:
--
Yeah... figured out with [~jhung] that isolating and ignoring specific tests
wont work since as you mentioned, when executed in the parallel, a thread
running a random batch of tests might go down.
Given that single YARN patch testing for instance should run fine (Am assuming
branch-2 Dockerfile is good currently since YARN-8658 ran fine), Can we disable
parallel tests for full builds then ? Will that help track down specific tests
that go OOM for instance ?
> Fix branch-2 builds
> ---
>
> Key: HADOOP-15711
> URL: https://issues.apache.org/jira/browse/HADOOP-15711
> Project: Hadoop Common
> Issue Type: Task
>Reporter: Jonathan Hung
>Priority: Critical
> Attachments: HADOOP-15711.001.branch-2.patch
>
>
> Branch-2 builds have been disabled for a while:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/
> A test run here causes hdfs tests to hang:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86-jhung/4/
> Running hadoop-hdfs tests locally reveal some errors such
> as:{noformat}[ERROR]
> testComplexAppend2(org.apache.hadoop.hdfs.TestFileAppend2) Time elapsed:
> 0.059 s <<< ERROR!
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1164)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1128)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:174)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1172)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:403)
> at
> org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:234)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1080)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:883)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:514)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:473)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend(TestFileAppend2.java:489)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend2(TestFileAppend2.java:543)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){noformat}
> I was able to get more tests passing locally by increasing the max user
> process count on my machine. But the error suggests that there's an issue in
> the tests themselves. Not sure if the error seen locally is the same reason
> as why jenkins builds are failing, I wasn't able to confirm based on the
> jenkins builds' lack of output.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15711) Fix branch-2 builds
[
https://issues.apache.org/jira/browse/HADOOP-15711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622769#comment-16622769
]
Arun Suresh commented on HADOOP-15711:
--
While I agree with [~aw]'s reservations.
* Partial information from the tests that do get run is better than no
information at all.
* Folks are working on patches/features targeting branch-2 that do not touch
HDFS code and hence probably the tests wont get executed anyway - atleast for
most YARN patches.
My vote is to commit the revert patch as part of this JIRA, \@ Ignore offending
tests for now and get them fixed as soon as possible ?
> Fix branch-2 builds
> ---
>
> Key: HADOOP-15711
> URL: https://issues.apache.org/jira/browse/HADOOP-15711
> Project: Hadoop Common
> Issue Type: Task
>Reporter: Jonathan Hung
>Priority: Critical
> Attachments: HADOOP-15711.001.branch-2.patch
>
>
> Branch-2 builds have been disabled for a while:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/
> A test run here causes hdfs tests to hang:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86-jhung/4/
> Running hadoop-hdfs tests locally reveal some errors such
> as:{noformat}[ERROR]
> testComplexAppend2(org.apache.hadoop.hdfs.TestFileAppend2) Time elapsed:
> 0.059 s <<< ERROR!
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1164)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1128)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:174)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1172)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:403)
> at
> org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:234)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1080)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:883)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:514)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:473)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend(TestFileAppend2.java:489)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend2(TestFileAppend2.java:543)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){noformat}
> I was able to get more tests passing locally by increasing the max user
> process count on my machine. But the error suggests that there's an issue in
> the tests themselves. Not sure if the error seen locally is the same reason
> as why jenkins builds are failing, I wasn't able to confirm based on the
> jenkins builds' lack of output.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15711) Fix branch-2 builds
[
https://issues.apache.org/jira/browse/HADOOP-15711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622318#comment-16622318
]
Arun Suresh commented on HADOOP-15711:
--
Attached a revert patch...
> Fix branch-2 builds
> ---
>
> Key: HADOOP-15711
> URL: https://issues.apache.org/jira/browse/HADOOP-15711
> Project: Hadoop Common
> Issue Type: Task
>Reporter: Jonathan Hung
>Priority: Critical
> Attachments: HADOOP-15711.001.branch-2.patch
>
>
> Branch-2 builds have been disabled for a while:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/
> A test run here causes hdfs tests to hang:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86-jhung/4/
> Running hadoop-hdfs tests locally reveal some errors such
> as:{noformat}[ERROR]
> testComplexAppend2(org.apache.hadoop.hdfs.TestFileAppend2) Time elapsed:
> 0.059 s <<< ERROR!
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1164)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1128)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:174)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1172)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:403)
> at
> org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:234)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1080)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:883)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:514)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:473)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend(TestFileAppend2.java:489)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend2(TestFileAppend2.java:543)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){noformat}
> I was able to get more tests passing locally by increasing the max user
> process count on my machine. But the error suggests that there's an issue in
> the tests themselves. Not sure if the error seen locally is the same reason
> as why jenkins builds are failing, I wasn't able to confirm based on the
> jenkins builds' lack of output.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15711) Fix branch-2 builds
[
https://issues.apache.org/jira/browse/HADOOP-15711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-15711:
-
Status: Patch Available (was: Open)
> Fix branch-2 builds
> ---
>
> Key: HADOOP-15711
> URL: https://issues.apache.org/jira/browse/HADOOP-15711
> Project: Hadoop Common
> Issue Type: Task
>Reporter: Jonathan Hung
>Priority: Critical
> Attachments: HADOOP-15711.001.branch-2.patch
>
>
> Branch-2 builds have been disabled for a while:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/
> A test run here causes hdfs tests to hang:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86-jhung/4/
> Running hadoop-hdfs tests locally reveal some errors such
> as:{noformat}[ERROR]
> testComplexAppend2(org.apache.hadoop.hdfs.TestFileAppend2) Time elapsed:
> 0.059 s <<< ERROR!
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1164)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1128)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:174)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1172)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:403)
> at
> org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:234)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1080)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:883)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:514)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:473)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend(TestFileAppend2.java:489)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend2(TestFileAppend2.java:543)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){noformat}
> I was able to get more tests passing locally by increasing the max user
> process count on my machine. But the error suggests that there's an issue in
> the tests themselves. Not sure if the error seen locally is the same reason
> as why jenkins builds are failing, I wasn't able to confirm based on the
> jenkins builds' lack of output.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15711) Fix branch-2 builds
[
https://issues.apache.org/jira/browse/HADOOP-15711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-15711:
-
Attachment: HADOOP-15711.001.branch-2.patch
> Fix branch-2 builds
> ---
>
> Key: HADOOP-15711
> URL: https://issues.apache.org/jira/browse/HADOOP-15711
> Project: Hadoop Common
> Issue Type: Task
>Reporter: Jonathan Hung
>Priority: Critical
> Attachments: HADOOP-15711.001.branch-2.patch
>
>
> Branch-2 builds have been disabled for a while:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/
> A test run here causes hdfs tests to hang:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86-jhung/4/
> Running hadoop-hdfs tests locally reveal some errors such
> as:{noformat}[ERROR]
> testComplexAppend2(org.apache.hadoop.hdfs.TestFileAppend2) Time elapsed:
> 0.059 s <<< ERROR!
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1164)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1128)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:174)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1172)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:403)
> at
> org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:234)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1080)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:883)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:514)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:473)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend(TestFileAppend2.java:489)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend2(TestFileAppend2.java:543)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){noformat}
> I was able to get more tests passing locally by increasing the max user
> process count on my machine. But the error suggests that there's an issue in
> the tests themselves. Not sure if the error seen locally is the same reason
> as why jenkins builds are failing, I wasn't able to confirm based on the
> jenkins builds' lack of output.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15446) WASB: PageBlobInputStream.skip breaks HBASE replication
[ https://issues.apache.org/jira/browse/HADOOP-15446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-15446: - Resolution: Fixed Status: Resolved (was: Patch Available) Resolving this. Please re-open if you need to brackport to a different branch > WASB: PageBlobInputStream.skip breaks HBASE replication > --- > > Key: HADOOP-15446 > URL: https://issues.apache.org/jira/browse/HADOOP-15446 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 2.9.0, 3.0.2 >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt >Priority: Major > Fix For: 2.10.0, 3.2.0 > > Attachments: HADOOP-15446-001.patch, HADOOP-15446-002.patch, > HADOOP-15446-003.patch, HADOOP-15446-branch-2.001.patch > > > Page Blobs are primarily used by HBASE. HBASE replication, which apparently > has not been used with WASB until recently, performs non-sequential reads on > log files using PageBlobInputStream. There are bugs in this stream > implementation which prevent skip and seek from working properly, and > eventually the stream state becomes corrupt and unusable. > I believe this bug affects all releases of WASB/HADOOP. It appears to be a > day-0 bug in PageBlobInputStream. There were similar bugs opened in the past > (HADOOP-15042) but the issue was not properly fixed, and no test coverage was > added. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15446) WASB: PageBlobInputStream.skip breaks HBASE replication
[ https://issues.apache.org/jira/browse/HADOOP-15446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-15446: - Fix Version/s: 3.2.0 2.10.0 > WASB: PageBlobInputStream.skip breaks HBASE replication > --- > > Key: HADOOP-15446 > URL: https://issues.apache.org/jira/browse/HADOOP-15446 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 2.9.0, 3.0.2 >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt >Priority: Major > Fix For: 2.10.0, 3.2.0 > > Attachments: HADOOP-15446-001.patch, HADOOP-15446-002.patch, > HADOOP-15446-003.patch, HADOOP-15446-branch-2.001.patch > > > Page Blobs are primarily used by HBASE. HBASE replication, which apparently > has not been used with WASB until recently, performs non-sequential reads on > log files using PageBlobInputStream. There are bugs in this stream > implementation which prevent skip and seek from working properly, and > eventually the stream state becomes corrupt and unusable. > I believe this bug affects all releases of WASB/HADOOP. It appears to be a > day-0 bug in PageBlobInputStream. There were similar bugs opened in the past > (HADOOP-15042) but the issue was not properly fixed, and no test coverage was > added. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15446) WASB: PageBlobInputStream.skip breaks HBASE replication
[ https://issues.apache.org/jira/browse/HADOOP-15446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470848#comment-16470848 ] Arun Suresh commented on HADOOP-15446: -- Committed this to branch-2 and branch-2.9 as well (I didn't see a difference in the code between branch-2 and trunk - so, instead of applying the provided patch, I just cherry-picked the trunk commit and ran the tests - which passed) > WASB: PageBlobInputStream.skip breaks HBASE replication > --- > > Key: HADOOP-15446 > URL: https://issues.apache.org/jira/browse/HADOOP-15446 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 2.9.0, 3.0.2 >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt >Priority: Major > Fix For: 2.10.0, 3.2.0 > > Attachments: HADOOP-15446-001.patch, HADOOP-15446-002.patch, > HADOOP-15446-003.patch, HADOOP-15446-branch-2.001.patch > > > Page Blobs are primarily used by HBASE. HBASE replication, which apparently > has not been used with WASB until recently, performs non-sequential reads on > log files using PageBlobInputStream. There are bugs in this stream > implementation which prevent skip and seek from working properly, and > eventually the stream state becomes corrupt and unusable. > I believe this bug affects all releases of WASB/HADOOP. It appears to be a > day-0 bug in PageBlobInputStream. There were similar bugs opened in the past > (HADOOP-15042) but the issue was not properly fixed, and no test coverage was > added. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15036) Update LICENSE.txt for HADOOP-14840
[ https://issues.apache.org/jira/browse/HADOOP-15036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh resolved HADOOP-15036. -- Resolution: Fixed Fix Version/s: 3.0.0 2.9.0 Thanks for the rev [~anu] Committed to trunk, branch-3.0, branch-2, branch-2.9 & branch-2.9.0 > Update LICENSE.txt for HADOOP-14840 > --- > > Key: HADOOP-15036 > URL: https://issues.apache.org/jira/browse/HADOOP-15036 > Project: Hadoop Common > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Fix For: 2.9.0, 3.0.0 > > Attachments: HADOOP-15036.001.patch > > > As noticed by [~anu]: > Looks like HADOOP-14840 added a dependency on “oj! Algorithms - version > 43.0”, but we have just added “oj! Algorithms - version 43.0” to the > “LICENSE.txt”. The right addition to the LICENESE.txt should contain the > original MIT License, especially “Copyright (c) 2003-2017 Optimatika”. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15036) Update LICENSE.txt for HADOOP-14840
[ https://issues.apache.org/jira/browse/HADOOP-15036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-15036: - Attachment: HADOOP-15036.001.patch Uploading initial patch. [~anu] Can you please take a look ? > Update LICENSE.txt for HADOOP-14840 > --- > > Key: HADOOP-15036 > URL: https://issues.apache.org/jira/browse/HADOOP-15036 > Project: Hadoop Common > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: HADOOP-15036.001.patch > > > As noticed by [~anu]: > Looks like HADOOP-14840 added a dependency on “oj! Algorithms - version > 43.0”, but we have just added “oj! Algorithms - version 43.0” to the > “LICENSE.txt”. The right addition to the LICENESE.txt should contain the > original MIT License, especially “Copyright (c) 2003-2017 Optimatika”. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15036) Update LICENSE.txt for HADOOP-14840
Arun Suresh created HADOOP-15036: Summary: Update LICENSE.txt for HADOOP-14840 Key: HADOOP-15036 URL: https://issues.apache.org/jira/browse/HADOOP-15036 Project: Hadoop Common Issue Type: Bug Reporter: Arun Suresh Assignee: Arun Suresh As noticed by [~anu]: Looks like HADOOP-14840 added a dependency on “oj! Algorithms - version 43.0”, but we have just added “oj! Algorithms - version 43.0” to the “LICENSE.txt”. The right addition to the LICENESE.txt should contain the original MIT License, especially “Copyright (c) 2003-2017 Optimatika”. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15025) Ensure singleton for ResourceEstimatorService
[
https://issues.apache.org/jira/browse/HADOOP-15025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245051#comment-16245051
]
Arun Suresh commented on HADOOP-15025:
--
The changes seem to have greatly simplified the tests. Thanks for the patch
[~subru] and [~Rui Li]
+1, pending jenkins
> Ensure singleton for ResourceEstimatorService
> -
>
> Key: HADOOP-15025
> URL: https://issues.apache.org/jira/browse/HADOOP-15025
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Subru Krishnan
>Assignee: Rui Li
> Attachments: HADOOP-15025-v1.patch, HADOOP-15025-v2.patch
>
>
> HADOOP-15013 fixed static findbugs warnings but this has lead to the
> singleton being broken for {{ResourceEstimatorService}}. This jira tracks the
> fix for the same.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15013) Fix ResourceEstimator findbugs issues
[ https://issues.apache.org/jira/browse/HADOOP-15013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-15013: - Resolution: Fixed Fix Version/s: 3.1.0 3.0.0 2.9.0 Status: Resolved (was: Patch Available) Thanks [~subru] and [~curino]. Committed to trunk, branch-2, branch-2.9 and branch-3.0 (oh so many branches !!) > Fix ResourceEstimator findbugs issues > - > > Key: HADOOP-15013 > URL: https://issues.apache.org/jira/browse/HADOOP-15013 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.9.0, 3.1.0 >Reporter: Allen Wittenauer >Assignee: Arun Suresh >Priority: Blocker > Fix For: 2.9.0, 3.0.0, 3.1.0 > > Attachments: YARN-7431.001.patch > > > Just see any recent report. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15013) Fix ResourceEstimator findbugs issues
[ https://issues.apache.org/jira/browse/HADOOP-15013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-15013: - Summary: Fix ResourceEstimator findbugs issues (was: Fix Resource Estimator findbugs issues) > Fix ResourceEstimator findbugs issues > - > > Key: HADOOP-15013 > URL: https://issues.apache.org/jira/browse/HADOOP-15013 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.9.0, 3.1.0 >Reporter: Allen Wittenauer >Assignee: Arun Suresh >Priority: Blocker > Attachments: YARN-7431.001.patch > > > Just see any recent report. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Moved] (HADOOP-15013) resource estimator has findbugs problems
[ https://issues.apache.org/jira/browse/HADOOP-15013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh moved YARN-7431 to HADOOP-15013: Affects Version/s: (was: 3.1.0) (was: 2.9.0) 3.1.0 2.9.0 Target Version/s: 2.9.0, 3.0.0, 3.1.0 (was: 2.9.0, 3.0.0, 3.1.0) Component/s: (was: resourcemanager) Key: HADOOP-15013 (was: YARN-7431) Project: Hadoop Common (was: Hadoop YARN) > resource estimator has findbugs problems > > > Key: HADOOP-15013 > URL: https://issues.apache.org/jira/browse/HADOOP-15013 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.9.0, 3.1.0 >Reporter: Allen Wittenauer >Assignee: Arun Suresh >Priority: Blocker > Attachments: YARN-7431.001.patch > > > Just see any recent report. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15013) Fix Resource Estimator findbugs issues
[ https://issues.apache.org/jira/browse/HADOOP-15013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-15013: - Summary: Fix Resource Estimator findbugs issues (was: resource estimator has findbugs problems) > Fix Resource Estimator findbugs issues > -- > > Key: HADOOP-15013 > URL: https://issues.apache.org/jira/browse/HADOOP-15013 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.9.0, 3.1.0 >Reporter: Allen Wittenauer >Assignee: Arun Suresh >Priority: Blocker > Attachments: YARN-7431.001.patch > > > Just see any recent report. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13556) Change Configuration.getPropsWithPrefix to use getProps instead of iterator
[ https://issues.apache.org/jira/browse/HADOOP-13556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-13556: - Fix Version/s: 3.0.0 2.9.0 > Change Configuration.getPropsWithPrefix to use getProps instead of iterator > --- > > Key: HADOOP-13556 > URL: https://issues.apache.org/jira/browse/HADOOP-13556 > Project: Hadoop Common > Issue Type: Bug >Reporter: Larry McCay >Assignee: Larry McCay > Fix For: 2.9.0, 3.0.0 > > Attachments: HADOOP-13556-001.patch, HADOOP-13556-002.patch > > > Current implementation of getPropsWithPrefix uses the > Configuration.iterator() method. This method is not threadsafe. > This patch will reimplement the gathering of properties that begin with a > prefix by using the safe getProps() method. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13556) Change Configuration.getPropsWithPrefix to use getProps instead of iterator
[ https://issues.apache.org/jira/browse/HADOOP-13556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-13556: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk, branch-3.0 and branch-2. Thanks [~lmccay] > Change Configuration.getPropsWithPrefix to use getProps instead of iterator > --- > > Key: HADOOP-13556 > URL: https://issues.apache.org/jira/browse/HADOOP-13556 > Project: Hadoop Common > Issue Type: Bug >Reporter: Larry McCay >Assignee: Larry McCay > Attachments: HADOOP-13556-001.patch, HADOOP-13556-002.patch > > > Current implementation of getPropsWithPrefix uses the > Configuration.iterator() method. This method is not threadsafe. > This patch will reimplement the gathering of properties that begin with a > prefix by using the safe getProps() method. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13556) Change Configuration.getPropsWithPrefix to use getProps instead of iterator
[ https://issues.apache.org/jira/browse/HADOOP-13556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201036#comment-16201036 ] Arun Suresh commented on HADOOP-13556: -- Looks straightforward. +1 > Change Configuration.getPropsWithPrefix to use getProps instead of iterator > --- > > Key: HADOOP-13556 > URL: https://issues.apache.org/jira/browse/HADOOP-13556 > Project: Hadoop Common > Issue Type: Bug >Reporter: Larry McCay >Assignee: Larry McCay > Attachments: HADOOP-13556-001.patch, HADOOP-13556-002.patch > > > Current implementation of getPropsWithPrefix uses the > Configuration.iterator() method. This method is not threadsafe. > This patch will reimplement the gathering of properties that begin with a > prefix by using the safe getProps() method. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13091) DistCp masks potential CRC check failures
[
https://issues.apache.org/jira/browse/HADOOP-13091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-13091:
-
Is this still on target for 2.9.0 ? If not, can we we push this out to the next
major release ?
> DistCp masks potential CRC check failures
> -
>
> Key: HADOOP-13091
> URL: https://issues.apache.org/jira/browse/HADOOP-13091
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Elliot West
>Assignee: Yiqun Lin
> Attachments: HADOOP-13091.003.patch, HADOOP-13091.004.patch,
> HDFS-10338.001.patch, HDFS-10338.002.patch
>
>
> There appear to be edge cases whereby CRC checks may be circumvented when
> requests for checksums from the source or target file system fail. In this
> event CRCs could differ between the source and target and yet the DistCp copy
> would succeed, even when the 'skip CRC check' option is not being used.
> The code in question is contained in the method
> [{{org.apache.hadoop.tools.util.DistCpUtils#checksumsAreEqual(...)}}|https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/DistCpUtils.java#L457]
> Specifically this code block suggests that if there is a failure when trying
> to read the source or target checksum then the method will return {{true}}
> (i.e. the checksums are equal), implying that the check succeeded. In actual
> fact we just failed to obtain the checksum and could not perform the check.
> {code}
> try {
> sourceChecksum = sourceChecksum != null ? sourceChecksum :
> sourceFS.getFileChecksum(source);
> targetChecksum = targetFS.getFileChecksum(target);
> } catch (IOException e) {
> LOG.error("Unable to retrieve checksum for " + source + " or "
> + target, e);
> }
> return (sourceChecksum == null || targetChecksum == null ||
> sourceChecksum.equals(targetChecksum));
> {code}
> I believe that at the very least the caught {{IOException}} should be
> re-thrown. If this is not deemed desirable then I believe an option
> ({{--strictCrc}}?) should be added to enforce a strict check where we require
> that both the source and target CRCs are retrieved, are not null, and are
> then compared for equality. If for any reason either of the CRCs retrievals
> fail then an exception is thrown.
> Clearly some {{FileSystems}} do not support CRCs and invocations to
> {{FileSystem.getFileChecksum(...)}} return {{null}} in these instances. I
> would suggest that these should fail a strict CRC check to prevent users
> developing a false sense of security in their copy pipeline.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14507) extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret,key
[
https://issues.apache.org/jira/browse/HADOOP-14507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-14507:
-
Is this still on target for 2.9.0 ? If not, can we we push this out to the next
major release ?
> extend per-bucket secret key config with explicit getPassword() on
> fs.s3a.$bucket.secret,key
>
>
> Key: HADOOP-14507
> URL: https://issues.apache.org/jira/browse/HADOOP-14507
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14507-001.patch
>
>
> Per-bucket jceks support turns out to be complex as you have to manage
> multiple jecks files & configure the client to ask for the right one. This is
> because we're calling {{Configuration.getPassword{"fs,s3a.secret.key")}.
> If before that, we do a check for the explict id, key, session key in the
> properties {{fs.s3a.$bucket.secret}} ( & c), we could have a single JCEKs
> file with all the secrets for different bucket. You would only need to
> explicitly point the base config to the secrets file, and the right
> credentials would be picked up, if set
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14845) Azure wasb: getFileStatus not making any auth checks
[
https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-14845:
-
Is this still on target for 2.9.0 ? If not, can we we push this out to the next
major release ?
> Azure wasb: getFileStatus not making any auth checks
>
>
> Key: HADOOP-14845
> URL: https://issues.apache.org/jira/browse/HADOOP-14845
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure, security
>Affects Versions: 2.8.0, 2.7.4
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch,
> HADOOP-14845.003.patch, HADOOP-14845-branch-2-001.patch.txt,
> HADOOP-14845-branch-2-002.patch, HADOOP-14845-branch-2-003.patch
>
>
> The HDFS spec requires only traverse checks for any file accessed via
> getFileStatus ... and since WASB does not support traverse checks, removing
> this call effectively removed all protections for the getFileStatus call. The
> reasoning at that time was that doing a performAuthCheck was the wrong thing
> to do, since it was going against the specand that the correct fix to the
> getFileStatus issue was to implement traverse checks rather than go against
> the spec by calling performAuthCheck. The side-effects of such a change were
> not fully clear at that time, but the thinking was that it was safer to
> remain true to the spec, as far as possible.
> The reasoning remains correct even today. But in view of the security hole
> introduced by this change (that anyone can load up any other user's data in
> hive), and keeping in mind that WASB does not intend to implement traverse
> checks, we propose a compromise.
> We propose (re)introducing a read-access check to getFileStatus(), that would
> check the existing ancestor for read-access whenever invoked. Although not
> perfect (in that it is a departure from the spec), we believe that it is a
> good compromise between having no checks at all; and implementing full-blown
> traverse checks.
> For scenarios that deal with intermediate folders like mkdirs, the call would
> check for read access against an existing ancestor (when invoked from shell)
> for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/"
> exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar"
> }}. This can be thought of, as being a close-enough substitute for the
> traverse checks that hdfs does.
> For other scenarios that don't deal with non-existent intermediate folders –
> like read, delete etc, the check will happen against the parent. Once again,
> we can think of the read-check against the parent as a substitute for the
> traverse check, which can be customized for various users with ranger
> policies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13514) Upgrade maven surefire plugin to 2.19.1
[
https://issues.apache.org/jira/browse/HADOOP-13514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-13514:
-
Is this still on target for 2.9.0 ? If not, can we we push this out to the next
major release ?
> Upgrade maven surefire plugin to 2.19.1
> ---
>
> Key: HADOOP-13514
> URL: https://issues.apache.org/jira/browse/HADOOP-13514
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 2.8.0
>Reporter: Ewan Higgs
>Assignee: Akira Ajisaka
> Attachments: HADOOP-13514.002.patch, HADOOP-13514.003.patch,
> HADOOP-13514-addendum.01.patch, HADOOP-13514-testing.001.patch,
> HADOOP-13514-testing.002.patch, HADOOP-13514-testing.003.patch,
> HADOOP-13514-testing.004.patch, surefire-2.19.patch
>
>
> A lot of people working on Hadoop don't want to run all the tests when they
> develop; only the bits they're working on. Surefire 2.19 introduced more
> useful test filters which let us run a subset of the tests that brings the
> build time down from 'come back tomorrow' to 'grab a coffee'.
> For instance, if I only care about the S3 adaptor, I might run:
> {code}
> mvn test -Dmaven.javadoc.skip=true -Pdist,native -Djava.awt.headless=true
> \"-Dtest=org.apache.hadoop.fs.*, org.apache.hadoop.hdfs.*,
> org.apache.hadoop.fs.s3a.*\"
> {code}
> We can work around this by specifying the surefire version on the command
> line but it would be better, imo, to just update the default surefire used.
> {code}
> mvn test -Dmaven.javadoc.skip=true -Pdist,native -Djava.awt.headless=true
> \"-Dtest=org.apache.hadoop.fs.*, org.apache.hadoop.hdfs.*,
> org.apache.hadoop.fs.s3a.*\" -Dmaven-surefire-plugin.version=2.19.1
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14912) FairCallQueue may defer servicing calls
[ https://issues.apache.org/jira/browse/HADOOP-14912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-14912: - Is this still on target for 2.9.0 ? If not, can we we push this out to the next major release ? > FairCallQueue may defer servicing calls > --- > > Key: HADOOP-14912 > URL: https://issues.apache.org/jira/browse/HADOOP-14912 > Project: Hadoop Common > Issue Type: Bug > Components: ipc >Affects Versions: 2.9.0, 3.0.0-alpha4, 2.8.2 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HADOOP-14912.patch > > > HADOOP-14033 switched a lock to a semaphore to allow concurrency for > producers & consumers to the underlying queues. A race condition was created > that may cause a consumer to acquire a permit but not extract an element, > leaving the semaphore with fewer permits than queued elements. This causes a > minimum number of calls to always be present in the call queue. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11626) Comment ReadStatistics to indicate that it tracks the actual read occurred
[
https://issues.apache.org/jira/browse/HADOOP-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-11626:
-
Is this still on target for 2.9.0 ? If not, can we we push this out to the next
major release ?
> Comment ReadStatistics to indicate that it tracks the actual read occurred
> --
>
> Key: HADOOP-11626
> URL: https://issues.apache.org/jira/browse/HADOOP-11626
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Trivial
> Attachments: HADOOP-11626.000.patch, HADOOP-11626.001.patch,
> HADOOP-11626.002.patch
>
>
> In {{DFSOutputStream#actualGetFromOneDataNode()}}, it updates the
> {{ReadStatistics}} even the read is failed:
> {code}
> int nread = reader.readAll(buf, offset, len);
> updateReadStatistics(readStatistics, nread, reader);
> if (nread != len) {
> throw new IOException("truncated return from reader.read(): " +
> "excpected " + len + ", got " + nread);
> }
> {code}
> It indicates that {{ReadStatistics}} tracks actual read occurred. Need to add
> comment to {{ReadStatistics}} to make this clear.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14519) Client$Connection#waitForWork may suffer from spurious wakeups
[
https://issues.apache.org/jira/browse/HADOOP-14519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-14519:
-
Is this still on target for 2.9.0 ? If not, can we we push this out to the next
major release ?
> Client$Connection#waitForWork may suffer from spurious wakeups
> --
>
> Key: HADOOP-14519
> URL: https://issues.apache.org/jira/browse/HADOOP-14519
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.8.0
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HADOOP-14519.001.patch
>
>
> {{Client$Connection#waitForWork}} may suffer spurious wakeup because the
> {{wait}} is not surrounded by a loop. See
> [https://docs.oracle.com/javase/7/docs/api/java/lang/Object.html#wait()].
> {code:title=Client$Connection#waitForWork}
> if (calls.isEmpty() && !shouldCloseConnection.get() && running.get()) {
> long timeout = maxIdleTime-
> (Time.now()-lastActivity.get());
> if (timeout>0) {
> try {
> wait(timeout); << spurious wakeup
> } catch (InterruptedException e) {}
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12455) fs.Globber breaks on colon in filename; doesn't use Path's handling for colons
[
https://issues.apache.org/jira/browse/HADOOP-12455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-12455:
-
Is this still on target for 2.9.0 ? If not, can we we push this out to the next
major release ?
> fs.Globber breaks on colon in filename; doesn't use Path's handling for colons
> --
>
> Key: HADOOP-12455
> URL: https://issues.apache.org/jira/browse/HADOOP-12455
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.2
>Reporter: Daniel Barclay
>Assignee: Rich Haase
> Attachments: HADOOP-12455.patch
>
>
> {{org.apache.hadoop.fs.Globber.glob()}} breaks when a searched directory
> contains a file whose simple name contains a colon.
> The problem seem to be in the code currently at lines 258 and 257
> [https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java#L257]:
> {noformat}
> 256: // Set the child path based on the parent path.
> 257: child.setPath(new Path(candidate.getPath(),
> 258: child.getPath().getName()));
> {noformat}
> That last line should probably be:
> {noformat}
> new Path(null, null, child.getPath().getName(;
> {noformat}
>
> The bug in the current code is that:
> 1) {{child.getPath().getName()}} gets the simple name (last segment) of the
> child {{Path}} as a _raw_ string (not necessarily the corresponding relative
> _{{Path}}_ string), and
> 2) that raw string is passed as {{Path(Path, String)}}'s second argument,
> which takes a _{{Path}}_ string.
> When that raw string contains a colon (e.g., {{xxx:yyy}}), it looks like a
> {{Path}} string that specifies a scheme ("{{xxx}}") and has a relative path
> "{{yyy}}}"--but that combination isn't allowed, so trying to constructing a
> {{Path}} with it (as {{Path(Path, String)}} does inside) throws an exception,
> aborting the entire {{glob()}} call.
>
> Adding the call to {{Path(String, String, String)}} does the equivalent of
> converting the raw string "{{xxx:yyy}}" to the {{Path}} string
> "{{./xxx:yyy}}", so the part before the colon is not taken as a scheme.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13102) Update GroupsMapping documentation to reflect the new changes
[
https://issues.apache.org/jira/browse/HADOOP-13102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-13102:
-
Is this still on target for 2.9.0 ? If not, can we we push this out to the next
major release ?
> Update GroupsMapping documentation to reflect the new changes
> -
>
> Key: HADOOP-13102
> URL: https://issues.apache.org/jira/browse/HADOOP-13102
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.8.0
>Reporter: Anu Engineer
>Assignee: Esther Kundin
> Labels: docs, ldap
> Attachments: HADOOP-13102-001.patch
>
>
> We need to update this section in the groupsMapping.md
> {noformat}
> Line 84:
> The implementation does not attempt to resolve group hierarchies. Therefore,
> a user must be an explicit member of a group object
> in order to be considered a member.
> {noformat}
> With changes in Hadoop-12291 this is no longer true since we will have the
> ability to walk the group hierarchies.
> We also should modify this line
> {noformat}
> Line : 81
> It is possible to set a maximum time limit when searching and awaiting a
> result.
> Set `hadoop.security.group.mapping.ldap.directory.search.timeout` to 0 if
> infinite wait period is desired. Default is 10,000 milliseconds (10 seconds).
> {noformat}
> we might want to document how the new settings affects the timeout.
> and also add the new settings into this doc.
> {noformat}
> hadoop.security.group.mapping.ldap.search.group.hierarchy.levels
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14565) Azure: Add Authorization support to ADLS
[ https://issues.apache.org/jira/browse/HADOOP-14565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-14565: - Is this still on target for 2.9.0 ? If not, can we we push this out to the next major release ? > Azure: Add Authorization support to ADLS > > > Key: HADOOP-14565 > URL: https://issues.apache.org/jira/browse/HADOOP-14565 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Affects Versions: 2.8.0 >Reporter: Ryan Waters >Assignee: Sivaguru Sankaridurg > Attachments: > HADOOP_14565__Added_authorizer_functionality_to_ADL_driver.patch > > > This task is meant to add an Authorizer interface to be used by the ADLS > driver in a similar way to the one used by WASB. The primary difference in > functionality being that the implementation of this Authorizer will be > provided by an external jar. This class will be specified through > configuration using "adl.external.authorization.class". > If this configuration is provided, an instance of the provided class will be > created and all file system calls will be passed through the authorizer, > allowing implementations to determine if the file path and access type > (create, open, delete, etc.) being requested is valid. If the requested > implementation class is not found or it fails to initialize, it will fail > initialization of the ADL driver. If no configuration is provided, calls to > the authorizer will be skipped and the driver will behave as it did > previously. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14870) backport HADOOP-14553 parallel tests to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-14870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-14870: - Is this still on target for 2.9.0 ? If not, can we we push this out to the next major release ? > backport HADOOP-14553 parallel tests to branch-2 > > > Key: HADOOP-14870 > URL: https://issues.apache.org/jira/browse/HADOOP-14870 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14870-branch-2-001.patch, > HADOOP-14870-branch-2-002.patch > > > Backport the HADOOP-14553 parallel test running from trunk to branch-2. > There's some complexity related to The FS Contract base test being JUnit4 in > branch -2, so its not a simple cherrypick. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14176) distcp reports beyond physical memory limits on 2.X
[
https://issues.apache.org/jira/browse/HADOOP-14176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-14176:
-
Is this still on target for 2.9.0 ? If not, can we we push this out to the next
major release ?
> distcp reports beyond physical memory limits on 2.X
> ---
>
> Key: HADOOP-14176
> URL: https://issues.apache.org/jira/browse/HADOOP-14176
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 2.9.0
>Reporter: Fei Hui
>Assignee: Fei Hui
> Attachments: HADOOP-14176-branch-2.001.patch,
> HADOOP-14176-branch-2.002.patch, HADOOP-14176-branch-2.003.patch,
> HADOOP-14176-branch-2.004.patch
>
>
> When i run distcp, i get some errors as follow
> {quote}
> 17/02/21 15:31:18 INFO mapreduce.Job: Task Id :
> attempt_1487645941615_0037_m_03_0, Status : FAILED
> Container [pid=24661,containerID=container_1487645941615_0037_01_05] is
> running beyond physical memory limits. Current usage: 1.1 GB of 1 GB physical
> memory used; 4.0 GB of 5 GB virtual memory used. Killing container.
> Dump of the process-tree for container_1487645941615_0037_01_05 :
> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> |- 24661 24659 24661 24661 (bash) 0 0 108650496 301 /bin/bash -c
> /usr/lib/jvm/java/bin/java -Djava.net.preferIPv4Stack=true
> -Dhadoop.metrics.log.level=WARN -Xmx2120m
> -Djava.io.tmpdir=/mnt/disk4/yarn/usercache/hadoop/appcache/application_1487645941615_0037/container_1487645941615_0037_01_05/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir=/mnt/disk2/log/hadoop-yarn/containers/application_1487645941615_0037/container_1487645941615_0037_01_05
> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
> -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.1.208
> 44048 attempt_1487645941615_0037_m_03_0 5
> 1>/mnt/disk2/log/hadoop-yarn/containers/application_1487645941615_0037/container_1487645941615_0037_01_05/stdout
>
> 2>/mnt/disk2/log/hadoop-yarn/containers/application_1487645941615_0037/container_1487645941615_0037_01_05/stderr
> |- 24665 24661 24661 24661 (java) 1766 336 4235558912 280699
> /usr/lib/jvm/java/bin/java -Djava.net.preferIPv4Stack=true
> -Dhadoop.metrics.log.level=WARN -Xmx2120m
> -Djava.io.tmpdir=/mnt/disk4/yarn/usercache/hadoop/appcache/application_1487645941615_0037/container_1487645941615_0037_01_05/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir=/mnt/disk2/log/hadoop-yarn/containers/application_1487645941615_0037/container_1487645941615_0037_01_05
> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
> -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.1.208
> 44048 attempt_1487645941615_0037_m_03_0 5
> Container killed on request. Exit code is 143
> Container exited with a non-zero exit code 143
> {quote}
> Deep into the code , i find that because distcp configuration covers
> mapred-site.xml
> {code}
>
> mapred.job.map.memory.mb
> 1024
>
>
> mapred.job.reduce.memory.mb
> 1024
>
> {code}
> When mapreduce.map.java.opts and mapreduce.map.memory.mb is setting in
> mapred-default.xml, and the value is larger than setted in
> distcp-default.xml, the error maybe occur.
> we should remove those two configurations in distcp-default.xml
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14521) KMS client needs retry logic
[ https://issues.apache.org/jira/browse/HADOOP-14521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-14521: - Is this still on target for 2.9.0 ? If not, can we we push this out to the next major release ? > KMS client needs retry logic > > > Key: HADOOP-14521 > URL: https://issues.apache.org/jira/browse/HADOOP-14521 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.6.0 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah > Attachments: HADOOP-14521.09.patch, HADOOP-14521.11.patch, > HADOOP-14521-branch-2.8.002.patch, HADOOP-14521-branch-2.8.2.patch, > HADOOP-14521-trunk-10.patch, HDFS-11804-branch-2.8.patch, > HDFS-11804-trunk-1.patch, HDFS-11804-trunk-2.patch, HDFS-11804-trunk-3.patch, > HDFS-11804-trunk-4.patch, HDFS-11804-trunk-5.patch, HDFS-11804-trunk-6.patch, > HDFS-11804-trunk-7.patch, HDFS-11804-trunk-8.patch, HDFS-11804-trunk.patch > > > The kms client appears to have no retry logic – at all. It's completely > decoupled from the ipc retry logic. This has major impacts if the KMS is > unreachable for any reason, including but not limited to network connection > issues, timeouts, the +restart during an upgrade+. > This has some major ramifications: > # Jobs may fail to submit, although oozie resubmit logic should mask it > # Non-oozie launchers may experience higher rates if they do not already have > retry logic. > # Tasks reading EZ files will fail, probably be masked by framework reattempts > # EZ file creation fails after creating a 0-length file – client receives > EDEK in the create response, then fails when decrypting the EDEK > # Bulk hadoop fs copies, and maybe distcp, will prematurely fail -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14624) Add GenericTestUtils.DelayAnswer that accept slf4j logger API
[ https://issues.apache.org/jira/browse/HADOOP-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-14624: - Is this still on target for 2.9.0 ? If not, can we we push this out to the next major release ? > Add GenericTestUtils.DelayAnswer that accept slf4j logger API > - > > Key: HADOOP-14624 > URL: https://issues.apache.org/jira/browse/HADOOP-14624 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Wenxin He >Assignee: Wenxin He > Attachments: HADOOP-14624.001.patch, HADOOP-14624.002.patch > > > Split from HADOOP-14539. > Now GenericTestUtils.DelayAnswer only accepts commons-logging logger API. Now > we are migrating the APIs to slf4j, slf4j logger API should be accepted as > well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-14768: - Is this still on target for 2.9.0 ? If not, can we we push this out to the next major release ? > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.8.1 >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch, > HADOOP-14768.004.patch, HADOOP-14768.005.patch, HADOOP-14768.006.patch, > HADOOP-14768.007.patch, HADOOP-14768-branch-2-008.patch, > HADOOP-14768-branch-2-009.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. > Note : Sticky bit implementation for 'Rename' operation is not done as part > of this JIRA -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14095) Document caveats about the default JavaKeyStoreProvider in KMS
[ https://issues.apache.org/jira/browse/HADOOP-14095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-14095: - Is this still on target for 2.9.0 ? If not, can we we push this out to the next major release ? > Document caveats about the default JavaKeyStoreProvider in KMS > -- > > Key: HADOOP-14095 > URL: https://issues.apache.org/jira/browse/HADOOP-14095 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation, kms >Affects Versions: 2.6.0 >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HADOOP-14095.01.patch > > > KMS doc provides and example to use JavaKeyStoreProvider. But we should > document the caveats of using it and setting it up, specifically about > keystore passwords. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13887) Support for client-side encryption in S3A file system
[ https://issues.apache.org/jira/browse/HADOOP-13887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-13887: - Is this still on target for 2.9.0 ? If not, can we we push this out to the next major release ? > Support for client-side encryption in S3A file system > - > > Key: HADOOP-13887 > URL: https://issues.apache.org/jira/browse/HADOOP-13887 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Jeeyoung Kim >Assignee: Igor Mazur >Priority: Minor > Attachments: HADOOP-13887-002.patch, HADOOP-13887-007.patch, > HADOOP-13887-branch-2-003.patch, HADOOP-13897-branch-2-004.patch, > HADOOP-13897-branch-2-005.patch, HADOOP-13897-branch-2-006.patch, > HADOOP-13897-branch-2-008.patch, HADOOP-13897-branch-2-009.patch, > HADOOP-13897-branch-2-010.patch, HADOOP-13897-branch-2-012.patch, > HADOOP-13897-branch-2-014.patch, HADOOP-13897-trunk-011.patch, > HADOOP-13897-trunk-013.patch, HADOOP-14171-001.patch, S3-CSE Proposal.pdf > > > Expose the client-side encryption option documented in Amazon S3 > documentation - > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingClientSideEncryption.html > Currently this is not exposed in Hadoop but it is exposed as an option in AWS > Java SDK, which Hadoop currently includes. It should be trivial to propagate > this as a parameter passed to the S3client used in S3AFileSystem.java -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14396) Add builder interface to FileContext
[
https://issues.apache.org/jira/browse/HADOOP-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-14396:
-
Is this still on target for 2.9.0 ? If not, can we we push this out to the next
major release ?
> Add builder interface to FileContext
>
>
> Key: HADOOP-14396
> URL: https://issues.apache.org/jira/browse/HADOOP-14396
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 2.9.0, 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HADOOP-14396.00.patch
>
>
> Add builder interface for {{FileContext#create}} and {{FileContext#append}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-10738) Dynamically adjust distcp configuration by adding distcp-site.xml into code base
[ https://issues.apache.org/jira/browse/HADOOP-10738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-10738: - Is this still on target for 2.9.0 ? If not, can we we push this out to the next major release ? > Dynamically adjust distcp configuration by adding distcp-site.xml into code > base > > > Key: HADOOP-10738 > URL: https://issues.apache.org/jira/browse/HADOOP-10738 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Siqi Li > Labels: BB2015-05-TBR > Attachments: HADOOP-10738-branch-2.001.patch, HADOOP-10738.v1.patch, > HADOOP-10738.v2.patch > > > For now, the configuration of distcp resides in hadoop-distcp.jar. This makes > it difficult to adjust the configuration dynamically. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9844) NPE when trying to create an error message response of RPC
[ https://issues.apache.org/jira/browse/HADOOP-9844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-9844: Is this still on target for 2.9.0 ? If not, can we we push this out to the next major release ? > NPE when trying to create an error message response of RPC > -- > > Key: HADOOP-9844 > URL: https://issues.apache.org/jira/browse/HADOOP-9844 > Project: Hadoop Common > Issue Type: Bug > Components: ipc >Affects Versions: 2.7.1, 3.0.0-alpha1 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-9844-001.patch, HADOOP-9844-002.patch, > HADOOP-9844-003.patch > > > I'm seeing an NPE which is raised when the server is trying to create an > error response to send back to the caller and there is no error text. > The root cause is probably somewhere in SASL, but sending something back to > the caller would seem preferable to NPE-ing server-side. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13730) After 5 connection failures, yarn stops sending metrics graphite until restarted
[ https://issues.apache.org/jira/browse/HADOOP-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-13730: - Is this still on target for 2.9.0 ? If not, can we we push this out to the next major release ? > After 5 connection failures, yarn stops sending metrics graphite until > restarted > > > Key: HADOOP-13730 > URL: https://issues.apache.org/jira/browse/HADOOP-13730 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 2.7.2 >Reporter: Sean Young >Priority: Minor > Attachments: > 0001-Graphite-can-be-unreachable-for-some-time-and-come-b.patch > > > We've had issues in production where metrics stopped. We found the following > in the log files: > 2016-09-02 21:44:32,493 WARN org.apache.hadoop.metrics2.sink.GraphiteSink: > Error sending metrics to Graphite > java.net.SocketException: Broken pipe > at java.net.SocketOutputStream.socketWrite0(Native Method) > at > java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:120) > at java.net.SocketOutputStream.write(SocketOutputStream.java:164) > at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233) > at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:294) > at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:137) > at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:147) > at java.io.OutputStreamWriter.write(OutputStreamWriter.java:270) > at java.io.Writer.write(Writer.java:154) > at > org.apache.hadoop.metrics2.sink.GraphiteSink$Graphite.write(GraphiteSink.java:170) > at > org.apache.hadoop.metrics2.sink.GraphiteSink.putMetrics(GraphiteSink.java:98) > at > org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186) > at > org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43) > at > org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87) > at > org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134) > at > org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88) > 2016-09-03 00:03:04,335 WARN org.apache.hadoop.metrics2.sink.GraphiteSink: > Error sending metrics to Graphite > java.net.SocketException: Broken pipe > at java.net.SocketOutputStream.socketWrite0(Native Method) > at > java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:120) > at java.net.SocketOutputStream.write(SocketOutputStream.java:164) > at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233) > at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:294) > at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:137) > at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:147) > at java.io.OutputStreamWriter.write(OutputStreamWriter.java:270) > at java.io.Writer.write(Writer.java:154) > at > org.apache.hadoop.metrics2.sink.GraphiteSink$Graphite.write(GraphiteSink.java:170) > at > org.apache.hadoop.metrics2.sink.GraphiteSink.putMetrics(GraphiteSink.java:98) > at > org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186) > at > org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43) > at > org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87) > at > org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134) > at > org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88) > 2016-09-03 00:20:35,436 WARN org.apache.hadoop.metrics2.sink.GraphiteSink: > Error sending metrics to Graphite > java.net.SocketException: Connection timed out > at java.net.SocketOutputStream.socketWrite0(Native Method) > at > java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:120) > at java.net.SocketOutputStream.write(SocketOutputStream.java:164) > at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233) > at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:294) > at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:137) > at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:147) > at java.io.OutputStreamWriter.write(OutputStreamWriter.java:270) > at java.io.Writer.write(Writer.java:154) > at > org.apache.hadoop.metrics2.sink.GraphiteSink$Graphite.write(GraphiteSink.java:170) > at > org.apache.hadoop.metrics2.sink.GraphiteSink.putMetrics(GraphiteSink.java:98) > at >
[jira] [Updated] (HADOOP-13556) Change Configuration.getPropsWithPrefix to use getProps instead of iterator
[ https://issues.apache.org/jira/browse/HADOOP-13556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-13556: - Is this still on target for 2.9.0 ? If not, can we we push this out to the next major release ? > Change Configuration.getPropsWithPrefix to use getProps instead of iterator > --- > > Key: HADOOP-13556 > URL: https://issues.apache.org/jira/browse/HADOOP-13556 > Project: Hadoop Common > Issue Type: Bug >Reporter: Larry McCay >Assignee: Larry McCay > Attachments: HADOOP-13556-001.patch, HADOOP-13556-002.patch > > > Current implementation of getPropsWithPrefix uses the > Configuration.iterator() method. This method is not threadsafe. > This patch will reimplement the gathering of properties that begin with a > prefix by using the safe getProps() method. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13592) Outputs errors and warnings by checkstyle at compile time
[ https://issues.apache.org/jira/browse/HADOOP-13592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-13592: - Is this still on target for 2.9.0 ? If not, can we we push this out to the next major release ? > Outputs errors and warnings by checkstyle at compile time > - > > Key: HADOOP-13592 > URL: https://issues.apache.org/jira/browse/HADOOP-13592 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Tsuyoshi Ozawa > Attachments: HADOOP-13592.001.patch > > > Currently, Apache Hadoop has lots checkstyle errors and warnings, but it's > not outputted at compile time. This prevents us from fixing the errors and > warnings. > We should output errors and warnings at compile time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12889) Make kdiag something services can use directly on startup
[ https://issues.apache.org/jira/browse/HADOOP-12889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-12889: - Is this still on target for 2.9.0 ? If not, can we we push this out to the next major release ? > Make kdiag something services can use directly on startup > - > > Key: HADOOP-12889 > URL: https://issues.apache.org/jira/browse/HADOOP-12889 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-12289-002.patch, HADOOP-12889-001.patch > > > I want the ability to start kdiag as a service launches, without doing > anything with side-effects other than usual UGI Init (that is: no keytab > login), and hook this up so that services can start it. Then add an option > for the YARN and HDFS services to do this on launch (Default: off) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9657) NetUtils.wrapException to have special handling for 0.0.0.0 addresses and :0 ports
[
https://issues.apache.org/jira/browse/HADOOP-9657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-9657:
Is this still on target for 2.9.0 ? If not, can we we push this out to the next
major release ?
> NetUtils.wrapException to have special handling for 0.0.0.0 addresses and :0
> ports
> --
>
> Key: HADOOP-9657
> URL: https://issues.apache.org/jira/browse/HADOOP-9657
> Project: Hadoop Common
> Issue Type: Improvement
> Components: net
>Affects Versions: 2.7.0
>Reporter: Steve Loughran
>Assignee: Varun Saxena
>Priority: Minor
> Attachments: HADOOP-9657.01.patch, HADOOP-9657.02.patch
>
>
> when an exception is wrapped, it may look like {{0.0.0.0:0 failed on
> connection exception: java.net.ConnectException: Connection refused; For more
> details see: http://wiki.apache.org/hadoop/ConnectionRefused}}
> We should recognise all zero ip addresses and 0 ports and flag them as "your
> configuration of the endpoint is wrong", as it is clearly the case
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14510) Use error code detail in AWS server responses for finer grained exceptions
[
https://issues.apache.org/jira/browse/HADOOP-14510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-14510:
-
Target Version/s: 3.1.0 (was: 2.9.0)
> Use error code detail in AWS server responses for finer grained exceptions
> --
>
> Key: HADOOP-14510
> URL: https://issues.apache.org/jira/browse/HADOOP-14510
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>
> {{S3Utils.translateException()}} maps HTTP status code to exceptions. We
> aren't looking at the body of the reponses though, except when handling a 301
> redirect.
> We should use the exit code to fine tune responses, especially 400 & 401/403.
> Right now I'm not sure we are even getting that error code into the text.
> see: http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14365) Stabilise FileSystem builder-based create API
[
https://issues.apache.org/jira/browse/HADOOP-14365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-14365:
-
Target Version/s: 3.1.0 (was: 2.9.0, 3.0.0)
> Stabilise FileSystem builder-based create API
> --
>
> Key: HADOOP-14365
> URL: https://issues.apache.org/jira/browse/HADOOP-14365
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Lei (Eddy) Xu
>
> HDFS-11170 added a builder-based create API for file creation which has a few
> issues to work out before it can be considered ready for use
> 1. There no specification in the filesystem.md of what it is meant to do,
> which means there's no public documentation on expected behaviour except on
> the Javadocs, which consists of the sentences "Create a new
> FSDataOutputStreamBuilder for the file with path" and "Base of specific file
> system FSDataOutputStreamBuilder".
> I propose:
> # Give the new method a relevant name rather than just define the return
> type, e.g. {{createFile()}}.
> # `Filesystem.md` to be extended with coverage of this method, and, sadly for
> the authors, coverage of what the semantics of
> {{FSDataOutputStreamBuilder.build()}} are.
> 2. There are only tests for HDFS and local, neither of them perfect.
> Proposed: move to {{AbstractContractCreateTest}}, test for all filesystems,
> fix tests and FS where appropriate.
> 3. Add more tests to generate the failure conditions implied by the updated
> filesystem spec. Eg. create over a an existing file, create over a directory,
> create with negative buffer size, negative block size, empty dest path, etc,
> etc.
> This will clarify when precondition checks are made, as well as whether. For
> example: should {{newFSDataOutputStreamBuilder()}} validate the path
> immediately?
> 4. Add to {{FileContext}}.
> 5. Take the opportunity to look at the flaws in today's {{create()}} calls
> and address them, rather than replicate. In particular, I'd like to end the
> behaviour "create all parent dirs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12876) [Azure Data Lake] Support for process level FileStatus cache to optimize GetFileStatus frequent opeations
[ https://issues.apache.org/jira/browse/HADOOP-12876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-12876: - Target Version/s: 3.1.0 (was: 2.9.0) > [Azure Data Lake] Support for process level FileStatus cache to optimize > GetFileStatus frequent opeations > - > > Key: HADOOP-12876 > URL: https://issues.apache.org/jira/browse/HADOOP-12876 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, fs/adl, tools >Reporter: Vishwajeet Dusane >Assignee: Vishwajeet Dusane > > Add support to cache GetFileStatus and ListStatus response locally for > limited period of time. Local cache for limited period of time would optimize > number of calls for GetFileStatus operation. > One of the example where local limited period cache would be useful - > terasort ListStatus on input directory follows with GetFileStatus operation > on each file within directory. For 2048 input files in a directory would save > 2048 GetFileStatus calls during start up (Using the ListStatus response to > cache FileStatus instances). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13308) S3A delete and rename may fail to preserve parent directory.
[ https://issues.apache.org/jira/browse/HADOOP-13308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-13308: - Target Version/s: 3.1.0 (was: 2.9.0) > S3A delete and rename may fail to preserve parent directory. > > > Key: HADOOP-13308 > URL: https://issues.apache.org/jira/browse/HADOOP-13308 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Chris Nauroth >Priority: Minor > > When a file or directory is deleted or renamed in S3A, and the result of that > operation makes the parent empty, S3A must store a fake directory (a pure > metadata object) at the parent to indicate that the directory still exists. > The logic for restoring fake directories is not resilient to a process death. > This may cause a directory to vanish unexpectedly after a deletion or rename > of its last child. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14132) Filesystem discovery to stop loading implementation classes
[ https://issues.apache.org/jira/browse/HADOOP-14132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-14132: - Target Version/s: 3.1.0 (was: 2.9.0) > Filesystem discovery to stop loading implementation classes > --- > > Key: HADOOP-14132 > URL: https://issues.apache.org/jira/browse/HADOOP-14132 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/adl, fs/azure, fs/oss, fs/s3, fs/swift >Affects Versions: 2.7.3 >Reporter: Steve Loughran >Assignee: Steve Loughran > > Integration testing of Hadoop with the HADOOP-14040 has shown up that the > move to a shaded AWS JAR is slowing all hadoop client code down. > I believe this is due to how we use service discovery to identify FS > implementations: the implementation classes themselves are instantiated. > This has known problems today with classloading, but clearly impacts > performance too, especially with complex transitive dependencies unique to > the loaded class. > Proposed: have lightweight service declaration classes which implement an > interface declaring > # schema > # classname of FileSystem impl > # classname of AbstractFS impl > # homepage (for third party code, support, etc) > These are what we register and scan in the FS to look for services. > This will leave the question about what to do for existing filesystems? I > think we'll need to retain the old code for external ones, while moving the > hadoop modules to the new ones -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13726) Enforce that FileSystem initializes only a single instance of the requested FileSystem.
[
https://issues.apache.org/jira/browse/HADOOP-13726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-13726:
-
Target Version/s: 3.1.0 (was: 2.9.0)
> Enforce that FileSystem initializes only a single instance of the requested
> FileSystem.
> ---
>
> Key: HADOOP-13726
> URL: https://issues.apache.org/jira/browse/HADOOP-13726
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Reporter: Chris Nauroth
>Assignee: Manjunath Anand
> Attachments: HADOOP-13726.001.patch
>
>
> The {{FileSystem}} cache is intended to guarantee reuse of instances by
> multiple call sites or multiple threads. The current implementation does
> provide this guarantee, but there is a brief race condition window during
> which multiple threads could perform redundant initialization. If the file
> system implementation has expensive initialization logic, then this is
> wasteful. This issue proposes to eliminate that race condition and guarantee
> initialization of only a single instance.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-10850) KerberosAuthenticator should not do the SPNEGO handshake
[
https://issues.apache.org/jira/browse/HADOOP-10850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-10850:
-
Target Version/s: 3.1.0 (was: 2.9.0)
> KerberosAuthenticator should not do the SPNEGO handshake
>
>
> Key: HADOOP-10850
> URL: https://issues.apache.org/jira/browse/HADOOP-10850
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Attachments: HADOOP-10850.patch, testFailures.png, testorder.patch
>
>
> As mentioned in HADOOP-10453, the JDK automatically does a SPNEGO handshake
> when opening a connection with a URL within a Kerberos login context, there
> is no need to do the SPNEGO handshake in the {{KerberosAuthenticator}},
> simply extract the auth token (hadoop-auth cookie) and do the fallback if
> necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14723) reinstate URI parameter in AWSCredentialProvider constructors
[ https://issues.apache.org/jira/browse/HADOOP-14723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-14723: - Target Version/s: 3.1.0 (was: 2.9.0, 3.0.0) > reinstate URI parameter in AWSCredentialProvider constructors > - > > Key: HADOOP-14723 > URL: https://issues.apache.org/jira/browse/HADOOP-14723 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > > I need to revert HADOOP-14135 "Remove URI parameter in AWSCredentialProvider > constructors", as knowing the bucket in use is needed for > * HADOOP-14507: per bucket secrets in JCEKS files > * HADOOP-14556: delegation tokens in S3A > these providers need the URI as it needs to it to decide which keys to scan > for/what token to look up. > I know we pulled it out to allow us to talk to DDB without needing a FS URI, > but for these specific cases, it is needed —we just won't be able to use the > specific auth providers to talk to AWS except to an S3 bucket. > Rather than just revert the patch, I propose waiting for s3guard phase I to > be merged in to trunk, then do it, with the JCEKS auth mech being set up to > skip looking for a per-bucket secret and key if it doesn't know its bucket > name. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13204) Über-jira: S3a phase III: scale and tuning
[ https://issues.apache.org/jira/browse/HADOOP-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-13204: - Target Version/s: 3.1.0 (was: 2.9.0, 3.0.0) > Über-jira: S3a phase III: scale and tuning > -- > > Key: HADOOP-13204 > URL: https://issues.apache.org/jira/browse/HADOOP-13204 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > > S3A Phase III work; post 2.8. > Areas could include > * customisation > * performance enhancement > * management and metrics > * error/failure handling > And of course any bugs which surface -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14695) Allow disabling chunked encoding on S3 client
[
https://issues.apache.org/jira/browse/HADOOP-14695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-14695:
-
Target Version/s: 3.1.0 (was: 2.9.0, 2.8.3, 3.0.0)
> Allow disabling chunked encoding on S3 client
> -
>
> Key: HADOOP-14695
> URL: https://issues.apache.org/jira/browse/HADOOP-14695
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>
> [Using the Amazon S3 Adapter for Snowball
> |http://docs.aws.amazon.com/snowball/latest/ug/using-adapter.html] indicates
> that we need to disable chunked coding and set path style access.
> HADOOP-12963 enables setting path style access.
> This JIRA will enable disabling chunked encoding. A new property
> {{fs.s3a.disable.chunked.encoding}} is proposed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14737) Sort out hadoop-common contract-test-options.xml
[ https://issues.apache.org/jira/browse/HADOOP-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-14737: - Target Version/s: 3.1.0 (was: 2.9.0, 3.0.0) > Sort out hadoop-common contract-test-options.xml > > > Key: HADOOP-14737 > URL: https://issues.apache.org/jira/browse/HADOOP-14737 > Project: Hadoop Common > Issue Type: Bug > Components: documentation, fs, test >Affects Versions: 2.8.0 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Minor > > Follow up to HADOOP-14103. Update hadoop-common testing.md in a similar > fashion. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13523) Implement a json format audit logger for KMS
[ https://issues.apache.org/jira/browse/HADOOP-13523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-13523: - Target Version/s: 3.1.0 (was: 2.9.0) > Implement a json format audit logger for KMS > > > Key: HADOOP-13523 > URL: https://issues.apache.org/jira/browse/HADOOP-13523 > Project: Hadoop Common > Issue Type: Improvement > Components: kms >Reporter: Xiao Chen >Assignee: Xiao Chen > > Follow-on work after HADOOP-13396. > A json format log extension will be added using the refactored API, and being > turned off by default. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14554) TestFileSystemOperationExceptionMessage to not rerun all of NativeAzureFileSystemBaseTest
[
https://issues.apache.org/jira/browse/HADOOP-14554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-14554:
-
Target Version/s: 3.1.0 (was: 2.9.0)
> TestFileSystemOperationExceptionMessage to not rerun all of
> NativeAzureFileSystemBaseTest
> -
>
> Key: HADOOP-14554
> URL: https://issues.apache.org/jira/browse/HADOOP-14554
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>
> the Wasb test {{TestFileSystemOperationExceptionMessage}} subclasses
> {{NativeAzureFileSystemBaseTest}} to test a new situation. As a result, when
> you run this test, it reruns all those base test cases.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14805) Upgrade to zstd 1.3.1
[ https://issues.apache.org/jira/browse/HADOOP-14805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-14805: - Target Version/s: 3.1.0 (was: 2.9.0, 3.0.0) > Upgrade to zstd 1.3.1 > - > > Key: HADOOP-14805 > URL: https://issues.apache.org/jira/browse/HADOOP-14805 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.9.0, 3.0.0-alpha2 >Reporter: Andrew Wang > > zstandard 1.3.1 has been dual licensed under GPL and BSD. This clears up the > concerns regarding the Facebook-specific PATENTS file. If we upgrade to > 1.3.1, we can bundle zstd with binary distributions of Hadoop. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12928) Update netty to 3.10.5.Final to sync with zookeeper
[ https://issues.apache.org/jira/browse/HADOOP-12928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-12928: - Target Version/s: 3.1.0 (was: 2.9.0, 3.0.0) > Update netty to 3.10.5.Final to sync with zookeeper > --- > > Key: HADOOP-12928 > URL: https://issues.apache.org/jira/browse/HADOOP-12928 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 2.7.2 >Reporter: Hendy Irawan >Assignee: Lei (Eddy) Xu > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-12928.01.patch, HADOOP-12928.02.patch, > HADOOP-12928.03.patch, HADOOP-12928-branch-2.00.patch, > HADOOP-12928-branch-2.01.patch, HADOOP-12928-branch-2.02.patch, > HDFS-12928.00.patch > > > Update netty to 3.7.1.Final because hadoop-client 2.7.2 depends on zookeeper > 3.4.6 which depends on netty 3.7.x. Related to HADOOP-12927 > Pull request: https://github.com/apache/hadoop/pull/85 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12708) SSL based embeded jetty for hadoop services should allow configuration for elect cipher suites
[ https://issues.apache.org/jira/browse/HADOOP-12708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-12708: - Target Version/s: 3.1.0 (was: 2.9.0) > SSL based embeded jetty for hadoop services should allow configuration for > elect cipher suites > -- > > Key: HADOOP-12708 > URL: https://issues.apache.org/jira/browse/HADOOP-12708 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Vijay Singh > > Currently all hadoop services use emebeded jetty server. Effort to exclude > cipher suite is being tracked under Hadoop-12668. However, hadoop should > allow to restrict the cipher suites using explicit list of cipher suites. > Current, pom dependency does not allow interface to include select cipher > suites. As a result, the options include following: > 1) Upgrade jetty from 6.x to 8.x or 9.x > 2) Contemplate and Implement WAR based implementation for these services > 3) Include ability to select specific cipher suites -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14025) Enhance AccessControlList to reject empty ugi#getGroups
[ https://issues.apache.org/jira/browse/HADOOP-14025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-14025: - Target Version/s: 3.1.0 (was: 2.9.0) > Enhance AccessControlList to reject empty ugi#getGroups > --- > > Key: HADOOP-14025 > URL: https://issues.apache.org/jira/browse/HADOOP-14025 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.6.0 >Reporter: Xiao Chen >Assignee: Xiao Chen > > It would be good to reconsider empty groups when checking user access. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11417) review filesystem seek logic, clarify/confirm spec, test & fix compliance
[
https://issues.apache.org/jira/browse/HADOOP-11417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-11417:
-
Target Version/s: 3.1.0 (was: 2.9.0)
> review filesystem seek logic, clarify/confirm spec, test & fix compliance
> -
>
> Key: HADOOP-11417
> URL: https://issues.apache.org/jira/browse/HADOOP-11417
> Project: Hadoop Common
> Issue Type: Task
> Components: fs, fs/s3, fs/swift
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
> HADOOP-11270 implies there's a diff in the way HDFS seeks and the object
> stores on the action {{seek(len(file))}}
> # review what HDFS does, add contract test to exactly demonstrate HDFS
> behaviour.
> # ensure FS spec is consistent with this
> # test/audit all supported filesystems to verify consistent behaviour
> # fix where appropriate
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14579) Add Kerberos and Delegation token support to ADLS client
[
https://issues.apache.org/jira/browse/HADOOP-14579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-14579:
-
Target Version/s: 3.1.0 (was: 2.9.0, 2.8.3, 3.0.0)
> Add Kerberos and Delegation token support to ADLS client
>
>
> Key: HADOOP-14579
> URL: https://issues.apache.org/jira/browse/HADOOP-14579
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/adl
>Affects Versions: 2.8.0
>Reporter: Anandsagar Kothapalli
>Assignee: Santhosh G Nayak
>
> Current implementation of Azure storage client for Hadoop ({{ADLS}}) does not
> support Kerberos Authentication and FileSystem authorization, which makes it
> unusable in secure environments with multi user setup.
> To make {{ADLS}} client more suitable to run in Secure environments,
> HADOOP-14565 is under way for providing the authorization and fine grained
> access control support.
> This JIRA is created to add Kerberos and delegation token support to {{ADLS}}
> client to fetch Azure Storage SAS keys, which provides fine grained timed
> access to containers and blobs.
> For delegation token management, the proposal is it use the same REST service
> which being used to generate the SAS Keys.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14206) TestSFTPFileSystem#testFileExists failure: Invalid encoding for signature
[
https://issues.apache.org/jira/browse/HADOOP-14206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-14206:
-
Target Version/s: 3.1.0 (was: 2.9.0)
> TestSFTPFileSystem#testFileExists failure: Invalid encoding for signature
> -
>
> Key: HADOOP-14206
> URL: https://issues.apache.org/jira/browse/HADOOP-14206
> Project: Hadoop Common
> Issue Type: Test
> Components: fs, test
>Affects Versions: 2.9.0
>Reporter: John Zhuge
>
> https://builds.apache.org/job/PreCommit-HADOOP-Build/11862/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_121.txt:
> {noformat}
> Tests run: 9, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 10.454 sec
> <<< FAILURE! - in org.apache.hadoop.fs.sftp.TestSFTPFileSystem
> testFileExists(org.apache.hadoop.fs.sftp.TestSFTPFileSystem) Time elapsed:
> 0.19 sec <<< ERROR!
> java.io.IOException: com.jcraft.jsch.JSchException: Session.connect:
> java.security.SignatureException: Invalid encoding for signature
> at com.jcraft.jsch.Session.connect(Session.java:565)
> at com.jcraft.jsch.Session.connect(Session.java:183)
> at
> org.apache.hadoop.fs.sftp.SFTPConnectionPool.connect(SFTPConnectionPool.java:168)
> at
> org.apache.hadoop.fs.sftp.SFTPFileSystem.connect(SFTPFileSystem.java:149)
> at
> org.apache.hadoop.fs.sftp.SFTPFileSystem.getFileStatus(SFTPFileSystem.java:663)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1626)
> at
> org.apache.hadoop.fs.sftp.TestSFTPFileSystem.testFileExists(TestSFTPFileSystem.java:190)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> at
> org.apache.hadoop.fs.sftp.SFTPConnectionPool.connect(SFTPConnectionPool.java:180)
> at
> org.apache.hadoop.fs.sftp.SFTPFileSystem.connect(SFTPFileSystem.java:149)
> at
> org.apache.hadoop.fs.sftp.SFTPFileSystem.getFileStatus(SFTPFileSystem.java:663)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1626)
> at
> org.apache.hadoop.fs.sftp.TestSFTPFileSystem.testFileExists(TestSFTPFileSystem.java:190)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13607) Specify and test contract for FileSystem#close.
[
https://issues.apache.org/jira/browse/HADOOP-13607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-13607:
-
Target Version/s: 3.1.0 (was: 2.9.0)
> Specify and test contract for FileSystem#close.
> ---
>
> Key: HADOOP-13607
> URL: https://issues.apache.org/jira/browse/HADOOP-13607
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Reporter: Chris Nauroth
>
> This issue proposes to enhance the {{FileSystem}} specification by describing
> the expected semantics of {{FileSystem#close}} and adding corresponding
> contract tests. Notable aspects are that the method must be idempotent as
> dictated by {{java.io.Closeable}} and closing also interacts with the
> delete-on-exit feature.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14778) AzureNativeFileSystemStore.getDirectorySet() to use getTrimmedStrings to get directory listings
[
https://issues.apache.org/jira/browse/HADOOP-14778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-14778:
-
Target Version/s: 3.1.0 (was: 2.9.0)
> AzureNativeFileSystemStore.getDirectorySet() to use getTrimmedStrings to get
> directory listings
> ---
>
> Key: HADOOP-14778
> URL: https://issues.apache.org/jira/browse/HADOOP-14778
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Priority: Minor
>
> {{AzureNativeFileSystemStore.getDirectorySet()}} should switch to
> {{Configuration.getTrimmedStrings()}} so the list of directories get
> whitespace and line endings removed.
> As it stands, all paths need to go one the same line, without whitespace
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13846) S3A to implement rename(final Path src, final Path dst, final Rename... options)
[
https://issues.apache.org/jira/browse/HADOOP-13846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-13846:
-
Target Version/s: 3.1.0 (was: 2.9.0)
> S3A to implement rename(final Path src, final Path dst, final Rename...
> options)
>
>
> Key: HADOOP-13846
> URL: https://issues.apache.org/jira/browse/HADOOP-13846
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>
> S3a now raises exceptions on invalid rename operations, but these get lost. I
> plan to use them in my s3guard committer HADOOP-13786.
> Rather than just make innerRename() private, S3A could implement
> {{FileSystem.rename(final Path src, final Path dst, final Rename...
> options)}} and so have an exception-raising rename which can be called
> without going more into the internals.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13144) Enhancing IPC client throughput via multiple connections per user
[
https://issues.apache.org/jira/browse/HADOOP-13144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-13144:
-
Target Version/s: 3.1.0 (was: 2.9.0)
> Enhancing IPC client throughput via multiple connections per user
> -
>
> Key: HADOOP-13144
> URL: https://issues.apache.org/jira/browse/HADOOP-13144
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
>Reporter: Jason Kace
>Priority: Minor
>
> The generic IPC client ({{org.apache.hadoop.ipc.Client}}) utilizes a single
> connection thread for each {{ConnectionId}}. The {{ConnectionId}} is unique
> to the connection's remote address, ticket and protocol. Each ConnectionId
> is 1:1 mapped to a connection thread by the client via a map cache.
> The result is to serialize all IPC read/write activity through a single
> thread for a each user/ticket + address. If a single user makes repeated
> calls (1k-100k/sec) to the same destination, the IPC client becomes a
> bottleneck.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13581) Expand FileSystem specification to describe differences between FileSystem and FileContext.
[
https://issues.apache.org/jira/browse/HADOOP-13581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-13581:
-
Target Version/s: 3.1.0 (was: 2.9.0)
> Expand FileSystem specification to describe differences between FileSystem
> and FileContext.
> ---
>
> Key: HADOOP-13581
> URL: https://issues.apache.org/jira/browse/HADOOP-13581
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Reporter: Chris Nauroth
>Priority: Minor
>
> Sometimes there are behavioral differences between using the {{FileSystem}}
> API and using the {{FileContext}} API. This issue proposes to enhance the
> specification to describe these differences in more detail.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13022) S3 MD5 check fails on Server Side Encryption-KMS with AWS and default key is used
[
https://issues.apache.org/jira/browse/HADOOP-13022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-13022:
-
Target Version/s: 3.1.0 (was: 2.9.0)
> S3 MD5 check fails on Server Side Encryption-KMS with AWS and default key is
> used
> -
>
> Key: HADOOP-13022
> URL: https://issues.apache.org/jira/browse/HADOOP-13022
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Leonardo Contreras
>Priority: Minor
>
> When server side encryption with "aws:kms" value and no custom key is used in
> S3A Filesystem, the AWSClient fails when verifing Md5:
> {noformat}
> Exception in thread "main" com.amazonaws.AmazonClientException: Unable to
> verify integrity of data upload. Client calculated content hash (contentMD5:
> 1B2M2Y8AsgTpgAmY7PhCfg== in base 64) didn't match hash (etag:
> c29fcc646e17c348bce9cca8f9d205f5 in hex) calculated by Amazon S3. You may
> need to delete the data stored in Amazon S3. (metadata.contentMD5: null,
> md5DigestStream:
> com.amazonaws.services.s3.internal.MD5DigestCalculatingInputStream@65d9e72a,
> bucketName: abuse-messages-nonprod, key:
> venus/raw_events/checkpoint/825eb6aa-543d-46b1-801f-42de9dbc1610/)
> at
> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1492)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.createEmptyObject(S3AFileSystem.java:1295)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.createFakeDirectory(S3AFileSystem.java:1272)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:969)
> at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1888)
> at
> org.apache.spark.SparkContext$$anonfun$setCheckpointDir$2.apply(SparkContext.scala:2077)
> at
> org.apache.spark.SparkContext$$anonfun$setCheckpointDir$2.apply(SparkContext.scala:2074)
> at scala.Option.map(Option.scala:145)
> at
> org.apache.spark.SparkContext.setCheckpointDir(SparkContext.scala:2074)
> at
> org.apache.spark.streaming.StreamingContext.checkpoint(StreamingContext.scala:237)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13637) improve setting of max connections in AWS client
[ https://issues.apache.org/jira/browse/HADOOP-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-13637: - Target Version/s: 3.1.0 (was: 2.9.0) > improve setting of max connections in AWS client > > > Key: HADOOP-13637 > URL: https://issues.apache.org/jira/browse/HADOOP-13637 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Priority: Minor > > things can go badly wrong if the S3A FS creates a thread pool for IO > than > the number of pooled AWS http connections (set by property > MAXIMUM_CONNECTIONS); you also need some for any other IO requests coming in. > The max connections property is currently independent of thread pool size, > and has a default value of 1. > this is why there is a troubleshooting section in the docs showing the stack > trace and instructions to fix". > Better: have a dynamic minimum like thread pool size + n, for a value of n to > be chosen. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13808) replace `Can't get Master Kerberos principal` message with useful information about principal and configuration item
[ https://issues.apache.org/jira/browse/HADOOP-13808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-13808: - Target Version/s: 3.1.0 (was: 2.9.0) > replace `Can't get Master Kerberos principal` message with useful information > about principal and configuration item > > > Key: HADOOP-13808 > URL: https://issues.apache.org/jira/browse/HADOOP-13808 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: 2.7.3 >Reporter: Steve Loughran > > Clients get the `Can't get Master Kerberos principal` error message when > their config doesn't identify the principal for a service. > This message is fairly useless as > # it doesn't identify the service for which a principal is sought > # it doesn't identify the configuration property looked for > # it doesn't provide any guidance of how to fix it > As a result, anyone seeing this doesn't have a clue what to do. > I propose > # naming the principal sought, e.g. "namenode", "YARN RM" > # naming the configuration item looked for > # adding a link to a wiki page covering the problem > KDiag should scan through the common set of principals if it doesn't already -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14552) Über-jira: WASB client phase II: performance and testing
[ https://issues.apache.org/jira/browse/HADOOP-14552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-14552: - Target Version/s: 3.1.0 (was: 2.9.0, 3.0.0) > Über-jira: WASB client phase II: performance and testing > > > Key: HADOOP-14552 > URL: https://issues.apache.org/jira/browse/HADOOP-14552 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 2.8.1 >Reporter: Steve Loughran > > Uber-JIRA for wasb:// phase II: the things we want for Hadoop 2.9. > I think the focus is on performance, but looking at the tests there are some > things we need there first: parallel test execution, move over to IT tests, > other tuning. > All patches must come with declarations of which azure endpoint they were > tested against. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13625) Document FileSystem actions that trigger update of modification time.
[ https://issues.apache.org/jira/browse/HADOOP-13625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-13625: - Target Version/s: 3.1.0 (was: 2.9.0) > Document FileSystem actions that trigger update of modification time. > - > > Key: HADOOP-13625 > URL: https://issues.apache.org/jira/browse/HADOOP-13625 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Reporter: Chris Nauroth > > Hadoop users and developers of Hadoop-compatible file systems have sometimes > asked questions about which file system actions trigger an update of the > path's modification time. This issue proposes to document which actions do > and do not update modification time, so that the information is easy to find > without reading HDFS code or manually testing individual cases. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13986) UGI.UgiMetrics.renewalFailureTotal is not printable
[
https://issues.apache.org/jira/browse/HADOOP-13986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-13986:
-
Target Version/s: 3.1.0 (was: 2.9.0, 3.0.0)
> UGI.UgiMetrics.renewalFailureTotal is not printable
> ---
>
> Key: HADOOP-13986
> URL: https://issues.apache.org/jira/browse/HADOOP-13986
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.8.0, 3.0.0-alpha2
>Reporter: Wei-Chiu Chuang
>Priority: Minor
>
> The metrics (renewalFailures and renewalFailuresTotal) in the following code
> snippet are not printable.
> {code:title=UserGroupInformation.java}
> metrics.renewalFailuresTotal.incr();
> final long tgtEndTime = tgt.getEndTime().getTime();
> LOG.warn("Exception encountered while running the renewal "
> + "command for {}. (TGT end time:{}, renewalFailures: {},"
> + "renewalFailuresTotal: {})", getUserName(), tgtEndTime,
> metrics.renewalFailures, metrics.renewalFailuresTotal, ie);
> {code}
> The output of the code is like the following:
> {quote}
> 2017-01-12 12:23:14,062 WARN security.UserGroupInformation
> (UserGroupInformation.java:run(1012)) - Exception encountered while running
> the renewal command for f...@example.com. (TGT end time:148425260,
> renewalFailures:
> org.apache.hadoop.metrics2.lib.MutableGaugeInt@323aa7f9,renewalFailuresTotal:
> org.apache.hadoop.metrics2.lib.MutableGaugeLong@c8af058)
> ExitCodeException exitCode=1: kinit: krb5_cc_get_principal: No credentials
> cache file found
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14184) Remove service loader config entry for ftp fs
[
https://issues.apache.org/jira/browse/HADOOP-14184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-14184:
-
Target Version/s: 3.1.0 (was: 2.9.0)
> Remove service loader config entry for ftp fs
> -
>
> Key: HADOOP-14184
> URL: https://issues.apache.org/jira/browse/HADOOP-14184
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.7.3
>Reporter: John Zhuge
>Priority: Minor
> Labels: newbie
>
> Per discussion in HADOOP-14132. Remove line
> {{org.apache.hadoop.fs.ftp.FTPFileSystem}} from the service loader config
> file
> hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
> and add property {{fs.ftp.impl}} to {{core-default.xml}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14612) Reduce memory copy in BlobOutputStreamInternal::dispatchWrite
[
https://issues.apache.org/jira/browse/HADOOP-14612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-14612:
-
Target Version/s: 3.1.0 (was: 2.9.0, 3.0.0)
> Reduce memory copy in BlobOutputStreamInternal::dispatchWrite
> -
>
> Key: HADOOP-14612
> URL: https://issues.apache.org/jira/browse/HADOOP-14612
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Rajesh Balamohan
>Priority: Minor
>
> Currently in {{BlobOutputStreamInternal::dispatchWrite}}, buffer is copied
> internally for every write. During large uploads this can be around 4 MB.
> This can be avoided if there is internal class which extends
> ByteArrayOutputStream with additional method "ByteArrayInputStream
> getInputStream()".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13728) S3A can support short user-friendly aliases for configuration of credential providers.
[ https://issues.apache.org/jira/browse/HADOOP-13728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-13728: - Target Version/s: 3.1.0 (was: 2.9.0) > S3A can support short user-friendly aliases for configuration of credential > providers. > -- > > Key: HADOOP-13728 > URL: https://issues.apache.org/jira/browse/HADOOP-13728 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Chris Nauroth >Priority: Minor > > This issue proposes to support configuration of the S3A credential provider > chain using short aliases to refer to the common credential providers in > addition to allowing full class names. Supporting short aliases would > provide a simpler operations experience for the most common cases. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13311) S3A shell entry point to support commands specific to S3A.
[
https://issues.apache.org/jira/browse/HADOOP-13311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-13311:
-
Target Version/s: 3.1.0 (was: 2.9.0)
> S3A shell entry point to support commands specific to S3A.
> --
>
> Key: HADOOP-13311
> URL: https://issues.apache.org/jira/browse/HADOOP-13311
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Chris Nauroth
>Priority: Minor
>
> Create a new {{s3a}} shell entry point. This can support diagnostic and
> administrative commands that are specific to S3A and wouldn't make sense to
> group under existing scripts like {{hadoop}} or {{hdfs}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13762) S3A: Set thread names with more specific information about the call.
[
https://issues.apache.org/jira/browse/HADOOP-13762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-13762:
-
Target Version/s: 3.1.0 (was: 2.9.0)
> S3A: Set thread names with more specific information about the call.
>
>
> Key: HADOOP-13762
> URL: https://issues.apache.org/jira/browse/HADOOP-13762
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Chris Nauroth
>
> Running {{jstack}} on a hung process and reading the stack traces is a
> helpful way to determine exactly what code in the process is stuck. This
> would be even more helpful if we included more descriptive information about
> the specific file system method call.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13733) Support WASB connections through an HTTP proxy server.
[ https://issues.apache.org/jira/browse/HADOOP-13733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-13733: - Target Version/s: 3.1.0 (was: 2.9.0) > Support WASB connections through an HTTP proxy server. > -- > > Key: HADOOP-13733 > URL: https://issues.apache.org/jira/browse/HADOOP-13733 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Chris Nauroth > > WASB currently does not support use of an HTTP proxy server to connect to the > Azure Storage back-end. The Azure Storage SDK does support use of a proxy, > so we can enhance WASB to read proxy settings from configuration and pass > them along in the Azure Storage SDK calls. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9363) AuthenticatedURL will NPE if server closes connection
[
https://issues.apache.org/jira/browse/HADOOP-9363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-9363:
Target Version/s: 3.1.0 (was: 2.9.0, 3.0.0)
> AuthenticatedURL will NPE if server closes connection
> -
>
> Key: HADOOP-9363
> URL: https://issues.apache.org/jira/browse/HADOOP-9363
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0-alpha1
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>
> A NPE occurs if the server unexpectedly closes the connection for an
> {{AuthenticatedURL}} w/o sending a response.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14591) wasb to support getStorageStatistics
[ https://issues.apache.org/jira/browse/HADOOP-14591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-14591: - Target Version/s: 3.1.0 (was: 2.9.0) > wasb to support getStorageStatistics > > > Key: HADOOP-14591 > URL: https://issues.apache.org/jira/browse/HADOOP-14591 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.8.1 >Reporter: Steve Loughran > > Wasb can support the storage statstics of HADOOP-13065, so any execution > framework gathering the statistics can include them, and tests can log them -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14753) wasb: Support FileContext
[
https://issues.apache.org/jira/browse/HADOOP-14753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-14753:
-
Target Version/s: 3.1.0 (was: 2.9.0, 2.8.3, 3.0.0)
> wasb: Support FileContext
> -
>
> Key: HADOOP-14753
> URL: https://issues.apache.org/jira/browse/HADOOP-14753
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: John Zhuge
>Priority: Minor
>
> Support the use of wasb with FileContext: {{FileContext.getFileContext(new
> URI("wasb://abc/"), conf)}}.
> * Add property {{fs.AbstractFileSystem.wasb.impl}} and
> {{fs.AbstractFileSystem.wasbs.impl}} to {{core-default.xml}}
> * Add FileContext contract tests. See ITestS3AFileContextURI and friends for
> example.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13754) Hadoop-Azure Update WASB URI format to support SAS token in it.
[ https://issues.apache.org/jira/browse/HADOOP-13754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-13754: - Target Version/s: 3.1.0 (was: 2.9.0) > Hadoop-Azure Update WASB URI format to support SAS token in it. > --- > > Key: HADOOP-13754 > URL: https://issues.apache.org/jira/browse/HADOOP-13754 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.7.3, 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Sumit Dubey >Assignee: Sumit Dubey > Attachments: HADOOP-13754-branch-2.7.3.patch > > Original Estimate: 3h > Remaining Estimate: 3h > > Currently Azure WASB adapter code supports wasb url in this format > wasb://[containername@]youraccount.blob.core.windows.net/testDir with the > credentials retrieved from configuration and scoped to a container. > With this change we want > 1) change the url to contain file level sas token in the url > wasb://[containername[:]]@youraccount.blob.core.windows.net/testDir > 2) Scope access to a blob/file level. > 3) Tests to test the new url format -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13250) jdiff and dependency reports aren't linked in site web pages
[ https://issues.apache.org/jira/browse/HADOOP-13250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-13250: - Target Version/s: 3.1.0 (was: 2.9.0, 3.0.0) > jdiff and dependency reports aren't linked in site web pages > > > Key: HADOOP-13250 > URL: https://issues.apache.org/jira/browse/HADOOP-13250 > Project: Hadoop Common > Issue Type: Bug > Components: build, documentation >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Vinod Kumar Vavilapalli > > Even though they are in the site tar ball (after HADOOP-13245), they aren't > actually reachable. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12877) [Azure Data Lake] Add Telemetry instrumentation
[ https://issues.apache.org/jira/browse/HADOOP-12877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-12877: - Target Version/s: 3.1.0 (was: 2.9.0) > [Azure Data Lake] Add Telemetry instrumentation > --- > > Key: HADOOP-12877 > URL: https://issues.apache.org/jira/browse/HADOOP-12877 > Project: Hadoop Common > Issue Type: New Feature > Components: fs, fs/adl, tools >Reporter: Vishwajeet Dusane >Assignee: Vishwajeet Dusane > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14840) Tool to estimate resource requirements of an application pipeline based on prior executions
[
https://issues.apache.org/jira/browse/HADOOP-14840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-14840:
-
Priority: Blocker (was: Major)
> Tool to estimate resource requirements of an application pipeline based on
> prior executions
> ---
>
> Key: HADOOP-14840
> URL: https://issues.apache.org/jira/browse/HADOOP-14840
> Project: Hadoop Common
> Issue Type: New Feature
> Components: tools
>Reporter: Subru Krishnan
>Assignee: Rui Li
>Priority: Blocker
> Attachments: ResourceEstimator-design-v1.pdf
>
>
> We have been working on providing SLAs for job execution on Hadoop. At high
> level this involves 2 parts: deriving the resource requirements of a job and
> guaranteeing the estimated resources at runtime. The {{YARN
> ReservationSystem}} (YARN-1051/YARN-2572/YARN-5326) enable the latter and in
> this JIRA, we propose to add a tool to Hadoop to predict the resource
> requirements of a job based on past executions of the job. The system (aka
> *Morpheus*) deep dive can be found in our OSDI'16 paper
> [here|https://www.usenix.org/conference/osdi16/technical-sessions/presentation/jyothi].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14771) hadoop-client does not include hadoop-yarn-client
[
https://issues.apache.org/jira/browse/HADOOP-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171101#comment-16171101
]
Arun Suresh commented on HADOOP-14771:
--
I guess you need to add one or more "allowed_expr" lines in
{{ensure-jars-have-correct-contents.sh}} to allow it to accept the following
lines:
{code}
javax/
javax/inject/
javax/inject/Inject.class
javax/inject/Named.class
javax/inject/Provider.class
javax/inject/Qualifier.class
javax/inject/Scope.class
javax/inject/Singleton.class
jersey/
jersey/repackaged/
jersey/repackaged/org/
jersey/repackaged/org/objectweb/
jersey/repackaged/org/objectweb/asm/
jersey/repackaged/org/objectweb/asm/AnnotationVisitor.class
jersey/repackaged/org/objectweb/asm/AnnotationWriter.class
jersey/repackaged/org/objectweb/asm/Attribute.class
jersey/repackaged/org/objectweb/asm/ByteVector.class
jersey/repackaged/org/objectweb/asm/ClassReader.class
jersey/repackaged/org/objectweb/asm/ClassVisitor.class
jersey/repackaged/org/objectweb/asm/ClassWriter.class
jersey/repackaged/org/objectweb/asm/Context.class
jersey/repackaged/org/objectweb/asm/Edge.class
jersey/repackaged/org/objectweb/asm/FieldVisitor.class
jersey/repackaged/org/objectweb/asm/FieldWriter.class
jersey/repackaged/org/objectweb/asm/Frame.class
jersey/repackaged/org/objectweb/asm/Handle.class
jersey/repackaged/org/objectweb/asm/Handler.class
jersey/repackaged/org/objectweb/asm/Item.class
jersey/repackaged/org/objectweb/asm/Label.class
jersey/repackaged/org/objectweb/asm/MethodVisitor.class
jersey/repackaged/org/objectweb/asm/MethodWriter.class
jersey/repackaged/org/objectweb/asm/Opcodes.class
jersey/repackaged/org/objectweb/asm/Type.class
jersey/repackaged/org/objectweb/asm/TypePath.class
{code}
> hadoop-client does not include hadoop-yarn-client
> -
>
> Key: HADOOP-14771
> URL: https://issues.apache.org/jira/browse/HADOOP-14771
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: common
>Reporter: Haibo Chen
>Assignee: Ajay Kumar
>Priority: Critical
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14771.01.patch, HADOOP-14771.02.patch
>
>
> The hadoop-client does not include hadoop-yarn-client, thus, the shared
> hadoop-client is incomplete.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14771) hadoop-client does not include hadoop-yarn-client
[ https://issues.apache.org/jira/browse/HADOOP-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170885#comment-16170885 ] Arun Suresh commented on HADOOP-14771: -- [~haibochen] / [~ajakumar], It looks like this has resulted in build failures - can you take a look ? > hadoop-client does not include hadoop-yarn-client > - > > Key: HADOOP-14771 > URL: https://issues.apache.org/jira/browse/HADOOP-14771 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Haibo Chen >Assignee: Ajay Kumar >Priority: Critical > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-14771.01.patch, HADOOP-14771.02.patch > > > The hadoop-client does not include hadoop-yarn-client, thus, the shared > hadoop-client is incomplete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16035218#comment-16035218
]
Arun Suresh commented on HADOOP-14445:
--
[~daryn], agreed that duplicating the entries is bad if the RM will blindly
renew all of them. Although, IIRC, the tokens are renewed only if they are
expired, so if they are renewed serially, it should not be a problem. But I do
agree, since RM also renews DTs on other app events as well (app recovery etc)
- for which duplicate renewal, might not be preventable.
bq. The cleanest way to manage a kms cluster is transparently via a cname or
multi-A record
True, in which case one would not even need the LoadBalancingKMSClientProvider.
I do like the idea of using a nameservice though, as [~yzhangal] suggested
which will ensure that we will still have only 1 single entry.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation, kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Rushabh S Shah
> Attachments: HADOOP-14445-branch-2.8.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16033346#comment-16033346
]
Arun Suresh commented on HADOOP-14445:
--
Not sure I follow. Why is the fact that RM depends on local conf to create the
key provider a bad thing ?
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation, kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Rushabh S Shah
> Attachments: HADOOP-14445-branch-2.8.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16033315#comment-16033315
]
Arun Suresh commented on HADOOP-14445:
--
[~shahrs87], As per
[SecurityUtil#buildTokenService|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java#L440-L453]
it should be the latter:
||TokenService||Token||
|{{kms01.example.com}}'s ip:9600|t1|
|{{kms02.example.com}}'s ip:9600|t1|
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation, kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Rushabh S Shah
> Attachments: HADOOP-14445-branch-2.8.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16032215#comment-16032215
]
Arun Suresh commented on HADOOP-14445:
--
[~yzhangal], Apologies for the delay
{quote}
Each host corresponds to one KMS, I thought we just need split the hosts part,
and make each one its own URI with the corresponding host, and use that for
tokenService key for each KMS.
{quote}
I think the KMSClientProvider.KeyProviderFactory does this.
{quote}
So when we collect a delegation token from one KMS, we associate the token with
each KMS and the corresponding URI.
{quote}
Yup... this was precisely what I was thinking.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation, kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Rushabh S Shah
> Attachments: HADOOP-14445-branch-2.8.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
