[jira] [Commented] (HADOOP-15607) AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
[
https://issues.apache.org/jira/browse/HADOOP-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564760#comment-16564760
]
Sammi Chen commented on HADOOP-15607:
-
I fired YETUS-652 to track the build issue. Will commit the patch to branch-2
and branch-2.9 soon.
> AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
> -
>
> Key: HADOOP-15607
> URL: https://issues.apache.org/jira/browse/HADOOP-15607
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.1.0, 2.10.0, 2.9.1, 3.2.0, 3.0.3
>Reporter: wujinhu
>Assignee: wujinhu
>Priority: Major
> Fix For: 3.2.0, 3.0.4, 3.1.2
>
> Attachments: HADOOP-15607-branch-2.001.patch, HADOOP-15607.001.patch,
> HADOOP-15607.002.patch, HADOOP-15607.003.patch, HADOOP-15607.004.patch
>
>
> When I generated data with hive-tpcds tool, I got exception below:
> 2018-07-16 14:50:43,680 INFO mapreduce.Job: Task Id :
> attempt_1531723399698_0001_m_52_0, Status : FAILED
> Error: com.aliyun.oss.OSSException: The list of parts was not in ascending
> order. Parts list must specified in order by part number.
> [ErrorCode]: InvalidPartOrder
> [RequestId]: 5B4C40425FCC208D79D1EAF5
> [HostId]: 100.103.0.137
> [ResponseError]:
>
>
> InvalidPartOrder
> The list of parts was not in ascending order. Parts list must
> specified in order by part number.
> 5B4C40425FCC208D79D1EAF5
> xx.xx.xx.xx
> current PartNumber 3, you given part number 3is not in
> ascending order
>
> at
> com.aliyun.oss.common.utils.ExceptionFactory.createOSSException(ExceptionFactory.java:99)
> at
> com.aliyun.oss.internal.OSSErrorResponseHandler.handle(OSSErrorResponseHandler.java:69)
> at
> com.aliyun.oss.common.comm.ServiceClient.handleResponse(ServiceClient.java:248)
> at
> com.aliyun.oss.common.comm.ServiceClient.sendRequestImpl(ServiceClient.java:130)
> at
> com.aliyun.oss.common.comm.ServiceClient.sendRequest(ServiceClient.java:68)
> at com.aliyun.oss.internal.OSSOperation.send(OSSOperation.java:94)
> at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:149)
> at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:113)
> at
> com.aliyun.oss.internal.OSSMultipartOperation.completeMultipartUpload(OSSMultipartOperation.java:185)
> at com.aliyun.oss.OSSClient.completeMultipartUpload(OSSClient.java:790)
> at
> org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.completeMultipartUpload(AliyunOSSFileSystemStore.java:643)
> at
> org.apache.hadoop.fs.aliyun.oss.AliyunOSSBlockOutputStream.close(AliyunOSSBlockOutputStream.java:120)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
> at
> org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:106)
> at
> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.close(MultipleOutputs.java:574)
> at org.notmysock.tpcds.GenTable$DSDGen.cleanup(GenTable.java:169)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686)
>
> I reviewed code below,
> {code:java}
> blockId {code}
> has thread synchronization problem
> {code:java}
> // code placeholder
> private void uploadCurrentPart() throws IOException {
> blockFiles.add(blockFile);
> blockStream.flush();
> blockStream.close();
> if (blockId == 0) {
> uploadId = store.getUploadId(key);
> }
> ListenableFuture partETagFuture =
> executorService.submit(() -> {
> PartETag partETag = store.uploadPart(blockFile, key, uploadId,
> blockId + 1);
> return partETag;
> });
> partETagsFutures.add(partETagFuture);
> blockFile = newBlockFile();
> blockId++;
> blockStream = new BufferedOutputStream(new FileOutputStream(blockFile));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
[
https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564734#comment-16564734
]
Aaron Fabbri commented on HADOOP-14154:
---
Ok.. I actually looked at the right patch today ;). This looks pretty good
overall. I'm excited to see this coming together. Inline comments:
{noformat}
@@ -518,12 +520,19 @@ public DirListingMetadata listChildren(final Path path)
throws IOException {
PathMetadata meta = itemToPathMetadata(item, username);
metas.add(meta);
}
+
+ PathMetadata dirPathMeta = get(path);
+ boolean isAuthoritative = false;
+ if(dirPathMeta != null) {
+isAuthoritative = dirPathMeta.isAuthoritativeDir();
+ }
{noformat}
We should try to avoid the change to the public API (PathMetadata) I would
create a new class (POJO) DDBPathMetadata which includes all the fields in
PathMetadata plus the isAuthoritative flag. This way we don't have to change
the public MetadataStore / PathMetadata interface.
DDBPathMetadata can have a constructor which takes a PathMetadata + isAuth flag
for when you are writing to dynamo. It can also have a method that creates
(transforms to) a PathMetadata for when you are reading. More ideas below...
{noformat}
if (deletionBatch.size() == S3GUARD_DDB_BATCH_WRITE_REQUEST_LIMIT) {
Thread.sleep(delay);
processBatchWriteRequest(pathToKey(deletionBatch), null);
+
+ // set authoritative false for each pruned dir listing
+ removeAuthoritativeDirFlag(parentPathSet);
+ parentPathSet.clear();
+
deletionBatch.clear();
}
}
if (deletionBatch.size() > 0) {
Thread.sleep(delay);
processBatchWriteRequest(pathToKey(deletionBatch), null);
+
+// set authoritative false for each pruned dir listing
+removeAuthoritativeDirFlag(parentPathSet);
+parentPathSet.clear();
{noformat}
looks good. You should get test coverage from existing
MetadataStoreTestBase#testPruneUnsetsAuthoritative case.
{noformat}
+ private void removeAuthoritativeDirFlag(Set pathSet) {
+pathSet.forEach(pp -> {
+ removeAuthoritativeDirFlag(pp);
+});
+ }
+
+ private void removeAuthoritativeDirFlag(Path path){
+try {
+ DirListingMetadata dlm = listChildren(path);
{noformat}
looks correct but could be optimized. We are getting the whole listing just
to clear the flag. I would probably have internal (private) getters/setters for
loading / storing a single DDBPathMetadata from dynamo. Then you can just deal
with the single row, instead of getting/putting the whole listing to set a
flag.
{noformat}
+ if(dlm == null){
{noformat}
nit: spacing (space after if and close paren.)
{noformat}
@@ -1157,7 +1206,7 @@ private static void checkPathMetadata(PathMetadata meta) {
map.put(WRITE_CAPACITY, throughput.getWriteCapacityUnits().toString());
map.put(TABLE, desc.toString());
map.put(MetadataStoreCapabilities.PERSISTS_AUTHORITATIVE_BIT,
- Boolean.toString(false));
+ Boolean.toString(true));
{noformat}
Woo hoo!
{noformat}
- public PathMetadata(FileStatus fileStatus, Tristate isEmptyDir, boolean
- isDeleted) {
+ public PathMetadata(FileStatus fileStatus, Tristate isEmptyDir,
+ boolean isDeleted) {
+this(fileStatus, isEmptyDir, isDeleted, false);
+ }
+
+ public PathMetadata(FileStatus fileStatus, Tristate isEmptyDir,
+ boolean isDeleted, boolean isAuthoritativeDir) {
{noformat}
IMO, this change should go away, instead happening on the dynamo-specific
PathMetadata POJO (e.g. DDBPathMetadata)
{noformat}
-return new PathMetadata(fileStatus, Tristate.UNKNOWN, isDeleted);
+return new PathMetadata(fileStatus, Tristate.UNKNOWN, isDeleted,
+isAuthoritativeDir);
}
{noformat}
here I'm thinking you could have a new private function, e.g.
private DDBPathMetadata itemToDDBPathMetadata()
then if you need an actual PathMetadata you can just
DDBPathMetadata dpm = itemToDDBPathMetadata(..);
...
return dpm.toPathMetadata()
{noformat}
+// set parent dir as authoritative
+if(!allowMissing()) {
{noformat}
nit: spacing
{noformat}
+ @Test
+ public void testPrunePreservesAuthoritative() throws Exception {
+ // assert that parent dir is still authoritative (no removed elements
+ // during prune)
+ assertFalse(rootDirMd.isAuthoritative());
+ assertFalse(grandParentDirMd.isAuthoritative());
+ assertTrue(parentDirMd.isAuthoritative());
+}
+ }
+
{noformat}
Nice, thank you for adding more test coverage to the base test class.
> Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)
> -
>
> Key: HADOOP-14154
> URL:
[jira] [Commented] (HADOOP-14840) Tool to estimate resource requirements of an application pipeline based on prior executions
[
https://issues.apache.org/jira/browse/HADOOP-14840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564651#comment-16564651
]
tangshangwen commented on HADOOP-14840:
---
hi [~Rui Li] [~subru], Is there a follow-up improvement plan for this feature
now?
> Tool to estimate resource requirements of an application pipeline based on
> prior executions
> ---
>
> Key: HADOOP-14840
> URL: https://issues.apache.org/jira/browse/HADOOP-14840
> Project: Hadoop Common
> Issue Type: New Feature
> Components: tools
>Reporter: Subru Krishnan
>Assignee: Rui Li
>Priority: Major
> Fix For: 2.9.0, 3.0.0, 3.1.0
>
> Attachments: HADOOP-14840-branch-2-v4.patch, HADOOP-14840-v1.patch,
> HADOOP-14840-v2.patch, HADOOP-14840-v3.patch, HADOOP-14840-v4.patch,
> ResourceEstimator-design-v1.pdf
>
>
> We have been working on providing SLAs for job execution on Hadoop. At high
> level this involves 2 parts: deriving the resource requirements of a job and
> guaranteeing the estimated resources at runtime. The {{YARN
> ReservationSystem}} (YARN-1051/YARN-2572/YARN-5326) enable the latter and in
> this JIRA, we propose to add a tool to Hadoop to predict the resource
> requirements of a job based on past executions of the job. The system (aka
> *Morpheus*) deep dive can be found in our OSDI'16 paper
> [here|https://www.usenix.org/conference/osdi16/technical-sessions/presentation/jyothi].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15642) Update to latest/recent version of aws-sdk for Hadoop 3.2
[
https://issues.apache.org/jira/browse/HADOOP-15642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564650#comment-16564650
]
genericqa commented on HADOOP-15642:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 44m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
93m 16s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
37s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 48m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 48m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 14s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
23s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
37s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
47s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}164m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HADOOP-15642 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933833/HADOOP-15642-003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml |
| uname | Linux 1af747e7ed1e 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c835fc0 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14974/artifact/out/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14974/testReport/ |
| Max. process+thread count | 337 (vs. ulimit of 1) |
| modules | C: hadoop-project hadoop-tools/hadoop-aws U: . |
| Console output |
[jira] [Commented] (HADOOP-15645) ITestS3GuardToolLocal.testDiffCommand fails if bucket has per-bucket binding to DDB
[
https://issues.apache.org/jira/browse/HADOOP-15645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564639#comment-16564639
]
genericqa commented on HADOOP-15645:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 39s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 19s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
17s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HADOOP-15645 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933840/HADOOP-15645-001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux a62a153f4717 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c835fc0 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14975/testReport/ |
| Max. process+thread count | 301 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14975/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ITestS3GuardToolLocal.testDiffCommand fails if bucket has per-bucket binding
> to DDB
> ---
>
>
[jira] [Commented] (HADOOP-15576) S3A Multipart Uploader to work with S3Guard and encryption
[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564625#comment-16564625
]
Steve Loughran commented on HADOOP-15576:
-
Playing with this
* tests take a v long time.
* they fail, at least for me
{code}
[INFO]
[ERROR] Failures:
[ERROR]
ITestS3AContractMultipartUploader>AbstractContractMultipartUploaderTest.testMultipartUpload:85->Assert.assertArrayEquals:294->Assert.internalArrayEquals:473
File contents differ: arrays first differed at element [0]; expected:<-123>
but was:<-44>
[ERROR]
ITestS3AContractMultipartUploader>AbstractContractMultipartUploaderTest.testMultipartUploadReverseOrder:122->Assert.assertArrayEquals:294->Assert.internalArrayEquals:473
File contents differ: arrays first differed at element [0]; expected:<-123>
but was:<-44>
[ERROR]
ITestS3AContractMultipartUploader>AbstractContractMultipartUploaderTest.testMultipartUploadReverseOrderNonContiguousPartNumbers:156->Assert.assertArrayEquals:294->Assert.internalArrayEquals:473
File contents differ: arrays first differed at element [0]; expected:<-47> but
was:<-44>
[INFO]
[ERROR] Tests run: 6, Failures: 3, Errors: 0, Skipped: 0
[INFO]
{code}
I'm going to do a revision of the patch, so I'll see if I can identify the
problem. I'll do what I can about scaling the patch down to be viable, and make
it a -Dscale test only.
> S3A Multipart Uploader to work with S3Guard and encryption
> ---
>
> Key: HADOOP-15576
> URL: https://issues.apache.org/jira/browse/HADOOP-15576
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Ewan Higgs
>Priority: Blocker
> Attachments: HADOOP-15576.001.patch, HADOOP-15576.002.patch,
> HADOOP-15576.003.patch, HADOOP-15576.004.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with
> the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper;
> adding any new methods are needed there.
> # Tests. the tests of HDFS-13713.
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and
> even if there was, it might not show that S3Guard was bypassed, because
> there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader,
> encryption settings are't picked up. Files being uploaded this way *are not
> being encrypted*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15398) StagingTestBase uses methods not available in Mockito 1.8.5
[
https://issues.apache.org/jira/browse/HADOOP-15398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564600#comment-16564600
]
genericqa commented on HADOOP-15398:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
58s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 31m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 31m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 19m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
95m 50s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 27m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 19m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 3s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
41s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}131m 40s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}324m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.client.impl.TestBlockReaderLocal |
| | hadoop.hdfs.TestPread |
| |
hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainersLauncher
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HADOOP-15398 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12922891/HADOOP-15398.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml |
| uname | Linux cbb15a55f2e6 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 4b540bb |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14972/artifact/out/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14972/testReport/ |
| Max. process+thread count | 3718 (vs. ulimit of 1) |
| modules | C: hadoop-project . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14972/console |
| Powered by | Apache
[jira] [Commented] (HADOOP-15642) Update to latest/recent version of aws-sdk for Hadoop 3.2
[ https://issues.apache.org/jira/browse/HADOOP-15642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564596#comment-16564596 ] Steve Loughran commented on HADOOP-15642: - PS: tests against US-west-1, with CLI tests done with s3guard enabled. > Update to latest/recent version of aws-sdk for Hadoop 3.2 > - > > Key: HADOOP-15642 > URL: https://issues.apache.org/jira/browse/HADOOP-15642 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15642-001.patch, HADOOP-15642-003.patch, Screen > Shot 2018-07-30 at 14.11.22.png > > > Move to a later version of the AWS SDK library for a different set of > features and issues. > proposed version: 1.11.375 > One thing which doesn't work on the one we ship with is the ability to create > assumed role sessions >1h, as there's a check in the client lib for > role-duration <= 3600 seconds. I'll assume more recent SDKs delegate duration > checks to the far end. > see: > [https://aws.amazon.com/about-aws/whats-new/2018/03/longer-role-sessions/] > * assuming later versions will extend assumed role life, docs will need > changing, > * Adding a test in HADOOP-15583 which expects an error message if you ask for > a duration of 3h; this should act as the test to see what happens. > * think this time would be good to explicitly write down the SDK update > process -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15645) ITestS3GuardToolLocal.testDiffCommand fails if bucket has per-bucket binding to DDB
[
https://issues.apache.org/jira/browse/HADOOP-15645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15645:
Assignee: Steve Loughran
Target Version/s: 3.2.0
Status: Patch Available (was: Open)
Testing: us-west-1 w/ test setup of S3Guard enabled in the bucket, options of
{{-Ds3guard}} and {{{-Ds3guard -Ddynamodb}}.
In the first of those options because my test bucket is per-bucket bonded to
s3, the local test is now skipped properly
> ITestS3GuardToolLocal.testDiffCommand fails if bucket has per-bucket binding
> to DDB
> ---
>
> Key: HADOOP-15645
> URL: https://issues.apache.org/jira/browse/HADOOP-15645
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-15645-001.patch
>
>
> If your bucket has a per-bucket setting to use S3Guard, then the
> ITestS3GuardToolLocal.testDiffCommand test can fail as fs-only data can creep
> into the metastore.
> The rawfs should use clearBucketOption to clear the metastore binding, so
> guarantee a real raw fs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15645) ITestS3GuardToolLocal.testDiffCommand fails if bucket has per-bucket binding to DDB
[
https://issues.apache.org/jira/browse/HADOOP-15645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564591#comment-16564591
]
Steve Loughran commented on HADOOP-15645:
-
Patch 001
* better setup of {{AbstractS3GuardToolTestBase}} implementations, especially
bucket config, where per-bucket settings are cleared
* {{ITestS3GuardToolDynamoDB}} and {{ITestS3GuardToolLocal}} both check the
metastore type of the test FS, skip if not correct -and include details on
store in the message.
* testDiff test erases test FS in testDiff setup through rawFS and metastore
FS, in case it gets contaminated
This addresses the double setup of dynamodb being picked up on a local test
run, the ddb metastore getting tainted with stuff which shouldn't be there, and
the s3 bucket getting stuff into it which the getFileSystem() fs doesn't
see/delete.
> ITestS3GuardToolLocal.testDiffCommand fails if bucket has per-bucket binding
> to DDB
> ---
>
> Key: HADOOP-15645
> URL: https://issues.apache.org/jira/browse/HADOOP-15645
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-15645-001.patch
>
>
> If your bucket has a per-bucket setting to use S3Guard, then the
> ITestS3GuardToolLocal.testDiffCommand test can fail as fs-only data can creep
> into the metastore.
> The rawfs should use clearBucketOption to clear the metastore binding, so
> guarantee a real raw fs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15642) Update to latest/recent version of aws-sdk for Hadoop 3.2
[ https://issues.apache.org/jira/browse/HADOOP-15642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564588#comment-16564588 ] Aaron Fabbri commented on HADOOP-15642: --- ok, on my list. keep bugging me if needed.. I also need to review your assumed role patch HADOOP-15583 > Update to latest/recent version of aws-sdk for Hadoop 3.2 > - > > Key: HADOOP-15642 > URL: https://issues.apache.org/jira/browse/HADOOP-15642 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15642-001.patch, HADOOP-15642-003.patch, Screen > Shot 2018-07-30 at 14.11.22.png > > > Move to a later version of the AWS SDK library for a different set of > features and issues. > proposed version: 1.11.375 > One thing which doesn't work on the one we ship with is the ability to create > assumed role sessions >1h, as there's a check in the client lib for > role-duration <= 3600 seconds. I'll assume more recent SDKs delegate duration > checks to the far end. > see: > [https://aws.amazon.com/about-aws/whats-new/2018/03/longer-role-sessions/] > * assuming later versions will extend assumed role life, docs will need > changing, > * Adding a test in HADOOP-15583 which expects an error message if you ask for > a duration of 3h; this should act as the test to see what happens. > * think this time would be good to explicitly write down the SDK update > process -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15645) ITestS3GuardToolLocal.testDiffCommand fails if bucket has per-bucket binding to DDB
[ https://issues.apache.org/jira/browse/HADOOP-15645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15645: Attachment: HADOOP-15645-001.patch > ITestS3GuardToolLocal.testDiffCommand fails if bucket has per-bucket binding > to DDB > --- > > Key: HADOOP-15645 > URL: https://issues.apache.org/jira/browse/HADOOP-15645 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Priority: Minor > Attachments: HADOOP-15645-001.patch > > > If your bucket has a per-bucket setting to use S3Guard, then the > ITestS3GuardToolLocal.testDiffCommand test can fail as fs-only data can creep > into the metastore. > The rawfs should use clearBucketOption to clear the metastore binding, so > guarantee a real raw fs -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14624) Add GenericTestUtils.DelayAnswer that accept slf4j logger API
[ https://issues.apache.org/jira/browse/HADOOP-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564585#comment-16564585 ] Ian Pickering commented on HADOOP-14624: I ran the tests that failed in my local environment and they all passed. > Add GenericTestUtils.DelayAnswer that accept slf4j logger API > - > > Key: HADOOP-14624 > URL: https://issues.apache.org/jira/browse/HADOOP-14624 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Wenxin He >Assignee: Wenxin He >Priority: Major > Attachments: HADOOP-14624.001.patch, HADOOP-14624.002.patch, > HADOOP-14624.003.patch, HADOOP-14624.004.patch, HADOOP-14624.005.patch, > HADOOP-14624.006.patch, HADOOP-14624.007.patch, HADOOP-14624.008.patch, > HADOOP-14624.009.patch, HADOOP-14624.010.patch, HADOOP-14624.011.patch, > HADOOP-14624.012.patch, HADOOP-14624.013.patch, HADOOP-14624.014.patch, > HADOOP-14624.015.patch, HADOOP-14624.016.patch, HADOOP-14624.017.patch, > HADOOP-14624.018.patch > > > Split from HADOOP-14539. > Now GenericTestUtils.DelayAnswer only accepts commons-logging logger API. Now > we are migrating the APIs to slf4j, slf4j logger API should be accepted as > well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564567#comment-16564567
]
Daryn Sharp commented on HADOOP-14445:
--
I'll review tomorrow.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14624) Add GenericTestUtils.DelayAnswer that accept slf4j logger API
[
https://issues.apache.org/jira/browse/HADOOP-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564565#comment-16564565
]
genericqa commented on HADOOP-14624:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 29m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 28m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 0s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
53s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 27m
36s{color} | {color:green} root generated 0 new + 1465 unchanged - 3 fixed =
1465 total (was 1468) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 9s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
16s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 99m 29s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
44s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}240m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HADOOP-14624 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933818/HADOOP-14624.018.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 0c91e35035d5 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15
17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564563#comment-16564563
]
Wei-Chiu Chuang commented on HADOOP-14445:
--
+1 from me. let's wait a day so others have an opportunity to review it.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15645) ITestS3GuardToolLocal.testDiffCommand fails if bucket has per-bucket binding to DDB
Steve Loughran created HADOOP-15645: --- Summary: ITestS3GuardToolLocal.testDiffCommand fails if bucket has per-bucket binding to DDB Key: HADOOP-15645 URL: https://issues.apache.org/jira/browse/HADOOP-15645 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.2.0 Reporter: Steve Loughran If your bucket has a per-bucket setting to use S3Guard, then the ITestS3GuardToolLocal.testDiffCommand test can fail as fs-only data can creep into the metastore. The rawfs should use clearBucketOption to clear the metastore binding, so guarantee a real raw fs -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15642) Update to latest/recent version of aws-sdk for Hadoop 3.2
[ https://issues.apache.org/jira/browse/HADOOP-15642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15642: Status: Patch Available (was: Open) > Update to latest/recent version of aws-sdk for Hadoop 3.2 > - > > Key: HADOOP-15642 > URL: https://issues.apache.org/jira/browse/HADOOP-15642 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15642-001.patch, HADOOP-15642-003.patch, Screen > Shot 2018-07-30 at 14.11.22.png > > > Move to a later version of the AWS SDK library for a different set of > features and issues. > proposed version: 1.11.375 > One thing which doesn't work on the one we ship with is the ability to create > assumed role sessions >1h, as there's a check in the client lib for > role-duration <= 3600 seconds. I'll assume more recent SDKs delegate duration > checks to the far end. > see: > [https://aws.amazon.com/about-aws/whats-new/2018/03/longer-role-sessions/] > * assuming later versions will extend assumed role life, docs will need > changing, > * Adding a test in HADOOP-15583 which expects an error message if you ask for > a duration of 3h; this should act as the test to see what happens. > * think this time would be good to explicitly write down the SDK update > process -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15642) Update to latest/recent version of aws-sdk for Hadoop 3.2
[ https://issues.apache.org/jira/browse/HADOOP-15642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564525#comment-16564525 ] Steve Loughran commented on HADOOP-15642: - good to have some testing here by others, esp [~ehiggs] [~fabbri], [~gabor.bota] & [~mackrorysd] > Update to latest/recent version of aws-sdk for Hadoop 3.2 > - > > Key: HADOOP-15642 > URL: https://issues.apache.org/jira/browse/HADOOP-15642 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15642-001.patch, HADOOP-15642-003.patch, Screen > Shot 2018-07-30 at 14.11.22.png > > > Move to a later version of the AWS SDK library for a different set of > features and issues. > proposed version: 1.11.375 > One thing which doesn't work on the one we ship with is the ability to create > assumed role sessions >1h, as there's a check in the client lib for > role-duration <= 3600 seconds. I'll assume more recent SDKs delegate duration > checks to the far end. > see: > [https://aws.amazon.com/about-aws/whats-new/2018/03/longer-role-sessions/] > * assuming later versions will extend assumed role life, docs will need > changing, > * Adding a test in HADOOP-15583 which expects an error message if you ask for > a duration of 3h; this should act as the test to see what happens. > * think this time would be good to explicitly write down the SDK update > process -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15642) Update to latest/recent version of aws-sdk for Hadoop 3.2
[ https://issues.apache.org/jira/browse/HADOOP-15642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15642: Status: Open (was: Patch Available) > Update to latest/recent version of aws-sdk for Hadoop 3.2 > - > > Key: HADOOP-15642 > URL: https://issues.apache.org/jira/browse/HADOOP-15642 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15642-001.patch, HADOOP-15642-003.patch, Screen > Shot 2018-07-30 at 14.11.22.png > > > Move to a later version of the AWS SDK library for a different set of > features and issues. > proposed version: 1.11.375 > One thing which doesn't work on the one we ship with is the ability to create > assumed role sessions >1h, as there's a check in the client lib for > role-duration <= 3600 seconds. I'll assume more recent SDKs delegate duration > checks to the far end. > see: > [https://aws.amazon.com/about-aws/whats-new/2018/03/longer-role-sessions/] > * assuming later versions will extend assumed role life, docs will need > changing, > * Adding a test in HADOOP-15583 which expects an error message if you ask for > a duration of 3h; this should act as the test to see what happens. > * think this time would be good to explicitly write down the SDK update > process -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15642) Update to latest/recent version of aws-sdk for Hadoop 3.2
[ https://issues.apache.org/jira/browse/HADOOP-15642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564515#comment-16564515 ] Steve Loughran commented on HADOOP-15642: - Patch 003 (skipped 002) * new AWS SDK is out, so update to 1.11.375; walk through the update qualification process to see how it works. Fine, just slow, especially looking through the output, where this time I only did a subset of the output.txt files (the seek ones) * added some more hadoop fs & s3guard commands to help validate stuff. Test run with s3guard, ddb and auth against US-west-1 saw the (known) failure ITestS3AFileContextMainOperations>FileContextMainOperationsBaseTest.testBuilderCreateAppendExistingFile Also got a diff failure in ITestS3GuardToolLocal; not sure if that is new or not, but as its the first time I've run auth for a while, assuming unrelated while I look at it > Update to latest/recent version of aws-sdk for Hadoop 3.2 > - > > Key: HADOOP-15642 > URL: https://issues.apache.org/jira/browse/HADOOP-15642 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15642-001.patch, HADOOP-15642-003.patch, Screen > Shot 2018-07-30 at 14.11.22.png > > > Move to a later version of the AWS SDK library for a different set of > features and issues. > proposed version: 1.11.375 > One thing which doesn't work on the one we ship with is the ability to create > assumed role sessions >1h, as there's a check in the client lib for > role-duration <= 3600 seconds. I'll assume more recent SDKs delegate duration > checks to the far end. > see: > [https://aws.amazon.com/about-aws/whats-new/2018/03/longer-role-sessions/] > * assuming later versions will extend assumed role life, docs will need > changing, > * Adding a test in HADOOP-15583 which expects an error message if you ask for > a duration of 3h; this should act as the test to see what happens. > * think this time would be good to explicitly write down the SDK update > process -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15642) Update to latest/recent version of aws-sdk for Hadoop 3.2
[ https://issues.apache.org/jira/browse/HADOOP-15642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15642: Attachment: HADOOP-15642-003.patch > Update to latest/recent version of aws-sdk for Hadoop 3.2 > - > > Key: HADOOP-15642 > URL: https://issues.apache.org/jira/browse/HADOOP-15642 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15642-001.patch, HADOOP-15642-003.patch, Screen > Shot 2018-07-30 at 14.11.22.png > > > Move to a later version of the AWS SDK library for a different set of > features and issues. > proposed version: 1.11.375 > One thing which doesn't work on the one we ship with is the ability to create > assumed role sessions >1h, as there's a check in the client lib for > role-duration <= 3600 seconds. I'll assume more recent SDKs delegate duration > checks to the far end. > see: > [https://aws.amazon.com/about-aws/whats-new/2018/03/longer-role-sessions/] > * assuming later versions will extend assumed role life, docs will need > changing, > * Adding a test in HADOOP-15583 which expects an error message if you ask for > a duration of 3h; this should act as the test to see what happens. > * think this time would be good to explicitly write down the SDK update > process -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15642) Update to latest/recent version of aws-sdk for Hadoop 3.2
[ https://issues.apache.org/jira/browse/HADOOP-15642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15642: Description: Move to a later version of the AWS SDK library for a different set of features and issues. proposed version: 1.11.375 One thing which doesn't work on the one we ship with is the ability to create assumed role sessions >1h, as there's a check in the client lib for role-duration <= 3600 seconds. I'll assume more recent SDKs delegate duration checks to the far end. see: [https://aws.amazon.com/about-aws/whats-new/2018/03/longer-role-sessions/] * assuming later versions will extend assumed role life, docs will need changing, * Adding a test in HADOOP-15583 which expects an error message if you ask for a duration of 3h; this should act as the test to see what happens. * think this time would be good to explicitly write down the SDK update process was: Move to a later version of the AWS SDK library for a different set of features and issues. proposed version: 1.11.374 One thing which doesn't work on the one we ship with is the ability to create assumed role sessions >1h, as there's a check in the client lib for role-duration <= 3600 seconds. I'll assume more recent SDKs delegate duration checks to the far end. see: [https://aws.amazon.com/about-aws/whats-new/2018/03/longer-role-sessions/] * assuming later versions will extend assumed role life, docs will need changing, * Adding a test in HADOOP-15583 which expects an error message if you ask for a duration of 3h; this should act as the test to see what happens. * think this time would be good to explicitly write down the SDK update process > Update to latest/recent version of aws-sdk for Hadoop 3.2 > - > > Key: HADOOP-15642 > URL: https://issues.apache.org/jira/browse/HADOOP-15642 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15642-001.patch, Screen Shot 2018-07-30 at > 14.11.22.png > > > Move to a later version of the AWS SDK library for a different set of > features and issues. > proposed version: 1.11.375 > One thing which doesn't work on the one we ship with is the ability to create > assumed role sessions >1h, as there's a check in the client lib for > role-duration <= 3600 seconds. I'll assume more recent SDKs delegate duration > checks to the far end. > see: > [https://aws.amazon.com/about-aws/whats-new/2018/03/longer-role-sessions/] > * assuming later versions will extend assumed role life, docs will need > changing, > * Adding a test in HADOOP-15583 which expects an error message if you ask for > a duration of 3h; this should act as the test to see what happens. > * think this time would be good to explicitly write down the SDK update > process -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15604) Bulk commits of S3A MPUs place needless excessive load on S3 & S3Guard
[ https://issues.apache.org/jira/browse/HADOOP-15604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15604: Summary: Bulk commits of S3A MPUs place needless excessive load on S3 & S3Guard (was: bulk commits of S3A MPUs place needless excessive load on S3 & S3Guard) > Bulk commits of S3A MPUs place needless excessive load on S3 & S3Guard > -- > > Key: HADOOP-15604 > URL: https://issues.apache.org/jira/browse/HADOOP-15604 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > > When there are ~50 files being committed; each in their own thread from the > commit pool; probably the DDB repo is being overloaded just from one single > process doing task commit. We should be backing off more, especially given > that failing on a write could potentially leave the store inconsistent with > the FS (renames, etc) > It would be nice to have some tests to prove that the I/O thresholds are the > reason for unprocessed items in DynamoDB metadata store -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15604) bulk commits of S3A MPUs place needless excessive load on S3 & S3Guard
[ https://issues.apache.org/jira/browse/HADOOP-15604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15604: Parent Issue: HADOOP-15620 (was: HADOOP-15226) > bulk commits of S3A MPUs place needless excessive load on S3 & S3Guard > -- > > Key: HADOOP-15604 > URL: https://issues.apache.org/jira/browse/HADOOP-15604 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > > When there are ~50 files being committed; each in their own thread from the > commit pool; probably the DDB repo is being overloaded just from one single > process doing task commit. We should be backing off more, especially given > that failing on a write could potentially leave the store inconsistent with > the FS (renames, etc) > It would be nice to have some tests to prove that the I/O thresholds are the > reason for unprocessed items in DynamoDB metadata store -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15604) bulk commits of S3A MPUs place needless excessive load on S3 & S3Guard
[ https://issues.apache.org/jira/browse/HADOOP-15604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15604: Summary: bulk commits of S3A MPUs place needless excessive load on S3 & S3Guard (was: Test if the unprocessed items in S3Guard DDB metadata store caused by I/O thresholds) > bulk commits of S3A MPUs place needless excessive load on S3 & S3Guard > -- > > Key: HADOOP-15604 > URL: https://issues.apache.org/jira/browse/HADOOP-15604 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > > When there are ~50 files being committed; each in their own thread from the > commit pool; probably the DDB repo is being overloaded just from one single > process doing task commit. We should be backing off more, especially given > that failing on a write could potentially leave the store inconsistent with > the FS (renames, etc) > It would be nice to have some tests to prove that the I/O thresholds are the > reason for unprocessed items in DynamoDB metadata store -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15604) Test if the unprocessed items in S3Guard DDB metadata store caused by I/O thresholds
[
https://issues.apache.org/jira/browse/HADOOP-15604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1656#comment-1656
]
Steve Loughran commented on HADOOP-15604:
-
reviewing the commit code
* If the problem is DDB, it'll be in {{finishedWrite}}
* there are retries going on, but we're not doing much with the retry callback.
Maybe
Looking at {{S3AFS.finishedWrite()}}, it's doing
{{deleteUnnecessaryFakeDirectories()}} every time. I don't see that being
needed when doing a full set of commits: if you are committing 50 files to a
dir, then you don't need to issue 50+ batch delete calls, just one at the start
of the commit (maybe a WriteOperationsHelper.prepareForCommit(path) call).
Similarly, it's calling S3Guard add ancestors every time. If we can track which
ancestors have already been updated as part of a commit, then many fewer DDB
queries will be needed. Looking at {{addAncestors()}}, it's at least one DDB
GET per file, even if they all share the same parent dir.
I think we can do stuff here to reduce load, and it'll matter for the MPU
upload of HADOOP-15576.
Moving under HADOOP-15620 as a Hadoop 3.3 S3A top level feature, renaming
"bulk commits of S3A MPUs place needless excessive load on S3 & S3Guard"
> Test if the unprocessed items in S3Guard DDB metadata store caused by I/O
> thresholds
>
>
> Key: HADOOP-15604
> URL: https://issues.apache.org/jira/browse/HADOOP-15604
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> When there are ~50 files being committed; each in their own thread from the
> commit pool; probably the DDB repo is being overloaded just from one single
> process doing task commit. We should be backing off more, especially given
> that failing on a write could potentially leave the store inconsistent with
> the FS (renames, etc)
> It would be nice to have some tests to prove that the I/O thresholds are the
> reason for unprocessed items in DynamoDB metadata store
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15363) (transient) ITestS3AInconsistency.testOpenFailOnRead S3Guard failure
[
https://issues.apache.org/jira/browse/HADOOP-15363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-15363.
-
Resolution: Cannot Reproduce
closing as cannot reproduce
> (transient) ITestS3AInconsistency.testOpenFailOnRead S3Guard failure
>
>
> Key: HADOOP-15363
> URL: https://issues.apache.org/jira/browse/HADOOP-15363
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Priority: Major
>
> Test failure
> {code}
> ITestS3AInconsistency.testOpenFailOnRead:162->doOpenFailOnReadTest:185
> S3Guard failed to handle fail-on-read
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15426) Make S3guard client resilient to DDB throttle events and network failures
[
https://issues.apache.org/jira/browse/HADOOP-15426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564433#comment-16564433
]
Steve Loughran commented on HADOOP-15426:
-
See also:
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Programming.Errors.html
States that the BatchGetItem/BatchWriteItem calls are really wrappers around a
series (parallelised?) GET/POST operations, rather than anything done server
side in some bulk operation. Which means that
* switching to things like a batch delete call may not save much, unless the
calls are parallelised.
* the IO load of a batch write of 10 items is 10 *1 item.
* Partial failures will happen regularly under an overloaded store: if each API
call has an probability of X chance of failing, then the chance of a batch
operation on Y items not failing is (1-X)^Y...except of course, throttle
failures are not independent events, are they. So in fact, a batch request of a
large #of items against a table with a small IO capacity will inevitably
trigger the throttling itself.
h2. Large batch operations against a table with a small allocated IO capacity
will inevitably trigger throttling and hence partial failures will be the
normal outcome, not an outlier event.
Which means that its critical that the events get reported to the FS, and that
we don't go overboard in reporting throttling events. They will happen, we need
to recover from them everywhere, and we don't want to fill the logs telling
people this is happening
> Make S3guard client resilient to DDB throttle events and network failures
> -
>
> Key: HADOOP-15426
> URL: https://issues.apache.org/jira/browse/HADOOP-15426
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-15426-001.patch, HADOOP-15426-002.patch,
> HADOOP-15426-003.patch, Screen Shot 2018-07-24 at 15.16.46.png, Screen Shot
> 2018-07-25 at 16.22.10.png, Screen Shot 2018-07-25 at 16.28.53.png, Screen
> Shot 2018-07-27 at 14.07.38.png,
> org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStoreScale-output.txt
>
>
> managed to create on a parallel test run
> {code}
> org.apache.hadoop.fs.s3a.AWSServiceThrottledException: delete on
> s3a://hwdev-steve-ireland-new/fork-0005/test/existing-dir/existing-file:
> com.amazonaws.services.dynamodbv2.model.ProvisionedThroughputExceededException:
> The level of configured provisioned throughput for the table was exceeded.
> Consider increasing your provisioning level with the UpdateTable API.
> (Service: AmazonDynamoDBv2; Status Code: 400; Error Code:
> ProvisionedThroughputExceededException; Request ID:
> RDM3370REDBBJQ0SLCLOFC8G43VV4KQNSO5AEMVJF66Q9ASUAAJG): The level of
> configured provisioned throughput for the table was exceeded. Consider
> increasing your provisioning level with the UpdateTable API. (Service:
> AmazonDynamoDBv2; Status Code: 400; Error Code:
> ProvisionedThroughputExceededException; Request ID:
> RDM3370REDBBJQ0SLCLOFC8G43VV4KQNSO5AEMVJF66Q9ASUAAJG)
> at
> {code}
> We should be able to handle this. 400 "bad things happened" error though, not
> the 503 from S3.
> h3. We need a retry handler for DDB throttle operations
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14576) s3guard DynamoDB resource not found: tables not ACTIVE state after initial connection
[ https://issues.apache.org/jira/browse/HADOOP-14576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564427#comment-16564427 ] Steve Loughran commented on HADOOP-14576: - HADOOP-15426 is the second iteration at DDB resilience, but I don't see us doing any handling of ResourceNotFoundException as anything which can be recovered from. > s3guard DynamoDB resource not found: tables not ACTIVE state after initial > connection > - > > Key: HADOOP-14576 > URL: https://issues.apache.org/jira/browse/HADOOP-14576 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Sean Mackrory >Priority: Major > > We currently only anticipate tables not being in the ACTIVE state when first > connecting. It is possible for a table to be in the ACTIVE state and move to > an UPDATING state during partitioning events. Attempts to read or write > during that time will result in an AmazonServerException getting thrown. We > should try to handle that better... -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14576) s3guard DynamoDB resource not found: tables not ACTIVE state after initial connection
[
https://issues.apache.org/jira/browse/HADOOP-14576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236463#comment-16236463
]
Steve Loughran edited comment on HADOOP-14576 at 7/31/18 10:12 PM:
---
I've seen similar issues intermittently just running S3A integration tests. I
will paste a fresh stack trace next time I hit it. For now, sharing another
stack trace our Hive folks provided a while back.
{code}
Caused by: org.apache.hadoop.fs.s3a.AWSServiceIOException: get on
s3a:///1000_unpartitioned_parquet_cdh_ip-10-0-0-158/catalog_sales/.hive-staging_hive_2017-06-21_00-27-07_895_5004400669443203568-24/_tmp.-ext-1/000530_0:
com.amazonaws.services.dynamodbv2.model.ResourceNotFoundException: Requested
resource not found (Service: AmazonDynamoDBv2; Status Code: 400; Error Code:
ResourceNotFoundException; Request ID:
L93OFC0JOT2N6FT45BIPRQVP1FVV4KQNSO5AEMVJF66Q9ASUAAJG): Requested resource not
found (Service: AmazonDynamoDBv2; Status Code: 400; Error Code:
ResourceNotFoundException; Request ID:
L93OFC0JOT2N6FT45BIPRQVP1FVV4KQNSO5AEMVJF66Q9ASUAAJG)
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:178)
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:101)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.get(DynamoDBMetadataStore.java:395)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:1775)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:776)
at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:734)
at
org.apache.hadoop.hive.ql.util.ParallelDirectoryRenamer$1.call(ParallelDirectoryRenamer.java:105)
at
org.apache.hadoop.hive.ql.util.ParallelDirectoryRenamer$1.call(ParallelDirectoryRenamer.java:101)
... 4 more
Caused by: com.amazonaws.services.dynamodbv2.model.ResourceNotFoundException:
Requested resource not found (Service: AmazonDynamoDBv2; Status Code: 400;
Error Code: ResourceNotFoundException; Request ID: L93OF...snip...)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
{code}
was (Author: fabbri):
I've seen similar issues intermittently just running S3A integration tests. I
will paste a fresh stack trace next time I hit it. For now, sharing another
stack trace our Hive folks provided a while back.
Caused by: org.apache.hadoop.fs.s3a.AWSServiceIOException: get on
s3a:///1000_unpartitioned_parquet_cdh_ip-10-0-0-158/catalog_sales/.hive-staging_hive_2017-06-21_00-27-07_895_5004400669443203568-24/_tmp.-ext-1/000530_0:
com.amazonaws.services.dynamodbv2.model.ResourceNotFoundException: Requested
resource not found (Service: AmazonDynamoDBv2; Status Code: 400; Error Code:
ResourceNotFoundException; Request ID:
L93OFC0JOT2N6FT45BIPRQVP1FVV4KQNSO5AEMVJF66Q9ASUAAJG): Requested resource not
found (Service: AmazonDynamoDBv2; Status Code: 400; Error Code:
ResourceNotFoundException; Request ID:
L93OFC0JOT2N6FT45BIPRQVP1FVV4KQNSO5AEMVJF66Q9ASUAAJG)
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:178)
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:101)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.get(DynamoDBMetadataStore.java:395)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:1775)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:776)
at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:734)
at
org.apache.hadoop.hive.ql.util.ParallelDirectoryRenamer$1.call(ParallelDirectoryRenamer.java:105)
at
org.apache.hadoop.hive.ql.util.ParallelDirectoryRenamer$1.call(ParallelDirectoryRenamer.java:101)
... 4 more
Caused by: com.amazonaws.services.dynamodbv2.model.ResourceNotFoundException:
Requested resource not found (Service: AmazonDynamoDBv2; Status Code: 400;
Error Code: ResourceNotFoundException; Request ID: L93OF...snip...)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
at
[jira] [Commented] (HADOOP-15644) Hadoop Docker Image Pip Install Fails on branch-2
[
https://issues.apache.org/jira/browse/HADOOP-15644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564419#comment-16564419
]
Haibo Chen commented on HADOOP-15644:
-
Thanks [~rkanter]
> Hadoop Docker Image Pip Install Fails on branch-2
> -
>
> Key: HADOOP-15644
> URL: https://issues.apache.org/jira/browse/HADOOP-15644
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 2.10.0
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Critical
> Fix For: 2.10.0
>
> Attachments: HADOOP-15644-branch-2.00.patch
>
>
> HADOOP-15610 fixes the pip install failures on branch 3.x releases, but it is
> still failing on branch-2, though with a slightly different error message
> {code:java}
> Downloading/unpacking pylint
> Running setup.py (path:/tmp/pip_build_root/pylint/setup.py) egg_info for
> package pylint
> /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown
> distribution option: 'python_requires'
> warnings.warn(msg)
> Traceback (most recent call last):
> File "", line 17, in
> File "/tmp/pip_build_root/pylint/setup.py", line 177, in
> install()
> File "/tmp/pip_build_root/pylint/setup.py", line 174, in install
> **kwargs)
> File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
> _setup_distribution = dist = klass(attrs)
> File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 239,
> in __init__
> self.fetch_build_eggs(attrs.pop('setup_requires'))
> File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 264,
> in fetch_build_eggs
> replace_conflicting=True
> File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 620, in
> resolve
> dist = best[req.key] = env.best_match(req, ws, installer)
> File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 858, in
> best_match
> return self.obtain(req, installer) # try and download/install
> File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 870, in
> obtain
> return installer(requirement)
> File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 314,
> in fetch_build_egg
> return cmd.easy_install(req)
> File
> "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
> 616, in easy_install
> return self.install_item(spec, dist.location, tmpdir, deps)
> File
> "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
> 646, in install_item
> dists = self.install_eggs(spec, download, tmpdir)
> File
> "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
> 834, in install_eggs
> return self.build_and_install(setup_script, setup_base)
> File
> "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
> 1040, in build_and_install
> self.run_setup(setup_script, setup_base, args)
> File
> "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
> 1025, in run_setup
> run_setup(setup_script, args)
> File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 50,
> in run_setup
> lambda: execfile(
> File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line
> 100, in run
> return func()
> File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 52,
> in
> {'__file__':setup_script, '__name__':'__main__'}
> File "setup.py", line 76, in
> File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
> _setup_distribution = dist = klass(attrs)
> File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 243,
> in __init__
> _Distribution.__init__(self,attrs)
> File "/usr/lib/python2.7/distutils/dist.py", line 287, in __init__
> self.finalize_options()
> File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 277,
> in finalize_options
> ep.load()(self, ep.name, value)
> File "build/bdist.linux-x86_64/egg/setuptools_scm/integration.py", line
> 10, in version_keyword
> File "build/bdist.linux-x86_64/egg/setuptools_scm/version.py", line 66,
> in _warn_if_setuptools_outdated
> setuptools_scm.version.SetuptoolsOutdatedWarning: your setuptools is too
> old (<12)
> Complete output from command python setup.py egg_info:
> /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown
> distribution option: 'python_requires'
> warnings.warn(msg)
> Traceback (most recent call last):
> File "", line 17, in
> File "/tmp/pip_build_root/pylint/setup.py", line 177, in
> install()
> File "/tmp/pip_build_root/pylint/setup.py", line 174,
[jira] [Comment Edited] (HADOOP-14936) S3Guard: remove "experimental" from documentation
[ https://issues.apache.org/jira/browse/HADOOP-14936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564397#comment-16564397 ] Steve Loughran edited comment on HADOOP-14936 at 7/31/18 9:42 PM: -- I've moved this to Hadoop 3.3. Looking at the outstanding issues, I'll link in the throttling & shared credential stuff, but the other big issue for me is remaining consistent in the presence of partial failures of rename/delete was (Author: ste...@apache.org): I've moved this to Hadoop 3.4. Looking at the outstanding issues, I'll link in the throttling & shared credential stuff, but the other big issue for me is remaining consistent in the presence of partial failures of rename/delete > S3Guard: remove "experimental" from documentation > - > > Key: HADOOP-14936 > URL: https://issues.apache.org/jira/browse/HADOOP-14936 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Priority: Major > > I think it is time to remove the "experimental feature" designation in the > site docs for S3Guard. Discuss. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14936) S3Guard: remove "experimental" from documentation
[ https://issues.apache.org/jira/browse/HADOOP-14936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564397#comment-16564397 ] Steve Loughran commented on HADOOP-14936: - I've moved this to Hadoop 3.4. Looking at the outstanding issues, I'll link in the throttling & shared credential stuff, but the other big issue for me is remaining consistent in the presence of partial failures of rename/delete > S3Guard: remove "experimental" from documentation > - > > Key: HADOOP-14936 > URL: https://issues.apache.org/jira/browse/HADOOP-14936 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Priority: Major > > I think it is time to remove the "experimental feature" designation in the > site docs for S3Guard. Discuss. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14936) S3Guard: remove "experimental" from documentation
[ https://issues.apache.org/jira/browse/HADOOP-14936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14936: Parent Issue: HADOOP-15619 (was: HADOOP-15226) > S3Guard: remove "experimental" from documentation > - > > Key: HADOOP-14936 > URL: https://issues.apache.org/jira/browse/HADOOP-14936 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Priority: Major > > I think it is time to remove the "experimental feature" designation in the > site docs for S3Guard. Discuss. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15314) Scheme assertion in S3Guard DynamoDBMetadataStore::checkPath is unnecessarily restrictive
[
https://issues.apache.org/jira/browse/HADOOP-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564394#comment-16564394
]
Steve Loughran commented on HADOOP-15314:
-
Reviewing this again.
I think you are both right: we don't need to be checking the schema. All we
care about is a path is being registered -it is up to the owning FS to worry
about schema equivalence, because path qualification should have done that
already.
However, the static
{{PathMetadataDynamoDBTranslation.itemToPathMetadata(Item)}} method is
prepending that s3a scheme when it builds paths, so removing the check in
{{checkPath}} will only get the path in: when you get FileStatus entries back
they'll be of the s3a:// scheme.
This issue here then is really one of "support URI schemes other than S3a in
DDB", as the instance of the metastore will need to know which scheme to use
when creating paths from items.
> Scheme assertion in S3Guard DynamoDBMetadataStore::checkPath is unnecessarily
> restrictive
> -
>
> Key: HADOOP-15314
> URL: https://issues.apache.org/jira/browse/HADOOP-15314
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0
>Reporter: DJ Hoffman
>Priority: Major
>
> In version 3.0.0, the checkPath method for dealing with paths prevents us
> from using the s3:// scheme when utilizing S3Guard. However, in our
> core-site.xml we have included
> {noformat}
>
> fs.s3.impl
> org.apache.hadoop.fs.s3a.S3AFileSystem
> {noformat}
> which should enforce that s3 prefixed paths go through s3a and are properly
> compatible with s3guard. We removed the assertion that paths use the s3a
> scheme (some of our paths use the s3 scheme) and our testing thus far with
> S3Guard enabled have been positive. We believe the assertion in checkPath is
> unnecessary and could be expanded to include the s3 and s3n schemes if not
> dropped altogether or altered in some other way. We're happy to develop and
> test a patch if the community is amenable to the change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15314) Scheme assertion in S3Guard DynamoDBMetadataStore::checkPath is unnecessarily restrictive
[
https://issues.apache.org/jira/browse/HADOOP-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15314:
Parent Issue: HADOOP-15619 (was: HADOOP-15226)
> Scheme assertion in S3Guard DynamoDBMetadataStore::checkPath is unnecessarily
> restrictive
> -
>
> Key: HADOOP-15314
> URL: https://issues.apache.org/jira/browse/HADOOP-15314
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0
>Reporter: DJ Hoffman
>Priority: Major
>
> In version 3.0.0, the checkPath method for dealing with paths prevents us
> from using the s3:// scheme when utilizing S3Guard. However, in our
> core-site.xml we have included
> {noformat}
>
> fs.s3.impl
> org.apache.hadoop.fs.s3a.S3AFileSystem
> {noformat}
> which should enforce that s3 prefixed paths go through s3a and are properly
> compatible with s3guard. We removed the assertion that paths use the s3a
> scheme (some of our paths use the s3 scheme) and our testing thus far with
> S3Guard enabled have been positive. We believe the assertion in checkPath is
> unnecessary and could be expanded to include the s3 and s3n schemes if not
> dropped altogether or altered in some other way. We're happy to develop and
> test a patch if the community is amenable to the change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14757) S3AFileSystem.innerRename() to size metadatastore lists better
[
https://issues.apache.org/jira/browse/HADOOP-14757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14757:
Parent Issue: HADOOP-15619 (was: HADOOP-15226)
> S3AFileSystem.innerRename() to size metadatastore lists better
> --
>
> Key: HADOOP-14757
> URL: https://issues.apache.org/jira/browse/HADOOP-14757
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Priority: Minor
>
> In {{S3AFileSystem.innerRename()}}, various ArrayLists are created to track
> paths to update; these are created with the default size. It could/should be
> possible to allocate better, so avoid expensive array growth & copy
> operations while iterating through the list of entries.
> # for a single file copy, sizes == 1
> # for a recursive copy, the outcome of the first real LIST will either
> provide the actual size, or, if the list == the max response, a very large
> minimum size.
> For #2, we'd need to get the hint of iterable length rather than just iterate
> through...some interface {{{IterableLength.expectedMinimumSize()}} could do
> that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14158) Possible for modified configuration to leak into metadatastore in S3GuardTool
[
https://issues.apache.org/jira/browse/HADOOP-14158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564391#comment-16564391
]
Steve Loughran commented on HADOOP-14158:
-
This is going to be an issue if people are using the entry point as an API
(i.e. management tools) rather than just a CLI.
I think we must consider that a legit use, and tag {{S3GuardTool}} with the
stability guarantees we want to offer, which I think could be
@LimitedPrivate("CLI", "management tools") and @Evolving.
> Possible for modified configuration to leak into metadatastore in S3GuardTool
> -
>
> Key: HADOOP-14158
> URL: https://issues.apache.org/jira/browse/HADOOP-14158
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Sean Mackrory
>Priority: Minor
>
> It doesn't appear to do it when run from the command-line, but when running
> the S3GuardTool.run (i.e. the parent function of most of the functions used
> in the unit tests) from a unit test, you end up with a NullMetadataStore,
> regardless of what else was configured.
> We create an instance of S3AFileSystem with the metadata store implementation
> overridden to NullMetadataStore so that we have distinct interfaces to S3 and
> the metadata store. S3Guard can later be called using this filesystem,
> causing it to pick up the filesystem's configuration, which instructs it to
> use the NullMetadataStore implementation. This shouldn't be possible.
> It is unknown if this happens in any real-world scenario - I've been unable
> to reproduce the problem from the command-line. But it definitely happens in
> a test, it shouldn't, and fixing this will at least allow HADOOP-14145 to
> have an automated test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14852) Intermittent failure of S3Guard TestConsistencyListFiles
[
https://issues.apache.org/jira/browse/HADOOP-14852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-14852.
-
Resolution: Cannot Reproduce
Fix Version/s: 3.2.0
I've not seen this for a while so closing as cannot-reproduce.
> Intermittent failure of S3Guard TestConsistencyListFiles
>
>
> Key: HADOOP-14852
> URL: https://issues.apache.org/jira/browse/HADOOP-14852
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Priority: Major
> Fix For: 3.2.0
>
>
> I'm seeing intermittent test failures with a test run of {{ -Dparallel-tests
> -DtestsThreadCount=8 -Ds3guard -Ddynamo}} (-Dauth set or unset) in which a
> file in DELAY-LISTING-ME isn't being returned in a listing.
> Theories
> * test is wrong
> * config is wrong
> * code is wrong
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15022) s3guard IT tests increase R/W capacity of the test table by 1
[ https://issues.apache.org/jira/browse/HADOOP-15022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564379#comment-16564379 ] Steve Loughran edited comment on HADOOP-15022 at 7/31/18 9:22 PM: -- This is going to be fixed by HADOOP-15426, as I cut the set-capacity operation entirely. There's a limit to how often you can call the API a day, so as well as running up bills, if you ran the tests enough time it'd end up failing was (Author: ste...@apache.org): This is going to be fixed by HADOOP-15583, as I cut the set-capacity operation entirely. There's a limit to how often you can call the API a day, so as well as running up bills, if you ran the tests enough time it'd end up failing > s3guard IT tests increase R/W capacity of the test table by 1 > - > > Key: HADOOP-15022 > URL: https://issues.apache.org/jira/browse/HADOOP-15022 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Priority: Minor > > Just noticed playing with the CLI that my allocated IOPs was 153; reset it to > 10 R & 10 W; after a few of the IT test runs it is now 13 each > assumption: every test run of the S3Guard CLI is increasing the allocated IO -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15022) s3guard IT tests increase R/W capacity of the test table by 1
[ https://issues.apache.org/jira/browse/HADOOP-15022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned HADOOP-15022: --- Assignee: Steve Loughran > s3guard IT tests increase R/W capacity of the test table by 1 > - > > Key: HADOOP-15022 > URL: https://issues.apache.org/jira/browse/HADOOP-15022 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > > Just noticed playing with the CLI that my allocated IOPs was 153; reset it to > 10 R & 10 W; after a few of the IT test runs it is now 13 each > assumption: every test run of the S3Guard CLI is increasing the allocated IO -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14750) s3guard to provide better diags on ddb init failures
[ https://issues.apache.org/jira/browse/HADOOP-14750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14750: Parent Issue: HADOOP-15620 (was: HADOOP-15226) > s3guard to provide better diags on ddb init failures > > > Key: HADOOP-14750 > URL: https://issues.apache.org/jira/browse/HADOOP-14750 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Priority: Minor > > When you can't connect to DDB you get an Http exception; it'd be good to > include more info here (table name & region in particular) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15022) s3guard IT tests increase R/W capacity of the test table by 1
[ https://issues.apache.org/jira/browse/HADOOP-15022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564379#comment-16564379 ] Steve Loughran edited comment on HADOOP-15022 at 7/31/18 9:20 PM: -- This is going to be fixed by HADOOP-15583, as I cut the set-capacity operation entirely. There's a limit to how often you can call the API a day, so as well as running up bills, if you ran the tests enough time it'd end up failing was (Author: ste...@apache.org): This is going to be fixed by HADOOP-15883, as I cut the set-capacity operation entirely. There's a limit to how often you can call the API a day, so as well as running up bills, if you ran the tests enough time it'd end up failing > s3guard IT tests increase R/W capacity of the test table by 1 > - > > Key: HADOOP-15022 > URL: https://issues.apache.org/jira/browse/HADOOP-15022 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Priority: Minor > > Just noticed playing with the CLI that my allocated IOPs was 153; reset it to > 10 R & 10 W; after a few of the IT test runs it is now 13 each > assumption: every test run of the S3Guard CLI is increasing the allocated IO -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15022) s3guard IT tests increase R/W capacity of the test table by 1
[ https://issues.apache.org/jira/browse/HADOOP-15022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564379#comment-16564379 ] Steve Loughran commented on HADOOP-15022: - This is going to be fixed by HADOOP-15883, as I cut the set-capacity operation entirely. There's a limit to how often you can call the API a day, so as well as running up bills, if you ran the tests enough time it'd end up failing > s3guard IT tests increase R/W capacity of the test table by 1 > - > > Key: HADOOP-15022 > URL: https://issues.apache.org/jira/browse/HADOOP-15022 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Priority: Minor > > Just noticed playing with the CLI that my allocated IOPs was 153; reset it to > 10 R & 10 W; after a few of the IT test runs it is now 13 each > assumption: every test run of the S3Guard CLI is increasing the allocated IO -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15426) Make S3guard client resilient to DDB throttle events and network failures
[
https://issues.apache.org/jira/browse/HADOOP-15426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564378#comment-16564378
]
Steve Loughran commented on HADOOP-15426:
-
I'm not going to worry about XML parser errors for now.
as far as I'm concerned, this is done & ready for review. The stabilise
AssumedRole patch of HADOOP-15883 adds retries to its calls of STS too
> Make S3guard client resilient to DDB throttle events and network failures
> -
>
> Key: HADOOP-15426
> URL: https://issues.apache.org/jira/browse/HADOOP-15426
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-15426-001.patch, HADOOP-15426-002.patch,
> HADOOP-15426-003.patch, Screen Shot 2018-07-24 at 15.16.46.png, Screen Shot
> 2018-07-25 at 16.22.10.png, Screen Shot 2018-07-25 at 16.28.53.png, Screen
> Shot 2018-07-27 at 14.07.38.png,
> org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStoreScale-output.txt
>
>
> managed to create on a parallel test run
> {code}
> org.apache.hadoop.fs.s3a.AWSServiceThrottledException: delete on
> s3a://hwdev-steve-ireland-new/fork-0005/test/existing-dir/existing-file:
> com.amazonaws.services.dynamodbv2.model.ProvisionedThroughputExceededException:
> The level of configured provisioned throughput for the table was exceeded.
> Consider increasing your provisioning level with the UpdateTable API.
> (Service: AmazonDynamoDBv2; Status Code: 400; Error Code:
> ProvisionedThroughputExceededException; Request ID:
> RDM3370REDBBJQ0SLCLOFC8G43VV4KQNSO5AEMVJF66Q9ASUAAJG): The level of
> configured provisioned throughput for the table was exceeded. Consider
> increasing your provisioning level with the UpdateTable API. (Service:
> AmazonDynamoDBv2; Status Code: 400; Error Code:
> ProvisionedThroughputExceededException; Request ID:
> RDM3370REDBBJQ0SLCLOFC8G43VV4KQNSO5AEMVJF66Q9ASUAAJG)
> at
> {code}
> We should be able to handle this. 400 "bad things happened" error though, not
> the 503 from S3.
> h3. We need a retry handler for DDB throttle operations
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15576) S3A Multipart Uploader to work with S3Guard and encryption
[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564363#comment-16564363
]
genericqa commented on HADOOP-15576:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 9 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 30m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 41s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 28m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 28m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 12s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
16s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}101m 32s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
39s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
46s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}251m 42s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| | hadoop.fs.viewfs.TestViewFileSystemLinkFallback |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HADOOP-15576 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933793/HADOOP-15576.004.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit
[jira] [Updated] (HADOOP-14624) Add GenericTestUtils.DelayAnswer that accept slf4j logger API
[ https://issues.apache.org/jira/browse/HADOOP-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Pickering updated HADOOP-14624: --- Attachment: HADOOP-14624.018.patch > Add GenericTestUtils.DelayAnswer that accept slf4j logger API > - > > Key: HADOOP-14624 > URL: https://issues.apache.org/jira/browse/HADOOP-14624 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Wenxin He >Assignee: Wenxin He >Priority: Major > Attachments: HADOOP-14624.001.patch, HADOOP-14624.002.patch, > HADOOP-14624.003.patch, HADOOP-14624.004.patch, HADOOP-14624.005.patch, > HADOOP-14624.006.patch, HADOOP-14624.007.patch, HADOOP-14624.008.patch, > HADOOP-14624.009.patch, HADOOP-14624.010.patch, HADOOP-14624.011.patch, > HADOOP-14624.012.patch, HADOOP-14624.013.patch, HADOOP-14624.014.patch, > HADOOP-14624.015.patch, HADOOP-14624.016.patch, HADOOP-14624.017.patch, > HADOOP-14624.018.patch > > > Split from HADOOP-14539. > Now GenericTestUtils.DelayAnswer only accepts commons-logging logger API. Now > we are migrating the APIs to slf4j, slf4j logger API should be accepted as > well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15644) Hadoop Docker Image Pip Install Fails on branch-2
[
https://issues.apache.org/jira/browse/HADOOP-15644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated HADOOP-15644:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.10.0
Status: Resolved (was: Patch Available)
Thanks [~haibochen]. Committed to branch-2!
> Hadoop Docker Image Pip Install Fails on branch-2
> -
>
> Key: HADOOP-15644
> URL: https://issues.apache.org/jira/browse/HADOOP-15644
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 2.10.0
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Critical
> Fix For: 2.10.0
>
> Attachments: HADOOP-15644-branch-2.00.patch
>
>
> HADOOP-15610 fixes the pip install failures on branch 3.x releases, but it is
> still failing on branch-2, though with a slightly different error message
> {code:java}
> Downloading/unpacking pylint
> Running setup.py (path:/tmp/pip_build_root/pylint/setup.py) egg_info for
> package pylint
> /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown
> distribution option: 'python_requires'
> warnings.warn(msg)
> Traceback (most recent call last):
> File "", line 17, in
> File "/tmp/pip_build_root/pylint/setup.py", line 177, in
> install()
> File "/tmp/pip_build_root/pylint/setup.py", line 174, in install
> **kwargs)
> File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
> _setup_distribution = dist = klass(attrs)
> File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 239,
> in __init__
> self.fetch_build_eggs(attrs.pop('setup_requires'))
> File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 264,
> in fetch_build_eggs
> replace_conflicting=True
> File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 620, in
> resolve
> dist = best[req.key] = env.best_match(req, ws, installer)
> File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 858, in
> best_match
> return self.obtain(req, installer) # try and download/install
> File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 870, in
> obtain
> return installer(requirement)
> File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 314,
> in fetch_build_egg
> return cmd.easy_install(req)
> File
> "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
> 616, in easy_install
> return self.install_item(spec, dist.location, tmpdir, deps)
> File
> "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
> 646, in install_item
> dists = self.install_eggs(spec, download, tmpdir)
> File
> "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
> 834, in install_eggs
> return self.build_and_install(setup_script, setup_base)
> File
> "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
> 1040, in build_and_install
> self.run_setup(setup_script, setup_base, args)
> File
> "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
> 1025, in run_setup
> run_setup(setup_script, args)
> File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 50,
> in run_setup
> lambda: execfile(
> File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line
> 100, in run
> return func()
> File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 52,
> in
> {'__file__':setup_script, '__name__':'__main__'}
> File "setup.py", line 76, in
> File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
> _setup_distribution = dist = klass(attrs)
> File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 243,
> in __init__
> _Distribution.__init__(self,attrs)
> File "/usr/lib/python2.7/distutils/dist.py", line 287, in __init__
> self.finalize_options()
> File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 277,
> in finalize_options
> ep.load()(self, ep.name, value)
> File "build/bdist.linux-x86_64/egg/setuptools_scm/integration.py", line
> 10, in version_keyword
> File "build/bdist.linux-x86_64/egg/setuptools_scm/version.py", line 66,
> in _warn_if_setuptools_outdated
> setuptools_scm.version.SetuptoolsOutdatedWarning: your setuptools is too
> old (<12)
> Complete output from command python setup.py egg_info:
> /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown
> distribution option: 'python_requires'
> warnings.warn(msg)
> Traceback (most recent call last):
> File "", line 17, in
> File
[jira] [Commented] (HADOOP-15644) Hadoop Docker Image Pip Install Fails on branch-2
[
https://issues.apache.org/jira/browse/HADOOP-15644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564336#comment-16564336
]
Robert Kanter commented on HADOOP-15644:
+1 LGTM
> Hadoop Docker Image Pip Install Fails on branch-2
> -
>
> Key: HADOOP-15644
> URL: https://issues.apache.org/jira/browse/HADOOP-15644
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 2.10.0
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Critical
> Attachments: HADOOP-15644-branch-2.00.patch
>
>
> HADOOP-15610 fixes the pip install failures on branch 3.x releases, but it is
> still failing on branch-2, though with a slightly different error message
> {code:java}
> Downloading/unpacking pylint
> Running setup.py (path:/tmp/pip_build_root/pylint/setup.py) egg_info for
> package pylint
> /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown
> distribution option: 'python_requires'
> warnings.warn(msg)
> Traceback (most recent call last):
> File "", line 17, in
> File "/tmp/pip_build_root/pylint/setup.py", line 177, in
> install()
> File "/tmp/pip_build_root/pylint/setup.py", line 174, in install
> **kwargs)
> File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
> _setup_distribution = dist = klass(attrs)
> File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 239,
> in __init__
> self.fetch_build_eggs(attrs.pop('setup_requires'))
> File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 264,
> in fetch_build_eggs
> replace_conflicting=True
> File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 620, in
> resolve
> dist = best[req.key] = env.best_match(req, ws, installer)
> File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 858, in
> best_match
> return self.obtain(req, installer) # try and download/install
> File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 870, in
> obtain
> return installer(requirement)
> File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 314,
> in fetch_build_egg
> return cmd.easy_install(req)
> File
> "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
> 616, in easy_install
> return self.install_item(spec, dist.location, tmpdir, deps)
> File
> "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
> 646, in install_item
> dists = self.install_eggs(spec, download, tmpdir)
> File
> "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
> 834, in install_eggs
> return self.build_and_install(setup_script, setup_base)
> File
> "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
> 1040, in build_and_install
> self.run_setup(setup_script, setup_base, args)
> File
> "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
> 1025, in run_setup
> run_setup(setup_script, args)
> File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 50,
> in run_setup
> lambda: execfile(
> File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line
> 100, in run
> return func()
> File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 52,
> in
> {'__file__':setup_script, '__name__':'__main__'}
> File "setup.py", line 76, in
> File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
> _setup_distribution = dist = klass(attrs)
> File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 243,
> in __init__
> _Distribution.__init__(self,attrs)
> File "/usr/lib/python2.7/distutils/dist.py", line 287, in __init__
> self.finalize_options()
> File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 277,
> in finalize_options
> ep.load()(self, ep.name, value)
> File "build/bdist.linux-x86_64/egg/setuptools_scm/integration.py", line
> 10, in version_keyword
> File "build/bdist.linux-x86_64/egg/setuptools_scm/version.py", line 66,
> in _warn_if_setuptools_outdated
> setuptools_scm.version.SetuptoolsOutdatedWarning: your setuptools is too
> old (<12)
> Complete output from command python setup.py egg_info:
> /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown
> distribution option: 'python_requires'
> warnings.warn(msg)
> Traceback (most recent call last):
> File "", line 17, in
> File "/tmp/pip_build_root/pylint/setup.py", line 177, in
> install()
> File "/tmp/pip_build_root/pylint/setup.py", line 174, in install
> **kwargs)
> File
[jira] [Commented] (HADOOP-15625) S3A input stream to use etags to detect changed source files
[ https://issues.apache.org/jira/browse/HADOOP-15625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564334#comment-16564334 ] Brahma Reddy Battula commented on HADOOP-15625: --- Uploaded the initial patch. [~ste...@apache.org] can you please review.? i) Added the etag while sending the request. ii) Throwing new exception when etags are not matched. Ran the tests after this changes,there were failures related to this ( seen some encryption errors). One simple test case was added in patch ( still need to optimize). > S3A input stream to use etags to detect changed source files > > > Key: HADOOP-15625 > URL: https://issues.apache.org/jira/browse/HADOOP-15625 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Brahma Reddy Battula >Priority: Major > Attachments: HADOOP-15625-001.patch > > > S3A input stream doesn't handle changing source files any better than the > other cloud store connectors. Specifically: it doesn't noticed it has > changed, caches the length from startup, and whenever a seek triggers a new > GET, you may get one of: old data, new data, and even perhaps go from new > data to old data due to eventual consistency. > We can't do anything to stop this, but we could detect changes by > # caching the etag of the first HEAD/GET (we don't get that HEAD on open with > S3Guard, BTW) > # on future GET requests, verify the etag of the response > # raise an IOE if the remote file changed during the read. > It's a more dramatic failure, but it stops changes silently corrupting things. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15625) S3A input stream to use etags to detect changed source files
[ https://issues.apache.org/jira/browse/HADOOP-15625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HADOOP-15625: -- Attachment: HADOOP-15625-001.patch > S3A input stream to use etags to detect changed source files > > > Key: HADOOP-15625 > URL: https://issues.apache.org/jira/browse/HADOOP-15625 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Brahma Reddy Battula >Priority: Major > Attachments: HADOOP-15625-001.patch > > > S3A input stream doesn't handle changing source files any better than the > other cloud store connectors. Specifically: it doesn't noticed it has > changed, caches the length from startup, and whenever a seek triggers a new > GET, you may get one of: old data, new data, and even perhaps go from new > data to old data due to eventual consistency. > We can't do anything to stop this, but we could detect changes by > # caching the etag of the first HEAD/GET (we don't get that HEAD on open with > S3Guard, BTW) > # on future GET requests, verify the etag of the response > # raise an IOE if the remote file changed during the read. > It's a more dramatic failure, but it stops changes silently corrupting things. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15501) [Umbrella] Upgrade efforts to Hadoop 3.x
[ https://issues.apache.org/jira/browse/HADOOP-15501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated HADOOP-15501: Target Version/s: 3.2.0, 3.1.2 (was: 3.2.0) > [Umbrella] Upgrade efforts to Hadoop 3.x > > > Key: HADOOP-15501 > URL: https://issues.apache.org/jira/browse/HADOOP-15501 > Project: Hadoop Common > Issue Type: Task >Reporter: Sunil Govindan >Priority: Major > > This is an umbrella ticket to manage all similar efforts to close gaps for > upgrade efforts to 3.x. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15602) Support SASL Rpc request handling in separate Handlers
[ https://issues.apache.org/jira/browse/HADOOP-15602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated HADOOP-15602: Target Version/s: 3.1.2 (was: 3.1.1) > Support SASL Rpc request handling in separate Handlers > --- > > Key: HADOOP-15602 > URL: https://issues.apache.org/jira/browse/HADOOP-15602 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Major > Attachments: HADOOP-15602.01.patch > > > Right now, during RPC Connection establishment, all SASL requests are > considered as OutOfBand requests and handled within the same Reader thread. > SASL handling involves authentication with Kerberos and SecretManagers(for > Token validation). During this time, Reader thread would be blocked, hence > blocking all the incoming RPC requests on other established connections. Some > secretManager impls require to communicate to external systems (ex: ZK) for > verification. > SASL RPC handling in separate dedicated handlers, would enable Reader threads > to read RPC requests from established connections without blocking. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15205) maven release: missing source attachments for hadoop-mapreduce-client-core
[ https://issues.apache.org/jira/browse/HADOOP-15205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated HADOOP-15205: Target Version/s: 3.1.2 (was: 3.1.1) > maven release: missing source attachments for hadoop-mapreduce-client-core > -- > > Key: HADOOP-15205 > URL: https://issues.apache.org/jira/browse/HADOOP-15205 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.5, 3.0.0, 3.1.0, 3.0.1 >Reporter: Zoltan Haindrich >Priority: Major > > I wanted to use the source attachment; however it looks like since 2.7.5 that > artifact is not present at maven central ; it looks like the last release > which had source attachments / javadocs was 2.7.4 > http://central.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.5/ > this seems to be not limited to mapreduce; as the same change is present for > yarn-common as well > http://central.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.7.5/ > and also hadoop-common > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/2.7.5/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/3.0.0/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/3.1.0/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15501) [Umbrella] Upgrade efforts to Hadoop 3.x
[ https://issues.apache.org/jira/browse/HADOOP-15501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated HADOOP-15501: Target Version/s: 3.2.0 (was: 3.2.0, 3.1.1) > [Umbrella] Upgrade efforts to Hadoop 3.x > > > Key: HADOOP-15501 > URL: https://issues.apache.org/jira/browse/HADOOP-15501 > Project: Hadoop Common > Issue Type: Task >Reporter: Sunil Govindan >Priority: Major > > This is an umbrella ticket to manage all similar efforts to close gaps for > upgrade efforts to 3.x. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15398) StagingTestBase uses methods not available in Mockito 1.8.5
[
https://issues.apache.org/jira/browse/HADOOP-15398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated HADOOP-15398:
Target Version/s: 3.2.0, 3.1.2 (was: 3.2.0, 3.1.1)
> StagingTestBase uses methods not available in Mockito 1.8.5
> ---
>
> Key: HADOOP-15398
> URL: https://issues.apache.org/jira/browse/HADOOP-15398
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Mohammad Arshad
>Assignee: Mohammad Arshad
>Priority: Major
> Attachments: HADOOP-15398.001.patch, HADOOP-15398.002.patch
>
>
> *Problem:* hadoop trunk compilation is failing
> *Root Cause:*
> compilation error is coming from
> {{org.apache.hadoop.fs.s3a.commit.staging.StagingTestBase}}. Compilation
> error is "The method getArgumentAt(int, Class) is
> undefined for the type InvocationOnMock".
> StagingTestBase is using getArgumentAt(int, Class) method
> which is not available in mockito-all 1.8.5 version. getArgumentAt(int,
> Class) method is available only from version 2.0.0-beta
> *Expectations:*
> Either mockito-all version to be upgraded or test case to be written only
> with available functions in 1.8.5.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15124) Slow FileSystem.Statistics counters implementation
[ https://issues.apache.org/jira/browse/HADOOP-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated HADOOP-15124: Target Version/s: 3.0.3, 3.1.0, 2.6.6, 2.10.0, 3.2.0, 2.9.2, 2.8.5, 2.7.8, 3.0.4, 3.1.2 (was: 2.6.6, 3.1.0, 2.10.0, 3.2.0, 3.1.1, 2.9.2, 3.0.3, 2.8.5, 2.7.8, 3.0.4) > Slow FileSystem.Statistics counters implementation > -- > > Key: HADOOP-15124 > URL: https://issues.apache.org/jira/browse/HADOOP-15124 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 2.9.0, 2.8.3, 2.7.5, 3.0.0, 3.1.0 >Reporter: Igor Dvorzhak >Assignee: Igor Dvorzhak >Priority: Major > Labels: common, filesystem, fs, statistics > Attachments: HADOOP-15124.001.patch > > > While profiling 1TB TeraGen job on Hadoop 2.8.2 cluster (Google Dataproc, 2 > workers, GCS connector) I saw that FileSystem.Statistics code paths Wall time > is 5.58% and CPU time is 26.5% of total execution time. > After switching FileSystem.Statistics implementation to LongAdder, consumed > Wall time decreased to 0.006% and CPU time to 0.104% of total execution time. > Total job runtime decreased from 66 mins to 61 mins. > These results are not conclusive, because I didn't benchmark multiple times > to average results, but regardless of performance gains switching to > LongAdder simplifies code and reduces its complexity. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15593) UserGroupInformation TGT renewer throws NPE
[
https://issues.apache.org/jira/browse/HADOOP-15593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564259#comment-16564259
]
Wangda Tan commented on HADOOP-15593:
-
cherry-picked to branch-3.1.1
> UserGroupInformation TGT renewer throws NPE
> ---
>
> Key: HADOOP-15593
> URL: https://issues.apache.org/jira/browse/HADOOP-15593
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 3.0.0
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Blocker
> Fix For: 3.2.0, 3.1.1
>
> Attachments: HADOOP-15593.001.patch, HADOOP-15593.002.patch,
> HADOOP-15593.003.patch, HADOOP-15593.004.patch, HADOOP-15593.005.patch
>
>
> Found the following NPE thrown in UGI tgt renewer. The NPE was thrown within
> an exception handler so the original exception was hidden, though it's likely
> caused by expired tgt.
> {noformat}
> 18/07/02 10:30:57 ERROR util.SparkUncaughtExceptionHandler: Uncaught
> exception in thread Thread[TGT Renewer for f...@example.com,5,main]
> java.lang.NullPointerException
> at
> javax.security.auth.kerberos.KerberosTicket.getEndTime(KerberosTicket.java:482)
> at
> org.apache.hadoop.security.UserGroupInformation$1.run(UserGroupInformation.java:894)
> at java.lang.Thread.run(Thread.java:748){noformat}
> Suspect it's related to [https://bugs.openjdk.java.net/browse/JDK-8154889].
> The relevant code was added in HADOOP-13590. File this jira to handle the
> exception better.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15593) UserGroupInformation TGT renewer throws NPE
[
https://issues.apache.org/jira/browse/HADOOP-15593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated HADOOP-15593:
Fix Version/s: (was: 3.1.2)
3.1.1
> UserGroupInformation TGT renewer throws NPE
> ---
>
> Key: HADOOP-15593
> URL: https://issues.apache.org/jira/browse/HADOOP-15593
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 3.0.0
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Blocker
> Fix For: 3.2.0, 3.1.1
>
> Attachments: HADOOP-15593.001.patch, HADOOP-15593.002.patch,
> HADOOP-15593.003.patch, HADOOP-15593.004.patch, HADOOP-15593.005.patch
>
>
> Found the following NPE thrown in UGI tgt renewer. The NPE was thrown within
> an exception handler so the original exception was hidden, though it's likely
> caused by expired tgt.
> {noformat}
> 18/07/02 10:30:57 ERROR util.SparkUncaughtExceptionHandler: Uncaught
> exception in thread Thread[TGT Renewer for f...@example.com,5,main]
> java.lang.NullPointerException
> at
> javax.security.auth.kerberos.KerberosTicket.getEndTime(KerberosTicket.java:482)
> at
> org.apache.hadoop.security.UserGroupInformation$1.run(UserGroupInformation.java:894)
> at java.lang.Thread.run(Thread.java:748){noformat}
> Suspect it's related to [https://bugs.openjdk.java.net/browse/JDK-8154889].
> The relevant code was added in HADOOP-13590. File this jira to handle the
> exception better.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15637) LocalFs#listLocatedStatus does not filter out hidden .crc files
[
https://issues.apache.org/jira/browse/HADOOP-15637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564219#comment-16564219
]
Erik Krogen commented on HADOOP-15637:
--
Thanks [~vagarychen]
> LocalFs#listLocatedStatus does not filter out hidden .crc files
> ---
>
> Key: HADOOP-15637
> URL: https://issues.apache.org/jira/browse/HADOOP-15637
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.8.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
> Fix For: 2.10.0, 3.2.0, 2.9.2, 2.8.5, 3.0.4, 3.1.2
>
> Attachments: HADOOP-15637-branch-2.8.001.patch,
> HADOOP-15637-branch-3.0.001.patch, HADOOP-15637.000.patch
>
>
> After HADOOP-7165, {{LocalFs#listLocatedStatus}} incorrectly returns the
> hidden {{.crc}} files used to store checksum information. This is because
> HADOOP-7165 implemented {{listLocatedStatus}} on {{FilterFs}}, so the default
> implementation is no longer used, and {{FilterFs}} directly calls the raw FS
> since {{listLocatedStatus}} is not defined in {{ChecksumFs}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15637) LocalFs#listLocatedStatus does not filter out hidden .crc files
[
https://issues.apache.org/jira/browse/HADOOP-15637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HADOOP-15637:
-
Fix Version/s: 2.10.0
> LocalFs#listLocatedStatus does not filter out hidden .crc files
> ---
>
> Key: HADOOP-15637
> URL: https://issues.apache.org/jira/browse/HADOOP-15637
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.8.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
> Fix For: 2.10.0, 3.2.0, 2.9.2, 2.8.5, 3.0.4, 3.1.2
>
> Attachments: HADOOP-15637-branch-2.8.001.patch,
> HADOOP-15637-branch-3.0.001.patch, HADOOP-15637.000.patch
>
>
> After HADOOP-7165, {{LocalFs#listLocatedStatus}} incorrectly returns the
> hidden {{.crc}} files used to store checksum information. This is because
> HADOOP-7165 implemented {{listLocatedStatus}} on {{FilterFs}}, so the default
> implementation is no longer used, and {{FilterFs}} directly calls the raw FS
> since {{listLocatedStatus}} is not defined in {{ChecksumFs}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15637) LocalFs#listLocatedStatus does not filter out hidden .crc files
[
https://issues.apache.org/jira/browse/HADOOP-15637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564218#comment-16564218
]
Chen Liang commented on HADOOP-15637:
-
Right, thanks for the catch [~xkrogen], committed to branch-2 as well.
> LocalFs#listLocatedStatus does not filter out hidden .crc files
> ---
>
> Key: HADOOP-15637
> URL: https://issues.apache.org/jira/browse/HADOOP-15637
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.8.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
> Fix For: 3.2.0, 2.9.2, 2.8.5, 3.0.4, 3.1.2
>
> Attachments: HADOOP-15637-branch-2.8.001.patch,
> HADOOP-15637-branch-3.0.001.patch, HADOOP-15637.000.patch
>
>
> After HADOOP-7165, {{LocalFs#listLocatedStatus}} incorrectly returns the
> hidden {{.crc}} files used to store checksum information. This is because
> HADOOP-7165 implemented {{listLocatedStatus}} on {{FilterFs}}, so the default
> implementation is no longer used, and {{FilterFs}} directly calls the raw FS
> since {{listLocatedStatus}} is not defined in {{ChecksumFs}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15644) Hadoop Docker Image Pip Install Fails on branch-2
[
https://issues.apache.org/jira/browse/HADOOP-15644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564157#comment-16564157
]
genericqa commented on HADOOP-15644:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 4m
48s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
10s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 6m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:f667ef1 |
| JIRA Issue | HADOOP-15644 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933802/HADOOP-15644-branch-2.00.patch
|
| Optional Tests | asflicense shellcheck shelldocs |
| uname | Linux cacbeea9ff6a 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15
17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2 / 5ef8fd5 |
| maven | version: Apache Maven 3.3.9
(bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00) |
| shellcheck | v0.4.7 |
| Max. process+thread count | 38 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14971/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Hadoop Docker Image Pip Install Fails on branch-2
> -
>
> Key: HADOOP-15644
> URL: https://issues.apache.org/jira/browse/HADOOP-15644
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 2.10.0
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Critical
> Attachments: HADOOP-15644-branch-2.00.patch
>
>
> HADOOP-15610 fixes the pip install failures on branch 3.x releases, but it is
> still failing on branch-2, though with a slightly different error message
> {code:java}
> Downloading/unpacking pylint
> Running setup.py (path:/tmp/pip_build_root/pylint/setup.py) egg_info for
> package pylint
> /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown
> distribution option: 'python_requires'
> warnings.warn(msg)
> Traceback (most recent call last):
> File "", line 17, in
> File "/tmp/pip_build_root/pylint/setup.py", line 177, in
> install()
> File "/tmp/pip_build_root/pylint/setup.py", line 174, in install
> **kwargs)
> File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
> _setup_distribution = dist = klass(attrs)
> File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 239,
> in __init__
> self.fetch_build_eggs(attrs.pop('setup_requires'))
> File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 264,
> in fetch_build_eggs
> replace_conflicting=True
> File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 620, in
> resolve
> dist = best[req.key] = env.best_match(req, ws, installer)
> File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 858, in
> best_match
> return self.obtain(req, installer) # try and download/install
> File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 870, in
> obtain
> return installer(requirement)
> File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 314,
> in fetch_build_egg
> return cmd.easy_install(req)
> File
> "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
> 616, in easy_install
> return self.install_item(spec, dist.location, tmpdir, deps)
> File
>
[jira] [Commented] (HADOOP-15644) Hadoop Docker Image Pip Install Fails on branch-2
[
https://issues.apache.org/jira/browse/HADOOP-15644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564141#comment-16564141
]
genericqa commented on HADOOP-15644:
(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at
https://builds.apache.org/job/PreCommit-HADOOP-Build/14971/console in case of
problems.
> Hadoop Docker Image Pip Install Fails on branch-2
> -
>
> Key: HADOOP-15644
> URL: https://issues.apache.org/jira/browse/HADOOP-15644
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 2.10.0
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Critical
> Attachments: HADOOP-15644-branch-2.00.patch
>
>
> HADOOP-15610 fixes the pip install failures on branch 3.x releases, but it is
> still failing on branch-2, though with a slightly different error message
> {code:java}
> Downloading/unpacking pylint
> Running setup.py (path:/tmp/pip_build_root/pylint/setup.py) egg_info for
> package pylint
> /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown
> distribution option: 'python_requires'
> warnings.warn(msg)
> Traceback (most recent call last):
> File "", line 17, in
> File "/tmp/pip_build_root/pylint/setup.py", line 177, in
> install()
> File "/tmp/pip_build_root/pylint/setup.py", line 174, in install
> **kwargs)
> File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
> _setup_distribution = dist = klass(attrs)
> File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 239,
> in __init__
> self.fetch_build_eggs(attrs.pop('setup_requires'))
> File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 264,
> in fetch_build_eggs
> replace_conflicting=True
> File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 620, in
> resolve
> dist = best[req.key] = env.best_match(req, ws, installer)
> File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 858, in
> best_match
> return self.obtain(req, installer) # try and download/install
> File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 870, in
> obtain
> return installer(requirement)
> File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 314,
> in fetch_build_egg
> return cmd.easy_install(req)
> File
> "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
> 616, in easy_install
> return self.install_item(spec, dist.location, tmpdir, deps)
> File
> "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
> 646, in install_item
> dists = self.install_eggs(spec, download, tmpdir)
> File
> "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
> 834, in install_eggs
> return self.build_and_install(setup_script, setup_base)
> File
> "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
> 1040, in build_and_install
> self.run_setup(setup_script, setup_base, args)
> File
> "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
> 1025, in run_setup
> run_setup(setup_script, args)
> File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 50,
> in run_setup
> lambda: execfile(
> File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line
> 100, in run
> return func()
> File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 52,
> in
> {'__file__':setup_script, '__name__':'__main__'}
> File "setup.py", line 76, in
> File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
> _setup_distribution = dist = klass(attrs)
> File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 243,
> in __init__
> _Distribution.__init__(self,attrs)
> File "/usr/lib/python2.7/distutils/dist.py", line 287, in __init__
> self.finalize_options()
> File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 277,
> in finalize_options
> ep.load()(self, ep.name, value)
> File "build/bdist.linux-x86_64/egg/setuptools_scm/integration.py", line
> 10, in version_keyword
> File "build/bdist.linux-x86_64/egg/setuptools_scm/version.py", line 66,
> in _warn_if_setuptools_outdated
> setuptools_scm.version.SetuptoolsOutdatedWarning: your setuptools is too
> old (<12)
> Complete output from command python setup.py egg_info:
> /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown
> distribution option: 'python_requires'
> warnings.warn(msg)
> Traceback
[jira] [Updated] (HADOOP-15644) Hadoop Docker Image Pip Install Fails on branch-2
[
https://issues.apache.org/jira/browse/HADOOP-15644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibo Chen updated HADOOP-15644:
Description:
HADOOP-15610 fixes the pip install failures on branch 3.x releases, but it is
still failing on branch-2, though with a slightly different error message
{code:java}
Downloading/unpacking pylint
Running setup.py (path:/tmp/pip_build_root/pylint/setup.py) egg_info for
package pylint
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution
option: 'python_requires'
warnings.warn(msg)
Traceback (most recent call last):
File "", line 17, in
File "/tmp/pip_build_root/pylint/setup.py", line 177, in
install()
File "/tmp/pip_build_root/pylint/setup.py", line 174, in install
**kwargs)
File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
_setup_distribution = dist = klass(attrs)
File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 239, in
__init__
self.fetch_build_eggs(attrs.pop('setup_requires'))
File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 264, in
fetch_build_eggs
replace_conflicting=True
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 620, in
resolve
dist = best[req.key] = env.best_match(req, ws, installer)
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 858, in
best_match
return self.obtain(req, installer) # try and download/install
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 870, in
obtain
return installer(requirement)
File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 314, in
fetch_build_egg
return cmd.easy_install(req)
File
"/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
616, in easy_install
return self.install_item(spec, dist.location, tmpdir, deps)
File
"/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
646, in install_item
dists = self.install_eggs(spec, download, tmpdir)
File
"/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
834, in install_eggs
return self.build_and_install(setup_script, setup_base)
File
"/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
1040, in build_and_install
self.run_setup(setup_script, setup_base, args)
File
"/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line
1025, in run_setup
run_setup(setup_script, args)
File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 50,
in run_setup
lambda: execfile(
File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 100,
in run
return func()
File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 52,
in
{'__file__':setup_script, '__name__':'__main__'}
File "setup.py", line 76, in
File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
_setup_distribution = dist = klass(attrs)
File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 243, in
__init__
_Distribution.__init__(self,attrs)
File "/usr/lib/python2.7/distutils/dist.py", line 287, in __init__
self.finalize_options()
File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 277, in
finalize_options
ep.load()(self, ep.name, value)
File "build/bdist.linux-x86_64/egg/setuptools_scm/integration.py", line
10, in version_keyword
File "build/bdist.linux-x86_64/egg/setuptools_scm/version.py", line 66,
in _warn_if_setuptools_outdated
setuptools_scm.version.SetuptoolsOutdatedWarning: your setuptools is too
old (<12)
Complete output from command python setup.py egg_info:
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution
option: 'python_requires'
warnings.warn(msg)
Traceback (most recent call last):
File "", line 17, in
File "/tmp/pip_build_root/pylint/setup.py", line 177, in
install()
File "/tmp/pip_build_root/pylint/setup.py", line 174, in install
**kwargs)
File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
_setup_distribution = dist = klass(attrs)
File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 239, in
__init__
self.fetch_build_eggs(attrs.pop('setup_requires'))
File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 264, in
fetch_build_eggs
replace_conflicting=True
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 620, in resolve
dist = best[req.key] = env.best_match(req, ws, installer)
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 858, in
best_match
return self.obtain(req, installer) # try and download/install
File
[jira] [Commented] (HADOOP-15610) Hadoop Docker Image Pip Install Fails
[ https://issues.apache.org/jira/browse/HADOOP-15610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564131#comment-16564131 ] Haibo Chen commented on HADOOP-15610: - I created HADOOP-15644 to fix branch-2 > Hadoop Docker Image Pip Install Fails > - > > Key: HADOOP-15610 > URL: https://issues.apache.org/jira/browse/HADOOP-15610 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jack Bearden >Assignee: Jack Bearden >Priority: Critical > Labels: docker, trunk > Fix For: 3.2.0, 3.1.1, 3.0.4 > > Attachments: HADOOP-15610.001.patch, HADOOP-15610.002.patch, > HADOOP-15610.003.patch, HADOOP-15610.004.patch, HADOOP-15610.005.patch > > > The Hadoop Docker image on trunk does not build. The pip package on the > Ubuntu Xenial repo is out of date and fails by throwing the following error > when attempting to install pylint: > "You are using pip version 8.1.1, however version 10.0.1 is available" > The following patch fixes this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15644) Hadoop Docker Image Pip Install Fails on branch-2
Haibo Chen created HADOOP-15644: --- Summary: Hadoop Docker Image Pip Install Fails on branch-2 Key: HADOOP-15644 URL: https://issues.apache.org/jira/browse/HADOOP-15644 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 2.10.0 Reporter: Haibo Chen Assignee: Haibo Chen Attachments: HADOOP-15644-branch-2.00.patch HADOOP-15610 fixes the pip install failures on branch 3.x releases, but it is still failing on branch-2, though with a slightly different error message -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15644) Hadoop Docker Image Pip Install Fails on branch-2
[ https://issues.apache.org/jira/browse/HADOOP-15644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated HADOOP-15644: Attachment: HADOOP-15644-branch-2.00.patch > Hadoop Docker Image Pip Install Fails on branch-2 > - > > Key: HADOOP-15644 > URL: https://issues.apache.org/jira/browse/HADOOP-15644 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 2.10.0 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Critical > Attachments: HADOOP-15644-branch-2.00.patch > > > HADOOP-15610 fixes the pip install failures on branch 3.x releases, but it is > still failing on branch-2, though with a slightly different error message -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15644) Hadoop Docker Image Pip Install Fails on branch-2
[ https://issues.apache.org/jira/browse/HADOOP-15644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated HADOOP-15644: Status: Patch Available (was: Open) > Hadoop Docker Image Pip Install Fails on branch-2 > - > > Key: HADOOP-15644 > URL: https://issues.apache.org/jira/browse/HADOOP-15644 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 2.10.0 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Critical > Attachments: HADOOP-15644-branch-2.00.patch > > > HADOOP-15610 fixes the pip install failures on branch 3.x releases, but it is > still failing on branch-2, though with a slightly different error message -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15610) Hadoop Docker Image Pip Install Fails
[
https://issues.apache.org/jira/browse/HADOOP-15610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564113#comment-16564113
]
Haibo Chen commented on HADOOP-15610:
-
I tried to backport this to branch-2
{code:java}
diff --git a/dev-support/docker/Dockerfile b/dev-support/docker/Dockerfile
index b1fc420..026109f 100644
--- a/dev-support/docker/Dockerfile
+++ b/dev-support/docker/Dockerfile
@@ -122,7 +122,7 @@ RUN apt-get -q install --no-install-recommends -y bats
# Install pylint
-RUN pip install pylint
+RUN pip install pylint==1.9.2{code}
It fixes the build failure that I ran across while running ./start-build-env.sh
> Hadoop Docker Image Pip Install Fails
> -
>
> Key: HADOOP-15610
> URL: https://issues.apache.org/jira/browse/HADOOP-15610
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Jack Bearden
>Assignee: Jack Bearden
>Priority: Critical
> Labels: docker, trunk
> Fix For: 3.2.0, 3.1.1, 3.0.4
>
> Attachments: HADOOP-15610.001.patch, HADOOP-15610.002.patch,
> HADOOP-15610.003.patch, HADOOP-15610.004.patch, HADOOP-15610.005.patch
>
>
> The Hadoop Docker image on trunk does not build. The pip package on the
> Ubuntu Xenial repo is out of date and fails by throwing the following error
> when attempting to install pylint:
> "You are using pip version 8.1.1, however version 10.0.1 is available"
> The following patch fixes this issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14212) Expose SecurityEnabled boolean field in JMX for other services besides NameNode
[
https://issues.apache.org/jira/browse/HADOOP-14212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564079#comment-16564079
]
genericqa commented on HADOOP-14212:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 24s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 27m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 7s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 3s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 18m
7s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 59s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
46s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}288m 5s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
| | hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HADOOP-14212 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933762/HADOOP-14212.009.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 228701117dc7 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
|
[jira] [Updated] (HADOOP-15642) Update to latest/recent version of aws-sdk for Hadoop 3.2
[ https://issues.apache.org/jira/browse/HADOOP-15642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15642: Description: Move to a later version of the AWS SDK library for a different set of features and issues. proposed version: 1.11.374 One thing which doesn't work on the one we ship with is the ability to create assumed role sessions >1h, as there's a check in the client lib for role-duration <= 3600 seconds. I'll assume more recent SDKs delegate duration checks to the far end. see: [https://aws.amazon.com/about-aws/whats-new/2018/03/longer-role-sessions/] * assuming later versions will extend assumed role life, docs will need changing, * Adding a test in HADOOP-15583 which expects an error message if you ask for a duration of 3h; this should act as the test to see what happens. * think this time would be good to explicitly write down the SDK update process was: Move to a later version of the AWS SDK library for a different set of features and issues. One thing which doesn't work on the one we ship with is the ability to create assumed role sessions >1h, as there's a check in the client lib for role-duration <= 3600 seconds. I'll assume more recent SDKs delegate duration checks to the far end. see: [https://aws.amazon.com/about-aws/whats-new/2018/03/longer-role-sessions/] * assuming later versions will extend assumed role life, docs will need changing, * Adding a test in HADOOP-15583 which expects an error message if you ask for a duration of 3h; this should act as the test to see what happens. * think this time would be good to explicitly write down the SDK update process > Update to latest/recent version of aws-sdk for Hadoop 3.2 > - > > Key: HADOOP-15642 > URL: https://issues.apache.org/jira/browse/HADOOP-15642 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15642-001.patch, Screen Shot 2018-07-30 at > 14.11.22.png > > > Move to a later version of the AWS SDK library for a different set of > features and issues. > proposed version: 1.11.374 > One thing which doesn't work on the one we ship with is the ability to create > assumed role sessions >1h, as there's a check in the client lib for > role-duration <= 3600 seconds. I'll assume more recent SDKs delegate duration > checks to the far end. > see: > [https://aws.amazon.com/about-aws/whats-new/2018/03/longer-role-sessions/] > * assuming later versions will extend assumed role life, docs will need > changing, > * Adding a test in HADOOP-15583 which expects an error message if you ask for > a duration of 3h; this should act as the test to see what happens. > * think this time would be good to explicitly write down the SDK update > process -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15613) KerberosAuthenticator should resolve the hostname to get the service principal
[ https://issues.apache.org/jira/browse/HADOOP-15613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564031#comment-16564031 ] Ajay Kumar commented on HADOOP-15613: - [~surendrasingh] Does this work when you replace ip with hostname? Replacing hostname with fully qualified domain name may break some existing applications where principals were generated with host name but not fqdn. > KerberosAuthenticator should resolve the hostname to get the service principal > -- > > Key: HADOOP-15613 > URL: https://issues.apache.org/jira/browse/HADOOP-15613 > Project: Hadoop Common > Issue Type: Bug > Components: auth >Affects Versions: 2.7.2 >Reporter: Surendra Singh Lilhore >Assignee: Surendra Singh Lilhore >Priority: Major > Attachments: HADOOP-15613.001.patch > > > When in rest URL, IP is used as a address then > "KerberosAuthenticator.this.url.getHost()" not able to resolve the hostname. > This hostname will be used to construct server HTTP principal. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15576) S3A Multipart Uploader to work with S3Guard and encryption
[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563967#comment-16563967
]
Ewan Higgs edited comment on HADOOP-15576 at 7/31/18 4:50 PM:
--
004
- Add test for empty uploadID
- Make the part size configurable by FS implementation as S3A requires 5MB part
sizes (except the last part).
- Use md5sum to compare the file contents so we don't need to compare multiple
5MB blocks together if they were a string.
- Fix META-INF/service for S3AMultipartUploader$Factory.
- Removed test based on comparing fs.open(pathHandle) since S3a doesn't support
opening by path handle.
{quote}
yes. If it's java serialization then it needs to be looked at to make sure it
defends against malicios stuff.
{quote}
Would you prefer protobuf for this? It would be an extra generation step + new
dependency.
was (Author: ehiggs):
004
- Add test for empty uploadID
- Make the part size configurable by FS implementation as S3A requires 5MB part
sizes (except the last part).
- Use md5sum to compare the file contents so we don't need to compare multiple
5MB blocks together if they were a string.
- Fix META-INF/service for S3AMultipartUploader$Factory.
{quote}
yes. If it's java serialization then it needs to be looked at to make sure it
defends against malicios stuff.
{quote}
Would you prefer protobuf for this? It would be an extra generation step + new
dependency.
> S3A Multipart Uploader to work with S3Guard and encryption
> ---
>
> Key: HADOOP-15576
> URL: https://issues.apache.org/jira/browse/HADOOP-15576
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Ewan Higgs
>Priority: Blocker
> Attachments: HADOOP-15576.001.patch, HADOOP-15576.002.patch,
> HADOOP-15576.003.patch, HADOOP-15576.004.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with
> the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper;
> adding any new methods are needed there.
> # Tests. the tests of HDFS-13713.
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and
> even if there was, it might not show that S3Guard was bypassed, because
> there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader,
> encryption settings are't picked up. Files being uploaded this way *are not
> being encrypted*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15576) S3A Multipart Uploader to work with S3Guard and encryption
[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563967#comment-16563967
]
Ewan Higgs commented on HADOOP-15576:
-
004
- Add test for empty uploadID
- Make the part size configurable by FS implementation as S3A requires 5MB part
sizes (except the last part).
- Use md5sum to compare the file contents so we don't need to compare multiple
5MB blocks together if they were a string.
- Fix META-INF/service for S3AMultipartUploader$Factory.
{quote}
yes. If it's java serialization then it needs to be looked at to make sure it
defends against malicios stuff.
{quote}
Would you prefer protobuf for this? It would be an extra generation step + new
dependency.
> S3A Multipart Uploader to work with S3Guard and encryption
> ---
>
> Key: HADOOP-15576
> URL: https://issues.apache.org/jira/browse/HADOOP-15576
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Ewan Higgs
>Priority: Blocker
> Attachments: HADOOP-15576.001.patch, HADOOP-15576.002.patch,
> HADOOP-15576.003.patch, HADOOP-15576.004.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with
> the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper;
> adding any new methods are needed there.
> # Tests. the tests of HDFS-13713.
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and
> even if there was, it might not show that S3Guard was bypassed, because
> there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader,
> encryption settings are't picked up. Files being uploaded this way *are not
> being encrypted*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15576) S3A Multipart Uploader to work with S3Guard and encryption
[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HADOOP-15576:
Status: Open (was: Patch Available)
> S3A Multipart Uploader to work with S3Guard and encryption
> ---
>
> Key: HADOOP-15576
> URL: https://issues.apache.org/jira/browse/HADOOP-15576
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Ewan Higgs
>Priority: Blocker
> Attachments: HADOOP-15576.001.patch, HADOOP-15576.002.patch,
> HADOOP-15576.003.patch, HADOOP-15576.004.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with
> the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper;
> adding any new methods are needed there.
> # Tests. the tests of HDFS-13713.
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and
> even if there was, it might not show that S3Guard was bypassed, because
> there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader,
> encryption settings are't picked up. Files being uploaded this way *are not
> being encrypted*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15576) S3A Multipart Uploader to work with S3Guard and encryption
[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HADOOP-15576:
Attachment: HADOOP-15576.004.patch
> S3A Multipart Uploader to work with S3Guard and encryption
> ---
>
> Key: HADOOP-15576
> URL: https://issues.apache.org/jira/browse/HADOOP-15576
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Ewan Higgs
>Priority: Blocker
> Attachments: HADOOP-15576.001.patch, HADOOP-15576.002.patch,
> HADOOP-15576.003.patch, HADOOP-15576.004.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with
> the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper;
> adding any new methods are needed there.
> # Tests. the tests of HDFS-13713.
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and
> even if there was, it might not show that S3Guard was bypassed, because
> there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader,
> encryption settings are't picked up. Files being uploaded this way *are not
> being encrypted*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15576) S3A Multipart Uploader to work with S3Guard and encryption
[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HADOOP-15576:
Status: Patch Available (was: Open)
> S3A Multipart Uploader to work with S3Guard and encryption
> ---
>
> Key: HADOOP-15576
> URL: https://issues.apache.org/jira/browse/HADOOP-15576
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Ewan Higgs
>Priority: Blocker
> Attachments: HADOOP-15576.001.patch, HADOOP-15576.002.patch,
> HADOOP-15576.003.patch, HADOOP-15576.004.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with
> the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper;
> adding any new methods are needed there.
> # Tests. the tests of HDFS-13713.
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and
> even if there was, it might not show that S3Guard was bypassed, because
> there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader,
> encryption settings are't picked up. Files being uploaded this way *are not
> being encrypted*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15637) LocalFs#listLocatedStatus does not filter out hidden .crc files
[
https://issues.apache.org/jira/browse/HADOOP-15637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563831#comment-16563831
]
Erik Krogen commented on HADOOP-15637:
--
Thank you [~vagarychen]! We should do branch-2 as well right?
> LocalFs#listLocatedStatus does not filter out hidden .crc files
> ---
>
> Key: HADOOP-15637
> URL: https://issues.apache.org/jira/browse/HADOOP-15637
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.8.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
> Fix For: 3.2.0, 2.9.2, 2.8.5, 3.0.4, 3.1.2
>
> Attachments: HADOOP-15637-branch-2.8.001.patch,
> HADOOP-15637-branch-3.0.001.patch, HADOOP-15637.000.patch
>
>
> After HADOOP-7165, {{LocalFs#listLocatedStatus}} incorrectly returns the
> hidden {{.crc}} files used to store checksum information. This is because
> HADOOP-7165 implemented {{listLocatedStatus}} on {{FilterFs}}, so the default
> implementation is no longer used, and {{FilterFs}} directly calls the raw FS
> since {{listLocatedStatus}} is not defined in {{ChecksumFs}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15083) Create base image for running hadoop in docker containers
[ https://issues.apache.org/jira/browse/HADOOP-15083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HADOOP-15083: -- Resolution: Fixed Status: Resolved (was: Patch Available) Closing this issue. The base image is at the dockerhub under apache/hadoop-runner and actively used during the ozone development. > Create base image for running hadoop in docker containers > - > > Key: HADOOP-15083 > URL: https://issues.apache.org/jira/browse/HADOOP-15083 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-15083-docker-hadoop-runner.003.patch, > HADOOP-15083.001.patch, HADOOP-15083.002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14212) Expose SecurityEnabled boolean field in JMX for other services besides NameNode
[
https://issues.apache.org/jira/browse/HADOOP-14212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563633#comment-16563633
]
Adam Antal commented on HADOOP-14212:
-
Thank you for the suggestions, [~gabor.bota]. I uploaded patch v9 with these
items, and also ran an additional checkstyle.
One minor side question of mine is: shouldn't the
{{@InterfaceAudience.Private}} annotation be used since end users shouldn't be
implementing those interfaces - they should use the API instead.
> Expose SecurityEnabled boolean field in JMX for other services besides
> NameNode
> ---
>
> Key: HADOOP-14212
> URL: https://issues.apache.org/jira/browse/HADOOP-14212
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Ray Burgemeestre
>Assignee: Adam Antal
>Priority: Minor
> Labels: security
> Attachments: HADOOP-14212.001.patch, HADOOP-14212.002.patch,
> HADOOP-14212.003.patch, HADOOP-14212.004.patch, HADOOP-14212.005.patch,
> HADOOP-14212.005.patch, HADOOP-14212.005.patch, HADOOP-14212.006.patch,
> HADOOP-14212.007.patch, HADOOP-14212.008.patch, HADOOP-14212.009.patch
>
>
> The following commit
> https://github.com/apache/hadoop/commit/dc17bda4b677e30c02c2a9a053895a43e41f7a12
> introduced a "SecurityEnabled" field in the JMX output for the NameNode. I
> believe it would be nice to add this same change to the JMX output of other
> services: Secondary Namenode, ResourceManager, NodeManagers, DataNodes, etc.
> So that it can be queried whether Security is enabled in all JMX resources.
> The reason I am suggesting this feature / improvement is that I think it
> would provide a clean way to check whether your cluster is completely
> Kerberized or not. I don't think there is an easy/clean way to do this now,
> other than checking the logs, checking ports etc.?
> The file where the change was made is
> hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
> has the following function now:
> {code:java}
> @Override // NameNodeStatusMXBean
> public boolean isSecurityEnabled() {
> return UserGroupInformation.isSecurityEnabled();
> }
> {code}
> I would be happy to develop a patch if it seems useful by others as well?
> This is a snippet from the JMX output from the NameNode in case security is
> not enabled:
> {code}
> {
> "name" : "Hadoop:service=NameNode,name=NameNodeStatus",
> "modelerType" : "org.apache.hadoop.hdfs.server.namenode.NameNode",
> "NNRole" : "NameNode",
> "HostAndPort" : "node001.cm.cluster:8020",
> "SecurityEnabled" : false,
> "LastHATransitionTime" : 0,
> "State" : "standby"
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14212) Expose SecurityEnabled boolean field in JMX for other services besides NameNode
[
https://issues.apache.org/jira/browse/HADOOP-14212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Antal updated HADOOP-14212:
Attachment: HADOOP-14212.009.patch
> Expose SecurityEnabled boolean field in JMX for other services besides
> NameNode
> ---
>
> Key: HADOOP-14212
> URL: https://issues.apache.org/jira/browse/HADOOP-14212
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Ray Burgemeestre
>Assignee: Adam Antal
>Priority: Minor
> Labels: security
> Attachments: HADOOP-14212.001.patch, HADOOP-14212.002.patch,
> HADOOP-14212.003.patch, HADOOP-14212.004.patch, HADOOP-14212.005.patch,
> HADOOP-14212.005.patch, HADOOP-14212.005.patch, HADOOP-14212.006.patch,
> HADOOP-14212.007.patch, HADOOP-14212.008.patch, HADOOP-14212.009.patch
>
>
> The following commit
> https://github.com/apache/hadoop/commit/dc17bda4b677e30c02c2a9a053895a43e41f7a12
> introduced a "SecurityEnabled" field in the JMX output for the NameNode. I
> believe it would be nice to add this same change to the JMX output of other
> services: Secondary Namenode, ResourceManager, NodeManagers, DataNodes, etc.
> So that it can be queried whether Security is enabled in all JMX resources.
> The reason I am suggesting this feature / improvement is that I think it
> would provide a clean way to check whether your cluster is completely
> Kerberized or not. I don't think there is an easy/clean way to do this now,
> other than checking the logs, checking ports etc.?
> The file where the change was made is
> hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
> has the following function now:
> {code:java}
> @Override // NameNodeStatusMXBean
> public boolean isSecurityEnabled() {
> return UserGroupInformation.isSecurityEnabled();
> }
> {code}
> I would be happy to develop a patch if it seems useful by others as well?
> This is a snippet from the JMX output from the NameNode in case security is
> not enabled:
> {code}
> {
> "name" : "Hadoop:service=NameNode,name=NameNodeStatus",
> "modelerType" : "org.apache.hadoop.hdfs.server.namenode.NameNode",
> "NNRole" : "NameNode",
> "HostAndPort" : "node001.cm.cluster:8020",
> "SecurityEnabled" : false,
> "LastHATransitionTime" : 0,
> "State" : "standby"
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15607) AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
[
https://issues.apache.org/jira/browse/HADOOP-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563531#comment-16563531
]
wujinhu edited comment on HADOOP-15607 at 7/31/18 11:58 AM:
[~Sammi] I have built in my local environment successfully. Can we get some
logs from remote servers?
[https://builds.apache.org/job/PreCommit-HADOOP-Build/14966/console]
I have found some error messages from the link like this:
setuptools_scm.version.SetuptoolsOutdatedWarning: your setuptools is too old
(<12)
Cleaning up...
Command python setup.py egg_info failed with error code 1 in
/tmp/pip_build_root/pylint
Storing debug log for failure in /root/.pip/pip.log
The command '/bin/sh -c pip install pylint' returned a non-zero code: 1
was (Author: wujinhu):
[~Sammi] I have built in my local environment successfully. Can we get some
logs from remote servers?
> AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
> -
>
> Key: HADOOP-15607
> URL: https://issues.apache.org/jira/browse/HADOOP-15607
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.1.0, 2.10.0, 2.9.1, 3.2.0, 3.0.3
>Reporter: wujinhu
>Assignee: wujinhu
>Priority: Major
> Fix For: 3.2.0, 3.0.4, 3.1.2
>
> Attachments: HADOOP-15607-branch-2.001.patch, HADOOP-15607.001.patch,
> HADOOP-15607.002.patch, HADOOP-15607.003.patch, HADOOP-15607.004.patch
>
>
> When I generated data with hive-tpcds tool, I got exception below:
> 2018-07-16 14:50:43,680 INFO mapreduce.Job: Task Id :
> attempt_1531723399698_0001_m_52_0, Status : FAILED
> Error: com.aliyun.oss.OSSException: The list of parts was not in ascending
> order. Parts list must specified in order by part number.
> [ErrorCode]: InvalidPartOrder
> [RequestId]: 5B4C40425FCC208D79D1EAF5
> [HostId]: 100.103.0.137
> [ResponseError]:
>
>
> InvalidPartOrder
> The list of parts was not in ascending order. Parts list must
> specified in order by part number.
> 5B4C40425FCC208D79D1EAF5
> xx.xx.xx.xx
> current PartNumber 3, you given part number 3is not in
> ascending order
>
> at
> com.aliyun.oss.common.utils.ExceptionFactory.createOSSException(ExceptionFactory.java:99)
> at
> com.aliyun.oss.internal.OSSErrorResponseHandler.handle(OSSErrorResponseHandler.java:69)
> at
> com.aliyun.oss.common.comm.ServiceClient.handleResponse(ServiceClient.java:248)
> at
> com.aliyun.oss.common.comm.ServiceClient.sendRequestImpl(ServiceClient.java:130)
> at
> com.aliyun.oss.common.comm.ServiceClient.sendRequest(ServiceClient.java:68)
> at com.aliyun.oss.internal.OSSOperation.send(OSSOperation.java:94)
> at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:149)
> at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:113)
> at
> com.aliyun.oss.internal.OSSMultipartOperation.completeMultipartUpload(OSSMultipartOperation.java:185)
> at com.aliyun.oss.OSSClient.completeMultipartUpload(OSSClient.java:790)
> at
> org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.completeMultipartUpload(AliyunOSSFileSystemStore.java:643)
> at
> org.apache.hadoop.fs.aliyun.oss.AliyunOSSBlockOutputStream.close(AliyunOSSBlockOutputStream.java:120)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
> at
> org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:106)
> at
> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.close(MultipleOutputs.java:574)
> at org.notmysock.tpcds.GenTable$DSDGen.cleanup(GenTable.java:169)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686)
>
> I reviewed code below,
> {code:java}
> blockId {code}
> has thread synchronization problem
> {code:java}
> // code placeholder
> private void uploadCurrentPart() throws IOException {
> blockFiles.add(blockFile);
> blockStream.flush();
> blockStream.close();
> if (blockId == 0) {
> uploadId = store.getUploadId(key);
> }
> ListenableFuture partETagFuture =
> executorService.submit(() -> {
> PartETag partETag = store.uploadPart(blockFile, key, uploadId,
> blockId + 1);
> return partETag;
> });
>
[jira] [Commented] (HADOOP-15607) AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
[
https://issues.apache.org/jira/browse/HADOOP-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563531#comment-16563531
]
wujinhu commented on HADOOP-15607:
--
[~Sammi] I have built in my local environment successfully. Can we get some
logs from remote servers?
> AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
> -
>
> Key: HADOOP-15607
> URL: https://issues.apache.org/jira/browse/HADOOP-15607
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.1.0, 2.10.0, 2.9.1, 3.2.0, 3.0.3
>Reporter: wujinhu
>Assignee: wujinhu
>Priority: Major
> Fix For: 3.2.0, 3.0.4, 3.1.2
>
> Attachments: HADOOP-15607-branch-2.001.patch, HADOOP-15607.001.patch,
> HADOOP-15607.002.patch, HADOOP-15607.003.patch, HADOOP-15607.004.patch
>
>
> When I generated data with hive-tpcds tool, I got exception below:
> 2018-07-16 14:50:43,680 INFO mapreduce.Job: Task Id :
> attempt_1531723399698_0001_m_52_0, Status : FAILED
> Error: com.aliyun.oss.OSSException: The list of parts was not in ascending
> order. Parts list must specified in order by part number.
> [ErrorCode]: InvalidPartOrder
> [RequestId]: 5B4C40425FCC208D79D1EAF5
> [HostId]: 100.103.0.137
> [ResponseError]:
>
>
> InvalidPartOrder
> The list of parts was not in ascending order. Parts list must
> specified in order by part number.
> 5B4C40425FCC208D79D1EAF5
> xx.xx.xx.xx
> current PartNumber 3, you given part number 3is not in
> ascending order
>
> at
> com.aliyun.oss.common.utils.ExceptionFactory.createOSSException(ExceptionFactory.java:99)
> at
> com.aliyun.oss.internal.OSSErrorResponseHandler.handle(OSSErrorResponseHandler.java:69)
> at
> com.aliyun.oss.common.comm.ServiceClient.handleResponse(ServiceClient.java:248)
> at
> com.aliyun.oss.common.comm.ServiceClient.sendRequestImpl(ServiceClient.java:130)
> at
> com.aliyun.oss.common.comm.ServiceClient.sendRequest(ServiceClient.java:68)
> at com.aliyun.oss.internal.OSSOperation.send(OSSOperation.java:94)
> at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:149)
> at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:113)
> at
> com.aliyun.oss.internal.OSSMultipartOperation.completeMultipartUpload(OSSMultipartOperation.java:185)
> at com.aliyun.oss.OSSClient.completeMultipartUpload(OSSClient.java:790)
> at
> org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.completeMultipartUpload(AliyunOSSFileSystemStore.java:643)
> at
> org.apache.hadoop.fs.aliyun.oss.AliyunOSSBlockOutputStream.close(AliyunOSSBlockOutputStream.java:120)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
> at
> org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:106)
> at
> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.close(MultipleOutputs.java:574)
> at org.notmysock.tpcds.GenTable$DSDGen.cleanup(GenTable.java:169)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686)
>
> I reviewed code below,
> {code:java}
> blockId {code}
> has thread synchronization problem
> {code:java}
> // code placeholder
> private void uploadCurrentPart() throws IOException {
> blockFiles.add(blockFile);
> blockStream.flush();
> blockStream.close();
> if (blockId == 0) {
> uploadId = store.getUploadId(key);
> }
> ListenableFuture partETagFuture =
> executorService.submit(() -> {
> PartETag partETag = store.uploadPart(blockFile, key, uploadId,
> blockId + 1);
> return partETag;
> });
> partETagsFutures.add(partETagFuture);
> blockFile = newBlockFile();
> blockId++;
> blockStream = new BufferedOutputStream(new FileOutputStream(blockFile));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14212) Expose SecurityEnabled boolean field in JMX for other services besides NameNode
[
https://issues.apache.org/jira/browse/HADOOP-14212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563352#comment-16563352
]
Gabor Bota commented on HADOOP-14212:
-
Thanks for working on this [~adam.antal]!
I've noticed the following while I did a review:
* In {{SecondaryNameNodeInfoMXBean}} you removed {{public}} modifiers. It's
generally not encouraged to do that until you have some real reason to do that
(like increasing visibility for testing purposes). I'd recommend just leaving
it as it is.
* In {{TestDataNodeMXBean}} you've added a new test, which is very good. But
for {{TestDataNodeMXBean #testDataNodeMXBean}} changing the try-catch to
try-with-resources could cause backporting problems in the future, so it would
be wise to change just what you have to in this case.
* In {{NodeManager#serviceInit) there's a whitespace before the parentheses -
it's just a minor flaw, but still would be better not to have it
* In {{NodeManagerMXBean}} and {{ResourceManagerMXBean}} interfaces you should
add {{@InterfaceAudience.Public}} and {{@InterfaceStability.Evolving}}. See eg.
[Interface
Taxonomy|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/InterfaceClassification.html]
* For the class {{ResourceManager}} line 144 is longer than just 80 chars
> Expose SecurityEnabled boolean field in JMX for other services besides
> NameNode
> ---
>
> Key: HADOOP-14212
> URL: https://issues.apache.org/jira/browse/HADOOP-14212
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Ray Burgemeestre
>Assignee: Adam Antal
>Priority: Minor
> Labels: security
> Attachments: HADOOP-14212.001.patch, HADOOP-14212.002.patch,
> HADOOP-14212.003.patch, HADOOP-14212.004.patch, HADOOP-14212.005.patch,
> HADOOP-14212.005.patch, HADOOP-14212.005.patch, HADOOP-14212.006.patch,
> HADOOP-14212.007.patch, HADOOP-14212.008.patch
>
>
> The following commit
> https://github.com/apache/hadoop/commit/dc17bda4b677e30c02c2a9a053895a43e41f7a12
> introduced a "SecurityEnabled" field in the JMX output for the NameNode. I
> believe it would be nice to add this same change to the JMX output of other
> services: Secondary Namenode, ResourceManager, NodeManagers, DataNodes, etc.
> So that it can be queried whether Security is enabled in all JMX resources.
> The reason I am suggesting this feature / improvement is that I think it
> would provide a clean way to check whether your cluster is completely
> Kerberized or not. I don't think there is an easy/clean way to do this now,
> other than checking the logs, checking ports etc.?
> The file where the change was made is
> hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
> has the following function now:
> {code:java}
> @Override // NameNodeStatusMXBean
> public boolean isSecurityEnabled() {
> return UserGroupInformation.isSecurityEnabled();
> }
> {code}
> I would be happy to develop a patch if it seems useful by others as well?
> This is a snippet from the JMX output from the NameNode in case security is
> not enabled:
> {code}
> {
> "name" : "Hadoop:service=NameNode,name=NameNodeStatus",
> "modelerType" : "org.apache.hadoop.hdfs.server.namenode.NameNode",
> "NNRole" : "NameNode",
> "HostAndPort" : "node001.cm.cluster:8020",
> "SecurityEnabled" : false,
> "LastHATransitionTime" : 0,
> "State" : "standby"
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14212) Expose SecurityEnabled boolean field in JMX for other services besides NameNode
[
https://issues.apache.org/jira/browse/HADOOP-14212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota updated HADOOP-14212:
Labels: security (was: newbie security)
> Expose SecurityEnabled boolean field in JMX for other services besides
> NameNode
> ---
>
> Key: HADOOP-14212
> URL: https://issues.apache.org/jira/browse/HADOOP-14212
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Ray Burgemeestre
>Assignee: Adam Antal
>Priority: Minor
> Labels: security
> Attachments: HADOOP-14212.001.patch, HADOOP-14212.002.patch,
> HADOOP-14212.003.patch, HADOOP-14212.004.patch, HADOOP-14212.005.patch,
> HADOOP-14212.005.patch, HADOOP-14212.005.patch, HADOOP-14212.006.patch,
> HADOOP-14212.007.patch, HADOOP-14212.008.patch
>
>
> The following commit
> https://github.com/apache/hadoop/commit/dc17bda4b677e30c02c2a9a053895a43e41f7a12
> introduced a "SecurityEnabled" field in the JMX output for the NameNode. I
> believe it would be nice to add this same change to the JMX output of other
> services: Secondary Namenode, ResourceManager, NodeManagers, DataNodes, etc.
> So that it can be queried whether Security is enabled in all JMX resources.
> The reason I am suggesting this feature / improvement is that I think it
> would provide a clean way to check whether your cluster is completely
> Kerberized or not. I don't think there is an easy/clean way to do this now,
> other than checking the logs, checking ports etc.?
> The file where the change was made is
> hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
> has the following function now:
> {code:java}
> @Override // NameNodeStatusMXBean
> public boolean isSecurityEnabled() {
> return UserGroupInformation.isSecurityEnabled();
> }
> {code}
> I would be happy to develop a patch if it seems useful by others as well?
> This is a snippet from the JMX output from the NameNode in case security is
> not enabled:
> {code}
> {
> "name" : "Hadoop:service=NameNode,name=NameNodeStatus",
> "modelerType" : "org.apache.hadoop.hdfs.server.namenode.NameNode",
> "NNRole" : "NameNode",
> "HostAndPort" : "node001.cm.cluster:8020",
> "SecurityEnabled" : false,
> "LastHATransitionTime" : 0,
> "State" : "standby"
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563290#comment-16563290
]
genericqa commented on HADOOP-14445:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 29m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 21s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 32m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 32m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 26s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
22s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
46s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
54s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}145m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HADOOP-14445 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933703/HADOOP-14445.16.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux acd9d3ae8bec 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15
17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 3e06a5d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| Test Results |
[jira] [Commented] (HADOOP-15642) Update to latest/recent version of aws-sdk for Hadoop 3.2
[
https://issues.apache.org/jira/browse/HADOOP-15642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563280#comment-16563280
]
genericqa commented on HADOOP-15642:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
57s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
68m 45s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 26m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 26m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 8s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
24s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
34s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}130m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HADOOP-15642 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933704/HADOOP-15642-001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml |
| uname | Linux 3188598dbb0c 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 3e06a5d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14968/artifact/out/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14968/testReport/ |
| Max. process+thread count | 318 (vs. ulimit of 1) |
| modules | C: hadoop-project hadoop-tools/hadoop-aws U: . |
| Console output |
[jira] [Commented] (HADOOP-15566) Remove HTrace support
[ https://issues.apache.org/jira/browse/HADOOP-15566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563253#comment-16563253 ] Elek, Marton commented on HADOOP-15566: --- Just for the reference, the links for the started mailing list discussions: https://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201807.mbox/%3CCADiq6%3DxdAuPT5q8PNdXBnSODzniKw2zBGo-z9PwCA2_mrDc7wg%40mail.gmail.com%3E https://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201807.mbox/%3cCADcMMgEkJ=OqhJ83-aPFQZ+TZ+5BH=7w6-tsahd9hlpuc3e...@mail.gmail.com%3e (Thanks to [~stack] and [~jojochuang]) > Remove HTrace support > - > > Key: HADOOP-15566 > URL: https://issues.apache.org/jira/browse/HADOOP-15566 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics >Affects Versions: 3.1.0 >Reporter: Todd Lipcon >Priority: Major > Labels: security > Attachments: Screen Shot 2018-06-29 at 11.59.16 AM.png, > ss-trace-s3a.png > > > The HTrace incubator project has voted to retire itself and won't be making > further releases. The Hadoop project currently has various hooks with HTrace. > It seems in some cases (eg HDFS-13702) these hooks have had measurable > performance overhead. Given these two factors, I think we should consider > removing the HTrace integration. If there is someone willing to do the work, > replacing it with OpenTracing might be a better choice since there is an > active community. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
