[jira] [Updated] (HADOOP-13934) S3Guard: DynamoDBMetadataStore#move() could be throwing exception due to BatchWriteItem limits
[
https://issues.apache.org/jira/browse/HADOOP-13934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-13934:
---
Summary: S3Guard: DynamoDBMetadataStore#move() could be throwing exception
due to BatchWriteItem limits (was: S3Guard: DynamoDBMetaStore::move could be
throwing exception due to BatchWriteItem limits)
> S3Guard: DynamoDBMetadataStore#move() could be throwing exception due to
> BatchWriteItem limits
> --
>
> Key: HADOOP-13934
> URL: https://issues.apache.org/jira/browse/HADOOP-13934
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Rajesh Balamohan
>Assignee: Mingliang Liu
>Priority: Minor
> Attachments: HADOOP-13934-HADOOP-13345.000.patch,
> HADOOP-13934-HADOOP-13345.001.patch, HADOOP-13934-HADOOP-13345.002.patch,
> HADOOP-13934-HADOOP-13345.003.patch
>
>
> When using {{DynamoDBMetadataStore}} with a insert heavy hive app , it
> started throwing exceptions in {{DynamoDBMetadataStore::move}}. But just with
> the following exception, it is relatively hard to debug on the real issue in
> DynamoDB side.
> {noformat}
> Caused by: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: 1
> validation error detected: Value
> '{ddb-table-name-334=[com.amazonaws.dynamodb.v20120810.WriteRequest@ca1da583,
> com.amazonaws.dynamodb.v20120810.WriteRequest@ca1fc7cd,
> com.amazonaws.dynamodb.v20120810.WriteRequest@ca4244e6,
> com.amazonaws.dynamodb.v20120810.WriteRequest@ca2f58a9,
> com.amazonaws.dynamodb.v20120810.WriteRequest@ca3525f8,
> ...
> ...
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1529)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1167)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:948)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:661)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:635)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:618)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:586)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:573)
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:445)
> at
> com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:1722)
> at
> com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1698)
> at
> com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.batchWriteItem(AmazonDynamoDBClient.java:668)
> at
> com.amazonaws.services.dynamodbv2.document.internal.BatchWriteItemImpl.doBatchWriteItem(BatchWriteItemImpl.java:111)
> at
> com.amazonaws.services.dynamodbv2.document.internal.BatchWriteItemImpl.batchWriteItem(BatchWriteItemImpl.java:52)
> at
> com.amazonaws.services.dynamodbv2.document.DynamoDB.batchWriteItem(DynamoDB.java:178)
> at
> org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.move(DynamoDBMetadataStore.java:351)
> ... 28 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13922) Some modules have dependencies on hadoop-client jar removed by HADOOP-11804
[ https://issues.apache.org/jira/browse/HADOOP-13922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15797463#comment-15797463 ] Marcel B commented on HADOOP-13922: --- Sorry for the noise - updating b31e195..e49e0a6 solved the issue. I have no clue what was going on yesterday. Thank you! > Some modules have dependencies on hadoop-client jar removed by HADOOP-11804 > --- > > Key: HADOOP-13922 > URL: https://issues.apache.org/jira/browse/HADOOP-13922 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.0.0-alpha2 >Reporter: Joe Pallas >Assignee: Sean Busbey >Priority: Blocker > Fix For: 3.0.0-alpha2 > > Attachments: HADOOP-13922.1.patch > > > As discussed in [HADOOP-11804 comment > 15758048|https://issues.apache.org/jira/browse/HADOOP-11804?focusedCommentId=15758048&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15758048] > and following comments, there are still dependencies on the now-removed > hadoop-client jar. The current code builds only because an obsolete snapshot > of the jar is found on the repository server. Changing the project version > to something new exposes the problem. > While the build currently dies at hadoop-tools/hadoop-sls, I'm seeing issues > with some Hadoop Client modules, too. > I'm filing a new bug because I can't reopen HADOOP-11804. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[
https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15797457#comment-15797457
]
Xiao Chen commented on HADOOP-13597:
Thanks [~jojochuang] for the review. Good catch, searching for tomcat I can
also see {{SecureMode.md}} has a mention of tomcat, maybe better changed in the
httpfs jira though (since that goes in later). Also 1 reference of tomcat in
{{hadoop-common-project/hadoop-kms/dev-support/findbugsExcludeFile.xml}}, can
be updated here.
+1 pending those 2 nits though.
Since it's a fairly big and impactful patch, will let it float a few days in
case other watchers have comments. Plan to commit by Thursday if no objections.
> Switch KMS from Tomcat to Jetty
> ---
>
> Key: HADOOP-13597
> URL: https://issues.apache.org/jira/browse/HADOOP-13597
> Project: Hadoop Common
> Issue Type: New Feature
> Components: kms
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch,
> HADOOP-13597.003.patch, HADOOP-13597.004.patch, HADOOP-13597.005.patch,
> HADOOP-13597.006.patch
>
>
> The Tomcat 6 we are using will reach EOL at the end of 2017. While there are
> other good options, I would propose switching to {{Jetty 9}} for the
> following reasons:
> * Easier migration. Both Tomcat and Jetty are based on {{Servlet
> Containers}}, so we don't have change client code that much. It would require
> more work to switch to {{JAX-RS}}.
> * Well established.
> * Good performance and scalability.
> Other alternatives:
> * Jersey + Grizzly
> * Tomcat 8
> Your opinions will be greatly appreciated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13934) S3Guard: DynamoDBMetaStore::move could be throwing exception due to BatchWriteItem limits
[
https://issues.apache.org/jira/browse/HADOOP-13934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15797455#comment-15797455
]
Hadoop QA commented on HADOOP-13934:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
38s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
6s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
38s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13934 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12845483/HADOOP-13934-HADOOP-13345.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux fe42e952249f 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HADOOP-13345 / b273171 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11350/testReport/ |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11350/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> S3Guard: DynamoDBMetaStore::move could be throwing exception due to
> BatchWriteItem limits
> -
>
> Key: HADOOP-13934
> URL: https://issues.apache.org/jira/browse/HADOOP-13934
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Rajesh Balamohan
>Assignee: Mingliang Liu
>Priority: Minor
> Attachments: HADO
[jira] [Updated] (HADOOP-13934) S3Guard: DynamoDBMetaStore::move could be throwing exception due to BatchWriteItem limits
[
https://issues.apache.org/jira/browse/HADOOP-13934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-13934:
---
Attachment: HADOOP-13934-HADOOP-13345.003.patch
Thanks [~ste...@apache.org] for the review and offline discussion. Attaching
the v3 patch addressing the comments and unit tests.
Will comment later integration tests against US-West-1 region.
> S3Guard: DynamoDBMetaStore::move could be throwing exception due to
> BatchWriteItem limits
> -
>
> Key: HADOOP-13934
> URL: https://issues.apache.org/jira/browse/HADOOP-13934
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Rajesh Balamohan
>Assignee: Mingliang Liu
>Priority: Minor
> Attachments: HADOOP-13934-HADOOP-13345.000.patch,
> HADOOP-13934-HADOOP-13345.001.patch, HADOOP-13934-HADOOP-13345.002.patch,
> HADOOP-13934-HADOOP-13345.003.patch
>
>
> When using {{DynamoDBMetadataStore}} with a insert heavy hive app , it
> started throwing exceptions in {{DynamoDBMetadataStore::move}}. But just with
> the following exception, it is relatively hard to debug on the real issue in
> DynamoDB side.
> {noformat}
> Caused by: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: 1
> validation error detected: Value
> '{ddb-table-name-334=[com.amazonaws.dynamodb.v20120810.WriteRequest@ca1da583,
> com.amazonaws.dynamodb.v20120810.WriteRequest@ca1fc7cd,
> com.amazonaws.dynamodb.v20120810.WriteRequest@ca4244e6,
> com.amazonaws.dynamodb.v20120810.WriteRequest@ca2f58a9,
> com.amazonaws.dynamodb.v20120810.WriteRequest@ca3525f8,
> ...
> ...
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1529)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1167)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:948)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:661)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:635)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:618)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:586)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:573)
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:445)
> at
> com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:1722)
> at
> com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1698)
> at
> com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.batchWriteItem(AmazonDynamoDBClient.java:668)
> at
> com.amazonaws.services.dynamodbv2.document.internal.BatchWriteItemImpl.doBatchWriteItem(BatchWriteItemImpl.java:111)
> at
> com.amazonaws.services.dynamodbv2.document.internal.BatchWriteItemImpl.batchWriteItem(BatchWriteItemImpl.java:52)
> at
> com.amazonaws.services.dynamodbv2.document.DynamoDB.batchWriteItem(DynamoDB.java:178)
> at
> org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.move(DynamoDBMetadataStore.java:351)
> ... 28 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13897) TestAdlFileContextMainOperationsLive#testGetFileContext1 fails consistently
[
https://issues.apache.org/jira/browse/HADOOP-13897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge resolved HADOOP-13897.
-
Resolution: Duplicate
> TestAdlFileContextMainOperationsLive#testGetFileContext1 fails consistently
> ---
>
> Key: HADOOP-13897
> URL: https://issues.apache.org/jira/browse/HADOOP-13897
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: Tony Wu
>
> {{TestAdlFileContextMainOperationsLive#testGetFileContext1}} (this is a live
> test against Azure Data Lake Store) fails consistently with the following
> error:
> {noformat}
> ---
> T E S T S
> ---
> Running org.apache.hadoop.fs.adl.live.TestAdlFileContextMainOperationsLive
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 11.55 sec <<<
> FAILURE! - in
> org.apache.hadoop.fs.adl.live.TestAdlFileContextMainOperationsLive
> testGetFileContext1(org.apache.hadoop.fs.adl.live.TestAdlFileContextMainOperationsLive)
> Time elapsed: 11.229 sec <<< ERROR!
> java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
> at
> org.apache.hadoop.fs.AbstractFileSystem.newInstance(AbstractFileSystem.java:136)
> at
> org.apache.hadoop.fs.AbstractFileSystem.createFileSystem(AbstractFileSystem.java:165)
> at
> org.apache.hadoop.fs.AbstractFileSystem.get(AbstractFileSystem.java:250)
> at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:331)
> at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:328)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1857)
> at
> org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:328)
> at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:320)
> at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:85)
> at org.apache.hadoop.fs.FileContext.create(FileContext.java:685)
> at
> org.apache.hadoop.fs.FileContextMainOperationsBaseTest.testGetFileContext1(FileContextMainOperationsBaseTest.java:1350)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:254)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:149)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Caused by: java.lang.reflect.InvocationTargetException: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstru
[jira] [Commented] (HADOOP-12733) Remove references to obsolete io.seqfile configuration variables
[ https://issues.apache.org/jira/browse/HADOOP-12733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15797265#comment-15797265 ] Hudson commented on HADOOP-12733: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11070 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11070/]) HADOOP-12733. Remove references to obsolete io.seqfile configuration (aajisaka: rev 01d31fe9389ccdc153d7f4bf6574bf8e509867c1) * (edit) hadoop-tools/hadoop-sls/src/main/data/2jobs2min-rumen-jh.json * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java * (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml > Remove references to obsolete io.seqfile configuration variables > > > Key: HADOOP-12733 > URL: https://issues.apache.org/jira/browse/HADOOP-12733 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Ray Chiang >Assignee: Ray Chiang >Priority: Minor > Labels: supportability > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HADOOP-12733.001.patch, HADOOP-12733.002.patch, > HADOOP-12733.003.patch > > > The following variables appear to no longer be used. > io.seqfile.lazydecompress > io.seqfile.sorter.recordlimit -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12733) Remove references to obsolete io.seqfile configuration variables
[ https://issues.apache.org/jira/browse/HADOOP-12733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-12733: --- Fix Version/s: 3.0.0-alpha2 > Remove references to obsolete io.seqfile configuration variables > > > Key: HADOOP-12733 > URL: https://issues.apache.org/jira/browse/HADOOP-12733 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Ray Chiang >Assignee: Ray Chiang >Priority: Minor > Labels: supportability > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HADOOP-12733.001.patch, HADOOP-12733.002.patch, > HADOOP-12733.003.patch > > > The following variables appear to no longer be used. > io.seqfile.lazydecompress > io.seqfile.sorter.recordlimit -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12733) Remove references to obsolete io.seqfile configuration variables
[ https://issues.apache.org/jira/browse/HADOOP-12733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-12733: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) Committed this to trunk, branch-2, and branch-2.8. Thanks [~rchiang] for the contribution! > Remove references to obsolete io.seqfile configuration variables > > > Key: HADOOP-12733 > URL: https://issues.apache.org/jira/browse/HADOOP-12733 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Ray Chiang >Assignee: Ray Chiang >Priority: Minor > Labels: supportability > Fix For: 2.8.0 > > Attachments: HADOOP-12733.001.patch, HADOOP-12733.002.patch, > HADOOP-12733.003.patch > > > The following variables appear to no longer be used. > io.seqfile.lazydecompress > io.seqfile.sorter.recordlimit -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12733) Remove references to obsolete io.seqfile configuration variables
[ https://issues.apache.org/jira/browse/HADOOP-12733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15797183#comment-15797183 ] Akira Ajisaka commented on HADOOP-12733: +1, checking this in. > Remove references to obsolete io.seqfile configuration variables > > > Key: HADOOP-12733 > URL: https://issues.apache.org/jira/browse/HADOOP-12733 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Ray Chiang >Assignee: Ray Chiang >Priority: Minor > Labels: supportability > Attachments: HADOOP-12733.001.patch, HADOOP-12733.002.patch, > HADOOP-12733.003.patch > > > The following variables appear to no longer be used. > io.seqfile.lazydecompress > io.seqfile.sorter.recordlimit -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13907) Fix KerberosUtil#getDefaultRealm() on Windows
[
https://issues.apache.org/jira/browse/HADOOP-13907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15796943#comment-15796943
]
Xiaoyu Yao commented on HADOOP-13907:
-
Thanks [~ste...@apache.org] for the heads up. The issue was found when I tried
it with 2.8 and trunk(3.0) on Windows. I have not try it on older Hadoop
versions yet.
> Fix KerberosUtil#getDefaultRealm() on Windows
> -
>
> Key: HADOOP-13907
> URL: https://issues.apache.org/jira/browse/HADOOP-13907
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.8.0
>Reporter: Xiaoyu Yao
> Labels: kerberos
>
> Running unit test
> TestWebDelegationToken#testKerberosDelegationTokenAuthenticator on windows
> will fail with {{java.lang.IllegalArgumentException: Can't get Kerberos
> realm}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13907) Fix KerberosUtil#getDefaultRealm() on Windows
[
https://issues.apache.org/jira/browse/HADOOP-13907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HADOOP-13907:
Affects Version/s: 2.8.0
> Fix KerberosUtil#getDefaultRealm() on Windows
> -
>
> Key: HADOOP-13907
> URL: https://issues.apache.org/jira/browse/HADOOP-13907
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.8.0
>Reporter: Xiaoyu Yao
> Labels: kerberos
>
> Running unit test
> TestWebDelegationToken#testKerberosDelegationTokenAuthenticator on windows
> will fail with {{java.lang.IllegalArgumentException: Can't get Kerberos
> realm}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13930) Azure: Add Authorization support to WASB
[
https://issues.apache.org/jira/browse/HADOOP-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15796817#comment-15796817
]
Hadoop QA commented on HADOOP-13930:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HADOOP-13930 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-13930 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12844284/HADOOP-13930.001.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11348/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Azure: Add Authorization support to WASB
>
>
> Key: HADOOP-13930
> URL: https://issues.apache.org/jira/browse/HADOOP-13930
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Dushyanth
>Assignee: Dushyanth
> Attachments: HADOOP-13930.001.patch
>
>
> As highlighted in HADOOP-13863, current implementation of WASB does not
> support authorization to any File System operations. This jira is created to
> add authorization support for WASB. The current approach is to enforce
> authorization via an external REST service (One approach could be to use
> component like Ranger to enforce authorization). The support for
> authorization would be hiding behind a configuration flag :
> "fs.azure.enable.authorization" and the remote service is expected to be
> provided via config : "fs.azure.remote.auth.service.url".
> The remote service is expected to provide support for the following REST
> call: {URL}/CHECK_AUTHORIZATION```
> An example request:
> {URL}/CHECK_AUTHORIZATION?wasb_absolute_path=&operation_type= type>&delegation_token=
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13896) distribution tarball is missing almost all of the hadoop jars
[ https://issues.apache.org/jira/browse/HADOOP-13896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-13896: - Status: Patch Available (was: Open) > distribution tarball is missing almost all of the hadoop jars > - > > Key: HADOOP-13896 > URL: https://issues.apache.org/jira/browse/HADOOP-13896 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.0.0-alpha2 >Reporter: Allen Wittenauer >Assignee: Andrew Wang >Priority: Blocker > Attachments: hadoop-13896.001.patch > > > From what I can tell, all of the hdfs and common jars from their respective > lib dirs are missing, excluding hadoop-hdfs-client and hadoop-hdfs-nfs. But > there are likely more. > Steps to reproduce: > 1. ./start-build-env.sh > 2. mvn install -Pdist,src -Dtar -DskipTests -Pnative > 3. ls hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common > hadoop-nfs-3.0.0-alpha2-SNAPSHOT.jar jdiff lib templates webapps > 4. ls hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs > hadoop-hdfs-nfs-3.0.0-alpha2-SNAPSHOT.jar jdiff lib templates webapps -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13896) distribution tarball is missing almost all of the hadoop jars
[
https://issues.apache.org/jira/browse/HADOOP-13896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HADOOP-13896:
-
Attachment: hadoop-13896.001.patch
A little more progress debugging this, the {{src}} profile is causing the
per-component assembly plugin invocation to run before the jars are created. I
ran a {{dist}} and a {{dist,src}} build, and then grepped as follows: {{egrep
"(maven-assembly-plugin|maven-jar-plugin)" dist.out | grep hadoop-common}}
dist:
{noformat}
[INFO] --- maven-jar-plugin:2.5:jar (prepare-jar) @ hadoop-common ---
[INFO] --- maven-jar-plugin:2.5:test-jar (prepare-test-jar) @ hadoop-common ---
[INFO] --- maven-assembly-plugin:2.4:single (dist) @ hadoop-common ---
[INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ hadoop-common ---
{noformat}
dist,src:
{noformat}
[INFO] --- maven-assembly-plugin:2.4:single (dist) @ hadoop-common ---
[INFO] --- maven-jar-plugin:2.5:jar (prepare-jar) @ hadoop-common ---
[INFO] --- maven-jar-plugin:2.5:test-jar (prepare-test-jar) @ hadoop-common ---
[INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ hadoop-common ---
{noformat}
Here's a patch that just changes the dist assembly plugin in "package" instead
of "prepare-package", which fixes the ordering problem. I did a build and
diffed the jars in alpha1 against trunk with this patch applied, and only found
the changed lib dependencies and the new shaded client and aliyun JARs.
> distribution tarball is missing almost all of the hadoop jars
> -
>
> Key: HADOOP-13896
> URL: https://issues.apache.org/jira/browse/HADOOP-13896
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha2
>Reporter: Allen Wittenauer
>Assignee: Andrew Wang
>Priority: Blocker
> Attachments: hadoop-13896.001.patch
>
>
> From what I can tell, all of the hdfs and common jars from their respective
> lib dirs are missing, excluding hadoop-hdfs-client and hadoop-hdfs-nfs. But
> there are likely more.
> Steps to reproduce:
> 1. ./start-build-env.sh
> 2. mvn install -Pdist,src -Dtar -DskipTests -Pnative
> 3. ls hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common
> hadoop-nfs-3.0.0-alpha2-SNAPSHOT.jar jdiff lib templates webapps
> 4. ls hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs
> hadoop-hdfs-nfs-3.0.0-alpha2-SNAPSHOT.jar jdiff lib templates webapps
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13930) Azure: Add Authorization support to WASB
[
https://issues.apache.org/jira/browse/HADOOP-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-13930:
---
Status: Patch Available (was: Open)
> Azure: Add Authorization support to WASB
>
>
> Key: HADOOP-13930
> URL: https://issues.apache.org/jira/browse/HADOOP-13930
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Dushyanth
>Assignee: Dushyanth
> Attachments: HADOOP-13930.001.patch
>
>
> As highlighted in HADOOP-13863, current implementation of WASB does not
> support authorization to any File System operations. This jira is created to
> add authorization support for WASB. The current approach is to enforce
> authorization via an external REST service (One approach could be to use
> component like Ranger to enforce authorization). The support for
> authorization would be hiding behind a configuration flag :
> "fs.azure.enable.authorization" and the remote service is expected to be
> provided via config : "fs.azure.remote.auth.service.url".
> The remote service is expected to provide support for the following REST
> call: {URL}/CHECK_AUTHORIZATION```
> An example request:
> {URL}/CHECK_AUTHORIZATION?wasb_absolute_path=&operation_type= type>&delegation_token=
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13805) UGI.getCurrentUser() fails if user does not have a keytab associated
[
https://issues.apache.org/jira/browse/HADOOP-13805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15796761#comment-15796761
]
Hadoop QA commented on HADOOP-13805:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
10s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 30s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 9 new + 101 unchanged - 0 fixed = 110 total (was 101) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
27s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13805 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12845453/HADOOP-13805.04.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 7a10fcdf0400 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 8fadd69 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11347/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11347/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11347/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> UGI.getCurrentUser() fails if user does not have a keytab associated
>
>
> Key: HADOOP-13805
> URL: https://issues.apache.org/jira/browse/HADOOP-13805
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>
[jira] [Updated] (HADOOP-12403) Enable multiple writes in flight for HBase WAL writing backed by WASB
[ https://issues.apache.org/jira/browse/HADOOP-12403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Xu updated HADOOP-12403: Status: In Progress (was: Patch Available) > Enable multiple writes in flight for HBase WAL writing backed by WASB > - > > Key: HADOOP-12403 > URL: https://issues.apache.org/jira/browse/HADOOP-12403 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Reporter: Duo Xu >Assignee: Duo Xu > Attachments: HADOOP-12403.01.patch, HADOOP-12403.02.patch, > HADOOP-12403.03.patch > > > Azure HDI HBase clusters use Azure blob storage as file system. We found that > the bottle neck was during writing to write ahead log (WAL). The latest HBase > WAL write model (HBASE-8755) uses multiple AsyncSyncer threads to sync data > to HDFS. However, our WASB driver is still based on a single thread model. > Thus when the sync threads call into WASB layer, every time only one thread > will be allowed to send data to Azure storage.This jira is to introduce a new > write model in WASB layer to allow multiple writes in parallel. > 1. Since We use page blob for WAL, this will cause "holes" in the page blob > as every write starts on a new page. We use the first two bytes of every page > to record the actual data size of the current page. > 2. When reading WAL, we need to know the actual size of the WAL. This should > be the sum of the number represented by the first two bytes of every page. > However looping over every page to get the size will be very slow, > considering normal WAL size is 128MB and each page is 512 bytes. So during > writing, every time a write succeeds, a metadata of the blob called > "total_data_uploaded" will be updated. > 3. Although we allow multiple writes in flight, we need to make sure the sync > threads which call into WASB layers return in order. Reading HBase source > code FSHLog.java, we find that every sync request is associated with a > transaction id. If the sync succeeds, all the transactions prior to this > transaction id are assumed to be in Azure Storage. We use a queue to store > the sync requests and make sure they return to HBase layer in order. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12733) Remove references to obsolete io.seqfile configuration variables
[
https://issues.apache.org/jira/browse/HADOOP-12733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15796636#comment-15796636
]
Hadoop QA commented on HADOOP-12733:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
0s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
6s{color} | {color:green} hadoop-mapreduce-client-hs in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
19s{color} | {color:green} hadoop-sls in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 99m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-12733 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12845443/HADOOP-12733.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml findbugs checkstyle |
| uname | Linux d59203ba7b41 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / ebdd2e0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11346/testReport/ |
| modules | C: hadoop-common-project/hadoop-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
hadoop-tools/hadoop-sls U: .
[jira] [Updated] (HADOOP-13805) UGI.getCurrentUser() fails if user does not have a keytab associated
[
https://issues.apache.org/jira/browse/HADOOP-13805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-13805:
---
Attachment: HADOOP-13805.04.patch
Patch 4 attached.
Tested this by running downstream smokes etc, didn't see any failure. But found
out there's a existing test {{TestUGIWithMiniKdc}} that needs update.
bq. The renewal thread should not be started if there is no keytab, there is
no point to do so because it will not have the credentials (the info in the
keytab) at renewal time.
[~tucu00] please correct me if I'm wrong, the renewal thread is doing {{kinit
-R}} so a TGT would be sufficient, and keytab doesn't need to be renewed or
present for the tgt renewal, right? In any case, I agree with your initial
proposal of having this done in HADOOP-13807 - feels cleaner and more separated
:)
> UGI.getCurrentUser() fails if user does not have a keytab associated
>
>
> Key: HADOOP-13805
> URL: https://issues.apache.org/jira/browse/HADOOP-13805
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.8.0, 2.9.0, 3.0.0-alpha2
>Reporter: Alejandro Abdelnur
>Assignee: Xiao Chen
> Attachments: HADOOP-13805.01.patch, HADOOP-13805.02.patch,
> HADOOP-13805.03.patch, HADOOP-13805.04.patch
>
>
> HADOOP-13558 intention was to avoid UGI from trying to renew the TGT when the
> UGI is created from an existing Subject as in that case the keytab is not
> 'own' by UGI but by the creator of the Subject.
> In HADOOP-13558 we introduced a new private UGI constructor
> {{UserGroupInformation(Subject subject, final boolean externalKeyTab)}} and
> we use with TRUE only when doing a {{UGI.loginUserFromSubject()}}.
> The problem is, when we call {{UGI.getCurrentUser()}}, and UGI was created
> via a Subject (via the {{UGI.loginUserFromSubject()}} method), we call {{new
> UserGroupInformation(subject)}} which will delegate to
> {{UserGroupInformation(Subject subject, final boolean externalKeyTab)}} and
> that will use externalKeyTab == *FALSE*.
> Then the UGI returned by {{UGI.getCurrentUser()}} will attempt to login using
> a non-existing keytab if the TGT expired.
> This problem is experienced in {{KMSClientProvider}} when used by the HDFS
> filesystem client accessing an an encryption zone.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13578) Add Codec for ZStandard Compression
[ https://issues.apache.org/jira/browse/HADOOP-13578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15796609#comment-15796609 ] churro morales commented on HADOOP-13578: - [~jlowe] thank you for taking the time to review this patch. > Add Codec for ZStandard Compression > --- > > Key: HADOOP-13578 > URL: https://issues.apache.org/jira/browse/HADOOP-13578 > Project: Hadoop Common > Issue Type: New Feature >Reporter: churro morales >Assignee: churro morales > Attachments: HADOOP-13578-branch-2.v9.patch, HADOOP-13578.patch, > HADOOP-13578.v1.patch, HADOOP-13578.v2.patch, HADOOP-13578.v3.patch, > HADOOP-13578.v4.patch, HADOOP-13578.v5.patch, HADOOP-13578.v6.patch, > HADOOP-13578.v7.patch, HADOOP-13578.v8.patch, HADOOP-13578.v9.patch > > > ZStandard: https://github.com/facebook/zstd has been used in production for 6 > months by facebook now. v1.0 was recently released. Create a codec for this > library. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13578) Add Codec for ZStandard Compression
[ https://issues.apache.org/jira/browse/HADOOP-13578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15796550#comment-15796550 ] Jason Lowe commented on HADOOP-13578: - Sorry for the long delay, as I just got back from vacation. +1 for the v9 trunk and branch-2 patches. The branch-2 test failures and ASF warnings are unrelated. I'll commit this tomorrow if there are no objections. > Add Codec for ZStandard Compression > --- > > Key: HADOOP-13578 > URL: https://issues.apache.org/jira/browse/HADOOP-13578 > Project: Hadoop Common > Issue Type: New Feature >Reporter: churro morales >Assignee: churro morales > Attachments: HADOOP-13578-branch-2.v9.patch, HADOOP-13578.patch, > HADOOP-13578.v1.patch, HADOOP-13578.v2.patch, HADOOP-13578.v3.patch, > HADOOP-13578.v4.patch, HADOOP-13578.v5.patch, HADOOP-13578.v6.patch, > HADOOP-13578.v7.patch, HADOOP-13578.v8.patch, HADOOP-13578.v9.patch > > > ZStandard: https://github.com/facebook/zstd has been used in production for 6 > months by facebook now. v1.0 was recently released. Create a codec for this > library. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13946) Document how HDFS updates timestamps in the FS spec; compare with object stores
[ https://issues.apache.org/jira/browse/HADOOP-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15796511#comment-15796511 ] Hudson commented on HADOOP-13946: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11067 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11067/]) Revert "HADOOP-13946. Document how HDFS updates timestamps in the FS (liuml07: rev 88731c731a7207edbd5d408a8ec67b2bb63a3eb5) * (edit) hadoop-common-project/hadoop-common/src/site/markdown/filesystem/introduction.md > Document how HDFS updates timestamps in the FS spec; compare with object > stores > --- > > Key: HADOOP-13946 > URL: https://issues.apache.org/jira/browse/HADOOP-13946 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation, fs >Affects Versions: 2.7.3 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HADOOP-13946-001.patch > > > SPARK-17159 shows that the behavior of when HDFS updates timestamps isn't > well documented. Document these in the FS spec. > I'm not going to add tests for this, as it is so very dependent on FS > implementations, as in "POSIX filesystems may behave differently from HDFS". > If someone knows what happens there, their contribution is welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13896) distribution tarball is missing almost all of the hadoop jars
[ https://issues.apache.org/jira/browse/HADOOP-13896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-13896: - Summary: distribution tarball is missing almost all of the hadoop jars (was: disribution tarball is missing almost all of the hadoop jars) > distribution tarball is missing almost all of the hadoop jars > - > > Key: HADOOP-13896 > URL: https://issues.apache.org/jira/browse/HADOOP-13896 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.0.0-alpha2 >Reporter: Allen Wittenauer >Assignee: Andrew Wang >Priority: Blocker > > From what I can tell, all of the hdfs and common jars from their respective > lib dirs are missing, excluding hadoop-hdfs-client and hadoop-hdfs-nfs. But > there are likely more. > Steps to reproduce: > 1. ./start-build-env.sh > 2. mvn install -Pdist,src -Dtar -DskipTests -Pnative > 3. ls hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common > hadoop-nfs-3.0.0-alpha2-SNAPSHOT.jar jdiff lib templates webapps > 4. ls hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs > hadoop-hdfs-nfs-3.0.0-alpha2-SNAPSHOT.jar jdiff lib templates webapps -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13946) Document how HDFS updates timestamps in the FS spec; compare with object stores
[ https://issues.apache.org/jira/browse/HADOOP-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15796440#comment-15796440 ] Mingliang Liu commented on HADOOP-13946: I've reverted for addressing the comments. Thanks, > Document how HDFS updates timestamps in the FS spec; compare with object > stores > --- > > Key: HADOOP-13946 > URL: https://issues.apache.org/jira/browse/HADOOP-13946 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation, fs >Affects Versions: 2.7.3 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HADOOP-13946-001.patch > > > SPARK-17159 shows that the behavior of when HDFS updates timestamps isn't > well documented. Document these in the FS spec. > I'm not going to add tests for this, as it is so very dependent on FS > implementations, as in "POSIX filesystems may behave differently from HDFS". > If someone knows what happens there, their contribution is welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-13946) Document how HDFS updates timestamps in the FS spec; compare with object stores
[ https://issues.apache.org/jira/browse/HADOOP-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu reopened HADOOP-13946: > Document how HDFS updates timestamps in the FS spec; compare with object > stores > --- > > Key: HADOOP-13946 > URL: https://issues.apache.org/jira/browse/HADOOP-13946 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation, fs >Affects Versions: 2.7.3 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HADOOP-13946-001.patch > > > SPARK-17159 shows that the behavior of when HDFS updates timestamps isn't > well documented. Document these in the FS spec. > I'm not going to add tests for this, as it is so very dependent on FS > implementations, as in "POSIX filesystems may behave differently from HDFS". > If someone knows what happens there, their contribution is welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13946) Document how HDFS updates timestamps in the FS spec; compare with object stores
[ https://issues.apache.org/jira/browse/HADOOP-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15796426#comment-15796426 ] Thomas Demoor commented on HADOOP-13946: Another late review remark: bq. The file only becomes visible at the end of the write operation; this also sets the creation time of the file. On S3 an object (file) is only visible at the end of the write operation, but the creation / modification timestamp (Last-Modified HTTP header) is that of the *start* of the write operation. > Document how HDFS updates timestamps in the FS spec; compare with object > stores > --- > > Key: HADOOP-13946 > URL: https://issues.apache.org/jira/browse/HADOOP-13946 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation, fs >Affects Versions: 2.7.3 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HADOOP-13946-001.patch > > > SPARK-17159 shows that the behavior of when HDFS updates timestamps isn't > well documented. Document these in the FS spec. > I'm not going to add tests for this, as it is so very dependent on FS > implementations, as in "POSIX filesystems may behave differently from HDFS". > If someone knows what happens there, their contribution is welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13929) ADLS should not check in contract-test-options.xml
[
https://issues.apache.org/jira/browse/HADOOP-13929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15796388#comment-15796388
]
John Zhuge commented on HADOOP-13929:
-
Another question: can we add another property {{fs.contract.test.enabled}}
(default true to be backwards compatible)?
It is awkward to use {{fs.contract.test.fs.%s}} being empty to decide whether
live/integration tests are enabled because 1) not clear, the property
essentially serves 2 purposes; 2) not easy to disable then re-enable.
> ADLS should not check in contract-test-options.xml
> --
>
> Key: HADOOP-13929
> URL: https://issues.apache.org/jira/browse/HADOOP-13929
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/adl, test
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: John Zhuge
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13929.001.patch, HADOOP-13929.002.patch,
> HADOOP-13929.003.patch, HADOOP-13929.004.patch, HADOOP-13929.005.patch
>
>
> Should not check in the file {{contract-test-options.xml}}. Make sure the
> file is excluded by {{.gitignore}}. Make sure ADLS {{index.md}} provides a
> complete example of this file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12733) Remove references to obsolete io.seqfile configuration variables
[ https://issues.apache.org/jira/browse/HADOOP-12733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated HADOOP-12733: Attachment: HADOOP-12733.003.patch Thanks [~ajisakaa]! Here's an updated patch for trunk. > Remove references to obsolete io.seqfile configuration variables > > > Key: HADOOP-12733 > URL: https://issues.apache.org/jira/browse/HADOOP-12733 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Ray Chiang >Assignee: Ray Chiang >Priority: Minor > Labels: supportability > Attachments: HADOOP-12733.001.patch, HADOOP-12733.002.patch, > HADOOP-12733.003.patch > > > The following variables appear to no longer be used. > io.seqfile.lazydecompress > io.seqfile.sorter.recordlimit -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13922) Some modules have dependencies on hadoop-client jar removed by HADOOP-11804
[ https://issues.apache.org/jira/browse/HADOOP-13922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15796304#comment-15796304 ] Hudson commented on HADOOP-13922: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11066 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11066/]) HADOOP-13922. Some modules have dependencies on hadoop-client jar (cnauroth: rev ebdd2e03b7b8573cc3531958dbfda72cdbc277fd) * (edit) hadoop-client-modules/hadoop-client/pom.xml > Some modules have dependencies on hadoop-client jar removed by HADOOP-11804 > --- > > Key: HADOOP-13922 > URL: https://issues.apache.org/jira/browse/HADOOP-13922 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.0.0-alpha2 >Reporter: Joe Pallas >Assignee: Sean Busbey >Priority: Blocker > Fix For: 3.0.0-alpha2 > > Attachments: HADOOP-13922.1.patch > > > As discussed in [HADOOP-11804 comment > 15758048|https://issues.apache.org/jira/browse/HADOOP-11804?focusedCommentId=15758048&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15758048] > and following comments, there are still dependencies on the now-removed > hadoop-client jar. The current code builds only because an obsolete snapshot > of the jar is found on the repository server. Changing the project version > to something new exposes the problem. > While the build currently dies at hadoop-tools/hadoop-sls, I'm seeing issues > with some Hadoop Client modules, too. > I'm filing a new bug because I can't reopen HADOOP-11804. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13946) Document how HDFS updates timestamps in the FS spec; compare with object stores
[
https://issues.apache.org/jira/browse/HADOOP-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15796303#comment-15796303
]
Chris Nauroth commented on HADOOP-13946:
Sorry to come late to the review, but I would have liked to see a mention of
how HDFS rename updates the modification time of both the source and the
destination folder (though not the modification time of the renamed file
itself).
Also, regarding this:
{quote}
Object stores have a significantly simpler view of time:
...
+ A file's modification time is always the same as its creation time.
{quote}
This makes it sound like this section covers all object stores, but the
statement about modification time is not necessarily true universally. For
example, on WASB, the {{FileStatus}} on read is always populated with the last
modified time field of the blob as reported by the Azure Storage service. I
think any kind of modification of the blob will result in a change in that
value. I specifically tested {{hadoop fs -chmod}} against WASB, and it updated
the blob's modification time, which is different from HDFS. Out-of-band blob
modifications directly through the Azure Storage service, bypassing the
{{FileSystem}} API, could be another source of perceived changes in the last
modification time.
I expect this is not consistent across services, and therefore it's unlikely we
can make accurate statements in the file system spec beyond just saying "it's
different." :-)
Please feel free to address this either by reverting and revising or filing a
new JIRA to track an addendum.
Thanks!
> Document how HDFS updates timestamps in the FS spec; compare with object
> stores
> ---
>
> Key: HADOOP-13946
> URL: https://issues.apache.org/jira/browse/HADOOP-13946
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation, fs
>Affects Versions: 2.7.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HADOOP-13946-001.patch
>
>
> SPARK-17159 shows that the behavior of when HDFS updates timestamps isn't
> well documented. Document these in the FS spec.
> I'm not going to add tests for this, as it is so very dependent on FS
> implementations, as in "POSIX filesystems may behave differently from HDFS".
> If someone knows what happens there, their contribution is welcome.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13946) Document how HDFS updates timestamps in the FS spec; compare with object stores
[ https://issues.apache.org/jira/browse/HADOOP-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15796250#comment-15796250 ] Hudson commented on HADOOP-13946: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11065 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11065/]) HADOOP-13946. Document how HDFS updates timestamps in the FS spec; (liuml07: rev 451efb08fe0680d002c6856c104ebb366acee8a0) * (edit) hadoop-common-project/hadoop-common/src/site/markdown/filesystem/introduction.md > Document how HDFS updates timestamps in the FS spec; compare with object > stores > --- > > Key: HADOOP-13946 > URL: https://issues.apache.org/jira/browse/HADOOP-13946 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation, fs >Affects Versions: 2.7.3 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HADOOP-13946-001.patch > > > SPARK-17159 shows that the behavior of when HDFS updates timestamps isn't > well documented. Document these in the FS spec. > I'm not going to add tests for this, as it is so very dependent on FS > implementations, as in "POSIX filesystems may behave differently from HDFS". > If someone knows what happens there, their contribution is welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13946) Document how HDFS updates timestamps in the FS spec; compare with object stores
[ https://issues.apache.org/jira/browse/HADOOP-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15796233#comment-15796233 ] Mingliang Liu commented on HADOOP-13946: Thanks [~cnauroth] for taking care of this. I didn't mean it but my mouse must have fooled me. > Document how HDFS updates timestamps in the FS spec; compare with object > stores > --- > > Key: HADOOP-13946 > URL: https://issues.apache.org/jira/browse/HADOOP-13946 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation, fs >Affects Versions: 2.7.3 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HADOOP-13946-001.patch > > > SPARK-17159 shows that the behavior of when HDFS updates timestamps isn't > well documented. Document these in the FS spec. > I'm not going to add tests for this, as it is so very dependent on FS > implementations, as in "POSIX filesystems may behave differently from HDFS". > If someone knows what happens there, their contribution is welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13946) Document how HDFS updates timestamps in the FS spec; compare with object stores
[ https://issues.apache.org/jira/browse/HADOOP-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth resolved HADOOP-13946. Resolution: Fixed Reopened just to change resolution from Cannot Reproduce to Fixed. > Document how HDFS updates timestamps in the FS spec; compare with object > stores > --- > > Key: HADOOP-13946 > URL: https://issues.apache.org/jira/browse/HADOOP-13946 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation, fs >Affects Versions: 2.7.3 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HADOOP-13946-001.patch > > > SPARK-17159 shows that the behavior of when HDFS updates timestamps isn't > well documented. Document these in the FS spec. > I'm not going to add tests for this, as it is so very dependent on FS > implementations, as in "POSIX filesystems may behave differently from HDFS". > If someone knows what happens there, their contribution is welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-13946) Document how HDFS updates timestamps in the FS spec; compare with object stores
[ https://issues.apache.org/jira/browse/HADOOP-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth reopened HADOOP-13946: > Document how HDFS updates timestamps in the FS spec; compare with object > stores > --- > > Key: HADOOP-13946 > URL: https://issues.apache.org/jira/browse/HADOOP-13946 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation, fs >Affects Versions: 2.7.3 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HADOOP-13946-001.patch > > > SPARK-17159 shows that the behavior of when HDFS updates timestamps isn't > well documented. Document these in the FS spec. > I'm not going to add tests for this, as it is so very dependent on FS > implementations, as in "POSIX filesystems may behave differently from HDFS". > If someone knows what happens there, their contribution is welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13937) Mock bucket locations in MockS3ClientFactory
[
https://issues.apache.org/jira/browse/HADOOP-13937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-13937:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: HADOOP-13345
Status: Resolved (was: Patch Available)
Committed to feature branch; thanks for your review [~ste...@apache.org].
> Mock bucket locations in MockS3ClientFactory
>
>
> Key: HADOOP-13937
> URL: https://issues.apache.org/jira/browse/HADOOP-13937
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Minor
> Fix For: HADOOP-13345
>
> Attachments: HADOOP-13937-HADOOP-13345.000.patch
>
>
> Currently the MockS3ClientFactory}} does not mock the bucket locations. One
> effect is that {{TestDynamoDBMetadataStore}} will have null region for the
> bucket.
> {code}
> INFO - Creating DynamoDB table TestDynamoDBMetadataStore in region null
> {code}
> The reason is that the DynamoDBMetadataStore will use the same region as the
> S3 bucket.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13922) Some modules have dependencies on hadoop-client jar removed by HADOOP-11804
[
https://issues.apache.org/jira/browse/HADOOP-13922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-13922:
---
Resolution: Fixed
Fix Version/s: 3.0.0-alpha2
Status: Resolved (was: Patch Available)
I have committed this to trunk. [~busbey], thank you for contributing the fix.
[~mblo], I chose to commit this, because it fixes the immediate problem that is
impacting some of us. If you continue to reproduce issues running {{mvn
eclipse:eclipse}}, then please feel free to file a new JIRA issue with the
details. Thank you.
> Some modules have dependencies on hadoop-client jar removed by HADOOP-11804
> ---
>
> Key: HADOOP-13922
> URL: https://issues.apache.org/jira/browse/HADOOP-13922
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha2
>Reporter: Joe Pallas
>Assignee: Sean Busbey
>Priority: Blocker
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13922.1.patch
>
>
> As discussed in [HADOOP-11804 comment
> 15758048|https://issues.apache.org/jira/browse/HADOOP-11804?focusedCommentId=15758048&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15758048]
> and following comments, there are still dependencies on the now-removed
> hadoop-client jar. The current code builds only because an obsolete snapshot
> of the jar is found on the repository server. Changing the project version
> to something new exposes the problem.
> While the build currently dies at hadoop-tools/hadoop-sls, I'm seeing issues
> with some Hadoop Client modules, too.
> I'm filing a new bug because I can't reopen HADOOP-11804.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13946) Document how HDFS updates timestamps in the FS spec; compare with object stores
[
https://issues.apache.org/jira/browse/HADOOP-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-13946:
---
Resolution: Cannot Reproduce
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
2.8.0
Status: Resolved (was: Patch Available)
Committed to {{trunk}} through {{branch-2.8}} branches. Thanks for the
contribution [~ste...@apache.org].
One trivial changes was that I replaced the two "+" with "*" for list chars
following convention in the markdown file.
> Document how HDFS updates timestamps in the FS spec; compare with object
> stores
> ---
>
> Key: HADOOP-13946
> URL: https://issues.apache.org/jira/browse/HADOOP-13946
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation, fs
>Affects Versions: 2.7.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HADOOP-13946-001.patch
>
>
> SPARK-17159 shows that the behavior of when HDFS updates timestamps isn't
> well documented. Document these in the FS spec.
> I'm not going to add tests for this, as it is so very dependent on FS
> implementations, as in "POSIX filesystems may behave differently from HDFS".
> If someone knows what happens there, their contribution is welcome.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13946) Document how HDFS updates timestamps in the FS spec; compare with object stores
[ https://issues.apache.org/jira/browse/HADOOP-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15796188#comment-15796188 ] Mingliang Liu commented on HADOOP-13946: +1 > Document how HDFS updates timestamps in the FS spec; compare with object > stores > --- > > Key: HADOOP-13946 > URL: https://issues.apache.org/jira/browse/HADOOP-13946 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation, fs >Affects Versions: 2.7.3 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-13946-001.patch > > > SPARK-17159 shows that the behavior of when HDFS updates timestamps isn't > well documented. Document these in the FS spec. > I'm not going to add tests for this, as it is so very dependent on FS > implementations, as in "POSIX filesystems may behave differently from HDFS". > If someone knows what happens there, their contribution is welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13896) disribution tarball is missing almost all of the hadoop jars
[
https://issues.apache.org/jira/browse/HADOOP-13896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15796024#comment-15796024
]
Andrew Wang commented on HADOOP-13896:
--
I looked at this a little more, and I think I was able to reproduce the issue
with {{mvn clean install -Pdist,src -Dtar -DskipTests}}. This makes me think
it's something to do with the src profile.
> disribution tarball is missing almost all of the hadoop jars
>
>
> Key: HADOOP-13896
> URL: https://issues.apache.org/jira/browse/HADOOP-13896
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha2
>Reporter: Allen Wittenauer
>Priority: Blocker
>
> From what I can tell, all of the hdfs and common jars from their respective
> lib dirs are missing, excluding hadoop-hdfs-client and hadoop-hdfs-nfs. But
> there are likely more.
> Steps to reproduce:
> 1. ./start-build-env.sh
> 2. mvn install -Pdist,src -Dtar -DskipTests -Pnative
> 3. ls hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common
> hadoop-nfs-3.0.0-alpha2-SNAPSHOT.jar jdiff lib templates webapps
> 4. ls hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs
> hadoop-hdfs-nfs-3.0.0-alpha2-SNAPSHOT.jar jdiff lib templates webapps
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-13896) disribution tarball is missing almost all of the hadoop jars
[ https://issues.apache.org/jira/browse/HADOOP-13896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reassigned HADOOP-13896: Assignee: Andrew Wang > disribution tarball is missing almost all of the hadoop jars > > > Key: HADOOP-13896 > URL: https://issues.apache.org/jira/browse/HADOOP-13896 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.0.0-alpha2 >Reporter: Allen Wittenauer >Assignee: Andrew Wang >Priority: Blocker > > From what I can tell, all of the hdfs and common jars from their respective > lib dirs are missing, excluding hadoop-hdfs-client and hadoop-hdfs-nfs. But > there are likely more. > Steps to reproduce: > 1. ./start-build-env.sh > 2. mvn install -Pdist,src -Dtar -DskipTests -Pnative > 3. ls hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common > hadoop-nfs-3.0.0-alpha2-SNAPSHOT.jar jdiff lib templates webapps > 4. ls hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs > hadoop-hdfs-nfs-3.0.0-alpha2-SNAPSHOT.jar jdiff lib templates webapps -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13945) Azure: Add Kerberos and Delegation token support to WASB client.
[
https://issues.apache.org/jira/browse/HADOOP-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795989#comment-15795989
]
Mingliang Liu commented on HADOOP-13945:
Very early comments are about 1) the logging messages: we should log more
meaningful messages especially in case of exceptions; 2) can we add unit tests
for this? 3) We can make Constants class final, along with a private
constructor 4) getSASKey() static? 5) I'm expecting some checkstyle warnings
(e.g. longer than 80 chars) 6) for selecting the tokens in
{{RemoteSASKeyGeneratorImpl#initialize()}}, can we borrow the code in
{{AbstractDelegationTokenSelector}}? I'm not sure about this.
> Azure: Add Kerberos and Delegation token support to WASB client.
>
>
> Key: HADOOP-13945
> URL: https://issues.apache.org/jira/browse/HADOOP-13945
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-13945.1.patch
>
>
> Current implementation of Azure storage client for Hadoop ({{WASB}}) does not
> support Kerberos Authentication and FileSystem authorization, which makes it
> unusable in secure environments with multi user setup.
> To make {{WASB}} client more suitable to run in Secure environments, there
> are 2 initiatives under way for providing the authorization (HADOOP-13930)
> and fine grained access control (HADOOP-13863) support.
> This JIRA is created to add Kerberos and delegation token support to {{WASB}}
> client to fetch Azure Storage SAS keys (from Remote service as discussed in
> HADOOP-13863), which provides fine grained timed access to containers and
> blobs.
> For delegation token management, the proposal is it use the same REST service
> which being used to generate the SAS Keys.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11804) Shaded Hadoop client artifacts and minicluster
[ https://issues.apache.org/jira/browse/HADOOP-11804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795970#comment-15795970 ] Sean Busbey commented on HADOOP-11804: -- thanks for checking [~andrew.wang], I was pretty sure I had added things to .gitignore but hadn't had a chance to follow up yet. [~ste...@apache.org] mind filing a follow up with the repro steps (ala HADOOP-13922)? If you assign it to me I'd be happy to chase it down. > Shaded Hadoop client artifacts and minicluster > -- > > Key: HADOOP-11804 > URL: https://issues.apache.org/jira/browse/HADOOP-11804 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 3.0.0-alpha2 > > Attachments: HADOOP-11804.1.patch, HADOOP-11804.10.patch, > HADOOP-11804.11.patch, HADOOP-11804.12.patch, HADOOP-11804.13.patch, > HADOOP-11804.14.patch, HADOOP-11804.2.patch, HADOOP-11804.3.patch, > HADOOP-11804.4.patch, HADOOP-11804.5.patch, HADOOP-11804.6.patch, > HADOOP-11804.7.patch, HADOOP-11804.8.patch, HADOOP-11804.9.patch, > hadoop-11804-client-test.tar.gz > > > make a hadoop-client-api and hadoop-client-runtime that i.e. HBase can use to > talk with a Hadoop cluster without seeing any of the implementation > dependencies. > see proposal on parent for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13927) ADLS TestAdlContractRootDirLive.testRmNonEmptyRootDirNonRecursive failed
[
https://issues.apache.org/jira/browse/HADOOP-13927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795923#comment-15795923
]
Tony Wu commented on HADOOP-13927:
--
Hi [~jzhuge], this error started appearing after HADOOP-13900. After reverting
the {{azure-data-lake-store-sdk}} version to {{2.0.4-SNAPSHOT}} the error went
away.
> ADLS TestAdlContractRootDirLive.testRmNonEmptyRootDirNonRecursive failed
>
>
> Key: HADOOP-13927
> URL: https://issues.apache.org/jira/browse/HADOOP-13927
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/adl, test
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Priority: Critical
>
> {noformat}
> Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 18.095 sec
> <<< FAILURE! - in org.apache.hadoop.fs.adl.live.TestAdlContractRootDirLive
> testRmNonEmptyRootDirNonRecursive(org.apache.hadoop.fs.adl.live.TestAdlContractRootDirLive)
> Time elapsed: 1.085 sec <<< FAILURE!
> java.lang.AssertionError: non recursive delete should have raised an
> exception, but completed with exit code false
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.testRmNonEmptyRootDirNonRecursive(AbstractContractRootDirectoryTest.java:132)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13922) Some modules have dependencies on hadoop-client jar removed by HADOOP-11804
[ https://issues.apache.org/jira/browse/HADOOP-13922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795917#comment-15795917 ] Sean Busbey commented on HADOOP-13922: -- [~mblo] please provide more context; the precommit test above shows the eclipse command working. > Some modules have dependencies on hadoop-client jar removed by HADOOP-11804 > --- > > Key: HADOOP-13922 > URL: https://issues.apache.org/jira/browse/HADOOP-13922 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.0.0-alpha2 >Reporter: Joe Pallas >Assignee: Sean Busbey >Priority: Blocker > Attachments: HADOOP-13922.1.patch > > > As discussed in [HADOOP-11804 comment > 15758048|https://issues.apache.org/jira/browse/HADOOP-11804?focusedCommentId=15758048&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15758048] > and following comments, there are still dependencies on the now-removed > hadoop-client jar. The current code builds only because an obsolete snapshot > of the jar is found on the repository server. Changing the project version > to something new exposes the problem. > While the build currently dies at hadoop-tools/hadoop-sls, I'm seeing issues > with some Hadoop Client modules, too. > I'm filing a new bug because I can't reopen HADOOP-11804. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13945) Azure: Add Kerberos and Delegation token support to WASB client.
[
https://issues.apache.org/jira/browse/HADOOP-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795905#comment-15795905
]
Mingliang Liu commented on HADOOP-13945:
The patch does not apply. I used {{patch}} command and it failed as well. Can
you have a look at this? The alternative approach is to use {{git format-patch
trunk}} for generating patches in your working directory.
> Azure: Add Kerberos and Delegation token support to WASB client.
>
>
> Key: HADOOP-13945
> URL: https://issues.apache.org/jira/browse/HADOOP-13945
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-13945.1.patch
>
>
> Current implementation of Azure storage client for Hadoop ({{WASB}}) does not
> support Kerberos Authentication and FileSystem authorization, which makes it
> unusable in secure environments with multi user setup.
> To make {{WASB}} client more suitable to run in Secure environments, there
> are 2 initiatives under way for providing the authorization (HADOOP-13930)
> and fine grained access control (HADOOP-13863) support.
> This JIRA is created to add Kerberos and delegation token support to {{WASB}}
> client to fetch Azure Storage SAS keys (from Remote service as discussed in
> HADOOP-13863), which provides fine grained timed access to containers and
> blobs.
> For delegation token management, the proposal is it use the same REST service
> which being used to generate the SAS Keys.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13945) Azure: Add Kerberos and Delegation token support to WASB client.
[
https://issues.apache.org/jira/browse/HADOOP-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795854#comment-15795854
]
Hadoop QA commented on HADOOP-13945:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HADOOP-13945 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-13945 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12845373/HADOOP-13945.1.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11345/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Azure: Add Kerberos and Delegation token support to WASB client.
>
>
> Key: HADOOP-13945
> URL: https://issues.apache.org/jira/browse/HADOOP-13945
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-13945.1.patch
>
>
> Current implementation of Azure storage client for Hadoop ({{WASB}}) does not
> support Kerberos Authentication and FileSystem authorization, which makes it
> unusable in secure environments with multi user setup.
> To make {{WASB}} client more suitable to run in Secure environments, there
> are 2 initiatives under way for providing the authorization (HADOOP-13930)
> and fine grained access control (HADOOP-13863) support.
> This JIRA is created to add Kerberos and delegation token support to {{WASB}}
> client to fetch Azure Storage SAS keys (from Remote service as discussed in
> HADOOP-13863), which provides fine grained timed access to containers and
> blobs.
> For delegation token management, the proposal is it use the same REST service
> which being used to generate the SAS Keys.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13945) Azure: Add Kerberos and Delegation token support to WASB client.
[
https://issues.apache.org/jira/browse/HADOOP-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-13945:
---
Status: Patch Available (was: Open)
> Azure: Add Kerberos and Delegation token support to WASB client.
>
>
> Key: HADOOP-13945
> URL: https://issues.apache.org/jira/browse/HADOOP-13945
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-13945.1.patch
>
>
> Current implementation of Azure storage client for Hadoop ({{WASB}}) does not
> support Kerberos Authentication and FileSystem authorization, which makes it
> unusable in secure environments with multi user setup.
> To make {{WASB}} client more suitable to run in Secure environments, there
> are 2 initiatives under way for providing the authorization (HADOOP-13930)
> and fine grained access control (HADOOP-13863) support.
> This JIRA is created to add Kerberos and delegation token support to {{WASB}}
> client to fetch Azure Storage SAS keys (from Remote service as discussed in
> HADOOP-13863), which provides fine grained timed access to containers and
> blobs.
> For delegation token management, the proposal is it use the same REST service
> which being used to generate the SAS Keys.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13945) Azure: Add Kerberos and Delegation token support to WASB client.
[
https://issues.apache.org/jira/browse/HADOOP-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795838#comment-15795838
]
Mingliang Liu commented on HADOOP-13945:
[~snayak], I added you to the Hadoop Common contributor list and assigned this
JIRA to you. Thanks for providing a patch.
> Azure: Add Kerberos and Delegation token support to WASB client.
>
>
> Key: HADOOP-13945
> URL: https://issues.apache.org/jira/browse/HADOOP-13945
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-13945.1.patch
>
>
> Current implementation of Azure storage client for Hadoop ({{WASB}}) does not
> support Kerberos Authentication and FileSystem authorization, which makes it
> unusable in secure environments with multi user setup.
> To make {{WASB}} client more suitable to run in Secure environments, there
> are 2 initiatives under way for providing the authorization (HADOOP-13930)
> and fine grained access control (HADOOP-13863) support.
> This JIRA is created to add Kerberos and delegation token support to {{WASB}}
> client to fetch Azure Storage SAS keys (from Remote service as discussed in
> HADOOP-13863), which provides fine grained timed access to containers and
> blobs.
> For delegation token management, the proposal is it use the same REST service
> which being used to generate the SAS Keys.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13945) Azure: Add Kerberos and Delegation token support to WASB client.
[
https://issues.apache.org/jira/browse/HADOOP-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-13945:
---
Assignee: Santhosh G Nayak
> Azure: Add Kerberos and Delegation token support to WASB client.
>
>
> Key: HADOOP-13945
> URL: https://issues.apache.org/jira/browse/HADOOP-13945
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Attachments: HADOOP-13945.1.patch
>
>
> Current implementation of Azure storage client for Hadoop ({{WASB}}) does not
> support Kerberos Authentication and FileSystem authorization, which makes it
> unusable in secure environments with multi user setup.
> To make {{WASB}} client more suitable to run in Secure environments, there
> are 2 initiatives under way for providing the authorization (HADOOP-13930)
> and fine grained access control (HADOOP-13863) support.
> This JIRA is created to add Kerberos and delegation token support to {{WASB}}
> client to fetch Azure Storage SAS keys (from Remote service as discussed in
> HADOOP-13863), which provides fine grained timed access to containers and
> blobs.
> For delegation token management, the proposal is it use the same REST service
> which being used to generate the SAS Keys.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11804) Shaded Hadoop client artifacts and minicluster
[ https://issues.apache.org/jira/browse/HADOOP-11804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795795#comment-15795795 ] Andrew Wang commented on HADOOP-11804: -- This should be covered by the initial commit, I see an entry for dependency-reduced-pom.xml in the top-level .gitignore. I did a full build and "git status" was clean. If this still crops up, could you provide additional repro steps? > Shaded Hadoop client artifacts and minicluster > -- > > Key: HADOOP-11804 > URL: https://issues.apache.org/jira/browse/HADOOP-11804 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 3.0.0-alpha2 > > Attachments: HADOOP-11804.1.patch, HADOOP-11804.10.patch, > HADOOP-11804.11.patch, HADOOP-11804.12.patch, HADOOP-11804.13.patch, > HADOOP-11804.14.patch, HADOOP-11804.2.patch, HADOOP-11804.3.patch, > HADOOP-11804.4.patch, HADOOP-11804.5.patch, HADOOP-11804.6.patch, > HADOOP-11804.7.patch, HADOOP-11804.8.patch, HADOOP-11804.9.patch, > hadoop-11804-client-test.tar.gz > > > make a hadoop-client-api and hadoop-client-runtime that i.e. HBase can use to > talk with a Hadoop cluster without seeing any of the implementation > dependencies. > see proposal on parent for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13947) Add a pre-commit step to check LICENSE/NOTICE issue
[ https://issues.apache.org/jira/browse/HADOOP-13947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795750#comment-15795750 ] Xiao Chen commented on HADOOP-13947: Thanks for the comments! Yea maybe Yetus or both sides, we'll see. :) > Add a pre-commit step to check LICENSE/NOTICE issue > --- > > Key: HADOOP-13947 > URL: https://issues.apache.org/jira/browse/HADOOP-13947 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Xiao Chen >Assignee: Xiao Chen > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12956) Inevitable Log4j2 migration via slf4j
[
https://issues.apache.org/jira/browse/HADOOP-12956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795719#comment-15795719
]
Andrew Wang commented on HADOOP-12956:
--
I'm fine with moving logging APIs over to slf4j in 2.x, as Steve described
above.
2.8 is trying to wrap a release right now, so it'd be safer to target this for
2.9 since this is going to touch a lot of code.
> Inevitable Log4j2 migration via slf4j
> -
>
> Key: HADOOP-12956
> URL: https://issues.apache.org/jira/browse/HADOOP-12956
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Gopal V
>
> {{5 August 2015 --The Apache Logging Services™ Project Management Committee
> (PMC) has announced that the Log4j™ 1.x logging framework has reached its end
> of life (EOL) and is no longer officially supported.}}
> https://blogs.apache.org/foundation/entry/apache_logging_services_project_announces
> A whole framework log4j2 upgrade has to be synchronized, partly for improved
> performance brought about by log4j2.
> https://logging.apache.org/log4j/2.x/manual/async.html#Performance
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13948) Create automated scripts to update LICENSE/NOTICE
[ https://issues.apache.org/jira/browse/HADOOP-13948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-13948: - Target Version/s: 3.0.0-alpha2 (was: 3.0.0-alpha1) > Create automated scripts to update LICENSE/NOTICE > - > > Key: HADOOP-13948 > URL: https://issues.apache.org/jira/browse/HADOOP-13948 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Xiao Chen >Assignee: Xiao Chen > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13947) Add a pre-commit step to check LICENSE/NOTICE issue
[ https://issues.apache.org/jira/browse/HADOOP-13947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795745#comment-15795745 ] Sangjin Lee commented on HADOOP-13947: -- This would be a great idea. Thanks Xiao! > Add a pre-commit step to check LICENSE/NOTICE issue > --- > > Key: HADOOP-13947 > URL: https://issues.apache.org/jira/browse/HADOOP-13947 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Xiao Chen >Assignee: Xiao Chen > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12956) Inevitable Log4j2 migration via slf4j
[
https://issues.apache.org/jira/browse/HADOOP-12956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795711#comment-15795711
]
Mikael Ståldal commented on HADOOP-12956:
-
Unfortunately, we did not manage to finalize the support for Log4j 1
configuration files in Log4j 2.7. There is some experimental support in 2.7,
but not complete or stable yet. We will continue to work on this, but I cannot
promise any particular release or time right now.
However, this should not block you from making progress in Hadoop. If you find
anything blocking you, please raise the issue as a comment here and we will
have a look.
> Inevitable Log4j2 migration via slf4j
> -
>
> Key: HADOOP-12956
> URL: https://issues.apache.org/jira/browse/HADOOP-12956
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Gopal V
>
> {{5 August 2015 --The Apache Logging Services™ Project Management Committee
> (PMC) has announced that the Log4j™ 1.x logging framework has reached its end
> of life (EOL) and is no longer officially supported.}}
> https://blogs.apache.org/foundation/entry/apache_logging_services_project_announces
> A whole framework log4j2 upgrade has to be synchronized, partly for improved
> performance brought about by log4j2.
> https://logging.apache.org/log4j/2.x/manual/async.html#Performance
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13897) TestAdlFileContextMainOperationsLive#testGetFileContext1 fails consistently
[
https://issues.apache.org/jira/browse/HADOOP-13897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HADOOP-13897:
Component/s: (was: fs/azure)
fs/adl
> TestAdlFileContextMainOperationsLive#testGetFileContext1 fails consistently
> ---
>
> Key: HADOOP-13897
> URL: https://issues.apache.org/jira/browse/HADOOP-13897
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: Tony Wu
>
> {{TestAdlFileContextMainOperationsLive#testGetFileContext1}} (this is a live
> test against Azure Data Lake Store) fails consistently with the following
> error:
> {noformat}
> ---
> T E S T S
> ---
> Running org.apache.hadoop.fs.adl.live.TestAdlFileContextMainOperationsLive
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 11.55 sec <<<
> FAILURE! - in
> org.apache.hadoop.fs.adl.live.TestAdlFileContextMainOperationsLive
> testGetFileContext1(org.apache.hadoop.fs.adl.live.TestAdlFileContextMainOperationsLive)
> Time elapsed: 11.229 sec <<< ERROR!
> java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
> at
> org.apache.hadoop.fs.AbstractFileSystem.newInstance(AbstractFileSystem.java:136)
> at
> org.apache.hadoop.fs.AbstractFileSystem.createFileSystem(AbstractFileSystem.java:165)
> at
> org.apache.hadoop.fs.AbstractFileSystem.get(AbstractFileSystem.java:250)
> at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:331)
> at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:328)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1857)
> at
> org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:328)
> at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:320)
> at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:85)
> at org.apache.hadoop.fs.FileContext.create(FileContext.java:685)
> at
> org.apache.hadoop.fs.FileContextMainOperationsBaseTest.testGetFileContext1(FileContextMainOperationsBaseTest.java:1350)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:254)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:149)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Caused by: java.lang.reflect.InvocationTargetException: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>
[jira] [Commented] (HADOOP-13780) LICENSE/NOTICE are out of date for source artifacts
[
https://issues.apache.org/jira/browse/HADOOP-13780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795696#comment-15795696
]
Xiao Chen commented on HADOOP-13780:
Thanks Andrew, filed 2 jiras linked here, for pre-commit and for automation.
Agree on getting rid of the gdoc - we can just use tsvs but for this jira, the
gdoc is the one place to include all of them. Interesting idea about sqlite db,
will play with it. :)
The fully-automatic is possible, but more work needed than current state (what
I do manually now):
- Those raw files (js/css/etc.) needs a doc to get managed, and merged
- Need to add a new entry for the overrides so we can intentionally ignore some
(jdiff, json, ldapsdk as we found out so far)
Once those are done, will need a wiki/instruction page to use it.
> LICENSE/NOTICE are out of date for source artifacts
> ---
>
> Key: HADOOP-13780
> URL: https://issues.apache.org/jira/browse/HADOOP-13780
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.0.0-alpha2
>Reporter: Sean Busbey
>Assignee: Xiao Chen
>Priority: Blocker
> Attachments: HADOOP-13780.01.patch, HADOOP-13780.02.patch,
> HADOOP-13780.03-with-scripts.patch, HADOOP-13780.03.patch
>
>
> we need to perform a check that all of our bundled works are properly
> accounted for in our LICENSE/NOTICE files.
> At a minimum, it looks like HADOOP-10075 introduced some changes that have
> not been accounted for.
> e.g. the jsTree plugin found at
> {{hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/jt/jquery.jstree.js}}
> does not show up in LICENSE.txt to (a) indicate that we're redistributing it
> under the MIT option and (b) give proper citation of the original copyright
> holder per ASF policy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13947) Add a pre-commit step to check LICENSE/NOTICE issue
[ https://issues.apache.org/jira/browse/HADOOP-13947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795656#comment-15795656 ] Wei-Chiu Chuang commented on HADOOP-13947: -- This should probably be a YETUS jira I suppose? > Add a pre-commit step to check LICENSE/NOTICE issue > --- > > Key: HADOOP-13947 > URL: https://issues.apache.org/jira/browse/HADOOP-13947 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Xiao Chen >Assignee: Xiao Chen > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13874) TestSSLHttpServer failures
[
https://issues.apache.org/jira/browse/HADOOP-13874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge resolved HADOOP-13874.
-
Resolution: Cannot Reproduce
Can't reproduce it, even at the same commit.
> TestSSLHttpServer failures
> --
>
> Key: HADOOP-13874
> URL: https://issues.apache.org/jira/browse/HADOOP-13874
> Project: Hadoop Common
> Issue Type: Bug
> Components: security, test
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Critical
>
> All exceptions look like "Cannot support ... with currently installed
> providers". I am running Centos 7.2.1511 and native enabled.
> {noformat}
> Tests run: 5, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 1.593 sec <<<
> FAILURE! - in org.apache.hadoop.http.TestSSLHttpServer
> testExclusiveEnabledCiphers(org.apache.hadoop.http.TestSSLHttpServer) Time
> elapsed: 0.012 sec <<< ERROR!
> java.lang.IllegalArgumentException: Cannot support
> TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA with currently installed providers
> at sun.security.ssl.CipherSuiteList.(CipherSuiteList.java:92)
> at
> sun.security.ssl.SSLSocketImpl.setEnabledCipherSuites(SSLSocketImpl.java:2461)
> at
> org.apache.hadoop.http.TestSSLHttpServer$PrefferedCipherSSLSocketFactory.createSocket(TestSSLHttpServer.java:269)
> at
> sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:436)
> at
> sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1513)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
> at
> sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
> at
> org.apache.hadoop.http.TestSSLHttpServer.testExclusiveEnabledCiphers(TestSSLHttpServer.java:227)
> testOneEnabledCiphers(org.apache.hadoop.http.TestSSLHttpServer) Time
> elapsed: 0.004 sec <<< ERROR!
> java.lang.IllegalArgumentException: Cannot support
> TLS_ECDHE_RSA_WITH_RC4_128_SHA with currently installed providers
> at sun.security.ssl.CipherSuiteList.(CipherSuiteList.java:92)
> at
> sun.security.ssl.SSLSocketImpl.setEnabledCipherSuites(SSLSocketImpl.java:2461)
> at
> org.apache.hadoop.http.TestSSLHttpServer$PrefferedCipherSSLSocketFactory.createSocket(TestSSLHttpServer.java:269)
> at
> sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:436)
> at
> sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1513)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
> at
> sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
> at
> org.apache.hadoop.http.TestSSLHttpServer.testOneEnabledCiphers(TestSSLHttpServer.java:200)
> testExcludedCiphers(org.apache.hadoop.http.TestSSLHttpServer) Time elapsed:
> 0.015 sec <<< ERROR!
> java.lang.IllegalArgumentException: Cannot support
> TLS_ECDHE_RSA_WITH_RC4_128_SHA with currently installed providers
> at sun.security.ssl.CipherSuiteList.(CipherSuiteList.java:92)
> at
> sun.security.ssl.SSLSocketImpl.setEnabledCipherSuites(SSLSocketImpl.java:2461)
> at
> org.apache.hadoop.http.TestSSLHttpServer$PrefferedCipherSSLSocketFactory.createSocket(TestSSLHttpServer.java:269)
> at
> sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:436)
> at
> sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1513)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
> at
> sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
> at
> org.apache.hadoop.http.TestSSLHttpServer.testExcludedCiphers(TestSSLHttpServer.java:176)
> {noformat}
> My source tree sync'd to:
> {noformat}
> 9ef89ed HDFS-11140. Directory Scanner should log startup message time
> correctly. Contributed by Yiqun Lin.
> {noformat}
> My SSL environment:
> {noformat}
> $ curl -sS https://www.howsmyssl.com/a/check | python -m json.tool
> {
> "able_to_detect_n_minus_one_splitting": false,
> "beast_vuln": false,
> "ephemeral_keys_supported": true,
> "given_cipher_suites": [
> "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_
[jira] [Created] (HADOOP-13948) Create automated scripts to update LICENSE/NOTICE
Xiao Chen created HADOOP-13948: -- Summary: Create automated scripts to update LICENSE/NOTICE Key: HADOOP-13948 URL: https://issues.apache.org/jira/browse/HADOOP-13948 Project: Hadoop Common Issue Type: Improvement Components: common Reporter: Xiao Chen Assignee: Xiao Chen -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13947) Add a pre-commit step to check LICENSE/NOTICE issue
Xiao Chen created HADOOP-13947: -- Summary: Add a pre-commit step to check LICENSE/NOTICE issue Key: HADOOP-13947 URL: https://issues.apache.org/jira/browse/HADOOP-13947 Project: Hadoop Common Issue Type: Improvement Components: common Reporter: Xiao Chen Assignee: Xiao Chen -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[
https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795603#comment-15795603
]
Hadoop QA commented on HADOOP-11223:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
10s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 28s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
41s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-11223 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12689236/HADOOP-11223.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 4380b9fd7d69 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / b31e195 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11344/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11344/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11344/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Offer a read-only conf alternative to new Configuration()
> -
>
> Key: HADOOP-11223
> URL: https://issues.apache.org/jira/browse/HADOOP-11223
[jira] [Commented] (HADOOP-13780) LICENSE/NOTICE are out of date for source artifacts
[
https://issues.apache.org/jira/browse/HADOOP-13780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795593#comment-15795593
]
Andrew Wang commented on HADOOP-13780:
--
Thanks for your hard work on this Xiao! +1 LGTM to unblock the release, though
we need a follow-on to improve the scripts. I'm sure these are already on your
todo list, but a few thoughts along those lines:
* We should try to remove the dependency on the externally managed GDoc.
Checking in an exported sqlite DB or some csvs would be an improvement.
* generate.py is still in the spreadsheet
* the manual merge step is unfortunate, ideally everything is fully-generated
by a single script and input data.
Also, did we ever file a JIRA to do a precommit check?
> LICENSE/NOTICE are out of date for source artifacts
> ---
>
> Key: HADOOP-13780
> URL: https://issues.apache.org/jira/browse/HADOOP-13780
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.0.0-alpha2
>Reporter: Sean Busbey
>Assignee: Xiao Chen
>Priority: Blocker
> Attachments: HADOOP-13780.01.patch, HADOOP-13780.02.patch,
> HADOOP-13780.03-with-scripts.patch, HADOOP-13780.03.patch
>
>
> we need to perform a check that all of our bundled works are properly
> accounted for in our LICENSE/NOTICE files.
> At a minimum, it looks like HADOOP-10075 introduced some changes that have
> not been accounted for.
> e.g. the jsTree plugin found at
> {{hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/jt/jquery.jstree.js}}
> does not show up in LICENSE.txt to (a) indicate that we're redistributing it
> under the MIT option and (b) give proper citation of the original copyright
> holder per ASF policy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13345) S3Guard: Improved Consistency for S3A
[ https://issues.apache.org/jira/browse/HADOOP-13345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795579#comment-15795579 ] Steve Loughran commented on HADOOP-13345: - the CLI needs to be able to rebuild its state from the bucket, including detecting & logging any inconsistencies. > S3Guard: Improved Consistency for S3A > - > > Key: HADOOP-13345 > URL: https://issues.apache.org/jira/browse/HADOOP-13345 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HADOOP-13345.prototype1.patch, > S3C-ConsistentListingonS3-Design.pdf, S3GuardImprovedConsistencyforS3A.pdf, > S3GuardImprovedConsistencyforS3AV2.pdf, s3c.001.patch > > > This issue proposes S3Guard, a new feature of S3A, to provide an option for a > stronger consistency model than what is currently offered. The solution > coordinates with a strongly consistent external store to resolve > inconsistencies caused by the S3 eventual consistency model. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13786) add output committer which uses s3guard for consistent commits to S3
[ https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13786: Status: Open (was: Patch Available) > add output committer which uses s3guard for consistent commits to S3 > > > Key: HADOOP-13786 > URL: https://issues.apache.org/jira/browse/HADOOP-13786 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13786-HADOOP-13345-001.patch > > > A goal of this code is "support O(1) commits to S3 repositories in the > presence of failures". Implement it, including whatever is needed to > demonstrate the correctness of the algorithm. (that is, assuming that s3guard > provides a consistent view of the presence/absence of blobs, show that we can > commit directly). > I consider ourselves free to expose the blobstore-ness of the s3 output > streams (ie. not visible until the close()), if we need to use that to allow > us to abort commit operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13934) S3Guard: DynamoDBMetaStore::move could be throwing exception due to BatchWriteItem limits
[
https://issues.apache.org/jira/browse/HADOOP-13934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795546#comment-15795546
]
Steve Loughran commented on HADOOP-13934:
-
test things
* 0 to delete, 1 to put, 25, to put, 26 to put
* 0 to put, 1 to delete, 25 to put, 26 to put
* null as one of the parameters
* null as both of the parameters
> S3Guard: DynamoDBMetaStore::move could be throwing exception due to
> BatchWriteItem limits
> -
>
> Key: HADOOP-13934
> URL: https://issues.apache.org/jira/browse/HADOOP-13934
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Rajesh Balamohan
>Assignee: Mingliang Liu
>Priority: Minor
> Attachments: HADOOP-13934-HADOOP-13345.000.patch,
> HADOOP-13934-HADOOP-13345.001.patch, HADOOP-13934-HADOOP-13345.002.patch
>
>
> When using {{DynamoDBMetadataStore}} with a insert heavy hive app , it
> started throwing exceptions in {{DynamoDBMetadataStore::move}}. But just with
> the following exception, it is relatively hard to debug on the real issue in
> DynamoDB side.
> {noformat}
> Caused by: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: 1
> validation error detected: Value
> '{ddb-table-name-334=[com.amazonaws.dynamodb.v20120810.WriteRequest@ca1da583,
> com.amazonaws.dynamodb.v20120810.WriteRequest@ca1fc7cd,
> com.amazonaws.dynamodb.v20120810.WriteRequest@ca4244e6,
> com.amazonaws.dynamodb.v20120810.WriteRequest@ca2f58a9,
> com.amazonaws.dynamodb.v20120810.WriteRequest@ca3525f8,
> ...
> ...
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1529)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1167)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:948)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:661)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:635)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:618)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:586)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:573)
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:445)
> at
> com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:1722)
> at
> com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1698)
> at
> com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.batchWriteItem(AmazonDynamoDBClient.java:668)
> at
> com.amazonaws.services.dynamodbv2.document.internal.BatchWriteItemImpl.doBatchWriteItem(BatchWriteItemImpl.java:111)
> at
> com.amazonaws.services.dynamodbv2.document.internal.BatchWriteItemImpl.batchWriteItem(BatchWriteItemImpl.java:52)
> at
> com.amazonaws.services.dynamodbv2.document.DynamoDB.batchWriteItem(DynamoDB.java:178)
> at
> org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.move(DynamoDBMetadataStore.java:351)
> ... 28 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13934) S3Guard: DynamoDBMetaStore::move could be throwing exception due to BatchWriteItem limits
[
https://issues.apache.org/jira/browse/HADOOP-13934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795535#comment-15795535
]
Steve Loughran commented on HADOOP-13934:
-
LGTM
# that debug statement on line 349 needs looking at
# I'd like that limit variable to be made a constant, even if just in the class
> S3Guard: DynamoDBMetaStore::move could be throwing exception due to
> BatchWriteItem limits
> -
>
> Key: HADOOP-13934
> URL: https://issues.apache.org/jira/browse/HADOOP-13934
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Rajesh Balamohan
>Assignee: Mingliang Liu
>Priority: Minor
> Attachments: HADOOP-13934-HADOOP-13345.000.patch,
> HADOOP-13934-HADOOP-13345.001.patch, HADOOP-13934-HADOOP-13345.002.patch
>
>
> When using {{DynamoDBMetadataStore}} with a insert heavy hive app , it
> started throwing exceptions in {{DynamoDBMetadataStore::move}}. But just with
> the following exception, it is relatively hard to debug on the real issue in
> DynamoDB side.
> {noformat}
> Caused by: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: 1
> validation error detected: Value
> '{ddb-table-name-334=[com.amazonaws.dynamodb.v20120810.WriteRequest@ca1da583,
> com.amazonaws.dynamodb.v20120810.WriteRequest@ca1fc7cd,
> com.amazonaws.dynamodb.v20120810.WriteRequest@ca4244e6,
> com.amazonaws.dynamodb.v20120810.WriteRequest@ca2f58a9,
> com.amazonaws.dynamodb.v20120810.WriteRequest@ca3525f8,
> ...
> ...
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1529)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1167)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:948)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:661)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:635)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:618)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:586)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:573)
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:445)
> at
> com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:1722)
> at
> com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1698)
> at
> com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.batchWriteItem(AmazonDynamoDBClient.java:668)
> at
> com.amazonaws.services.dynamodbv2.document.internal.BatchWriteItemImpl.doBatchWriteItem(BatchWriteItemImpl.java:111)
> at
> com.amazonaws.services.dynamodbv2.document.internal.BatchWriteItemImpl.batchWriteItem(BatchWriteItemImpl.java:52)
> at
> com.amazonaws.services.dynamodbv2.document.DynamoDB.batchWriteItem(DynamoDB.java:178)
> at
> org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.move(DynamoDBMetadataStore.java:351)
> ... 28 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12956) Inevitable Log4j2 migration via slf4j
[
https://issues.apache.org/jira/browse/HADOOP-12956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795462#comment-15795462
]
Roni Burd commented on HADOOP-12956:
Sounds good. Just to be on the same page: you guys are friendly to the proposal
to bake this into 2.8 in addition to trunk?
> Inevitable Log4j2 migration via slf4j
> -
>
> Key: HADOOP-12956
> URL: https://issues.apache.org/jira/browse/HADOOP-12956
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Gopal V
>
> {{5 August 2015 --The Apache Logging Services™ Project Management Committee
> (PMC) has announced that the Log4j™ 1.x logging framework has reached its end
> of life (EOL) and is no longer officially supported.}}
> https://blogs.apache.org/foundation/entry/apache_logging_services_project_announces
> A whole framework log4j2 upgrade has to be synchronized, partly for improved
> performance brought about by log4j2.
> https://logging.apache.org/log4j/2.x/manual/async.html#Performance
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[
https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795391#comment-15795391
]
Colin P. McCabe commented on HADOOP-11223:
--
If the real problem is reloading all those XML files all the time, why not
change that behavior in Hadoop 3.x? At the very least, we could have some kind
of mapping between classpath and default configuration values, and only
actually load the XML files when we saw a new classpath which might cause us to
load some different files.
> Offer a read-only conf alternative to new Configuration()
> -
>
> Key: HADOOP-11223
> URL: https://issues.apache.org/jira/browse/HADOOP-11223
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
>Reporter: Gopal V
>Assignee: Varun Saxena
> Labels: Performance
> Attachments: HADOOP-11223.001.patch
>
>
> new Configuration() is called from several static blocks across Hadoop.
> This is incredibly inefficient, since each one of those involves primarily
> XML parsing at a point where the JIT won't be triggered & interpreter mode is
> essentially forced on the JVM.
> The alternate solution would be to offer a {{Configuration::getDefault()}}
> alternative which disallows any modifications.
> At the very least, such a method would need to be called from
> # org.apache.hadoop.io.nativeio.NativeIO::()
> # org.apache.hadoop.security.SecurityUtil::()
> # org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider::
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[
https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795365#comment-15795365
]
Wei-Chiu Chuang commented on HADOOP-13597:
--
Thanks [~jzhuge] and [~xiaochen] for driving forward this far. I think the
patch is good to me.
There's one nit in doc CommandsManual.md though:
{quote}
However, the command does not support KMS server, because its web interface is
based on Tomcat, which does not support the servlet.
{quote}
This statement will no longer be valid after this change.
> Switch KMS from Tomcat to Jetty
> ---
>
> Key: HADOOP-13597
> URL: https://issues.apache.org/jira/browse/HADOOP-13597
> Project: Hadoop Common
> Issue Type: New Feature
> Components: kms
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch,
> HADOOP-13597.003.patch, HADOOP-13597.004.patch, HADOOP-13597.005.patch,
> HADOOP-13597.006.patch
>
>
> The Tomcat 6 we are using will reach EOL at the end of 2017. While there are
> other good options, I would propose switching to {{Jetty 9}} for the
> following reasons:
> * Easier migration. Both Tomcat and Jetty are based on {{Servlet
> Containers}}, so we don't have change client code that much. It would require
> more work to switch to {{JAX-RS}}.
> * Well established.
> * Good performance and scalability.
> Other alternatives:
> * Jersey + Grizzly
> * Tomcat 8
> Your opinions will be greatly appreciated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13946) Document how HDFS updates timestamps in the FS spec; compare with object stores
[
https://issues.apache.org/jira/browse/HADOOP-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795295#comment-15795295
]
Hadoop QA commented on HADOOP-13946:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13946 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12845397/HADOOP-13946-001.patch
|
| Optional Tests | asflicense mvnsite |
| uname | Linux 2aa1636e3457 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / b31e195 |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11343/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Document how HDFS updates timestamps in the FS spec; compare with object
> stores
> ---
>
> Key: HADOOP-13946
> URL: https://issues.apache.org/jira/browse/HADOOP-13946
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation, fs
>Affects Versions: 2.7.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13946-001.patch
>
>
> SPARK-17159 shows that the behavior of when HDFS updates timestamps isn't
> well documented. Document these in the FS spec.
> I'm not going to add tests for this, as it is so very dependent on FS
> implementations, as in "POSIX filesystems may behave differently from HDFS".
> If someone knows what happens there, their contribution is welcome.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13946) Document how HDFS updates timestamps in the FS spec; compare with object stores
[ https://issues.apache.org/jira/browse/HADOOP-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13946: Attachment: HADOOP-13946-001.patch Patch 001 > Document how HDFS updates timestamps in the FS spec; compare with object > stores > --- > > Key: HADOOP-13946 > URL: https://issues.apache.org/jira/browse/HADOOP-13946 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation, fs >Affects Versions: 2.7.3 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-13946-001.patch > > > SPARK-17159 shows that the behavior of when HDFS updates timestamps isn't > well documented. Document these in the FS spec. > I'm not going to add tests for this, as it is so very dependent on FS > implementations, as in "POSIX filesystems may behave differently from HDFS". > If someone knows what happens there, their contribution is welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13946) Document how HDFS updates timestamps in the FS spec; compare with object stores
[ https://issues.apache.org/jira/browse/HADOOP-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13946: Status: Patch Available (was: Open) > Document how HDFS updates timestamps in the FS spec; compare with object > stores > --- > > Key: HADOOP-13946 > URL: https://issues.apache.org/jira/browse/HADOOP-13946 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation, fs >Affects Versions: 2.7.3 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-13946-001.patch > > > SPARK-17159 shows that the behavior of when HDFS updates timestamps isn't > well documented. Document these in the FS spec. > I'm not going to add tests for this, as it is so very dependent on FS > implementations, as in "POSIX filesystems may behave differently from HDFS". > If someone knows what happens there, their contribution is welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13937) Mock bucket locations in MockS3ClientFactory
[
https://issues.apache.org/jira/browse/HADOOP-13937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795218#comment-15795218
]
Steve Loughran commented on HADOOP-13937:
-
+1
> Mock bucket locations in MockS3ClientFactory
>
>
> Key: HADOOP-13937
> URL: https://issues.apache.org/jira/browse/HADOOP-13937
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Minor
> Attachments: HADOOP-13937-HADOOP-13345.000.patch
>
>
> Currently the MockS3ClientFactory}} does not mock the bucket locations. One
> effect is that {{TestDynamoDBMetadataStore}} will have null region for the
> bucket.
> {code}
> INFO - Creating DynamoDB table TestDynamoDBMetadataStore in region null
> {code}
> The reason is that the DynamoDBMetadataStore will use the same region as the
> S3 bucket.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13946) Document how HDFS updates timestamps in the FS spec; compare with object stores
Steve Loughran created HADOOP-13946: --- Summary: Document how HDFS updates timestamps in the FS spec; compare with object stores Key: HADOOP-13946 URL: https://issues.apache.org/jira/browse/HADOOP-13946 Project: Hadoop Common Issue Type: Improvement Components: documentation, fs Affects Versions: 2.7.3 Reporter: Steve Loughran Assignee: Steve Loughran Priority: Minor SPARK-17159 shows that the behavior of when HDFS updates timestamps isn't well documented. Document these in the FS spec. I'm not going to add tests for this, as it is so very dependent on FS implementations, as in "POSIX filesystems may behave differently from HDFS". If someone knows what happens there, their contribution is welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13922) Some modules have dependencies on hadoop-client jar removed by HADOOP-11804
[ https://issues.apache.org/jira/browse/HADOOP-13922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795145#comment-15795145 ] Marcel B commented on HADOOP-13922: --- The patch works for me (mvn package), however mvn eclipse:eclipse still fails with same error. I had a clean new env (osx 10.12.2, jdk8). I'll provide more details on request. > Some modules have dependencies on hadoop-client jar removed by HADOOP-11804 > --- > > Key: HADOOP-13922 > URL: https://issues.apache.org/jira/browse/HADOOP-13922 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.0.0-alpha2 >Reporter: Joe Pallas >Assignee: Sean Busbey >Priority: Blocker > Attachments: HADOOP-13922.1.patch > > > As discussed in [HADOOP-11804 comment > 15758048|https://issues.apache.org/jira/browse/HADOOP-11804?focusedCommentId=15758048&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15758048] > and following comments, there are still dependencies on the now-removed > hadoop-client jar. The current code builds only because an obsolete snapshot > of the jar is found on the repository server. Changing the project version > to something new exposes the problem. > While the build currently dies at hadoop-tools/hadoop-sls, I'm seeing issues > with some Hadoop Client modules, too. > I'm filing a new bug because I can't reopen HADOOP-11804. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13874) TestSSLHttpServer failures
[
https://issues.apache.org/jira/browse/HADOOP-13874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795116#comment-15795116
]
Wei-Chiu Chuang commented on HADOOP-13874:
--
Hi [~jzhuge] is this test failure reproducible?
Looking at test code, I am pretty sure it's a test code issue.
> TestSSLHttpServer failures
> --
>
> Key: HADOOP-13874
> URL: https://issues.apache.org/jira/browse/HADOOP-13874
> Project: Hadoop Common
> Issue Type: Bug
> Components: security, test
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Critical
>
> All exceptions look like "Cannot support ... with currently installed
> providers". I am running Centos 7.2.1511 and native enabled.
> {noformat}
> Tests run: 5, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 1.593 sec <<<
> FAILURE! - in org.apache.hadoop.http.TestSSLHttpServer
> testExclusiveEnabledCiphers(org.apache.hadoop.http.TestSSLHttpServer) Time
> elapsed: 0.012 sec <<< ERROR!
> java.lang.IllegalArgumentException: Cannot support
> TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA with currently installed providers
> at sun.security.ssl.CipherSuiteList.(CipherSuiteList.java:92)
> at
> sun.security.ssl.SSLSocketImpl.setEnabledCipherSuites(SSLSocketImpl.java:2461)
> at
> org.apache.hadoop.http.TestSSLHttpServer$PrefferedCipherSSLSocketFactory.createSocket(TestSSLHttpServer.java:269)
> at
> sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:436)
> at
> sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1513)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
> at
> sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
> at
> org.apache.hadoop.http.TestSSLHttpServer.testExclusiveEnabledCiphers(TestSSLHttpServer.java:227)
> testOneEnabledCiphers(org.apache.hadoop.http.TestSSLHttpServer) Time
> elapsed: 0.004 sec <<< ERROR!
> java.lang.IllegalArgumentException: Cannot support
> TLS_ECDHE_RSA_WITH_RC4_128_SHA with currently installed providers
> at sun.security.ssl.CipherSuiteList.(CipherSuiteList.java:92)
> at
> sun.security.ssl.SSLSocketImpl.setEnabledCipherSuites(SSLSocketImpl.java:2461)
> at
> org.apache.hadoop.http.TestSSLHttpServer$PrefferedCipherSSLSocketFactory.createSocket(TestSSLHttpServer.java:269)
> at
> sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:436)
> at
> sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1513)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
> at
> sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
> at
> org.apache.hadoop.http.TestSSLHttpServer.testOneEnabledCiphers(TestSSLHttpServer.java:200)
> testExcludedCiphers(org.apache.hadoop.http.TestSSLHttpServer) Time elapsed:
> 0.015 sec <<< ERROR!
> java.lang.IllegalArgumentException: Cannot support
> TLS_ECDHE_RSA_WITH_RC4_128_SHA with currently installed providers
> at sun.security.ssl.CipherSuiteList.(CipherSuiteList.java:92)
> at
> sun.security.ssl.SSLSocketImpl.setEnabledCipherSuites(SSLSocketImpl.java:2461)
> at
> org.apache.hadoop.http.TestSSLHttpServer$PrefferedCipherSSLSocketFactory.createSocket(TestSSLHttpServer.java:269)
> at
> sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:436)
> at
> sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1513)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
> at
> sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
> at
> org.apache.hadoop.http.TestSSLHttpServer.testExcludedCiphers(TestSSLHttpServer.java:176)
> {noformat}
> My source tree sync'd to:
> {noformat}
> 9ef89ed HDFS-11140. Directory Scanner should log startup message time
> correctly. Contributed by Yiqun Lin.
> {noformat}
> My SSL environment:
> {noformat}
> $ curl -sS https://www.howsmyssl.com/a/check | python -m json.tool
> {
> "able_to_detect_n_minus_one_splitting": false,
> "beast_vuln": false,
> "ephemeral_
[jira] [Updated] (HADOOP-13912) S3a Multipart Committer (avoid rename)
[
https://issues.apache.org/jira/browse/HADOOP-13912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13912:
Issue Type: New Feature (was: Sub-task)
Parent: (was: HADOOP-13204)
> S3a Multipart Committer (avoid rename)
> --
>
> Key: HADOOP-13912
> URL: https://issues.apache.org/jira/browse/HADOOP-13912
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
>Reporter: Thomas Demoor
>Assignee: Thomas Demoor
>
> Object stores do not have an efficient rename operation, which is used by the
> Hadoop FileOutputCommitter to atomically promote the "winning" attempt out of
> the multiple (speculative) attempts to the final path. These slow job commits
> are one of the main friction points when using object stores in Hadoop.There
> have been quite some attempts at resolving this: HADOOP-9565, Apache Spark
> DirectOutputCommitters, ... but they have proven not to be robust in face of
> adversity (network partitions, ...).
> The current ticket proposes to do the atomic commit by using the S3 Multipart
> API, which allows multiple concurrent uploads on the same objectname, each in
> its own "temporary space, identified by the UploadId which is returned as a
> response to InitiateMultipartUpload. Every attempt writes directly to the
> final {{outputPath}}. Data is uploaded using Put Part and as a response an
> ETag for the part is returned and stored. The CompleteMultipartUpload is
> postponed. Instead, we persist the UploadId (using a _temporary subdir or
> elsewhere) and the ETags. When a certain "job" wins
> {{CompleteMultipartUpload}} is called for each of its files using the proper
> list of Part ETags.
> Completing a MultipartUpload is a metadata only operation (internally in S3)
> and is thus orders of magnitude faster than the rename-based approach which
> moves all the data.
> Required work:
> * Expose the multipart initiate and complete calls in S3AOutputStream to
> S3AFilesystem
> * Use these multipart calls in a custom committer as described above. I
> propose to build on the S3ACommitter [~ste...@apache.org] is doing for
> HADOOP-13786
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13181) WASB append support: getPos incorrect
[
https://issues.apache.org/jira/browse/HADOOP-13181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15794933#comment-15794933
]
Hadoop QA commented on HADOOP-13181:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
33s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
34s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 81m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13181 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12805632/HADOOP-13181.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 85c08094ffb5 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 6938b67 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11341/testReport/ |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure U:
. |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11341/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> WASB append support: getPos incorrect
> -
>
> Key: HADOOP-13181
> URL: https://issues.apache.org/jira/browse
[jira] [Updated] (HADOOP-13945) Azure: Add Kerberos and Delegation token support to WASB client.
[
https://issues.apache.org/jira/browse/HADOOP-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Santhosh G Nayak updated HADOOP-13945:
--
Attachment: HADOOP-13945.1.patch
Attaching initial version of the patch containing proposed changes.
> Azure: Add Kerberos and Delegation token support to WASB client.
>
>
> Key: HADOOP-13945
> URL: https://issues.apache.org/jira/browse/HADOOP-13945
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Santhosh G Nayak
> Attachments: HADOOP-13945.1.patch
>
>
> Current implementation of Azure storage client for Hadoop ({{WASB}}) does not
> support Kerberos Authentication and FileSystem authorization, which makes it
> unusable in secure environments with multi user setup.
> To make {{WASB}} client more suitable to run in Secure environments, there
> are 2 initiatives under way for providing the authorization (HADOOP-13930)
> and fine grained access control (HADOOP-13863) support.
> This JIRA is created to add Kerberos and delegation token support to {{WASB}}
> client to fetch Azure Storage SAS keys (from Remote service as discussed in
> HADOOP-13863), which provides fine grained timed access to containers and
> blobs.
> For delegation token management, the proposal is it use the same REST service
> which being used to generate the SAS Keys.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11804) Shaded Hadoop client artifacts and minicluster
[ https://issues.apache.org/jira/browse/HADOOP-11804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15794862#comment-15794862 ] Steve Loughran commented on HADOOP-11804: - is this patch the cause of some dependency-reduced-pom.xml files appearing in my source? If so, they need adding to .gitignore > Shaded Hadoop client artifacts and minicluster > -- > > Key: HADOOP-11804 > URL: https://issues.apache.org/jira/browse/HADOOP-11804 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 3.0.0-alpha2 > > Attachments: HADOOP-11804.1.patch, HADOOP-11804.10.patch, > HADOOP-11804.11.patch, HADOOP-11804.12.patch, HADOOP-11804.13.patch, > HADOOP-11804.14.patch, HADOOP-11804.2.patch, HADOOP-11804.3.patch, > HADOOP-11804.4.patch, HADOOP-11804.5.patch, HADOOP-11804.6.patch, > HADOOP-11804.7.patch, HADOOP-11804.8.patch, HADOOP-11804.9.patch, > hadoop-11804-client-test.tar.gz > > > make a hadoop-client-api and hadoop-client-runtime that i.e. HBase can use to > talk with a Hadoop cluster without seeing any of the implementation > dependencies. > see proposal on parent for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13754) Hadoop-Azure Update WASB URI format to support SAS token in it.
[
https://issues.apache.org/jira/browse/HADOOP-13754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15794812#comment-15794812
]
Steve Loughran commented on HADOOP-13754:
-
h2. General
* This'll need documentation.
* Alongside hitting "submit patch" for yetus/jenkins test runs, can you declare
that you've tested this, including which azure endpoint, as per
[https://wiki.apache.org/hadoop/HowToContribute#Submitting_patches_against_object_stores_such_as_Amazon_S3.2C_OpenStack_Swift_and_Microsoft_Azure]
h2. {{AzureNativeFileSystemStore}}
* What if {{containerSasParts[0]==null}}?
h2. Style.
* Please use lower case method name, e.g {{GetSASFromAuthority}} goes to
{{getSASFromAuthority}};
* two chars for indentation.
h2. Tests
* {{testBadUriFormatWithSasDelimiter}}. If inner cause doesn't match, throw a
new AssertionError with the cause being the caught exception. That way, no data
is lost.
* You can use {{ContractTestUtils}} methods to write/touch files, and assert
that paths exist. That assertion lists the parent directory on a failure, so is
better for debugging.
* Cleanup/close the {{fs}} variable at the end of the test methods in a finally
clause.
* We've had problems with base64 encoding in other stores, problems that are
hard to replicate as they only surface with some logins. Can you add a test for
{{Base64UrlDecodeSAS}} .
> Hadoop-Azure Update WASB URI format to support SAS token in it.
> ---
>
> Key: HADOOP-13754
> URL: https://issues.apache.org/jira/browse/HADOOP-13754
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.7.3, 3.0.0-alpha1, 3.0.0-alpha2
>Reporter: Sumit Dubey
>Assignee: Sumit Dubey
> Fix For: 2.7.3
>
> Attachments: HADOOP-13754-branch-2.7.3.patch
>
> Original Estimate: 3h
> Remaining Estimate: 3h
>
> Currently Azure WASB adapter code supports wasb url in this format
> wasb://[containername@]youraccount.blob.core.windows.net/testDir with the
> credentials retrieved from configuration and scoped to a container.
> With this change we want
> 1) change the url to contain file level sas token in the url
> wasb://[containername[:]]@youraccount.blob.core.windows.net/testDir
> 2) Scope access to a blob/file level.
> 3) Tests to test the new url format
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13754) Hadoop-Azure Update WASB URI format to support SAS token in it.
[ https://issues.apache.org/jira/browse/HADOOP-13754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13754: Assignee: Sumit Dubey > Hadoop-Azure Update WASB URI format to support SAS token in it. > --- > > Key: HADOOP-13754 > URL: https://issues.apache.org/jira/browse/HADOOP-13754 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 2.7.3, 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Sumit Dubey >Assignee: Sumit Dubey > Fix For: 2.7.3 > > Attachments: HADOOP-13754-branch-2.7.3.patch > > Original Estimate: 3h > Remaining Estimate: 3h > > Currently Azure WASB adapter code supports wasb url in this format > wasb://[containername@]youraccount.blob.core.windows.net/testDir with the > credentials retrieved from configuration and scoped to a container. > With this change we want > 1) change the url to contain file level sas token in the url > wasb://[containername[:]]@youraccount.blob.core.windows.net/testDir > 2) Scope access to a blob/file level. > 3) Tests to test the new url format -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13783) Improve efficiency of WASB over page blobs
[ https://issues.apache.org/jira/browse/HADOOP-13783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13783: Component/s: (was: azure) fs/azure > Improve efficiency of WASB over page blobs > -- > > Key: HADOOP-13783 > URL: https://issues.apache.org/jira/browse/HADOOP-13783 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: NITIN VERMA >Assignee: NITIN VERMA > > 1)Add telemetry to WASB driver. WASB driver is lack of any log or > telemetry which makes trouble shoot very difficult. For example, we don’t > know where is high e2e latency between HBase and Azure storage came from when > Azure storage server latency was very low. Also we don’t know why WASB can > only do 166 IOPs which is way below azure storage 500 IOPs. And we had > several incidents before related to storage latency, because of lacking logs, > we couldn’t find the ownership of the incident quickly. > 2)Resolving the hot spotting issue when WASB driver partition the azure > page blobs by changing the key. Current key design is causing the hot > spotting on azure storage. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13754) Hadoop-Azure Update WASB URI format to support SAS token in it.
[ https://issues.apache.org/jira/browse/HADOOP-13754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13754: Component/s: (was: azure) fs/azure > Hadoop-Azure Update WASB URI format to support SAS token in it. > --- > > Key: HADOOP-13754 > URL: https://issues.apache.org/jira/browse/HADOOP-13754 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 2.7.3, 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Sumit Dubey > Fix For: 2.7.3 > > Attachments: HADOOP-13754-branch-2.7.3.patch > > Original Estimate: 3h > Remaining Estimate: 3h > > Currently Azure WASB adapter code supports wasb url in this format > wasb://[containername@]youraccount.blob.core.windows.net/testDir with the > credentials retrieved from configuration and scoped to a container. > With this change we want > 1) change the url to contain file level sas token in the url > wasb://[containername[:]]@youraccount.blob.core.windows.net/testDir > 2) Scope access to a blob/file level. > 3) Tests to test the new url format -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13475) Adding Append Blob support for WASB
[ https://issues.apache.org/jira/browse/HADOOP-13475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13475: Component/s: (was: azure) fs/azure > Adding Append Blob support for WASB > --- > > Key: HADOOP-13475 > URL: https://issues.apache.org/jira/browse/HADOOP-13475 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/azure >Affects Versions: 2.7.1 >Reporter: Raul da Silva Martins >Assignee: Raul da Silva Martins > Attachments: 0001-Added-Support-for-Azure-AppendBlobs.patch, > HADOOP-13475.001.patch, HADOOP-13475.002.patch > > > Currently the WASB implementation of the HDFS interface does not support the > utilization of Azure AppendBlobs underneath. As owners of a large scale > service who intend to start writing to Append blobs, we need this support in > order to be able to keep using our HDI capabilities. > This JIRA is added to implement Azure AppendBlob support to WASB. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13181) WASB append support: getPos incorrect
[
https://issues.apache.org/jira/browse/HADOOP-13181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15794786#comment-15794786
]
Steve Loughran commented on HADOOP-13181:
-
Submitting the patch, but I'd like to see a revision where the {{/* expected
*/}} inline comments are cut. We don't like inline comments, and people writing
tests (or at least reviewers) are expected to know the ordering of
{{assertEquals()}} clauses.
As with all object store patches, we'll need a declaration that [you've
manually run the azure
tests|https://wiki.apache.org/hadoop/HowToContribute#Submitting_patches_against_object_stores_such_as_Amazon_S3.2C_OpenStack_Swift_and_Microsoft_Azure].
Ideally, as the contract tests are being updated, the other stores too, though
as they don't yet support append, it's moot here. Do run the hadoop-hdfs tests
though, as jenkins won't test that module on a patch to hadoop-common
> WASB append support: getPos incorrect
> -
>
> Key: HADOOP-13181
> URL: https://issues.apache.org/jira/browse/HADOOP-13181
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Bogdan Raducanu
> Attachments: HADOOP-13181.001.patch, append.java
>
>
> See attached code.
> Cause:
> In NativeAzureFileSystem.java: the append method returns
> {code}
> new FSDataOutputStream(appendStream, statistics)
> {code}
> Instead, it should probably return
> {code}
> new FSDataOutputStream(appendStream, statistics, meta.getLength())
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12403) Enable multiple writes in flight for HBase WAL writing backed by WASB
[
https://issues.apache.org/jira/browse/HADOOP-12403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15794782#comment-15794782
]
Hadoop QA commented on HADOOP-12403:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HADOOP-12403 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-12403 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12755309/HADOOP-12403.03.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11342/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Enable multiple writes in flight for HBase WAL writing backed by WASB
> -
>
> Key: HADOOP-12403
> URL: https://issues.apache.org/jira/browse/HADOOP-12403
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Reporter: Duo Xu
>Assignee: Duo Xu
> Attachments: HADOOP-12403.01.patch, HADOOP-12403.02.patch,
> HADOOP-12403.03.patch
>
>
> Azure HDI HBase clusters use Azure blob storage as file system. We found that
> the bottle neck was during writing to write ahead log (WAL). The latest HBase
> WAL write model (HBASE-8755) uses multiple AsyncSyncer threads to sync data
> to HDFS. However, our WASB driver is still based on a single thread model.
> Thus when the sync threads call into WASB layer, every time only one thread
> will be allowed to send data to Azure storage.This jira is to introduce a new
> write model in WASB layer to allow multiple writes in parallel.
> 1. Since We use page blob for WAL, this will cause "holes" in the page blob
> as every write starts on a new page. We use the first two bytes of every page
> to record the actual data size of the current page.
> 2. When reading WAL, we need to know the actual size of the WAL. This should
> be the sum of the number represented by the first two bytes of every page.
> However looping over every page to get the size will be very slow,
> considering normal WAL size is 128MB and each page is 512 bytes. So during
> writing, every time a write succeeds, a metadata of the blob called
> "total_data_uploaded" will be updated.
> 3. Although we allow multiple writes in flight, we need to make sure the sync
> threads which call into WASB layers return in order. Reading HBase source
> code FSHLog.java, we find that every sync request is associated with a
> transaction id. If the sync succeeds, all the transactions prior to this
> transaction id are assumed to be in Azure Storage. We use a queue to store
> the sync requests and make sure they return to HBase layer in order.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13181) WASB append support: getPos incorrect
[
https://issues.apache.org/jira/browse/HADOOP-13181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13181:
Status: Patch Available (was: Open)
> WASB append support: getPos incorrect
> -
>
> Key: HADOOP-13181
> URL: https://issues.apache.org/jira/browse/HADOOP-13181
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Bogdan Raducanu
> Attachments: HADOOP-13181.001.patch, append.java
>
>
> See attached code.
> Cause:
> In NativeAzureFileSystem.java: the append method returns
> {code}
> new FSDataOutputStream(appendStream, statistics)
> {code}
> Instead, it should probably return
> {code}
> new FSDataOutputStream(appendStream, statistics, meta.getLength())
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13475) Adding Append Blob support for WASB
[ https://issues.apache.org/jira/browse/HADOOP-13475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15794777#comment-15794777 ] Steve Loughran commented on HADOOP-13475: - does this duplicate HADOOP-13195 ? if so, as this is the one with a code patch, the other should be closed as a duplicate of this one > Adding Append Blob support for WASB > --- > > Key: HADOOP-13475 > URL: https://issues.apache.org/jira/browse/HADOOP-13475 > Project: Hadoop Common > Issue Type: New Feature > Components: azure >Affects Versions: 2.7.1 >Reporter: Raul da Silva Martins >Assignee: Raul da Silva Martins > Attachments: 0001-Added-Support-for-Azure-AppendBlobs.patch, > HADOOP-13475.001.patch, HADOOP-13475.002.patch > > > Currently the WASB implementation of the HDFS interface does not support the > utilization of Azure AppendBlobs underneath. As owners of a large scale > service who intend to start writing to Append blobs, we need this support in > order to be able to keep using our HDI capabilities. > This JIRA is added to implement Azure AppendBlob support to WASB. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13195) hadoop-azure: page blob append support
[
https://issues.apache.org/jira/browse/HADOOP-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13195:
Component/s: (was: azure)
fs/azure
> hadoop-azure: page blob append support
> --
>
> Key: HADOOP-13195
> URL: https://issues.apache.org/jira/browse/HADOOP-13195
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Reporter: Bogdan Raducanu
>
> The use case for this is storing transaction logs, which, unlike HBase logs,
> need to be appended, instead of created every time.
> Currently, hadoop-azure has append support but only for Block Blobs, which
> are not suited for transaction logging.
> After a quick look, I think the existing {{PageBlobOutputStream}} can be
> easily adapted to support appends. It already contains logic to re-upload the
> last page if it's not full.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13181) WASB append support: getPos incorrect
[
https://issues.apache.org/jira/browse/HADOOP-13181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13181:
Component/s: (was: azure)
fs/azure
> WASB append support: getPos incorrect
> -
>
> Key: HADOOP-13181
> URL: https://issues.apache.org/jira/browse/HADOOP-13181
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Bogdan Raducanu
> Attachments: HADOOP-13181.001.patch, append.java
>
>
> See attached code.
> Cause:
> In NativeAzureFileSystem.java: the append method returns
> {code}
> new FSDataOutputStream(appendStream, statistics)
> {code}
> Instead, it should probably return
> {code}
> new FSDataOutputStream(appendStream, statistics, meta.getLength())
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13134) WASB's file delete still throwing Blob not found exception
[ https://issues.apache.org/jira/browse/HADOOP-13134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15794772#comment-15794772 ] Steve Loughran commented on HADOOP-13134: - The implication is that there is no parent folder here...is that an issue? Should a parent folder be created? > WASB's file delete still throwing Blob not found exception > -- > > Key: HADOOP-13134 > URL: https://issues.apache.org/jira/browse/HADOOP-13134 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 2.7.1 >Reporter: Lin Chan >Assignee: Dushyanth > > WASB is still throwing blob not found exception as shown in the following > stack. Need to catch that and convert to Boolean return code in WASB delete. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13134) WASB's file delete still throwing Blob not found exception
[ https://issues.apache.org/jira/browse/HADOOP-13134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13134: Target Version/s: (was: 2.7.1) Description: WASB is still throwing blob not found exception as shown in the following stack. Need to catch that and convert to Boolean return code in WASB delete. was: WASB is still throwing blob not found exception as shown in the following stack. Need to catch that and convert to Boolean return code in WASB delete. 16/05/07 01:24:57 ERROR InsertIntoHadoopFsRelation: Aborting job. org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.storage.StorageException: The specified blob does not exist. at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2682) at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2693) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.updateParentFolderLastModifiedTime(NativeAzureFileSystem.java:2495) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1860) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1603) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1836) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1603) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1836) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1603) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1836) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1603) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1836) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1603) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1836) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1603) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1836) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1603) at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.cleanupJob(FileOutputCommitter.java:510) at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJobInternal(FileOutputCommitter.java:403) at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:364) at org.apache.parquet.hadoop.ParquetOutputCommitter.commitJob(ParquetOutputCommitter.java:46) at org.apache.spark.sql.execution.datasources.BaseWriterContainer.commitJob(WriterContainer.scala:230) at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply$mcV$sp(InsertIntoHadoopFsRelation.scala:151) at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:108) at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:108) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:56) at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation.run(InsertIntoHadoopFsRelation.scala:108) Component/s: (was: azure) fs/azure > WASB's file delete still throwing Blob not found exception > -- > > Key: HADOOP-13134 > URL: https://issues.apache.org/jira/browse/HADOOP-13134 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 2.7.1 >Reporter: Lin Chan >Assignee: Dushyanth > > WASB is still throwing blob not found exception as shown in the following > stack. Need to catch that and convert to Boolean return code in WASB delete. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13134) WASB's file delete still throwing Blob not found exception
[
https://issues.apache.org/jira/browse/HADOOP-13134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15794767#comment-15794767
]
Steve Loughran commented on HADOOP-13134:
-
{code}
16/05/07 01:24:57 ERROR InsertIntoHadoopFsRelation: Aborting job.
org.apache.hadoop.fs.azure.AzureException:
com.microsoft.azure.storage.StorageException: The specified blob does not exist.
at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2682)
at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2693)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.updateParentFolderLastModifiedTime(NativeAzureFileSystem.java:2495)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1860)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1603)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1836)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1603)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1836)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1603)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1836)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1603)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1836)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1603)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1836)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1603)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1836)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1603)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.cleanupJob(FileOutputCommitter.java:510)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJobInternal(FileOutputCommitter.java:403)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:364)
at
org.apache.parquet.hadoop.ParquetOutputCommitter.commitJob(ParquetOutputCommitter.java:46)
at
org.apache.spark.sql.execution.datasources.BaseWriterContainer.commitJob(WriterContainer.scala:230)
at
org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply$mcV$sp(InsertIntoHadoopFsRelation.scala:151)
at
org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:108)
at
org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:108)
at
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:56)
at
org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation.run(InsertIntoHadoopFsRelation.scala:108)
{code}
> WASB's file delete still throwing Blob not found exception
> --
>
> Key: HADOOP-13134
> URL: https://issues.apache.org/jira/browse/HADOOP-13134
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.7.1
>Reporter: Lin Chan
>Assignee: Dushyanth
>
> WASB is still throwing blob not found exception as shown in the following
> stack. Need to catch that and convert to Boolean return code in WASB delete.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12403) Enable multiple writes in flight for HBase WAL writing backed by WASB
[ https://issues.apache.org/jira/browse/HADOOP-12403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-12403: Component/s: (was: azure) fs/azure > Enable multiple writes in flight for HBase WAL writing backed by WASB > - > > Key: HADOOP-12403 > URL: https://issues.apache.org/jira/browse/HADOOP-12403 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Reporter: Duo Xu >Assignee: Duo Xu > Attachments: HADOOP-12403.01.patch, HADOOP-12403.02.patch, > HADOOP-12403.03.patch > > > Azure HDI HBase clusters use Azure blob storage as file system. We found that > the bottle neck was during writing to write ahead log (WAL). The latest HBase > WAL write model (HBASE-8755) uses multiple AsyncSyncer threads to sync data > to HDFS. However, our WASB driver is still based on a single thread model. > Thus when the sync threads call into WASB layer, every time only one thread > will be allowed to send data to Azure storage.This jira is to introduce a new > write model in WASB layer to allow multiple writes in parallel. > 1. Since We use page blob for WAL, this will cause "holes" in the page blob > as every write starts on a new page. We use the first two bytes of every page > to record the actual data size of the current page. > 2. When reading WAL, we need to know the actual size of the WAL. This should > be the sum of the number represented by the first two bytes of every page. > However looping over every page to get the size will be very slow, > considering normal WAL size is 128MB and each page is 512 bytes. So during > writing, every time a write succeeds, a metadata of the blob called > "total_data_uploaded" will be updated. > 3. Although we allow multiple writes in flight, we need to make sure the sync > threads which call into WASB layers return in order. Reading HBase source > code FSHLog.java, we find that every sync request is associated with a > transaction id. If the sync succeeds, all the transactions prior to this > transaction id are assumed to be in Azure Storage. We use a queue to store > the sync requests and make sure they return to HBase layer in order. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12836) Change hadoop-azure to support full semantics required for the root directory.
[
https://issues.apache.org/jira/browse/HADOOP-12836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-12836:
Component/s: (was: azure)
fs/azure
> Change hadoop-azure to support full semantics required for the root directory.
> --
>
> Key: HADOOP-12836
> URL: https://issues.apache.org/jira/browse/HADOOP-12836
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure, test
>Reporter: Chris Nauroth
>
> HADOOP-12535 introduced usage of the file system contract tests in
> hadoop-azure. During development of that patch, an attempt was made to
> execute the root directory tests defined in
> {{AbstractContractRootDirectoryTest}}, but some of the tests failed. This
> issue tracks the work of reattempting to add those tests and implementing any
> fixes required in hadoop-azure to make those tests pass.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13945) Azure: Add Kerberos and Delegation token support to WASB client.
[
https://issues.apache.org/jira/browse/HADOOP-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13945:
Component/s: (was: azure)
> Azure: Add Kerberos and Delegation token support to WASB client.
>
>
> Key: HADOOP-13945
> URL: https://issues.apache.org/jira/browse/HADOOP-13945
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Santhosh G Nayak
>
> Current implementation of Azure storage client for Hadoop ({{WASB}}) does not
> support Kerberos Authentication and FileSystem authorization, which makes it
> unusable in secure environments with multi user setup.
> To make {{WASB}} client more suitable to run in Secure environments, there
> are 2 initiatives under way for providing the authorization (HADOOP-13930)
> and fine grained access control (HADOOP-13863) support.
> This JIRA is created to add Kerberos and delegation token support to {{WASB}}
> client to fetch Azure Storage SAS keys (from Remote service as discussed in
> HADOOP-13863), which provides fine grained timed access to containers and
> blobs.
> For delegation token management, the proposal is it use the same REST service
> which being used to generate the SAS Keys.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13945) Azure: Add Kerberos and Delegation token support to WASB client.
Santhosh G Nayak created HADOOP-13945:
-
Summary: Azure: Add Kerberos and Delegation token support to WASB
client.
Key: HADOOP-13945
URL: https://issues.apache.org/jira/browse/HADOOP-13945
Project: Hadoop Common
Issue Type: Improvement
Components: azure, fs/azure
Affects Versions: 2.8.0
Reporter: Santhosh G Nayak
Current implementation of Azure storage client for Hadoop ({{WASB}}) does not
support Kerberos Authentication and FileSystem authorization, which makes it
unusable in secure environments with multi user setup.
To make {{WASB}} client more suitable to run in Secure environments, there are
2 initiatives under way for providing the authorization (HADOOP-13930) and fine
grained access control (HADOOP-13863) support.
This JIRA is created to add Kerberos and delegation token support to {{WASB}}
client to fetch Azure Storage SAS keys (from Remote service as discussed in
HADOOP-13863), which provides fine grained timed access to containers and
blobs.
For delegation token management, the proposal is it use the same REST service
which being used to generate the SAS Keys.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
