[GitHub] elek commented on issue #463: HDDS-905. Create informative landing page for Ozone S3 gateway
elek commented on issue #463: HDDS-905. Create informative landing page for Ozone S3 gateway URL: https://github.com/apache/hadoop/pull/463#issuecomment-453444347 rerun tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] elek opened a new pull request #464: HDDS-977. Exclude dependency-reduced-pom.xml from ozone rat check.
elek opened a new pull request #464: HDDS-977. Exclude dependency-reduced-pom.xml from ozone rat check. URL: https://github.com/apache/hadoop/pull/464 See: https://issues.apache.org/jira/browse/HDDS-977 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15994) Upgrade Jackson2 to 2.9.8
[ https://issues.apache.org/jira/browse/HADOOP-15994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740284#comment-16740284 ] lqjacklee commented on HADOOP-15994: [~ajisakaa] sorry to update the status. > Upgrade Jackson2 to 2.9.8 > - > > Key: HADOOP-15994 > URL: https://issues.apache.org/jira/browse/HADOOP-15994 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Akira Ajisaka >Assignee: lqjacklee >Priority: Major > Attachments: 460.patch, HADOOP-15994-001.patch, > HADOOP-15994-002.patch, HADOOP-15994-003.patch > > > Now Jackson 2.9.5 is used and it is vulnerable (CVE-2018-11307). Let's > upgrade to the latest version. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15994) Upgrade Jackson2 to 2.9.8
[ https://issues.apache.org/jira/browse/HADOOP-15994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lqjacklee updated HADOOP-15994: --- Status: Patch Available (was: Reopened) > Upgrade Jackson2 to 2.9.8 > - > > Key: HADOOP-15994 > URL: https://issues.apache.org/jira/browse/HADOOP-15994 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Akira Ajisaka >Assignee: lqjacklee >Priority: Major > Attachments: 460.patch, HADOOP-15994-001.patch, > HADOOP-15994-002.patch, HADOOP-15994-003.patch > > > Now Jackson 2.9.5 is used and it is vulnerable (CVE-2018-11307). Let's > upgrade to the latest version. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] elek commented on issue #464: HDDS-977. Exclude dependency-reduced-pom.xml from ozone rat check.
elek commented on issue #464: HDDS-977. Exclude dependency-reduced-pom.xml from ozone rat check. URL: https://github.com/apache/hadoop/pull/464#issuecomment-453484700 rerun tests please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15975) ABFS: remove timeout check for DELETE and RENAME
[ https://issues.apache.org/jira/browse/HADOOP-15975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15975: Resolution: Fixed Fix Version/s: 3.2.1 Status: Resolved (was: Patch Available) +1 couldn't see any way to test this at all, other than having a mock store simulate an operation taking so long the timeouts kicked in, and then verify that with this patch that timeout didn't happen. But that would be a fairly brittle test, i.e. it wouldn't necessarily catch regressions, maybe even have false positives. Taking it as is. > ABFS: remove timeout check for DELETE and RENAME > > > Key: HADOOP-15975 > URL: https://issues.apache.org/jira/browse/HADOOP-15975 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.1 > > Attachments: HADOOP-15975-001.patch, HADOOP-15975-002.patch, > HADOOP-15975-003.patch > > > Currently, ABFS rename and delete is doing a timeout check, which will fail > the request for rename/delete when the target contains tons of file/dirs. > Because timeout check is already there for each HTTP call, we should remove > the timeout check in RENAME and DELETE. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15662) ABFS: Better exception handling of DNS errors
[ https://issues.apache.org/jira/browse/HADOOP-15662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740292#comment-16740292 ] Steve Loughran commented on HADOOP-15662: - Thomas, Da Zhou I've already got the 01 patch in; the second iteration should really be done as a followup . > ABFS: Better exception handling of DNS errors > - > > Key: HADOOP-15662 > URL: https://issues.apache.org/jira/browse/HADOOP-15662 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.1 > > Attachments: HADOOP-15662-001.patch, HADOOP-15662-002.patch > > > DNS errors are common during testing due to typos or misconfiguration. They > can also occur in production, as some transient DNS issues occur from time to > time. > 1) Let's investigate if we can distinguish between the two and fail fast for > the test issues, but continue to have retry logic for the transient DNS > issues in production. > 2) Let's improve the error handling of DNS failures, so the user has an > actionable error message. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15975) ABFS: remove timeout check for DELETE and RENAME
[ https://issues.apache.org/jira/browse/HADOOP-15975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740301#comment-16740301 ] Hudson commented on HADOOP-15975: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15761 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15761/]) HADOOP-15975. ABFS: remove timeout check for DELETE and RENAME. (stevel: rev dddad985d7ff54448c871515c1e8d1773dfbc3df) * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java > ABFS: remove timeout check for DELETE and RENAME > > > Key: HADOOP-15975 > URL: https://issues.apache.org/jira/browse/HADOOP-15975 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.1 > > Attachments: HADOOP-15975-001.patch, HADOOP-15975-002.patch, > HADOOP-15975-003.patch > > > Currently, ABFS rename and delete is doing a timeout check, which will fail > the request for rename/delete when the target contains tons of file/dirs. > Because timeout check is already there for each HTTP call, we should remove > the timeout check in RENAME and DELETE. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16029) Consecutive Append Should Reuse
[
https://issues.apache.org/jira/browse/HADOOP-16029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740382#comment-16740382
]
Ayush Saxena commented on HADOOP-16029:
---
[~giovanni.fumarola]
Checkstyle warning is existing one only.Attaching from trunk report.As Decided
will fix separately.
>From trunk build ::
{noformat}
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/GraphiteSink.java:79:
'if' child has incorrect indentation level 16, expected level should be 8.
[Indentation]
{noformat}
Pls Review :)
> Consecutive Append Should Reuse
> ---
>
> Key: HADOOP-16029
> URL: https://issues.apache.org/jira/browse/HADOOP-16029
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HADOOP-16029-01.patch, HADOOP-16029-02.patch,
> HADOOP-16029-03.patch, HADOOP-16029-04.patch, HADOOP-16029-05.patch
>
>
> Consecutive calls to StringBuffer/StringBuilder .append should be chained,
> reusing the target object. This can improve the performance by producing a
> smaller bytecode, reducing overhead and improving inlining.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16043) NPE in ITestDynamoDBMetadataStore when fs.s3a.s3guard.ddb.table is not set
Adam Antal created HADOOP-16043:
---
Summary: NPE in ITestDynamoDBMetadataStore when
fs.s3a.s3guard.ddb.table is not set
Key: HADOOP-16043
URL: https://issues.apache.org/jira/browse/HADOOP-16043
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 3.2.0
Reporter: Adam Antal
When running {{org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore}}
integration test, I got the following stack trace:
{code:java}
[ERROR] org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore Time
elapsed: 0.333 s <<< ERROR!
java.lang.NullPointerException
at
org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore.beforeClassSetup(ITestDynamoDBMetadataStore.java:164)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
{code}
The NPE happened here:
{code:java}
assertTrue("Test DynamoDB table name: '"
+ S3GUARD_DDB_TEST_TABLE_NAME_KEY + "' and production table name: '"
+ S3GUARD_DDB_TABLE_NAME_KEY + "' can not be the same.",
!conf.get(S3GUARD_DDB_TABLE_NAME_KEY).equals(testDynamoDBTableName));
{code}
The problem is that though we check previously whether the variable
testDynamoDBTableName ({{fs.s3a.s3guard.ddb.test.table}} config) is not null,
but we don't do the same for {{fs.s3a.s3guard.ddb.table}}
({{S3GUARD_DDB_TABLE_NAME_KEY}}) before calling the .equals(), thus causing an
NPE.
Since we don't need the {{fs.s3a.s3guard.ddb.table}} config for the test, we
should check first whether that config is given or not, and only comparing the
two configs if they both exist.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16043) NPE in ITestDynamoDBMetadataStore when fs.s3a.s3guard.ddb.table is not set
[
https://issues.apache.org/jira/browse/HADOOP-16043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota reassigned HADOOP-16043:
---
Assignee: Gabor Bota
> NPE in ITestDynamoDBMetadataStore when fs.s3a.s3guard.ddb.table is not set
> --
>
> Key: HADOOP-16043
> URL: https://issues.apache.org/jira/browse/HADOOP-16043
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Gabor Bota
>Priority: Major
>
> When running {{org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore}}
> integration test, I got the following stack trace:
> {code:java}
> [ERROR] org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore Time
> elapsed: 0.333 s <<< ERROR!
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore.beforeClassSetup(ITestDynamoDBMetadataStore.java:164)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
> The NPE happened here:
> {code:java}
> assertTrue("Test DynamoDB table name: '"
> + S3GUARD_DDB_TEST_TABLE_NAME_KEY + "' and production table name: '"
> + S3GUARD_DDB_TABLE_NAME_KEY + "' can not be the same.",
> !conf.get(S3GUARD_DDB_TABLE_NAME_KEY).equals(testDynamoDBTableName));
> {code}
> The problem is that though we check previously whether the variable
> testDynamoDBTableName ({{fs.s3a.s3guard.ddb.test.table}} config) is not null,
> but we don't do the same for {{fs.s3a.s3guard.ddb.table}}
> ({{S3GUARD_DDB_TABLE_NAME_KEY}}) before calling the .equals(), thus causing
> an NPE.
> Since we don't need the {{fs.s3a.s3guard.ddb.table}} config for the test, we
> should check first whether that config is given or not, and only comparing
> the two configs if they both exist.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] elek opened a new pull request #465: HDDS-793. Support custom key/value annotations on volume/bucket/key.
elek opened a new pull request #465: HDDS-793. Support custom key/value annotations on volume/bucket/key. URL: https://github.com/apache/hadoop/pull/465 See: https://issues.apache.org/jira/browse/HDDS-793 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15843) s3guard bucket-info command to not print a stack trace on bucket-not-found
[
https://issues.apache.org/jira/browse/HADOOP-15843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740520#comment-16740520
]
Adam Antal commented on HADOOP-15843:
-
I looked through the patch, and tested it on eu-ireland ({{eu-west-1}}). The
tests are failing, and I'm on fixing them and add new ones.
Also during the test I got an NPE as well, but it is discussed separately here:
HADOOP-16043.
> s3guard bucket-info command to not print a stack trace on bucket-not-found
> --
>
> Key: HADOOP-15843
> URL: https://issues.apache.org/jira/browse/HADOOP-15843
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Adam Antal
>Priority: Minor
> Attachments: HADOOP-15843-001.patch
>
>
> when you go {{hadoop s3guard bucket-info s3a://bucket-which-doesnt-exist}}
> you get a full stack trace on the failure. This is overkill: all the caller
> needs to know is the bucket isn't there.
> Proposed: catch FNFE and treat as special, have return code of "44", "not
> found".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15843) s3guard bucket-info command to not print a stack trace on bucket-not-found
[
https://issues.apache.org/jira/browse/HADOOP-15843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740520#comment-16740520
]
Adam Antal edited comment on HADOOP-15843 at 1/11/19 4:08 PM:
--
I looked through the patch, and tested it on eu-ireland ({{eu-west-1}}). Some
tests are failing, and I'm on fixing them and add new ones.
Also during the test I got an NPE as well, but it is discussed separately here:
HADOOP-16043.
was (Author: adam.antal):
I looked through the patch, and tested it on eu-ireland ({{eu-west-1}}). The
tests are failing, and I'm on fixing them and add new ones.
Also during the test I got an NPE as well, but it is discussed separately here:
HADOOP-16043.
> s3guard bucket-info command to not print a stack trace on bucket-not-found
> --
>
> Key: HADOOP-15843
> URL: https://issues.apache.org/jira/browse/HADOOP-15843
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Adam Antal
>Priority: Minor
> Attachments: HADOOP-15843-001.patch
>
>
> when you go {{hadoop s3guard bucket-info s3a://bucket-which-doesnt-exist}}
> you get a full stack trace on the failure. This is overkill: all the caller
> needs to know is the bucket isn't there.
> Proposed: catch FNFE and treat as special, have return code of "44", "not
> found".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15481) Emit FairCallQueue stats as metrics
[
https://issues.apache.org/jira/browse/HADOOP-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740563#comment-16740563
]
Erik Krogen commented on HADOOP-15481:
--
Looking great [~cgregori]! I have a few more minor nits:
* I don't think you need the {{@Metrics}} annotation anymore – the
{{DefaultMetricsSystem.instance().register()}} should take care of this, IIUC.
* In the definition of {{MetricsProxy}}, we usually use indentation like:
(hanging indent of 4 spaces)
{code:java}
private static final class MetricsProxy implements FairCallQueueMXBean,
MetricsSource {
{code}
There's some discussion of this
[here|https://www.oracle.com/technetwork/java/javase/documentation/codeconventions-136091.html],
though we typically use 4 spaces instead of 8.
* For the {{name}}, now you do "FairCallQueue.namespace", I'm thinking that
"namespace.FairCallQueue" might be better to match more closely with how the
configurations look?
* I'm starting to think that names like "FairCallQueueSize_p0" might be
better, to make it more clear what the number represents. Let me know what you
think.
> Emit FairCallQueue stats as metrics
> ---
>
> Key: HADOOP-15481
> URL: https://issues.apache.org/jira/browse/HADOOP-15481
> Project: Hadoop Common
> Issue Type: Improvement
> Components: metrics, rpc-server
>Reporter: Erik Krogen
>Assignee: Christopher Gregorian
>Priority: Major
> Attachments: HADOOP-15481.001.patch, HADOOP-15481.001.patch,
> HADOOP-15481.002.patch
>
>
> Currently FairCallQueue has some statistics which are exported via JMX: the
> size of each queue, and the number of overflowed calls per queue. These are
> useful statistics to track over time to determine, for example, if queues
> need to be resized. We should emit them via the standard metrics system.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15481) Emit FairCallQueue stats as metrics
[ https://issues.apache.org/jira/browse/HADOOP-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740586#comment-16740586 ] Christopher Gregorian commented on HADOOP-15481: Good call on the naming—updated the test and docs to reflect this as well. > Emit FairCallQueue stats as metrics > --- > > Key: HADOOP-15481 > URL: https://issues.apache.org/jira/browse/HADOOP-15481 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics, rpc-server >Reporter: Erik Krogen >Assignee: Christopher Gregorian >Priority: Major > Attachments: HADOOP-15481.001.patch, HADOOP-15481.001.patch, > HADOOP-15481.002.patch, HADOOP-15481.003.patch > > > Currently FairCallQueue has some statistics which are exported via JMX: the > size of each queue, and the number of overflowed calls per queue. These are > useful statistics to track over time to determine, for example, if queues > need to be resized. We should emit them via the standard metrics system. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15481) Emit FairCallQueue stats as metrics
[ https://issues.apache.org/jira/browse/HADOOP-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Gregorian updated HADOOP-15481: --- Attachment: HADOOP-15481.003.patch > Emit FairCallQueue stats as metrics > --- > > Key: HADOOP-15481 > URL: https://issues.apache.org/jira/browse/HADOOP-15481 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics, rpc-server >Reporter: Erik Krogen >Assignee: Christopher Gregorian >Priority: Major > Attachments: HADOOP-15481.001.patch, HADOOP-15481.001.patch, > HADOOP-15481.002.patch, HADOOP-15481.003.patch > > > Currently FairCallQueue has some statistics which are exported via JMX: the > size of each queue, and the number of overflowed calls per queue. These are > useful statistics to track over time to determine, for example, if queues > need to be resized. We should emit them via the standard metrics system. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15481) Emit FairCallQueue stats as metrics
[ https://issues.apache.org/jira/browse/HADOOP-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740595#comment-16740595 ] Erik Krogen commented on HADOOP-15481: -- Great, LGTM pending Jenkins. +1 (non-binding) [~vagarychen] or [~shv], care to help review & commit this? > Emit FairCallQueue stats as metrics > --- > > Key: HADOOP-15481 > URL: https://issues.apache.org/jira/browse/HADOOP-15481 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics, rpc-server >Reporter: Erik Krogen >Assignee: Christopher Gregorian >Priority: Major > Attachments: HADOOP-15481.001.patch, HADOOP-15481.001.patch, > HADOOP-15481.002.patch, HADOOP-15481.003.patch > > > Currently FairCallQueue has some statistics which are exported via JMX: the > size of each queue, and the number of overflowed calls per queue. These are > useful statistics to track over time to determine, for example, if queues > need to be resized. We should emit them via the standard metrics system. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15662) ABFS: Better exception handling of DNS errors
[ https://issues.apache.org/jira/browse/HADOOP-15662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740600#comment-16740600 ] Da Zhou commented on HADOOP-15662: -- [~tmarquardt] L73: Will fix the msg format. BTW, the HostNotFountException.getMessage() will only print out the unresolved host name, just as "httpOperation.getUrl().getHost())" does. L714: I see what you mean, yes the else if will break the exception trace. [~ste...@apache.org] Sorry, I will open a new jira for this follow up, closing this one. > ABFS: Better exception handling of DNS errors > - > > Key: HADOOP-15662 > URL: https://issues.apache.org/jira/browse/HADOOP-15662 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.1 > > Attachments: HADOOP-15662-001.patch, HADOOP-15662-002.patch > > > DNS errors are common during testing due to typos or misconfiguration. They > can also occur in production, as some transient DNS issues occur from time to > time. > 1) Let's investigate if we can distinguish between the two and fail fast for > the test issues, but continue to have retry logic for the transient DNS > issues in production. > 2) Let's improve the error handling of DNS failures, so the user has an > actionable error message. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15229) Add FileSystem builder-based openFile() API to match createFile() + S3 Select
[ https://issues.apache.org/jira/browse/HADOOP-15229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740602#comment-16740602 ] Steve Loughran commented on HADOOP-15229: - # I'm thinking we could add an option textinputformat.splitable (yes, wrong spelling, but its consistent), which has TextInputFormat return false, even when there's no codec # Even then though, the split will be that of the file length. How does that work on unsplittable formats? Because we may get back less than expected. (This is the same problem which S3 client-side-encryption has; something which fixes this should fix that too) > Add FileSystem builder-based openFile() API to match createFile() + S3 Select > - > > Key: HADOOP-15229 > URL: https://issues.apache.org/jira/browse/HADOOP-15229 > Project: Hadoop Common > Issue Type: New Feature > Components: fs, fs/azure, fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15229-001.patch, HADOOP-15229-002.patch, > HADOOP-15229-003.patch, HADOOP-15229-004.patch, HADOOP-15229-004.patch, > HADOOP-15229-005.patch, HADOOP-15229-006.patch, HADOOP-15229-007.patch, > HADOOP-15229-009.patch, HADOOP-15229-010.patch, HADOOP-15229-011.patch, > HADOOP-15229-012.patch, HADOOP-15229-013.patch, HADOOP-15229-014.patch, > HADOOP-15229-015.patch, HADOOP-15229-016.patch, HADOOP-15229-017.patch, > HADOOP-15229-018.patch > > > Replicate HDFS-1170 and HADOOP-14365 with an API to open files. > A key requirement of this is not HDFS, it's to put in the fadvise policy for > working with object stores, where getting the decision to do a full GET and > TCP abort on seek vs smaller GETs is fundamentally different: the wrong > option can cost you minutes. S3A and Azure both have adaptive policies now > (first backward seek), but they still don't do it that well. > Columnar formats (ORC, Parquet) should be able to say "fs.input.fadvise" > "random" as an option when they open files; I can imagine other options too. > The Builder model of [~eddyxu] is the one to mimic, method for method. > Ideally with as much code reuse as possible -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16044) ABFS: Better exception handling of DNS errors followup
Da Zhou created HADOOP-16044: Summary: ABFS: Better exception handling of DNS errors followup Key: HADOOP-16044 URL: https://issues.apache.org/jira/browse/HADOOP-16044 Project: Hadoop Common Issue Type: Sub-task Reporter: Da Zhou Assignee: Da Zhou This is a follow up for HADOOP-15662 as the 001 patch of HADOOP-15662 is already committed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16029) Consecutive StringBuilder.append should be reused
[ https://issues.apache.org/jira/browse/HADOOP-16029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated HADOOP-16029: -- Summary: Consecutive StringBuilder.append should be reused (was: Consecutive Append Should Reuse) > Consecutive StringBuilder.append should be reused > - > > Key: HADOOP-16029 > URL: https://issues.apache.org/jira/browse/HADOOP-16029 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HADOOP-16029-01.patch, HADOOP-16029-02.patch, > HADOOP-16029-03.patch, HADOOP-16029-04.patch, HADOOP-16029-05.patch > > > Consecutive calls to StringBuffer/StringBuilder .append should be chained, > reusing the target object. This can improve the performance by producing a > smaller bytecode, reducing overhead and improving inlining. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16029) Consecutive StringBuilder.append can be reused
[ https://issues.apache.org/jira/browse/HADOOP-16029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated HADOOP-16029: -- Summary: Consecutive StringBuilder.append can be reused (was: Consecutive StringBuilder.append should be reused) > Consecutive StringBuilder.append can be reused > -- > > Key: HADOOP-16029 > URL: https://issues.apache.org/jira/browse/HADOOP-16029 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HADOOP-16029-01.patch, HADOOP-16029-02.patch, > HADOOP-16029-03.patch, HADOOP-16029-04.patch, HADOOP-16029-05.patch > > > Consecutive calls to StringBuffer/StringBuilder .append should be chained, > reusing the target object. This can improve the performance by producing a > smaller bytecode, reducing overhead and improving inlining. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16041) UserAgent string for ABFS
[ https://issues.apache.org/jira/browse/HADOOP-16041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740641#comment-16740641 ] Thomas Marquardt commented on HADOOP-16041: --- What are your goals for this change? There were two things we wanted to accomplish with our current User-Agent implementation: 1) Ability to identity the client as ABFS. That is, as opposed to some other client outside Hadoop. 2) Ability to identify the service or partner that build and distributed the client. E.g., Cloudera, Hortonworks, HDInsight, Databricks, etc. Looking at the implementation of VersionInfo.getVersion, it appears to be returning a version number like "3.1". That would make it difficult to do 1), as we would not be searching for a static string but rather a dynamic string which is perhaps nothing but a version number. Perhaps we can leave the static part "Azure Blob FS" or shorten it to "ABFS" and append the Hadoop version or hadoop-azure jar version, so the format is something like "ABFS/%s" where %s is the version string? > UserAgent string for ABFS > - > > Key: HADOOP-16041 > URL: https://issues.apache.org/jira/browse/HADOOP-16041 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Shweta >Assignee: Shweta >Priority: Major > Fix For: 3.3.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15481) Emit FairCallQueue stats as metrics
[ https://issues.apache.org/jira/browse/HADOOP-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740660#comment-16740660 ] Chen Liang commented on HADOOP-15481: - +1 on v003 patch, pending Jenkins. > Emit FairCallQueue stats as metrics > --- > > Key: HADOOP-15481 > URL: https://issues.apache.org/jira/browse/HADOOP-15481 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics, rpc-server >Reporter: Erik Krogen >Assignee: Christopher Gregorian >Priority: Major > Attachments: HADOOP-15481.001.patch, HADOOP-15481.001.patch, > HADOOP-15481.002.patch, HADOOP-15481.003.patch > > > Currently FairCallQueue has some statistics which are exported via JMX: the > size of each queue, and the number of overflowed calls per queue. These are > useful statistics to track over time to determine, for example, if queues > need to be resized. We should emit them via the standard metrics system. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16029) Consecutive StringBuilder.append can be reused
[ https://issues.apache.org/jira/browse/HADOOP-16029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated HADOOP-16029: -- Fix Version/s: 3.3.0 > Consecutive StringBuilder.append can be reused > -- > > Key: HADOOP-16029 > URL: https://issues.apache.org/jira/browse/HADOOP-16029 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-16029-01.patch, HADOOP-16029-02.patch, > HADOOP-16029-03.patch, HADOOP-16029-04.patch, HADOOP-16029-05.patch > > > Consecutive calls to StringBuffer/StringBuilder .append should be chained, > reusing the target object. This can improve the performance by producing a > smaller bytecode, reducing overhead and improving inlining. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16029) Consecutive StringBuilder.append can be reused
[ https://issues.apache.org/jira/browse/HADOOP-16029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740661#comment-16740661 ] Giovanni Matteo Fumarola commented on HADOOP-16029: --- Thanks [~ayushtkn] for the hard work. Committed to trunk. > Consecutive StringBuilder.append can be reused > -- > > Key: HADOOP-16029 > URL: https://issues.apache.org/jira/browse/HADOOP-16029 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-16029-01.patch, HADOOP-16029-02.patch, > HADOOP-16029-03.patch, HADOOP-16029-04.patch, HADOOP-16029-05.patch > > > Consecutive calls to StringBuffer/StringBuilder .append should be chained, > reusing the target object. This can improve the performance by producing a > smaller bytecode, reducing overhead and improving inlining. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16029) Consecutive StringBuilder.append can be reused
[ https://issues.apache.org/jira/browse/HADOOP-16029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated HADOOP-16029: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Consecutive StringBuilder.append can be reused > -- > > Key: HADOOP-16029 > URL: https://issues.apache.org/jira/browse/HADOOP-16029 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-16029-01.patch, HADOOP-16029-02.patch, > HADOOP-16029-03.patch, HADOOP-16029-04.patch, HADOOP-16029-05.patch > > > Consecutive calls to StringBuffer/StringBuilder .append should be chained, > reusing the target object. This can improve the performance by producing a > smaller bytecode, reducing overhead and improving inlining. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16041) UserAgent string for ABFS
[ https://issues.apache.org/jira/browse/HADOOP-16041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740666#comment-16740666 ] Sean Mackrory commented on HADOOP-16041: In the case of CDH (and I believe Hortonworks / HDInsight) the version number includes an identification of the vendor and the vendor release. For example "3.0.0-cdh6.0.0". That's how the vendor has been identified in the other connectors as far as I'm aware - that's definitely all we did for ADLS Gen1 and I know the required information was still collected. In these Hadoop distros the user-configurable prefix is also modifiable by the user, and has been used (not on Azure specifically in the cases I'm thinking of) to identify workloads from a particular large user or something like that. It's not as good a fit for identifying the vendor. With the Databricks model where it's more of a service, then it does make sense to identify themselves using the configured prefix (which is actually a suffix right now) unless their version string embeds enough information. +1 to ABFS/, if that allows you to identify Hadoop vendors sufficiently well. > UserAgent string for ABFS > - > > Key: HADOOP-16041 > URL: https://issues.apache.org/jira/browse/HADOOP-16041 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Shweta >Assignee: Shweta >Priority: Major > Fix For: 3.3.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] BELUGABEHR commented on issue #46: Simplify logging logic
BELUGABEHR commented on issue #46: Simplify logging logic URL: https://github.com/apache/hadoop/pull/46#issuecomment-453624359 This PR was resolved instead with a patch from: HDFS-9457 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] BELUGABEHR closed pull request #46: Simplify logging logic
BELUGABEHR closed pull request #46: Simplify logging logic
URL: https://github.com/apache/hadoop/pull/46
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
index 29bcd795a520..116cbd032575 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
@@ -1628,21 +1628,16 @@ public BlockLocalPathInfo
getBlockLocalPathInfo(ExtendedBlock block,
checkBlockLocalPathAccess();
checkBlockToken(block, token, BlockTokenIdentifier.AccessMode.READ);
Preconditions.checkNotNull(data, "Storage not yet initialized");
-BlockLocalPathInfo info = data.getBlockLocalPathInfo(block);
-if (LOG.isDebugEnabled()) {
- if (info != null) {
-if (LOG.isTraceEnabled()) {
- LOG.trace("getBlockLocalPathInfo successful block=" + block
- + " blockfile " + info.getBlockPath() + " metafile "
- + info.getMetaPath());
-}
- } else {
-if (LOG.isTraceEnabled()) {
- LOG.trace("getBlockLocalPathInfo for block=" + block
- + " returning null");
+final BlockLocalPathInfo info = data.getBlockLocalPathInfo(block);
+if (LOG.isTraceEnabled()) {
+if (info != null) {
+LOG.trace("getBlockLocalPathInfo successful block=" + block
++ " blockfile " + info.getBlockPath() + " metafile "
++ info.getMetaPath());
+} else {
+LOG.trace("getBlockLocalPathInfo for block=null returning null");
}
}
-}
metrics.incrBlocksGetLocalPathInfo();
return info;
}
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15662) ABFS: Better exception handling of DNS errors
[ https://issues.apache.org/jira/browse/HADOOP-15662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Zhou updated HADOOP-15662: - Resolution: Fixed Status: Resolved (was: Patch Available) > ABFS: Better exception handling of DNS errors > - > > Key: HADOOP-15662 > URL: https://issues.apache.org/jira/browse/HADOOP-15662 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.1 > > Attachments: HADOOP-15662-001.patch, HADOOP-15662-002.patch > > > DNS errors are common during testing due to typos or misconfiguration. They > can also occur in production, as some transient DNS issues occur from time to > time. > 1) Let's investigate if we can distinguish between the two and fail fast for > the test issues, but continue to have retry logic for the transient DNS > issues in production. > 2) Let's improve the error handling of DNS failures, so the user has an > actionable error message. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15662) ABFS: Better exception handling of DNS errors
[ https://issues.apache.org/jira/browse/HADOOP-15662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740676#comment-16740676 ] Da Zhou commented on HADOOP-15662: -- HADOOP-16044 has been created for the followup. > ABFS: Better exception handling of DNS errors > - > > Key: HADOOP-15662 > URL: https://issues.apache.org/jira/browse/HADOOP-15662 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.1 > > Attachments: HADOOP-15662-001.patch, HADOOP-15662-002.patch > > > DNS errors are common during testing due to typos or misconfiguration. They > can also occur in production, as some transient DNS issues occur from time to > time. > 1) Let's investigate if we can distinguish between the two and fail fast for > the test issues, but continue to have retry logic for the transient DNS > issues in production. > 2) Let's improve the error handling of DNS failures, so the user has an > actionable error message. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16029) Consecutive StringBuilder.append can be reused
[ https://issues.apache.org/jira/browse/HADOOP-16029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740679#comment-16740679 ] Hudson commented on HADOOP-16029: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15763 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15763/]) HADOOP-16029. Consecutive StringBuilder.append can be reused. (gifuma: rev fb8932a727f757b2e9c1c61a18145878d0eb77bd) * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Path.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CipherSuite.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/BlockLocation.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/RedundantEditLogInputStream.java * (edit) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyListingFileStatus.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/launcher/InterruptEscalator.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECSchema.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/GraphiteSink.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/AccessControlList.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeInfo.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/SemaphoredDelegatingExecutor.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/bloom/DynamicBloomFilter.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StoragePolicySummary.java * (edit) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/PathLocation.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Ls.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/PlanCommand.java * (edit) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionContext.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSZKFailoverController.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/CpuTimeTracker.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/SignalLogger.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileStatus.java * (edit) hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterQuotaUsage.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EncryptionZoneManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/MD5Hash.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/CredentialProvider.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WritableRpcEngine.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/
[jira] [Commented] (HADOOP-15481) Emit FairCallQueue stats as metrics
[
https://issues.apache.org/jira/browse/HADOOP-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740702#comment-16740702
]
Hadoop QA commented on HADOOP-15481:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 30s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} hadoop-common-project/hadoop-common: The patch
generated 0 new + 32 unchanged - 1 fixed = 32 total (was 33) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 59s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
18s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
44s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 99m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15481 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12954612/HADOOP-15481.003.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux c0d82c5b1764 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct
31 10:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / dddad98 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15776/testReport/ |
| Max. process+thread count | 1447 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15776/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Emit FairCallQueue stats as metrics
>
[jira] [Updated] (HADOOP-16044) ABFS: Better exception handling of DNS errors followup
[ https://issues.apache.org/jira/browse/HADOOP-16044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Zhou updated HADOOP-16044: - Attachment: HADOOP-16044-001.patch > ABFS: Better exception handling of DNS errors followup > -- > > Key: HADOOP-16044 > URL: https://issues.apache.org/jira/browse/HADOOP-16044 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Attachments: HADOOP-16044-001.patch > > > This is a follow up for HADOOP-15662 as the 001 patch of HADOOP-15662 is > already committed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16044) ABFS: Better exception handling of DNS errors followup
[ https://issues.apache.org/jira/browse/HADOOP-16044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Zhou updated HADOOP-16044: - Attachment: (was: HADOOP-16044-001.patch) > ABFS: Better exception handling of DNS errors followup > -- > > Key: HADOOP-16044 > URL: https://issues.apache.org/jira/browse/HADOOP-16044 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Attachments: HADOOP-16044-001.patch > > > This is a follow up for HADOOP-15662 as the 001 patch of HADOOP-15662 is > already committed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16044) ABFS: Better exception handling of DNS errors followup
[ https://issues.apache.org/jira/browse/HADOOP-16044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Zhou updated HADOOP-16044: - Attachment: HADOOP-16044-001.patch > ABFS: Better exception handling of DNS errors followup > -- > > Key: HADOOP-16044 > URL: https://issues.apache.org/jira/browse/HADOOP-16044 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Attachments: HADOOP-16044-001.patch > > > This is a follow up for HADOOP-15662 as the 001 patch of HADOOP-15662 is > already committed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16044) ABFS: Better exception handling of DNS errors followup
[ https://issues.apache.org/jira/browse/HADOOP-16044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Zhou updated HADOOP-16044: - Attachment: HADOOP-16044-002.patch > ABFS: Better exception handling of DNS errors followup > -- > > Key: HADOOP-16044 > URL: https://issues.apache.org/jira/browse/HADOOP-16044 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Attachments: HADOOP-16044-001.patch, HADOOP-16044-002.patch > > > This is a follow up for HADOOP-15662 as the 001 patch of HADOOP-15662 is > already committed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16044) ABFS: Better exception handling of DNS errors followup
[ https://issues.apache.org/jira/browse/HADOOP-16044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Zhou updated HADOOP-16044: - Attachment: HADOOP-16044-002.patch > ABFS: Better exception handling of DNS errors followup > -- > > Key: HADOOP-16044 > URL: https://issues.apache.org/jira/browse/HADOOP-16044 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Attachments: HADOOP-16044-001.patch, HADOOP-16044-002.patch > > > This is a follow up for HADOOP-15662 as the 001 patch of HADOOP-15662 is > already committed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16044) ABFS: Better exception handling of DNS errors followup
[ https://issues.apache.org/jira/browse/HADOOP-16044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Zhou updated HADOOP-16044: - Status: Patch Available (was: Open) > ABFS: Better exception handling of DNS errors followup > -- > > Key: HADOOP-16044 > URL: https://issues.apache.org/jira/browse/HADOOP-16044 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Attachments: HADOOP-16044-001.patch, HADOOP-16044-002.patch > > > This is a follow up for HADOOP-15662 as the 001 patch of HADOOP-15662 is > already committed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16044) ABFS: Better exception handling of DNS errors followup
[ https://issues.apache.org/jira/browse/HADOOP-16044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Zhou updated HADOOP-16044: - Attachment: (was: HADOOP-16044-002.patch) > ABFS: Better exception handling of DNS errors followup > -- > > Key: HADOOP-16044 > URL: https://issues.apache.org/jira/browse/HADOOP-16044 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Attachments: HADOOP-16044-001.patch > > > This is a follow up for HADOOP-15662 as the 001 patch of HADOOP-15662 is > already committed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16044) ABFS: Better exception handling of DNS errors followup
[
https://issues.apache.org/jira/browse/HADOOP-16044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740734#comment-16740734
]
Da Zhou commented on HADOOP-16044:
--
Following up for -HADOOP-15662, c-ompared to HADOOP-15662-002.patch:
L173:
- Updated the format to: *Unknown host name: %s. Retrying to resolve the host
name...*
- Replace *"ex.getMessage()*" with "*httpOperation.getUrl().getHost()*)",
although they both provide the host name, but the *getHost()* seems to be more
readable.
- Removed the else if logic as it breaks the current exception trace.
Test:
testUnknownHost() would take about 14 minutes due to the retry, I verified the
warning format in the console:
{noformat}
Unknown host name:
fake7c299e30-d477-4839-b0bc-84271c7ab281.dfs.core.windows.net. Retrying to
resolve the host name...
{noformat}
I haven't come up with a good way to retrieve the log msg for test, any
suggestions?
Testes against US west account passed:
All tests passed my US west account:
XNS account oauth
Tests run: 35, Failures: 0, Errors: 0, Skipped: 0
Tests run: 324, Failures: 0, Errors: 0, Skipped: 22
Tests run: 168, Failures: 0, Errors: 0, Skipped: 21
XNS account sharedKey:
Tests run: 35, Failures: 0, Errors: 0, Skipped: 0
Tests run: 324, Failures: 0, Errors: 0, Skipped: 20
Tests run: 168, Failures: 0, Errors: 0, Skipped: 15
non-xns account sharedKe:
Tests run: 35, Failures: 0, Errors: 0, Skipped: 0
Tests run: 324, Failures: 0, Errors: 0, Skipped: 206
Tests run: 168, Failures: 0, Errors: 0, Skipped: 15
> ABFS: Better exception handling of DNS errors followup
> --
>
> Key: HADOOP-16044
> URL: https://issues.apache.org/jira/browse/HADOOP-16044
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Da Zhou
>Assignee: Da Zhou
>Priority: Major
> Attachments: HADOOP-16044-001.patch, HADOOP-16044-002.patch
>
>
> This is a follow up for HADOOP-15662 as the 001 patch of HADOOP-15662 is
> already committed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16044) ABFS: Better exception handling of DNS errors followup
[
https://issues.apache.org/jira/browse/HADOOP-16044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740734#comment-16740734
]
Da Zhou edited comment on HADOOP-16044 at 1/11/19 8:01 PM:
---
Following up for -HADOOP-15662, c-ompared to HADOOP-15662-002.patch:
L173:
- Updated the format to: *Unknown host name: %s. Retrying to resolve the host
name...*
- Replace *"ex.getMessage()*" with "*httpOperation.getUrl().getHost()*)",
although they both return host name string, but the *getHost()* seems to be
more readable code.
- Removed the else if logic as it breaks the current exception trace.
Test:
testUnknownHost() would take about 14 minutes due to the retry, I verified the
warning format in the console:
{noformat}
Unknown host name:
fake7c299e30-d477-4839-b0bc-84271c7ab281.dfs.core.windows.net. Retrying to
resolve the host name...
{noformat}
I haven't come up with a good way to retrieve the log msg for test, any
suggestions?
Testes against US west account passed:
All tests passed my US west account:
XNS account oauth
Tests run: 35, Failures: 0, Errors: 0, Skipped: 0
Tests run: 324, Failures: 0, Errors: 0, Skipped: 22
Tests run: 168, Failures: 0, Errors: 0, Skipped: 21
XNS account sharedKey:
Tests run: 35, Failures: 0, Errors: 0, Skipped: 0
Tests run: 324, Failures: 0, Errors: 0, Skipped: 20
Tests run: 168, Failures: 0, Errors: 0, Skipped: 15
non-xns account sharedKe:
Tests run: 35, Failures: 0, Errors: 0, Skipped: 0
Tests run: 324, Failures: 0, Errors: 0, Skipped: 206
Tests run: 168, Failures: 0, Errors: 0, Skipped: 15
was (Author: danielzhou):
Following up for -HADOOP-15662, c-ompared to HADOOP-15662-002.patch:
L173:
- Updated the format to: *Unknown host name: %s. Retrying to resolve the host
name...*
- Replace *"ex.getMessage()*" with "*httpOperation.getUrl().getHost()*)",
although they both provide the host name, but the *getHost()* seems to be more
readable.
- Removed the else if logic as it breaks the current exception trace.
Test:
testUnknownHost() would take about 14 minutes due to the retry, I verified the
warning format in the console:
{noformat}
Unknown host name:
fake7c299e30-d477-4839-b0bc-84271c7ab281.dfs.core.windows.net. Retrying to
resolve the host name...
{noformat}
I haven't come up with a good way to retrieve the log msg for test, any
suggestions?
Testes against US west account passed:
All tests passed my US west account:
XNS account oauth
Tests run: 35, Failures: 0, Errors: 0, Skipped: 0
Tests run: 324, Failures: 0, Errors: 0, Skipped: 22
Tests run: 168, Failures: 0, Errors: 0, Skipped: 21
XNS account sharedKey:
Tests run: 35, Failures: 0, Errors: 0, Skipped: 0
Tests run: 324, Failures: 0, Errors: 0, Skipped: 20
Tests run: 168, Failures: 0, Errors: 0, Skipped: 15
non-xns account sharedKe:
Tests run: 35, Failures: 0, Errors: 0, Skipped: 0
Tests run: 324, Failures: 0, Errors: 0, Skipped: 206
Tests run: 168, Failures: 0, Errors: 0, Skipped: 15
> ABFS: Better exception handling of DNS errors followup
> --
>
> Key: HADOOP-16044
> URL: https://issues.apache.org/jira/browse/HADOOP-16044
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Da Zhou
>Assignee: Da Zhou
>Priority: Major
> Attachments: HADOOP-16044-001.patch, HADOOP-16044-002.patch
>
>
> This is a follow up for HADOOP-15662 as the 001 patch of HADOOP-15662 is
> already committed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16041) UserAgent string for ABFS
[
https://issues.apache.org/jira/browse/HADOOP-16041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740776#comment-16740776
]
Da Zhou commented on HADOOP-16041:
--
yes currently the so called "configured prefix" is actually a suffix, the
current userAgent string is:
{noformat}
Azure Blob FS/1.0 (JavaJRE JAVA_VERSION; OS_VERSION) {optional :
CONFIGURED_SUFFIX}
{noformat}
+1 for the proposal:
{noformat}
Azure Blob FS/{HADOOP_VENDOR_VERSION} (JavaJRE JAVA_VERSION; OS_VERSION)
{optional : CONFIGURED_SUFFIX}
{noformat}
> UserAgent string for ABFS
> -
>
> Key: HADOOP-16041
> URL: https://issues.apache.org/jira/browse/HADOOP-16041
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Shweta
>Assignee: Shweta
>Priority: Major
> Fix For: 3.3.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16013) DecayRpcScheduler decay thread should run as a daemon
[
https://issues.apache.org/jira/browse/HADOOP-16013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740778#comment-16740778
]
Chen Liang commented on HADOOP-16013:
-
+1 on v000 patch, the failed tests seem unrelated and passed in my local run.
I've committed to trunk, thanks for the contribution [~xkrogen]!
> DecayRpcScheduler decay thread should run as a daemon
> -
>
> Key: HADOOP-16013
> URL: https://issues.apache.org/jira/browse/HADOOP-16013
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 3.3.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HADOOP-16013.000.patch
>
>
> {{DecayRpcScheduler}} sets up a {{Timer}} to schedule a decay of the weights
> it tracks:
> {code}
> // Setup delay timer
> Timer timer = new Timer();
> DecayTask task = new DecayTask(this, timer);
> timer.scheduleAtFixedRate(task, decayPeriodMillis, decayPeriodMillis);
> {code}
> However this Timer is not set up as a daemon thread. I have seen this cause
> my JVM to refuse to exit when running, for example, NNThroughputBenchmark
> with FairCallQueue enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16013) DecayRpcScheduler decay thread should run as a daemon
[
https://issues.apache.org/jira/browse/HADOOP-16013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740781#comment-16740781
]
Erik Krogen commented on HADOOP-16013:
--
Nice! Thanks [~vagarychen]!
> DecayRpcScheduler decay thread should run as a daemon
> -
>
> Key: HADOOP-16013
> URL: https://issues.apache.org/jira/browse/HADOOP-16013
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 3.3.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HADOOP-16013.000.patch
>
>
> {{DecayRpcScheduler}} sets up a {{Timer}} to schedule a decay of the weights
> it tracks:
> {code}
> // Setup delay timer
> Timer timer = new Timer();
> DecayTask task = new DecayTask(this, timer);
> timer.scheduleAtFixedRate(task, decayPeriodMillis, decayPeriodMillis);
> {code}
> However this Timer is not set up as a daemon thread. I have seen this cause
> my JVM to refuse to exit when running, for example, NNThroughputBenchmark
> with FairCallQueue enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16013) DecayRpcScheduler decay thread should run as a daemon
[
https://issues.apache.org/jira/browse/HADOOP-16013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740778#comment-16740778
]
Chen Liang edited comment on HADOOP-16013 at 1/11/19 8:57 PM:
--
+1 on v000 patch, the failed tests seem unrelated and passed in my local run.
I've committed to trunk and branch-2, thanks for the contribution [~xkrogen]!
was (Author: vagarychen):
+1 on v000 patch, the failed tests seem unrelated and passed in my local run.
I've committed to trunk, thanks for the contribution [~xkrogen]!
> DecayRpcScheduler decay thread should run as a daemon
> -
>
> Key: HADOOP-16013
> URL: https://issues.apache.org/jira/browse/HADOOP-16013
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 3.3.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HADOOP-16013.000.patch
>
>
> {{DecayRpcScheduler}} sets up a {{Timer}} to schedule a decay of the weights
> it tracks:
> {code}
> // Setup delay timer
> Timer timer = new Timer();
> DecayTask task = new DecayTask(this, timer);
> timer.scheduleAtFixedRate(task, decayPeriodMillis, decayPeriodMillis);
> {code}
> However this Timer is not set up as a daemon thread. I have seen this cause
> my JVM to refuse to exit when running, for example, NNThroughputBenchmark
> with FairCallQueue enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16013) DecayRpcScheduler decay thread should run as a daemon
[
https://issues.apache.org/jira/browse/HADOOP-16013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HADOOP-16013:
Resolution: Fixed
Fix Version/s: 3.3.0
2.10.0
Status: Resolved (was: Patch Available)
> DecayRpcScheduler decay thread should run as a daemon
> -
>
> Key: HADOOP-16013
> URL: https://issues.apache.org/jira/browse/HADOOP-16013
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 3.3.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Fix For: 2.10.0, 3.3.0
>
> Attachments: HADOOP-16013.000.patch
>
>
> {{DecayRpcScheduler}} sets up a {{Timer}} to schedule a decay of the weights
> it tracks:
> {code}
> // Setup delay timer
> Timer timer = new Timer();
> DecayTask task = new DecayTask(this, timer);
> timer.scheduleAtFixedRate(task, decayPeriodMillis, decayPeriodMillis);
> {code}
> However this Timer is not set up as a daemon thread. I have seen this cause
> my JVM to refuse to exit when running, for example, NNThroughputBenchmark
> with FairCallQueue enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16041) UserAgent string for ABFS
[ https://issues.apache.org/jira/browse/HADOOP-16041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740785#comment-16740785 ] Thomas Marquardt commented on HADOOP-16041: --- For 2), I did not mention that the design is such that the service or partner can set "fs.azure.user.agent.prefix" to a value like "3.0.0-cdh6.0.0", and then it will be appended to the User-Agent string in a format similar to "Azure Blob FS/1.0 (JavaJRE 1.8.0_172; MacOSX 10.13.6) %s", where %s is "3.0.0-cdh6.0.0" in this example. I realize this is a suffix and not a prefix, but I did not come up with the name for the configuration property. :) With that said, your proposal improves the situation by making it automatic, so there is less of a need for the service or partner to further customize the User-Agent with our config setting. Specifically, replacing "Azure Blob FS/1.0" with "ABFS/" sounds great to me. > UserAgent string for ABFS > - > > Key: HADOOP-16041 > URL: https://issues.apache.org/jira/browse/HADOOP-16041 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Shweta >Assignee: Shweta >Priority: Major > Fix For: 3.3.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16044) ABFS: Better exception handling of DNS errors followup
[ https://issues.apache.org/jira/browse/HADOOP-16044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740792#comment-16740792 ] Thomas Marquardt commented on HADOOP-16044: --- +1, LGTM > ABFS: Better exception handling of DNS errors followup > -- > > Key: HADOOP-16044 > URL: https://issues.apache.org/jira/browse/HADOOP-16044 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Attachments: HADOOP-16044-001.patch, HADOOP-16044-002.patch > > > This is a follow up for HADOOP-15662 as the 001 patch of HADOOP-15662 is > already committed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16013) DecayRpcScheduler decay thread should run as a daemon
[
https://issues.apache.org/jira/browse/HADOOP-16013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740795#comment-16740795
]
Hudson commented on HADOOP-16013:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15764 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/15764/])
HADOOP-16013. DecayRpcScheduler decay thread should run as a daemon. (cliang:
rev 01cb958af44b2376bcf579cc65d90566530f733d)
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/DecayRpcScheduler.java
> DecayRpcScheduler decay thread should run as a daemon
> -
>
> Key: HADOOP-16013
> URL: https://issues.apache.org/jira/browse/HADOOP-16013
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 3.3.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Fix For: 2.10.0, 3.3.0
>
> Attachments: HADOOP-16013.000.patch
>
>
> {{DecayRpcScheduler}} sets up a {{Timer}} to schedule a decay of the weights
> it tracks:
> {code}
> // Setup delay timer
> Timer timer = new Timer();
> DecayTask task = new DecayTask(this, timer);
> timer.scheduleAtFixedRate(task, decayPeriodMillis, decayPeriodMillis);
> {code}
> However this Timer is not set up as a daemon thread. I have seen this cause
> my JVM to refuse to exit when running, for example, NNThroughputBenchmark
> with FairCallQueue enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16044) ABFS: Better exception handling of DNS errors followup
[
https://issues.apache.org/jira/browse/HADOOP-16044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740790#comment-16740790
]
Hadoop QA commented on HADOOP-16044:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 5s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 35s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
12s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-16044 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12954630/HADOOP-16044-002.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux f8faa8078ced 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / fb8932a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15777/testReport/ |
| Max. process+thread count | 342 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15777/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> ABFS: Better exception handling of DNS errors followup
> --
>
> Key: HADOOP-16044
>
[jira] [Updated] (HADOOP-15481) Emit FairCallQueue stats as metrics
[ https://issues.apache.org/jira/browse/HADOOP-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HADOOP-15481: Resolution: Fixed Fix Version/s: 3.3.0 Status: Resolved (was: Patch Available) > Emit FairCallQueue stats as metrics > --- > > Key: HADOOP-15481 > URL: https://issues.apache.org/jira/browse/HADOOP-15481 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics, rpc-server >Reporter: Erik Krogen >Assignee: Christopher Gregorian >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15481.001.patch, HADOOP-15481.001.patch, > HADOOP-15481.002.patch, HADOOP-15481.003.patch > > > Currently FairCallQueue has some statistics which are exported via JMX: the > size of each queue, and the number of overflowed calls per queue. These are > useful statistics to track over time to determine, for example, if queues > need to be resized. We should emit them via the standard metrics system. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15481) Emit FairCallQueue stats as metrics
[ https://issues.apache.org/jira/browse/HADOOP-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740831#comment-16740831 ] Hudson commented on HADOOP-15481: - FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #15765 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15765/]) HADOOP-15481. Emit FairCallQueue stats as metrics. Contributed by (cliang: rev bf08f4abae43d706a305af3f14e00f01c00dba7c) * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestFairCallQueue.java * (edit) hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/FairCallQueue.java > Emit FairCallQueue stats as metrics > --- > > Key: HADOOP-15481 > URL: https://issues.apache.org/jira/browse/HADOOP-15481 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics, rpc-server >Reporter: Erik Krogen >Assignee: Christopher Gregorian >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15481.001.patch, HADOOP-15481.001.patch, > HADOOP-15481.002.patch, HADOOP-15481.003.patch > > > Currently FairCallQueue has some statistics which are exported via JMX: the > size of each queue, and the number of overflowed calls per queue. These are > useful statistics to track over time to determine, for example, if queues > need to be resized. We should emit them via the standard metrics system. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15481) Emit FairCallQueue stats as metrics
[ https://issues.apache.org/jira/browse/HADOOP-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740830#comment-16740830 ] Chen Liang commented on HADOOP-15481: - I've committed v003 patch to trunk, thanks [~cgregori] for the contribution! > Emit FairCallQueue stats as metrics > --- > > Key: HADOOP-15481 > URL: https://issues.apache.org/jira/browse/HADOOP-15481 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics, rpc-server >Reporter: Erik Krogen >Assignee: Christopher Gregorian >Priority: Major > Attachments: HADOOP-15481.001.patch, HADOOP-15481.001.patch, > HADOOP-15481.002.patch, HADOOP-15481.003.patch > > > Currently FairCallQueue has some statistics which are exported via JMX: the > size of each queue, and the number of overflowed calls per queue. These are > useful statistics to track over time to determine, for example, if queues > need to be resized. We should emit them via the standard metrics system. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth
[ https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Zhou updated HADOOP-15954: - Attachment: HADOOP-15954-011.patch > ABFS: Enable owner and group conversion for MSI and login user using OAuth > -- > > Key: HADOOP-15954 > URL: https://issues.apache.org/jira/browse/HADOOP-15954 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: junhua gu >Assignee: Da Zhou >Priority: Major > Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, > HADOOP-15954-003.patch, HADOOP-15954-004.patch, HADOOP-15954-005.patch, > HADOOP-15954-006.patch, HADOOP-15954-007.patch, HADOOP-15954-008.patch, > HADOOP-15954-009.patch, HADOOP-15954-010.patch, HADOOP-15954-011.patch > > > Add support for overwriting owner and group in set/get operations to be the > service principal id when OAuth is used. Add support for upn short name > format. > > Add Standard Transformer for SharedKey / Service > Add interface provides an extensible model for customizing the acquisition of > Identity Transformer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] apache-yetus commented on issue #459: HADOOP-16035. Jenkinsfile for Hadoop
apache-yetus commented on issue #459: HADOOP-16035. Jenkinsfile for Hadoop URL: https://github.com/apache/hadoop/pull/459#issuecomment-453681576 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 32 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | ||| _ trunk Compile Tests _ | | +1 | shadedclient | 806 | branch has no errors when building and testing our client artifacts. | ||| _ Patch Compile Tests _ | | +1 | shellcheck | 0 | There were no new shellcheck issues. | | +1 | shelldocs | 18 | There were no new shelldocs issues. | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 715 | patch has no errors when building and testing our client artifacts. | ||| _ Other Tests _ | | +1 | asflicense | 28 | The patch does not generate ASF License warnings. | | | | 1638 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-459/8/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/459 | | Optional Tests | dupname asflicense shellcheck shelldocs | | uname | Linux 708479b19413 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / bf08f4a | | maven | version: Apache Maven 3.3.9 | | shellcheck | v0.4.6 | | Max. process+thread count | 445 (vs. ulimit of 5500) | | modules | C: . U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-459/8/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16035) Jenkinsfile for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-16035: -- Attachment: HADOOP-16035.00.patch > Jenkinsfile for Hadoop > -- > > Key: HADOOP-16035 > URL: https://issues.apache.org/jira/browse/HADOOP-16035 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Major > Attachments: HADOOP-16035.00.patch > > > Just a JIRA for me to test yetus github PR support against hadoop -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16035) Jenkinsfile for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-16035: -- Summary: Jenkinsfile for Hadoop (was: test yetus master) > Jenkinsfile for Hadoop > -- > > Key: HADOOP-16035 > URL: https://issues.apache.org/jira/browse/HADOOP-16035 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Major > Attachments: HADOOP-16035.00.patch > > > Just a JIRA for me to test yetus github PR support against hadoop -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth
[ https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740851#comment-16740851 ] Da Zhou commented on HADOOP-15954: -- patch 010 doesn't apply to trunk due to recent commits. Submitting patch 011. > ABFS: Enable owner and group conversion for MSI and login user using OAuth > -- > > Key: HADOOP-15954 > URL: https://issues.apache.org/jira/browse/HADOOP-15954 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: junhua gu >Assignee: Da Zhou >Priority: Major > Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, > HADOOP-15954-003.patch, HADOOP-15954-004.patch, HADOOP-15954-005.patch, > HADOOP-15954-006.patch, HADOOP-15954-007.patch, HADOOP-15954-008.patch, > HADOOP-15954-009.patch, HADOOP-15954-010.patch, HADOOP-15954-011.patch > > > Add support for overwriting owner and group in set/get operations to be the > service principal id when OAuth is used. Add support for upn short name > format. > > Add Standard Transformer for SharedKey / Service > Add interface provides an extensible model for customizing the acquisition of > Identity Transformer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16035) Jenkinsfile for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-16035: -- Description: In order to enable Github Branch Source plugin on Jenkins to test Github PRs with Apache Yetus: - an account that can read Github - Apache Yetus 0.9.0+ - a Jenkinsfile that uses the above was:Just a JIRA for me to test yetus github PR support against hadoop > Jenkinsfile for Hadoop > -- > > Key: HADOOP-16035 > URL: https://issues.apache.org/jira/browse/HADOOP-16035 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Major > Attachments: HADOOP-16035.00.patch > > > In order to enable Github Branch Source plugin on Jenkins to test Github PRs > with Apache Yetus: > - an account that can read Github > - Apache Yetus 0.9.0+ > - a Jenkinsfile that uses the above -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16035) Jenkinsfile for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740863#comment-16740863 ] Allen Wittenauer commented on HADOOP-16035: --- This patch uses the pre-release branch of Apache Yetus 0.9.0. Branches for releases in Apache Yetus disappear after the release gets tagged, so this should be considered a temporary patch for testing. > Jenkinsfile for Hadoop > -- > > Key: HADOOP-16035 > URL: https://issues.apache.org/jira/browse/HADOOP-16035 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Major > Attachments: HADOOP-16035.00.patch > > > In order to enable Github Branch Source plugin on Jenkins to test Github PRs > with Apache Yetus: > - an account that can read Github > - Apache Yetus 0.9.0+ > - a Jenkinsfile that uses the above -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-13656) fs -expunge to take a filesystem
[
https://issues.apache.org/jira/browse/HADOOP-13656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shweta reassigned HADOOP-13656:
---
Assignee: Shweta
> fs -expunge to take a filesystem
>
>
> Key: HADOOP-13656
> URL: https://issues.apache.org/jira/browse/HADOOP-13656
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 2.7.3
>Reporter: Steve Loughran
>Assignee: Shweta
>Priority: Minor
>
> you can't pass in a filesystem or object store to {{fs -expunge}; you have to
> change the default fs
> {code}
> hadoop fs -expunge -D fs.defaultFS=s3a://bucket/
> {code}
> If the command took an optional filesystem argument, it'd be better at
> cleaning up object stores. Given that even deleted object store data runs up
> bills, this could be appreciated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15205) maven release: missing source attachments for hadoop-mapreduce-client-core
[ https://issues.apache.org/jira/browse/HADOOP-15205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740875#comment-16740875 ] Elek, Marton commented on HADOOP-15205: --- As I see in the source the -Pdist profile is required for the source upload: [https://github.com/apache/hadoop/blob/bf08f4abae43d706a305af3f14e00f01c00dba7c/hadoop-project/pom.xml#L1932] I modified the [release guideline|https://wiki.apache.org/hadoop/HowToRelease#preview] to include dist profile in the the mvn deploy command execution. Don't know what is the plan to fix older releases. > maven release: missing source attachments for hadoop-mapreduce-client-core > -- > > Key: HADOOP-15205 > URL: https://issues.apache.org/jira/browse/HADOOP-15205 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.9.0, 2.8.2, 2.8.3, 2.7.5, 3.0.0, 3.1.0, 3.0.1, 2.8.4, > 2.9.2, 2.8.5 >Reporter: Zoltan Haindrich >Priority: Major > Attachments: chk.bash > > > I wanted to use the source attachment; however it looks like since 2.7.5 that > artifact is not present at maven central ; it looks like the last release > which had source attachments / javadocs was 2.7.4 > http://central.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.5/ > this seems to be not limited to mapreduce; as the same change is present for > yarn-common as well > http://central.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.7.5/ > and also hadoop-common > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/2.7.5/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/3.0.0/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/3.1.0/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth
[
https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740891#comment-16740891
]
Hadoop QA commented on HADOOP-15954:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 12s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} hadoop-tools/hadoop-azure: The patch generated 0 new
+ 2 unchanged - 1 fixed = 2 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 41s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
11s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15954 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12954645/HADOOP-15954-011.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 6955c6a39e8e 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / bf08f4a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15778/testReport/ |
| Max. process+thread count | 341 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15778/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org
[jira] [Updated] (HADOOP-16045) Don't run TestDU on Windows
[ https://issues.apache.org/jira/browse/HADOOP-16045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Majercak updated HADOOP-16045: Attachment: HADOOP-16045.001.patch > Don't run TestDU on Windows > --- > > Key: HADOOP-16045 > URL: https://issues.apache.org/jira/browse/HADOOP-16045 > Project: Hadoop Common > Issue Type: Bug > Components: common, test >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Trivial > Attachments: HADOOP-16045.001.patch > > > DU is not supported on Windows, ignore the test. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16045) Don't run TestDU on Windows
[ https://issues.apache.org/jira/browse/HADOOP-16045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HADOOP-16045: - Issue Type: Sub-task (was: Bug) Parent: HADOOP-15475 > Don't run TestDU on Windows > --- > > Key: HADOOP-16045 > URL: https://issues.apache.org/jira/browse/HADOOP-16045 > Project: Hadoop Common > Issue Type: Sub-task > Components: common, test >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Trivial > Attachments: HADOOP-16045.001.patch > > > DU is not supported on Windows, ignore the test. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16045) Don't run TestDU on Windows
Lukas Majercak created HADOOP-16045: --- Summary: Don't run TestDU on Windows Key: HADOOP-16045 URL: https://issues.apache.org/jira/browse/HADOOP-16045 Project: Hadoop Common Issue Type: Bug Components: common, test Reporter: Lukas Majercak Assignee: Lukas Majercak DU is not supported on Windows, ignore the test. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16045) Don't run TestDU on Windows
[ https://issues.apache.org/jira/browse/HADOOP-16045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740896#comment-16740896 ] Íñigo Goiri commented on HADOOP-16045: -- This looks straightforward. Can you point to the code that uses the command not available in Windows? > Don't run TestDU on Windows > --- > > Key: HADOOP-16045 > URL: https://issues.apache.org/jira/browse/HADOOP-16045 > Project: Hadoop Common > Issue Type: Sub-task > Components: common, test >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Trivial > Attachments: HADOOP-16045.001.patch > > > DU is not supported on Windows, ignore the test. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16045) Don't run TestDU on Windows
[ https://issues.apache.org/jira/browse/HADOOP-16045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740897#comment-16740897 ] Lukas Majercak commented on HADOOP-16045: - DU line 72 > Don't run TestDU on Windows > --- > > Key: HADOOP-16045 > URL: https://issues.apache.org/jira/browse/HADOOP-16045 > Project: Hadoop Common > Issue Type: Sub-task > Components: common, test >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Trivial > Attachments: HADOOP-16045.001.patch > > > DU is not supported on Windows, ignore the test. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16045) Don't run TestDU on Windows
[ https://issues.apache.org/jira/browse/HADOOP-16045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740899#comment-16740899 ] Lukas Majercak commented on HADOOP-16045: - Windows uses WindowsGetSpaceUsed > Don't run TestDU on Windows > --- > > Key: HADOOP-16045 > URL: https://issues.apache.org/jira/browse/HADOOP-16045 > Project: Hadoop Common > Issue Type: Sub-task > Components: common, test >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Trivial > Attachments: HADOOP-16045.001.patch > > > DU is not supported on Windows, ignore the test. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16045) Don't run TestDU on Windows
[ https://issues.apache.org/jira/browse/HADOOP-16045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Majercak updated HADOOP-16045: Status: Patch Available (was: Open) > Don't run TestDU on Windows > --- > > Key: HADOOP-16045 > URL: https://issues.apache.org/jira/browse/HADOOP-16045 > Project: Hadoop Common > Issue Type: Bug > Components: common, test >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Trivial > Attachments: HADOOP-16045.001.patch > > > DU is not supported on Windows, ignore the test. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16045) Don't run TestDU on Windows
[
https://issues.apache.org/jira/browse/HADOOP-16045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740958#comment-16740958
]
Hadoop QA commented on HADOOP-16045:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 24s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 22s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
6s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 96m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-16045 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12954651/HADOOP-16045.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 76fe673a3c71 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug
17 11:07:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / bf08f4a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15779/testReport/ |
| Max. process+thread count | 1510 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15779/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Don't run TestDU on Windows
> ---
>
> Key: HADOOP-16045
> URL: https://is
[jira] [Commented] (HADOOP-16045) Don't run TestDU on Windows
[ https://issues.apache.org/jira/browse/HADOOP-16045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740961#comment-16740961 ] Íñigo Goiri commented on HADOOP-16045: -- +1 on [^HADOOP-16045.001.patch]. Committing all the way to branch-2.9. > Don't run TestDU on Windows > --- > > Key: HADOOP-16045 > URL: https://issues.apache.org/jira/browse/HADOOP-16045 > Project: Hadoop Common > Issue Type: Sub-task > Components: common, test >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Trivial > Attachments: HADOOP-16045.001.patch > > > DU is not supported on Windows, ignore the test. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16045) Don't run TestDU on Windows
[ https://issues.apache.org/jira/browse/HADOOP-16045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HADOOP-16045: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.1.3 3.2.1 3.3.0 3.0.4 Status: Resolved (was: Patch Available) > Don't run TestDU on Windows > --- > > Key: HADOOP-16045 > URL: https://issues.apache.org/jira/browse/HADOOP-16045 > Project: Hadoop Common > Issue Type: Sub-task > Components: common, test >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Trivial > Fix For: 3.0.4, 3.3.0, 3.2.1, 3.1.3 > > Attachments: HADOOP-16045.001.patch > > > DU is not supported on Windows, ignore the test. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16045) Don't run TestDU on Windows
[ https://issues.apache.org/jira/browse/HADOOP-16045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740972#comment-16740972 ] Íñigo Goiri commented on HADOOP-16045: -- Thanks [~lukmajercak] for the patch. Committed to trunk, branch-3.2, branch-3.1, and branch-3.0. Apparently, it does not apply to branch-2. [~lukmajercak] can you provide a patch there if so? > Don't run TestDU on Windows > --- > > Key: HADOOP-16045 > URL: https://issues.apache.org/jira/browse/HADOOP-16045 > Project: Hadoop Common > Issue Type: Sub-task > Components: common, test >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Trivial > Fix For: 3.0.4, 3.3.0, 3.2.1, 3.1.3 > > Attachments: HADOOP-16045.001.patch > > > DU is not supported on Windows, ignore the test. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16045) Don't run TestDU on Windows
[ https://issues.apache.org/jira/browse/HADOOP-16045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740977#comment-16740977 ] Hudson commented on HADOOP-16045: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15766 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15766/]) HADOOP-16045. Don't run TestDU on Windows. Contributed by Lukas (inigoiri: rev 35fa3bd685605d8b3639e6c5cbe83cd9acd8cbe7) * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDU.java > Don't run TestDU on Windows > --- > > Key: HADOOP-16045 > URL: https://issues.apache.org/jira/browse/HADOOP-16045 > Project: Hadoop Common > Issue Type: Sub-task > Components: common, test >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Trivial > Fix For: 3.0.4, 3.3.0, 3.2.1, 3.1.3 > > Attachments: HADOOP-16045.001.patch > > > DU is not supported on Windows, ignore the test. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] elek removed a comment on issue #143: YARN-4597: initial commit
elek removed a comment on issue #143: YARN-4597: initial commit URL: https://github.com/apache/hadoop/pull/143#issuecomment-453179327 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] elek commented on issue #143: YARN-4597: initial commit
elek commented on issue #143: YARN-4597: initial commit URL: https://github.com/apache/hadoop/pull/143#issuecomment-453721648 Sorry for the noise, It was just accidentally picked by a jenkins job which didn't work as expected. But as it's already committed: I will close this PR if no objections This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] elek removed a comment on issue #7: YARN-1964 Launching containers from docker
elek removed a comment on issue #7: YARN-1964 Launching containers from docker URL: https://github.com/apache/hadoop/pull/7#issuecomment-453180646 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] elek removed a comment on issue #1: MAPREDUCE-6096.SummarizedJob Class Improvment
elek removed a comment on issue #1: MAPREDUCE-6096.SummarizedJob Class Improvment URL: https://github.com/apache/hadoop/pull/1#issuecomment-453180725 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] elek removed a comment on issue #3: [HADOOP-10724] better interoperation with `sort -h`
elek removed a comment on issue #3: [HADOOP-10724] better interoperation with `sort -h` URL: https://github.com/apache/hadoop/pull/3#issuecomment-453180710 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] elek removed a comment on issue #6: YARN-1964 Launching containers from docker
elek removed a comment on issue #6: YARN-1964 Launching containers from docker URL: https://github.com/apache/hadoop/pull/6#issuecomment-453180669 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] elek removed a comment on issue #4: Update TaskInputOutputContext.java javadoc
elek removed a comment on issue #4: Update TaskInputOutputContext.java javadoc URL: https://github.com/apache/hadoop/pull/4#issuecomment-453180687 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16046) [JDK 11] hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/hamlet classes make compilation fail
Devaraj K created HADOOP-16046:
--
Summary: [JDK 11]
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/hamlet
classes make compilation fail
Key: HADOOP-16046
URL: https://issues.apache.org/jira/browse/HADOOP-16046
Project: Hadoop Common
Issue Type: Sub-task
Components: build
Reporter: Devaraj K
I see that YARN-8123 handled it but still see that this fails with "mvn
compile".
{code:xml}
[ERROR] COMPILATION ERROR :
[INFO] -
[ERROR]
/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/hamlet/HamletSpec.java:[306,20]
as of release 9, '_' is a keyword, and may not be used as an identifier
...
{code}
{code:xml}
[ERROR]
/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/webapp/hamlet/TestHamlet.java:[40,24]
as of release 9, '_' is a keyword, and may not be used as an identifier
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15994) Upgrade Jackson2 to 2.9.8
[ https://issues.apache.org/jira/browse/HADOOP-15994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16741025#comment-16741025 ] Akira Ajisaka commented on HADOOP-15994: +1, committing this. > Upgrade Jackson2 to 2.9.8 > - > > Key: HADOOP-15994 > URL: https://issues.apache.org/jira/browse/HADOOP-15994 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Akira Ajisaka >Assignee: lqjacklee >Priority: Major > Attachments: 460.patch, HADOOP-15994-001.patch, > HADOOP-15994-002.patch, HADOOP-15994-003.patch > > > Now Jackson 2.9.5 is used and it is vulnerable (CVE-2018-11307). Let's > upgrade to the latest version. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15994) Upgrade Jackson2 to 2.9.8
[ https://issues.apache.org/jira/browse/HADOOP-15994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-15994: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.2.1 3.3.0 Status: Resolved (was: Patch Available) Committed this to trunk and branch-3.2. Thanks [~Jack-Lee] for the contribution! > Upgrade Jackson2 to 2.9.8 > - > > Key: HADOOP-15994 > URL: https://issues.apache.org/jira/browse/HADOOP-15994 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Akira Ajisaka >Assignee: lqjacklee >Priority: Major > Fix For: 3.3.0, 3.2.1 > > Attachments: 460.patch, HADOOP-15994-001.patch, > HADOOP-15994-002.patch, HADOOP-15994-003.patch > > > Now Jackson 2.9.5 is used and it is vulnerable (CVE-2018-11307). Let's > upgrade to the latest version. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15994) Upgrade Jackson2 to 2.9.8
[ https://issues.apache.org/jira/browse/HADOOP-15994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16741128#comment-16741128 ] Hudson commented on HADOOP-15994: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15767 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15767/]) HADOOP-15994. Upgrade Jackson2 to 2.9.8. Contributed by lqjacklee. (aajisaka: rev 3bb745df18669e2ae400dc0d1a37a81cdc270eb2) * (edit) hadoop-project/pom.xml > Upgrade Jackson2 to 2.9.8 > - > > Key: HADOOP-15994 > URL: https://issues.apache.org/jira/browse/HADOOP-15994 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Akira Ajisaka >Assignee: lqjacklee >Priority: Major > Fix For: 3.3.0, 3.2.1 > > Attachments: 460.patch, HADOOP-15994-001.patch, > HADOOP-15994-002.patch, HADOOP-15994-003.patch > > > Now Jackson 2.9.5 is used and it is vulnerable (CVE-2018-11307). Let's > upgrade to the latest version. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
