[jira] [Comment Edited] (HADOOP-15621) s3guard: implement time-based (TTL) expiry for DynamoDB Metadata Store
[
https://issues.apache.org/jira/browse/HADOOP-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624144#comment-16624144
]
Gabor Bota edited comment on HADOOP-15621 at 9/21/18 8:51 PM:
--
Just a note: There is no call from {{S3AFileSytem}} to {{MetadataStore#put}}
directly, only from {{S3Guard}} (with static methods).
Both {{S3Guard}} AND {{S3AFileSytem}} should be analyzed more closely how it
interacts with {{MetadataStore}}, and all [get, listChildren, put, move]
interactions should be modified to handle TTL.
Also note: we have usages from S3GuardTool as well, directly calling
MetadataStore#put. We need to handle those as well.
was (Author: gabor.bota):
Just a note: There is no call from {{S3AFileSytem}} to {{MetadataStore#put}}
directly, only from {{S3Guard}} (with static methods).
Both {{S3Guard}} AND {{S3AFileSytem}} should be analyzed more closely how it
interacts with {{MetadataStore}}, and all [get, listChildren, put, move]
interactions should be modified to handle TTL.
> s3guard: implement time-based (TTL) expiry for DynamoDB Metadata Store
> --
>
> Key: HADOOP-15621
> URL: https://issues.apache.org/jira/browse/HADOOP-15621
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Aaron Fabbri
>Assignee: Gabor Bota
>Priority: Minor
> Attachments: HADOOP-15621.001.patch
>
>
> Similar to HADOOP-13649, I think we should add a TTL (time to live) feature
> to the Dynamo metadata store (MS) for S3Guard.
> This is a similar concept to an "online algorithm" version of the CLI prune()
> function, which is the "offline algorithm".
> Why:
> 1. Self healing (soft state): since we do not implement transactions around
> modification of the two systems (s3 and metadata store), certain failures can
> lead to inconsistency between S3 and the metadata store (MS) state. Having a
> time to live (TTL) on each entry in S3Guard means that any inconsistencies
> will be time bound. Thus "wait and restart your job" becomes a valid, if
> ugly, way to get around any issues with FS client failure leaving things in a
> bad state.
> 2. We could make manual invocation of `hadoop s3guard prune ...`
> unnecessary, depending on the implementation.
> 3. Makes it possible to fix the problem that dynamo MS prune() doesn't prune
> directories due to the lack of true modification time.
> How:
> I think we need a new column in the dynamo table "entry last written time".
> This is updated each time the entry is written to dynamo.
> After that we can either
> 1. Have the client simply ignore / elide any entries that are older than the
> configured TTL.
> 2. Have the client delete entries older than the TTL.
> The issue with #2 is it will increase latency if done inline in the context
> of an FS operation. We could mitigate this some by using an async helper
> thread, or probabilistically doing it "some times" to amortize the expense of
> deleting stale entries (allowing some batching as well).
> Caveats:
> - Clock synchronization as usual is a concern. Many clusters already keep
> clocks close enough via NTP. We should at least document the requirement
> along with the configuration knob that enables the feature.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15621) s3guard: implement time-based (TTL) expiry for DynamoDB Metadata Store
[
https://issues.apache.org/jira/browse/HADOOP-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624144#comment-16624144
]
Gabor Bota edited comment on HADOOP-15621 at 9/21/18 8:51 PM:
--
There is no call from {{S3AFileSytem}} to {{MetadataStore#put}} directly, only
from {{S3Guard}} (with static methods).
Both {{S3Guard}} AND {{S3AFileSytem}} should be analyzed more closely how it
interacts with {{MetadataStore}}, and all [get, listChildren, put, move]
interactions should be modified to handle TTL.
Also note: we have usages from S3GuardTool as well, directly calling
MetadataStore#put. We need to handle those as well.
was (Author: gabor.bota):
Just a note: There is no call from {{S3AFileSytem}} to {{MetadataStore#put}}
directly, only from {{S3Guard}} (with static methods).
Both {{S3Guard}} AND {{S3AFileSytem}} should be analyzed more closely how it
interacts with {{MetadataStore}}, and all [get, listChildren, put, move]
interactions should be modified to handle TTL.
Also note: we have usages from S3GuardTool as well, directly calling
MetadataStore#put. We need to handle those as well.
> s3guard: implement time-based (TTL) expiry for DynamoDB Metadata Store
> --
>
> Key: HADOOP-15621
> URL: https://issues.apache.org/jira/browse/HADOOP-15621
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Aaron Fabbri
>Assignee: Gabor Bota
>Priority: Minor
> Attachments: HADOOP-15621.001.patch
>
>
> Similar to HADOOP-13649, I think we should add a TTL (time to live) feature
> to the Dynamo metadata store (MS) for S3Guard.
> This is a similar concept to an "online algorithm" version of the CLI prune()
> function, which is the "offline algorithm".
> Why:
> 1. Self healing (soft state): since we do not implement transactions around
> modification of the two systems (s3 and metadata store), certain failures can
> lead to inconsistency between S3 and the metadata store (MS) state. Having a
> time to live (TTL) on each entry in S3Guard means that any inconsistencies
> will be time bound. Thus "wait and restart your job" becomes a valid, if
> ugly, way to get around any issues with FS client failure leaving things in a
> bad state.
> 2. We could make manual invocation of `hadoop s3guard prune ...`
> unnecessary, depending on the implementation.
> 3. Makes it possible to fix the problem that dynamo MS prune() doesn't prune
> directories due to the lack of true modification time.
> How:
> I think we need a new column in the dynamo table "entry last written time".
> This is updated each time the entry is written to dynamo.
> After that we can either
> 1. Have the client simply ignore / elide any entries that are older than the
> configured TTL.
> 2. Have the client delete entries older than the TTL.
> The issue with #2 is it will increase latency if done inline in the context
> of an FS operation. We could mitigate this some by using an async helper
> thread, or probabilistically doing it "some times" to amortize the expense of
> deleting stale entries (allowing some batching as well).
> Caveats:
> - Clock synchronization as usual is a concern. Many clusters already keep
> clocks close enough via NTP. We should at least document the requirement
> along with the configuration knob that enables the feature.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15621) s3guard: implement time-based (TTL) expiry for DynamoDB Metadata Store
[
https://issues.apache.org/jira/browse/HADOOP-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624144#comment-16624144
]
Gabor Bota edited comment on HADOOP-15621 at 9/21/18 8:44 PM:
--
Just a note: There is no call from {{S3AFileSytem}} to {{MetadataStore#put}}
directly, only from {{S3Guard}} (with static methods).
Both {{S3Guard}} AND {{S3AFileSytem}} should be analyzed more closely how it
interacts with {{MetadataStore}}, and all {get, listChildren, put, move}
interactions should be modified to handle TTL.
was (Author: gabor.bota):
Just a note: There is no call from {{S3AFileSytem}} to {{MetadataStore#put}}
directly, only from {{S3Guard}} (with static methods).
Both {{S3Guard}} AND {{S3AFileSytem}} should be analyzed more closely how it
interacts with {{MetadataStore}}, and all interactions should be modified to
handle TTL.
> s3guard: implement time-based (TTL) expiry for DynamoDB Metadata Store
> --
>
> Key: HADOOP-15621
> URL: https://issues.apache.org/jira/browse/HADOOP-15621
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Aaron Fabbri
>Assignee: Gabor Bota
>Priority: Minor
> Attachments: HADOOP-15621.001.patch
>
>
> Similar to HADOOP-13649, I think we should add a TTL (time to live) feature
> to the Dynamo metadata store (MS) for S3Guard.
> This is a similar concept to an "online algorithm" version of the CLI prune()
> function, which is the "offline algorithm".
> Why:
> 1. Self healing (soft state): since we do not implement transactions around
> modification of the two systems (s3 and metadata store), certain failures can
> lead to inconsistency between S3 and the metadata store (MS) state. Having a
> time to live (TTL) on each entry in S3Guard means that any inconsistencies
> will be time bound. Thus "wait and restart your job" becomes a valid, if
> ugly, way to get around any issues with FS client failure leaving things in a
> bad state.
> 2. We could make manual invocation of `hadoop s3guard prune ...`
> unnecessary, depending on the implementation.
> 3. Makes it possible to fix the problem that dynamo MS prune() doesn't prune
> directories due to the lack of true modification time.
> How:
> I think we need a new column in the dynamo table "entry last written time".
> This is updated each time the entry is written to dynamo.
> After that we can either
> 1. Have the client simply ignore / elide any entries that are older than the
> configured TTL.
> 2. Have the client delete entries older than the TTL.
> The issue with #2 is it will increase latency if done inline in the context
> of an FS operation. We could mitigate this some by using an async helper
> thread, or probabilistically doing it "some times" to amortize the expense of
> deleting stale entries (allowing some batching as well).
> Caveats:
> - Clock synchronization as usual is a concern. Many clusters already keep
> clocks close enough via NTP. We should at least document the requirement
> along with the configuration knob that enables the feature.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15621) s3guard: implement time-based (TTL) expiry for DynamoDB Metadata Store
[
https://issues.apache.org/jira/browse/HADOOP-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624144#comment-16624144
]
Gabor Bota edited comment on HADOOP-15621 at 9/21/18 8:44 PM:
--
Just a note: There is no call from {{S3AFileSytem}} to {{MetadataStore#put}}
directly, only from {{S3Guard}} (with static methods).
Both {{S3Guard}} AND {{S3AFileSytem}} should be analyzed more closely how it
interacts with {{MetadataStore}}, and all [get, listChildren, put, move]
interactions should be modified to handle TTL.
was (Author: gabor.bota):
Just a note: There is no call from {{S3AFileSytem}} to {{MetadataStore#put}}
directly, only from {{S3Guard}} (with static methods).
Both {{S3Guard}} AND {{S3AFileSytem}} should be analyzed more closely how it
interacts with {{MetadataStore}}, and all {get, listChildren, put, move}
interactions should be modified to handle TTL.
> s3guard: implement time-based (TTL) expiry for DynamoDB Metadata Store
> --
>
> Key: HADOOP-15621
> URL: https://issues.apache.org/jira/browse/HADOOP-15621
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Aaron Fabbri
>Assignee: Gabor Bota
>Priority: Minor
> Attachments: HADOOP-15621.001.patch
>
>
> Similar to HADOOP-13649, I think we should add a TTL (time to live) feature
> to the Dynamo metadata store (MS) for S3Guard.
> This is a similar concept to an "online algorithm" version of the CLI prune()
> function, which is the "offline algorithm".
> Why:
> 1. Self healing (soft state): since we do not implement transactions around
> modification of the two systems (s3 and metadata store), certain failures can
> lead to inconsistency between S3 and the metadata store (MS) state. Having a
> time to live (TTL) on each entry in S3Guard means that any inconsistencies
> will be time bound. Thus "wait and restart your job" becomes a valid, if
> ugly, way to get around any issues with FS client failure leaving things in a
> bad state.
> 2. We could make manual invocation of `hadoop s3guard prune ...`
> unnecessary, depending on the implementation.
> 3. Makes it possible to fix the problem that dynamo MS prune() doesn't prune
> directories due to the lack of true modification time.
> How:
> I think we need a new column in the dynamo table "entry last written time".
> This is updated each time the entry is written to dynamo.
> After that we can either
> 1. Have the client simply ignore / elide any entries that are older than the
> configured TTL.
> 2. Have the client delete entries older than the TTL.
> The issue with #2 is it will increase latency if done inline in the context
> of an FS operation. We could mitigate this some by using an async helper
> thread, or probabilistically doing it "some times" to amortize the expense of
> deleting stale entries (allowing some batching as well).
> Caveats:
> - Clock synchronization as usual is a concern. Many clusters already keep
> clocks close enough via NTP. We should at least document the requirement
> along with the configuration knob that enables the feature.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15621) s3guard: implement time-based (TTL) expiry for DynamoDB Metadata Store
[
https://issues.apache.org/jira/browse/HADOOP-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624144#comment-16624144
]
Gabor Bota edited comment on HADOOP-15621 at 9/21/18 8:40 PM:
--
Just a note: There is no call from {{S3AFileSytem}} to {{MetadataStore#put}}
directly, only from {{S3Guard}} (with static methods).
Both {{S3Guard}} AND {{S3AFileSytem}} should be analyzed more closely how it
interacts with {{MetadataStore}}, and all interactions should be modified to
handle TTL.
was (Author: gabor.bota):
Just a note: There is no call from {{S3AFileSytem}} to {{MetadataStore#put}}
directly, only from {{S3Guard}} (with static methods).
{{S3Guard}} should be analyzed more closely how it interacts with
{{MetadataStore}}, and all interactions should be modified to handle TTL.
> s3guard: implement time-based (TTL) expiry for DynamoDB Metadata Store
> --
>
> Key: HADOOP-15621
> URL: https://issues.apache.org/jira/browse/HADOOP-15621
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Aaron Fabbri
>Assignee: Gabor Bota
>Priority: Minor
> Attachments: HADOOP-15621.001.patch
>
>
> Similar to HADOOP-13649, I think we should add a TTL (time to live) feature
> to the Dynamo metadata store (MS) for S3Guard.
> This is a similar concept to an "online algorithm" version of the CLI prune()
> function, which is the "offline algorithm".
> Why:
> 1. Self healing (soft state): since we do not implement transactions around
> modification of the two systems (s3 and metadata store), certain failures can
> lead to inconsistency between S3 and the metadata store (MS) state. Having a
> time to live (TTL) on each entry in S3Guard means that any inconsistencies
> will be time bound. Thus "wait and restart your job" becomes a valid, if
> ugly, way to get around any issues with FS client failure leaving things in a
> bad state.
> 2. We could make manual invocation of `hadoop s3guard prune ...`
> unnecessary, depending on the implementation.
> 3. Makes it possible to fix the problem that dynamo MS prune() doesn't prune
> directories due to the lack of true modification time.
> How:
> I think we need a new column in the dynamo table "entry last written time".
> This is updated each time the entry is written to dynamo.
> After that we can either
> 1. Have the client simply ignore / elide any entries that are older than the
> configured TTL.
> 2. Have the client delete entries older than the TTL.
> The issue with #2 is it will increase latency if done inline in the context
> of an FS operation. We could mitigate this some by using an async helper
> thread, or probabilistically doing it "some times" to amortize the expense of
> deleting stale entries (allowing some batching as well).
> Caveats:
> - Clock synchronization as usual is a concern. Many clusters already keep
> clocks close enough via NTP. We should at least document the requirement
> along with the configuration knob that enables the feature.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15621) s3guard: implement time-based (TTL) expiry for DynamoDB Metadata Store
[
https://issues.apache.org/jira/browse/HADOOP-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624144#comment-16624144
]
Gabor Bota commented on HADOOP-15621:
-
Just a note: There is no call from {{S3AFileSytem}} to {{MetadataStore#put}}
directly, only from {{S3Guard}} (with static methods).
{{S3Guard}} should be analyzed more closely how it interacts with
{{MetadataStore}}, and all interactions should be modified to handle TTL.
> s3guard: implement time-based (TTL) expiry for DynamoDB Metadata Store
> --
>
> Key: HADOOP-15621
> URL: https://issues.apache.org/jira/browse/HADOOP-15621
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Aaron Fabbri
>Assignee: Gabor Bota
>Priority: Minor
> Attachments: HADOOP-15621.001.patch
>
>
> Similar to HADOOP-13649, I think we should add a TTL (time to live) feature
> to the Dynamo metadata store (MS) for S3Guard.
> This is a similar concept to an "online algorithm" version of the CLI prune()
> function, which is the "offline algorithm".
> Why:
> 1. Self healing (soft state): since we do not implement transactions around
> modification of the two systems (s3 and metadata store), certain failures can
> lead to inconsistency between S3 and the metadata store (MS) state. Having a
> time to live (TTL) on each entry in S3Guard means that any inconsistencies
> will be time bound. Thus "wait and restart your job" becomes a valid, if
> ugly, way to get around any issues with FS client failure leaving things in a
> bad state.
> 2. We could make manual invocation of `hadoop s3guard prune ...`
> unnecessary, depending on the implementation.
> 3. Makes it possible to fix the problem that dynamo MS prune() doesn't prune
> directories due to the lack of true modification time.
> How:
> I think we need a new column in the dynamo table "entry last written time".
> This is updated each time the entry is written to dynamo.
> After that we can either
> 1. Have the client simply ignore / elide any entries that are older than the
> configured TTL.
> 2. Have the client delete entries older than the TTL.
> The issue with #2 is it will increase latency if done inline in the context
> of an FS operation. We could mitigate this some by using an async helper
> thread, or probabilistically doing it "some times" to amortize the expense of
> deleting stale entries (allowing some batching as well).
> Caveats:
> - Clock synchronization as usual is a concern. Many clusters already keep
> clocks close enough via NTP. We should at least document the requirement
> along with the configuration knob that enables the feature.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15621) s3guard: implement time-based (TTL) expiry for DynamoDB Metadata Store
[ https://issues.apache.org/jira/browse/HADOOP-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607040#comment-16607040 ] Gabor Bota edited comment on HADOOP-15621 at 9/21/18 8:31 PM: -- Based on your comments [~fabbri] I think we could rename this jira. It's more like a TTL for authoritative directory listings and files for all metadata store rather than a ttl expiry for dynamodb. was (Author: gabor.bota): Based on your comments [~fabbri] I think we could rename this jira. It's more like a TTL for authoritative directory listings for all metadata store rather than a ttl expiry for dynamodb. > s3guard: implement time-based (TTL) expiry for DynamoDB Metadata Store > -- > > Key: HADOOP-15621 > URL: https://issues.apache.org/jira/browse/HADOOP-15621 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-15621.001.patch > > > Similar to HADOOP-13649, I think we should add a TTL (time to live) feature > to the Dynamo metadata store (MS) for S3Guard. > This is a similar concept to an "online algorithm" version of the CLI prune() > function, which is the "offline algorithm". > Why: > 1. Self healing (soft state): since we do not implement transactions around > modification of the two systems (s3 and metadata store), certain failures can > lead to inconsistency between S3 and the metadata store (MS) state. Having a > time to live (TTL) on each entry in S3Guard means that any inconsistencies > will be time bound. Thus "wait and restart your job" becomes a valid, if > ugly, way to get around any issues with FS client failure leaving things in a > bad state. > 2. We could make manual invocation of `hadoop s3guard prune ...` > unnecessary, depending on the implementation. > 3. Makes it possible to fix the problem that dynamo MS prune() doesn't prune > directories due to the lack of true modification time. > How: > I think we need a new column in the dynamo table "entry last written time". > This is updated each time the entry is written to dynamo. > After that we can either > 1. Have the client simply ignore / elide any entries that are older than the > configured TTL. > 2. Have the client delete entries older than the TTL. > The issue with #2 is it will increase latency if done inline in the context > of an FS operation. We could mitigate this some by using an async helper > thread, or probabilistically doing it "some times" to amortize the expense of > deleting stale entries (allowing some batching as well). > Caveats: > - Clock synchronization as usual is a concern. Many clusters already keep > clocks close enough via NTP. We should at least document the requirement > along with the configuration knob that enables the feature. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15774) Discovery of HA servers
[
https://issues.apache.org/jira/browse/HADOOP-15774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624003#comment-16624003
]
Anu Engineer commented on HADOOP-15774:
---
{quote}[~anu] any thoughts on the general approach? I'm not very familiar with
the OZone architecture; do you guys also need to specify servers manually?
{quote}
[~elgoiri] Thanks for posting the design docs. My apologies for the delayed
response, been busy with Ozone Alpha release.
I have proposed the same idea a couple of times internally. Each time it got
shot down because of the issue that [~ste...@apache.org] is referring to.
{quote}Being part of the YARN may bring some weird dependencies (HDFS->YARN),
any ideas here? Should it be move to a separate place? Maybe parts?
{quote}
The most important concern was always HDFS taking a dependency on YARN. It
creates a cyclical dependency.
Ozone needs and plans to build something very similar to this, but our thought
has been to create something like this in SCM.
For Hadoop and also from Ozone perspective, The consideration for such a
service should be slightly different. It is possible for someone to run a
cluster without HDFS or even without YARN. A Discovery service should be
independent of these specific services.
If we are planning to make this a Hadoop level discovery service we should
probably build this independent of all other services. If possible, even
independent of ZooKeeper.
Then this service can be the bootup service, and all other services including
HDFS, YARN and even Zookeeper can use this service to discover endpoints. Many
services need to find the address of zookeeper too.
If this service can be built independent of all other services, Ozone can
certainly use this, and we don't need to reinvent discovery.
Another critical issue that Ozone struggles with this the issue of config
changes – for example, in the HA case if a server fails and a new server is
added Datanodes need to be told of that.
Ozone solves it by reading these changes from SCM, not only config via
Heartbeat.
But this problem is more generic, in the sense that there is a class of
configuration changes that need to be pushed to all entities in the cluster. We
have been thinking about building a discovery and config store for Ozone.
> Discovery of HA servers
> ---
>
> Key: HADOOP-15774
> URL: https://issues.apache.org/jira/browse/HADOOP-15774
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Íñigo Goiri
>Priority: Major
> Attachments: Discovery Service.pdf
>
>
> Currently, Hadoop relies on configuration files to specify the servers.
> This requires maintaining these configuration files and propagating the
> changes.
> Hadoop should have a framework to provide discovery.
> For example, in HDFS, we could define the Namenodes in a shared location and
> the DNs would use the framework to find the Namenodes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15775) [JDK9] Add missing javax.activation-api dependency
[
https://issues.apache.org/jira/browse/HADOOP-15775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623975#comment-16623975
]
Andrew Purtell commented on HADOOP-15775:
-
FWIW the 003 patch doesn't address some cases, like
TestWebHDFSStoragePolicyCommands. You need to specify this dependency in the
hadoop-hdfs-project POM too.
Cause is:
{noformat}
Caused by: java.lang.ClassNotFoundException: javax.activation.DataSource
at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)
at
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
{noformat}
This is Java 11+28
> [JDK9] Add missing javax.activation-api dependency
> --
>
> Key: HADOOP-15775
> URL: https://issues.apache.org/jira/browse/HADOOP-15775
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: test
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Critical
> Attachments: HADOOP-15775.01.patch, HADOOP-15775.02.patch,
> HADOOP-15775.03.patch
>
>
> Many unit tests fail due to missing java.activation module. This failure can
> be fixed by adding javax.activation-api as third-party dependency.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15775) [JDK9] Add missing javax.activation-api dependency
[
https://issues.apache.org/jira/browse/HADOOP-15775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623975#comment-16623975
]
Andrew Purtell edited comment on HADOOP-15775 at 9/21/18 6:02 PM:
--
FWIW the 003 patch doesn't address some cases, like
TestWebHDFSStoragePolicyCommands. You need to specify this dependency in the
hadoop-hdfs-project POM too.
Cause for TestWebHDFSStoragePolicyCommands failure (and others) is:
{noformat}
Caused by: java.lang.ClassNotFoundException: javax.activation.DataSource
at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)
at
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
{noformat}
This is Java 11+28
was (Author: apurtell):
FWIW the 003 patch doesn't address some cases, like
TestWebHDFSStoragePolicyCommands. You need to specify this dependency in the
hadoop-hdfs-project POM too.
Cause is:
{noformat}
Caused by: java.lang.ClassNotFoundException: javax.activation.DataSource
at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)
at
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
{noformat}
This is Java 11+28
> [JDK9] Add missing javax.activation-api dependency
> --
>
> Key: HADOOP-15775
> URL: https://issues.apache.org/jira/browse/HADOOP-15775
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: test
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Critical
> Attachments: HADOOP-15775.01.patch, HADOOP-15775.02.patch,
> HADOOP-15775.03.patch
>
>
> Many unit tests fail due to missing java.activation module. This failure can
> be fixed by adding javax.activation-api as third-party dependency.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15711) Fix branch-2 builds
[
https://issues.apache.org/jira/browse/HADOOP-15711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623935#comment-16623935
]
Konstantin Shvachko commented on HADOOP-15711:
--
[~aw] thanks a lot for your insight, this helps understanding the problem and
differences between the branches.
Looked at the PreCommit runs with the attached patch more closely. Definitely
some cheating there, as many tests were not executed at all (Just as [~aw]
commented). This explains fast running time.
I also made Jenkins [execute branch-3.1
build|https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86-jhung/12/console],
which completed in 12 hours.
I don't see any shortcut solutions to the branch-2 build problem at this point.
We should probably take the long path and understand the fixes that were done
to unit tests on branch-3, that were not backported to branch-2.
> Fix branch-2 builds
> ---
>
> Key: HADOOP-15711
> URL: https://issues.apache.org/jira/browse/HADOOP-15711
> Project: Hadoop Common
> Issue Type: Task
>Reporter: Jonathan Hung
>Priority: Critical
> Attachments: HADOOP-15711.001.branch-2.patch
>
>
> Branch-2 builds have been disabled for a while:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/
> A test run here causes hdfs tests to hang:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86-jhung/4/
> Running hadoop-hdfs tests locally reveal some errors such
> as:{noformat}[ERROR]
> testComplexAppend2(org.apache.hadoop.hdfs.TestFileAppend2) Time elapsed:
> 0.059 s <<< ERROR!
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1164)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1128)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:174)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1172)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:403)
> at
> org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:234)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1080)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:883)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:514)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:473)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend(TestFileAppend2.java:489)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend2(TestFileAppend2.java:543)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){noformat}
> I was able to get more tests passing locally by increasing the max user
> process count on my machine. But the error suggests that there's an issue in
> the tests themselves. Not sure if the error seen locally is the same reason
> as why jenkins builds are failing, I wasn't able to confirm based on the
> jenkins builds' lack of output.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15723) ABFS: Ranger Support
[ https://issues.apache.org/jira/browse/HADOOP-15723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623926#comment-16623926 ] Da Zhou commented on HADOOP-15723: -- The error indicates the reason: "URI scheme must be *abfss* when authenticating using Oauth", "ITestAbfsFileSystemContractDistCp" is only used to test in secure mode (abfs://XXX), so when using Oauth (Oauth only support abfss://), "ITestAbfsFileSystemContractDistCp" should be skipped. In future, we should merge "ITestAbfsFileSystemContractDistCp" and "ITestAbfsFileSystemContractSecureDistCp" since they are running same tests but in different mode. But for now, you can just skip this test set by checking if it is using Oauth Type. > ABFS: Ranger Support > > > Key: HADOOP-15723 > URL: https://issues.apache.org/jira/browse/HADOOP-15723 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: HADOOP-15407 >Reporter: Thomas Marquardt >Assignee: Yuan Gao >Priority: Major > Attachments: HADOOP-15273-HADOOP-15407-001.patch, > HADOOP-15273-HADOOP-15407-002.patch, HADOOP-15273-HADOOP-15407-003.patch, > HADOOP-15273-HADOOP-15407-004.patch > > > Add support for Ranger -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15723) ABFS: Ranger Support
[ https://issues.apache.org/jira/browse/HADOOP-15723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623926#comment-16623926 ] Da Zhou edited comment on HADOOP-15723 at 9/21/18 5:35 PM: --- The error indicates the reason: "URI scheme must be *abfss* when authenticating using Oauth", "ITestAbfsFileSystemContractDistCp" is only used to test in secure mode (abfs://XXX), so when using Oauth (Oauth only support abfss://), "ITestAbfsFileSystemContractDistCp" should be skipped. In future, we should merge "ITestAbfsFileSystemContractDistCp" and "ITestAbfsFileSystemContractSecureDistCp" since they are running same tests but in different mode. But for now, you can just skip this test set by checking if it is using Oauth authentication. was (Author: danielzhou): The error indicates the reason: "URI scheme must be *abfss* when authenticating using Oauth", "ITestAbfsFileSystemContractDistCp" is only used to test in secure mode (abfs://XXX), so when using Oauth (Oauth only support abfss://), "ITestAbfsFileSystemContractDistCp" should be skipped. In future, we should merge "ITestAbfsFileSystemContractDistCp" and "ITestAbfsFileSystemContractSecureDistCp" since they are running same tests but in different mode. But for now, you can just skip this test set by checking if it is using Oauth Type. > ABFS: Ranger Support > > > Key: HADOOP-15723 > URL: https://issues.apache.org/jira/browse/HADOOP-15723 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: HADOOP-15407 >Reporter: Thomas Marquardt >Assignee: Yuan Gao >Priority: Major > Attachments: HADOOP-15273-HADOOP-15407-001.patch, > HADOOP-15273-HADOOP-15407-002.patch, HADOOP-15273-HADOOP-15407-003.patch, > HADOOP-15273-HADOOP-15407-004.patch > > > Add support for Ranger -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15723) ABFS: Ranger Support
[
https://issues.apache.org/jira/browse/HADOOP-15723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623725#comment-16623725
]
Yuan Gao edited comment on HADOOP-15723 at 9/21/18 5:14 PM:
patch 004:
- called makeQualified() before authz checks
- moved configuration and factory method to AbfsConfiguration
- moved the authorization interface to org.apache.hadoop.fs.azurebfs.extensions
- renamed from adl to abfs
- added test cases for authz checks in ITestAzureBlobFileSystemAuthorization
I was able to run:
{{mvn -T 1C -Dparallel-tests -Dscale -DtestsThreadCount=8 clean verify}}
{{[INFO] Results:}}
{{[WARNING] Tests run: 277, Failures: 0, Errors: 0, Skipped: 11}}
But there are errors when running:
{{mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify}}
{{[ERROR] Tests run: 6, Failures: 0, Errors: 6, Skipped: 0, Time elapsed: 5.955
s <<< FAILURE! - in
org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDistCp}}
{{[ERROR]
testUpdateDeepDirectoryStructureNoChange(org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDistCp)
Time elapsed: 3.038 s <<< ERROR!}}
{{java.io.IOException: Unable to initialize filesystem, URI scheme must be
abfss when authenticating using Oauth.}}
was (Author: kowon2008):
patch 004:
- called makeQualified() before authz checks
- moved configuration and factory method to AbfsConfiguration
- moved the authorization interface to org.apache.hadoop.fs.azurebfs.extensions
- renamed from adl to abfs
- added test cases for authz checks in ITestAzureBlobFileSystemAuthorization
I was able to run:
{{mvn -T 1C -Dparallel-tests -Dscale -DtestsThreadCount=8 clean verify}}
{{[INFO] Results:}}
{{[WARNING] Tests run: 277, Failures: 0, Errors: 0, Skipped: 11}}
But there are errors when running:
{{mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify}}
{{[ERROR] Tests run: 6, Failures: 0, Errors: 6, Skipped: 0, Time elapsed: 5.955
s <<< FAILURE! - in
org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDistCp}}
{{[ERROR]
testUpdateDeepDirectoryStructureNoChange(org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDistCp)
Time elapsed: 3.038 s <<< ERROR!}}
{{java.io.IOException: Unable to initialize filesystem
abfs://dfsafast...@rajeevbstorageac.dfs.core.windows.net:
java.lang.IllegalArgumentException: Incorrect URI
abfs://dfsafast...@rajeevbstorageac.dfs.core.windows.net, URI scheme must be
abfss when authenticating using Oauth.}}
> ABFS: Ranger Support
>
>
> Key: HADOOP-15723
> URL: https://issues.apache.org/jira/browse/HADOOP-15723
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: HADOOP-15407
>Reporter: Thomas Marquardt
>Assignee: Yuan Gao
>Priority: Major
> Attachments: HADOOP-15273-HADOOP-15407-001.patch,
> HADOOP-15273-HADOOP-15407-002.patch, HADOOP-15273-HADOOP-15407-003.patch,
> HADOOP-15273-HADOOP-15407-004.patch
>
>
> Add support for Ranger
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15765) Can not find login module class for IBM due to hard codes
[ https://issues.apache.org/jira/browse/HADOOP-15765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623890#comment-16623890 ] Eric Yang commented on HADOOP-15765: [~jiangjianfei] How about IBM Java version detection to make decision which security class to use? I believe that was the approach taken by existing developers. > Can not find login module class for IBM due to hard codes > - > > Key: HADOOP-15765 > URL: https://issues.apache.org/jira/browse/HADOOP-15765 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.0.3 >Reporter: Jianfei Jiang >Priority: Major > Attachments: HADOOP-15765_000.patch > > > As the differences between various versions of IBM, the login module class is > sometimes different. However, the class for specified jdk (no matter the > version) is hard coded in Hadoop code. We have faced the error like following: > *javax.security.auth.login.LoginException: unable to find LoginModule class: > com.ibm.security.auth.module.LinuxLoginModule* > > Should we set the value as a config which can be set by users? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623885#comment-16623885
]
Hadoop QA commented on HADOOP-15691:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
58s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 46s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
28s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m
54s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 30s{color} | {color:orange} root: The patch generated 4 new + 525 unchanged
- 0 fixed = 529 total (was 525) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 12 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 34s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
58s{color} | {color:red} hadoop-common-project_hadoop-common generated 2 new +
0 unchanged - 0 fixed = 2 total (was 0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
27s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
40s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
58s{color} | {color:green} hadoop-hdfs-httpfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
38s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
14s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
55s{color} | {color:green} hadoop-azure-datalake in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}153m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce
[jira] [Commented] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[
https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623882#comment-16623882
]
Hadoop QA commented on HADOOP-15679:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m
3s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
47s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
20s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 36s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 115 unchanged - 1 fixed = 116 total (was 116) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
0s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 75m 24s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
44s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}140m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:a716388 |
| JIRA Issue | HADOOP-15679 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12940790/HADOOP-15679-branch-2-004.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 1408c8a2646c 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2 / 3a6ad9c |
| maven | version: Apache Maven 3.3.9
(bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00) |
| Default Java | 1.7.0_181 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15249/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15249/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15249/testReport/ |
| asflicense |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15249/artifact/out/patch-asflicense-problems.txt
|
| Max. process+thread count | 1361 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15249/console |
| Powered by | Apache Yetus 0.8.0
[jira] [Commented] (HADOOP-15774) Discovery of HA servers
[ https://issues.apache.org/jira/browse/HADOOP-15774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623860#comment-16623860 ] Eric Yang commented on HADOOP-15774: RegistryDNS is a tool to provide DNS resolution. If namenode hostname follows certain convention, then it might be possible to provide multi-A record to provide information about all namenodes. This is assuming that namdenode would know how to find out about location of ZooKeeper servers via configuration file to publish themselves. If ZooKeeper quorum can only be discovered by configuration, then there is still no real service discovery. It would be nice to move registryDNS from YARN to Hadoop common, but dynamically spawned ZooKeeper server is broken currently. ZooKeeper needs to be fixed in the following area to meet the requirements for service discovery for Hadoop core system: # ZooKeeper client session affinity. If ZooKeeper server spawned somewhere else, existing client still try to connect to old IP. ZOOKEEPER-2929 # ZooKeeper Kerberos security fails, but continue to allow ZooKeeper connections. ZOOKEEPER-1634 YARN usage of ZooKeeper and RegistryDNS are less prone to ZooKeeper defects because assumption of ZooKeeper deployment is static, and curator handles some odd conditions for ZooKeeper connections. Curator dependency might generate a wave of realignment for HDFS dependency on ZooKeeper to get subpar results. Alternative approach is to use multicast DNS for daemon process that supports service discovery. Similar to Apple products that can find out about devices in surrounding environment. There is a implementation of service discovery in [this project|https://github.com/macroadster/HMS/blob/master/beacon/src/main/java/org/apache/hms/beacon/Beacon.java] for ZooKeeper. This was canned 6 years ago when [~aw] and others provided feedback that said multicast DNS is too expensive and possible vulnerabilities. Many issues were addressed by multicast DNS implementations in the last 6 years. Not sure if it is worth while to revisit multicast DNS for service discovery purpose. DNS or multicast DNS are most likely the best approach for providing discovery of HA servers. However, the information are provided to be aware of potential pitfalls in the current implementations. > Discovery of HA servers > --- > > Key: HADOOP-15774 > URL: https://issues.apache.org/jira/browse/HADOOP-15774 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Íñigo Goiri >Priority: Major > Attachments: Discovery Service.pdf > > > Currently, Hadoop relies on configuration files to specify the servers. > This requires maintaining these configuration files and propagating the > changes. > Hadoop should have a framework to provide discovery. > For example, in HDFS, we could define the Namenodes in a shared location and > the DNs would use the framework to find the Namenodes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15723) ABFS: Ranger Support
[
https://issues.apache.org/jira/browse/HADOOP-15723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623805#comment-16623805
]
Steve Loughran commented on HADOOP-15723:
-
h3. ITestAzureBlobFileSystemAuthorization
* needs ASF header (as do the mock classes)
* tests should close the (opened) files after opening/creating/appending them.
you can simply go fs.open(path).close() for this
L129, L139: use delete(path, false) to eliminate deprecation warnings.
I don't think the unauthed tests are really stressing things properly, because
a test like {{testSetAclUnauthorized}} is going to fail on the fs.create(),
before the setAcl() call goes through.
I'd recommend
* having the normal setup() in place, with the FS created there being
authorized for full R/W
* Creating a second the other filesystems (readOnly, writeOnly) filesystems in
test setup too; closing them in teardown.
* in each test, creating all files in the r/w fs
* and only use the readonly FS for those FS operations where failures are
expected.
{code}
getFileSystem().create(TEST_FILE_PATH)
intercept(AbfsAuthorizationException.class,
()-> {
readOnlyFS.removeACL(TEST_FILE_PATH)
});
Why LambdaTestUtils.intercept()? it lets you restrict exactly where the auth
exceptions will be expected.
> ABFS: Ranger Support
>
>
> Key: HADOOP-15723
> URL: https://issues.apache.org/jira/browse/HADOOP-15723
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: HADOOP-15407
>Reporter: Thomas Marquardt
>Assignee: Yuan Gao
>Priority: Major
> Attachments: HADOOP-15273-HADOOP-15407-001.patch,
> HADOOP-15273-HADOOP-15407-002.patch, HADOOP-15273-HADOOP-15407-003.patch,
> HADOOP-15273-HADOOP-15407-004.patch
>
>
> Add support for Ranger
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15760) Include Apache Commons Collections4
[ https://issues.apache.org/jira/browse/HADOOP-15760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623771#comment-16623771 ] Steve Loughran commented on HADOOP-15760: - LGTM. what's your overall migration plan? > Include Apache Commons Collections4 > --- > > Key: HADOOP-15760 > URL: https://issues.apache.org/jira/browse/HADOOP-15760 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.10.0, 3.0.3 >Reporter: BELUGA BEHR >Priority: Major > Attachments: HADOOP-15760.1.patch > > > Please allow for use of Apache Commons Collections 4 library with the end > goal of migrating from Apache Commons Collections 3. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15774) Discovery of HA servers
[ https://issues.apache.org/jira/browse/HADOOP-15774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623767#comment-16623767 ] Steve Loughran commented on HADOOP-15774: - bq. Would it make sense to make the registry a standalone service instead of running in the RM? Being part of the YARN may bring some weird dependencies (HDFS->YARN), any ideas here? Should it be move to a separate place? Maybe parts? It can run standalone —its just a subclass of org.apache.hadoop.utils.AbstractService with its normal lifecycle. It's integrated in YARN so that * the RM can set things up with its kerberos permissions. * it can do cleanup after an app is shut down. Provided HDFS is set up with the permissions to edit the /services part of the tree, it could run the ZK "manager" service internally, registering itself on startup. This'd be easier if the HDFS NN went over to using that same Hadoop util.Service lifecycle, but if not, it could still wire it up. All you really need to keep running is ZK. > Discovery of HA servers > --- > > Key: HADOOP-15774 > URL: https://issues.apache.org/jira/browse/HADOOP-15774 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Íñigo Goiri >Priority: Major > Attachments: Discovery Service.pdf > > > Currently, Hadoop relies on configuration files to specify the servers. > This requires maintaining these configuration files and propagating the > changes. > Hadoop should have a framework to provide discovery. > For example, in HDFS, we could define the Namenodes in a shared location and > the DNs would use the framework to find the Namenodes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623761#comment-16623761 ] Steve Loughran commented on HADOOP-15679: - issue was that in the backport, the checkstyle warning about an unused import was wrong: the import was used in the javadocs. Remove the import and javadoc errors surface. latest patch reinstates the import, essentially reverting to the first branch-2 patch > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 2.8.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch, > HADOOP-15679-branch-2-003.patch, HADOOP-15679-branch-2-003.patch, > HADOOP-15679-branch-2-004.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15774) Discovery of HA servers
[ https://issues.apache.org/jira/browse/HADOOP-15774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623736#comment-16623736 ] Billie Rinaldi commented on HADOOP-15774: - bq. Would it make sense to make the registry a standalone service instead of running in the RM? There isn't really anything running in the RM. The registry is more of a convention about how things are stored in ZooKeeper under /registry. The Registry DNS server that hosts DNS records based on registry data runs as a separate daemon. bq. Being part of the YARN may bring some weird dependencies I think it would be possible to refactor the hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry module and move it to hadoop common if there are HDFS/YARN dependency issues. > Discovery of HA servers > --- > > Key: HADOOP-15774 > URL: https://issues.apache.org/jira/browse/HADOOP-15774 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Íñigo Goiri >Priority: Major > Attachments: Discovery Service.pdf > > > Currently, Hadoop relies on configuration files to specify the servers. > This requires maintaining these configuration files and propagating the > changes. > Hadoop should have a framework to provide discovery. > For example, in HDFS, we could define the Namenodes in a shared location and > the DNs would use the framework to find the Namenodes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15723) ABFS: Ranger Support
[
https://issues.apache.org/jira/browse/HADOOP-15723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623725#comment-16623725
]
Yuan Gao edited comment on HADOOP-15723 at 9/21/18 2:45 PM:
patch 004:
- called makeQualified() before authz checks
- moved configuration and factory method to AbfsConfiguration
- moved the authorization interface to org.apache.hadoop.fs.azurebfs.extensions
- renamed from adl to abfs
- added test cases for authz checks in ITestAzureBlobFileSystemAuthorization
I was able to run:
{{mvn -T 1C -Dparallel-tests -Dscale -DtestsThreadCount=8 clean verify}}
{{[INFO] Results:}}
{{[WARNING] Tests run: 277, Failures: 0, Errors: 0, Skipped: 11}}
But there are errors when running:
{{mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify}}
{{[ERROR] Tests run: 6, Failures: 0, Errors: 6, Skipped: 0, Time elapsed: 5.955
s <<< FAILURE! - in
org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDistCp}}
{{[ERROR]
testUpdateDeepDirectoryStructureNoChange(org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDistCp)
Time elapsed: 3.038 s <<< ERROR!}}
{{java.io.IOException: Unable to initialize filesystem
abfs://dfsafast...@rajeevbstorageac.dfs.core.windows.net:
java.lang.IllegalArgumentException: Incorrect URI
abfs://dfsafast...@rajeevbstorageac.dfs.core.windows.net, URI scheme must be
abfss when authenticating using Oauth.}}
was (Author: kowon2008):
patch 004:
- called makeQualified() before authz checks
- moved configuration and factory method to AbfsConfiguration
- moved the authorization interface to org.apache.hadoop.fs.azurebfs.extensions
- renamed from adl to abfs
- added test cases for authz checks in ITestAzureBlobFileSystemAuthorization
I was able to run:
{{ mvn -T 1C -Dparallel-tests -Dscale -DtestsThreadCount=8 clean verify}}
{{ [INFO] Results:}}
{{ [WARNING] Tests run: 277, Failures: 0, Errors: 0, Skipped: 11}}
But there are errors when running:
{{mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify}}
{{[ERROR] Tests run: 6, Failures: 0, Errors: 6, Skipped: 0, Time elapsed: 5.955
s <<< FAILURE! - in
org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDistCp}}
{{[ERROR]
testUpdateDeepDirectoryStructureNoChange(org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDistCp)
Time elapsed: 3.038 s <<< ERROR!}}
{{java.io.IOException: Unable to initialize filesystem
abfs://dfsafast...@rajeevbstorageac.dfs.core.windows.net:
java.lang.IllegalArgumentException: Incorrect URI
abfs://dfsafast...@rajeevbstorageac.dfs.core.windows.net, URI scheme must be
abfss when authenticating using Oauth.}}
> ABFS: Ranger Support
>
>
> Key: HADOOP-15723
> URL: https://issues.apache.org/jira/browse/HADOOP-15723
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: HADOOP-15407
>Reporter: Thomas Marquardt
>Assignee: Yuan Gao
>Priority: Major
> Attachments: HADOOP-15273-HADOOP-15407-001.patch,
> HADOOP-15273-HADOOP-15407-002.patch, HADOOP-15273-HADOOP-15407-003.patch,
> HADOOP-15273-HADOOP-15407-004.patch
>
>
> Add support for Ranger
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15723) ABFS: Ranger Support
[
https://issues.apache.org/jira/browse/HADOOP-15723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623725#comment-16623725
]
Yuan Gao commented on HADOOP-15723:
---
patch 004:
- called makeQualified() before authz checks
- moved configuration and factory method to AbfsConfiguration
- moved the authorization interface to org.apache.hadoop.fs.azurebfs.extensions
- renamed from adl to abfs
- added test cases for authz checks in ITestAzureBlobFileSystemAuthorization
I was able to run:
{{ mvn -T 1C -Dparallel-tests -Dscale -DtestsThreadCount=8 clean verify}}
{{ [INFO] Results:}}
{{ [WARNING] Tests run: 277, Failures: 0, Errors: 0, Skipped: 11}}
But there are errors when running:
{{mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify}}
{{[ERROR] Tests run: 6, Failures: 0, Errors: 6, Skipped: 0, Time elapsed: 5.955
s <<< FAILURE! - in
org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDistCp}}
{{[ERROR]
testUpdateDeepDirectoryStructureNoChange(org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDistCp)
Time elapsed: 3.038 s <<< ERROR!}}
{{java.io.IOException: Unable to initialize filesystem
abfs://dfsafast...@rajeevbstorageac.dfs.core.windows.net:
java.lang.IllegalArgumentException: Incorrect URI
abfs://dfsafast...@rajeevbstorageac.dfs.core.windows.net, URI scheme must be
abfss when authenticating using Oauth.}}
> ABFS: Ranger Support
>
>
> Key: HADOOP-15723
> URL: https://issues.apache.org/jira/browse/HADOOP-15723
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: HADOOP-15407
>Reporter: Thomas Marquardt
>Assignee: Yuan Gao
>Priority: Major
> Attachments: HADOOP-15273-HADOOP-15407-001.patch,
> HADOOP-15273-HADOOP-15407-002.patch, HADOOP-15273-HADOOP-15407-003.patch,
> HADOOP-15273-HADOOP-15407-004.patch
>
>
> Add support for Ranger
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Attachment: HADOOP-15679-branch-2-004.patch > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 2.8.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch, > HADOOP-15679-branch-2-003.patch, HADOOP-15679-branch-2-003.patch, > HADOOP-15679-branch-2-004.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Target Version/s: 3.0.3, 2.10.0, 2.9.2, 2.8.5 (was: 2.10.0, 2.9.2, 3.0.3, 2.8.5) Status: Patch Available (was: Open) > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch, > HADOOP-15679-branch-2-003.patch, HADOOP-15679-branch-2-003.patch, > HADOOP-15679-branch-2-004.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Target Version/s: 3.0.3, 2.10.0, 2.9.2, 2.8.5 (was: 2.10.0, 2.9.2, 3.0.3, 2.8.5) Status: Open (was: Patch Available) > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch, > HADOOP-15679-branch-2-003.patch, HADOOP-15679-branch-2-003.patch, > HADOOP-15679-branch-2-004.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[
https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623692#comment-16623692
]
Steve Loughran commented on HADOOP-15679:
-
{code}
[WARNING]
/testptch/hadoop/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ShutdownHookManager.java:64:
warning - Tag @link: reference not found:
CommonConfigurationKeysPublic#SERVICE_SHUTDOWN_TIMEOUT
[WARNING]
/testptch/hadoop/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ShutdownHookManager.java:64:
warning - Tag @link: reference not found:
CommonConfigurationKeysPublic#SERVICE_SHUTDOWN_TIMEOUT_DEFAULT
{code}
> ShutdownHookManager shutdown time needs to be configurable & extended
> -
>
> Key: HADOOP-15679
> URL: https://issues.apache.org/jira/browse/HADOOP-15679
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
>Affects Versions: 2.8.0, 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch,
> HADOOP-15679-002.patch, HADOOP-15679-003.patch,
> HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch,
> HADOOP-15679-branch-2-003.patch, HADOOP-15679-branch-2-003.patch
>
>
> HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging
> shutdowns. But the timeout is too short for applications where a large flush
> of data is needed on shutdown.
> A key example of this is Spark apps which save their history to object
> stores, where the file close() call triggers an upload of the final local
> cached block of data (could be 32+MB), and then execute the final mutipart
> commit.
> Proposed
> # make the default sleep time 30s, not 10s
> # make it configurable with a time duration property (with minimum time of
> 1s.?)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15691:
Description:
Add a {{PathCapabilities}} interface to both FileSystem and FileContext to
declare the capabilities under the path of an FS
This is needed for
* HADOOP-14707: declare that a dest FS supports permissions
* object stores to declare that they offer PUT-in-place alongside (slow-rename)
* Anything else where the implementation semantics of an FS is so different
caller apps would benefit from probing for the underlying semantics
I know, we want all filesystem to work *exactly* the same. But it doesn't hold,
especially for object stores —and to efficiently use them, callers need to be
able to ask for specific features.
was:
Add a {{PathCapabilities}} interface to both FileSystem and FileContext to
declare the capabilities under the path of an FS
This is needed for
* HADOOP-15407: declare that a dest FS supports permissions
* object stores to declare that they offer PUT-in-place alongside (slow-rename)
* Anything else where the implementation semantics of an FS is so different
caller apps would benefit from probing for the underlying semantics
I know, we want all filesystem to work *exactly* the same. But it doesn't hold,
especially for object stores —and to efficiently use them, callers need to be
able to ask for specific features.
> Add PathCapabilities to FS and FC to complement StreamCapabilities
> --
>
> Key: HADOOP-15691
> URL: https://issues.apache.org/jira/browse/HADOOP-15691
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15691-001.patch, HADOOP-15691-002.patch,
> HADOOP-15691-003.patch
>
>
> Add a {{PathCapabilities}} interface to both FileSystem and FileContext to
> declare the capabilities under the path of an FS
> This is needed for
> * HADOOP-14707: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside
> (slow-rename)
> * Anything else where the implementation semantics of an FS is so different
> caller apps would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't
> hold, especially for object stores —and to efficiently use them, callers need
> to be able to ask for specific features.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15691:
Status: Patch Available (was: Open)
> Add PathCapabilities to FS and FC to complement StreamCapabilities
> --
>
> Key: HADOOP-15691
> URL: https://issues.apache.org/jira/browse/HADOOP-15691
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15691-001.patch, HADOOP-15691-002.patch,
> HADOOP-15691-003.patch
>
>
> Add a {{PathCapabilities}} interface to both FileSystem and FileContext to
> declare the capabilities under the path of an FS
> This is needed for
> * HADOOP-15407: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside
> (slow-rename)
> * Anything else where the implementation semantics of an FS is so different
> caller apps would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't
> hold, especially for object stores —and to efficiently use them, callers need
> to be able to ask for specific features.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623684#comment-16623684
]
Steve Loughran commented on HADOOP-15691:
-
Patch 003
* revise based on feedback
* catch up with trunk (pathhandles)
* add option for an FS connector to say "We know we are read only" (http, har)
* add checksum capability, wire up so those filesystems with it (DFS, S3A when
its enabled)
* symlink capability is based on {{symlinksEnabled}} flag in FileSystem class.
* expanded docs
* ~0 tests
I now realise that a bool "true/false" isn't quite right, as it doesn't cover
the "we think we'd support it but you'd have to try". Symlinks being a key
example. There's DFS: we know we have it, other stores "we know we don't", but
localFS is either "known disabled" or "try it and see"
An enum seems the outcome here, something like
{code}
known-available: its a known feature and known to work in this instance (ex:
symlinks on hdfs)
known-unavailable: we know of it, we know we don't support it (ex: append on
s3a)
known-maybe-available: we know of it, but to see if it works, you will have to
try the operation. (symlink on local)
unknown: we don't know of this feature.
{code}
the base FS would say "unknown feature" for ~everything; implementations would
list what they know about and their interpretation, so that everything with
rename would add "fs:rename"; posix-ish filesytems with atomic rename,
"fs-rename-atomic"; the object stores would mark the latter as
"known-unavailable", etc.
We'd also need to qualify everything with "depending on your permissions". An
FS may know of append, but it might be a read-only mount. I've already added to
filesystem.md the fact that the state may change depending on external events.
> Add PathCapabilities to FS and FC to complement StreamCapabilities
> --
>
> Key: HADOOP-15691
> URL: https://issues.apache.org/jira/browse/HADOOP-15691
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15691-001.patch, HADOOP-15691-002.patch,
> HADOOP-15691-003.patch
>
>
> Add a {{PathCapabilities}} interface to both FileSystem and FileContext to
> declare the capabilities under the path of an FS
> This is needed for
> * HADOOP-15407: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside
> (slow-rename)
> * Anything else where the implementation semantics of an FS is so different
> caller apps would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't
> hold, especially for object stores —and to efficiently use them, callers need
> to be able to ask for specific features.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15691:
Attachment: HADOOP-15691-003.patch
> Add PathCapabilities to FS and FC to complement StreamCapabilities
> --
>
> Key: HADOOP-15691
> URL: https://issues.apache.org/jira/browse/HADOOP-15691
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15691-001.patch, HADOOP-15691-002.patch,
> HADOOP-15691-003.patch
>
>
> Add a {{PathCapabilities}} interface to both FileSystem and FileContext to
> declare the capabilities under the path of an FS
> This is needed for
> * HADOOP-15407: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside
> (slow-rename)
> * Anything else where the implementation semantics of an FS is so different
> caller apps would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't
> hold, especially for object stores —and to efficiently use them, callers need
> to be able to ask for specific features.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15566) Remove HTrace support
[ https://issues.apache.org/jira/browse/HADOOP-15566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623548#comment-16623548 ] Carlos Alberto Cortez commented on HADOOP-15566: Hi all, I went ahead and did a Proof-Of-Concept migration from HTrace to OpenTracing (using Zipkin as the backend). You can inspect (and play with the code) here: [https://github.com/apache/hadoop/compare/trunk...carlosalberto:ot_initial_integration] Some notes: 1. It creates by default a Tracer instance based on Zipkin running in localhost (for simplicity purposes). 2. It uses the notion of a GlobalTracer so create and register and use the Tracer from a single place. 3. As Wei-Chiu mentioned, it needed some small extra work to pass around the parent-child relationship (which is done trough `SpanId` in HTrace, and `SpanContext` in OpenTracing). 4. Added a new SpanContext field in the clases using protobuf to pass trace info. As mentioned, this is a POC, but hope this can throw light into this (and I'm happy to answer questions or contribute with this as an actual migration ;) ) > Remove HTrace support > - > > Key: HADOOP-15566 > URL: https://issues.apache.org/jira/browse/HADOOP-15566 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics >Affects Versions: 3.1.0 >Reporter: Todd Lipcon >Priority: Major > Labels: security > Attachments: Screen Shot 2018-06-29 at 11.59.16 AM.png, > ss-trace-s3a.png > > > The HTrace incubator project has voted to retire itself and won't be making > further releases. The Hadoop project currently has various hooks with HTrace. > It seems in some cases (eg HDFS-13702) these hooks have had measurable > performance overhead. Given these two factors, I think we should consider > removing the HTrace integration. If there is someone willing to do the work, > replacing it with OpenTracing might be a better choice since there is an > active community. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-8977) multiple FsShell test failures on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623505#comment-16623505 ] Vinayakumar B commented on HADOOP-8977: --- HDFS-13911 seems to be broken by this change. After this change, all local paths with special characters (ex: space) should be encoded first before passing as argument to command line. > multiple FsShell test failures on Windows > - > > Key: HADOOP-8977 > URL: https://issues.apache.org/jira/browse/HADOOP-8977 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: trunk-win >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Major > Fix For: trunk-win > > Attachments: HADOOP-8977-branch-trunk-win.patch, > HADOOP-8977-branch-trunk-win.patch, HADOOP-8977-branch-trunk-win.patch, > HADOOP-8977.patch > > > Multiple FsShell-related tests fail on Windows. Commands are returning > non-zero exit status. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15775) [JDK9] Add missing javax.activation-api dependency
[
https://issues.apache.org/jira/browse/HADOOP-15775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623491#comment-16623491
]
Hadoop QA commented on HADOOP-15775:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 5m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
49m 50s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 26s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 18m 58s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 75m
20s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} | {color:green} hadoop-client-minicluster in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}181m 24s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.containermanager.TestNMProxy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | HADOOP-15775 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12940749/HADOOP-15775.03.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml |
| uname | Linux f80156c6aac8 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 524f7cd |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| unit |
[jira] [Commented] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623450#comment-16623450
]
Steve Loughran commented on HADOOP-15691:
-
bq. Should we document a well defined capabilities? For example, we can say in
Hadoop 3.3, we define the following list of path capabilities (like
fs:snapshots) and file system implementations should support probing of these
capabilities. Right now these capabilities exist in the code, not in user docs.
bq. Let's leave to the javadocs, so that its more likely to be in sync with the
code. We can add a ref to it though in the FS docs.
bq. Instead of defining the capabilities in StreamCapabilities.java, why not
create a PathCapabilities class for these capability definitions?
I'd just added them alongside the existing ones, but yes, it makes sense. Will
do this.
bq. Are these capabilities well defined somewhere? I am not a file system guy
so I am not sure if people interpret these capabilities differently.
Not yet, we can be specific in the javadocs
> Add PathCapabilities to FS and FC to complement StreamCapabilities
> --
>
> Key: HADOOP-15691
> URL: https://issues.apache.org/jira/browse/HADOOP-15691
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15691-001.patch, HADOOP-15691-002.patch
>
>
> Add a {{PathCapabilities}} interface to both FileSystem and FileContext to
> declare the capabilities under the path of an FS
> This is needed for
> * HADOOP-15407: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside
> (slow-rename)
> * Anything else where the implementation semantics of an FS is so different
> caller apps would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't
> hold, especially for object stores —and to efficiently use them, callers need
> to be able to ask for specific features.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[
https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623447#comment-16623447
]
Hadoop QA commented on HADOOP-15679:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
9s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
2s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} hadoop-common-project/hadoop-common: The patch
generated 0 new + 115 unchanged - 1 fixed = 115 total (was 116) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m
1s{color} | {color:red} hadoop-common-project_hadoop-common generated 2 new +
10 unchanged - 0 fixed = 12 total (was 10) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
41s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
42s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 70m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:a716388 |
| JIRA Issue | HADOOP-15679 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12940760/HADOOP-15679-branch-2-003.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux e48cccf61331 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2 / 3a6ad9c |
| maven | version: Apache Maven 3.3.9
(bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00) |
| Default Java | 1.7.0_181 |
| javadoc |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15247/artifact/out/diff-javadoc-javadoc-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15247/testReport/ |
| Max. process+thread count | 1286 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15247/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> ShutdownHookManager shutdown time needs to be configurable & extended
>
[jira] [Commented] (HADOOP-15775) [JDK9] Add missing javax.activation-api dependency
[
https://issues.apache.org/jira/browse/HADOOP-15775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623443#comment-16623443
]
Hadoop QA commented on HADOOP-15775:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
44m 48s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 12m
44s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 49s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 16s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}168m 51s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.containermanager.TestNMProxy |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueManagementDynamicEditPolicy
|
| | hadoop.yarn.server.resourcemanager.TestApplicationMasterService |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | HADOOP-15775 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12940745/HADOOP-15775.02.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml |
| uname | Linux db4dfd446c98 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 524f7cd |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| unit |
[jira] [Updated] (HADOOP-15662) ABFS: Better exception handling of DNS errors
[ https://issues.apache.org/jira/browse/HADOOP-15662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15662: Parent Issue: HADOOP-15763 (was: HADOOP-15407) > ABFS: Better exception handling of DNS errors > - > > Key: HADOOP-15662 > URL: https://issues.apache.org/jira/browse/HADOOP-15662 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Da Zhou >Priority: Major > > DNS errors are common during testing due to typos or misconfiguration. They > can also occur in production, as some transient DNS issues occur from time to > time. > 1) Let's investigate if we can distinguish between the two and fail fast for > the test issues, but continue to have retry logic for the transient DNS > issues in production. > 2) Let's improve the error handling of DNS failures, so the user has an > actionable error message. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15778) ABFS: Fix client side throttling for read
[ https://issues.apache.org/jira/browse/HADOOP-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15778: Resolution: Fixed Fix Version/s: HADOOP-15407 Status: Resolved (was: Patch Available) +1 committed. Thanks! > ABFS: Fix client side throttling for read > - > > Key: HADOOP-15778 > URL: https://issues.apache.org/jira/browse/HADOOP-15778 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sneha Varma >Assignee: Sneha Varma >Priority: Minor > Fix For: HADOOP-15407 > > Attachments: HADOOP-15778-HADOOP-15407-001.patch, > HADOOP-15778-HADOOP-15407-002.patch > > > 1. The content length for ReadFile in updateMetrics of > AbfsClientThrottlingIntercept is incorrect for cases when the request fails. > It is currently equal to the number of bytes that are read whereas it should > be equal to the number of bytes requested. > 2. sendingRequest of AbfsClientThrottlingIntercept at AbfsRestOperation > should be called irrespective of whether the request has body. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15779) S3guard: add inconsistency detection metrics
[ https://issues.apache.org/jira/browse/HADOOP-15779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623362#comment-16623362 ] Steve Loughran commented on HADOOP-15779: - this'd be good. I could imagine pumping out (CSV? Avro) logs of the events to the yarn log dir, for aggregation and analysis > S3guard: add inconsistency detection metrics > > > Key: HADOOP-15779 > URL: https://issues.apache.org/jira/browse/HADOOP-15779 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Aaron Fabbri >Priority: Major > > S3Guard uses a trailing log of metadata changes made to an S3 bucket to add > consistency to the eventually-consistent AWS S3 service. We should add some > metrics that are incremented when we detect inconsistency (eventual > consistency) in S3. > I'm thinking at least two counters: (1) getFileStatus() (HEAD) inconsistency > detected, and (2) listing inconsistency detected. We may want to further > separate into categories (present / not present etc.) > This is related to Auth. Mode and TTL work that is ongoing, so let me outline > how I think this should all evolve: > This should happen after HADOOP-15621 (TTL for dynamo MetadataStore), since > that will change *when* we query both S3 and the MetadataStore (e.g. Dynamo) > for metadata. There I suggest that: > # Prune time is different than TTL. Prune time: "how long until > inconsistency is no longer a problem" . TTL time "how long a MetadataStore > entry is considered authoritative before refresh" > # Prune expired: delete entries (when hadoop CLI prune command is run). TTL > Expired: entries become non-authoritative. > # Prune implemented in each MetadataStore, but TTL filtering happens in S3A. > Once we have this, S3A will be consulting both S3 and MetadataStore depending > on configuration and/or age of the entry in the MetadataStore. Today > HEAD/getFileStatus() is always short-circuit (skips S3 if MetadataStore > returns results). I think S3A should consult both when TTL is stale, and > invoke a callback on inconsistency that increments the new metrics. For > listing, we already are comparing both sources of truth (except when S3A auth > mode is on and a directory is marked authoritative in MS), so it would be > pretty simple to invoke a callback on inconsistency and bump some metrics. > Comments / suggestions / questions welcomed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Target Version/s: 3.0.3, 2.10.0, 2.9.2, 2.8.5 (was: 2.10.0, 2.9.2, 3.0.3, 2.8.5) Status: Patch Available (was: Open) > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch, > HADOOP-15679-branch-2-003.patch, HADOOP-15679-branch-2-003.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Target Version/s: 3.0.3, 2.10.0, 2.9.2, 2.8.5 (was: 2.10.0, 2.9.2, 3.0.3, 2.8.5) Status: Open (was: Patch Available) > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch, > HADOOP-15679-branch-2-003.patch, HADOOP-15679-branch-2-003.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Attachment: HADOOP-15679-branch-2-003.patch > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 2.8.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch, > HADOOP-15679-branch-2-003.patch, HADOOP-15679-branch-2-003.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15711) Fix branch-2 builds
[
https://issues.apache.org/jira/browse/HADOOP-15711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623338#comment-16623338
]
Steve Loughran commented on HADOOP-15711:
-
bq. My guess is something in hadoop-common (and FileSystem in particular) that
doesn't work well on JDK7.
something in the native code binding? As I can't imagine much else
> Fix branch-2 builds
> ---
>
> Key: HADOOP-15711
> URL: https://issues.apache.org/jira/browse/HADOOP-15711
> Project: Hadoop Common
> Issue Type: Task
>Reporter: Jonathan Hung
>Priority: Critical
> Attachments: HADOOP-15711.001.branch-2.patch
>
>
> Branch-2 builds have been disabled for a while:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/
> A test run here causes hdfs tests to hang:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86-jhung/4/
> Running hadoop-hdfs tests locally reveal some errors such
> as:{noformat}[ERROR]
> testComplexAppend2(org.apache.hadoop.hdfs.TestFileAppend2) Time elapsed:
> 0.059 s <<< ERROR!
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1164)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1128)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:174)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1172)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:403)
> at
> org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:234)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1080)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:883)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:514)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:473)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend(TestFileAppend2.java:489)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend2(TestFileAppend2.java:543)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){noformat}
> I was able to get more tests passing locally by increasing the max user
> process count on my machine. But the error suggests that there's an issue in
> the tests themselves. Not sure if the error seen locally is the same reason
> as why jenkins builds are failing, I wasn't able to confirm based on the
> jenkins builds' lack of output.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15735) backport HADOOP-11687 for hadoop-aws 2.7.x
[ https://issues.apache.org/jira/browse/HADOOP-15735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15735: Resolution: Fixed Fix Version/s: 2.7.8 Status: Resolved (was: Patch Available) thanks for doing this, especially the testing. Fix will be in 2.7.8 Note that spark rolled back from 2.7.7 to 2.7.6 because part of a security lockdown broke things...that will need to be addressed in the same release before they can move up > backport HADOOP-11687 for hadoop-aws 2.7.x > -- > > Key: HADOOP-15735 > URL: https://issues.apache.org/jira/browse/HADOOP-15735 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.7.7 >Reporter: MunShik JOUNG >Assignee: MunShik JOUNG >Priority: Trivial > Fix For: 2.7.8 > > Attachments: > 0001-HADOOP-12970-Intermitent-SignatureDoesNotMatch-Excep.patch > > > HADOOP-12970 Intermitent SignatureDoesNotMatch Exception is fixed by > HADOOP-11687. > And from hadoop-aws >= 2.8, HADOOP-11687 is applied. but Spark 2.3.1 is still > using hadoop-aws 2.7.7. > I was stumpled upon Intermitent SignatureDoesNotMatch Exception with s3 with > spark 2.3.1. So I guess backporting might benefit some spark users. In my > situation, SignatureDoesNotMatch was reliably reproducible (4 out of 8 > attemp). And after using patched version symptom was not seen (0 out of 10) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15735) backport HADOOP-11687 for hadoop-aws 2.7.x
[ https://issues.apache.org/jira/browse/HADOOP-15735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned HADOOP-15735: --- Assignee: MunShik JOUNG > backport HADOOP-11687 for hadoop-aws 2.7.x > -- > > Key: HADOOP-15735 > URL: https://issues.apache.org/jira/browse/HADOOP-15735 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.7.7 >Reporter: MunShik JOUNG >Assignee: MunShik JOUNG >Priority: Trivial > Attachments: > 0001-HADOOP-12970-Intermitent-SignatureDoesNotMatch-Excep.patch > > > HADOOP-12970 Intermitent SignatureDoesNotMatch Exception is fixed by > HADOOP-11687. > And from hadoop-aws >= 2.8, HADOOP-11687 is applied. but Spark 2.3.1 is still > using hadoop-aws 2.7.7. > I was stumpled upon Intermitent SignatureDoesNotMatch Exception with s3 with > spark 2.3.1. So I guess backporting might benefit some spark users. In my > situation, SignatureDoesNotMatch was reliably reproducible (4 out of 8 > attemp). And after using patched version symptom was not seen (0 out of 10) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15775) [JDK9] Add missing javax.activation-api dependency
[ https://issues.apache.org/jira/browse/HADOOP-15775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-15775: --- Summary: [JDK9] Add missing javax.activation-api dependency (was: [JDK9] Add "--add-modules java.activation" to java option in unit tests) Description: Many unit tests fail due to missing java.activation module. This failure can be fixed by adding javax.activation-api as third-party dependency. (was: Many unit tests fail due to missing java.activation module. We need to configure maven surefire plugin to add "--add-modules java.activation" option to test JVM.) > [JDK9] Add missing javax.activation-api dependency > -- > > Key: HADOOP-15775 > URL: https://issues.apache.org/jira/browse/HADOOP-15775 > Project: Hadoop Common > Issue Type: Sub-task > Components: test >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Critical > Attachments: HADOOP-15775.01.patch, HADOOP-15775.02.patch, > HADOOP-15775.03.patch > > > Many unit tests fail due to missing java.activation module. This failure can > be fixed by adding javax.activation-api as third-party dependency. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15775) [JDK9] Add "--add-modules java.activation" to java option in unit tests
[ https://issues.apache.org/jira/browse/HADOOP-15775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623302#comment-16623302 ] Akira Ajisaka commented on HADOOP-15775: 03 patch * Shaded the third-party dependency from hadoop-client-minicluster > [JDK9] Add "--add-modules java.activation" to java option in unit tests > --- > > Key: HADOOP-15775 > URL: https://issues.apache.org/jira/browse/HADOOP-15775 > Project: Hadoop Common > Issue Type: Sub-task > Components: test >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Critical > Attachments: HADOOP-15775.01.patch, HADOOP-15775.02.patch, > HADOOP-15775.03.patch > > > Many unit tests fail due to missing java.activation module. We need to > configure maven surefire plugin to add "--add-modules java.activation" option > to test JVM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15775) [JDK9] Add "--add-modules java.activation" to java option in unit tests
[ https://issues.apache.org/jira/browse/HADOOP-15775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-15775: --- Attachment: HADOOP-15775.03.patch > [JDK9] Add "--add-modules java.activation" to java option in unit tests > --- > > Key: HADOOP-15775 > URL: https://issues.apache.org/jira/browse/HADOOP-15775 > Project: Hadoop Common > Issue Type: Sub-task > Components: test >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Critical > Attachments: HADOOP-15775.01.patch, HADOOP-15775.02.patch, > HADOOP-15775.03.patch > > > Many unit tests fail due to missing java.activation module. We need to > configure maven surefire plugin to add "--add-modules java.activation" option > to test JVM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15711) Fix branch-2 builds
[
https://issues.apache.org/jira/browse/HADOOP-15711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623288#comment-16623288
]
Allen Wittenauer commented on HADOOP-15711:
---
bq. Can we disable parallel tests for full builds then ?
I don't even want to contemplate the runtime. With parallelization, it is
already in the 14-15 hour mark, easily.
[BTW, have a client that was using Yetus internally for 2.7. As a result, one
of the changes in Yetus 0.8.0 was to disable parallelization for 2.7 and lower
since those aren't thread safe anyway. That's why the 2.7 run timed out after
20 hours. ]
bq. Will that help track down specific tests that go OOM for instance ?
Probably not, and definitely not without also setting the test order. Also: no
one has really done the homework to figure out why things are going catatonic.
My guess is something in hadoop-common (and FileSystem in particular) that
doesn't work well on JDK7. That's why trunk doesn't have these issues. My
other hunch is that commit date of the problematic code is was committed
between June 2016-June 2017. Because we had other issues and no one was aware
of how bad SUREFIRE-524 was, the issues had been hidden.
> Fix branch-2 builds
> ---
>
> Key: HADOOP-15711
> URL: https://issues.apache.org/jira/browse/HADOOP-15711
> Project: Hadoop Common
> Issue Type: Task
>Reporter: Jonathan Hung
>Priority: Critical
> Attachments: HADOOP-15711.001.branch-2.patch
>
>
> Branch-2 builds have been disabled for a while:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/
> A test run here causes hdfs tests to hang:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86-jhung/4/
> Running hadoop-hdfs tests locally reveal some errors such
> as:{noformat}[ERROR]
> testComplexAppend2(org.apache.hadoop.hdfs.TestFileAppend2) Time elapsed:
> 0.059 s <<< ERROR!
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1164)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1128)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:174)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1172)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:403)
> at
> org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:234)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1080)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:883)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:514)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:473)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend(TestFileAppend2.java:489)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend2(TestFileAppend2.java:543)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){noformat}
> I was able to get more tests passing locally by increasing the max user
> process count on my machine. But the error suggests that there's an issue in
> the tests themselves. Not sure if the error seen locally is the same reason
> as why jenkins builds are failing, I wasn't able to confirm based on the
> jenkins builds' lack of output.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15774) Discovery of HA servers
[ https://issues.apache.org/jira/browse/HADOOP-15774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623272#comment-16623272 ] Steve Loughran commented on HADOOP-15774: - even though it says YARN, is explicitly not depending on any of the YARN libraries; it's always had a vision (from Eric14) to be broader than this. Essentially: discovery and binding to dynamically deployed applications, with Kerberos in the mix for security. For things YARN schedules, this has always been essential; but if you want dynamic deployment of core Hadoop services, it should work too. If not, something to fix Pull [~sanjay.radia] [~billie.rinaldi] &[~elserj] for their insight here too. > Discovery of HA servers > --- > > Key: HADOOP-15774 > URL: https://issues.apache.org/jira/browse/HADOOP-15774 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Íñigo Goiri >Priority: Major > Attachments: Discovery Service.pdf > > > Currently, Hadoop relies on configuration files to specify the servers. > This requires maintaining these configuration files and propagating the > changes. > Hadoop should have a framework to provide discovery. > For example, in HDFS, we could define the Namenodes in a shared location and > the DNs would use the framework to find the Namenodes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15775) [JDK9] Add "--add-modules java.activation" to java option in unit tests
[ https://issues.apache.org/jira/browse/HADOOP-15775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623259#comment-16623259 ] Akira Ajisaka commented on HADOOP-15775: 02 patch: * Add javax.activation-api dependency. * TODO: Shade this dependency. > [JDK9] Add "--add-modules java.activation" to java option in unit tests > --- > > Key: HADOOP-15775 > URL: https://issues.apache.org/jira/browse/HADOOP-15775 > Project: Hadoop Common > Issue Type: Sub-task > Components: test >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Critical > Attachments: HADOOP-15775.01.patch, HADOOP-15775.02.patch > > > Many unit tests fail due to missing java.activation module. We need to > configure maven surefire plugin to add "--add-modules java.activation" option > to test JVM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15775) [JDK9] Add "--add-modules java.activation" to java option in unit tests
[ https://issues.apache.org/jira/browse/HADOOP-15775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-15775: --- Attachment: HADOOP-15775.02.patch > [JDK9] Add "--add-modules java.activation" to java option in unit tests > --- > > Key: HADOOP-15775 > URL: https://issues.apache.org/jira/browse/HADOOP-15775 > Project: Hadoop Common > Issue Type: Sub-task > Components: test >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Critical > Attachments: HADOOP-15775.01.patch, HADOOP-15775.02.patch > > > Many unit tests fail due to missing java.activation module. We need to > configure maven surefire plugin to add "--add-modules java.activation" option > to test JVM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15775) [JDK9] Add "--add-modules java.activation" to java option in unit tests
[ https://issues.apache.org/jira/browse/HADOOP-15775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623248#comment-16623248 ] Akira Ajisaka commented on HADOOP-15775: bq. Is the third party dependency is available in Java 9 and 10? It is available in Java 9 and 10. Great. > [JDK9] Add "--add-modules java.activation" to java option in unit tests > --- > > Key: HADOOP-15775 > URL: https://issues.apache.org/jira/browse/HADOOP-15775 > Project: Hadoop Common > Issue Type: Sub-task > Components: test >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Critical > Attachments: HADOOP-15775.01.patch > > > Many unit tests fail due to missing java.activation module. We need to > configure maven surefire plugin to add "--add-modules java.activation" option > to test JVM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15711) Fix branch-2 builds
[
https://issues.apache.org/jira/browse/HADOOP-15711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623202#comment-16623202
]
Arun Suresh commented on HADOOP-15711:
--
Yeah... figured out with [~jhung] that isolating and ignoring specific tests
wont work since as you mentioned, when executed in the parallel, a thread
running a random batch of tests might go down.
Given that single YARN patch testing for instance should run fine (Am assuming
branch-2 Dockerfile is good currently since YARN-8658 ran fine), Can we disable
parallel tests for full builds then ? Will that help track down specific tests
that go OOM for instance ?
> Fix branch-2 builds
> ---
>
> Key: HADOOP-15711
> URL: https://issues.apache.org/jira/browse/HADOOP-15711
> Project: Hadoop Common
> Issue Type: Task
>Reporter: Jonathan Hung
>Priority: Critical
> Attachments: HADOOP-15711.001.branch-2.patch
>
>
> Branch-2 builds have been disabled for a while:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/
> A test run here causes hdfs tests to hang:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86-jhung/4/
> Running hadoop-hdfs tests locally reveal some errors such
> as:{noformat}[ERROR]
> testComplexAppend2(org.apache.hadoop.hdfs.TestFileAppend2) Time elapsed:
> 0.059 s <<< ERROR!
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1164)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1128)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:174)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1172)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:403)
> at
> org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:234)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1080)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:883)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:514)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:473)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend(TestFileAppend2.java:489)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend2(TestFileAppend2.java:543)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){noformat}
> I was able to get more tests passing locally by increasing the max user
> process count on my machine. But the error suggests that there's an issue in
> the tests themselves. Not sure if the error seen locally is the same reason
> as why jenkins builds are failing, I wasn't able to confirm based on the
> jenkins builds' lack of output.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15775) [JDK9] Add "--add-modules java.activation" to java option in unit tests
[ https://issues.apache.org/jira/browse/HADOOP-15775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623165#comment-16623165 ] Akira Ajisaka commented on HADOOP-15775: Thank you for the information, [~apurtell]! Is the third party dependency is available in Java 9 and 10? If so, adding the dependency in Java 9+ is better. Otherwise we need another profile to add the third party dependency only in Java 11+. > [JDK9] Add "--add-modules java.activation" to java option in unit tests > --- > > Key: HADOOP-15775 > URL: https://issues.apache.org/jira/browse/HADOOP-15775 > Project: Hadoop Common > Issue Type: Sub-task > Components: test >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Critical > Attachments: HADOOP-15775.01.patch > > > Many unit tests fail due to missing java.activation module. We need to > configure maven surefire plugin to add "--add-modules java.activation" option > to test JVM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14775) Change junit dependency in parent pom file to junit 5 while maintaining backward compatibility to junit4.
[ https://issues.apache.org/jira/browse/HADOOP-14775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623162#comment-16623162 ] Akira Ajisaka commented on HADOOP-14775: Note: Apache Hadoop sets reuseForks=false and when reuseForks=false JUnit 5.3.0 and Maven Surefire Plugin 2.22.0 fail to find tests. https://github.com/junit-team/junit5/issues/1579 Probably we need to wait for Maven Surefire Plugin 2.22.1 release. > Change junit dependency in parent pom file to junit 5 while maintaining > backward compatibility to junit4. > -- > > Key: HADOOP-14775 > URL: https://issues.apache.org/jira/browse/HADOOP-14775 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha4 >Reporter: Ajay Kumar >Assignee: Akira Ajisaka >Priority: Major > Labels: junit5 > Attachments: HADOOP-14775.01.patch, HADOOP-14775.02.patch, > HADOOP-14775.03.patch, HADOOP-14775.04.patch, HADOOP-14775.05.patch > > > Change junit dependency in parent pom file to junit 5 while maintaining > backward compatibility to junit4. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14775) Change junit dependency in parent pom file to junit 5 while maintaining backward compatibility to junit4.
[ https://issues.apache.org/jira/browse/HADOOP-14775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623105#comment-16623105 ] Akira Ajisaka commented on HADOOP-14775: Thanks [~ajayydv]! Yes. If we want to run JUnit4 and JUnit5 tests in the same module, add vintage and jupitor engine and leave junit dependency as-is in pom.xml. Sample: https://github.com/junit-team/junit5-samples/tree/master/junit5-migration-maven > Change junit dependency in parent pom file to junit 5 while maintaining > backward compatibility to junit4. > -- > > Key: HADOOP-14775 > URL: https://issues.apache.org/jira/browse/HADOOP-14775 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha4 >Reporter: Ajay Kumar >Assignee: Akira Ajisaka >Priority: Major > Labels: junit5 > Attachments: HADOOP-14775.01.patch, HADOOP-14775.02.patch, > HADOOP-14775.03.patch, HADOOP-14775.04.patch, HADOOP-14775.05.patch > > > Change junit dependency in parent pom file to junit 5 while maintaining > backward compatibility to junit4. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
