[jira] [Updated] (HADOOP-18279) Cancel fileMonitoringTimer even if trustManager isn't defined
[ https://issues.apache.org/jira/browse/HADOOP-18279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-18279: -- Fix Version/s: 3.3.9 > Cancel fileMonitoringTimer even if trustManager isn't defined > - > > Key: HADOOP-18279 > URL: https://issues.apache.org/jira/browse/HADOOP-18279 > Project: Hadoop Common > Issue Type: Bug > Components: common, test >Affects Versions: 3.4.0, 3.3.5 > Environment: This problem was identified using the Hadoop Development > Environment during unit testing. >Reporter: Steve Vaughan >Assignee: Steve Vaughan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > The key stores factory starts a timer for monitoring file changes that should > be reloaded. The call to destroy() doesn't cancel the timer when a trust > manager isn't defined. This leaves the timer running, during shutdown. This > can be seen in unit tests that do not stop when the test completes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18279) Cancel fileMonitoringTimer even if trustManager isn't defined
[ https://issues.apache.org/jira/browse/HADOOP-18279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun resolved HADOOP-18279. --- Resolution: Fixed > Cancel fileMonitoringTimer even if trustManager isn't defined > - > > Key: HADOOP-18279 > URL: https://issues.apache.org/jira/browse/HADOOP-18279 > Project: Hadoop Common > Issue Type: Bug > Components: common, test >Affects Versions: 3.4.0, 3.3.5 > Environment: This problem was identified using the Hadoop Development > Environment during unit testing. >Reporter: Steve Vaughan >Assignee: Steve Vaughan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > The key stores factory starts a timer for monitoring file changes that should > be reloaded. The call to destroy() doesn't cancel the timer when a trust > manager isn't defined. This leaves the timer running, during shutdown. This > can be seen in unit tests that do not stop when the test completes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18383) Codecs with @DoNotPool annotation are not closed causing memory leak
[ https://issues.apache.org/jira/browse/HADOOP-18383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-18383: -- Fix Version/s: 3.3.9 > Codecs with @DoNotPool annotation are not closed causing memory leak > > > Key: HADOOP-18383 > URL: https://issues.apache.org/jira/browse/HADOOP-18383 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.3.2 >Reporter: Kevin Sewell >Assignee: Kevin Sewell >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > Time Spent: 20m > Remaining Estimate: 0h > > Compressors and Decompressions with a @DoNotPool annotation are not closed > when they are returned to the CodecPool, which causes a native memory leak. > > I have included a link to a [Demo > Project|https://github.com/kevins-29/hadoop-gzip-memory-leak] demonstrating > the leak -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18383) Codecs with @DoNotPool annotation are not closed causing memory leak
[ https://issues.apache.org/jira/browse/HADOOP-18383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun resolved HADOOP-18383. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Codecs with @DoNotPool annotation are not closed causing memory leak > > > Key: HADOOP-18383 > URL: https://issues.apache.org/jira/browse/HADOOP-18383 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.3.2 >Reporter: Kevin Sewell >Assignee: Kevin Sewell >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Compressors and Decompressions with a @DoNotPool annotation are not closed > when they are returned to the CodecPool, which causes a native memory leak. > > I have included a link to a [Demo > Project|https://github.com/kevins-29/hadoop-gzip-memory-leak] demonstrating > the leak -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-18383) Codecs with @DoNotPool annotation are not closed causing memory leak
[ https://issues.apache.org/jira/browse/HADOOP-18383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun reassigned HADOOP-18383: - Assignee: Kevin Sewell > Codecs with @DoNotPool annotation are not closed causing memory leak > > > Key: HADOOP-18383 > URL: https://issues.apache.org/jira/browse/HADOOP-18383 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.3.2 >Reporter: Kevin Sewell >Assignee: Kevin Sewell >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Compressors and Decompressions with a @DoNotPool annotation are not closed > when they are returned to the CodecPool, which causes a native memory leak. > > I have included a link to a [Demo > Project|https://github.com/kevins-29/hadoop-gzip-memory-leak] demonstrating > the leak -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18331) Shouldn't relocate org/wildfly/openssl in shaded client
[
https://issues.apache.org/jira/browse/HADOOP-18331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17564615#comment-17564615
]
Chao Sun commented on HADOOP-18331:
---
No I was on 3.3.2. I applied the same fix in HADOOP-18160 and it worked
afterwards.
> Shouldn't relocate org/wildfly/openssl in shaded client
> ---
>
> Key: HADOOP-18331
> URL: https://issues.apache.org/jira/browse/HADOOP-18331
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3, security
>Affects Versions: 3.3.3
>Reporter: Chao Sun
>Priority: Major
>
> We shouldn't shade {{org.wildfly.openssl}} since it's provided dependency and
> will give error like the following:
> {code}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/shaded/org/wildfly/openssl/OpenSSLProvider
> at
> org.apache.hadoop.security.ssl.DelegatingSSLSocketFactory.bindToOpenSSLProvider(DelegatingSSLSocketFactory.java:196)
> at
> org.apache.hadoop.security.ssl.DelegatingSSLSocketFactory.initializeSSLContext(DelegatingSSLSocketFactory.java:169)
> at
> org.apache.hadoop.security.ssl.DelegatingSSLSocketFactory.(DelegatingSSLSocketFactory.java:134)
> at
> org.apache.hadoop.security.ssl.DelegatingSSLSocketFactory.initializeDefaultFactory(DelegatingSSLSocketFactory.java:105)
> at
> org.apache.hadoop.fs.s3a.impl.NetworkBinding.bindSSLChannelMode(NetworkBinding.java:82)
> at
> org.apache.hadoop.fs.s3a.S3AUtils.initProtocolSettings(S3AUtils.java:1347)
> at
> org.apache.hadoop.fs.s3a.S3AUtils.initConnectionSettings(S3AUtils.java:1302)
> at org.apache.hadoop.fs.s3a.S3AUtils.createAwsConf(S3AUtils.java:1259)
> at
> org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:114)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:846)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:503)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18331) Shouldn't relocate org/wildfly/openssl in shaded client
[
https://issues.apache.org/jira/browse/HADOOP-18331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HADOOP-18331:
--
Affects Version/s: 3.3.2
(was: 3.3.3)
> Shouldn't relocate org/wildfly/openssl in shaded client
> ---
>
> Key: HADOOP-18331
> URL: https://issues.apache.org/jira/browse/HADOOP-18331
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3, security
>Affects Versions: 3.3.2
>Reporter: Chao Sun
>Priority: Major
>
> We shouldn't shade {{org.wildfly.openssl}} since it's provided dependency and
> will give error like the following:
> {code}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/shaded/org/wildfly/openssl/OpenSSLProvider
> at
> org.apache.hadoop.security.ssl.DelegatingSSLSocketFactory.bindToOpenSSLProvider(DelegatingSSLSocketFactory.java:196)
> at
> org.apache.hadoop.security.ssl.DelegatingSSLSocketFactory.initializeSSLContext(DelegatingSSLSocketFactory.java:169)
> at
> org.apache.hadoop.security.ssl.DelegatingSSLSocketFactory.(DelegatingSSLSocketFactory.java:134)
> at
> org.apache.hadoop.security.ssl.DelegatingSSLSocketFactory.initializeDefaultFactory(DelegatingSSLSocketFactory.java:105)
> at
> org.apache.hadoop.fs.s3a.impl.NetworkBinding.bindSSLChannelMode(NetworkBinding.java:82)
> at
> org.apache.hadoop.fs.s3a.S3AUtils.initProtocolSettings(S3AUtils.java:1347)
> at
> org.apache.hadoop.fs.s3a.S3AUtils.initConnectionSettings(S3AUtils.java:1302)
> at org.apache.hadoop.fs.s3a.S3AUtils.createAwsConf(S3AUtils.java:1259)
> at
> org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:114)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:846)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:503)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18331) Shouldn't relocate org/wildfly/openssl in shaded client
[
https://issues.apache.org/jira/browse/HADOOP-18331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun resolved HADOOP-18331.
---
Resolution: Duplicate
> Shouldn't relocate org/wildfly/openssl in shaded client
> ---
>
> Key: HADOOP-18331
> URL: https://issues.apache.org/jira/browse/HADOOP-18331
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3, security
>Affects Versions: 3.3.3
>Reporter: Chao Sun
>Priority: Major
>
> We shouldn't shade {{org.wildfly.openssl}} since it's provided dependency and
> will give error like the following:
> {code}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/shaded/org/wildfly/openssl/OpenSSLProvider
> at
> org.apache.hadoop.security.ssl.DelegatingSSLSocketFactory.bindToOpenSSLProvider(DelegatingSSLSocketFactory.java:196)
> at
> org.apache.hadoop.security.ssl.DelegatingSSLSocketFactory.initializeSSLContext(DelegatingSSLSocketFactory.java:169)
> at
> org.apache.hadoop.security.ssl.DelegatingSSLSocketFactory.(DelegatingSSLSocketFactory.java:134)
> at
> org.apache.hadoop.security.ssl.DelegatingSSLSocketFactory.initializeDefaultFactory(DelegatingSSLSocketFactory.java:105)
> at
> org.apache.hadoop.fs.s3a.impl.NetworkBinding.bindSSLChannelMode(NetworkBinding.java:82)
> at
> org.apache.hadoop.fs.s3a.S3AUtils.initProtocolSettings(S3AUtils.java:1347)
> at
> org.apache.hadoop.fs.s3a.S3AUtils.initConnectionSettings(S3AUtils.java:1302)
> at org.apache.hadoop.fs.s3a.S3AUtils.createAwsConf(S3AUtils.java:1259)
> at
> org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:114)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:846)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:503)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18331) Shouldn't relocate org/wildfly/openssl in shaded client
[
https://issues.apache.org/jira/browse/HADOOP-18331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17564591#comment-17564591
]
Chao Sun commented on HADOOP-18331:
---
Oops sorry, it's already resolved by HADOOP-18160
> Shouldn't relocate org/wildfly/openssl in shaded client
> ---
>
> Key: HADOOP-18331
> URL: https://issues.apache.org/jira/browse/HADOOP-18331
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3, security
>Affects Versions: 3.3.3
>Reporter: Chao Sun
>Priority: Major
>
> We shouldn't shade {{org.wildfly.openssl}} since it's provided dependency and
> will give error like the following:
> {code}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/shaded/org/wildfly/openssl/OpenSSLProvider
> at
> org.apache.hadoop.security.ssl.DelegatingSSLSocketFactory.bindToOpenSSLProvider(DelegatingSSLSocketFactory.java:196)
> at
> org.apache.hadoop.security.ssl.DelegatingSSLSocketFactory.initializeSSLContext(DelegatingSSLSocketFactory.java:169)
> at
> org.apache.hadoop.security.ssl.DelegatingSSLSocketFactory.(DelegatingSSLSocketFactory.java:134)
> at
> org.apache.hadoop.security.ssl.DelegatingSSLSocketFactory.initializeDefaultFactory(DelegatingSSLSocketFactory.java:105)
> at
> org.apache.hadoop.fs.s3a.impl.NetworkBinding.bindSSLChannelMode(NetworkBinding.java:82)
> at
> org.apache.hadoop.fs.s3a.S3AUtils.initProtocolSettings(S3AUtils.java:1347)
> at
> org.apache.hadoop.fs.s3a.S3AUtils.initConnectionSettings(S3AUtils.java:1302)
> at org.apache.hadoop.fs.s3a.S3AUtils.createAwsConf(S3AUtils.java:1259)
> at
> org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:114)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:846)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:503)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18331) Shouldn't relocate org/wildfly/openssl in shaded client
Chao Sun created HADOOP-18331:
-
Summary: Shouldn't relocate org/wildfly/openssl in shaded client
Key: HADOOP-18331
URL: https://issues.apache.org/jira/browse/HADOOP-18331
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3, security
Affects Versions: 3.3.3
Reporter: Chao Sun
We shouldn't shade {{org.wildfly.openssl}} since it's provided dependency and
will give error like the following:
{code}
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/shaded/org/wildfly/openssl/OpenSSLProvider
at
org.apache.hadoop.security.ssl.DelegatingSSLSocketFactory.bindToOpenSSLProvider(DelegatingSSLSocketFactory.java:196)
at
org.apache.hadoop.security.ssl.DelegatingSSLSocketFactory.initializeSSLContext(DelegatingSSLSocketFactory.java:169)
at
org.apache.hadoop.security.ssl.DelegatingSSLSocketFactory.(DelegatingSSLSocketFactory.java:134)
at
org.apache.hadoop.security.ssl.DelegatingSSLSocketFactory.initializeDefaultFactory(DelegatingSSLSocketFactory.java:105)
at
org.apache.hadoop.fs.s3a.impl.NetworkBinding.bindSSLChannelMode(NetworkBinding.java:82)
at
org.apache.hadoop.fs.s3a.S3AUtils.initProtocolSettings(S3AUtils.java:1347)
at
org.apache.hadoop.fs.s3a.S3AUtils.initConnectionSettings(S3AUtils.java:1302)
at org.apache.hadoop.fs.s3a.S3AUtils.createAwsConf(S3AUtils.java:1259)
at
org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:114)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:846)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:503)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18138) UserGroupInformation shouldn't log exception
Chao Sun created HADOOP-18138:
-
Summary: UserGroupInformation shouldn't log exception
Key: HADOOP-18138
URL: https://issues.apache.org/jira/browse/HADOOP-18138
Project: Hadoop Common
Issue Type: Improvement
Reporter: Chao Sun
Assignee: Chao Sun
In {{UserGroupInformation.doAs}}, we currently create a new {{Exception}} and
log it in {{LOG.debug}}. This doesn't look necessary:
{code}
if (LOG.isDebugEnabled()) {
LOG.debug("PrivilegedAction [as: {}][action: {}]", this, action,
new Exception());
}
{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18066) AbstractJavaKeyStoreProvider: need a way to read credential store password from Configuration
[
https://issues.apache.org/jira/browse/HADOOP-18066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HADOOP-18066:
--
Fix Version/s: (was: 3.3.2)
> AbstractJavaKeyStoreProvider: need a way to read credential store password
> from Configuration
> -
>
> Key: HADOOP-18066
> URL: https://issues.apache.org/jira/browse/HADOOP-18066
> Project: Hadoop Common
> Issue Type: Wish
> Components: security
>Reporter: László Bodor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> Codepath in focus is
> [this|https://github.com/apache/hadoop/blob/c3006be516ce7d4f970e24e7407b401318ceec3c/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.java#L316]
> {code}
> password = ProviderUtils.locatePassword(CREDENTIAL_PASSWORD_ENV_VAR,
> conf.get(CREDENTIAL_PASSWORD_FILE_KEY));
> {code}
> Since HIVE-14822, we can use custom keystore that Hiveserver2 propagates to
> jobs/tasks of different execution engines (mr, tez, spark).
> We're able to pass any "jceks:" url, but not a password, e.g. on this
> codepath:
> {code}
> Caused by: java.security.UnrecoverableKeyException: Password verification
> failed
> at com.sun.crypto.provider.JceKeyStore.engineLoad(JceKeyStore.java:879)
> ~[sunjce_provider.jar:1.8.0_232]
> at java.security.KeyStore.load(KeyStore.java:1445) ~[?:1.8.0_232]
> at
> org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.locateKeystore(AbstractJavaKeyStoreProvider.java:326)
> ~[hadoop-common-3.1.1.7.1.7.0-551.jar:?]
> at
> org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.(AbstractJavaKeyStoreProvider.java:86)
> ~[hadoop-common-3.1.1.7.1.7.0-551.jar:?]
> at
> org.apache.hadoop.security.alias.KeyStoreProvider.(KeyStoreProvider.java:49)
> ~[hadoop-common-3.1.1.7.1.7.0-551.jar:?]
> at
> org.apache.hadoop.security.alias.JavaKeyStoreProvider.(JavaKeyStoreProvider.java:42)
> ~[hadoop-common-3.1.1.7.1.7.0-551.jar:?]
> at
> org.apache.hadoop.security.alias.JavaKeyStoreProvider.(JavaKeyStoreProvider.java:35)
> ~[hadoop-common-3.1.1.7.1.7.0-551.jar:?]
> at
> org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory.createProvider(JavaKeyStoreProvider.java:68)
> ~[hadoop-common-3.1.1.7.1.7.0-551.jar:?]
> at
> org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:73)
> ~[hadoop-common-3.1.1.7.1.7.0-551.jar:?]
> at
> org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2409)
> ~[hadoop-common-3.1.1.7.1.7.0-551.jar:?]
> at
> org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2347)
> ~[hadoop-common-3.1.1.7.1.7.0-551.jar:?]
> at
> org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getPasswordString(AbfsConfiguration.java:295)
> ~[hadoop-azure-3.1.1.7.1.7.0-551.jar:?]
> at
> org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getTokenProvider(AbfsConfiguration.java:525)
> ~[hadoop-azure-3.1.1.7.1.7.0-551.jar:?]
> {code}
> Even there is a chance of reading a text file, it's not secure, we need to
> try reading a Configuration property first and if it's null, we can go to the
> environment variable.
> Hacking the System.getenv() is only possible with reflection, doesn't look so
> good.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17873) ABFS: Fix transient failures in ITestAbfsStreamStatistics and ITestAbfsRestOperationException
[ https://issues.apache.org/jira/browse/HADOOP-17873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-17873: -- Fix Version/s: (was: 3.3.2) > ABFS: Fix transient failures in ITestAbfsStreamStatistics and > ITestAbfsRestOperationException > - > > Key: HADOOP-17873 > URL: https://issues.apache.org/jira/browse/HADOOP-17873 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 7h > Remaining Estimate: 0h > > To address transient failures in the following test classes: > * ITestAbfsStreamStatistics: Uses a filesystem level instance to record > read/write statistics, which also tracks these operations in other tests. > running parallelly. To be marked for sequential run only to avoid transient > failure > * ITestAbfsRestOperationException: The use of a static member to track retry > count causes transient failures when two tests of this class happen to run > together. Switch to non-static variable for assertions on retry count -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17728) Fix issue of the StatisticsDataReferenceCleaner cleanUp
[
https://issues.apache.org/jira/browse/HADOOP-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HADOOP-17728:
--
Fix Version/s: (was: 3.3.2)
> Fix issue of the StatisticsDataReferenceCleaner cleanUp
> ---
>
> Key: HADOOP-17728
> URL: https://issues.apache.org/jira/browse/HADOOP-17728
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.2.1
>Reporter: yikf
>Assignee: yikf
>Priority: Minor
> Labels: pull-request-available, reverted
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> Cleaner thread will be blocked if we remove reference from ReferenceQueue
> unless the `queue.enqueue` called.
>
> As shown below, We call ReferenceQueue.remove() now while cleanUp, Call
> chain as follow：
> *StatisticsDataReferenceCleaner#queue.remove() ->
> ReferenceQueue.remove(0) -> lock.wait(0)*
> But, lock.notifyAll is called when queue.enqueue only, so Cleaner thread
> will be blocked.
>
> ThreadDump:
> {code:java}
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x7f7afc088800
> nid=0x2119 in Object.wait() [0x7f7b0023]
>java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xc00c2f58> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
> - locked <0xc00c2f58> (a java.lang.ref.Reference$Lock)
> at
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153){code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17728) Fix issue of the StatisticsDataReferenceCleaner cleanUp
[
https://issues.apache.org/jira/browse/HADOOP-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HADOOP-17728:
--
Fix Version/s: 3.3.2
> Fix issue of the StatisticsDataReferenceCleaner cleanUp
> ---
>
> Key: HADOOP-17728
> URL: https://issues.apache.org/jira/browse/HADOOP-17728
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.2.1
>Reporter: yikf
>Assignee: yikf
>Priority: Minor
> Labels: pull-request-available, reverted
> Fix For: 3.3.2
>
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> Cleaner thread will be blocked if we remove reference from ReferenceQueue
> unless the `queue.enqueue` called.
>
> As shown below, We call ReferenceQueue.remove() now while cleanUp, Call
> chain as follow：
> *StatisticsDataReferenceCleaner#queue.remove() ->
> ReferenceQueue.remove(0) -> lock.wait(0)*
> But, lock.notifyAll is called when queue.enqueue only, so Cleaner thread
> will be blocked.
>
> ThreadDump:
> {code:java}
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x7f7afc088800
> nid=0x2119 in Object.wait() [0x7f7b0023]
>java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xc00c2f58> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
> - locked <0xc00c2f58> (a java.lang.ref.Reference$Lock)
> at
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153){code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17988) Disable JIRA plugin for YETUS on Hadoop
[
https://issues.apache.org/jira/browse/HADOOP-17988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HADOOP-17988:
--
Fix Version/s: 3.3.2
(was: 3.3.3)
> Disable JIRA plugin for YETUS on Hadoop
> ---
>
> Key: HADOOP-17988
> URL: https://issues.apache.org/jira/browse/HADOOP-17988
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.3.3
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.3.2
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> I’ve been noticing an issue with Jenkins CI where a file jira-json goes
> missing all of a sudden – jenkins / hadoop-multibranch / PR-3588 / #2
> (apache.org)
> {code}
> [2021-10-27T17:52:58.787Z] Processing:
> https://github.com/apache/hadoop/pull/3588
> [2021-10-27T17:52:58.787Z] GITHUB PR #3588 is being downloaded from
> [2021-10-27T17:52:58.787Z]
> https://api.github.com/repos/apache/hadoop/pulls/3588
> [2021-10-27T17:52:58.787Z] JSON data at Wed Oct 27 17:52:55 UTC 2021
> [2021-10-27T17:52:58.787Z] Patch data at Wed Oct 27 17:52:56 UTC 2021
> [2021-10-27T17:52:58.787Z] Diff data at Wed Oct 27 17:52:56 UTC 2021
> [2021-10-27T17:52:59.814Z] awk: cannot open
> /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-3588/centos-7/out/jira-json
> (No such file or directory)
> [2021-10-27T17:52:59.814Z] ERROR: https://github.com/apache/hadoop/pull/3588
> issue status is not matched with "Patch Available".
> [2021-10-27T17:52:59.814Z]
> {code}
> This causes the pipeline run to fail. I’ve seen this in my multiple attempts
> to re-run the CI on my PR –
> # After 45 minutes – [jenkins / hadoop-multibranch / PR-3588 / #1
> (apache.org)|https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-3588/1/pipeline/]
> # After 1 minute – [jenkins / hadoop-multibranch / PR-3588 / #2
> (apache.org)|https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-3588/2/pipeline/]
> # After 17 minutes – [jenkins / hadoop-multibranch / PR-3588 / #3
> (apache.org)|https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-3588/3/pipeline/]
> The hadoop-multibranch pipeline doesn't use ASF JIRA, thus, we're disabling
> the *jira* plugin to fix this issue.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17290) ABFS: Add Identifiers to Client Request Header
[ https://issues.apache.org/jira/browse/HADOOP-17290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-17290: -- Fix Version/s: 3.3.2 (was: 3.3.3) > ABFS: Add Identifiers to Client Request Header > -- > > Key: HADOOP-17290 > URL: https://issues.apache.org/jira/browse/HADOOP-17290 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: abfsactive, pull-request-available > Fix For: 3.3.2 > > Time Spent: 10h 50m > Remaining Estimate: 0h > > Adding unique values to the client request header to assist in correlating > requests > the value of "fs.azure.client.correlationid" will be included in Azure > storage logs for requests -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17819) Add extensions to ProtobufRpcEngine RequestHeaderProto
[ https://issues.apache.org/jira/browse/HADOOP-17819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-17819: -- Fix Version/s: 3.3.2 > Add extensions to ProtobufRpcEngine RequestHeaderProto > -- > > Key: HADOOP-17819 > URL: https://issues.apache.org/jira/browse/HADOOP-17819 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Hector Sandoval Chaverri >Assignee: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 2.10.2, 3.2.3, 3.3.2 > > Time Spent: 0.5h > Remaining Estimate: 0h > > The header used in ProtobufRpcEngine messages doesn't allow for new > properties to be added by child classes. We can add a range of extensions > that can be useful for proto classes that need to extend RequestHeaderProto. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17195) Intermittent OutOfMemory error while performing hdfs CopyFromLocal to abfs
[ https://issues.apache.org/jira/browse/HADOOP-17195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-17195: -- Fix Version/s: 3.3.2 (was: 3.3.3) > Intermittent OutOfMemory error while performing hdfs CopyFromLocal to abfs > --- > > Key: HADOOP-17195 > URL: https://issues.apache.org/jira/browse/HADOOP-17195 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Mehakmeet Singh >Assignee: Mehakmeet Singh >Priority: Major > Labels: abfsactive, pull-request-available > Fix For: 3.3.2 > > Time Spent: 7h 50m > Remaining Estimate: 0h > > OutOfMemory error due to new ThreadPools being made each time > AbfsOutputStream is created. Since threadPool aren't limited a lot of data is > loaded in buffer and thus it causes OutOfMemory error. > Possible fixes: > - Limit the number of ThreadCounts while performing hdfs copyFromLocal (Using > -t property). > - Reducing OUTPUT_BUFFER_SIZE significantly which would limit the amount of > buffer to be loaded in threads. > - Don't create new ThreadPools each time AbfsOutputStream is created and > limit the number of ThreadPools each AbfsOutputStream could create. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17857) Check real user ACLs in addition to proxied user ACLs
[
https://issues.apache.org/jira/browse/HADOOP-17857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HADOOP-17857:
--
Fix Version/s: 3.3.2
(was: 3.3.3)
> Check real user ACLs in addition to proxied user ACLs
> -
>
> Key: HADOOP-17857
> URL: https://issues.apache.org/jira/browse/HADOOP-17857
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.2.2, 2.10.1, 3.3.1
>Reporter: Eric Payne
>Assignee: Eric Payne
>Priority: Major
> Fix For: 3.4.0, 2.10.2, 3.3.2, 3.2.4
>
> Attachments: HADOOP-17857.001.patch, HADOOP-17857.002.patch
>
>
> In a secure cluster, it is possible to configure the services to allow a
> super-user to proxy to a regular user and perform actions on behalf of the
> proxied user (see [Proxy user - Superusers Acting On Behalf Of Other
> Users|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html]).
> This is useful for automating server access for multiple different users in a
> multi-tenant cluster. For example, this can be used by a super user
> submitting jobs to a YARN queue, accessing HDFS files, scheduling Oozie
> workflows, etc, which will then execute the service as the proxied user.
> Usually when these services check ACLs to determine if the user has access to
> the requested resources, the service only needs to check the ACLs for the
> proxied user. However, it is sometimes desirable to allow the proxied user to
> have access to the resources when only the real user has open ACLs.
> For instance, let's say the user {{adm}} is the only user with submit ACLs to
> the {{dataload}} queue, and the {{adm}} user wants to submit apps to the
> {{dataload}} queue on behalf of users {{headless1}} and {{headless2}}. In
> addition, we want to be able to bill {{headless1}} and {{headless2}}
> separately for the YARN resources used in the {{dataload}} queue. In order to
> do this, the apps need to run in the {{dataload}} queue as the respective
> headless users. We could open up the ACLs to the {{dataload}} queue to allow
> {{headless1}} and {{headless2}} to submit apps. But this would allow those
> users to submit any app to that queue, and not be limited to just the data
> loading apps, and we don't trust the {{headless1}} and {{headless2}} owners
> to honor that restriction.
> This JIRA proposes that we define a way to set up ACLs to restrict a
> resource's access to a super-user, but when the access happens, run it as
> the proxied user.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17975) Fallback to simple auth does not work for a secondary DistributedFileSystem instance
[
https://issues.apache.org/jira/browse/HADOOP-17975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HADOOP-17975:
--
Fix Version/s: 3.3.2
(was: 3.3.3)
> Fallback to simple auth does not work for a secondary DistributedFileSystem
> instance
>
>
> Key: HADOOP-17975
> URL: https://issues.apache.org/jira/browse/HADOOP-17975
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Reporter: István Fajth
>Assignee: István Fajth
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.2, 3.2.4
>
> Time Spent: 10h 10m
> Remaining Estimate: 0h
>
> The following code snippet demonstrates what is necessary to cause a failure
> in connection to a non secure cluster with fallback to SIMPLE auth allowed
> from a secure cluster.
> {code:java}
> Configuration conf = new Configuration();
> conf.setBoolean("ipc.client.fallback-to-simple-auth-allowed", true);
> URI fsUri = new URI("hdfs://");
> conf.setBoolean("fs.hdfs.impl.disable.cache", true);
> FileSystem fs = FileSystem.get(fsUri, conf);
> FSDataInputStream src = fs.open(new Path("/path/to/a/file"));
> FileOutputStream dst = new FileOutputStream(File.createTempFile("foo",
> "bar"));
> IOUtils.copyBytes(src, dst, 1024);
> // The issue happens even if we re-enable cache at this point
> //conf.setBoolean("fs.hdfs.impl.disable.cache", false);
> // The issue does not happen when we close the first FileSystem object
> // before creating the second.
> //fs.close();
> FileSystem fs2 = FileSystem.get(fsUri, conf);
> FSDataInputStream src2 = fs2.open(new Path("/path/to/a/file"));
> FileOutputStream dst2 = new FileOutputStream(File.createTempFile("foo",
> "bar"));
> IOUtils.copyBytes(src2, dst2, 1024);
> {code}
> The problem is that when the DfsClient is created it creates an instance of
> AtomicBoolean, which is propagated down into the IPC layer, where the
> Client.Connection instance in setupIOStreams sets its value. This connection
> object is cached and re-used to multiplex requests against the same DataNode.
> In case of creating a second DfsClient, the AtomicBoolean reference in the
> client is a new AtomicBoolean, but the Client.Connection instance is the
> same, and as it has a socket already open to the DataNode, it returns
> immediatelly from setupIOStreams, leaving the fallbackToSimpleAuth
> AtomicBoolean false as it is created in the DfsClient.
> This AtomicBoolean on the other hand controls how the SaslDataTransferClient
> handles the connection in the above level, and with this value left on the
> default false, the SaslDataTransferClient of the second DfsClient will not
> fall back to SIMPLE authentication but will try to send a SASL handshake when
> connecting to the DataNode.
>
> The access to the FileSystem via the second DfsClient fails with exceptions
> like the following one, then fails the read with a BlockMissingException like
> below:
> {code}
> WARN hdfs.DFSClient: Failed to connect to /: for file
> for block BP-531773307--1634685133591:blk_1073741826_1002, add to
> deadNodes and continue.
> java.io.EOFException: Unexpected EOF while trying to read response from server
> at
> org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:552)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.readSaslMessage(DataTransferSaslUtil.java:215)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:455)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getSaslStreams(SaslDataTransferClient.java:393)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:267)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:215)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.peerSend(SaslDataTransferClient.java:160)
> at
> org.apache.hadoop.hdfs.DFSUtilClient.peerFromSocketAndKey(DFSUtilClient.java:648)
> at
> org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:2980)
> at
> org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:822)
> at
> org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:747)
> at
> org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.build(BlockReaderFactory.java:380)
> at
> org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInput
[jira] [Updated] (HADOOP-17198) Support S3 Access Points
[ https://issues.apache.org/jira/browse/HADOOP-17198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-17198: -- Fix Version/s: 3.3.2 (was: 3.3.3) > Support S3 Access Points > > > Key: HADOOP-17198 > URL: https://issues.apache.org/jira/browse/HADOOP-17198 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Bogdan Stolojan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 16.5h > Remaining Estimate: 0h > > Improve VPC integration by supporting access points for buckets > https://docs.aws.amazon.com/AmazonS3/latest/dev/access-points.html > *important*: when backporting, always include HADOOP-17951 as the followup > patch -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17939) Support building on Apple Silicon
[ https://issues.apache.org/jira/browse/HADOOP-17939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17489832#comment-17489832 ] Chao Sun commented on HADOOP-17939: --- Ehh NVM it is already there 5 months ago > Support building on Apple Silicon > -- > > Key: HADOOP-17939 > URL: https://issues.apache.org/jira/browse/HADOOP-17939 > Project: Hadoop Common > Issue Type: Improvement > Components: build, common >Affects Versions: 3.4.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17939) Support building on Apple Silicon
[ https://issues.apache.org/jira/browse/HADOOP-17939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17489826#comment-17489826 ] Chao Sun commented on HADOOP-17939: --- [~ste...@apache.org] are you going to backport this into branch-3.3.2? > Support building on Apple Silicon > -- > > Key: HADOOP-17939 > URL: https://issues.apache.org/jira/browse/HADOOP-17939 > Project: Hadoop Common > Issue Type: Improvement > Components: build, common >Affects Versions: 3.4.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18096) Distcp: Sync moves filtered file to home directory rather than deleting
[ https://issues.apache.org/jira/browse/HADOOP-18096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17486633#comment-17486633 ] Chao Sun commented on HADOOP-18096: --- Got it [~ayushtkn]. Any rough estimate when this will be resolved? > Distcp: Sync moves filtered file to home directory rather than deleting > --- > > Key: HADOOP-18096 > URL: https://issues.apache.org/jira/browse/HADOOP-18096 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Critical > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > Distcp sync with snapshot, if the file being copied is renamed to a path > which is in the exclusion filter, tries to delete the file. > But instead of deleting, it moves the file to home directory > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17198) Support S3 Access Points
[ https://issues.apache.org/jira/browse/HADOOP-17198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17484845#comment-17484845 ] Chao Sun commented on HADOOP-17198: --- [~bogthe] have asked this to be added in Hadoop 3.3.2 release. [~ste...@apache.org] do you think it's possible? > Support S3 Access Points > > > Key: HADOOP-17198 > URL: https://issues.apache.org/jira/browse/HADOOP-17198 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Bogdan Stolojan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 13h 40m > Remaining Estimate: 0h > > Improve VPC integration by supporting access points for buckets > https://docs.aws.amazon.com/AmazonS3/latest/dev/access-points.html > *important*: when backporting, always include HADOOP-17951 as the followup > patch -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18091) S3A auditing leaks memory through ThreadLocal references
[
https://issues.apache.org/jira/browse/HADOOP-18091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HADOOP-18091:
--
Priority: Blocker (was: Critical)
> S3A auditing leaks memory through ThreadLocal references
>
>
> Key: HADOOP-18091
> URL: https://issues.apache.org/jira/browse/HADOOP-18091
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
>
> {{ActiveAuditManagerS3A}} uses thread locals to map to active audit spans,
> which (because they are wrapped) include back reference to the audit manager
> instance and the config it was created with.
> these *do not* get cleaned up when the FS instance is closed.
> if you have a long lived process creating and destroying many FS instances,
> then memory gets used up. l
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16223) remove misleading fs.s3a.delegation.tokens.enabled prompt
[ https://issues.apache.org/jira/browse/HADOOP-16223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun resolved HADOOP-16223. --- Fix Version/s: 3.3.2 (was: 3.3.3) Hadoop Flags: Reviewed Resolution: Fixed > remove misleading fs.s3a.delegation.tokens.enabled prompt > - > > Key: HADOOP-16223 > URL: https://issues.apache.org/jira/browse/HADOOP-16223 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Priority: Minor > Labels: pull-request-available > Fix For: 3.3.2 > > Time Spent: 50m > Remaining Estimate: 0h > > There's a reference to the option fs.s3a.delegation.tokens.enabled, but > that's not used in the final HADOOP-14556, which runs straight off the > fs.s3a.delegation.token.binding value. > Remove the entry to avoid confusion -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18065) ExecutorHelper.logThrowableFromAfterExecute() is too noisy.
[
https://issues.apache.org/jira/browse/HADOOP-18065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HADOOP-18065:
--
Fix Version/s: 3.3.2
(was: 3.3.3)
> ExecutorHelper.logThrowableFromAfterExecute() is too noisy.
>
>
> Key: HADOOP-18065
> URL: https://issues.apache.org/jira/browse/HADOOP-18065
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.1
>Reporter: Mukund Thakur
>Assignee: Mukund Thakur
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.3.2
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> {code:java}
> if (t == null && r instanceof Future && ((Future) r).isDone()) {
> try {
> ((Future) r).get();
> } catch (ExecutionException ee) {
> LOG.warn(
> "Execution exception when running task in " + Thread.currentThread()
> .getName());
> t = ee.getCause();
> } catch (InterruptedException ie) {
> LOG.warn("Thread (" + Thread.currentThread() + ") interrupted: ", ie);
> Thread.currentThread().interrupt();
> } catch (Throwable throwable) {
> t = throwable;
> }
> }
> if (t != null) {
> LOG.warn("Caught exception in thread " + Thread
> .currentThread().getName() + ": ", t);
> } {code}
> We should downgrade the logging here from warn to debug.
>
> CC [~ste...@apache.org] [~mehakmeetSingh]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14334) S3 SSEC tests to downgrade when running against a mandatory encryption object store
[ https://issues.apache.org/jira/browse/HADOOP-14334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-14334: -- Fix Version/s: 3.3.2 (was: 3.3.3) > S3 SSEC tests to downgrade when running against a mandatory encryption > object store > > > Key: HADOOP-14334 > URL: https://issues.apache.org/jira/browse/HADOOP-14334 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Monthon Klongklaew >Priority: Minor > Labels: pull-request-available > Fix For: 3.3.2 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > If you run the S3A tests against a bucket with mandatory encryption, you need > to set the encryption in auth keys. This breaks the SSEC tests because the > encryption.key property being set breaks them. > That can be fixed (unset it), but they also need to handle failure from the > wrong encryption mech being used. Proposed: catch and downgrade to skipped. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18056) DistCp: Filter duplicates in the source paths
[ https://issues.apache.org/jira/browse/HADOOP-18056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-18056: -- Fix Version/s: 3.3.2 (was: 3.3.3) > DistCp: Filter duplicates in the source paths > - > > Key: HADOOP-18056 > URL: https://issues.apache.org/jira/browse/HADOOP-18056 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Add a basic filtering to remove the exact duplicate paths exposed for copying. > In case two same srcPath say /tmp/file1 is passed in the list twice. DistCp > fails with DuplicateFileException, post building the listing. > Would be better if we do a basic filtering of duplicate paths. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18031) Build arm64 (aarch64) and x86_64 image with the same Dockerfile
[
https://issues.apache.org/jira/browse/HADOOP-18031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HADOOP-18031:
--
Target Version/s: 3.4.0, 3.3.3 (was: 3.4.0, 3.3.2, 3.3.3)
> Build arm64 (aarch64) and x86_64 image with the same Dockerfile
> ---
>
> Key: HADOOP-18031
> URL: https://issues.apache.org/jira/browse/HADOOP-18031
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HADOOP-18031.branch-3.3.1.001.testing.patch
>
>
> -Support building Linux arm64 (aarch64) Docker images. And bump up some
> dependency versions.-
> -Note: This only provides a arm64 *runtime* environment for Hadoop 3. But not
> a full environment for compiling Hadoop 3 in arm64 yet. For the latter, gRPC
> may well need to be compiled from source because it hasn't started
> distributing Linux arm64 binaries yet.-
> -The patch for branch-3.3 is ready. I developed this patch on branch-3.3.1
> when I was trying to build arm64 Linux Hadoop Docker image. For trunk
> (3.4.0), due to HADOOP-17727, I need to post a different PR.-
> Just realized we already had {{Dockerfile_aarch64}}. Will try it out.
> My approach builds the Docker images for both architectures (x86_64 and
> aarch64) with the same {{Dockerfile}}.
> We should push the built arm64 image to Docker Hub. I only see amd64
> [there|https://hub.docker.com/r/apache/hadoop/tags] so I assumed we didn't
> have arm64 Docker image, hmm.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18065) ExecutorHelper.logThrowableFromAfterExecute() is too noisy.
[
https://issues.apache.org/jira/browse/HADOOP-18065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HADOOP-18065:
--
Target Version/s: 3.3.3 (was: 3.3.2)
> ExecutorHelper.logThrowableFromAfterExecute() is too noisy.
>
>
> Key: HADOOP-18065
> URL: https://issues.apache.org/jira/browse/HADOOP-18065
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.1
>Reporter: Mukund Thakur
>Assignee: Mukund Thakur
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {code:java}
> if (t == null && r instanceof Future && ((Future) r).isDone()) {
> try {
> ((Future) r).get();
> } catch (ExecutionException ee) {
> LOG.warn(
> "Execution exception when running task in " + Thread.currentThread()
> .getName());
> t = ee.getCause();
> } catch (InterruptedException ie) {
> LOG.warn("Thread (" + Thread.currentThread() + ") interrupted: ", ie);
> Thread.currentThread().interrupt();
> } catch (Throwable throwable) {
> t = throwable;
> }
> }
> if (t != null) {
> LOG.warn("Caught exception in thread " + Thread
> .currentThread().getName() + ": ", t);
> } {code}
> We should downgrade the logging here from warn to debug.
>
> CC [~ste...@apache.org] [~mehakmeetSingh]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18033) Upgrade fasterxml Jackson to 2.13.0
[ https://issues.apache.org/jira/browse/HADOOP-18033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-18033: -- Fix Version/s: 3.3.2 (was: 3.3.3) > Upgrade fasterxml Jackson to 2.13.0 > --- > > Key: HADOOP-18033 > URL: https://issues.apache.org/jira/browse/HADOOP-18033 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Akira Ajisaka >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 3h 40m > Remaining Estimate: 0h > > Spark 3.2.0 depends on Jackson 2.12.3. Let's upgrade to 2.12.5 (2.12.x latest > as of now) or upper. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18040) Use maven.test.failure.ignore instead of ignoreTestFailure
[ https://issues.apache.org/jira/browse/HADOOP-18040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-18040: -- Fix Version/s: 3.3.2 (was: 3.3.3) > Use maven.test.failure.ignore instead of ignoreTestFailure > -- > > Key: HADOOP-18040 > URL: https://issues.apache.org/jira/browse/HADOOP-18040 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 2.10.2, 3.3.2, 3.2.4 > > Time Spent: 50m > Remaining Estimate: 0h > > In HADOOP-16596, "ignoreTestFailure" variable was introduced to ignore unit > test failure, however, Maven property "maven.test.failure.ignore" can be used > instead and it can simplify the pom.xml. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13500) Synchronizing iteration of Configuration properties object
[ https://issues.apache.org/jira/browse/HADOOP-13500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-13500: -- Fix Version/s: 3.3.2 (was: 3.3.3) > Synchronizing iteration of Configuration properties object > -- > > Key: HADOOP-13500 > URL: https://issues.apache.org/jira/browse/HADOOP-13500 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Jason Darrell Lowe >Assignee: Dhananjay Badaya >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 2.10.2, 3.2.3, 3.3.2 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > It is possible to encounter a ConcurrentModificationException while trying to > iterate a Configuration object. The iterator method tries to walk the > underlying Property object without proper synchronization, so another thread > simultaneously calling the set method can trigger it. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18052) Support Apple Silicon in start-build-env.sh
[ https://issues.apache.org/jira/browse/HADOOP-18052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-18052: -- Fix Version/s: 3.3.2 (was: 3.3.3) > Support Apple Silicon in start-build-env.sh > --- > > Key: HADOOP-18052 > URL: https://issues.apache.org/jira/browse/HADOOP-18052 > Project: Hadoop Common > Issue Type: Improvement > Components: build > Environment: M1 Pro. MacOS 12.0.1. Docker for Mac. >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 40m > Remaining Estimate: 0h > > start-build-env.sh uses Dockerfile for x86 in M1 Mac, and the Dockerfile sets > wrong JAVA_HOME. Dockerfile_aarch64 should be used instead. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18045) Disable TestDynamometerInfra
[ https://issues.apache.org/jira/browse/HADOOP-18045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-18045: -- Fix Version/s: 3.3.2 (was: 3.3.3) > Disable TestDynamometerInfra > > > Key: HADOOP-18045 > URL: https://issues.apache.org/jira/browse/HADOOP-18045 > Project: Hadoop Common > Issue Type: Bug > Components: test >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 0.5h > Remaining Estimate: 0h > > This test is broken and there is no fix provided for a long time. Let's > disable the test to reduce the noise in the daily qbt job. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18063) Remove unused import AbstractJavaKeyStoreProvider in Shell class
[ https://issues.apache.org/jira/browse/HADOOP-18063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-18063: -- Fix Version/s: 3.3.2 (was: 3.3.3) > Remove unused import AbstractJavaKeyStoreProvider in Shell class > > > Key: HADOOP-18063 > URL: https://issues.apache.org/jira/browse/HADOOP-18063 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.4.0 >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2, 3.2.4 > > Attachments: image-2022-01-01-22-40-50-604.png > > Time Spent: 1h > Remaining Estimate: 0h > > In Shell, there are some invalid imports. > For example: > !image-2022-01-01-22-40-50-604.png! > Among them, AbstractJavaKeyStoreProvider does not seem to be referenced > anywhere. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13464) update GSON to 2.7+
[ https://issues.apache.org/jira/browse/HADOOP-13464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-13464: -- Fix Version/s: 3.3.2 (was: 3.3.3) > update GSON to 2.7+ > --- > > Key: HADOOP-13464 > URL: https://issues.apache.org/jira/browse/HADOOP-13464 > Project: Hadoop Common > Issue Type: Task > Components: build >Reporter: Sean Busbey >Assignee: Igor Dvorzhak >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2, 3.2.4 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > our GSON version is from ~3 years ago. update to latest release. > try to check release notes to see if this is incompatible. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18002) abfs rename idempotency broken -remove recovery
[ https://issues.apache.org/jira/browse/HADOOP-18002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-18002: -- Fix Version/s: 3.3.2 (was: 3.3.3) > abfs rename idempotency broken -remove recovery > --- > > Key: HADOOP-18002 > URL: https://issues.apache.org/jira/browse/HADOOP-18002 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.3.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 1.5h > Remaining Estimate: 0h > > rename idempotency logic of HADOOP-17105 is broken as modtimes aren't > uodated on rename. > remove, with the changes from the PR of HADOOP-17981. > also fix delete recovery test after HADOOP-17934 -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17999) No-op implementation of setWriteChecksum and setVerifyChecksum in ViewFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-17999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-17999: -- Fix Version/s: 3.3.2 (was: 3.3.3) > No-op implementation of setWriteChecksum and setVerifyChecksum in > ViewFileSystem > > > Key: HADOOP-17999 > URL: https://issues.apache.org/jira/browse/HADOOP-17999 > Project: Hadoop Common > Issue Type: Bug >Reporter: Abhishek Das >Assignee: Abhishek Das >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 2.10.2, 3.3.2, 3.2.4 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Currently setVerifyChecksum and setWriteChecksum initializes all target file > systems which causes delay in hadoop shell copy commands such as get, put, > copyFromLocal etc. > This also eventually causes OOM. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18026) Fix default value of Magic committer
[ https://issues.apache.org/jira/browse/HADOOP-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17468883#comment-17468883 ] Chao Sun commented on HADOOP-18026: --- [~stevel] could you add [~philipse] as contributor and assign this JIRA to him? thanks. > Fix default value of Magic committer > > > Key: HADOOP-18026 > URL: https://issues.apache.org/jira/browse/HADOOP-18026 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.3.1 >Reporter: guophilipse >Priority: Minor > Labels: pull-request-available > Fix For: 3.3.2 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > `fs.s3a.committer.magic.enabled` was set to true by default after > HADOOP-17483, we can improve the doc -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18026) Fix default value of Magic committer
[ https://issues.apache.org/jira/browse/HADOOP-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-18026: -- Fix Version/s: 3.3.2 (was: 3.3.3) > Fix default value of Magic committer > > > Key: HADOOP-18026 > URL: https://issues.apache.org/jira/browse/HADOOP-18026 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.3.1 >Reporter: guophilipse >Priority: Minor > Labels: pull-request-available > Fix For: 3.3.2 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > `fs.s3a.committer.magic.enabled` was set to true by default after > HADOOP-17483, we can improve the doc -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18048) [branch-3.3] Dockerfile_aarch64 build fails with fatal error: Python.h: No such file or directory
[ https://issues.apache.org/jira/browse/HADOOP-18048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-18048: -- Fix Version/s: 3.3.2 Resolution: Fixed Status: Resolved (was: Patch Available) > [branch-3.3] Dockerfile_aarch64 build fails with fatal error: Python.h: No > such file or directory > - > > Key: HADOOP-18048 > URL: https://issues.apache.org/jira/browse/HADOOP-18048 > Project: Hadoop Common > Issue Type: Bug >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available > Fix For: 3.3.2 > > Time Spent: 50m > Remaining Estimate: 0h > > See previous discussion: > https://issues.apache.org/jira/browse/HADOOP-17723?focusedCommentId=17452329&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17452329 -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17946) Update commons-lang to latest 3.x
[ https://issues.apache.org/jira/browse/HADOOP-17946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-17946: -- Fix Version/s: 3.3.2 > Update commons-lang to latest 3.x > - > > Key: HADOOP-17946 > URL: https://issues.apache.org/jira/browse/HADOOP-17946 > Project: Hadoop Common > Issue Type: Task >Reporter: Sean Busbey >Assignee: Renukaprasad C >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > our commons-lang3 dependency is currently 3.7, which is nearly 4 years old. > latest right now is 3.12 and there are at least some fixes that would make us > more robust on JDKs newer than openjdk8 (e.g. LANG-1384. [release notes > indicate 3.9 is the first to support > jdk11|https://commons.apache.org/proper/commons-lang/changes-report.html]). -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16963) HADOOP-16582 changed mkdirs() behavior
[
https://issues.apache.org/jira/browse/HADOOP-16963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HADOOP-16963:
--
Target Version/s: 3.4.0, 3.3.3 (was: 3.4.0, 3.3.2)
> HADOOP-16582 changed mkdirs() behavior
> --
>
> Key: HADOOP-16963
> URL: https://issues.apache.org/jira/browse/HADOOP-16963
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.10.0, 3.3.0, 2.8.6, 2.9.3, 3.1.3, 3.2.2
>Reporter: Wei-Chiu Chuang
>Priority: Critical
>
> HADOOP-16582 changed behavior of {{mkdirs()}}
> Some Hive tests depend on the old behavior and they fail miserably.
> {quote}
> earlier:
> all plain mkdirs(somePath) were fast-tracked to FileSystem.mkdirs which have
> rerouted them to mkdirs(somePath, somePerm) method with some defaults (which
> were static)
> an implementation of FileSystem have only needed implement "mkdirs(somePath,
> somePerm)" - because the other was not neccessarily called if it was always
> in a FilterFileSystem or something like that
> now:
> especially FilterFileSystem forwards the call of mkdirs(p) to the actual fs
> implementation...which may skip overriden mkdirs(somPath,somePerm) methods
> ...and could cause issues for existing FileSystem implementations
> {quote}
> File this jira to address this problem.
> [~kgyrtkirk] [~ste...@apache.org] [~kihwal]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17006) Fix the CosCrendentials Provider in core-site.xml for unit tests.
[ https://issues.apache.org/jira/browse/HADOOP-17006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-17006: -- Target Version/s: 3.3.3 (was: 3.3.2) > Fix the CosCrendentials Provider in core-site.xml for unit tests. > - > > Key: HADOOP-17006 > URL: https://issues.apache.org/jira/browse/HADOOP-17006 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/cos >Reporter: Yang Yu >Assignee: Yang Yu >Priority: Blocker > > Fix the CosCredentials Provider classpath in core-site.xml for unit tests. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17006) Fix the CosCrendentials Provider in core-site.xml for unit tests.
[ https://issues.apache.org/jira/browse/HADOOP-17006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17440764#comment-17440764 ] Chao Sun commented on HADOOP-17006: --- Hey [~yuyang733], any update on this? do you have a ETA? > Fix the CosCrendentials Provider in core-site.xml for unit tests. > - > > Key: HADOOP-17006 > URL: https://issues.apache.org/jira/browse/HADOOP-17006 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/cos >Reporter: Yang Yu >Assignee: Yang Yu >Priority: Blocker > > Fix the CosCredentials Provider classpath in core-site.xml for unit tests. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16752) ABFS: test failure testLastModifiedTime()
[ https://issues.apache.org/jira/browse/HADOOP-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-16752: -- Target Version/s: 3.3.3 (was: 3.3.2) > ABFS: test failure testLastModifiedTime() > - > > Key: HADOOP-16752 > URL: https://issues.apache.org/jira/browse/HADOOP-16752 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Da Zhou >Assignee: Sneha Vijayarajan >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > java.lang.AssertionError: lastModifiedTime should be after minCreateStartTime > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.assertTrue(Assert.java:41) > at > org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemFileStatus.testLastModifiedTime(ITestAzureBlobFileSystemFileStatus.java:138) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16955) Umbrella Jira for improving the Hadoop-cos support in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-16955: -- Target Version/s: 3.3.3 (was: 3.3.2) > Umbrella Jira for improving the Hadoop-cos support in Hadoop > > > Key: HADOOP-16955 > URL: https://issues.apache.org/jira/browse/HADOOP-16955 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/cos >Reporter: Yang Yu >Assignee: Yang Yu >Priority: Major > Attachments: HADOOP-16955-branch-3.3.001.patch > > Original Estimate: 48h > Time Spent: 4h > Remaining Estimate: 44h > > This Umbrella Jira focus on fixing some known bugs and adding some important > features. > > bugfix: > # resolve the dependency conflict; > # fix the upload buffer returning failed when some exceptions occur; > # fix the issue that the single file upload can not be retried; > # fix the bug of checking if a file exists through listing the file > frequently. > features: > # support SessionCredentialsProvider and InstanceCredentialsProvider, which > allows users to specify the credentials in URI or get it from the CVM > (Tencent Cloud Virtual Machine) bound to the CAM role that can access the COS > bucket; > # support the server encryption based on SSE-COS and SSE-C; > # support the HTTP proxy settings; > # support the storage class settings; > # support the CRC64 checksum. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16968) Optimizing the upload buffer in Hadoop-cos
[ https://issues.apache.org/jira/browse/HADOOP-16968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-16968: -- Target Version/s: 3.3.3 (was: 3.3.2) > Optimizing the upload buffer in Hadoop-cos > -- > > Key: HADOOP-16968 > URL: https://issues.apache.org/jira/browse/HADOOP-16968 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/cos >Reporter: Yang Yu >Assignee: Yang Yu >Priority: Major > Attachments: HADOOP-16968-branch-3.3.001.patch > > > This task focus on fixing the bug of returning an upload buffer failed when > some exceptions occur. > > What's more, the optimizing upload buffer management would be provided. > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16963) HADOOP-16582 changed mkdirs() behavior
[
https://issues.apache.org/jira/browse/HADOOP-16963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17440763#comment-17440763
]
Chao Sun commented on HADOOP-16963:
---
Hey [~weichiu], what's the status of this JIRA? I'm currently working on the
3.3.2 release.
> HADOOP-16582 changed mkdirs() behavior
> --
>
> Key: HADOOP-16963
> URL: https://issues.apache.org/jira/browse/HADOOP-16963
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.10.0, 3.3.0, 2.8.6, 2.9.3, 3.1.3, 3.2.2
>Reporter: Wei-Chiu Chuang
>Priority: Critical
>
> HADOOP-16582 changed behavior of {{mkdirs()}}
> Some Hive tests depend on the old behavior and they fail miserably.
> {quote}
> earlier:
> all plain mkdirs(somePath) were fast-tracked to FileSystem.mkdirs which have
> rerouted them to mkdirs(somePath, somePerm) method with some defaults (which
> were static)
> an implementation of FileSystem have only needed implement "mkdirs(somePath,
> somePerm)" - because the other was not neccessarily called if it was always
> in a FilterFileSystem or something like that
> now:
> especially FilterFileSystem forwards the call of mkdirs(p) to the actual fs
> implementation...which may skip overriden mkdirs(somPath,somePerm) methods
> ...and could cause issues for existing FileSystem implementations
> {quote}
> File this jira to address this problem.
> [~kgyrtkirk] [~ste...@apache.org] [~kihwal]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16970) Supporting the new credentials provider in Hadoop-cos
[ https://issues.apache.org/jira/browse/HADOOP-16970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-16970: -- Target Version/s: 3.3.3 (was: 3.3.2) > Supporting the new credentials provider in Hadoop-cos > - > > Key: HADOOP-16970 > URL: https://issues.apache.org/jira/browse/HADOOP-16970 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/cos >Reporter: Yang Yu >Assignee: Yang Yu >Priority: Major > > This task aims to support three credentials provider in Hadoop-cos: > * SessionCredentialsProvider > * InstanceCredentialsProvider -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17063) S3A deleteObjects hanging/retrying forever
[
https://issues.apache.org/jira/browse/HADOOP-17063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HADOOP-17063:
--
Target Version/s: 3.3.3 (was: 3.3.2)
> S3A deleteObjects hanging/retrying forever
> --
>
> Key: HADOOP-17063
> URL: https://issues.apache.org/jira/browse/HADOOP-17063
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.1
> Environment: hadoop 3.2.1
> spark 2.4.4
>
>Reporter: Dyno
>Priority: Minor
> Attachments: jstack_exec-34.log, jstack_exec-40.log,
> jstack_exec-74.log
>
>
> {code}
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:523)
> com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:82)
>
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream.putObject(S3ABlockOutputStream.java:446)
>
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream.close(S3ABlockOutputStream.java:365)
>
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
> org.apache.parquet.hadoop.util.HadoopPositionOutputStream.close(HadoopPositionOutputStream.java:64)
> org.apache.parquet.hadoop.ParquetFileWriter.end(ParquetFileWriter.java:685)
> org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:122)
>
> org.apache.parquet.hadoop.ParquetRecordWriter.close(ParquetRecordWriter.java:165)
>
> org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.close(ParquetOutputWriter.scala:42)
>
> org.apache.spark.sql.execution.datasources.FileFormatDataWriter.releaseResources(FileFormatDataWriter.scala:57)
>
> org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:74)
>
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:247)
>
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:242)
>
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394)
>
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:248)
>
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)
>
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
> org.apache.spark.scheduler.Task.run(Task.scala:123)
> org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
> org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> {code}
>
> we are using spark 2.4.4 with hadoop 3.2.1 on kubernetes/spark-operator,
> sometimes we see this hang with the stacktrace above. it looks like the
> putObject never return, we have to kill the executor to make the job move
> forward.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17583) Enable shelldoc check in GitHub PR
[ https://issues.apache.org/jira/browse/HADOOP-17583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-17583: -- Target Version/s: 3.4.0, 3.2.4, 3.3.3 (was: 3.4.0, 3.3.2, 3.2.4) > Enable shelldoc check in GitHub PR > -- > > Key: HADOOP-17583 > URL: https://issues.apache.org/jira/browse/HADOOP-17583 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Akira Ajisaka >Priority: Major > > After HADOOP-17570, we can enable shelldoc check again because the commit > hash of Yetus includes YETUS-1099. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17682) ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters
[ https://issues.apache.org/jira/browse/HADOOP-17682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-17682: -- Target Version/s: 3.3.3 (was: 3.3.2) > ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters > -- > > Key: HADOOP-17682 > URL: https://issues.apache.org/jira/browse/HADOOP-17682 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 8h 10m > Remaining Estimate: 0h > > ABFS open methods require certain information (contentLength, eTag, etc) to > to create an InputStream for the file at the given path. This information is > retrieved via a GetFileStatus request to backend. > However, client applications may often have access to the FileStatus prior to > invoking the open API. Providing this FileStatus to the driver through the > OpenFileParameters argument of openFileWithOptions() can help avoid the call > to Store for FileStatus. > This PR adds handling for the FileStatus instance (if any) provided via the > OpenFileParameters argument. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17446) Print the thread parker and lock information in stacks page
[ https://issues.apache.org/jira/browse/HADOOP-17446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-17446: -- Target Version/s: 3.4.0, 3.2.4, 3.3.3 (was: 3.4.0, 3.3.2, 3.2.4) > Print the thread parker and lock information in stacks page > --- > > Key: HADOOP-17446 > URL: https://issues.apache.org/jira/browse/HADOOP-17446 > Project: Hadoop Common > Issue Type: New Feature > Components: common >Affects Versions: 3.4.0 >Reporter: Baolong Mao >Assignee: Baolong Mao >Priority: Major > Labels: pull-request-available > Attachments: image-2020-12-25-08-32-32-982.png > > Time Spent: 0.5h > Remaining Estimate: 0h > > Sometimes, our service stuck because of some lock held by other thread, but > we can get nothing from "stacks" for ReadWriteLock, and it is widely used in > our services, like the fslock, cplock, dirlock of namenode. > Luckily, we can get thread parker from Thread object, it can help us see the > thread parker clearly. > !image-2020-12-25-08-32-32-982.png! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17500) S3A doesn't calculate Content-MD5 on uploads
[
https://issues.apache.org/jira/browse/HADOOP-17500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HADOOP-17500:
--
Target Version/s: 3.3.3 (was: 3.3.2)
> S3A doesn't calculate Content-MD5 on uploads
>
>
> Key: HADOOP-17500
> URL: https://issues.apache.org/jira/browse/HADOOP-17500
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Pedro Tôrres
>Priority: Major
>
> Hadoop doesn't specify the Content-MD5 of an object when uploading it to an
> S3 Bucket. This prevents uploads to buckets with Object Lock, that require
> the Content-MD5 to be specified.
>
> {code:java}
> com.amazonaws.services.s3.model.AmazonS3Exception: Content-MD5 HTTP header is
> required for Put Part requests with Object Lock parameters (Service: Amazon
> S3; Status Code: 400; Error Code: InvalidRequest; Request ID:
> ; S3 Extended Request ID:
> ;
> Proxy: null), S3 Extended Request ID:
>
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1819)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1403)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1372)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
> at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5248)
> at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5195)
> at
> com.amazonaws.services.s3.AmazonS3Client.doUploadPart(AmazonS3Client.java:3768)
> at
> com.amazonaws.services.s3.AmazonS3Client.uploadPart(AmazonS3Client.java:3753)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.uploadPart(S3AFileSystem.java:2230)
> at
> org.apache.hadoop.fs.s3a.WriteOperationHelper.lambda$uploadPart$8(WriteOperationHelper.java:558)
> at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:110)
> ... 15 more{code}
>
> Similar to https://issues.apache.org/jira/browse/JCLOUDS-1549
> Related to https://issues.apache.org/jira/browse/HADOOP-13076
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17543) HDFS Put was failed with IPV6 cluster
[ https://issues.apache.org/jira/browse/HADOOP-17543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-17543: -- Target Version/s: 3.3.3 (was: 3.3.2) > HDFS Put was failed with IPV6 cluster > -- > > Key: HADOOP-17543 > URL: https://issues.apache.org/jira/browse/HADOOP-17543 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: ANANDA G B >Priority: Minor > Labels: ipv6 > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17669) Port HADOOP-17079, HADOOP-17505 to branch-3.3
[ https://issues.apache.org/jira/browse/HADOOP-17669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-17669: -- Target Version/s: 3.3.3 (was: 3.3.2) > Port HADOOP-17079, HADOOP-17505 to branch-3.3 > -- > > Key: HADOOP-17669 > URL: https://issues.apache.org/jira/browse/HADOOP-17669 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.1 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17673) IOStatistics API in branch-3.3 break compatibility
[
https://issues.apache.org/jira/browse/HADOOP-17673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HADOOP-17673:
--
Target Version/s: 3.3.3 (was: 3.3.2)
> IOStatistics API in branch-3.3 break compatibility
> --
>
> Key: HADOOP-17673
> URL: https://issues.apache.org/jira/browse/HADOOP-17673
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Steve Loughran
>Priority: Critical
> Labels: release-blocker
>
> The S3 delegation token feature (3.3.0) added API
> {code:java}
> AmazonS3 createS3Client(URI name,
> String bucket,
> AWSCredentialsProvider credentialSet,
> String userAgentSuffix) throws IOException;
> {code}
> However, the IOStatistics API (HADOOP-17271, HADOOP-13551. in 3.3.1) changed
> it to
> {code:java}
> AmazonS3 createS3Client(URI name,
> String bucket,
> AWSCredentialsProvider credentialSet,
> String userAgentSuffix) throws IOException; {code}
> The API is declared evolving, so we're not supposed to break compat between
> maintenance releases.
> [~ste...@apache.org]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17717) Update wildfly openssl to 1.1.3.Final
[ https://issues.apache.org/jira/browse/HADOOP-17717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-17717: -- Target Version/s: 3.3.3 (was: 3.3.2) > Update wildfly openssl to 1.1.3.Final > - > > Key: HADOOP-17717 > URL: https://issues.apache.org/jira/browse/HADOOP-17717 > Project: Hadoop Common > Issue Type: Bug >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > HADOOP-17649 got stalled. IMO we can bump the version to 1.1.3.Final instead, > at least, for branch-3.3. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17915) ABFS AbfsDelegationTokenManager to generate canonicalServiceName if DT plugin doesn't
[
https://issues.apache.org/jira/browse/HADOOP-17915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HADOOP-17915:
--
Target Version/s: 3.3.3 (was: 3.3.2)
> ABFS AbfsDelegationTokenManager to generate canonicalServiceName if DT plugin
> doesn't
> -
>
> Key: HADOOP-17915
> URL: https://issues.apache.org/jira/browse/HADOOP-17915
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.3.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Currently in {{AbfsDelegationTokenManager}}, any
> {{CustomDelegationTokenManager}} only provides a canonical service name if it
> implements {{BoundDTExtension}} and its {{getCanonicalServiceName()}} method.
> If this doesn't hold, {{AbfsDelegationTokenManager}} returns null, which
> causes {{AzureBlobFileSystem.getCanonicalServiceName()}}
> to call {{super.getCanonicalServiceName()}} *which resolves the IP address of
> the abfs endpoint, and then the FQDN of that IPAddr
> If a storage account is served over >1 endpoint, then the DT will only have a
> valid service name for one of the possible
> endpoints, so _only work if all process get the same IP address when the look
> up the storage account address_
> Fix
> # DT plugins SHOULD generate the canonical service name
> # If they don't, and DTs are enabled: {{AbfsDelegationTokenManager}} to
> create a default one
> # and {{AzureBlobFileSystem.getCanonicalServiceName()}} MUST NOT call
> superclass.
> The default canonical service name of a store will be {{abfs:// +
> FsURI.getHost() + "/"}}, so all containers in same storage account has the
> same service name
> {code}
> abfs://buc...@stevel-testing.dfs.core.windows.net/path
> {code}
> maps to
> {code}
> abfs://stevel-testing.dfs.core.windows.net/
> {code}
> This will mean that only one DT will be created per storage a/c; Applications
> will not need to list all containers which deployed processes will wish to
> interact with. Today's behaviour, based on rDNS lookup of storage account, is
> possibly slightly broader in that all storage accounts which map to the same
> IPAddr share a DT. The proposed scheme will still be much broader than that
> of S3A, where every bucket has its unique service name, so apps need to list
> all target filesystems at launch time (easy for MR, source of trouble in
> spark).
> Fix: straightforward.
> Test
> * no DTs: service name == null
> * DTs: will match proposed pattern, even if extension returns null.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17928) s3a: set fs.s3a.downgrade.syncable.exceptions = true by default
[ https://issues.apache.org/jira/browse/HADOOP-17928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-17928: -- Target Version/s: 3.3.3 (was: 3.3.2) > s3a: set fs.s3a.downgrade.syncable.exceptions = true by default > --- > > Key: HADOOP-17928 > URL: https://issues.apache.org/jira/browse/HADOOP-17928 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 3.5h > Remaining Estimate: 0h > > HADOOP-17597 set policy of reacting to hsync() on an s3 output stream to be > one of :Fail, warn, with default == fail. > I propose downgrading this to warn. We've done it internally, after having it > on fail long enough to identify which processes were doing either of > * having unrealistic expectations about the output stream (fix: move off s3) > * were using hflush() as a variant of flush(), with the failure being an > over-reaction -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-17702) HADOOP-17065 breaks compatibility between 3.3.0 and 3.3.1
[
https://issues.apache.org/jira/browse/HADOOP-17702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17436967#comment-17436967
]
Chao Sun edited comment on HADOOP-17702 at 11/1/21, 6:30 PM:
-
Hi [~ste...@apache.org] [~weichiu], is this still a blocker for 3.3.2?
was (Author: csun):
Hi, is this still a blocker for 3.3.2?
> HADOOP-17065 breaks compatibility between 3.3.0 and 3.3.1
> -
>
> Key: HADOOP-17702
> URL: https://issues.apache.org/jira/browse/HADOOP-17702
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 3.3.1
>Reporter: Wei-Chiu Chuang
>Priority: Blocker
>
> AzureBlobFileSystemStore is a Public, Evolving class, by contract can't break
> compatibility between maintenance releases.
> HADOOP-17065 changed its constructor signature from
> {noformat}
> public AzureBlobFileSystemStore(URI uri, boolean isSecureScheme,
> Configuration configuration,
> AbfsCounters abfsCounters) throws
> IOException {
> {noformat}
> to
> {noformat}
> public AzureBlobFileSystemStore(URI uri, boolean isSecureScheme,
> Configuration configuration,
> AbfsCounters abfsCounters) throws
> IOException {
> {noformat}
> between 3.3.0 and 3.3.1.
> cc: [~mehakmeetSingh], [~tmarquardt]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17006) Fix the CosCrendentials Provider in core-site.xml for unit tests.
[ https://issues.apache.org/jira/browse/HADOOP-17006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17436971#comment-17436971 ] Chao Sun commented on HADOOP-17006: --- [~yuyang733] [~weichiu] what's the progress on this one? still a blocker? I'm working on the 3.3.2 release right now. > Fix the CosCrendentials Provider in core-site.xml for unit tests. > - > > Key: HADOOP-17006 > URL: https://issues.apache.org/jira/browse/HADOOP-17006 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/cos >Reporter: Yang Yu >Assignee: Yang Yu >Priority: Blocker > > Fix the CosCredentials Provider classpath in core-site.xml for unit tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-7370) Optimize pread on ChecksumFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-7370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17436969#comment-17436969 ] Chao Sun commented on HADOOP-7370: -- [~ste...@apache.org] what's the progress on this? is it still a blocker for 3.3.2? I'm working on the release right now. > Optimize pread on ChecksumFileSystem > > > Key: HADOOP-7370 > URL: https://issues.apache.org/jira/browse/HADOOP-7370 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Blocker > Attachments: checksumfs-pread-0.20.txt > > > Currently the implementation of positional read in ChecksumFileSystem is > verify inefficient - it actually re-opens the underlying file and checksum > file, then seeks and uses normal read. Instead, it can push down positional > read directly to the underlying FS and verify checksum. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17702) HADOOP-17065 breaks compatibility between 3.3.0 and 3.3.1
[
https://issues.apache.org/jira/browse/HADOOP-17702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17436967#comment-17436967
]
Chao Sun commented on HADOOP-17702:
---
Hi, is this still a blocker for 3.3.2?
> HADOOP-17065 breaks compatibility between 3.3.0 and 3.3.1
> -
>
> Key: HADOOP-17702
> URL: https://issues.apache.org/jira/browse/HADOOP-17702
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 3.3.1
>Reporter: Wei-Chiu Chuang
>Priority: Blocker
>
> AzureBlobFileSystemStore is a Public, Evolving class, by contract can't break
> compatibility between maintenance releases.
> HADOOP-17065 changed its constructor signature from
> {noformat}
> public AzureBlobFileSystemStore(URI uri, boolean isSecureScheme,
> Configuration configuration,
> AbfsCounters abfsCounters) throws
> IOException {
> {noformat}
> to
> {noformat}
> public AzureBlobFileSystemStore(URI uri, boolean isSecureScheme,
> Configuration configuration,
> AbfsCounters abfsCounters) throws
> IOException {
> {noformat}
> between 3.3.0 and 3.3.1.
> cc: [~mehakmeetSingh], [~tmarquardt]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17936) TestLocalFSCopyFromLocal.testDestinationFileIsToParentDirectory failure after reverting HADOOP-16878
[
https://issues.apache.org/jira/browse/HADOOP-17936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun resolved HADOOP-17936.
---
Fix Version/s: 3.3.2
3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
> TestLocalFSCopyFromLocal.testDestinationFileIsToParentDirectory failure after
> reverting HADOOP-16878
>
>
> Key: HADOOP-17936
> URL: https://issues.apache.org/jira/browse/HADOOP-17936
> Project: Hadoop Common
> Issue Type: Test
>Reporter: Chao Sun
>Assignee: Chao Sun
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.2
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> After reverting HADOOP-16878 from branch-3.3, test
> {{TestLocalFSCopyFromLocal.testDestinationFileIsToParentDirectory}} started
> to fail because it expects an exception but the copying succeeded.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17936) Fix test failure after reverting HADOOP-16878
[
https://issues.apache.org/jira/browse/HADOOP-17936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HADOOP-17936:
--
Summary: Fix test failure after reverting HADOOP-16878 (was: Fix test
failure after reverting HADOOP-16878 from branch-3.3)
> Fix test failure after reverting HADOOP-16878
> -
>
> Key: HADOOP-17936
> URL: https://issues.apache.org/jira/browse/HADOOP-17936
> Project: Hadoop Common
> Issue Type: Test
>Reporter: Chao Sun
>Assignee: Chao Sun
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> After reverting HADOOP-16878 from branch-3.3, test
> {{TestLocalFSCopyFromLocal.testDestinationFileIsToParentDirectory}} started
> to fail because it expects an exception but the copying succeeded.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17936) Fix test failure after reverting HADOOP-16878 from branch-3.3
[
https://issues.apache.org/jira/browse/HADOOP-17936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HADOOP-17936:
--
Fix Version/s: (was: 3.3.2)
> Fix test failure after reverting HADOOP-16878 from branch-3.3
> -
>
> Key: HADOOP-17936
> URL: https://issues.apache.org/jira/browse/HADOOP-17936
> Project: Hadoop Common
> Issue Type: Test
>Reporter: Chao Sun
>Assignee: Chao Sun
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> After reverting HADOOP-16878 from branch-3.3, test
> {{TestLocalFSCopyFromLocal.testDestinationFileIsToParentDirectory}} started
> to fail because it expects an exception but the copying succeeded.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17936) Fix test failure after reverting HADOOP-16878 from branch-3.3
Chao Sun created HADOOP-17936:
-
Summary: Fix test failure after reverting HADOOP-16878 from
branch-3.3
Key: HADOOP-17936
URL: https://issues.apache.org/jira/browse/HADOOP-17936
Project: Hadoop Common
Issue Type: Test
Reporter: Chao Sun
Assignee: Chao Sun
Fix For: 3.3.2
After reverting HADOOP-16878 from branch-3.3, test
{{TestLocalFSCopyFromLocal.testDestinationFileIsToParentDirectory}} started to
fail because it expects an exception but the copying succeeded.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17868) Add more test for the BuiltInGzipCompressor
[ https://issues.apache.org/jira/browse/HADOOP-17868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun resolved HADOOP-17868. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Assignee: L. C. Hsieh Resolution: Fixed > Add more test for the BuiltInGzipCompressor > --- > > Key: HADOOP-17868 > URL: https://issues.apache.org/jira/browse/HADOOP-17868 > Project: Hadoop Common > Issue Type: Test >Reporter: L. C. Hsieh >Assignee: L. C. Hsieh >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 6h 10m > Remaining Estimate: 0h > > We added BuiltInGzipCompressor recently. It is better to add more > compatibility tests for the compressor. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17418363#comment-17418363
]
Chao Sun commented on HADOOP-17891:
---
Also check SPARK-36669 for a workaround on this (linked to this JIRA).
> lz4-java and snappy-java should be excluded from relocation in shaded Hadoop
> libraries
> --
>
> Key: HADOOP-17891
> URL: https://issues.apache.org/jira/browse/HADOOP-17891
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.3.1
>Reporter: L. C. Hsieh
>Assignee: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.2
>
> Attachments: HADOOP-17891-Addendum-01.patch
>
> Time Spent: 17h 40m
> Remaining Estimate: 0h
>
> lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g.
> hadoop-client-api, if we don't exclude lz4 dependency, the downstream will
> still see the exception even they include lz4 dependency.
> {code:java}
> [info] Cause: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
> [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> [info] at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.(Lz4Compressor.java:66)
> [info] at
> org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
> {code}
> Currently snappy-java is included and relocated in Hadoop shaded client
> libraries. But as it includes native methods, it should not be relocated too
> due to JNI method resolution. The downstream will see the exception:
> {code}
> [info] Cause: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
> Method)
>
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
>
>
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17418362#comment-17418362
]
Chao Sun commented on HADOOP-17891:
---
Hi [~ivan.sadikov], it should be fixed in the upcoming Hadoop 3.3.2 release. I
plan to work on it soon. Feel free to subscribe to common-...@hadoop.apache.org
for update on this.
> lz4-java and snappy-java should be excluded from relocation in shaded Hadoop
> libraries
> --
>
> Key: HADOOP-17891
> URL: https://issues.apache.org/jira/browse/HADOOP-17891
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.3.1
>Reporter: L. C. Hsieh
>Assignee: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.2
>
> Attachments: HADOOP-17891-Addendum-01.patch
>
> Time Spent: 17h 40m
> Remaining Estimate: 0h
>
> lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g.
> hadoop-client-api, if we don't exclude lz4 dependency, the downstream will
> still see the exception even they include lz4 dependency.
> {code:java}
> [info] Cause: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
> [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> [info] at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.(Lz4Compressor.java:66)
> [info] at
> org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
> {code}
> Currently snappy-java is included and relocated in Hadoop shaded client
> libraries. But as it includes native methods, it should not be relocated too
> due to JNI method resolution. The downstream will see the exception:
> {code}
> [info] Cause: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
> Method)
>
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
>
>
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17833) Improve Magic Committer Performance
[ https://issues.apache.org/jira/browse/HADOOP-17833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17417773#comment-17417773 ] Chao Sun commented on HADOOP-17833: --- [~ste...@apache.org] is this targeting 3.3.2 release? since it is a point release I'm thinking perhaps we should stick to bug fixes and avoid putting new features or improvements (esp. since this looks like a big PR). > Improve Magic Committer Performance > --- > > Key: HADOOP-17833 > URL: https://issues.apache.org/jira/browse/HADOOP-17833 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.1 >Reporter: Steve Loughran >Priority: Minor > Labels: pull-request-available > Time Spent: 5h 10m > Remaining Estimate: 0h > > Magic committer tasks can be slow because every file created with > overwrite=false triggers a HEAD (verify there's no file) and a LIST (that > there's no dir). And because of delayed manifestations, it may not behave as > expected. > ParquetOutputFormat is one example of a library which does this. > we could fix parquet to use overwrite=true, but (a) there may be surprises in > other uses (b) it'd still leave the list and (c) do nothing for other formats > call > Proposed: createFile() under a magic path to skip all probes for file/dir at > end of path > Only a single task attempt Will be writing to that directory and it should > know what it is doing. If there is conflicting file names and parts across > tasks that won't even get picked up at this point. Oh and none of the > committers ever check for this: you'll get the last file manifested (s3a) or > renamed (file) > If we skip the checks we will save 2 HTTP requests/file. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415752#comment-17415752
]
Chao Sun commented on HADOOP-17891:
---
Thanks [~ebadger] and [~ayushtkn]! sorry for introducing the bug. We had a test
in the PR but it mis-used {{noshade}} for {{skipShade}} and didn't catch it.
[~viirya]: do you mind open another PR to address the issue. You can follow the
format of this: https://github.com/apache/hadoop/pull/3276. Thanks.
> lz4-java and snappy-java should be excluded from relocation in shaded Hadoop
> libraries
> --
>
> Key: HADOOP-17891
> URL: https://issues.apache.org/jira/browse/HADOOP-17891
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.3.1
>Reporter: L. C. Hsieh
>Assignee: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.2
>
> Attachments: HADOOP-17891-Addendum-01.patch
>
> Time Spent: 14h
> Remaining Estimate: 0h
>
> lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g.
> hadoop-client-api, if we don't exclude lz4 dependency, the downstream will
> still see the exception even they include lz4 dependency.
> {code:java}
> [info] Cause: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
> [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> [info] at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.(Lz4Compressor.java:66)
> [info] at
> org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
> {code}
> Currently snappy-java is included and relocated in Hadoop shaded client
> libraries. But as it includes native methods, it should not be relocated too
> due to JNI method resolution. The downstream will see the exception:
> {code}
> [info] Cause: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
> Method)
>
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
>
>
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415252#comment-17415252
]
Chao Sun commented on HADOOP-17891:
---
Could you share your build command? I just did {{mvn clean install
-DskipTests}} and it worked for me.
> lz4-java and snappy-java should be excluded from relocation in shaded Hadoop
> libraries
> --
>
> Key: HADOOP-17891
> URL: https://issues.apache.org/jira/browse/HADOOP-17891
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.3.1
>Reporter: L. C. Hsieh
>Assignee: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.2
>
> Time Spent: 14h
> Remaining Estimate: 0h
>
> lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g.
> hadoop-client-api, if we don't exclude lz4 dependency, the downstream will
> still see the exception even they include lz4 dependency.
> {code:java}
> [info] Cause: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
> [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> [info] at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.(Lz4Compressor.java:66)
> [info] at
> org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
> {code}
> Currently snappy-java is included and relocated in Hadoop shaded client
> libraries. But as it includes native methods, it should not be relocated too
> due to JNI method resolution. The downstream will see the exception:
> {code}
> [info] Cause: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
> Method)
>
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
>
>
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415245#comment-17415245
]
Chao Sun commented on HADOOP-17891:
---
Hmm interesting, I checked the yetus result and it's a +1 there.
> lz4-java and snappy-java should be excluded from relocation in shaded Hadoop
> libraries
> --
>
> Key: HADOOP-17891
> URL: https://issues.apache.org/jira/browse/HADOOP-17891
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.3.1
>Reporter: L. C. Hsieh
>Assignee: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.2
>
> Time Spent: 14h
> Remaining Estimate: 0h
>
> lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g.
> hadoop-client-api, if we don't exclude lz4 dependency, the downstream will
> still see the exception even they include lz4 dependency.
> {code:java}
> [info] Cause: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
> [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> [info] at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.(Lz4Compressor.java:66)
> [info] at
> org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
> {code}
> Currently snappy-java is included and relocated in Hadoop shaded client
> libraries. But as it includes native methods, it should not be relocated too
> due to JNI method resolution. The downstream will see the exception:
> {code}
> [info] Cause: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
> Method)
>
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
>
>
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HADOOP-17891:
--
Fix Version/s: 3.3.2
> lz4-java and snappy-java should be excluded from relocation in shaded Hadoop
> libraries
> --
>
> Key: HADOOP-17891
> URL: https://issues.apache.org/jira/browse/HADOOP-17891
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.3.1
>Reporter: L. C. Hsieh
>Assignee: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.2
>
> Time Spent: 14h
> Remaining Estimate: 0h
>
> lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g.
> hadoop-client-api, if we don't exclude lz4 dependency, the downstream will
> still see the exception even they include lz4 dependency.
> {code:java}
> [info] Cause: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
> [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> [info] at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.(Lz4Compressor.java:66)
> [info] at
> org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
> {code}
> Currently snappy-java is included and relocated in Hadoop shaded client
> libraries. But as it includes native methods, it should not be relocated too
> due to JNI method resolution. The downstream will see the exception:
> {code}
> [info] Cause: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
> Method)
>
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
>
>
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HADOOP-17891:
--
Affects Version/s: 3.3.1
> lz4-java and snappy-java should be excluded from relocation in shaded Hadoop
> libraries
> --
>
> Key: HADOOP-17891
> URL: https://issues.apache.org/jira/browse/HADOOP-17891
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.3.1
>Reporter: L. C. Hsieh
>Assignee: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 13h 50m
> Remaining Estimate: 0h
>
> lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g.
> hadoop-client-api, if we don't exclude lz4 dependency, the downstream will
> still see the exception even they include lz4 dependency.
> {code:java}
> [info] Cause: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
> [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> [info] at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.(Lz4Compressor.java:66)
> [info] at
> org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
> {code}
> Currently snappy-java is included and relocated in Hadoop shaded client
> libraries. But as it includes native methods, it should not be relocated too
> due to JNI method resolution. The downstream will see the exception:
> {code}
> [info] Cause: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
> Method)
>
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
>
>
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun resolved HADOOP-17891.
---
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Assignee: L. C. Hsieh
Resolution: Fixed
> lz4-java and snappy-java should be excluded from relocation in shaded Hadoop
> libraries
> --
>
> Key: HADOOP-17891
> URL: https://issues.apache.org/jira/browse/HADOOP-17891
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: L. C. Hsieh
>Assignee: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 13h 50m
> Remaining Estimate: 0h
>
> lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g.
> hadoop-client-api, if we don't exclude lz4 dependency, the downstream will
> still see the exception even they include lz4 dependency.
> {code:java}
> [info] Cause: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
> [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> [info] at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.(Lz4Compressor.java:66)
> [info] at
> org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
> {code}
> Currently snappy-java is included and relocated in Hadoop shaded client
> libraries. But as it includes native methods, it should not be relocated too
> due to JNI method resolution. The downstream will see the exception:
> {code}
> [info] Cause: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
> Method)
>
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
>
>
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17825) Add BuiltInGzipCompressor
[ https://issues.apache.org/jira/browse/HADOOP-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-17825: -- Component/s: common > Add BuiltInGzipCompressor > - > > Key: HADOOP-17825 > URL: https://issues.apache.org/jira/browse/HADOOP-17825 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: L. C. Hsieh >Assignee: L. C. Hsieh >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 25h 20m > Remaining Estimate: 0h > > Currently, GzipCodec only supports BuiltInGzipDecompressor, if native zlib is > not loaded. So, without Hadoop native codec installed, saving SequenceFile > using GzipCodec will throw exception like "SequenceFile doesn't work with > GzipCodec without native-hadoop code!" > Same as other codecs which we migrated to using prepared packages (lz4, > snappy), it will be better if we support GzipCodec generally without Hadoop > native codec installed. Similar to BuiltInGzipDecompressor, we can use Java > Deflater to support BuiltInGzipCompressor. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17877) BuiltInGzipCompressor header and trailer should not be static variables
[ https://issues.apache.org/jira/browse/HADOOP-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun resolved HADOOP-17877. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Assignee: L. C. Hsieh Resolution: Fixed > BuiltInGzipCompressor header and trailer should not be static variables > --- > > Key: HADOOP-17877 > URL: https://issues.apache.org/jira/browse/HADOOP-17877 > Project: Hadoop Common > Issue Type: Bug >Reporter: L. C. Hsieh >Assignee: L. C. Hsieh >Priority: Critical > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 2h > Remaining Estimate: 0h > > In the newly added BuiltInGzipCompressor, we should not let header and > trailer as static variables as they are for different instances. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17887) Remove GzipOutputStream
[ https://issues.apache.org/jira/browse/HADOOP-17887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun resolved HADOOP-17887. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Assignee: L. C. Hsieh Resolution: Fixed > Remove GzipOutputStream > --- > > Key: HADOOP-17887 > URL: https://issues.apache.org/jira/browse/HADOOP-17887 > Project: Hadoop Common > Issue Type: Improvement >Reporter: L. C. Hsieh >Assignee: L. C. Hsieh >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 4h > Remaining Estimate: 0h > > As we provide built-in gzip compressor, we can use it in compressor stream. > The wrapper GzipOutputStream can be removed now. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17877) BuiltInGzipCompressor header and trailer should not be static variables
[ https://issues.apache.org/jira/browse/HADOOP-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-17877: -- Priority: Critical (was: Major) > BuiltInGzipCompressor header and trailer should not be static variables > --- > > Key: HADOOP-17877 > URL: https://issues.apache.org/jira/browse/HADOOP-17877 > Project: Hadoop Common > Issue Type: Bug >Reporter: L. C. Hsieh >Priority: Critical > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > In the newly added BuiltInGzipCompressor, we should not let header and > trailer as static variables as they are for different instances. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17825) Add BuiltInGzipCompressor
[ https://issues.apache.org/jira/browse/HADOOP-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun resolved HADOOP-17825. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Assignee: L. C. Hsieh Resolution: Fixed > Add BuiltInGzipCompressor > - > > Key: HADOOP-17825 > URL: https://issues.apache.org/jira/browse/HADOOP-17825 > Project: Hadoop Common > Issue Type: Improvement >Reporter: L. C. Hsieh >Assignee: L. C. Hsieh >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 24h 40m > Remaining Estimate: 0h > > Currently, GzipCodec only supports BuiltInGzipDecompressor, if native zlib is > not loaded. So, without Hadoop native codec installed, saving SequenceFile > using GzipCodec will throw exception like "SequenceFile doesn't work with > GzipCodec without native-hadoop code!" > Same as other codecs which we migrated to using prepared packages (lz4, > snappy), it will be better if we support GzipCodec generally without Hadoop > native codec installed. Similar to BuiltInGzipDecompressor, we can use Java > Deflater to support BuiltInGzipCompressor. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17825) Add BuiltInGzipCompressor
[ https://issues.apache.org/jira/browse/HADOOP-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17391797#comment-17391797 ] Chao Sun commented on HADOOP-17825: --- No there isn't. You can just use "Improvement" I think. > Add BuiltInGzipCompressor > - > > Key: HADOOP-17825 > URL: https://issues.apache.org/jira/browse/HADOOP-17825 > Project: Hadoop Common > Issue Type: Improvement >Reporter: L. C. Hsieh >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > Currently, GzipCodec only supports BuiltInGzipDecompressor, if native zlib is > not loaded. So, without Hadoop native codec installed, saving SequenceFile > using GzipCodec will throw exception like "SequenceFile doesn't work with > GzipCodec without native-hadoop code!" > Same as other codecs which we migrated to using prepared packages (lz4, > snappy), it will be better if we support GzipCodec generally without Hadoop > native codec installed. Similar to BuiltInGzipDecompressor, we can use Java > Deflater to support BuiltInGzipCompressor. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17825) Add BuiltInGzipCompressor
[ https://issues.apache.org/jira/browse/HADOOP-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17391036#comment-17391036 ] Chao Sun commented on HADOOP-17825: --- Will take a look later. BTW shall we create an umbrella JIRA covering all the work (e.g., HADOOP-17125, HADOOP-17292, HADOOP-17464) of replacing native lib with their Java wrappers? > Add BuiltInGzipCompressor > - > > Key: HADOOP-17825 > URL: https://issues.apache.org/jira/browse/HADOOP-17825 > Project: Hadoop Common > Issue Type: Improvement >Reporter: L. C. Hsieh >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Currently, GzipCodec only supports BuiltInGzipDecompressor, if native zlib is > not loaded. So, without Hadoop native codec installed, saving SequenceFile > using GzipCodec will throw exception like "SequenceFile doesn't work with > GzipCodec without native-hadoop code!" > Same as other codecs which we migrated to using prepared packages (lz4, > snappy), it will be better if we support GzipCodec generally without Hadoop > native codec installed. Similar to BuiltInGzipDecompressor, we can use Java > Deflater to support BuiltInGzipCompressor. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16822) Provide source artifacts for hadoop-client-api
[ https://issues.apache.org/jira/browse/HADOOP-16822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17343509#comment-17343509 ] Chao Sun commented on HADOOP-16822: --- Any update on this issue? [~karel.kolman] [~weichiu] [~busbey]. It'd be great if we can get this in to 3.3.1 release. > Provide source artifacts for hadoop-client-api > -- > > Key: HADOOP-16822 > URL: https://issues.apache.org/jira/browse/HADOOP-16822 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Karel Kolman >Assignee: Karel Kolman >Priority: Major > Attachments: HADOOP-16822-hadoop-client-api-source-jar.patch > > > h5. Improvement request > The third-party libraries shading hadoop-client-api (& hadoop-client-runtime) > artifacts are super useful. > > Having uber source jar for hadoop-client-api (maybe even > hadoop-client-runtime) would be great for downstream development & debugging > purposes. > Are there any obstacles or objections against providing fat jar with all the > hadoop client api as well ? > h5. Dev links > - *maven-shaded-plugin* and its *shadeSourcesContent* attribute > - > https://maven.apache.org/plugins/maven-shade-plugin/shade-mojo.html#shadeSourcesContent -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17532) Yarn Job execution get failed when LZ4 Compression Codec is used
[
https://issues.apache.org/jira/browse/HADOOP-17532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HADOOP-17532:
--
Component/s: common
> Yarn Job execution get failed when LZ4 Compression Codec is used
>
>
> Key: HADOOP-17532
> URL: https://issues.apache.org/jira/browse/HADOOP-17532
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Reporter: Bhavik Patel
>Priority: Major
> Fix For: 3.3.1, 3.4.0
>
> Attachments: HADOOP-17532.001.patch, HADOOP-17532.002.patch, LZ4.png,
> lz4-test.jpg
>
>
> When we try to compress a file using the LZ4 codec compression type then the
> yarn job gets failed with the error message :
> {code:java}
> net.jpountz.lz4.LZ4Compressorcompres(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)V
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17532) Yarn Job execution get failed when LZ4 Compression Codec is used
[
https://issues.apache.org/jira/browse/HADOOP-17532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HADOOP-17532:
--
Fix Version/s: 3.3.1
> Yarn Job execution get failed when LZ4 Compression Codec is used
>
>
> Key: HADOOP-17532
> URL: https://issues.apache.org/jira/browse/HADOOP-17532
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Bhavik Patel
>Priority: Major
> Fix For: 3.3.1, 3.4.0
>
> Attachments: HADOOP-17532.001.patch, HADOOP-17532.002.patch, LZ4.png,
> lz4-test.jpg
>
>
> When we try to compress a file using the LZ4 codec compression type then the
> yarn job gets failed with the error message :
> {code:java}
> net.jpountz.lz4.LZ4Compressorcompres(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)V
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17532) Yarn Job execution get failed when LZ4 Compression Codec is used
[
https://issues.apache.org/jira/browse/HADOOP-17532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17301374#comment-17301374
]
Chao Sun commented on HADOOP-17532:
---
Done. Sorry for the delay [~bpatel]. I was planning to commit this Friday but
somehow forgot. :(
Seems you'll need to be added to contributor list in order to assign the JIRA
to you. [~tasanuma] could you help with this?
I'll also backport this to branch-3.3 as well.
> Yarn Job execution get failed when LZ4 Compression Codec is used
>
>
> Key: HADOOP-17532
> URL: https://issues.apache.org/jira/browse/HADOOP-17532
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Bhavik Patel
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: HADOOP-17532.001.patch, HADOOP-17532.002.patch, LZ4.png,
> lz4-test.jpg
>
>
> When we try to compress a file using the LZ4 codec compression type then the
> yarn job gets failed with the error message :
> {code:java}
> net.jpountz.lz4.LZ4Compressorcompres(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)V
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17532) Yarn Job execution get failed when LZ4 Compression Codec is used
[
https://issues.apache.org/jira/browse/HADOOP-17532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HADOOP-17532:
--
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Yarn Job execution get failed when LZ4 Compression Codec is used
>
>
> Key: HADOOP-17532
> URL: https://issues.apache.org/jira/browse/HADOOP-17532
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Bhavik Patel
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: HADOOP-17532.001.patch, HADOOP-17532.002.patch, LZ4.png,
> lz4-test.jpg
>
>
> When we try to compress a file using the LZ4 codec compression type then the
> yarn job gets failed with the error message :
> {code:java}
> net.jpountz.lz4.LZ4Compressorcompres(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)V
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17532) Yarn Job execution get failed when LZ4 Compression Codec is used
[
https://issues.apache.org/jira/browse/HADOOP-17532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17300426#comment-17300426
]
Chao Sun commented on HADOOP-17532:
---
Thanks [~bpatel]. +1 on patch v2.
> Yarn Job execution get failed when LZ4 Compression Codec is used
>
>
> Key: HADOOP-17532
> URL: https://issues.apache.org/jira/browse/HADOOP-17532
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Bhavik Patel
>Priority: Major
> Attachments: HADOOP-17532.001.patch, HADOOP-17532.002.patch, LZ4.png,
> lz4-test.jpg
>
>
> When we try to compress a file using the LZ4 codec compression type then the
> yarn job gets failed with the error message :
> {code:java}
> net.jpountz.lz4.LZ4Compressorcompres(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)V
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16080) hadoop-aws does not work with hadoop-client-api
[
https://issues.apache.org/jira/browse/HADOOP-16080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HADOOP-16080:
--
Fix Version/s: 3.4.0
> hadoop-aws does not work with hadoop-client-api
> ---
>
> Key: HADOOP-16080
> URL: https://issues.apache.org/jira/browse/HADOOP-16080
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.2.0, 3.1.1, 3.4.0
>Reporter: Keith Turner
>Assignee: Chao Sun
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.2, 3.3.1, 3.4.0
>
> Time Spent: 8h
> Remaining Estimate: 0h
>
> I attempted to use Accumulo and S3a with the following jars on the classpath.
> * hadoop-client-api-3.1.1.jar
> * hadoop-client-runtime-3.1.1.jar
> * hadoop-aws-3.1.1.jar
> This failed with the following exception.
> {noformat}
> Exception in thread "init" java.lang.NoSuchMethodError:
> org.apache.hadoop.util.SemaphoredDelegatingExecutor.(Lcom/google/common/util/concurrent/ListeningExecutorService;IZ)V
> at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:769)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1169)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1149)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1108)
> at org.apache.hadoop.fs.FileSystem.createNewFile(FileSystem.java:1413)
> at
> org.apache.accumulo.server.fs.VolumeManagerImpl.createNewFile(VolumeManagerImpl.java:184)
> at
> org.apache.accumulo.server.init.Initialize.initDirs(Initialize.java:479)
> at
> org.apache.accumulo.server.init.Initialize.initFileSystem(Initialize.java:487)
> at
> org.apache.accumulo.server.init.Initialize.initialize(Initialize.java:370)
> at org.apache.accumulo.server.init.Initialize.doInit(Initialize.java:348)
> at org.apache.accumulo.server.init.Initialize.execute(Initialize.java:967)
> at org.apache.accumulo.start.Main.lambda$execKeyword$0(Main.java:129)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> The problem is that {{S3AFileSystem.create()}} looks for
> {{SemaphoredDelegatingExecutor(com.google.common.util.concurrent.ListeningExecutorService)}}
> which does not exist in hadoop-client-api-3.1.1.jar. What does exist is
> {{SemaphoredDelegatingExecutor(org.apache.hadoop.shaded.com.google.common.util.concurrent.ListeningExecutorService)}}.
> To work around this issue I created a version of hadoop-aws-3.1.1.jar that
> relocated references to Guava.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17532) Yarn Job execution get failed when LZ4 Compression Codec is used
[
https://issues.apache.org/jira/browse/HADOOP-17532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291888#comment-17291888
]
Chao Sun commented on HADOOP-17532:
---
Sorry for chiming in late here. Yes I agree we should keep the dependency
either provided or optional in case Hadoop users do not need it. Also, is the
issue here caused by the conflicting version from Kafka? and the change on
provided scope is not that related?
> Yarn Job execution get failed when LZ4 Compression Codec is used
>
>
> Key: HADOOP-17532
> URL: https://issues.apache.org/jira/browse/HADOOP-17532
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Bhavik Patel
>Priority: Major
> Attachments: HADOOP-17532.001.patch, LZ4.png, lz4-test.jpg
>
>
> When we try to compress a file using the LZ4 codec compression type then the
> yarn job gets failed with the error message :
> {code:java}
> net.jpountz.lz4.LZ4Compressorcompres(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)V
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16080) hadoop-aws does not work with hadoop-client-api
[
https://issues.apache.org/jira/browse/HADOOP-16080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HADOOP-16080:
--
Target Version/s: 3.1.1, 3.2.2, 3.4.0
Affects Version/s: 3.4.0
> hadoop-aws does not work with hadoop-client-api
> ---
>
> Key: HADOOP-16080
> URL: https://issues.apache.org/jira/browse/HADOOP-16080
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.2.0, 3.1.1, 3.4.0
>Reporter: Keith Turner
>Assignee: Chao Sun
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.2, 3.3.1
>
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> I attempted to use Accumulo and S3a with the following jars on the classpath.
> * hadoop-client-api-3.1.1.jar
> * hadoop-client-runtime-3.1.1.jar
> * hadoop-aws-3.1.1.jar
> This failed with the following exception.
> {noformat}
> Exception in thread "init" java.lang.NoSuchMethodError:
> org.apache.hadoop.util.SemaphoredDelegatingExecutor.(Lcom/google/common/util/concurrent/ListeningExecutorService;IZ)V
> at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:769)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1169)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1149)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1108)
> at org.apache.hadoop.fs.FileSystem.createNewFile(FileSystem.java:1413)
> at
> org.apache.accumulo.server.fs.VolumeManagerImpl.createNewFile(VolumeManagerImpl.java:184)
> at
> org.apache.accumulo.server.init.Initialize.initDirs(Initialize.java:479)
> at
> org.apache.accumulo.server.init.Initialize.initFileSystem(Initialize.java:487)
> at
> org.apache.accumulo.server.init.Initialize.initialize(Initialize.java:370)
> at org.apache.accumulo.server.init.Initialize.doInit(Initialize.java:348)
> at org.apache.accumulo.server.init.Initialize.execute(Initialize.java:967)
> at org.apache.accumulo.start.Main.lambda$execKeyword$0(Main.java:129)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> The problem is that {{S3AFileSystem.create()}} looks for
> {{SemaphoredDelegatingExecutor(com.google.common.util.concurrent.ListeningExecutorService)}}
> which does not exist in hadoop-client-api-3.1.1.jar. What does exist is
> {{SemaphoredDelegatingExecutor(org.apache.hadoop.shaded.com.google.common.util.concurrent.ListeningExecutorService)}}.
> To work around this issue I created a version of hadoop-aws-3.1.1.jar that
> relocated references to Guava.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16822) Provide source artifacts for hadoop-client-api
[ https://issues.apache.org/jira/browse/HADOOP-16822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17256190#comment-17256190 ] Chao Sun commented on HADOOP-16822: --- actually [~karel.kolman] HADOOP-13989 had problems when attempting to add createSourcesJar=true. Will this meet the same issues? > Provide source artifacts for hadoop-client-api > -- > > Key: HADOOP-16822 > URL: https://issues.apache.org/jira/browse/HADOOP-16822 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Karel Kolman >Assignee: Karel Kolman >Priority: Major > Attachments: HADOOP-16822-hadoop-client-api-source-jar.patch > > > h5. Improvement request > The third-party libraries shading hadoop-client-api (& hadoop-client-runtime) > artifacts are super useful. > > Having uber source jar for hadoop-client-api (maybe even > hadoop-client-runtime) would be great for downstream development & debugging > purposes. > Are there any obstacles or objections against providing fat jar with all the > hadoop client api as well ? > h5. Dev links > - *maven-shaded-plugin* and its *shadeSourcesContent* attribute > - > https://maven.apache.org/plugins/maven-shade-plugin/shade-mojo.html#shadeSourcesContent -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
