[jira] [Commented] (HADOOP-13363) Upgrade protobuf from 2.5.0 to something newer
[ https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923856#comment-16923856 ] Sunil Govindan commented on HADOOP-13363: - [https://github.com/apache/hadoop/pull/1408/files] > Upgrade protobuf from 2.5.0 to something newer > -- > > Key: HADOOP-13363 > URL: https://issues.apache.org/jira/browse/HADOOP-13363 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Allen Wittenauer >Assignee: Vinayakumar B >Priority: Major > Labels: security > Attachments: HADOOP-13363.001.patch, HADOOP-13363.002.patch, > HADOOP-13363.003.patch, HADOOP-13363.004.patch, HADOOP-13363.005.patch > > > Standard protobuf 2.5.0 does not work properly on many platforms. (See, for > example, https://gist.github.com/BennettSmith/7111094 ). In order for us to > avoid crazy work arounds in the build environment and the fact that 2.5.0 is > starting to slowly disappear as a standard install-able package for even > Linux/x86, we need to either upgrade or self bundle or something else. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16534) Exclude submarine from hadoop source build
[ https://issues.apache.org/jira/browse/HADOOP-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16917941#comment-16917941 ] Sunil Govindan commented on HADOOP-16534: - +1. [~nandakumar131] cud u pls run jenkins ? > Exclude submarine from hadoop source build > -- > > Key: HADOOP-16534 > URL: https://issues.apache.org/jira/browse/HADOOP-16534 > Project: Hadoop Common > Issue Type: Bug >Reporter: Nanda kumar >Assignee: Nanda kumar >Priority: Major > > When we do source package of hadoop, it should not contain submarine > project/code. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16366) Fix YARNUIV2 failing with "Request is a replay attack"
[
https://issues.apache.org/jira/browse/HADOOP-16366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862266#comment-16862266
]
Sunil Govindan commented on HADOOP-16366:
-
However code which you are removing is not UI2 specific, correct. Hence i think
there will be impact.
we need to recheck again from ui2 perspective here.
> Fix YARNUIV2 failing with "Request is a replay attack"
> --
>
> Key: HADOOP-16366
> URL: https://issues.apache.org/jira/browse/HADOOP-16366
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: security
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: HADOOP-16366-001.patch
>
>
> YARNUIV2 fails with "Request is a replay attack" when below settings
> configured.
> {code:java}
> hadoop.security.authentication = kerberos
> hadoop.http.authentication.type = kerberos
> hadoop.http.filter.initializers =
> org.apache.hadoop.security.AuthenticationFilterInitializer
> yarn.resourcemanager.webapp.delegation-token-auth-filter.enabled = false{code}
> AuthenticationFilter is added twice by the Yarn UI2 Context causing the
> issue.
> {code:java}
> 2019-06-12 11:59:43,900 INFO webapp.RMWebAppUtil
> (RMWebAppUtil.java:addFiltersForUI2Context(483)) - UI2 context filter
> Name:authentication,
> className=org.apache.hadoop.security.authentication.server.AuthenticationFilter
> 2019-06-12 11:59:43,900 INFO webapp.RMWebAppUtil
> (RMWebAppUtil.java:addFiltersForUI2Context(483)) - UI2 context filter
> Name:authentication,
> className=org.apache.hadoop.security.authentication.server.AuthenticationFilter
> {code}
>
> Another issue with {{TimelineReaderServer}} which ignores
> {{ProxyUserAuthenticationFilterInitializer}} when
> {{hadoop.http.filter.initializers}} is configured.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16366) Fix YARNUIV2 failing with "Request is a replay attack"
[
https://issues.apache.org/jira/browse/HADOOP-16366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862151#comment-16862151
]
Sunil Govindan commented on HADOOP-16366:
-
[~Prabhu Joseph] This is a SPENGO filter which is added second time.
If this is causing replay issue, will it be happening for UI1? Could you pls
check.
In case where we are using SPNEGO for some endpoints, we may need this.
> Fix YARNUIV2 failing with "Request is a replay attack"
> --
>
> Key: HADOOP-16366
> URL: https://issues.apache.org/jira/browse/HADOOP-16366
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: security
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: HADOOP-16366-001.patch
>
>
> YARNUIV2 fails with "Request is a replay attack" when below settings
> configured.
> {code:java}
> hadoop.security.authentication = kerberos
> hadoop.http.authentication.type = kerberos
> hadoop.http.filter.initializers =
> org.apache.hadoop.security.AuthenticationFilterInitializer
> yarn.resourcemanager.webapp.delegation-token-auth-filter.enabled = false{code}
> AuthenticationFilter is added twice by the Yarn UI2 Context causing the
> issue.
> {code:java}
> 2019-06-12 11:59:43,900 INFO webapp.RMWebAppUtil
> (RMWebAppUtil.java:addFiltersForUI2Context(483)) - UI2 context filter
> Name:authentication,
> className=org.apache.hadoop.security.authentication.server.AuthenticationFilter
> 2019-06-12 11:59:43,900 INFO webapp.RMWebAppUtil
> (RMWebAppUtil.java:addFiltersForUI2Context(483)) - UI2 context filter
> Name:authentication,
> className=org.apache.hadoop.security.authentication.server.AuthenticationFilter
> {code}
>
> Another issue with {{TimelineReaderServer}} which ignores
> {{ProxyUserAuthenticationFilterInitializer}} when
> {{hadoop.http.filter.initializers}} is configured.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16287) KerberosAuthenticationHandler Trusted Proxy Support for Knox
[ https://issues.apache.org/jira/browse/HADOOP-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836930#comment-16836930 ] Sunil Govindan commented on HADOOP-16287: - Thanks [~Prabhu Joseph] and [~eyang] In Summary, we ensure that in a secure cluster (where authentication is kerberos), we will be adding this new filter (by using from an initialiser for this filter which will be done in another Jira ) at the end of the filter chain. With this new filter, incoming authenticated user will be also checked for 2 things. # Whether incoming authenticated user (say for eg: knox) is a valid proxy user or not (from proxy user configurations) # Whether any doAs param is present in the same requests. In such case, doAs user will be considered as the requested user for Hadoop. Since this is very much configuration driven and not coming in default path, it looks like a cleaner solution. Appreciate your thoughts [~lmccay] [~vinodkv] [~daryn] [~leftnoteasy] > KerberosAuthenticationHandler Trusted Proxy Support for Knox > > > Key: HADOOP-16287 > URL: https://issues.apache.org/jira/browse/HADOOP-16287 > Project: Hadoop Common > Issue Type: New Feature > Components: auth >Affects Versions: 3.2.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: HADOOP-16287-001.patch, HADOOP-16287-002.patch, > HADOOP-16287-004.patch, HADOOP-16287-005.patch, HADOOP-16827-003.patch > > > Knox passes doAs with end user while accessing RM, WebHdfs Rest Api. > Currently KerberosAuthenticationHandler sets the remote user to Knox. Need > Trusted Proxy Support by reading doAs query parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15205) maven release: missing source attachments for hadoop-mapreduce-client-core
[ https://issues.apache.org/jira/browse/HADOOP-15205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750884#comment-16750884 ] Sunil Govindan commented on HADOOP-15205: - Yes. release 3.2.0 is updated with this. [~kgyrtkirk] could we close this given documentation is updated > maven release: missing source attachments for hadoop-mapreduce-client-core > -- > > Key: HADOOP-15205 > URL: https://issues.apache.org/jira/browse/HADOOP-15205 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.9.0, 2.8.2, 2.8.3, 2.7.5, 3.0.0, 3.1.0, 3.0.1, 2.8.4, > 2.9.2, 2.8.5 >Reporter: Zoltan Haindrich >Priority: Major > Attachments: chk.bash > > > I wanted to use the source attachment; however it looks like since 2.7.5 that > artifact is not present at maven central ; it looks like the last release > which had source attachments / javadocs was 2.7.4 > http://central.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.5/ > this seems to be not limited to mapreduce; as the same change is present for > yarn-common as well > http://central.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.7.5/ > and also hadoop-common > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/2.7.4/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/2.7.5/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/3.0.0/ > http://central.maven.org/maven2/org/apache/hadoop/hadoop-common/3.1.0/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15617) Reduce node.js and npm package loading in the Dockerfile
[
https://issues.apache.org/jira/browse/HADOOP-15617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744095#comment-16744095
]
Sunil Govindan commented on HADOOP-15617:
-
In trunk code, i can see this part is different.
{code:java}
RUN apt-get -q update \
&& apt-get install -y --no-install-recommends nodejs npm \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* \
&& ln -s /usr/bin/nodejs /usr/bin/node \
&& npm install npm@latest -g \
&& npm install -g jshint{code}
> Reduce node.js and npm package loading in the Dockerfile
>
>
> Key: HADOOP-15617
> URL: https://issues.apache.org/jira/browse/HADOOP-15617
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 2.9.3
>Reporter: Allen Wittenauer
>Priority: Major
> Attachments: HADOOP-15617-branch-2-001.patch
>
>
> {code}
> RUN apt-get -y install nodejs && \
> ln -s /usr/bin/nodejs /usr/bin/node && \
> apt-get -y install npm && \
> npm install npm@latest -g && \
> npm install -g bower && \
> npm install -g ember-cli
> {code}
> should get reduced to
> {code}
> RUN apt-get -y install nodejs && \
> ln -s /usr/bin/nodejs /usr/bin/node && \
> apt-get -y install npm && \
> npm install npm@latest -g
> {code}
> The locally installed versions of bower and ember-cli aren't being used
> anymore. Removing these cuts the docker build time significantly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15617) Reduce node.js and npm package loading in the Dockerfile
[
https://issues.apache.org/jira/browse/HADOOP-15617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744082#comment-16744082
]
Sunil Govindan commented on HADOOP-15617:
-
Hi [~ste...@apache.org]
Still it is failing, I ll take a look now. cc [~akhilpb]
{code:java}
npm ERR! Error: CERT_UNTRUSTED
npm ERR! at SecurePair. (tls.js:1370:32)
npm ERR! at SecurePair.EventEmitter.emit (events.js:92:17)
npm ERR! at SecurePair.maybeInitFinished (tls.js:982:10)
npm ERR! at CleartextStream.read [as _read] (tls.js:469:13)
npm ERR! at CleartextStream.Readable.read (_stream_readable.js:320:10)
npm ERR! at EncryptedStream.write [as _write] (tls.js:366:25)
npm ERR! at doWrite (_stream_writable.js:223:10)
npm ERR! at writeOrBuffer (_stream_writable.js:213:5)
npm ERR! at EncryptedStream.Writable.write (_stream_writable.js:180:11)
npm ERR! at write (_stream_readable.js:583:24)
npm ERR! If you need help, you may report this log at:
npm ERR!
npm ERR! or email it to:
npm ERR!
npm ERR! System Linux 4.4.0-138-generic
npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "install" "npm@latest" "-g"
npm ERR! cwd /root
npm ERR! node -v v0.10.25
npm ERR! npm -v 1.3.10
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR! /root/npm-debug.log
npm ERR! not ok code 0{code}
> Reduce node.js and npm package loading in the Dockerfile
>
>
> Key: HADOOP-15617
> URL: https://issues.apache.org/jira/browse/HADOOP-15617
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 2.9.3
>Reporter: Allen Wittenauer
>Priority: Major
> Attachments: HADOOP-15617-branch-2-001.patch
>
>
> {code}
> RUN apt-get -y install nodejs && \
> ln -s /usr/bin/nodejs /usr/bin/node && \
> apt-get -y install npm && \
> npm install npm@latest -g && \
> npm install -g bower && \
> npm install -g ember-cli
> {code}
> should get reduced to
> {code}
> RUN apt-get -y install nodejs && \
> ln -s /usr/bin/nodejs /usr/bin/node && \
> apt-get -y install npm && \
> npm install npm@latest -g
> {code}
> The locally installed versions of bower and ember-cli aren't being used
> anymore. Removing these cuts the docker build time significantly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16031) TestSecureLogins#testValidKerberosName fails
[
https://issues.apache.org/jira/browse/HADOOP-16031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil Govindan updated HADOOP-16031:
Fix Version/s: (was: 3.2.1)
3.2.0
> TestSecureLogins#testValidKerberosName fails
>
>
> Key: HADOOP-16031
> URL: https://issues.apache.org/jira/browse/HADOOP-16031
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Fix For: 3.2.0, 3.3.0, 3.1.3
>
> Attachments: HADOOP-16031.01.patch
>
>
> {noformat}
> [INFO] Running org.apache.hadoop.registry.secure.TestSecureLogins
> [ERROR] Tests run: 11, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 2.724 s <<< FAILURE! - in org.apache.hadoop.registry.secure.TestSecureLogins
> [ERROR]
> testValidKerberosName(org.apache.hadoop.registry.secure.TestSecureLogins)
> Time elapsed: 0.01 s <<< ERROR!
> org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule:
> No rules applied to zookeeper/localhost
> at
> org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:429)
> at
> org.apache.hadoop.registry.secure.TestSecureLogins.testValidKerberosName(TestSecureLogins.java:203)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15996) Plugin interface to support more complex usernames in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-15996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15996: Fix Version/s: (was: 3.2.1) 3.2.0 > Plugin interface to support more complex usernames in Hadoop > > > Key: HADOOP-15996 > URL: https://issues.apache.org/jira/browse/HADOOP-15996 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Reporter: Eric Yang >Assignee: Bolke de Bruin >Priority: Major > Fix For: 3.2.0, 3.3.0, 3.1.2 > > Attachments: 0001-HADOOP-15996-Make-auth-to-local-configurable.patch, > 0001-Make-allowing-or-configurable.patch, > 0001-Simple-trial-of-using-krb5.conf-for-auth_to_local-ru.patch, > 0002-HADOOP-15996-Make-auth-to-local-configurable.patch, > 0003-HADOOP-15996-Make-auth-to-local-configurable.patch, > 0004-HADOOP-15996-Make-auth-to-local-configurable.patch, > 0005-HADOOP-15996-Make-auth-to-local-configurable.patch, > HADOOP-15996.0005.patch, HADOOP-15996.0006.patch, HADOOP-15996.0007.patch, > HADOOP-15996.0008.patch, HADOOP-15996.0009.patch, HADOOP-15996.0010.patch, > HADOOP-15996.0011.patch, HADOOP-15996.0012.patch > > > Hadoop does not allow support of @ character in username in recent security > mailing list vote to revert HADOOP-12751. Hadoop auth_to_local rule must > match to authorize user to login to Hadoop cluster. This design does not > work well in multi-realm environment where identical username between two > realms do not map to the same user. There is also possibility that lossy > regex can incorrectly map users. In the interest of supporting multi-realms, > it maybe preferred to pass principal name without rewrite to uniquely > distinguish users. This jira is to revisit if Hadoop can support full > principal names without rewrite and provide a plugin to override Hadoop's > default implementation of auth_to_local for multi-realm use case. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15996) Plugin interface to support more complex usernames in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-15996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15996: Fix Version/s: 3.2.1 > Plugin interface to support more complex usernames in Hadoop > > > Key: HADOOP-15996 > URL: https://issues.apache.org/jira/browse/HADOOP-15996 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Reporter: Eric Yang >Assignee: Bolke de Bruin >Priority: Major > Fix For: 3.3.0, 3.1.2, 3.2.1 > > Attachments: 0001-HADOOP-15996-Make-auth-to-local-configurable.patch, > 0001-Make-allowing-or-configurable.patch, > 0001-Simple-trial-of-using-krb5.conf-for-auth_to_local-ru.patch, > 0002-HADOOP-15996-Make-auth-to-local-configurable.patch, > 0003-HADOOP-15996-Make-auth-to-local-configurable.patch, > 0004-HADOOP-15996-Make-auth-to-local-configurable.patch, > 0005-HADOOP-15996-Make-auth-to-local-configurable.patch, > HADOOP-15996.0005.patch, HADOOP-15996.0006.patch, HADOOP-15996.0007.patch, > HADOOP-15996.0008.patch, HADOOP-15996.0009.patch, HADOOP-15996.0010.patch, > HADOOP-15996.0011.patch, HADOOP-15996.0012.patch > > > Hadoop does not allow support of @ character in username in recent security > mailing list vote to revert HADOOP-12751. Hadoop auth_to_local rule must > match to authorize user to login to Hadoop cluster. This design does not > work well in multi-realm environment where identical username between two > realms do not map to the same user. There is also possibility that lossy > regex can incorrectly map users. In the interest of supporting multi-realms, > it maybe preferred to pass principal name without rewrite to uniquely > distinguish users. This jira is to revisit if Hadoop can support full > principal names without rewrite and provide a plugin to override Hadoop's > default implementation of auth_to_local for multi-realm use case. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15996) Plugin interface to support more complex usernames in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-15996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15996: Fix Version/s: (was: 3.2.0) > Plugin interface to support more complex usernames in Hadoop > > > Key: HADOOP-15996 > URL: https://issues.apache.org/jira/browse/HADOOP-15996 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Reporter: Eric Yang >Assignee: Bolke de Bruin >Priority: Major > Fix For: 3.3.0, 3.1.2 > > Attachments: 0001-HADOOP-15996-Make-auth-to-local-configurable.patch, > 0001-Make-allowing-or-configurable.patch, > 0001-Simple-trial-of-using-krb5.conf-for-auth_to_local-ru.patch, > 0002-HADOOP-15996-Make-auth-to-local-configurable.patch, > 0003-HADOOP-15996-Make-auth-to-local-configurable.patch, > 0004-HADOOP-15996-Make-auth-to-local-configurable.patch, > 0005-HADOOP-15996-Make-auth-to-local-configurable.patch, > HADOOP-15996.0005.patch, HADOOP-15996.0006.patch, HADOOP-15996.0007.patch, > HADOOP-15996.0008.patch, HADOOP-15996.0009.patch, HADOOP-15996.0010.patch, > HADOOP-15996.0011.patch, HADOOP-15996.0012.patch > > > Hadoop does not allow support of @ character in username in recent security > mailing list vote to revert HADOOP-12751. Hadoop auth_to_local rule must > match to authorize user to login to Hadoop cluster. This design does not > work well in multi-realm environment where identical username between two > realms do not map to the same user. There is also possibility that lossy > regex can incorrectly map users. In the interest of supporting multi-realms, > it maybe preferred to pass principal name without rewrite to uniquely > distinguish users. This jira is to revisit if Hadoop can support full > principal names without rewrite and provide a plugin to override Hadoop's > default implementation of auth_to_local for multi-realm use case. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16025) Update the year to 2019
[ https://issues.apache.org/jira/browse/HADOOP-16025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-16025: Fix Version/s: (was: 3.2.1) 3.2.0 > Update the year to 2019 > --- > > Key: HADOOP-16025 > URL: https://issues.apache.org/jira/browse/HADOOP-16025 > Project: Hadoop Common > Issue Type: Task > Components: build >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Fix For: 3.2.0, 2.7.8, 3.0.4, 2.8.6, 2.9.3, 3.1.3 > > Attachments: HADOOP-16025-01.patch > > > Update the year to 2019 from 2018. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15997) KMS client uses wrong UGI after HADOOP-14445
[
https://issues.apache.org/jira/browse/HADOOP-15997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734301#comment-16734301
]
Sunil Govindan commented on HADOOP-15997:
-
Thanks. Committing this shortly
> KMS client uses wrong UGI after HADOOP-14445
>
>
> Key: HADOOP-15997
> URL: https://issues.apache.org/jira/browse/HADOOP-15997
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 3.2.0, 3.0.4, 3.1.2
> Environment: Hadoop 3.0.x (CDH6.x), Kerberized, HDFS at-rest
> encryption, multiple KMS
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Blocker
> Attachments: HADOOP-15997.001.patch, HADOOP-15997.02.patch
>
>
> After HADOOP-14445, KMS client always authenticates itself using the
> credentials from login user, rather than current user.
> {noformat}
> 2018-12-07 15:58:30,663 DEBUG [main]
> org.apache.hadoop.crypto.key.kms.KMSClientProvider: Using loginUser when
> Kerberos is enabled but the actual user does not have either KMS Delegation
> Token or Kerberos Credentials
> {noformat}
> The log message {{"Using loginUser when Kerberos is enabled but the actual
> user does not have either KMS Delegation Token or Kerberos Credentials"}} is
> printed because {{KMSClientProvider#containsKmsDt()}} is null when it
> definitely has the kms delegation token.
> In fact, {{KMSClientProvider#containsKmsDt()}} should select delegation token
> using {{clientTokenProvider.selectDelegationToken(creds)}} rather than
> checking if its dtService is in the user credentials.
> This is done correctly in {{KMSClientProvider#createAuthenticatedURL}} though.
> We found this bug when it broke Cloudera's Backup and Disaster Recovery tool.
>
> [~daryn] [~xiaochen] mind taking a look? HADOOP-14445 is a huge patch but it
> is almost perfect except for this bug.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15992) JSON License is included in the transitive dependency of aliyun-sdk-oss 3.0.0
[
https://issues.apache.org/jira/browse/HADOOP-15992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734300#comment-16734300
]
Sunil Govindan commented on HADOOP-15992:
-
Hi [~ajisakaa]
Attaching a new revert from top of trunk after fixing the conflict.
Could you please help to review and commit. Thanks
> JSON License is included in the transitive dependency of aliyun-sdk-oss 3.0.0
> -
>
> Key: HADOOP-15992
> URL: https://issues.apache.org/jira/browse/HADOOP-15992
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.9.2
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Blocker
> Attachments: HADOOP-15992.01.patch, HADOOP-15992.02.patch
>
>
> This is the output of {{mvn dependency:tree}}
> {noformat}
> [INFO] +- com.aliyun.oss:aliyun-sdk-oss:jar:3.0.0:compile
> [INFO] | +- org.jdom:jdom:jar:1.1:compile
> [INFO] | +- com.sun.jersey:jersey-json:jar:1.19:compile
> [INFO] | | +- org.codehaus.jettison:jettison:jar:1.1:compile
> [INFO] | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
> [INFO] | | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
> [INFO] | | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
> [INFO] | | +- org.codehaus.jackson:jackson-jaxrs:jar:1.9.13:compile
> [INFO] | | \- org.codehaus.jackson:jackson-xc:jar:1.9.13:compile
> [INFO] | +- com.aliyun:aliyun-java-sdk-core:jar:3.4.0:compile
> [INFO] | | \- org.json:json:jar:20170516:compile
> {noformat}
> The license of org.json:json:jar:20170516:compile is JSON License, which
> cannot be included.
> https://www.apache.org/legal/resolved.html#json
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15997) KMS client uses wrong UGI after HADOOP-14445
[
https://issues.apache.org/jira/browse/HADOOP-15997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil Govindan updated HADOOP-15997:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.1.2
3.3.0
3.2.0
Status: Resolved (was: Patch Available)
Thanks [~jojochuang] Committed to trunk/branch-3.2/branch-3.1
Cherrypick is failing for branch-3.0. Could you please help. Thanks
> KMS client uses wrong UGI after HADOOP-14445
>
>
> Key: HADOOP-15997
> URL: https://issues.apache.org/jira/browse/HADOOP-15997
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 3.2.0, 3.0.4, 3.1.2
> Environment: Hadoop 3.0.x (CDH6.x), Kerberized, HDFS at-rest
> encryption, multiple KMS
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Blocker
> Fix For: 3.2.0, 3.3.0, 3.1.2
>
> Attachments: HADOOP-15997.001.patch, HADOOP-15997.02.patch
>
>
> After HADOOP-14445, KMS client always authenticates itself using the
> credentials from login user, rather than current user.
> {noformat}
> 2018-12-07 15:58:30,663 DEBUG [main]
> org.apache.hadoop.crypto.key.kms.KMSClientProvider: Using loginUser when
> Kerberos is enabled but the actual user does not have either KMS Delegation
> Token or Kerberos Credentials
> {noformat}
> The log message {{"Using loginUser when Kerberos is enabled but the actual
> user does not have either KMS Delegation Token or Kerberos Credentials"}} is
> printed because {{KMSClientProvider#containsKmsDt()}} is null when it
> definitely has the kms delegation token.
> In fact, {{KMSClientProvider#containsKmsDt()}} should select delegation token
> using {{clientTokenProvider.selectDelegationToken(creds)}} rather than
> checking if its dtService is in the user credentials.
> This is done correctly in {{KMSClientProvider#createAuthenticatedURL}} though.
> We found this bug when it broke Cloudera's Backup and Disaster Recovery tool.
>
> [~daryn] [~xiaochen] mind taking a look? HADOOP-14445 is a huge patch but it
> is almost perfect except for this bug.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15992) JSON License is included in the transitive dependency of aliyun-sdk-oss 3.0.0
[
https://issues.apache.org/jira/browse/HADOOP-15992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil Govindan updated HADOOP-15992:
Attachment: HADOOP-15992.02.patch
> JSON License is included in the transitive dependency of aliyun-sdk-oss 3.0.0
> -
>
> Key: HADOOP-15992
> URL: https://issues.apache.org/jira/browse/HADOOP-15992
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.9.2
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Blocker
> Attachments: HADOOP-15992.01.patch, HADOOP-15992.02.patch
>
>
> This is the output of {{mvn dependency:tree}}
> {noformat}
> [INFO] +- com.aliyun.oss:aliyun-sdk-oss:jar:3.0.0:compile
> [INFO] | +- org.jdom:jdom:jar:1.1:compile
> [INFO] | +- com.sun.jersey:jersey-json:jar:1.19:compile
> [INFO] | | +- org.codehaus.jettison:jettison:jar:1.1:compile
> [INFO] | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
> [INFO] | | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
> [INFO] | | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
> [INFO] | | +- org.codehaus.jackson:jackson-jaxrs:jar:1.9.13:compile
> [INFO] | | \- org.codehaus.jackson:jackson-xc:jar:1.9.13:compile
> [INFO] | +- com.aliyun:aliyun-java-sdk-core:jar:3.4.0:compile
> [INFO] | | \- org.json:json:jar:20170516:compile
> {noformat}
> The license of org.json:json:jar:20170516:compile is JSON License, which
> cannot be included.
> https://www.apache.org/legal/resolved.html#json
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15992) JSON License is included in the transitive dependency of aliyun-sdk-oss 3.0.0
[
https://issues.apache.org/jira/browse/HADOOP-15992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733857#comment-16733857
]
Sunil Govindan commented on HADOOP-15992:
-
Hi [~ajisakaa]
Latest patch is not applying on to trunk. Could u pls help. Thanks
> JSON License is included in the transitive dependency of aliyun-sdk-oss 3.0.0
> -
>
> Key: HADOOP-15992
> URL: https://issues.apache.org/jira/browse/HADOOP-15992
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.9.2
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Blocker
> Attachments: HADOOP-15992.01.patch
>
>
> This is the output of {{mvn dependency:tree}}
> {noformat}
> [INFO] +- com.aliyun.oss:aliyun-sdk-oss:jar:3.0.0:compile
> [INFO] | +- org.jdom:jdom:jar:1.1:compile
> [INFO] | +- com.sun.jersey:jersey-json:jar:1.19:compile
> [INFO] | | +- org.codehaus.jettison:jettison:jar:1.1:compile
> [INFO] | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
> [INFO] | | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
> [INFO] | | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
> [INFO] | | +- org.codehaus.jackson:jackson-jaxrs:jar:1.9.13:compile
> [INFO] | | \- org.codehaus.jackson:jackson-xc:jar:1.9.13:compile
> [INFO] | +- com.aliyun:aliyun-java-sdk-core:jar:3.4.0:compile
> [INFO] | | \- org.json:json:jar:20170516:compile
> {noformat}
> The license of org.json:json:jar:20170516:compile is JSON License, which
> cannot be included.
> https://www.apache.org/legal/resolved.html#json
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15992) JSON License is included in the transitive dependency of aliyun-sdk-oss 3.0.0
[
https://issues.apache.org/jira/browse/HADOOP-15992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732781#comment-16732781
]
Sunil Govindan commented on HADOOP-15992:
-
+1. Committing shortly to trunk/branch-3.2 and branch-3.2.0
[~ajisakaa] do we need to put this to any other branches?
> JSON License is included in the transitive dependency of aliyun-sdk-oss 3.0.0
> -
>
> Key: HADOOP-15992
> URL: https://issues.apache.org/jira/browse/HADOOP-15992
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.9.2
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Blocker
> Attachments: HADOOP-15992.01.patch
>
>
> This is the output of {{mvn dependency:tree}}
> {noformat}
> [INFO] +- com.aliyun.oss:aliyun-sdk-oss:jar:3.0.0:compile
> [INFO] | +- org.jdom:jdom:jar:1.1:compile
> [INFO] | +- com.sun.jersey:jersey-json:jar:1.19:compile
> [INFO] | | +- org.codehaus.jettison:jettison:jar:1.1:compile
> [INFO] | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
> [INFO] | | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
> [INFO] | | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
> [INFO] | | +- org.codehaus.jackson:jackson-jaxrs:jar:1.9.13:compile
> [INFO] | | \- org.codehaus.jackson:jackson-xc:jar:1.9.13:compile
> [INFO] | +- com.aliyun:aliyun-java-sdk-core:jar:3.4.0:compile
> [INFO] | | \- org.json:json:jar:20170516:compile
> {noformat}
> The license of org.json:json:jar:20170516:compile is JSON License, which
> cannot be included.
> https://www.apache.org/legal/resolved.html#json
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15997) KMS client uses wrong UGI after HADOOP-14445
[
https://issues.apache.org/jira/browse/HADOOP-15997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16731878#comment-16731878
]
Sunil Govindan commented on HADOOP-15997:
-
Jenkins seems fine with the patch apart from test issue. [~jojochuang] cud u
pls check test failure. if no objections, i ll help to commit this patch by
tomorrow.
cc [~templedf] [~daryn] [~xiaochen]
> KMS client uses wrong UGI after HADOOP-14445
>
>
> Key: HADOOP-15997
> URL: https://issues.apache.org/jira/browse/HADOOP-15997
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 3.2.0, 3.0.4, 3.1.2
> Environment: Hadoop 3.0.x (CDH6.x), Kerberized, HDFS at-rest
> encryption, multiple KMS
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Blocker
> Attachments: HADOOP-15997.001.patch, HADOOP-15997.02.patch
>
>
> After HADOOP-14445, KMS client always authenticates itself using the
> credentials from login user, rather than current user.
> {noformat}
> 2018-12-07 15:58:30,663 DEBUG [main]
> org.apache.hadoop.crypto.key.kms.KMSClientProvider: Using loginUser when
> Kerberos is enabled but the actual user does not have either KMS Delegation
> Token or Kerberos Credentials
> {noformat}
> The log message {{"Using loginUser when Kerberos is enabled but the actual
> user does not have either KMS Delegation Token or Kerberos Credentials"}} is
> printed because {{KMSClientProvider#containsKmsDt()}} is null when it
> definitely has the kms delegation token.
> In fact, {{KMSClientProvider#containsKmsDt()}} should select delegation token
> using {{clientTokenProvider.selectDelegationToken(creds)}} rather than
> checking if its dtService is in the user credentials.
> This is done correctly in {{KMSClientProvider#createAuthenticatedURL}} though.
> We found this bug when it broke Cloudera's Backup and Disaster Recovery tool.
>
> [~daryn] [~xiaochen] mind taking a look? HADOOP-14445 is a huge patch but it
> is almost perfect except for this bug.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15992) JSON License is included in the transitive dependency of aliyun-sdk-oss 3.0.0
[
https://issues.apache.org/jira/browse/HADOOP-15992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16728915#comment-16728915
]
Sunil Govindan commented on HADOOP-15992:
-
Hi [~ajisakaa],
Thank you. will u have some cycles to update same? I can help to commit.
> JSON License is included in the transitive dependency of aliyun-sdk-oss 3.0.0
> -
>
> Key: HADOOP-15992
> URL: https://issues.apache.org/jira/browse/HADOOP-15992
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.9.2
>Reporter: Akira Ajisaka
>Priority: Blocker
>
> This is the output of {{mvn dependency:tree}}
> {noformat}
> [INFO] +- com.aliyun.oss:aliyun-sdk-oss:jar:3.0.0:compile
> [INFO] | +- org.jdom:jdom:jar:1.1:compile
> [INFO] | +- com.sun.jersey:jersey-json:jar:1.19:compile
> [INFO] | | +- org.codehaus.jettison:jettison:jar:1.1:compile
> [INFO] | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
> [INFO] | | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
> [INFO] | | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
> [INFO] | | +- org.codehaus.jackson:jackson-jaxrs:jar:1.9.13:compile
> [INFO] | | \- org.codehaus.jackson:jackson-xc:jar:1.9.13:compile
> [INFO] | +- com.aliyun:aliyun-java-sdk-core:jar:3.4.0:compile
> [INFO] | | \- org.json:json:jar:20170516:compile
> {noformat}
> The license of org.json:json:jar:20170516:compile is JSON License, which
> cannot be included.
> https://www.apache.org/legal/resolved.html#json
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15992) JSON License is included in the transitive dependency of aliyun-sdk-oss 3.0.0
[
https://issues.apache.org/jira/browse/HADOOP-15992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16728849#comment-16728849
]
Sunil Govindan commented on HADOOP-15992:
-
[~ajisakaa] [~Sammi] [~wujinhu] could we revert HADOOP-15759 to unblock 3.2.0
release. Thanks.
> JSON License is included in the transitive dependency of aliyun-sdk-oss 3.0.0
> -
>
> Key: HADOOP-15992
> URL: https://issues.apache.org/jira/browse/HADOOP-15992
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.9.2
>Reporter: Akira Ajisaka
>Priority: Blocker
>
> This is the output of {{mvn dependency:tree}}
> {noformat}
> [INFO] +- com.aliyun.oss:aliyun-sdk-oss:jar:3.0.0:compile
> [INFO] | +- org.jdom:jdom:jar:1.1:compile
> [INFO] | +- com.sun.jersey:jersey-json:jar:1.19:compile
> [INFO] | | +- org.codehaus.jettison:jettison:jar:1.1:compile
> [INFO] | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
> [INFO] | | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
> [INFO] | | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
> [INFO] | | +- org.codehaus.jackson:jackson-jaxrs:jar:1.9.13:compile
> [INFO] | | \- org.codehaus.jackson:jackson-xc:jar:1.9.13:compile
> [INFO] | +- com.aliyun:aliyun-java-sdk-core:jar:3.4.0:compile
> [INFO] | | \- org.json:json:jar:20170516:compile
> {noformat}
> The license of org.json:json:jar:20170516:compile is JSON License, which
> cannot be included.
> https://www.apache.org/legal/resolved.html#json
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16007) Order of property settings is incorrect when includes are processed
[
https://issues.apache.org/jira/browse/HADOOP-16007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723935#comment-16723935
]
Sunil Govindan commented on HADOOP-16007:
-
Hi [~jlowe] and [~eepayne]
Thanks for reporting same. I think this is the same issue reported during 3.2.0
RC0 time. Correct?
Does this need some revert or partial revert of HADOOP-15554 as I could see a
good set of changes in that work?
I think I need to hold on to RC once this is resolved. Pls correct me
otherwise. Thanks.
> Order of property settings is incorrect when includes are processed
> ---
>
> Key: HADOOP-16007
> URL: https://issues.apache.org/jira/browse/HADOOP-16007
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
>Affects Versions: 3.2.0, 3.1.1, 3.0.4
>Reporter: Jason Lowe
>Assignee: Eric Payne
>Priority: Blocker
>
> If a configuration file contains a setting for a property then later includes
> another file that also sets that property to a different value then the
> property will be parsed incorrectly. For example, consider the following
> configuration file:
> {noformat}
> http://www.w3.org/2001/XInclude;>
>
> myprop
> val1
>
>
>
> {noformat}
> with the contents of /some/other/file.xml as:
> {noformat}
>
>myprop
>val2
>
> {noformat}
> Parsing this configuration should result in myprop=val2, but it actually
> results in myprop=val1.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15992) JSON License is included in the transitive dependency of aliyun-sdk-oss 3.0.0
[
https://issues.apache.org/jira/browse/HADOOP-15992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723930#comment-16723930
]
Sunil Govindan commented on HADOOP-15992:
-
[~ajisakaa] is this a blocker for 3.2.0. I am planning to cut RC1 tomorrow. Is
this possible to be fixed sooner?
If not a blocker, could this be moved.
> JSON License is included in the transitive dependency of aliyun-sdk-oss 3.0.0
> -
>
> Key: HADOOP-15992
> URL: https://issues.apache.org/jira/browse/HADOOP-15992
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.9.2
>Reporter: Akira Ajisaka
>Priority: Blocker
>
> This is the output of {{mvn dependency:tree}}
> {noformat}
> [INFO] +- com.aliyun.oss:aliyun-sdk-oss:jar:3.0.0:compile
> [INFO] | +- org.jdom:jdom:jar:1.1:compile
> [INFO] | +- com.sun.jersey:jersey-json:jar:1.19:compile
> [INFO] | | +- org.codehaus.jettison:jettison:jar:1.1:compile
> [INFO] | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
> [INFO] | | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
> [INFO] | | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
> [INFO] | | +- org.codehaus.jackson:jackson-jaxrs:jar:1.9.13:compile
> [INFO] | | \- org.codehaus.jackson:jackson-xc:jar:1.9.13:compile
> [INFO] | +- com.aliyun:aliyun-java-sdk-core:jar:3.4.0:compile
> [INFO] | | \- org.json:json:jar:20170516:compile
> {noformat}
> The license of org.json:json:jar:20170516:compile is JSON License, which
> cannot be included.
> https://www.apache.org/legal/resolved.html#json
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15997) KMS client uses wrong UGI after HADOOP-14445
[
https://issues.apache.org/jira/browse/HADOOP-15997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723927#comment-16723927
]
Sunil Govindan commented on HADOOP-15997:
-
ping [~templedf] [~jojochuang] [~xiaochen] [~daryn]
> KMS client uses wrong UGI after HADOOP-14445
>
>
> Key: HADOOP-15997
> URL: https://issues.apache.org/jira/browse/HADOOP-15997
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 3.2.0, 3.0.4, 3.1.2
> Environment: Hadoop 3.0.x (CDH6.x), Kerberized, HDFS at-rest
> encryption, multiple KMS
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Blocker
> Attachments: HADOOP-15997.001.patch
>
>
> After HADOOP-14445, KMS client always authenticates itself using the
> credentials from login user, rather than current user.
> {noformat}
> 2018-12-07 15:58:30,663 DEBUG [main]
> org.apache.hadoop.crypto.key.kms.KMSClientProvider: Using loginUser when
> Kerberos is enabled but the actual user does not have either KMS Delegation
> Token or Kerberos Credentials
> {noformat}
> The log message {{"Using loginUser when Kerberos is enabled but the actual
> user does not have either KMS Delegation Token or Kerberos Credentials"}} is
> printed because {{KMSClientProvider#containsKmsDt()}} is null when it
> definitely has the kms delegation token.
> In fact, {{KMSClientProvider#containsKmsDt()}} should select delegation token
> using {{clientTokenProvider.selectDelegationToken(creds)}} rather than
> checking if its dtService is in the user credentials.
> This is done correctly in {{KMSClientProvider#createAuthenticatedURL}} though.
> We found this bug when it broke Cloudera's Backup and Disaster Recovery tool.
>
> [~daryn] [~xiaochen] mind taking a look? HADOOP-14445 is a huge patch but it
> is almost perfect except for this bug.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15997) KMS client uses wrong UGI after HADOOP-14445
[
https://issues.apache.org/jira/browse/HADOOP-15997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721899#comment-16721899
]
Sunil Govindan commented on HADOOP-15997:
-
[~templedf] [~xiaochen] [~daryn] pls help to review.
I marked this as a blocker for 3.2.0 and 3.1.2
> KMS client uses wrong UGI after HADOOP-14445
>
>
> Key: HADOOP-15997
> URL: https://issues.apache.org/jira/browse/HADOOP-15997
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 3.2.0, 3.0.4, 3.1.2
> Environment: Hadoop 3.0.x (CDH6.x), Kerberized, HDFS at-rest
> encryption, multiple KMS
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Blocker
> Attachments: HADOOP-15997.001.patch
>
>
> After HADOOP-14445, KMS client always authenticates itself using the
> credentials from login user, rather than current user.
> {noformat}
> 2018-12-07 15:58:30,663 DEBUG [main]
> org.apache.hadoop.crypto.key.kms.KMSClientProvider: Using loginUser when
> Kerberos is enabled but the actual user does not have either KMS Delegation
> Token or Kerberos Credentials
> {noformat}
> The log message {{"Using loginUser when Kerberos is enabled but the actual
> user does not have either KMS Delegation Token or Kerberos Credentials"}} is
> printed because {{KMSClientProvider#containsKmsDt()}} is null when it
> definitely has the kms delegation token.
> In fact, {{KMSClientProvider#containsKmsDt()}} should select delegation token
> using {{clientTokenProvider.selectDelegationToken(creds)}} rather than
> checking if its dtService is in the user credentials.
> This is done correctly in {{KMSClientProvider#createAuthenticatedURL}} though.
> We found this bug when it broke Cloudera's Backup and Disaster Recovery tool.
>
> [~daryn] [~xiaochen] mind taking a look? HADOOP-14445 is a huge patch but it
> is almost perfect except for this bug.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15997) KMS client uses wrong UGI after HADOOP-14445
[
https://issues.apache.org/jira/browse/HADOOP-15997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil Govindan updated HADOOP-15997:
Priority: Blocker (was: Major)
> KMS client uses wrong UGI after HADOOP-14445
>
>
> Key: HADOOP-15997
> URL: https://issues.apache.org/jira/browse/HADOOP-15997
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 3.2.0, 3.0.4, 3.1.2
> Environment: Hadoop 3.0.x (CDH6.x), Kerberized, HDFS at-rest
> encryption, multiple KMS
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Blocker
> Attachments: HADOOP-15997.001.patch
>
>
> After HADOOP-14445, KMS client always authenticates itself using the
> credentials from login user, rather than current user.
> {noformat}
> 2018-12-07 15:58:30,663 DEBUG [main]
> org.apache.hadoop.crypto.key.kms.KMSClientProvider: Using loginUser when
> Kerberos is enabled but the actual user does not have either KMS Delegation
> Token or Kerberos Credentials
> {noformat}
> The log message {{"Using loginUser when Kerberos is enabled but the actual
> user does not have either KMS Delegation Token or Kerberos Credentials"}} is
> printed because {{KMSClientProvider#containsKmsDt()}} is null when it
> definitely has the kms delegation token.
> In fact, {{KMSClientProvider#containsKmsDt()}} should select delegation token
> using {{clientTokenProvider.selectDelegationToken(creds)}} rather than
> checking if its dtService is in the user credentials.
> This is done correctly in {{KMSClientProvider#createAuthenticatedURL}} though.
> We found this bug when it broke Cloudera's Backup and Disaster Recovery tool.
>
> [~daryn] [~xiaochen] mind taking a look? HADOOP-14445 is a huge patch but it
> is almost perfect except for this bug.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15959) revert HADOOP-12751
[ https://issues.apache.org/jira/browse/HADOOP-15959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15959: Fix Version/s: (was: 3.2.1) 3.2.0 > revert HADOOP-12751 > --- > > Key: HADOOP-15959 > URL: https://issues.apache.org/jira/browse/HADOOP-15959 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 3.2.0, 3.1.1, 2.9.2, 3.0.3, 2.7.7, 2.8.5 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 3.2.0, 3.0.4, 3.1.2 > > Attachments: HADOOP-15959-001.patch, HADOOP-15959-branch-2-002.patch > > > HADOOP-12751 doesn't quite work right. Revert. > (this patch is so jenkins can do the test runs) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15586) Fix wrong log statement in AbstractService
[
https://issues.apache.org/jira/browse/HADOOP-15586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676753#comment-16676753
]
Sunil Govindan commented on HADOOP-15586:
-
Updated Fixed versions.
> Fix wrong log statement in AbstractService
> --
>
> Key: HADOOP-15586
> URL: https://issues.apache.org/jira/browse/HADOOP-15586
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
>Affects Versions: 2.9.0, 3.1.0
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Fix For: 3.2.0, 3.3.0
>
> Attachments: HADOOP-15586-001.patch, HADOOP-15586-002.patch,
> HADOOP-15586-003.patch
>
>
> There are some wrong logging statements in AbstractService, here is one
> example:
> {code:java}
> LOG.debug("noteFailure {}" + exception);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15586) Fix wrong log statement in AbstractService
[
https://issues.apache.org/jira/browse/HADOOP-15586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil Govindan updated HADOOP-15586:
Fix Version/s: 3.3.0
3.2.0
> Fix wrong log statement in AbstractService
> --
>
> Key: HADOOP-15586
> URL: https://issues.apache.org/jira/browse/HADOOP-15586
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
>Affects Versions: 2.9.0, 3.1.0
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Fix For: 3.2.0, 3.3.0
>
> Attachments: HADOOP-15586-001.patch, HADOOP-15586-002.patch,
> HADOOP-15586-003.patch
>
>
> There are some wrong logging statements in AbstractService, here is one
> example:
> {code:java}
> LOG.debug("noteFailure {}" + exception);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15740) ABFS: Check variable names during initialization of AbfsClientThrottlingIntercept
[ https://issues.apache.org/jira/browse/HADOOP-15740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15740: Fix Version/s: 3.2.0 > ABFS: Check variable names during initialization of > AbfsClientThrottlingIntercept > -- > > Key: HADOOP-15740 > URL: https://issues.apache.org/jira/browse/HADOOP-15740 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sneha Varma >Assignee: Sneha Varma >Priority: Minor > Fix For: 3.2.0 > > Attachments: HADOOP-15740-HADOOP-15407-001.patch > > > In the initializeSingleton function of the AbfsClientThrottlingIntercept the > local variable name is same as global variable isAutoThrottlingEnabled > because of which the global isAutoThrottlingEnabled is not being set to true. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15356) Make HTTP timeout configurable in ADLS Connector
[ https://issues.apache.org/jira/browse/HADOOP-15356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15356: Fix Version/s: 3.3.0 3.2.0 > Make HTTP timeout configurable in ADLS Connector > > > Key: HADOOP-15356 > URL: https://issues.apache.org/jira/browse/HADOOP-15356 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Reporter: Atul Sikaria >Assignee: Atul Sikaria >Priority: Major > Fix For: 3.2.0, 3.3.0 > > Attachments: HADOOP-15356.001.patch, HADOOP-15356.002.patch, > HADOOP-15356.003.patch, HADOOP-15356.004.patch, HADOOP-15356.005.patch > > > Currently the HTTP timeout for the connections to ADLS are not configurable > in Hadoop. This patch enables the timeouts to be configurable based on a > core-site config setting. Also, up the ADLS SDK version to 2.2.8, that has > default value of 60 seconds - any optimizations to that setting can now be > done in Hadoop through core-site. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15356) Make HTTP timeout configurable in ADLS Connector
[ https://issues.apache.org/jira/browse/HADOOP-15356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676736#comment-16676736 ] Sunil Govindan commented on HADOOP-15356: - Updated Fixed version to 3.3.0 and 3.2.0 > Make HTTP timeout configurable in ADLS Connector > > > Key: HADOOP-15356 > URL: https://issues.apache.org/jira/browse/HADOOP-15356 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Reporter: Atul Sikaria >Assignee: Atul Sikaria >Priority: Major > Fix For: 3.2.0, 3.3.0 > > Attachments: HADOOP-15356.001.patch, HADOOP-15356.002.patch, > HADOOP-15356.003.patch, HADOOP-15356.004.patch, HADOOP-15356.005.patch > > > Currently the HTTP timeout for the connections to ADLS are not configurable > in Hadoop. This patch enables the timeouts to be configurable based on a > core-site config setting. Also, up the ADLS SDK version to 2.2.8, that has > default value of 60 seconds - any optimizations to that setting can now be > done in Hadoop through core-site. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15349) S3Guard DDB retryBackoff to be more informative on limits exceeded
[ https://issues.apache.org/jira/browse/HADOOP-15349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15349: Fix Version/s: 3.3.0 3.2.0 > S3Guard DDB retryBackoff to be more informative on limits exceeded > -- > > Key: HADOOP-15349 > URL: https://issues.apache.org/jira/browse/HADOOP-15349 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Major > Fix For: 3.2.0, 3.3.0 > > Attachments: HADOOP-15349.001.patch, failure.log > > > When S3Guard can't update the DB and so throws an IOE after the retry limit > is exceeded, it's not at all informative. Improve logging & exception -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15349) S3Guard DDB retryBackoff to be more informative on limits exceeded
[ https://issues.apache.org/jira/browse/HADOOP-15349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676483#comment-16676483 ] Sunil Govindan commented on HADOOP-15349: - I added 3.2.0 and 3.3.0 to Fixed version. Pls add if i missed any versions. > S3Guard DDB retryBackoff to be more informative on limits exceeded > -- > > Key: HADOOP-15349 > URL: https://issues.apache.org/jira/browse/HADOOP-15349 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Major > Fix For: 3.2.0, 3.3.0 > > Attachments: HADOOP-15349.001.patch, failure.log > > > When S3Guard can't update the DB and so throws an IOE after the retry limit > is exceeded, it's not at all informative. Improve logging & exception -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15668) mvn test goal fails on HADOOP-15407 branch
[
https://issues.apache.org/jira/browse/HADOOP-15668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil Govindan resolved HADOOP-15668.
-
Resolution: Not A Problem
Closed with correct status
> mvn test goal fails on HADOOP-15407 branch
> --
>
> Key: HADOOP-15668
> URL: https://issues.apache.org/jira/browse/HADOOP-15668
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>Priority: Major
>
> It's very easy to reproduce:
> {code}cd hadoop-common-project/hadoop-common/
> mvn test -Dtest=Whatever{code}
> The error is due to
> {code} [exec] Running bats -t hadoop_stop_daemon.bats
> [exec] 1..2
> [exec] ok 1 hadoop_stop_daemon_changing_pid
> [exec] not ok 2 hadoop_stop_daemon_force_kill
> [exec] # (in test file hadoop_stop_daemon.bats, line 43)
> [exec] # `[ -f ${TMP}/pidfile ]' failed
> [exec] # bindir:
> /Users/abokor/work/hadoop/hadoop-common-project/hadoop-common/src/test/scripts
> [exec] # sh:
> /Users/abokor/work/hadoop/hadoop-common-project/hadoop-common/src/test/scripts/process_with_sigterm_trap.sh:
> No such file or directory{code}
> This happens because actually 3 commits belong to HADOOP-15527 but
> HADOOP-15407 branch contains only one of them so the test won't find
> process_with_sigterm_trap.sh.
> I am not sure what is the best practice to solve this kind of issues. Is
> patch required or can somebody just cherry-pick the missing commits?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-15668) mvn test goal fails on HADOOP-15407 branch
[
https://issues.apache.org/jira/browse/HADOOP-15668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil Govindan reopened HADOOP-15668:
-
> mvn test goal fails on HADOOP-15407 branch
> --
>
> Key: HADOOP-15668
> URL: https://issues.apache.org/jira/browse/HADOOP-15668
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>Priority: Major
>
> It's very easy to reproduce:
> {code}cd hadoop-common-project/hadoop-common/
> mvn test -Dtest=Whatever{code}
> The error is due to
> {code} [exec] Running bats -t hadoop_stop_daemon.bats
> [exec] 1..2
> [exec] ok 1 hadoop_stop_daemon_changing_pid
> [exec] not ok 2 hadoop_stop_daemon_force_kill
> [exec] # (in test file hadoop_stop_daemon.bats, line 43)
> [exec] # `[ -f ${TMP}/pidfile ]' failed
> [exec] # bindir:
> /Users/abokor/work/hadoop/hadoop-common-project/hadoop-common/src/test/scripts
> [exec] # sh:
> /Users/abokor/work/hadoop/hadoop-common-project/hadoop-common/src/test/scripts/process_with_sigterm_trap.sh:
> No such file or directory{code}
> This happens because actually 3 commits belong to HADOOP-15527 but
> HADOOP-15407 branch contains only one of them so the test won't find
> process_with_sigterm_trap.sh.
> I am not sure what is the best practice to solve this kind of issues. Is
> patch required or can somebody just cherry-pick the missing commits?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15688) ABFS: InputStream wrapped in FSDataInputStream twice
[ https://issues.apache.org/jira/browse/HADOOP-15688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15688: Fix Version/s: 3.2.0 > ABFS: InputStream wrapped in FSDataInputStream twice > > > Key: HADOOP-15688 > URL: https://issues.apache.org/jira/browse/HADOOP-15688 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15688-HADOOP-15407-002.patch, > HADOOP-15688.001.patch > > > I can't read Parquet files from ABFS. It has 2 different implementations to > read seekable streams, and it'll use the one that uses ByteBuffer reads if it > can. It currently decides to use the ByteBuffer read implementation because > the FSDataInputStream it gets back wraps another FSDataInputStream, which > implements ByteBufferReadable. > That's not the most robust way to check that ByteBufferReads are supported by > the ultimately underlying InputStream, but it's unnecessary and probably a > mistake to double-wrap the InputStream, so let's not. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15659) ABFS: Code changes for bug fix and new tests
[ https://issues.apache.org/jira/browse/HADOOP-15659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15659: Fix Version/s: 3.2.0 > ABFS: Code changes for bug fix and new tests > > > Key: HADOOP-15659 > URL: https://issues.apache.org/jira/browse/HADOOP-15659 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15659-HADOOP-15407-001.patch, > HADOOP-15659-HADOOP-15407-002.patch, HADOOP-15659-HADOOP-15407-003.patch > > > - add bug fixes. > - remove unnecessary dependencies. > - add new tests for code changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15669) ABFS: Improve HTTPS Performance
[ https://issues.apache.org/jira/browse/HADOOP-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676368#comment-16676368 ] Sunil Govindan commented on HADOOP-15669: - Updated fixed version to 3.2.0 as HADOOP-15407 branch is merged to trunk > ABFS: Improve HTTPS Performance > --- > > Key: HADOOP-15669 > URL: https://issues.apache.org/jira/browse/HADOOP-15669 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Vishwajeet Dusane >Priority: Major > Fix For: 3.2.0 > > Attachments: ABFS - Improve HTTPS Performance Over Java Based > Client.pdf, HADOOP-15669-HADOOP-15407-01.patch, > HADOOP-15669-HADOOP-15407-02.patch, HADOOP-15669-HADOOP-15407-03.patch, > HADOOP-15669-HADOOP-15407-04.patch > > > We see approximately 50% worse throughput for ABFS over HTTPs vs HTTP. Lets > perform a detailed measurement and see what can be done to improve throughput. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15659) ABFS: Code changes for bug fix and new tests
[ https://issues.apache.org/jira/browse/HADOOP-15659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676370#comment-16676370 ] Sunil Govindan commented on HADOOP-15659: - Updated fixed version to 3.2.0 as HADOOP-15407 branch is merged to trunk > ABFS: Code changes for bug fix and new tests > > > Key: HADOOP-15659 > URL: https://issues.apache.org/jira/browse/HADOOP-15659 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15659-HADOOP-15407-001.patch, > HADOOP-15659-HADOOP-15407-002.patch, HADOOP-15659-HADOOP-15407-003.patch > > > - add bug fixes. > - remove unnecessary dependencies. > - add new tests for code changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15660) ABFS: Add support for OAuth
[ https://issues.apache.org/jira/browse/HADOOP-15660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15660: Fix Version/s: 3.2.0 > ABFS: Add support for OAuth > --- > > Key: HADOOP-15660 > URL: https://issues.apache.org/jira/browse/HADOOP-15660 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15660-HADOOP-15407-001.patch, > HADOOP-15660-HADOOP-15407-002.patch, HADOOP-15660-HADOOP-15407-003.patch > > > - Add support for OAuth -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15669) ABFS: Improve HTTPS Performance
[ https://issues.apache.org/jira/browse/HADOOP-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15669: Fix Version/s: 3.2.0 > ABFS: Improve HTTPS Performance > --- > > Key: HADOOP-15669 > URL: https://issues.apache.org/jira/browse/HADOOP-15669 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Vishwajeet Dusane >Priority: Major > Fix For: 3.2.0 > > Attachments: ABFS - Improve HTTPS Performance Over Java Based > Client.pdf, HADOOP-15669-HADOOP-15407-01.patch, > HADOOP-15669-HADOOP-15407-02.patch, HADOOP-15669-HADOOP-15407-03.patch, > HADOOP-15669-HADOOP-15407-04.patch > > > We see approximately 50% worse throughput for ABFS over HTTPs vs HTTP. Lets > perform a detailed measurement and see what can be done to improve throughput. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15661) ABFS: Add support for ACL
[ https://issues.apache.org/jira/browse/HADOOP-15661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676366#comment-16676366 ] Sunil Govindan commented on HADOOP-15661: - Updated fixed version to 3.2.0 as HADOOP-15407 branch is merged to trunk > ABFS: Add support for ACL > - > > Key: HADOOP-15661 > URL: https://issues.apache.org/jira/browse/HADOOP-15661 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15661-HADOOP-15407-001.patch, > HADOOP-15661-HADOOP-15407-002.patch, HADOOP-15661-HADOOP-15407-003.patch, > HADOOP-15661-HADOOP-15407-004.patch, HADOOP-15661-HADOOP-15407-005.patch > > > - Add support for ACL -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15661) ABFS: Add support for ACL
[ https://issues.apache.org/jira/browse/HADOOP-15661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15661: Fix Version/s: 3.2.0 > ABFS: Add support for ACL > - > > Key: HADOOP-15661 > URL: https://issues.apache.org/jira/browse/HADOOP-15661 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15661-HADOOP-15407-001.patch, > HADOOP-15661-HADOOP-15407-002.patch, HADOOP-15661-HADOOP-15407-003.patch, > HADOOP-15661-HADOOP-15407-004.patch, HADOOP-15661-HADOOP-15407-005.patch > > > - Add support for ACL -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15660) ABFS: Add support for OAuth
[ https://issues.apache.org/jira/browse/HADOOP-15660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676367#comment-16676367 ] Sunil Govindan commented on HADOOP-15660: - Updated fixed version to 3.2.0 as HADOOP-15407 branch is merged to trunk > ABFS: Add support for OAuth > --- > > Key: HADOOP-15660 > URL: https://issues.apache.org/jira/browse/HADOOP-15660 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15660-HADOOP-15407-001.patch, > HADOOP-15660-HADOOP-15407-002.patch, HADOOP-15660-HADOOP-15407-003.patch > > > - Add support for OAuth -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15682) ABFS: Add support for StreamCapabilities. Fix javadoc and checkstyle
[ https://issues.apache.org/jira/browse/HADOOP-15682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676365#comment-16676365 ] Sunil Govindan commented on HADOOP-15682: - Updated fixed version to 3.2.0 as HADOOP-15407 branch is merged to trunk > ABFS: Add support for StreamCapabilities. Fix javadoc and checkstyle > > > Key: HADOOP-15682 > URL: https://issues.apache.org/jira/browse/HADOOP-15682 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15682-HADOOP-15407-001.patch > > > Add support for the new StreamCapabilities interface. This work is similar > to what was done for WASB > [HADOOP-15677|https://jira.apache.org/jira/browse/HADOOP-15677]. > Also fix javadoc and checkstyle errors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15682) ABFS: Add support for StreamCapabilities. Fix javadoc and checkstyle
[ https://issues.apache.org/jira/browse/HADOOP-15682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15682: Fix Version/s: 3.2.0 > ABFS: Add support for StreamCapabilities. Fix javadoc and checkstyle > > > Key: HADOOP-15682 > URL: https://issues.apache.org/jira/browse/HADOOP-15682 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15682-HADOOP-15407-001.patch > > > Add support for the new StreamCapabilities interface. This work is similar > to what was done for WASB > [HADOOP-15677|https://jira.apache.org/jira/browse/HADOOP-15677]. > Also fix javadoc and checkstyle errors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15663) ABFS: Simplify configuration
[ https://issues.apache.org/jira/browse/HADOOP-15663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15663: Fix Version/s: 3.2.0 > ABFS: Simplify configuration > > > Key: HADOOP-15663 > URL: https://issues.apache.org/jira/browse/HADOOP-15663 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15663-HADOOP-15407-001.patch, > HADOOP-15663-HADOOP-15407-002.patch, HADOOP-15663-HADOOP-15407-003.patch, > HADOOP-15663-HADOOP-15407-004.patch, HADOOP-15663-HADOOP-15407-005.patch > > > Configuration for WASB and ABFS is too complex. The current approach is to > use four files for test configuration. > Both WASB and ABFS have basic test configuration which is committed to the > repo (azure-test.xml and azure-bfs-test.xml). Currently these contain the > fs.AbstractFileSystem.[scheme].impl configuration, but otherwise are empty > except for an include reference to a file containing the endpoint > credentials. > Both WASB and ABFS have endpoint credential configuration files > (azure-auth-keys.xml and azure-bfs-auth-keys.xml). These have been added to > .gitignore to prevent them from accidentally being submitted in a patch, > which would leak the developers storage account credentials. These files > contain account names, storage account keys, and service endpoints. > There is some overlap of the configuration for WASB and ABFS, where they use > the same property name but use different values. > 1) Let's reduce the number of test configuration files to one, if possible. > 2) Let's simplify the account name, key, and endpoint configuration for WASB > and ABFS if possible, but still support the legacy way of doing it, which is > very error prone. > 3) Let's improve error handling, so that typos or misconfiguration are not so > difficult to troubleshoot. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15664) ABFS: Reduce test run time via parallelization and grouping
[ https://issues.apache.org/jira/browse/HADOOP-15664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15664: Fix Version/s: 3.2.0 > ABFS: Reduce test run time via parallelization and grouping > --- > > Key: HADOOP-15664 > URL: https://issues.apache.org/jira/browse/HADOOP-15664 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15664-HADOOP-15407-001.patch, > HADOOP-15664-HADOOP-15407-002.patch, HADOOP-15664-HADOOP-15407-003.patch, > HADOOP-15664-HADOOP-15407-004.patch, HADOOP-15664-HADOOP-15407-005.patch > > > 1) Let's reduce the total test runtime by improving parallelization of the > tests. > 2) Let's make it possible to select WASB tests, ABFS tests, or both so > developers can run only the tests appropriate for the change they've made. > 3) Update the testing-azure.md accordingly -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15692) ABFS: extensible support for custom oauth
[ https://issues.apache.org/jira/browse/HADOOP-15692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676361#comment-16676361 ] Sunil Govindan commented on HADOOP-15692: - Updated fixed version to 3.2.0 as HADOOP-15407 branch is merged to trunk > ABFS: extensible support for custom oauth > - > > Key: HADOOP-15692 > URL: https://issues.apache.org/jira/browse/HADOOP-15692 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15692-HADOOP-15407-001.patch, > HADOOP-15692-HADOOP-15407-002.patch > > > ABFS supports oauth in various forms and needs to export interfaces for > customization of FileSystem.getDelegationToken and getAccessToken. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15664) ABFS: Reduce test run time via parallelization and grouping
[ https://issues.apache.org/jira/browse/HADOOP-15664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676359#comment-16676359 ] Sunil Govindan commented on HADOOP-15664: - Updated fixed version to 3.2.0 as HADOOP-15407 branch is merged to trunk > ABFS: Reduce test run time via parallelization and grouping > --- > > Key: HADOOP-15664 > URL: https://issues.apache.org/jira/browse/HADOOP-15664 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15664-HADOOP-15407-001.patch, > HADOOP-15664-HADOOP-15407-002.patch, HADOOP-15664-HADOOP-15407-003.patch, > HADOOP-15664-HADOOP-15407-004.patch, HADOOP-15664-HADOOP-15407-005.patch > > > 1) Let's reduce the total test runtime by improving parallelization of the > tests. > 2) Let's make it possible to select WASB tests, ABFS tests, or both so > developers can run only the tests appropriate for the change they've made. > 3) Update the testing-azure.md accordingly -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15692) ABFS: extensible support for custom oauth
[ https://issues.apache.org/jira/browse/HADOOP-15692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15692: Fix Version/s: 3.2.0 > ABFS: extensible support for custom oauth > - > > Key: HADOOP-15692 > URL: https://issues.apache.org/jira/browse/HADOOP-15692 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15692-HADOOP-15407-001.patch, > HADOOP-15692-HADOOP-15407-002.patch > > > ABFS supports oauth in various forms and needs to export interfaces for > customization of FileSystem.getDelegationToken and getAccessToken. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15663) ABFS: Simplify configuration
[ https://issues.apache.org/jira/browse/HADOOP-15663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676360#comment-16676360 ] Sunil Govindan commented on HADOOP-15663: - Updated fixed version to 3.2.0 as HADOOP-15407 branch is merged to trunk > ABFS: Simplify configuration > > > Key: HADOOP-15663 > URL: https://issues.apache.org/jira/browse/HADOOP-15663 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15663-HADOOP-15407-001.patch, > HADOOP-15663-HADOOP-15407-002.patch, HADOOP-15663-HADOOP-15407-003.patch, > HADOOP-15663-HADOOP-15407-004.patch, HADOOP-15663-HADOOP-15407-005.patch > > > Configuration for WASB and ABFS is too complex. The current approach is to > use four files for test configuration. > Both WASB and ABFS have basic test configuration which is committed to the > repo (azure-test.xml and azure-bfs-test.xml). Currently these contain the > fs.AbstractFileSystem.[scheme].impl configuration, but otherwise are empty > except for an include reference to a file containing the endpoint > credentials. > Both WASB and ABFS have endpoint credential configuration files > (azure-auth-keys.xml and azure-bfs-auth-keys.xml). These have been added to > .gitignore to prevent them from accidentally being submitted in a patch, > which would leak the developers storage account credentials. These files > contain account names, storage account keys, and service endpoints. > There is some overlap of the configuration for WASB and ABFS, where they use > the same property name but use different values. > 1) Let's reduce the number of test configuration files to one, if possible. > 2) Let's simplify the account name, key, and endpoint configuration for WASB > and ABFS if possible, but still support the legacy way of doing it, which is > very error prone. > 3) Let's improve error handling, so that typos or misconfiguration are not so > difficult to troubleshoot. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15745) Add ABFS configuration to ConfigRedactor
[ https://issues.apache.org/jira/browse/HADOOP-15745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15745: Fix Version/s: 3.2.0 > Add ABFS configuration to ConfigRedactor > > > Key: HADOOP-15745 > URL: https://issues.apache.org/jira/browse/HADOOP-15745 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15735-HADOOP-15407.001.patch, > HADOOP-15745-HADOOP-15407-002.patch > > > Sensitive information like credentials should be detected by ConfigRedactor > so they never appear in logs or other channels. ABFS credentials are not all > currently detected correctly so we should amend the default list of config > patterns. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15703) ABFS - Implement client-side throttling
[ https://issues.apache.org/jira/browse/HADOOP-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676358#comment-16676358 ] Sunil Govindan commented on HADOOP-15703: - Updated fixed version to 3.2.0 as HADOOP-15407 branch is merged to trunk > ABFS - Implement client-side throttling > > > Key: HADOOP-15703 > URL: https://issues.apache.org/jira/browse/HADOOP-15703 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sneha Varma >Assignee: Thomas Marquardt >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15703-HADOOP-15407-001.patch, > HADOOP-15703-HADOOP-15407-002.patch, HADOOP-15703-HADOOP-15407-003.patch, > HADOOP-15703-HADOOP-15407-004.patch > > > Big data workloads frequently exceed the AzureBlobFS max ingress and egress > limits > (https://docs.microsoft.com/en-us/azure/storage/common/storage-scalability-targets). > For example, the max ingress limit for a GRS account in the United States is > currently 10 Gbps. When the limit is exceeded, the AzureBlobFS service fails > a percentage of incoming requests, and this causes the client to initiate the > retry policy. The retry policy delays requests by sleeping, but the sleep > duration is independent of the client throughput and account limit. This > results in low throughput, due to the high number of failed requests and > thrashing causes by the retry policy. > To fix this, we introduce a client-side throttle which minimizes failed > requests and maximizes throughput. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15728) ABFS: Add backward compatibility to handle Unsupported Operation for storage account with no namespace feature
[ https://issues.apache.org/jira/browse/HADOOP-15728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15728: Fix Version/s: 3.2.0 > ABFS: Add backward compatibility to handle Unsupported Operation for storage > account with no namespace feature > -- > > Key: HADOOP-15728 > URL: https://issues.apache.org/jira/browse/HADOOP-15728 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15728-HADOOP-15407-001.patch, > HADOOP-15728-HADOOP-15407-002.patch, HADOOP-15728-HADOOP-15407-003.patch > > > For non-namespaceenbaled account, UnsupportedOperationException is thrown > when setOwner() or setPermission() is called, which cause problems for others. > They should behave the same as before, so should make them no-op if the > storage account is not namespace enabled. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15694) ABFS: Allow OAuth credentials to not be tied to accounts
[ https://issues.apache.org/jira/browse/HADOOP-15694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676355#comment-16676355 ] Sunil Govindan commented on HADOOP-15694: - Updated fixed version to 3.2.0 as HADOOP-15407 branch is merged to trunk > ABFS: Allow OAuth credentials to not be tied to accounts > > > Key: HADOOP-15694 > URL: https://issues.apache.org/jira/browse/HADOOP-15694 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15694-HADOOP-15407-005.patch, > HADOOP-15694-HADOOP-15407.003.patch, HADOOP-15694-HADOOP-15407.004.patch, > HADOOP-15694-HADOOP-15407.006.patch, HADOOP-15694-HADOOP-15407.007.patch, > HADOOP-15694-HADOOP-15407.008.patch, HADOOP-15694.001.patch, > HADOOP-15694.002.patch, HADOOP-15694.003.patch > > > Now that there's OAuth support, it's possible to have a notion of identity > that's distinct from the account itself. If a cluster is configured via OAuth > with it's own identity, it's likely operators will want to use that identity > regardless of which storage account a job uses. > So OAuth configs right now (and probably others) are looked up with > .. I propose that we add a function for looking up these > configs that returns an account-specific value if it exists, but in the event > it does not will also try to return , if that exists. > I can work on a patch for this if nobody has any objections. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15728) ABFS: Add backward compatibility to handle Unsupported Operation for storage account with no namespace feature
[ https://issues.apache.org/jira/browse/HADOOP-15728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676356#comment-16676356 ] Sunil Govindan commented on HADOOP-15728: - Updated fixed version to 3.2.0 as HADOOP-15407 branch is merged to trunk > ABFS: Add backward compatibility to handle Unsupported Operation for storage > account with no namespace feature > -- > > Key: HADOOP-15728 > URL: https://issues.apache.org/jira/browse/HADOOP-15728 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15728-HADOOP-15407-001.patch, > HADOOP-15728-HADOOP-15407-002.patch, HADOOP-15728-HADOOP-15407-003.patch > > > For non-namespaceenbaled account, UnsupportedOperationException is thrown > when setOwner() or setPermission() is called, which cause problems for others. > They should behave the same as before, so should make them no-op if the > storage account is not namespace enabled. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15703) ABFS - Implement client-side throttling
[ https://issues.apache.org/jira/browse/HADOOP-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15703: Fix Version/s: 3.2.0 > ABFS - Implement client-side throttling > > > Key: HADOOP-15703 > URL: https://issues.apache.org/jira/browse/HADOOP-15703 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sneha Varma >Assignee: Thomas Marquardt >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15703-HADOOP-15407-001.patch, > HADOOP-15703-HADOOP-15407-002.patch, HADOOP-15703-HADOOP-15407-003.patch, > HADOOP-15703-HADOOP-15407-004.patch > > > Big data workloads frequently exceed the AzureBlobFS max ingress and egress > limits > (https://docs.microsoft.com/en-us/azure/storage/common/storage-scalability-targets). > For example, the max ingress limit for a GRS account in the United States is > currently 10 Gbps. When the limit is exceeded, the AzureBlobFS service fails > a percentage of incoming requests, and this causes the client to initiate the > retry policy. The retry policy delays requests by sleeping, but the sleep > duration is independent of the client throughput and account limit. This > results in low throughput, due to the high number of failed requests and > thrashing causes by the retry policy. > To fix this, we introduce a client-side throttle which minimizes failed > requests and maximizes throughput. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15745) Add ABFS configuration to ConfigRedactor
[ https://issues.apache.org/jira/browse/HADOOP-15745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676357#comment-16676357 ] Sunil Govindan commented on HADOOP-15745: - Updated fixed version to 3.2.0 as HADOOP-15407 branch is merged to trunk > Add ABFS configuration to ConfigRedactor > > > Key: HADOOP-15745 > URL: https://issues.apache.org/jira/browse/HADOOP-15745 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15735-HADOOP-15407.001.patch, > HADOOP-15745-HADOOP-15407-002.patch > > > Sensitive information like credentials should be detected by ConfigRedactor > so they never appear in logs or other channels. ABFS credentials are not all > currently detected correctly so we should amend the default list of config > patterns. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15757) ABFS: remove dependency on common-codec Base64
[ https://issues.apache.org/jira/browse/HADOOP-15757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676352#comment-16676352 ] Sunil Govindan commented on HADOOP-15757: - Updated fixed version to 3.2.0 as HADOOP-15407 branch is merged to trunk > ABFS: remove dependency on common-codec Base64 > -- > > Key: HADOOP-15757 > URL: https://issues.apache.org/jira/browse/HADOOP-15757 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15757-HADOOP-15407-001.patch, > HADOOP-15757-HADOOP-15407-002.patch > > > Currently ABFS relies on common-codec Base64, because different versions of > common-codec are widely used and some are missing the methods needed by ABFS, > it cause lots of "no such method" exception in customer's env, hence we > decide to add util for Base64 to avoid such issues in future. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15753) ABFS: support path "abfs://mycluster/file/path"
[ https://issues.apache.org/jira/browse/HADOOP-15753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676353#comment-16676353 ] Sunil Govindan commented on HADOOP-15753: - Updated fixed version to 3.2.0 as HADOOP-15407 branch is merged to trunk > ABFS: support path "abfs://mycluster/file/path" > --- > > Key: HADOOP-15753 > URL: https://issues.apache.org/jira/browse/HADOOP-15753 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15753-HADOOP-15407-001.patch > > > WASB support path format: "wasb://mycluster/file/path", but ABFS doesn't, > which caused some issues for customer. I will add support for this path > format. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15753) ABFS: support path "abfs://mycluster/file/path"
[ https://issues.apache.org/jira/browse/HADOOP-15753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15753: Fix Version/s: 3.2.0 > ABFS: support path "abfs://mycluster/file/path" > --- > > Key: HADOOP-15753 > URL: https://issues.apache.org/jira/browse/HADOOP-15753 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15753-HADOOP-15407-001.patch > > > WASB support path format: "wasb://mycluster/file/path", but ABFS doesn't, > which caused some issues for customer. I will add support for this path > format. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15757) ABFS: remove dependency on common-codec Base64
[ https://issues.apache.org/jira/browse/HADOOP-15757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15757: Fix Version/s: 3.2.0 > ABFS: remove dependency on common-codec Base64 > -- > > Key: HADOOP-15757 > URL: https://issues.apache.org/jira/browse/HADOOP-15757 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15757-HADOOP-15407-001.patch, > HADOOP-15757-HADOOP-15407-002.patch > > > Currently ABFS relies on common-codec Base64, because different versions of > common-codec are widely used and some are missing the methods needed by ABFS, > it cause lots of "no such method" exception in customer's env, hence we > decide to add util for Base64 to avoid such issues in future. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15694) ABFS: Allow OAuth credentials to not be tied to accounts
[ https://issues.apache.org/jira/browse/HADOOP-15694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15694: Fix Version/s: 3.2.0 > ABFS: Allow OAuth credentials to not be tied to accounts > > > Key: HADOOP-15694 > URL: https://issues.apache.org/jira/browse/HADOOP-15694 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15694-HADOOP-15407-005.patch, > HADOOP-15694-HADOOP-15407.003.patch, HADOOP-15694-HADOOP-15407.004.patch, > HADOOP-15694-HADOOP-15407.006.patch, HADOOP-15694-HADOOP-15407.007.patch, > HADOOP-15694-HADOOP-15407.008.patch, HADOOP-15694.001.patch, > HADOOP-15694.002.patch, HADOOP-15694.003.patch > > > Now that there's OAuth support, it's possible to have a notion of identity > that's distinct from the account itself. If a cluster is configured via OAuth > with it's own identity, it's likely operators will want to use that identity > regardless of which storage account a job uses. > So OAuth configs right now (and probably others) are looked up with > .. I propose that we add a function for looking up these > configs that returns an account-specific value if it exists, but in the event > it does not will also try to return , if that exists. > I can work on a patch for this if nobody has any objections. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15773) ABFS: Fix issues raised by Yetus
[ https://issues.apache.org/jira/browse/HADOOP-15773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15773: Fix Version/s: 3.2.0 > ABFS: Fix issues raised by Yetus > > > Key: HADOOP-15773 > URL: https://issues.apache.org/jira/browse/HADOOP-15773 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15773-HADOOP-15407.001.patch, > HADOOP-15773-HADOOP-15407.002.patch > > > I aggregated the HADOOP-15407 branch into a single patch and posted it on > HADOOP-15770 just to get an aggregate report of all current issues raised by > Yetus. There was a javac deprecation warning, a number of checkstyle issues, > some whitespace issues, and there are a couple of valid javadoc errors I see > locally. Let's fix them before we merge. > I see a number of wildcard imports, all for the contents of large classes of > configuration constants, and I think those should stay the way they are. > There are a number of existing checkstyle issues in WASB, too that are > irrelevant for the merge, and there are some field visibility issues in tests > that are required that way for the tests to work as designed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15773) ABFS: Fix issues raised by Yetus
[ https://issues.apache.org/jira/browse/HADOOP-15773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676351#comment-16676351 ] Sunil Govindan commented on HADOOP-15773: - Updated fixed version to 3.2.0 > ABFS: Fix issues raised by Yetus > > > Key: HADOOP-15773 > URL: https://issues.apache.org/jira/browse/HADOOP-15773 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15773-HADOOP-15407.001.patch, > HADOOP-15773-HADOOP-15407.002.patch > > > I aggregated the HADOOP-15407 branch into a single patch and posted it on > HADOOP-15770 just to get an aggregate report of all current issues raised by > Yetus. There was a javac deprecation warning, a number of checkstyle issues, > some whitespace issues, and there are a couple of valid javadoc errors I see > locally. Let's fix them before we merge. > I see a number of wildcard imports, all for the contents of large classes of > configuration constants, and I think those should stay the way they are. > There are a number of existing checkstyle issues in WASB, too that are > irrelevant for the merge, and there are some field visibility issues in tests > that are required that way for the tests to work as designed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15456) create base image for running secure ozone cluster
[ https://issues.apache.org/jira/browse/HADOOP-15456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676335#comment-16676335 ] Sunil Govindan commented on HADOOP-15456: - [~elek] cud u pls update correct Fixed version. > create base image for running secure ozone cluster > -- > > Key: HADOOP-15456 > URL: https://issues.apache.org/jira/browse/HADOOP-15456 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HADOOP-15456-docker-hadoop-runner.001.patch, > HADOOP-15456-docker-hadoop-runner.002.patch, > HADOOP-15456-docker-hadoop-runner.003.patch, secure-ozone.tar > > > Create docker image to run secure ozone cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15825) ABFS: Enable some tests for namespace not enabled account using OAuth
[ https://issues.apache.org/jira/browse/HADOOP-15825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676333#comment-16676333 ] Sunil Govindan commented on HADOOP-15825: - Updated Fixed Version as 3.3.0 > ABFS: Enable some tests for namespace not enabled account using OAuth > - > > Key: HADOOP-15825 > URL: https://issues.apache.org/jira/browse/HADOOP-15825 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15825-001.patch, HADOOP-15825-002.patch > > > When testing namespace not enabled account using Oauth, some tests were > skipped. So need to update the tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15825) ABFS: Enable some tests for namespace not enabled account using OAuth
[ https://issues.apache.org/jira/browse/HADOOP-15825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15825: Fix Version/s: 3.3.0 > ABFS: Enable some tests for namespace not enabled account using OAuth > - > > Key: HADOOP-15825 > URL: https://issues.apache.org/jira/browse/HADOOP-15825 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15825-001.patch, HADOOP-15825-002.patch > > > When testing namespace not enabled account using Oauth, some tests were > skipped. So need to update the tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15626) FileContextMainOperationsBaseTest.testBuilderCreateAppendExistingFile fails on filesystems without append.
[ https://issues.apache.org/jira/browse/HADOOP-15626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15626: Fix Version/s: 3.2.0 > FileContextMainOperationsBaseTest.testBuilderCreateAppendExistingFile fails > on filesystems without append. > -- > > Key: HADOOP-15626 > URL: https://issues.apache.org/jira/browse/HADOOP-15626 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Fix For: 3.2.0 > > Attachments: HADOOP-15626-001.patch > > > After HADOOP-14396. one of the new tests fails on S3A because append() isn't > supported there -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15626) FileContextMainOperationsBaseTest.testBuilderCreateAppendExistingFile fails on filesystems without append.
[ https://issues.apache.org/jira/browse/HADOOP-15626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676226#comment-16676226 ] Sunil Govindan commented on HADOOP-15626: - Updating Fixed version to 3.2.0 > FileContextMainOperationsBaseTest.testBuilderCreateAppendExistingFile fails > on filesystems without append. > -- > > Key: HADOOP-15626 > URL: https://issues.apache.org/jira/browse/HADOOP-15626 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15626-001.patch > > > After HADOOP-14396. one of the new tests fails on S3A because append() isn't > supported there -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15583) Stabilize S3A Assumed Role support
[
https://issues.apache.org/jira/browse/HADOOP-15583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil Govindan updated HADOOP-15583:
Fix Version/s: 3.2.0
> Stabilize S3A Assumed Role support
> --
>
> Key: HADOOP-15583
> URL: https://issues.apache.org/jira/browse/HADOOP-15583
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Fix For: 3.2.0
>
> Attachments: HADOOP-15583-001.patch, HADOOP-15583-002.patch,
> HADOOP-15583-003.patch, HADOOP-15583-004.patch, HADOOP-15583-005.patch
>
>
> started off just on sharing credentials across S3A and S3Guard, but in the
> process it has grown to becoming one of stabilising the assumed role support
> so it can be used for more than just testing.
> Was: "S3Guard to get AWS Credential chain from S3AFS; credentials closed() on
> shutdown"
> h3. Issue: lack of auth chain sharing causes ddb and s3 to get out of sync
> S3Guard builds its DDB auth chain itself, which stops it having to worry
> about being created standalone vs part of an S3AFS, but it means its
> authenticators are in a separate chain.
> When you are using short-lived assumed roles or other session credentials
> updated in the S3A FS authentication chain, you need that same set of
> credentials picked up by DDB. Otherwise, at best you are doubling load, at
> worse: the DDB connector may not get refreshed credentials.
> Proposed: {{DynamoDBClientFactory.createDynamoDBClient()}} to take an
> optional ref to aws credentials. If set: don't create a new set.
> There's one little complication here: our {{AWSCredentialProviderList}} list
> is autocloseable; it's close() will go through all children and close them.
> Apparently the AWS S3 client (And hopefully the DDB client) will close this
> when they are closed themselves. If DDB has the same set of credentials as
> the FS, then there could be trouble if they are closed in one place when the
> other still wants to use them.
> Solution; have a use count the uses of the credentials list, starting at one:
> every close() call decrements, and when this hits zero the cleanup is kicked
> off
> h3. Issue: {{AssumedRoleCredentialProvider}} connector to STS not picking up
> the s3a connection settings, including proxy.
> h3. issue: we're not using getPassword() to get user/password for proxy
> binding for STS. Fix: use that and pass down the bucket ref for per-bucket
> secrets in a JCEKS file.
> h3. Issue; hard to debug what's going wrong :)
> h3. Issue: docs about KMS permissions for SSE-KMS are wrong, and the
> ITestAssumedRole* tests don't request KMS permissions, so fail in a bucket
> when the base s3 FS is using SSE-KMS. KMS permissions need to be included in
> generated profiles
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15583) Stabilize S3A Assumed Role support
[
https://issues.apache.org/jira/browse/HADOOP-15583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676225#comment-16676225
]
Sunil Govindan commented on HADOOP-15583:
-
Updating Fixed version to 3.2.0
> Stabilize S3A Assumed Role support
> --
>
> Key: HADOOP-15583
> URL: https://issues.apache.org/jira/browse/HADOOP-15583
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Fix For: 3.2.0
>
> Attachments: HADOOP-15583-001.patch, HADOOP-15583-002.patch,
> HADOOP-15583-003.patch, HADOOP-15583-004.patch, HADOOP-15583-005.patch
>
>
> started off just on sharing credentials across S3A and S3Guard, but in the
> process it has grown to becoming one of stabilising the assumed role support
> so it can be used for more than just testing.
> Was: "S3Guard to get AWS Credential chain from S3AFS; credentials closed() on
> shutdown"
> h3. Issue: lack of auth chain sharing causes ddb and s3 to get out of sync
> S3Guard builds its DDB auth chain itself, which stops it having to worry
> about being created standalone vs part of an S3AFS, but it means its
> authenticators are in a separate chain.
> When you are using short-lived assumed roles or other session credentials
> updated in the S3A FS authentication chain, you need that same set of
> credentials picked up by DDB. Otherwise, at best you are doubling load, at
> worse: the DDB connector may not get refreshed credentials.
> Proposed: {{DynamoDBClientFactory.createDynamoDBClient()}} to take an
> optional ref to aws credentials. If set: don't create a new set.
> There's one little complication here: our {{AWSCredentialProviderList}} list
> is autocloseable; it's close() will go through all children and close them.
> Apparently the AWS S3 client (And hopefully the DDB client) will close this
> when they are closed themselves. If DDB has the same set of credentials as
> the FS, then there could be trouble if they are closed in one place when the
> other still wants to use them.
> Solution; have a use count the uses of the credentials list, starting at one:
> every close() call decrements, and when this hits zero the cleanup is kicked
> off
> h3. Issue: {{AssumedRoleCredentialProvider}} connector to STS not picking up
> the s3a connection settings, including proxy.
> h3. issue: we're not using getPassword() to get user/password for proxy
> binding for STS. Fix: use that and pass down the bucket ref for per-bucket
> secrets in a JCEKS file.
> h3. Issue; hard to debug what's going wrong :)
> h3. Issue: docs about KMS permissions for SSE-KMS are wrong, and the
> ITestAssumedRole* tests don't request KMS permissions, so fail in a bucket
> when the base s3 FS is using SSE-KMS. KMS permissions need to be included in
> generated profiles
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10783) apache-commons-lang.jar 2.6 does not support FreeBSD -upgrade to 3.x needed
[ https://issues.apache.org/jira/browse/HADOOP-10783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676197#comment-16676197 ] Sunil Govindan commented on HADOOP-10783: - HADOOP-15495, HDFS-13621, MAPREDUCE-7098 and YARN-8363 HDDS-157 are subtasks of this and all are fixed. Reference for future RM's > apache-commons-lang.jar 2.6 does not support FreeBSD -upgrade to 3.x needed > --- > > Key: HADOOP-10783 > URL: https://issues.apache.org/jira/browse/HADOOP-10783 > Project: Hadoop Common > Issue Type: Bug >Reporter: Dmitry Sivachenko >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-10783.2.patch, HADOOP-10783.3.patch, > HADOOP-10783.4.patch, HADOOP-10783.5.patch, HADOOP-10783.6.patch, > commons-lang3_1.patch > > > Hadoop-2.4.1 ships with apache-commons.jar version 2.6. > It does not support FreeBSD (IS_OS_UNIX returns False). > This is fixed in recent versions of apache-commons.jar > Please update apache-commons.jar to recent version so it correctly recognizes > FreeBSD as UNIX-like system. > Right now I get in datanode's log: > 2014-07-04 11:58:10,459 DEBUG > org.apache.hadoop.hdfs.server.datanode.ShortCircui > tRegistry: Disabling ShortCircuitRegistry > java.io.IOException: The OS is not UNIX. > at > org.apache.hadoop.io.nativeio.SharedFileDescriptorFactory.create(SharedFileDescriptorFactory.java:77) > at > org.apache.hadoop.hdfs.server.datanode.ShortCircuitRegistry.(ShortCircuitRegistry.java:169) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.initDataXceiver(DataNode.java:583) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:771) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:289) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1931) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1818) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1865) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2041) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2065) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14785) Specify the behavior of handling conflicts between must and opt parameters
[
https://issues.apache.org/jira/browse/HADOOP-14785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676194#comment-16676194
]
Sunil Govindan commented on HADOOP-14785:
-
Removing Fixed Version as this issue doesnt need any fix.
> Specify the behavior of handling conflicts between must and opt parameters
> ---
>
> Key: HADOOP-14785
> URL: https://issues.apache.org/jira/browse/HADOOP-14785
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Major
>
> It is flexible to allow user to use strings as key/values to specify the
> behaviors of {{FSOutputStream}}, but this flexibility offers the potential
> conflicts between parameters.
> It should specify a general rule of how to handle such conflicts for differnt
> file system implementations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14785) Specify the behavior of handling conflicts between must and opt parameters
[
https://issues.apache.org/jira/browse/HADOOP-14785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil Govindan updated HADOOP-14785:
Fix Version/s: (was: 3.2.0)
> Specify the behavior of handling conflicts between must and opt parameters
> ---
>
> Key: HADOOP-14785
> URL: https://issues.apache.org/jira/browse/HADOOP-14785
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Major
>
> It is flexible to allow user to use strings as key/values to specify the
> behaviors of {{FSOutputStream}}, but this flexibility offers the potential
> conflicts between parameters.
> It should specify a general rule of how to handle such conflicts for differnt
> file system implementations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14852) Intermittent failure of S3Guard TestConsistencyListFiles
[
https://issues.apache.org/jira/browse/HADOOP-14852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil Govindan updated HADOOP-14852:
Fix Version/s: (was: 3.2.0)
> Intermittent failure of S3Guard TestConsistencyListFiles
>
>
> Key: HADOOP-14852
> URL: https://issues.apache.org/jira/browse/HADOOP-14852
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Priority: Major
>
> I'm seeing intermittent test failures with a test run of {{ -Dparallel-tests
> -DtestsThreadCount=8 -Ds3guard -Ddynamo}} (-Dauth set or unset) in which a
> file in DELAY-LISTING-ME isn't being returned in a listing.
> Theories
> * test is wrong
> * config is wrong
> * code is wrong
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14852) Intermittent failure of S3Guard TestConsistencyListFiles
[
https://issues.apache.org/jira/browse/HADOOP-14852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676185#comment-16676185
]
Sunil Govindan commented on HADOOP-14852:
-
Removing Fixed Version as this issue cannot be reproducible.
> Intermittent failure of S3Guard TestConsistencyListFiles
>
>
> Key: HADOOP-14852
> URL: https://issues.apache.org/jira/browse/HADOOP-14852
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Priority: Major
>
> I'm seeing intermittent test failures with a test run of {{ -Dparallel-tests
> -DtestsThreadCount=8 -Ds3guard -Ddynamo}} (-Dauth set or unset) in which a
> file in DELAY-LISTING-ME isn't being returned in a listing.
> Theories
> * test is wrong
> * config is wrong
> * code is wrong
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15367) Update the initialization code in the docker hadoop-runner baseimage
[ https://issues.apache.org/jira/browse/HADOOP-15367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676183#comment-16676183 ] Sunil Govindan commented on HADOOP-15367: - [~elek] [~anu] pls help to update correct Fixed Version for this. I cannot find this patch in trunk and branch-3.2. > Update the initialization code in the docker hadoop-runner baseimage > - > > Key: HADOOP-15367 > URL: https://issues.apache.org/jira/browse/HADOOP-15367 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15367-hadoop-docker-runner.001.patch, > HADOOP-15367-hadoop-docker-runner.002.patch > > > The hadoop-runner baseimage contains initialization code for both the HDFS > namenode/datanode and Ozone/Hdds scm/ksm. > The script name for Ozone/Hdds is changed (from oz to ozone) therefore we > need to updated the base image. > This commit also would be a test for the dockerhub automated build. > Please apply the patch on the top of the _docker-hadoop-runner_ branch. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[ https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15546: Fix Version/s: 3.2.0 > ABFS: tune imports & javadocs; stabilise tests > -- > > Key: HADOOP-15546 > URL: https://issues.apache.org/jira/browse/HADOOP-15546 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: HADOOP-15407 >Reporter: Steve Loughran >Assignee: Thomas Marquardt >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15546-001.patch, > HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch, > HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch, > HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch, > HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-007.patch, > HADOOP-15546-HADOOP-15407-008.patch, HADOOP-15546-HADOOP-15407-009.patch, > HADOOP-15546-HADOOP-15407-010.patch, HADOOP-15546-HADOOP-15407-011.patch, > HADOOP-15546-HADOOP-15407-012.patch, azure-auth-keys.xml > > > Followup on HADOOP-15540 with some initial review tuning > h2. Tuning > * ordering of imports > * rely on azure-auth-keys.xml to store credentials (change imports, > docs,.gitignore) > * log4j -> info > * add a "." to the first sentence of all the javadocs I noticed. > * remove @Public annotations except for some constants (which includes some > commitment to maintain them). > * move the AbstractFS declarations out of the src/test/resources XML file > into core-default.xml for all to use > * other IDE-suggested tweaks > h2. Testing > Review the tests, move to ContractTestUtil assertions, make more consistent > to contract test setup, and general work to make the tests work well over > slower links, document, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15545) ABFS initialize() throws string out of bounds exception of the URI isn't fully qualified
[
https://issues.apache.org/jira/browse/HADOOP-15545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil Govindan updated HADOOP-15545:
Fix Version/s: (was: 3.2.0)
> ABFS initialize() throws string out of bounds exception of the URI isn't
> fully qualified
>
>
> Key: HADOOP-15545
> URL: https://issues.apache.org/jira/browse/HADOOP-15545
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
>
> if you try to connect to a store like {{abfs://user@something/}} you'll see a
> StringIndexOutOfBoundsExceptionbetter to have something useful like "not
> a FQDN"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15676) Cleanup TestSSLHttpServer
[ https://issues.apache.org/jira/browse/HADOOP-15676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15676: Fix Version/s: (was: 3.2.0) 3.2.1 > Cleanup TestSSLHttpServer > - > > Key: HADOOP-15676 > URL: https://issues.apache.org/jira/browse/HADOOP-15676 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 2.6.0 >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > Fix For: 3.2.1 > > Attachments: HADOOP-15676.001.patch, HADOOP-15676.002.patch, > HADOOP-15676.003.patch, HADOOP-15676.004.patch, HADOOP-15676.005.patch > > > This issue will fix: > * Several typos in this class > * Code is not very well readable in some of the places. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15676) Cleanup TestSSLHttpServer
[ https://issues.apache.org/jira/browse/HADOOP-15676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676145#comment-16676145 ] Sunil Govindan commented on HADOOP-15676: - Updating Fixed Version to 3.2.1 as this patch landed on branch-3.2 instead on branch-3.2.0 which was closed for commits. > Cleanup TestSSLHttpServer > - > > Key: HADOOP-15676 > URL: https://issues.apache.org/jira/browse/HADOOP-15676 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 2.6.0 >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > Fix For: 3.2.1 > > Attachments: HADOOP-15676.001.patch, HADOOP-15676.002.patch, > HADOOP-15676.003.patch, HADOOP-15676.004.patch, HADOOP-15676.005.patch > > > This issue will fix: > * Several typos in this class > * Code is not very well readable in some of the places. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15741) [JDK10] Upgrade Maven Javadoc Plugin from 3.0.0-M1 to 3.0.1
[ https://issues.apache.org/jira/browse/HADOOP-15741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676142#comment-16676142 ] Sunil Govindan commented on HADOOP-15741: - Thanks [~ajisakaa] for pointing. For future RM's reference, Commit message for this Jira is "[JDK10] Upgrade Maven Javadoc Plugin from 3.0.0-M1 to 3.0.1". > [JDK10] Upgrade Maven Javadoc Plugin from 3.0.0-M1 to 3.0.1 > --- > > Key: HADOOP-15741 > URL: https://issues.apache.org/jira/browse/HADOOP-15741 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Reporter: Akira Ajisaka >Assignee: Takanobu Asanuma >Priority: Minor > Fix For: 3.2.0 > > Attachments: HADOOP-15741.1.patch, HADOOP-15741.2.patch > > > MJAVADOC-517 is fixed in 3.0.1. Let's upgrade the plugin to 3.0.1 or upper to > support Java 10. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15851) Disable wildfly logs to the console
[
https://issues.apache.org/jira/browse/HADOOP-15851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil Govindan updated HADOOP-15851:
Fix Version/s: (was: 3.2.0)
3.2.1
> Disable wildfly logs to the console
> ---
>
> Key: HADOOP-15851
> URL: https://issues.apache.org/jira/browse/HADOOP-15851
> Project: Hadoop Common
> Issue Type: Task
> Components: fs/azure
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Fix For: 3.2.1
>
> Attachments: HADOOP-15851-001.patch
>
>
> On loading OpenSSL library successfully, Wildfly logging messages like below
> {code:java}
> Oct 15, 2018 6:47:24 AM org.wildfly.openssl.SSL init
> INFO: WFOPENSSL0002 OpenSSL Version OpenSSL 1.1.0g 2 Nov 2017
> {code}
> These messages may fiddle with existing scripts which parses logs with a
> predefined schema.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15851) Disable wildfly logs to the console
[
https://issues.apache.org/jira/browse/HADOOP-15851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676132#comment-16676132
]
Sunil Govindan commented on HADOOP-15851:
-
Updated Fixed Version to 3.2.1 as the fix has not landed on branch-3.2.0 (which
was closed for RC prep during this commit). Please let me know if this is to be
landed on 3.2.0 release.
> Disable wildfly logs to the console
> ---
>
> Key: HADOOP-15851
> URL: https://issues.apache.org/jira/browse/HADOOP-15851
> Project: Hadoop Common
> Issue Type: Task
> Components: fs/azure
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HADOOP-15851-001.patch
>
>
> On loading OpenSSL library successfully, Wildfly logging messages like below
> {code:java}
> Oct 15, 2018 6:47:24 AM org.wildfly.openssl.SSL init
> INFO: WFOPENSSL0002 OpenSSL Version OpenSSL 1.1.0g 2 Nov 2017
> {code}
> These messages may fiddle with existing scripts which parses logs with a
> predefined schema.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15868) AliyunOSS: update document for properties of multiple part download, multiple part upload and directory copy
[ https://issues.apache.org/jira/browse/HADOOP-15868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676131#comment-16676131 ] Sunil Govindan commented on HADOOP-15868: - Updated Fixed Version to 3.2.1 as the fix has not landed on branch-3.2.0 (which was closed for RC prep during this commit) > AliyunOSS: update document for properties of multiple part download, multiple > part upload and directory copy > > > Key: HADOOP-15868 > URL: https://issues.apache.org/jira/browse/HADOOP-15868 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 2.10.0, 3.2.0, 2.9.2, 3.0.4, 3.1.2 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Fix For: 2.10.0, 2.9.2, 3.0.4, 3.3.0, 3.1.2, 3.2.1 > > Attachments: HADOOP-15868.001.patch, HADOOP-15868.002.patch > > > Update document for properties of multiple part download, multiple part > upload and directory copy. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15868) AliyunOSS: update document for properties of multiple part download, multiple part upload and directory copy
[ https://issues.apache.org/jira/browse/HADOOP-15868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15868: Fix Version/s: (was: 3.2.0) 3.2.1 > AliyunOSS: update document for properties of multiple part download, multiple > part upload and directory copy > > > Key: HADOOP-15868 > URL: https://issues.apache.org/jira/browse/HADOOP-15868 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 2.10.0, 3.2.0, 2.9.2, 3.0.4, 3.1.2 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Fix For: 2.10.0, 2.9.2, 3.0.4, 3.3.0, 3.1.2, 3.2.1 > > Attachments: HADOOP-15868.001.patch, HADOOP-15868.002.patch > > > Update document for properties of multiple part download, multiple part > upload and directory copy. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15146) Remove DataOutputByteBuffer
[
https://issues.apache.org/jira/browse/HADOOP-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676001#comment-16676001
]
Sunil Govindan commented on HADOOP-15146:
-
Thanks [~ste...@apache.org]. Updated same.
> Remove DataOutputByteBuffer
> ---
>
> Key: HADOOP-15146
> URL: https://issues.apache.org/jira/browse/HADOOP-15146
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Fix For: 3.2.0
>
> Attachments: HADOOP-15146.1.patch, HADOOP-15146.2.patch,
> HADOOP-15146.3.patch, HADOOP-15146.4.patch, HADOOP-15146.5.patch
>
>
> I can't seem to find any references to {{DataOutputByteBuffer}} maybe it
> should be deprecated or simply removed?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15146) Remove DataOutputByteBuffer
[
https://issues.apache.org/jira/browse/HADOOP-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil Govindan updated HADOOP-15146:
Release Note: This jira helped to remove the private and unused class
DataOutputByteBuffer.
> Remove DataOutputByteBuffer
> ---
>
> Key: HADOOP-15146
> URL: https://issues.apache.org/jira/browse/HADOOP-15146
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Fix For: 3.2.0
>
> Attachments: HADOOP-15146.1.patch, HADOOP-15146.2.patch,
> HADOOP-15146.3.patch, HADOOP-15146.4.patch, HADOOP-15146.5.patch
>
>
> I can't seem to find any references to {{DataOutputByteBuffer}} maybe it
> should be deprecated or simply removed?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15146) Remove DataOutputByteBuffer
[
https://issues.apache.org/jira/browse/HADOOP-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16675150#comment-16675150
]
Sunil Govindan commented on HADOOP-15146:
-
Hi [~ste...@apache.org]
While preparing RC, I got a warning in the ReleaseNotes.md file mentioning like
below.
{code:java}
* [HADOOP-15146](https://issues.apache.org/jira/browse/HADOOP-15146) | *Minor*
| **Remove DataOutputByteBuffer**
**WARNING: No release note provided for this change.*{code}
> Remove DataOutputByteBuffer
> ---
>
> Key: HADOOP-15146
> URL: https://issues.apache.org/jira/browse/HADOOP-15146
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Fix For: 3.2.0
>
> Attachments: HADOOP-15146.1.patch, HADOOP-15146.2.patch,
> HADOOP-15146.3.patch, HADOOP-15146.4.patch, HADOOP-15146.5.patch
>
>
> I can't seem to find any references to {{DataOutputByteBuffer}} maybe it
> should be deprecated or simply removed?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15146) Remove DataOutputByteBuffer
[
https://issues.apache.org/jira/browse/HADOOP-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674895#comment-16674895
]
Sunil Govindan commented on HADOOP-15146:
-
[~belugabehr] [~ste...@apache.org]
This is an incompatible change. Hence pls help to update Release Notes by
editing this Jira at the earliest. Appreciate the support. Thank You.
> Remove DataOutputByteBuffer
> ---
>
> Key: HADOOP-15146
> URL: https://issues.apache.org/jira/browse/HADOOP-15146
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Fix For: 3.2.0
>
> Attachments: HADOOP-15146.1.patch, HADOOP-15146.2.patch,
> HADOOP-15146.3.patch, HADOOP-15146.4.patch, HADOOP-15146.5.patch
>
>
> I can't seem to find any references to {{DataOutputByteBuffer}} maybe it
> should be deprecated or simply removed?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15815) Upgrade Eclipse Jetty version to 9.3.24
[ https://issues.apache.org/jira/browse/HADOOP-15815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15815: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.1.2 3.3.0 3.0.4 3.2.0 Status: Resolved (was: Patch Available) Thanks [~borisvu]. Thanks [~tasanuma0829] [~bharatviswa]. I committed to all branches which HADOOP-15882 is committed to. trunk/branch-3.2/branch-3.2.0/branch-3.1//branch-3.0 Pls let me know if any issues. Sorry for committing sooner. As 3.2 is closing, i would like clear all blockers soon. Thanks. > Upgrade Eclipse Jetty version to 9.3.24 > --- > > Key: HADOOP-15815 > URL: https://issues.apache.org/jira/browse/HADOOP-15815 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.1.1, 3.0.3 >Reporter: Boris Vulikh >Assignee: Boris Vulikh >Priority: Major > Fix For: 3.2.0, 3.0.4, 3.3.0, 3.1.2 > > Attachments: HADOOP-15815.01-2.patch > > > * > [CVE-2017-7657|https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2017-7657] > * > [CVE-2017-7658|https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2017-7658] > * > [CVE-2017-7656|https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2017-7656] > * > [CVE-2018-12536|https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2018-12536] > We should upgrade the dependency to version 9.3.24 or the latest, if possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15815) Upgrade Eclipse Jetty version to
[ https://issues.apache.org/jira/browse/HADOOP-15815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15815: Summary: Upgrade Eclipse Jetty version to (was: Upgrade Eclipse Jetty version due to security concerns) > Upgrade Eclipse Jetty version to > - > > Key: HADOOP-15815 > URL: https://issues.apache.org/jira/browse/HADOOP-15815 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.1.1, 3.0.3 >Reporter: Boris Vulikh >Assignee: Boris Vulikh >Priority: Major > Attachments: HADOOP-15815.01-2.patch > > > * > [CVE-2017-7657|https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2017-7657] > * > [CVE-2017-7658|https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2017-7658] > * > [CVE-2017-7656|https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2017-7656] > * > [CVE-2018-12536|https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2018-12536] > We should upgrade the dependency to version 9.3.24 or the latest, if possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15815) Upgrade Eclipse Jetty version to 9.3.24
[ https://issues.apache.org/jira/browse/HADOOP-15815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated HADOOP-15815: Summary: Upgrade Eclipse Jetty version to 9.3.24 (was: Upgrade Eclipse Jetty version to ) > Upgrade Eclipse Jetty version to 9.3.24 > --- > > Key: HADOOP-15815 > URL: https://issues.apache.org/jira/browse/HADOOP-15815 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.1.1, 3.0.3 >Reporter: Boris Vulikh >Assignee: Boris Vulikh >Priority: Major > Attachments: HADOOP-15815.01-2.patch > > > * > [CVE-2017-7657|https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2017-7657] > * > [CVE-2017-7658|https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2017-7658] > * > [CVE-2017-7656|https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2017-7656] > * > [CVE-2018-12536|https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2018-12536] > We should upgrade the dependency to version 9.3.24 or the latest, if possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15815) Upgrade Eclipse Jetty version due to security concerns
[ https://issues.apache.org/jira/browse/HADOOP-15815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664607#comment-16664607 ] Sunil Govindan commented on HADOOP-15815: - +1 from me as well. Lets get this in. > Upgrade Eclipse Jetty version due to security concerns > -- > > Key: HADOOP-15815 > URL: https://issues.apache.org/jira/browse/HADOOP-15815 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.1.1, 3.0.3 >Reporter: Boris Vulikh >Assignee: Boris Vulikh >Priority: Major > Attachments: HADOOP-15815.01-2.patch > > > * > [CVE-2017-7657|https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2017-7657] > * > [CVE-2017-7658|https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2017-7658] > * > [CVE-2017-7656|https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2017-7656] > * > [CVE-2018-12536|https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2018-12536] > We should upgrade the dependency to version 9.3.24 or the latest, if possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15882) Upgrade maven-shade-plugin from 2.4.3 to 3.2.0
[ https://issues.apache.org/jira/browse/HADOOP-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664606#comment-16664606 ] Sunil Govindan commented on HADOOP-15882: - Thanks. I pulled to branch-3.2.0 as well. > Upgrade maven-shade-plugin from 2.4.3 to 3.2.0 > -- > > Key: HADOOP-15882 > URL: https://issues.apache.org/jira/browse/HADOOP-15882 > Project: Hadoop Common > Issue Type: Task >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Fix For: 3.2.0, 3.0.4, 3.3.0, 3.1.2 > > Attachments: HADOOP-15882.1.patch > > > While working on HADOOP-15815, we have faced a shaded-client error. Please > see [~bharatviswa]'s comment > [here|https://issues.apache.org/jira/browse/HADOOP-15815?focusedCommentId=16662718=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16662718]. > MSHADE-242 and MSHADE-258 are needed to fix it. Let's upgrade > maven-shade-plugin to 3.1.0 or later. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15882) Upgrade maven-shade-plugin from 2.4.3 to 3.2.0
[ https://issues.apache.org/jira/browse/HADOOP-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663872#comment-16663872 ] Sunil Govindan commented on HADOOP-15882: - cc [~bharatviswa] for review. > Upgrade maven-shade-plugin from 2.4.3 to 3.2.0 > -- > > Key: HADOOP-15882 > URL: https://issues.apache.org/jira/browse/HADOOP-15882 > Project: Hadoop Common > Issue Type: Task >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Attachments: HADOOP-15882.1.patch > > > While working on HADOOP-15815, we have faced a shaded-client error. Please > see [~bharatviswa]'s comment > [here|https://issues.apache.org/jira/browse/HADOOP-15815?focusedCommentId=16662718=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16662718]. > MSHADE-242 and MSHADE-258 are needed to fix it. Let's upgrade > maven-shade-plugin to 3.1.0 or later. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
