[jira] [Commented] (HADOOP-15981) Add mutual TLS support for RPC
[ https://issues.apache.org/jira/browse/HADOOP-15981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16972304#comment-16972304 ] Anu Engineer commented on HADOOP-15981: --- I just wanted to flag that Ozone has a complete Certificate Server and all the associated interface. We can bring it into Hadoop Common if there is some interest. it has an interface based approach, so that if we want to use an external CA, we can like the Key Management server or even have it running independently. > Add mutual TLS support for RPC > -- > > Key: HADOOP-15981 > URL: https://issues.apache.org/jira/browse/HADOOP-15981 > Project: Hadoop Common > Issue Type: Sub-task > Components: ipc, security >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Major > > The RPC server should allow optionally enabling mutual TLS as 1st class > authentication. If enabled, a client cert may provide the user's identity or > fallback to kerberos or token. Essentially the placeholder CERTIFICATE > authentication method will be implemented and offered as an authentication > method during connection negotiation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16692) UserGroupInformation Treats kerberosMinSecondsBeforeRelogin as Millis
[ https://issues.apache.org/jira/browse/HADOOP-16692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969736#comment-16969736 ] Anu Engineer commented on HADOOP-16692: --- Yes, I also realized that this will kill all the existing configs if we check this in. So renaming the variable is a better idea for sure. > UserGroupInformation Treats kerberosMinSecondsBeforeRelogin as Millis > - > > Key: HADOOP-16692 > URL: https://issues.apache.org/jira/browse/HADOOP-16692 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.2 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HADOOP-16692.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16692) UserGroupInformation Treats kerberosMinSecondsBeforeRelogin, as Millis
[ https://issues.apache.org/jira/browse/HADOOP-16692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969628#comment-16969628 ] Anu Engineer commented on HADOOP-16692: --- +1. LGTM. Pending Jenkins. > UserGroupInformation Treats kerberosMinSecondsBeforeRelogin, as Millis > -- > > Key: HADOOP-16692 > URL: https://issues.apache.org/jira/browse/HADOOP-16692 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.2 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HADOOP-16692.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16678) Review of ArrayWritable
[
https://issues.apache.org/jira/browse/HADOOP-16678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HADOOP-16678:
--
Hadoop Flags: Reviewed
Resolution: Fixed
Status: Resolved (was: Patch Available)
[~belugabehr] Thank you for the contribution. I have committed this patch to
the trunk.
> Review of ArrayWritable
> ---
>
> Key: HADOOP-16678
> URL: https://issues.apache.org/jira/browse/HADOOP-16678
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: HADOOP-16678.1.patch, HADOOP-16678.2.patch
>
>
> I originally wanted to include a {{toString()}} method so I could debug an
> issue in another project, but since I opened up, figured I'd put in a few
> more changes for cleanup.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16678) Review of ArrayWritable
[
https://issues.apache.org/jira/browse/HADOOP-16678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16967059#comment-16967059
]
Anu Engineer edited comment on HADOOP-16678 at 11/4/19 10:27 PM:
-
[~belugabehr] Thank you for the contribution. I have committed this patch to
the trunk.
Please note: I committed the patch from the Github version not the patch that
is posted here.
was (Author: anu):
[~belugabehr] Thank you for the contribution. I have committed this patch to
the trunk.
> Review of ArrayWritable
> ---
>
> Key: HADOOP-16678
> URL: https://issues.apache.org/jira/browse/HADOOP-16678
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: HADOOP-16678.1.patch, HADOOP-16678.2.patch
>
>
> I originally wanted to include a {{toString()}} method so I could debug an
> issue in another project, but since I opened up, figured I'd put in a few
> more changes for cleanup.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13363) Upgrade protobuf from 2.5.0 to something newer
[ https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16935990#comment-16935990 ] Anu Engineer edited comment on HADOOP-13363 at 9/23/19 4:00 PM: There [~aw] I took one for the team; updated my name as the reporter :). Feel free to switch it back to your name .. if the email thread is not bad; especially given the fact that this is a fun thread to read :) Feel free to report in about the adventures in the Millennium land. I have quite forgotten all that happened during that time. was (Author: anu): There [~aw] I took one for the team; updated my name as the reporter :). Feel free to switch it back to your name .. if the email thread is not bad; especially given the fact that this is a fun thread to read :) > Upgrade protobuf from 2.5.0 to something newer > -- > > Key: HADOOP-13363 > URL: https://issues.apache.org/jira/browse/HADOOP-13363 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Anu Engineer >Assignee: Vinayakumar B >Priority: Major > Labels: security > Attachments: HADOOP-13363.001.patch, HADOOP-13363.002.patch, > HADOOP-13363.003.patch, HADOOP-13363.004.patch, HADOOP-13363.005.patch > > > Standard protobuf 2.5.0 does not work properly on many platforms. (See, for > example, https://gist.github.com/BennettSmith/7111094 ). In order for us to > avoid crazy work arounds in the build environment and the fact that 2.5.0 is > starting to slowly disappear as a standard install-able package for even > Linux/x86, we need to either upgrade or self bundle or something else. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13363) Upgrade protobuf from 2.5.0 to something newer
[ https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16935990#comment-16935990 ] Anu Engineer commented on HADOOP-13363: --- There [~aw] I took one for the team; updated my name as the reporter :). Feel free to switch it back to your name .. if the email thread is not bad; especially given the fact that this is a fun thread to read :) > Upgrade protobuf from 2.5.0 to something newer > -- > > Key: HADOOP-13363 > URL: https://issues.apache.org/jira/browse/HADOOP-13363 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Anu Engineer >Assignee: Vinayakumar B >Priority: Major > Labels: security > Attachments: HADOOP-13363.001.patch, HADOOP-13363.002.patch, > HADOOP-13363.003.patch, HADOOP-13363.004.patch, HADOOP-13363.005.patch > > > Standard protobuf 2.5.0 does not work properly on many platforms. (See, for > example, https://gist.github.com/BennettSmith/7111094 ). In order for us to > avoid crazy work arounds in the build environment and the fact that 2.5.0 is > starting to slowly disappear as a standard install-able package for even > Linux/x86, we need to either upgrade or self bundle or something else. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13363) Upgrade protobuf from 2.5.0 to something newer
[ https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HADOOP-13363: -- Reporter: Anu Engineer (was: Allen Wittenauer) > Upgrade protobuf from 2.5.0 to something newer > -- > > Key: HADOOP-13363 > URL: https://issues.apache.org/jira/browse/HADOOP-13363 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Anu Engineer >Assignee: Vinayakumar B >Priority: Major > Labels: security > Attachments: HADOOP-13363.001.patch, HADOOP-13363.002.patch, > HADOOP-13363.003.patch, HADOOP-13363.004.patch, HADOOP-13363.005.patch > > > Standard protobuf 2.5.0 does not work properly on many platforms. (See, for > example, https://gist.github.com/BennettSmith/7111094 ). In order for us to > avoid crazy work arounds in the build environment and the fact that 2.5.0 is > starting to slowly disappear as a standard install-able package for even > Linux/x86, we need to either upgrade or self bundle or something else. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16061) Update Apache Yetus to 0.10.0
[
https://issues.apache.org/jira/browse/HADOOP-16061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16933837#comment-16933837
]
Anu Engineer commented on HADOOP-16061:
---
This upgrade also breaks some of the tools. I am not able to run
smart-apply-patch on MacOS. This used to work till the upgrade.
{code:java}
~/a/hadoop> ./dev-support/bin/smart-apply-patch
~/Downloads/review-jiras/1369.patch.1
ERROR: yetus-dl: unable to download
https://archive.apache.org/dist/yetus/0.10.0//apache-yetus-0.10.0-bin.tar.gz
{code}
> Update Apache Yetus to 0.10.0
> -
>
> Key: HADOOP-16061
> URL: https://issues.apache.org/jira/browse/HADOOP-16061
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Fix For: 3.3.0, 3.2.1
>
>
> Yetus 0.10.0 is out. Let's upgrade.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13363) Upgrade protobuf from 2.5.0 to something newer
[ https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16924523#comment-16924523 ] Anu Engineer commented on HADOOP-13363: --- +1, on the protoc maven approach. Ozone also uses it and we have never had a problem in the Jenkins or other build systems (we build and test using Argo/K8s internally). > Upgrade protobuf from 2.5.0 to something newer > -- > > Key: HADOOP-13363 > URL: https://issues.apache.org/jira/browse/HADOOP-13363 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Allen Wittenauer >Assignee: Vinayakumar B >Priority: Major > Labels: security > Attachments: HADOOP-13363.001.patch, HADOOP-13363.002.patch, > HADOOP-13363.003.patch, HADOOP-13363.004.patch, HADOOP-13363.005.patch > > > Standard protobuf 2.5.0 does not work properly on many platforms. (See, for > example, https://gist.github.com/BennettSmith/7111094 ). In order for us to > avoid crazy work arounds in the build environment and the fact that 2.5.0 is > starting to slowly disappear as a standard install-able package for even > Linux/x86, we need to either upgrade or self bundle or something else. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13363) Upgrade protobuf from 2.5.0 to something newer
[ https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16923785#comment-16923785 ] Anu Engineer commented on HADOOP-13363: --- Rebase or incomplete patch? > Upgrade protobuf from 2.5.0 to something newer > -- > > Key: HADOOP-13363 > URL: https://issues.apache.org/jira/browse/HADOOP-13363 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Allen Wittenauer >Assignee: Vinayakumar B >Priority: Major > Labels: security > Attachments: HADOOP-13363.001.patch, HADOOP-13363.002.patch, > HADOOP-13363.003.patch, HADOOP-13363.004.patch, HADOOP-13363.005.patch > > > Standard protobuf 2.5.0 does not work properly on many platforms. (See, for > example, https://gist.github.com/BennettSmith/7111094 ). In order for us to > avoid crazy work arounds in the build environment and the fact that 2.5.0 is > starting to slowly disappear as a standard install-able package for even > Linux/x86, we need to either upgrade or self bundle or something else. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16398) Exports Hadoop metrics to Prometheus
[
https://issues.apache.org/jira/browse/HADOOP-16398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891224#comment-16891224
]
Anu Engineer edited comment on HADOOP-16398 at 7/23/19 5:24 PM:
just wondering if we should do this change instead of the todo comment?
{code:java}
// If hadoop prometheus endpoint is enabled, then HDDS/Ozone
// will reuse the same end point.
if(conf.getBoolean("hadoop.prometheus.endpoint.enabled")) {
conf.setBoolean("hdds.prometheus.endpoint.enabled", false);
}{code}
The assumption here is that baseHTTPServer when used by HDDS and Ozone will
actually be an OzoneConfiguration and it will allow us to set the hdds config
value. If not, in the pure Hadoop case, it will be a no-op, yes; the config
space will have an extra entry but not used by anyone. However, that entry will
be visible in the /config servelet.
was (Author: anu):
just wondering if we should do this change instead of the todo comment?
{code:java}
// If hadoop prometheus endpoint is enabled, then HDDS/Ozone
// will reuse the same end point.
if(conf.getBoolean("hadoop.prometheus.endpoint.enabled")) {
conf.setBoolean("hdds.prometheus.endpoint.enabled", false);
}{code}
> Exports Hadoop metrics to Prometheus
>
>
> Key: HADOOP-16398
> URL: https://issues.apache.org/jira/browse/HADOOP-16398
> Project: Hadoop Common
> Issue Type: New Feature
> Components: metrics
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HADOOP-16398.001.patch, HADOOP-16398.002.patch
>
>
> Hadoop common side of HDDS-846. HDDS already have its own
> PrometheusMetricsSink, so we can reuse the implementation.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16398) Exports Hadoop metrics to Prometheus
[
https://issues.apache.org/jira/browse/HADOOP-16398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891224#comment-16891224
]
Anu Engineer commented on HADOOP-16398:
---
just wondering if we should do this change instead of the todo comment?
{code:java}
// If hadoop prometheus endpoint is enabled, then HDDS/Ozone
// will reuse the same end point.
if(conf.getBoolean("hadoop.prometheus.endpoint.enabled")) {
conf.setBoolean("hdds.prometheus.endpoint.enabled", false);
}{code}
> Exports Hadoop metrics to Prometheus
>
>
> Key: HADOOP-16398
> URL: https://issues.apache.org/jira/browse/HADOOP-16398
> Project: Hadoop Common
> Issue Type: New Feature
> Components: metrics
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HADOOP-16398.001.patch, HADOOP-16398.002.patch
>
>
> Hadoop common side of HDDS-846. HDDS already have its own
> PrometheusMetricsSink, so we can reuse the implementation.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16448) Connection to Hadoop homepage is not secure
[ https://issues.apache.org/jira/browse/HADOOP-16448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891162#comment-16891162 ] Anu Engineer commented on HADOOP-16448: --- I am presuming that the fix is to make the "htttp:" in that page to "https:". Since you are the first person to file this JIRA, would you like to provide a fix? If not, I will assign this to someone else. If you are interested in providing a fix, you can create a pull request against [https://github.com/apache/hadoop-site] or the actual markdown files are located in the documentation directories of Hadoop. [https://github.com/apache/hadoop/tree/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown] Once more, thank you very much for flagging this; if you are not interested in providing a fix; one of us will gladly do so. > Connection to Hadoop homepage is not secure > --- > > Key: HADOOP-16448 > URL: https://issues.apache.org/jira/browse/HADOOP-16448 > Project: Hadoop Common > Issue Type: Bug > Components: website >Affects Versions: 0.23.11, 2.4.0, 2.5.0, 2.4.1, 2.5.1, 2.5.2, 2.6.0, > 2.6.1, 2.7.0, 2.8.0, 2.7.1, 2.7.2, 2.6.2, 2.6.3, 2.7.3, 2.9.0, 2.6.4, 2.6.5, > 2.7.4, 2.8.1, 2.8.2, 2.8.3, 2.7.5, 3.0.0, 3.1.0, 2.9.1, 3.0.1, 2.8.4, 2.7.6, > 3.2.0, 3.0.2, 3.1.1, 2.9.2, 3.0.3, 2.7.7, 2.8.5 >Reporter: Kaspar Tint >Priority: Major > Labels: newbie > Attachments: Screen Shot 2019-07-23 at 9.37.54 AM.png > > > When visiting the [Hadoop > website|https://hadoop.apache.org/docs/r3.2.0/index.html] with the latest > Firefox browser (v 68.0.1) it appears that the website cannot be reached > through secure means by default. > The culprit seems to be the fact that the two header images presented on the > page are loaded in via *HTTP* > !Screen Shot 2019-07-23 at 9.37.54 AM.png!. > These images are located in the respective locations: > http://hadoop.apache.org/images/hadoop-logo.jpg > http://www.apache.org/images/asf_logo_wide.png > These images can be reached also from the following locations: > https://hadoop.apache.org/images/hadoop-logo.jpg > https://www.apache.org/images/asf_logo_wide.png > As one can see, a fix could be made to use a more safe way of including in > the two header pictures to the page. > I feel like I am in danger when reading the Hadoop documentation from the > official Hadoop webpage in a non secure way. Thus I felt the need to open > this ticket and raise the issue in order to have a future where everyone can > learn from Hadoop documentation in a safe and secure way. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16448) Connection to Hadoop homepage is not secure
[ https://issues.apache.org/jira/browse/HADOOP-16448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HADOOP-16448: -- Labels: newbie (was: ) > Connection to Hadoop homepage is not secure > --- > > Key: HADOOP-16448 > URL: https://issues.apache.org/jira/browse/HADOOP-16448 > Project: Hadoop Common > Issue Type: Bug > Components: website >Affects Versions: 0.23.11, 2.4.0, 2.5.0, 2.4.1, 2.5.1, 2.5.2, 2.6.0, > 2.6.1, 2.7.0, 2.8.0, 2.7.1, 2.7.2, 2.6.2, 2.6.3, 2.7.3, 2.9.0, 2.6.4, 2.6.5, > 2.7.4, 2.8.1, 2.8.2, 2.8.3, 2.7.5, 3.0.0, 3.1.0, 2.9.1, 3.0.1, 2.8.4, 2.7.6, > 3.2.0, 3.0.2, 3.1.1, 2.9.2, 3.0.3, 2.7.7, 2.8.5 >Reporter: Kaspar Tint >Priority: Major > Labels: newbie > Attachments: Screen Shot 2019-07-23 at 9.37.54 AM.png > > > When visiting the [Hadoop > website|https://hadoop.apache.org/docs/r3.2.0/index.html] with the latest > Firefox browser (v 68.0.1) it appears that the website cannot be reached > through secure means by default. > The culprit seems to be the fact that the two header images presented on the > page are loaded in via *HTTP* > !Screen Shot 2019-07-23 at 9.37.54 AM.png!. > These images are located in the respective locations: > http://hadoop.apache.org/images/hadoop-logo.jpg > http://www.apache.org/images/asf_logo_wide.png > These images can be reached also from the following locations: > https://hadoop.apache.org/images/hadoop-logo.jpg > https://www.apache.org/images/asf_logo_wide.png > As one can see, a fix could be made to use a more safe way of including in > the two header pictures to the page. > I feel like I am in danger when reading the Hadoop documentation from the > official Hadoop webpage in a non secure way. Thus I felt the need to open > this ticket and raise the issue in order to have a future where everyone can > learn from Hadoop documentation in a safe and secure way. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16452) Increase ipc.maximum.data.length default from 64MB to 128MB
[ https://issues.apache.org/jira/browse/HADOOP-16452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891147#comment-16891147 ] Anu Engineer commented on HADOOP-16452: --- what is the impact on block processing? we take some locks..so these kinds of block reports can be hard on NN? Is it possible to break these messages into multiple messages at least inside NN? > Increase ipc.maximum.data.length default from 64MB to 128MB > --- > > Key: HADOOP-16452 > URL: https://issues.apache.org/jira/browse/HADOOP-16452 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Affects Versions: 2.6.0 >Reporter: Wei-Chiu Chuang >Priority: Major > > Reason for bumping the default: > Denser DataNodes are common. It is not uncommon to find a DataNode with > 7 > million blocks these days. > With such a high number of blocks, the block report message can exceed the > 64mb limit (defined by ipc.maximum.data.length). The block reports are > rejected, causing missing blocks in HDFS. We had to double this configuration > value in order to work around the issue. > We are seeing an increasing number of these cases. I think it's time to > revisit some of these default values as the hardware evolves. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16398) Exports Hadoop metrics to Prometheus
[ https://issues.apache.org/jira/browse/HADOOP-16398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16888552#comment-16888552 ] Anu Engineer commented on HADOOP-16398: --- Thanks for getting this done for Hadoop. +1, LGTM. > Exports Hadoop metrics to Prometheus > > > Key: HADOOP-16398 > URL: https://issues.apache.org/jira/browse/HADOOP-16398 > Project: Hadoop Common > Issue Type: New Feature > Components: metrics >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Attachments: HADOOP-16398.001.patch > > > Hadoop common side of HDDS-846. HDDS already have its own > PrometheusMetricsSink, so we can reuse the implementation. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13363) Upgrade protobuf from 2.5.0 to something newer
[
https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16882209#comment-16882209
]
Anu Engineer edited comment on HADOOP-13363 at 7/10/19 4:04 PM:
{quote}Who watching this JIRA thinks "it is time" to make Hadoop 3.3 a protobuf
upgrade event?
{quote}
Me. The support of 2.5 series in non-existent now. Either we need to fork
protobuf 2.5 and make it part of Hadoop tool chain or move. We cannot forever
depend on 2.5 being around.
was (Author: anu):
{quote}Who watching this JIRA thinks "it is time" to make Hadoop 3.3 a protobuf
upgrade event?
{quote}
Me.
> Upgrade protobuf from 2.5.0 to something newer
> --
>
> Key: HADOOP-13363
> URL: https://issues.apache.org/jira/browse/HADOOP-13363
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2
>Reporter: Allen Wittenauer
>Priority: Major
> Labels: security
> Attachments: HADOOP-13363.001.patch, HADOOP-13363.002.patch,
> HADOOP-13363.003.patch, HADOOP-13363.004.patch, HADOOP-13363.005.patch
>
>
> Standard protobuf 2.5.0 does not work properly on many platforms. (See, for
> example, https://gist.github.com/BennettSmith/7111094 ). In order for us to
> avoid crazy work arounds in the build environment and the fact that 2.5.0 is
> starting to slowly disappear as a standard install-able package for even
> Linux/x86, we need to either upgrade or self bundle or something else.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13363) Upgrade protobuf from 2.5.0 to something newer
[
https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16882209#comment-16882209
]
Anu Engineer commented on HADOOP-13363:
---
{quote}Who watching this JIRA thinks "it is time" to make Hadoop 3.3 a protobuf
upgrade event?
{quote}
Me.
> Upgrade protobuf from 2.5.0 to something newer
> --
>
> Key: HADOOP-13363
> URL: https://issues.apache.org/jira/browse/HADOOP-13363
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2
>Reporter: Allen Wittenauer
>Priority: Major
> Labels: security
> Attachments: HADOOP-13363.001.patch, HADOOP-13363.002.patch,
> HADOOP-13363.003.patch, HADOOP-13363.004.patch, HADOOP-13363.005.patch
>
>
> Standard protobuf 2.5.0 does not work properly on many platforms. (See, for
> example, https://gist.github.com/BennettSmith/7111094 ). In order for us to
> avoid crazy work arounds in the build environment and the fact that 2.5.0 is
> starting to slowly disappear as a standard install-able package for even
> Linux/x86, we need to either upgrade or self bundle or something else.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15566) Support Opentracing
[
https://issues.apache.org/jira/browse/HADOOP-15566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16872581#comment-16872581
]
Anu Engineer commented on HADOOP-15566:
---
{quote}probably will be available in the next couple of months (deadline is 6th
of July)
{quote}
[~bogdandrutu] I looked at the OpenTelemetry.io and was not able to see
anything related to a release. So are we still on track for a 6th of July
release? I am just interested in the Java SDK for time being. Thanks
> Support Opentracing
> ---
>
> Key: HADOOP-15566
> URL: https://issues.apache.org/jira/browse/HADOOP-15566
> Project: Hadoop Common
> Issue Type: New Feature
> Components: metrics, tracing
>Affects Versions: 3.1.0
>Reporter: Todd Lipcon
>Assignee: Siyao Meng
>Priority: Major
> Labels: security
> Attachments: Screen Shot 2018-06-29 at 11.59.16 AM.png,
> ss-trace-s3a.png
>
>
> The HTrace incubator project has voted to retire itself and won't be making
> further releases. The Hadoop project currently has various hooks with HTrace.
> It seems in some cases (eg HDFS-13702) these hooks have had measurable
> performance overhead. Given these two factors, I think we should consider
> removing the HTrace integration. If there is someone willing to do the work,
> replacing it with OpenTracing might be a better choice since there is an
> active community.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16144) Create a Hadoop RPC based KMS client
[ https://issues.apache.org/jira/browse/HADOOP-16144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837473#comment-16837473 ] Anu Engineer commented on HADOOP-16144: --- Thanks for taking over and getting this to completion. Appreciate it. I am not even sure if we need Genesis, It is a habit carried over from Ozone. You have real-world workload, so having a workload or a perf test is far less important in HDFS world. Since I already wrote code, I will file a Jira and move it when your code is in, so I can accommodate any changes you might have made. > Create a Hadoop RPC based KMS client > > > Key: HADOOP-16144 > URL: https://issues.apache.org/jira/browse/HADOOP-16144 > Project: Hadoop Common > Issue Type: Sub-task > Components: kms >Reporter: Wei-Chiu Chuang >Assignee: Anu Engineer >Priority: Major > Attachments: HADOOP-16144.001.patch, KMS.RPC.patch > > > Create a new KMS client implementation that speaks Hadoop RPC. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16144) Create a Hadoop RPC based KMS client
[ https://issues.apache.org/jira/browse/HADOOP-16144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832838#comment-16832838 ] Anu Engineer commented on HADOOP-16144: --- Great, way to go. I wrote genesis stuff to get the benchmarks, it is not important at all. Please post the patch when you are ready. I am traveling for the next few weeks, so my responses might be slow. > Create a Hadoop RPC based KMS client > > > Key: HADOOP-16144 > URL: https://issues.apache.org/jira/browse/HADOOP-16144 > Project: Hadoop Common > Issue Type: Sub-task > Components: kms >Reporter: Wei-Chiu Chuang >Assignee: Anu Engineer >Priority: Major > Attachments: HADOOP-16144.001.patch, KMS.RPC.patch > > > Create a new KMS client implementation that speaks Hadoop RPC. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16026) Replace incorrect use of system property user.name
[ https://issues.apache.org/jira/browse/HADOOP-16026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer resolved HADOOP-16026. --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.3.0 Target Version/s: 3.3.0 [~dineshchitlangia] Thanks for the contribution. [~jojochuang] Thanks for the review. I have committed this patch to trunk. > Replace incorrect use of system property user.name > -- > > Key: HADOOP-16026 > URL: https://issues.apache.org/jira/browse/HADOOP-16026 > Project: Hadoop Common > Issue Type: Improvement > Environment: Kerberized >Reporter: Dinesh Chitlangia >Assignee: Dinesh Chitlangia >Priority: Major > Fix For: 3.3.0 > > > This jira has been created to track the suggested changes for Hadoop Common > as identified in HDFS-14176 > Following occurrence need to be corrected: > Common/FileSystem L2233 > Common/AbstractFileSystem L451 > Common/SshFenceByTcpPort L239 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16026) Replace incorrect use of system property user.name
[ https://issues.apache.org/jira/browse/HADOOP-16026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16823348#comment-16823348 ] Anu Engineer commented on HADOOP-16026: --- [~dineshchitlangia] Thanks for the patch. [~jojochuang] Thanks for the review. I will commit this patch. > Replace incorrect use of system property user.name > -- > > Key: HADOOP-16026 > URL: https://issues.apache.org/jira/browse/HADOOP-16026 > Project: Hadoop Common > Issue Type: Improvement > Environment: Kerberized >Reporter: Dinesh Chitlangia >Assignee: Dinesh Chitlangia >Priority: Major > > This jira has been created to track the suggested changes for Hadoop Common > as identified in HDFS-14176 > Following occurrence need to be corrected: > Common/FileSystem L2233 > Common/AbstractFileSystem L451 > Common/SshFenceByTcpPort L239 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16262) Missing some modules in pom.xml
[
https://issues.apache.org/jira/browse/HADOOP-16262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16820311#comment-16820311
]
Anu Engineer commented on HADOOP-16262:
---
Yes, that might be a good idea. Here is some guidance for Ozone in case you are
adding this info to the BUILDING.txt. Feel free to reopen this Jira in case you
are providing a new patch.
[https://cwiki.apache.org/confluence/display/HADOOP/Downloading+Sources]
Thanks
> Missing some modules in pom.xml
> ---
>
> Key: HADOOP-16262
> URL: https://issues.apache.org/jira/browse/HADOOP-16262
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Wanqiang Ji
>Assignee: Wanqiang Ji
>Priority: Major
> Attachments: HADOOP-16262.001.patch
>
>
> Had three project modules missing in pom.xml. Such as:
> {code:xml}
> hadoop-hdds
> hadoop-ozone
> hadoop-submarine
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16262) Missing some modules in pom.xml
[
https://issues.apache.org/jira/browse/HADOOP-16262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HADOOP-16262:
--
Resolution: Not A Problem
Status: Resolved (was: Patch Available)
> Missing some modules in pom.xml
> ---
>
> Key: HADOOP-16262
> URL: https://issues.apache.org/jira/browse/HADOOP-16262
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Wanqiang Ji
>Assignee: Wanqiang Ji
>Priority: Major
> Attachments: HADOOP-16262.001.patch
>
>
> Had three project modules missing in pom.xml. Such as:
> {code:xml}
> hadoop-hdds
> hadoop-ozone
> hadoop-submarine
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16262) Missing some modules in pom.xml
[
https://issues.apache.org/jira/browse/HADOOP-16262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16820286#comment-16820286
]
Anu Engineer commented on HADOOP-16262:
---
These are optional components that are designed to be build when needed. For
example, to build ozone, you can do something like this.
*mvn -f pom.ozone.xml install -DskipTests -Dmaven.javadoc.skip -Dskipshade*
In short, this is a feature, not a bug.:) Thanks for flagging it though. I will
resolve this as not an issue.
> Missing some modules in pom.xml
> ---
>
> Key: HADOOP-16262
> URL: https://issues.apache.org/jira/browse/HADOOP-16262
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Wanqiang Ji
>Assignee: Wanqiang Ji
>Priority: Major
> Attachments: HADOOP-16262.001.patch
>
>
> Had three project modules missing in pom.xml. Such as:
> {code:xml}
> hadoop-hdds
> hadoop-ozone
> hadoop-submarine
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16144) Create a Hadoop RPC based KMS client
[ https://issues.apache.org/jira/browse/HADOOP-16144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16815992#comment-16815992 ] Anu Engineer commented on HADOOP-16144: --- Thanks appreciate it, I will spend some time cleaning up the checkstyle and find bugs issues too. > Create a Hadoop RPC based KMS client > > > Key: HADOOP-16144 > URL: https://issues.apache.org/jira/browse/HADOOP-16144 > Project: Hadoop Common > Issue Type: Sub-task > Components: kms >Reporter: Wei-Chiu Chuang >Assignee: Anu Engineer >Priority: Major > Attachments: HADOOP-16144.001.patch, KMS.RPC.patch > > > Create a new KMS client implementation that speaks Hadoop RPC. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16240) start-build-env.sh can consume all disk space during image creation
[ https://issues.apache.org/jira/browse/HADOOP-16240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HADOOP-16240: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.3.0 Status: Resolved (was: Patch Available) [~ccondit] Thank you for your contribution. I have committed this patch to the trunk branch. > start-build-env.sh can consume all disk space during image creation > --- > > Key: HADOOP-16240 > URL: https://issues.apache.org/jira/browse/HADOOP-16240 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.3.0, 3.2.1 > Environment: macOS Mojave 10.14.4 >Reporter: Craig Condit >Assignee: Craig Condit >Priority: Minor > Fix For: 3.3.0 > > Attachments: HADOOP-16240.001.patch > > > The start-build-env.sh creates a Docker image and creates a user within it > which maps to the user ID from the host. In the case where the host UID is > very large (> 1 billion or so, not uncommon in large AD deployments), the > resultant image fails to build due to /var/log/lastlog and /var/log/faillog > growing to consume all available disk space. > These files are not necessary for the build process and if they do not exist, > they will not be grown. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16240) start-build-env.sh can consume all disk space during image creation
[ https://issues.apache.org/jira/browse/HADOOP-16240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812837#comment-16812837 ] Anu Engineer commented on HADOOP-16240: --- [~ccondit] I have also added you to the Hadoop-common contributor's group, so you can assign these JIRAs to yourself. > start-build-env.sh can consume all disk space during image creation > --- > > Key: HADOOP-16240 > URL: https://issues.apache.org/jira/browse/HADOOP-16240 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.3.0, 3.2.1 > Environment: macOS Mojave 10.14.4 >Reporter: Craig Condit >Assignee: Craig Condit >Priority: Minor > Attachments: HADOOP-16240.001.patch > > > The start-build-env.sh creates a Docker image and creates a user within it > which maps to the user ID from the host. In the case where the host UID is > very large (> 1 billion or so, not uncommon in large AD deployments), the > resultant image fails to build due to /var/log/lastlog and /var/log/faillog > growing to consume all available disk space. > These files are not necessary for the build process and if they do not exist, > they will not be grown. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16240) start-build-env.sh can consume all disk space during image creation
[ https://issues.apache.org/jira/browse/HADOOP-16240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer reassigned HADOOP-16240: - Assignee: Craig Condit > start-build-env.sh can consume all disk space during image creation > --- > > Key: HADOOP-16240 > URL: https://issues.apache.org/jira/browse/HADOOP-16240 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.3.0, 3.2.1 > Environment: macOS Mojave 10.14.4 >Reporter: Craig Condit >Assignee: Craig Condit >Priority: Minor > Attachments: HADOOP-16240.001.patch > > > The start-build-env.sh creates a Docker image and creates a user within it > which maps to the user ID from the host. In the case where the host UID is > very large (> 1 billion or so, not uncommon in large AD deployments), the > resultant image fails to build due to /var/log/lastlog and /var/log/faillog > growing to consume all available disk space. > These files are not necessary for the build process and if they do not exist, > they will not be grown. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16240) start-build-env.sh can consume all disk space during image creation
[ https://issues.apache.org/jira/browse/HADOOP-16240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812833#comment-16812833 ] Anu Engineer commented on HADOOP-16240: --- +1, I will commit this as soon as we have Jenkins run. Jenkins does not run on MacOS. So I will run this locally before committing, Jenkins run is just to ensure that there are no regressions. Thanks for your contribution. > start-build-env.sh can consume all disk space during image creation > --- > > Key: HADOOP-16240 > URL: https://issues.apache.org/jira/browse/HADOOP-16240 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.3.0, 3.2.1 > Environment: macOS Mojave 10.14.4 >Reporter: Craig Condit >Priority: Minor > Attachments: HADOOP-16240.001.patch > > > The start-build-env.sh creates a Docker image and creates a user within it > which maps to the user ID from the host. In the case where the host UID is > very large (> 1 billion or so, not uncommon in large AD deployments), the > resultant image fails to build due to /var/log/lastlog and /var/log/faillog > growing to consume all available disk space. > These files are not necessary for the build process and if they do not exist, > they will not be grown. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16144) Create a Hadoop RPC based KMS client
[ https://issues.apache.org/jira/browse/HADOOP-16144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16810498#comment-16810498 ] Anu Engineer commented on HADOOP-16144: --- [~daryn], [~jojochuang], [~xyao], [~elek] I have posted the first patch. I can setup a meeting and walk you thru the code if you are interested. I have written some wiki pages under ozone to make to easy for you folks to know how to run and test this tool. This is still a work in progress, that is there are missing commands in the client and benchmarking tool. I thought you folks would like to see where we have reached. There are three pages at the following wiki location, feel free to leave questions/comments here or in the wiki. [https://cwiki.apache.org/confluence/display/HADOOP/Ozone+KMS] > Create a Hadoop RPC based KMS client > > > Key: HADOOP-16144 > URL: https://issues.apache.org/jira/browse/HADOOP-16144 > Project: Hadoop Common > Issue Type: Sub-task > Components: kms >Reporter: Wei-Chiu Chuang >Assignee: Anu Engineer >Priority: Major > Attachments: HADOOP-16144.001.patch, KMS.RPC.patch > > > Create a new KMS client implementation that speaks Hadoop RPC. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16144) Create a Hadoop RPC based KMS client
[ https://issues.apache.org/jira/browse/HADOOP-16144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HADOOP-16144: -- Attachment: HADOOP-16144.001.patch > Create a Hadoop RPC based KMS client > > > Key: HADOOP-16144 > URL: https://issues.apache.org/jira/browse/HADOOP-16144 > Project: Hadoop Common > Issue Type: Sub-task > Components: kms >Reporter: Wei-Chiu Chuang >Assignee: Anu Engineer >Priority: Major > Attachments: HADOOP-16144.001.patch, KMS.RPC.patch > > > Create a new KMS client implementation that speaks Hadoop RPC. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16144) Create a Hadoop RPC based KMS client
[ https://issues.apache.org/jira/browse/HADOOP-16144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HADOOP-16144: -- Status: Patch Available (was: Open) > Create a Hadoop RPC based KMS client > > > Key: HADOOP-16144 > URL: https://issues.apache.org/jira/browse/HADOOP-16144 > Project: Hadoop Common > Issue Type: Sub-task > Components: kms >Reporter: Wei-Chiu Chuang >Assignee: Anu Engineer >Priority: Major > Attachments: HADOOP-16144.001.patch, KMS.RPC.patch > > > Create a new KMS client implementation that speaks Hadoop RPC. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16144) Create a Hadoop RPC based KMS client
[ https://issues.apache.org/jira/browse/HADOOP-16144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807112#comment-16807112 ] Anu Engineer commented on HADOOP-16144: --- Hi All, I am hoping that we have consensus in this direction, and I will post a real patch soon. As usual ["Silence means consent"|https://en.wiktionary.org/wiki/qui_tacet_consentire_videtur] so this is a really good time to speak up if there are any concerns. Thanks > Create a Hadoop RPC based KMS client > > > Key: HADOOP-16144 > URL: https://issues.apache.org/jira/browse/HADOOP-16144 > Project: Hadoop Common > Issue Type: Sub-task > Components: kms >Reporter: Wei-Chiu Chuang >Assignee: Anu Engineer >Priority: Major > Attachments: KMS.RPC.patch > > > Create a new KMS client implementation that speaks Hadoop RPC. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16144) Create a Hadoop RPC based KMS client
[ https://issues.apache.org/jira/browse/HADOOP-16144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16804667#comment-16804667 ] Anu Engineer commented on HADOOP-16144: --- Hi [~daryn], This is a just the first draft of the KMSv2 based on Hadoop RPC. This is also my first ever patch on KMS, please holler if I have done something colossally stupid. I really appreciate the chance to write a KMS from scratch, Thanks. Here overview and patterns (and rationale) used in this patch. 1. KeyManagementProtos.proto - Contains all our proto definitions. It follows a pattern that we have used extensively in Ozone. That has one RPC call (submitRequest) and a union of commands for both request and responses. This is very useful when we want to add tracing, audit logging, and profiling, etc. Single entry and exit point in the server makes it easy for the rest of the code. Other than this protobuf file, all files other files are in _??/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/v2??_ 2. Under the v2 directory, you will find the *protocol directory*. The classes under it follow the standard Hadoop RPC pattern. The client and Server side stubs. 2. The server code is in *v2/server* -- I have broken up each function to xxxOps.java file, so that is where the core logic is. For example, deleteKey logic will be in DeleteKeyOps.java. 3. KMSv2Server.java -- all calls follow a standard pattern; they are decodeRequest, call the main function and then call makeResponse. 4. The Server Dispatch functions in KMSV2ProtocolServerSideTranslatorPB decode the request and get us to type safety. This is where we will add the Audit functionality. 5. What is missing -- 1. Tests -- I have just started working on it. 2. CLI Client -- I will get to that soon. 3. Genesis -- A Microbench marking tool 4. Kerberos -- I have commented out the Kerberos code for now. 5. ACL check -- this is kind of strange -- when I got the first calls to createkey working, the providers were checking and enforcing Key ACLS. Looks like in the current KMS code path we might be doing ACLs check across different places. We might want to clean that up. I will get to that once this core code is done. 6. At some point, I would like to introduce Ratis based provider, so Ozone can run independently of ZooKeeper. That is long term. 7. KMS audit, it looks like a small change on this patch, I will probably add that before the final patch. +Again, Work under progress+, just posting a patch so that you are not left wondering what the hell is this guy doing. I will get to tests tomorrow and post a better patch over the weekend. Please think of this patch as a proposal and I am very open to re-architecture based on new ideas, so feel free to comment including future looking ideas. Since we are rewriting this from scratch, we can accommodate any future looking ideas. Thanks Anu [~jojochuang], [~xyao], [~elek] FYI. > Create a Hadoop RPC based KMS client > > > Key: HADOOP-16144 > URL: https://issues.apache.org/jira/browse/HADOOP-16144 > Project: Hadoop Common > Issue Type: Sub-task > Components: kms >Reporter: Wei-Chiu Chuang >Assignee: Anu Engineer >Priority: Major > Attachments: KMS.RPC.patch > > > Create a new KMS client implementation that speaks Hadoop RPC. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16144) Create a Hadoop RPC based KMS client
[ https://issues.apache.org/jira/browse/HADOOP-16144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HADOOP-16144: -- Attachment: KMS.RPC.patch > Create a Hadoop RPC based KMS client > > > Key: HADOOP-16144 > URL: https://issues.apache.org/jira/browse/HADOOP-16144 > Project: Hadoop Common > Issue Type: Sub-task > Components: kms >Reporter: Wei-Chiu Chuang >Assignee: Anu Engineer >Priority: Major > Attachments: KMS.RPC.patch > > > Create a new KMS client implementation that speaks Hadoop RPC. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16215) Genconfig does not generate LOG4j configs
Anu Engineer created HADOOP-16215: - Summary: Genconfig does not generate LOG4j configs Key: HADOOP-16215 URL: https://issues.apache.org/jira/browse/HADOOP-16215 Project: Hadoop Common Issue Type: Task Affects Versions: 0.3.0 Reporter: Hrishikesh Gadre Assignee: Hrishikesh Gadre Genconfig does not generate Log4J configs, This is needed for Ozone configs to work correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16144) Create a Hadoop RPC based KMS client
[ https://issues.apache.org/jira/browse/HADOOP-16144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16803195#comment-16803195 ] Anu Engineer edited comment on HADOOP-16144 at 3/27/19 6:36 PM: I have a KMS server with Hadoop RPC working on my machine. It is not complete, the client side and tools is what I am working on now. We are on track. Sorry, I got pulled into some Ozone-4.0 release too work too. But I will have this done by end of this week. At least something that you will be able to take and start using. was (Author: anu): I have a KMS server with Hadoop RPC working on my machine. It is not complete, the client side and tools is what I am working on now. We are on track. Sorry, I got pulled into some Ozone-4.0 release too work too. But I have this done by end of this week. At least something that you will be able to take and start using. > Create a Hadoop RPC based KMS client > > > Key: HADOOP-16144 > URL: https://issues.apache.org/jira/browse/HADOOP-16144 > Project: Hadoop Common > Issue Type: Sub-task > Components: kms >Reporter: Wei-Chiu Chuang >Assignee: Anu Engineer >Priority: Major > > Create a new KMS client implementation that speaks Hadoop RPC. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16144) Create a Hadoop RPC based KMS client
[ https://issues.apache.org/jira/browse/HADOOP-16144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16803195#comment-16803195 ] Anu Engineer commented on HADOOP-16144: --- I have a KMS server with Hadoop RPC working on my machine. It is not complete, the client side and tools is what I am working on now. We are on track. Sorry, I got pulled into some Ozone-4.0 release too work too. But I have this done by end of this week. At least something that you will be able to take and start using. > Create a Hadoop RPC based KMS client > > > Key: HADOOP-16144 > URL: https://issues.apache.org/jira/browse/HADOOP-16144 > Project: Hadoop Common > Issue Type: Sub-task > Components: kms >Reporter: Wei-Chiu Chuang >Assignee: Anu Engineer >Priority: Major > > Create a new KMS client implementation that speaks Hadoop RPC. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16144) Create a Hadoop RPC based KMS client
[ https://issues.apache.org/jira/browse/HADOOP-16144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16798328#comment-16798328 ] Anu Engineer commented on HADOOP-16144: --- [~jojochuang] Thanks for letting us know. I will be glad to take care of this. > Create a Hadoop RPC based KMS client > > > Key: HADOOP-16144 > URL: https://issues.apache.org/jira/browse/HADOOP-16144 > Project: Hadoop Common > Issue Type: Sub-task > Components: kms >Reporter: Wei-Chiu Chuang >Assignee: Anu Engineer >Priority: Major > > Create a new KMS client implementation that speaks Hadoop RPC. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16144) Create a Hadoop RPC based KMS client
[ https://issues.apache.org/jira/browse/HADOOP-16144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16797572#comment-16797572 ] Anu Engineer commented on HADOOP-16144: --- [~daryn] I thought someone like [~jojochuang] was working on this. If that is not the case, I would love to work on this. Ozone has TDE now, So this is really important for us too. I will post a patch for the discussion purpose and if [~jojochuang] or someone else is working on this, I will step aside. I would rather that you keep the momentum and not blocked on these minor details. > Create a Hadoop RPC based KMS client > > > Key: HADOOP-16144 > URL: https://issues.apache.org/jira/browse/HADOOP-16144 > Project: Hadoop Common > Issue Type: Sub-task > Components: kms >Reporter: Wei-Chiu Chuang >Priority: Major > > Create a new KMS client implementation that speaks Hadoop RPC. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16144) Create a Hadoop RPC based KMS client
[ https://issues.apache.org/jira/browse/HADOOP-16144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer reassigned HADOOP-16144: - Assignee: Anu Engineer > Create a Hadoop RPC based KMS client > > > Key: HADOOP-16144 > URL: https://issues.apache.org/jira/browse/HADOOP-16144 > Project: Hadoop Common > Issue Type: Sub-task > Components: kms >Reporter: Wei-Chiu Chuang >Assignee: Anu Engineer >Priority: Major > > Create a new KMS client implementation that speaks Hadoop RPC. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[
https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16769711#comment-16769711
]
Anu Engineer commented on HADOOP-11223:
---
[~milleruntime] Sorry to join the party so late, just want to understand how
Accumulo uses this feature, as this is still not read-only config, But surely
benefits you someway. I would like to understand the use case a little better.
Thanks
> Offer a read-only conf alternative to new Configuration()
> -
>
> Key: HADOOP-11223
> URL: https://issues.apache.org/jira/browse/HADOOP-11223
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
>Reporter: Gopal V
>Assignee: Michael Miller
>Priority: Major
> Labels: Performance
> Attachments: HADOOP-11223.001.patch, HADOOP-11223.002.patch,
> HADOOP-11223.003.patch
>
>
> new Configuration() is called from several static blocks across Hadoop.
> This is incredibly inefficient, since each one of those involves primarily
> XML parsing at a point where the JIT won't be triggered & interpreter mode is
> essentially forced on the JVM.
> The alternate solution would be to offer a {{Configuration::getDefault()}}
> alternative which disallows any modifications.
> At the very least, such a method would need to be called from
> # org.apache.hadoop.io.nativeio.NativeIO::()
> # org.apache.hadoop.security.SecurityUtil::()
> # org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider::
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16113) Your project apache/hadoop is using buggy third-party libraries [WARNING]
[ https://issues.apache.org/jira/browse/HADOOP-16113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16769496#comment-16769496 ] Anu Engineer commented on HADOOP-16113: --- bq. Ozone team to upgrade log4j 2 and then tell us how we went, Will do; > Your project apache/hadoop is using buggy third-party libraries [WARNING] > - > > Key: HADOOP-16113 > URL: https://issues.apache.org/jira/browse/HADOOP-16113 > Project: Hadoop Common > Issue Type: Bug >Reporter: Kaifeng Huang >Priority: Major > > Hi, there! > We are a research team working on third-party library analysis. We have > found that some widely-used third-party libraries in your project have > major/critical bugs, which will degrade the quality of your project. We > highly recommend you to update those libraries to new versions. > We have attached the buggy third-party libraries and corresponding jira > issue links below for you to have more detailed information. > 1. org.apache.logging.log4j log4j-core(hadoop-hdds/common/pom.xml) > version: 2.11.0 > Jira issues: > Log4j2 throws NoClassDefFoundError in Java 9 > affectsVersions:2.10.0,2.11.0 > > https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2129?filter=allopenissues > Empty Automatic-Module-Name Header > affectsVersions:2.10.0,2.11.0,3.0.0 > > https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2254?filter=allopenissues > gc-free mixed async loging loses parameter values after the first > appender > affectsVersions:2.11.0 > > https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2301?filter=allopenissues > Log4j 2.10+not working with SLF4J 1.8 in OSGI environment > affectsVersions:2.10.0,2.11.0 > > https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2305?filter=allopenissues > AsyncQueueFullMessageUtil causes unparsable message output > affectsVersions:2.11.0 > > https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2318?filter=allopenissues > AbstractLogger NPE hides actual cause when getFormat returns null > affectsVersions:2.11.0 > > https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2320?filter=allopenissues > AsyncLogger without specifying a level always uses ERROR > affectsVersions:2.11.0 > > https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2321?filter=allopenissues > Errors thrown in formatting may stop background threads > affectsVersions:2.11.0 > > https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2333?filter=allopenissues > JsonLayout not working with AsyncLoggerContextSelector in 2.11.0 > affectsVersions:2.11.0 > > https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2341?filter=allopenissues > Typo in log4j-api Activator > affectsVersions:2.11.0 > > https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2343?filter=allopenissues > PropertiesUtil.reload() might throw NullPointerException > affectsVersions:2.11.0 > > https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2355?filter=allopenissues > NameAbbreviator skips first fragments > affectsVersions:2.11.0,2.11.1 > > https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2365?filter=allopenissues > Outputs wrong message when used within overridden Throwable method > affectsVersions:2.8.1,2.11.0 > > https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2368?filter=allopenissues > StringBuilder escapeJson performs unnecessary Memory Allocations > affectsVersions:2.11.0 > > https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2373?filter=allopenissues > fix the CacheEntry map in ThrowableProxy#toExtendedStackTrace to be put > and gotten with same key > affectsVersions:2.6.2,2.7,2.8,2.8.1,2.8.2,2.9.0,2.9.1,2.10.0,2.11.0 > > https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2389?filter=allopenissues > Fix incorrect links in Log4j web documentation. > affectsVersions:2.11.0 > > https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2390?filter=allopenissues > 2. org.apache.httpcomponents httpclient(hadoop-project/pom.xml) > version: 4.5.2 > Jira issues: > > org.apache.http.impl.client.AbstractHttpClient#createClientConnectionManager > Does not account for context class loader > affectsVersions:4.4.1;4.5;4.5.1;4.5.2 > > https://issues.apache.org/jira/projects/HTTPCLIENT/issues/HTTPCLIENT-1727?filter=allopenissues > Memory Leak in OSGi support > affectsVersions:4.4.1;4.5.2 > > https://issues.ap
[jira] [Commented] (HADOOP-16091) Create hadoop/ozone docker images with inline build process
[
https://issues.apache.org/jira/browse/HADOOP-16091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16761033#comment-16761033
]
Anu Engineer commented on HADOOP-16091:
---
bq. Note: based on LEGAL-270 and linked discussion both approaches (inline
build process / external build process) are compatible with the apache release.
If legal is saying both of these approaches are part of Apache way, then we are
okay in the current state. Of course, anyone is welcome to improve the state
of Hadoop as they deem fit.
> Create hadoop/ozone docker images with inline build process
> ---
>
> Key: HADOOP-16091
> URL: https://issues.apache.org/jira/browse/HADOOP-16091
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Elek, Marton
>Priority: Major
>
> This is proposed by [~eyang] in
> [this|https://lists.apache.org/thread.html/33ac54bdeacb4beb023ebd452464603aaffa095bd104cb43c22f484e@%3Chdfs-dev.hadoop.apache.org%3E]
> mailing thread.
> bq. 1, 3. There are 38 Apache projects hosting docker images on Docker hub
> using Apache Organization. By browsing Apache github mirror. There are only
> 7 projects using a separate repository for docker image build. Popular
> projects official images are not from Apache organization, such as zookeeper,
> tomcat, httpd. We may not disrupt what other Apache projects are doing, but
> it looks like inline build process is widely employed by majority of projects
> such as Nifi, Brooklyn, thrift, karaf, syncope and others. The situation
> seems a bit chaotic for Apache as a whole. However, Hadoop community can
> decide what is best for Hadoop. My preference is to remove ozone from source
> tree naming, if Ozone is intended to be subproject of Hadoop for long period
> of time. This enables Hadoop community to host docker images for various
> subproject without having to check out several source tree to trigger a grand
> build. However, inline build process seems more popular than separated
> process. Hence, I highly recommend making docker build inline if possible.
> The main challenges are also discussed in the thread:
> {code}
> 3. Technically it would be possible to add the Dockerfile to the source
> tree and publish the docker image together with the release by the
> release manager but it's also problematic:
> {code}
> a) there is no easy way to stage the images for the vote
> c) it couldn't be flagged as automated on dockerhub
> d) It couldn't support the critical updates.
> * Updating existing images (for example in case of an ssl bug, rebuild
> all the existing images with exactly the same payload but updated base
> image/os environment)
> * Creating image for older releases (We would like to provide images,
> for hadoop 2.6/2.7/2.7/2.8/2.9. Especially for doing automatic testing
> with different versions).
> {code}
> The a) can be solved (as [~eyang] suggested) with using a personal docker
> image during the vote and publish it to the dockerhub after the vote (in case
> the permission can be set by the INFRA)
> Note: based on LEGAL-270 and linked discussion both approaches (inline build
> process / external build process) are compatible with the apache release.
> Note: HDDS-851 and HADOOP-14898 contains more information about these
> problems.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15990) S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2
[
https://issues.apache.org/jira/browse/HADOOP-15990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16742750#comment-16742750
]
Anu Engineer commented on HADOOP-15990:
---
{quote}[~Jitendra] [~anu] : what's the ozone S3 gateway going to do when it
sees a list v2 request?
{quote}
We support only V2 list in the S3 Gateway, not v1. cc: [~elek], [~bharatviswa]
> S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2
>
>
> Key: HADOOP-15990
> URL: https://issues.apache.org/jira/browse/HADOOP-15990
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: lqjacklee
>Priority: Major
> Attachments: HADOOP-15409-005.patch, HADOOP-15990-006.patch
>
>
> in S3AFileSystem.initialize(), we check for the bucket existing with
> verifyBucketExists(), which calls s3.doesBucketExist(). But that doesn't
> check for auth issues.
> s3. doesBucketExistV2() does at least validate credentials, and should be
> switched to. This will help things fail faster
> See SPARK-24000
> (this is a dupe of HADOOP-15409; moving off git PRs so we can get yetus to
> test everything)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16012) Typo in daemonlog docs
[ https://issues.apache.org/jira/browse/HADOOP-16012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16722392#comment-16722392 ] Anu Engineer commented on HADOOP-16012: --- [~brahmareddy] Thanks for the info. resolved. > Typo in daemonlog docs > -- > > Key: HADOOP-16012 > URL: https://issues.apache.org/jira/browse/HADOOP-16012 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Reporter: Anu Engineer >Priority: Major > Labels: newbie > > From the user mailing thread: -- From tzq > I might find a spelling error in > "http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CommandsManual.html"; > hadoop daemonlog -getlevel [-protocol (http|https)] > *hdoop* daemonlog -setlevel[-protocol > (http|https)] > The second hadoop has a typo. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16012) Typo in daemonlog docs
[ https://issues.apache.org/jira/browse/HADOOP-16012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer resolved HADOOP-16012. --- Resolution: Fixed > Typo in daemonlog docs > -- > > Key: HADOOP-16012 > URL: https://issues.apache.org/jira/browse/HADOOP-16012 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Reporter: Anu Engineer >Priority: Major > Labels: newbie > > From the user mailing thread: -- From tzq > I might find a spelling error in > "http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CommandsManual.html"; > hadoop daemonlog -getlevel [-protocol (http|https)] > *hdoop* daemonlog -setlevel[-protocol > (http|https)] > The second hadoop has a typo. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16012) Typo in daemonlog docs
Anu Engineer created HADOOP-16012: - Summary: Typo in daemonlog docs Key: HADOOP-16012 URL: https://issues.apache.org/jira/browse/HADOOP-16012 Project: Hadoop Common Issue Type: Bug Components: documentation Reporter: Anu Engineer >From the user mailing thread: -- From tzq I might find a spelling error in "http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CommandsManual.html"; hadoop daemonlog -getlevel [-protocol (http|https)] *hdoop* daemonlog -setlevel[-protocol (http|https)] The second hadoop has a typo. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15932) Oozie unable to create sharelib in s3a filesystem
[
https://issues.apache.org/jira/browse/HADOOP-15932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700859#comment-16700859
]
Anu Engineer commented on HADOOP-15932:
---
+1, Thanks for taking care of this.
> Oozie unable to create sharelib in s3a filesystem
> -
>
> Key: HADOOP-15932
> URL: https://issues.apache.org/jira/browse/HADOOP-15932
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/s3
>Affects Versions: 3.0.0
>Reporter: Soumitra Sulav
>Assignee: Steve Loughran
>Priority: Critical
> Attachments: HADOOP-15932-001.patch
>
>
> Oozie server unable to start cause of below exception.
> s3a expects a file to copy it in store but sharelib is a folder containing
> all the needed components jars.
> Hence throws the exception :
> _Not a file: /usr/hdp/current/oozie-server/share/lib_
> {code:java}
> [oozie@sg-hdp1 ~]$ /usr/hdp/current/oozie-server/bin/oozie-setup.sh sharelib
> create -fs s3a://hdp -locallib /usr/hdp/current/oozie-server/share
> setting OOZIE_CONFIG=${OOZIE_CONFIG:-/usr/hdp/current/oozie-client/conf}
> setting
> CATALINA_BASE=${CATALINA_BASE:-/usr/hdp/current/oozie-client/oozie-server}
> setting CATALINA_TMPDIR=${CATALINA_TMPDIR:-/var/tmp/oozie}
> setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat
> setting JAVA_HOME=/usr/jdk64/jdk1.8.0_112
> setting JRE_HOME=${JAVA_HOME}
> setting CATALINA_OPTS="$CATALINA_OPTS -Xmx2048m"
> setting OOZIE_LOG=/var/log/oozie
> setting CATALINA_PID=/var/run/oozie/oozie.pid
> setting OOZIE_DATA=/hadoop/oozie/data
> setting OOZIE_HTTP_PORT=11000
> setting OOZIE_ADMIN_PORT=11001
> setting
> JAVA_LIBRARY_PATH=/usr/hdp/3.0.0.0-1634/hadoop/lib/native/Linux-amd64-64
> setting OOZIE_CLIENT_OPTS="${OOZIE_CLIENT_OPTS}
> -Doozie.connection.retry.count=5 "
> setting OOZIE_CONFIG=${OOZIE_CONFIG:-/usr/hdp/current/oozie-client/conf}
> setting
> CATALINA_BASE=${CATALINA_BASE:-/usr/hdp/current/oozie-client/oozie-server}
> setting CATALINA_TMPDIR=${CATALINA_TMPDIR:-/var/tmp/oozie}
> setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat
> setting JAVA_HOME=/usr/jdk64/jdk1.8.0_112
> setting JRE_HOME=${JAVA_HOME}
> setting CATALINA_OPTS="$CATALINA_OPTS -Xmx2048m"
> setting OOZIE_LOG=/var/log/oozie
> setting CATALINA_PID=/var/run/oozie/oozie.pid
> setting OOZIE_DATA=/hadoop/oozie/data
> setting OOZIE_HTTP_PORT=11000
> setting OOZIE_ADMIN_PORT=11001
> setting
> JAVA_LIBRARY_PATH=/usr/hdp/3.0.0.0-1634/hadoop/lib/native/Linux-amd64-64
> setting OOZIE_CLIENT_OPTS="${OOZIE_CLIENT_OPTS}
> -Doozie.connection.retry.count=5 "
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.0.0.0-1634/oozie/lib/slf4j-simple-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.0.0.0-1634/oozie/libserver/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.0.0.0-1634/oozie/libserver/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.SimpleLoggerFactory]
> 518 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 605 [main] INFO org.apache.hadoop.conf.Configuration.deprecation -
> mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
> 619 [main] INFO org.apache.hadoop.security.SecurityUtil - Updating
> Configuration
> the destination path for sharelib is: /user/oozie/share/lib/lib_20181114154552
> log4j:WARN No appenders could be found for logger
> (org.apache.htrace.core.Tracer).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> 1118 [main] WARN org.apache.hadoop.metrics2.impl.MetricsConfig - Cannot
> locate configuration: tried
> hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
> 1172 [main] INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl -
> Scheduled Metric snapshot period at 10 second(s).
> 1172 [main] INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl -
> s3a-file-system metrics system started
> 2255 [main] INFO org.apache.hadoop.conf.Configuration.deprecation -
> fs.s3a.server-side-encryption-key is deprecated. Instead, use
> fs.s3a.server-side-encryption.key
> Error: Not a file: /usr/hdp/current/oozie-server/share/lib
> Stack trace for the error was (for debug purposes):
> --
> java.io.FileNotFoundException: Not a file:
> /usr/hdp/current/oozie-server/share/lib
>
[jira] [Commented] (HADOOP-15904) [JDK 11] Javadoc build failed due to "bad use of '>'" in hadoop-hdfs
[
https://issues.apache.org/jira/browse/HADOOP-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16675994#comment-16675994
]
Anu Engineer commented on HADOOP-15904:
---
+1, Thanks for fixing it.
> [JDK 11] Javadoc build failed due to "bad use of '>'" in hadoop-hdfs
>
>
> Key: HADOOP-15904
> URL: https://issues.apache.org/jira/browse/HADOOP-15904
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Attachments: HADOOP-15904.1.patch
>
>
> {noformat}
> $ mvn javadoc:javadoc -pl hadoop-hdfs-project/hadoop-hdfs
> ...
> [ERROR]
> /hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/planner/package-info.java:32:
> error: bad use of '>'
> [ERROR] * // Creates a plan , like move 20 GB data from v1 -> v2
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15339) Support additional key/value propereties in JMX bean registration
[ https://issues.apache.org/jira/browse/HADOOP-15339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667500#comment-16667500 ] Anu Engineer commented on HADOOP-15339: --- +1, on this change. thanks > Support additional key/value propereties in JMX bean registration > - > > Key: HADOOP-15339 > URL: https://issues.apache.org/jira/browse/HADOOP-15339 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15339-branch-3.1.004.patch, > HADOOP-15339.001.patch, HADOOP-15339.002.patch, HADOOP-15339.003.patch > > > org.apache.hadoop.metrics2.util.MBeans.register is a utility function to > register objects to the JMX registry with a given name prefix and name. > JMX supports any additional key value pairs which could be part the the > address of the jmx bean. For example: > _java.lang:type=MemoryManager,name=CodeCacheManager_ > Using this method we can query a group of mbeans, for example we can add the > same tag to similar mbeans from namenode and datanode. > This patch adds a small modification to support custom key value pairs and > also introduce a new unit test for MBeans utility which was missing until now. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15840) Slow RPC logger too spammy
[
https://issues.apache.org/jira/browse/HADOOP-15840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645296#comment-16645296
]
Anu Engineer commented on HADOOP-15840:
---
good to know, may the microsecond precision approach will yield better results.
But 4th dev. is easier to do and test :)
> Slow RPC logger too spammy
> --
>
> Key: HADOOP-15840
> URL: https://issues.apache.org/jira/browse/HADOOP-15840
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.8.0, 2.7.4, 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Priority: Minor
>
> HADOOP-12325 added a capability where "slow" RPCs are logged in NN log when
> ipc.server.log.slow.rpc is enabled.
> The "slow" RPCs are supposed to be those whose processing time is outside 3
> standard deviation, and it supposed to account for 0.3% of total RPCs.
> However, I found in practice, NN marks more than 1% of total RPCs as slow,
> and I've seen RPCs whose processing time is 1ms declared slow too.
> {noformat}
> 2018-10-08 01:48:33,203 WARN org.apache.hadoop.ipc.Server: Slow RPC :
> sendHeartbeat took 1 milliseconds to process from client 10.17.199.16:56645
> 2018-10-08 01:48:33,219 WARN org.apache.hadoop.ipc.Server: Slow RPC :
> sendHeartbeat took 1 milliseconds to process from client 10.17.190.44:36435
> 2018-10-08 01:48:33,308 WARN org.apache.hadoop.ipc.Server: Slow RPC :
> sendHeartbeat took 1 milliseconds to process from client 10.17.190.43:56530
> {noformat}
> This is too many. 1% means NN spits hundreds slow RPCs per second on average
> in NN log.
> How about:
> # use 4 stddev?
> # use microsecond precision. Majority of RPCs takes less than 1 millisecond
> anyway. It makes the stddev calculation imprecise. An RPC could be calculated
> to spend 1 millisecond and be marked as "slow", since it starts from one
> millisecond and ends in the next millisecond.
>
> [~anu] any thoughts?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15840) Slow RPC logger too spammy
[
https://issues.apache.org/jira/browse/HADOOP-15840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645296#comment-16645296
]
Anu Engineer edited comment on HADOOP-15840 at 10/10/18 5:20 PM:
-
good to know, may be the microsecond precision approach will yield better
results. But 4th dev. is easier to do and test :)
was (Author: anu):
good to know, may the microsecond precision approach will yield better results.
But 4th dev. is easier to do and test :)
> Slow RPC logger too spammy
> --
>
> Key: HADOOP-15840
> URL: https://issues.apache.org/jira/browse/HADOOP-15840
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.8.0, 2.7.4, 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Priority: Minor
>
> HADOOP-12325 added a capability where "slow" RPCs are logged in NN log when
> ipc.server.log.slow.rpc is enabled.
> The "slow" RPCs are supposed to be those whose processing time is outside 3
> standard deviation, and it supposed to account for 0.3% of total RPCs.
> However, I found in practice, NN marks more than 1% of total RPCs as slow,
> and I've seen RPCs whose processing time is 1ms declared slow too.
> {noformat}
> 2018-10-08 01:48:33,203 WARN org.apache.hadoop.ipc.Server: Slow RPC :
> sendHeartbeat took 1 milliseconds to process from client 10.17.199.16:56645
> 2018-10-08 01:48:33,219 WARN org.apache.hadoop.ipc.Server: Slow RPC :
> sendHeartbeat took 1 milliseconds to process from client 10.17.190.44:36435
> 2018-10-08 01:48:33,308 WARN org.apache.hadoop.ipc.Server: Slow RPC :
> sendHeartbeat took 1 milliseconds to process from client 10.17.190.43:56530
> {noformat}
> This is too many. 1% means NN spits hundreds slow RPCs per second on average
> in NN log.
> How about:
> # use 4 stddev?
> # use microsecond precision. Majority of RPCs takes less than 1 millisecond
> anyway. It makes the stddev calculation imprecise. An RPC could be calculated
> to spend 1 millisecond and be marked as "slow", since it starts from one
> millisecond and ends in the next millisecond.
>
> [~anu] any thoughts?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15840) Slow RPC logger too spammy
[
https://issues.apache.org/jira/browse/HADOOP-15840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645276#comment-16645276
]
Anu Engineer edited comment on HADOOP-15840 at 10/10/18 5:13 PM:
-
[~jojochuang] Thanks for tagging me. I am plus one either of these approaches.
We might have to test and figure out what is a better approach.
Since I don't have the full context, it might also be that we are looking at a
very small samples
in Server.java#logSlowRpcCalls – final int minSampleSize=1024; if you seeing
this right at the start of the system – that could be an issue too.
was (Author: anu):
[~jojochuang] Thanks for tagging me. I am plus one either of these approaches.
We might have to test and figure out what is a better approach.
> Slow RPC logger too spammy
> --
>
> Key: HADOOP-15840
> URL: https://issues.apache.org/jira/browse/HADOOP-15840
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.8.0, 2.7.4, 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Priority: Minor
>
> HADOOP-12325 added a capability where "slow" RPCs are logged in NN log when
> ipc.server.log.slow.rpc is enabled.
> The "slow" RPCs are supposed to be those whose processing time is outside 3
> standard deviation, and it supposed to account for 0.3% of total RPCs.
> However, I found in practice, NN marks more than 1% of total RPCs as slow,
> and I've seen RPCs whose processing time is 1ms declared slow too.
> {noformat}
> 2018-10-08 01:48:33,203 WARN org.apache.hadoop.ipc.Server: Slow RPC :
> sendHeartbeat took 1 milliseconds to process from client 10.17.199.16:56645
> 2018-10-08 01:48:33,219 WARN org.apache.hadoop.ipc.Server: Slow RPC :
> sendHeartbeat took 1 milliseconds to process from client 10.17.190.44:36435
> 2018-10-08 01:48:33,308 WARN org.apache.hadoop.ipc.Server: Slow RPC :
> sendHeartbeat took 1 milliseconds to process from client 10.17.190.43:56530
> {noformat}
> This is too many. 1% means NN spits hundreds slow RPCs per second on average
> in NN log.
> How about:
> # use 4 stddev?
> # use microsecond precision. Majority of RPCs takes less than 1 millisecond
> anyway. It makes the stddev calculation imprecise. An RPC could be calculated
> to spend 1 millisecond and be marked as "slow", since it starts from one
> millisecond and ends in the next millisecond.
>
> [~anu] any thoughts?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15840) Slow RPC logger too spammy
[
https://issues.apache.org/jira/browse/HADOOP-15840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645276#comment-16645276
]
Anu Engineer commented on HADOOP-15840:
---
[~jojochuang] Thanks for tagging me. I am plus one either of these approaches.
We might have to test and figure out what is a better approach.
> Slow RPC logger too spammy
> --
>
> Key: HADOOP-15840
> URL: https://issues.apache.org/jira/browse/HADOOP-15840
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.8.0, 2.7.4, 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Priority: Minor
>
> HADOOP-12325 added a capability where "slow" RPCs are logged in NN log when
> ipc.server.log.slow.rpc is enabled.
> The "slow" RPCs are supposed to be those whose processing time is outside 3
> standard deviation, and it supposed to account for 0.3% of total RPCs.
> However, I found in practice, NN marks more than 1% of total RPCs as slow,
> and I've seen RPCs whose processing time is 1ms declared slow too.
> {noformat}
> 2018-10-08 01:48:33,203 WARN org.apache.hadoop.ipc.Server: Slow RPC :
> sendHeartbeat took 1 milliseconds to process from client 10.17.199.16:56645
> 2018-10-08 01:48:33,219 WARN org.apache.hadoop.ipc.Server: Slow RPC :
> sendHeartbeat took 1 milliseconds to process from client 10.17.190.44:36435
> 2018-10-08 01:48:33,308 WARN org.apache.hadoop.ipc.Server: Slow RPC :
> sendHeartbeat took 1 milliseconds to process from client 10.17.190.43:56530
> {noformat}
> This is too many. 1% means NN spits hundreds slow RPCs per second on average
> in NN log.
> How about:
> # use 4 stddev?
> # use microsecond precision. Majority of RPCs takes less than 1 millisecond
> anyway. It makes the stddev calculation imprecise. An RPC could be calculated
> to spend 1 millisecond and be marked as "slow", since it starts from one
> millisecond and ends in the next millisecond.
>
> [~anu] any thoughts?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15805) Hadoop logo not showed correctly in old site
[
https://issues.apache.org/jira/browse/HADOOP-15805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16634392#comment-16634392
]
Anu Engineer commented on HADOOP-15805:
---
[~Sandeep Nemuri] Added you to the Contributors group. You should be able
upload patches now.
> Hadoop logo not showed correctly in old site
>
>
> Key: HADOOP-15805
> URL: https://issues.apache.org/jira/browse/HADOOP-15805
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.1.1
>Reporter: Yiqun Lin
>Priority: Major
> Attachments: Error-page.jpg
>
>
> Hadoop logo not showed correctly in old site. In old site pages, we use the
> address {{[http://hadoop.apache.org/images/hadoop-logo.jpg]}} to show the
> hadoop logo. Actually, this address is outdated. The right address now is
> [http://hadoop.apache.org/hadoop-logo.jpg].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15805) Hadoop logo not showed correctly in old site
[
https://issues.apache.org/jira/browse/HADOOP-15805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer reassigned HADOOP-15805:
-
Assignee: Sandeep Nemuri
> Hadoop logo not showed correctly in old site
>
>
> Key: HADOOP-15805
> URL: https://issues.apache.org/jira/browse/HADOOP-15805
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.1.1
>Reporter: Yiqun Lin
>Assignee: Sandeep Nemuri
>Priority: Major
> Attachments: Error-page.jpg
>
>
> Hadoop logo not showed correctly in old site. In old site pages, we use the
> address {{[http://hadoop.apache.org/images/hadoop-logo.jpg]}} to show the
> hadoop logo. Actually, this address is outdated. The right address now is
> [http://hadoop.apache.org/hadoop-logo.jpg].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HADOOP-15805) Hadoop logo not showed correctly in old site
[
https://issues.apache.org/jira/browse/HADOOP-15805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HADOOP-15805:
--
Comment: was deleted
(was: This is a hadoop Jira [~linyiqun] surely has the permissions in this
JIRA. How can I help?)
> Hadoop logo not showed correctly in old site
>
>
> Key: HADOOP-15805
> URL: https://issues.apache.org/jira/browse/HADOOP-15805
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.1.1
>Reporter: Yiqun Lin
>Priority: Major
> Attachments: Error-page.jpg
>
>
> Hadoop logo not showed correctly in old site. In old site pages, we use the
> address {{[http://hadoop.apache.org/images/hadoop-logo.jpg]}} to show the
> hadoop logo. Actually, this address is outdated. The right address now is
> [http://hadoop.apache.org/hadoop-logo.jpg].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15805) Hadoop logo not showed correctly in old site
[
https://issues.apache.org/jira/browse/HADOOP-15805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16634390#comment-16634390
]
Anu Engineer commented on HADOOP-15805:
---
This is a hadoop Jira [~linyiqun] surely has the permissions in this JIRA. How
can I help?
> Hadoop logo not showed correctly in old site
>
>
> Key: HADOOP-15805
> URL: https://issues.apache.org/jira/browse/HADOOP-15805
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.1.1
>Reporter: Yiqun Lin
>Priority: Major
> Attachments: Error-page.jpg
>
>
> Hadoop logo not showed correctly in old site. In old site pages, we use the
> address {{[http://hadoop.apache.org/images/hadoop-logo.jpg]}} to show the
> hadoop logo. Actually, this address is outdated. The right address now is
> [http://hadoop.apache.org/hadoop-logo.jpg].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15774) Discovery of HA servers
[
https://issues.apache.org/jira/browse/HADOOP-15774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16624003#comment-16624003
]
Anu Engineer commented on HADOOP-15774:
---
{quote}[~anu] any thoughts on the general approach? I'm not very familiar with
the OZone architecture; do you guys also need to specify servers manually?
{quote}
[~elgoiri] Thanks for posting the design docs. My apologies for the delayed
response, been busy with Ozone Alpha release.
I have proposed the same idea a couple of times internally. Each time it got
shot down because of the issue that [~ste...@apache.org] is referring to.
{quote}Being part of the YARN may bring some weird dependencies (HDFS->YARN),
any ideas here? Should it be move to a separate place? Maybe parts?
{quote}
The most important concern was always HDFS taking a dependency on YARN. It
creates a cyclical dependency.
Ozone needs and plans to build something very similar to this, but our thought
has been to create something like this in SCM.
For Hadoop and also from Ozone perspective, The consideration for such a
service should be slightly different. It is possible for someone to run a
cluster without HDFS or even without YARN. A Discovery service should be
independent of these specific services.
If we are planning to make this a Hadoop level discovery service we should
probably build this independent of all other services. If possible, even
independent of ZooKeeper.
Then this service can be the bootup service, and all other services including
HDFS, YARN and even Zookeeper can use this service to discover endpoints. Many
services need to find the address of zookeeper too.
If this service can be built independent of all other services, Ozone can
certainly use this, and we don't need to reinvent discovery.
Another critical issue that Ozone struggles with this the issue of config
changes – for example, in the HA case if a server fails and a new server is
added Datanodes need to be told of that.
Ozone solves it by reading these changes from SCM, not only config via
Heartbeat.
But this problem is more generic, in the sense that there is a class of
configuration changes that need to be pushed to all entities in the cluster. We
have been thinking about building a discovery and config store for Ozone.
> Discovery of HA servers
> ---
>
> Key: HADOOP-15774
> URL: https://issues.apache.org/jira/browse/HADOOP-15774
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Íñigo Goiri
>Priority: Major
> Attachments: Discovery Service.pdf
>
>
> Currently, Hadoop relies on configuration files to specify the servers.
> This requires maintaining these configuration files and propagating the
> changes.
> Hadoop should have a framework to provide discovery.
> For example, in HDFS, we could define the Namenodes in a shared location and
> the DNs would use the framework to find the Namenodes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15774) Discovery of HA servers
[ https://issues.apache.org/jira/browse/HADOOP-15774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16621037#comment-16621037 ] Anu Engineer commented on HADOOP-15774: --- This seems like quite a big feature. Two requests. One have a proposal and design doc. Two Please open a branch if you planning to write code for this. So we don't get into situations like HDFS-13312. > Discovery of HA servers > --- > > Key: HADOOP-15774 > URL: https://issues.apache.org/jira/browse/HADOOP-15774 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Íñigo Goiri >Priority: Major > > Currently, Hadoop relies on configuration files to specify the servers. > This requires maintaining these configuration files and propagating the > changes. > Hadoop should have a framework to provide discovery. > For example, in HDFS, we could define the Namenodes in a shared location and > the DNs would use the framework to find the Namenodes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12760) sun.misc.Cleaner has moved to a new location in OpenJDK 9
[ https://issues.apache.org/jira/browse/HADOOP-12760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16619400#comment-16619400 ] Anu Engineer commented on HADOOP-12760: --- [~ajisakaa],[~tasanuma0829], @all Awesome, thanks for all the hard work. Just wanted to reach out and say that I just build and tested Ozone with Java 9 and it works like a charm. Once more, Hats off for such amazing work. > sun.misc.Cleaner has moved to a new location in OpenJDK 9 > - > > Key: HADOOP-12760 > URL: https://issues.apache.org/jira/browse/HADOOP-12760 > Project: Hadoop Common > Issue Type: Bug >Reporter: Chris Hegarty >Assignee: Akira Ajisaka >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-12760.00.patch, HADOOP-12760.01.patch, > HADOOP-12760.02.patch, HADOOP-12760.03.patch, HADOOP-12760.04.patch, > HADOOP-12760.05.patch, HADOOP-12760.06.patch, HADOOP-12760.07.patch, > HADOOP-12760.08.patch > > > This is a heads-up: there are upcoming changes in JDK 9 that will require, at > least, a small update to org.apache.hadoop.crypto.CryptoStreamUtils & > org.apache.hadoop.io.nativeio.NativeIO. > OpenJDK issue no. 8148117: "Move sun.misc.Cleaner to jdk.internal.ref" [1], > will move the Cleaner class from sun.misc to jdk.internal.ref. There is > ongoing discussion about the possibility of providing a public supported API, > maybe in the JDK 9 timeframe, for releasing NIO direct buffer native memory, > see the core-libs-dev mail thread [2]. At the very least CryptoStreamUtils & > NativeIO [3] should be updated to have knowledge of the new location of the > JDK Cleaner. > [1] https://bugs.openjdk.java.net/browse/JDK-8148117 > [2] > http://mail.openjdk.java.net/pipermail/core-libs-dev/2016-January/038243.html > [3] https://github.com/apache/hadoop/search?utf8=✓&q=sun.misc.Cleaner -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15656) Support byteman in hadoop-runner baseimage
[ https://issues.apache.org/jira/browse/HADOOP-15656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589383#comment-16589383 ] Anu Engineer edited comment on HADOOP-15656 at 8/22/18 9:59 PM: +1, Please feel free to commit. I tried to apply this patch but probably needs a rebase. As I said, +1. was (Author: anu): +1, Please feel free to commit. > Support byteman in hadoop-runner baseimage > -- > > Key: HADOOP-15656 > URL: https://issues.apache.org/jira/browse/HADOOP-15656 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-15656-docker-hadoop-runner.001.patch > > > [Byteman|http://byteman.jboss.org/] is an easy to use tool to instrument a > java process with agent string. > For example [this > script|https://gist.githubusercontent.com/elek/0589a91b4d55afb228279f6c4f04a525/raw/8bb4e03de7397c8a9d9bb74a5ec80028b42575c4/hadoop.btm] > defines a rule to print out all the hadoop rpc traffic to the standard > output (which is extremely useful for testing development). > This patch adds the byteman.jar to the baseimage and defines a simple logic > to add agent instrumentation string to the HADOOP_OPTS (optional it also > could download the byteman script from an url) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15656) Support byteman in hadoop-runner baseimage
[ https://issues.apache.org/jira/browse/HADOOP-15656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589383#comment-16589383 ] Anu Engineer commented on HADOOP-15656: --- +1, Please feel free to commit. > Support byteman in hadoop-runner baseimage > -- > > Key: HADOOP-15656 > URL: https://issues.apache.org/jira/browse/HADOOP-15656 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-15656-docker-hadoop-runner.001.patch > > > [Byteman|http://byteman.jboss.org/] is an easy to use tool to instrument a > java process with agent string. > For example [this > script|https://gist.githubusercontent.com/elek/0589a91b4d55afb228279f6c4f04a525/raw/8bb4e03de7397c8a9d9bb74a5ec80028b42575c4/hadoop.btm] > defines a rule to print out all the hadoop rpc traffic to the standard > output (which is extremely useful for testing development). > This patch adds the byteman.jar to the baseimage and defines a simple logic > to add agent instrumentation string to the HADOOP_OPTS (optional it also > could download the byteman script from an url) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15552) Move logging APIs over to slf4j in hadoop-tools - Part2
[ https://issues.apache.org/jira/browse/HADOOP-15552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16581425#comment-16581425 ] Anu Engineer commented on HADOOP-15552: --- [~giovanni.fumarola] Thank you, just committed that patch. We are good now. > Move logging APIs over to slf4j in hadoop-tools - Part2 > --- > > Key: HADOOP-15552 > URL: https://issues.apache.org/jira/browse/HADOOP-15552 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Ian Pickering >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15552.addendum.patch, HADOOP-15552.v1.patch, > HADOOP-15552.v2.patch, HADOOP-15552.v3.patch, HADOOP-15552.v4.patch, > HADOOP-15552.v5.patch, HADOOP-15552.v6.patch, HADOOP-15552.v7.patch, > HADOOP-15552.v8.patch > > > Some classes in Hadoop-tools were not moved to slf4j > e.g. AliyunOSSInputStream.java, HadoopArchiveLogs.java, > HadoopArchiveLogsRunner.java -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15552) Move logging APIs over to slf4j in hadoop-tools - Part2
[ https://issues.apache.org/jira/browse/HADOOP-15552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16581418#comment-16581418 ] Anu Engineer commented on HADOOP-15552: --- Just attached a patch that fixes this issue, since this HDDS/Ozone is broken can someone please give me a +1, so I can commit this soon? > Move logging APIs over to slf4j in hadoop-tools - Part2 > --- > > Key: HADOOP-15552 > URL: https://issues.apache.org/jira/browse/HADOOP-15552 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Ian Pickering >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15552.addendum.patch, HADOOP-15552.v1.patch, > HADOOP-15552.v2.patch, HADOOP-15552.v3.patch, HADOOP-15552.v4.patch, > HADOOP-15552.v5.patch, HADOOP-15552.v6.patch, HADOOP-15552.v7.patch, > HADOOP-15552.v8.patch > > > Some classes in Hadoop-tools were not moved to slf4j > e.g. AliyunOSSInputStream.java, HadoopArchiveLogs.java, > HadoopArchiveLogsRunner.java -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15552) Move logging APIs over to slf4j in hadoop-tools - Part2
[ https://issues.apache.org/jira/browse/HADOOP-15552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HADOOP-15552: -- Attachment: HADOOP-15552.addendum.patch > Move logging APIs over to slf4j in hadoop-tools - Part2 > --- > > Key: HADOOP-15552 > URL: https://issues.apache.org/jira/browse/HADOOP-15552 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Ian Pickering >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15552.addendum.patch, HADOOP-15552.v1.patch, > HADOOP-15552.v2.patch, HADOOP-15552.v3.patch, HADOOP-15552.v4.patch, > HADOOP-15552.v5.patch, HADOOP-15552.v6.patch, HADOOP-15552.v7.patch, > HADOOP-15552.v8.patch > > > Some classes in Hadoop-tools were not moved to slf4j > e.g. AliyunOSSInputStream.java, HadoopArchiveLogs.java, > HadoopArchiveLogsRunner.java -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15552) Move logging APIs over to slf4j in hadoop-tools - Part2
[
https://issues.apache.org/jira/browse/HADOOP-15552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16581407#comment-16581407
]
Anu Engineer commented on HADOOP-15552:
---
This patch breaks the HDDS/Ozone. The renaming of
*AbstractFSContractTestBase#getLog* to *AbstractFSContractTestBase#getLogger*
is the root cause. I can post a patch that fixes the issue or revert this patch
for now.
{noformat}
mvn clean install -T 1C -Phdds -DskipTests=true -Dmaven.javadoc.skip=true
-DskipShade
{noformat}
Here is a command to build and test the issue.
Please let me know how you would like to proceed and I can fix this either way.
> Move logging APIs over to slf4j in hadoop-tools - Part2
> ---
>
> Key: HADOOP-15552
> URL: https://issues.apache.org/jira/browse/HADOOP-15552
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Giovanni Matteo Fumarola
>Assignee: Ian Pickering
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HADOOP-15552.v1.patch, HADOOP-15552.v2.patch,
> HADOOP-15552.v3.patch, HADOOP-15552.v4.patch, HADOOP-15552.v5.patch,
> HADOOP-15552.v6.patch, HADOOP-15552.v7.patch, HADOOP-15552.v8.patch
>
>
> Some classes in Hadoop-tools were not moved to slf4j
> e.g. AliyunOSSInputStream.java, HadoopArchiveLogs.java,
> HadoopArchiveLogsRunner.java
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15551) Avoid use of Java8 streams in Configuration.addTags
[
https://issues.apache.org/jira/browse/HADOOP-15551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519917#comment-16519917
]
Anu Engineer commented on HADOOP-15551:
---
[~tlipcon] Thank you very much for catching this and fixing this. This came via
Ozone.
> Avoid use of Java8 streams in Configuration.addTags
> ---
>
> Key: HADOOP-15551
> URL: https://issues.apache.org/jira/browse/HADOOP-15551
> Project: Hadoop Common
> Issue Type: Improvement
> Components: performance
>Affects Versions: 3.2
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: hadoop-15551.txt
>
>
> Configuration.addTags oddly uses Arrays.stream instead of a more conventional
> mechanism. When profiling a simple program that uses Configuration, I found
> that addTags was taking tens of millis of CPU to do very little work the
> first time it's called, accounting for ~8% of total profiler samples in my
> program.
> {code}
> [9] 4.52% 253 self: 0.00% 0 java/lang/invoke/MethodHandleNatives.linkCallSite
> [9] 3.71% 208 self: 0.00% 0
> java/lang/invoke/MethodHandleNatives.linkMethodHandleConstant
> {code}
> I don't know much about the implementation details of the Streams stuff, but
> it seems it's probably meant more for cases with very large arrays or
> somesuch. Switching to a normal Set.addAll() call eliminates this from the
> profile.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14898) Create official Docker images for development and testing features
[
https://issues.apache.org/jira/browse/HADOOP-14898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HADOOP-14898:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.1.0
Status: Resolved (was: Patch Available)
This is not a feature that is released directly. Hence ignore the release
version tag. [~elek] Thanks for the contribution.
> Create official Docker images for development and testing features
> ---
>
> Key: HADOOP-14898
> URL: https://issues.apache.org/jira/browse/HADOOP-14898
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HADOOP-14898.001.tar.gz, HADOOP-14898.002.tar.gz,
> HADOOP-14898.003.tgz, docker_design.pdf
>
>
> This is the original mail from the mailing list:
> {code}
> TL;DR: I propose to create official hadoop images and upload them to the
> dockerhub.
> GOAL/SCOPE: I would like improve the existing documentation with easy-to-use
> docker based recipes to start hadoop clusters with various configuration.
> The images also could be used to test experimental features. For example
> ozone could be tested easily with these compose file and configuration:
> https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6
> Or even the configuration could be included in the compose file:
> https://github.com/elek/hadoop/blob/docker-2.8.0/example/docker-compose.yaml
> I would like to create separated example compose files for federation, ha,
> metrics usage, etc. to make it easier to try out and understand the features.
> CONTEXT: There is an existing Jira
> https://issues.apache.org/jira/browse/HADOOP-13397
> But it’s about a tool to generate production quality docker images (multiple
> types, in a flexible way). If no objections, I will create a separated issue
> to create simplified docker images for rapid prototyping and investigating
> new features. And register the branch to the dockerhub to create the images
> automatically.
> MY BACKGROUND: I am working with docker based hadoop/spark clusters quite a
> while and run them succesfully in different environments (kubernetes,
> docker-swarm, nomad-based scheduling, etc.) My work is available from here:
> https://github.com/flokkr but they could handle more complex use cases (eg.
> instrumenting java processes with btrace, or read/reload configuration from
> consul).
> And IMHO in the official hadoop documentation it’s better to suggest to use
> official apache docker images and not external ones (which could be changed).
> {code}
> The next list will enumerate the key decision points regarding to docker
> image creating
> A. automated dockerhub build / jenkins build
> Docker images could be built on the dockerhub (a branch pattern should be
> defined for a github repository and the location of the Docker files) or
> could be built on a CI server and pushed.
> The second one is more flexible (it's more easy to create matrix build, for
> example)
> The first one had the advantage that we can get an additional flag on the
> dockerhub that the build is automated (and built from the source by the
> dockerhub).
> The decision is easy as ASF supports the first approach: (see
> https://issues.apache.org/jira/browse/INFRA-12781?focusedCommentId=15824096&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15824096)
> B. source: binary distribution or source build
> The second question is about creating the docker image. One option is to
> build the software on the fly during the creation of the docker image the
> other one is to use the binary releases.
> I suggest to use the second approach as:
> 1. In that case the hadoop:2.7.3 could contain exactly the same hadoop
> distrubution as the downloadable one
> 2. We don't need to add development tools to the image, the image could be
> more smaller (which is important as the goal for this image to getting
> started as fast as possible)
> 3. The docker definition will be more simple (and more easy to maintain)
> Usually this approach is used in other projects (I checked Apache Zeppelin
> and Apache Nutch)
> C. branch usage
> Other question is the location of the Docker file. It could be on the
> official source-code branches (branch-2, trunk, etc.) or we can create
> separated branches for the dockerhub (eg. docker/2.7 docker/2.8 docker/3.0)
> For the first approach it's easier to find the docker images, but it's less
> flexible. For example if we had a Dockerfile for on the source code it should
> be used for every release (for example the Docker file from the tag
> release-3.0.0 should be used for the 3.0 hadoop docker image). In that case
> the release
[jira] [Commented] (HADOOP-15443) hadoop shell should allow non-privileged user to start secure daemons.
[ https://issues.apache.org/jira/browse/HADOOP-15443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461731#comment-16461731 ] Anu Engineer commented on HADOOP-15443: --- why? lots of actions like setting up the secure DN path would need the privilage, why change this? I am trying to understand what problem it is solving. Thanks Anu > hadoop shell should allow non-privileged user to start secure daemons. > -- > > Key: HADOOP-15443 > URL: https://issues.apache.org/jira/browse/HADOOP-15443 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > > With [HDFS-13081] now secure Datanode can be started without root privileges > if rpc port is protected via sasl and ssl is enabled for http. However hadoop > shell still has check for privilged user in hadoop-functions.sh. Jira intends > to amend it, at-least for hdfs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15367) Update the initialization code in the docker hadoop-runner baseimage
[ https://issues.apache.org/jira/browse/HADOOP-15367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HADOOP-15367: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.2.0 Status: Resolved (was: Patch Available) [~elek] I have committed this change to the feature branch. Thanks for the contribution. > Update the initialization code in the docker hadoop-runner baseimage > - > > Key: HADOOP-15367 > URL: https://issues.apache.org/jira/browse/HADOOP-15367 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15367-hadoop-docker-runner.001.patch, > HADOOP-15367-hadoop-docker-runner.002.patch > > > The hadoop-runner baseimage contains initialization code for both the HDFS > namenode/datanode and Ozone/Hdds scm/ksm. > The script name for Ozone/Hdds is changed (from oz to ozone) therefore we > need to updated the base image. > This commit also would be a test for the dockerhub automated build. > Please apply the patch on the top of the _docker-hadoop-runner_ branch. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15367) Update the initialization code in the docker hadoop-runner baseimage
[ https://issues.apache.org/jira/browse/HADOOP-15367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16427607#comment-16427607 ] Anu Engineer commented on HADOOP-15367: --- [~elek] Thanks for the contribution. I will commit this now. I have reviewed the change and we don't have any tests for this branch. > Update the initialization code in the docker hadoop-runner baseimage > - > > Key: HADOOP-15367 > URL: https://issues.apache.org/jira/browse/HADOOP-15367 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-15367-hadoop-docker-runner.001.patch, > HADOOP-15367-hadoop-docker-runner.002.patch > > > The hadoop-runner baseimage contains initialization code for both the HDFS > namenode/datanode and Ozone/Hdds scm/ksm. > The script name for Ozone/Hdds is changed (from oz to ozone) therefore we > need to updated the base image. > This commit also would be a test for the dockerhub automated build. > Please apply the patch on the top of the _docker-hadoop-runner_ branch. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15340) Fix the RPC server name usage to provide information about the metrics
[ https://issues.apache.org/jira/browse/HADOOP-15340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16427535#comment-16427535 ] Anu Engineer commented on HADOOP-15340: --- [~elek] Thanks for the update, +1, pending Jenkins. > Fix the RPC server name usage to provide information about the metrics > -- > > Key: HADOOP-15340 > URL: https://issues.apache.org/jira/browse/HADOOP-15340 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.2.0 >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-15340.001.patch, HADOOP-15340.002.patch > > > In case of multiple RPC servers in the same JVM it's hard to identify the > metric data. The only available information as of now is the port number. > Server name is also added in the constructor of Server.java but it's not used > at all. > This patch fix this behaviour: > 1. The server name is saved to a field in Server.java (constructor signature > is not changed) > 2. ServerName is added as a tag to the metrics in RpcMetrics > 3. The naming convention for the severs are fix. > About 3: if the server name is not defined the current code tries to identify > the name from the class name. Which is not always an easy task as in some > cases the server has a protobuf generated dirty name which also could be an > inner class. > The patch also improved the detection of the name (if it's not defined). It's > a compatible change as the current name is not user ad all. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15295) Remove redundant logging related to tags from Configuration
[
https://issues.apache.org/jira/browse/HADOOP-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HADOOP-15295:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.2.0
Status: Resolved (was: Patch Available)
[~ajayydv] Thank you for the contribution. I have committed this to the trunk.
> Remove redundant logging related to tags from Configuration
> ---
>
> Key: HADOOP-15295
> URL: https://issues.apache.org/jira/browse/HADOOP-15295
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HADOOP-15295.000.patch, HADOOP-15295.001.patch,
> HADOOP-15295.002.patch
>
>
> Remove redundant logging related to tags from Configuration.
> {code}
> 2018-03-06 18:55:46,164 INFO conf.Configuration: Removed undeclared tags:
> 2018-03-06 18:55:46,237 INFO conf.Configuration: Removed undeclared tags:
> 2018-03-06 18:55:46,249 INFO conf.Configuration: Removed undeclared tags:
> 2018-03-06 18:55:46,256 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14898) Create official Docker images for development and testing features
[
https://issues.apache.org/jira/browse/HADOOP-14898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16388801#comment-16388801
]
Anu Engineer commented on HADOOP-14898:
---
[~chris.douglas] Will do, [~elek] is filing a set of INFRA jiras to link these
branches to Apache DockerHub Accounts. After that I will commit the rest of the
patches(three of them) and will update the documentation too.
> Create official Docker images for development and testing features
> ---
>
> Key: HADOOP-14898
> URL: https://issues.apache.org/jira/browse/HADOOP-14898
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HADOOP-14898.001.tar.gz, HADOOP-14898.002.tar.gz,
> HADOOP-14898.003.tgz, docker_design.pdf
>
>
> This is the original mail from the mailing list:
> {code}
> TL;DR: I propose to create official hadoop images and upload them to the
> dockerhub.
> GOAL/SCOPE: I would like improve the existing documentation with easy-to-use
> docker based recipes to start hadoop clusters with various configuration.
> The images also could be used to test experimental features. For example
> ozone could be tested easily with these compose file and configuration:
> https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6
> Or even the configuration could be included in the compose file:
> https://github.com/elek/hadoop/blob/docker-2.8.0/example/docker-compose.yaml
> I would like to create separated example compose files for federation, ha,
> metrics usage, etc. to make it easier to try out and understand the features.
> CONTEXT: There is an existing Jira
> https://issues.apache.org/jira/browse/HADOOP-13397
> But it’s about a tool to generate production quality docker images (multiple
> types, in a flexible way). If no objections, I will create a separated issue
> to create simplified docker images for rapid prototyping and investigating
> new features. And register the branch to the dockerhub to create the images
> automatically.
> MY BACKGROUND: I am working with docker based hadoop/spark clusters quite a
> while and run them succesfully in different environments (kubernetes,
> docker-swarm, nomad-based scheduling, etc.) My work is available from here:
> https://github.com/flokkr but they could handle more complex use cases (eg.
> instrumenting java processes with btrace, or read/reload configuration from
> consul).
> And IMHO in the official hadoop documentation it’s better to suggest to use
> official apache docker images and not external ones (which could be changed).
> {code}
> The next list will enumerate the key decision points regarding to docker
> image creating
> A. automated dockerhub build / jenkins build
> Docker images could be built on the dockerhub (a branch pattern should be
> defined for a github repository and the location of the Docker files) or
> could be built on a CI server and pushed.
> The second one is more flexible (it's more easy to create matrix build, for
> example)
> The first one had the advantage that we can get an additional flag on the
> dockerhub that the build is automated (and built from the source by the
> dockerhub).
> The decision is easy as ASF supports the first approach: (see
> https://issues.apache.org/jira/browse/INFRA-12781?focusedCommentId=15824096&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15824096)
> B. source: binary distribution or source build
> The second question is about creating the docker image. One option is to
> build the software on the fly during the creation of the docker image the
> other one is to use the binary releases.
> I suggest to use the second approach as:
> 1. In that case the hadoop:2.7.3 could contain exactly the same hadoop
> distrubution as the downloadable one
> 2. We don't need to add development tools to the image, the image could be
> more smaller (which is important as the goal for this image to getting
> started as fast as possible)
> 3. The docker definition will be more simple (and more easy to maintain)
> Usually this approach is used in other projects (I checked Apache Zeppelin
> and Apache Nutch)
> C. branch usage
> Other question is the location of the Docker file. It could be on the
> official source-code branches (branch-2, trunk, etc.) or we can create
> separated branches for the dockerhub (eg. docker/2.7 docker/2.8 docker/3.0)
> For the first approach it's easier to find the docker images, but it's less
> flexible. For example if we had a Dockerfile for on the source code it should
> be used for every release (for example the Docker file from the tag
> release-3.0.0 should be used for the 3.0 hadoop docker image). In that case
> the release proce
[jira] [Commented] (HADOOP-14898) Create official Docker images for development and testing features
[
https://issues.apache.org/jira/browse/HADOOP-14898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16388747#comment-16388747
]
Anu Engineer commented on HADOOP-14898:
---
I have created three branches.
* docker-hadoop-runner
* docker-hadoop-2
* docker-hadoop-3
Thanks all, Please let me know if you everything looks ok.
> Create official Docker images for development and testing features
> ---
>
> Key: HADOOP-14898
> URL: https://issues.apache.org/jira/browse/HADOOP-14898
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HADOOP-14898.001.tar.gz, HADOOP-14898.002.tar.gz,
> HADOOP-14898.003.tgz, docker_design.pdf
>
>
> This is the original mail from the mailing list:
> {code}
> TL;DR: I propose to create official hadoop images and upload them to the
> dockerhub.
> GOAL/SCOPE: I would like improve the existing documentation with easy-to-use
> docker based recipes to start hadoop clusters with various configuration.
> The images also could be used to test experimental features. For example
> ozone could be tested easily with these compose file and configuration:
> https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6
> Or even the configuration could be included in the compose file:
> https://github.com/elek/hadoop/blob/docker-2.8.0/example/docker-compose.yaml
> I would like to create separated example compose files for federation, ha,
> metrics usage, etc. to make it easier to try out and understand the features.
> CONTEXT: There is an existing Jira
> https://issues.apache.org/jira/browse/HADOOP-13397
> But it’s about a tool to generate production quality docker images (multiple
> types, in a flexible way). If no objections, I will create a separated issue
> to create simplified docker images for rapid prototyping and investigating
> new features. And register the branch to the dockerhub to create the images
> automatically.
> MY BACKGROUND: I am working with docker based hadoop/spark clusters quite a
> while and run them succesfully in different environments (kubernetes,
> docker-swarm, nomad-based scheduling, etc.) My work is available from here:
> https://github.com/flokkr but they could handle more complex use cases (eg.
> instrumenting java processes with btrace, or read/reload configuration from
> consul).
> And IMHO in the official hadoop documentation it’s better to suggest to use
> official apache docker images and not external ones (which could be changed).
> {code}
> The next list will enumerate the key decision points regarding to docker
> image creating
> A. automated dockerhub build / jenkins build
> Docker images could be built on the dockerhub (a branch pattern should be
> defined for a github repository and the location of the Docker files) or
> could be built on a CI server and pushed.
> The second one is more flexible (it's more easy to create matrix build, for
> example)
> The first one had the advantage that we can get an additional flag on the
> dockerhub that the build is automated (and built from the source by the
> dockerhub).
> The decision is easy as ASF supports the first approach: (see
> https://issues.apache.org/jira/browse/INFRA-12781?focusedCommentId=15824096&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15824096)
> B. source: binary distribution or source build
> The second question is about creating the docker image. One option is to
> build the software on the fly during the creation of the docker image the
> other one is to use the binary releases.
> I suggest to use the second approach as:
> 1. In that case the hadoop:2.7.3 could contain exactly the same hadoop
> distrubution as the downloadable one
> 2. We don't need to add development tools to the image, the image could be
> more smaller (which is important as the goal for this image to getting
> started as fast as possible)
> 3. The docker definition will be more simple (and more easy to maintain)
> Usually this approach is used in other projects (I checked Apache Zeppelin
> and Apache Nutch)
> C. branch usage
> Other question is the location of the Docker file. It could be on the
> official source-code branches (branch-2, trunk, etc.) or we can create
> separated branches for the dockerhub (eg. docker/2.7 docker/2.8 docker/3.0)
> For the first approach it's easier to find the docker images, but it's less
> flexible. For example if we had a Dockerfile for on the source code it should
> be used for every release (for example the Docker file from the tag
> release-3.0.0 should be used for the 3.0 hadoop docker image). In that case
> the release process is much more harder: in case of a Dockerfile error (which
> c
[jira] [Commented] (HADOOP-14898) Create official Docker images for development and testing features
[
https://issues.apache.org/jira/browse/HADOOP-14898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16388715#comment-16388715
]
Anu Engineer commented on HADOOP-14898:
---
[~chris.douglas] Thanks for the comments. I will create the branches now, and
update here with the branch info once it is done.
> Create official Docker images for development and testing features
> ---
>
> Key: HADOOP-14898
> URL: https://issues.apache.org/jira/browse/HADOOP-14898
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HADOOP-14898.001.tar.gz, HADOOP-14898.002.tar.gz,
> HADOOP-14898.003.tgz, docker_design.pdf
>
>
> This is the original mail from the mailing list:
> {code}
> TL;DR: I propose to create official hadoop images and upload them to the
> dockerhub.
> GOAL/SCOPE: I would like improve the existing documentation with easy-to-use
> docker based recipes to start hadoop clusters with various configuration.
> The images also could be used to test experimental features. For example
> ozone could be tested easily with these compose file and configuration:
> https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6
> Or even the configuration could be included in the compose file:
> https://github.com/elek/hadoop/blob/docker-2.8.0/example/docker-compose.yaml
> I would like to create separated example compose files for federation, ha,
> metrics usage, etc. to make it easier to try out and understand the features.
> CONTEXT: There is an existing Jira
> https://issues.apache.org/jira/browse/HADOOP-13397
> But it’s about a tool to generate production quality docker images (multiple
> types, in a flexible way). If no objections, I will create a separated issue
> to create simplified docker images for rapid prototyping and investigating
> new features. And register the branch to the dockerhub to create the images
> automatically.
> MY BACKGROUND: I am working with docker based hadoop/spark clusters quite a
> while and run them succesfully in different environments (kubernetes,
> docker-swarm, nomad-based scheduling, etc.) My work is available from here:
> https://github.com/flokkr but they could handle more complex use cases (eg.
> instrumenting java processes with btrace, or read/reload configuration from
> consul).
> And IMHO in the official hadoop documentation it’s better to suggest to use
> official apache docker images and not external ones (which could be changed).
> {code}
> The next list will enumerate the key decision points regarding to docker
> image creating
> A. automated dockerhub build / jenkins build
> Docker images could be built on the dockerhub (a branch pattern should be
> defined for a github repository and the location of the Docker files) or
> could be built on a CI server and pushed.
> The second one is more flexible (it's more easy to create matrix build, for
> example)
> The first one had the advantage that we can get an additional flag on the
> dockerhub that the build is automated (and built from the source by the
> dockerhub).
> The decision is easy as ASF supports the first approach: (see
> https://issues.apache.org/jira/browse/INFRA-12781?focusedCommentId=15824096&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15824096)
> B. source: binary distribution or source build
> The second question is about creating the docker image. One option is to
> build the software on the fly during the creation of the docker image the
> other one is to use the binary releases.
> I suggest to use the second approach as:
> 1. In that case the hadoop:2.7.3 could contain exactly the same hadoop
> distrubution as the downloadable one
> 2. We don't need to add development tools to the image, the image could be
> more smaller (which is important as the goal for this image to getting
> started as fast as possible)
> 3. The docker definition will be more simple (and more easy to maintain)
> Usually this approach is used in other projects (I checked Apache Zeppelin
> and Apache Nutch)
> C. branch usage
> Other question is the location of the Docker file. It could be on the
> official source-code branches (branch-2, trunk, etc.) or we can create
> separated branches for the dockerhub (eg. docker/2.7 docker/2.8 docker/3.0)
> For the first approach it's easier to find the docker images, but it's less
> flexible. For example if we had a Dockerfile for on the source code it should
> be used for every release (for example the Docker file from the tag
> release-3.0.0 should be used for the 3.0 hadoop docker image). In that case
> the release process is much more harder: in case of a Dockerfile error (which
> could be test on dockerhub only
[jira] [Commented] (HADOOP-14652) Update metrics-core version to 3.2.4
[ https://issues.apache.org/jira/browse/HADOOP-14652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384352#comment-16384352 ] Anu Engineer commented on HADOOP-14652: --- Just leaving a note here for other branches which are merging from trunk. If you have used this library in any child POMs you will get an error saying dependency versions not found for _com.codahale.metrics_. The way to fix is to change the groupId to _io.dropwizard.metrics_ from _com.codahale.metrics_ Discovered while merging trunk to ozone. > Update metrics-core version to 3.2.4 > > > Key: HADOOP-14652 > URL: https://issues.apache.org/jira/browse/HADOOP-14652 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.0.0-beta1 >Reporter: Ray Chiang >Assignee: Ray Chiang >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-14652.001.patch, HADOOP-14652.002.patch, > HADOOP-14652.003.patch, HADOOP-14652.004.patch, HADOOP-14652.005.patch, > HADOOP-14652.006.patch > > > The current artifact is: > com.codehale.metrics:metrics-core:3.0.1 > That version could either be bumped to 3.0.2 (the latest of that line), or > use the latest artifact: > io.dropwizard.metrics:metrics-core:3.2.4 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14898) Create official Docker images for development and testing features
[
https://issues.apache.org/jira/browse/HADOOP-14898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381264#comment-16381264
]
Anu Engineer commented on HADOOP-14898:
---
{quote}The infra process linked through the document (INFRA-12781) wasn't clear
to me. We create branches- this proposes a base image, {{hadoop2}}, and
{{hadoop3}}- and that will be pushed to the ASF repo on Docker Hub?
{quote}
[~chris.douglas] Thanks for the comments. ([~elek] Please correct me I am
saying something stupid. With the caveat that my understanding is not very deep
on this. I will try to answer this to the best of my understanding)
# We will have 2 base images – one for version 2 of Hadoop and one from
version 3 of Hadoop.
# When we deploy an image, for example, {{docker-compose pull; docker-compose
up}} what happens is that we will pull down these base images and start 2 or
more containers. One or more Namenode and a set of data nodes.
# docker-compose command will work directly against the docker-hub. When we
release a new version of Hadoop, we will create the corresponding base images –
I asked [~elek] in a private conversation if we need one image per release and
we thought we don't need it for now, since the base images are very similar for
each release, so we don't expect to see much change between the images – hence
one base image for Hadoop 2 and another one for Hadoop 3.
# In the release tarball, we will ship the base image or leave a pointer to
the new base image – so that PMC can try this out and approve it. One major
win, at least in my mind is that more developers(PMC members will be willing to
try out these applications, hence it is a single command to launch a docker
based Hadoop Cluster).
# Once the release is signed off, we will push those base images to DockerHub.
The dockerHub account credentials are something that INFRA owns, so will have
to file a ticket to update these images, since we are going to use Apache
DockerHub Account. I am told many other Apache projects follow this process.
If you like, *I can set up a call so we discuss and hash out the details*.
Please let me know if that is something that will be useful.
>From personal experience in Ozone branch, I can tell you that most devs make a
>change and actually test it before pushing them ever since [~elek] supported
>that feature for ozone. It has been a great QE tool for Ozone, and I feel that
>it will be very useful to have this feature for HDFS.
> Create official Docker images for development and testing features
> ---
>
> Key: HADOOP-14898
> URL: https://issues.apache.org/jira/browse/HADOOP-14898
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HADOOP-14898.001.tar.gz, HADOOP-14898.002.tar.gz,
> HADOOP-14898.003.tgz, docker_design.pdf
>
>
> This is the original mail from the mailing list:
> {code}
> TL;DR: I propose to create official hadoop images and upload them to the
> dockerhub.
> GOAL/SCOPE: I would like improve the existing documentation with easy-to-use
> docker based recipes to start hadoop clusters with various configuration.
> The images also could be used to test experimental features. For example
> ozone could be tested easily with these compose file and configuration:
> https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6
> Or even the configuration could be included in the compose file:
> https://github.com/elek/hadoop/blob/docker-2.8.0/example/docker-compose.yaml
> I would like to create separated example compose files for federation, ha,
> metrics usage, etc. to make it easier to try out and understand the features.
> CONTEXT: There is an existing Jira
> https://issues.apache.org/jira/browse/HADOOP-13397
> But it’s about a tool to generate production quality docker images (multiple
> types, in a flexible way). If no objections, I will create a separated issue
> to create simplified docker images for rapid prototyping and investigating
> new features. And register the branch to the dockerhub to create the images
> automatically.
> MY BACKGROUND: I am working with docker based hadoop/spark clusters quite a
> while and run them succesfully in different environments (kubernetes,
> docker-swarm, nomad-based scheduling, etc.) My work is available from here:
> https://github.com/flokkr but they could handle more complex use cases (eg.
> instrumenting java processes with btrace, or read/reload configuration from
> consul).
> And IMHO in the official hadoop documentation it’s better to suggest to use
> official apache docker images and not external ones (which could be changed).
> {code}
> The next list will enumerate the key decision points regarding
[jira] [Updated] (HADOOP-15007) Stabilize and document Configuration element
[ https://issues.apache.org/jira/browse/HADOOP-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HADOOP-15007: -- Resolution: Fixed Status: Resolved (was: Patch Available) I have committed to branch-3.1, thanks for the updated patch. [~ajayydv] > Stabilize and document Configuration element > -- > > Key: HADOOP-15007 > URL: https://issues.apache.org/jira/browse/HADOOP-15007 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Ajay Kumar >Priority: Blocker > Fix For: 3.1.0, 3.2.0 > > Attachments: HADOOP-15007-branch-3.1.000.patch, > HADOOP-15007.000.patch, HADOOP-15007.001.patch, HADOOP-15007.002.patch, > HADOOP-15007.003.patch > > > HDFS-12350 (moved to HADOOP-15005). Adds the ability to tag properties with a > value. > We need to make sure that this feature is backwards compatible & usable in > production. That's docs, testing, marshalling etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14898) Create official Docker images for development and testing features
[
https://issues.apache.org/jira/browse/HADOOP-14898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16377272#comment-16377272
]
Anu Engineer commented on HADOOP-14898:
---
[~aw], [~chris.douglas], [~ste...@apache.org] I plan to create these branches -
HADOOP-15083 / HADOOP-15084 / HADOOP-15256 and file a ticket with INFRA so that
they can push this image as an Apache Image to DockerHub. This will allow INFRA
to push these dockerHub Images if and when we make the changes. That is, we
will have official Apache base images which can be trusted by end-users and
updated with various releases if needed.
{quote}HADOOP-15257, HADOOP-15258, HADOOP-15259 should be committed to trunk.
{quote}
Once that is done, I will commit these patches to the trunk.
Please let me know if you see any issues or have any concerns.
> Create official Docker images for development and testing features
> ---
>
> Key: HADOOP-14898
> URL: https://issues.apache.org/jira/browse/HADOOP-14898
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HADOOP-14898.001.tar.gz, HADOOP-14898.002.tar.gz,
> HADOOP-14898.003.tgz, docker_design.pdf
>
>
> This is the original mail from the mailing list:
> {code}
> TL;DR: I propose to create official hadoop images and upload them to the
> dockerhub.
> GOAL/SCOPE: I would like improve the existing documentation with easy-to-use
> docker based recipes to start hadoop clusters with various configuration.
> The images also could be used to test experimental features. For example
> ozone could be tested easily with these compose file and configuration:
> https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6
> Or even the configuration could be included in the compose file:
> https://github.com/elek/hadoop/blob/docker-2.8.0/example/docker-compose.yaml
> I would like to create separated example compose files for federation, ha,
> metrics usage, etc. to make it easier to try out and understand the features.
> CONTEXT: There is an existing Jira
> https://issues.apache.org/jira/browse/HADOOP-13397
> But it’s about a tool to generate production quality docker images (multiple
> types, in a flexible way). If no objections, I will create a separated issue
> to create simplified docker images for rapid prototyping and investigating
> new features. And register the branch to the dockerhub to create the images
> automatically.
> MY BACKGROUND: I am working with docker based hadoop/spark clusters quite a
> while and run them succesfully in different environments (kubernetes,
> docker-swarm, nomad-based scheduling, etc.) My work is available from here:
> https://github.com/flokkr but they could handle more complex use cases (eg.
> instrumenting java processes with btrace, or read/reload configuration from
> consul).
> And IMHO in the official hadoop documentation it’s better to suggest to use
> official apache docker images and not external ones (which could be changed).
> {code}
> The next list will enumerate the key decision points regarding to docker
> image creating
> A. automated dockerhub build / jenkins build
> Docker images could be built on the dockerhub (a branch pattern should be
> defined for a github repository and the location of the Docker files) or
> could be built on a CI server and pushed.
> The second one is more flexible (it's more easy to create matrix build, for
> example)
> The first one had the advantage that we can get an additional flag on the
> dockerhub that the build is automated (and built from the source by the
> dockerhub).
> The decision is easy as ASF supports the first approach: (see
> https://issues.apache.org/jira/browse/INFRA-12781?focusedCommentId=15824096&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15824096)
> B. source: binary distribution or source build
> The second question is about creating the docker image. One option is to
> build the software on the fly during the creation of the docker image the
> other one is to use the binary releases.
> I suggest to use the second approach as:
> 1. In that case the hadoop:2.7.3 could contain exactly the same hadoop
> distrubution as the downloadable one
> 2. We don't need to add development tools to the image, the image could be
> more smaller (which is important as the goal for this image to getting
> started as fast as possible)
> 3. The docker definition will be more simple (and more easy to maintain)
> Usually this approach is used in other projects (I checked Apache Zeppelin
> and Apache Nutch)
> C. branch usage
> Other question is the location of the Docker file. It could be on the
> official source-code branches (branch-2, trunk,
[jira] [Commented] (HADOOP-15007) Stabilize and document Configuration element
[ https://issues.apache.org/jira/browse/HADOOP-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16374822#comment-16374822 ] Anu Engineer commented on HADOOP-15007: --- [~leftnoteasy] if you are ok with this, Can I commit this to 3.1 ? Thanks. > Stabilize and document Configuration element > -- > > Key: HADOOP-15007 > URL: https://issues.apache.org/jira/browse/HADOOP-15007 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Ajay Kumar >Priority: Blocker > Fix For: 3.2.0 > > Attachments: HADOOP-15007.000.patch, HADOOP-15007.001.patch, > HADOOP-15007.002.patch, HADOOP-15007.003.patch > > > HDFS-12350 (moved to HADOOP-15005). Adds the ability to tag properties with a > value. > We need to make sure that this feature is backwards compatible & usable in > production. That's docs, testing, marshalling etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15007) Stabilize and document Configuration element
[ https://issues.apache.org/jira/browse/HADOOP-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HADOOP-15007: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.2.0 Status: Resolved (was: Patch Available) [~ajayydv] I have committed this to the trunk. Thank you for the contribution. > Stabilize and document Configuration element > -- > > Key: HADOOP-15007 > URL: https://issues.apache.org/jira/browse/HADOOP-15007 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Ajay Kumar >Priority: Blocker > Fix For: 3.2.0 > > Attachments: HADOOP-15007.000.patch, HADOOP-15007.001.patch, > HADOOP-15007.002.patch, HADOOP-15007.003.patch > > > HDFS-12350 (moved to HADOOP-15005). Adds the ability to tag properties with a > value. > We need to make sure that this feature is backwards compatible & usable in > production. That's docs, testing, marshalling etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15007) Stabilize and document Configuration element
[ https://issues.apache.org/jira/browse/HADOOP-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16374809#comment-16374809 ] Anu Engineer commented on HADOOP-15007: --- [~ste...@apache.org] and [~elek] I am committing this change. With this change, all of our three perspectives are addressed. # Lose types and Enums - This patch makes tags simple strings. # Detect tags which are not correct - The systems tags and custom tags list in the XML config can be used to cross check if a tag is valid or not. If it is not, we add a single line of log that tells us that tag is being ignored. # Remove unnecessary logging - With this patch, Ajay has removed the repeated warnings that were filling up the logs. [~ajayydv], +1, thanks for getting this done, I will commit this shortly. > Stabilize and document Configuration element > -- > > Key: HADOOP-15007 > URL: https://issues.apache.org/jira/browse/HADOOP-15007 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Ajay Kumar >Priority: Blocker > Attachments: HADOOP-15007.000.patch, HADOOP-15007.001.patch, > HADOOP-15007.002.patch, HADOOP-15007.003.patch > > > HDFS-12350 (moved to HADOOP-15005). Adds the ability to tag properties with a > value. > We need to make sure that this feature is backwards compatible & usable in > production. That's docs, testing, marshalling etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15007) Stabilize and document Configuration element
[
https://issues.apache.org/jira/browse/HADOOP-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366512#comment-16366512
]
Anu Engineer commented on HADOOP-15007:
---
[~ajayydv] Overall it looks very good. Some minor comments/questions.
# Since we have the hadoop.system.tags in core-default.xml, do we need {{public
static final String HADOOP_SYSTEM_TAGS_DEFAULT =}} in
{{CommonConfigurationKeysPublic.java}}
# Not sure why we callthe {{removeUndeclaredTags}}, if we don't do that we get
custom tag support right? Am I missing something here?
> Stabilize and document Configuration element
> --
>
> Key: HADOOP-15007
> URL: https://issues.apache.org/jira/browse/HADOOP-15007
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Ajay Kumar
>Priority: Blocker
> Attachments: HADOOP-15007.000.patch, HADOOP-15007.001.patch
>
>
> HDFS-12350 (moved to HADOOP-15005). Adds the ability to tag properties with a
> value.
> We need to make sure that this feature is backwards compatible & usable in
> production. That's docs, testing, marshalling etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15204) Add Configuration API for parsing storage sizes
[
https://issues.apache.org/jira/browse/HADOOP-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365921#comment-16365921
]
Anu Engineer commented on HADOOP-15204:
---
{quote}
Given its in, I'd like to see a followup "move existing uses of getLongBytes to
getStorageUnits"
{quote}
Will do, I plan to post an initial patch where I will convert the Ozone usage
to getStorageUnits and I will follow up with trunk later.
> Add Configuration API for parsing storage sizes
> ---
>
> Key: HADOOP-15204
> URL: https://issues.apache.org/jira/browse/HADOOP-15204
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
>Affects Versions: 3.1.0
>Reporter: Anu Engineer
>Assignee: Anu Engineer
>Priority: Minor
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HADOOP-15204.001.patch, HADOOP-15204.002.patch,
> HADOOP-15204.003.patch
>
>
> Hadoop has a lot of configurations that specify memory and disk size. This
> JIRA proposes to add an API like {{Configuration.getStorageSize}} which will
> allow users
> to specify units like KB, MB, GB etc. This is JIRA is inspired by
> HADOOP-8608 and Ozone. Adding {{getTimeDuration}} support was a great
> improvement for ozone code base, this JIRA hopes to do the same thing for
> configs that deal with disk and memory usage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15204) Add Configuration API for parsing storage sizes
[
https://issues.apache.org/jira/browse/HADOOP-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HADOOP-15204:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.1
Status: Resolved (was: Patch Available)
[~chris.douglas] and [~ste...@apache.org] Thanks for code reviews. I have
committed this to trunk, branch-3.0, branch-3.0.1, and branch-3.1
> Add Configuration API for parsing storage sizes
> ---
>
> Key: HADOOP-15204
> URL: https://issues.apache.org/jira/browse/HADOOP-15204
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
>Affects Versions: 3.1.0
>Reporter: Anu Engineer
>Assignee: Anu Engineer
>Priority: Minor
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HADOOP-15204.001.patch, HADOOP-15204.002.patch,
> HADOOP-15204.003.patch
>
>
> Hadoop has a lot of configurations that specify memory and disk size. This
> JIRA proposes to add an API like {{Configuration.getStorageSize}} which will
> allow users
> to specify units like KB, MB, GB etc. This is JIRA is inspired by
> HADOOP-8608 and Ozone. Adding {{getTimeDuration}} support was a great
> improvement for ozone code base, this JIRA hopes to do the same thing for
> configs that deal with disk and memory usage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15204) Add Configuration API for parsing storage sizes
[
https://issues.apache.org/jira/browse/HADOOP-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364761#comment-16364761
]
Anu Engineer commented on HADOOP-15204:
---
[~chris.douglas] / [~ste...@apache.org] Please let me know if you have any more
comments. If this looks good, I will make corresponding changes in Ozone branch
to use this feature. Thank you for the time and comments.
> Add Configuration API for parsing storage sizes
> ---
>
> Key: HADOOP-15204
> URL: https://issues.apache.org/jira/browse/HADOOP-15204
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
>Affects Versions: 3.1.0
>Reporter: Anu Engineer
>Assignee: Anu Engineer
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HADOOP-15204.001.patch, HADOOP-15204.002.patch,
> HADOOP-15204.003.patch
>
>
> Hadoop has a lot of configurations that specify memory and disk size. This
> JIRA proposes to add an API like {{Configuration.getStorageSize}} which will
> allow users
> to specify units like KB, MB, GB etc. This is JIRA is inspired by
> HADOOP-8608 and Ozone. Adding {{getTimeDuration}} support was a great
> improvement for ozone code base, this JIRA hopes to do the same thing for
> configs that deal with disk and memory usage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15204) Add Configuration API for parsing storage sizes
[
https://issues.apache.org/jira/browse/HADOOP-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352956#comment-16352956
]
Anu Engineer commented on HADOOP-15204:
---
[~ste...@apache.org], [~chris.douglas] Thanks for the comments. Patch v3
addresses all the comments.
Details below:
bq. IDE shuffled imports; please revert
Thanks for catching this, Fixed.
bq. parseFromString() can just use Precondition.checkArgument for validation
Fixed.
bq. validation/parse errors to include value at error, and, ideally, config
option too. Compare a stack trace saying "Value not in expected format", with
one saying "value of option 'buffer.size' not in expected format "54exa"
Fixed.
bq. sanitizedValue.toLowerCase() should specify local for case conversion, same
everywhere else used.
Fixed.
bq. What if a caller doesn't want to provide a string default value of the new
getters, but just a number? That would let me return something like -1 to mean
"no value set", which I can't do with the current API.
There is an API that takes a default float argument, and a default string
argument with the storage unit.
bq. getStorageSize(String name, String defaultValue,
+ StorageUnit targetUnit) -- Does this come up often?
We define the standard defaults as "5 GB", etc., so yes it is a convenient
function.
bq. I'd lean toward MB instead of MEGABYTES, and similar.
Fixed. I agree, thanks for this suggestion, that does improve code readability.
bq. Please, no. This is the silliest dependency we have on Guava.
Fixed. I still use it in Configuration, since it is already in the file as an
import.
> Add Configuration API for parsing storage sizes
> ---
>
> Key: HADOOP-15204
> URL: https://issues.apache.org/jira/browse/HADOOP-15204
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
>Affects Versions: 3.1.0
>Reporter: Anu Engineer
>Assignee: Anu Engineer
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HADOOP-15204.001.patch, HADOOP-15204.002.patch,
> HADOOP-15204.003.patch
>
>
> Hadoop has a lot of configurations that specify memory and disk size. This
> JIRA proposes to add an API like {{Configuration.getStorageSize}} which will
> allow users
> to specify units like KB, MB, GB etc. This is JIRA is inspired by
> HADOOP-8608 and Ozone. Adding {{getTimeDuration}} support was a great
> improvement for ozone code base, this JIRA hopes to do the same thing for
> configs that deal with disk and memory usage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15204) Add Configuration API for parsing storage sizes
[
https://issues.apache.org/jira/browse/HADOOP-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HADOOP-15204:
--
Attachment: HADOOP-15204.003.patch
> Add Configuration API for parsing storage sizes
> ---
>
> Key: HADOOP-15204
> URL: https://issues.apache.org/jira/browse/HADOOP-15204
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
>Affects Versions: 3.1.0
>Reporter: Anu Engineer
>Assignee: Anu Engineer
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HADOOP-15204.001.patch, HADOOP-15204.002.patch,
> HADOOP-15204.003.patch
>
>
> Hadoop has a lot of configurations that specify memory and disk size. This
> JIRA proposes to add an API like {{Configuration.getStorageSize}} which will
> allow users
> to specify units like KB, MB, GB etc. This is JIRA is inspired by
> HADOOP-8608 and Ozone. Adding {{getTimeDuration}} support was a great
> improvement for ozone code base, this JIRA hopes to do the same thing for
> configs that deal with disk and memory usage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15007) Stabilize and document Configuration element
[
https://issues.apache.org/jira/browse/HADOOP-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351659#comment-16351659
]
Anu Engineer commented on HADOOP-15007:
---
{quote}Please, stop trying to argue this. I'm not going to be convinced. Sorry.
{quote}
Sorry If i came across as having a strong opinion on this, in fact, I don't
have a strong preference of enums over strings. I was thinking the core issue
was extra logging.
Let us by all means go over to strings, I just want this feature to work, and I
don't see any strong benefit in having Enums, after all XML is strings as you
pointed out.
cc: [~ajayydv]
> Stabilize and document Configuration element
> --
>
> Key: HADOOP-15007
> URL: https://issues.apache.org/jira/browse/HADOOP-15007
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Ajay Kumar
>Priority: Blocker
>
> HDFS-12350 (moved to HADOOP-15005). Adds the ability to tag properties with a
> value.
> We need to make sure that this feature is backwards compatible & usable in
> production. That's docs, testing, marshalling etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15204) Add Configuration API for parsing storage sizes
[
https://issues.apache.org/jira/browse/HADOOP-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349709#comment-16349709
]
Anu Engineer commented on HADOOP-15204:
---
[~chris.douglas] I have attached patch v2 that takes care of the comments and
checkstyle issues.
{quote}I find the API intuitive, but that is not universal (e.g., HDFS-9847).
Explaining it has taken more cycles than I expected, and perhaps more than a
good API should.
{quote}
Thank you for the time and comments. Personally, I find _getTimeDuration_ API
extremely intuitive, hence imitating that for this API; As for others, you have
done the heavy lifting of educating the crowd, I will just ride on your
coattails.
{quote}TERRABYTES is misspelled.
{quote}
Thanks for catching that, Fixed.
{quote}Is long insufficient as a return type for getStorageSize? I appreciate
future-proofing, but for Configuration values, that's what, ~8 petabytes?
{quote}
I started with long; the real issue was returning rounded numbers for large
storage units. Rounding causes a significant loss when we convert from _x
bytes_ to _y exabytes_. Hence I voted for the least element of surprise and
decided to return double.
{quote}Why ROUND_UP of the options? Just curious.
{quote}
I was using RoundingMode.HALF_UP in divide and now I do that for multiply too,
just to be consistent.
The reason for using
[HALF_UP|https://docs.oracle.com/javase/8/docs/api/java/math/BigDecimal.html#ROUND_HALF_UP]
is that it is probably the least surprising result for most users. From the
Doc: {{Note that this is the rounding mode that most of us were taught in grade
school.}}
{quote}Storage units are more likely to be exact powers
{quote}
This is the curse of writing a unit as a library; we need to be cognizant of
that single use case which will break us. Hence I have used bigDecimal to be
safe and correct and return doubles. It yields values that people expect.
> Add Configuration API for parsing storage sizes
> ---
>
> Key: HADOOP-15204
> URL: https://issues.apache.org/jira/browse/HADOOP-15204
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
>Affects Versions: 3.1.0
>Reporter: Anu Engineer
>Assignee: Anu Engineer
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HADOOP-15204.001.patch, HADOOP-15204.002.patch
>
>
> Hadoop has a lot of configurations that specify memory and disk size. This
> JIRA proposes to add an API like {{Configuration.getStorageSize}} which will
> allow users
> to specify units like KB, MB, GB etc. This is JIRA is inspired by
> HADOOP-8608 and Ozone. Adding {{getTimeDuration}} support was a great
> improvement for ozone code base, this JIRA hopes to do the same thing for
> configs that deal with disk and memory usage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15204) Add Configuration API for parsing storage sizes
[
https://issues.apache.org/jira/browse/HADOOP-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HADOOP-15204:
--
Attachment: HADOOP-15204.002.patch
> Add Configuration API for parsing storage sizes
> ---
>
> Key: HADOOP-15204
> URL: https://issues.apache.org/jira/browse/HADOOP-15204
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
>Affects Versions: 3.1.0
>Reporter: Anu Engineer
>Assignee: Anu Engineer
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HADOOP-15204.001.patch, HADOOP-15204.002.patch
>
>
> Hadoop has a lot of configurations that specify memory and disk size. This
> JIRA proposes to add an API like {{Configuration.getStorageSize}} which will
> allow users
> to specify units like KB, MB, GB etc. This is JIRA is inspired by
> HADOOP-8608 and Ozone. Adding {{getTimeDuration}} support was a great
> improvement for ozone code base, this JIRA hopes to do the same thing for
> configs that deal with disk and memory usage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15204) Add Configuration API for parsing storage sizes
[
https://issues.apache.org/jira/browse/HADOOP-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349121#comment-16349121
]
Anu Engineer edited comment on HADOOP-15204 at 2/1/18 8:22 PM:
---
[~ste...@apache.org] I did look at getLongBytes. The issue is that it does not
provide the same code improvement as {{getTimeDuration}} and
{{setTimeDuration}}.
Let me support my assertion with some examples that you can look at.
Let us suppose that I have a case to read the standard configured size of
containers. This is a configurable parameter in Ozone.
Here is how the code would look like with get/setStorageSize.
{code:java}
setStorageSize(OZONE_CONTAINER_SIZE, 1, GIGABYTES);
// let us suppose we want to read back this config value as MBs.
long valueInMB = getStorageSize(OZONE_CONTAINER_SIZE, "5 GB", MEGABYTES);{code}
now let us see how we write this code using getLongBytes..
{code:java}
// there is no symmetric function called setLongBytes.. So I have to fall back
to setLong
//
// Here is my first attempt.
setLong(OZONE_CONTAINER_SIZE, 1048576);
// This looks bad, since I am not sure if the OZONE_CONTAINER_SIZE is in MB/GB
or what,
// So the many keys get tagged as OZONE_CONTAINER_SIZE_GB
// Now second attempt.
setLong(OZONE_CONTAINER_SIZE_GB, 1048576);
// But this is bad too, now I can not set container size in MB, since setLong
takes a
// whole number and not a fraction.
// So now third attempt -- convert all fields to bytes
setLong(OZONE_CONTAINER_SIZE_BYTES, 5368709120); // The default is 5 GG.
{code}
Before you think this is a made up example, this is part of changes that we
tried which triggered this change.
Now let us go to back get examples –
{code:java}
// getLongBytes forces us to write code in a certain way which does not match
with
// rest of code like getTimeDuration. For example, if I have a case where I
want to read // the read the value in MB, and the ozone-default.xml is
configured to GB.
// the case that this line solves
// getStorageSize(OZONE_CONTAINER_SIZE, "5 GB", MEGABYTES);
long defaultValueInBytes = getDefaultValue(OZONE_CONTAINER_SIZE_DEFAULT); // in
bytes.
long valueInMB = BYTES.fromBytes(getLongBytes(OZONE_CONTAINER_SIZE,
defaultValueInBytes)).toMBs();{code}
Now imagine repeating this code many times all over the code base, plus, the
BYTES.fromBytes(xxx).ToMBs() is a fuction from this patch. We need some
equivalent code.
In other words, I submit that these following factors make getLongBytes a less
desirable function compared to getTimeDuration/getStorageSize.
* lack of symetric function in getLongBytes - Without a set function it
degenerates to a messy set of multiplication and division each time we have to
use a storage Unit. With this those issue are cleanly isolated to a single
place.
* Lack of a formal storage unit - lack of a formal value like to
TimeUnit/StorageUnit makes the code less readable (see the example
setStorageSize) where the context also tells you the unit that we are operating
with.
* Does not suit our usage pattern - Ozone code follows the patterns in HDFS.
Hence the default value mapping is not handled well in getLongBytes.
* Units and conversion is needed - In ozone, there are several places where we
convert these numbers. Users can specify Quota as a easy to read Storage Units,
like 5 GB or 10 TB. We have a dedicated code for handling that, we use the
storage numbers to specify how large the off-heap size should be, or as I am
showing in this example, container sizes etc.
* The getLongBytes by itself does not address the lack of StorageUnits, what
this patch does is that it introduces a class that is very similar to
{{TimeUnit}}. This makes the code more readable and easy to maintain.
was (Author: anu):
[~ste...@apache.org] I did look at getLongBytes. The issue is that it does not
provide the same code improvement as {{getTimeDuration}} and
{{setTimeDuration}}.
Let me support my assertion with some examples that you can look at.
Let us suppose that I have a case to read the standard configured size of
containers. This is a configurable parameter in Ozone.
Here is how the code would look like with get/setStorageSize.
{code:java}
setStorageSize(OZONE_CONTAINER_SIZE, 1, GIGABYTES);
// let us suppose we want to read back this config value as MBs.
long valueInMB = getStorageSize(OZONE_CONTAINER_SIZE, "5 GB", MEGABYTES);{code}
now let us see how we write this code using getLongBytes..
{code:java}
// there is no symetric function called setLongBytes.. So I have to fall back
to setLong
//
// Here is my first attempt.
setLong(OZONE_CONTAINER_SIZE, 1048576);
// This looks bad, since I am not sure if the OZONE_CONTAINER_SIZE is in MB/GB
or what,
// So the many keys get tagged as OZONE_CONTAINER_SIZE_GB
// Now second attempt.
setLong(OZONE_CONTAINER_SIZE_GB, 1048576);
// But this is bad too
[jira] [Commented] (HADOOP-15204) Add Configuration API for parsing storage sizes
[
https://issues.apache.org/jira/browse/HADOOP-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349191#comment-16349191
]
Anu Engineer commented on HADOOP-15204:
---
bq.[~anu] , you mean HADOOP-8608?
[~xyao] Thanks for catching that. Fixed.
> Add Configuration API for parsing storage sizes
> ---
>
> Key: HADOOP-15204
> URL: https://issues.apache.org/jira/browse/HADOOP-15204
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
>Affects Versions: 3.1.0
>Reporter: Anu Engineer
>Assignee: Anu Engineer
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HADOOP-15204.001.patch
>
>
> Hadoop has a lot of configurations that specify memory and disk size. This
> JIRA proposes to add an API like {{Configuration.getStorageSize}} which will
> allow users
> to specify units like KB, MB, GB etc. This is JIRA is inspired by
> HADOOP-8608 and Ozone. Adding {{getTimeDuration}} support was a great
> improvement for ozone code base, this JIRA hopes to do the same thing for
> configs that deal with disk and memory usage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15204) Add Configuration API for parsing storage sizes
[
https://issues.apache.org/jira/browse/HADOOP-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HADOOP-15204:
--
Description:
Hadoop has a lot of configurations that specify memory and disk size. This JIRA
proposes to add an API like {{Configuration.getStorageSize}} which will allow
users
to specify units like KB, MB, GB etc. This is JIRA is inspired by HADOOP-8608
and Ozone. Adding {{getTimeDuration}} support was a great improvement for ozone
code base, this JIRA hopes to do the same thing for configs that deal with disk
and memory usage.
was:
Hadoop has a lot of configurations that specify memory and disk size. This JIRA
proposes to add an API like {{Configuration.getStorageSize}} which will allow
users
to specify units like KB, MB, GB etc. This is JIRA is inspired by HDFS-8608 and
Ozone. Adding {{getTimeDuration}} support was a great improvement for ozone
code base, this JIRA hopes to do the same thing for configs that deal with disk
and memory usage.
> Add Configuration API for parsing storage sizes
> ---
>
> Key: HADOOP-15204
> URL: https://issues.apache.org/jira/browse/HADOOP-15204
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
>Affects Versions: 3.1.0
>Reporter: Anu Engineer
>Assignee: Anu Engineer
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HADOOP-15204.001.patch
>
>
> Hadoop has a lot of configurations that specify memory and disk size. This
> JIRA proposes to add an API like {{Configuration.getStorageSize}} which will
> allow users
> to specify units like KB, MB, GB etc. This is JIRA is inspired by
> HADOOP-8608 and Ozone. Adding {{getTimeDuration}} support was a great
> improvement for ozone code base, this JIRA hopes to do the same thing for
> configs that deal with disk and memory usage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
