[jira] [Updated] (YARN-5244) Documentation required for DNS Server implementation
[ https://issues.apache.org/jira/browse/YARN-5244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-5244: - Attachment: dns record creation.jpeg dns record removal.jpeg dns overview.png yarn_dns_server.md Initial documentation > Documentation required for DNS Server implementation > > > Key: YARN-5244 > URL: https://issues.apache.org/jira/browse/YARN-5244 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Maron > Attachments: dns overview.png, dns record creation.jpeg, dns record > removal.jpeg, yarn_dns_server.md > > > The DNS server requires documentation describing its functionality etc -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-4951) large IP ranges require a different naming strategy
[ https://issues.apache.org/jira/browse/YARN-4951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron resolved YARN-4951. -- Resolution: Fixed Fixed with the specification of zone-mask property > large IP ranges require a different naming strategy > --- > > Key: YARN-4951 > URL: https://issues.apache.org/jira/browse/YARN-4951 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Maron >Assignee: Jonathan Maron > Attachments: > 0001-YARN-4757-simplified-reverse-lookup-zone-approach-fo.patch > > > Large subnet definitions (e.g. specifying a mask value of 255.255.224.0) > yield a large number of potential network addresses. Therefore, the standard > naming convention of xx.xx.xx.in-addr.arpa needs to be modified to be more > general: xx.xx.in-addr.arpa. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5244) Documentation required for DNS Server implementation
Jonathan Maron created YARN-5244: Summary: Documentation required for DNS Server implementation Key: YARN-5244 URL: https://issues.apache.org/jira/browse/YARN-5244 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jonathan Maron The DNS server requires documentation describing its functionality etc -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5218) Submit initial DNS server approach for review
[ https://issues.apache.org/jira/browse/YARN-5218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-5218: - Attachment: YARN-5218-YARN-4757.003.patch > Submit initial DNS server approach for review > - > > Key: YARN-5218 > URL: https://issues.apache.org/jira/browse/YARN-5218 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Jonathan Maron >Assignee: Jonathan Maron > Attachments: YARN-5218-YARN-4757.001.patch, > YARN-5218-YARN-4757.002.patch, YARN-5218-YARN-4757.003.patch > > > Submit the initial dnsjava based solution for review -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5218) Submit initial DNS server approach for review
[ https://issues.apache.org/jira/browse/YARN-5218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-5218: - Attachment: YARN-5218-YARN-4757.002.patch > Submit initial DNS server approach for review > - > > Key: YARN-5218 > URL: https://issues.apache.org/jira/browse/YARN-5218 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Jonathan Maron >Assignee: Jonathan Maron > Attachments: YARN-5218-YARN-4757.001.patch, > YARN-5218-YARN-4757.002.patch > > > Submit the initial dnsjava based solution for review -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5218) Submit initial DNS server approach for review
[ https://issues.apache.org/jira/browse/YARN-5218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-5218: - Attachment: YARN-5218-YARN-4757.001.patch latest implementation > Submit initial DNS server approach for review > - > > Key: YARN-5218 > URL: https://issues.apache.org/jira/browse/YARN-5218 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Jonathan Maron >Assignee: Jonathan Maron > Attachments: YARN-5218-YARN-4757.001.patch > > > Submit the initial dnsjava based solution for review -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5218) Submit initial DNS server approach for review
Jonathan Maron created YARN-5218: Summary: Submit initial DNS server approach for review Key: YARN-5218 URL: https://issues.apache.org/jira/browse/YARN-5218 Project: Hadoop YARN Issue Type: Sub-task Components: yarn Reporter: Jonathan Maron Assignee: Jonathan Maron Submit the initial dnsjava based solution for review -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
[ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-4757: - Attachment: YARN-4757-YARN-4757.005.patch > [Umbrella] Simplified discovery of services via DNS mechanisms > -- > > Key: YARN-4757 > URL: https://issues.apache.org/jira/browse/YARN-4757 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Maron > Attachments: > 0001-YARN-4757-Initial-code-submission-for-DNS-Service.patch, YARN-4757- > Simplified discovery of services via DNS mechanisms.pdf, > YARN-4757-YARN-4757.001.patch, YARN-4757-YARN-4757.002.patch, > YARN-4757-YARN-4757.003.patch, YARN-4757-YARN-4757.004.patch, > YARN-4757-YARN-4757.005.patch, YARN-4757.001.patch, YARN-4757.002.patch > > > [See overview doc at YARN-4692, copying the sub-section (3.2.10.2) to track > all related efforts.] > In addition to completing the present story of service-registry (YARN-913), > we also need to simplify the access to the registry entries. The existing > read mechanisms of the YARN Service Registry are currently limited to a > registry specific (java) API and a REST interface. In practice, this makes it > very difficult for wiring up existing clients and services. For e.g, dynamic > configuration of dependent endpoints of a service is not easy to implement > using the present registry-read mechanisms, *without* code-changes to > existing services. > A good solution to this is to expose the registry information through a more > generic and widely used discovery mechanism: DNS. Service Discovery via DNS > uses the well-known DNS interfaces to browse the network for services. > YARN-913 in fact talked about such a DNS based mechanism but left it as a > future task. (Task) Having the registry information exposed via DNS > simplifies the life of services. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
[ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-4757: - Attachment: YARN-4757-YARN-4757.004.patch > [Umbrella] Simplified discovery of services via DNS mechanisms > -- > > Key: YARN-4757 > URL: https://issues.apache.org/jira/browse/YARN-4757 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Maron > Attachments: > 0001-YARN-4757-Initial-code-submission-for-DNS-Service.patch, YARN-4757- > Simplified discovery of services via DNS mechanisms.pdf, > YARN-4757-YARN-4757.001.patch, YARN-4757-YARN-4757.002.patch, > YARN-4757-YARN-4757.003.patch, YARN-4757-YARN-4757.004.patch, > YARN-4757.001.patch, YARN-4757.002.patch > > > [See overview doc at YARN-4692, copying the sub-section (3.2.10.2) to track > all related efforts.] > In addition to completing the present story of service-registry (YARN-913), > we also need to simplify the access to the registry entries. The existing > read mechanisms of the YARN Service Registry are currently limited to a > registry specific (java) API and a REST interface. In practice, this makes it > very difficult for wiring up existing clients and services. For e.g, dynamic > configuration of dependent endpoints of a service is not easy to implement > using the present registry-read mechanisms, *without* code-changes to > existing services. > A good solution to this is to expose the registry information through a more > generic and widely used discovery mechanism: DNS. Service Discovery via DNS > uses the well-known DNS interfaces to browse the network for services. > YARN-913 in fact talked about such a DNS based mechanism but left it as a > future task. (Task) Having the registry information exposed via DNS > simplifies the life of services. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
[ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-4757: - Attachment: YARN-4757.002.patch Addressing checkstyle issues > [Umbrella] Simplified discovery of services via DNS mechanisms > -- > > Key: YARN-4757 > URL: https://issues.apache.org/jira/browse/YARN-4757 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Maron > Attachments: > 0001-YARN-4757-Initial-code-submission-for-DNS-Service.patch, YARN-4757- > Simplified discovery of services via DNS mechanisms.pdf, > YARN-4757-YARN-4757.001.patch, YARN-4757-YARN-4757.002.patch, > YARN-4757-YARN-4757.003.patch, YARN-4757.001.patch, YARN-4757.002.patch > > > [See overview doc at YARN-4692, copying the sub-section (3.2.10.2) to track > all related efforts.] > In addition to completing the present story of service-registry (YARN-913), > we also need to simplify the access to the registry entries. The existing > read mechanisms of the YARN Service Registry are currently limited to a > registry specific (java) API and a REST interface. In practice, this makes it > very difficult for wiring up existing clients and services. For e.g, dynamic > configuration of dependent endpoints of a service is not easy to implement > using the present registry-read mechanisms, *without* code-changes to > existing services. > A good solution to this is to expose the registry information through a more > generic and widely used discovery mechanism: DNS. Service Discovery via DNS > uses the well-known DNS interfaces to browse the network for services. > YARN-913 in fact talked about such a DNS based mechanism but left it as a > future task. (Task) Having the registry information exposed via DNS > simplifies the life of services. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
[ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-4757: - Attachment: YARN-4757.001.patch > [Umbrella] Simplified discovery of services via DNS mechanisms > -- > > Key: YARN-4757 > URL: https://issues.apache.org/jira/browse/YARN-4757 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Maron > Attachments: > 0001-YARN-4757-Initial-code-submission-for-DNS-Service.patch, YARN-4757- > Simplified discovery of services via DNS mechanisms.pdf, > YARN-4757-YARN-4757.001.patch, YARN-4757-YARN-4757.002.patch, > YARN-4757-YARN-4757.003.patch, YARN-4757.001.patch > > > [See overview doc at YARN-4692, copying the sub-section (3.2.10.2) to track > all related efforts.] > In addition to completing the present story of service-registry (YARN-913), > we also need to simplify the access to the registry entries. The existing > read mechanisms of the YARN Service Registry are currently limited to a > registry specific (java) API and a REST interface. In practice, this makes it > very difficult for wiring up existing clients and services. For e.g, dynamic > configuration of dependent endpoints of a service is not easy to implement > using the present registry-read mechanisms, *without* code-changes to > existing services. > A good solution to this is to expose the registry information through a more > generic and widely used discovery mechanism: DNS. Service Discovery via DNS > uses the well-known DNS interfaces to browse the network for services. > YARN-913 in fact talked about such a DNS based mechanism but left it as a > future task. (Task) Having the registry information exposed via DNS > simplifies the life of services. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
[
https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317372#comment-15317372
]
Jonathan Maron commented on YARN-4757:
--
{quote}
Do you mean this flag will be used to enable/disable dns functionality if the
DNS server is hosted in RM ?
{quote}
Oh - good point. At one point I was looking at embedding the server, but at
this point that is not the case so the flag is probably unnecessary. I'll
remove it.
{quote}
dont quite know what the SimpleResolver can do. Does it behave like a normal
DNS server which can answer non-YARN queries ? I thought the flow is that if
the primary server cannot answer the query, that will be forwarded to yarn dns.
Not that yarn dns forward to the primary server.
{quote}
The SimpleResolver acts as a DNS client in this instance. I was exploring the
idea of allowing the YARN DNS server to server as a "primary" server by
indirectly supporting record retrieval for records outside of the YARN zone. I
now feel that is probably very unlikely, so I can remove that feature.
{quote}
After closer look, IIUC, this patch assumes the last entry of the zk path is
container Id only. If it is component name, then the
BaseServiceRecordProcessor#getContainerIDName will break.
{quote}
It is container ID in practice, which may actually conflict with the YARN
Registry documentation. I may have mis-spoken when I indicated the component
name is related in the path. So my belief is that the correct and is
correlated to the real work implementation.
> [Umbrella] Simplified discovery of services via DNS mechanisms
> --
>
> Key: YARN-4757
> URL: https://issues.apache.org/jira/browse/YARN-4757
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Jonathan Maron
> Attachments:
> 0001-YARN-4757-Initial-code-submission-for-DNS-Service.patch, YARN-4757-
> Simplified discovery of services via DNS mechanisms.pdf,
> YARN-4757-YARN-4757.001.patch, YARN-4757-YARN-4757.002.patch,
> YARN-4757-YARN-4757.003.patch
>
>
> [See overview doc at YARN-4692, copying the sub-section (3.2.10.2) to track
> all related efforts.]
> In addition to completing the present story of service-registry (YARN-913),
> we also need to simplify the access to the registry entries. The existing
> read mechanisms of the YARN Service Registry are currently limited to a
> registry specific (java) API and a REST interface. In practice, this makes it
> very difficult for wiring up existing clients and services. For e.g, dynamic
> configuration of dependent endpoints of a service is not easy to implement
> using the present registry-read mechanisms, *without* code-changes to
> existing services.
> A good solution to this is to expose the registry information through a more
> generic and widely used discovery mechanism: DNS. Service Discovery via DNS
> uses the well-known DNS interfaces to browse the network for services.
> YARN-913 in fact talked about such a DNS based mechanism but left it as a
> future task. (Task) Having the registry information exposed via DNS
> simplifies the life of services.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
[
https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316435#comment-15316435
]
Jonathan Maron commented on YARN-4757:
--
[~jianhe] - Thanks for the review! Answers:
{quote}
Here, it’s using the ‘description’ filed for constructing the DNS name. is this
expected ? seems not mentioned in the doc
{quote}
The description field is currently the service record field that Slider (the
current primary user of the ZK registry) is leveraging to relate the component
name (role). I assume the reason is because the ZK path to the node relates
the user, application, and component name. [~steve_l] - is that correct? Any
objection to an explicit "name" attribute in the service record?
{quote}
we can close the CuratorService#treeCache when stop the service?
{quote}
Probably makes sense.
{quote}
only readLock is being used. wondering whether these locks are needed.
{quote}
This is probably due to some refactorings - it was leveraged in previous
iterations of the code. I think I will re-introduce it. I see nothing in the
dnsjava code to indicate that Zone implementations are thread safe. Given the
dynamic nature of record registration and deletion in the yarn use case I think
it would be base to synchronize access to the zone object to ensure
deterministic results.
{quote}
question about the dnsEnabled config, if the dnsEnabled is false, what else
does the RegistryDNSServer do ? Asking this because I'm wondering whether this
config is actually needed.
{quote}
I guess this depends on what our expectations are regarding the use of DNS - is
it expected to be available as a default service. If the answer is no, we
could manage the inclusion of the service at this level or perhaps one level
(have the RM not even add the service based on flag value?)
{quote}
RecordCreatorFactory: The RecordCreatroFactory#getRecordCreator is
instantiating the creator instance every time this method gets called. May be
singleton pattern could be useful to avoid creating a new instance every time.
{quote}
Certainly a possibility. The thinking here was to create simple, lightweight,
stateless objects that could be used with little regard to multi-threading
concerns etc. However, if some profiling indicates an issue a singleton
approach may be preferable.
{quote}
DNSManagementOperations class is not used anywhere , can be removed?
{quote}
Yes - probably a leftover from a previous code iteration.
{quote}
a few unused methods in RegistryDNS, e.g. addDSRecord, signZones. is this
intended ?
{quote}
For the time being - yes. DS records appear to play a role in some DNS
negative response processing. Though we have made strides in better support
for negative responses (NXT records), it was still somewhat unclear whether we
ultimately would need to enhance support with full DS record capabilities. So
I have left these methods in place until such time that I could make a better
determination.
{quote}
what does the RegistryDNS#primaryDNS do ?
{quote}
The idea here was to support a use case in which the YARN DNS server was
designated as a resolver for a suite of hosts in the cluster. In those
instances, queries that it itself could not resolve would have to be forwarded
to a "primary" DNS server for resolution. I now think this probably less
likely, so we could certainly look at removing that feature.
> [Umbrella] Simplified discovery of services via DNS mechanisms
> --
>
> Key: YARN-4757
> URL: https://issues.apache.org/jira/browse/YARN-4757
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Jonathan Maron
> Attachments:
> 0001-YARN-4757-Initial-code-submission-for-DNS-Service.patch, YARN-4757-
> Simplified discovery of services via DNS mechanisms.pdf,
> YARN-4757-YARN-4757.001.patch, YARN-4757-YARN-4757.002.patch,
> YARN-4757-YARN-4757.003.patch
>
>
> [See overview doc at YARN-4692, copying the sub-section (3.2.10.2) to track
> all related efforts.]
> In addition to completing the present story of service-registry (YARN-913),
> we also need to simplify the access to the registry entries. The existing
> read mechanisms of the YARN Service Registry are currently limited to a
> registry specific (java) API and a REST interface. In practice, this makes it
> very difficult for wiring up existing clients and services. For e.g, dynamic
> configuration of dependent endpoints of a service is not easy to implement
> using the present registry-read mechanisms, *without* code-changes to
> existing services.
> A good solution to this is to expose the registry information through a more
> generic and widely used discovery mechanism: DNS. Service Discovery via DNS
> uses the well-known DNS
[jira] [Updated] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
[ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-4757: - Attachment: YARN-4757-YARN-4757.003.patch > [Umbrella] Simplified discovery of services via DNS mechanisms > -- > > Key: YARN-4757 > URL: https://issues.apache.org/jira/browse/YARN-4757 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Maron > Attachments: > 0001-YARN-4757-Initial-code-submission-for-DNS-Service.patch, YARN-4757- > Simplified discovery of services via DNS mechanisms.pdf, > YARN-4757-YARN-4757.001.patch, YARN-4757-YARN-4757.002.patch, > YARN-4757-YARN-4757.003.patch > > > [See overview doc at YARN-4692, copying the sub-section (3.2.10.2) to track > all related efforts.] > In addition to completing the present story of service-registry (YARN-913), > we also need to simplify the access to the registry entries. The existing > read mechanisms of the YARN Service Registry are currently limited to a > registry specific (java) API and a REST interface. In practice, this makes it > very difficult for wiring up existing clients and services. For e.g, dynamic > configuration of dependent endpoints of a service is not easy to implement > using the present registry-read mechanisms, *without* code-changes to > existing services. > A good solution to this is to expose the registry information through a more > generic and widely used discovery mechanism: DNS. Service Discovery via DNS > uses the well-known DNS interfaces to browse the network for services. > YARN-913 in fact talked about such a DNS based mechanism but left it as a > future task. (Task) Having the registry information exposed via DNS > simplifies the life of services. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
[ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-4757: - Attachment: YARN-4757-YARN-4757.002.patch Updated patch with minor fixes and more testing > [Umbrella] Simplified discovery of services via DNS mechanisms > -- > > Key: YARN-4757 > URL: https://issues.apache.org/jira/browse/YARN-4757 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Maron > Attachments: > 0001-YARN-4757-Initial-code-submission-for-DNS-Service.patch, YARN-4757- > Simplified discovery of services via DNS mechanisms.pdf, > YARN-4757-YARN-4757.001.patch, YARN-4757-YARN-4757.002.patch > > > [See overview doc at YARN-4692, copying the sub-section (3.2.10.2) to track > all related efforts.] > In addition to completing the present story of service-registry (YARN-913), > we also need to simplify the access to the registry entries. The existing > read mechanisms of the YARN Service Registry are currently limited to a > registry specific (java) API and a REST interface. In practice, this makes it > very difficult for wiring up existing clients and services. For e.g, dynamic > configuration of dependent endpoints of a service is not easy to implement > using the present registry-read mechanisms, *without* code-changes to > existing services. > A good solution to this is to expose the registry information through a more > generic and widely used discovery mechanism: DNS. Service Discovery via DNS > uses the well-known DNS interfaces to browse the network for services. > YARN-913 in fact talked about such a DNS based mechanism but left it as a > future task. (Task) Having the registry information exposed via DNS > simplifies the life of services. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
[ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-4757: - Attachment: 0001-YARN-4757-Initial-code-submission-for-DNS-Service.patch Updated patch consolidating three local commits and rebased to branch YARN-4757. > [Umbrella] Simplified discovery of services via DNS mechanisms > -- > > Key: YARN-4757 > URL: https://issues.apache.org/jira/browse/YARN-4757 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Maron > Attachments: > 0001-YARN-4757-Initial-code-submission-for-DNS-Service.patch, YARN-4757- > Simplified discovery of services via DNS mechanisms.pdf > > > [See overview doc at YARN-4692, copying the sub-section (3.2.10.2) to track > all related efforts.] > In addition to completing the present story of service-registry (YARN-913), > we also need to simplify the access to the registry entries. The existing > read mechanisms of the YARN Service Registry are currently limited to a > registry specific (java) API and a REST interface. In practice, this makes it > very difficult for wiring up existing clients and services. For e.g, dynamic > configuration of dependent endpoints of a service is not easy to implement > using the present registry-read mechanisms, *without* code-changes to > existing services. > A good solution to this is to expose the registry information through a more > generic and widely used discovery mechanism: DNS. Service Discovery via DNS > uses the well-known DNS interfaces to browse the network for services. > YARN-913 in fact talked about such a DNS based mechanism but left it as a > future task. (Task) Having the registry information exposed via DNS > simplifies the life of services. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
[ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-4757: - Attachment: (was: 0001-YARN-4757-Initial-code-submission-for-DNS-Service.patch) > [Umbrella] Simplified discovery of services via DNS mechanisms > -- > > Key: YARN-4757 > URL: https://issues.apache.org/jira/browse/YARN-4757 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Maron > Attachments: YARN-4757- Simplified discovery of services via DNS > mechanisms.pdf > > > [See overview doc at YARN-4692, copying the sub-section (3.2.10.2) to track > all related efforts.] > In addition to completing the present story of service-registry (YARN-913), > we also need to simplify the access to the registry entries. The existing > read mechanisms of the YARN Service Registry are currently limited to a > registry specific (java) API and a REST interface. In practice, this makes it > very difficult for wiring up existing clients and services. For e.g, dynamic > configuration of dependent endpoints of a service is not easy to implement > using the present registry-read mechanisms, *without* code-changes to > existing services. > A good solution to this is to expose the registry information through a more > generic and widely used discovery mechanism: DNS. Service Discovery via DNS > uses the well-known DNS interfaces to browse the network for services. > YARN-913 in fact talked about such a DNS based mechanism but left it as a > future task. (Task) Having the registry information exposed via DNS > simplifies the life of services. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4951) large IP ranges require a different naming strategy
[ https://issues.apache.org/jira/browse/YARN-4951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15296241#comment-15296241 ] Jonathan Maron commented on YARN-4951: -- I've been trying to work against the YARN-4757 branch, though I actually don't have branch committer permissions. I believe this patch is one of three commits that are stacked up waiting for an ability to commit to that branch. > large IP ranges require a different naming strategy > --- > > Key: YARN-4951 > URL: https://issues.apache.org/jira/browse/YARN-4951 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Maron >Assignee: Jonathan Maron > Attachments: > 0001-YARN-4757-simplified-reverse-lookup-zone-approach-fo.patch > > > Large subnet definitions (e.g. specifying a mask value of 255.255.224.0) > yield a large number of potential network addresses. Therefore, the standard > naming convention of xx.xx.xx.in-addr.arpa needs to be modified to be more > general: xx.xx.in-addr.arpa. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5076) YARN web interfaces lack XFS protection
[
https://issues.apache.org/jira/browse/YARN-5076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291572#comment-15291572
]
Jonathan Maron commented on YARN-5076:
--
Does a package-info.java file need to exist for
org.apache.hadoop.yarn.server.resourcemanager.webapp? I don't see one at the
moment.
{noformat}
HW10386:hadoop jmaron$ find . -name package-info.java | grep resource
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/package-info.java
{noformat}
> YARN web interfaces lack XFS protection
> ---
>
> Key: YARN-5076
> URL: https://issues.apache.org/jira/browse/YARN-5076
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager, resourcemanager, timelineserver
>Reporter: Jonathan Maron
>Assignee: Jonathan Maron
> Attachments: YARN-5076.002.patch, YARN-5076.003.patch,
> YARN-5076.004.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> There are web interfaces in YARN that do not provide protection against cross
> frame scripting
> (https://www.owasp.org/index.php/Clickjacking_Defense_Cheat_Sheet).
> HADOOP-13008 provides a common filter for addressing this vulnerability, so
> this filter should be integrated into the YARN web interfaces.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5076) YARN web interfaces lack XFS protection
[ https://issues.apache.org/jira/browse/YARN-5076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291474#comment-15291474 ] Jonathan Maron commented on YARN-5076: -- Will that address this problem? I have no idea why this would be required simply to accommodate a test class. > YARN web interfaces lack XFS protection > --- > > Key: YARN-5076 > URL: https://issues.apache.org/jira/browse/YARN-5076 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, resourcemanager, timelineserver >Reporter: Jonathan Maron >Assignee: Jonathan Maron > Attachments: YARN-5076.002.patch, YARN-5076.003.patch, > YARN-5076.004.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > There are web interfaces in YARN that do not provide protection against cross > frame scripting > (https://www.owasp.org/index.php/Clickjacking_Defense_Cheat_Sheet). > HADOOP-13008 provides a common filter for addressing this vulnerability, so > this filter should be integrated into the YARN web interfaces. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5076) YARN web interfaces lack XFS protection
[ https://issues.apache.org/jira/browse/YARN-5076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291162#comment-15291162 ] Jonathan Maron commented on YARN-5076: -- You kicked it off or do you want me to? I have been unsuccessful in the past... > YARN web interfaces lack XFS protection > --- > > Key: YARN-5076 > URL: https://issues.apache.org/jira/browse/YARN-5076 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, resourcemanager, timelineserver >Reporter: Jonathan Maron >Assignee: Jonathan Maron > Attachments: YARN-5076.002.patch, YARN-5076.003.patch, > YARN-5076.004.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > There are web interfaces in YARN that do not provide protection against cross > frame scripting > (https://www.owasp.org/index.php/Clickjacking_Defense_Cheat_Sheet). > HADOOP-13008 provides a common filter for addressing this vulnerability, so > this filter should be integrated into the YARN web interfaces. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5076) YARN web interfaces lack XFS protection
[ https://issues.apache.org/jira/browse/YARN-5076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291137#comment-15291137 ] Jonathan Maron commented on YARN-5076: -- The test actually verified that a value set for the property is returned as a header, so the value isn't as important as the fact that the values are the same. > YARN web interfaces lack XFS protection > --- > > Key: YARN-5076 > URL: https://issues.apache.org/jira/browse/YARN-5076 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, resourcemanager, timelineserver >Reporter: Jonathan Maron >Assignee: Jonathan Maron > Attachments: YARN-5076.002.patch, YARN-5076.003.patch, > YARN-5076.004.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > There are web interfaces in YARN that do not provide protection against cross > frame scripting > (https://www.owasp.org/index.php/Clickjacking_Defense_Cheat_Sheet). > HADOOP-13008 provides a common filter for addressing this vulnerability, so > this filter should be integrated into the YARN web interfaces. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5076) YARN web interfaces lack XFS protection
[ https://issues.apache.org/jira/browse/YARN-5076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291028#comment-15291028 ] Jonathan Maron commented on YARN-5076: -- Thanks - missed that. Will upload new patch. > YARN web interfaces lack XFS protection > --- > > Key: YARN-5076 > URL: https://issues.apache.org/jira/browse/YARN-5076 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, resourcemanager, timelineserver >Reporter: Jonathan Maron >Assignee: Jonathan Maron > Attachments: YARN-5076.002.patch, YARN-5076.003.patch, > YARN-5076.004.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > There are web interfaces in YARN that do not provide protection against cross > frame scripting > (https://www.owasp.org/index.php/Clickjacking_Defense_Cheat_Sheet). > HADOOP-13008 provides a common filter for addressing this vulnerability, so > this filter should be integrated into the YARN web interfaces. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5076) YARN web interfaces lack XFS protection
[ https://issues.apache.org/jira/browse/YARN-5076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-5076: - Attachment: YARN-5076.004.patch > YARN web interfaces lack XFS protection > --- > > Key: YARN-5076 > URL: https://issues.apache.org/jira/browse/YARN-5076 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, resourcemanager, timelineserver >Reporter: Jonathan Maron >Assignee: Jonathan Maron > Attachments: YARN-5076.002.patch, YARN-5076.003.patch, > YARN-5076.004.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > There are web interfaces in YARN that do not provide protection against cross > frame scripting > (https://www.owasp.org/index.php/Clickjacking_Defense_Cheat_Sheet). > HADOOP-13008 provides a common filter for addressing this vulnerability, so > this filter should be integrated into the YARN web interfaces. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5076) YARN web interfaces lack XFS protection
[ https://issues.apache.org/jira/browse/YARN-5076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-5076: - Attachment: YARN-5076.003.patch [~vvasudev] - please review and see if this addresses the changes requested. Thanks! > YARN web interfaces lack XFS protection > --- > > Key: YARN-5076 > URL: https://issues.apache.org/jira/browse/YARN-5076 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, resourcemanager, timelineserver >Reporter: Jonathan Maron >Assignee: Jonathan Maron > Attachments: YARN-5076.002.patch, YARN-5076.003.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > There are web interfaces in YARN that do not provide protection against cross > frame scripting > (https://www.owasp.org/index.php/Clickjacking_Defense_Cheat_Sheet). > HADOOP-13008 provides a common filter for addressing this vulnerability, so > this filter should be integrated into the YARN web interfaces. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5076) YARN web interfaces lack XFS protection
[ https://issues.apache.org/jira/browse/YARN-5076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289362#comment-15289362 ] Jonathan Maron commented on YARN-5076: -- [~djp] - if you can confirm that the settings [~vvasudev] lists above are a reflection of the way you'd like to see the configuration I can make the change - I have no real strong objection. I can look into making the default SAMEORIGIN. > YARN web interfaces lack XFS protection > --- > > Key: YARN-5076 > URL: https://issues.apache.org/jira/browse/YARN-5076 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, resourcemanager, timelineserver >Reporter: Jonathan Maron >Assignee: Jonathan Maron > Attachments: YARN-5076.002.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > There are web interfaces in YARN that do not provide protection against cross > frame scripting > (https://www.owasp.org/index.php/Clickjacking_Defense_Cheat_Sheet). > HADOOP-13008 provides a common filter for addressing this vulnerability, so > this filter should be integrated into the YARN web interfaces. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5076) YARN web interfaces lack XFS protection
[ https://issues.apache.org/jira/browse/YARN-5076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288999#comment-15288999 ] Jonathan Maron commented on YARN-5076: -- I agree with a [~vvasudev] - there are certainly use cases in which the granular control of this filter would be required. > YARN web interfaces lack XFS protection > --- > > Key: YARN-5076 > URL: https://issues.apache.org/jira/browse/YARN-5076 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, resourcemanager, timelineserver >Reporter: Jonathan Maron >Assignee: Jonathan Maron > Attachments: YARN-5076.002.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > There are web interfaces in YARN that do not provide protection against cross > frame scripting > (https://www.owasp.org/index.php/Clickjacking_Defense_Cheat_Sheet). > HADOOP-13008 provides a common filter for addressing this vulnerability, so > this filter should be integrated into the YARN web interfaces. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5076) YARN web interfaces lack XFS protection
[ https://issues.apache.org/jira/browse/YARN-5076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287399#comment-15287399 ] Jonathan Maron commented on YARN-5076: -- [~vvasudev] - the unit test failures above don't seem related to my changes. Is there anyway to find out if they are known issues? > YARN web interfaces lack XFS protection > --- > > Key: YARN-5076 > URL: https://issues.apache.org/jira/browse/YARN-5076 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, resourcemanager, timelineserver >Reporter: Jonathan Maron >Assignee: Jonathan Maron > Attachments: YARN-5076.002.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > There are web interfaces in YARN that do not provide protection against cross > frame scripting > (https://www.owasp.org/index.php/Clickjacking_Defense_Cheat_Sheet). > HADOOP-13008 provides a common filter for addressing this vulnerability, so > this filter should be integrated into the YARN web interfaces. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5076) YARN web interfaces lack XFS protection
[ https://issues.apache.org/jira/browse/YARN-5076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-5076: - Attachment: (was: YARN-5076.001.patch) > YARN web interfaces lack XFS protection > --- > > Key: YARN-5076 > URL: https://issues.apache.org/jira/browse/YARN-5076 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, resourcemanager, timelineserver >Reporter: Jonathan Maron >Assignee: Jonathan Maron > Attachments: YARN-5076.002.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > There are web interfaces in YARN that do not provide protection against cross > frame scripting > (https://www.owasp.org/index.php/Clickjacking_Defense_Cheat_Sheet). > HADOOP-13008 provides a common filter for addressing this vulnerability, so > this filter should be integrated into the YARN web interfaces. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
[ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15282968#comment-15282968 ] Jonathan Maron commented on YARN-4757: -- We did take a look - I even POC'd an approach. We found some limitations that pointed us in other directions: - No DNSSEC support - documentation indicated you should just turn it off ;) - Lack of support for some desirable record types (TXT, for example) - It is primarily a distributed key/value storage solution The last probably can't be considered a limitation, but it would be a somewhat extraneous facility in Hadoop. > [Umbrella] Simplified discovery of services via DNS mechanisms > -- > > Key: YARN-4757 > URL: https://issues.apache.org/jira/browse/YARN-4757 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Maron > Attachments: > 0001-YARN-4757-Initial-code-submission-for-DNS-Service.patch, YARN-4757- > Simplified discovery of services via DNS mechanisms.pdf > > > [See overview doc at YARN-4692, copying the sub-section (3.2.10.2) to track > all related efforts.] > In addition to completing the present story of service-registry (YARN-913), > we also need to simplify the access to the registry entries. The existing > read mechanisms of the YARN Service Registry are currently limited to a > registry specific (java) API and a REST interface. In practice, this makes it > very difficult for wiring up existing clients and services. For e.g, dynamic > configuration of dependent endpoints of a service is not easy to implement > using the present registry-read mechanisms, *without* code-changes to > existing services. > A good solution to this is to expose the registry information through a more > generic and widely used discovery mechanism: DNS. Service Discovery via DNS > uses the well-known DNS interfaces to browse the network for services. > YARN-913 in fact talked about such a DNS based mechanism but left it as a > future task. (Task) Having the registry information exposed via DNS > simplifies the life of services. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5076) YARN web interfaces lack XFS protection
[ https://issues.apache.org/jira/browse/YARN-5076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-5076: - Attachment: YARN-5076.002.patch Fix for NPE discovered through wider testing > YARN web interfaces lack XFS protection > --- > > Key: YARN-5076 > URL: https://issues.apache.org/jira/browse/YARN-5076 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, resourcemanager, timelineserver >Reporter: Jonathan Maron >Assignee: Jonathan Maron > Attachments: YARN-5076.001.patch, YARN-5076.002.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > There are web interfaces in YARN that do not provide protection against cross > frame scripting > (https://www.owasp.org/index.php/Clickjacking_Defense_Cheat_Sheet). > HADOOP-13008 provides a common filter for addressing this vulnerability, so > this filter should be integrated into the YARN web interfaces. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5076) YARN web interfaces lack XFS protection
[ https://issues.apache.org/jira/browse/YARN-5076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-5076: - Attachment: YARN-5076.001.patch This patch integrated the XFrame Options Filter from Hadoop Common. Some notes: - The filter is enabled by default. It can be disabled via 'xframe-options-enabled' properties (properties have been created for all web interfaces in this integration) - I've run checkstyle locally and there are no new style issues introduced by the code modified or added in this patch > YARN web interfaces lack XFS protection > --- > > Key: YARN-5076 > URL: https://issues.apache.org/jira/browse/YARN-5076 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, resourcemanager, timelineserver >Reporter: Jonathan Maron >Assignee: Jonathan Maron > Attachments: YARN-5076.001.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > There are web interfaces in YARN that do not provide protection against cross > frame scripting > (https://www.owasp.org/index.php/Clickjacking_Defense_Cheat_Sheet). > HADOOP-13008 provides a common filter for addressing this vulnerability, so > this filter should be integrated into the YARN web interfaces. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5076) YARN web interfaces lack XFS protection
Jonathan Maron created YARN-5076: Summary: YARN web interfaces lack XFS protection Key: YARN-5076 URL: https://issues.apache.org/jira/browse/YARN-5076 Project: Hadoop YARN Issue Type: Bug Components: nodemanager, resourcemanager, timelineserver Reporter: Jonathan Maron Assignee: Jonathan Maron There are web interfaces in YARN that do not provide protection against cross frame scripting (https://www.owasp.org/index.php/Clickjacking_Defense_Cheat_Sheet). HADOOP-13008 provides a common filter for addressing this vulnerability, so this filter should be integrated into the YARN web interfaces. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4951) large IP ranges require a different naming strategy
[ https://issues.apache.org/jira/browse/YARN-4951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-4951: - Attachment: 0001-YARN-4757-simplified-reverse-lookup-zone-approach-fo.patch > large IP ranges require a different naming strategy > --- > > Key: YARN-4951 > URL: https://issues.apache.org/jira/browse/YARN-4951 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Maron >Assignee: Jonathan Maron > Attachments: > 0001-YARN-4757-simplified-reverse-lookup-zone-approach-fo.patch > > > Large subnet definitions (e.g. specifying a mask value of 255.255.224.0) > yield a large number of potential network addresses. Therefore, the standard > naming convention of xx.xx.xx.in-addr.arpa needs to be modified to be more > general: xx.xx.in-addr.arpa. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4951) large IP ranges require a different naming strategy
[ https://issues.apache.org/jira/browse/YARN-4951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15240129#comment-15240129 ] Jonathan Maron commented on YARN-4951: -- The original approach was unnecessary. Uploaded a new patch that addresses the need to simply generalize the reverse lookup zone name. > large IP ranges require a different naming strategy > --- > > Key: YARN-4951 > URL: https://issues.apache.org/jira/browse/YARN-4951 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Maron >Assignee: Jonathan Maron > > Large subnet definitions (e.g. specifying a mask value of 255.255.224.0) > yield a large number of potential network addresses. Therefore, the standard > naming convention of xx.xx.xx.in-addr.arpa needs to be modified to be more > general: xx.xx.in-addr.arpa. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4951) large IP ranges require a different naming strategy
[ https://issues.apache.org/jira/browse/YARN-4951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-4951: - Description: Large subnet definitions (e.g. specifying a mask value of 255.255.224.0) yield a large number of potential network addresses. Therefore, the standard naming convention of xx.xx.xx.in-addr.arpa needs to be modified to be more general: xx.xx.in-addr.arpa. (was: Large subnet definitions (e.g. specifying a mask value of 255.255.224.0) yield a large number of potential network addresses, each requiring a separate reverse zone definition (given that reverse zones include the first 3 IP bytes in reverse order).) > large IP ranges require a different naming strategy > --- > > Key: YARN-4951 > URL: https://issues.apache.org/jira/browse/YARN-4951 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Maron >Assignee: Jonathan Maron > > Large subnet definitions (e.g. specifying a mask value of 255.255.224.0) > yield a large number of potential network addresses. Therefore, the standard > naming convention of xx.xx.xx.in-addr.arpa needs to be modified to be more > general: xx.xx.in-addr.arpa. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4951) large IP ranges require a different naming strategy
[ https://issues.apache.org/jira/browse/YARN-4951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-4951: - Attachment: (was: 0001-YARN-4757-address-multiple-reverse-lookup-zones-and-.patch) > large IP ranges require a different naming strategy > --- > > Key: YARN-4951 > URL: https://issues.apache.org/jira/browse/YARN-4951 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Maron >Assignee: Jonathan Maron > > Large subnet definitions (e.g. specifying a mask value of 255.255.224.0) > yield a large number of potential network addresses. Therefore, the standard > naming convention of xx.xx.xx.in-addr.arpa needs to be modified to be more > general: xx.xx.in-addr.arpa. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4951) large IP ranges require a different naming strategy
[ https://issues.apache.org/jira/browse/YARN-4951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-4951: - Summary: large IP ranges require a different naming strategy (was: large IP ranges require the creation of multiple reverse lookup zones) > large IP ranges require a different naming strategy > --- > > Key: YARN-4951 > URL: https://issues.apache.org/jira/browse/YARN-4951 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Maron >Assignee: Jonathan Maron > Attachments: > 0001-YARN-4757-address-multiple-reverse-lookup-zones-and-.patch > > > Large subnet definitions (e.g. specifying a mask value of 255.255.224.0) > yield a large number of potential network addresses, each requiring a > separate reverse zone definition (given that reverse zones include the first > 3 IP bytes in reverse order). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4951) large IP ranges require the creation of multiple reverse lookup zones
[ https://issues.apache.org/jira/browse/YARN-4951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-4951: - Attachment: 0001-YARN-4757-address-multiple-reverse-lookup-zones-and-.patch An approach that: 1) Add config properties for netmask, IP range min, and IP range max 2) Uses SubnetUtils to find the list addresses based on subnet and mask, selects the network addresses (end in ".0") for resulting list, and allows for the addresses to be filtered based on range values. > large IP ranges require the creation of multiple reverse lookup zones > - > > Key: YARN-4951 > URL: https://issues.apache.org/jira/browse/YARN-4951 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Maron >Assignee: Jonathan Maron > Attachments: > 0001-YARN-4757-address-multiple-reverse-lookup-zones-and-.patch > > > Large subnet definitions (e.g. specifying a mask value of 255.255.224.0) > yield a large number of potential network addresses, each requiring a > separate reverse zone definition (given that reverse zones include the first > 3 IP bytes in reverse order). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4952) need configuration mechanism for specifying per-host network interface
Jonathan Maron created YARN-4952: Summary: need configuration mechanism for specifying per-host network interface Key: YARN-4952 URL: https://issues.apache.org/jira/browse/YARN-4952 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jonathan Maron Assignee: Jonathan Maron The initial configuration approach for the DNS service specified a bind-address that designated the network interface to which the service should bind its listener port. However, there is a need to potentially specify multiple DNS service instances (HA approach) and therefore a need to specify bind addresses for each instance (and those interfaces may vary between hosts). This may take a for similar to the RM HA approach (rm1, rm2) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4951) large IP ranges require the creation of multiple reverse lookup zones
Jonathan Maron created YARN-4951: Summary: large IP ranges require the creation of multiple reverse lookup zones Key: YARN-4951 URL: https://issues.apache.org/jira/browse/YARN-4951 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jonathan Maron Assignee: Jonathan Maron Large subnet definitions (e.g. specifying a mask value of 255.255.224.0) yield a large number of potential network addresses, each requiring a separate reverse zone definition (given that reverse zones include the first 3 IP bytes in reverse order). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
[ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-4757: - Attachment: 0001-YARN-4757-Initial-code-submission-for-DNS-Service.patch While I await branch-committer status, I am uploading an initial patch that should give a more concrete sense of a DNS service implementation. I made use of the dnsjava library and implemented a good portion of the specification. I plan to provide relatively frequent updates and record sub-tasks to address any issues unearthed. > [Umbrella] Simplified discovery of services via DNS mechanisms > -- > > Key: YARN-4757 > URL: https://issues.apache.org/jira/browse/YARN-4757 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Maron > Attachments: > 0001-YARN-4757-Initial-code-submission-for-DNS-Service.patch, YARN-4757- > Simplified discovery of services via DNS mechanisms.pdf > > > [See overview doc at YARN-4692, copying the sub-section (3.2.10.2) to track > all related efforts.] > In addition to completing the present story of service-registry (YARN-913), > we also need to simplify the access to the registry entries. The existing > read mechanisms of the YARN Service Registry are currently limited to a > registry specific (java) API and a REST interface. In practice, this makes it > very difficult for wiring up existing clients and services. For e.g, dynamic > configuration of dependent endpoints of a service is not easy to implement > using the present registry-read mechanisms, *without* code-changes to > existing services. > A good solution to this is to expose the registry information through a more > generic and widely used discovery mechanism: DNS. Service Discovery via DNS > uses the well-known DNS interfaces to browse the network for services. > YARN-913 in fact talked about such a DNS based mechanism but left it as a > future task. (Task) Having the registry information exposed via DNS > simplifies the life of services. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
[ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235712#comment-15235712 ] Jonathan Maron commented on YARN-4757: -- thanks! I believe I'll need to be voted on as a branch committer to commit my work... > [Umbrella] Simplified discovery of services via DNS mechanisms > -- > > Key: YARN-4757 > URL: https://issues.apache.org/jira/browse/YARN-4757 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Maron > Attachments: YARN-4757- Simplified discovery of services via DNS > mechanisms.pdf > > > [See overview doc at YARN-4692, copying the sub-section (3.2.10.2) to track > all related efforts.] > In addition to completing the present story of service-registry (YARN-913), > we also need to simplify the access to the registry entries. The existing > read mechanisms of the YARN Service Registry are currently limited to a > registry specific (java) API and a REST interface. In practice, this makes it > very difficult for wiring up existing clients and services. For e.g, dynamic > configuration of dependent endpoints of a service is not easy to implement > using the present registry-read mechanisms, *without* code-changes to > existing services. > A good solution to this is to expose the registry information through a more > generic and widely used discovery mechanism: DNS. Service Discovery via DNS > uses the well-known DNS interfaces to browse the network for services. > YARN-913 in fact talked about such a DNS based mechanism but left it as a > future task. (Task) Having the registry information exposed via DNS > simplifies the life of services. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
[ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15232423#comment-15232423 ] Jonathan Maron commented on YARN-4757: -- Filed a sub task to explicitly assess the zone transfer approach. > [Umbrella] Simplified discovery of services via DNS mechanisms > -- > > Key: YARN-4757 > URL: https://issues.apache.org/jira/browse/YARN-4757 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Maron > Attachments: YARN-4757- Simplified discovery of services via DNS > mechanisms.pdf > > > [See overview doc at YARN-4692, copying the sub-section (3.2.10.2) to track > all related efforts.] > In addition to completing the present story of service-registry (YARN-913), > we also need to simplify the access to the registry entries. The existing > read mechanisms of the YARN Service Registry are currently limited to a > registry specific (java) API and a REST interface. In practice, this makes it > very difficult for wiring up existing clients and services. For e.g, dynamic > configuration of dependent endpoints of a service is not easy to implement > using the present registry-read mechanisms, *without* code-changes to > existing services. > A good solution to this is to expose the registry information through a more > generic and widely used discovery mechanism: DNS. Service Discovery via DNS > uses the well-known DNS interfaces to browse the network for services. > YARN-913 in fact talked about such a DNS based mechanism but left it as a > future task. (Task) Having the registry information exposed via DNS > simplifies the life of services. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4933) Evaluate parent-slave DNS options to assess deployment options for DNS service
Jonathan Maron created YARN-4933: Summary: Evaluate parent-slave DNS options to assess deployment options for DNS service Key: YARN-4933 URL: https://issues.apache.org/jira/browse/YARN-4933 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jonathan Maron Assignee: Jonathan Maron Comments on YARN-4757 indicate that, in addition to the primary server to YARN DNS service zone request forwarding implementation currently suggested, it may be appropriate to also offer the ability to configure the DNS service as a master server that can support zone transfers to slaves. Some other features that are related and should be examined are: - DNS NOTIFY - AXFR - IXFR -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
[ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15230523#comment-15230523 ] Jonathan Maron commented on YARN-4757: -- Sounds like a good approach to me. > [Umbrella] Simplified discovery of services via DNS mechanisms > -- > > Key: YARN-4757 > URL: https://issues.apache.org/jira/browse/YARN-4757 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Maron > Attachments: YARN-4757- Simplified discovery of services via DNS > mechanisms.pdf > > > [See overview doc at YARN-4692, copying the sub-section (3.2.10.2) to track > all related efforts.] > In addition to completing the present story of service-registry (YARN-913), > we also need to simplify the access to the registry entries. The existing > read mechanisms of the YARN Service Registry are currently limited to a > registry specific (java) API and a REST interface. In practice, this makes it > very difficult for wiring up existing clients and services. For e.g, dynamic > configuration of dependent endpoints of a service is not easy to implement > using the present registry-read mechanisms, *without* code-changes to > existing services. > A good solution to this is to expose the registry information through a more > generic and widely used discovery mechanism: DNS. Service Discovery via DNS > uses the well-known DNS interfaces to browse the network for services. > YARN-913 in fact talked about such a DNS based mechanism but left it as a > future task. (Task) Having the registry information exposed via DNS > simplifies the life of services. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
[ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226722#comment-15226722 ] Jonathan Maron commented on YARN-4757: -- That's great information. Thanks! I'll start digging into zone transfer support. > [Umbrella] Simplified discovery of services via DNS mechanisms > -- > > Key: YARN-4757 > URL: https://issues.apache.org/jira/browse/YARN-4757 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Maron > Attachments: YARN-4757- Simplified discovery of services via DNS > mechanisms.pdf > > > [See overview doc at YARN-4692, copying the sub-section (3.2.10.2) to track > all related efforts.] > In addition to completing the present story of service-registry (YARN-913), > we also need to simplify the access to the registry entries. The existing > read mechanisms of the YARN Service Registry are currently limited to a > registry specific (java) API and a REST interface. In practice, this makes it > very difficult for wiring up existing clients and services. For e.g, dynamic > configuration of dependent endpoints of a service is not easy to implement > using the present registry-read mechanisms, *without* code-changes to > existing services. > A good solution to this is to expose the registry information through a more > generic and widely used discovery mechanism: DNS. Service Discovery via DNS > uses the well-known DNS interfaces to browse the network for services. > YARN-913 in fact talked about such a DNS based mechanism but left it as a > future task. (Task) Having the registry information exposed via DNS > simplifies the life of services. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
[ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226326#comment-15226326 ] Jonathan Maron commented on YARN-4757: -- two questions: 1) May I ask why? If a zone key is created and properly configured, isn't the authenticity of the records assured? 2) If this truly a non-starter for many environments - what is the alternative? Do we need to require the support of secure zone transfers? > [Umbrella] Simplified discovery of services via DNS mechanisms > -- > > Key: YARN-4757 > URL: https://issues.apache.org/jira/browse/YARN-4757 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Maron > Attachments: YARN-4757- Simplified discovery of services via DNS > mechanisms.pdf > > > [See overview doc at YARN-4692, copying the sub-section (3.2.10.2) to track > all related efforts.] > In addition to completing the present story of service-registry (YARN-913), > we also need to simplify the access to the registry entries. The existing > read mechanisms of the YARN Service Registry are currently limited to a > registry specific (java) API and a REST interface. In practice, this makes it > very difficult for wiring up existing clients and services. For e.g, dynamic > configuration of dependent endpoints of a service is not easy to implement > using the present registry-read mechanisms, *without* code-changes to > existing services. > A good solution to this is to expose the registry information through a more > generic and widely used discovery mechanism: DNS. Service Discovery via DNS > uses the well-known DNS interfaces to browse the network for services. > YARN-913 in fact talked about such a DNS based mechanism but left it as a > future task. (Task) Having the registry information exposed via DNS > simplifies the life of services. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
[ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226267#comment-15226267 ] Jonathan Maron commented on YARN-4757: -- So in my internal testing I setup the following (this is the approach related in the "High Availability" section): 1) Run an instance of the DNS service serving up records of the hadoop cluster and configured with a specific zone name 2) Configured a BIND server with a forwarding zone configuration, forwarding all requests for the hadoop configured zone to the DNS service instance 3) Queries to the BIND server for cluster related records are returned successfully This also assumes that the DNSSEC setup is correct (zone key created and configured on both ends). Assuming that a corporate server is configured appropriately to serve external requests, the hadoop cluster zone records should be available externally, I believe (this is one thing I haven't had the ability to test). > [Umbrella] Simplified discovery of services via DNS mechanisms > -- > > Key: YARN-4757 > URL: https://issues.apache.org/jira/browse/YARN-4757 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Maron > Attachments: YARN-4757- Simplified discovery of services via DNS > mechanisms.pdf > > > [See overview doc at YARN-4692, copying the sub-section (3.2.10.2) to track > all related efforts.] > In addition to completing the present story of service-registry (YARN-913), > we also need to simplify the access to the registry entries. The existing > read mechanisms of the YARN Service Registry are currently limited to a > registry specific (java) API and a REST interface. In practice, this makes it > very difficult for wiring up existing clients and services. For e.g, dynamic > configuration of dependent endpoints of a service is not easy to implement > using the present registry-read mechanisms, *without* code-changes to > existing services. > A good solution to this is to expose the registry information through a more > generic and widely used discovery mechanism: DNS. Service Discovery via DNS > uses the well-known DNS interfaces to browse the network for services. > YARN-913 in fact talked about such a DNS based mechanism but left it as a > future task. (Task) Having the registry information exposed via DNS > simplifies the life of services. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
[ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15224524#comment-15224524 ] Jonathan Maron commented on YARN-4757: -- Are you looking for examples here (as a response) or suggesting those examples should exist in the specification? And by examples, I imagine you mean a clear delineation of these possible race conditions and suggestions as to how to manage the risks (i.e. you are not suggesting that there is a DNS-based solution)? > [Umbrella] Simplified discovery of services via DNS mechanisms > -- > > Key: YARN-4757 > URL: https://issues.apache.org/jira/browse/YARN-4757 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Maron > Attachments: YARN-4757- Simplified discovery of services via DNS > mechanisms.pdf > > > [See overview doc at YARN-4692, copying the sub-section (3.2.10.2) to track > all related efforts.] > In addition to completing the present story of service-registry (YARN-913), > we also need to simplify the access to the registry entries. The existing > read mechanisms of the YARN Service Registry are currently limited to a > registry specific (java) API and a REST interface. In practice, this makes it > very difficult for wiring up existing clients and services. For e.g, dynamic > configuration of dependent endpoints of a service is not easy to implement > using the present registry-read mechanisms, *without* code-changes to > existing services. > A good solution to this is to expose the registry information through a more > generic and widely used discovery mechanism: DNS. Service Discovery via DNS > uses the well-known DNS interfaces to browse the network for services. > YARN-913 in fact talked about such a DNS based mechanism but left it as a > future task. (Task) Having the registry information exposed via DNS > simplifies the life of services. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
[ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15214216#comment-15214216 ] Jonathan Maron commented on YARN-4757: -- Actually, I forgot that there is a yarn:persistence attribute that allows you to distinguish the record type, so it is probably a valid indicator as to the appropriate parsing. Current defined types are "application" and "container". I'll explore the possibility of leveraging both persistence type and JSON element type (e.g. "API") to clarify the mappings. > [Umbrella] Simplified discovery of services via DNS mechanisms > -- > > Key: YARN-4757 > URL: https://issues.apache.org/jira/browse/YARN-4757 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Maron > Attachments: YARN-4757- Simplified discovery of services via DNS > mechanisms.pdf > > > [See overview doc at YARN-4692, copying the sub-section (3.2.10.2) to track > all related efforts.] > In addition to completing the present story of service-registry (YARN-913), > we also need to simplify the access to the registry entries. The existing > read mechanisms of the YARN Service Registry are currently limited to a > registry specific (java) API and a REST interface. In practice, this makes it > very difficult for wiring up existing clients and services. For e.g, dynamic > configuration of dependent endpoints of a service is not easy to implement > using the present registry-read mechanisms, *without* code-changes to > existing services. > A good solution to this is to expose the registry information through a more > generic and widely used discovery mechanism: DNS. Service Discovery via DNS > uses the well-known DNS interfaces to browse the network for services. > YARN-913 in fact talked about such a DNS based mechanism but left it as a > future task. (Task) Having the registry information exposed via DNS > simplifies the life of services. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
[
https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15214175#comment-15214175
]
Jonathan Maron commented on YARN-4757:
--
{quote}
As for the SRV records and which IP address is returned it might be good to
make it more clear in the document what you are proposing.
{quote}
That is a good point. I think you've actually found a flaw in the document. I
was focused on the current Slider Service Record structure, which leverages the
defined JSON structures in specific ways to define the application endpoints
and containers that make up a Slider application. I think it may be clearer to
simply map the service record JSON objects to their corresponding DNS records,
allowing future applications to structure their Service Records as needed with
a clear understanding of the DNS records that will end up in the DNS service
based on those records.
> [Umbrella] Simplified discovery of services via DNS mechanisms
> --
>
> Key: YARN-4757
> URL: https://issues.apache.org/jira/browse/YARN-4757
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Jonathan Maron
> Attachments: YARN-4757- Simplified discovery of services via DNS
> mechanisms.pdf
>
>
> [See overview doc at YARN-4692, copying the sub-section (3.2.10.2) to track
> all related efforts.]
> In addition to completing the present story of service-registry (YARN-913),
> we also need to simplify the access to the registry entries. The existing
> read mechanisms of the YARN Service Registry are currently limited to a
> registry specific (java) API and a REST interface. In practice, this makes it
> very difficult for wiring up existing clients and services. For e.g, dynamic
> configuration of dependent endpoints of a service is not easy to implement
> using the present registry-read mechanisms, *without* code-changes to
> existing services.
> A good solution to this is to expose the registry information through a more
> generic and widely used discovery mechanism: DNS. Service Discovery via DNS
> uses the well-known DNS interfaces to browse the network for services.
> YARN-913 in fact talked about such a DNS based mechanism but left it as a
> future task. (Task) Having the registry information exposed via DNS
> simplifies the life of services.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
[ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15214170#comment-15214170 ] Jonathan Maron commented on YARN-4757: -- Let me respond with a deployment option that would address the concern, though I doubt we can try address your security concerns given the inherent nature of DNS: Given the scenario you describe, I believe a more viable/secure approach would be to: 1) Deploy a load balancer fronting the web services 2) Web services would come up as YARN components, assigned ports that are available on the given hosts (not necessarily ) and advertising their address leveraging the registry and DNS service 3) It would be up to the load balancer to discover the available hosts (lookup name, test connection etc), based on configuration (e.g. the names of components to lookup etc) The load balancer is on a trusted host and is configured "outside" the YARN cluster, so the opportunity for a "spoofing" server is mitigated. For a given deployment, the location (SRV record) of the load balancer could be statically defined in a DNS zone file that is configured as the initial configuration file for the DNS service. This scheme would require some configuration to come up with appropriate TTL values etc. Interestingly enough, I've had some discussion in the past with some of the Knox developers about its potential role in the management of connections to dynamically deployed cluster elements... > [Umbrella] Simplified discovery of services via DNS mechanisms > -- > > Key: YARN-4757 > URL: https://issues.apache.org/jira/browse/YARN-4757 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Maron > Attachments: YARN-4757- Simplified discovery of services via DNS > mechanisms.pdf > > > [See overview doc at YARN-4692, copying the sub-section (3.2.10.2) to track > all related efforts.] > In addition to completing the present story of service-registry (YARN-913), > we also need to simplify the access to the registry entries. The existing > read mechanisms of the YARN Service Registry are currently limited to a > registry specific (java) API and a REST interface. In practice, this makes it > very difficult for wiring up existing clients and services. For e.g, dynamic > configuration of dependent endpoints of a service is not easy to implement > using the present registry-read mechanisms, *without* code-changes to > existing services. > A good solution to this is to expose the registry information through a more > generic and widely used discovery mechanism: DNS. Service Discovery via DNS > uses the well-known DNS interfaces to browse the network for services. > YARN-913 in fact talked about such a DNS based mechanism but left it as a > future task. (Task) Having the registry information exposed via DNS > simplifies the life of services. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
[
https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209288#comment-15209288
]
Jonathan Maron commented on YARN-4757:
--
{quote}
Are there situations where you would just return the IP Address of the node the
container is running on?
{quote}
One situration I can readily think of is YARN linux containers etc that are
not assigned an IP. The appropriate way to manage those should be considered
(I can add to Open Issues on next revision)
{quote}
Does that mean that we will return records for any service API no matter how
the IP Addresses are assigned, or there is no way for the IP Address to not be
available?
{quote}
Application records are generally associated with the AM and the host on which
it resided (at least that's true of the Slider use cases that are the only ones
currently making use of the YARN registry and service records). So most to the
CNAME/TXT records mapping to an API will leverage that host IP.
{quote}
How is authentication with zookeeper handled? Is it always SASL+kerberos?
{quote}
Probably best to just point you to this writeup:
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/registry/registry-security.html
{quote}
Would we be exposing SRV records for both of these combinations? If so how
would they be named?
{quote}
Yes. The current design calls for the creation and registration of SRV records
that have both the application name as well as the API names
{quote}
I am not an expert on DNS so if I say something silly after you stop laughing
please let me know
{quote}
I have been working with dnsjava and BIND trying to learn the internals for the
last few months, so I'm by no means an expert. And I'm not going to laugh - if
anything I'm going to thank you profusely for the help!
{quote}
What about limits on the number of IP addresses that can be returned for a
given name. I could not find anything specific but I have to assume that in
practice most systems don't support a huge number of these, and large clusters
on YARN can easily launch hundreds or even thousands of containers for a given
service.
{quote}
I'd have to look into the relevant RFC's and other literature to see if there
is a length limit. Generally documentation point to the host name RFC (1123?).
I think limits on length of name would also be dictated by other software
products (DBs etc). So we'd have to consider any "shortening" that may be
required.
You can have multiple addresses mapped to a single name, e.g.
{code}
HW10386:hadoop jmaron$ nslookup www.google.com
Server: 192.168.1.1
Address:192.168.1.1#53
Non-authoritative answer:
Name: www.google.com
Address: 63.117.14.150
Name: www.google.com
Address: 63.117.14.151
Name: www.google.com
Address: 63.117.14.155
Name: www.google.com
Address: 63.117.14.154
Name: www.google.com
Address: 63.117.14.153
Name: www.google.com
Address: 63.117.14.148
Name: www.google.com
Address: 63.117.14.149
Name: www.google.com
Address: 63.117.14.152
{code}
So, some of the naming conventions (e.g. component name) may point to multiple
container IPs. Addressing that through component name uniqueness (there is a
slider JIRA for that) may be one possibility.
{quote}
In addition to Allen's concerns the document does not seem to address/call out
my initial concerns about requiring mutual authentication, or handling of port
availability in scheduling.
{quote}
I'm going to need a little more help in understanding these concerns. The
approach we provide is targeted at supporting standard DNS clients, and DNS
does not provide for mutual authentication - the concept of restricting who can
query the DNS for records is considered outside the scope of the protocol. As
for port availability - currently the DNS implementation is targeted at
relaying the port assignments as designated by YARN scheduler, rather than
actively participating in the scheduling itself. So I assume I'm
misunderstanding
> [Umbrella] Simplified discovery of services via DNS mechanisms
> --
>
> Key: YARN-4757
> URL: https://issues.apache.org/jira/browse/YARN-4757
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Jonathan Maron
> Attachments: YARN-4757- Simplified discovery of services via DNS
> mechanisms.pdf
>
>
> [See overview doc at YARN-4692, copying the sub-section (3.2.10.2) to track
> all related efforts.]
> In addition to completing the present story of service-registry (YARN-913),
> we also need to simplify the access to the registry entries. The existing
> read mechanisms of the YARN Service Registry are currently limited to a
> registry specific (java) API and a REST interface. In practice, this makes it
>
[jira] [Commented] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
[ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15208950#comment-15208950 ] Jonathan Maron commented on YARN-4757: -- a) A records are more usable from an existing client interaction perspective. For example, you can use a tool such as nslookup to map from a known name to its IP. You could potentially leverage an SRV record in that instance, but you'd have to go into the interactive mode of nslookup, set the type, and then perform the query - a less intuitive and well known approach. b) It's not a matter of managing a named.conf file as much as setting up bind to support the dynamic update protocol (YARN containers will come up and go down and those record updates may be relatively frequent). In addition, the stateful complaint has more to do with the need to synch state in multiple processes rather than rely on one source of truth. Finally, the security needs for an internal zone server are finite enough that, if security was the primary driver, would make the BIND selection overkill. c) Not familiar with manta (even initial web searches didn't seem to bring anything up)? If there is an open source, available solution I'd be more happy to evaluate its potential use. d) I'm not sure the problem is necessarily solved. DNS is well understood, obviously. But the use case here - mirroring the details of an existing ZK-based registry or, more accurately, the state of the YARN cluster - present some requirements that perhaps can be best addressed by a tailored solution. Given the availability of APIs such as dnsjava etc. the approach is not necessarily daunting from a development perspective. As such, testing can be performed to address security and performance concerns, though I'm not naive - I understand some issues will not manifest till actual deployment. > [Umbrella] Simplified discovery of services via DNS mechanisms > -- > > Key: YARN-4757 > URL: https://issues.apache.org/jira/browse/YARN-4757 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Maron > Attachments: YARN-4757- Simplified discovery of services via DNS > mechanisms.pdf > > > [See overview doc at YARN-4692, copying the sub-section (3.2.10.2) to track > all related efforts.] > In addition to completing the present story of service-registry (YARN-913), > we also need to simplify the access to the registry entries. The existing > read mechanisms of the YARN Service Registry are currently limited to a > registry specific (java) API and a REST interface. In practice, this makes it > very difficult for wiring up existing clients and services. For e.g, dynamic > configuration of dependent endpoints of a service is not easy to implement > using the present registry-read mechanisms, *without* code-changes to > existing services. > A good solution to this is to expose the registry information through a more > generic and widely used discovery mechanism: DNS. Service Discovery via DNS > uses the well-known DNS interfaces to browse the network for services. > YARN-913 in fact talked about such a DNS based mechanism but left it as a > future task. (Task) Having the registry information exposed via DNS > simplifies the life of services. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
[ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-4757: - Attachment: YARN-4757- Simplified discovery of services via DNS mechanisms.pdf I’ve posted a document providing greater detail concerning this effort. It is intended as a description of the background, a proposed architectural approach, implementation details, and some open issues. I've already had some initial reviews that were of great help in both describing existing points and identifying additional ones. /cc [~vvasudev], [~vinodkv], [~sidharta-s], [~ste...@apache.org], [~elserj] > [Umbrella] Simplified discovery of services via DNS mechanisms > -- > > Key: YARN-4757 > URL: https://issues.apache.org/jira/browse/YARN-4757 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Maron > Attachments: YARN-4757- Simplified discovery of services via DNS > mechanisms.pdf > > > [See overview doc at YARN-4692, copying the sub-section (3.2.10.2) to track > all related efforts.] > In addition to completing the present story of service-registry (YARN-913), > we also need to simplify the access to the registry entries. The existing > read mechanisms of the YARN Service Registry are currently limited to a > registry specific (java) API and a REST interface. In practice, this makes it > very difficult for wiring up existing clients and services. For e.g, dynamic > configuration of dependent endpoints of a service is not easy to implement > using the present registry-read mechanisms, *without* code-changes to > existing services. > A good solution to this is to expose the registry information through a more > generic and widely used discovery mechanism: DNS. Service Discovery via DNS > uses the well-known DNS interfaces to browse the network for services. > YARN-913 in fact talked about such a DNS based mechanism but left it as a > future task. (Task) Having the registry information exposed via DNS > simplifies the life of services. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
[ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15194047#comment-15194047 ] Jonathan Maron commented on YARN-4757: -- I'm trying to address all of these issues/concerns in the document I reference above - it'll probably be a good way to structure the discussion. I hope to have it posted to this JIRA this week. Some quick points: - I'm trying to address security by leveraging the existing DNS security extensions (DNSSEC). The exposed DNS facility will have to accommodate both Java and non-Java clients, and as such should probably not provide proprietary or non-compliant security mechanisms. In addition, for the DNS facility will more than likely need to interoperate with existing DNS resources (e.g. a corporate BIND server). DNS security is structured more around the idea of validating the authenticity of returned information rather than authenticating identities. In addition, I believe the approach I'm proposing will address the authentication concerns. - As Allen mentioned - there are existing approaches for interacting with DNS name servers. I have been utilizing dnsjava to prototype some approaches. > [Umbrella] Simplified discovery of services via DNS mechanisms > -- > > Key: YARN-4757 > URL: https://issues.apache.org/jira/browse/YARN-4757 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Maron > > [See overview doc at YARN-4692, copying the sub-section (3.2.10.2) to track > all related efforts.] > In addition to completing the present story of service-registry (YARN-913), > we also need to simplify the access to the registry entries. The existing > read mechanisms of the YARN Service Registry are currently limited to a > registry specific (java) API and a REST interface. In practice, this makes it > very difficult for wiring up existing clients and services. For e.g, dynamic > configuration of dependent endpoints of a service is not easy to implement > using the present registry-read mechanisms, *without* code-changes to > existing services. > A good solution to this is to expose the registry information through a more > generic and widely used discovery mechanism: DNS. Service Discovery via DNS > uses the well-known DNS interfaces to browse the network for services. > YARN-913 in fact talked about such a DNS based mechanism but left it as a > future task. (Task) Having the registry information exposed via DNS > simplifies the life of services. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4737) Use CSRF Filter in YARN
[ https://issues.apache.org/jira/browse/YARN-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-4737: - Attachment: YARN-4737.004.patch > Use CSRF Filter in YARN > --- > > Key: YARN-4737 > URL: https://issues.apache.org/jira/browse/YARN-4737 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, resourcemanager, webapp >Reporter: Jonathan Maron >Assignee: Jonathan Maron > Attachments: YARN-4737.001.patch, YARN-4737.002.patch, > YARN-4737.003.patch, YARN-4737.004.patch > > > A CSRF filter was added to hadoop common > (https://issues.apache.org/jira/browse/HADOOP-12691). The aim of this JIRA > is to come up with a mechanism to integrate this filter into the webapps for > which it is applicable (web apps that may establish an authenticated > identity). That includes the RM, NM, and mapreduce jobhistory web app. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
[ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15178629#comment-15178629 ] Jonathan Maron commented on YARN-4757: -- I've actually been working on a DNS approach for some time. I'll upload a document describing the approach soon. > [Umbrella] Simplified discovery of services via DNS mechanisms > -- > > Key: YARN-4757 > URL: https://issues.apache.org/jira/browse/YARN-4757 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Maron > > [See overview doc at YARN-4692, copying the sub-section (3.2.10.2) to track > all related efforts.] > In addition to completing the present story of service-registry (YARN-913), > we also need to simplify the access to the registry entries. The existing > read mechanisms of the YARN Service Registry are currently limited to a > registry specific (java) API and a REST interface. In practice, this makes it > very difficult for wiring up existing clients and services. For e.g, dynamic > configuration of dependent endpoints of a service is not easy to implement > using the present registry-read mechanisms, *without* code-changes to > existing services. > A good solution to this is to expose the registry information through a more > generic and widely used discovery mechanism: DNS. Service Discovery via DNS > uses the well-known DNS interfaces to browse the network for services. > YARN-913 in fact talked about such a DNS based mechanism but left it as a > future task. (Task) Having the registry information exposed via DNS > simplifies the life of services. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
[ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron reassigned YARN-4757: Assignee: Jonathan Maron > [Umbrella] Simplified discovery of services via DNS mechanisms > -- > > Key: YARN-4757 > URL: https://issues.apache.org/jira/browse/YARN-4757 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Maron > > [See overview doc at YARN-4692, copying the sub-section (3.2.10.2) to track > all related efforts.] > In addition to completing the present story of service-registry (YARN-913), > we also need to simplify the access to the registry entries. The existing > read mechanisms of the YARN Service Registry are currently limited to a > registry specific (java) API and a REST interface. In practice, this makes it > very difficult for wiring up existing clients and services. For e.g, dynamic > configuration of dependent endpoints of a service is not easy to implement > using the present registry-read mechanisms, *without* code-changes to > existing services. > A good solution to this is to expose the registry information through a more > generic and widely used discovery mechanism: DNS. Service Discovery via DNS > uses the well-known DNS interfaces to browse the network for services. > YARN-913 in fact talked about such a DNS based mechanism but left it as a > future task. (Task) Having the registry information exposed via DNS > simplifies the life of services. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4737) Use CSRF Filter in YARN
[ https://issues.apache.org/jira/browse/YARN-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-4737: - Attachment: YARN-4737.003.patch > Use CSRF Filter in YARN > --- > > Key: YARN-4737 > URL: https://issues.apache.org/jira/browse/YARN-4737 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, resourcemanager, webapp >Reporter: Jonathan Maron >Assignee: Jonathan Maron > Attachments: YARN-4737.001.patch, YARN-4737.002.patch, > YARN-4737.003.patch > > > A CSRF filter was added to hadoop common > (https://issues.apache.org/jira/browse/HADOOP-12691). The aim of this JIRA > is to come up with a mechanism to integrate this filter into the webapps for > which it is applicable (web apps that may establish an authenticated > identity). That includes the RM, NM, and mapreduce jobhistory web app. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4737) Use CSRF Filter in YARN
[ https://issues.apache.org/jira/browse/YARN-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-4737: - Attachment: YARN-4737.002.patch I believe all code issues have been addressed > Use CSRF Filter in YARN > --- > > Key: YARN-4737 > URL: https://issues.apache.org/jira/browse/YARN-4737 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, resourcemanager, webapp >Reporter: Jonathan Maron >Assignee: Jonathan Maron > Attachments: YARN-4737.001.patch, YARN-4737.002.patch > > > A CSRF filter was added to hadoop common > (https://issues.apache.org/jira/browse/HADOOP-12691). The aim of this JIRA > is to come up with a mechanism to integrate this filter into the webapps for > which it is applicable (web apps that may establish an authenticated > identity). That includes the RM, NM, and mapreduce jobhistory web app. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4737) Use CSRF Filter in YARN
[ https://issues.apache.org/jira/browse/YARN-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15175759#comment-15175759 ] Jonathan Maron commented on YARN-4737: -- Enabling CSRF w/o auth will require the inclusion of the custom header for all invocations, regardless of whether they are secure invocations or not. I don't believe that is the expected usage model for the filter. As far as identifying auth mechanisms - I'm trying to find instances that would show the use of custom auth filters but I'm not really finding any. One theory I have is that looking up a value other than "Simple" for "hadoop.http.authentication.type" might provide a more general indicator of auth being enabled? Does that seem correct? POST requests from java clients should not be an issue - the filter only executes when a browser user agent is detected. BTW, the license issues (asflicense) don't appear even remotely related to this patch. > Use CSRF Filter in YARN > --- > > Key: YARN-4737 > URL: https://issues.apache.org/jira/browse/YARN-4737 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, resourcemanager, webapp >Reporter: Jonathan Maron >Assignee: Jonathan Maron > Attachments: YARN-4737.001.patch > > > A CSRF filter was added to hadoop common > (https://issues.apache.org/jira/browse/HADOOP-12691). The aim of this JIRA > is to come up with a mechanism to integrate this filter into the webapps for > which it is applicable (web apps that may establish an authenticated > identity). That includes the RM, NM, and mapreduce jobhistory web app. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4737) Use CSRF Filter in YARN
[ https://issues.apache.org/jira/browse/YARN-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15175697#comment-15175697 ] Jonathan Maron commented on YARN-4737: -- 1) Will do 2) will perform renaming. As for the ATS - the only three web apps instances I identified that have an authentication mechanism enabled were the three I modified. Is the ATS leveraging another auth mechanism (or not using WebApps to construct the endpoint)? 3) The CSRF protection doesn't make sense in the context of not auth mechanism, and the only auth mechanism I see enabled with WebApps in SPNEGO? Is there another auth mechanism that can be enabled independent of API calls to WebApps.Builder? > Use CSRF Filter in YARN > --- > > Key: YARN-4737 > URL: https://issues.apache.org/jira/browse/YARN-4737 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, resourcemanager, webapp >Reporter: Jonathan Maron >Assignee: Jonathan Maron > Attachments: YARN-4737.001.patch > > > A CSRF filter was added to hadoop common > (https://issues.apache.org/jira/browse/HADOOP-12691). The aim of this JIRA > is to come up with a mechanism to integrate this filter into the webapps for > which it is applicable (web apps that may establish an authenticated > identity). That includes the RM, NM, and mapreduce jobhistory web app. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4737) Use CSRF Filter in YARN
[ https://issues.apache.org/jira/browse/YARN-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-4737: - Attachment: YARN-4737.001.patch > Use CSRF Filter in YARN > --- > > Key: YARN-4737 > URL: https://issues.apache.org/jira/browse/YARN-4737 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, resourcemanager, webapp >Reporter: Jonathan Maron >Assignee: Jonathan Maron > Attachments: YARN-4737.001.patch > > > A CSRF filter was added to hadoop common > (https://issues.apache.org/jira/browse/HADOOP-12691). The aim of this JIRA > is to come up with a mechanism to integrate this filter into the webapps for > which it is applicable (web apps that may establish an authenticated > identity). That includes the RM, NM, and mapreduce jobhistory web app. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4737) Use CSRF Filter in YARN
[ https://issues.apache.org/jira/browse/YARN-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-4737: - Attachment: (was: YARN-4737.patch.001) > Use CSRF Filter in YARN > --- > > Key: YARN-4737 > URL: https://issues.apache.org/jira/browse/YARN-4737 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, resourcemanager, webapp >Reporter: Jonathan Maron >Assignee: Jonathan Maron > Attachments: YARN-4737.001.patch > > > A CSRF filter was added to hadoop common > (https://issues.apache.org/jira/browse/HADOOP-12691). The aim of this JIRA > is to come up with a mechanism to integrate this filter into the webapps for > which it is applicable (web apps that may establish an authenticated > identity). That includes the RM, NM, and mapreduce jobhistory web app. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4737) Use CSRF Filter in YARN
[ https://issues.apache.org/jira/browse/YARN-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-4737: - Attachment: YARN-4737.patch.001 The key elements of the uploaded patch: - Provides a CSRF enabling call to WebApps.Builder, taking the configuration prefix as an argument. - Adds the call to web apps currently capable of an SPNEGO authentication (and thus susceptible to CSRF) - RM, NM, and Job History - Defines the properties associated with configuration of the filter for these given web apps - Tests added based on TestRMWebServices (used the test as an example of client invocations of RM web endpoint) NOTE: Could use some assistance in ascertaining whether web apps currently have javascript invocations of the exposed REST services. Those calls will fail if CSRF is enabled. > Use CSRF Filter in YARN > --- > > Key: YARN-4737 > URL: https://issues.apache.org/jira/browse/YARN-4737 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, resourcemanager, webapp >Reporter: Jonathan Maron >Assignee: Jonathan Maron > Attachments: YARN-4737.patch.001 > > > A CSRF filter was added to hadoop common > (https://issues.apache.org/jira/browse/HADOOP-12691). The aim of this JIRA > is to come up with a mechanism to integrate this filter into the webapps for > which it is applicable (web apps that may establish an authenticated > identity). That includes the RM, NM, and mapreduce jobhistory web app. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4737) Use CSRF Filter in YARN
[ https://issues.apache.org/jira/browse/YARN-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167973#comment-15167973 ] Jonathan Maron commented on YARN-4737: -- Thank you! > Use CSRF Filter in YARN > --- > > Key: YARN-4737 > URL: https://issues.apache.org/jira/browse/YARN-4737 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, resourcemanager, webapp >Reporter: Jonathan Maron >Assignee: Jonathan Maron > > A CSRF filter was added to hadoop common > (https://issues.apache.org/jira/browse/HADOOP-12691). The aim of this JIRA > is to come up with a mechanism to integrate this filter into the webapps for > which it is applicable (web apps that may establish an authenticated > identity). That includes the RM, NM, and mapreduce jobhistory web app. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4737) Use CSRF Filter in YARN
[ https://issues.apache.org/jira/browse/YARN-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167965#comment-15167965 ] Jonathan Maron commented on YARN-4737: -- Could this be reassigned to me ([~jmaron]? > Use CSRF Filter in YARN > --- > > Key: YARN-4737 > URL: https://issues.apache.org/jira/browse/YARN-4737 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, resourcemanager, webapp >Reporter: Jonathan Maron > > A CSRF filter was added to hadoop common > (https://issues.apache.org/jira/browse/HADOOP-12691). The aim of this JIRA > is to come up with a mechanism to integrate this filter into the webapps for > which it is applicable (web apps that may establish an authenticated > identity). That includes the RM, NM, and mapreduce jobhistory web app. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4737) Use CSRF Filter in YARN
Jonathan Maron created YARN-4737: Summary: Use CSRF Filter in YARN Key: YARN-4737 URL: https://issues.apache.org/jira/browse/YARN-4737 Project: Hadoop YARN Issue Type: Bug Components: nodemanager, resourcemanager, webapp Reporter: Jonathan Maron A CSRF filter was added to hadoop common (https://issues.apache.org/jira/browse/HADOOP-12691). The aim of this JIRA is to come up with a mechanism to integrate this filter into the webapps for which it is applicable (web apps that may establish an authenticated identity). That includes the RM, NM, and mapreduce jobhistory web app. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4430) registry security validation can fail when downgrading to insecure would work
[ https://issues.apache.org/jira/browse/YARN-4430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045386#comment-15045386 ] Jonathan Maron commented on YARN-4430: -- Given the nature of the underlying issue, the logging of the error rather than raising the exception for read-only access seems appropriate. > registry security validation can fail when downgrading to insecure would work > - > > Key: YARN-4430 > URL: https://issues.apache.org/jira/browse/YARN-4430 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 2.7.1 > Environment: Kerberos, java 7 & 8 >Reporter: Steve Loughran > > The Registry ZK client code does a pre-emptive validation of registry > security settings, in the interest of being helpful when things fail. > But: sometimes it fails fast (see stack trace in SLIDER-993) when it could > just warn and continue -because if the registry is only being read, skipping > SASL auth works -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2031) YARN Proxy model doesn't support REST APIs in AMs
[ https://issues.apache.org/jira/browse/YARN-2031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307473#comment-14307473 ] Jonathan Maron commented on YARN-2031: -- I believe most, if not all, HTTP 1.1 client's will handle both 302 and 307. But assuming an HTTP 1.0 client will even work, or an HTTP 1.1 client exists that doesn't handle 307, we could implement that sort of logic. So, to be clear, the plan is: 1) Wait for this patch to get merged 2) Subsequently add logic that returns 302 for GET/HEAD requests, 307 otherwise? YARN Proxy model doesn't support REST APIs in AMs - Key: YARN-2031 URL: https://issues.apache.org/jira/browse/YARN-2031 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.4.0 Reporter: Steve Loughran Assignee: Steve Loughran Attachments: YARN-2031.patch.001 AMs can't support REST APIs because # the AM filter redirects all requests to the proxy with a 302 response (not 307) # the proxy doesn't forward PUT/POST/DELETE verbs Either the AM filter needs to return 307 and the proxy to forward the verbs, or Am filter should not filter a REST bit of the web site -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2031) YARN Proxy model doesn't support REST APIs in AMs
[ https://issues.apache.org/jira/browse/YARN-2031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-2031: - Attachment: YARN-2031.patch.001 initial attempt at supplying the correct HTTP response code (307) from the proxy servlet YARN Proxy model doesn't support REST APIs in AMs - Key: YARN-2031 URL: https://issues.apache.org/jira/browse/YARN-2031 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.4.0 Reporter: Steve Loughran Assignee: Steve Loughran Attachments: YARN-2031.patch.001 AMs can't support REST APIs because # the AM filter redirects all requests to the proxy with a 302 response (not 307) # the proxy doesn't forward PUT/POST/DELETE verbs Either the AM filter needs to return 307 and the proxy to forward the verbs, or Am filter should not filter a REST bit of the web site -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3068) Support secure HTTP communications between RM proxy and AM web endpoint
[ https://issues.apache.org/jira/browse/YARN-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-3068: - Description: When exposing a web endpoint for UI and REST, an AM is dependent on the RM as a proxy for incoming interactions. The RM web proxy supports security features such as SSL and SPNEGO. However, those security mechanisms are not supported by the AM, and supporting them directly at the AM would require some complex implementation details and configuration (not to mention that given the proxying relationship they may be considered somewhat redundant). In order to ensure that there is a measure of security (trust) between the RM web proxy and the AM, the following mechanism is suggested: - The RM will create a shared secret and propagate it to the AM during AM launch (e.g. it could be part of the existing credentials). - The web proxy will leverage the shared secret to encrypt an agreed upon text (e.g. the container ID) and an associated expiry time (to mitigate potential request spoofing). - The AM will decrypt the text leveraging the shared secret and, if successful and the expiry time has not been reached, proceed with the request processing (probably appropriate to perform these checks in the existing AmIpFilter or a specific trust filter). Note that this feature is key to supporting interactions between Knox and AM REST resources, since those interactions depend on trusted proxy support the RM can provide (via its current SPNEGO and doAs support), allowing AM's to focus on performing their processing based on the established doAs identity (established at the RM and related to the AM via a trusted path). was: When exposing a web endpoint for UI and REST, an AM is dependent on the RM as a proxy for incoming interactions. The RM web proxy supports security features such as SSL and SPNEGO. However, those security mechanisms are not supported by the AM, and supporting them directly at the AM would require some complex implementation details and configuration (not to mention that given the proxying relationship they may be considered somewhat redundant). In order to ensure that there is a measure of security (trust) between the RM web proxy and the AM, the following mechanism is suggested: - The AM will create a shared secret and propagate it to the AM during AM launch (e.g. it could be part of the existing credentials). - The web proxy will leverage the shared secret to encrypt an agreed upon text (e.g. the container ID) and an associated expiry time (to mitigate potential request spoofing). - The AM will decrypt the text leveraging the shared secret and, if successful and the expiry time has not been reached, proceed with the request processing (probably appropriate to perform these checks in the existing AmIpFilter or a specific trust filter). Note that this feature is key to supporting interactions between Knox and AM REST resources, since those interactions depend on trusted proxy support the RM can provide (via its current SPNEGO and doAs support), allowing AM's to focus on performing their processing based on the established doAs identity (established at the RM and related to the AM via a trusted path). Support secure HTTP communications between RM proxy and AM web endpoint --- Key: YARN-3068 URL: https://issues.apache.org/jira/browse/YARN-3068 Project: Hadoop YARN Issue Type: Bug Components: applications, resourcemanager Reporter: Jonathan Maron When exposing a web endpoint for UI and REST, an AM is dependent on the RM as a proxy for incoming interactions. The RM web proxy supports security features such as SSL and SPNEGO. However, those security mechanisms are not supported by the AM, and supporting them directly at the AM would require some complex implementation details and configuration (not to mention that given the proxying relationship they may be considered somewhat redundant). In order to ensure that there is a measure of security (trust) between the RM web proxy and the AM, the following mechanism is suggested: - The RM will create a shared secret and propagate it to the AM during AM launch (e.g. it could be part of the existing credentials). - The web proxy will leverage the shared secret to encrypt an agreed upon text (e.g. the container ID) and an associated expiry time (to mitigate potential request spoofing). - The AM will decrypt the text leveraging the shared secret and, if successful and the expiry time has not been reached, proceed with the request processing (probably appropriate to perform these checks in the existing AmIpFilter or a specific trust filter). Note that this feature is key to supporting interactions between Knox and AM REST resources,
[jira] [Commented] (YARN-3068) Support secure HTTP communications between RM proxy and AM web endpoint
[ https://issues.apache.org/jira/browse/YARN-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14282862#comment-14282862 ] Jonathan Maron commented on YARN-3068: -- Correct - should be RM. Edited and fixed. Support secure HTTP communications between RM proxy and AM web endpoint --- Key: YARN-3068 URL: https://issues.apache.org/jira/browse/YARN-3068 Project: Hadoop YARN Issue Type: Bug Components: applications, resourcemanager Reporter: Jonathan Maron When exposing a web endpoint for UI and REST, an AM is dependent on the RM as a proxy for incoming interactions. The RM web proxy supports security features such as SSL and SPNEGO. However, those security mechanisms are not supported by the AM, and supporting them directly at the AM would require some complex implementation details and configuration (not to mention that given the proxying relationship they may be considered somewhat redundant). In order to ensure that there is a measure of security (trust) between the RM web proxy and the AM, the following mechanism is suggested: - The RM will create a shared secret and propagate it to the AM during AM launch (e.g. it could be part of the existing credentials). - The web proxy will leverage the shared secret to encrypt an agreed upon text (e.g. the container ID) and an associated expiry time (to mitigate potential request spoofing). - The AM will decrypt the text leveraging the shared secret and, if successful and the expiry time has not been reached, proceed with the request processing (probably appropriate to perform these checks in the existing AmIpFilter or a specific trust filter). Note that this feature is key to supporting interactions between Knox and AM REST resources, since those interactions depend on trusted proxy support the RM can provide (via its current SPNEGO and doAs support), allowing AM's to focus on performing their processing based on the established doAs identity (established at the RM and related to the AM via a trusted path). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-3068) shared secret for trust between RM and AM
Jonathan Maron created YARN-3068: Summary: shared secret for trust between RM and AM Key: YARN-3068 URL: https://issues.apache.org/jira/browse/YARN-3068 Project: Hadoop YARN Issue Type: Bug Components: applications, resourcemanager Reporter: Jonathan Maron When exposing a web endpoint for UI and REST, an AM is dependent on the RM as a proxy for incoming interactions. The RM web proxy supports security features such as SSL and SPNEGO. However, those security mechanisms are not supported by the AM, and supporting them directly at the AM would require some complex implementation details and configuration (not to mention that given the proxying relationship they may be considered somewhat redundant). In order to ensure that there is a measure of security (trust) between the RM web proxy and the AM, the following mechanism is suggested: - The AM will create a shared secret and propagate it to the AM during AM launch (e.g. it could be part of the existing credentials). - The web proxy will leverage the shared secret to encrypt an agreed upon text (e.g. the container ID) and an associated expiry time (to mitigate potential request spoofing). - The AM will decrypt the text leveraging the shared secret and, if successful and the expiry time has not been reached, proceed with the request processing (probably appropriate to perform these checks in the existing AmIpFilter or a specific trust filter). Note that this feature is key to supporting interactions between Knox and AM REST resources, since those interactions depend on trusted proxy support the RM can provide (via its current SPNEGO and doAs support), allowing AM's to focus on performing their processing based on the established doAs identity (established at the RM and related to the AM via a trusted path). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2648) need mechanism for updating HDFS delegation tokens associated with container launch contexts
Jonathan Maron created YARN-2648: Summary: need mechanism for updating HDFS delegation tokens associated with container launch contexts Key: YARN-2648 URL: https://issues.apache.org/jira/browse/YARN-2648 Project: Hadoop YARN Issue Type: Bug Components: nodemanager, resourcemanager Reporter: Jonathan Maron During the launch of a container, the required delegation tokens (e.g. HDFS) are passed to the launch context. If those tokens expire and the container requires a restart the restart attempt will fail. Sample log output: 2014-10-06 18:37:28,609 WARN ipc.Client (Client.java:run(675)) - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 124 for hbase) can't be found in cache -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2554) Slider AM Web UI is inaccessible if HTTPS/SSL is specified as the HTTP policy
[ https://issues.apache.org/jira/browse/YARN-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14142488#comment-14142488 ] Jonathan Maron commented on YARN-2554: -- Let me try to address each of these concerns: - the key-store needs to present on all machine to distribute certificates: AMs may come up anywhere. The client trust store appears to an already defined keystore in the secure deployments I've seen so far. - the key-store used by Hadoop daemons CANNOT be shared with AMs: AMs run user-code as the user - the key-store cannot be shared across AMs of different users: Assuming I am running three different Slider apps as three different users, you don't want to have a single key-store instance accessible by all Slider AMs. Your concern about sharing elements between daemon processes and users is correct in the context of kerberos - principals are used for authentication and authorization and it wouldn't be prudent to have those shared. When code is running within an AM, the identity that is established and governs its authorized interactions is done via kerberos. But SSL is a security offering targeted for the transport layer. Typically with SSL, for client C and server S, SSL only establishes that: C believes S is who it intended to contact C believes it has a secure connection to S S believes it has a secure connection to C In addition, typically SSL ensures authentication of the server alone (the CN is generally the host name), so the trust between C and S is more along the lines of trusting the C is actually talking to host S. The session key for the RM (C) and AM (S), given the host certificate of the AM, will be unique - it is created by the RM, encrypted using the AM cert (using the AM public key), and sent to the AM to be decrypted by the AM's private key. I guess I don't see the issue with hadoop applications, whether daemon or AM, using the available hadoop keystores. The tranport layer mechanism doesn't play a role in the application level authorization since it has no role to play in establishing the user's subject etc. Having said all that, I may not be aware of some uses of the keystores in a hadoop cluster (or perhaps my assumption about the availability of a client trust store is wrong). In addition, it'd be nice to get the perspective of some of the security folks about the role of SSL in the security layers provided in a secure cluster. Slider AM Web UI is inaccessible if HTTPS/SSL is specified as the HTTP policy - Key: YARN-2554 URL: https://issues.apache.org/jira/browse/YARN-2554 Project: Hadoop YARN Issue Type: Bug Components: webapp Affects Versions: 2.6.0 Reporter: Jonathan Maron Attachments: YARN-2554.1.patch, YARN-2554.2.patch, YARN-2554.3.patch, YARN-2554.3.patch If the HTTP policy to enable HTTPS is specified, the RM and AM are initialized with SSL listeners. The RM has a web app proxy servlet that acts as a proxy for incoming AM requests. In order to forward the requests to the AM the proxy servlet makes use of HttpClient. However, the HttpClient utilized is not initialized correctly with the necessary certs to allow for successful one way SSL invocations to the other nodes in the cluster (it is not configured to access/load the client truststore specified in ssl-client.xml). I imagine SSLFactory.createSSLSocketFactory() could be utilized to create an instance that can be assigned to the HttpClient. The symptoms of this issue are: AM: Displays unknown_certificate exception RM: Displays an exception such as javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2554) Slider AM Web UI is inaccessible if HTTPS/SSL is specified as the HTTP policy
[ https://issues.apache.org/jira/browse/YARN-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-2554: - Attachment: YARN-2554.3.patch Slider AM Web UI is inaccessible if HTTPS/SSL is specified as the HTTP policy - Key: YARN-2554 URL: https://issues.apache.org/jira/browse/YARN-2554 Project: Hadoop YARN Issue Type: Bug Components: webapp Affects Versions: 2.6.0 Reporter: Jonathan Maron Attachments: YARN-2554.1.patch, YARN-2554.2.patch, YARN-2554.3.patch If the HTTP policy to enable HTTPS is specified, the RM and AM are initialized with SSL listeners. The RM has a web app proxy servlet that acts as a proxy for incoming AM requests. In order to forward the requests to the AM the proxy servlet makes use of HttpClient. However, the HttpClient utilized is not initialized correctly with the necessary certs to allow for successful one way SSL invocations to the other nodes in the cluster (it is not configured to access/load the client truststore specified in ssl-client.xml). I imagine SSLFactory.createSSLSocketFactory() could be utilized to create an instance that can be assigned to the HttpClient. The symptoms of this issue are: AM: Displays unknown_certificate exception RM: Displays an exception such as javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2554) Slider AM Web UI is inaccessible if HTTPS/SSL is specified as the HTTP policy
[ https://issues.apache.org/jira/browse/YARN-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-2554: - Attachment: YARN-2554.3.patch Slider AM Web UI is inaccessible if HTTPS/SSL is specified as the HTTP policy - Key: YARN-2554 URL: https://issues.apache.org/jira/browse/YARN-2554 Project: Hadoop YARN Issue Type: Bug Components: webapp Affects Versions: 2.6.0 Reporter: Jonathan Maron Attachments: YARN-2554.1.patch, YARN-2554.2.patch, YARN-2554.3.patch, YARN-2554.3.patch If the HTTP policy to enable HTTPS is specified, the RM and AM are initialized with SSL listeners. The RM has a web app proxy servlet that acts as a proxy for incoming AM requests. In order to forward the requests to the AM the proxy servlet makes use of HttpClient. However, the HttpClient utilized is not initialized correctly with the necessary certs to allow for successful one way SSL invocations to the other nodes in the cluster (it is not configured to access/load the client truststore specified in ssl-client.xml). I imagine SSLFactory.createSSLSocketFactory() could be utilized to create an instance that can be assigned to the HttpClient. The symptoms of this issue are: AM: Displays unknown_certificate exception RM: Displays an exception such as javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2554) Slider AM Web UI is inaccessible if HTTPS/SSL is specified as the HTTP policy
[ https://issues.apache.org/jira/browse/YARN-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14142256#comment-14142256 ] Jonathan Maron commented on YARN-2554: -- I'm not certain I understand your comment about the keys. The client trust store configured via ssl-client.xml generally contains the certificates for the cluster hosts, it is not specific to users or applications. In any usage scenario it would by necessity contain those certificates. Slider AM Web UI is inaccessible if HTTPS/SSL is specified as the HTTP policy - Key: YARN-2554 URL: https://issues.apache.org/jira/browse/YARN-2554 Project: Hadoop YARN Issue Type: Bug Components: webapp Affects Versions: 2.6.0 Reporter: Jonathan Maron Attachments: YARN-2554.1.patch, YARN-2554.2.patch, YARN-2554.3.patch, YARN-2554.3.patch If the HTTP policy to enable HTTPS is specified, the RM and AM are initialized with SSL listeners. The RM has a web app proxy servlet that acts as a proxy for incoming AM requests. In order to forward the requests to the AM the proxy servlet makes use of HttpClient. However, the HttpClient utilized is not initialized correctly with the necessary certs to allow for successful one way SSL invocations to the other nodes in the cluster (it is not configured to access/load the client truststore specified in ssl-client.xml). I imagine SSLFactory.createSSLSocketFactory() could be utilized to create an instance that can be assigned to the HttpClient. The symptoms of this issue are: AM: Displays unknown_certificate exception RM: Displays an exception such as javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2554) Slider AM Web UI is inaccessible if HTTPS/SSL is specified as the HTTP policy
[ https://issues.apache.org/jira/browse/YARN-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-2554: - Attachment: YARN-2554.1.patch Slider AM Web UI is inaccessible if HTTPS/SSL is specified as the HTTP policy - Key: YARN-2554 URL: https://issues.apache.org/jira/browse/YARN-2554 Project: Hadoop YARN Issue Type: Bug Components: webapp Affects Versions: 2.6.0 Reporter: Jonathan Maron Attachments: YARN-2554.1.patch If the HTTP policy to enable HTTPS is specified, the RM and AM are initialized with SSL listeners. The RM has a web app proxy servlet that acts as a proxy for incoming AM requests. In order to forward the requests to the AM the proxy servlet makes use of HttpClient. However, the HttpClient utilized is not initialized correctly with the necessary certs to allow for successful one way SSL invocations to the other nodes in the cluster (it is not configured to access/load the client truststore specified in ssl-client.xml). I imagine SSLFactory.createSSLSocketFactory() could be utilized to create an instance that can be assigned to the HttpClient. The symptoms of this issue are: AM: Displays unknown_certificate exception RM: Displays an exception such as javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2554) Slider AM Web UI is inaccessible if HTTPS/SSL is specified as the HTTP policy
[ https://issues.apache.org/jira/browse/YARN-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-2554: - Attachment: YARN-2554.2.patch Slider AM Web UI is inaccessible if HTTPS/SSL is specified as the HTTP policy - Key: YARN-2554 URL: https://issues.apache.org/jira/browse/YARN-2554 Project: Hadoop YARN Issue Type: Bug Components: webapp Affects Versions: 2.6.0 Reporter: Jonathan Maron Attachments: YARN-2554.1.patch, YARN-2554.2.patch If the HTTP policy to enable HTTPS is specified, the RM and AM are initialized with SSL listeners. The RM has a web app proxy servlet that acts as a proxy for incoming AM requests. In order to forward the requests to the AM the proxy servlet makes use of HttpClient. However, the HttpClient utilized is not initialized correctly with the necessary certs to allow for successful one way SSL invocations to the other nodes in the cluster (it is not configured to access/load the client truststore specified in ssl-client.xml). I imagine SSLFactory.createSSLSocketFactory() could be utilized to create an instance that can be assigned to the HttpClient. The symptoms of this issue are: AM: Displays unknown_certificate exception RM: Displays an exception such as javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-2554) Slider AM Web UI is inaccessible if HTTPS/SSL is enabled as the HTTP policy
Jonathan Maron created YARN-2554: Summary: Slider AM Web UI is inaccessible if HTTPS/SSL is enabled as the HTTP policy Key: YARN-2554 URL: https://issues.apache.org/jira/browse/YARN-2554 Project: Hadoop YARN Issue Type: Bug Components: webapp Affects Versions: 2.6.0 Reporter: Jonathan Maron If the HTTP policy to enable HTTPS is specified, the RM and AM are initialized with SSL listeners. The RM has a web app proxy servlet that acts as a proxy for incoming AM requests. In order to forward the requests to the AM the proxy servlet makes use of HttpClient. However, the HttpClient utilized is not initialized correctly with the necessary certs to allow for successful one way SSL invocations to the other nodes in the cluster (it is not configured to access/load the client truststore specified in ssl-client.xml). I imagine SSLFactory.createSSLSocketFactory() could be utilized to create an instance that can be assigned to the HttpClient. The symptoms of this issue are: AM: Displays unknown_certificate exception RM: Displays an exception such as javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2554) Slider AM Web UI is inaccessible if HTTPS/SSL is specified as the HTTP policy
[ https://issues.apache.org/jira/browse/YARN-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron updated YARN-2554: - Summary: Slider AM Web UI is inaccessible if HTTPS/SSL is specified as the HTTP policy (was: Slider AM Web UI is inaccessible if HTTPS/SSL is enabled as the HTTP policy) Slider AM Web UI is inaccessible if HTTPS/SSL is specified as the HTTP policy - Key: YARN-2554 URL: https://issues.apache.org/jira/browse/YARN-2554 Project: Hadoop YARN Issue Type: Bug Components: webapp Affects Versions: 2.6.0 Reporter: Jonathan Maron If the HTTP policy to enable HTTPS is specified, the RM and AM are initialized with SSL listeners. The RM has a web app proxy servlet that acts as a proxy for incoming AM requests. In order to forward the requests to the AM the proxy servlet makes use of HttpClient. However, the HttpClient utilized is not initialized correctly with the necessary certs to allow for successful one way SSL invocations to the other nodes in the cluster (it is not configured to access/load the client truststore specified in ssl-client.xml). I imagine SSLFactory.createSSLSocketFactory() could be utilized to create an instance that can be assigned to the HttpClient. The symptoms of this issue are: AM: Displays unknown_certificate exception RM: Displays an exception such as javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2554) Slider AM Web UI is inaccessible if HTTPS/SSL is enabled as the HTTP policy
[ https://issues.apache.org/jira/browse/YARN-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14133934#comment-14133934 ] Jonathan Maron commented on YARN-2554: -- A workaround (though not necessarily a production recommended one) is to add the client trust store certs to the the JDK's cacerts file (export the trust store certs, import them to JDK/jre/lib/security/cacerts) Slider AM Web UI is inaccessible if HTTPS/SSL is enabled as the HTTP policy --- Key: YARN-2554 URL: https://issues.apache.org/jira/browse/YARN-2554 Project: Hadoop YARN Issue Type: Bug Components: webapp Affects Versions: 2.6.0 Reporter: Jonathan Maron If the HTTP policy to enable HTTPS is specified, the RM and AM are initialized with SSL listeners. The RM has a web app proxy servlet that acts as a proxy for incoming AM requests. In order to forward the requests to the AM the proxy servlet makes use of HttpClient. However, the HttpClient utilized is not initialized correctly with the necessary certs to allow for successful one way SSL invocations to the other nodes in the cluster (it is not configured to access/load the client truststore specified in ssl-client.xml). I imagine SSLFactory.createSSLSocketFactory() could be utilized to create an instance that can be assigned to the HttpClient. The symptoms of this issue are: AM: Displays unknown_certificate exception RM: Displays an exception such as javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target -- This message was sent by Atlassian JIRA (v6.3.4#6332)
