Hi,
I am running Spark-on-YARN on a secure cluster with yarn log aggregation set
up. Once a job completes, when viewing stdout/stderr executor logs in the Spark
history server UI it redirects me to the local nodemanager where a page appears
for a second saying ‘Redirecting to log server….’ and then redirects me to the
aggregated job history server log page. However, the aggregated job history
page sends me to
http://<jobhistoryserver>:<port>/jobhistory<http://%3cjobhistoryserver%3e:%3cport%3e/jobhistory>......
Instead of https://..., causing odd characters to appear. If you manually
specify https:// in the URL, this works as expected, however the page
automatically refreshes and causes this to go back to http again.
I have set the job.log.server.url property in yarn-site.xml to include https:
<property>
<name>yarn.log.server.url</name>
<value>https://<jobhistoryserver>.domain.com:port/jobhistory/logs/</value>
</property>
I know this isn’t a Spark issue specifically, but I wondered if anyone else has
experienced this issue and knows how to get around it?
Thanks,
Mike
This e-mail (including any attachments) is private and confidential, may
contain proprietary or privileged information and is intended for the named
recipient(s) only. Unintended recipients are strictly prohibited from taking
action on the basis of information in this e-mail and must contact the sender
immediately, delete this e-mail (and all attachments) and destroy any hard
copies. Nomura will not accept responsibility or liability for the accuracy or
completeness of, or the presence of any virus or disabling code in, this
e-mail. If verification is sought please request a hard copy. Any reference to
the terms of executed transactions should be treated as preliminary only and
subject to formal written confirmation by Nomura. Nomura reserves the right to
retain, monitor and intercept e-mail communications through its networks
(subject to and in accordance with applicable laws). No confidentiality or
privilege is waived or lost by Nomura by any mistransmission of this e-mail.
Any reference to "Nomura" is a reference to any entity in the Nomura Holdings,
Inc. group. Please read our Electronic Communications Legal Notice which forms
part of this e-mail: http://www.Nomura.com/email_disclaimer.htm
