[jira] [Commented] (SPARK-15829) spark master webpage links to application UI broke when running in cluster mode
[ https://issues.apache.org/jira/browse/SPARK-15829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16141304#comment-16141304 ] drain.miao commented on SPARK-15829: Hi Andy, I'm also facing this issue, is there any solutions for it? > spark master webpage links to application UI broke when running in cluster > mode > --- > > Key: SPARK-15829 > URL: https://issues.apache.org/jira/browse/SPARK-15829 > Project: Spark > Issue Type: Bug > Components: EC2 >Affects Versions: 1.6.1 > Environment: AWS ec2 cluster >Reporter: Andrew Davidson > > Hi > I created a cluster using the spark-1.6.1-bin-hadoop2.6/ec2/spark-ec2 > I use the stand alone cluster manager. I have a streaming app running in > cluster mode. I notice the master webpage links to the application UI page > are incorrect > It does not look like jira will let my upload images. I'll try and describe > the web pages and the bug > My master is running on > http://ec2-54-215-230-73.us-west-1.compute.amazonaws.com:8080/ > It has a section marked "applications". If I click on one of the running > application ids I am taken to a page showing "Executor Summary". This page > has a link to teh 'application detail UI' the url is > http://ec2-54-215-230-73.us-west-1.compute.amazonaws.com:4041/ > Notice it things the application UI is running on the cluster master. > It is actually running on the same machine as the driver on port 4041. I was > able to reverse engine the url by noticing the private ip address is part of > the worker id . For example worker-20160322041632-172.31.23.201-34909 > next I went on the aws ec2 console to find the public DNS name for this > machine > http://ec2-54-193-104-169.us-west-1.compute.amazonaws.com:4041/streaming/ > Kind regards > Andy -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15829) spark master webpage links to application UI broke when running in cluster mode
[ https://issues.apache.org/jira/browse/SPARK-15829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359307#comment-15359307 ] Sean Owen commented on SPARK-15829: --- It's not present in the 2.x branches. It lives on in the amplab repo; the docs in 2.x point to it. > spark master webpage links to application UI broke when running in cluster > mode > --- > > Key: SPARK-15829 > URL: https://issues.apache.org/jira/browse/SPARK-15829 > Project: Spark > Issue Type: Bug > Components: EC2 >Affects Versions: 1.6.1 > Environment: AWS ec2 cluster >Reporter: Andrew Davidson >Priority: Critical > > Hi > I created a cluster using the spark-1.6.1-bin-hadoop2.6/ec2/spark-ec2 > I use the stand alone cluster manager. I have a streaming app running in > cluster mode. I notice the master webpage links to the application UI page > are incorrect > It does not look like jira will let my upload images. I'll try and describe > the web pages and the bug > My master is running on > http://ec2-54-215-230-73.us-west-1.compute.amazonaws.com:8080/ > It has a section marked "applications". If I click on one of the running > application ids I am taken to a page showing "Executor Summary". This page > has a link to teh 'application detail UI' the url is > http://ec2-54-215-230-73.us-west-1.compute.amazonaws.com:4041/ > Notice it things the application UI is running on the cluster master. > It is actually running on the same machine as the driver on port 4041. I was > able to reverse engine the url by noticing the private ip address is part of > the worker id . For example worker-20160322041632-172.31.23.201-34909 > next I went on the aws ec2 console to find the public DNS name for this > machine > http://ec2-54-193-104-169.us-west-1.compute.amazonaws.com:4041/streaming/ > Kind regards > Andy -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15829) spark master webpage links to application UI broke when running in cluster mode
[ https://issues.apache.org/jira/browse/SPARK-15829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359277#comment-15359277 ] Andrew Davidson commented on SPARK-15829: - Hi Sean you mention the ec2 script is not supported anymore? What was the last release it was supported in? Its still part of the 1.6.x documentation Is there a replacement or alternative? thanks Andy > spark master webpage links to application UI broke when running in cluster > mode > --- > > Key: SPARK-15829 > URL: https://issues.apache.org/jira/browse/SPARK-15829 > Project: Spark > Issue Type: Bug > Components: EC2 >Affects Versions: 1.6.1 > Environment: AWS ec2 cluster >Reporter: Andrew Davidson >Priority: Critical > > Hi > I created a cluster using the spark-1.6.1-bin-hadoop2.6/ec2/spark-ec2 > I use the stand alone cluster manager. I have a streaming app running in > cluster mode. I notice the master webpage links to the application UI page > are incorrect > It does not look like jira will let my upload images. I'll try and describe > the web pages and the bug > My master is running on > http://ec2-54-215-230-73.us-west-1.compute.amazonaws.com:8080/ > It has a section marked "applications". If I click on one of the running > application ids I am taken to a page showing "Executor Summary". This page > has a link to teh 'application detail UI' the url is > http://ec2-54-215-230-73.us-west-1.compute.amazonaws.com:4041/ > Notice it things the application UI is running on the cluster master. > It is actually running on the same machine as the driver on port 4041. I was > able to reverse engine the url by noticing the private ip address is part of > the worker id . For example worker-20160322041632-172.31.23.201-34909 > next I went on the aws ec2 console to find the public DNS name for this > machine > http://ec2-54-193-104-169.us-west-1.compute.amazonaws.com:4041/streaming/ > Kind regards > Andy -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15829) spark master webpage links to application UI broke when running in cluster mode
[ https://issues.apache.org/jira/browse/SPARK-15829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359224#comment-15359224 ] Sean Owen commented on SPARK-15829: --- No, it certainly doesn't assume the driver is the master. I'm not sure, it actually doesn't look like a private/public host name thing, so maybe SPARK_MASTER_HOST is irrelevant. No, it's not set for you. Although the ec2 script is not supported now, it may not be ec2-specific, if there's a problem. It's not my area so much, but maybe you can track down a series of log messages that shows more clearly what's going on. > spark master webpage links to application UI broke when running in cluster > mode > --- > > Key: SPARK-15829 > URL: https://issues.apache.org/jira/browse/SPARK-15829 > Project: Spark > Issue Type: Bug > Components: EC2 >Affects Versions: 1.6.1 > Environment: AWS ec2 cluster >Reporter: Andrew Davidson >Priority: Critical > > Hi > I created a cluster using the spark-1.6.1-bin-hadoop2.6/ec2/spark-ec2 > I use the stand alone cluster manager. I have a streaming app running in > cluster mode. I notice the master webpage links to the application UI page > are incorrect > It does not look like jira will let my upload images. I'll try and describe > the web pages and the bug > My master is running on > http://ec2-54-215-230-73.us-west-1.compute.amazonaws.com:8080/ > It has a section marked "applications". If I click on one of the running > application ids I am taken to a page showing "Executor Summary". This page > has a link to teh 'application detail UI' the url is > http://ec2-54-215-230-73.us-west-1.compute.amazonaws.com:4041/ > Notice it things the application UI is running on the cluster master. > It is actually running on the same machine as the driver on port 4041. I was > able to reverse engine the url by noticing the private ip address is part of > the worker id . For example worker-20160322041632-172.31.23.201-34909 > next I went on the aws ec2 console to find the public DNS name for this > machine > http://ec2-54-193-104-169.us-west-1.compute.amazonaws.com:4041/streaming/ > Kind regards > Andy -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15829) spark master webpage links to application UI broke when running in cluster mode
[ https://issues.apache.org/jira/browse/SPARK-15829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359215#comment-15359215 ] Andrew Davidson commented on SPARK-15829: - Hi Sean I am not sure how to check the value of 'SPARK_MASTER_HOST'. I looked at the documentation pags http://spark.apache.org/docs/latest/configuration.html and http://spark.apache.org/docs/latest/ec2-scripts.html. They do not mention SPARK_MASTER_HOST when I submit my jobs I use MASTER_URL=spark://ec2-54-215-230-73.us-west-1.compute.amazonaws.com:6066 I use the stand alone cluster manager I think the problem may be that the web UI assume the driver is always running on the master machine. I assume the cluster manager decides which worker the driver will run on. Is there a way for the web UI to discover where the driver is running? On my master [ec2-user@ip-172-31-22-140 conf]$ pwd /root/spark/conf [ec2-user@ip-172-31-22-140 conf]$ cat slaves ec2-54-193-94-207.us-west-1.compute.amazonaws.com ec2-54-67-13-246.us-west-1.compute.amazonaws.com ec2-54-67-48-49.us-west-1.compute.amazonaws.com ec2-54-193-104-169.us-west-1.compute.amazonaws.com [ec2-user@ip-172-31-22-140 conf]$ [ec2-user@ip-172-31-22-140 conf]$ grep SPARK_MASTER_HOST * [ec2-user@ip-172-31-22-140 conf]$ pwd /root/spark/conf [ec2-user@ip-172-31-22-140 conf]$ grep SPARK_MASTER_HOST * [ec2-user@ip-172-31-22-140 conf]$ ec2-user@ip-172-31-22-140 sbin]$ pwd /root/spark/sbin [ec2-user@ip-172-31-22-140 sbin]$ grep SPARK_MASTER_HOST * [ec2-user@ip-172-31-22-140 bin]$ !grep grep SPARK_MASTER_HOST * [ec2-user@ip-172-31-22-140 bin]$ Thanks for looking into this Andy > spark master webpage links to application UI broke when running in cluster > mode > --- > > Key: SPARK-15829 > URL: https://issues.apache.org/jira/browse/SPARK-15829 > Project: Spark > Issue Type: Bug > Components: EC2 >Affects Versions: 1.6.1 > Environment: AWS ec2 cluster >Reporter: Andrew Davidson >Priority: Critical > > Hi > I created a cluster using the spark-1.6.1-bin-hadoop2.6/ec2/spark-ec2 > I use the stand alone cluster manager. I have a streaming app running in > cluster mode. I notice the master webpage links to the application UI page > are incorrect > It does not look like jira will let my upload images. I'll try and describe > the web pages and the bug > My master is running on > http://ec2-54-215-230-73.us-west-1.compute.amazonaws.com:8080/ > It has a section marked "applications". If I click on one of the running > application ids I am taken to a page showing "Executor Summary". This page > has a link to teh 'application detail UI' the url is > http://ec2-54-215-230-73.us-west-1.compute.amazonaws.com:4041/ > Notice it things the application UI is running on the cluster master. > It is actually running on the same machine as the driver on port 4041. I was > able to reverse engine the url by noticing the private ip address is part of > the worker id . For example worker-20160322041632-172.31.23.201-34909 > next I went on the aws ec2 console to find the public DNS name for this > machine > http://ec2-54-193-104-169.us-west-1.compute.amazonaws.com:4041/streaming/ > Kind regards > Andy -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15829) spark master webpage links to application UI broke when running in cluster mode
[ https://issues.apache.org/jira/browse/SPARK-15829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15355121#comment-15355121 ] Sean Owen commented on SPARK-15829: --- I think this is a question of whether you are using internal or external host names in your config. You can look into SPARK_MASTER_HOST? > spark master webpage links to application UI broke when running in cluster > mode > --- > > Key: SPARK-15829 > URL: https://issues.apache.org/jira/browse/SPARK-15829 > Project: Spark > Issue Type: Bug > Components: EC2 >Affects Versions: 1.6.1 > Environment: AWS ec2 cluster >Reporter: Andrew Davidson >Priority: Critical > > Hi > I created a cluster using the spark-1.6.1-bin-hadoop2.6/ec2/spark-ec2 > I use the stand alone cluster manager. I have a streaming app running in > cluster mode. I notice the master webpage links to the application UI page > are incorrect > It does not look like jira will let my upload images. I'll try and describe > the web pages and the bug > My master is running on > http://ec2-54-215-230-73.us-west-1.compute.amazonaws.com:8080/ > It has a section marked "applications". If I click on one of the running > application ids I am taken to a page showing "Executor Summary". This page > has a link to teh 'application detail UI' the url is > http://ec2-54-215-230-73.us-west-1.compute.amazonaws.com:4041/ > Notice it things the application UI is running on the cluster master. > It is actually running on the same machine as the driver on port 4041. I was > able to reverse engine the url by noticing the private ip address is part of > the worker id . For example worker-20160322041632-172.31.23.201-34909 > next I went on the aws ec2 console to find the public DNS name for this > machine > http://ec2-54-193-104-169.us-west-1.compute.amazonaws.com:4041/streaming/ > Kind regards > Andy -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15829) spark master webpage links to application UI broke when running in cluster mode
[ https://issues.apache.org/jira/browse/SPARK-15829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325073#comment-15325073 ] Xin Ren commented on SPARK-15829: - sorry Andy, my bad. I'm running on port 7077 and client mode. > spark master webpage links to application UI broke when running in cluster > mode > --- > > Key: SPARK-15829 > URL: https://issues.apache.org/jira/browse/SPARK-15829 > Project: Spark > Issue Type: Bug > Components: EC2 >Affects Versions: 1.6.1 > Environment: AWS ec2 cluster >Reporter: Andrew Davidson >Priority: Critical > > Hi > I created a cluster using the spark-1.6.1-bin-hadoop2.6/ec2/spark-ec2 > I use the stand alone cluster manager. I have a streaming app running in > cluster mode. I notice the master webpage links to the application UI page > are incorrect > It does not look like jira will let my upload images. I'll try and describe > the web pages and the bug > My master is running on > http://ec2-54-215-230-73.us-west-1.compute.amazonaws.com:8080/ > It has a section marked "applications". If I click on one of the running > application ids I am taken to a page showing "Executor Summary". This page > has a link to teh 'application detail UI' the url is > http://ec2-54-215-230-73.us-west-1.compute.amazonaws.com:4041/ > Notice it things the application UI is running on the cluster master. > It is actually running on the same machine as the driver on port 4041. I was > able to reverse engine the url by noticing the private ip address is part of > the worker id . For example worker-20160322041632-172.31.23.201-34909 > next I went on the aws ec2 console to find the public DNS name for this > machine > http://ec2-54-193-104-169.us-west-1.compute.amazonaws.com:4041/streaming/ > Kind regards > Andy -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15829) spark master webpage links to application UI broke when running in cluster mode
[ https://issues.apache.org/jira/browse/SPARK-15829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325001#comment-15325001 ] Andrew Davidson commented on SPARK-15829: - Hi Xin I ran netstat on my master. I do not think the port are in use. To submit in cluster mode I use port 6066. If you are using port 7077 you are in client mode. In client mode the application UI will run on the spark master. In cluster mode the application UI runs on which ever slave the driver is running on. If you notice in my original description the url is incorrect. the ip is wrong, the port is correct. Kind regards Andy # bash-4.2# netstat -tulpn Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp0 0 0.0.0.0:86520.0.0.0:* LISTEN 3832/gmetad tcp0 0 0.0.0.0:87870.0.0.0:* LISTEN 2584/rserver tcp0 0 0.0.0.0:36757 0.0.0.0:* LISTEN 2905/java tcp0 0 0.0.0.0:50070 0.0.0.0:* LISTEN 2905/java tcp0 0 0.0.0.0:22 0.0.0.0:* LISTEN 2144/sshd tcp0 0 127.0.0.1:631 0.0.0.0:* LISTEN 2095/cupsd tcp0 0 127.0.0.1:7000 0.0.0.0:* LISTEN 6512/python3.4 tcp0 0 127.0.0.1:250.0.0.0:* LISTEN 2183/sendmail tcp0 0 0.0.0.0:43813 0.0.0.0:* LISTEN 3093/java tcp0 0 172.31.22.140:9000 0.0.0.0:* LISTEN 2905/java tcp0 0 0.0.0.0:86490.0.0.0:* LISTEN 3810/gmond tcp0 0 0.0.0.0:50090 0.0.0.0:* LISTEN 3093/java tcp0 0 0.0.0.0:86510.0.0.0:* LISTEN 3832/gmetad tcp0 0 :::8080 :::* LISTEN 23719/java tcp0 0 :::8081 :::* LISTEN 5588/java tcp0 0 :::172.31.22.140:6066 :::* LISTEN 23719/java tcp0 0 :::172.31.22.140:6067 :::* LISTEN 5588/java tcp0 0 :::22 :::* LISTEN 2144/sshd tcp0 0 ::1:631 :::* LISTEN 2095/cupsd tcp0 0 :::19998:::* LISTEN 3709/java tcp0 0 :::1:::* LISTEN 3709/java tcp0 0 :::172.31.22.140:7077 :::* LISTEN 23719/java tcp0 0 :::172.31.22.140:7078 :::* LISTEN 5588/java udp0 0 0.0.0.0:86490.0.0.0:* 3810/gmond udp0 0 0.0.0.0:631 0.0.0.0:* 2095/cupsd udp0 0 0.0.0.0:38546 0.0.0.0:* 2905/java udp0 0 0.0.0.0:68 0.0.0.0:* 1142/dhclient udp0 0 172.31.22.140:123 0.0.0.0:* 2168/ntpd udp0 0 127.0.0.1:123 0.0.0.0:* 2168/ntpd udp0 0 0.0.0.0:123 0.0.0.0:* 2168/ntpd bash-4.2# > spark master webpage links to application UI broke when running in cluster > mode > --- > > Key: SPARK-15829 > URL: https://issues.apache.org/jira/browse/SPARK-15829 > Project: Spark > Issue Type: Bug > Components: EC2 >Affects Versions: 1.6.1 > Environment: AWS ec2 cluster >Reporter: Andrew Davidson >Priority: Critical > > Hi > I created a cluster using the spark-1.6.1-bin-hadoop2.6/ec2/spark-ec2 > I use the stand alone cluster manager. I have a streaming app running in > cluster mode. I notice the master webpage links to the application UI page > are incorrect > It does not look like jira will let my upload images. I'll try and
[jira] [Commented] (SPARK-15829) spark master webpage links to application UI broke when running in cluster mode
[ https://issues.apache.org/jira/browse/SPARK-15829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15324789#comment-15324789 ] Xin Ren commented on SPARK-15829: - Hi Andy, maybe you want to check your port configuration to make sure the port is not in use. I just tried it on my cluster which is also on EC2 with v-1.6.1, and 'application detail UI' link is working properly. Just for your information. > spark master webpage links to application UI broke when running in cluster > mode > --- > > Key: SPARK-15829 > URL: https://issues.apache.org/jira/browse/SPARK-15829 > Project: Spark > Issue Type: Bug > Components: EC2 >Affects Versions: 1.6.1 > Environment: AWS ec2 cluster >Reporter: Andrew Davidson >Priority: Critical > > Hi > I created a cluster using the spark-1.6.1-bin-hadoop2.6/ec2/spark-ec2 > I use the stand alone cluster manager. I have a streaming app running in > cluster mode. I notice the master webpage links to the application UI page > are incorrect > It does not look like jira will let my upload images. I'll try and describe > the web pages and the bug > My master is running on > http://ec2-54-215-230-73.us-west-1.compute.amazonaws.com:8080/ > It has a section marked "applications". If I click on one of the running > application ids I am taken to a page showing "Executor Summary". This page > has a link to teh 'application detail UI' the url is > http://ec2-54-215-230-73.us-west-1.compute.amazonaws.com:4041/ > Notice it things the application UI is running on the cluster master. > It is actually running on the same machine as the driver on port 4041. I was > able to reverse engine the url by noticing the private ip address is part of > the worker id . For example worker-20160322041632-172.31.23.201-34909 > next I went on the aws ec2 console to find the public DNS name for this > machine > http://ec2-54-193-104-169.us-west-1.compute.amazonaws.com:4041/streaming/ > Kind regards > Andy -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
