Starting httpd: http: Syntax error on line 154
I’m unable to access ganglia, it looks like due the web server not starting as I receive this error when I launch spark: Starting httpd: http: Syntax error on line 154 of /etc/httpd/conf/httpd.conf: Cannot load /etc/httpd/modules/mod_authz_core.so This occurs when I’m using the vanilla script. I’ve also tried modifying my spark-ec2 script in various ways in an effort to correct this problem including using different instance types and modifying the instance virtualization types. Thanks for any help!
Recent Git Builds Application WebUI Problem and Exception Stating Log directory /tmp/spark-events does not exist.
I posted about the Application WebUI error (specifically application WebUI not the master WebUI generally) and have spent at least a few hours a day for over week trying to resolve it so I’d be very grateful for any suggestions. It is quite troubling that I appear to be the only one encountering this issue and I’ve tried to include everything here which might be relevant (sorry for the length). Please see the thread Current Build Gives HTTP ERROR” https://www.mail-archive.com/user@spark.apache.org/msg18752.html https://www.mail-archive.com/user@spark.apache.org/msg18752.html to see specifics about the application webUI issue and the master log. Environment: I’m doing my spark builds and application programming in scala locally on my macbook pro in eclipse, using modified ec2 launch scripts to launch my cluster, uploading my spark builds and models to s3, and uploading applications to and submitting them from ec2. I’m using java 8 locally and also installing and using java 8 on my ec2 instances (which works with spark 1.2.0). I have a windows machine at home (macbook is work machine), but have not yet attempted to launch from there. Errors: I’ve built two different recent git versions of spark both multiple times, and when running applications both have produced an Application WebUI error and this exception: Exception in thread main java.lang.IllegalArgumentException: Log directory /tmp/spark-events does not exist. While both will display the master webUI just fine including running/completed applications, registered workers etc, when I try to access a running or completed application’s WebUI by clicking their respective link, I receive a server error. When I manually create the above log directory, the exception goes away, but the WebUI problem does not. I don’t have any strong evidence, but I suspect these errors and whatever is causing them are related. Why and How of Modifications to Launch Scripts for Installation of Unreleased Spark Versions: When using a prebuilt version of spark on my cluster everything works except the new methods I need, which I had previously added to my custom version of spark and used by building the spark-assembly.jar locally and then replacing the assembly file produced through the 1.1.0 ec2 launch scripts. However, since my pull request was accepted and can now be found in the apache/spark repository along with some additional features I’d like to use and because I’d like a more elegant permanent solution for launching a cluster and installing unreleased versions of spark to my ec2 clusters, I’ve modified the included ec2 launch scripts in this way (credit to gen tang here: https://www.mail-archive.com/user%40spark.apache.org/msg18761.html https://www.mail-archive.com/user@spark.apache.org/msg18761.html): 1. Clone the most recent git version of spark 2. Use the make-dist script 3. Tar the dist folder and upload the resulting spark-1.3.0-snapshot-hadoop1.tgz to s3 and change file permissions 4. Fork the mesos/spark-ec2 repository and modify the spark/init.sh script to do a wget of my hosted distribution instead of spark’s stable release 5. Modify my spark_ec2.py script to point to my repository. 6. Modify my spark_ec2.py script to install java 8 on my ec2 instances. (This works and does not produce the above stated errors when using a stable release like 1.2.0). Additional Possibly Related Info: As far as I can tell (I went through line by line), when I launch my recent build vs when I launch the most recent stable release the console prints almost identical INFO and WARNINGS except where you would expect things to be different e.g. version numbers. I’ve noted that after launch the prebuilt stable version does not have a /tmp/spark-events directory, but it is created when the application is launched, while it is never created in my build. Further, in my unreleased builds the application logs that I find are always stored as .inprogress files (when I set the logging directory to /root/ or add the /tmp/spark-events directory manually) even after completion, which I believe is supposed to change to .completed (or something similar) when the application finishes. Thanks for any help!
Re: Current Build Gives HTTP ERROR
Yeah upon running the test locally I receive: Pi is roughly 3.139948” So spark is working, it’s just the application ui that is not… On Jan 13, 2015, at 1:13 PM, Ganon Pierce ganon.pie...@me.com wrote: My application logs remain stored as .inprogress files, e.g. app-20150113190025-0004.inprogress” even after completion, could this have something to do with what is going on. @ Ted Yu Where do I find the master log? It’s not very obviously labeled in my /tmp/ directory. Sorry if I should know, I’ve read through the documentation many times, but probably missed it. Do I need to set this in my configuration somehow? Could the problem actually be that I’m just not setting a conf property properly somewhere? @Robin East All of these work fine, the only issue I’m having is viewing the application ui after I submit a job to run. Further, the application runs and will even compute some of the outputs I want (they get stored to s3 and I’ve used the outputs in other models I’m making). It’s simply when I access the master web ui, click the application that is currently running, and then click “Application Detail UI” that I receive this error. When I an application has completed and I click the same link I receive: “Application history not found (app-201501131190025-004) On Jan 13, 2015, at 8:07 AM, Ted Yu yuzhih...@gmail.com mailto:yuzhih...@gmail.com wrote: Gabon: Can you check the master log to see if there is some clue ? Cheers On Jan 13, 2015, at 2:03 AM, Robin East robin.e...@xense.co.uk mailto:robin.e...@xense.co.uk wrote: I’ve just pulled down the latest commits from github, and done the following: 1) mvn clean package -DskipTests builds fine 2) ./bin/spark-shell works 3) run SparkPi example with no problems: ./bin/run-example SparkPi 10 4) Started a master ./sbin/start-master.sh grabbed the MasterWebUI from the master log - Started MasterWebUI at http://x.x.x.x:8080 http://x.x.x.x:8080/ Can view the MasterWebUI from local browser 5) grabbed the spark url from the master log and started a local slave: ./bin/spark-class org.apache.spark.deploy.worker.Worker spark://hostname:7077 6) Ran jps to confirm both Master and Worker processes are present. 7) Ran SparkPi on the mini-cluster: MASTER=spark://host:7077 ./bin/run-example SparkPi 10 All worked fine, can see information in the MasterWebUI Which of these stops doesn’t work for you? I presume you’ve tried re-pulling from git and a clean build again. Robin On 13 Jan 2015, at 08:07, Ganon Pierce ganon.pie...@me.com mailto:ganon.pie...@me.com wrote: After clean build still receiving the same error. On Jan 6, 2015, at 3:59 PM, Sean Owen so...@cloudera.com mailto:so...@cloudera.com wrote: FWIW I do not see any such error, after a mvn -DskipTests clean package and ./bin/spark-shell from master. Maybe double-check you have done a full clean build. On Tue, Jan 6, 2015 at 9:09 PM, Ganon Pierce ganon.pie...@me.com mailto:ganon.pie...@me.com wrote: I’m attempting to build from the latest commit on git and receive the following error upon attempting to access the application web ui: HTTP ERROR: 500 Problem accessing /jobs/. Reason: Server Error Powered by Jetty:// My driver also prints this error: java.lang.UnsupportedOperationException: empty.max at scala.collection.TraversableOnce$class.max(TraversableOnce.scala:216) at scala.collection.AbstractTraversable.max(Traversable.scala:105) at org.apache.spark.ui.jobs.AllJobsPage.org http://org.apache.spark.ui.jobs.alljobspage.org/$apache$spark$ui$jobs$AllJobsPage$$makeRow$1(AllJobsPage.scala:46) at org.apache.spark.ui.jobs.AllJobsPage$$anonfun$jobsTable$1.apply(AllJobsPage.scala:91) at org.apache.spark.ui.jobs.AllJobsPage$$anonfun$jobsTable$1.apply(AllJobsPage.scala:91) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.List.foreach(List.scala:318) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at org.apache.spark.ui.jobs.AllJobsPage.jobsTable(AllJobsPage.scala:91) at org.apache.spark.ui.jobs.AllJobsPage.render(AllJobsPage.scala:106) at org.apache.spark.ui.WebUI$$anonfun$attachPage$1.apply(WebUI.scala:68) at org.apache.spark.ui.WebUI$$anonfun$attachPage$1.apply(WebUI.scala:68) at org.apache.spark.ui.JettyUtils$$anon$1.doGet(JettyUtils.scala:68) at javax.servlet.http.HttpServlet.service(HttpServlet.java:735) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501
Re: Current Build Gives HTTP ERROR
Perhaps I need to change my spark.eventLog.dir to an hdfs directory? Could this have something to do with the “history server” not having access to my application logs? On Jan 13, 2015, at 1:13 PM, Ganon Pierce ganon.pie...@me.com wrote: My application logs remain stored as .inprogress files, e.g. app-20150113190025-0004.inprogress” even after completion, could this have something to do with what is going on. @ Ted Yu Where do I find the master log? It’s not very obviously labeled in my /tmp/ directory. Sorry if I should know, I’ve read through the documentation many times, but probably missed it. Do I need to set this in my configuration somehow? Could the problem actually be that I’m just not setting a conf property properly somewhere? @Robin East All of these work fine, the only issue I’m having is viewing the application ui after I submit a job to run. Further, the application runs and will even compute some of the outputs I want (they get stored to s3 and I’ve used the outputs in other models I’m making). It’s simply when I access the master web ui, click the application that is currently running, and then click “Application Detail UI” that I receive this error. When I an application has completed and I click the same link I receive: “Application history not found (app-201501131190025-004) On Jan 13, 2015, at 8:07 AM, Ted Yu yuzhih...@gmail.com mailto:yuzhih...@gmail.com wrote: Gabon: Can you check the master log to see if there is some clue ? Cheers On Jan 13, 2015, at 2:03 AM, Robin East robin.e...@xense.co.uk mailto:robin.e...@xense.co.uk wrote: I’ve just pulled down the latest commits from github, and done the following: 1) mvn clean package -DskipTests builds fine 2) ./bin/spark-shell works 3) run SparkPi example with no problems: ./bin/run-example SparkPi 10 4) Started a master ./sbin/start-master.sh grabbed the MasterWebUI from the master log - Started MasterWebUI at http://x.x.x.x:8080 http://x.x.x.x:8080/ Can view the MasterWebUI from local browser 5) grabbed the spark url from the master log and started a local slave: ./bin/spark-class org.apache.spark.deploy.worker.Worker spark://hostname:7077 6) Ran jps to confirm both Master and Worker processes are present. 7) Ran SparkPi on the mini-cluster: MASTER=spark://host:7077 ./bin/run-example SparkPi 10 All worked fine, can see information in the MasterWebUI Which of these stops doesn’t work for you? I presume you’ve tried re-pulling from git and a clean build again. Robin On 13 Jan 2015, at 08:07, Ganon Pierce ganon.pie...@me.com mailto:ganon.pie...@me.com wrote: After clean build still receiving the same error. On Jan 6, 2015, at 3:59 PM, Sean Owen so...@cloudera.com mailto:so...@cloudera.com wrote: FWIW I do not see any such error, after a mvn -DskipTests clean package and ./bin/spark-shell from master. Maybe double-check you have done a full clean build. On Tue, Jan 6, 2015 at 9:09 PM, Ganon Pierce ganon.pie...@me.com mailto:ganon.pie...@me.com wrote: I’m attempting to build from the latest commit on git and receive the following error upon attempting to access the application web ui: HTTP ERROR: 500 Problem accessing /jobs/. Reason: Server Error Powered by Jetty:// My driver also prints this error: java.lang.UnsupportedOperationException: empty.max at scala.collection.TraversableOnce$class.max(TraversableOnce.scala:216) at scala.collection.AbstractTraversable.max(Traversable.scala:105) at org.apache.spark.ui.jobs.AllJobsPage.org http://org.apache.spark.ui.jobs.alljobspage.org/$apache$spark$ui$jobs$AllJobsPage$$makeRow$1(AllJobsPage.scala:46) at org.apache.spark.ui.jobs.AllJobsPage$$anonfun$jobsTable$1.apply(AllJobsPage.scala:91) at org.apache.spark.ui.jobs.AllJobsPage$$anonfun$jobsTable$1.apply(AllJobsPage.scala:91) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.List.foreach(List.scala:318) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at org.apache.spark.ui.jobs.AllJobsPage.jobsTable(AllJobsPage.scala:91) at org.apache.spark.ui.jobs.AllJobsPage.render(AllJobsPage.scala:106) at org.apache.spark.ui.WebUI$$anonfun$attachPage$1.apply(WebUI.scala:68) at org.apache.spark.ui.WebUI$$anonfun$attachPage$1.apply(WebUI.scala:68) at org.apache.spark.ui.JettyUtils$$anon$1.doGet(JettyUtils.scala:68) at javax.servlet.http.HttpServlet.service(HttpServlet.java:735) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) at org.eclipse.jetty.servlet.ServletHandler.doHandle
Re: Current Build Gives HTTP ERROR
My application logs remain stored as .inprogress files, e.g. app-20150113190025-0004.inprogress” even after completion, could this have something to do with what is going on. @ Ted Yu Where do I find the master log? It’s not very obviously labeled in my /tmp/ directory. Sorry if I should know, I’ve read through the documentation many times, but probably missed it. Do I need to set this in my configuration somehow? Could the problem actually be that I’m just not setting a conf property properly somewhere? @Robin East All of these work fine, the only issue I’m having is viewing the application ui after I submit a job to run. Further, the application runs and will even compute some of the outputs I want (they get stored to s3 and I’ve used the outputs in other models I’m making). It’s simply when I access the master web ui, click the application that is currently running, and then click “Application Detail UI” that I receive this error. When I an application has completed and I click the same link I receive: “Application history not found (app-201501131190025-004) On Jan 13, 2015, at 8:07 AM, Ted Yu yuzhih...@gmail.com wrote: Gabon: Can you check the master log to see if there is some clue ? Cheers On Jan 13, 2015, at 2:03 AM, Robin East robin.e...@xense.co.uk mailto:robin.e...@xense.co.uk wrote: I’ve just pulled down the latest commits from github, and done the following: 1) mvn clean package -DskipTests builds fine 2) ./bin/spark-shell works 3) run SparkPi example with no problems: ./bin/run-example SparkPi 10 4) Started a master ./sbin/start-master.sh grabbed the MasterWebUI from the master log - Started MasterWebUI at http://x.x.x.x:8080 http://x.x.x.x:8080/ Can view the MasterWebUI from local browser 5) grabbed the spark url from the master log and started a local slave: ./bin/spark-class org.apache.spark.deploy.worker.Worker spark://hostname:7077 6) Ran jps to confirm both Master and Worker processes are present. 7) Ran SparkPi on the mini-cluster: MASTER=spark://host:7077 ./bin/run-example SparkPi 10 All worked fine, can see information in the MasterWebUI Which of these stops doesn’t work for you? I presume you’ve tried re-pulling from git and a clean build again. Robin On 13 Jan 2015, at 08:07, Ganon Pierce ganon.pie...@me.com mailto:ganon.pie...@me.com wrote: After clean build still receiving the same error. On Jan 6, 2015, at 3:59 PM, Sean Owen so...@cloudera.com mailto:so...@cloudera.com wrote: FWIW I do not see any such error, after a mvn -DskipTests clean package and ./bin/spark-shell from master. Maybe double-check you have done a full clean build. On Tue, Jan 6, 2015 at 9:09 PM, Ganon Pierce ganon.pie...@me.com mailto:ganon.pie...@me.com wrote: I’m attempting to build from the latest commit on git and receive the following error upon attempting to access the application web ui: HTTP ERROR: 500 Problem accessing /jobs/. Reason: Server Error Powered by Jetty:// My driver also prints this error: java.lang.UnsupportedOperationException: empty.max at scala.collection.TraversableOnce$class.max(TraversableOnce.scala:216) at scala.collection.AbstractTraversable.max(Traversable.scala:105) at org.apache.spark.ui.jobs.AllJobsPage.org http://org.apache.spark.ui.jobs.alljobspage.org/$apache$spark$ui$jobs$AllJobsPage$$makeRow$1(AllJobsPage.scala:46) at org.apache.spark.ui.jobs.AllJobsPage$$anonfun$jobsTable$1.apply(AllJobsPage.scala:91) at org.apache.spark.ui.jobs.AllJobsPage$$anonfun$jobsTable$1.apply(AllJobsPage.scala:91) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.List.foreach(List.scala:318) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at org.apache.spark.ui.jobs.AllJobsPage.jobsTable(AllJobsPage.scala:91) at org.apache.spark.ui.jobs.AllJobsPage.render(AllJobsPage.scala:106) at org.apache.spark.ui.WebUI$$anonfun$attachPage$1.apply(WebUI.scala:68) at org.apache.spark.ui.WebUI$$anonfun$attachPage$1.apply(WebUI.scala:68) at org.apache.spark.ui.JettyUtils$$anon$1.doGet(JettyUtils.scala:68) at javax.servlet.http.HttpServlet.service(HttpServlet.java:735) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java
Re: Current Build Gives HTTP ERROR
Also, thanks for everyone’s help so far! On Jan 13, 2015, at 2:04 PM, Ganon Pierce ganon.pie...@me.com wrote: Yeah upon running the test locally I receive: Pi is roughly 3.139948” So spark is working, it’s just the application ui that is not… On Jan 13, 2015, at 1:13 PM, Ganon Pierce ganon.pie...@me.com mailto:ganon.pie...@me.com wrote: My application logs remain stored as .inprogress files, e.g. app-20150113190025-0004.inprogress” even after completion, could this have something to do with what is going on. @ Ted Yu Where do I find the master log? It’s not very obviously labeled in my /tmp/ directory. Sorry if I should know, I’ve read through the documentation many times, but probably missed it. Do I need to set this in my configuration somehow? Could the problem actually be that I’m just not setting a conf property properly somewhere? @Robin East All of these work fine, the only issue I’m having is viewing the application ui after I submit a job to run. Further, the application runs and will even compute some of the outputs I want (they get stored to s3 and I’ve used the outputs in other models I’m making). It’s simply when I access the master web ui, click the application that is currently running, and then click “Application Detail UI” that I receive this error. When I an application has completed and I click the same link I receive: “Application history not found (app-201501131190025-004) On Jan 13, 2015, at 8:07 AM, Ted Yu yuzhih...@gmail.com mailto:yuzhih...@gmail.com wrote: Gabon: Can you check the master log to see if there is some clue ? Cheers On Jan 13, 2015, at 2:03 AM, Robin East robin.e...@xense.co.uk mailto:robin.e...@xense.co.uk wrote: I’ve just pulled down the latest commits from github, and done the following: 1) mvn clean package -DskipTests builds fine 2) ./bin/spark-shell works 3) run SparkPi example with no problems: ./bin/run-example SparkPi 10 4) Started a master ./sbin/start-master.sh grabbed the MasterWebUI from the master log - Started MasterWebUI at http://x.x.x.x:8080 http://x.x.x.x:8080/ Can view the MasterWebUI from local browser 5) grabbed the spark url from the master log and started a local slave: ./bin/spark-class org.apache.spark.deploy.worker.Worker spark://hostname:7077 6) Ran jps to confirm both Master and Worker processes are present. 7) Ran SparkPi on the mini-cluster: MASTER=spark://host:7077 ./bin/run-example SparkPi 10 All worked fine, can see information in the MasterWebUI Which of these stops doesn’t work for you? I presume you’ve tried re-pulling from git and a clean build again. Robin On 13 Jan 2015, at 08:07, Ganon Pierce ganon.pie...@me.com mailto:ganon.pie...@me.com wrote: After clean build still receiving the same error. On Jan 6, 2015, at 3:59 PM, Sean Owen so...@cloudera.com mailto:so...@cloudera.com wrote: FWIW I do not see any such error, after a mvn -DskipTests clean package and ./bin/spark-shell from master. Maybe double-check you have done a full clean build. On Tue, Jan 6, 2015 at 9:09 PM, Ganon Pierce ganon.pie...@me.com mailto:ganon.pie...@me.com wrote: I’m attempting to build from the latest commit on git and receive the following error upon attempting to access the application web ui: HTTP ERROR: 500 Problem accessing /jobs/. Reason: Server Error Powered by Jetty:// My driver also prints this error: java.lang.UnsupportedOperationException: empty.max at scala.collection.TraversableOnce$class.max(TraversableOnce.scala:216) at scala.collection.AbstractTraversable.max(Traversable.scala:105) at org.apache.spark.ui.jobs.AllJobsPage.org http://org.apache.spark.ui.jobs.alljobspage.org/$apache$spark$ui$jobs$AllJobsPage$$makeRow$1(AllJobsPage.scala:46) at org.apache.spark.ui.jobs.AllJobsPage$$anonfun$jobsTable$1.apply(AllJobsPage.scala:91) at org.apache.spark.ui.jobs.AllJobsPage$$anonfun$jobsTable$1.apply(AllJobsPage.scala:91) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.List.foreach(List.scala:318) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at org.apache.spark.ui.jobs.AllJobsPage.jobsTable(AllJobsPage.scala:91) at org.apache.spark.ui.jobs.AllJobsPage.render(AllJobsPage.scala:106) at org.apache.spark.ui.WebUI$$anonfun$attachPage$1.apply(WebUI.scala:68) at org.apache.spark.ui.WebUI$$anonfun$attachPage$1.apply(WebUI.scala:68) at org.apache.spark.ui.JettyUtils$$anon$1.doGet(JettyUtils.scala:68) at javax.servlet.http.HttpServlet.service(HttpServlet.java:735) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848
Re: Current Build Gives HTTP ERROR
After clean build still receiving the same error. On Jan 6, 2015, at 3:59 PM, Sean Owen so...@cloudera.com wrote: FWIW I do not see any such error, after a mvn -DskipTests clean package and ./bin/spark-shell from master. Maybe double-check you have done a full clean build. On Tue, Jan 6, 2015 at 9:09 PM, Ganon Pierce ganon.pie...@me.com wrote: I’m attempting to build from the latest commit on git and receive the following error upon attempting to access the application web ui: HTTP ERROR: 500 Problem accessing /jobs/. Reason: Server Error Powered by Jetty:// My driver also prints this error: java.lang.UnsupportedOperationException: empty.max at scala.collection.TraversableOnce$class.max(TraversableOnce.scala:216) at scala.collection.AbstractTraversable.max(Traversable.scala:105) at org.apache.spark.ui.jobs.AllJobsPage.org$apache$spark$ui$jobs$AllJobsPage$$makeRow$1(AllJobsPage.scala:46) at org.apache.spark.ui.jobs.AllJobsPage$$anonfun$jobsTable$1.apply(AllJobsPage.scala:91) at org.apache.spark.ui.jobs.AllJobsPage$$anonfun$jobsTable$1.apply(AllJobsPage.scala:91) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.List.foreach(List.scala:318) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at org.apache.spark.ui.jobs.AllJobsPage.jobsTable(AllJobsPage.scala:91) at org.apache.spark.ui.jobs.AllJobsPage.render(AllJobsPage.scala:106) at org.apache.spark.ui.WebUI$$anonfun$attachPage$1.apply(WebUI.scala:68) at org.apache.spark.ui.WebUI$$anonfun$attachPage$1.apply(WebUI.scala:68) at org.apache.spark.ui.JettyUtils$$anon$1.doGet(JettyUtils.scala:68) at javax.servlet.http.HttpServlet.service(HttpServlet.java:735) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) Has the ui been disabled intentionally for development purposes, have I not set something up correctly, or is this a bug?
Re: Current Build Gives HTTP ERROR
Yep did this and can view the masterwebui no problem: 4) Started a master ./sbin/start-master.sh grabbed the MasterWebUI from the master log - Started MasterWebUI at http://x.x.x.x:8080 http://x.x.x.x:8080/ Can view the MasterWebUI from local browser However, cannot see view the app UI in this case as the app completes too quickly, and it is not set to log events. I set up logging and see what happens. On Jan 13, 2015, at 4:03 AM, Robin East robin.e...@xense.co.uk wrote: 4) Started a master ./sbin/start-master.sh grabbed the MasterWebUI from the master log - Started MasterWebUI at http://x.x.x.x:8080 http://x.x.x.x:8080/ Can view the MasterWebUI from local browser
Re: Current Build Gives HTTP ERROR
Here is the master log: Spark Command: /usr/lib/jvm/java-1.7.0/bin/java -cp ::/root/ephemeral-hdfs/conf:/root/spark/sbin/../conf:/root/spark/lib/spark-assembly-1.3.0-SNAPSHOT-hadoop1.0.4.jar -XX:MaxPermSize=128m -Dspark.akka.logLifecycleEvents=true -Xms512m -Xmx512m org.apache.spark.deploy.master.Master --ip ec2-54-148-20-235.us-west-2.compute.amazonaws.com --port 7077 --webui-port 8080 15/01/13 16:25:54 INFO master.Master: Registered signal handlers for [TERM, HUP, INT] 15/01/13 16:25:54 INFO spark.SecurityManager: Changing view acls to: root 15/01/13 16:25:54 INFO spark.SecurityManager: Changing modify acls to: root 15/01/13 16:25:54 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root) 15/01/13 16:25:55 INFO slf4j.Slf4jLogger: Slf4jLogger started 15/01/13 16:25:55 INFO Remoting: Starting remoting 15/01/13 16:25:55 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkmas...@ec2-54-148-20-235.us-west-2.compute.amazonaws.com:7077] 15/01/13 16:25:55 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkmas...@ec2-54-148-20-235.us-west-2.compute.amazonaws.com:7077] 15/01/13 16:25:55 INFO util.Utils: Successfully started service 'sparkMaster' on port 7077. 15/01/13 16:25:55 INFO master.Master: Starting Spark master at spark://ec2-54-148-20-235.us-west-2.compute.amazonaws.com:7077 15/01/13 16:25:55 INFO master.Master: Running Spark version 1.3.0-SNAPSHOT 15/01/13 16:25:55 INFO server.Server: jetty-8.y.z-SNAPSHOT 15/01/13 16:25:55 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:8080 15/01/13 16:25:55 INFO util.Utils: Successfully started service 'MasterUI' on port 8080. 15/01/13 16:25:55 INFO ui.MasterWebUI: Started MasterWebUI at http://ec2-54-148-20-235.us-west-2.compute.amazonaws.com:8080 15/01/13 16:25:55 INFO master.Master: I have been elected leader! New state: ALIVE 15/01/13 16:26:17 INFO master.Master: Registering worker ip-172-31-14-253.us-west-2.compute.internal:36568 with 32 cores, 57.7 GB RAM 15/01/13 16:26:17 INFO master.Master: Registering worker ip-172-31-10-227.us-west-2.compute.internal:50140 with 32 cores, 57.7 GB RAM 15/01/13 16:26:17 INFO master.Master: Registering worker ip-172-31-9-37.us-west-2.compute.internal:34991 with 32 cores, 57.7 GB RAM 15/01/13 16:26:17 INFO master.Master: Registering worker ip-172-31-2-174.us-west-2.compute.internal:58511 with 32 cores, 57.7 GB RAM 15/01/13 16:31:49 INFO master.Master: Registering app test2 15/01/13 16:31:49 INFO master.Master: Registered app test2 with ID app-20150113163149- 15/01/13 16:31:49 INFO master.Master: Launching executor app-20150113163149-/0 on worker worker-20150113162617-ip-172-31-2-174.us-west-2.compute.internal-58511 15/01/13 16:31:49 INFO master.Master: Launching executor app-20150113163149-/1 on worker worker-20150113162617-ip-172-31-14-253.us-west-2.compute.internal-36568 15/01/13 16:31:49 INFO master.Master: Launching executor app-20150113163149-/2 on worker worker-20150113162617-ip-172-31-9-37.us-west-2.compute.internal-34991 15/01/13 16:31:49 INFO master.Master: Launching executor app-20150113163149-/3 on worker worker-20150113162617-ip-172-31-10-227.us-west-2.compute.internal-50140 15/01/13 16:32:50 INFO master.Master: akka.tcp://sparkdri...@ip-172-31-10-58.us-west-2.compute.internal:37612 got disassociated, removing it. 15/01/13 16:32:50 INFO master.Master: Removing app app-20150113163149- 15/01/13 16:32:50 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkdri...@ip-172-31-10-58.us-west-2.compute.internal:37612] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. 15/01/13 16:32:50 INFO actor.LocalActorRef: Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%40172.31.10.58%3A35980-5#873817571] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 15/01/13 16:32:50 WARN master.Master: Application test2 is still in progress. 15/01/13 16:32:50 INFO master.Master: akka.tcp://sparkdri...@ip-172-31-10-58.us-west-2.compute.internal:37612 got disassociated, removing it. 15/01/13 16:32:51 WARN master.Master: Got status update for unknown executor app-20150113163149-/1 15/01/13 16:32:51 WARN master.Master: Got status update for unknown executor app-20150113163149-/3 15/01/13 16:32:51 WARN master.Master: Got status update for unknown executor app-20150113163149-/2 15/01/13 16:32:52 WARN master.Master: Got status update for unknown executor app-20150113163149-/0 15/01/13
Using ec2 launch script with locally built version of spark?
Is there a way to use the ec2 launch script with a locally built version of spark? I launch and destroy clusters pretty frequently and would like to not have to wait each time for the master instance to compile the source as happens when I set the -v tag with the latest git commit. To be clear, I would like to launch a non-release version of spark compiled locally as quickly as I can launch a release version (e.g. -v 1.2.0) which does not have to be compiled upon launch. Up to this point, I have just used the launch script included with the latest release to set up the cluster and then manually replaced the assembly file on the master and slaves with the version I built locally and then stored on s3. Is there anything wrong with doing it this way? Further, is there a better or more standard way of accomplishing this? - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Driver hangs on running mllib word2vec
Two billion words is a very large vocabulary… You can try solving this issue by by setting the number of times words must occur in order to be included in the vocabulary using setMinCount, this will be prevent common misspellings, websites, and other things from being included and may improve the quality of your model overall. On Jan 6, 2015, at 12:59 AM, Eric Zhen zhpeng...@gmail.com wrote: Thanks Zhan, I'm also confused about the jstack output, why the driver gets stuck at org.apache.spark.SparkContext.clean ? On Tue, Jan 6, 2015 at 2:10 PM, Zhan Zhang zzh...@hortonworks.com mailto:zzh...@hortonworks.com wrote: I think it is overflow. The training data is quite big. The algorithms scalability highly depends on the vocabSize. Even without overflow, there are still other bottlenecks, for example, syn0Global and syn1Global, each of them has vocabSize * vectorSize elements. Thanks. Zhan Zhang On Jan 5, 2015, at 7:47 PM, Eric Zhen zhpeng...@gmail.com mailto:zhpeng...@gmail.com wrote: Hi Xiangrui, Our dataset is about 80GB(10B lines). In the driver's log, we foud this: INFO Word2Vec: trainWordsCount = -1610413239 it seems that there is a integer overflow? On Tue, Jan 6, 2015 at 5:44 AM, Xiangrui Meng men...@gmail.com mailto:men...@gmail.com wrote: How big is your dataset, and what is the vocabulary size? -Xiangrui On Sun, Jan 4, 2015 at 11:18 PM, Eric Zhen zhpeng...@gmail.com mailto:zhpeng...@gmail.com wrote: Hi, When we run mllib word2vec(spark-1.1.0), driver get stuck with 100% cup usage. Here is the jstack output: main prio=10 tid=0x40112800 nid=0x46f2 runnable [0x4162e000] java.lang.Thread.State: RUNNABLE at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1847) at java.io.ObjectOutputStream$BlockDataOutputStream.write(ObjectOutputStream.java:1778) at java.io.DataOutputStream.writeInt(DataOutputStream.java:182) at java.io.DataOutputStream.writeFloat(DataOutputStream.java:225) at java.io.ObjectOutputStream$BlockDataOutputStream.writeFloats(ObjectOutputStream.java:2064) at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1310) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1154) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1518) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1483) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1400) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1518) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1483) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1400) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1518) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1483) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1400) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330) at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42) at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:73) at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:164) at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:158) at org.apache.spark.SparkContext.clean(SparkContext.scala:1242) at org.apache.spark.rdd.RDD.mapPartitionsWithIndex(RDD.scala:610) at org.apache.spark.mllib.feature.Word2Vec$$anonfun$fit$1.apply$mcVI$sp(Word2Vec.scala:291) at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141) at org.apache.spark.mllib.feature.Word2Vec.fit(Word2Vec.scala:290) at com.baidu.inf.WordCount$.main(WordCount.scala:31) at com.baidu.inf.WordCount.main(WordCount.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:328) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Re: Driver hangs on running mllib word2vec
Oops, just kidding, this method is not in the current release. However, it is included in the latest commit on git if you want to do a build. On Jan 6, 2015, at 2:56 PM, Ganon Pierce ganon.pie...@me.com wrote: Two billion words is a very large vocabulary… You can try solving this issue by by setting the number of times words must occur in order to be included in the vocabulary using setMinCount, this will be prevent common misspellings, websites, and other things from being included and may improve the quality of your model overall. On Jan 6, 2015, at 12:59 AM, Eric Zhen zhpeng...@gmail.com mailto:zhpeng...@gmail.com wrote: Thanks Zhan, I'm also confused about the jstack output, why the driver gets stuck at org.apache.spark.SparkContext.clean ? On Tue, Jan 6, 2015 at 2:10 PM, Zhan Zhang zzh...@hortonworks.com mailto:zzh...@hortonworks.com wrote: I think it is overflow. The training data is quite big. The algorithms scalability highly depends on the vocabSize. Even without overflow, there are still other bottlenecks, for example, syn0Global and syn1Global, each of them has vocabSize * vectorSize elements. Thanks. Zhan Zhang On Jan 5, 2015, at 7:47 PM, Eric Zhen zhpeng...@gmail.com mailto:zhpeng...@gmail.com wrote: Hi Xiangrui, Our dataset is about 80GB(10B lines). In the driver's log, we foud this: INFO Word2Vec: trainWordsCount = -1610413239 it seems that there is a integer overflow? On Tue, Jan 6, 2015 at 5:44 AM, Xiangrui Meng men...@gmail.com mailto:men...@gmail.com wrote: How big is your dataset, and what is the vocabulary size? -Xiangrui On Sun, Jan 4, 2015 at 11:18 PM, Eric Zhen zhpeng...@gmail.com mailto:zhpeng...@gmail.com wrote: Hi, When we run mllib word2vec(spark-1.1.0), driver get stuck with 100% cup usage. Here is the jstack output: main prio=10 tid=0x40112800 nid=0x46f2 runnable [0x4162e000] java.lang.Thread.State: RUNNABLE at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1847) at java.io.ObjectOutputStream$BlockDataOutputStream.write(ObjectOutputStream.java:1778) at java.io.DataOutputStream.writeInt(DataOutputStream.java:182) at java.io.DataOutputStream.writeFloat(DataOutputStream.java:225) at java.io.ObjectOutputStream$BlockDataOutputStream.writeFloats(ObjectOutputStream.java:2064) at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1310) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1154) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1518) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1483) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1400) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1518) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1483) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1400) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1518) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1483) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1400) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330) at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42) at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:73) at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:164) at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:158) at org.apache.spark.SparkContext.clean(SparkContext.scala:1242) at org.apache.spark.rdd.RDD.mapPartitionsWithIndex(RDD.scala:610) at org.apache.spark.mllib.feature.Word2Vec$$anonfun$fit$1.apply$mcVI$sp(Word2Vec.scala:291) at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141) at org.apache.spark.mllib.feature.Word2Vec.fit(Word2Vec.scala:290) at com.baidu.inf.WordCount$.main(WordCount.scala:31) at com.baidu.inf.WordCount.main(WordCount.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25
Current Build Gives HTTP ERROR
I’m attempting to build from the latest commit on git and receive the following error upon attempting to access the application web ui: HTTP ERROR: 500 Problem accessing /jobs/. Reason: Server Error Powered by Jetty:// My driver also prints this error: java.lang.UnsupportedOperationException: empty.max at scala.collection.TraversableOnce$class.max(TraversableOnce.scala:216) at scala.collection.AbstractTraversable.max(Traversable.scala:105) at org.apache.spark.ui.jobs.AllJobsPage.org$apache$spark$ui$jobs$AllJobsPage$$makeRow$1(AllJobsPage.scala:46) at org.apache.spark.ui.jobs.AllJobsPage$$anonfun$jobsTable$1.apply(AllJobsPage.scala:91) at org.apache.spark.ui.jobs.AllJobsPage$$anonfun$jobsTable$1.apply(AllJobsPage.scala:91) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.List.foreach(List.scala:318) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at org.apache.spark.ui.jobs.AllJobsPage.jobsTable(AllJobsPage.scala:91) at org.apache.spark.ui.jobs.AllJobsPage.render(AllJobsPage.scala:106) at org.apache.spark.ui.WebUI$$anonfun$attachPage$1.apply(WebUI.scala:68) at org.apache.spark.ui.WebUI$$anonfun$attachPage$1.apply(WebUI.scala:68) at org.apache.spark.ui.JettyUtils$$anon$1.doGet(JettyUtils.scala:68) at javax.servlet.http.HttpServlet.service(HttpServlet.java:735) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) Has the ui been disabled intentionally for development purposes, have I not set something up correctly, or is this a bug?
