[jira] [Commented] (SPARK-7909) spark-ec2 and associated tools not py3 ready
[ https://issues.apache.org/jira/browse/SPARK-7909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14619799#comment-14619799 ] Shivaram Venkataraman commented on SPARK-7909: -- I think we merged all the Python3 patches that were open in mesos/spark-ec2 -- [~meawoppl] [~davies] can we mark this as fixed now ? > spark-ec2 and associated tools not py3 ready > > > Key: SPARK-7909 > URL: https://issues.apache.org/jira/browse/SPARK-7909 > Project: Spark > Issue Type: Improvement > Components: EC2 > Environment: ec2 python3 >Reporter: Matthew Goodman >Priority: Blocker > > At present there is not a possible permutation of tools that supports Python3 > on both the launching computer and running cluster. There are a couple > problems involved: > - There is no prebuilt spark binary with python3 support. > - spark-ec2/spark/init.sh contains inline py3 unfriendly print statements > - Config files for cluster processes don't seem to make it to all nodes in a > working format. > I have fixes for some of this, but the config and running context debugging > remains elusive to me. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7909) spark-ec2 and associated tools not py3 ready
[ https://issues.apache.org/jira/browse/SPARK-7909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14619764#comment-14619764 ] Sean Owen commented on SPARK-7909: -- In parallel, there's a suggestion to move the EC2 support out of apache/spark entirely into its own repo, where it can rejoin the other EC2 Spark support from mesos/spark-ec2. Would that make this any easier, if it could evolve separately and as one unit? > spark-ec2 and associated tools not py3 ready > > > Key: SPARK-7909 > URL: https://issues.apache.org/jira/browse/SPARK-7909 > Project: Spark > Issue Type: Improvement > Components: EC2 > Environment: ec2 python3 >Reporter: Matthew Goodman >Priority: Blocker > > At present there is not a possible permutation of tools that supports Python3 > on both the launching computer and running cluster. There are a couple > problems involved: > - There is no prebuilt spark binary with python3 support. > - spark-ec2/spark/init.sh contains inline py3 unfriendly print statements > - Config files for cluster processes don't seem to make it to all nodes in a > working format. > I have fixes for some of this, but the config and running context debugging > remains elusive to me. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7909) spark-ec2 and associated tools not py3 ready
[ https://issues.apache.org/jira/browse/SPARK-7909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14567735#comment-14567735 ] Shivaram Venkataraman commented on SPARK-7909: -- Yeah feel free to open a PR for the `print` fixes. > spark-ec2 and associated tools not py3 ready > > > Key: SPARK-7909 > URL: https://issues.apache.org/jira/browse/SPARK-7909 > Project: Spark > Issue Type: Improvement > Components: EC2 > Environment: ec2 python3 >Reporter: Matthew Goodman > > At present there is not a possible permutation of tools that supports Python3 > on both the launching computer and running cluster. There are a couple > problems involved: > - There is no prebuilt spark binary with python3 support. > - spark-ec2/spark/init.sh contains inline py3 unfriendly print statements > - Config files for cluster processes don't seem to make it to all nodes in a > working format. > I have fixes for some of this, but the config and running context debugging > remains elusive to me. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7909) spark-ec2 and associated tools not py3 ready
[
https://issues.apache.org/jira/browse/SPARK-7909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14567685#comment-14567685
]
Matthew Goodman commented on SPARK-7909:
Awesome, thanks for all the help on this. There is one (possibly unrelated)
issue remains, which is that httpd seems to fail to startup, giving the
following traceback:
{code:title=HTTPD Failure Traceback|borderStyle=solid}
Starting httpd: httpd: Syntax error on line 154 of /etc/httpd/conf/httpd.conf:
Cannot load /etc/httpd/modules/mod_authz_core.so into server:
/etc/httpd/modules/mod_authz_core.so: cannot open shared object file: No such
file or directory
{code}
Should I send in a PR [for this
change|https://github.com/3Scan/spark-ec2/commit/3416dd07c492b0cddcc98c4fa83f9e4284ed8fc9]?
> spark-ec2 and associated tools not py3 ready
>
>
> Key: SPARK-7909
> URL: https://issues.apache.org/jira/browse/SPARK-7909
> Project: Spark
> Issue Type: Improvement
> Components: EC2
> Environment: ec2 python3
>Reporter: Matthew Goodman
>
> At present there is not a possible permutation of tools that supports Python3
> on both the launching computer and running cluster. There are a couple
> problems involved:
> - There is no prebuilt spark binary with python3 support.
> - spark-ec2/spark/init.sh contains inline py3 unfriendly print statements
> - Config files for cluster processes don't seem to make it to all nodes in a
> working format.
> I have fixes for some of this, but the config and running context debugging
> remains elusive to me.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7909) spark-ec2 and associated tools not py3 ready
[ https://issues.apache.org/jira/browse/SPARK-7909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14564168#comment-14564168 ] Shivaram Venkataraman commented on SPARK-7909: -- The packages will get to S3 once the 1.4 release is finalized. We are still testing / voting on release candidates and you can follow these on the Spark developer mailing list. BTW I also have a change open at spark-ec2 for substituting the Spark version based on pattern https://github.com/mesos/spark-ec2/pull/116/files#diff-1d040c3294246f2b59643d63868fc2ad, so that should take care of picking up the binary once its released. However, feel free to send out PRs for the other python3 print fixes you had to make in init.sh etc. > spark-ec2 and associated tools not py3 ready > > > Key: SPARK-7909 > URL: https://issues.apache.org/jira/browse/SPARK-7909 > Project: Spark > Issue Type: Improvement > Components: EC2 > Environment: ec2 python3 >Reporter: Matthew Goodman > > At present there is not a possible permutation of tools that supports Python3 > on both the launching computer and running cluster. There are a couple > problems involved: > - There is no prebuilt spark binary with python3 support. > - spark-ec2/spark/init.sh contains inline py3 unfriendly print statements > - Config files for cluster processes don't seem to make it to all nodes in a > working format. > I have fixes for some of this, but the config and running context debugging > remains elusive to me. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7909) spark-ec2 and associated tools not py3 ready
[ https://issues.apache.org/jira/browse/SPARK-7909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14564166#comment-14564166 ] Matthew Goodman commented on SPARK-7909: Using the prebuilt binaries from the links provided yields a working cluster. Is there a timeline for when the spark 1.4.0 binaries make the s3 bucket? I can add the link to the spark/init.sh script, but it will bounce until the binary is actually place in the bucket. In either case I suspect the naming convention will be similar, so would a PR for the changes outlined above be a good step at this stage? > spark-ec2 and associated tools not py3 ready > > > Key: SPARK-7909 > URL: https://issues.apache.org/jira/browse/SPARK-7909 > Project: Spark > Issue Type: Improvement > Components: EC2 > Environment: ec2 python3 >Reporter: Matthew Goodman > > At present there is not a possible permutation of tools that supports Python3 > on both the launching computer and running cluster. There are a couple > problems involved: > - There is no prebuilt spark binary with python3 support. > - spark-ec2/spark/init.sh contains inline py3 unfriendly print statements > - Config files for cluster processes don't seem to make it to all nodes in a > working format. > I have fixes for some of this, but the config and running context debugging > remains elusive to me. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7909) spark-ec2 and associated tools not py3 ready
[ https://issues.apache.org/jira/browse/SPARK-7909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14563634#comment-14563634 ] Shivaram Venkataraman commented on SPARK-7909: -- Yeah I think that error comes from trying to use Hadoop2 while the cluster is configured for Hadoop1. I think the pre-built binaries shouldn't have that problem > spark-ec2 and associated tools not py3 ready > > > Key: SPARK-7909 > URL: https://issues.apache.org/jira/browse/SPARK-7909 > Project: Spark > Issue Type: Improvement > Components: EC2 > Environment: ec2 python3 >Reporter: Matthew Goodman > > At present there is not a possible permutation of tools that supports Python3 > on both the launching computer and running cluster. There are a couple > problems involved: > - There is no prebuilt spark binary with python3 support. > - spark-ec2/spark/init.sh contains inline py3 unfriendly print statements > - Config files for cluster processes don't seem to make it to all nodes in a > working format. > I have fixes for some of this, but the config and running context debugging > remains elusive to me. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7909) spark-ec2 and associated tools not py3 ready
[
https://issues.apache.org/jira/browse/SPARK-7909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14563609#comment-14563609
]
Matthew Goodman commented on SPARK-7909:
There are 11 folders in /root/spark/work/app-20150528200603-/, all with the
same traceback below different only in time of error
{code:title=Spark worker Traceback|borderStyle=solid}
15/05/28 20:06:04 INFO executor.CoarseGrainedExecutorBackend: Registered signal
handlers for [TERM, HUP, INT]
Exception in thread "main" java.lang.ExceptionInInitializerError
at
org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:146)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:245)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
Caused by: java.lang.RuntimeException:
java.lang.reflect.InvocationTargetException
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at org.apache.hadoop.security.Groups.(Groups.java:55)
at
org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:182)
at
org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:235)
at
org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:249)
at
org.apache.spark.deploy.SparkHadoopUtil.(SparkHadoopUtil.scala:50)
at
org.apache.spark.deploy.SparkHadoopUtil$.(SparkHadoopUtil.scala:353)
at
org.apache.spark.deploy.SparkHadoopUtil$.(SparkHadoopUtil.scala)
... 3 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:129)
... 10 more
Caused by: java.lang.UnsatisfiedLinkError:
org.apache.hadoop.security.JniBasedUnixGroupsMapping.anchorNative()V
at
org.apache.hadoop.security.JniBasedUnixGroupsMapping.anchorNative(Native Method)
at
org.apache.hadoop.security.JniBasedUnixGroupsMapping.(JniBasedUnixGroupsMapping.java:49)
at
org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.(JniBasedUnixGroupsMappingWithFallback.java:38)
... 15 more
{code}
My launch script is as follows:
{code:title=Spark Launch Call|borderStyle=solid}
bash spark-ec2 --spark-version=ab62d73ddb973c25de043e8e9ade7800adf244e8
--spark-ec2-git-repo=https://github.com/3scan/spark-ec2
--spark-ec2-git-branch=branch-1.4 --key-pair=blahblahblah
--identity-file=blahblahblah.pem --region us-west-2 --user-data
/home/meawoppl/repos/3scan-analysis/spark/linux-bootstrap.sh login test-cluster
{code}
I am going to try the prebuilt spark next. I suspect this is surrounding the
compiled/checked out version that I am running? Not sure.
> spark-ec2 and associated tools not py3 ready
>
>
> Key: SPARK-7909
> URL: https://issues.apache.org/jira/browse/SPARK-7909
> Project: Spark
> Issue Type: Improvement
> Components: EC2
> Environment: ec2 python3
>Reporter: Matthew Goodman
>
> At present there is not a possible permutation of tools that supports Python3
> on both the launching computer and running cluster. There are a couple
> problems involved:
> - There is no prebuilt spark binary with python3 support.
> - spark-ec2/spark/init.sh contains inline py3 unfriendly print statements
> - Config files for cluster processes don't seem to make it to all nodes in a
> working format.
> I have fixes for some of this, but the config and running context debugging
> remains elusive to me.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7909) spark-ec2 and associated tools not py3 ready
[ https://issues.apache.org/jira/browse/SPARK-7909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14563581#comment-14563581 ] Shivaram Venkataraman commented on SPARK-7909: -- You can find the 1.4 RC2 binaries at http://people.apache.org/~pwendell/spark-releases/spark-1.4.0-rc2-bin/ Also the worker logs are at `/root/spark/work///stderr` on the slave machines > spark-ec2 and associated tools not py3 ready > > > Key: SPARK-7909 > URL: https://issues.apache.org/jira/browse/SPARK-7909 > Project: Spark > Issue Type: Improvement > Components: EC2 > Environment: ec2 python3 >Reporter: Matthew Goodman > > At present there is not a possible permutation of tools that supports Python3 > on both the launching computer and running cluster. There are a couple > problems involved: > - There is no prebuilt spark binary with python3 support. > - spark-ec2/spark/init.sh contains inline py3 unfriendly print statements > - Config files for cluster processes don't seem to make it to all nodes in a > working format. > I have fixes for some of this, but the config and running context debugging > remains elusive to me. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7909) spark-ec2 and associated tools not py3 ready
[ https://issues.apache.org/jira/browse/SPARK-7909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14563488#comment-14563488 ] Matthew Goodman commented on SPARK-7909: Thanks for the leads. I had some major problems with the JIRA yesterday, was I alone in that? [~shivaram] Where are the worker logs stored? [~davies] I didn't see any of the RC binaries for spark in the EC2 in the s3 bucket: https://github.com/mesos/spark-ec2/blob/branch-1.4/spark/init.sh http://s3.amazonaws.com/spark-related-packages/ Is there somewhere else they are stored? I recall seeing the RC branches in github, but that appears to have just dissapeared, and the branch-1.4 no longer has a RC. Is the release happening today or some such? The first thing I did was [add the 1.3.1| https://github.com/3Scan/spark-ec2/commit/08d210dc8d44c07383e46fcd303c8f0c20828bcf] build that I found there, and try that. That didn't feature any of the py3 support, and crashes fast and early. Next I had it build from a checkout of the current master, which is where the above errors come from. To get a source build to work, I had to make [this change | https://github.com/3Scan/spark-ec2/commit/3416dd07c492b0cddcc98c4fa83f9e4284ed8fc9], and at least one other . . . to be determined following sorting out the above trace. > spark-ec2 and associated tools not py3 ready > > > Key: SPARK-7909 > URL: https://issues.apache.org/jira/browse/SPARK-7909 > Project: Spark > Issue Type: Improvement > Components: EC2 > Environment: ec2 python3 >Reporter: Matthew Goodman > > At present there is not a possible permutation of tools that supports Python3 > on both the launching computer and running cluster. There are a couple > problems involved: > - There is no prebuilt spark binary with python3 support. > - spark-ec2/spark/init.sh contains inline py3 unfriendly print statements > - Config files for cluster processes don't seem to make it to all nodes in a > working format. > I have fixes for some of this, but the config and running context debugging > remains elusive to me. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7909) spark-ec2 and associated tools not py3 ready
[ https://issues.apache.org/jira/browse/SPARK-7909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14563352#comment-14563352 ] Davies Liu commented on SPARK-7909: --- [~meawoppl] It's true that some tools don't work with Python3, but PySpark 1.4 should work with Python3 (it's tested with Python3 for every PR), you can test with the 1.4-RC2 package. The loggings didn't say something wrong with Python, It said the executors are keep dying. Could you check the logging of executors? > spark-ec2 and associated tools not py3 ready > > > Key: SPARK-7909 > URL: https://issues.apache.org/jira/browse/SPARK-7909 > Project: Spark > Issue Type: Improvement > Components: EC2 > Environment: ec2 python3 >Reporter: Matthew Goodman > > At present there is not a possible permutation of tools that supports Python3 > on both the launching computer and running cluster. There are a couple > problems involved: > - There is no prebuilt spark binary with python3 support. > - spark-ec2/spark/init.sh contains inline py3 unfriendly print statements > - Config files for cluster processes don't seem to make it to all nodes in a > working format. > I have fixes for some of this, but the config and running context debugging > remains elusive to me. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7909) spark-ec2 and associated tools not py3 ready
[ https://issues.apache.org/jira/browse/SPARK-7909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14562161#comment-14562161 ] Shivaram Venkataraman commented on SPARK-7909: -- Could you check the worker logs to figure out why the executors were dying ? Also regarding the points described in the JIRA 1. Regd. pre-built Spark binary, I think Spark 1.4 will be Python3 friendly, but [~davies] might be able to confirm 2. I am not sure how print statements in shell script can interfere with Python3 (cc [~nchammas] who knows more than me about Python 3) > spark-ec2 and associated tools not py3 ready > > > Key: SPARK-7909 > URL: https://issues.apache.org/jira/browse/SPARK-7909 > Project: Spark > Issue Type: Improvement > Components: EC2 > Environment: ec2 python3 >Reporter: Matthew Goodman > > At present there is not a possible permutation of tools that supports Python3 > on both the launching computer and running cluster. There are a couple > problems involved: > - There is no prebuilt spark binary with python3 support. > - spark-ec2/spark/init.sh contains inline py3 unfriendly print statements > - Config files for cluster processes don't seem to make it to all nodes in a > working format. > I have fixes for some of this, but the config and running context debugging > remains elusive to me. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7909) spark-ec2 and associated tools not py3 ready
[
https://issues.apache.org/jira/browse/SPARK-7909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14562121#comment-14562121
]
Matthew Goodman commented on SPARK-7909:
I have almost everything working, but I am getting hung-up on getting the
pyspark binary to launch things correctly. When logging into ec2:
{code:title=PySpark Output Loop|borderStyle=solid}
root@ip-172-31-6-84 ~]$ ./spark/bin/pyspark
Python 3.4.3 |Continuum Analytics, Inc.| (default, Mar 6 2015, 12:03:53)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/05/28 00:58:28 INFO SparkContext: Running Spark version 1.4.0-SNAPSHOT
15/05/28 00:58:28 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
15/05/28 00:58:28 INFO SecurityManager: Changing view acls to: root
15/05/28 00:58:28 INFO SecurityManager: Changing modify acls to: root
15/05/28 00:58:28 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(root); users with
modify permissions: Set(root)
15/05/28 00:58:29 INFO Slf4jLogger: Slf4jLogger started
15/05/28 00:58:29 INFO Remoting: Starting remoting
15/05/28 00:58:29 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://sparkDriver@172.31.6.84:59125]
15/05/28 00:58:30 INFO Utils: Successfully started service 'sparkDriver' on
port 59125.
15/05/28 00:58:30 INFO SparkEnv: Registering MapOutputTracker
15/05/28 00:58:30 INFO SparkEnv: Registering BlockManagerMaster
15/05/28 00:58:30 INFO DiskBlockManager: Created local directory at
/mnt/spark/spark-985d5a6c-150e-40ad-875f-351733a40276/blockmgr-e36c9174-ff48-42e1-bbd0-c2b0649ab751
15/05/28 00:58:30 INFO DiskBlockManager: Created local directory at
/mnt2/spark/spark-fb2a7e42-2998-4ad2-be5f-d25472727d57/blockmgr-660e39f5-3561-4bec-a042-7cab1ea8cf54
15/05/28 00:58:30 INFO MemoryStore: MemoryStore started with capacity 265.4 MB
15/05/28 00:58:30 INFO HttpFileServer: HTTP File server directory is
/mnt/spark/spark-985d5a6c-150e-40ad-875f-351733a40276/httpd-40a7ad26-25d9-482b-bc7f-68d9f126b32d
15/05/28 00:58:30 INFO HttpServer: Starting HTTP Server
15/05/28 00:58:30 INFO Server: jetty-8.y.z-SNAPSHOT
15/05/28 00:58:30 INFO AbstractConnector: Started SocketConnector@0.0.0.0:46564
15/05/28 00:58:30 INFO Utils: Successfully started service 'HTTP file server'
on port 46564.
15/05/28 00:58:30 INFO SparkEnv: Registering OutputCommitCoordinator
15/05/28 00:58:30 INFO Server: jetty-8.y.z-SNAPSHOT
15/05/28 00:58:30 INFO AbstractConnector: Started
SelectChannelConnector@0.0.0.0:4040
15/05/28 00:58:30 INFO Utils: Successfully started service 'SparkUI' on port
4040.
15/05/28 00:58:30 INFO SparkUI: Started SparkUI at
http://ec2-52-24-65-198.us-west-2.compute.amazonaws.com:4040
15/05/28 00:58:30 INFO AppClient$ClientActor: Connecting to master
akka.tcp://sparkmas...@ec2-52-24-65-198.us-west-2.compute.amazonaws.com:7077/user/Master...
15/05/28 00:58:31 INFO SparkDeploySchedulerBackend: Connected to Spark cluster
with app ID app-20150528005831-0005
15/05/28 00:58:31 INFO AppClient$ClientActor: Executor added:
app-20150528005831-0005/0 on worker-20150527230803-172.31.13.150-50730
(172.31.13.150:50730) with 2 cores
15/05/28 00:58:31 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20150528005831-0005/0 on hostPort 172.31.13.150:50730 with 2 cores, 6.0 GB
RAM
15/05/28 00:58:31 INFO AppClient$ClientActor: Executor updated:
app-20150528005831-0005/0 is now LOADING
15/05/28 00:58:31 INFO AppClient$ClientActor: Executor updated:
app-20150528005831-0005/0 is now RUNNING
15/05/28 00:58:31 INFO Utils: Successfully started service
'org.apache.spark.network.netty.NettyBlockTransferService' on port 34430.
15/05/28 00:58:31 INFO NettyBlockTransferService: Server created on 34430
15/05/28 00:58:31 INFO BlockManagerMaster: Trying to register BlockManager
15/05/28 00:58:31 INFO BlockManagerMasterEndpoint: Registering block manager
172.31.6.84:34430 with 265.4 MB RAM, BlockManagerId(driver, 172.31.6.84, 34430)
15/05/28 00:58:31 INFO BlockManagerMaster: Registered BlockManager
15/05/28 00:58:31 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready
for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 1.4.0-SNAPSHOT
/_/
Using Python version 3.4.3 (default, Mar 6 2015 12:03:53)
SparkContext available as sc, SQLContext available as sqlContext.
>>> 15/05/28 00:58:32 INFO AppClient$ClientActor: Executor updated:
>>> app-20150528005831-0005/0 is now EXITED (Command exited with code 1)
15/05/28 00:58:32 INFO SparkDeploySchedulerBackend: Executor
app-2015052800
