Spark 1.6.1 packages on S3 corrupt?

2016-04-12 Thread Augustus Hong
tar: Error is not recoverable: exiting now mv: missing destination file operand after `spark' Try `mv --help' for more information. -- [image: Branch] <https://branch.io/?bmp=xink-sig> Augustus Hong Software Engineer

Spark Streaming Running Out Of Memory in 1.5.0.

2015-12-03 Thread Augustus Hong
L$2.apply(Utility.scala:256) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) > at scala.xml.Utility$.sequenceToXML(Utility.scala:256) > at scala.xml.Utility$.serialize(Utility.scala:227) > at s

Re: Upgrading Spark in EC2 clusters

2015-11-12 Thread Augustus Hong
luster in place, you'll probably have to do >> that manually. Otherwise, perhaps spark-ec2 is not the right tool, and >> instead you want one of those "grown-up" management tools like Ansible >> which can be setup to allow in-place upgrades. That'll take a bit of work, &g

Upgrading Spark in EC2 clusters

2015-11-11 Thread Augustus Hong
to spin up a new cluster every time we need to upgrade. Thanks, Augustus -- [image: Branch Metrics mobile deep linking] <http://branch.io/>* Augustus Hong* Data Analytics | Branch Metrics m 650-391-3369 | e augus...@branch.io

Re: Multiple Spark Streaming Jobs on Single Master

2015-10-23 Thread Augustus Hong
ocs/latest/submitting-applications.html >> >> # Run on a Spark standalone cluster in client deploy mode >> ./bin/spark-submit \ >> --class org.apache.spark.examples.SparkPi \ >> --master spark://207.184.161.138:7077 \ >> --executor-memory 20G \ >> *-

Multiple Spark Streaming Jobs on Single Master

2015-10-19 Thread Augustus Hong
only one job will be run, even though there are a lot of idle cores. Best, Augustus -- [image: Branch Metrics mobile deep linking] <http://branch.io/>* Augustus Hong* Data Analytics | Branch Metrics m 650-391-3369 | e augus...@branch.io

Adding / Removing worker nodes for Spark Streaming

2015-09-28 Thread Augustus Hong
branch.io/>* Augustus Hong* Data Analytics | Branch Metrics m 650-391-3369 | e augus...@branch.io