java.lang.IllegalArgumentException: requirement failed: sizeInBytes was negative: -9223372036842471144

2014-10-28 Thread Ruebenacker, Oliver A
Hello, I have a Spark app which I run with master local[3]. When running without any persist calls, it seems to work fine, but as soon as I add persist calls (at default storage level), it fails at the first persist call with the message below. Unfortunately, I can't post the code.

What is akka-actor_2.10-2.2.3-shaded-protobuf.jar?

2014-10-17 Thread Ruebenacker, Oliver A
Hello, My SBT pulls in, among others, the following dependency for Spark 1.1.0: akka-actor_2.10-2.2.3-shaded-protobuf.jar What is this? How is this different from the regular Akka Actor JAR? How do I reconcile with other libs that use Akka, such as Play? Thanks! Best,

Spark as a Library

2014-09-16 Thread Ruebenacker, Oliver A
Hello, Suppose I want to use Spark from an application that I already submit to run in another container (e.g. Tomcat). Is this at all possible? Or do I have to split the app into two components, and submit one to Spark and one to the other container? In that case, what is the

RE: Spark as a Library

2014-09-16 Thread Ruebenacker, Oliver A
the script. Thanks! Best, Oliver From: Matei Zaharia [mailto:matei.zaha...@gmail.com] Sent: Tuesday, September 16, 2014 1:31 PM To: Ruebenacker, Oliver A; user@spark.apache.org Subject: Re: Spark as a Library If you want to run the computation on just one machine (using Spark's local mode

RE: Web UI

2014-09-05 Thread Ruebenacker, Oliver A
, September 04, 2014 2:11 PM To: Ruebenacker, Oliver A Cc: Akhil Das; Wonha Ryu; user@spark.apache.org Subject: Re: Web UI Hi all, The JSON version of the web UI is not officially supported; I don't believe this is documented anywhere. The alternative is to set `spark.eventLog.enabled` to true

RE: Programatically running of the Spark Jobs.

2014-09-04 Thread Ruebenacker, Oliver A
Hello, Can this be used as a library from within another application? Thanks! Best, Oliver From: Matt Chu [mailto:m...@kabam.com] Sent: Thursday, September 04, 2014 2:46 AM To: Vicky Kak Cc: user Subject: Re: Programatically running of the Spark Jobs.

RE: Web UI

2014-09-04 Thread Ruebenacker, Oliver A
, 2014 3:09 AM To: Ruebenacker, Oliver A Cc: Wonha Ryu; user@spark.apache.org Subject: Re: Web UI Hi You can see this dochttps://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security for all the available webUI ports. Yes there are ways to get the data metrics

Is cluster manager same as master?

2014-09-04 Thread Ruebenacker, Oliver A
Hello, Is cluster manager mentioned herehttps://spark.apache.org/docs/latest/cluster-overview.html the same thing as master mentioned herehttps://spark.apache.org/docs/latest/spark-standalone.html#starting-a-cluster-manually? Thanks! Best, Oliver Oliver Ruebenacker | Solutions

Setting Java properties for Standalone on Windows 7?

2014-09-04 Thread Ruebenacker, Oliver A
Hello, I'm running Spark on Windows 7 as standalone, with everything on the same machine. No Hadoop installed. My app throws exception and worker reports: Could not locate executable null\bin\winutils.exe in the Hadoop binaries. I had the same problem earlier when deploying local. I

Reduce truncates RDD in standalone, but fine when local.

2014-09-04 Thread Ruebenacker, Oliver A
Hello, In the app below, when I run it with local[1] or local [3], I get the expected result - a list of the square roots of the numbers from 1 to 20. When I try the same app as standalone with one or two workers on the same machine, it will only print 1.0. Adding print statements

RE: Reduce truncates RDD in standalone, but fine when local.

2014-09-04 Thread Ruebenacker, Oliver A
field, it works fine. Best, Oliver From: Ruebenacker, Oliver A [mailto:oliver.ruebenac...@altisource.com] Sent: Thursday, September 04, 2014 4:15 PM To: user@spark.apache.org Subject: Reduce truncates RDD in standalone, but fine when local. Hello, In the app below, when I run

Web UI

2014-09-03 Thread Ruebenacker, Oliver A
Hello, What is included in the Spark web UI? What are the available URLs? Can the information be obtained in a machine-readable way (e.g. JSON, XML, etc)? Thanks! Best, Oliver Oliver Ruebenacker | Solutions Architect Altisource(tm) 290 Congress St, 7th Floor | Boston,

RE: Web UI

2014-09-03 Thread Ruebenacker, Oliver A
] Sent: Wednesday, September 03, 2014 3:36 PM To: Ruebenacker, Oliver A Cc: user@spark.apache.org Subject: Re: Web UI Hi Oliver, Spark standalone master and worker support '/json' endpoint in web UI, which returns some of the information in JSON format. I wasn't able to find relevant documentation

RE: Web UI

2014-09-03 Thread Ruebenacker, Oliver A
. Best, Oliver From: Wonha Ryu [mailto:wonha@gmail.com] Sent: Wednesday, September 03, 2014 4:27 PM To: Ruebenacker, Oliver A Cc: user@spark.apache.org Subject: Re: Web UI Hey Oliver, IIRC there's no JSON endpoint for application web UI. They only exist for cluster master and worker

If master is local, where are master and workers?

2014-09-03 Thread Ruebenacker, Oliver A
Hello, If launched with local as master, where are master and workers? Do they each have a web UI? How can they be monitored? Thanks! Best, Oliver Oliver Ruebenacker | Solutions Architect Altisource(tm) 290 Congress St, 7th Floor | Boston, Massachusetts 02210 P: (617)

RE: If master is local, where are master and workers?

2014-09-03 Thread Ruebenacker, Oliver A
How can that single process be monitored? Thanks! -Original Message- From: Marcelo Vanzin [mailto:van...@cloudera.com] Sent: Wednesday, September 03, 2014 6:32 PM To: Ruebenacker, Oliver A Cc: user@spark.apache.org Subject: Re: If master is local, where are master and workers? local