Hello, I'm using the Github snapshot of PySpark and having trouble setting
the worker memory correctly. I've set spark.executor.memory to 5g, but
somewhere along the way Xmx is getting capped to 512M. This was not
occurring with the same setup and 0.9.0. How many places do I need to
configure the m
env.sh on the workers as well. Maybe code there is
> somehow overriding the spark.executor.memory setting.
>
> Matei
>
> On Mar 18, 2014, at 6:17 PM, Jim Blomo wrote:
>
> Hello, I'm using the Github snapshot of PySpark and having trouble setting
> the worker memory correctly.
n the machine running the driver
(typically the master host) and will affect the memory available to
the Executor running on a slave node
-D
SPARK_DAEMON_OPTS:
On Wed, Mar 19, 2014 at 12:48 AM, Jim Blomo wrote:
> Thanks for the suggestion, Matei. I've tracked this down to a se
I think I've encountered the same problem and filed
https://spark-project.atlassian.net/plugins/servlet/mobile#issue/SPARK-1284
For me it hung the worker, though. Can you add reproducible steps and what
version you're running?
On Mar 19, 2014 10:13 PM, "Nicholas Chammas"
wrote:
> So I have the p
Hi all, I'm wondering if there's any settings I can use to reduce the
memory needed by the PythonRDD when computing simple stats. I am
getting OutOfMemoryError exceptions while calculating count() on big,
but not absurd, records. It seems like PythonRDD is trying to keep
too many of these records
efault. Try passing batchSize=10 to your
> SparkContext constructor to lower it (the default is 1024). Or even
> batchSize=1 to match earlier versions.
>
> Matei
>
> On Mar 21, 2014, at 6:18 PM, Jim Blomo wrote:
>
>> Hi all, I'm wondering if there's any settings
ions of Spark (0.9 or 0.8)? We'll
> try to look into these, seems like a serious error.
>
> Matei
>
> On Mar 27, 2014, at 7:27 PM, Jim Blomo wrote:
>
>> Thanks, Matei. I am running "Spark 1.0.0-SNAPSHOT built for Hadoop
>> 1.0.4" from GitHub on 2014-
, Mar 29, 2014 at 3:17 PM, Jim Blomo wrote:
> I've only tried 0.9, in which I ran into the `stdin writer to Python
> finished early` so frequently I wasn't able to load even a 1GB file.
> Let me know if I can provide any other info!
>
> On Thu, Mar 27, 2014 at 8:48 PM, Mat
might happen only if records are
> large, or something like that. How much heap are you giving to your
> executors, and does it show that much in the web UI?
>
> Matei
>
> On Mar 29, 2014, at 10:44 PM, Jim Blomo wrote:
>
>> I think the problem I ran into in 0.9 is c
7;d like to reproduce and fix this.
>
> Matei
>
> On Apr 9, 2014, at 3:52 PM, Jim Blomo wrote:
>
>> Hi Matei, thanks for working with me to find these issues.
>>
>> To summarize, the issues I've seen are:
>> 0.9.0:
>> - https://issues.apache.org/jira/b
On Thu, Apr 10, 2014 at 12:24 PM, Andrew Ash wrote:
> The biggest issue I've come across is that the cluster is somewhat unstable
> when under memory pressure. Meaning that if you attempt to persist an RDD
> that's too big for memory, even with MEMORY_AND_DISK, you'll often still get
> OOMs. I h
I'm using PySpark to load some data and getting an error while
parsing it. Is it possible to find the source file and line of the bad
data? I imagine that this would be extremely tricky when dealing with
multiple derived RRDs, so an answer with the caveat of "this only
works when running .map() o
cess
+
case e: IOException =>
// This can happen for legitimate reasons if the Python code
stops returning data
// before we are done passing elements through, e.g., for
take(). Just log a message to
On Wed, Apr 9, 2014 at 7:04 PM, Jim Blomo wrote:
> This d
ader knows
>> what is going on.
>>
>> Anyways maybe if you have a simpler solution you could sketch it out in
>> the JIRA and we could talk over there. The current proposal in the JIRA is
>> somewhat complicated...
>>
>> - Patrick
>>
>>
>&g
Should add that I had to tweak the numbers a bit to keep above swap
threshold, but below the "Too many open files" error (`ulimit -n` is
32768).
On Wed, May 14, 2014 at 10:47 AM, Jim Blomo wrote:
> That worked amazingly well, thank you Matei! Numbers that worked for
> me
.
>
> Matei
>
> On May 12, 2014, at 5:47 PM, Jim Blomo wrote:
>
>> Thanks, Aaron, this looks like a good solution! Will be trying it out
>> shortly.
>>
>> I noticed that the S3 exception seem to occur more frequently when the
>> box is swappin
I've seen these caused by the OOM killer. I recommend checking
/var/log/syslog to see if it was activated due to lack of system
memory.
On Thu, Jun 12, 2014 at 11:45 PM, libl <271592...@qq.com> wrote:
> I use standalone mode submit task.But often,I got an error.The stacktrace as
>
> 2014-06-12 11
Hi, I'm having trouble compiling a snapshot, any advice would be
appreciated. I get the error below when compiling either master or
branch-1.1. The key error is, I believe, "[ERROR] File name too long"
but I don't understand what it is referring to. Thanks!
./make-distribution.sh --tgz --skip-
Tracked this down to incompatibility with Scala and encryptfs.
Resolved by compiling in a directory not mounted with encryption (eg
/tmp).
On Thu, Aug 14, 2014 at 3:25 PM, Jim Blomo wrote:
> Hi, I'm having trouble compiling a snapshot, any advice would be
> appreciated. I get the
19 matches
Mail list logo
