Re: Very long pause/hang at end of execution

I have the same issue with Spark 2.0.1, Java 1.8.x and pyspark. I also use SparkSQL and JDBC. My application runs locally. It happens only of I connect to the UI during Spark execution and even if I close the browser before the execution ends. I observed this behaviour both on macOS Sierra and Red

Re: TaskMemoryManager: Failed to allocate a page

ink this is caused by BroadcastHashJoin try to use more memory > than the amount driver have, could you decrease the > spark.sql.autoBroadcastJoinThreshold (-1 or 0 means disable it)? > > On Thu, Oct 27, 2016 at 9:19 AM, Pietro Pugni <pietro.pu...@gmail.com> wrote: >> I’

Re: TaskMemoryManager: Failed to allocate a page

I’m sorry, here’s the formatted message text: I'm running an ETL process that joins table1 with other tables (CSV files), one table at time (for example table1 with table2, table1 with table3, and so on). The join is written inside a PostgreSQL istance using JDBC. The entire process runs

Re: pyspark doesn't recognize MMM dateFormat pattern in spark.read.load() for dates like 1989Dec31 and 31Dec1989

And what if the month abbreviation is upper-case? Java doesn’t parse the month-name, for example if it's “JAN" instead of “Jan” or “DEC” instead of “Dec". Is it possible to solve this issue without using UDFs? Many thanks again Pietro > Il giorno 24 ott 2016, alle ore 17:33,

Re: pyspark doesn't recognize MMM dateFormat pattern in spark.read.load() for dates like 1989Dec31 and 31Dec1989

e it will be too late to set it there, and these are JVM flags, not > app or Spark flags. See spark.driver.extraJavaOptions and likewise for the > executor. > > On Mon, Oct 24, 2016 at 4:04 PM Pietro Pugni <pietro.pu...@gmail.com > <mailto:pietro.pu...@gmail.com>> wrote:

Re: pyspark doesn't recognize MMM dateFormat pattern in spark.read.load() for dates like 1989Dec31 and 31Dec1989

Sean Owen <so...@cloudera.com> ha > scritto: > > This is more of an OS-level thing, but I think that if you can manage to set > -Duser.language=en to the JVM, it might do the trick. > > I summarized what I think I know about this at > https://issues.apache.org/j

Re: pyspark doesn't recognize MMM dateFormat pattern in spark.read.load() for dates like 1989Dec31 and 31Dec1989

Thank you, I’ll appreciate that. I have no experience with Python, Java and Spark, so I the question can be translated to: “How can I set JVM locale when using spark-submit and pyspark?”. Probably this is possible only by changing the system defaul locale and not within the Spark session,

pyspark doesn't recognize MMM dateFormat pattern in spark.read.load() for dates like 1989Dec31 and 31Dec1989

Hi there, I opened a question on StackOverflow at this link: http://stackoverflow.com/questions/40007972/pyspark-doesnt-recognize-mmm-dateformat-pattern-in-spark-read-load-for-dates?noredirect=1#comment67297930_40007972 I didn’t get any useful answer, so I’m writing here hoping that someone can