[jira] [Commented] (SPARK-40873) Spark doesn't see some Parquet columns written from r-arrow

2022-10-21 Thread Daniel Darabos (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17622311#comment-17622311 ] Daniel Darabos commented on SPARK-40873: This works on the R side for dropping the metadata:

[jira] [Commented] (SPARK-40873) Spark doesn't see some Parquet columns written from r-arrow

2022-10-21 Thread Daniel Darabos (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17622251#comment-17622251 ] Daniel Darabos commented on SPARK-40873: Oh, I think I got it! With debug logging Spark prints a

[jira] [Created] (SPARK-40873) Spark doesn't see some Parquet columns written from r-arrow

2022-10-21 Thread Daniel Darabos (Jira)
Daniel Darabos created SPARK-40873: -- Summary: Spark doesn't see some Parquet columns written from r-arrow Key: SPARK-40873 URL: https://issues.apache.org/jira/browse/SPARK-40873 Project: Spark

[jira] [Updated] (SPARK-40873) Spark doesn't see some Parquet columns written from r-arrow

2022-10-21 Thread Daniel Darabos (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Darabos updated SPARK-40873: --- Attachment: part-0.parquet > Spark doesn't see some Parquet columns written from r-arrow >

[jira] [Commented] (SPARK-37690) Recursive view `df` detected (cycle: `df` -> `df`)

2022-08-12 Thread Daniel Darabos (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578901#comment-17578901 ] Daniel Darabos commented on SPARK-37690: It's fixed in Spark 3.3.0.

[jira] [Commented] (SPARK-37690) Recursive view `df` detected (cycle: `df` -> `df`)

2022-01-17 Thread Daniel Darabos (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17477141#comment-17477141 ] Daniel Darabos commented on SPARK-37690: We've hit this too with Spark 3.2.0. Could this be

[jira] [Commented] (SPARK-20144) spark.read.parquet no long maintains ordering of the data

2018-11-23 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16697319#comment-16697319 ] Daniel Darabos commented on SPARK-20144: So where do we go from here? Should I try to find a

[jira] [Commented] (SPARK-20144) spark.read.parquet no long maintains ordering of the data

2018-10-15 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650817#comment-16650817 ] Daniel Darabos commented on SPARK-20144: Thanks, those are good questions. # The global option

[jira] [Commented] (SPARK-20144) spark.read.parquet no long maintains ordering of the data

2018-10-15 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650722#comment-16650722 ] Daniel Darabos commented on SPARK-20144: Yeah, I'm not too happy about the alphabetical ordering

[jira] [Commented] (SPARK-20144) spark.read.parquet no long maintains ordering of the data

2018-10-15 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650492#comment-16650492 ] Daniel Darabos commented on SPARK-20144: Thanks Victor! I've expanded the test with a case where

[jira] [Commented] (SPARK-20144) spark.read.parquet no long maintains ordering of the data

2018-10-08 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642401#comment-16642401 ] Daniel Darabos commented on SPARK-20144: Sorry, I had an idea for a quick fix for this and sent

[jira] [Commented] (SPARK-23207) Shuffle+Repartition on an DataFrame could lead to incorrect answers

2018-08-23 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590435#comment-16590435 ] Daniel Darabos commented on SPARK-23207: Sorry, could you clarify the fix version please?

[jira] [Commented] (SPARK-25146) avg() returns null on some decimals

2018-08-17 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16584015#comment-16584015 ] Daniel Darabos commented on SPARK-25146: Wonderful, thanks! Sorry I missed the fix. > avg()

[jira] [Updated] (SPARK-25146) avg() returns null on some decimals

2018-08-17 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Darabos updated SPARK-25146: --- Description: We compute some 0-10 numbers in a pipeline using Spark SQL. Then we average

[jira] [Created] (SPARK-25146) avg() returns null on some decimals

2018-08-17 Thread Daniel Darabos (JIRA)
Daniel Darabos created SPARK-25146: -- Summary: avg() returns null on some decimals Key: SPARK-25146 URL: https://issues.apache.org/jira/browse/SPARK-25146 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-23666) Undeterministic column name with UDFs

2018-03-13 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396832#comment-16396832 ] Daniel Darabos commented on SPARK-23666: I've looked at the code and both {{ScalaUDF.scala}} and

[jira] [Created] (SPARK-23666) Undeterministic column name with UDFs

2018-03-13 Thread Daniel Darabos (JIRA)
Daniel Darabos created SPARK-23666: -- Summary: Undeterministic column name with UDFs Key: SPARK-23666 URL: https://issues.apache.org/jira/browse/SPARK-23666 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-21418) NoSuchElementException: None.get in DataSourceScanExec with sun.io.serialization.extendedDebugInfo=true

2017-09-14 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16165946#comment-16165946 ] Daniel Darabos commented on SPARK-21418: Sean's fix should cover you no matter what triggers the

[jira] [Commented] (SPARK-21418) NoSuchElementException: None.get on DataFrame.rdd

2017-09-04 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152742#comment-16152742 ] Daniel Darabos commented on SPARK-21418: Sorry for the delay. I can confirm that removing

[jira] [Comment Edited] (SPARK-21418) NoSuchElementException: None.get on DataFrame.rdd

2017-09-04 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088952#comment-16088952 ] Daniel Darabos edited comment on SPARK-21418 at 9/4/17 3:49 PM: I'm on

[jira] [Commented] (SPARK-21418) NoSuchElementException: None.get on DataFrame.rdd

2017-07-16 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088952#comment-16088952 ] Daniel Darabos commented on SPARK-21418: I'm on holiday without a computer through the coming

[jira] [Commented] (SPARK-20070) Redact datasource explain output

2017-07-14 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087378#comment-16087378 ] Daniel Darabos commented on SPARK-20070: I think this change has broken our application. Your

[jira] [Created] (SPARK-21418) NoSuchElementException: None.get on DataFrame.rdd

2017-07-14 Thread Daniel Darabos (JIRA)
Daniel Darabos created SPARK-21418: -- Summary: NoSuchElementException: None.get on DataFrame.rdd Key: SPARK-21418 URL: https://issues.apache.org/jira/browse/SPARK-21418 Project: Spark Issue

[jira] [Created] (SPARK-21136) Misleading error message for typo in SQL

2017-06-19 Thread Daniel Darabos (JIRA)
Daniel Darabos created SPARK-21136: -- Summary: Misleading error message for typo in SQL Key: SPARK-21136 URL: https://issues.apache.org/jira/browse/SPARK-21136 Project: Spark Issue Type: Bug

[jira] [Comment Edited] (SPARK-19209) "No suitable driver" on first try

2017-01-16 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824224#comment-15824224 ] Daniel Darabos edited comment on SPARK-19209 at 1/16/17 4:26 PM: - Sorry,

[jira] [Commented] (SPARK-19209) "No suitable driver" on first try

2017-01-16 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824224#comment-15824224 ] Daniel Darabos commented on SPARK-19209: Sorry, Jira had some issues when I was trying to file

[jira] [Updated] (SPARK-19209) "No suitable driver" on first try

2017-01-13 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Darabos updated SPARK-19209: --- Description: This is a regression from Spark 2.0.2. Observe! {code} $

[jira] [Commented] (SPARK-19209) "No suitable driver" on first try

2017-01-13 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821578#comment-15821578 ] Daniel Darabos commented on SPARK-19209: Puzzlingly this only happens in the application when the

[jira] [Created] (SPARK-19209) "No suitable driver" on first try

2017-01-13 Thread Daniel Darabos (JIRA)
Daniel Darabos created SPARK-19209: -- Summary: "No suitable driver" on first try Key: SPARK-19209 URL: https://issues.apache.org/jira/browse/SPARK-19209 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-16625) Oracle JDBC table creation fails with ORA-00902: invalid datatype

2016-07-19 Thread Daniel Darabos (JIRA)
Daniel Darabos created SPARK-16625: -- Summary: Oracle JDBC table creation fails with ORA-00902: invalid datatype Key: SPARK-16625 URL: https://issues.apache.org/jira/browse/SPARK-16625 Project: Spark

[jira] [Updated] (SPARK-12964) SparkContext.localProperties leaked

2016-07-15 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Darabos updated SPARK-12964: --- Description: I have a non-deterministic but quite reliable reproduction for a case where

[jira] [Commented] (SPARK-15796) Spark 1.6 default memory settings can cause heavy GC when caching

2016-06-07 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318523#comment-15318523 ] Daniel Darabos commented on SPARK-15796: > The only argument against it was that it's specific to

[jira] [Commented] (SPARK-15796) Spark 1.6 default memory settings can cause heavy GC when caching

2016-06-07 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318486#comment-15318486 ] Daniel Darabos commented on SPARK-15796: The example program takes less than a minute on Spark

[jira] [Updated] (SPARK-11293) Spillable collections leak shuffle memory

2016-03-23 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Darabos updated SPARK-11293: --- Affects Version/s: 1.6.1 > Spillable collections leak shuffle memory >

[jira] [Updated] (SPARK-13620) Avoid reverse DNS lookup for 0.0.0.0 on startup

2016-03-03 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Darabos updated SPARK-13620: --- Fix Version/s: 2.0.0 > Avoid reverse DNS lookup for 0.0.0.0 on startup >

[jira] [Resolved] (SPARK-13620) Avoid reverse DNS lookup for 0.0.0.0 on startup

2016-03-03 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Darabos resolved SPARK-13620. Resolution: Fixed > Avoid reverse DNS lookup for 0.0.0.0 on startup >

[jira] [Commented] (SPARK-13620) Avoid reverse DNS lookup for 0.0.0.0 on startup

2016-03-03 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15177784#comment-15177784 ] Daniel Darabos commented on SPARK-13620: I just tested with the latest 2.0 nightly. Starting

[jira] [Commented] (SPARK-13620) Avoid reverse DNS lookup for 0.0.0.0 on startup

2016-03-03 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15177655#comment-15177655 ] Daniel Darabos commented on SPARK-13620: For the record I figured out a workaround. Strace shows

[jira] [Commented] (SPARK-13620) Avoid reverse DNS lookup for 0.0.0.0 on startup

2016-03-02 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15175974#comment-15175974 ] Daniel Darabos commented on SPARK-13620: Looks like the whole {{InetSocketAddress}} line has

[jira] [Updated] (SPARK-13620) Avoid reverse DNS lookup for 0.0.0.0 on startup

2016-03-02 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Darabos updated SPARK-13620: --- Description: I noticed we spend 5+ seconds during application startup with the following

[jira] [Updated] (SPARK-13620) Avoid reverse DNS lookup for 0.0.0.0 on startup

2016-03-02 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Darabos updated SPARK-13620: --- Description: I noticed we spend 5+ seconds during application startup with the following

[jira] [Created] (SPARK-13620) Avoid reverse DNS lookup for 0.0.0.0 on startup

2016-03-02 Thread Daniel Darabos (JIRA)
Daniel Darabos created SPARK-13620: -- Summary: Avoid reverse DNS lookup for 0.0.0.0 on startup Key: SPARK-13620 URL: https://issues.apache.org/jira/browse/SPARK-13620 Project: Spark Issue

[jira] [Reopened] (SPARK-10057) Faill to load class org.slf4j.impl.StaticLoggerBinder

2016-03-01 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Darabos reopened SPARK-10057: Thanks for the repro, Stephen! It also works in Spark 1.6.0. This was mentioned in

[jira] [Updated] (SPARK-10057) Faill to load class org.slf4j.impl.StaticLoggerBinder

2016-03-01 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Darabos updated SPARK-10057: --- Affects Version/s: 1.6.0 > Faill to load class org.slf4j.impl.StaticLoggerBinder >

[jira] [Created] (SPARK-13212) Provide a way to unregister data sources from a SQLContext

2016-02-05 Thread Daniel Darabos (JIRA)
Daniel Darabos created SPARK-13212: -- Summary: Provide a way to unregister data sources from a SQLContext Key: SPARK-13212 URL: https://issues.apache.org/jira/browse/SPARK-13212 Project: Spark

[jira] [Commented] (SPARK-1239) Don't fetch all map output statuses at each reducer during shuffles

2016-02-03 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15130351#comment-15130351 ] Daniel Darabos commented on SPARK-1239: --- I've read an interesting article about the "Kylix"

[jira] [Updated] (SPARK-12964) SparkContext.localProperties leaked

2016-01-22 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Darabos updated SPARK-12964: --- Description: I have a non-deterministic but quite reliable reproduction for a case where

[jira] [Created] (SPARK-12964) SparkContext.localProperties leaked

2016-01-22 Thread Daniel Darabos (JIRA)
Daniel Darabos created SPARK-12964: -- Summary: SparkContext.localProperties leaked Key: SPARK-12964 URL: https://issues.apache.org/jira/browse/SPARK-12964 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-2309) Generalize the binary logistic regression into multinomial logistic regression

2016-01-21 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110628#comment-15110628 ] Daniel Darabos commented on SPARK-2309: ---

[jira] [Commented] (SPARK-11293) Spillable collections leak shuffle memory

2016-01-14 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15097987#comment-15097987 ] Daniel Darabos commented on SPARK-11293: > so should be reopened or not? is there still a memory

[jira] [Reopened] (SPARK-11293) Spillable collections leak shuffle memory

2016-01-14 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Darabos reopened SPARK-11293: > Spillable collections leak shuffle memory > - > >

[jira] [Updated] (SPARK-11293) Spillable collections leak shuffle memory

2016-01-14 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Darabos updated SPARK-11293: --- Affects Version/s: 1.6.0 > Spillable collections leak shuffle memory >

[jira] [Commented] (SPARK-11293) Spillable collections leak shuffle memory

2016-01-14 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098011#comment-15098011 ] Daniel Darabos commented on SPARK-11293: > so add 1.6.0 as affected version... Done. >

[jira] [Commented] (SPARK-11293) Spillable collections leak shuffle memory

2016-01-05 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083311#comment-15083311 ] Daniel Darabos commented on SPARK-11293: I have a somewhat contrived example that still leaks in

[jira] [Commented] (SPARK-11293) Spillable collections leak shuffle memory

2016-01-05 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083326#comment-15083326 ] Daniel Darabos commented on SPARK-11293: Sorry, my example was overly complicated. This one

[jira] [Commented] (SPARK-11652) Remote code execution with InvokerTransformer

2015-11-13 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15003982#comment-15003982 ] Daniel Darabos commented on SPARK-11652: > I may be missing some point, but Spark isn't consuming

[jira] [Created] (SPARK-11652) Remote code execution with InvokerTransformer

2015-11-11 Thread Daniel Darabos (JIRA)
Daniel Darabos created SPARK-11652: -- Summary: Remote code execution with InvokerTransformer Key: SPARK-11652 URL: https://issues.apache.org/jira/browse/SPARK-11652 Project: Spark Issue

[jira] [Commented] (SPARK-1239) Don't fetch all map output statuses at each reducer during shuffles

2015-11-03 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987046#comment-14987046 ] Daniel Darabos commented on SPARK-1239: --- I can also add some data. I have a ShuffleMapStage with

[jira] [Created] (SPARK-11403) Log something when dying from OnOutOfMemoryError

2015-10-29 Thread Daniel Darabos (JIRA)
Daniel Darabos created SPARK-11403: -- Summary: Log something when dying from OnOutOfMemoryError Key: SPARK-11403 URL: https://issues.apache.org/jira/browse/SPARK-11403 Project: Spark Issue

[jira] [Closed] (SPARK-10792) Spark + YARN – executor is not re-created

2015-10-06 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Darabos closed SPARK-10792. -- Resolution: Duplicate > Spark + YARN – executor is not re-created >

[jira] [Comment Edited] (SPARK-10792) Spark + YARN – executor is not re-created

2015-10-05 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14943405#comment-14943405 ] Daniel Darabos edited comment on SPARK-10792 at 10/5/15 2:17 PM: - I work

[jira] [Updated] (SPARK-10792) Spark + YARN – executor is not re-created

2015-10-05 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Darabos updated SPARK-10792: --- Priority: Critical (was: Minor) Component/s: (was: Streaming) Summary:

[jira] [Commented] (SPARK-10792) Spark streaming + YARN – executor is not re-created on machine restart

2015-10-05 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14943405#comment-14943405 ] Daniel Darabos commented on SPARK-10792: I work with Andras and can now substantiate his account

[jira] [Commented] (SPARK-10792) Spark + YARN – executor is not re-created

2015-10-05 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14943407#comment-14943407 ] Daniel Darabos commented on SPARK-10792: I've edited the priority, the title, and the component

[jira] [Commented] (SPARK-5077) Map output statuses can still exceed spark.akka.frameSize

2015-09-11 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740816#comment-14740816 ] Daniel Darabos commented on SPARK-5077: --- Hi Josh, The MapOutputTracker errors are a source of pain

[jira] [Commented] (SPARK-6497) Class is not registered: scala.reflect.ManifestFactory$$anon$9

2015-08-28 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14718559#comment-14718559 ] Daniel Darabos commented on SPARK-6497: --- I've checked 1.5.0-RC2 and the repro still

[jira] [Updated] (SPARK-6497) Class is not registered: scala.reflect.ManifestFactory$$anon$9

2015-08-28 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Darabos updated SPARK-6497: -- Priority: Major (was: Minor) Class is not registered: scala.reflect.ManifestFactory$$anon$9

[jira] [Updated] (SPARK-6497) Class is not registered: scala.reflect.ManifestFactory$$anon$9

2015-08-28 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Darabos updated SPARK-6497: -- Affects Version/s: 1.5.0 1.4.0 Class is not registered:

[jira] [Created] (SPARK-9382) Tachyon version mismatch

2015-07-27 Thread Daniel Darabos (JIRA)
Daniel Darabos created SPARK-9382: - Summary: Tachyon version mismatch Key: SPARK-9382 URL: https://issues.apache.org/jira/browse/SPARK-9382 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-9382) Tachyon version mismatch

2015-07-27 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14642908#comment-14642908 ] Daniel Darabos commented on SPARK-9382: --- Ah, looks like this was just an oversight

[jira] [Closed] (SPARK-9382) Tachyon version mismatch

2015-07-27 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Darabos closed SPARK-9382. - Resolution: Fixed Fix Version/s: 1.4.1 Tachyon version mismatch

[jira] [Commented] (SPARK-8898) Jets3t hangs with more than 1 core

2015-07-17 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631153#comment-14631153 ] Daniel Darabos commented on SPARK-8898: --- Unfortunately the {{spark-ec2}} script does

[jira] [Commented] (SPARK-4879) Missing output partitions after job completes with speculative execution

2015-07-15 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14628380#comment-14628380 ] Daniel Darabos commented on SPARK-4879: --- Thanks, Josh! I wonder when the

[jira] [Commented] (SPARK-4879) Missing output partitions after job completes with speculative execution

2015-07-15 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14628086#comment-14628086 ] Daniel Darabos commented on SPARK-4879: --- I've managed to reproduce on Spark 1.3.1

[jira] [Commented] (SPARK-8836) Sorted join

2015-07-15 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14628030#comment-14628030 ] Daniel Darabos commented on SPARK-8836: --- I just noticed this has actually been

[jira] [Commented] (SPARK-4879) Missing output partitions after job completes with speculative execution

2015-07-12 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14624000#comment-14624000 ] Daniel Darabos commented on SPARK-4879: --- Good idea! I'll try with 1.3.1 next week.

[jira] [Commented] (SPARK-4879) Missing output partitions after job completes with speculative execution

2015-07-10 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1466#comment-1466 ] Daniel Darabos commented on SPARK-4879: --- I wonder if this issue is serious enough to

[jira] [Comment Edited] (SPARK-4879) Missing output partitions after job completes with speculative execution

2015-07-10 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1466#comment-1466 ] Daniel Darabos edited comment on SPARK-4879 at 7/10/15 12:25 PM:

[jira] [Commented] (SPARK-4879) Missing output partitions after job completes with speculative execution

2015-07-09 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14620570#comment-14620570 ] Daniel Darabos commented on SPARK-4879: --- I have a fairly reliable reproduction with

[jira] [Comment Edited] (SPARK-4879) Missing output partitions after job completes with speculative execution

2015-07-09 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14620570#comment-14620570 ] Daniel Darabos edited comment on SPARK-4879 at 7/9/15 6:15 PM:

[jira] [Comment Edited] (SPARK-4879) Missing output partitions after job completes with speculative execution

2015-07-09 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14620570#comment-14620570 ] Daniel Darabos edited comment on SPARK-4879 at 7/9/15 6:15 PM:

[jira] [Created] (SPARK-8960) Style cleanup of spark_ec2.py

2015-07-09 Thread Daniel Darabos (JIRA)
Daniel Darabos created SPARK-8960: - Summary: Style cleanup of spark_ec2.py Key: SPARK-8960 URL: https://issues.apache.org/jira/browse/SPARK-8960 Project: Spark Issue Type: Task

[jira] [Commented] (SPARK-8960) Style cleanup of spark_ec2.py

2015-07-09 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14621397#comment-14621397 ] Daniel Darabos commented on SPARK-8960: --- So, can I sort of hijack this to continue

[jira] [Updated] (SPARK-8893) Require positive partition counts in RDD.repartition

2015-07-08 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Darabos updated SPARK-8893: -- Description: What does {{sc.parallelize(1 to 3).repartition(p).collect}} return? I would

[jira] [Created] (SPARK-8893) Require positive partition counts in RDD.repartition

2015-07-08 Thread Daniel Darabos (JIRA)
Daniel Darabos created SPARK-8893: - Summary: Require positive partition counts in RDD.repartition Key: SPARK-8893 URL: https://issues.apache.org/jira/browse/SPARK-8893 Project: Spark Issue

[jira] [Created] (SPARK-8898) Jets3t hangs with more than 1 core

2015-07-08 Thread Daniel Darabos (JIRA)
Daniel Darabos created SPARK-8898: - Summary: Jets3t hangs with more than 1 core Key: SPARK-8898 URL: https://issues.apache.org/jira/browse/SPARK-8898 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-8902) Hostname missing in spark-ec2 error message

2015-07-08 Thread Daniel Darabos (JIRA)
Daniel Darabos created SPARK-8902: - Summary: Hostname missing in spark-ec2 error message Key: SPARK-8902 URL: https://issues.apache.org/jira/browse/SPARK-8902 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-8902) Hostname missing in spark-ec2 error message

2015-07-08 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Darabos updated SPARK-8902: -- Description: When SSH to a machine fails, the following message is printed: {noformat} Failed

[jira] [Created] (SPARK-8836) Sorted join

2015-07-06 Thread Daniel Darabos (JIRA)
Daniel Darabos created SPARK-8836: - Summary: Sorted join Key: SPARK-8836 URL: https://issues.apache.org/jira/browse/SPARK-8836 Project: Spark Issue Type: Improvement Components:

[jira] [Commented] (SPARK-5945) Spark should not retry a stage infinitely on a FetchFailedException

2015-07-02 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14611960#comment-14611960 ] Daniel Darabos commented on SPARK-5945: --- At the moment we have a ton of these

[jira] [Commented] (SPARK-4357) Modify release publishing to work with Scala 2.11

2015-06-30 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14608150#comment-14608150 ] Daniel Darabos commented on SPARK-4357: --- Hi, what is the obstacle to providing

[jira] [Commented] (SPARK-8310) Spark EC2 branch in 1.4 is wrong

2015-06-29 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14605411#comment-14605411 ] Daniel Darabos commented on SPARK-8310: --- It's an easy mistake to make, and one of

[jira] [Commented] (SPARK-6944) Mechanism to associate generic operator scope with RDD's

2015-05-28 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14562823#comment-14562823 ] Daniel Darabos commented on SPARK-6944: --- Thanks for this fantastic feature! Please

[jira] [Created] (SPARK-6497) Class is not registered: scala.reflect.ManifestFactory$$anon$9

2015-03-24 Thread Daniel Darabos (JIRA)
Daniel Darabos created SPARK-6497: - Summary: Class is not registered: scala.reflect.ManifestFactory$$anon$9 Key: SPARK-6497 URL: https://issues.apache.org/jira/browse/SPARK-6497 Project: Spark

[jira] [Created] (SPARK-5102) CompressedMapStatus needs to be registered with Kryo

2015-01-06 Thread Daniel Darabos (JIRA)
Daniel Darabos created SPARK-5102: - Summary: CompressedMapStatus needs to be registered with Kryo Key: SPARK-5102 URL: https://issues.apache.org/jira/browse/SPARK-5102 Project: Spark Issue

[jira] [Created] (SPARK-4831) Current directory always on classpath with spark-submit

2014-12-11 Thread Daniel Darabos (JIRA)
Daniel Darabos created SPARK-4831: - Summary: Current directory always on classpath with spark-submit Key: SPARK-4831 URL: https://issues.apache.org/jira/browse/SPARK-4831 Project: Spark

[jira] [Commented] (SPARK-4831) Current directory always on classpath with spark-submit

2014-12-11 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14243074#comment-14243074 ] Daniel Darabos commented on SPARK-4831: --- bq. Is it perhaps finding and exploded

[jira] [Commented] (SPARK-3721) Broadcast Variables above 2GB break in PySpark

2014-10-27 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14185189#comment-14185189 ] Daniel Darabos commented on SPARK-3721: --- We are hitting all kinds of MaxInt and

[jira] [Commented] (SPARK-3644) REST API for Spark application info (jobs / stages / tasks / storage info)

2014-10-09 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14165364#comment-14165364 ] Daniel Darabos commented on SPARK-3644: --- Hi Dev, thanks for the offer! Have you seen

[jira] [Commented] (SPARK-3644) REST API for Spark application info (jobs / stages / tasks / storage info)

2014-10-06 Thread Daniel Darabos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14160183#comment-14160183 ] Daniel Darabos commented on SPARK-3644: --- Notes from another consumer with a

  1   2   >