[jira] [Commented] (SPARK-18640) Fix minor synchronization issue in TaskSchedulerImpl.runningTasksByExecutors

2016-12-02 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715712#comment-15715712 ] Josh Rosen commented on SPARK-18640: Actually, this doesn't look necessary because that method isn't

[jira] [Commented] (SPARK-18640) Fix minor synchronization issue in TaskSchedulerImpl.runningTasksByExecutors

2016-12-02 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715690#comment-15715690 ] Josh Rosen commented on SPARK-18640: I'm also going to backport this into branch-1.6. > Fix minor

[jira] [Resolved] (SPARK-18553) Executor loss may cause TaskSetManager to be leaked

2016-12-01 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-18553. Resolution: Fixed Fix Version/s: 1.6.4 > Executor loss may cause TaskSetManager to be

[jira] [Updated] (SPARK-18362) Use TextFileFormat in implementation of CSVFileFormat

2016-11-30 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-18362: --- Summary: Use TextFileFormat in implementation of CSVFileFormat (was: Use TextFileFormat in

[jira] [Updated] (SPARK-18362) Use TextFileFormat in implementation of CSVFileFormat

2016-11-30 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-18362: --- Description: Spark's CSVFileFormat data source uses inefficient methods for reading files during

[jira] [Created] (SPARK-18640) Fix minor synchronization issue in TaskSchedulerImpl.runningTasksByExecutors

2016-11-29 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-18640: -- Summary: Fix minor synchronization issue in TaskSchedulerImpl.runningTasksByExecutors Key: SPARK-18640 URL: https://issues.apache.org/jira/browse/SPARK-18640 Project:

[jira] [Assigned] (SPARK-18640) Fix minor synchronization issue in TaskSchedulerImpl.runningTasksByExecutors

2016-11-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen reassigned SPARK-18640: -- Assignee: Josh Rosen > Fix minor synchronization issue in

[jira] [Updated] (SPARK-18553) Executor loss may cause TaskSetManager to be leaked

2016-11-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-18553: --- Fix Version/s: 2.2.0 2.1.0 > Executor loss may cause TaskSetManager to be leaked

[jira] [Commented] (SPARK-18352) Parse normal, multi-line JSON files (not just JSON Lines)

2016-11-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706213#comment-15706213 ] Josh Rosen commented on SPARK-18352: Yeah, I'll update my patch to roll back my JSON changes so it

[jira] [Updated] (SPARK-18553) Executor loss may cause TaskSetManager to be leaked

2016-11-28 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-18553: --- Fix Version/s: 2.0.3 > Executor loss may cause TaskSetManager to be leaked >

[jira] [Created] (SPARK-18553) Executor loss may cause TaskSetManager to be leaked

2016-11-22 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-18553: -- Summary: Executor loss may cause TaskSetManager to be leaked Key: SPARK-18553 URL: https://issues.apache.org/jira/browse/SPARK-18553 Project: Spark Issue Type:

[jira] [Updated] (SPARK-18462) SparkListenerDriverAccumUpdates event does not deserialize properly in history server

2016-11-17 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-18462: --- Target Version/s: 2.0.3, 2.1.0 > SparkListenerDriverAccumUpdates event does not deserialize properly

[jira] [Updated] (SPARK-1267) Add a pip installer for PySpark

2016-11-16 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-1267: -- Fix Version/s: 2.1.0 > Add a pip installer for PySpark > --- > >

[jira] [Updated] (SPARK-18129) Sign pip artifacts

2016-11-16 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-18129: --- Fix Version/s: 2.1.0 > Sign pip artifacts > -- > > Key: SPARK-18129

[jira] [Resolved] (SPARK-1267) Add a pip installer for PySpark

2016-11-16 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-1267. --- Resolution: Fixed Fix Version/s: 2.2.0 Merged into master (2.2) and will consider for 2.1. >

[jira] [Updated] (SPARK-18129) Sign pip artifacts

2016-11-16 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-18129: --- Assignee: holdenk > Sign pip artifacts > -- > > Key: SPARK-18129 >

[jira] [Resolved] (SPARK-18129) Sign pip artifacts

2016-11-16 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-18129. Resolution: Fixed Fix Version/s: 2.2.0 Merged to master (2.2). > Sign pip artifacts >

[jira] [Updated] (SPARK-1267) Add a pip installer for PySpark

2016-11-16 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-1267: -- Assignee: holdenk > Add a pip installer for PySpark > --- > >

[jira] [Commented] (SPARK-18462) SparkListenerDriverAccumUpdates event does not deserialize properly in history server

2016-11-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15668990#comment-15668990 ] Josh Rosen commented on SPARK-18462: {code} [info] - roundtripping SparkListenerDriverAccumUpdates

[jira] [Created] (SPARK-18462) SparkListenerDriverAccumUpdates event does not deserialize properly in history server

2016-11-15 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-18462: -- Summary: SparkListenerDriverAccumUpdates event does not deserialize properly in history server Key: SPARK-18462 URL: https://issues.apache.org/jira/browse/SPARK-18462

[jira] [Updated] (SPARK-18418) Make release script hadoop profiles aren't correctly specified.

2016-11-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-18418: --- Fix Version/s: 2.2.0 > Make release script hadoop profiles aren't correctly specified. >

[jira] [Resolved] (SPARK-18418) Make release script hadoop profiles aren't correctly specified.

2016-11-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-18418. Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 15860

[jira] [Updated] (SPARK-18418) Make release script hadoop profiles aren't correctly specified.

2016-11-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-18418: --- Assignee: holdenk Affects Version/s: 2.1.0 Target Version/s: 2.1.0, 2.2.0

[jira] [Commented] (SPARK-18418) Make release script hadoop profiles aren't correctly specified.

2016-11-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15660416#comment-15660416 ] Josh Rosen commented on SPARK-18418: For reference: this patch fixes a bug which was introduced in

[jira] [Updated] (SPARK-18418) Make release script hadoop profiles aren't correctly specified.

2016-11-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-18418: --- Component/s: Project Infra > Make release script hadoop profiles aren't correctly specified. >

[jira] [Updated] (SPARK-18406) Race between end-of-task and completion iterator read lock release

2016-11-10 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-18406: --- Description: The following log comes from a production streaming job where executors periodically

[jira] [Created] (SPARK-18406) Race between end-of-task and completion iterator read lock release

2016-11-10 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-18406: -- Summary: Race between end-of-task and completion iterator read lock release Key: SPARK-18406 URL: https://issues.apache.org/jira/browse/SPARK-18406 Project: Spark

[jira] [Created] (SPARK-18362) Use TextFileFormat in implementation of JsonFileFormat and CSVFileFormat

2016-11-08 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-18362: -- Summary: Use TextFileFormat in implementation of JsonFileFormat and CSVFileFormat Key: SPARK-18362 URL: https://issues.apache.org/jira/browse/SPARK-18362 Project: Spark

[jira] [Resolved] (SPARK-18236) Reduce memory usage of Spark UI and HistoryServer by reducing duplicate objects

2016-11-07 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-18236. Resolution: Fixed Fix Version/s: 2.2.0 Merged into master (2.2.0). > Reduce memory usage

[jira] [Commented] (SPARK-14220) Build and test Spark against Scala 2.12

2016-11-03 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634050#comment-15634050 ] Josh Rosen commented on SPARK-14220: SPARK-14643 is likely to be the hardest task. > Build and test

[jira] [Created] (SPARK-18256) Improve performance of event log replay in HistoryServer based on profiler results

2016-11-03 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-18256: -- Summary: Improve performance of event log replay in HistoryServer based on profiler results Key: SPARK-18256 URL: https://issues.apache.org/jira/browse/SPARK-18256

[jira] [Updated] (SPARK-18256) Improve performance of event log replay in HistoryServer based on profiler results

2016-11-03 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-18256: --- Issue Type: Improvement (was: Bug) > Improve performance of event log replay in HistoryServer based

[jira] [Updated] (SPARK-18254) UDFs don't see aliased column names

2016-11-03 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-18254: --- Labels: correctness (was: ) > UDFs don't see aliased column names >

[jira] [Closed] (SPARK-14960) Don't perform treeAggregation in local mode

2016-11-03 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen closed SPARK-14960. -- Resolution: Won't Fix > Don't perform treeAggregation in local mode >

[jira] [Commented] (SPARK-14960) Don't perform treeAggregation in local mode

2016-11-03 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633478#comment-15633478 ] Josh Rosen commented on SPARK-14960: It turns out that {{treeAggregation}}'s extra costs in local

[jira] [Created] (SPARK-18236) Reduce memory usage of Spark UI and HistoryServer by reducing duplicate objects

2016-11-02 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-18236: -- Summary: Reduce memory usage of Spark UI and HistoryServer by reducing duplicate objects Key: SPARK-18236 URL: https://issues.apache.org/jira/browse/SPARK-18236 Project:

[jira] [Created] (SPARK-18182) Expose ReplayListenerBus.replay() overload which accepts Iterator

2016-10-31 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-18182: -- Summary: Expose ReplayListenerBus.replay() overload which accepts Iterator Key: SPARK-18182 URL: https://issues.apache.org/jira/browse/SPARK-18182 Project: Spark

[jira] [Updated] (SPARK-18034) Upgrade to MiMa 0.1.11

2016-10-21 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-18034: --- Fix Version/s: 2.0.2 > Upgrade to MiMa 0.1.11 > -- > > Key:

[jira] [Updated] (SPARK-18034) Upgrade to MiMa 0.1.11

2016-10-21 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-18034: --- Target Version/s: 2.0.2, 2.1.0 (was: 2.1.0) > Upgrade to MiMa 0.1.11 > -- > >

[jira] [Resolved] (SPARK-18034) Upgrade to MiMa 0.1.11

2016-10-21 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-18034. Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 15571

[jira] [Commented] (SPARK-18037) Event listener should be aware of multiple tries of same stage

2016-10-20 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593347#comment-15593347 ] Josh Rosen commented on SPARK-18037: Ahhh, I remember there being other JIRAs related to a negative

[jira] [Created] (SPARK-18034) Upgrade to MiMa 0.1.11

2016-10-20 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-18034: -- Summary: Upgrade to MiMa 0.1.11 Key: SPARK-18034 URL: https://issues.apache.org/jira/browse/SPARK-18034 Project: Spark Issue Type: Bug Components:

[jira] [Updated] (SPARK-18003) RDD zipWithIndex generate wrong result when one partition contains more than 2147483647 records.

2016-10-19 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-18003: --- Labels: correctness (was: ) > RDD zipWithIndex generate wrong result when one partition contains

[jira] [Updated] (SPARK-17981) Incorrectly Set Nullability to False in FilterExec

2016-10-18 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17981: --- Labels: correctness (was: ) > Incorrectly Set Nullability to False in FilterExec >

[jira] [Commented] (SPARK-7721) Generate test coverage report from Python

2016-10-17 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583642#comment-15583642 ] Josh Rosen commented on SPARK-7721: --- IIRC when I looked into this I hit problems with the HTML Publisher

[jira] [Commented] (SPARK-3132) Avoid serialization for Array[Byte] in TorrentBroadcast

2016-10-17 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583605#comment-15583605 ] Josh Rosen commented on SPARK-3132: --- I don't think that this is being actively worked on. I remember

[jira] [Updated] (SPARK-3132) Avoid serialization for Array[Byte] in TorrentBroadcast

2016-10-17 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-3132: -- Assignee: (was: Davies Liu) > Avoid serialization for Array[Byte] in TorrentBroadcast >

[jira] [Updated] (SPARK-17806) Incorrect result when work with data from parquet

2016-10-07 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17806: --- Component/s: SQL > Incorrect result when work with data from parquet >

[jira] [Updated] (SPARK-17803) Docker integration tests don't run with "Docker for Mac"

2016-10-06 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17803: --- Assignee: Christian Kadner > Docker integration tests don't run with "Docker for Mac" >

[jira] [Resolved] (SPARK-17803) Docker integration tests don't run with "Docker for Mac"

2016-10-06 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-17803. Resolution: Fixed Fix Version/s: 2.1.0 2.0.2 Issue resolved by pull

[jira] [Resolved] (SPARK-17809) scala.MatchError: BooleanType when casting a struct

2016-10-06 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-17809. Resolution: Fixed Fix Version/s: 2.0.1 > scala.MatchError: BooleanType when casting a

[jira] [Commented] (SPARK-17809) scala.MatchError: BooleanType when casting a struct

2016-10-06 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552868#comment-15552868 ] Josh Rosen commented on SPARK-17809: This appears to be fixed in the Spark 2.0.1 release, so I'm

[jira] [Commented] (SPARK-17809) scala.MatchError: BooleanType when casting a struct

2016-10-06 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552864#comment-15552864 ] Josh Rosen commented on SPARK-17809: For context, here's the stacktrace as of Spark 2.0.0: {code}

[jira] [Commented] (SPARK-17806) Incorrect result when work with data from parquet

2016-10-06 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552842#comment-15552842 ] Josh Rosen commented on SPARK-17806: I think that broadcast join may be the culprit here since I seem

[jira] [Commented] (SPARK-17806) Incorrect result when work with data from parquet

2016-10-06 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552830#comment-15552830 ] Josh Rosen commented on SPARK-17806: I was able to confirm that this is still a problem as of 2.0.1.

[jira] [Updated] (SPARK-17806) Incorrect result when work with data from parquet

2016-10-06 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17806: --- Labels: correctness (was: ) > Incorrect result when work with data from parquet >

[jira] [Updated] (SPARK-13210) NPE in Sort

2016-10-03 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-13210: --- Fix Version/s: 1.6.1 > NPE in Sort > --- > > Key: SPARK-13210 >

[jira] [Updated] (SPARK-13210) NPE in Sort

2016-10-03 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-13210: --- Affects Version/s: 1.6.0 > NPE in Sort > --- > > Key: SPARK-13210 >

[jira] [Commented] (SPARK-13210) NPE in Sort

2016-10-03 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15543975#comment-15543975 ] Josh Rosen commented on SPARK-13210: Hmm, weird. It looks like this was cherry-picked in

[jira] [Commented] (SPARK-17733) InferFiltersFromConstraints rule never terminates for query

2016-09-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534127#comment-15534127 ] Josh Rosen commented on SPARK-17733: Here's an even simpler test case: {code} sql("""CREATE

[jira] [Comment Edited] (SPARK-17733) InferFiltersFromConstraints rule never terminates for query

2016-09-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534082#comment-15534082 ] Josh Rosen edited comment on SPARK-17733 at 9/29/16 9:30 PM: - I managed to

[jira] [Comment Edited] (SPARK-17733) InferFiltersFromConstraints rule never terminates for query

2016-09-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534082#comment-15534082 ] Josh Rosen edited comment on SPARK-17733 at 9/29/16 9:28 PM: - I managed to

[jira] [Commented] (SPARK-17733) InferFiltersFromConstraints rule never terminates for query

2016-09-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534082#comment-15534082 ] Josh Rosen commented on SPARK-17733: I managed to shrink to a smaller case which freezes {{explain}}:

[jira] [Commented] (SPARK-17733) InferFiltersFromConstraints rule never terminates for query

2016-09-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533996#comment-15533996 ] Josh Rosen commented on SPARK-17733: Actually, the above log segment wasn't super useful, so let me

[jira] [Updated] (SPARK-17733) InferFiltersFromConstraints rule never terminates for query

2016-09-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17733: --- Attachment: constraints.png > InferFiltersFromConstraints rule never terminates for query >

[jira] [Updated] (SPARK-17733) InferFiltersFromConstraints rule never terminates for query

2016-09-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17733: --- Attachment:

[jira] [Created] (SPARK-17733) InferFiltersFromConstraints rule never terminates for query

2016-09-29 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-17733: -- Summary: InferFiltersFromConstraints rule never terminates for query Key: SPARK-17733 URL: https://issues.apache.org/jira/browse/SPARK-17733 Project: Spark

[jira] [Updated] (SPARK-17666) take() or isEmpty() on dataset leaks s3a connections

2016-09-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17666: --- Target Version/s: 2.0.1, 2.1.0 (was: 2.0.2, 2.1.0) > take() or isEmpty() on dataset leaks s3a

[jira] [Assigned] (SPARK-17666) take() or isEmpty() on dataset leaks s3a connections

2016-09-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen reassigned SPARK-17666: -- Assignee: Josh Rosen > take() or isEmpty() on dataset leaks s3a connections >

[jira] [Updated] (SPARK-17712) Incorrect result due to invalid pushdown of data-independent filter beneath aggregate

2016-09-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17712: --- Fix Version/s: 2.0.2 > Incorrect result due to invalid pushdown of data-independent filter beneath

[jira] [Updated] (SPARK-16343) Improve the PushDownPredicate rule to pushdown predicates currectly in non-deterministic condition

2016-09-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-16343: --- Fix Version/s: 2.0.2 > Improve the PushDownPredicate rule to pushdown predicates currectly in >

[jira] [Updated] (SPARK-17721) Erroneous computation in multiplication of transposed SparseMatrix with SparseVector

2016-09-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17721: --- Labels: correctness (was: ) > Erroneous computation in multiplication of transposed SparseMatrix

[jira] [Commented] (SPARK-16343) Improve the PushDownPredicate rule to pushdown predicates currectly in non-deterministic condition

2016-09-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533632#comment-15533632 ] Josh Rosen commented on SPARK-16343: This is actually a correctness issue, albeit pretty minor since

[jira] [Updated] (SPARK-16343) Improve the PushDownPredicate rule to pushdown predicates currectly in non-deterministic condition

2016-09-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-16343: --- Labels: correctness (was: ) > Improve the PushDownPredicate rule to pushdown predicates currectly

[jira] [Assigned] (SPARK-17712) Incorrect result due to invalid pushdown of data-independent filter beneath aggregate

2016-09-28 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen reassigned SPARK-17712: -- Assignee: Josh Rosen > Incorrect result due to invalid pushdown of data-independent filter

[jira] [Updated] (SPARK-17712) Incorrect result due to invalid pushdown of data-independent filter beneath aggregate

2016-09-28 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17712: --- Priority: Minor (was: Major) > Incorrect result due to invalid pushdown of data-independent filter

[jira] [Commented] (SPARK-17712) Incorrect result due to invalid pushdown of data-independent filter beneath aggregate

2016-09-28 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531105#comment-15531105 ] Josh Rosen commented on SPARK-17712: Intuitively, the only case where you can push a filter beneath

[jira] [Updated] (SPARK-17712) Incorrect result due to invalid pushdown of data-independent filter beneath aggregate

2016-09-28 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17712: --- Summary: Incorrect result due to invalid pushdown of data-independent filter beneath aggregate

[jira] [Updated] (SPARK-17712) Incorrect result due to invalid pushdown of filter beneath aggregate

2016-09-28 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17712: --- Summary: Incorrect result due to invalid pushdown of filter beneath aggregate (was: Incorrect

[jira] [Commented] (SPARK-17712) Incorrect result when selecting from aggregate subquery where outer WHERE clause constant-folds to false

2016-09-28 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531051#comment-15531051 ] Josh Rosen commented on SPARK-17712: This appears to be an optimizer bug: {code} 16/09/28 15:18:57

[jira] [Created] (SPARK-17712) Incorrect result when selecting from aggregate subquery where outer WHERE clause constant-folds to false

2016-09-28 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-17712: -- Summary: Incorrect result when selecting from aggregate subquery where outer WHERE clause constant-folds to false Key: SPARK-17712 URL:

[jira] [Created] (SPARK-17710) ReplSuite fails with ClassCircularityError in master Maven builds

2016-09-28 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-17710: -- Summary: ReplSuite fails with ClassCircularityError in master Maven builds Key: SPARK-17710 URL: https://issues.apache.org/jira/browse/SPARK-17710 Project: Spark

[jira] [Updated] (SPARK-17710) ReplSuite fails with ClassCircularityError in master Maven builds

2016-09-28 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17710: --- Affects Version/s: (was: 2.0.0) 2.1.0 > ReplSuite fails with

[jira] [Resolved] (SPARK-17056) Fix a wrong assert in MemoryStore

2016-09-27 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-17056. Resolution: Fixed Fix Version/s: 2.1.0 2.0.2 Issue resolved by pull

[jira] [Updated] (SPARK-17056) Fix a wrong assert in MemoryStore

2016-09-27 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17056: --- Assignee: Liang-Chi Hsieh > Fix a wrong assert in MemoryStore > - >

[jira] [Updated] (SPARK-17056) Fix a wrong assert in MemoryStore

2016-09-27 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17056: --- Component/s: (was: Spark Core) Block Manager > Fix a wrong assert in

[jira] [Updated] (SPARK-17618) Dataframe except returns incorrect results when combined with coalesce

2016-09-27 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17618: --- Fix Version/s: 2.1.0 2.0.2 > Dataframe except returns incorrect results when

[jira] [Commented] (SPARK-17681) Empty DataFrame with non-zero rows after using drop

2016-09-27 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526929#comment-15526929 ] Josh Rosen commented on SPARK-17681: I don't think that the current behavior is wrong. If {{drop()}}

[jira] [Updated] (SPARK-17666) take() or isEmpty() on dataset leaks s3a connections

2016-09-26 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17666: --- Target Version/s: 2.0.2, 2.1.0 Priority: Critical (was: Major) > take() or isEmpty() on

[jira] [Updated] (SPARK-17666) take() or isEmpty() on dataset leaks s3a connections

2016-09-26 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17666: --- Component/s: (was: Java API) SQL > take() or isEmpty() on dataset leaks s3a

[jira] [Commented] (SPARK-17666) take() or isEmpty() on dataset leaks s3a connections

2016-09-26 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523921#comment-15523921 ] Josh Rosen commented on SPARK-17666: I think that one problem with that approach is that any

[jira] [Commented] (SPARK-17666) take() or isEmpty() on dataset leaks s3a connections

2016-09-26 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523742#comment-15523742 ] Josh Rosen commented on SPARK-17666: My hunch is that there's cleanup which is performed in a

[jira] [Comment Edited] (SPARK-17647) SQL LIKE/RLIKE do not handle backslashes correctly

2016-09-23 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517656#comment-15517656 ] Josh Rosen edited comment on SPARK-17647 at 9/23/16 9:52 PM: - Another piece

[jira] [Commented] (SPARK-17647) SQL LIKE/RLIKE do not handle backslashes correctly

2016-09-23 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517656#comment-15517656 ] Josh Rosen commented on SPARK-17647: Another piece of evidence to help untangle this: In MySQL,

[jira] [Commented] (SPARK-17647) SQL LIKE/RLIKE do not handle backslashes correctly

2016-09-23 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517636#comment-15517636 ] Josh Rosen commented on SPARK-17647: On the other hand, running {code} select '' like '%\\%'

[jira] [Commented] (SPARK-17647) SQL LIKE/RLIKE do not handle backslashes correctly

2016-09-23 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517589#comment-15517589 ] Josh Rosen commented on SPARK-17647: I think that the first case is clearly a bug (and have a fix)

[jira] [Updated] (SPARK-17650) Adding a malformed URL to sc.addJar and/or sc.addFile bricks Executors

2016-09-23 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17650: --- Component/s: Spark Core > Adding a malformed URL to sc.addJar and/or sc.addFile bricks Executors >

[jira] [Resolved] (SPARK-17646) SparkType::add method does not work in 2.0.0 (in Java)

2016-09-23 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-17646. Resolution: Not A Problem Yep, this is the intended behavior. From the Scaladoc: {code} /**

[jira] [Updated] (SPARK-16240) model loading backward compatibility for ml.clustering.LDA

2016-09-23 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-16240: --- Fix Version/s: (was: 2.0.1) 2.0.2 > model loading backward compatibility for

[jira] [Updated] (SPARK-14387) Enable Hive-1.x ORC compatibility with spark.sql.hive.convertMetastoreOrc

2016-09-23 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-14387: --- Target Version/s: 2.0.2 (was: 2.0.1) > Enable Hive-1.x ORC compatibility with

<    1   2   3   4   5   6   7   8   9   10   >