[jira] [Resolved] (SPARK-19872) UnicodeDecodeError in Pyspark on sc.textFile read with repartition

2017-03-15 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-19872. Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > UnicodeDecodeError in

[jira] [Assigned] (SPARK-19872) UnicodeDecodeError in Pyspark on sc.textFile read with repartition

2017-03-15 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reassigned SPARK-19872: -- Assignee: Hyukjin Kwon > UnicodeDecodeError in Pyspark on sc.textFile read with repartition >

[jira] [Resolved] (SPARK-19561) Pyspark Dataframes don't allow timestamps near epoch

2017-03-07 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-19561. Resolution: Fixed Assignee: Jason White Fix Version/s: 2.2.0

[jira] [Resolved] (SPARK-19500) Fail to spill the aggregated hash map when radix sort is used

2017-02-17 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-19500. Resolution: Fixed Fix Version/s: 2.2.0 2.0.3 2.1.1

[jira] [Resolved] (SPARK-19481) Fix flaky test: o.a.s.repl.ReplSuite should clone and clean line object in ClosureCleaner

2017-02-09 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-19481. Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 Issue resolved by pull

[jira] [Assigned] (SPARK-19500) Fail to spill the aggregated hash map when radix sort is used

2017-02-07 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reassigned SPARK-19500: -- Assignee: Davies Liu > Fail to spill the aggregated hash map when radix sort is used >

[jira] [Created] (SPARK-19500) Fail to spill the aggregated hash map when radix sort is used

2017-02-07 Thread Davies Liu (JIRA)
Davies Liu created SPARK-19500: -- Summary: Fail to spill the aggregated hash map when radix sort is used Key: SPARK-19500 URL: https://issues.apache.org/jira/browse/SPARK-19500 Project: Spark

[jira] [Resolved] (SPARK-19415) Improve the implicit type conversion between numeric type and string to avoid precesion loss

2017-01-31 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-19415. Resolution: Duplicate Fix Version/s: 2.2.0 > Improve the implicit type conversion between

[jira] [Created] (SPARK-19415) Improve the implicit type conversion between numeric type and string to avoid precesion loss

2017-01-31 Thread Davies Liu (JIRA)
Davies Liu created SPARK-19415: -- Summary: Improve the implicit type conversion between numeric type and string to avoid precesion loss Key: SPARK-19415 URL: https://issues.apache.org/jira/browse/SPARK-19415

[jira] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

2017-01-30 Thread Davies Liu (JIRA)
Title: Message Title Davies Liu commented on SPARK-18105

[jira] [Reopened] (SPARK-14480) Remove meaningless StringIteratorReader for CSV data source for better performance

2017-01-26 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reopened SPARK-14480: This patch have a regression: A column that have escaped newline can't be correctly parsed anymore.

[jira] [Closed] (SPARK-19375) na.fill() should not change the data type of column

2017-01-26 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu closed SPARK-19375. -- Resolution: Duplicate > na.fill() should not change the data type of column >

[jira] [Created] (SPARK-19375) na.fill() should not change the data type of column

2017-01-26 Thread Davies Liu (JIRA)
Davies Liu created SPARK-19375: -- Summary: na.fill() should not change the data type of column Key: SPARK-19375 URL: https://issues.apache.org/jira/browse/SPARK-19375 Project: Spark Issue Type:

[jira] [Updated] (SPARK-19370) Flaky test: MetadataCacheSuite

2017-01-26 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-19370: --- Affects Version/s: 2.1.0 > Flaky test: MetadataCacheSuite > -- > >

[jira] [Updated] (SPARK-19370) Flaky test: MetadataCacheSuite

2017-01-26 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-19370: --- Description:

[jira] [Created] (SPARK-19370) Flaky test: MetadataCacheSuite

2017-01-26 Thread Davies Liu (JIRA)
Davies Liu created SPARK-19370: -- Summary: Flaky test: MetadataCacheSuite Key: SPARK-19370 URL: https://issues.apache.org/jira/browse/SPARK-19370 Project: Spark Issue Type: Test

[jira] [Resolved] (SPARK-17912) Refactor code generation to get data for ColumnVector/ColumnarBatch

2017-01-19 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-17912. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 15467

[jira] [Updated] (SPARK-17912) Refactor code generation to get data for ColumnVector/ColumnarBatch

2017-01-19 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-17912: --- Fix Version/s: (was: 3.0.0) 2.2.0 > Refactor code generation to get data for

[jira] [Commented] (SPARK-17602) PySpark - Performance Optimization Large Size of Broadcast Variable

2017-01-19 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830644#comment-15830644 ] Davies Liu commented on SPARK-17602: The Python workers are reused by default, could you re-run the

[jira] [Resolved] (SPARK-19019) PySpark does not work with Python 3.6.0

2017-01-17 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-19019. Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 Issue resolved by pull

[jira] [Resolved] (SPARK-19180) the offset of short is 4 in OffHeapColumnVector's putShorts

2017-01-13 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-19180. Resolution: Fixed Fix Version/s: 2.0.3 2.1.1 Issue resolved by pull

[jira] [Updated] (SPARK-18589) persist() resolves "java.lang.RuntimeException: Invalid PythonUDF (...), requires attributes from more than one child"

2017-01-13 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-18589: --- Priority: Critical (was: Minor) > persist() resolves "java.lang.RuntimeException: Invalid PythonUDF

[jira] [Assigned] (SPARK-18589) persist() resolves "java.lang.RuntimeException: Invalid PythonUDF (...), requires attributes from more than one child"

2017-01-13 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reassigned SPARK-18589: -- Assignee: Davies Liu > persist() resolves "java.lang.RuntimeException: Invalid PythonUDF >

[jira] [Resolved] (SPARK-18281) toLocalIterator yields time out error on pyspark2

2016-12-20 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-18281. Resolution: Fixed Fix Version/s: 2.0.3 2.1.1 Issue resolved by pull

[jira] [Comment Edited] (SPARK-18676) Spark 2.x query plan data size estimation can crash join queries versus 1.x

2016-12-13 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15746370#comment-15746370 ] Davies Liu edited comment on SPARK-18676 at 12/13/16 9:47 PM: -- I had a

[jira] [Commented] (SPARK-18676) Spark 2.x query plan data size estimation can crash join queries versus 1.x

2016-12-13 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15746370#comment-15746370 ] Davies Liu commented on SPARK-18676: I had a working prototype, but in introduce some weird behavior,

[jira] [Commented] (SPARK-18676) Spark 2.x query plan data size estimation can crash join queries versus 1.x

2016-12-08 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733569#comment-15733569 ] Davies Liu commented on SPARK-18676: Yes, it can, see WholeStageCodegen.doExecute() as an example.

[jira] [Updated] (SPARK-16589) Chained cartesian produces incorrect number of records

2016-12-08 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-16589: --- Fix Version/s: (was: 2.1.0) 2.1.1 > Chained cartesian produces incorrect

[jira] [Resolved] (SPARK-16589) Chained cartesian produces incorrect number of records

2016-12-08 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-16589. Resolution: Fixed Fix Version/s: 2.1.0 2.0.3 > Chained cartesian

[jira] [Updated] (SPARK-16589) Chained cartesian produces incorrect number of records

2016-12-08 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-16589: --- Assignee: Andrew Ray > Chained cartesian produces incorrect number of records >

[jira] [Commented] (SPARK-18676) Spark 2.x query plan data size estimation can crash join queries versus 1.x

2016-12-06 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15726247#comment-15726247 ] Davies Liu commented on SPARK-18676: What's the schema and plan of the child looks like? It's

[jira] [Updated] (SPARK-18719) Document spark.ui.showConsoleProgress

2016-12-05 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-18719: --- Assignee: Nicholas > Document spark.ui.showConsoleProgress > - >

[jira] [Resolved] (SPARK-18719) Document spark.ui.showConsoleProgress

2016-12-05 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-18719. Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 16151

[jira] [Updated] (SPARK-18719) Document spark.ui.showConsoleProgress

2016-12-05 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-18719: --- Assignee: Nicholas Chammas (was: Nicholas) > Document spark.ui.showConsoleProgress >

[jira] [Updated] (SPARK-18188) Add checksum for block of broadcast

2016-11-18 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-18188: --- Description: There is an understanding issue for a long time:

[jira] [Updated] (SPARK-18188) Add checksum for block of broadcast

2016-11-18 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-18188: --- Summary: Add checksum for block of broadcast (was: Add checksum for block in Spark) > Add checksum

[jira] [Assigned] (SPARK-4105) FAILED_TO_UNCOMPRESS(5) errors when fetching shuffle data with sort-based shuffle

2016-11-17 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reassigned SPARK-4105: - Assignee: Davies Liu (was: Josh Rosen) > FAILED_TO_UNCOMPRESS(5) errors when fetching shuffle

[jira] [Commented] (SPARK-18097) Can't drop a table from Hive if the schema is corrupt

2016-11-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15654590#comment-15654590 ] Davies Liu commented on SPARK-18097: I have no idea why the schema is corrupt, we could catch the

[jira] [Resolved] (SPARK-18254) UDFs don't see aliased column names

2016-11-03 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-18254. Resolution: Fixed Assignee: Eyal Farago (was: Davies Liu) > UDFs don't see aliased column

[jira] [Commented] (SPARK-18254) UDFs don't see aliased column names

2016-11-03 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634416#comment-15634416 ] Davies Liu commented on SPARK-18254: Could you also try 2.0.2? > UDFs don't see aliased column names

[jira] [Commented] (SPARK-18254) UDFs don't see aliased column names

2016-11-03 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634333#comment-15634333 ] Davies Liu commented on SPARK-18254: I tried the following in master (2.1), it works {code}

[jira] [Commented] (SPARK-18254) UDFs don't see aliased column names

2016-11-03 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634228#comment-15634228 ] Davies Liu commented on SPARK-18254: I doubt it's a bug in ExtractPythonUDFs, not operator push down,

[jira] [Assigned] (SPARK-18254) UDFs don't see aliased column names

2016-11-03 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reassigned SPARK-18254: -- Assignee: Davies Liu > UDFs don't see aliased column names >

[jira] [Created] (SPARK-18233) Failed to deserialize the task

2016-11-02 Thread Davies Liu (JIRA)
Davies Liu created SPARK-18233: -- Summary: Failed to deserialize the task Key: SPARK-18233 URL: https://issues.apache.org/jira/browse/SPARK-18233 Project: Spark Issue Type: Bug

[jira] [Closed] (SPARK-4549) Support BigInt -> Decimal in convertToCatalyst in SparkSQL

2016-11-02 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu closed SPARK-4549. - Resolution: Incomplete > Support BigInt -> Decimal in convertToCatalyst in SparkSQL >

[jira] [Commented] (SPARK-18212) Flaky test: org.apache.spark.sql.kafka010.KafkaSourceSuite.assign from specific offsets

2016-11-01 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626935#comment-15626935 ] Davies Liu commented on SPARK-18212: cc [~zsxwing] > Flaky test:

[jira] [Created] (SPARK-18212) Flaky test: org.apache.spark.sql.kafka010.KafkaSourceSuite.assign from specific offsets

2016-11-01 Thread Davies Liu (JIRA)
Davies Liu created SPARK-18212: -- Summary: Flaky test: org.apache.spark.sql.kafka010.KafkaSourceSuite.assign from specific offsets Key: SPARK-18212 URL: https://issues.apache.org/jira/browse/SPARK-18212

[jira] [Created] (SPARK-18188) Add checksum for block in Spark

2016-10-31 Thread Davies Liu (JIRA)
Davies Liu created SPARK-18188: -- Summary: Add checksum for block in Spark Key: SPARK-18188 URL: https://issues.apache.org/jira/browse/SPARK-18188 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

2016-10-27 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612686#comment-15612686 ] Davies Liu commented on SPARK-18105: It turned out that the bug in LZ4 is a false alarm, so close the

[jira] [Updated] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

2016-10-27 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-18105: --- Priority: Major (was: Blocker) > LZ4 failed to decompress a stream of shuffled data >

[jira] [Updated] (SPARK-16078) from_utc_timestamp/to_utc_timestamp may give different result in different timezone

2016-10-27 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-16078: --- Fix Version/s: 1.6.3 > from_utc_timestamp/to_utc_timestamp may give different result in different >

[jira] [Updated] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

2016-10-26 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-18105: --- Description: When lz4 is used to compress the shuffle files, it may fail to decompress it as

[jira] [Commented] (SPARK-18100) Improve the performance of get_json_object using Gson

2016-10-26 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609078#comment-15609078 ] Davies Liu commented on SPARK-18100: [~viirya] Jackson does not support it either > Improve the

[jira] [Updated] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

2016-10-26 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-18105: --- Priority: Blocker (was: Major) > LZ4 failed to decompress a stream of shuffled data >

[jira] [Created] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

2016-10-25 Thread Davies Liu (JIRA)
Davies Liu created SPARK-18105: -- Summary: LZ4 failed to decompress a stream of shuffled data Key: SPARK-18105 URL: https://issues.apache.org/jira/browse/SPARK-18105 Project: Spark Issue Type:

[jira] [Created] (SPARK-18102) Failed to deserialize the result of task

2016-10-25 Thread Davies Liu (JIRA)
Davies Liu created SPARK-18102: -- Summary: Failed to deserialize the result of task Key: SPARK-18102 URL: https://issues.apache.org/jira/browse/SPARK-18102 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-18100) Improve the performance of get_json_object using Gson

2016-10-25 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-18100: --- Issue Type: Improvement (was: Bug) > Improve the performance of get_json_object using Gson >

[jira] [Created] (SPARK-18100) Improve the performance of get_json_object using Gson

2016-10-25 Thread Davies Liu (JIRA)
Davies Liu created SPARK-18100: -- Summary: Improve the performance of get_json_object using Gson Key: SPARK-18100 URL: https://issues.apache.org/jira/browse/SPARK-18100 Project: Spark Issue

[jira] [Updated] (SPARK-18097) Can't drop a table from Hive if the schema is corrupt

2016-10-25 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-18097: --- Description: When the schema of Hive table is broken, we can't drop the table using Spark SQL, for

[jira] [Updated] (SPARK-18097) Can't drop a table from Hive if the schema is corrupt

2016-10-25 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-18097: --- Description: When the schema of Hive table is broken, we can't drop the table using Spark SQL, for

[jira] [Created] (SPARK-18097) Can't drop a table from Hive if the schema is corrupt

2016-10-25 Thread Davies Liu (JIRA)
Davies Liu created SPARK-18097: -- Summary: Can't drop a table from Hive if the schema is corrupt Key: SPARK-18097 URL: https://issues.apache.org/jira/browse/SPARK-18097 Project: Spark Issue

[jira] [Updated] (SPARK-18055) Dataset.flatMap can't work with types from customized jar

2016-10-21 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-18055: --- Description: Try to apply flatMap() on Dataset column which of of type com.A.B Here's a schema of a

[jira] [Updated] (SPARK-18055) Dataset.flatMap can't work with types from customized jar

2016-10-21 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-18055: --- Description: Try to apply flatMap() on Dataset column which of of type com.A.B Here's a schema of a

[jira] [Created] (SPARK-18055) Dataset.flatMap can't work with types from customized jar

2016-10-21 Thread Davies Liu (JIRA)
Davies Liu created SPARK-18055: -- Summary: Dataset.flatMap can't work with types from customized jar Key: SPARK-18055 URL: https://issues.apache.org/jira/browse/SPARK-18055 Project: Spark Issue

[jira] [Updated] (SPARK-18053) ARRAY equality is broken in Spark 2.0

2016-10-21 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-18053: --- Assignee: Wenchen Fan > ARRAY equality is broken in Spark 2.0 >

[jira] [Created] (SPARK-18037) Event listener should be aware of multiple tries of same stage

2016-10-20 Thread Davies Liu (JIRA)
Davies Liu created SPARK-18037: -- Summary: Event listener should be aware of multiple tries of same stage Key: SPARK-18037 URL: https://issues.apache.org/jira/browse/SPARK-18037 Project: Spark

[jira] [Commented] (SPARK-10915) Add support for UDAFs in Python

2016-10-20 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592514#comment-15592514 ] Davies Liu commented on SPARK-10915: [~jason.white] When a aggregate function is applied, the order

[jira] [Created] (SPARK-18032) Spark test failed as OOM in jenkins

2016-10-20 Thread Davies Liu (JIRA)
Davies Liu created SPARK-18032: -- Summary: Spark test failed as OOM in jenkins Key: SPARK-18032 URL: https://issues.apache.org/jira/browse/SPARK-18032 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-18031) Flaky test: org.apache.spark.streaming.scheduler.ExecutorAllocationManagerSuite basic functionality

2016-10-20 Thread Davies Liu (JIRA)
Davies Liu created SPARK-18031: -- Summary: Flaky test: org.apache.spark.streaming.scheduler.ExecutorAllocationManagerSuite basic functionality Key: SPARK-18031 URL: https://issues.apache.org/jira/browse/SPARK-18031

[jira] [Created] (SPARK-18030) Flaky test: org.apache.spark.sql.streaming.FileStreamSourceSuite

2016-10-20 Thread Davies Liu (JIRA)
Davies Liu created SPARK-18030: -- Summary: Flaky test: org.apache.spark.sql.streaming.FileStreamSourceSuite Key: SPARK-18030 URL: https://issues.apache.org/jira/browse/SPARK-18030 Project: Spark

[jira] [Resolved] (SPARK-17388) Support for inferring type date/timestamp/decimal for partition column

2016-10-18 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-17388. Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 14947

[jira] [Commented] (SPARK-10915) Add support for UDAFs in Python

2016-10-17 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583638#comment-15583638 ] Davies Liu commented on SPARK-10915: Currently all the aggregate functions are implemented in Scala,

[jira] [Commented] (SPARK-10915) Add support for UDAFs in Python

2016-10-17 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582918#comment-15582918 ] Davies Liu commented on SPARK-10915: Python UDF is executed in batch mode to have reasonable

[jira] [Updated] (SPARK-17845) Improve window function frame boundary API in DataFrame

2016-10-12 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-17845: --- Fix Version/s: 2.1.0 > Improve window function frame boundary API in DataFrame >

[jira] [Resolved] (SPARK-17845) Improve window function frame boundary API in DataFrame

2016-10-12 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-17845. Resolution: Fixed > Improve window function frame boundary API in DataFrame >

[jira] [Commented] (SPARK-15621) BatchEvalPythonExec fails with OOM

2016-10-11 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566858#comment-15566858 ] Davies Liu commented on SPARK-15621: [~rezasafi] We usually do not backport this kind of

[jira] [Created] (SPARK-17856) JVM Crash during tests: pyspark.mllib.linalg.distributed

2016-10-10 Thread Davies Liu (JIRA)
Davies Liu created SPARK-17856: -- Summary: JVM Crash during tests: pyspark.mllib.linalg.distributed Key: SPARK-17856 URL: https://issues.apache.org/jira/browse/SPARK-17856 Project: Spark Issue

[jira] [Resolved] (SPARK-17806) Incorrect result when work with data from parquet

2016-10-07 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-17806. Resolution: Fixed Fix Version/s: 2.1.0 2.0.2 Issue resolved by pull

[jira] [Commented] (SPARK-17738) Flaky test: org.apache.spark.sql.execution.columnar.ColumnTypeSuite MAP append/extract

2016-10-07 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1759#comment-1759 ] Davies Liu commented on SPARK-17738: I will looking into that. > Flaky test:

[jira] [Assigned] (SPARK-17806) Incorrect result when work with data from parquet

2016-10-07 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reassigned SPARK-17806: -- Assignee: Davies Liu > Incorrect result when work with data from parquet >

[jira] [Updated] (SPARK-17806) Incorrect result when work with data from parquet

2016-10-07 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-17806: --- Priority: Blocker (was: Critical) > Incorrect result when work with data from parquet >

[jira] [Commented] (SPARK-16922) Query with Broadcast Hash join fails due to executor OOM in Spark 2.0

2016-10-05 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15549494#comment-15549494 ] Davies Liu commented on SPARK-16922: Thanks for the feedback, that's reasonable. > Query with

[jira] [Resolved] (SPARK-15390) Memory management issue in complex DataFrame join and filter

2016-10-04 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-15390. Resolution: Fixed > Memory management issue in complex DataFrame join and filter >

[jira] [Updated] (SPARK-15390) Memory management issue in complex DataFrame join and filter

2016-10-04 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-15390: --- Fix Version/s: 2.0.1 > Memory management issue in complex DataFrame join and filter >

[jira] [Comment Edited] (SPARK-15390) Memory management issue in complex DataFrame join and filter

2016-10-04 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15546675#comment-15546675 ] Davies Liu edited comment on SPARK-15390 at 10/4/16 9:11 PM: - @lulian Dragos

[jira] [Updated] (SPARK-15390) Memory management issue in complex DataFrame join and filter

2016-10-04 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-15390: --- Fix Version/s: (was: 2.0.0) > Memory management issue in complex DataFrame join and filter >

[jira] [Commented] (SPARK-15390) Memory management issue in complex DataFrame join and filter

2016-10-04 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15546675#comment-15546675 ] Davies Liu commented on SPARK-15390: @lulian Dragos I think this is a different issue, fixed by

[jira] [Resolved] (SPARK-17679) Remove unnecessary Py4J ListConverter patch

2016-10-03 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-17679. Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 15254

[jira] [Updated] (SPARK-17738) Flaky test: org.apache.spark.sql.execution.columnar.ColumnTypeSuite MAP append/extract

2016-09-30 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-17738: --- Fix Version/s: (was: 2.2.0) 2.1.0 > Flaky test:

[jira] [Resolved] (SPARK-17738) Flaky test: org.apache.spark.sql.execution.columnar.ColumnTypeSuite MAP append/extract

2016-09-30 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-17738. Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 15305

[jira] [Created] (SPARK-17738) Flaky test: org.apache.spark.sql.execution.columnar.ColumnTypeSuite MAP append/extract

2016-09-29 Thread Davies Liu (JIRA)
Davies Liu created SPARK-17738: -- Summary: Flaky test: org.apache.spark.sql.execution.columnar.ColumnTypeSuite MAP append/extract Key: SPARK-17738 URL: https://issues.apache.org/jira/browse/SPARK-17738

[jira] [Updated] (SPARK-17494) Floor/ceil of decimal returns wrong result if it's in compact format

2016-09-19 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-17494: --- Summary: Floor/ceil of decimal returns wrong result if it's in compact format (was: Floor function

[jira] [Updated] (SPARK-17100) pyspark filter on a udf column after join gives java.lang.UnsupportedOperationException

2016-09-19 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-17100: --- Fix Version/s: (was: 2.2.0) 2.1.0 > pyspark filter on a udf column after join

[jira] [Resolved] (SPARK-17100) pyspark filter on a udf column after join gives java.lang.UnsupportedOperationException

2016-09-19 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-17100. Resolution: Fixed Fix Version/s: 2.2.0 2.0.1 Issue resolved by pull

[jira] [Updated] (SPARK-16439) Incorrect information in SQL Query details

2016-09-19 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-16439: --- Fix Version/s: (was: 2.2.0) 2.1.0 > Incorrect information in SQL Query

[jira] [Assigned] (SPARK-17494) Floor function rounds up during join

2016-09-19 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reassigned SPARK-17494: -- Assignee: Davies Liu > Floor function rounds up during join >

[jira] [Updated] (SPARK-16439) Incorrect information in SQL Query details

2016-09-19 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-16439: --- Assignee: Davies Liu (was: Maciej BryƄski) > Incorrect information in SQL Query details >

[jira] [Resolved] (SPARK-16439) Incorrect information in SQL Query details

2016-09-19 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-16439. Resolution: Fixed Fix Version/s: (was: 2.0.0) 2.2.0

[jira] [Reopened] (SPARK-16439) Incorrect information in SQL Query details

2016-09-14 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reopened SPARK-16439: We could bring the seperator back for better readability. > Incorrect information in SQL Query

[jira] [Commented] (SPARK-16439) Incorrect information in SQL Query details

2016-09-14 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491744#comment-15491744 ] Davies Liu commented on SPARK-16439: The separator was added on purpose, otherwise it's very

  1   2   3   4   5   6   7   8   9   10   >