[jira] [Created] (SPARK-18730) Ask the build script to link to Jenkins test report page instead of full console output page when posting to GitHub

2016-12-05 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-18730: -- Summary: Ask the build script to link to Jenkins test report page instead of full console output page when posting to GitHub Key: SPARK-18730 URL:

[jira] [Commented] (SPARK-18539) Cannot filter by nonexisting column in parquet file

2016-12-05 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723781#comment-15723781 ] Cheng Lian commented on SPARK-18539: Please remind me if I missed anything important, otherwise, we

[jira] [Comment Edited] (SPARK-18539) Cannot filter by nonexisting column in parquet file

2016-12-05 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723747#comment-15723747 ] Cheng Lian edited comment on SPARK-18539 at 12/5/16 11:43 PM: --

[jira] [Commented] (SPARK-18539) Cannot filter by nonexisting column in parquet file

2016-12-05 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723747#comment-15723747 ] Cheng Lian commented on SPARK-18539: [~v-gerasimov], [~smilegator], and [~xwu0226], after some

[jira] [Commented] (SPARK-18539) Cannot filter by nonexisting column in parquet file

2016-12-05 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723718#comment-15723718 ] Cheng Lian commented on SPARK-18539: As commented on GitHub, there're two issues right now: # This

[jira] [Commented] (SPARK-18539) Cannot filter by nonexisting column in parquet file

2016-12-05 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722891#comment-15722891 ] Cheng Lian commented on SPARK-18539: Haven't looked deeply into this issue, but my hunch is that this

[jira] [Assigned] (SPARK-17213) Parquet String Pushdown for Non-Eq Comparisons Broken

2016-12-01 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian reassigned SPARK-17213: -- Assignee: Cheng Lian > Parquet String Pushdown for Non-Eq Comparisons Broken >

[jira] [Commented] (SPARK-17213) Parquet String Pushdown for Non-Eq Comparisons Broken

2016-12-01 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15712707#comment-15712707 ] Cheng Lian commented on SPARK-17213: Agree that we should disable string and binary filter push down

[jira] [Resolved] (SPARK-9876) Upgrade parquet-mr to 1.8.1

2016-12-01 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-9876. --- Resolution: Fixed Fix Version/s: 2.1.0 > Upgrade parquet-mr to 1.8.1 >

[jira] [Commented] (SPARK-18251) DataSet API | RuntimeException: Null value appeared in non-nullable field when holding Option Case Class

2016-11-30 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15709869#comment-15709869 ] Cheng Lian commented on SPARK-18251: One more comment about why we shouldn't allow a {{Option\[T <:

[jira] [Updated] (SPARK-18251) DataSet API | RuntimeException: Null value appeared in non-nullable field when holding Option Case Class

2016-11-30 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-18251: --- Assignee: Wenchen Fan > DataSet API | RuntimeException: Null value appeared in non-nullable field >

[jira] [Resolved] (SPARK-18251) DataSet API | RuntimeException: Null value appeared in non-nullable field when holding Option Case Class

2016-11-30 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-18251. Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 15979

[jira] [Comment Edited] (SPARK-18403) ObjectHashAggregateSuite is being flaky (occasional OOM errors)

2016-11-21 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684659#comment-15684659 ] Cheng Lian edited comment on SPARK-18403 at 11/22/16 6:54 AM: -- Here is a

[jira] [Commented] (SPARK-18403) ObjectHashAggregateSuite is being flaky (occasional OOM errors)

2016-11-21 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15685389#comment-15685389 ] Cheng Lian commented on SPARK-18403: Figured it out. It's caused by a false sharing issue inside

[jira] [Commented] (SPARK-18403) ObjectHashAggregateSuite is being flaky (occasional OOM errors)

2016-11-21 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684659#comment-15684659 ] Cheng Lian commented on SPARK-18403: Here is a minimal test case (add it to

[jira] [Commented] (SPARK-11785) When deployed against remote Hive metastore with lower versions, JDBC metadata calls throws exception

2016-11-18 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677469#comment-15677469 ] Cheng Lian commented on SPARK-11785: But I'm not sure which PR fixes this issue, though. > When

[jira] [Commented] (SPARK-11785) When deployed against remote Hive metastore with lower versions, JDBC metadata calls throws exception

2016-11-18 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677468#comment-15677468 ] Cheng Lian commented on SPARK-11785: Confirmed that this is no longer an issue for 2.1 > When

[jira] [Comment Edited] (SPARK-18251) DataSet API | RuntimeException: Null value appeared in non-nullable field when holding Option Case Class

2016-11-18 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677396#comment-15677396 ] Cheng Lian edited comment on SPARK-18251 at 11/18/16 6:38 PM: -- I'd prefer

[jira] [Comment Edited] (SPARK-18251) DataSet API | RuntimeException: Null value appeared in non-nullable field when holding Option Case Class

2016-11-18 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677396#comment-15677396 ] Cheng Lian edited comment on SPARK-18251 at 11/18/16 6:37 PM: -- I'd prefer

[jira] [Commented] (SPARK-18251) DataSet API | RuntimeException: Null value appeared in non-nullable field when holding Option Case Class

2016-11-18 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677396#comment-15677396 ] Cheng Lian commented on SPARK-18251: I'd prefer option 1 because of consistency of the semantics, and

[jira] [Created] (SPARK-18451) Always set -XX:+HeapDumpOnOutOfMemoryError for Spark tests

2016-11-15 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-18451: -- Summary: Always set -XX:+HeapDumpOnOutOfMemoryError for Spark tests Key: SPARK-18451 URL: https://issues.apache.org/jira/browse/SPARK-18451 Project: Spark Issue

[jira] [Created] (SPARK-18403) ObjectHashAggregateSuite is being flaky (occasional OOM errors)

2016-11-10 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-18403: -- Summary: ObjectHashAggregateSuite is being flaky (occasional OOM errors) Key: SPARK-18403 URL: https://issues.apache.org/jira/browse/SPARK-18403 Project: Spark

[jira] [Commented] (SPARK-18390) Optimized plan tried to use Cartesian join when it is not enabled

2016-11-09 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15652202#comment-15652202 ] Cheng Lian commented on SPARK-18390: I think this issue has already been fixed by SPARK-17298 and

[jira] [Updated] (SPARK-18390) Optimized plan tried to use Cartesian join when it is not enabled

2016-11-09 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-18390: --- Description: {code} val df2 = spark.range(1e9.toInt).withColumn("one", lit(1)) val df3 =

[jira] [Updated] (SPARK-18338) ObjectHashAggregateSuite fails under Maven builds

2016-11-07 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-18338: --- Description: Test case initialization order under Maven and SBT are different. Maven always creates

[jira] [Created] (SPARK-18338) ObjectHashAggregateSuite fails under Maven builds

2016-11-07 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-18338: -- Summary: ObjectHashAggregateSuite fails under Maven builds Key: SPARK-18338 URL: https://issues.apache.org/jira/browse/SPARK-18338 Project: Spark Issue Type:

[jira] [Updated] (SPARK-17972) Query planning slows down dramatically for large query plans even when sub-trees are cached

2016-11-02 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-17972: --- Description: The following Spark shell snippet creates a series of query plans that grow

[jira] [Resolved] (SPARK-11879) Checkpoint support for DataFrame/Dataset

2016-11-02 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-11879. Resolution: Duplicate > Checkpoint support for DataFrame/Dataset >

[jira] [Commented] (SPARK-11879) Checkpoint support for DataFrame/Dataset

2016-11-02 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15630537#comment-15630537 ] Cheng Lian commented on SPARK-11879: Sorry that I didn't notice this ticket while working on

[jira] [Commented] (SPARK-18209) More robust view canonicalization without full SQL expansion

2016-11-01 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626823#comment-15626823 ] Cheng Lian commented on SPARK-18209: One problem of the proposed approach is that our SQL parser

[jira] [Created] (SPARK-18186) Migrate HiveUDAFFunction to TypedImperativeAggregate for partial aggregation support

2016-10-31 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-18186: -- Summary: Migrate HiveUDAFFunction to TypedImperativeAggregate for partial aggregation support Key: SPARK-18186 URL: https://issues.apache.org/jira/browse/SPARK-18186

[jira] [Commented] (SPARK-18053) ARRAY equality is broken in Spark 2.0

2016-10-24 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602974#comment-15602974 ] Cheng Lian commented on SPARK-18053: Yea, reproduced using 2.0. > ARRAY equality is broken in Spark

[jira] [Commented] (SPARK-18053) ARRAY equality is broken in Spark 2.0

2016-10-24 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602969#comment-15602969 ] Cheng Lian commented on SPARK-18053: Hm, the user mailing list thread said that it fails under 2.0

[jira] [Created] (SPARK-18058) AnalysisException may be thrown when union two DFs whose struct fields have different nullability

2016-10-21 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-18058: -- Summary: AnalysisException may be thrown when union two DFs whose struct fields have different nullability Key: SPARK-18058 URL: https://issues.apache.org/jira/browse/SPARK-18058

[jira] [Updated] (SPARK-17949) Introduce a JVM object based aggregate operator

2016-10-21 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-17949: --- Description: The new Tungsten execution engine has very robust memory management and speed for

[jira] [Updated] (SPARK-17949) Introduce a JVM object based aggregate operator

2016-10-21 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-17949: --- Description: The new Tungsten execution engine has very robust memory management and speed for

[jira] [Updated] (SPARK-18053) ARRAY equality is broken in Spark 2.0

2016-10-21 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-18053: --- Labels: correctness (was: ) > ARRAY equality is broken in Spark 2.0 >

[jira] [Updated] (SPARK-18053) ARRAY equality is broken in Spark 2.0

2016-10-21 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-18053: --- Description: The following Spark shell reproduces this issue: {code} case class Test(a: Seq[Int])

[jira] [Created] (SPARK-18053) ARRAY equality is broken in Spark 2.0

2016-10-21 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-18053: -- Summary: ARRAY equality is broken in Spark 2.0 Key: SPARK-18053 URL: https://issues.apache.org/jira/browse/SPARK-18053 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-18012) Simplify WriterContainer code

2016-10-19 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-18012. Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 15551

[jira] [Updated] (SPARK-17949) Introduce a JVM object based aggregate operator

2016-10-19 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-17949: --- Attachment: [Design Doc] Support for Arbitrary Aggregation States.pdf > Introduce a JVM object based

[jira] [Updated] (SPARK-17949) Introduce a JVM object based aggregate operator

2016-10-19 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-17949: --- Attachment: (was: [Design Doc] Support for Arbitrary Aggregation States.pdf) > Introduce a JVM

[jira] [Updated] (SPARK-17949) Introduce a JVM object based aggregate operator

2016-10-19 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-17949: --- Attachment: [Design Doc] Support for Arbitrary Aggregation States.pdf > Introduce a JVM object based

[jira] [Created] (PARQUET-754) Deprecate the "strict" argument in MessageType.union()

2016-10-17 Thread Cheng Lian (JIRA)
Cheng Lian created PARQUET-754: -- Summary: Deprecate the "strict" argument in MessageType.union() Key: PARQUET-754 URL: https://issues.apache.org/jira/browse/PARQUET-754 Project: Parquet Issue

[jira] [Commented] (PARQUET-753) GroupType.union() doesn't merge the original type

2016-10-17 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/PARQUET-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583942#comment-15583942 ] Cheng Lian commented on PARQUET-753: PARQUET-379 resolves the {{union}} issue related to primitive

[jira] [Created] (SPARK-17972) Query planning slows down dramatically for large query plans even when sub-trees are cached

2016-10-17 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-17972: -- Summary: Query planning slows down dramatically for large query plans even when sub-trees are cached Key: SPARK-17972 URL: https://issues.apache.org/jira/browse/SPARK-17972

[jira] [Assigned] (SPARK-17949) Introduce a JVM object based aggregate operator

2016-10-14 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian reassigned SPARK-17949: -- Assignee: Cheng Lian > Introduce a JVM object based aggregate operator >

[jira] [Commented] (SPARK-10954) Parquet version in the "created_by" metadata field of Parquet files written by Spark 1.5 and 1.6 is wrong

2016-10-14 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576623#comment-15576623 ] Cheng Lian commented on SPARK-10954: [~hyukjin.kwon], yes, confirmed. Thanks! > Parquet version in

[jira] [Closed] (SPARK-9783) Use SqlNewHadoopRDD in JSONRelation to eliminate extra refresh() call

2016-10-14 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian closed SPARK-9783. - Resolution: Not A Problem This issue is no longer a problem since we re-implemented the JSON data source

[jira] [Commented] (SPARK-9783) Use SqlNewHadoopRDD in JSONRelation to eliminate extra refresh() call

2016-10-14 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576523#comment-15576523 ] Cheng Lian commented on SPARK-9783: --- Yes, I'm closing this. Thanks! > Use SqlNewHadoopRDD in

[jira] [Commented] (SPARK-17636) Parquet filter push down doesn't handle struct fields

2016-10-14 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576513#comment-15576513 ] Cheng Lian commented on SPARK-17636: [~MasterDDT], yes, just as what [~hyukjin.kwon] explained

[jira] [Updated] (SPARK-17636) Parquet filter push down doesn't handle struct fields

2016-10-14 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-17636: --- Description: There's a *PushedFilters* for a simple numeric field, but not for a numeric field

[jira] [Comment Edited] (SPARK-17845) Improve window function frame boundary API in DataFrame

2016-10-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15561376#comment-15561376 ] Cheng Lian edited comment on SPARK-17845 at 10/10/16 6:43 AM: -- One thing is

[jira] [Comment Edited] (SPARK-17845) Improve window function frame boundary API in DataFrame

2016-10-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15561376#comment-15561376 ] Cheng Lian edited comment on SPARK-17845 at 10/10/16 6:00 AM: -- One thing is

[jira] [Commented] (SPARK-17845) Improve window function frame boundary API in DataFrame

2016-10-09 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15561376#comment-15561376 ] Cheng Lian commented on SPARK-17845: One thing is that ANSI SQL also allows using arbitrary integral

[jira] [Commented] (SPARK-17725) Spark should not write out parquet files with schema containing non-nullable fields

2016-09-29 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533109#comment-15533109 ] Cheng Lian commented on SPARK-17725: Reproducing this issue by writing a Parquet file using: {code}

[jira] [Resolved] (SPARK-16516) Support for pushing down filters for decimal and timestamp types in ORC

2016-09-27 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-16516. Resolution: Fixed Issue resolved by pull request 14172

[jira] [Updated] (SPARK-16777) Parquet schema converter depends on deprecated APIs

2016-09-27 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16777: --- Fix Version/s: (was: 2.2.0) 2.1.0 > Parquet schema converter depends on

[jira] [Updated] (SPARK-16516) Support for pushing down filters for decimal and timestamp types in ORC

2016-09-27 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16516: --- Fix Version/s: 2.1.0 > Support for pushing down filters for decimal and timestamp types in ORC >

[jira] [Updated] (SPARK-16516) Support for pushing down filters for decimal and timestamp types in ORC

2016-09-27 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16516: --- Assignee: Hyukjin Kwon > Support for pushing down filters for decimal and timestamp types in ORC >

[jira] [Updated] (SPARK-16777) Parquet schema converter depends on deprecated APIs

2016-09-27 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16777: --- Fix Version/s: 2.1.0 2.0.2 > Parquet schema converter depends on deprecated APIs

[jira] [Resolved] (SPARK-16777) Parquet schema converter depends on deprecated APIs

2016-09-27 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-16777. Resolution: Fixed Fix Version/s: (was: 2.0.2) (was: 2.1.0)

[jira] [Updated] (SPARK-16777) Parquet schema converter depends on deprecated APIs

2016-09-27 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16777: --- Assignee: Hyukjin Kwon > Parquet schema converter depends on deprecated APIs >

[jira] [Comment Edited] (SPARK-8824) Support Parquet time related logical types

2016-09-27 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525313#comment-15525313 ] Cheng Lian edited comment on SPARK-8824 at 9/27/16 7:09 AM: Since we've

[jira] [Commented] (SPARK-8824) Support Parquet time related logical types

2016-09-27 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525313#comment-15525313 ] Cheng Lian commented on SPARK-8824: --- Since we've already upgraded parquet-mr in Spark master to 1.8.1,

[jira] [Commented] (SPARK-17572) Write.df is failing on spark cluster

2016-09-20 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506051#comment-15506051 ] Cheng Lian commented on SPARK-17572: Yea, I know you are not using HDFS. But Spark always uses Hadoop

[jira] [Commented] (SPARK-17572) Write.df is failing on spark cluster

2016-09-20 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505921#comment-15505921 ] Cheng Lian commented on SPARK-17572: Which version of Hadoop are you using? Does it work when you

[jira] [Updated] (SPARK-17572) Write.df is failing on spark cluster

2016-09-20 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-17572: --- Description: Hi, We have spark cluster with four nodes, all four nodes have NFS partition

[jira] [Resolved] (SPARK-17289) Sort based partial aggregation breaks due to SPARK-12978

2016-08-30 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-17289. Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 14865

[jira] [Updated] (SPARK-17289) Sort based partial aggregation breaks due to SPARK-12978

2016-08-30 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-17289: --- Assignee: Takeshi Yamamuro > Sort based partial aggregation breaks due to SPARK-12978 >

[jira] [Updated] (SPARK-16283) Implement percentile_approx SQL function

2016-08-25 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16283: --- Assignee: (was: Sean Zhong) > Implement percentile_approx SQL function >

[jira] [Updated] (SPARK-16283) Implement percentile_approx SQL function

2016-08-25 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16283: --- Assignee: Sean Zhong > Implement percentile_approx SQL function >

[jira] [Created] (SPARK-17182) CollectList and CollectSet should be marked as non-deterministic

2016-08-22 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-17182: -- Summary: CollectList and CollectSet should be marked as non-deterministic Key: SPARK-17182 URL: https://issues.apache.org/jira/browse/SPARK-17182 Project: Spark

[jira] [Resolved] (SPARK-16975) Spark-2.0.0 unable to infer schema for parquet data written by Spark-1.6.2

2016-08-12 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-16975. Resolution: Fixed Fix Version/s: 2.1.0 2.0.1 Issue resolved by pull

[jira] [Updated] (SPARK-16975) Spark-2.0.0 unable to infer schema for parquet data written by Spark-1.6.2

2016-08-12 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16975: --- Assignee: Dongjoon Hyun > Spark-2.0.0 unable to infer schema for parquet data written by Spark-1.6.2

[jira] [Resolved] (SPARK-16867) createTable and alterTable in ExternalCatalog should not take db

2016-08-04 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-16867. Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 14476

[jira] [Commented] (SPARK-16842) Concern about disallowing user-given schema for Parquet and ORC

2016-08-02 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403567#comment-15403567 ] Cheng Lian commented on SPARK-16842: First of all, the cost of schema discovery can be heavy when

[jira] [Updated] (SPARK-16621) Generate stable SQLs in SQLBuilder

2016-07-26 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16621: --- Assignee: Dongjoon Hyun > Generate stable SQLs in SQLBuilder > -- >

[jira] [Resolved] (SPARK-16621) Generate stable SQLs in SQLBuilder

2016-07-26 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-16621. Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 14257

[jira] [Updated] (SPARK-16666) Kryo encoder for custom complex classes

2016-07-26 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-1: --- Description: I'm trying to create a dataset with some geo data using spark and esri. If `Foo` only

[jira] [Updated] (SPARK-16666) Kryo encoder for custom complex classes

2016-07-26 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-1: --- Description: I'm trying to create a dataset with some geo data using spark and esri. If `Foo` only

[jira] [Updated] (SPARK-16734) Make sure examples in all language bindings are consistent

2016-07-26 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16734: --- Priority: Minor (was: Major) > Make sure examples in all language bindings are consistent >

[jira] [Resolved] (SPARK-16663) desc table should be consistent between data source and hive serde tables

2016-07-26 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-16663. Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 14302

[jira] [Created] (SPARK-16734) Make sure examples in all language bindings are consistent

2016-07-26 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-16734: -- Summary: Make sure examples in all language bindings are consistent Key: SPARK-16734 URL: https://issues.apache.org/jira/browse/SPARK-16734 Project: Spark Issue

[jira] [Resolved] (SPARK-16706) support java map in encoder

2016-07-26 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-16706. Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 14344

[jira] [Updated] (SPARK-16698) json parsing regression - "." in keys

2016-07-25 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16698: --- Assignee: Hyukjin Kwon > json parsing regression - "." in keys >

[jira] [Resolved] (SPARK-16698) json parsing regression - "." in keys

2016-07-25 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-16698. Resolution: Fixed Fix Version/s: 2.1.0 2.0.1 Issue resolved by pull

[jira] [Updated] (SPARK-16668) Test parquet reader for row groups containing both dictionary and plain encoded pages

2016-07-25 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16668: --- Assignee: Sameer Agarwal > Test parquet reader for row groups containing both dictionary and plain

[jira] [Resolved] (SPARK-16668) Test parquet reader for row groups containing both dictionary and plain encoded pages

2016-07-25 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-16668. Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 14304

[jira] [Resolved] (SPARK-16691) move BucketSpec to catalyst module and use it in CatalogTable

2016-07-25 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-16691. Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 14331

[jira] [Resolved] (SPARK-16660) CreateViewCommand should not take CatalogTable

2016-07-25 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-16660. Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 14297

[jira] [Updated] (SPARK-16703) Extra space in WindowSpecDefinition SQL representation

2016-07-24 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16703: --- Description: For a {{WindowSpecDefinition}} whose {{partitionSpec}} is empty, there's an extra

[jira] [Updated] (SPARK-16703) Extra space in WindowSpecDefinition SQL representation

2016-07-24 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16703: --- Description: For a {{WindowSpecDefinition}} whose {{partitionSpec}} is empty, there's an extra

[jira] [Created] (SPARK-16703) Extra space in WindowSpecDefinition SQL representation

2016-07-24 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-16703: -- Summary: Extra space in WindowSpecDefinition SQL representation Key: SPARK-16703 URL: https://issues.apache.org/jira/browse/SPARK-16703 Project: Spark Issue

[jira] [Commented] (SPARK-16646) LEAST doesn't accept numeric arguments with different data types

2016-07-22 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15389141#comment-15389141 ] Cheng Lian commented on SPARK-16646: Could you please help check Hive's behavior here? Especially

[jira] [Commented] (SPARK-16632) Vectorized parquet reader fails to read certain fields from Hive tables

2016-07-21 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387791#comment-15387791 ] Cheng Lian commented on SPARK-16632: Oh, I see, thanks for the explanation. > Vectorized parquet

[jira] [Commented] (SPARK-16646) LEAST doesn't accept numeric arguments with different data types

2016-07-20 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387172#comment-15387172 ] Cheng Lian commented on SPARK-16646: Thanks for the help! I'm not working on this. > LEAST doesn't

[jira] [Updated] (SPARK-16646) LEAST doesn't accept numeric arguments with different data types

2016-07-20 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16646: --- Reporter: Cheng Lian (was: liancheng) > LEAST doesn't accept numeric arguments with different data

[jira] [Updated] (SPARK-16646) LEAST doesn't accept numeric arguments with different data types

2016-07-20 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16646: --- Assignee: Hyukjin Kwon > LEAST doesn't accept numeric arguments with different data types >

[jira] [Updated] (SPARK-16648) LAST_VALUE(FALSE) OVER () throws IndexOutOfBoundsException

2016-07-20 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16648: --- Reporter: Cheng Lian (was: liancheng) > LAST_VALUE(FALSE) OVER () throws IndexOutOfBoundsException

<    1   2   3   4   5   6   7   8   9   10   >