[jira] [Updated] (SPARK-10136) Parquet support fail to decode Avro/Thrift arrays of primitive array (e.g. arrayarrayint)

2015-08-21 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-10136: --- Summary: Parquet support fail to decode Avro/Thrift arrays of primitive array (e.g. arrayarrayint)

[jira] [Commented] (PARQUET-364) Parque-avro cannot decode Avro/Thrift array of primitive array (e.g. arrayarrayint)

2015-08-21 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/PARQUET-364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14707150#comment-14707150 ] Cheng Lian commented on PARQUET-364: [~rdblue] The suggested fix has been verified by

[jira] [Created] (PARQUET-363) Cannot construct empty MessageType for ReadContext.requestedSchema

2015-08-21 Thread Cheng Lian (JIRA)
Cheng Lian created PARQUET-363: -- Summary: Cannot construct empty MessageType for ReadContext.requestedSchema Key: PARQUET-363 URL: https://issues.apache.org/jira/browse/PARQUET-363 Project: Parquet

[jira] [Commented] (HIVE-11611) A bad performance regression issue with Parquet happens if Hive does not select any columns

2015-08-21 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-11611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706312#comment-14706312 ] Cheng Lian commented on HIVE-11611: --- PARQUET-363 provided more background and details

[jira] [Resolved] (SPARK-10092) Multi-DB support follow up

2015-08-20 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-10092. Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 8336

[jira] [Updated] (SPARK-10136) Parquet support fail to decode Avro arrays of primitive array (e.g. arrayarrayint)

2015-08-20 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-10136: --- Description: The following Avro schema {noformat} record AvroNonNullableArrays { arrayarrayint

[jira] [Updated] (SPARK-10136) Parquet support fail to decode Avro arrays of primitive array (e.g. arrayarrayint)

2015-08-20 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-10136: --- Description: The following Avro schema {noformat} record AvroNonNullableArrays { arrayarrayint

[jira] [Created] (SPARK-10136) Parquet support fail to decode Avro arrays of primitive array (e.g. arrayarrayint)

2015-08-20 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-10136: -- Summary: Parquet support fail to decode Avro arrays of primitive array (e.g. arrayarrayint) Key: SPARK-10136 URL: https://issues.apache.org/jira/browse/SPARK-10136

[jira] [Updated] (SPARK-10136) Parquet support fail to decode Avro arrays of primitive array (e.g. arrayarrayint)

2015-08-20 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-10136: --- Description: The following Avro schema {noformat} record AvroNonNullableArrays { arrayarrayint

[jira] [Updated] (SPARK-10136) Parquet support fail to decode Avro arrays of primitive array (e.g. arrayarrayint)

2015-08-20 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-10136: --- Priority: Blocker (was: Major) Parquet support fail to decode Avro arrays of primitive array (e.g.

[jira] [Commented] (SPARK-10136) Parquet support fail to decode Avro arrays of primitive array (e.g. arrayarrayint)

2015-08-20 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14705189#comment-14705189 ] Cheng Lian commented on SPARK-10136: Marked this as BLOCKER since it's a regression

[jira] [Updated] (SPARK-9899) JSON/Parquet writing on retry or speculation broken with direct output committer

2015-08-19 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-9899: -- Description: If the first task fails all subsequent tasks will. We probably need to set a different

[jira] [Updated] (SPARK-9899) JSON/Parquet writing on retry or speculation broken with direct output committer

2015-08-19 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-9899: -- Description: If the first task fails all subsequent tasks will. We probably need to set a different

[jira] [Updated] (SPARK-9899) JSON/Parquet writing on retry or speculation broken with direct output committer

2015-08-19 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-9899: -- Description: If the first task fails all subsequent tasks will. We probably need to set a different

[jira] [Updated] (SPARK-9899) JSON/Parquet writing on retry or speculation broken with direct output committer

2015-08-19 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-9899: -- Description: If the first task fails all subsequent tasks will. We probably need to set a different

[jira] [Resolved] (SPARK-10035) Parquet filters does not process EqualNullSafe filter.

2015-08-19 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-10035. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8275

[jira] [Updated] (SPARK-10035) Parquet filters does not process EqualNullSafe filter.

2015-08-19 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-10035: --- Shepherd: Cheng Lian Affects Version/s: 1.5.0 1.4.1 Parquet

[jira] [Resolved] (SPARK-9600) DataFrameWriter.saveAsTable always writes data to /user/hive/warehouse

2015-08-19 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-9600. --- Resolution: Not A Problem DataFrameWriter.saveAsTable always writes data to /user/hive/warehouse

[jira] [Commented] (SPARK-9600) DataFrameWriter.saveAsTable always writes data to /user/hive/warehouse

2015-08-19 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14702747#comment-14702747 ] Cheng Lian commented on SPARK-9600: --- [~sthotaibeam] Sorry for my late reply. With the

[jira] [Commented] (SPARK-9627) SQL job failed if the dataframe with string columns is cached

2015-08-19 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14702661#comment-14702661 ] Cheng Lian commented on SPARK-9627: --- OK I finally reproduced this issue. The tricky part

[jira] [Commented] (SPARK-9627) SQL job failed if the dataframe with string columns is cached

2015-08-19 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14702671#comment-14702671 ] Cheng Lian commented on SPARK-9627: --- A quick Googling suggesting that it's probably

[jira] [Commented] (SPARK-9627) SQL job failed if the dataframe with string columns is cached

2015-08-18 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14701167#comment-14701167 ] Cheng Lian commented on SPARK-9627: --- [~davies] I tried to reproduce this issue locally

[jira] [Resolved] (SPARK-8118) Turn off noisy log output produced by Parquet 1.7.0

2015-08-18 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-8118. --- Resolution: Fixed Issue resolved by pull request 8196 [https://github.com/apache/spark/pull/8196]

[jira] [Resolved] (SPARK-9606) HiveThriftServer tests failing.

2015-08-18 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-9606. --- Resolution: Fixed Fix Version/s: 1.5.0 Fixed by SPARK-9939 HiveThriftServer tests failing.

[jira] [Resolved] (SPARK-9939) Resort to Java process API in test suites forking subprocesses

2015-08-18 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-9939. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 8168

[jira] [Updated] (SPARK-10035) Parquet filters does not process EqualNullSafe filter.

2015-08-17 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-10035: --- Assignee: Hyukjin Kwon Parquet filters does not process EqualNullSafe filter.

[jira] [Commented] (SPARK-10035) Parquet filters does not process EqualNullSafe filter.

2015-08-17 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699316#comment-14699316 ] Cheng Lian commented on SPARK-10035: Done, thanks for working on this! Parquet

[jira] [Commented] (SPARK-10030) Managed memory leak detected when cache table

2015-08-17 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699372#comment-14699372 ] Cheng Lian commented on SPARK-10030: [~joshrosen] Seems to be related to Tungsten?

[jira] [Resolved] (SPARK-7837) NPE when save as parquet in speculative tasks

2015-08-17 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-7837. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 8236

[jira] [Updated] (SPARK-9973) Wrong initial size of in-memory columnar buffers

2015-08-16 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-9973: -- Summary: Wrong initial size of in-memory columnar buffers (was: wrong buffle size) Wrong initial

[jira] [Updated] (SPARK-9973) wrong buffle size

2015-08-16 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-9973: -- Assignee: xukun wrong buffle size - Key: SPARK-9973

[jira] [Updated] (SPARK-9973) Wrong initial size of in-memory columnar buffers

2015-08-16 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-9973: -- Shepherd: Cheng Lian Sprint: Spark 1.5 doc/QA sprint Affects Version/s:

[jira] [Commented] (SPARK-9973) Wrong initial size of in-memory columnar buffers

2015-08-16 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14698553#comment-14698553 ] Cheng Lian commented on SPARK-9973: --- I've updated the title and description. Wrong

[jira] [Updated] (SPARK-10005) Parquet reader doesn't handle schema merging properly for nested structs

2015-08-16 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-10005: --- Description: Spark shell snippet to reproduce this issue (note that both {{DataFrame}} written

[jira] [Resolved] (SPARK-9973) Wrong initial size of in-memory columnar buffers

2015-08-16 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-9973. --- Resolution: Fixed Resolved by https://github.com/apache/spark/pull/8189 Wrong initial size of

[jira] [Updated] (SPARK-9973) Wrong initial size of in-memory columnar buffers

2015-08-16 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-9973: -- Fix Version/s: 1.5.0 Wrong initial size of in-memory columnar buffers

[jira] [Commented] (SPARK-7837) NPE when save as parquet in speculative tasks

2015-08-16 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14698767#comment-14698767 ] Cheng Lian commented on SPARK-7837: --- Just a note to people who want to reproduce this

[jira] [Commented] (SPARK-7837) NPE when save as parquet in speculative tasks

2015-08-16 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699036#comment-14699036 ] Cheng Lian commented on SPARK-7837: --- Good job! NPE when save as parquet in speculative

[jira] [Updated] (SPARK-9958) HiveThriftServer2Listener is not thread-safe

2015-08-14 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-9958: -- Assignee: Shixiong Zhu HiveThriftServer2Listener is not thread-safe

[jira] [Resolved] (SPARK-9958) HiveThriftServer2Listener is not thread-safe

2015-08-14 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-9958. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 8185

[jira] [Commented] (SPARK-8118) Turn off noisy log output produced by Parquet 1.7.0

2015-08-14 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697203#comment-14697203 ] Cheng Lian commented on SPARK-8118: --- Unfortunately no. Turn off noisy log output

[jira] [Updated] (SPARK-9606) HiveThriftServer tests failing.

2015-08-14 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-9606: -- Sprint: Spark 1.5 doc/QA sprint (was: Spark 1.5 release) HiveThriftServer tests failing.

[jira] [Created] (SPARK-10005) Parquet reader doesn't handle schema merging properly for nested structs

2015-08-14 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-10005: -- Summary: Parquet reader doesn't handle schema merging properly for nested structs Key: SPARK-10005 URL: https://issues.apache.org/jira/browse/SPARK-10005 Project: Spark

[jira] [Updated] (SPARK-9974) SBT build: com.twitter:parquet-hadoop-bundle:1.6.0 is not packaged into the assembly jar

2015-08-14 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-9974: -- Description: One of the consequence of this issue is that Parquet tables created in Hive are not

[jira] [Updated] (SPARK-6624) Convert filters into CNF for data sources

2015-08-14 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-6624: -- Assignee: Yijie Shen Convert filters into CNF for data sources

[jira] [Created] (SPARK-9974) SBT build: com.twitter:parquet-hadoop-bundle:1.6.0 is not packaged into the assembly jar

2015-08-14 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-9974: - Summary: SBT build: com.twitter:parquet-hadoop-bundle:1.6.0 is not packaged into the assembly jar Key: SPARK-9974 URL: https://issues.apache.org/jira/browse/SPARK-9974

[jira] [Updated] (SPARK-9974) SBT build: com.twitter:parquet-hadoop-bundle:1.6.0 is not packaged into the assembly jar

2015-08-14 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-9974: -- Description: One of the consequence of this issue is that Parquet tables created in Hive are not

[jira] [Updated] (PARQUET-173) StatisticsFilter doesn't handle And properly

2015-08-13 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/PARQUET-173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated PARQUET-173: --- Description: I guess it's [a pretty straightforward

[jira] [Updated] (SPARK-6795) Avoid reading Parquet footers on driver side when an global arbitrative schema is available

2015-08-13 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-6795: -- Fix Version/s: 1.5.0 Avoid reading Parquet footers on driver side when an global arbitrative schema

[jira] [Updated] (SPARK-6795) Avoid reading Parquet footers on driver side when an global arbitrative schema is available

2015-08-13 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-6795: -- Target Version/s: 1.5.0 (was: 1.6.0) Avoid reading Parquet footers on driver side when an global

[jira] [Updated] (SPARK-9757) Can't create persistent data source tables with decimal

2015-08-13 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-9757: -- Description: {{ParquetHiveSerDe}} in Hive versions 1.2.0 doesn't support decimal. Persisting Parquet

[jira] [Updated] (PARQUET-136) NPE thrown in StatisticsFilter when all values in a string/binary column trunk are null

2015-08-13 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/PARQUET-136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated PARQUET-136: --- Description: For a string or a binary column, if all values in a single column trunk are null, so

[jira] [Resolved] (SPARK-9885) IsolatedClientLoader ignores shared prefixes and barrier prefixes when spark.sql.hive.metastore.jars is set to maven

2015-08-13 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-9885. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 8158

[jira] [Assigned] (SPARK-9757) Can't create persistent data source tables with decimal

2015-08-13 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian reassigned SPARK-9757: - Assignee: Cheng Lian Can't create persistent data source tables with decimal

[jira] [Resolved] (SPARK-9757) Can't create persistent data source tables with decimal

2015-08-13 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-9757. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 8130

[jira] [Commented] (SPARK-6795) Avoid reading Parquet footers on driver side when an global arbitrative schema is available

2015-08-13 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694899#comment-14694899 ] Cheng Lian commented on SPARK-6795: --- As explained on GitHub, usually we only backport

[jira] [Commented] (SPARK-9725) spark sql query string field return empty/garbled string

2015-08-13 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14696235#comment-14696235 ] Cheng Lian commented on SPARK-9725: --- Seems that the result of {{show tables}} is

[jira] [Commented] (SPARK-9725) spark sql query string field return empty/garbled string

2015-08-13 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14696230#comment-14696230 ] Cheng Lian commented on SPARK-9725: --- I don't have so much memory to reproduce this issue

[jira] [Created] (SPARK-9939) Resort to Java process API in test suites forking subprocesses

2015-08-13 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-9939: - Summary: Resort to Java process API in test suites forking subprocesses Key: SPARK-9939 URL: https://issues.apache.org/jira/browse/SPARK-9939 Project: Spark

[jira] [Updated] (SPARK-9939) Resort to Java process API in test suites forking subprocesses

2015-08-13 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-9939: -- Description: The following SQL test suites fork subprocesses and have been flaky for quite a while and

[jira] [Updated] (SPARK-9939) Resort to Java process API in test suites forking subprocesses

2015-08-13 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-9939: -- Description: The following SQL test suites fork subprocesses and have been flaky for quite a while and

[jira] [Resolved] (SPARK-9927) Revert fix of 9182 since it's pushing the wrong filter down

2015-08-12 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-9927. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 8157

[jira] [Updated] (SPARK-9182) filter and groupBy on DataFrames are not passed through to jdbc source

2015-08-12 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-9182: -- Priority: Blocker (was: Critical) filter and groupBy on DataFrames are not passed through to jdbc

[jira] [Commented] (SPARK-9927) Revert fix of 9182 since it's pushing the wrong filter down

2015-08-12 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694684#comment-14694684 ] Cheng Lian commented on SPARK-9927: --- Could you provide a snippet that reproduces this

[jira] [Reopened] (SPARK-9182) filter and groupBy on DataFrames are not passed through to jdbc source

2015-08-12 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian reopened SPARK-9182: --- Found a regression in https://github.com/apache/spark/pull/8049 and reverted it via

[jira] [Commented] (SPARK-9182) filter and groupBy on DataFrames are not passed through to jdbc source

2015-08-12 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694739#comment-14694739 ] Cheng Lian commented on SPARK-9182: --- [~grahn] Unfortunately we found a regression in the

[jira] [Created] (SPARK-9885) IsolatedClientLoader ignores shared prefixes and barrier prefixes when spark.sql.hive.metastore.jars is set to maven

2015-08-12 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-9885: - Summary: IsolatedClientLoader ignores shared prefixes and barrier prefixes when spark.sql.hive.metastore.jars is set to maven Key: SPARK-9885 URL:

[jira] [Commented] (SPARK-9885) IsolatedClientLoader ignores shared prefixes and barrier prefixes when spark.sql.hive.metastore.jars is set to maven

2015-08-12 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14693851#comment-14693851 ] Cheng Lian commented on SPARK-9885: --- cc [~yhuai] This is the issue we discussed

[jira] [Resolved] (SPARK-9182) filter and groupBy on DataFrames are not passed through to jdbc source

2015-08-12 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-9182. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 8049

[jira] [Resolved] (SPARK-9407) Parquet shouldn't fail when pushing down predicates over a column whose underlying Parquet type is an ENUM

2015-08-12 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-9407. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 8107

[jira] [Created] (SPARK-9876) Upgrade parquet-mr to 1.8.1

2015-08-12 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-9876: - Summary: Upgrade parquet-mr to 1.8.1 Key: SPARK-9876 URL: https://issues.apache.org/jira/browse/SPARK-9876 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-9407) Parquet shouldn't fail when pushing down predicates over a column whose underlying Parquet type is an ENUM

2015-08-11 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-9407: -- Summary: Parquet shouldn't fail when pushing down predicates over a column whose underlying Parquet

[jira] [Comment Edited] (SPARK-8824) Support Parquet logical types TIMESTAMP_MILLIS and TIMESTAMP_MICROS

2015-08-11 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14681331#comment-14681331 ] Cheng Lian edited comment on SPARK-8824 at 8/11/15 6:55 AM: Oh

[jira] [Commented] (SPARK-8824) Support Parquet logical types TIMESTAMP_MILLIS and TIMESTAMP_MICROS

2015-08-11 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14681331#comment-14681331 ] Cheng Lian commented on SPARK-8824: --- Oh sorry, I mistook your request for

[jira] [Resolved] (SPARK-9340) CatalystSchemaConverter and CatalystRowConverter don't handle unannotated repeated fields correctly

2015-08-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-9340. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 8070

[jira] [Updated] (SPARK-9783) Use SqlNewHadoopRDD in JSONRelation to eliminate extra refresh() call

2015-08-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-9783: -- Sprint: Spark 1.5 doc/QA sprint Environment: (was: PR #8035 made a quick fix for SPARK-9743

[jira] [Updated] (SPARK-9340) CatalystSchemaConverter and CatalystRowConverter don't handle unannotated repeated fields correctly

2015-08-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-9340: -- Sprint: Spark 1.5 doc/QA sprint Target Version/s: 1.5.0 CatalystSchemaConverter and

[jira] [Commented] (SPARK-9783) Use SqlNewHadoopRDD in JSONRelation to eliminate extra refresh() call

2015-08-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680457#comment-14680457 ] Cheng Lian commented on SPARK-9783: --- cc [~yhuai] Use SqlNewHadoopRDD in JSONRelation

[jira] [Commented] (SPARK-9340) ParquetTypeConverter incorrectly handling of repeated types results in schema mismatch

2015-08-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680418#comment-14680418 ] Cheng Lian commented on SPARK-9340: --- Thanks for the clarification. In [PR

[jira] [Updated] (SPARK-9340) CatalystSchemaConverter and CatalystRowConverter don't handle unannotated repeated fields correctly

2015-08-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-9340: -- Summary: CatalystSchemaConverter and CatalystRowConverter don't handle unannotated repeated fields

[jira] [Created] (SPARK-9783) Use SqlNewHadoopRDD in JSONRelation to eliminate extra refresh() call

2015-08-10 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-9783: - Summary: Use SqlNewHadoopRDD in JSONRelation to eliminate extra refresh() call Key: SPARK-9783 URL: https://issues.apache.org/jira/browse/SPARK-9783 Project: Spark

[jira] [Commented] (SPARK-9340) CatalystSchemaConverter and CatalystRowConverter don't handle unannotated repeated fields correctly

2015-08-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680523#comment-14680523 ] Cheng Lian commented on SPARK-9340: --- Great, would you mind to leave a LGTM on the GitHub

[jira] [Assigned] (SPARK-9340) CatalystSchemaConverter and CatalystRowConverter don't handle unannotated repeated fields correctly

2015-08-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian reassigned SPARK-9340: - Assignee: Cheng Lian CatalystSchemaConverter and CatalystRowConverter don't handle unannotated

[jira] [Commented] (SPARK-9340) ParquetTypeConverter incorrectly handling of repeated types results in schema mismatch

2015-08-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680389#comment-14680389 ] Cheng Lian commented on SPARK-9340: --- [~damianguy] Would you mind to help reviewing [PR

[jira] [Updated] (SPARK-9340) CatalystSchemaConverter and CatalystRowConverter don't handle unannotated repeated fields correctly

2015-08-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-9340: -- Description: SPARK-6776 and SPARK-6777 followed {{parquet-avro}} to implement backwards-compatibility

[jira] [Commented] (SPARK-9340) ParquetTypeConverter incorrectly handling of repeated types results in schema mismatch

2015-08-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680277#comment-14680277 ] Cheng Lian commented on SPARK-9340: --- Ah, thanks a lot! I see the problem now.

[jira] [Commented] (SPARK-9340) ParquetTypeConverter incorrectly handling of repeated types results in schema mismatch

2015-08-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679719#comment-14679719 ] Cheng Lian commented on SPARK-9340: --- Would like to add that, parquet-avro can be from

[jira] [Comment Edited] (SPARK-9340) ParquetTypeConverter incorrectly handling of repeated types results in schema mismatch

2015-08-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679719#comment-14679719 ] Cheng Lian edited comment on SPARK-9340 at 8/10/15 7:45 AM:

[jira] [Commented] (SPARK-9340) ParquetTypeConverter incorrectly handling of repeated types results in schema mismatch

2015-08-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679684#comment-14679684 ] Cheng Lian commented on SPARK-9340: --- [~damianguy] This is actually a Parquet complex

[jira] [Comment Edited] (SPARK-9340) ParquetTypeConverter incorrectly handling of repeated types results in schema mismatch

2015-08-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679719#comment-14679719 ] Cheng Lian edited comment on SPARK-9340 at 8/10/15 7:40 AM:

[jira] [Comment Edited] (SPARK-9340) ParquetTypeConverter incorrectly handling of repeated types results in schema mismatch

2015-08-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679719#comment-14679719 ] Cheng Lian edited comment on SPARK-9340 at 8/10/15 7:43 AM:

[jira] [Commented] (SPARK-9600) DataFrameWriter.saveAsTable always writes data to /user/hive/warehouse

2015-08-08 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14662894#comment-14662894 ] Cheng Lian commented on SPARK-9600: --- [~sthotaibeam] Thanks for investigating this issue.

[jira] [Updated] (SPARK-9600) DataFrameWriter.saveAsTable always writes data to /user/hive/warehouse

2015-08-08 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-9600: -- Description: Get a clean Spark 1.4.1 build: {noformat} $ git checkout v1.4.1 $ ./build/sbt -Phive

[jira] [Commented] (SPARK-9701) allow not automatically using HiveContext with spark-shell when hive support built in

2015-08-08 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14662895#comment-14662895 ] Cheng Lian commented on SPARK-9701: --- I targeted it to 1.5.0. If we can't make it, we can

[jira] [Updated] (SPARK-9701) allow not automatically using HiveContext with spark-shell when hive support built in

2015-08-08 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-9701: -- Target Version/s: 1.5.0 allow not automatically using HiveContext with spark-shell when hive support

[jira] [Updated] (SPARK-9182) filter and groupBy on DataFrames are not passed through to jdbc source

2015-08-08 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-9182: -- Assignee: Yijie Shen (was: Cheng Lian) filter and groupBy on DataFrames are not passed through to

[jira] [Updated] (SPARK-9689) Cache doesn't refresh for HadoopFsRelation based table

2015-08-07 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-9689: -- Assignee: Cheng Hao Affects Version/s: 1.5.0 1.4.1 Cache doesn't

[jira] [Updated] (SPARK-9689) Cache doesn't refresh for HadoopFsRelation based table

2015-08-07 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-9689: -- Shepherd: Cheng Lian Cache doesn't refresh for HadoopFsRelation based table

[jira] [Created] (SPARK-9743) Scanning a HadoopFsRelation shouldn't requrire refreshing

2015-08-07 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-9743: - Summary: Scanning a HadoopFsRelation shouldn't requrire refreshing Key: SPARK-9743 URL: https://issues.apache.org/jira/browse/SPARK-9743 Project: Spark Issue

[jira] [Commented] (SPARK-9182) filter and groupBy on DataFrames are not passed through to jdbc source

2015-08-06 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660278#comment-14660278 ] Cheng Lian commented on SPARK-9182: --- Hey [~grahn], sorry for the late reply, I somehow

<    6   7   8   9   10   11   12   13   14   15   >