[jira] [Commented] (SPARK-18539) Cannot filter by nonexisting column in parquet file

[ https://issues.apache.org/jira/browse/SPARK-18539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15723296#comment-15723296 ] Xin Wu commented on SPARK-18539: Yes. I have the fix and will submit PR and cc everyone f

[jira] [Commented] (SPARK-18539) Cannot filter by nonexisting column in parquet file

[ https://issues.apache.org/jira/browse/SPARK-18539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15723888#comment-15723888 ] Xin Wu commented on SPARK-18539: I think we will hit the issue if we use user-specified s

[jira] [Comment Edited] (SPARK-18539) Cannot filter by nonexisting column in parquet file

[ https://issues.apache.org/jira/browse/SPARK-18539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15723888#comment-15723888 ] Xin Wu edited comment on SPARK-18539 at 12/6/16 12:46 AM: -- I thi

[jira] [Commented] (SPARK-18727) Support schema evolution as new files are inserted into table

[ https://issues.apache.org/jira/browse/SPARK-18727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729696#comment-15729696 ] Xin Wu commented on SPARK-18727: I am currently working on ALTER TABLE ADD COLUMNS, to ta

[jira] [Created] (SPARK-15025) creating datasource table with option (PATH) results in duplicate path key in serdeProperties

Xin Wu created SPARK-15025: -- Summary: creating datasource table with option (PATH) results in duplicate path key in serdeProperties Key: SPARK-15025 URL: https://issues.apache.org/jira/browse/SPARK-15025 Pro

[jira] [Commented] (SPARK-15025) creating datasource table with option (PATH) results in duplicate path key in serdeProperties

[ https://issues.apache.org/jira/browse/SPARK-15025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265127#comment-15265127 ] Xin Wu commented on SPARK-15025: I am testing a fix on this . will submit a PR soon. >

[jira] [Commented] (SPARK-14927) DataFrame. saveAsTable creates RDD partitions but not Hive partitions

[ https://issues.apache.org/jira/browse/SPARK-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265591#comment-15265591 ] Xin Wu commented on SPARK-14927: Since Spark 2.0.0 has moved around a lot of stuff, inclu

[jira] [Commented] (SPARK-14495) Distinct aggregation cannot be used in the having clause

[ https://issues.apache.org/jira/browse/SPARK-14495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15266069#comment-15266069 ] Xin Wu commented on SPARK-14495: I can recreated it on branch-1.6. and another workaround

[jira] [Comment Edited] (SPARK-14495) Distinct aggregation cannot be used in the having clause

[ https://issues.apache.org/jira/browse/SPARK-14495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15266069#comment-15266069 ] Xin Wu edited comment on SPARK-14495 at 5/2/16 2:21 AM: I can rec

[jira] [Commented] (SPARK-15044) spark-sql will throw "input path does not exist" exception if it handles a partition which exists in hive table, but the path is removed manually

[ https://issues.apache.org/jira/browse/SPARK-15044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15266116#comment-15266116 ] Xin Wu commented on SPARK-15044: I tried {code}alter table test drop partition (p=1){code

[jira] [Commented] (SPARK-14927) DataFrame. saveAsTable creates RDD partitions but not Hive partitions

[ https://issues.apache.org/jira/browse/SPARK-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15267576#comment-15267576 ] Xin Wu commented on SPARK-14927: right now, when a datasource table is created with parti

[jira] [Comment Edited] (SPARK-14927) DataFrame. saveAsTable creates RDD partitions but not Hive partitions

[ https://issues.apache.org/jira/browse/SPARK-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15267576#comment-15267576 ] Xin Wu edited comment on SPARK-14927 at 5/2/16 10:01 PM: - right n

[jira] [Commented] (SPARK-15044) spark-sql will throw "input path does not exist" exception if it handles a partition which exists in hive table, but the path is removed manually

[ https://issues.apache.org/jira/browse/SPARK-15044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15268071#comment-15268071 ] Xin Wu commented on SPARK-15044: Sorry. What I meant was that after I removed the path ma

[jira] [Commented] (SPARK-14495) Distinct aggregation cannot be used in the having clause

[ https://issues.apache.org/jira/browse/SPARK-14495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271956#comment-15271956 ] Xin Wu commented on SPARK-14495: [~smilegator] I got the fix and running regtest now. Wil

[jira] [Comment Edited] (SPARK-14495) Distinct aggregation cannot be used in the having clause

[ https://issues.apache.org/jira/browse/SPARK-14495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271956#comment-15271956 ] Xin Wu edited comment on SPARK-14495 at 5/5/16 6:25 AM: [~smilega

[jira] [Created] (SPARK-15206) Add testcases for Distinct Aggregation in Having clause

Xin Wu created SPARK-15206: -- Summary: Add testcases for Distinct Aggregation in Having clause Key: SPARK-15206 URL: https://issues.apache.org/jira/browse/SPARK-15206 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-15206) Add testcases for Distinct Aggregation in Having clause

[ https://issues.apache.org/jira/browse/SPARK-15206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Wu updated SPARK-15206: --- Issue Type: Test (was: Bug) > Add testcases for Distinct Aggregation in Having clause >

[jira] [Commented] (SPARK-15269) Creating external table in test code leaves empty directory under warehouse directory

[ https://issues.apache.org/jira/browse/SPARK-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15280877#comment-15280877 ] Xin Wu commented on SPARK-15269: The root cause maybe the following? When the first tabl

[jira] [Comment Edited] (SPARK-15269) Creating external table in test code leaves empty directory under warehouse directory

[ https://issues.apache.org/jira/browse/SPARK-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15280877#comment-15280877 ] Xin Wu edited comment on SPARK-15269 at 5/11/16 9:47 PM: - The roo

[jira] [Commented] (SPARK-15269) Creating external table in test code leaves empty directory under warehouse directory

[ https://issues.apache.org/jira/browse/SPARK-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15280888#comment-15280888 ] Xin Wu commented on SPARK-15269: In spark-shell, I can recreate it as following: {code} s

[jira] [Comment Edited] (SPARK-15269) Creating external table in test code leaves empty directory under warehouse directory

[ https://issues.apache.org/jira/browse/SPARK-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15280877#comment-15280877 ] Xin Wu edited comment on SPARK-15269 at 5/11/16 10:13 PM: -- The r

[jira] [Commented] (SPARK-15269) Creating external table in test code leaves empty directory under warehouse directory

[ https://issues.apache.org/jira/browse/SPARK-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15281621#comment-15281621 ] Xin Wu commented on SPARK-15269: For the case where we can not recreate this issue, it is

[jira] [Comment Edited] (SPARK-15269) Creating external table in test code leaves empty directory under warehouse directory

[ https://issues.apache.org/jira/browse/SPARK-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15281621#comment-15281621 ] Xin Wu edited comment on SPARK-15269 at 5/12/16 3:37 PM: - For the

[jira] [Commented] (SPARK-15269) Creating external table leaves empty directory under warehouse directory

[ https://issues.apache.org/jira/browse/SPARK-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15281748#comment-15281748 ] Xin Wu commented on SPARK-15269: Yes, I can . Thanks! > Creating external table leaves e

[jira] [Commented] (SPARK-15269) Creating external table leaves empty directory under warehouse directory

[ https://issues.apache.org/jira/browse/SPARK-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15282215#comment-15282215 ] Xin Wu commented on SPARK-15269: FYI.. The reason why the default database paths obtaine

[jira] [Comment Edited] (SPARK-15269) Creating external table leaves empty directory under warehouse directory

[ https://issues.apache.org/jira/browse/SPARK-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15282215#comment-15282215 ] Xin Wu edited comment on SPARK-15269 at 5/12/16 11:04 PM: -- FYI..

[jira] [Commented] (SPARK-15236) No way to disable Hive support in REPL

[ https://issues.apache.org/jira/browse/SPARK-15236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15282297#comment-15282297 ] Xin Wu commented on SPARK-15236: i am looking into this > No way to disable Hive support

[jira] [Created] (SPARK-15431) Support LIST FILE(s)|JAR(s) command natively

Xin Wu created SPARK-15431: -- Summary: Support LIST FILE(s)|JAR(s) command natively Key: SPARK-15431 URL: https://issues.apache.org/jira/browse/SPARK-15431 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-15463) Support for creating a dataframe from CSV in RDD[String]

[ https://issues.apache.org/jira/browse/SPARK-15463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15299167#comment-15299167 ] Xin Wu commented on SPARK-15463: I am looking into this. > Support for creating a dataf

[jira] [Updated] (SPARK-14361) Support EXCLUDE clause in Window function framing

[ https://issues.apache.org/jira/browse/SPARK-14361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Wu updated SPARK-14361: --- Issue Type: New Feature (was: Improvement) > Support EXCLUDE clause in Window function framing > ---

[jira] [Created] (SPARK-15681) Allow case-insensitiveness in sc.setLogLevel and support sc.resetLogLevel

Xin Wu created SPARK-15681: -- Summary: Allow case-insensitiveness in sc.setLogLevel and support sc.resetLogLevel Key: SPARK-15681 URL: https://issues.apache.org/jira/browse/SPARK-15681 Project: Spark

[jira] [Commented] (SPARK-10505) windowed form of count ( star ) fails with No handler for udf class

[ https://issues.apache.org/jira/browse/SPARK-10505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14940675#comment-14940675 ] Xin Wu commented on SPARK-10505: This error is triggered in function HiveFunctionRegistry

[jira] [Commented] (SPARK-10505) windowed form of count ( star ) fails with No handler for udf class

[ https://issues.apache.org/jira/browse/SPARK-10505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14945791#comment-14945791 ] Xin Wu commented on SPARK-10505: [~the6campbells], I am trying to recreate the issue with

[jira] [Commented] (SPARK-10505) windowed form of count ( star ) fails with No handler for udf class

[ https://issues.apache.org/jira/browse/SPARK-10505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14953580#comment-14953580 ] Xin Wu commented on SPARK-10505: [~the6campbells] I also tried this on Spark 1.5.1 and it

[jira] [Commented] (SPARK-10747) add support for window specification to include how NULLS are ordered

[ https://issues.apache.org/jira/browse/SPARK-10747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14956094#comment-14956094 ] Xin Wu commented on SPARK-10747: I ran this query on the released Hive 1.2.1 version, and

[jira] [Commented] (SPARK-11229) NPE in JoinedRow.isNullAt when spark.shuffle.memoryFraction=0

[ https://issues.apache.org/jira/browse/SPARK-11229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14967694#comment-14967694 ] Xin Wu commented on SPARK-11229: Here is what I tried: 1. start spark-shell with command

[jira] [Commented] (SPARK-11229) NPE in JoinedRow.isNullAt when spark.shuffle.memoryFraction=0

[ https://issues.apache.org/jira/browse/SPARK-11229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14967811#comment-14967811 ] Xin Wu commented on SPARK-11229: I am able to recreate it on 1.5.1 {code} scala> data02.

[jira] [Commented] (SPARK-11229) NPE in JoinedRow.isNullAt when spark.shuffle.memoryFraction=0

[ https://issues.apache.org/jira/browse/SPARK-11229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14968089#comment-14968089 ] Xin Wu commented on SPARK-11229: [~marmbrus] Please take a look if this is safe to close

[jira] [Commented] (SPARK-11246) [1.5] Table cache for Parquet broken in 1.5

[ https://issues.apache.org/jira/browse/SPARK-11246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14968293#comment-14968293 ] Xin Wu commented on SPARK-11246: I am able to recreate this issue on 1.6.0 with hive 1.2.

[jira] [Commented] (SPARK-11246) [1.5] Table cache for Parquet broken in 1.5

[ https://issues.apache.org/jira/browse/SPARK-11246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970071#comment-14970071 ] Xin Wu commented on SPARK-11246: Root cause found, fix is being tested. will submit PR sh

[jira] [Commented] (SPARK-11246) [1.5] Table cache for Parquet broken in 1.5

[ https://issues.apache.org/jira/browse/SPARK-11246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14980652#comment-14980652 ] Xin Wu commented on SPARK-11246: [~yhuai] Thank you! > [1.5] Table cache for Parquet bro

[jira] [Commented] (SPARK-10046) Hive warehouse dir not set in current directory when not providing hive-site.xml

[ https://issues.apache.org/jira/browse/SPARK-10046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14981282#comment-14981282 ] Xin Wu commented on SPARK-10046: The default HiveConf without hite-site.xml has "hive.me

[jira] [Commented] (SPARK-10046) Hive warehouse dir not set in current directory when not providing hive-site.xml

[ https://issues.apache.org/jira/browse/SPARK-10046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14981331#comment-14981331 ] Xin Wu commented on SPARK-10046: Ok. Let me update the doc then. > Hive warehouse dir no

[jira] [Issue Comment Deleted] (SPARK-10046) Hive warehouse dir not set in current directory when not providing hive-site.xml

[ https://issues.apache.org/jira/browse/SPARK-10046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Wu updated SPARK-10046: --- Comment: was deleted (was: pull request is created: https://github.com/apache/spark/pull/9365) > Hive wareho

[jira] [Commented] (SPARK-10046) Hive warehouse dir not set in current directory when not providing hive-site.xml

[ https://issues.apache.org/jira/browse/SPARK-10046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14981439#comment-14981439 ] Xin Wu commented on SPARK-10046: pull request is created: https://github.com/apache/spark

[jira] [Commented] (SPARK-9465) Could not read parquet table after recreating it with the same table name

[ https://issues.apache.org/jira/browse/SPARK-9465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14990890#comment-14990890 ] Xin Wu commented on SPARK-9465: --- I tried on both 1.5.1 and 1.6.0, I can not recreate the iss

[jira] [Commented] (SPARK-9465) Could not read parquet table after recreating it with the same table name

[ https://issues.apache.org/jira/browse/SPARK-9465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14990909#comment-14990909 ] Xin Wu commented on SPARK-9465: --- I can not recreate the issue on 1.5.1 or 1.6.0.. {code} sc

[jira] [Issue Comment Deleted] (SPARK-9465) Could not read parquet table after recreating it with the same table name

[ https://issues.apache.org/jira/browse/SPARK-9465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Wu updated SPARK-9465: -- Comment: was deleted (was: I can not recreate the issue on 1.5.1 or 1.6.0.. {code} scala> sqlContext.sql("creat

[jira] [Commented] (SPARK-11522) input_file_name() returns "" for external tables

[ https://issues.apache.org/jira/browse/SPARK-11522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992183#comment-14992183 ] Xin Wu commented on SPARK-11522: I can recreate this issue with simpler case on 1.6.0 cod

[jira] [Commented] (SPARK-11522) input_file_name() returns "" for external tables

[ https://issues.apache.org/jira/browse/SPARK-11522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993212#comment-14993212 ] Xin Wu commented on SPARK-11522: Actually this seems not related to whether the table is

[jira] [Commented] (SPARK-11522) input_file_name() returns "" for external tables

[ https://issues.apache.org/jira/browse/SPARK-11522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14994763#comment-14994763 ] Xin Wu commented on SPARK-11522: running full test against the fix now. will submit the P

[jira] [Updated] (SPARK-11522) input_file_name() returns "" for external tables

[ https://issues.apache.org/jira/browse/SPARK-11522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Wu updated SPARK-11522: --- Description: Given an external table definition where the data consists of many CSV files, {{input_file_name

[jira] [Commented] (SPARK-9465) Could not read parquet table after recreating it with the same table name

[ https://issues.apache.org/jira/browse/SPARK-9465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14995771#comment-14995771 ] Xin Wu commented on SPARK-9465: --- [~stanzhai] Would you be able to recreate on the later rele

[jira] [Commented] (SPARK-11356) Option to refresh information about parquet partitions

[ https://issues.apache.org/jira/browse/SPARK-11356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14997651#comment-14997651 ] Xin Wu commented on SPARK-11356: [~maver1ck] If I only execute {code}sqlContext.sql("sele

[jira] [Closed] (SPARK-15463) Support for creating a dataframe from CSV in Dataset[String]

[ https://issues.apache.org/jira/browse/SPARK-15463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Wu closed SPARK-15463. -- Resolution: Later > Support for creating a dataframe from CSV in Dataset[String] >

[jira] [Created] (SPARK-19539) CREATE TEMPORARY TABLE need to avoid existing temp view

Xin Wu created SPARK-19539: -- Summary: CREATE TEMPORARY TABLE need to avoid existing temp view Key: SPARK-19539 URL: https://issues.apache.org/jira/browse/SPARK-19539 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-19539) CREATE TEMPORARY TABLE needs to avoid existing temp view

[ https://issues.apache.org/jira/browse/SPARK-19539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Wu updated SPARK-19539: --- Summary: CREATE TEMPORARY TABLE needs to avoid existing temp view (was: CREATE TEMPORARY TABLE need to avoid

[jira] [Created] (SPARK-20256) Fail to start SparkContext/SparkSession with Hive support enabled when user does not have read/write privilege to Hive metastore warehouse dir

Xin Wu created SPARK-20256: -- Summary: Fail to start SparkContext/SparkSession with Hive support enabled when user does not have read/write privilege to Hive metastore warehouse dir Key: SPARK-20256 URL: https://issues.a

[jira] [Commented] (SPARK-20256) Fail to start SparkContext/SparkSession with Hive support enabled when user does not have read/write privilege to Hive metastore warehouse dir

[ https://issues.apache.org/jira/browse/SPARK-20256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15961193#comment-15961193 ] Xin Wu commented on SPARK-20256: I am working on a fix and creating simulated test cases

[jira] [Updated] (SPARK-20256) Fail to start SparkContext/SparkSession with Hive support enabled when user does not have read/write privilege to Hive metastore warehouse dir

[ https://issues.apache.org/jira/browse/SPARK-20256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Wu updated SPARK-20256: --- Description: In a cluster setup with production Hive running, when the user wants to run spark-shell using t

[jira] [Commented] (SPARK-20256) Fail to start SparkContext/SparkSession with Hive support enabled when user does not have read/write privilege to Hive metastore warehouse dir

[ https://issues.apache.org/jira/browse/SPARK-20256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15964969#comment-15964969 ] Xin Wu commented on SPARK-20256: Yes. I am working on it. My proposal is to revert the S

[jira] [Commented] (SPARK-18727) Support schema evolution as new files are inserted into table

[ https://issues.apache.org/jira/browse/SPARK-18727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987642#comment-15987642 ] Xin Wu commented on SPARK-18727: FYI. I have https://github.com/apache/spark/pull/16626 f

[jira] [Commented] (SPARK-18727) Support schema evolution as new files are inserted into table

[ https://issues.apache.org/jira/browse/SPARK-18727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987830#comment-15987830 ] Xin Wu commented on SPARK-18727: [~simeons] You are right.. My PR does not include the fe

[jira] [Commented] (SPARK-18727) Support schema evolution as new files are inserted into table

[ https://issues.apache.org/jira/browse/SPARK-18727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987874#comment-15987874 ] Xin Wu commented on SPARK-18727: [~ekhliang] First of all, i am not sure whether it is wi

[jira] [Commented] (SPARK-18727) Support schema evolution as new files are inserted into table

[ https://issues.apache.org/jira/browse/SPARK-18727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987939#comment-15987939 ] Xin Wu commented on SPARK-18727: [~ekhliang] I see. I will try to support ALTER TABLE SCH

[jira] [Commented] (SPARK-18727) Support schema evolution as new files are inserted into table

[ https://issues.apache.org/jira/browse/SPARK-18727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987976#comment-15987976 ] Xin Wu commented on SPARK-18727: Thanks! > Support schema evolution as new files are in

[jira] [Comment Edited] (SPARK-16605) Spark2.0 cannot "select" data from a table stored as an orc file which has been created by hive while hive or spark1.6 supports

[ https://issues.apache.org/jira/browse/SPARK-16605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383082#comment-15383082 ] Xin Wu edited comment on SPARK-16605 at 7/18/16 9:17 PM: - The cur

[jira] [Commented] (SPARK-16605) Spark2.0 cannot "select" data from a table stored as an orc file which has been created by hive while hive or spark1.6 supports

[ https://issues.apache.org/jira/browse/SPARK-16605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383082#comment-15383082 ] Xin Wu commented on SPARK-16605: The current issue for dealing with ORC data inserted by

[jira] [Commented] (SPARK-9761) Inconsistent metadata handling with ALTER TABLE

[ https://issues.apache.org/jira/browse/SPARK-9761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408405#comment-15408405 ] Xin Wu commented on SPARK-9761: --- [~drwinters] Spark 2.0 has support DDL commands, which mean

[jira] [Created] (SPARK-16924) DataStreamReader can not support option("inferSchema", true/false) for csv and json file source

Xin Wu created SPARK-16924: -- Summary: DataStreamReader can not support option("inferSchema", true/false) for csv and json file source Key: SPARK-16924 URL: https://issues.apache.org/jira/browse/SPARK-16924 P

[jira] [Updated] (SPARK-16924) DataStreamReader can not support option("inferSchema", true/false) for csv and json file source

[ https://issues.apache.org/jira/browse/SPARK-16924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Wu updated SPARK-16924: --- Issue Type: Improvement (was: Bug) > DataStreamReader can not support option("inferSchema", true/false) for

[jira] [Commented] (SPARK-10747) add support for window specification to include how NULLS are ordered

[ https://issues.apache.org/jira/browse/SPARK-10747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425547#comment-15425547 ] Xin Wu commented on SPARK-10747: [~hvanhovell] Yes. Since we have native parser now, we c

[jira] [Updated] (SPARK-15681) Allow case-insensitiveness in sc.setLogLevel

[ https://issues.apache.org/jira/browse/SPARK-15681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Wu updated SPARK-15681: --- Summary: Allow case-insensitiveness in sc.setLogLevel (was: Allow case-insensitiveness in sc.setLogLevel and

[jira] [Updated] (SPARK-15681) Allow case-insensitiveness in sc.setLogLevel

[ https://issues.apache.org/jira/browse/SPARK-15681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Wu updated SPARK-15681: --- Description: Currently SparkContext API setLogLevel(level: String) can not handle lower case or mixed case i

[jira] [Commented] (SPARK-14959) ​Problem Reading partitioned ORC or Parquet files

[ https://issues.apache.org/jira/browse/SPARK-14959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15311409#comment-15311409 ] Xin Wu commented on SPARK-14959: I can recreate the problem with hdfs location. and I hav

[jira] [Commented] (SPARK-15710) Exception with WHERE clause in SQL for non-default Hive database

[ https://issues.apache.org/jira/browse/SPARK-15710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15312836#comment-15312836 ] Xin Wu commented on SPARK-15710: Hmm. I can not recreate it on latest master branch. Here

[jira] [Commented] (SPARK-15710) Exception with WHERE clause in SQL for non-default Hive database

[ https://issues.apache.org/jira/browse/SPARK-15710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15312851#comment-15312851 ] Xin Wu commented on SPARK-15710: I see. pyspark does not work. > Exception with WHERE cl

[jira] [Commented] (SPARK-15710) Exception with WHERE clause in SQL for non-default Hive database

[ https://issues.apache.org/jira/browse/SPARK-15710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15313193#comment-15313193 ] Xin Wu commented on SPARK-15710: hmm.. after another rebase of the master. it seems that

[jira] [Commented] (SPARK-15705) Spark won't read ORC schema from metastore for partitioned tables

[ https://issues.apache.org/jira/browse/SPARK-15705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15313257#comment-15313257 ] Xin Wu commented on SPARK-15705: I can recreate it now. and will look into it. > Spark

[jira] [Comment Edited] (SPARK-15705) Spark won't read ORC schema from metastore for partitioned tables

[ https://issues.apache.org/jira/browse/SPARK-15705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15313257#comment-15313257 ] Xin Wu edited comment on SPARK-15705 at 6/2/16 11:15 PM: - I can r

[jira] [Created] (SPARK-15970) WARNing message related to persisting table to Hive Megastore while Spark SQL is running in-memory catalog mode

Xin Wu created SPARK-15970: -- Summary: WARNing message related to persisting table to Hive Megastore while Spark SQL is running in-memory catalog mode Key: SPARK-15970 URL: https://issues.apache.org/jira/browse/SPARK-1597

[jira] [Commented] (SPARK-14927) DataFrame. saveAsTable creates RDD partitions but not Hive partitions

[ https://issues.apache.org/jira/browse/SPARK-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15438457#comment-15438457 ] Xin Wu commented on SPARK-14927: [~smilegator] Do you think what you are working on regar

[jira] [Comment Edited] (SPARK-14927) DataFrame. saveAsTable creates RDD partitions but not Hive partitions

[ https://issues.apache.org/jira/browse/SPARK-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15438457#comment-15438457 ] Xin Wu edited comment on SPARK-14927 at 8/26/16 4:46 AM: - [~smile

[jira] [Updated] (SPARK-10747) add support for window specification to include how NULLS are ordered

[ https://issues.apache.org/jira/browse/SPARK-10747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Wu updated SPARK-10747: --- Issue Type: New Feature (was: Improvement) > add support for window specification to include how NULLS are o

[jira] [Commented] (SPARK-10747) add support for window specification to include how NULLS are ordered

[ https://issues.apache.org/jira/browse/SPARK-10747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15440656#comment-15440656 ] Xin Wu commented on SPARK-10747: This JIRA may be changed to support NULLS FIRST|LAST fea

[jira] [Updated] (SPARK-10747) add support for NULLS FIRST|LAST in ORDER BY clause

[ https://issues.apache.org/jira/browse/SPARK-10747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Wu updated SPARK-10747: --- Summary: add support for NULLS FIRST|LAST in ORDER BY clause (was: add support for window specification to i

[jira] [Created] (SPARK-17551) support null ordering for DataFrame API

Xin Wu created SPARK-17551: -- Summary: support null ordering for DataFrame API Key: SPARK-17551 URL: https://issues.apache.org/jira/browse/SPARK-17551 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-11657) Bad Dataframe data read from parquet

[ https://issues.apache.org/jira/browse/SPARK-11657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15000607#comment-15000607 ] Xin Wu commented on SPARK-11657: I tried this sample data as local file mode. and it seem

[jira] [Issue Comment Deleted] (SPARK-11657) Bad Dataframe data read from parquet

[ https://issues.apache.org/jira/browse/SPARK-11657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Wu updated SPARK-11657: --- Comment: was deleted (was: I tried this sample data as local file mode. and it seems working to me. Have you

[jira] [Commented] (SPARK-10848) Applied JSON Schema Works for json RDD but not when loading json file

[ https://issues.apache.org/jira/browse/SPARK-10848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001404#comment-15001404 ] Xin Wu commented on SPARK-10848: I can recreate this as describe. Looking into the code.

[jira] [Commented] (SPARK-10848) Applied JSON Schema Works for json RDD but not when loading json file

[ https://issues.apache.org/jira/browse/SPARK-10848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001667#comment-15001667 ] Xin Wu commented on SPARK-10848: Debugging through the code and found that in ResolvedDat

[jira] [Commented] (SPARK-10848) Applied JSON Schema Works for json RDD but not when loading json file

[ https://issues.apache.org/jira/browse/SPARK-10848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001685#comment-15001685 ] Xin Wu commented on SPARK-10848: One case I found where there is inconsistency between us

[jira] [Commented] (SPARK-10848) Applied JSON Schema Works for json RDD but not when loading json file

[ https://issues.apache.org/jira/browse/SPARK-10848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15003178#comment-15003178 ] Xin Wu commented on SPARK-10848: With rdd created in the application, as following: {code

[jira] [Commented] (SPARK-11683) DataFrameWriter.insertInto() fails on non-default schema

[ https://issues.apache.org/jira/browse/SPARK-11683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15003251#comment-15003251 ] Xin Wu commented on SPARK-11683: It seems like in 1.6.0, this issue is resolved: {code} s

[jira] [Commented] (SPARK-10673) spark.sql.hive.verifyPartitionPath Attempts to Verify Unregistered Partitions

[ https://issues.apache.org/jira/browse/SPARK-10673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15004240#comment-15004240 ] Xin Wu commented on SPARK-10673: I will look into this one. > spark.sql.hive.verifyPart

[jira] [Commented] (SPARK-10673) spark.sql.hive.verifyPartitionPath Attempts to Verify Unregistered Partitions

[ https://issues.apache.org/jira/browse/SPARK-10673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005249#comment-15005249 ] Xin Wu commented on SPARK-10673: if the default is false, {code} if (!sc.conf.verifyParti

[jira] [Comment Edited] (SPARK-10673) spark.sql.hive.verifyPartitionPath Attempts to Verify Unregistered Partitions

[ https://issues.apache.org/jira/browse/SPARK-10673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005249#comment-15005249 ] Xin Wu edited comment on SPARK-10673 at 11/14/15 8:19 AM: -- if th

[jira] [Commented] (SPARK-10673) spark.sql.hive.verifyPartitionPath Attempts to Verify Unregistered Partitions

[ https://issues.apache.org/jira/browse/SPARK-10673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005577#comment-15005577 ] Xin Wu commented on SPARK-10673: The fix is being tested.. will submit PR shortly. > spa

[jira] [Issue Comment Deleted] (SPARK-10673) spark.sql.hive.verifyPartitionPath Attempts to Verify Unregistered Partitions

[ https://issues.apache.org/jira/browse/SPARK-10673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Wu updated SPARK-10673: --- Comment: was deleted (was: The fix is being tested.. will submit PR shortly.) > spark.sql.hive.verifyPartiti

[jira] [Issue Comment Deleted] (SPARK-10673) spark.sql.hive.verifyPartitionPath Attempts to Verify Unregistered Partitions

[ https://issues.apache.org/jira/browse/SPARK-10673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Wu updated SPARK-10673: --- Comment: was deleted (was: if the default is false, {code} if (!sc.conf.verifyPartitionPath) { partitionT

  1   2   >