[jira] [Created] (SPARK-17838) Strict type checking for arguments with a better messages across APIs.

Hyukjin Kwon created SPARK-17838: Summary: Strict type checking for arguments with a better messages across APIs. Key: SPARK-17838 URL: https://issues.apache.org/jira/browse/SPARK-17838 Project: Spark

[jira] [Commented] (SPARK-10501) support UUID as an atomic type

[ https://issues.apache.org/jira/browse/SPARK-10501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15559405#comment-15559405 ] Hyukjin Kwon commented on SPARK-10501: -- Ah, it was type not the function. I just rus

[jira] [Resolved] (SPARK-8115) Remove TestData

[ https://issues.apache.org/jira/browse/SPARK-8115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-8115. Resolution: Later It sounds like this is not being fixed in the short term. Please reopen it if it is still

[jira] [Commented] (SPARK-10501) support UUID as an atomic type

[ https://issues.apache.org/jira/browse/SPARK-10501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15559271#comment-15559271 ] Russell Spitzer commented on SPARK-10501: - It's not that we need it as a unique i

[jira] [Commented] (SPARK-11758) Missing Index column while creating a DataFrame from Pandas

[ https://issues.apache.org/jira/browse/SPARK-11758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15559240#comment-15559240 ] holdenk commented on SPARK-11758: - I believe dropping the index field is intentional (but

[jira] [Updated] (SPARK-14420) keepLastCheckpoint Param for Python LDA with EM

[ https://issues.apache.org/jira/browse/SPARK-14420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk updated SPARK-14420: Fix Version/s: 2.0.0 > keepLastCheckpoint Param for Python LDA with EM > --

[jira] [Closed] (SPARK-14420) keepLastCheckpoint Param for Python LDA with EM

[ https://issues.apache.org/jira/browse/SPARK-14420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk closed SPARK-14420. --- Resolution: Duplicate > keepLastCheckpoint Param for Python LDA with EM > ---

[jira] [Closed] (SPARK-7659) Sort by attributes that are not present in the SELECT clause when there is windowfunction analysis error

[ https://issues.apache.org/jira/browse/SPARK-7659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li closed SPARK-7659. -- Resolution: Not A Problem > Sort by attributes that are not present in the SELECT clause when there is > window

[jira] [Commented] (SPARK-7659) Sort by attributes that are not present in the SELECT clause when there is windowfunction analysis error

[ https://issues.apache.org/jira/browse/SPARK-7659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15559117#comment-15559117 ] Xiao Li commented on SPARK-7659: This should have been fixed in 2.0. Please reopen it if y

[jira] [Closed] (SPARK-11479) add kmeans example for Dataset

[ https://issues.apache.org/jira/browse/SPARK-11479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li closed SPARK-11479. --- Resolution: Won't Fix > add kmeans example for Dataset > -- > > K

[jira] [Commented] (SPARK-11479) add kmeans example for Dataset

[ https://issues.apache.org/jira/browse/SPARK-11479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15559114#comment-15559114 ] Xiao Li commented on SPARK-11479: - Based on the PR, we should close it now. Please reopen

[jira] [Commented] (SPARK-10318) Getting issue in spark connectivity with cassandra

[ https://issues.apache.org/jira/browse/SPARK-10318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15559103#comment-15559103 ] Xiao Li commented on SPARK-10318: - Yeah. Will follow your guideline in the future. Thanks

[jira] [Closed] (SPARK-6413) For data source tables, we should provide better output for DESCRIBE FORMATTED

[ https://issues.apache.org/jira/browse/SPARK-6413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li closed SPARK-6413. -- Resolution: Not A Problem > For data source tables, we should provide better output for DESCRIBE FORMATTED > ---

[jira] [Commented] (SPARK-6413) For data source tables, we should provide better output for DESCRIBE FORMATTED

[ https://issues.apache.org/jira/browse/SPARK-6413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15559074#comment-15559074 ] Xiao Li commented on SPARK-6413: This has been well supported since Spark 2.0. Thus, close

[jira] [Commented] (SPARK-11523) spark_partition_id() considered invalid function

[ https://issues.apache.org/jira/browse/SPARK-11523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15559072#comment-15559072 ] Xiao Li commented on SPARK-11523: - Native views are supported since 2.0. Thus, this JIRA

[jira] [Closed] (SPARK-11523) spark_partition_id() considered invalid function

[ https://issues.apache.org/jira/browse/SPARK-11523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li closed SPARK-11523. --- Resolution: Not A Problem > spark_partition_id() considered invalid function > --

[jira] [Commented] (SPARK-11087) spark.sql.orc.filterPushdown does not work, No ORC pushdown predicate

[ https://issues.apache.org/jira/browse/SPARK-11087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15559068#comment-15559068 ] Xiao Li commented on SPARK-11087: - Can you retry it using the latest master/2.0.1 branch?

[jira] [Closed] (SPARK-9359) Support IntervalType for Parquet

[ https://issues.apache.org/jira/browse/SPARK-9359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li closed SPARK-9359. -- Assignee: (was: Liang-Chi Hsieh) > Support IntervalType for Parquet > > >

[jira] [Resolved] (SPARK-9359) Support IntervalType for Parquet

[ https://issues.apache.org/jira/browse/SPARK-9359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-9359. Resolution: Duplicate > Support IntervalType for Parquet > > >

[jira] [Commented] (SPARK-9205) org.apache.spark.sql.hive.HiveSparkSubmitSuite failing for Scala 2.11

[ https://issues.apache.org/jira/browse/SPARK-9205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15559052#comment-15559052 ] Xiao Li commented on SPARK-9205: This is not an issue, right? Since this JIRA is stale, le

[jira] [Closed] (SPARK-9205) org.apache.spark.sql.hive.HiveSparkSubmitSuite failing for Scala 2.11

[ https://issues.apache.org/jira/browse/SPARK-9205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li closed SPARK-9205. -- Resolution: Cannot Reproduce > org.apache.spark.sql.hive.HiveSparkSubmitSuite failing for Scala 2.11 > -

[jira] [Commented] (SPARK-17626) TPC-DS performance improvements using star-schema heuristics

[ https://issues.apache.org/jira/browse/SPARK-17626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15559044#comment-15559044 ] Ron Hu commented on SPARK-17626: In the CBO design spec we posted in https://issues.apac

[jira] [Closed] (SPARK-10101) Spark JDBC writer mapping String to TEXT or VARCHAR

[ https://issues.apache.org/jira/browse/SPARK-10101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li closed SPARK-10101. --- Resolution: Not A Problem > Spark JDBC writer mapping String to TEXT or VARCHAR > ---

[jira] [Commented] (SPARK-10101) Spark JDBC writer mapping String to TEXT or VARCHAR

[ https://issues.apache.org/jira/browse/SPARK-10101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558901#comment-15558901 ] Xiao Li commented on SPARK-10101: - This has been resolved in the master. If you still hit

[jira] [Closed] (SPARK-9265) Dataframe.limit joined with another dataframe can be non-deterministic

[ https://issues.apache.org/jira/browse/SPARK-9265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li closed SPARK-9265. -- Resolution: Not A Problem > Dataframe.limit joined with another dataframe can be non-deterministic > ---

[jira] [Commented] (SPARK-9265) Dataframe.limit joined with another dataframe can be non-deterministic

[ https://issues.apache.org/jira/browse/SPARK-9265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558875#comment-15558875 ] Xiao Li commented on SPARK-9265: This has been resolved since our Optimizer push down `Lim

[jira] [Commented] (SPARK-17626) TPC-DS performance improvements using star-schema heuristics

[ https://issues.apache.org/jira/browse/SPARK-17626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558762#comment-15558762 ] Ioana Delaney commented on SPARK-17626: --- [~mikewzh] Thank you. Yes, having informat

[jira] [Commented] (SPARK-10496) Efficient DataFrame cumulative sum

[ https://issues.apache.org/jira/browse/SPARK-10496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558724#comment-15558724 ] Reynold Xin commented on SPARK-10496: - I think there are two separate issues here: 1

[jira] [Closed] (SPARK-5818) unable to use "add jar" in hql

[ https://issues.apache.org/jira/browse/SPARK-5818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li closed SPARK-5818. -- Resolution: Not A Problem This has been supported. Please try the latest branch. Thanks! > unable to use "add j

[jira] [Commented] (SPARK-7097) Partitioned tables should only consider referred partitions in query during size estimation for checking against autoBroadcastJoinThreshold

[ https://issues.apache.org/jira/browse/SPARK-7097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558718#comment-15558718 ] Xiao Li commented on SPARK-7097: This will be resolved in the ongoing CBO work. Thus, clos

[jira] [Closed] (SPARK-7097) Partitioned tables should only consider referred partitions in query during size estimation for checking against autoBroadcastJoinThreshold

[ https://issues.apache.org/jira/browse/SPARK-7097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li closed SPARK-7097. -- Resolution: Won't Fix > Partitioned tables should only consider referred partitions in query during > size esti

[jira] [Resolved] (SPARK-11062) Thrift server does not support operationLog

[ https://issues.apache.org/jira/browse/SPARK-11062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-11062. - Resolution: Duplicate > Thrift server does not support operationLog > ---

[jira] [Commented] (SPARK-11055) Use mixing hash-based and sort-based aggregation in TungstenAggregationIterator

[ https://issues.apache.org/jira/browse/SPARK-11055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558706#comment-15558706 ] Xiao Li commented on SPARK-11055: - Based on the PR, Davies did a similar work. [SPARK-114

[jira] [Closed] (SPARK-11055) Use mixing hash-based and sort-based aggregation in TungstenAggregationIterator

[ https://issues.apache.org/jira/browse/SPARK-11055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li closed SPARK-11055. --- Resolution: Duplicate > Use mixing hash-based and sort-based aggregation in > TungstenAggregationIterator >

[jira] [Commented] (SPARK-10794) Spark-SQL- select query on table column with binary Data Type displays error message- java.lang.ClassCastException: java.lang.String cannot be cast to [B

[ https://issues.apache.org/jira/browse/SPARK-10794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558631#comment-15558631 ] Xiao Li commented on SPARK-10794: - The related parts are changed a lot. Could you retry i

[jira] [Closed] (SPARK-10933) Spark SQL Joins should have option to fail query when row multiplication is encountered

[ https://issues.apache.org/jira/browse/SPARK-10933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li closed SPARK-10933. --- Resolution: Won't Fix > Spark SQL Joins should have option to fail query when row multiplication is > encoun

[jira] [Commented] (SPARK-10933) Spark SQL Joins should have option to fail query when row multiplication is encountered

[ https://issues.apache.org/jira/browse/SPARK-10933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558625#comment-15558625 ] Xiao Li commented on SPARK-10933: - Now, we have a conf `spark.sql.crossJoin.enabled`. L

[jira] [Commented] (SPARK-10805) JSON Data Frame does not return correct string lengths

[ https://issues.apache.org/jira/browse/SPARK-10805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558595#comment-15558595 ] Xiao Li commented on SPARK-10805: - This is pretty expensive to find the max length for ea

[jira] [Resolved] (SPARK-10805) JSON Data Frame does not return correct string lengths

[ https://issues.apache.org/jira/browse/SPARK-10805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-10805. - Resolution: Won't Fix > JSON Data Frame does not return correct string lengths >

[jira] [Comment Edited] (SPARK-10972) UDFs in SQL joins

[ https://issues.apache.org/jira/browse/SPARK-10972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558581#comment-15558581 ] Xiao Li edited comment on SPARK-10972 at 10/8/16 7:34 PM: -- Also

[jira] [Commented] (SPARK-10972) UDFs in SQL joins

[ https://issues.apache.org/jira/browse/SPARK-10972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558581#comment-15558581 ] Xiao Li commented on SPARK-10972: - Also try to use the SQL interface? > UDFs in SQL join

[jira] [Commented] (SPARK-10972) UDFs in SQL joins

[ https://issues.apache.org/jira/browse/SPARK-10972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558577#comment-15558577 ] Xiao Li commented on SPARK-10972: - There is a workaround to fix it. You can specify the f

[jira] [Commented] (SPARK-17147) Spark Streaming Kafka 0.10 Consumer Can't Handle Non-consecutive Offsets

[ https://issues.apache.org/jira/browse/SPARK-17147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558572#comment-15558572 ] Cody Koeninger commented on SPARK-17147: I talked with Sean in person about this,

[jira] [Updated] (SPARK-10860) Bivariate Statistics: Chi-Squared independence test

[ https://issues.apache.org/jira/browse/SPARK-10860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-10860: Assignee: (was: Jihong MA) > Bivariate Statistics: Chi-Squared independence test >

[jira] [Commented] (SPARK-4960) Interceptor pattern in receivers

[ https://issues.apache.org/jira/browse/SPARK-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558557#comment-15558557 ] Cody Koeninger commented on SPARK-4960: --- Is this idea pretty much dead at this point

[jira] [Updated] (SPARK-10646) Bivariate Statistics: Pearson's Chi-Squared goodness of fit test

[ https://issues.apache.org/jira/browse/SPARK-10646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-10646: Assignee: (was: Jihong MA) > Bivariate Statistics: Pearson's Chi-Squared goodness of fit test > ---

[jira] [Updated] (SPARK-10860) Bivariate Statistics: Chi-Squared independence test

[ https://issues.apache.org/jira/browse/SPARK-10860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-10860: Component/s: (was: SQL) > Bivariate Statistics: Chi-Squared independence test > ---

[jira] [Resolved] (SPARK-3146) Improve the flexibility of Spark Streaming Kafka API to offer user the ability to process message before storing into BM

[ https://issues.apache.org/jira/browse/SPARK-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cody Koeninger resolved SPARK-3146. --- Resolution: Fixed Fix Version/s: 1.3.0 > Improve the flexibility of Spark Streaming Kaf

[jira] [Commented] (SPARK-3146) Improve the flexibility of Spark Streaming Kafka API to offer user the ability to process message before storing into BM

[ https://issues.apache.org/jira/browse/SPARK-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558550#comment-15558550 ] Cody Koeninger commented on SPARK-3146: --- SPARK-4964 / the direct stream added a mess

[jira] [Resolved] (SPARK-6649) DataFrame created through SQLContext.jdbc() failed if columns table must be quoted

[ https://issues.apache.org/jira/browse/SPARK-6649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-6649. Resolution: Fixed > DataFrame created through SQLContext.jdbc() failed if columns table must be > quoted >

[jira] [Updated] (SPARK-14212) Add configuration element for --packages option

[ https://issues.apache.org/jira/browse/SPARK-14212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk updated SPARK-14212: Labels: config starter (was: config fun happy pants spark-shell) > Add configuration element for --package

[jira] [Updated] (SPARK-14212) Add configuration element for --packages option

[ https://issues.apache.org/jira/browse/SPARK-14212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk updated SPARK-14212: Priority: Trivial (was: Major) > Add configuration element for --packages option > ---

[jira] [Updated] (SPARK-14212) Add configuration element for --packages option

[ https://issues.apache.org/jira/browse/SPARK-14212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk updated SPARK-14212: Component/s: (was: Spark Shell) (was: Spark Core) Documentation >

[jira] [Commented] (SPARK-14212) Add configuration element for --packages option

[ https://issues.apache.org/jira/browse/SPARK-14212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558540#comment-15558540 ] holdenk commented on SPARK-14212: - So I think this would be a good option to document for

[jira] [Closed] (SPARK-14017) dataframe.dtypes -> pyspark.sql.types aliases

[ https://issues.apache.org/jira/browse/SPARK-14017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk closed SPARK-14017. --- Resolution: Won't Fix Thanks for bringing this issue up - I don't think we necessarily want to add these ali

[jira] [Commented] (SPARK-10804) "LOCAL" in LOAD DATA LOCAL INPATH means "remote"

[ https://issues.apache.org/jira/browse/SPARK-10804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558532#comment-15558532 ] Xiao Li commented on SPARK-10804: - In Spark 2.0, we rewrote the whole part, especially th

[jira] [Closed] (SPARK-10804) "LOCAL" in LOAD DATA LOCAL INPATH means "remote"

[ https://issues.apache.org/jira/browse/SPARK-10804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li closed SPARK-10804. --- Resolution: Not A Problem > "LOCAL" in LOAD DATA LOCAL INPATH means "remote" > --

[jira] [Updated] (SPARK-17837) Disaster recovery of offsets from WAL

[ https://issues.apache.org/jira/browse/SPARK-17837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cody Koeninger updated SPARK-17837: --- Summary: Disaster recovery of offsets from WAL (was: Disaster recover of offsets from WAL)

[jira] [Created] (SPARK-17837) Disaster recover of offsets from WAL

Cody Koeninger created SPARK-17837: -- Summary: Disaster recover of offsets from WAL Key: SPARK-17837 URL: https://issues.apache.org/jira/browse/SPARK-17837 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-17815) Report committed offsets

[ https://issues.apache.org/jira/browse/SPARK-17815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558528#comment-15558528 ] Cody Koeninger commented on SPARK-17815: So if you start committing offsets to ka

[jira] [Commented] (SPARK-10427) Spark-sql -f or -e will output some

[ https://issues.apache.org/jira/browse/SPARK-10427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558525#comment-15558525 ] Xiao Li commented on SPARK-10427: - This is not an issue since 2.0. Thanks! > Spark-sql -

[jira] [Closed] (SPARK-10427) Spark-sql -f or -e will output some

[ https://issues.apache.org/jira/browse/SPARK-10427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li closed SPARK-10427. --- Resolution: Not A Problem > Spark-sql -f or -e will output some > --- > >

[jira] [Commented] (SPARK-9442) java.lang.ArithmeticException: / by zero when reading Parquet

[ https://issues.apache.org/jira/browse/SPARK-9442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558512#comment-15558512 ] Xiao Li commented on SPARK-9442: Is it still a problem in the latest branch? > java.lang.

[jira] [Closed] (SPARK-10703) Physical filter operators should replace the general AND/OR/equality/etc with a special version that treats null as false

[ https://issues.apache.org/jira/browse/SPARK-10703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li closed SPARK-10703. --- Resolution: Not A Problem > Physical filter operators should replace the general AND/OR/equality/etc with >

[jira] [Commented] (SPARK-10703) Physical filter operators should replace the general AND/OR/equality/etc with a special version that treats null as false

[ https://issues.apache.org/jira/browse/SPARK-10703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558509#comment-15558509 ] Xiao Li commented on SPARK-10703: - The problem has been resolved, I think. Try the latest

[jira] [Commented] (SPARK-17812) More granular control of starting offsets

[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558506#comment-15558506 ] Cody Koeninger commented on SPARK-17812: So I'm willing to do this work, mostly b

[jira] [Commented] (SPARK-10545) HiveMetastoreTypes.toMetastoreType should handle interval type

[ https://issues.apache.org/jira/browse/SPARK-10545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558489#comment-15558489 ] Xiao Li commented on SPARK-10545: - Both Hive and Spark do not support INTERVAL as a colum

[jira] [Commented] (SPARK-17344) Kafka 0.8 support for Structured Streaming

[ https://issues.apache.org/jira/browse/SPARK-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558474#comment-15558474 ] Cody Koeninger commented on SPARK-17344: I think this is premature until you have

[jira] [Closed] (SPARK-10044) AnalysisException in resolving reference for sorting with aggregation

[ https://issues.apache.org/jira/browse/SPARK-10044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li closed SPARK-10044. --- Resolution: Not A Problem > AnalysisException in resolving reference for sorting with aggregation > -

[jira] [Commented] (SPARK-10044) AnalysisException in resolving reference for sorting with aggregation

[ https://issues.apache.org/jira/browse/SPARK-10044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558470#comment-15558470 ] Xiao Li commented on SPARK-10044: - This has been resolved at least in the Spark 2.0. Thus

[jira] [Closed] (SPARK-7012) Add support for NOT NULL modifier for column definitions on DDLParser

[ https://issues.apache.org/jira/browse/SPARK-7012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li closed SPARK-7012. -- Resolution: Not A Problem > Add support for NOT NULL modifier for column definitions on DDLParser >

[jira] [Commented] (SPARK-7012) Add support for NOT NULL modifier for column definitions on DDLParser

[ https://issues.apache.org/jira/browse/SPARK-7012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558466#comment-15558466 ] Xiao Li commented on SPARK-7012: Since 2.0, we have a native Parser. Thus, this has been r

[jira] [Comment Edited] (SPARK-5511) [SQL] Possible optimisations for predicate pushdowns from Spark SQL to Parquet

[ https://issues.apache.org/jira/browse/SPARK-5511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15557667#comment-15557667 ] Hyukjin Kwon edited comment on SPARK-5511 at 10/8/16 4:52 PM: --

[jira] [Closed] (SPARK-17540) SparkR array serde cannot work correctly when array length == 0

[ https://issues.apache.org/jira/browse/SPARK-17540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu closed SPARK-17540. -- Resolution: Won't Fix > SparkR array serde cannot work correctly when array length == 0 > -

[jira] [Commented] (SPARK-11428) Schema Merging Broken for Some Queries

[ https://issues.apache.org/jira/browse/SPARK-11428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558095#comment-15558095 ] Hyukjin Kwon commented on SPARK-11428: -- How about https://issues.apache.org/jira/bro

[jira] [Commented] (SPARK-8128) Schema Merging Broken: Dataframe Fails to Recognize Column in Schema

[ https://issues.apache.org/jira/browse/SPARK-8128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558088#comment-15558088 ] Hyukjin Kwon commented on SPARK-8128: - I am not 100% sure but I recall I saw similar i

[jira] [Resolved] (SPARK-16903) nullValue in first field is not respected by CSV source when read

[ https://issues.apache.org/jira/browse/SPARK-16903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-16903. -- Resolution: Duplicate [~falaki] I am going to make this as a duplicate because the PR was merge

[jira] [Commented] (SPARK-16720) Loading CSV file with 2k+ columns fails during attribute resolution on action

[ https://issues.apache.org/jira/browse/SPARK-16720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558067#comment-15558067 ] Hyukjin Kwon commented on SPARK-16720: -- Hi [~holdenk] Do you mind if I ask to close

[jira] [Comment Edited] (SPARK-16386) SQLContext and HiveContext parse a query string differently

[ https://issues.apache.org/jira/browse/SPARK-16386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558062#comment-15558062 ] Hyukjin Kwon edited comment on SPARK-16386 at 10/8/16 2:29 PM:

[jira] [Resolved] (SPARK-16386) SQLContext and HiveContext parse a query string differently

[ https://issues.apache.org/jira/browse/SPARK-16386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-16386. -- Resolution: Cannot Reproduce I can;t reproduce the problematic case 2 {code} context.sql("sele

[jira] [Commented] (SPARK-15577) Java can't import DataFrame type alias

[ https://issues.apache.org/jira/browse/SPARK-15577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558049#comment-15558049 ] Hyukjin Kwon commented on SPARK-15577: -- If I were a naive user from Spark 1.x, I thi

[jira] [Resolved] (SPARK-14621) add oracle hint optimizer

[ https://issues.apache.org/jira/browse/SPARK-14621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-14621. --- Resolution: Not A Problem > add oracle hint optimizer > - > >

[jira] [Resolved] (SPARK-14766) Attribute reference mismatch with Dataset filter + mapPartitions

[ https://issues.apache.org/jira/browse/SPARK-14766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-14766. -- Resolution: Cannot Reproduce {code} scala> Seq((1, 1)).toDS().filter(_._1 != 0).mapPartitions {

[jira] [Commented] (SPARK-14621) add oracle hint optimizer

[ https://issues.apache.org/jira/browse/SPARK-14621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558034#comment-15558034 ] Hyukjin Kwon commented on SPARK-14621: -- +1 for closing this. > add oracle hint opti

[jira] [Commented] (SPARK-14393) monotonicallyIncreasingId not monotonically increasing with downstream coalesce

[ https://issues.apache.org/jira/browse/SPARK-14393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558029#comment-15558029 ] Hyukjin Kwon commented on SPARK-14393: -- Still happens in the current master/2.0 {co

[jira] [Commented] (SPARK-13699) Spark SQL drops the table in "overwrite" mode while writing into table

[ https://issues.apache.org/jira/browse/SPARK-13699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15557996#comment-15557996 ] Hyukjin Kwon commented on SPARK-13699: -- Thank you. I will try to follow it. > Spark

[jira] [Resolved] (SPARK-13699) Spark SQL drops the table in "overwrite" mode while writing into table

[ https://issues.apache.org/jira/browse/SPARK-13699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-13699. --- Resolution: Duplicate > Spark SQL drops the table in "overwrite" mode while writing into table >

[jira] [Reopened] (SPARK-13699) Spark SQL drops the table in "overwrite" mode while writing into table

[ https://issues.apache.org/jira/browse/SPARK-13699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reopened SPARK-13699: --- I think it's clearer to resolve these as Duplicate when they are clearly a duplicate, even if it's true

[jira] [Resolved] (SPARK-13699) Spark SQL drops the table in "overwrite" mode while writing into table

[ https://issues.apache.org/jira/browse/SPARK-13699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-13699. -- Resolution: Fixed We introduced {{truncate}} option in https://github.com/apache/spark/pull/140

[jira] [Updated] (SPARK-16295) Extract SQL programming guide example snippets from source files instead of hard code them

[ https://issues.apache.org/jira/browse/SPARK-16295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-16295: -- Assignee: Cheng Lian > Extract SQL programming guide example snippets from source files instead of > h

[jira] [Resolved] (SPARK-16223) Codegen failure with a Dataframe program using an array

[ https://issues.apache.org/jira/browse/SPARK-16223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-16223. --- Resolution: Duplicate Fix Version/s: (was: 2.1.0) > Codegen failure with a Dataframe progr

[jira] [Reopened] (SPARK-16223) Codegen failure with a Dataframe program using an array

[ https://issues.apache.org/jira/browse/SPARK-16223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reopened SPARK-16223: --- Actually I think this is a duplicate > Codegen failure with a Dataframe program using an array > ---

[jira] [Updated] (SPARK-8144) For PySpark SQL, automatically convert values provided in readwriter options to string

[ https://issues.apache.org/jira/browse/SPARK-8144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-8144: - Assignee: Yijie Shen > For PySpark SQL, automatically convert values provided in readwriter options > to

[jira] [Commented] (SPARK-12916) Support Row.fromSeq and Row.toSeq methods in pyspark

[ https://issues.apache.org/jira/browse/SPARK-12916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15557964#comment-15557964 ] Hyukjin Kwon commented on SPARK-12916: -- I am pretty sure we don't need this but I wo

[jira] [Closed] (SPARK-12497) thriftServer does not support semicolon in sql

[ https://issues.apache.org/jira/browse/SPARK-12497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon closed SPARK-12497. Resolution: Duplicate It seems the duplicate is clearly linked. Please reopen this if you strongly

[jira] [Closed] (SPARK-11995) Partitioning Parquet by DateType

[ https://issues.apache.org/jira/browse/SPARK-11995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon closed SPARK-11995. Resolution: Duplicate As SPARK-17388 has a PR, I will mark this as duplicate. Please reopen if you

[jira] [Comment Edited] (SPARK-11868) wrong results returned from dataframe create from Rows without consistent schma on pyspark

[ https://issues.apache.org/jira/browse/SPARK-11868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15557944#comment-15557944 ] Hyukjin Kwon edited comment on SPARK-11868 at 10/8/16 1:12 PM:

[jira] [Comment Edited] (SPARK-11868) wrong results returned from dataframe create from Rows without consistent schma on pyspark

[ https://issues.apache.org/jira/browse/SPARK-11868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15557944#comment-15557944 ] Hyukjin Kwon edited comment on SPARK-11868 at 10/8/16 1:11 PM:

[jira] [Commented] (SPARK-11868) wrong results returned from dataframe create from Rows without consistent schma on pyspark

[ https://issues.apache.org/jira/browse/SPARK-11868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15557944#comment-15557944 ] Hyukjin Kwon commented on SPARK-11868: -- FYI, it now prints differently: {code} >>>

[jira] [Commented] (SPARK-11784) enable Timestamp filter pushdown

[ https://issues.apache.org/jira/browse/SPARK-11784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15557940#comment-15557940 ] Hyukjin Kwon commented on SPARK-11784: -- Could you feel up the description? it seems

  1   2   3   >