[jira] [Created] (SPARK-30293) In HiveThriftServer2 will remove wrong statement

angerszhu created SPARK-30293: - Summary: In HiveThriftServer2 will remove wrong statement Key: SPARK-30293 URL: https://issues.apache.org/jira/browse/SPARK-30293 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-30292) Throw Exception when invalid string is cast to decimal in ANSI mode

Rakesh Raushan created SPARK-30292: -- Summary: Throw Exception when invalid string is cast to decimal in ANSI mode Key: SPARK-30292 URL: https://issues.apache.org/jira/browse/SPARK-30292 Project: Spar

[jira] [Commented] (SPARK-30283) V2 Command logical plan should use UnresolvedV2Relation for a table

[ https://issues.apache.org/jira/browse/SPARK-30283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998829#comment-16998829 ] Terry Kim commented on SPARK-30283: --- PR will be submitted once https://issues.apache.o

[jira] [Commented] (SPARK-30283) V2 Command logical plan should use UnresolvedV2Relation for a table

[ https://issues.apache.org/jira/browse/SPARK-30283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998827#comment-16998827 ] Terry Kim commented on SPARK-30283: --- [~aman_omer], I already started working on this (

[jira] [Commented] (SPARK-30283) V2 Command logical plan should use UnresolvedV2Relation for a table

[ https://issues.apache.org/jira/browse/SPARK-30283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998825#comment-16998825 ] Aman Omer commented on SPARK-30283: --- [~imback82] I would like to work on this. Please

[jira] [Commented] (SPARK-30288) Failed to write valid Parquet files when column names contains special characters like spaces

[ https://issues.apache.org/jira/browse/SPARK-30288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998812#comment-16998812 ] Hyukjin Kwon commented on SPARK-30288: -- You can fix it if Parquet supports it. > F

[jira] [Comment Edited] (SPARK-30288) Failed to write valid Parquet files when column names contains special characters like spaces

[ https://issues.apache.org/jira/browse/SPARK-30288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998809#comment-16998809 ] Hyukjin Kwon edited comment on SPARK-30288 at 12/18/19 4:52 AM: --

[jira] [Comment Edited] (SPARK-30288) Failed to write valid Parquet files when column names contains special characters like spaces

[ https://issues.apache.org/jira/browse/SPARK-30288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998809#comment-16998809 ] Rakesh Raushan edited comment on SPARK-30288 at 12/18/19 4:49 AM:

[jira] [Comment Edited] (SPARK-30288) Failed to write valid Parquet files when column names contains special characters like spaces

[ https://issues.apache.org/jira/browse/SPARK-30288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998809#comment-16998809 ] Rakesh Raushan edited comment on SPARK-30288 at 12/18/19 4:48 AM:

[jira] [Commented] (SPARK-30288) Failed to write valid Parquet files when column names contains special characters like spaces

[ https://issues.apache.org/jira/browse/SPARK-30288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998809#comment-16998809 ] Rakesh Raushan commented on SPARK-30288: [~dongjoon] I have checked locally afte

[jira] [Commented] (SPARK-30286) Some thoughts on new features for MLLIB

[ https://issues.apache.org/jira/browse/SPARK-30286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998792#comment-16998792 ] zhengruifeng commented on SPARK-30286: -- [~srowen] Thanks for the reply. I will make

[jira] [Updated] (SPARK-30049) SQL fails to parse when comment contains an unmatched quote character

[ https://issues.apache.org/jira/browse/SPARK-30049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-30049: Target Version/s: 3.0.0 > SQL fails to parse when comment contains an unmatched quote character >

[jira] [Commented] (SPARK-30049) SQL fails to parse when comment contains an unmatched quote character

[ https://issues.apache.org/jira/browse/SPARK-30049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998776#comment-16998776 ] Yuming Wang commented on SPARK-30049: - Thank you [~jlowe]. Hive also has this issue:

[jira] [Commented] (SPARK-30288) Failed to write valid Parquet files when column names contains special characters like spaces

[ https://issues.apache.org/jira/browse/SPARK-30288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998774#comment-16998774 ] Ankit Raj Boudh commented on SPARK-30288: - i will start working on this issue >

[jira] [Comment Edited] (SPARK-1476) 2GB limit in spark for blocks

[ https://issues.apache.org/jira/browse/SPARK-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998752#comment-16998752 ] Shirish edited comment on SPARK-1476 at 12/18/19 2:38 AM: -- This

[jira] [Commented] (SPARK-1476) 2GB limit in spark for blocks

[ https://issues.apache.org/jira/browse/SPARK-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998752#comment-16998752 ] Shirish commented on SPARK-1476: This is an old chain that I happen to land on to. I am i

[jira] [Updated] (SPARK-30285) Fix deadlock between LiveListenerBus#stop and AsyncEventQueue#removeListenerOnError

[ https://issues.apache.org/jira/browse/SPARK-30285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang Shuo updated SPARK-30285: -- Description: There is a deadlock between LiveListenerBus#stop and AsyncEventQueue#removeListenerOnErr

[jira] [Updated] (SPARK-30285) Fix deadlock between LiveListenerBus#stop and AsyncEventQueue#removeListenerOnError

[ https://issues.apache.org/jira/browse/SPARK-30285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang Shuo updated SPARK-30285: -- Summary: Fix deadlock between LiveListenerBus#stop and AsyncEventQueue#removeListenerOnError (was: Fi

[jira] [Created] (SPARK-30291) Catch the exception when do materialize in AQE

Ke Jia created SPARK-30291: -- Summary: Catch the exception when do materialize in AQE Key: SPARK-30291 URL: https://issues.apache.org/jira/browse/SPARK-30291 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-30290) Count for merged block when fetch continuous blocks in batch

L. C. Hsieh created SPARK-30290: --- Summary: Count for merged block when fetch continuous blocks in batch Key: SPARK-30290 URL: https://issues.apache.org/jira/browse/SPARK-30290 Project: Spark I

[jira] [Created] (SPARK-30289) DSv2 partitioning should not accept nested columns

DB Tsai created SPARK-30289: --- Summary: DSv2 partitioning should not accept nested columns Key: SPARK-30289 URL: https://issues.apache.org/jira/browse/SPARK-30289 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-30288) Failed to write valid Parquet files when column names contains special characters like spaces

Jingyuan Wang created SPARK-30288: - Summary: Failed to write valid Parquet files when column names contains special characters like spaces Key: SPARK-30288 URL: https://issues.apache.org/jira/browse/SPARK-30288

[jira] [Commented] (SPARK-6235) Address various 2G limits

[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998577#comment-16998577 ] Imran Rashid commented on SPARK-6235: - [~sammysheep] Spark's ML library uses the same

[jira] [Commented] (SPARK-29106) Add jenkins arm test for spark

[ https://issues.apache.org/jira/browse/SPARK-29106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998573#comment-16998573 ] Shane Knapp commented on SPARK-29106: - also got scipy compiled from source! pyarrow

[jira] [Resolved] (SPARK-30274) Avoid BytesToBytesMap lookup hang forever when holding keys reaching max capacity

[ https://issues.apache.org/jira/browse/SPARK-30274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-30274. --- Fix Version/s: 3.0.0 2.4.5 Resolution: Fixed Issue resolved by pul

[jira] [Assigned] (SPARK-29106) Add jenkins arm test for spark

[ https://issues.apache.org/jira/browse/SPARK-29106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Knapp reassigned SPARK-29106: --- Assignee: Shane Knapp > Add jenkins arm test for spark > -- > >

[jira] [Assigned] (SPARK-29600) array_contains built in function is not backward compatible in 3.0

[ https://issues.apache.org/jira/browse/SPARK-29600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-29600: --- Assignee: Aman Omer > array_contains built in function is not backward compatible in 3.0 >

[jira] [Commented] (SPARK-29106) Add jenkins arm test for spark

[ https://issues.apache.org/jira/browse/SPARK-29106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998476#comment-16998476 ] Shane Knapp commented on SPARK-29106: - alright, i installed gfortran and successfull

[jira] [Assigned] (SPARK-30235) Keeping compatibility with 2.4 external shuffle service regarding host local shuffle blocks reading

[ https://issues.apache.org/jira/browse/SPARK-30235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Masiero Vanzin reassigned SPARK-30235: -- Assignee: Attila Zsolt Piros > Keeping compatibility with 2.4 externa

[jira] [Resolved] (SPARK-30235) Keeping compatibility with 2.4 external shuffle service regarding host local shuffle blocks reading

[ https://issues.apache.org/jira/browse/SPARK-30235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Masiero Vanzin resolved SPARK-30235. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull reque

[jira] [Commented] (SPARK-29600) array_contains built in function is not backward compatible in 3.0

[ https://issues.apache.org/jira/browse/SPARK-29600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998413#comment-16998413 ] Aman Omer commented on SPARK-29600: --- ID: aman_omer [~cloud_fan] > array_contains buil

[jira] [Resolved] (SPARK-29600) array_contains built in function is not backward compatible in 3.0

[ https://issues.apache.org/jira/browse/SPARK-29600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-29600. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26811 [https://gith

[jira] [Reopened] (SPARK-27300) Create the new graph projects in Spark and set up build/test

[ https://issues.apache.org/jira/browse/SPARK-27300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-27300: --- Assignee: (was: Martin Junghanns) This is reverted via https://github.com/apache/spark

[jira] [Updated] (SPARK-27300) Create the new graph projects in Spark and set up build/test

[ https://issues.apache.org/jira/browse/SPARK-27300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27300: -- Fix Version/s: (was: 3.0.0) > Create the new graph projects in Spark and set up build/test

[jira] [Commented] (SPARK-29432) nullable flag of new column changes when persisting a pyspark dataframe

[ https://issues.apache.org/jira/browse/SPARK-29432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998357#comment-16998357 ] Martin Studer commented on SPARK-29432: --- We're facing the same issue with Spark 2.

[jira] [Commented] (SPARK-27300) Create the new graph projects in Spark and set up build/test

[ https://issues.apache.org/jira/browse/SPARK-27300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998344#comment-16998344 ] Dongjoon Hyun commented on SPARK-27300: --- +1 for [~srowen]'s decision. > Create th

[jira] [Commented] (SPARK-30049) SQL fails to parse when comment contains an unmatched quote character

[ https://issues.apache.org/jira/browse/SPARK-30049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998331#comment-16998331 ] Jason Darrell Lowe commented on SPARK-30049: Found some time to track this d

[jira] [Updated] (SPARK-29018) Spark ThriftServer change to it's own API

[ https://issues.apache.org/jira/browse/SPARK-29018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-29018: -- Description: With the development of Spark and Hive，in current `sql/hive-thriftserver`, we need to do

[jira] [Updated] (SPARK-30287) Add new module spark-service as thrift server module

[ https://issues.apache.org/jira/browse/SPARK-30287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-30287: -- Description: Add a new module of `sql/service` as spark thrift server module (was: Add a new module o

[jira] [Commented] (SPARK-30286) Some thoughts on new features for MLLIB

[ https://issues.apache.org/jira/browse/SPARK-30286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998274#comment-16998274 ] Sean R. Owen commented on SPARK-30286: -- I'm not so sure about a param server, or wh

[jira] [Updated] (SPARK-29018) Spark ThriftServer change to it's own API

[ https://issues.apache.org/jira/browse/SPARK-29018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-29018: -- Description: With the development of Spark and Hive，in current `sql/hive-thriftserver`, we need to do

[jira] [Updated] (SPARK-29018) Spark ThriftServer change to it's own API

[ https://issues.apache.org/jira/browse/SPARK-29018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-29018: -- Description: With the development of Spark and Hive，in current `sql/hive-thriftserver`, we need to do

[jira] [Updated] (SPARK-29018) Spark ThriftServer change to it's own API

[ https://issues.apache.org/jira/browse/SPARK-29018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-29018: -- Description: ### What changes were proposed in this pull request? With the development of Spark and H

[jira] [Created] (SPARK-30287) Add new module spark-service as thrift server module

angerszhu created SPARK-30287: - Summary: Add new module spark-service as thrift server module Key: SPARK-30287 URL: https://issues.apache.org/jira/browse/SPARK-30287 Project: Spark Issue Type: Su

[jira] [Commented] (SPARK-30229) java.lang.NullPointerException at org.apache.spark.SparkContext.getPreferredLocs(SparkContext.scala:1783)

[ https://issues.apache.org/jira/browse/SPARK-30229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998247#comment-16998247 ] Ankit Raj Boudh commented on SPARK-30229: - please provide me what operation you

[jira] [Commented] (SPARK-30229) java.lang.NullPointerException at org.apache.spark.SparkContext.getPreferredLocs(SparkContext.scala:1783)

[ https://issues.apache.org/jira/browse/SPARK-30229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998245#comment-16998245 ] Ankit Raj Boudh commented on SPARK-30229: - no , today i will analyse this > jav

[jira] [Commented] (SPARK-29708) Different answers in aggregates of multiple grouping sets

[ https://issues.apache.org/jira/browse/SPARK-29708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998229#comment-16998229 ] Takeshi Yamamuro commented on SPARK-29708: -- I'm looking into this issue, SPARK-

[jira] [Updated] (SPARK-29708) Different answers in aggregates of multiple grouping sets

[ https://issues.apache.org/jira/browse/SPARK-29708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-29708: - Labels: correctness (was: ) > Different answers in aggregates of multiple grouping sets

[jira] [Updated] (SPARK-30269) Should use old partition stats to decide whether to update stats when analyzing partition

[ https://issues.apache.org/jira/browse/SPARK-30269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-30269: - Fix Version/s: (was: 2.3.5) > Should use old partition stats to decide whether to update sta

[jira] [Commented] (SPARK-30269) Should use old partition stats to decide whether to update stats when analyzing partition

[ https://issues.apache.org/jira/browse/SPARK-30269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998215#comment-16998215 ] Hyukjin Kwon commented on SPARK-30269: -- (Spark 2.3.x is EOL so there would be no mo

[jira] [Commented] (SPARK-30270) Can't pickle abstract classes (with cloudpickle)

[ https://issues.apache.org/jira/browse/SPARK-30270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998124#comment-16998124 ] Sebastian Straub commented on SPARK-30270: -- So... no? There won't be a backport

[jira] [Commented] (SPARK-30270) Can't pickle abstract classes (with cloudpickle)

[ https://issues.apache.org/jira/browse/SPARK-30270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998109#comment-16998109 ] Hyukjin Kwon commented on SPARK-30270: -- Ah yes, there was a lack of explanation abo

[jira] [Updated] (SPARK-29018) Spark ThriftServer change to it's own API

[ https://issues.apache.org/jira/browse/SPARK-29018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-29018: Issue Type: Umbrella (was: Improvement) > Spark ThriftServer change to it's own API > ---

[jira] [Updated] (SPARK-30286) Some thoughts on new features for MLLIB

[ https://issues.apache.org/jira/browse/SPARK-30286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-30286: - Description: Some thoughts on new features for ML: 1, clustering: *mini-batch KMeans*: KMeans m

[jira] [Commented] (SPARK-30286) Some thoughts on new features for MLLIB

[ https://issues.apache.org/jira/browse/SPARK-30286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998096#comment-16998096 ] zhengruifeng commented on SPARK-30286: --  It seem that the last roadmap for mllib is

[jira] [Updated] (SPARK-30286) Some thoughts on new features for MLLIB

[ https://issues.apache.org/jira/browse/SPARK-30286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-30286: - Description: Some thoughts on new features for ML: 1, clustering: *mini-batch KMeans*: KMeans m

[jira] [Created] (SPARK-30286) Some thoughts on new features for MLLIB

zhengruifeng created SPARK-30286: Summary: Some thoughts on new features for MLLIB Key: SPARK-30286 URL: https://issues.apache.org/jira/browse/SPARK-30286 Project: Spark Issue Type: Wish

[jira] [Resolved] (SPARK-30181) Throws runtime exception when filter metastore partition key that's not string type or integral types

[ https://issues.apache.org/jira/browse/SPARK-30181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu-Jhe Li resolved SPARK-30181. --- Resolution: Duplicate > Throws runtime exception when filter metastore partition key that's not > s

[jira] [Commented] (SPARK-30181) Throws runtime exception when filter metastore partition key that's not string type or integral types

[ https://issues.apache.org/jira/browse/SPARK-30181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998053#comment-16998053 ] Yu-Jhe Li commented on SPARK-30181: --- Yes, the bug is fixed in the master branch and la

[jira] [Commented] (SPARK-29481) all the commands should look up catalog/table like v2 commands

[ https://issues.apache.org/jira/browse/SPARK-29481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998026#comment-16998026 ] Xiang Li commented on SPARK-29481: -- [~cloud_fan] and [~viirya], would you please share

[jira] [Created] (SPARK-30285) Fix race condition between LiveListenerBus#stop and AsyncEventQueue#removeListenerOnError

Wang Shuo created SPARK-30285: - Summary: Fix race condition between LiveListenerBus#stop and AsyncEventQueue#removeListenerOnError Key: SPARK-30285 URL: https://issues.apache.org/jira/browse/SPARK-30285 P

[jira] [Commented] (SPARK-30270) Can't pickle abstract classes (with cloudpickle)

[ https://issues.apache.org/jira/browse/SPARK-30270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16997998#comment-16997998 ] Sebastian Straub commented on SPARK-30270: -- True, it's fixed on master, but not

[jira] [Updated] (SPARK-30284) CREATE VIEW should track the current catalog and namespace

[ https://issues.apache.org/jira/browse/SPARK-30284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-30284: Summary: CREATE VIEW should track the current catalog and namespace (was: CREATE VIEW should keep

[jira] [Created] (SPARK-30284) CREATE VIEW should keep the current catalog and namespace

Wenchen Fan created SPARK-30284: --- Summary: CREATE VIEW should keep the current catalog and namespace Key: SPARK-30284 URL: https://issues.apache.org/jira/browse/SPARK-30284 Project: Spark Issue