[jira] [Resolved] (SPARK-27664) Performance issue with FileStatusCache, while reading from object stores.

2019-06-12 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma resolved SPARK-27664. - Resolution: Won't Fix I am marking it as won't fix, because, this is now difficult to

[jira] [Created] (SPARK-28032) DataFrame.saveAsTable( in AVRO format with Timestamps create bad Hive tables

2019-06-12 Thread Mathew Wicks (JIRA)
Mathew Wicks created SPARK-28032: Summary: DataFrame.saveAsTable( in AVRO format with Timestamps create bad Hive tables Key: SPARK-28032 URL: https://issues.apache.org/jira/browse/SPARK-28032

[jira] [Updated] (SPARK-27930) List all built-in UDFs have different names

2019-06-12 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27930: Summary: List all built-in UDFs have different names (was: Add built-in Math Function: RANDOM)

[jira] [Updated] (SPARK-27930) Add built-in Math Function: RANDOM

2019-06-12 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27930: Description: This ticket list all built-in UDFs have different names: |PostgreSQL|Spark SQL|

[jira] [Issue Comment Deleted] (SPARK-27930) List all built-in UDFs have different names

2019-06-12 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27930: Comment: was deleted (was: Workaround: {code:sql} select rand() {code} {code:sql} select

[jira] [Commented] (SPARK-28024) Incorrect numeric values when out of range

2019-06-12 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862712#comment-16862712 ] Yuming Wang commented on SPARK-28024: - JDK also has this issue: !SPARK-28024.png! > Incorrect

[jira] [Updated] (SPARK-28024) Incorrect numeric values when out of range

2019-06-12 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-28024: Attachment: SPARK-28024.png > Incorrect numeric values when out of range >

[jira] [Commented] (SPARK-28008) Default values & column comments in AVRO schema converters

2019-06-12 Thread Mathew Wicks (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862699#comment-16862699 ] Mathew Wicks commented on SPARK-28008: -- The only issue I could think, would be that the column

[jira] [Commented] (SPARK-27290) remove unneed sort under Aggregate

2019-06-12 Thread Xiaoju Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862698#comment-16862698 ] Xiaoju Wu commented on SPARK-27290: --- [~joshrosen] Got it. I think we should identify in which patterns

[jira] [Resolved] (SPARK-28016) Spark hangs when an execution plan has many projections on nested structs

2019-06-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28016. -- Resolution: Cannot Reproduce > Spark hangs when an execution plan has many projections on

[jira] [Commented] (SPARK-28016) Spark hangs when an execution plan has many projections on nested structs

2019-06-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862693#comment-16862693 ] Hyukjin Kwon commented on SPARK-28016: -- Cannot reproduce in the current master. {code} scala>

[jira] [Commented] (SPARK-28008) Default values & column comments in AVRO schema converters

2019-06-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862691#comment-16862691 ] Hyukjin Kwon commented on SPARK-28008: --

[jira] [Resolved] (SPARK-28009) PipedRDD: Block not locked for reading failure

2019-06-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28009. -- Resolution: Duplicate > PipedRDD: Block not locked for reading failure >

[jira] [Updated] (SPARK-28009) PipedRDD: Block not locked for reading failure

2019-06-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28009: - Description: PipedRDD operation fails with the below stack trace. Failure primarily occurs

[jira] [Commented] (SPARK-28016) Spark hangs when an execution plan has many projections on nested structs

2019-06-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862682#comment-16862682 ] Hyukjin Kwon commented on SPARK-28016: -- Can you narrow down and make a self-contained reproducer

[jira] [Resolved] (SPARK-28021) A unappropriate exception in StaticMemoryManager.getMaxExecutionMemory

2019-06-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28021. -- Resolution: Invalid Let's ask a question to mailing list before filing it as an issue. See

[jira] [Resolved] (SPARK-28026) How to get the second row from 1 minute window

2019-06-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28026. -- Resolution: Invalid > How to get the second row from 1 minute window >

[jira] [Commented] (SPARK-28026) How to get the second row from 1 minute window

2019-06-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862671#comment-16862671 ] Hyukjin Kwon commented on SPARK-28026: -- Let's ask a question to mailing list before filing as an

[jira] [Commented] (SPARK-13882) Remove org.apache.spark.sql.execution.local

2019-06-12 Thread Lai Zhou (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862641#comment-16862641 ] Lai Zhou commented on SPARK-13882: -- hi,[~rxin], is this iterator-based local mode  will be 

[jira] [Assigned] (SPARK-28031) Improve or remove doctest on over function of Column

2019-06-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-28031: Assignee: Liang-Chi Hsieh > Improve or remove doctest on over function of Column >

[jira] [Resolved] (SPARK-28031) Improve or remove doctest on over function of Column

2019-06-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28031. -- Resolution: Fixed Fix Version/s: 2.4.4 3.0.0 Issue resolved by pull

[jira] [Issue Comment Deleted] (SPARK-24791) Spark Structured Streaming randomly does not process batch

2019-06-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-24791: -- Comment: was deleted (was: User 'zhangmeng0426' has created a pull request for this issue:

[jira] [Updated] (SPARK-27943) Implement default constraint with Column for Hive table

2019-06-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27943: -- Affects Version/s: (was: 2.4.0) (was: 2.3.0)

[jira] [Comment Edited] (SPARK-28025) HDFSBackedStateStoreProvider should not leak .crc files

2019-06-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862548#comment-16862548 ] Stavros Kontopoulos edited comment on SPARK-28025 at 6/12/19 11:32 PM:

[jira] [Commented] (SPARK-28025) HDFSBackedStateStoreProvider should not leak .crc files

2019-06-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862548#comment-16862548 ] Stavros Kontopoulos commented on SPARK-28025: - [~kabhwan] yes we discussed this internally

[jira] [Updated] (SPARK-27919) DataSourceV2: Add v2 session catalog

2019-06-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27919: -- Issue Type: Improvement (was: Bug) > DataSourceV2: Add v2 session catalog >

[jira] [Updated] (SPARK-28006) User-defined grouped transform pandas_udf for window operations

2019-06-12 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Jin updated SPARK-28006: --- Description: Currently, pandas_udf supports "grouped aggregate" type that can be used with unbounded and

[jira] [Commented] (SPARK-28006) User-defined grouped transform pandas_udf for window operations

2019-06-12 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862505#comment-16862505 ] Li Jin commented on SPARK-28006: Thanks [~hyukjin.kwon] for the comments! I updated the description to

[jira] [Commented] (SPARK-28025) HDFSBackedStateStoreProvider should not leak .crc files

2019-06-12 Thread Jungtaek Lim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862449#comment-16862449 ] Jungtaek Lim commented on SPARK-28025: -- Personally I would respect the reporter and encourage to

[jira] [Resolved] (SPARK-28030) Binary file data source doesn't support space in file names

2019-06-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-28030. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24855

[jira] [Commented] (SPARK-18112) Spark2.x does not support read data from Hive 2.x metastore

2019-06-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862432#comment-16862432 ] Dongjoon Hyun commented on SPARK-18112: --- [~honglun]. You should not replace Apache Spark jars in

[jira] [Comment Edited] (SPARK-28025) HDFSBackedStateStoreProvider should not leak .crc files

2019-06-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862339#comment-16862339 ] Stavros Kontopoulos edited comment on SPARK-28025 at 6/12/19 6:26 PM:

[jira] [Comment Edited] (SPARK-28025) HDFSBackedStateStoreProvider should not leak .crc files

2019-06-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862339#comment-16862339 ] Stavros Kontopoulos edited comment on SPARK-28025 at 6/12/19 6:20 PM:

[jira] [Updated] (SPARK-27907) HiveUDAF should return NULL in case of 0 rows

2019-06-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27907: -- Priority: Blocker (was: Major) > HiveUDAF should return NULL in case of 0 rows >

[jira] [Comment Edited] (SPARK-28025) HDFSBackedStateStoreProvider should not leak .crc files

2019-06-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862339#comment-16862339 ] Stavros Kontopoulos edited comment on SPARK-28025 at 6/12/19 6:18 PM:

[jira] [Comment Edited] (SPARK-28025) HDFSBackedStateStoreProvider should not leak .crc files

2019-06-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862339#comment-16862339 ] Stavros Kontopoulos edited comment on SPARK-28025 at 6/12/19 6:17 PM:

[jira] [Updated] (SPARK-26045) Error in the spark 2.4 release package with the spark-avro_2.11 depdency

2019-06-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26045: -- Priority: Blocker (was: Major) > Error in the spark 2.4 release package with the

[jira] [Commented] (SPARK-28025) HDFSBackedStateStoreProvider should not leak .crc files

2019-06-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862339#comment-16862339 ] Stavros Kontopoulos commented on SPARK-28025: - There is a workaround by setting `--conf

[jira] [Updated] (SPARK-28006) User-defined grouped transform pandas_udf for window operations

2019-06-12 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Jin updated SPARK-28006: --- Description: Currently, pandas_udf supports "grouped aggregate" type that can be used with unbounded and

[jira] [Updated] (SPARK-28006) User-defined grouped transform pandas_udf for window operations

2019-06-12 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Jin updated SPARK-28006: --- Description: Currently, pandas_udf supports "grouped aggregate" type that can be used with unbounded and

[jira] [Assigned] (SPARK-26949) Prevent "purge" to remove needed batch files in CompactibleFileStreamLog

2019-06-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-26949: - Assignee: Jungtaek Lim > Prevent "purge" to remove needed batch files in

[jira] [Updated] (SPARK-26949) Prevent "purge" to remove needed batch files in CompactibleFileStreamLog

2019-06-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26949: -- Target Version/s: 3.0.0 > Prevent "purge" to remove needed batch files in

[jira] [Commented] (SPARK-27463) Support Dataframe Cogroup via Pandas UDFs

2019-06-12 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862287#comment-16862287 ] Li Jin commented on SPARK-27463: I think one way to design this API to mimic the existing dataset

[jira] [Assigned] (SPARK-27823) Add an abstraction layer for accelerator resource handling to avoid manipulating raw confs

2019-06-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-27823: - Assignee: Xiangrui Meng (was: Thomas Graves) > Add an abstraction layer for

[jira] [Resolved] (SPARK-28018) Allow upcasting decimal to double/float

2019-06-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-28018. --- Resolution: Invalid This is closed by the author due to the loss of precision. Please see

[jira] [Assigned] (SPARK-28030) Binary file data source doesn't support space in file names

2019-06-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-28030: Assignee: Apache Spark (was: Xiangrui Meng) > Binary file data source doesn't support

[jira] [Assigned] (SPARK-28030) Binary file data source doesn't support space in file names

2019-06-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-28030: Assignee: Xiangrui Meng (was: Apache Spark) > Binary file data source doesn't support

[jira] [Assigned] (SPARK-28031) Improve or remove doctest on over function of Column

2019-06-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-28031: Assignee: Apache Spark > Improve or remove doctest on over function of Column >

[jira] [Assigned] (SPARK-28031) Improve or remove doctest on over function of Column

2019-06-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-28031: Assignee: (was: Apache Spark) > Improve or remove doctest on over function of Column

[jira] [Created] (SPARK-28031) Improve or remove doctest on over function of Column

2019-06-12 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-28031: --- Summary: Improve or remove doctest on over function of Column Key: SPARK-28031 URL: https://issues.apache.org/jira/browse/SPARK-28031 Project: Spark

[jira] [Updated] (SPARK-28030) Binary file data source doesn't support space in file names

2019-06-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-28030: -- Description: {code} echo 123 > "/tmp/test space.txt"

[jira] [Created] (SPARK-28030) Binary file data source doesn't support space in file names

2019-06-12 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-28030: - Summary: Binary file data source doesn't support space in file names Key: SPARK-28030 URL: https://issues.apache.org/jira/browse/SPARK-28030 Project: Spark

[jira] [Commented] (SPARK-28025) HDFSBackedStateStoreProvider should not leak .crc files

2019-06-12 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862227#comment-16862227 ] Steve Loughran commented on SPARK-28025: looking at the previous patch, you don't need to call

[jira] [Commented] (SPARK-27463) Support Dataframe Cogroup via Pandas UDFs

2019-06-12 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862225#comment-16862225 ] Li Jin commented on SPARK-27463: For cogroup, I don't think there is analogous API in pandas. There is

[jira] [Resolved] (SPARK-28013) Upgrade to Kafka 2.2.1

2019-06-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-28013. --- Resolution: Fixed Assignee: Dongjoon Hyun Fix Version/s: 3.0.0 This is

[jira] [Assigned] (SPARK-28029) Add int2.sql

2019-06-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-28029: Assignee: (was: Apache Spark) > Add int2.sql > > > Key:

[jira] [Assigned] (SPARK-28029) Add int2.sql

2019-06-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-28029: Assignee: Apache Spark > Add int2.sql > > > Key:

[jira] [Created] (SPARK-28029) Add int2.sql

2019-06-12 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-28029: --- Summary: Add int2.sql Key: SPARK-28029 URL: https://issues.apache.org/jira/browse/SPARK-28029 Project: Spark Issue Type: Sub-task Components: SQL

[jira] [Updated] (SPARK-28024) Incorrect numeric values when out of range

2019-06-12 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-28024: Description: For example: {code:sql} select tinyint(128) * tinyint(2); -- 0 select

[jira] [Updated] (SPARK-28024) Incorrect numeric values when out of range

2019-06-12 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-28024: Description: For example: {code:sql} select tinyint(128) * tinyint(2); -- 0 select

[jira] [Commented] (SPARK-28009) PipedRDD: Block not locked for reading failure

2019-06-12 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862180#comment-16862180 ] Liang-Chi Hsieh commented on SPARK-28009: - I think this looks like duplicate to SPARK-27666. >

[jira] [Created] (SPARK-28028) Cast numeric to integral type need round

2019-06-12 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-28028: --- Summary: Cast numeric to integral type need round Key: SPARK-28028 URL: https://issues.apache.org/jira/browse/SPARK-28028 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-27750) Standalone scheduler - ability to prioritize applications over drivers, many drivers act like Denial of Service

2019-06-12 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-27750: - Description: If I submit 1000 spark submit drivers then they consume all the cores on my cluster (essentially

[jira] [Assigned] (SPARK-28027) Add Bitwise shift left/right

2019-06-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-28027: Assignee: (was: Apache Spark) > Add Bitwise shift left/right >

[jira] [Assigned] (SPARK-28027) Add Bitwise shift left/right

2019-06-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-28027: Assignee: Apache Spark > Add Bitwise shift left/right > > >

[jira] [Updated] (SPARK-28007) Caret operator (^) means bitwise XOR in Spark/Hive and exponentiation in Postgres/Redshift

2019-06-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-28007: --- Description: The expression {{expr1 ^ expr2}} has different meanings in Spark and Postgres: * [In

[jira] [Created] (SPARK-28027) Add Bitwise shift left/right

2019-06-12 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-28027: --- Summary: Add Bitwise shift left/right Key: SPARK-28027 URL: https://issues.apache.org/jira/browse/SPARK-28027 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-27309) CypherSession implementation in spark-cypher

2019-06-12 Thread Martin Junghanns (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Junghanns updated SPARK-27309: - Description: Implement CypherSession and RelationalCypherSession in spark-cypher

[jira] [Updated] (SPARK-27309) CypherSession implementation in spark-cypher

2019-06-12 Thread Martin Junghanns (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Junghanns updated SPARK-27309: - Summary: CypherSession implementation in spark-cypher (was: PropertyGraph SparkSession

[jira] [Updated] (SPARK-27307) PropertyGraph save/load API (Python)

2019-06-12 Thread Martin Junghanns (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Junghanns updated SPARK-27307: - Summary: PropertyGraph save/load API (Python) (was: PropertyGraph save/load (Python))

[jira] [Updated] (SPARK-27308) PropertyGraph Cypher query API (Python)

2019-06-12 Thread Martin Junghanns (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Junghanns updated SPARK-27308: - Summary: PropertyGraph Cypher query API (Python) (was: PropertyGraph Cypher query

[jira] [Updated] (SPARK-27305) PropertyGraph Cypher query API (Java/Scala)

2019-06-12 Thread Martin Junghanns (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Junghanns updated SPARK-27305: - Summary: PropertyGraph Cypher query API (Java/Scala) (was: PropertyGraph Cypher query

[jira] [Updated] (SPARK-27304) PropertyGraph save/load API (Scala/Java)

2019-06-12 Thread Martin Junghanns (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Junghanns updated SPARK-27304: - Summary: PropertyGraph save/load API (Scala/Java) (was: PropertyGraph save/load

[jira] [Updated] (SPARK-27306) PropertyGraph construction API (Python)

2019-06-12 Thread Martin Junghanns (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Junghanns updated SPARK-27306: - Summary: PropertyGraph construction API (Python) (was: PropertyGraph construction

[jira] [Updated] (SPARK-27303) PropertyGraph construction API (Scala/Java)

2019-06-12 Thread Martin Junghanns (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Junghanns updated SPARK-27303: - Summary: PropertyGraph construction API (Scala/Java) (was: PropertyGraph construction

[jira] [Commented] (SPARK-28025) HDFSBackedStateStoreProvider should not leak .crc files

2019-06-12 Thread Jungtaek Lim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862065#comment-16862065 ] Jungtaek Lim commented on SPARK-28025: -- [~gmaas] Nice finding. Would you like to submit a PR for

[jira] [Assigned] (SPARK-27303) PropertyGraph construction (Scala/Java)

2019-06-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27303: Assignee: (was: Apache Spark) > PropertyGraph construction (Scala/Java) >

[jira] [Assigned] (SPARK-27303) PropertyGraph construction (Scala/Java)

2019-06-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27303: Assignee: Apache Spark > PropertyGraph construction (Scala/Java) >

[jira] [Created] (SPARK-28026) How to get the second row from 1 minute window

2019-06-12 Thread Devendra Vishwakarma (JIRA)
Devendra Vishwakarma created SPARK-28026: Summary: How to get the second row from 1 minute window Key: SPARK-28026 URL: https://issues.apache.org/jira/browse/SPARK-28026 Project: Spark

[jira] [Resolved] (SPARK-28011) SQL parse error when there are too many aliases in the table

2019-06-12 Thread U Shaw (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] U Shaw resolved SPARK-28011. Resolution: Invalid > SQL parse error when there are too many aliases in the table >

[jira] [Commented] (SPARK-27463) Support Dataframe Cogroup via Pandas UDFs

2019-06-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16861999#comment-16861999 ] Hyukjin Kwon commented on SPARK-27463: -- It's easier and safer to find a reference to justify new

[jira] [Commented] (SPARK-27463) Support Dataframe Cogroup via Pandas UDFs

2019-06-12 Thread Chris Martin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16861995#comment-16861995 ] Chris Martin commented on SPARK-27463: -- Also my assumption is that the most difficult part of this

[jira] [Commented] (SPARK-27463) Support Dataframe Cogroup via Pandas UDFs

2019-06-12 Thread Chris Martin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16861993#comment-16861993 ] Chris Martin commented on SPARK-27463: -- Hi [~hyukjin.kwon]- I've just started working on the code

[jira] [Commented] (SPARK-28025) HDFSBackedStateStoreProvider should not leak .crc files

2019-06-12 Thread Gerard Maas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16861991#comment-16861991 ] Gerard Maas commented on SPARK-28025: - I reproduced  the issue in a  spark-shell session:

[jira] [Updated] (SPARK-28025) HDFSBackedStateStoreProvider should not leak .crc files

2019-06-12 Thread Gerard Maas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gerard Maas updated SPARK-28025: Summary: HDFSBackedStateStoreProvider should not leak .crc files (was:

[jira] [Updated] (SPARK-28024) Incorrect numeric values when out of range

2019-06-12 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-28024: Summary: Incorrect numeric values when out of range (was: Incorrect value when out of range) >

[jira] [Commented] (SPARK-28025) HDFSBackedStateStoreProvider leaks .crc files

2019-06-12 Thread Gerard Maas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16861983#comment-16861983 ] Gerard Maas commented on SPARK-28025: - Same problem. A different part of the code. >

[jira] [Created] (SPARK-28025) HDFSBackedStateStoreProvider leaks .crc files

2019-06-12 Thread Gerard Maas (JIRA)
Gerard Maas created SPARK-28025: --- Summary: HDFSBackedStateStoreProvider leaks .crc files Key: SPARK-28025 URL: https://issues.apache.org/jira/browse/SPARK-28025 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-28024) Incorrect value when out of range

2019-06-12 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-28024: --- Summary: Incorrect value when out of range Key: SPARK-28024 URL: https://issues.apache.org/jira/browse/SPARK-28024 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-28021) A unappropriate exception in StaticMemoryManager.getMaxExecutionMemory

2019-06-12 Thread JustDoDT (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16861957#comment-16861957 ] JustDoDT commented on SPARK-28021: --    I can't understand this question either.Begging for a big

[jira] [Updated] (SPARK-27997) kubernetes client token expired

2019-06-12 Thread Henry Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henry Yu updated SPARK-27997: - Description: Hi , when I try to submit spark to k8s in cluster mode, I need an authtoken to talk with

[jira] [Updated] (SPARK-28022) k8s pod affinity achieve cloud native friendly autoscaling

2019-06-12 Thread Henry Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henry Yu updated SPARK-28022: - Description: Hi, in order to achieve cloud native friendly autoscaling , I propose to add a pod

[jira] [Updated] (SPARK-27923) List all cases that PostgreSQL throws an exception but Spark SQL is NULL

2019-06-12 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27923: Description: In this ticket, we plan to list all cases that PostgreSQL throws an exception but

[jira] [Updated] (SPARK-28022) k8s pod affinity achieve cloud native friendly autoscaling

2019-06-12 Thread Henry Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henry Yu updated SPARK-28022: - Description: Hi, in order to achieve cloud native friendly autoscaling , I propose to add a pod

[jira] [Updated] (SPARK-28022) k8s pod affinity achieve cloud native friendly autoscaling

2019-06-12 Thread Henry Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henry Yu updated SPARK-28022: - Summary: k8s pod affinity achieve cloud native friendly autoscaling (was: k8s pod affinity achieve

[jira] [Updated] (SPARK-27997) kubernetes client token expired

2019-06-12 Thread Henry Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henry Yu updated SPARK-27997: - Shepherd: Dongjoon Hyun > kubernetes client token expired > > >

[jira] [Updated] (SPARK-28023) Trim the string when cast string type to other types

2019-06-12 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-28023: Description: For example: {code:sql} select int2 ' 21234 '; SELECT bool ' f ';

[jira] [Updated] (SPARK-28023) Trim the string when cast string type to other types

2019-06-12 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-28023: Description: For example: {code:sql} SELECT bool ' f '; select int2 ' 21234 ';

[jira] [Updated] (SPARK-28023) Trim the string when cast string type to other types

2019-06-12 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-28023: Description: For example: {code:sql} select int2 ' 21234 ' select bool ' 21234 ' {code} >

[jira] [Updated] (SPARK-28022) k8s pod affinity achieve cloud native friendly auto scaling

2019-06-12 Thread Henry Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henry Yu updated SPARK-28022: - Description: Hi, in order to achieve cloud native friendly autoscaling , I propose to add a pod

  1   2   >