[jira] [Comment Edited] (SPARK-12216) Spark failed to delete temp directory

2019-01-17 Thread Kingsley Jones (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745850#comment-16745850 ] Kingsley Jones edited comment on SPARK-12216 at 1/18/19 7:21 AM: - I

[jira] [Commented] (SPARK-12216) Spark failed to delete temp directory

2019-01-17 Thread Kingsley Jones (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745850#comment-16745850 ] Kingsley Jones commented on SPARK-12216: I don't know why I persist with posting anything on

[jira] [Closed] (SPARK-26577) Add input optimizer when reading Hive table by SparkSQL

2019-01-17 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-26577. - > Add input optimizer when reading Hive table by SparkSQL >

[jira] [Commented] (SPARK-26630) ClassCastException in TableReader while creating HadoopRDD

2019-01-17 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745846#comment-16745846 ] Dongjoon Hyun commented on SPARK-26630: --- Hi, [~Deegue]. Since this is reported as a bug, could you

[jira] [Resolved] (SPARK-26659) Duplicate cmd.nodeName in the explain output of DataWritingCommandExec

2019-01-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-26659. - Resolution: Fixed Assignee: Xiao Li Fix Version/s: 3.0.0 > Duplicate cmd.nodeName in

[jira] [Assigned] (SPARK-26659) Duplicate cmd.nodeName in the explain output of DataWritingCommandExec

2019-01-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-26659: --- Assignee: Kris Mok (was: Xiao Li) > Duplicate cmd.nodeName in the explain output of

[jira] [Updated] (SPARK-26644) Add documents for SQL metrics

2019-01-17 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-26644: - Description: To make Spark users understand SQL metrics more, it would be nice to

[jira] [Commented] (SPARK-26644) Add documents for SQL metrics

2019-01-17 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745777#comment-16745777 ] Takeshi Yamamuro commented on SPARK-26644: -- If nobody takes it, I will after v2.3.3 release

[jira] [Assigned] (SPARK-26660) Add warning logs for large taskBinary size

2019-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26660: Assignee: Apache Spark > Add warning logs for large taskBinary size >

[jira] [Assigned] (SPARK-26660) Add warning logs for large taskBinary size

2019-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26660: Assignee: (was: Apache Spark) > Add warning logs for large taskBinary size >

[jira] [Updated] (SPARK-26660) Add warning logs for large taskBinary size

2019-01-17 Thread liupengcheng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liupengcheng updated SPARK-26660: - Attachment: screenshot-1.png > Add warning logs for large taskBinary size >

[jira] [Created] (SPARK-26660) Add warning logs for large taskBinary size

2019-01-17 Thread liupengcheng (JIRA)
liupengcheng created SPARK-26660: Summary: Add warning logs for large taskBinary size Key: SPARK-26660 URL: https://issues.apache.org/jira/browse/SPARK-26660 Project: Spark Issue Type:

[jira] [Commented] (SPARK-26646) Flaky test: pyspark.mllib.tests.test_streaming_algorithms StreamingLogisticRegressionWithSGDTests.test_training_and_prediction

2019-01-17 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745739#comment-16745739 ] Hyukjin Kwon commented on SPARK-26646: -- Thank you [~viirya]! It's not urgent. Please take your

[jira] [Commented] (SPARK-26659) Duplicate cmd.nodeName in the explain output of DataWritingCommandExec

2019-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745706#comment-16745706 ] Apache Spark commented on SPARK-26659: -- User 'rednaxelafx' has created a pull request for this

[jira] [Assigned] (SPARK-26659) Duplicate cmd.nodeName in the explain output of DataWritingCommandExec

2019-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26659: Assignee: (was: Apache Spark) > Duplicate cmd.nodeName in the explain output of

[jira] [Assigned] (SPARK-26659) Duplicate cmd.nodeName in the explain output of DataWritingCommandExec

2019-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26659: Assignee: Apache Spark > Duplicate cmd.nodeName in the explain output of

[jira] [Created] (SPARK-26659) Duplicate cmd.nodeName in the explain output of DataWritingCommandExec

2019-01-17 Thread Kris Mok (JIRA)
Kris Mok created SPARK-26659: Summary: Duplicate cmd.nodeName in the explain output of DataWritingCommandExec Key: SPARK-26659 URL: https://issues.apache.org/jira/browse/SPARK-26659 Project: Spark

[jira] [Resolved] (SPARK-26640) Code cleanup from lgtm.com analysis

2019-01-17 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26640. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23571

[jira] [Commented] (SPARK-26646) Flaky test: pyspark.mllib.tests.test_streaming_algorithms StreamingLogisticRegressionWithSGDTests.test_training_and_prediction

2019-01-17 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745685#comment-16745685 ] Liang-Chi Hsieh commented on SPARK-26646: - Thanks for pinging me [~hyukjin.kwon]! Sure, let me

[jira] [Updated] (SPARK-26658) Pyspark job is unable to serialize large objects

2019-01-17 Thread Parth Gandhi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Parth Gandhi updated SPARK-26658: - Description: When a pyspark job using python 3 tries to serialize large objects, it throws a

[jira] [Updated] (SPARK-26658) Pyspark job is unable to serialize large objects

2019-01-17 Thread Parth Gandhi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Parth Gandhi updated SPARK-26658: - Description: When a pyspark job using python 3 tries to serialize large objects, it throws a

[jira] [Assigned] (SPARK-26658) Pyspark job is unable to serialize large objects

2019-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26658: Assignee: Apache Spark > Pyspark job is unable to serialize large objects >

[jira] [Assigned] (SPARK-26658) Pyspark job is unable to serialize large objects

2019-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26658: Assignee: (was: Apache Spark) > Pyspark job is unable to serialize large objects >

[jira] [Created] (SPARK-26658) Pyspark job is unable to serialize large objects

2019-01-17 Thread Parth Gandhi (JIRA)
Parth Gandhi created SPARK-26658: Summary: Pyspark job is unable to serialize large objects Key: SPARK-26658 URL: https://issues.apache.org/jira/browse/SPARK-26658 Project: Spark Issue

[jira] [Commented] (SPARK-23904) Big execution plan cause OOM

2019-01-17 Thread Dave DeCaprio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745531#comment-16745531 ] Dave DeCaprio commented on SPARK-23904: --- In that Pull Request, the traversal of the tree will only

[jira] [Created] (SPARK-26657) Port DayWeek, DayOfWeek and WeekDay on Proleptic Gregorian calendar

2019-01-17 Thread Maxim Gekk (JIRA)
Maxim Gekk created SPARK-26657: -- Summary: Port DayWeek, DayOfWeek and WeekDay on Proleptic Gregorian calendar Key: SPARK-26657 URL: https://issues.apache.org/jira/browse/SPARK-26657 Project: Spark

[jira] [Created] (SPARK-26656) Benchmark for date/time functions and expressions

2019-01-17 Thread Maxim Gekk (JIRA)
Maxim Gekk created SPARK-26656: -- Summary: Benchmark for date/time functions and expressions Key: SPARK-26656 URL: https://issues.apache.org/jira/browse/SPARK-26656 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-26655) Support multiple aggregates in Structured Streaming append mode

2019-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26655: Assignee: (was: Apache Spark) > Support multiple aggregates in Structured Streaming

[jira] [Assigned] (SPARK-26655) Support multiple aggregates in Structured Streaming append mode

2019-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26655: Assignee: Apache Spark > Support multiple aggregates in Structured Streaming append mode

[jira] [Created] (SPARK-26655) Support multiple aggregates in Structured Streaming append mode

2019-01-17 Thread Arun Mahadevan (JIRA)
Arun Mahadevan created SPARK-26655: -- Summary: Support multiple aggregates in Structured Streaming append mode Key: SPARK-26655 URL: https://issues.apache.org/jira/browse/SPARK-26655 Project: Spark

[jira] [Created] (SPARK-26654) Use Timestamp/DateFormatter in CatalogColumnStat

2019-01-17 Thread Maxim Gekk (JIRA)
Maxim Gekk created SPARK-26654: -- Summary: Use Timestamp/DateFormatter in CatalogColumnStat Key: SPARK-26654 URL: https://issues.apache.org/jira/browse/SPARK-26654 Project: Spark Issue Type:

[jira] [Commented] (SPARK-26648) Noop Streaming Sink using DSV2

2019-01-17 Thread Pablo Langa Blanco (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745506#comment-16745506 ] Pablo Langa Blanco commented on SPARK-26648: Duplicated with SPARK-26649? > Noop Streaming

[jira] [Created] (SPARK-26653) Use Timestamp/DateFormatter in JDBCRelation

2019-01-17 Thread Maxim Gekk (JIRA)
Maxim Gekk created SPARK-26653: -- Summary: Use Timestamp/DateFormatter in JDBCRelation Key: SPARK-26653 URL: https://issues.apache.org/jira/browse/SPARK-26653 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-26652) Use Proleptic Gregorian calendar in creation of Timestamp/Date literals from strings

2019-01-17 Thread Maxim Gekk (JIRA)
Maxim Gekk created SPARK-26652: -- Summary: Use Proleptic Gregorian calendar in creation of Timestamp/Date literals from strings Key: SPARK-26652 URL: https://issues.apache.org/jira/browse/SPARK-26652

[jira] [Updated] (SPARK-26424) Use java.time API in timestamp/date functions

2019-01-17 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-26424: --- Issue Type: Sub-task (was: Improvement) Parent: SPARK-26651 > Use java.time API in

[jira] [Commented] (SPARK-26651) Use Proleptic Gregorian calendar

2019-01-17 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745482#comment-16745482 ] Maxim Gekk commented on SPARK-26651: [~hvanhovell][~cloud_fan][~srowen] This is the umbrella ticket

[jira] [Updated] (SPARK-26178) Use java.time API for parsing timestamps and dates from CSV

2019-01-17 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-26178: --- Issue Type: Sub-task (was: Improvement) Parent: SPARK-26651 > Use java.time API for

[jira] [Updated] (SPARK-26243) Use java.time API for parsing timestamps and dates from JSON

2019-01-17 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-26243: --- Issue Type: Sub-task (was: Improvement) Parent: SPARK-26651 > Use java.time API for

[jira] [Updated] (SPARK-26374) Support new date/timestamp parser in HadoopFsRelationTest

2019-01-17 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-26374: --- Issue Type: Sub-task (was: Bug) Parent: SPARK-26651 > Support new date/timestamp parser in

[jira] [Updated] (SPARK-26384) CSV schema inferring does not respect spark.sql.legacy.timeParser.enabled

2019-01-17 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-26384: --- Issue Type: Sub-task (was: Bug) Parent: SPARK-26651 > CSV schema inferring does not

[jira] [Resolved] (SPARK-26638) Pyspark vector classes always return error for unary negation

2019-01-17 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26638. --- Resolution: Fixed Fix Version/s: 2.4.1 3.0.0 2.3.3

[jira] [Updated] (SPARK-26456) Cast date/timestamp by Date/TimestampFormatter

2019-01-17 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-26456: --- Issue Type: Sub-task (was: Improvement) Parent: SPARK-26651 > Cast date/timestamp by

[jira] [Updated] (SPARK-26503) Get rid of spark.sql.legacy.timeParser.enabled

2019-01-17 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-26503: --- Issue Type: Sub-task (was: Task) Parent: SPARK-26651 > Get rid of

[jira] [Updated] (SPARK-26546) Caching of DateTimeFormatter

2019-01-17 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-26546: --- Issue Type: Sub-task (was: Improvement) Parent: SPARK-26651 > Caching of DateTimeFormatter

[jira] [Updated] (SPARK-26593) Use Proleptic Gregorian calendar in casting UTF8String to date/timestamp types

2019-01-17 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-26593: --- Issue Type: Sub-task (was: Improvement) Parent: SPARK-26651 > Use Proleptic Gregorian

[jira] [Updated] (SPARK-26618) Make typed Timestamp/Date literals consistent to casting

2019-01-17 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-26618: --- Issue Type: Sub-task (was: Improvement) Parent: SPARK-26651 > Make typed Timestamp/Date

[jira] [Created] (SPARK-26651) Use Proleptic Gregorian calendar

2019-01-17 Thread Maxim Gekk (JIRA)
Maxim Gekk created SPARK-26651: -- Summary: Use Proleptic Gregorian calendar Key: SPARK-26651 URL: https://issues.apache.org/jira/browse/SPARK-26651 Project: Spark Issue Type: Umbrella

[jira] [Commented] (SPARK-23904) Big execution plan cause OOM

2019-01-17 Thread Izek Greenfield (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745454#comment-16745454 ] Izek Greenfield commented on SPARK-23904: - [~staslos] The plan description is created in any

[jira] [Commented] (SPARK-26154) Stream-stream joins - left outer join gives inconsistent output

2019-01-17 Thread Pavel Chernikov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745445#comment-16745445 ] Pavel Chernikov commented on SPARK-26154: - Just in case a bit more reproducible  example is

[jira] [Created] (SPARK-26650) Yarn Client throws 'ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration'

2019-01-17 Thread Devaraj K (JIRA)
Devaraj K created SPARK-26650: - Summary: Yarn Client throws 'ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration' Key: SPARK-26650 URL: https://issues.apache.org/jira/browse/SPARK-26650

[jira] [Created] (SPARK-26649) Noop Streaming Sink using DSV2

2019-01-17 Thread Xiao Li (JIRA)
Xiao Li created SPARK-26649: --- Summary: Noop Streaming Sink using DSV2 Key: SPARK-26649 URL: https://issues.apache.org/jira/browse/SPARK-26649 Project: Spark Issue Type: Bug Components:

[jira] [Created] (SPARK-26648) Noop Streaming Sink using DSV2

2019-01-17 Thread Xiao Li (JIRA)
Xiao Li created SPARK-26648: --- Summary: Noop Streaming Sink using DSV2 Key: SPARK-26648 URL: https://issues.apache.org/jira/browse/SPARK-26648 Project: Spark Issue Type: Bug Components:

[jira] [Resolved] (SPARK-26622) Improve wording in SQLMetrics labels

2019-01-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-26622. - Resolution: Fixed Assignee: Juliusz Sompolski Fix Version/s: 3.0.0 > Improve wording in

[jira] [Resolved] (SPARK-26621) Use ConfigEntry for hardcoded configs for shuffle categories.

2019-01-17 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26621. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23550

[jira] [Assigned] (SPARK-26621) Use ConfigEntry for hardcoded configs for shuffle categories.

2019-01-17 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-26621: - Assignee: liuxian > Use ConfigEntry for hardcoded configs for shuffle categories. >

[jira] [Commented] (SPARK-26329) ExecutorMetrics should poll faster than heartbeats

2019-01-17 Thread Edwina Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745362#comment-16745362 ] Edwina Lu commented on SPARK-26329: --- [~wypoon], this sounds like a good solution. Sending the

[jira] [Commented] (SPARK-25960) Support subpath mounting with Kubernetes

2019-01-17 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745353#comment-16745353 ] Marcelo Vanzin commented on SPARK-25960: I general we don't backport new features to previous

[jira] [Commented] (SPARK-25960) Support subpath mounting with Kubernetes

2019-01-17 Thread Luciano Resende (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745347#comment-16745347 ] Luciano Resende commented on SPARK-25960: - [~vanzin] Any chance this can be backported to 2.4.x

[jira] [Commented] (SPARK-26329) ExecutorMetrics should poll faster than heartbeats

2019-01-17 Thread Wing Yew Poon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745313#comment-16745313 ] Wing Yew Poon commented on SPARK-26329: --- I am working on this. In the executor, we can track what

[jira] [Resolved] (SPARK-26593) Use Proleptic Gregorian calendar in casting UTF8String to date/timestamp types

2019-01-17 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-26593. --- Resolution: Fixed Fix Version/s: 3.0.0 > Use Proleptic Gregorian calendar in

[jira] [Assigned] (SPARK-26593) Use Proleptic Gregorian calendar in casting UTF8String to date/timestamp types

2019-01-17 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell reassigned SPARK-26593: - Assignee: Maxim Gekk > Use Proleptic Gregorian calendar in casting UTF8String

[jira] [Updated] (SPARK-26647) Spark Job marked as success when data is still being written to GCS

2019-01-17 Thread Grid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grid updated SPARK-26647: - Attachment: 51244468-1971b700-197d-11e9-9682-f021f1bc64e7.png > Spark Job marked as success when data is still

[jira] [Updated] (SPARK-26647) Spark Job marked as success when data is still being written to GCS

2019-01-17 Thread Grid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grid updated SPARK-26647: - Environment: Spark Kubernetes {{2.4.0}} on gke {{1.11.5-gke.5}}   was: Using spark kubernetes {{2.4.0}} on

[jira] [Updated] (SPARK-26647) Spark Job marked as success when data is still being written to GCS

2019-01-17 Thread Grid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grid updated SPARK-26647: - Description: When using Spark on Kubernetes and the latest jar

[jira] [Created] (SPARK-26647) Spark Job marked as success when data is still being written to GCS

2019-01-17 Thread Grid (JIRA)
Grid created SPARK-26647: Summary: Spark Job marked as success when data is still being written to GCS Key: SPARK-26647 URL: https://issues.apache.org/jira/browse/SPARK-26647 Project: Spark Issue

[jira] [Commented] (SPARK-26646) Flaky test: pyspark.mllib.tests.test_streaming_algorithms StreamingLogisticRegressionWithSGDTests.test_training_and_prediction

2019-01-17 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745254#comment-16745254 ] Hyukjin Kwon commented on SPARK-26646: -- Hey [~bryanc] and [~viirya], would you guys mind if I ask

[jira] [Updated] (SPARK-26646) Flaky test: pyspark.mllib.tests.test_streaming_algorithms StreamingLogisticRegressionWithSGDTests.test_training_and_prediction

2019-01-17 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-26646: - Description: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101356/console

[jira] [Created] (SPARK-26646) Flaky test: pyspark.mllib.tests.test_streaming_algorithms StreamingLogisticRegressionWithSGDTests.test_training_and_prediction

2019-01-17 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-26646: Summary: Flaky test: pyspark.mllib.tests.test_streaming_algorithms StreamingLogisticRegressionWithSGDTests.test_training_and_prediction Key: SPARK-26646 URL:

[jira] [Resolved] (SPARK-23817) Migrate ORC file format read path to data source V2

2019-01-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-23817. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23383

[jira] [Assigned] (SPARK-26645) CSV infer schema bug infers decimal(9,-1)

2019-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26645: Assignee: Apache Spark > CSV infer schema bug infers decimal(9,-1) >

[jira] [Assigned] (SPARK-26645) CSV infer schema bug infers decimal(9,-1)

2019-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26645: Assignee: (was: Apache Spark) > CSV infer schema bug infers decimal(9,-1) >

[jira] [Commented] (SPARK-26645) CSV infer schema bug infers decimal(9,-1)

2019-01-17 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745146#comment-16745146 ] Marco Gaido commented on SPARK-26645: - The error is on python side, I will submit a PR shortly,

[jira] [Created] (SPARK-26645) CSV infer schema bug infers decimal(9,-1)

2019-01-17 Thread Ohad Raviv (JIRA)
Ohad Raviv created SPARK-26645: -- Summary: CSV infer schema bug infers decimal(9,-1) Key: SPARK-26645 URL: https://issues.apache.org/jira/browse/SPARK-26645 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-26644) Add documents for SQL metrics

2019-01-17 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745011#comment-16745011 ] Takeshi Yamamuro commented on SPARK-26644: -- An example I made is: 

[jira] [Created] (SPARK-26644) Add documents for SQL metrics

2019-01-17 Thread Takeshi Yamamuro (JIRA)
Takeshi Yamamuro created SPARK-26644: Summary: Add documents for SQL metrics Key: SPARK-26644 URL: https://issues.apache.org/jira/browse/SPARK-26644 Project: Spark Issue Type:

[jira] [Updated] (SPARK-26576) Broadcast hint not applied to partitioned table

2019-01-17 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-26576: - Fix Version/s: (was: 2.3.3) > Broadcast hint not applied to partitioned table >

[jira] [Resolved] (SPARK-26457) Show hadoop configurations in HistoryServer environment tab

2019-01-17 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26457. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23486

[jira] [Assigned] (SPARK-26457) Show hadoop configurations in HistoryServer environment tab

2019-01-17 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-26457: - Assignee: deshanxiao > Show hadoop configurations in HistoryServer environment tab >

[jira] [Assigned] (SPARK-26643) Spark hive throwing an incorrect analysis exception,when set table properties.

2019-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26643: Assignee: (was: Apache Spark) > Spark hive throwing an incorrect analysis

[jira] [Assigned] (SPARK-26643) Spark hive throwing an incorrect analysis exception,when set table properties.

2019-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26643: Assignee: Apache Spark > Spark hive throwing an incorrect analysis exception,when set

[jira] [Updated] (SPARK-26643) Spark hive throwing an incorrect analysis exception,when set table properties.

2019-01-17 Thread jiaan.geng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-26643: --- Description: When I execute a DDL in spark-sql,throwing a AnalysisException as follows: {code:java}

[jira] [Updated] (SPARK-26643) Spark hive throwing an incorrect analysis exception,when set table properties.

2019-01-17 Thread jiaan.geng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-26643: --- Description: When I execute a DDL in spark-sql,throwing a AnalysisException as follows:   I found

[jira] [Updated] (SPARK-26643) Spark hive throwing an incorrect analysis exception,when set table properties.

2019-01-17 Thread jiaan.geng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-26643: --- Description: When I execute a DDL in spark-sql,throwing a AnalysisException  (was: When I execute

[jira] [Updated] (SPARK-26643) Spark hive throwing an incorrect analysis exception,when set table properties.

2019-01-17 Thread jiaan.geng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-26643: --- Description: When I execute a DDL in spark-sql,throwing a AnalysisException as follows: {code:java}

[jira] [Updated] (SPARK-26643) Spark hive throwing an incorrect analysis exception,when set table properties.

2019-01-17 Thread jiaan.geng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-26643: --- Description: When I execute a DDL in spark-sql,throwing a AnalysisException as follows: {code:java}

[jira] [Updated] (SPARK-26643) Spark hive throwing an incorrect analysis exception,when set table properties.

2019-01-17 Thread jiaan.geng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-26643: --- Description: When I execute a DDL in spark-sql,throwing a AnalysisException as follows: {code:java}

[jira] [Created] (SPARK-26643) Spark hive

2019-01-17 Thread jiaan.geng (JIRA)
jiaan.geng created SPARK-26643: -- Summary: Spark hive Key: SPARK-26643 URL: https://issues.apache.org/jira/browse/SPARK-26643 Project: Spark Issue Type: Bug Components: SQL

[jira] [Updated] (SPARK-26643) Spark hive throwing an incorrect analysis exception,when set table properties.

2019-01-17 Thread jiaan.geng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-26643: --- Description: When I execute a DDL in spark-sql,throwing a AnalysisException as follows: {code:java}

[jira] [Updated] (SPARK-26643) Spark hive throwing an incorrect analysis exception,when set table properties.

2019-01-17 Thread jiaan.geng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-26643: --- Description: When I execute a DDL in spark-sql,throwing a AnalysisException as follows: {code:java}

[jira] [Updated] (SPARK-26643) Spark hive throwing an incorrect analysis exception,when set table properties.

2019-01-17 Thread jiaan.geng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-26643: --- Description: When I execute a DDL in spark-sql,throwing a AnalysisException as follows: (was: When

[jira] [Updated] (SPARK-26643) Spark hive throwing an incorrect analysis exception,when set table properties.

2019-01-17 Thread jiaan.geng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-26643: --- Description: When I execute a DDL in spark-sql,a AnalysisException  (was: When I execute a DDL in

[jira] [Updated] (SPARK-26643) Spark hive throwing an incorrect analysis exception,when set table properties.

2019-01-17 Thread jiaan.geng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-26643: --- Summary: Spark hive throwing an incorrect analysis exception,when set table properties. (was:

[jira] [Updated] (SPARK-26643) Spark hive

2019-01-17 Thread jiaan.geng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-26643: --- Description: When I execute a DDL in spark-sql,a AnalysisException > Spark hive > --- > >

[jira] [Updated] (SPARK-26642) Add --num-executors option to spark-submit for Spark on K8S

2019-01-17 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-26642: Description: Currently spark-submit supports the option "--num-executors NUM" only for Spark on

[jira] [Created] (SPARK-26642) Add --num-executor option to spark-submit for Spark on K8S

2019-01-17 Thread Luca Canali (JIRA)
Luca Canali created SPARK-26642: --- Summary: Add --num-executor option to spark-submit for Spark on K8S Key: SPARK-26642 URL: https://issues.apache.org/jira/browse/SPARK-26642 Project: Spark

[jira] [Updated] (SPARK-26642) Add --num-executor option to spark-submit for Spark on K8S

2019-01-17 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-26642: Description: Currently spark-submit supports the option "--num-executors NUM" only for Spark on

[jira] [Updated] (SPARK-26642) Add --num-executors option to spark-submit for Spark on K8S

2019-01-17 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-26642: Description: Currently spark-submit supports the option --num-executors NUM only for Spark on

[jira] [Updated] (SPARK-26642) Add --num-executors option to spark-submit for Spark on K8S

2019-01-17 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-26642: Description: Currently spark-submit supports the option "--num-executors NUM" only for Spark on

[jira] [Assigned] (SPARK-26642) Add --num-executors option to spark-submit for Spark on K8S

2019-01-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26642: Assignee: Apache Spark > Add --num-executors option to spark-submit for Spark on K8S >

[jira] [Updated] (SPARK-26642) Add --num-executors option to spark-submit for Spark on K8S

2019-01-17 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-26642: Description: Currently spark-submit supports the option "--num-executors NUM" only for Spark on

  1   2   >