[jira] [Commented] (SPARK-9278) DataFrameWriter.insertInto inserts incorrect data

[ https://issues.apache.org/jira/browse/SPARK-9278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374048#comment-16374048 ] Hyukjin Kwon commented on SPARK-9278: - Mind opening another JIRA ticket with its affected version and

[jira] [Comment Edited] (SPARK-9278) DataFrameWriter.insertInto inserts incorrect data

[ https://issues.apache.org/jira/browse/SPARK-9278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374040#comment-16374040 ] Xiaoju Wu edited comment on SPARK-9278 at 2/23/18 7:48 AM: --- Seems the issue

[jira] [Commented] (SPARK-9278) DataFrameWriter.insertInto inserts incorrect data

[ https://issues.apache.org/jira/browse/SPARK-9278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374040#comment-16374040 ] Xiaoju Wu commented on SPARK-9278: -- Seems the issue still exists, here's the test: val data = Seq( (7,

[jira] [Created] (SPARK-23492) Application shows up as running in history server even when latest attempt has completed

Kevin Kim created SPARK-23492: - Summary: Application shows up as running in history server even when latest attempt has completed Key: SPARK-23492 URL: https://issues.apache.org/jira/browse/SPARK-23492

[jira] [Updated] (SPARK-23400) Add the extra constructors for ScalaUDF

[ https://issues.apache.org/jira/browse/SPARK-23400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23400: Fix Version/s: (was: 2.3.1) (was: 2.4.0) 2.3.0 > Add the

[jira] [Updated] (SPARK-23422) YarnShuffleIntegrationSuite failure when SPARK_PREPEND_CLASSES set to 1

[ https://issues.apache.org/jira/browse/SPARK-23422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23422: Fix Version/s: (was: 2.3.1) (was: 2.4.0) 2.3.0 >

[jira] [Updated] (SPARK-23352) Explicitly specify supported types in Pandas UDFs

[ https://issues.apache.org/jira/browse/SPARK-23352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23352: Fix Version/s: (was: 2.3.1) 2.3.0 > Explicitly specify supported types in Pandas

[jira] [Updated] (SPARK-23399) Register a task completion listener first for OrcColumnarBatchReader

[ https://issues.apache.org/jira/browse/SPARK-23399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23399: Fix Version/s: (was: 2.3.1) 2.3.0 > Register a task completion listener first for

[jira] [Updated] (SPARK-23177) PySpark parameter-less UDFs raise exception if applied after distinct

[ https://issues.apache.org/jira/browse/SPARK-23177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23177: Fix Version/s: (was: 2.4.0) > PySpark parameter-less UDFs raise exception if applied after distinct >

[jira] [Updated] (SPARK-23388) Support for Parquet Binary DecimalType in VectorizedColumnReader

[ https://issues.apache.org/jira/browse/SPARK-23388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23388: Fix Version/s: (was: 2.3.1) 2.3.0 > Support for Parquet Binary DecimalType in

[jira] [Updated] (SPARK-23177) PySpark parameter-less UDFs raise exception if applied after distinct

[ https://issues.apache.org/jira/browse/SPARK-23177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23177: Fix Version/s: (was: 2.3.1) 2.3.0 > PySpark parameter-less UDFs raise exception if

[jira] [Updated] (SPARK-23154) Document backwards compatibility guarantees for ML persistence

[ https://issues.apache.org/jira/browse/SPARK-23154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23154: Fix Version/s: (was: 2.3.1) (was: 2.4.0) 2.3.0 > Document

[jira] [Updated] (SPARK-23384) When it has no incomplete(completed) applications found, the last updated time is not formatted and client local time zone is not show in history server web ui.

[ https://issues.apache.org/jira/browse/SPARK-23384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23384: Fix Version/s: (was: 2.3.1) (was: 2.4.0) 2.3.0 > When it has

[jira] [Updated] (SPARK-23394) Storage info's Cached Partitions doesn't consider the replications (but sc.getRDDStorageInfo does)

[ https://issues.apache.org/jira/browse/SPARK-23394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23394: Fix Version/s: (was: 2.3.1) 2.3.0 > Storage info's Cached Partitions doesn't

[jira] [Updated] (SPARK-23192) Hint is lost after using cached data

[ https://issues.apache.org/jira/browse/SPARK-23192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23192: Fix Version/s: (was: 2.3.1) 2.3.0 > Hint is lost after using cached data >

[jira] [Updated] (SPARK-22956) Union Stream Failover Cause `IllegalStateException`

[ https://issues.apache.org/jira/browse/SPARK-22956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-22956: Fix Version/s: (was: 2.3.1) 2.3.0 > Union Stream Failover Cause

[jira] [Updated] (SPARK-23148) spark.read.csv with multiline=true gives FileNotFoundException if path contains spaces

[ https://issues.apache.org/jira/browse/SPARK-23148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23148: Fix Version/s: (was: 2.3.1) (was: 2.4.0) 2.3.0 >

[jira] [Updated] (SPARK-23053) taskBinarySerialization and task partitions calculate in DagScheduler.submitMissingTasks should keep the same RDD checkpoint status

[ https://issues.apache.org/jira/browse/SPARK-23053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23053: Fix Version/s: (was: 2.3.1) (was: 2.4.0) 2.3.0 >

[jira] [Updated] (SPARK-23419) data source v2 write path should re-throw interruption exceptions directly

[ https://issues.apache.org/jira/browse/SPARK-23419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23419: Fix Version/s: (was: 2.3.1) 2.3.0 > data source v2 write path should re-throw

[jira] [Updated] (SPARK-23416) flaky test: org.apache.spark.sql.kafka010.KafkaSourceStressForDontFailOnDataLossSuite.stress test for failOnDataLoss=false

[ https://issues.apache.org/jira/browse/SPARK-23416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23416: Fix Version/s: (was: 2.3.1) 2.3.0 > flaky test: >

[jira] [Updated] (SPARK-23406) Stream-stream self joins does not work

[ https://issues.apache.org/jira/browse/SPARK-23406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23406: Target Version/s: 2.3.1, 2.4.0 (was: 2.4.0) > Stream-stream self joins does not work >

[jira] [Updated] (SPARK-23195) Hint of cached data is lost

[ https://issues.apache.org/jira/browse/SPARK-23195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23195: Target Version/s: 2.3.0 (was: 2.3.1) > Hint of cached data is lost > --- > >

[jira] [Updated] (SPARK-23406) Stream-stream self joins does not work

[ https://issues.apache.org/jira/browse/SPARK-23406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23406: Target Version/s: 2.4.0 (was: 2.3.1, 2.4.0) > Stream-stream self joins does not work >

[jira] [Updated] (SPARK-23454) Add Trigger information to the Structured Streaming programming guide

[ https://issues.apache.org/jira/browse/SPARK-23454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23454: Target Version/s: 2.3.0, 2.4.0 (was: 2.3.1, 2.4.0) > Add Trigger information to the Structured Streaming

[jira] [Updated] (SPARK-22809) pyspark is sensitive to imports with dots

[ https://issues.apache.org/jira/browse/SPARK-22809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-22809: Target Version/s: 2.3.0, 2.4.0 (was: 2.3.1, 2.4.0) > pyspark is sensitive to imports with dots >

[jira] [Updated] (SPARK-23468) Failure to authenticate with old shuffle service

[ https://issues.apache.org/jira/browse/SPARK-23468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23468: Target Version/s: 2.3.0 (was: 2.3.1) > Failure to authenticate with old shuffle service >

[jira] [Updated] (SPARK-23197) Flaky test: spark.streaming.ReceiverSuite."receiver_life_cycle"

[ https://issues.apache.org/jira/browse/SPARK-23197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23197: Target Version/s: 2.3.0 (was: 2.3.1) > Flaky test: spark.streaming.ReceiverSuite."receiver_life_cycle" >

[jira] [Resolved] (SPARK-23490) Check storage.locationUri with existing table in CreateTable

[ https://issues.apache.org/jira/browse/SPARK-23490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23490. - Resolution: Fixed Fix Version/s: 2.4.0 > Check storage.locationUri with existing table in

[jira] [Assigned] (SPARK-23490) Check storage.locationUri with existing table in CreateTable

[ https://issues.apache.org/jira/browse/SPARK-23490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-23490: --- Assignee: Gengliang Wang > Check storage.locationUri with existing table in CreateTable >

[jira] [Commented] (SPARK-23371) Parquet Footer data is wrong on window in parquet format partition table

[ https://issues.apache.org/jira/browse/SPARK-23371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373862#comment-16373862 ] pin_zhang commented on SPARK-23371: --- # It's the spark that bundles two versions(1.6 and 1.8) parquet

[jira] [Commented] (SPARK-23486) LookupFunctions should not check the same function name more than once

[ https://issues.apache.org/jira/browse/SPARK-23486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373851#comment-16373851 ] kevin yu commented on SPARK-23486: -- Hello [~lian cheng]: I think the easy way is to build a hash map 

[jira] [Reopened] (SPARK-23405) The task will hang up when a small table left semi join a big table

[ https://issues.apache.org/jira/browse/SPARK-23405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXinXIaoLei reopened SPARK-23405: --- [~q79969786] In the left semi join, the left table will not broadcast. And  there is many null

[jira] [Comment Edited] (SPARK-23485) Kubernetes should support node blacklist

[ https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373652#comment-16373652 ] Stavros Kontopoulos edited comment on SPARK-23485 at 2/23/18 12:04 AM:

[jira] [Comment Edited] (SPARK-23485) Kubernetes should support node blacklist

[ https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373652#comment-16373652 ] Stavros Kontopoulos edited comment on SPARK-23485 at 2/22/18 11:09 PM:

[jira] [Commented] (SPARK-23485) Kubernetes should support node blacklist

[ https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373652#comment-16373652 ] Stavros Kontopoulos commented on SPARK-23485: - Does the scheduler know all the reasons an app

[jira] [Issue Comment Deleted] (SPARK-23485) Kubernetes should support node blacklist

[ https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stavros Kontopoulos updated SPARK-23485: Comment: was deleted (was: [~liyinan926] I understand the default behavior of the

[jira] [Commented] (SPARK-23485) Kubernetes should support node blacklist

[ https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373634#comment-16373634 ] Stavros Kontopoulos commented on SPARK-23485: - [~liyinan926] I understand the default

[jira] [Commented] (SPARK-23485) Kubernetes should support node blacklist

[ https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373620#comment-16373620 ] Yinan Li commented on SPARK-23485: -- The Kubernetes scheduler backend simply creates executor pods

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.2.0

[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373611#comment-16373611 ] Michael Armbrust commented on SPARK-18057: -- My only concern is that it is stable and backwards

[jira] [Comment Edited] (SPARK-23485) Kubernetes should support node blacklist

[ https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373558#comment-16373558 ] Stavros Kontopoulos edited comment on SPARK-23485 at 2/22/18 10:32 PM:

[jira] [Comment Edited] (SPARK-23485) Kubernetes should support node blacklist

[ https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373558#comment-16373558 ] Stavros Kontopoulos edited comment on SPARK-23485 at 2/22/18 10:29 PM:

[jira] [Comment Edited] (SPARK-23485) Kubernetes should support node blacklist

[ https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373558#comment-16373558 ] Stavros Kontopoulos edited comment on SPARK-23485 at 2/22/18 10:22 PM:

[jira] [Commented] (SPARK-23485) Kubernetes should support node blacklist

[ https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373558#comment-16373558 ] Stavros Kontopoulos commented on SPARK-23485: - I guess everything is covered via

[jira] [Commented] (SPARK-23485) Kubernetes should support node blacklist

[ https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373544#comment-16373544 ] Yinan Li commented on SPARK-23485: -- I'm not sure if node blacklisting applies to Kubernetes. In the

[jira] [Comment Edited] (SPARK-18278) SPIP: Support native submission of spark jobs to a kubernetes cluster

[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373416#comment-16373416 ] Bob Treacy edited comment on SPARK-18278 at 2/22/18 8:49 PM: - Is this release

[jira] [Comment Edited] (SPARK-18278) SPIP: Support native submission of spark jobs to a kubernetes cluster

[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373416#comment-16373416 ] Bob Treacy edited comment on SPARK-18278 at 2/22/18 8:45 PM: - Is this release

[jira] [Commented] (SPARK-18278) SPIP: Support native submission of spark jobs to a kubernetes cluster

[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373416#comment-16373416 ] Bob Treacy commented on SPARK-18278: Is this release going to support using SparkLauncher? I have run

[jira] [Assigned] (SPARK-23476) Spark will not start in local mode with authentication on

[ https://issues.apache.org/jira/browse/SPARK-23476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-23476: -- Assignee: Gabor Somogyi > Spark will not start in local mode with authentication on >

[jira] [Resolved] (SPARK-23476) Spark will not start in local mode with authentication on

[ https://issues.apache.org/jira/browse/SPARK-23476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-23476. Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20652

[jira] [Commented] (SPARK-23475) The "stages" page doesn't show any completed stages

[ https://issues.apache.org/jira/browse/SPARK-23475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373254#comment-16373254 ] Marcelo Vanzin commented on SPARK-23475: Marco's PR was also merged into master:

[jira] [Assigned] (SPARK-23491) continuous symptom

[ https://issues.apache.org/jira/browse/SPARK-23491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23491: Assignee: (was: Apache Spark) > continuous symptom > -- > >

[jira] [Commented] (SPARK-23491) continuous symptom

[ https://issues.apache.org/jira/browse/SPARK-23491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373235#comment-16373235 ] Apache Spark commented on SPARK-23491: -- User 'jose-torres' has created a pull request for this

[jira] [Assigned] (SPARK-23491) continuous symptom

[ https://issues.apache.org/jira/browse/SPARK-23491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23491: Assignee: Apache Spark > continuous symptom > -- > > Key:

[jira] [Created] (SPARK-23491) continuous symptom

Jose Torres created SPARK-23491: --- Summary: continuous symptom Key: SPARK-23491 URL: https://issues.apache.org/jira/browse/SPARK-23491 Project: Spark Issue Type: Sub-task Components:

[jira] [Updated] (SPARK-23488) Add other missing Catalog methods to Python API

[ https://issues.apache.org/jira/browse/SPARK-23488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-23488: - Issue Type: Sub-task (was: Improvement) Parent: SPARK-23483 > Add other missing Catalog

[jira] [Assigned] (SPARK-23490) Check storage.locationUri with existing table in CreateTable

[ https://issues.apache.org/jira/browse/SPARK-23490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23490: Assignee: Apache Spark > Check storage.locationUri with existing table in CreateTable >

[jira] [Assigned] (SPARK-23490) Check storage.locationUri with existing table in CreateTable

[ https://issues.apache.org/jira/browse/SPARK-23490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23490: Assignee: (was: Apache Spark) > Check storage.locationUri with existing table in

[jira] [Commented] (SPARK-23490) Check storage.locationUri with existing table in CreateTable

[ https://issues.apache.org/jira/browse/SPARK-23490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372958#comment-16372958 ] Apache Spark commented on SPARK-23490: -- User 'gengliangwang' has created a pull request for this

[jira] [Created] (SPARK-23490) Check storage.locationUri with existing table in CreateTable

Gengliang Wang created SPARK-23490: -- Summary: Check storage.locationUri with existing table in CreateTable Key: SPARK-23490 URL: https://issues.apache.org/jira/browse/SPARK-23490 Project: Spark

[jira] [Created] (SPARK-23489) HiveExternalCatalogVersionsSuite flaky test

Marco Gaido created SPARK-23489: --- Summary: HiveExternalCatalogVersionsSuite flaky test Key: SPARK-23489 URL: https://issues.apache.org/jira/browse/SPARK-23489 Project: Spark Issue Type: Task

[jira] [Commented] (SPARK-21157) Report Total Memory Used by Spark Executors

[ https://issues.apache.org/jira/browse/SPARK-21157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372789#comment-16372789 ] assia ydroudj commented on SPARK-21157: --- this paragraph is well interesting for me: "Perhaps we

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.2.0

[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372627#comment-16372627 ] Sönke Liebau commented on SPARK-18057: -- Fine with me :) Any thoughts on which version we want to

[jira] [Commented] (SPARK-15516) Schema merging in driver fails for parquet when merging LongType and IntegerType

[ https://issues.apache.org/jira/browse/SPARK-15516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372536#comment-16372536 ] Heinrich van den Heever commented on SPARK-15516: - Hi All I am fairly new to Spark so