[jira] [Closed] (SPARK-25391) Make behaviors consistent when converting parquet hive table to parquet data source

[ https://issues.apache.org/jira/browse/SPARK-25391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chenxiao Mao closed SPARK-25391. > Make behaviors consistent when converting parquet hive table to parquet data > source >

[jira] [Commented] (SPARK-25437) Using OpenHashMap replace HashMap improve Encoder Performance

[ https://issues.apache.org/jira/browse/SPARK-25437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617102#comment-16617102 ] Kazuaki Ishizaki commented on SPARK-25437: -- Is such a feature for major release, not for

[jira] [Created] (SPARK-25444) Refactor GenArrayData.genCodeToCreateArrayData() method

Kazuaki Ishizaki created SPARK-25444: Summary: Refactor GenArrayData.genCodeToCreateArrayData() method Key: SPARK-25444 URL: https://issues.apache.org/jira/browse/SPARK-25444 Project: Spark

[jira] [Assigned] (SPARK-24768) Have a built-in AVRO data source implementation

[ https://issues.apache.org/jira/browse/SPARK-24768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-24768: --- Assignee: Gengliang Wang > Have a built-in AVRO data source implementation >

[jira] [Resolved] (SPARK-24768) Have a built-in AVRO data source implementation

[ https://issues.apache.org/jira/browse/SPARK-24768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24768. - Resolution: Fixed Fix Version/s: 2.4.0 > Have a built-in AVRO data source implementation >

[jira] [Commented] (SPARK-23874) Upgrade apache/arrow to 0.10.0

[ https://issues.apache.org/jira/browse/SPARK-23874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617090#comment-16617090 ] Xiao Li commented on SPARK-23874: - [~bryanc] Thanks! It is very helpful. > Upgrade apache/arrow to

[jira] [Created] (SPARK-25443) fix issues when building docs with release scripts in docker

Wenchen Fan created SPARK-25443: --- Summary: fix issues when building docs with release scripts in docker Key: SPARK-25443 URL: https://issues.apache.org/jira/browse/SPARK-25443 Project: Spark

[jira] [Commented] (SPARK-25430) Add map parameter for withColumnRenamed

[ https://issues.apache.org/jira/browse/SPARK-25430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617066#comment-16617066 ] Goun Na commented on SPARK-25430: - Thanks [~hyukjin.kwon]. I will remember your guide. > Add map

[jira] [Closed] (SPARK-25435) df = sqlContext.read.json("examples/src/main/resources/people.json")

[ https://issues.apache.org/jira/browse/SPARK-25435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WEI PENG closed SPARK-25435. > df = sqlContext.read.json("examples/src/main/resources/people.json") >

[jira] [Resolved] (SPARK-25435) df = sqlContext.read.json("examples/src/main/resources/people.json")

[ https://issues.apache.org/jira/browse/SPARK-25435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WEI PENG resolved SPARK-25435. -- Resolution: Incomplete > df = sqlContext.read.json("examples/src/main/resources/people.json") >

[jira] [Closed] (SPARK-25434) failed to locate the winutils binary in the hadoop binary path

[ https://issues.apache.org/jira/browse/SPARK-25434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WEI PENG closed SPARK-25434. Problem resolved > failed to locate the winutils binary in the hadoop binary path >

[jira] [Commented] (SPARK-25442) Support STS to run in K8S deployment with spark deployment mode as cluster

[ https://issues.apache.org/jira/browse/SPARK-25442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617018#comment-16617018 ] Suryanarayana Garlapati commented on SPARK-25442: - There was an earlier 

[jira] [Commented] (SPARK-23200) Reset configuration when restarting from checkpoints

[ https://issues.apache.org/jira/browse/SPARK-23200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617016#comment-16617016 ] Wenchen Fan commented on SPARK-23200: - We should definitely merge it to branch 2.4, but I won't

[jira] [Commented] (SPARK-25437) Using OpenHashMap replace HashMap improve Encoder Performance

[ https://issues.apache.org/jira/browse/SPARK-25437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16616994#comment-16616994 ] Hyukjin Kwon commented on SPARK-25437: -- Please feel the JIRA description. > Using OpenHashMap

[jira] [Commented] (SPARK-25435) df = sqlContext.read.json("examples/src/main/resources/people.json")

[ https://issues.apache.org/jira/browse/SPARK-25435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16616993#comment-16616993 ] Hyukjin Kwon commented on SPARK-25435: -- Can you provide reproduible steps to verify this issue

[jira] [Comment Edited] (SPARK-25437) Using OpenHashMap replace HashMap improve Encoder Performance

[ https://issues.apache.org/jira/browse/SPARK-25437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16616994#comment-16616994 ] Hyukjin Kwon edited comment on SPARK-25437 at 9/17/18 2:13 AM: --- Please

[jira] [Commented] (SPARK-25433) Add support for PEX in PySpark

[ https://issues.apache.org/jira/browse/SPARK-25433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16616992#comment-16616992 ] Hyukjin Kwon commented on SPARK-25433: -- What's advantages of adding this and what are the

[jira] [Commented] (SPARK-25424) Window duration and slide duration with negative values should fail fast

[ https://issues.apache.org/jira/browse/SPARK-25424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16616984#comment-16616984 ] Hyukjin Kwon commented on SPARK-25424: -- Please avoid to set a target version which is usually

[jira] [Commented] (SPARK-25429) SparkListenerBus inefficient due to 'LiveStageMetrics#accumulatorIds:Array[Long]' data structure

[ https://issues.apache.org/jira/browse/SPARK-25429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16616985#comment-16616985 ] Hyukjin Kwon commented on SPARK-25429: -- PR https://github.com/apache/spark/pull/22420 >

[jira] [Updated] (SPARK-25424) Window duration and slide duration with negative values should fail fast

[ https://issues.apache.org/jira/browse/SPARK-25424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-25424: - Target Version/s: (was: 2.4.0) > Window duration and slide duration with negative values

[jira] [Commented] (SPARK-25430) Add map parameter for withColumnRenamed

[ https://issues.apache.org/jira/browse/SPARK-25430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16616981#comment-16616981 ] Hyukjin Kwon commented on SPARK-25430: -- Please avoid to set the target version which is usually

[jira] [Updated] (SPARK-25430) Add map parameter for withColumnRenamed

[ https://issues.apache.org/jira/browse/SPARK-25430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-25430: - Target Version/s: (was: 2.4.0) > Add map parameter for withColumnRenamed >

[jira] [Resolved] (SPARK-25420) Dataset.count() every time is different.

[ https://issues.apache.org/jira/browse/SPARK-25420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huanghuai resolved SPARK-25420. --- Resolution: Fixed > Dataset.count() every time is different. >

[jira] [Commented] (SPARK-25420) Dataset.count() every time is different.

[ https://issues.apache.org/jira/browse/SPARK-25420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16616978#comment-16616978 ] huanghuai commented on SPARK-25420: --- Because every time I use dropDuplicates() ，it's all in disorder,

[jira] [Commented] (SPARK-25423) Output "dataFilters" in DataSourceScanExec.metadata

[ https://issues.apache.org/jira/browse/SPARK-25423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16616955#comment-16616955 ] Dongjoon Hyun commented on SPARK-25423: --- [~yumwang]'s PR link is added. > Output "dataFilters" in

[jira] [Commented] (SPARK-25423) Output "dataFilters" in DataSourceScanExec.metadata

[ https://issues.apache.org/jira/browse/SPARK-25423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16616953#comment-16616953 ] Dongjoon Hyun commented on SPARK-25423: --- For the new feature, `Affects Version/s` should be the

[jira] [Assigned] (SPARK-25423) Output "dataFilters" in DataSourceScanExec.metadata

[ https://issues.apache.org/jira/browse/SPARK-25423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-25423: - Assignee: Yuming Wang > Output "dataFilters" in DataSourceScanExec.metadata >

[jira] [Updated] (SPARK-25423) Output "dataFilters" in DataSourceScanExec.metadata

[ https://issues.apache.org/jira/browse/SPARK-25423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25423: -- Affects Version/s: (was: 2.3.1) 2.5.0 > Output "dataFilters" in

[jira] [Issue Comment Deleted] (SPARK-25153) Improve error messages for columns with dots/periods

[ https://issues.apache.org/jira/browse/SPARK-25153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fernando Díaz updated SPARK-25153: -- Comment: was deleted (was: I will take a look at it. Quick question: Given a dataframe with

[jira] [Comment Edited] (SPARK-25380) Generated plans occupy over 50% of Spark driver memory

[ https://issues.apache.org/jira/browse/SPARK-25380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16616675#comment-16616675 ] Nir Hedvat edited comment on SPARK-25380 at 9/16/18 11:21 AM: -- Experiencing

[jira] [Commented] (SPARK-25380) Generated plans occupy over 50% of Spark driver memory

[ https://issues.apache.org/jira/browse/SPARK-25380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16616675#comment-16616675 ] Nir Hedvat commented on SPARK-25380: Same problem here (using Spark 2.3.1) > Generated plans occupy

[jira] [Commented] (SPARK-24315) Multiple streaming jobs detected error causing job failure

[ https://issues.apache.org/jira/browse/SPARK-24315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16616643#comment-16616643 ] Marco Gaido commented on SPARK-24315: - [~joeyfezster] it has been a while ago, so I may be wrong,

[jira] [Commented] (SPARK-24315) Multiple streaming jobs detected error causing job failure

[ https://issues.apache.org/jira/browse/SPARK-24315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16616622#comment-16616622 ] Joey commented on SPARK-24315: -- [~mgaido] , can you please explain why this is not a bug?  What could be

[jira] [Resolved] (SPARK-25391) Make behaviors consistent when converting parquet hive table to parquet data source

[ https://issues.apache.org/jira/browse/SPARK-25391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chenxiao Mao resolved SPARK-25391. -- Resolution: Won't Do > Make behaviors consistent when converting parquet hive table to