[jira] [Commented] (SPARK-24572) "eager execution" for R shell, IDE

[ https://issues.apache.org/jira/browse/SPARK-24572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16617120#comment-16617120 ] Felix Cheung commented on SPARK-24572: -- thanks! very close -  showDF doesn't return

[jira] [Commented] (SPARK-21914) Running examples as tests in SQL builtin function documentation

[ https://issues.apache.org/jira/browse/SPARK-21914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16617123#comment-16617123 ] Takuya Ueshin commented on SPARK-21914: --- Actually I was just thinking of the tests

[jira] [Comment Edited] (SPARK-21291) R bucketBy partitionBy API

[ https://issues.apache.org/jira/browse/SPARK-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16617118#comment-16617118 ] Felix Cheung edited comment on SPARK-21291 at 9/17/18 6:07 AM: ---

[jira] [Commented] (SPARK-21291) R bucketBy partitionBy API

[ https://issues.apache.org/jira/browse/SPARK-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16617118#comment-16617118 ] Felix Cheung commented on SPARK-21291: -- I think it should be like this one   [htt

[jira] [Commented] (SPARK-21914) Running examples as tests in SQL builtin function documentation

[ https://issues.apache.org/jira/browse/SPARK-21914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16617113#comment-16617113 ] Hyukjin Kwon commented on SPARK-21914: -- [~rxin], how about if I proceed this regard

[jira] [Closed] (SPARK-25391) Make behaviors consistent when converting parquet hive table to parquet data source

[ https://issues.apache.org/jira/browse/SPARK-25391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chenxiao Mao closed SPARK-25391. > Make behaviors consistent when converting parquet hive table to parquet data > source > ---

[jira] [Commented] (SPARK-25437) Using OpenHashMap replace HashMap improve Encoder Performance

[ https://issues.apache.org/jira/browse/SPARK-25437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16617102#comment-16617102 ] Kazuaki Ishizaki commented on SPARK-25437: -- Is such a feature for major release

[jira] [Created] (SPARK-25444) Refactor GenArrayData.genCodeToCreateArrayData() method

Kazuaki Ishizaki created SPARK-25444: Summary: Refactor GenArrayData.genCodeToCreateArrayData() method Key: SPARK-25444 URL: https://issues.apache.org/jira/browse/SPARK-25444 Project: Spark

[jira] [Assigned] (SPARK-24768) Have a built-in AVRO data source implementation

[ https://issues.apache.org/jira/browse/SPARK-24768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-24768: --- Assignee: Gengliang Wang > Have a built-in AVRO data source implementation > --

[jira] [Resolved] (SPARK-24768) Have a built-in AVRO data source implementation

[ https://issues.apache.org/jira/browse/SPARK-24768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24768. - Resolution: Fixed Fix Version/s: 2.4.0 > Have a built-in AVRO data source implementation > --

[jira] [Commented] (SPARK-23874) Upgrade apache/arrow to 0.10.0

[ https://issues.apache.org/jira/browse/SPARK-23874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16617090#comment-16617090 ] Xiao Li commented on SPARK-23874: - [~bryanc] Thanks! It is very helpful. > Upgrade apac

[jira] [Created] (SPARK-25443) fix issues when building docs with release scripts in docker

Wenchen Fan created SPARK-25443: --- Summary: fix issues when building docs with release scripts in docker Key: SPARK-25443 URL: https://issues.apache.org/jira/browse/SPARK-25443 Project: Spark I

[jira] [Commented] (SPARK-25430) Add map parameter for withColumnRenamed

[ https://issues.apache.org/jira/browse/SPARK-25430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16617066#comment-16617066 ] Goun Na commented on SPARK-25430: - Thanks [~hyukjin.kwon]. I will remember your guide.

[jira] [Closed] (SPARK-25435) df = sqlContext.read.json("examples/src/main/resources/people.json")

[ https://issues.apache.org/jira/browse/SPARK-25435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WEI PENG closed SPARK-25435. > df = sqlContext.read.json("examples/src/main/resources/people.json") > -

[jira] [Resolved] (SPARK-25435) df = sqlContext.read.json("examples/src/main/resources/people.json")

[ https://issues.apache.org/jira/browse/SPARK-25435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WEI PENG resolved SPARK-25435. -- Resolution: Incomplete > df = sqlContext.read.json("examples/src/main/resources/people.json") > --

[jira] [Closed] (SPARK-25434) failed to locate the winutils binary in the hadoop binary path

[ https://issues.apache.org/jira/browse/SPARK-25434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WEI PENG closed SPARK-25434. Problem resolved > failed to locate the winutils binary in the hadoop binary path > -

[jira] [Commented] (SPARK-25442) Support STS to run in K8S deployment with spark deployment mode as cluster

[ https://issues.apache.org/jira/browse/SPARK-25442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16617018#comment-16617018 ] Suryanarayana Garlapati commented on SPARK-25442: - There was an earlier 

[jira] [Commented] (SPARK-23200) Reset configuration when restarting from checkpoints

[ https://issues.apache.org/jira/browse/SPARK-23200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16617016#comment-16617016 ] Wenchen Fan commented on SPARK-23200: - We should definitely merge it to branch 2.4,

[jira] [Commented] (SPARK-25437) Using OpenHashMap replace HashMap improve Encoder Performance

[ https://issues.apache.org/jira/browse/SPARK-25437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616994#comment-16616994 ] Hyukjin Kwon commented on SPARK-25437: -- Please feel the JIRA description. > Using

[jira] [Commented] (SPARK-25435) df = sqlContext.read.json("examples/src/main/resources/people.json")

[ https://issues.apache.org/jira/browse/SPARK-25435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616993#comment-16616993 ] Hyukjin Kwon commented on SPARK-25435: -- Can you provide reproduible steps to verify

[jira] [Comment Edited] (SPARK-25437) Using OpenHashMap replace HashMap improve Encoder Performance

[ https://issues.apache.org/jira/browse/SPARK-25437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616994#comment-16616994 ] Hyukjin Kwon edited comment on SPARK-25437 at 9/17/18 2:13 AM: ---

[jira] [Commented] (SPARK-25433) Add support for PEX in PySpark

[ https://issues.apache.org/jira/browse/SPARK-25433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616992#comment-16616992 ] Hyukjin Kwon commented on SPARK-25433: -- What's advantages of adding this and what a

[jira] [Commented] (SPARK-25424) Window duration and slide duration with negative values should fail fast

[ https://issues.apache.org/jira/browse/SPARK-25424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616984#comment-16616984 ] Hyukjin Kwon commented on SPARK-25424: -- Please avoid to set a target version which

[jira] [Commented] (SPARK-25429) SparkListenerBus inefficient due to 'LiveStageMetrics#accumulatorIds:Array[Long]' data structure

[ https://issues.apache.org/jira/browse/SPARK-25429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616985#comment-16616985 ] Hyukjin Kwon commented on SPARK-25429: -- PR https://github.com/apache/spark/pull/224

[jira] [Updated] (SPARK-25424) Window duration and slide duration with negative values should fail fast

[ https://issues.apache.org/jira/browse/SPARK-25424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-25424: - Target Version/s: (was: 2.4.0) > Window duration and slide duration with negative values shoul

[jira] [Commented] (SPARK-25430) Add map parameter for withColumnRenamed

[ https://issues.apache.org/jira/browse/SPARK-25430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616981#comment-16616981 ] Hyukjin Kwon commented on SPARK-25430: -- Please avoid to set the target version whic

[jira] [Updated] (SPARK-25430) Add map parameter for withColumnRenamed

[ https://issues.apache.org/jira/browse/SPARK-25430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-25430: - Target Version/s: (was: 2.4.0) > Add map parameter for withColumnRenamed > ---

[jira] [Resolved] (SPARK-25420) Dataset.count() every time is different.

[ https://issues.apache.org/jira/browse/SPARK-25420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huanghuai resolved SPARK-25420. --- Resolution: Fixed > Dataset.count() every time is different. >

[jira] [Commented] (SPARK-25420) Dataset.count() every time is different.

[ https://issues.apache.org/jira/browse/SPARK-25420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616978#comment-16616978 ] huanghuai commented on SPARK-25420: --- Because every time I use dropDuplicates() ，it's a

[jira] [Commented] (SPARK-25423) Output "dataFilters" in DataSourceScanExec.metadata

[ https://issues.apache.org/jira/browse/SPARK-25423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616955#comment-16616955 ] Dongjoon Hyun commented on SPARK-25423: --- [~yumwang]'s PR link is added. > Output

[jira] [Commented] (SPARK-25423) Output "dataFilters" in DataSourceScanExec.metadata

[ https://issues.apache.org/jira/browse/SPARK-25423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616953#comment-16616953 ] Dongjoon Hyun commented on SPARK-25423: --- For the new feature, `Affects Version/s`

[jira] [Assigned] (SPARK-25423) Output "dataFilters" in DataSourceScanExec.metadata

[ https://issues.apache.org/jira/browse/SPARK-25423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-25423: - Assignee: Yuming Wang > Output "dataFilters" in DataSourceScanExec.metadata > -

[jira] [Updated] (SPARK-25423) Output "dataFilters" in DataSourceScanExec.metadata

[ https://issues.apache.org/jira/browse/SPARK-25423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25423: -- Affects Version/s: (was: 2.3.1) 2.5.0 > Output "dataFilters" in Dat

[jira] [Issue Comment Deleted] (SPARK-25153) Improve error messages for columns with dots/periods

[ https://issues.apache.org/jira/browse/SPARK-25153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fernando Díaz updated SPARK-25153: -- Comment: was deleted (was: I will take a look at it. Quick question: Given a dataframe with j

[jira] [Comment Edited] (SPARK-25380) Generated plans occupy over 50% of Spark driver memory

[ https://issues.apache.org/jira/browse/SPARK-25380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616675#comment-16616675 ] Nir Hedvat edited comment on SPARK-25380 at 9/16/18 11:21 AM:

[jira] [Commented] (SPARK-25380) Generated plans occupy over 50% of Spark driver memory

[ https://issues.apache.org/jira/browse/SPARK-25380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616675#comment-16616675 ] Nir Hedvat commented on SPARK-25380: Same problem here (using Spark 2.3.1) > Genera

[jira] [Commented] (SPARK-24315) Multiple streaming jobs detected error causing job failure

[ https://issues.apache.org/jira/browse/SPARK-24315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616643#comment-16616643 ] Marco Gaido commented on SPARK-24315: - [~joeyfezster] it has been a while ago, so I

[jira] [Commented] (SPARK-24315) Multiple streaming jobs detected error causing job failure

[ https://issues.apache.org/jira/browse/SPARK-24315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616622#comment-16616622 ] Joey commented on SPARK-24315: -- [~mgaido] , can you please explain why this is not a bug?