[jira] [Created] (SPARK-25437) Using OpenHashMap replace HashMap improve Encoder Performance

2018-09-14 Thread wangjiaochun (JIRA)
wangjiaochun created SPARK-25437: Summary: Using OpenHashMap replace HashMap improve Encoder Performance Key: SPARK-25437 URL: https://issues.apache.org/jira/browse/SPARK-25437 Project: Spark

[jira] [Created] (SPARK-25436) Bump master branch version to 2.5.0-SNAPSHOT

2018-09-14 Thread Xiao Li (JIRA)
Xiao Li created SPARK-25436: --- Summary: Bump master branch version to 2.5.0-SNAPSHOT Key: SPARK-25436 URL: https://issues.apache.org/jira/browse/SPARK-25436 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-25435) df = sqlContext.read.json("examples/src/main/resources/people.json")

2018-09-14 Thread WEI PENG (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16615602#comment-16615602 ] WEI PENG commented on SPARK-25435: -- So disappointing, anyone can help? > df = sqlConte

[jira] [Updated] (SPARK-25435) df = sqlContext.read.json("examples/src/main/resources/people.json")

2018-09-14 Thread WEI PENG (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WEI PENG updated SPARK-25435: - Component/s: PySpark > df = sqlContext.read.json("examples/src/main/resources/people.json") > --

[jira] [Created] (SPARK-25435) df = sqlContext.read.json("examples/src/main/resources/people.json")

2018-09-14 Thread WEI PENG (JIRA)
WEI PENG created SPARK-25435: Summary: df = sqlContext.read.json("examples/src/main/resources/people.json") Key: SPARK-25435 URL: https://issues.apache.org/jira/browse/SPARK-25435 Project: Spark

[jira] [Commented] (SPARK-23367) Include python document style checking

2018-09-14 Thread Rekha Joshi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16615570#comment-16615570 ] Rekha Joshi commented on SPARK-23367: - Updated PR https://github.com/apache/spark/pu

[jira] [Updated] (SPARK-25434) failed to locate the winutils binary in the hadoop binary path

2018-09-14 Thread WEI PENG (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WEI PENG updated SPARK-25434: - Component/s: PySpark > failed to locate the winutils binary in the hadoop binary path >

[jira] [Commented] (SPARK-25434) failed to locate the winutils binary in the hadoop binary path

2018-09-14 Thread WEI PENG (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16615550#comment-16615550 ] WEI PENG commented on SPARK-25434: -- I tried the whole night, but I can't figure it out,

[jira] [Created] (SPARK-25434) failed to locate the winutils binary in the hadoop binary path

2018-09-14 Thread WEI PENG (JIRA)
WEI PENG created SPARK-25434: Summary: failed to locate the winutils binary in the hadoop binary path Key: SPARK-25434 URL: https://issues.apache.org/jira/browse/SPARK-25434 Project: Spark Issue

[jira] [Resolved] (SPARK-25238) Lint-Python: Upgrading to the current version of pycodestyle fails

2018-09-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-25238. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22231 [https://github.c

[jira] [Assigned] (SPARK-25238) Lint-Python: Upgrading to the current version of pycodestyle fails

2018-09-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-25238: - Assignee: cclauss > Lint-Python: Upgrading to the current version of pycodestyle fails > --

[jira] [Updated] (SPARK-24233) Union Operation on Read of Dataframe does NOT produce correct result

2018-09-14 Thread smohr003 (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] smohr003 updated SPARK-24233: - Summary: Union Operation on Read of Dataframe does NOT produce correct result (was: union operation on

[jira] [Commented] (SPARK-14948) Exception when joining DataFrames derived form the same DataFrame

2018-09-14 Thread Ashish Shrowty (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16615434#comment-16615434 ] Ashish Shrowty commented on SPARK-14948: I have hit this issue and is blocking s

[jira] [Commented] (SPARK-25246) When the spark.eventLog.compress is enabled, the Application is not showing in the History server UI ('incomplete application' page), initially.

2018-09-14 Thread Devaraj K (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16615398#comment-16615398 ] Devaraj K commented on SPARK-25246: --- I think it is not a problem, the behavior might b

[jira] [Updated] (SPARK-19480) Higher order functions in SQL

2018-09-14 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-19480: Fix Version/s: 2.4.0 > Higher order functions in SQL > - > > K

[jira] [Updated] (SPARK-25433) Add support for PEX in PySpark

2018-09-14 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-25433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Höring updated SPARK-25433: -- Description: This has been partly discussed in SPARK-13587 I would like to provision the exec

[jira] [Updated] (SPARK-25433) Add support for PEX in PySpark

2018-09-14 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-25433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Höring updated SPARK-25433: -- Description: This has been partly discussed in SPARK-13587 I would like to provision the exec

[jira] [Updated] (SPARK-25433) Add support for PEX in PySpark

2018-09-14 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-25433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Höring updated SPARK-25433: -- Description: This has been partly discussed in SPARK-13587 I would like to provision the exec

[jira] [Updated] (SPARK-25433) Add support for PEX in PySpark

2018-09-14 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-25433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Höring updated SPARK-25433: -- Description: This has been partly discussed in SPARK-13587 I would like to provision the exec

[jira] [Created] (SPARK-25433) Add support for PEX in PySpark

2018-09-14 Thread JIRA
Fabian Höring created SPARK-25433: - Summary: Add support for PEX in PySpark Key: SPARK-25433 URL: https://issues.apache.org/jira/browse/SPARK-25433 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-25344) Break large tests.py files into smaller files

2018-09-14 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16615091#comment-16615091 ] Imran Rashid commented on SPARK-25344: -- {quote} 1. When to create a separate test f

[jira] [Resolved] (SPARK-25431) Fix function examples and unify the format of the example results.

2018-09-14 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-25431. - Resolution: Fixed > Fix function examples and unify the format of the example results. > ---

[jira] [Assigned] (SPARK-25431) Fix function examples and unify the format of the example results.

2018-09-14 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-25431: --- Assignee: Takuya Ueshin > Fix function examples and unify the format of the example results. >

[jira] [Updated] (SPARK-25431) Fix function examples and unify the format of the example results.

2018-09-14 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25431: Fix Version/s: 2.4.0 > Fix function examples and unify the format of the example results. > --

[jira] [Created] (SPARK-25432) Consider if using standard getOrCreate from PySpark into JVM SparkSession would simplify code

2018-09-14 Thread holdenk (JIRA)
holdenk created SPARK-25432: --- Summary: Consider if using standard getOrCreate from PySpark into JVM SparkSession would simplify code Key: SPARK-25432 URL: https://issues.apache.org/jira/browse/SPARK-25432 P

[jira] [Updated] (SPARK-23899) Built-in SQL Function Improvement

2018-09-14 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23899: Target Version/s: 2.4.0 (was: 2.4.0, 3.0.0) > Built-in SQL Function Improvement > ---

[jira] [Commented] (SPARK-25431) Fix function examples and unify the format of the example results.

2018-09-14 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16614865#comment-16614865 ] Liang-Chi Hsieh commented on SPARK-25431: - Don't know why the PR link is not att

[jira] [Commented] (SPARK-24410) Missing optimization for Union on bucketed tables

2018-09-14 Thread Eyal Farago (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16614683#comment-16614683 ] Eyal Farago commented on SPARK-24410: - [~viirya], I see that the PR is now closed (p

[jira] [Commented] (SPARK-24410) Missing optimization for Union on bucketed tables

2018-09-14 Thread Eyal Farago (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16614678#comment-16614678 ] Eyal Farago commented on SPARK-24410: - [~viirya], I've opened SPARK-25203 because of

[jira] [Comment Edited] (SPARK-21652) Optimizer cannot reach a fixed point on certain queries

2018-09-14 Thread Pengfei Chang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16614664#comment-16614664 ] Pengfei Chang edited comment on SPARK-21652 at 9/14/18 11:09 AM: -

[jira] [Commented] (SPARK-21652) Optimizer cannot reach a fixed point on certain queries

2018-09-14 Thread Pengfei Chang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16614664#comment-16614664 ] Pengfei Chang commented on SPARK-21652: --- Hi, after this change, there are some cas

[jira] [Updated] (SPARK-25430) Add map parameter for withColumnRenamed

2018-09-14 Thread Goun Na (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Goun Na updated SPARK-25430: Description: WithColumnRenamed method should work with map parameter. It removes code redundancy. {code:j

[jira] [Updated] (SPARK-25426) Remove the duplicate fallback logic in UnsafeProjection

2018-09-14 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-25426: - Summary: Remove the duplicate fallback logic in UnsafeProjection (was: Handles subexpre

[jira] [Commented] (SPARK-25339) Refactor FilterPushdownBenchmark to use main method

2018-09-14 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16614585#comment-16614585 ] Yuming Wang commented on SPARK-25339: - Thank you [~dongjoon], I'll start next week.

[jira] [Updated] (SPARK-25431) Fix function examples and unify the format of the example results.

2018-09-14 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-25431: -- Summary: Fix function examples and unify the format of the example results. (was: Fix functio

[jira] [Created] (SPARK-25431) Fix function examples and unify the format of the functions results.

2018-09-14 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-25431: - Summary: Fix function examples and unify the format of the functions results. Key: SPARK-25431 URL: https://issues.apache.org/jira/browse/SPARK-25431 Project: Spark

[jira] [Commented] (SPARK-25430) Add map parameter for withColumnRenamed

2018-09-14 Thread Goun Na (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16614530#comment-16614530 ] Goun Na commented on SPARK-25430: - I am working on it. > Add map parameter for withColu

[jira] [Created] (SPARK-25430) Add map parameter for withColumnRenamed

2018-09-14 Thread Goun Na (JIRA)
Goun Na created SPARK-25430: --- Summary: Add map parameter for withColumnRenamed Key: SPARK-25430 URL: https://issues.apache.org/jira/browse/SPARK-25430 Project: Spark Issue Type: Improvement

[jira] [Comment Edited] (SPARK-25374) SafeProjection supports fallback to an interpreted mode

2018-09-14 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16614458#comment-16614458 ] Liang-Chi Hsieh edited comment on SPARK-25374 at 9/14/18 8:03 AM:

[jira] [Updated] (SPARK-25429) SparkListenerBus inefficient due to 'LiveStageMetrics#accumulatorIds:Array[Long]' data structure

2018-09-14 Thread DENG FEI (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DENG FEI updated SPARK-25429: - Description: {code:java} private def updateStageMetrics( stageId: Int, attemptId: Int,

[jira] [Updated] (SPARK-25429) SparkListenerBus inefficient due to 'LiveStageMetrics#accumulatorIds:Array[Long]' data structure

2018-09-14 Thread DENG FEI (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DENG FEI updated SPARK-25429: - Description: {code:java} private def updateStageMetrics( stageId: Int, attemptId: Int,

[jira] [Commented] (SPARK-25374) SafeProjection supports fallback to an interpreted mode

2018-09-14 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16614458#comment-16614458 ] Liang-Chi Hsieh commented on SPARK-25374: - Though this is not a bug fix, will we