[jira] [Updated] (SPARK-25084) "distribute by" on multiple columns may lead to codegen issue

2018-08-09 Thread yucai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yucai updated SPARK-25084: -- Description: Test Query: {code:java} select * from store_sales distribute by (ss_sold_time_sk, ss_item_sk, ss

[jira] [Commented] (SPARK-25084) "distribute by" on multiple columns may lead to codegen issue

2018-08-09 Thread yucai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575808#comment-16575808 ] yucai commented on SPARK-25084: --- It is a regression, when the generated codes size is more

[jira] [Commented] (SPARK-25084) "distribute by" on multiple columns may lead to codegen issue

2018-08-09 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575791#comment-16575791 ] Xiao Li commented on SPARK-25084: - Could you investigate which PR introduced this bug? W

[jira] [Commented] (SPARK-25084) "distribute by" on multiple columns may lead to codegen issue

2018-08-09 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575790#comment-16575790 ] Xiao Li commented on SPARK-25084: - Let us mark it as a blocker. How about the master bra

[jira] [Updated] (SPARK-25084) "distribute by" on multiple columns may lead to codegen issue

2018-08-09 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25084: Priority: Blocker (was: Major) > "distribute by" on multiple columns may lead to codegen issue >

[jira] [Commented] (SPARK-25084) "distribute by" on multiple columns may lead to codegen issue

2018-08-09 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575778#comment-16575778 ] Saisai Shao commented on SPARK-25084: - I see. Unfortunately I've cut the RC4, if it

[jira] [Commented] (SPARK-25084) "distribute by" on multiple columns may lead to codegen issue

2018-08-09 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575777#comment-16575777 ] Yuming Wang commented on SPARK-25084: - It's a regression. > "distribute by" on mult

[jira] [Commented] (SPARK-25084) "distribute by" on multiple columns may lead to codegen issue

2018-08-09 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575774#comment-16575774 ] Saisai Shao commented on SPARK-25084: - Is this a regression or just a bug existed in

[jira] [Updated] (SPARK-25067) Active tasks exceed total cores of an executor in WebUI

2018-08-09 Thread StanZhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] StanZhai updated SPARK-25067: - Attachment: (was: 1533128203469_2.png) > Active tasks exceed total cores of an executor in WebUI > -

[jira] [Commented] (SPARK-25084) "distribute by" on multiple columns may lead to codegen issue

2018-08-09 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575772#comment-16575772 ] Saisai Shao commented on SPARK-25084: - I'm already preparing new RC4. If this is not

[jira] [Commented] (SPARK-25084) "distribute by" on multiple columns may lead to codegen issue

2018-08-09 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575771#comment-16575771 ] Yuming Wang commented on SPARK-25084: - [~smilegator], [~jerryshao] I think It should

[jira] [Assigned] (SPARK-25084) "distribute by" on multiple columns may lead to codegen issue

2018-08-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25084: Assignee: Apache Spark > "distribute by" on multiple columns may lead to codegen issue >

[jira] [Assigned] (SPARK-25084) "distribute by" on multiple columns may lead to codegen issue

2018-08-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25084: Assignee: (was: Apache Spark) > "distribute by" on multiple columns may lead to codeg

[jira] [Commented] (SPARK-25084) "distribute by" on multiple columns may lead to codegen issue

2018-08-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575767#comment-16575767 ] Apache Spark commented on SPARK-25084: -- User 'yucai' has created a pull request for

[jira] [Created] (SPARK-25084) "distribute by" on multiple columns may lead to codegen issue

2018-08-09 Thread yucai (JIRA)
yucai created SPARK-25084: - Summary: "distribute by" on multiple columns may lead to codegen issue Key: SPARK-25084 URL: https://issues.apache.org/jira/browse/SPARK-25084 Project: Spark Issue Type:

[jira] [Commented] (SPARK-23992) ShuffleDependency does not need to be deserialized every time

2018-08-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575747#comment-16575747 ] Apache Spark commented on SPARK-23992: -- User '10110346' has created a pull request

[jira] [Closed] (SPARK-25052) Is there any possibility that spark structured streaming generate duplicates in the output?

2018-08-09 Thread bharath kumar avusherla (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bharath kumar avusherla closed SPARK-25052. --- > Is there any possibility that spark structured streaming generate duplicates

[jira] [Updated] (SPARK-23243) Shuffle+Repartition on an RDD could lead to incorrect answers

2018-08-09 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-23243: Labels: correctness (was: ) > Shuffle+Repartition on an RDD could lead to incorrect answers > ---

[jira] [Commented] (SPARK-25044) Address translation of LMF closure primitive args to Object in Scala 2.12

2018-08-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575672#comment-16575672 ] Apache Spark commented on SPARK-25044: -- User 'srowen' has created a pull request fo

[jira] [Assigned] (SPARK-25044) Address translation of LMF closure primitive args to Object in Scala 2.12

2018-08-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25044: Assignee: (was: Apache Spark) > Address translation of LMF closure primitive args to

[jira] [Assigned] (SPARK-25044) Address translation of LMF closure primitive args to Object in Scala 2.12

2018-08-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25044: Assignee: Apache Spark > Address translation of LMF closure primitive args to Object in S

[jira] [Created] (SPARK-25083) remove the type erasure hack in data source scan

2018-08-09 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-25083: --- Summary: remove the type erasure hack in data source scan Key: SPARK-25083 URL: https://issues.apache.org/jira/browse/SPARK-25083 Project: Spark Issue Type: Im

[jira] [Updated] (SPARK-25076) SQLConf should not be retrieved from a stopped SparkSession

2018-08-09 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-25076: Fix Version/s: 2.3.2 > SQLConf should not be retrieved from a stopped SparkSession > -

[jira] [Commented] (SPARK-24502) flaky test: UnsafeRowSerializerSuite

2018-08-09 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575642#comment-16575642 ] Wenchen Fan commented on SPARK-24502: - This is enough. The places you need to pay at

[jira] [Resolved] (SPARK-24886) Increase Jenkins build time

2018-08-09 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24886. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21845 [https://gi

[jira] [Commented] (SPARK-22236) CSV I/O: does not respect RFC 4180

2018-08-09 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575629#comment-16575629 ] Steve Loughran commented on SPARK-22236: I wouldn't recommend changing multiline

[jira] [Created] (SPARK-25082) Documentation for Spark Function expm1 is incomplete

2018-08-09 Thread Alexander Belov (JIRA)
Alexander Belov created SPARK-25082: --- Summary: Documentation for Spark Function expm1 is incomplete Key: SPARK-25082 URL: https://issues.apache.org/jira/browse/SPARK-25082 Project: Spark Is

[jira] [Updated] (SPARK-25081) Nested spill in ShuffleExternalSorter may access a released memory page

2018-08-09 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25081: - Labels: correctness (was: ) > Nested spill in ShuffleExternalSorter may access a released memor

[jira] [Assigned] (SPARK-25081) Nested spill in ShuffleExternalSorter may access a released memory page

2018-08-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25081: Assignee: Apache Spark (was: Shixiong Zhu) > Nested spill in ShuffleExternalSorter may a

[jira] [Commented] (SPARK-25081) Nested spill in ShuffleExternalSorter may access a released memory page

2018-08-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575598#comment-16575598 ] Apache Spark commented on SPARK-25081: -- User 'zsxwing' has created a pull request f

[jira] [Assigned] (SPARK-25081) Nested spill in ShuffleExternalSorter may access a released memory page

2018-08-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25081: Assignee: Shixiong Zhu (was: Apache Spark) > Nested spill in ShuffleExternalSorter may a

[jira] [Updated] (SPARK-25081) Nested spill in ShuffleExternalSorter may access a released memory page

2018-08-09 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25081: - Description: This issue is pretty similar to SPARK-21907. "allocateArray" in [ShuffleInMemoryS

[jira] [Updated] (SPARK-25081) Nested spill in ShuffleExternalSorter may access a released memory page

2018-08-09 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25081: - Description: This issue is pretty similar to SPARK-21907. "allocateArray" in [ShuffleInMemoryS

[jira] [Updated] (SPARK-25081) Nested spill in ShuffleExternalSorter may access a released memory page

2018-08-09 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25081: - Description: This issue is pretty similar to SPARK-21907. "allocateArray" in [ShuffleInMemoryS

[jira] [Created] (SPARK-25081) Nested spill in ShuffleExternalSorter may access a released memory page

2018-08-09 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-25081: Summary: Nested spill in ShuffleExternalSorter may access a released memory page Key: SPARK-25081 URL: https://issues.apache.org/jira/browse/SPARK-25081 Project: Spa

[jira] [Commented] (SPARK-25079) [PYTHON] upgrade python 3.4 -> 3.5

2018-08-09 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575554#comment-16575554 ] shane knapp commented on SPARK-25079: - i've confirmed, for the master branch at leas

[jira] [Commented] (SPARK-24950) scala DateTimeUtilsSuite daysToMillis and millisToDays fails w/java 8 181-b13

2018-08-09 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575521#comment-16575521 ] shane knapp commented on SPARK-24950: - do we care about 2.0 and 1.6? > scala DateTi

[jira] [Resolved] (SPARK-24950) scala DateTimeUtilsSuite daysToMillis and millisToDays fails w/java 8 181-b13

2018-08-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-24950. --- Resolution: Fixed Fix Version/s: 2.3.2 2.2.3 2.1.4 > sc

[jira] [Commented] (SPARK-24950) scala DateTimeUtilsSuite daysToMillis and millisToDays fails w/java 8 181-b13

2018-08-09 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575504#comment-16575504 ] shane knapp commented on SPARK-24950: - booyah!  i love watching the build queue pile

[jira] [Commented] (SPARK-24950) scala DateTimeUtilsSuite daysToMillis and millisToDays fails w/java 8 181-b13

2018-08-09 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575493#comment-16575493 ] shane knapp commented on SPARK-24950: - word.  let's leave this open until for a bit

[jira] [Commented] (SPARK-24950) scala DateTimeUtilsSuite daysToMillis and millisToDays fails w/java 8 181-b13

2018-08-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575488#comment-16575488 ] Sean Owen commented on SPARK-24950: --- Roger that, will back-port back to 2.1 as best I

[jira] [Commented] (SPARK-24950) scala DateTimeUtilsSuite daysToMillis and millisToDays fails w/java 8 181-b13

2018-08-09 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575479#comment-16575479 ] shane knapp commented on SPARK-24950: - [~srowen] [~d80tb7] i'm thinking that we actu

[jira] [Updated] (SPARK-24950) scala DateTimeUtilsSuite daysToMillis and millisToDays fails w/java 8 181-b13

2018-08-09 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shane knapp updated SPARK-24950: Affects Version/s: 2.1.3 > scala DateTimeUtilsSuite daysToMillis and millisToDays fails w/java 8 1

[jira] [Reopened] (SPARK-24950) scala DateTimeUtilsSuite daysToMillis and millisToDays fails w/java 8 181-b13

2018-08-09 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shane knapp reopened SPARK-24950: - > scala DateTimeUtilsSuite daysToMillis and millisToDays fails w/java 8 181-b13 > --

[jira] [Resolved] (SPARK-25068) High-order function: exists(array, function) → boolean

2018-08-09 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-25068. - Resolution: Fixed Fix Version/s: 2.4.0 > High-order function: exists(array, function) → boolean >

[jira] [Resolved] (SPARK-25076) SQLConf should not be retrieved from a stopped SparkSession

2018-08-09 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-25076. - Resolution: Fixed Fix Version/s: 2.4.0 > SQLConf should not be retrieved from a stopped SparkSess

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 0.10.0.1 to 2.0.0

2018-08-09 Thread Steve Bairos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575373#comment-16575373 ] Steve Bairos commented on SPARK-18057: -- Hey, long shot but any chance this change c

[jira] [Resolved] (SPARK-25077) Delete unused variable in WindowExec

2018-08-09 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-25077. - Resolution: Fixed Assignee: Li Yuanjian Fix Version/s: 2.4.0 > Delete unused variable in

[jira] [Updated] (SPARK-23298) distinct.count on Dataset/DataFrame yields non-deterministic results

2018-08-09 Thread Mateusz Jukiewicz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mateusz Jukiewicz updated SPARK-23298: -- Labels: Correctness CorrectnessBug correctness (was: CorrectnessBug correctness) > d

[jira] [Updated] (SPARK-23298) distinct.count on Dataset/DataFrame yields non-deterministic results

2018-08-09 Thread Mateusz Jukiewicz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mateusz Jukiewicz updated SPARK-23298: -- Labels: CorrectnessBug correctness (was: ) > distinct.count on Dataset/DataFrame yiel

[jira] [Commented] (SPARK-22236) CSV I/O: does not respect RFC 4180

2018-08-09 Thread Joe Pallas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575305#comment-16575305 ] Joe Pallas commented on SPARK-22236: If this can't change before 3.0, how about a no

[jira] [Commented] (SPARK-25059) Exception while executing an action on DataFrame that read Json

2018-08-09 Thread Kunal Goswami (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575288#comment-16575288 ] Kunal Goswami commented on SPARK-25059: --- Thank you so much for the prompt response

[jira] [Created] (SPARK-25080) NPE in HiveShim$.toCatalystDecimal(HiveShim.scala:110)

2018-08-09 Thread Andrew K Long (JIRA)
Andrew K Long created SPARK-25080: - Summary: NPE in HiveShim$.toCatalystDecimal(HiveShim.scala:110) Key: SPARK-25080 URL: https://issues.apache.org/jira/browse/SPARK-25080 Project: Spark Issu

[jira] [Commented] (SPARK-25024) Update mesos documentation to be clear about security supported

2018-08-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575267#comment-16575267 ] Thomas Graves commented on SPARK-25024: --- ok, I'm not familiar with mesos hardly at

[jira] [Commented] (SPARK-25044) Address translation of LMF closure primitive args to Object in Scala 2.12

2018-08-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575247#comment-16575247 ] Sean Owen commented on SPARK-25044: --- Next thought: use ScalaUDF's inputTypes field to

[jira] [Commented] (SPARK-25079) [PYTHON] upgrade python 3.4 -> 3.5

2018-08-09 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575197#comment-16575197 ] shane knapp commented on SPARK-25079: - SO.  MANY.  MOVING.  PARTS. > [PYTHON] upgra

[jira] [Assigned] (SPARK-25079) [PYTHON] upgrade python 3.4 -> 3.5

2018-08-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25079: Assignee: Apache Spark (was: shane knapp) > [PYTHON] upgrade python 3.4 -> 3.5 > ---

[jira] [Commented] (SPARK-25079) [PYTHON] upgrade python 3.4 -> 3.5

2018-08-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575186#comment-16575186 ] Apache Spark commented on SPARK-25079: -- User 'shaneknapp' has created a pull reques

[jira] [Assigned] (SPARK-25079) [PYTHON] upgrade python 3.4 -> 3.5

2018-08-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25079: Assignee: shane knapp (was: Apache Spark) > [PYTHON] upgrade python 3.4 -> 3.5 > ---

[jira] [Commented] (SPARK-23874) Upgrade apache/arrow to 0.10.0

2018-08-09 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575170#comment-16575170 ] shane knapp commented on SPARK-23874: - this issue depends on this:  https://issues.a

[jira] [Created] (SPARK-25079) [PYTHON] upgrade python 3.4 -> 3.5

2018-08-09 Thread shane knapp (JIRA)
shane knapp created SPARK-25079: --- Summary: [PYTHON] upgrade python 3.4 -> 3.5 Key: SPARK-25079 URL: https://issues.apache.org/jira/browse/SPARK-25079 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-25036) Scala 2.12 issues: Compilation error with sbt

2018-08-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25036: Assignee: Apache Spark (was: Kazuaki Ishizaki) > Scala 2.12 issues: Compilation error wi

[jira] [Assigned] (SPARK-25036) Scala 2.12 issues: Compilation error with sbt

2018-08-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25036: Assignee: Kazuaki Ishizaki (was: Apache Spark) > Scala 2.12 issues: Compilation error wi

[jira] [Commented] (SPARK-25036) Scala 2.12 issues: Compilation error with sbt

2018-08-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575158#comment-16575158 ] Apache Spark commented on SPARK-25036: -- User 'kiszk' has created a pull request for

[jira] [Updated] (SPARK-25036) Scala 2.12 issues: Compilation error with sbt

2018-08-09 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-25036: - Description: When compiling with sbt, the following errors occur: There are -two- three

[jira] [Comment Edited] (SPARK-25036) Scala 2.12 issues: Compilation error with sbt

2018-08-09 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575137#comment-16575137 ] Kazuaki Ishizaki edited comment on SPARK-25036 at 8/9/18 5:05 PM:

[jira] [Reopened] (SPARK-25036) Scala 2.12 issues: Compilation error with sbt

2018-08-09 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki reopened SPARK-25036: -- Another type of compilation error is found > Scala 2.12 issues: Compilation error with sb

[jira] [Commented] (SPARK-25024) Update mesos documentation to be clear about security supported

2018-08-09 Thread Arthur Rand (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575134#comment-16575134 ] Arthur Rand commented on SPARK-25024: - Just to chime in here. Unless a lot has chang

[jira] [Commented] (SPARK-25059) Exception while executing an action on DataFrame that read Json

2018-08-09 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575129#comment-16575129 ] Kazuaki Ishizaki commented on SPARK-25059: -- Thank you for reporting the issue.

[jira] [Commented] (SPARK-25036) Scala 2.12 issues: Compilation error with sbt

2018-08-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575116#comment-16575116 ] Apache Spark commented on SPARK-25036: -- User 'kiszk' has created a pull request for

[jira] [Commented] (SPARK-23207) Shuffle+Repartition on an DataFrame could lead to incorrect answers

2018-08-09 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575097#comment-16575097 ] Imran Rashid commented on SPARK-23207: -- yeah I agree with Tom, silent data loss is

[jira] [Commented] (SPARK-22634) Update Bouncy castle dependency

2018-08-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575066#comment-16575066 ] Sean Owen commented on SPARK-22634: --- That text is part of Netty's NOTICE file, which m

[jira] [Updated] (SPARK-25078) Standalone does not work with spark.authenticate.secret and deploy-mode=cluster

2018-08-09 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-25078: - Summary: Standalone does not work with spark.authenticate.secret and deploy-mode=cluster (was:

[jira] [Created] (SPARK-25078) Standalone cluster mode does not work with spark.authenticate.secret

2018-08-09 Thread Imran Rashid (JIRA)
Imran Rashid created SPARK-25078: Summary: Standalone cluster mode does not work with spark.authenticate.secret Key: SPARK-25078 URL: https://issues.apache.org/jira/browse/SPARK-25078 Project: Spark

[jira] [Assigned] (SPARK-25077) Delete unused variable in WindowExec

2018-08-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25077: Assignee: Apache Spark > Delete unused variable in WindowExec > -

[jira] [Commented] (SPARK-25077) Delete unused variable in WindowExec

2018-08-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575053#comment-16575053 ] Apache Spark commented on SPARK-25077: -- User 'xuanyuanking' has created a pull requ

[jira] [Assigned] (SPARK-25077) Delete unused variable in WindowExec

2018-08-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25077: Assignee: (was: Apache Spark) > Delete unused variable in WindowExec > --

[jira] [Created] (SPARK-25077) Delete unused variable in WindowExec

2018-08-09 Thread Li Yuanjian (JIRA)
Li Yuanjian created SPARK-25077: --- Summary: Delete unused variable in WindowExec Key: SPARK-25077 URL: https://issues.apache.org/jira/browse/SPARK-25077 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

2018-08-09 Thread Kyle Prifogle (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575044#comment-16575044 ] Kyle Prifogle commented on SPARK-12449: --- [~oae]  as far as I can tell the issue li

[jira] [Commented] (SPARK-24502) flaky test: UnsafeRowSerializerSuite

2018-08-09 Thread Paul Praet (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575026#comment-16575026 ] Paul Praet commented on SPARK-24502: We are only creating SparkSessions with a Spark

[jira] [Resolved] (SPARK-25063) Rename class KnowNotNull to KnownNotNull

2018-08-09 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-25063. - Resolution: Fixed Assignee: Maryann Xue Fix Version/s: 2.4.0 > Rename class KnowNotNull

[jira] [Commented] (SPARK-23207) Shuffle+Repartition on an DataFrame could lead to incorrect answers

2018-08-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574956#comment-16574956 ] Thomas Graves commented on SPARK-23207: --- ok, I guess I disagree with that. Any cor

[jira] [Commented] (SPARK-25076) SQLConf should not be retrieved from a stopped SparkSession

2018-08-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574915#comment-16574915 ] Apache Spark commented on SPARK-25076: -- User 'cloud-fan' has created a pull request

[jira] [Assigned] (SPARK-25076) SQLConf should not be retrieved from a stopped SparkSession

2018-08-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25076: Assignee: Wenchen Fan (was: Apache Spark) > SQLConf should not be retrieved from a stopp

[jira] [Assigned] (SPARK-25076) SQLConf should not be retrieved from a stopped SparkSession

2018-08-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25076: Assignee: Apache Spark (was: Wenchen Fan) > SQLConf should not be retrieved from a stopp

[jira] [Commented] (SPARK-24502) flaky test: UnsafeRowSerializerSuite

2018-08-09 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574908#comment-16574908 ] Wenchen Fan commented on SPARK-24502: - There is no resource leakage. Users need to m

[jira] [Commented] (SPARK-23207) Shuffle+Repartition on an DataFrame could lead to incorrect answers

2018-08-09 Thread Jiang Xingbo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574903#comment-16574903 ] Jiang Xingbo commented on SPARK-23207: -- This affects the 2.2 and lower versions, th

[jira] [Created] (SPARK-25076) SQLConf should not be retrieved from a stopped SparkSession

2018-08-09 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-25076: --- Summary: SQLConf should not be retrieved from a stopped SparkSession Key: SPARK-25076 URL: https://issues.apache.org/jira/browse/SPARK-25076 Project: Spark Is

[jira] [Commented] (SPARK-23207) Shuffle+Repartition on an DataFrame could lead to incorrect answers

2018-08-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574886#comment-16574886 ] Thomas Graves commented on SPARK-23207: --- [~jiangxb1987] ^ > Shuffle+Repartition o

[jira] [Commented] (SPARK-25073) Spark-submit on Yarn Task : When the yarn.nodemanager.resource.memory-mb and/or yarn.scheduler.maximum-allocation-mb is insufficient, Spark always reports an error req

2018-08-09 Thread Sujith (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574864#comment-16574864 ] Sujith commented on SPARK-25073: Seems to be you are right, Message is bit misleading to

[jira] [Commented] (SPARK-25044) Address translation of LMF closure primitive args to Object in Scala 2.12

2018-08-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574841#comment-16574841 ] Sean Owen commented on SPARK-25044: --- I tried this – it's not hard – but the implementa

[jira] [Commented] (SPARK-25075) Build and test Spark against Scala 2.13

2018-08-09 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574839#comment-16574839 ] Guillaume Massé commented on SPARK-25075: - Scala 2.13 is currently in the milest

[jira] [Created] (SPARK-25075) Build and test Spark against Scala 2.13

2018-08-09 Thread JIRA
Guillaume Massé created SPARK-25075: --- Summary: Build and test Spark against Scala 2.13 Key: SPARK-25075 URL: https://issues.apache.org/jira/browse/SPARK-25075 Project: Spark Issue Type: Umb

[jira] [Resolved] (SPARK-25047) Can't assign SerializedLambda to scala.Function1 in deserialization of BucketedRandomProjectionLSHModel

2018-08-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-25047. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22032 [https://github.c

[jira] [Assigned] (SPARK-25047) Can't assign SerializedLambda to scala.Function1 in deserialization of BucketedRandomProjectionLSHModel

2018-08-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-25047: - Assignee: Sean Owen > Can't assign SerializedLambda to scala.Function1 in deserialization of >

[jira] [Updated] (SPARK-25073) Spark-submit on Yarn Task : When the yarn.nodemanager.resource.memory-mb and/or yarn.scheduler.maximum-allocation-mb is insufficient, Spark always reports an error reque

2018-08-09 Thread vivek kumar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vivek kumar updated SPARK-25073: Description: When the yarn.nodemanager.resource.memory-mb and/or yarn.scheduler.maximum-allocatio

[jira] [Created] (SPARK-25074) Implement maxNumConcurrentTasks() in MesosFineGrainedSchedulerBackend

2018-08-09 Thread Jiang Xingbo (JIRA)
Jiang Xingbo created SPARK-25074: Summary: Implement maxNumConcurrentTasks() in MesosFineGrainedSchedulerBackend Key: SPARK-25074 URL: https://issues.apache.org/jira/browse/SPARK-25074 Project: Spark

[jira] [Assigned] (SPARK-23415) BufferHolderSparkSubmitSuite is flaky

2018-08-09 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-23415: --- Assignee: Kazuaki Ishizaki > BufferHolderSparkSubmitSuite is flaky > --

[jira] [Resolved] (SPARK-23415) BufferHolderSparkSubmitSuite is flaky

2018-08-09 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-23415. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20636 [https://gith

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

2018-08-09 Thread Johannes Zillmann (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574770#comment-16574770 ] Johannes Zillmann commented on SPARK-12449: --- I'm a bit confused. Reading http

  1   2   >