[jira] [Commented] (SPARK-23989) When using `SortShuffleWriter`, the data will be overwritten

2018-04-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441997#comment-16441997 ] Wenchen Fan commented on SPARK-23989: - You have to provide an end-to-end use case to

[jira] [Updated] (SPARK-23989) When using `SortShuffleWriter`, the data will be overwritten

2018-04-17 Thread liuxian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuxian updated SPARK-23989: Attachment: (was: 无标题2.png) > When using `SortShuffleWriter`, the data will be overwritten > --

[jira] [Commented] (SPARK-23989) When using `SortShuffleWriter`, the data will be overwritten

2018-04-17 Thread liuxian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441982#comment-16441982 ] liuxian commented on SPARK-23989: - We assume that: numPartitions > {color:#9876aa}MAX_SH

[jira] [Commented] (SPARK-23989) When using `SortShuffleWriter`, the data will be overwritten

2018-04-17 Thread liuxian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441980#comment-16441980 ] liuxian commented on SPARK-23989: - {color:#9876aa}I think '{color:#33}SortShuffleWrit

[jira] [Commented] (SPARK-23989) When using `SortShuffleWriter`, the data will be overwritten

2018-04-17 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441957#comment-16441957 ] Saisai Shao commented on SPARK-23989: - I've no idea what are you trying to express.

[jira] [Comment Edited] (SPARK-23989) When using `SortShuffleWriter`, the data will be overwritten

2018-04-17 Thread liuxian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441952#comment-16441952 ] liuxian edited comment on SPARK-23989 at 4/18/18 6:21 AM: -- 1.  M

[jira] [Commented] (SPARK-23989) When using `SortShuffleWriter`, the data will be overwritten

2018-04-17 Thread liuxian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441952#comment-16441952 ] liuxian commented on SPARK-23989: - 1.  Make 'BypassMergeSortShuffleHandle' and 'Serialize

[jira] [Updated] (SPARK-23989) When using `SortShuffleWriter`, the data will be overwritten

2018-04-17 Thread liuxian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuxian updated SPARK-23989: Attachment: 无标题2.png > When using `SortShuffleWriter`, the data will be overwritten > -

[jira] [Assigned] (SPARK-24007) EqualNullSafe for FloatType and DoubleType might generate a wrong result by codegen.

2018-04-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24007: Assignee: Takuya Ueshin (was: Apache Spark) > EqualNullSafe for FloatType and DoubleType

[jira] [Assigned] (SPARK-24007) EqualNullSafe for FloatType and DoubleType might generate a wrong result by codegen.

2018-04-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24007: Assignee: Apache Spark (was: Takuya Ueshin) > EqualNullSafe for FloatType and DoubleType

[jira] [Commented] (SPARK-24007) EqualNullSafe for FloatType and DoubleType might generate a wrong result by codegen.

2018-04-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441940#comment-16441940 ] Apache Spark commented on SPARK-24007: -- User 'ueshin' has created a pull request for

[jira] [Updated] (SPARK-24008) SQL/Hive Context fails with NullPointerException

2018-04-17 Thread Prabhu Joseph (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph updated SPARK-24008: -- Attachment: Repro > SQL/Hive Context fails with NullPointerException > ---

[jira] [Created] (SPARK-24008) SQL/Hive Context fails with NullPointerException

2018-04-17 Thread Prabhu Joseph (JIRA)
Prabhu Joseph created SPARK-24008: - Summary: SQL/Hive Context fails with NullPointerException Key: SPARK-24008 URL: https://issues.apache.org/jira/browse/SPARK-24008 Project: Spark Issue Typ

[jira] [Commented] (SPARK-23843) Deploy yarn meets incorrect LOCALIZED_CONF_DIR

2018-04-17 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441930#comment-16441930 ] Saisai Shao commented on SPARK-23843: - I think this issue is due to your "new Hadoop-

[jira] [Commented] (SPARK-23340) Upgrade Apache ORC to 1.4.3

2018-04-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441926#comment-16441926 ] Apache Spark commented on SPARK-23340: -- User 'dongjoon-hyun' has created a pull requ

[jira] [Assigned] (SPARK-23984) PySpark Bindings for K8S

2018-04-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23984: Assignee: Apache Spark > PySpark Bindings for K8S > > >

[jira] [Commented] (SPARK-23984) PySpark Bindings for K8S

2018-04-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441921#comment-16441921 ] Apache Spark commented on SPARK-23984: -- User 'ifilonenko' has created a pull request

[jira] [Assigned] (SPARK-23984) PySpark Bindings for K8S

2018-04-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23984: Assignee: (was: Apache Spark) > PySpark Bindings for K8S > >

[jira] [Commented] (SPARK-23830) Spark on YARN in cluster deploy mode fail with NullPointerException when a Spark application is a Scala class not object

2018-04-17 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441919#comment-16441919 ] Saisai Shao commented on SPARK-23830: - What is the reason to use {{class}} instead of

[jira] [Commented] (SPARK-24001) Multinode cluster

2018-04-17 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441918#comment-16441918 ] Saisai Shao commented on SPARK-24001: - Question should go to mail list. > Multinode

[jira] [Resolved] (SPARK-24001) Multinode cluster

2018-04-17 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24001. - Resolution: Invalid > Multinode cluster > -- > > Key: SPARK-2400

[jira] [Updated] (SPARK-24007) EqualNullSafe for FloatType and DoubleType might generate a wrong result by codegen.

2018-04-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24007: Labels: correctness (was: ) > EqualNullSafe for FloatType and DoubleType might generate a wrong result by

[jira] [Assigned] (SPARK-24007) EqualNullSafe for FloatType and DoubleType might generate a wrong result by codegen.

2018-04-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-24007: --- Assignee: Takuya Ueshin > EqualNullSafe for FloatType and DoubleType might generate a wrong result b

[jira] [Created] (SPARK-24007) EqualNullSafe for FloatType and DoubleType might generate a wrong result by codegen.

2018-04-17 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-24007: - Summary: EqualNullSafe for FloatType and DoubleType might generate a wrong result by codegen. Key: SPARK-24007 URL: https://issues.apache.org/jira/browse/SPARK-24007

[jira] [Updated] (SPARK-24006) ExecutorAllocationManager.onExecutorAdded is an O(n) operation

2018-04-17 Thread Xianjin YE (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianjin YE updated SPARK-24006: --- Description: The ExecutorAllocationManager.onExecutorAdded is an O(n) operations, I believe it will

[jira] [Created] (SPARK-24006) ExecutorAllocationManager.onExecutorAdded is an O(n) operation

2018-04-17 Thread Xianjin YE (JIRA)
Xianjin YE created SPARK-24006: -- Summary: ExecutorAllocationManager.onExecutorAdded is an O(n) operation Key: SPARK-24006 URL: https://issues.apache.org/jira/browse/SPARK-24006 Project: Spark I

[jira] [Commented] (SPARK-23989) When using `SortShuffleWriter`, the data will be overwritten

2018-04-17 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441906#comment-16441906 ] Saisai Shao commented on SPARK-23989: - Please provide a reproducible case. Did you r

[jira] [Commented] (SPARK-23982) NoSuchMethodException: There is no startCredentialUpdater method in the object YarnSparkHadoopUtil

2018-04-17 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441881#comment-16441881 ] Saisai Shao commented on SPARK-23982: - This method should be existed. Would you pleas

[jira] [Commented] (SPARK-7132) Add fit with validation set to spark.ml GBT

2018-04-17 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441864#comment-16441864 ] Weichen Xu commented on SPARK-7132: --- I dicussed with [~josephkb] and paste the proposal

[jira] [Updated] (SPARK-7132) Add fit with validation set to spark.ml GBT

2018-04-17 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-7132: -- Description: In spark.mllib GradientBoostedTrees, we have a method runWithValidation which takes a vali

[jira] [Assigned] (SPARK-23341) DataSourceOptions should handle path and table names to avoid confusion.

2018-04-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-23341: --- Assignee: Wenchen Fan > DataSourceOptions should handle path and table names to avoid confus

[jira] [Resolved] (SPARK-23341) DataSourceOptions should handle path and table names to avoid confusion.

2018-04-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-23341. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20535 [https://githu

[jira] [Comment Edited] (SPARK-24000) S3A: Create Table should fail on invalid AK/SK

2018-04-17 Thread Brahma Reddy Battula (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16440841#comment-16440841 ] Brahma Reddy Battula edited comment on SPARK-24000 at 4/18/18 3:41 AM:

[jira] [Commented] (SPARK-22676) Avoid iterating all partition paths when spark.sql.hive.verifyPartitionPath=true

2018-04-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441849#comment-16441849 ] Apache Spark commented on SPARK-22676: -- User 'jinxing64' has created a pull request

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 0.10.0.1 to 1.1.0

2018-04-17 Thread Jordan Moore (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441826#comment-16441826 ] Jordan Moore commented on SPARK-18057: -- Based on my Github searches, looks like it's

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 0.10.0.1 to 1.1.0

2018-04-17 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441818#comment-16441818 ] Cody Koeninger commented on SPARK-18057: Ok, if you can figure out what version o

[jira] [Comment Edited] (SPARK-18057) Update structured streaming kafka from 0.10.0.1 to 1.1.0

2018-04-17 Thread Richard Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441802#comment-16441802 ] Richard Yu edited comment on SPARK-18057 at 4/18/18 2:46 AM: -

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 0.10.0.1 to 1.1.0

2018-04-17 Thread Richard Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441802#comment-16441802 ] Richard Yu commented on SPARK-18057: Kafka contributors / developers are currently st

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 0.10.0.1 to 1.1.0

2018-04-17 Thread Jordan Moore (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441803#comment-16441803 ] Jordan Moore commented on SPARK-18057: -- {quote}probably won't work {quote} I figured

[jira] [Assigned] (SPARK-21479) Outer join filter pushdown in null supplying table when condition is on one of the joined columns

2018-04-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-21479: --- Assignee: Maryann Xue > Outer join filter pushdown in null supplying table when condition is

[jira] [Resolved] (SPARK-21479) Outer join filter pushdown in null supplying table when condition is on one of the joined columns

2018-04-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-21479. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20816 [https://githu

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 0.10.0.1 to 1.1.0

2018-04-17 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441793#comment-16441793 ] Cody Koeninger commented on SPARK-18057: Just adding the extra dependency on 0.11

[jira] [Resolved] (SPARK-22968) java.lang.IllegalStateException: No current assignment for partition kssh-2

2018-04-17 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cody Koeninger resolved SPARK-22968. Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21038 [https:/

[jira] [Assigned] (SPARK-22968) java.lang.IllegalStateException: No current assignment for partition kssh-2

2018-04-17 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cody Koeninger reassigned SPARK-22968: -- Assignee: Saisai Shao > java.lang.IllegalStateException: No current assignment for par

[jira] [Comment Edited] (SPARK-18057) Update structured streaming kafka from 0.10.0.1 to 1.1.0

2018-04-17 Thread Jordan Moore (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441718#comment-16441718 ] Jordan Moore edited comment on SPARK-18057 at 4/18/18 12:53 AM: ---

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 0.10.0.1 to 1.1.0

2018-04-17 Thread Jordan Moore (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441718#comment-16441718 ] Jordan Moore commented on SPARK-18057: -- Hello Cody,  No it is not. And no other spe

[jira] [Commented] (SPARK-18693) BinaryClassificationEvaluator, RegressionEvaluator, and MulticlassClassificationEvaluator should use sample weight data

2018-04-17 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441713#comment-16441713 ] Joseph K. Bradley commented on SPARK-18693: --- [~imatiach] Would you mind creatin

[jira] [Commented] (SPARK-23990) Instruments logging improvements - ML regression package

2018-04-17 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441701#comment-16441701 ] Joseph K. Bradley commented on SPARK-23990: --- A complication was brought up by t

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 0.10.0.1 to 1.1.0

2018-04-17 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441606#comment-16441606 ] Cody Koeninger commented on SPARK-18057: Out of curiosity, was that a compacted t

[jira] [Created] (SPARK-24005) Remove usage of Scala’s parallel collection

2018-04-17 Thread Xiao Li (JIRA)
Xiao Li created SPARK-24005: --- Summary: Remove usage of Scala’s parallel collection Key: SPARK-24005 URL: https://issues.apache.org/jira/browse/SPARK-24005 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-23948) Trigger mapstage's job listener in submitMissingTasks

2018-04-17 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-23948: - Fix Version/s: 2.3.1 > Trigger mapstage's job listener in submitMissingTasks > --

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 0.10.0.1 to 1.1.0

2018-04-17 Thread Jordan Moore (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441485#comment-16441485 ] Jordan Moore commented on SPARK-18057: -- Hi all, chiming in here to point out a produ

[jira] [Commented] (SPARK-15784) Add Power Iteration Clustering to spark.ml

2018-04-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441448#comment-16441448 ] Apache Spark commented on SPARK-15784: -- User 'jkbradley' has created a pull request

[jira] [Commented] (SPARK-23963) Queries on text-based Hive tables grow disproportionately slower as the number of columns increase

2018-04-17 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441378#comment-16441378 ] Bruce Robbins commented on SPARK-23963: --- [~Tagar] Yes, although I am a little fuzzy

[jira] [Commented] (SPARK-24004) Tests of from_json for MapType

2018-04-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441369#comment-16441369 ] Apache Spark commented on SPARK-24004: -- User 'MaxGekk' has created a pull request fo

[jira] [Assigned] (SPARK-24004) Tests of from_json for MapType

2018-04-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24004: Assignee: Apache Spark > Tests of from_json for MapType > -- >

[jira] [Assigned] (SPARK-24004) Tests of from_json for MapType

2018-04-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24004: Assignee: (was: Apache Spark) > Tests of from_json for MapType > -

[jira] [Created] (SPARK-24004) Tests of from_json for MapType

2018-04-17 Thread Maxim Gekk (JIRA)
Maxim Gekk created SPARK-24004: -- Summary: Tests of from_json for MapType Key: SPARK-24004 URL: https://issues.apache.org/jira/browse/SPARK-24004 Project: Spark Issue Type: Test Compone

[jira] [Commented] (SPARK-15784) Add Power Iteration Clustering to spark.ml

2018-04-17 Thread Miao Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441353#comment-16441353 ] Miao Wang commented on SPARK-15784: --- [~josephkb] You can start the new PR now. :) > Ad

[jira] [Assigned] (SPARK-24003) Add support to provide spark.executor.extraJavaOptions in terms of App Id and/or Executor Id's

2018-04-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24003: Assignee: (was: Apache Spark) > Add support to provide spark.executor.extraJavaOptions

[jira] [Assigned] (SPARK-24003) Add support to provide spark.executor.extraJavaOptions in terms of App Id and/or Executor Id's

2018-04-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24003: Assignee: Apache Spark > Add support to provide spark.executor.extraJavaOptions in terms o

[jira] [Commented] (SPARK-24003) Add support to provide spark.executor.extraJavaOptions in terms of App Id and/or Executor Id's

2018-04-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441328#comment-16441328 ] Apache Spark commented on SPARK-24003: -- User 'devaraj-kavali' has created a pull req

[jira] [Updated] (SPARK-22884) ML test for StructuredStreaming: spark.ml.clustering

2018-04-17 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-22884: -- Shepherd: Joseph K. Bradley > ML test for StructuredStreaming: spark.ml.clustering > --

[jira] [Created] (SPARK-24003) Add support to provide spark.executor.extraJavaOptions in terms of App Id and/or Executor Id's

2018-04-17 Thread Devaraj K (JIRA)
Devaraj K created SPARK-24003: - Summary: Add support to provide spark.executor.extraJavaOptions in terms of App Id and/or Executor Id's Key: SPARK-24003 URL: https://issues.apache.org/jira/browse/SPARK-24003

[jira] [Commented] (SPARK-23933) High-order function: map(array, array) → map

2018-04-17 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441268#comment-16441268 ] Kazuaki Ishizaki commented on SPARK-23933: -- ping [~smilegator] > High-order fun

[jira] [Commented] (SPARK-21063) Spark return an empty result from remote hadoop cluster

2018-04-17 Thread Carlos Bribiescas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441257#comment-16441257 ] Carlos Bribiescas commented on SPARK-21063: --- Any update or workarounds for this

[jira] [Updated] (SPARK-8799) OneVsRestModel should extend ClassificationModel

2018-04-17 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-8799: - Shepherd: Joseph K. Bradley > OneVsRestModel should extend ClassificationModel > -

[jira] [Commented] (SPARK-8799) OneVsRestModel should extend ClassificationModel

2018-04-17 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441207#comment-16441207 ] Joseph K. Bradley commented on SPARK-8799: -- The missing functionality was added i

[jira] [Updated] (SPARK-8799) OneVsRestModel should extend ClassificationModel

2018-04-17 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-8799: - Target Version/s: 3.0.0 > OneVsRestModel should extend ClassificationModel > -

[jira] [Assigned] (SPARK-21741) Python API for DataFrame-based multivariate summarizer

2018-04-17 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley reassigned SPARK-21741: - Assignee: Weichen Xu > Python API for DataFrame-based multivariate summarizer >

[jira] [Resolved] (SPARK-21741) Python API for DataFrame-based multivariate summarizer

2018-04-17 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-21741. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20695 [h

[jira] [Commented] (SPARK-23997) Configurable max number of buckets

2018-04-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441146#comment-16441146 ] Apache Spark commented on SPARK-23997: -- User 'ferdonline' has created a pull request

[jira] [Assigned] (SPARK-23997) Configurable max number of buckets

2018-04-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23997: Assignee: Apache Spark > Configurable max number of buckets >

[jira] [Assigned] (SPARK-23997) Configurable max number of buckets

2018-04-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23997: Assignee: (was: Apache Spark) > Configurable max number of buckets > -

[jira] [Resolved] (SPARK-23986) CompileException when using too many avg aggregation after joining

2018-04-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-23986. - Resolution: Fixed Fix Version/s: 2.3.1 2.4.0 Issue resolved by pull req

[jira] [Assigned] (SPARK-23986) CompileException when using too many avg aggregation after joining

2018-04-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-23986: --- Assignee: Marco Gaido > CompileException when using too many avg aggregation after joining >

[jira] [Resolved] (SPARK-23999) Spark SQL shell is a Stable one ? Can we use Spark SQL shell in our production environment?

2018-04-17 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-23999. Resolution: Invalid Fix Version/s: (was: 2.3.0) (was: 3.0.

[jira] [Assigned] (SPARK-24002) Task not serializable caused by org.apache.parquet.io.api.Binary$ByteBufferBackedBinary.getBytes

2018-04-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24002: Assignee: Xiao Li (was: Apache Spark) > Task not serializable caused by > org.apache.par

[jira] [Commented] (SPARK-24002) Task not serializable caused by org.apache.parquet.io.api.Binary$ByteBufferBackedBinary.getBytes

2018-04-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441071#comment-16441071 ] Apache Spark commented on SPARK-24002: -- User 'gatorsmile' has created a pull request

[jira] [Assigned] (SPARK-24002) Task not serializable caused by org.apache.parquet.io.api.Binary$ByteBufferBackedBinary.getBytes

2018-04-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24002: Assignee: Apache Spark (was: Xiao Li) > Task not serializable caused by > org.apache.par

[jira] [Updated] (SPARK-24002) Task not serializable caused by org.apache.parquet.io.api.Binary$ByteBufferBackedBinary.getBytes

2018-04-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24002: Description: Having two queries one is a 1000-line SQL query and a 3000-line SQL query. Need to run at lea

[jira] [Updated] (SPARK-24002) Task not serializable caused by org.apache.parquet.io.api.Binary$ByteBufferBackedBinary.getBytes

2018-04-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24002: Description: {code} java.lang.IllegalArgumentException at java.nio.Buffer.position(Buffer.java:244)

[jira] [Created] (SPARK-24002) Task not serializable caused by org.apache.parquet.io.api.Binary$ByteBufferBackedBinary.getBytes

2018-04-17 Thread Xiao Li (JIRA)
Xiao Li created SPARK-24002: --- Summary: Task not serializable caused by org.apache.parquet.io.api.Binary$ByteBufferBackedBinary.getBytes Key: SPARK-24002 URL: https://issues.apache.org/jira/browse/SPARK-24002

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

2018-04-17 Thread Edwina Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441006#comment-16441006 ] Edwina Lu commented on SPARK-23206: --- [~assia6], could you please try the new link, [ht

[jira] [Updated] (SPARK-23888) speculative task should not run on a given host where another attempt is already running on

2018-04-17 Thread wuyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-23888: - Description:   There's a bug in: {code:java} /** Check whether a task is currently running an attempt on a given

[jira] [Updated] (SPARK-24001) Multinode cluster

2018-04-17 Thread Direselign (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Direselign updated SPARK-24001: --- Attachment: Screenshot from 2018-04-17 22-47-39.png > Multinode cluster > -- > >

[jira] [Commented] (SPARK-23963) Queries on text-based Hive tables grow disproportionately slower as the number of columns increase

2018-04-17 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16440942#comment-16440942 ] Ruslan Dautkhanov commented on SPARK-23963: --- Thanks a lot [~bersprockets]  Wou

[jira] [Created] (SPARK-24001) Multinode cluster

2018-04-17 Thread Direselign (JIRA)
Direselign created SPARK-24001: -- Summary: Multinode cluster Key: SPARK-24001 URL: https://issues.apache.org/jira/browse/SPARK-24001 Project: Spark Issue Type: Bug Components: PySpark

[jira] [Comment Edited] (SPARK-22371) dag-scheduler-event-loop thread stopped with error Attempted to access garbage collected accumulator 5605982

2018-04-17 Thread Ben Doerr (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16402271#comment-16402271 ] Ben Doerr edited comment on SPARK-22371 at 4/17/18 2:04 PM: W

[jira] [Commented] (SPARK-23948) Trigger mapstage's job listener in submitMissingTasks

2018-04-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16440887#comment-16440887 ] Apache Spark commented on SPARK-23948: -- User 'squito' has created a pull request for

[jira] [Assigned] (SPARK-23948) Trigger mapstage's job listener in submitMissingTasks

2018-04-17 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-23948: Assignee: jin xing > Trigger mapstage's job listener in submitMissingTasks > -

[jira] [Resolved] (SPARK-23948) Trigger mapstage's job listener in submitMissingTasks

2018-04-17 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-23948. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21019 [https://git

[jira] [Resolved] (SPARK-22676) Avoid iterating all partition paths when spark.sql.hive.verifyPartitionPath=true

2018-04-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-22676. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 19868 [https://githu

[jira] [Assigned] (SPARK-22676) Avoid iterating all partition paths when spark.sql.hive.verifyPartitionPath=true

2018-04-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-22676: --- Assignee: jin xing > Avoid iterating all partition paths when > spark.sql.hive.verifyPartit

[jira] [Resolved] (SPARK-23835) When Dataset.as converts column from nullable to non-nullable type, null Doubles are converted silently to -1

2018-04-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-23835. - Resolution: Fixed Assignee: Marco Gaido Fix Version/s: 2.4.0 2.

[jira] [Commented] (SPARK-15703) Make ListenerBus event queue size configurable

2018-04-17 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16440855#comment-16440855 ] Thomas Graves commented on SPARK-15703: --- this Jira is purely making the size of the

[jira] [Commented] (SPARK-24000) S3A: Create Table should fail on invalid AK/SK

2018-04-17 Thread Brahma Reddy Battula (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16440841#comment-16440841 ] Brahma Reddy Battula commented on SPARK-24000: -- Discussed [~ste...@apache.or

[jira] [Resolved] (SPARK-23875) Create IndexedSeq wrapper for ArrayData

2018-04-17 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-23875. --- Resolution: Fixed Assignee: Liang-Chi Hsieh Fix Version/s: 2.4.0 > Cr

[jira] [Created] (SPARK-24000) S3A: Create Table should fail on invalid AK/SK

2018-04-17 Thread Brahma Reddy Battula (JIRA)
Brahma Reddy Battula created SPARK-24000: Summary: S3A: Create Table should fail on invalid AK/SK Key: SPARK-24000 URL: https://issues.apache.org/jira/browse/SPARK-24000 Project: Spark

[jira] [Created] (SPARK-23999) Spark SQL shell is a Stable one ? Can we use Spark SQL shell in our production environment?

2018-04-17 Thread Prabhu Bentick (JIRA)
Prabhu Bentick created SPARK-23999: -- Summary: Spark SQL shell is a Stable one ? Can we use Spark SQL shell in our production environment? Key: SPARK-23999 URL: https://issues.apache.org/jira/browse/SPARK-23999

  1   2   >