[jira] [Commented] (SPARK-24498) Add JDK compiler for runtime codegen

2018-06-21 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16520056#comment-16520056 ] Takeshi Yamamuro commented on SPARK-24498: -- The results of my investigation (sf

[jira] [Commented] (SPARK-24569) Spark Aggregator with output type Option[Boolean] creates column of type Row

2018-06-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16520042#comment-16520042 ] Apache Spark commented on SPARK-24569: -- User 'viirya' has created a pull request fo

[jira] [Assigned] (SPARK-24569) Spark Aggregator with output type Option[Boolean] creates column of type Row

2018-06-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24569: Assignee: Apache Spark > Spark Aggregator with output type Option[Boolean] creates column

[jira] [Assigned] (SPARK-24569) Spark Aggregator with output type Option[Boolean] creates column of type Row

2018-06-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24569: Assignee: (was: Apache Spark) > Spark Aggregator with output type Option[Boolean] cre

[jira] [Commented] (SPARK-24082) [Spark SQL] Tables are not listing under DB

2018-06-21 Thread sandeep katta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16520023#comment-16520023 ] sandeep katta commented on SPARK-24082: --- In your case both the tables created are

[jira] [Commented] (SPARK-24422) Add JDK9+ in our Jenkins' build servers

2018-06-21 Thread vaquar khan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16520014#comment-16520014 ] vaquar khan commented on SPARK-24422: - Where i will get required information and acc

[jira] [Commented] (SPARK-24201) IllegalArgumentException originating from ClosureCleaner in Java 9+

2018-06-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16520013#comment-16520013 ] Hyukjin Kwon commented on SPARK-24201: -- That's true but shell we see what's going t

[jira] [Resolved] (SPARK-13184) Support minPartitions parameter for JSON and CSV datasources as options

2018-06-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-13184. -- Resolution: Not A Problem > Support minPartitions parameter for JSON and CSV datasources as op

[jira] [Commented] (SPARK-13184) Support minPartitions parameter for JSON and CSV datasources as options

2018-06-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16520007#comment-16520007 ] Hyukjin Kwon commented on SPARK-13184: -- I think it's rather almost close to an obso

[jira] [Commented] (SPARK-24201) IllegalArgumentException originating from ClosureCleaner in Java 9+

2018-06-21 Thread vaquar khan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16520006#comment-16520006 ] vaquar khan commented on SPARK-24201: - Still  we need to modify doc for old version

[jira] [Commented] (SPARK-22357) SparkContext.binaryFiles ignore minPartitions parameter

2018-06-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16520002#comment-16520002 ] Hyukjin Kwon commented on SPARK-22357: -- Please open a PR and go ahead if you think

[jira] [Resolved] (SPARK-24458) Invalid PythonUDF check_1(), requires attributes from more than one child

2018-06-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24458. -- Resolution: Cannot Reproduce Let me leave this resolved per ^ for the current status. Please

[jira] [Resolved] (SPARK-24201) IllegalArgumentException originating from ClosureCleaner in Java 9+

2018-06-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24201. -- Resolution: Duplicate > IllegalArgumentException originating from ClosureCleaner in Java 9+ >

[jira] [Commented] (SPARK-24201) IllegalArgumentException originating from ClosureCleaner in Java 9+

2018-06-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519997#comment-16519997 ] Hyukjin Kwon commented on SPARK-24201: -- JDK 9 is in progress in SPARK-24417 too. Le

[jira] [Commented] (SPARK-20295) when spark.sql.adaptive.enabled is enabled, have conflict with Exchange Resue

2018-06-21 Thread Kevin Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519994#comment-16519994 ] Kevin Zhang commented on SPARK-20295: - Still hit this bug in spark 2.3.0 > when sp

[jira] [Updated] (SPARK-23710) Upgrade Hive to 2.3.2

2018-06-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-23710: - Priority: Critical (was: Major) > Upgrade Hive to 2.3.2 > - > >

[jira] [Commented] (SPARK-23710) Upgrade Hive to 2.3.2

2018-06-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519979#comment-16519979 ] Hyukjin Kwon commented on SPARK-23710: -- [~q79969786], per Xiao's comment, I think t

[jira] [Commented] (SPARK-24579) SPIP: Standardize Optimized Data Exchange between Spark and DL/AI frameworks

2018-06-21 Thread Jepson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519956#comment-16519956 ] Jepson commented on SPARK-24579: Very nice. > SPIP: Standardize Optimized Data Exchange

[jira] [Commented] (SPARK-24374) SPIP: Support Barrier Scheduling in Apache Spark

2018-06-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519934#comment-16519934 ] Saisai Shao commented on SPARK-24374: - Hi [~mengxr] [~jiangxb1987] SPARK-24615 would

[jira] [Comment Edited] (SPARK-24201) IllegalArgumentException originating from ClosureCleaner in Java 9+

2018-06-21 Thread vaquar khan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519913#comment-16519913 ] vaquar khan edited comment on SPARK-24201 at 6/22/18 2:36 AM:

[jira] [Comment Edited] (SPARK-24201) IllegalArgumentException originating from ClosureCleaner in Java 9+

2018-06-21 Thread vaquar khan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519913#comment-16519913 ] vaquar khan edited comment on SPARK-24201 at 6/22/18 2:36 AM:

[jira] [Comment Edited] (SPARK-24201) IllegalArgumentException originating from ClosureCleaner in Java 9+

2018-06-21 Thread vaquar khan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519913#comment-16519913 ] vaquar khan edited comment on SPARK-24201 at 6/22/18 2:28 AM:

[jira] [Commented] (SPARK-24201) IllegalArgumentException originating from ClosureCleaner in Java 9+

2018-06-21 Thread vaquar khan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519913#comment-16519913 ] vaquar khan commented on SPARK-24201: - Let me fix Spark doc > IllegalArgumentExcept

[jira] [Commented] (SPARK-22897) Expose stageAttemptId in TaskContext

2018-06-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519902#comment-16519902 ] Apache Spark commented on SPARK-22897: -- User 'tgravescs' has created a pull request

[jira] [Updated] (SPARK-24596) Non-cascading Cache Invalidation

2018-06-21 Thread Maryann Xue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maryann Xue updated SPARK-24596: Description: When invalidating a cache, we invalid other caches dependent on this cache to ensure

[jira] [Updated] (SPARK-24621) WebUI - application 'name' urls point to http instead of https (even when ssl enabled)

2018-06-21 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-24621: - Description: See attached ApplicationID correctly points to DNS url but Name points to IP address Update: I fo

[jira] [Updated] (SPARK-24621) WebUI - application 'name' urls point to http instead of https (even when ssl enabled)

2018-06-21 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-24621: - Summary: WebUI - application 'name' urls point to http instead of https (even when ssl enabled) (was: WebUI - a

[jira] [Resolved] (SPARK-24588) StreamingSymmetricHashJoinExec should require HashClusteredPartitioning from children

2018-06-21 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24588. - Resolution: Fixed Fix Version/s: 2.4.0 2.3.2 > StreamingSymmetricHashJoinExec

[jira] [Updated] (SPARK-24588) StreamingSymmetricHashJoinExec should require HashClusteredPartitioning from children

2018-06-21 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24588: Description: In [https://github.com/apache/spark/pull/19080], we simplified the distribution/partitioning

[jira] [Assigned] (SPARK-24626) Improve Analyze Table command

2018-06-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24626: Assignee: Apache Spark > Improve Analyze Table command > - >

[jira] [Commented] (SPARK-24626) Improve Analyze Table command

2018-06-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519813#comment-16519813 ] Apache Spark commented on SPARK-24626: -- User 'Achuth17' has created a pull request

[jira] [Assigned] (SPARK-24626) Improve Analyze Table command

2018-06-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24626: Assignee: (was: Apache Spark) > Improve Analyze Table command > -

[jira] [Updated] (SPARK-24626) Improve Analyze Table command

2018-06-21 Thread Achuth Narayan Rajagopal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Achuth Narayan Rajagopal updated SPARK-24626: - Fix Version/s: (was: 2.4.0) > Improve Analyze Table command > --

[jira] [Created] (SPARK-24626) Improve Analyze Table command

2018-06-21 Thread Achuth Narayan Rajagopal (JIRA)
Achuth Narayan Rajagopal created SPARK-24626: Summary: Improve Analyze Table command Key: SPARK-24626 URL: https://issues.apache.org/jira/browse/SPARK-24626 Project: Spark Issue T

[jira] [Commented] (SPARK-24589) OutputCommitCoordinator may allow duplicate commits

2018-06-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519799#comment-16519799 ] Apache Spark commented on SPARK-24589: -- User 'vanzin' has created a pull request fo

[jira] [Commented] (SPARK-22897) Expose stageAttemptId in TaskContext

2018-06-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519800#comment-16519800 ] Apache Spark commented on SPARK-22897: -- User 'vanzin' has created a pull request fo

[jira] [Resolved] (SPARK-24589) OutputCommitCoordinator may allow duplicate commits

2018-06-21 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24589. --- Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 2.4.0

[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried

2018-06-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519723#comment-16519723 ] Apache Spark commented on SPARK-24552: -- User 'vanzin' has created a pull request fo

[jira] [Updated] (SPARK-24615) Accelerator aware task scheduling for Spark

2018-06-21 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24615: -- Labels: Hydrogen SPIP (was: SPIP) > Accelerator aware task scheduling for Spark > ---

[jira] [Resolved] (SPARK-24613) Cache with UDF could not be matched with subsequent dependent caches

2018-06-21 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24613. - Resolution: Fixed Assignee: Maryann Xue > Cache with UDF could not be matched with subsequent depe

[jira] [Created] (SPARK-24625) put all the backward compatible behavior change configs under spark.sql.legacy.*

2018-06-21 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-24625: --- Summary: put all the backward compatible behavior change configs under spark.sql.legacy.* Key: SPARK-24625 URL: https://issues.apache.org/jira/browse/SPARK-24625 Projec

[jira] [Commented] (SPARK-24374) SPIP: Support Barrier Scheduling in Apache Spark

2018-06-21 Thread Mingjie Tang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519622#comment-16519622 ] Mingjie Tang commented on SPARK-24374: -- [~mengxr].  [~jerryshao] create a SISP [SP

[jira] [Commented] (SPARK-24624) Can not mix vectorized and non-vectorized UDFs

2018-06-21 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519590#comment-16519590 ] Li Jin commented on SPARK-24624: I can take a look at this one > Can not mix vectorized

[jira] [Commented] (SPARK-24624) Can not mix vectorized and non-vectorized UDFs

2018-06-21 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519586#comment-16519586 ] Xiao Li commented on SPARK-24624: - [~icexelloss] [~bryanc] [~hyukjin.kwon] Could you hel

[jira] [Created] (SPARK-24624) Can not mix vectorized and non-vectorized UDFs

2018-06-21 Thread Xiao Li (JIRA)
Xiao Li created SPARK-24624: --- Summary: Can not mix vectorized and non-vectorized UDFs Key: SPARK-24624 URL: https://issues.apache.org/jira/browse/SPARK-24624 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-17636) Parquet filter push down doesn't handle struct fields

2018-06-21 Thread Justin Miller (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519568#comment-16519568 ] Justin Miller commented on SPARK-17636: --- Any updates on this ticket? Thanks! > Pa

[jira] [Updated] (SPARK-24588) StreamingSymmetricHashJoinExec should require HashClusteredPartitioning from children

2018-06-21 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24588: Description: In [https://github.com/apache/spark/pull/19080], we simplified the distribution/partitioning

[jira] [Created] (SPARK-24623) Hadoop - Spark Cluster - Python XGBoost - Not working in distributed mode

2018-06-21 Thread Abhishek Reddy Chamakura (JIRA)
Abhishek Reddy Chamakura created SPARK-24623: Summary: Hadoop - Spark Cluster - Python XGBoost - Not working in distributed mode Key: SPARK-24623 URL: https://issues.apache.org/jira/browse/SPARK-24623

[jira] [Commented] (SPARK-24385) Trivially-true EqualNullSafe should be handled like EqualTo in Dataset.join

2018-06-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519541#comment-16519541 ] Apache Spark commented on SPARK-24385: -- User 'mgaido91' has created a pull request

[jira] [Resolved] (SPARK-24620) [security] Disable 'kill' button next to applications on Spark WebUI

2018-06-21 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24620. Resolution: Not A Bug Like "spark.ui.killEnabled"? > [security] Disable 'kill' button nex

[jira] [Comment Edited] (SPARK-24552) Task attempt numbers are reused when stages are retried

2018-06-21 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519445#comment-16519445 ] Thomas Graves edited comment on SPARK-24552 at 6/21/18 3:02 PM: --

[jira] [Comment Edited] (SPARK-24552) Task attempt numbers are reused when stages are retried

2018-06-21 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519445#comment-16519445 ] Thomas Graves edited comment on SPARK-24552 at 6/21/18 3:01 PM: --

[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried

2018-06-21 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519445#comment-16519445 ] Thomas Graves commented on SPARK-24552: --- more details on hadoop committer side: S

[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried

2018-06-21 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519430#comment-16519430 ] Thomas Graves commented on SPARK-24552: --- this is actually a problem with hadoop co

[jira] [Commented] (SPARK-24611) Clean up OutputCommitCoordinator

2018-06-21 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519420#comment-16519420 ] Thomas Graves commented on SPARK-24611: --- [~joshrosen]  just noticed you were the l

[jira] [Commented] (SPARK-24622) Task attempts in other stage attempts not killed when one task attempt succeeds

2018-06-21 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519416#comment-16519416 ] Thomas Graves commented on SPARK-24622: --- Need to investigate further/test to make

[jira] [Assigned] (SPARK-16630) Blacklist a node if executors won't launch on it.

2018-06-21 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-16630: Assignee: Attila Zsolt Piros > Blacklist a node if executors won't launch on it. > --

[jira] [Resolved] (SPARK-16630) Blacklist a node if executors won't launch on it.

2018-06-21 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-16630. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21068 [https://gi

[jira] [Created] (SPARK-24622) Task attempts in other stage attempts not killed when one task attempt succeeds

2018-06-21 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-24622: - Summary: Task attempts in other stage attempts not killed when one task attempt succeeds Key: SPARK-24622 URL: https://issues.apache.org/jira/browse/SPARK-24622 Pro

[jira] [Updated] (SPARK-24621) WebUI - application 'name' urls point to IP instead of DNS name

2018-06-21 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-24621: - Attachment: spark_master-one-app.png > WebUI - application 'name' urls point to IP instead of DNS name > ---

[jira] [Updated] (SPARK-24620) [security] Disable 'kill' button next to applications on Spark WebUI

2018-06-21 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-24620: - Attachment: spark_master-one-app.png > [security] Disable 'kill' button next to applications on Spark WebUI > --

[jira] [Updated] (SPARK-24620) [security] Disable 'kill' button next to applications on Spark WebUI

2018-06-21 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-24620: - Attachment: (was: spark_master-one-app.png) > [security] Disable 'kill' button next to applications on Spark

[jira] [Created] (SPARK-24621) WebUI - application 'name' urls point to IP instead of DNS name

2018-06-21 Thread t oo (JIRA)
t oo created SPARK-24621: Summary: WebUI - application 'name' urls point to IP instead of DNS name Key: SPARK-24621 URL: https://issues.apache.org/jira/browse/SPARK-24621 Project: Spark Issue Type:

[jira] [Created] (SPARK-24620) [security] Disable 'kill' button next to applications on Spark WebUI

2018-06-21 Thread t oo (JIRA)
t oo created SPARK-24620: Summary: [security] Disable 'kill' button next to applications on Spark WebUI Key: SPARK-24620 URL: https://issues.apache.org/jira/browse/SPARK-24620 Project: Spark Issue T

[jira] [Created] (SPARK-24619) DOCO - usage of authenticate.secret between masters/workers not clear

2018-06-21 Thread t oo (JIRA)
t oo created SPARK-24619: Summary: DOCO - usage of authenticate.secret between masters/workers not clear Key: SPARK-24619 URL: https://issues.apache.org/jira/browse/SPARK-24619 Project: Spark Issue

[jira] [Created] (SPARK-24618) Allow ability to consume driver memory on worker hosts not master (option for clustermode to wait for returncode?)

2018-06-21 Thread t oo (JIRA)
t oo created SPARK-24618: Summary: Allow ability to consume driver memory on worker hosts not master (option for clustermode to wait for returncode?) Key: SPARK-24618 URL: https://issues.apache.org/jira/browse/SPARK-24618

[jira] [Created] (SPARK-24617) Spark driver not requesting another executor once original executor exits due to 'lost worker'

2018-06-21 Thread t oo (JIRA)
t oo created SPARK-24617: Summary: Spark driver not requesting another executor once original executor exits due to 'lost worker' Key: SPARK-24617 URL: https://issues.apache.org/jira/browse/SPARK-24617 Projec

[jira] [Commented] (SPARK-14643) Remove overloaded methods which become ambiguous in Scala 2.12

2018-06-21 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519312#comment-16519312 ] Stavros Kontopoulos commented on SPARK-14643: - [~srowen] [~joshrosen] there

[jira] [Commented] (SPARK-14540) Support Scala 2.12 closures and Java 8 lambdas in ClosureCleaner

2018-06-21 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519310#comment-16519310 ] Stavros Kontopoulos commented on SPARK-14540: - [~srowen] There is a new disc

[jira] [Updated] (SPARK-22911) Migrate structured streaming sources to new DataSourceV2 APIs

2018-06-21 Thread Teng Peng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teng Peng updated SPARK-22911: -- Issue Type: Umbrella (was: Improvement) > Migrate structured streaming sources to new DataSourceV2 AP

[jira] [Commented] (SPARK-17248) Add native Scala enum support to Dataset Encoders

2018-06-21 Thread Angel Cervera Claudio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519214#comment-16519214 ] Angel Cervera Claudio commented on SPARK-17248: --- [~dongjin] suggestion is

[jira] [Commented] (SPARK-24347) df.alias() in python API should not clear metadata by default

2018-06-21 Thread Ruben Berenguel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519191#comment-16519191 ] Ruben Berenguel commented on SPARK-24347: - [~holdenkarau] [~ueshin] (I ping you

[jira] [Commented] (SPARK-21918) HiveClient shouldn't share Hive object between different thread

2018-06-21 Thread fengchaoge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519149#comment-16519149 ] fengchaoge commented on SPARK-21918: Hello Hu Liu, can you share you patch? we are

[jira] [Commented] (SPARK-24607) Distribute by rand() can lead to data inconsistency

2018-06-21 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519119#comment-16519119 ] Liang-Chi Hsieh commented on SPARK-24607: - [~gostop_zlx] If you don't give a se

[jira] [Resolved] (SPARK-24614) PySpark - Fix SyntaxWarning on tests.py

2018-06-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24614. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21604 [https://gi

[jira] [Assigned] (SPARK-24614) PySpark - Fix SyntaxWarning on tests.py

2018-06-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-24614: Assignee: Rekha Joshi > PySpark - Fix SyntaxWarning on tests.py > ---

[jira] [Updated] (SPARK-24615) Accelerator aware task scheduling for Spark

2018-06-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-24615: Labels: SPIP (was: ) > Accelerator aware task scheduling for Spark >

[jira] [Updated] (SPARK-24615) Accelerator aware task scheduling for Spark

2018-06-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-24615: Description: In the machine learning area, accelerator card (GPU, FPGA, TPU) is predominant compa

[jira] [Commented] (SPARK-24347) df.alias() in python API should not clear metadata by default

2018-06-21 Thread Ruben Berenguel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519074#comment-16519074 ] Ruben Berenguel commented on SPARK-24347: - Looking into this > df.alias() in py

[jira] [Updated] (SPARK-24615) Accelerator aware task scheduling for Spark

2018-06-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-24615: Description: In the machine learning area, accelerator card (GPU, FPGA, TPU) is predominant compa

[jira] [Created] (SPARK-24615) Accelerator aware task scheduling for Spark

2018-06-21 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-24615: --- Summary: Accelerator aware task scheduling for Spark Key: SPARK-24615 URL: https://issues.apache.org/jira/browse/SPARK-24615 Project: Spark Issue Type: Improve

[jira] [Created] (SPARK-24616) Need to retreive free memory on command prompt on DSE cluster

2018-06-21 Thread Kushagra Bindal (JIRA)
Kushagra Bindal created SPARK-24616: --- Summary: Need to retreive free memory on command prompt on DSE cluster Key: SPARK-24616 URL: https://issues.apache.org/jira/browse/SPARK-24616 Project: Spark

[jira] [Resolved] (SPARK-24574) improve array_contains function of the sql component to deal with Column type

2018-06-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24574. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21581 [https://gi

[jira] [Assigned] (SPARK-24574) improve array_contains function of the sql component to deal with Column type

2018-06-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-24574: Assignee: Chongguang LIU > improve array_contains function of the sql component to deal w