[jira] [Commented] (SPARK-24082) [Spark SQL] Tables are not listing under DB

2018-06-21 Thread sandeep katta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520023#comment-16520023 ] sandeep katta commented on SPARK-24082: --- In your case both the tables created are External tables.

[jira] [Commented] (SPARK-24422) Add JDK9+ in our Jenkins' build servers

2018-06-21 Thread vaquar khan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520014#comment-16520014 ] vaquar khan commented on SPARK-24422: - Where i will get required information and access to add JDK

[jira] [Commented] (SPARK-24201) IllegalArgumentException originating from ClosureCleaner in Java 9+

2018-06-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520013#comment-16520013 ] Hyukjin Kwon commented on SPARK-24201: -- That's true but shell we see what's going to happen in

[jira] [Resolved] (SPARK-13184) Support minPartitions parameter for JSON and CSV datasources as options

2018-06-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-13184. -- Resolution: Not A Problem > Support minPartitions parameter for JSON and CSV datasources as

[jira] [Commented] (SPARK-13184) Support minPartitions parameter for JSON and CSV datasources as options

2018-06-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520007#comment-16520007 ] Hyukjin Kwon commented on SPARK-13184: -- I think it's rather almost close to an obsolete one because

[jira] [Commented] (SPARK-24201) IllegalArgumentException originating from ClosureCleaner in Java 9+

2018-06-21 Thread vaquar khan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520006#comment-16520006 ] vaquar khan commented on SPARK-24201: - Still  we need to modify doc for old version also as doc

[jira] [Commented] (SPARK-22357) SparkContext.binaryFiles ignore minPartitions parameter

2018-06-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520002#comment-16520002 ] Hyukjin Kwon commented on SPARK-22357: -- Please open a PR and go ahead if you think it's right. I

[jira] [Resolved] (SPARK-24458) Invalid PythonUDF check_1(), requires attributes from more than one child

2018-06-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24458. -- Resolution: Cannot Reproduce Let me leave this resolved per ^ for the current status. Please

[jira] [Resolved] (SPARK-24201) IllegalArgumentException originating from ClosureCleaner in Java 9+

2018-06-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24201. -- Resolution: Duplicate > IllegalArgumentException originating from ClosureCleaner in Java 9+

[jira] [Commented] (SPARK-24201) IllegalArgumentException originating from ClosureCleaner in Java 9+

2018-06-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519997#comment-16519997 ] Hyukjin Kwon commented on SPARK-24201: -- JDK 9 is in progress in SPARK-24417 too. Let's leave this

[jira] [Commented] (SPARK-20295) when spark.sql.adaptive.enabled is enabled, have conflict with Exchange Resue

2018-06-21 Thread Kevin Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519994#comment-16519994 ] Kevin Zhang commented on SPARK-20295: - Still hit this bug in spark 2.3.0 > when

[jira] [Updated] (SPARK-23710) Upgrade Hive to 2.3.2

2018-06-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-23710: - Priority: Critical (was: Major) > Upgrade Hive to 2.3.2 > - > >

[jira] [Commented] (SPARK-23710) Upgrade Hive to 2.3.2

2018-06-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519979#comment-16519979 ] Hyukjin Kwon commented on SPARK-23710: -- [~q79969786], per Xiao's comment, I think the investigation

[jira] [Commented] (SPARK-24579) SPIP: Standardize Optimized Data Exchange between Spark and DL/AI frameworks

2018-06-21 Thread Jepson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519956#comment-16519956 ] Jepson commented on SPARK-24579: Very nice. > SPIP: Standardize Optimized Data Exchange between Spark

[jira] [Commented] (SPARK-24374) SPIP: Support Barrier Scheduling in Apache Spark

2018-06-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519934#comment-16519934 ] Saisai Shao commented on SPARK-24374: - Hi [~mengxr] [~jiangxb1987] SPARK-24615 would be a part of

[jira] [Comment Edited] (SPARK-24201) IllegalArgumentException originating from ClosureCleaner in Java 9+

2018-06-21 Thread vaquar khan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519913#comment-16519913 ] vaquar khan edited comment on SPARK-24201 at 6/22/18 2:36 AM: -- Let me 

[jira] [Comment Edited] (SPARK-24201) IllegalArgumentException originating from ClosureCleaner in Java 9+

2018-06-21 Thread vaquar khan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519913#comment-16519913 ] vaquar khan edited comment on SPARK-24201 at 6/22/18 2:36 AM: -- Let me 

[jira] [Comment Edited] (SPARK-24201) IllegalArgumentException originating from ClosureCleaner in Java 9+

2018-06-21 Thread vaquar khan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519913#comment-16519913 ] vaquar khan edited comment on SPARK-24201 at 6/22/18 2:28 AM: -- Let me 

[jira] [Commented] (SPARK-24201) IllegalArgumentException originating from ClosureCleaner in Java 9+

2018-06-21 Thread vaquar khan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519913#comment-16519913 ] vaquar khan commented on SPARK-24201: - Let me fix Spark doc > IllegalArgumentException originating

[jira] [Commented] (SPARK-22897) Expose stageAttemptId in TaskContext

2018-06-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519902#comment-16519902 ] Apache Spark commented on SPARK-22897: -- User 'tgravescs' has created a pull request for this issue:

[jira] [Updated] (SPARK-24596) Non-cascading Cache Invalidation

2018-06-21 Thread Maryann Xue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maryann Xue updated SPARK-24596: Description: When invalidating a cache, we invalid other caches dependent on this cache to

[jira] [Updated] (SPARK-24621) WebUI - application 'name' urls point to http instead of https (even when ssl enabled)

2018-06-21 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-24621: - Description: See attached ApplicationID correctly points to DNS url but Name points to IP address Update: I

[jira] [Updated] (SPARK-24621) WebUI - application 'name' urls point to http instead of https (even when ssl enabled)

2018-06-21 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-24621: - Summary: WebUI - application 'name' urls point to http instead of https (even when ssl enabled) (was: WebUI -

[jira] [Resolved] (SPARK-24588) StreamingSymmetricHashJoinExec should require HashClusteredPartitioning from children

2018-06-21 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24588. - Resolution: Fixed Fix Version/s: 2.4.0 2.3.2 > StreamingSymmetricHashJoinExec

[jira] [Updated] (SPARK-24588) StreamingSymmetricHashJoinExec should require HashClusteredPartitioning from children

2018-06-21 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24588: Description: In [https://github.com/apache/spark/pull/19080], we simplified the

[jira] [Commented] (SPARK-24626) Improve Analyze Table command

2018-06-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519813#comment-16519813 ] Apache Spark commented on SPARK-24626: -- User 'Achuth17' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24626) Improve Analyze Table command

2018-06-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24626: Assignee: Apache Spark > Improve Analyze Table command > - >

[jira] [Assigned] (SPARK-24626) Improve Analyze Table command

2018-06-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24626: Assignee: (was: Apache Spark) > Improve Analyze Table command >

[jira] [Updated] (SPARK-24626) Improve Analyze Table command

2018-06-21 Thread Achuth Narayan Rajagopal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Achuth Narayan Rajagopal updated SPARK-24626: - Fix Version/s: (was: 2.4.0) > Improve Analyze Table command >

[jira] [Created] (SPARK-24626) Improve Analyze Table command

2018-06-21 Thread Achuth Narayan Rajagopal (JIRA)
Achuth Narayan Rajagopal created SPARK-24626: Summary: Improve Analyze Table command Key: SPARK-24626 URL: https://issues.apache.org/jira/browse/SPARK-24626 Project: Spark Issue

[jira] [Commented] (SPARK-24589) OutputCommitCoordinator may allow duplicate commits

2018-06-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519799#comment-16519799 ] Apache Spark commented on SPARK-24589: -- User 'vanzin' has created a pull request for this issue:

[jira] [Commented] (SPARK-22897) Expose stageAttemptId in TaskContext

2018-06-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519800#comment-16519800 ] Apache Spark commented on SPARK-22897: -- User 'vanzin' has created a pull request for this issue:

[jira] [Resolved] (SPARK-24589) OutputCommitCoordinator may allow duplicate commits

2018-06-21 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24589. --- Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 2.4.0

[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried

2018-06-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519723#comment-16519723 ] Apache Spark commented on SPARK-24552: -- User 'vanzin' has created a pull request for this issue:

[jira] [Updated] (SPARK-24615) Accelerator aware task scheduling for Spark

2018-06-21 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24615: -- Labels: Hydrogen SPIP (was: SPIP) > Accelerator aware task scheduling for Spark >

[jira] [Resolved] (SPARK-24613) Cache with UDF could not be matched with subsequent dependent caches

2018-06-21 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24613. - Resolution: Fixed Assignee: Maryann Xue > Cache with UDF could not be matched with subsequent

[jira] [Created] (SPARK-24625) put all the backward compatible behavior change configs under spark.sql.legacy.*

2018-06-21 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-24625: --- Summary: put all the backward compatible behavior change configs under spark.sql.legacy.* Key: SPARK-24625 URL: https://issues.apache.org/jira/browse/SPARK-24625

[jira] [Commented] (SPARK-24374) SPIP: Support Barrier Scheduling in Apache Spark

2018-06-21 Thread Mingjie Tang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519622#comment-16519622 ] Mingjie Tang commented on SPARK-24374: -- [~mengxr].  [~jerryshao] create a SISP [SPARK-24615] for

[jira] [Commented] (SPARK-24624) Can not mix vectorized and non-vectorized UDFs

2018-06-21 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519590#comment-16519590 ] Li Jin commented on SPARK-24624: I can take a look at this one > Can not mix vectorized and

[jira] [Commented] (SPARK-24624) Can not mix vectorized and non-vectorized UDFs

2018-06-21 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519586#comment-16519586 ] Xiao Li commented on SPARK-24624: - [~icexelloss] [~bryanc] [~hyukjin.kwon] Could you help this? > Can

[jira] [Created] (SPARK-24624) Can not mix vectorized and non-vectorized UDFs

2018-06-21 Thread Xiao Li (JIRA)
Xiao Li created SPARK-24624: --- Summary: Can not mix vectorized and non-vectorized UDFs Key: SPARK-24624 URL: https://issues.apache.org/jira/browse/SPARK-24624 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-17636) Parquet filter push down doesn't handle struct fields

2018-06-21 Thread Justin Miller (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519568#comment-16519568 ] Justin Miller commented on SPARK-17636: --- Any updates on this ticket? Thanks! > Parquet filter

[jira] [Updated] (SPARK-24588) StreamingSymmetricHashJoinExec should require HashClusteredPartitioning from children

2018-06-21 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24588: Description: In [https://github.com/apache/spark/pull/19080], we simplified the

[jira] [Created] (SPARK-24623) Hadoop - Spark Cluster - Python XGBoost - Not working in distributed mode

2018-06-21 Thread Abhishek Reddy Chamakura (JIRA)
Abhishek Reddy Chamakura created SPARK-24623: Summary: Hadoop - Spark Cluster - Python XGBoost - Not working in distributed mode Key: SPARK-24623 URL: https://issues.apache.org/jira/browse/SPARK-24623

[jira] [Commented] (SPARK-24385) Trivially-true EqualNullSafe should be handled like EqualTo in Dataset.join

2018-06-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519541#comment-16519541 ] Apache Spark commented on SPARK-24385: -- User 'mgaido91' has created a pull request for this issue:

[jira] [Resolved] (SPARK-24620) [security] Disable 'kill' button next to applications on Spark WebUI

2018-06-21 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24620. Resolution: Not A Bug Like "spark.ui.killEnabled"? > [security] Disable 'kill' button

[jira] [Comment Edited] (SPARK-24552) Task attempt numbers are reused when stages are retried

2018-06-21 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519445#comment-16519445 ] Thomas Graves edited comment on SPARK-24552 at 6/21/18 3:02 PM: more

[jira] [Comment Edited] (SPARK-24552) Task attempt numbers are reused when stages are retried

2018-06-21 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519445#comment-16519445 ] Thomas Graves edited comment on SPARK-24552 at 6/21/18 3:01 PM: more

[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried

2018-06-21 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519445#comment-16519445 ] Thomas Graves commented on SPARK-24552: --- more details on hadoop committer side: So I think the

[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried

2018-06-21 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519430#comment-16519430 ] Thomas Graves commented on SPARK-24552: --- this is actually a problem with hadoop committers, v1 and

[jira] [Commented] (SPARK-24611) Clean up OutputCommitCoordinator

2018-06-21 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519420#comment-16519420 ] Thomas Graves commented on SPARK-24611: --- [~joshrosen]  just noticed you were the last one to

[jira] [Commented] (SPARK-24622) Task attempts in other stage attempts not killed when one task attempt succeeds

2018-06-21 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519416#comment-16519416 ] Thomas Graves commented on SPARK-24622: --- Need to investigate further/test to make sure I am not

[jira] [Assigned] (SPARK-16630) Blacklist a node if executors won't launch on it.

2018-06-21 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-16630: Assignee: Attila Zsolt Piros > Blacklist a node if executors won't launch on it. >

[jira] [Resolved] (SPARK-16630) Blacklist a node if executors won't launch on it.

2018-06-21 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-16630. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21068

[jira] [Created] (SPARK-24622) Task attempts in other stage attempts not killed when one task attempt succeeds

2018-06-21 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-24622: - Summary: Task attempts in other stage attempts not killed when one task attempt succeeds Key: SPARK-24622 URL: https://issues.apache.org/jira/browse/SPARK-24622

[jira] [Updated] (SPARK-24620) [security] Disable 'kill' button next to applications on Spark WebUI

2018-06-21 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-24620: - Attachment: spark_master-one-app.png > [security] Disable 'kill' button next to applications on Spark WebUI >

[jira] [Updated] (SPARK-24621) WebUI - application 'name' urls point to IP instead of DNS name

2018-06-21 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-24621: - Attachment: spark_master-one-app.png > WebUI - application 'name' urls point to IP instead of DNS name >

[jira] [Updated] (SPARK-24620) [security] Disable 'kill' button next to applications on Spark WebUI

2018-06-21 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-24620: - Attachment: (was: spark_master-one-app.png) > [security] Disable 'kill' button next to applications on

[jira] [Created] (SPARK-24621) WebUI - application 'name' urls point to IP instead of DNS name

2018-06-21 Thread t oo (JIRA)
t oo created SPARK-24621: Summary: WebUI - application 'name' urls point to IP instead of DNS name Key: SPARK-24621 URL: https://issues.apache.org/jira/browse/SPARK-24621 Project: Spark Issue Type:

[jira] [Created] (SPARK-24620) [security] Disable 'kill' button next to applications on Spark WebUI

2018-06-21 Thread t oo (JIRA)
t oo created SPARK-24620: Summary: [security] Disable 'kill' button next to applications on Spark WebUI Key: SPARK-24620 URL: https://issues.apache.org/jira/browse/SPARK-24620 Project: Spark Issue

[jira] [Created] (SPARK-24619) DOCO - usage of authenticate.secret between masters/workers not clear

2018-06-21 Thread t oo (JIRA)
t oo created SPARK-24619: Summary: DOCO - usage of authenticate.secret between masters/workers not clear Key: SPARK-24619 URL: https://issues.apache.org/jira/browse/SPARK-24619 Project: Spark Issue

[jira] [Created] (SPARK-24618) Allow ability to consume driver memory on worker hosts not master (option for clustermode to wait for returncode?)

2018-06-21 Thread t oo (JIRA)
t oo created SPARK-24618: Summary: Allow ability to consume driver memory on worker hosts not master (option for clustermode to wait for returncode?) Key: SPARK-24618 URL:

[jira] [Created] (SPARK-24617) Spark driver not requesting another executor once original executor exits due to 'lost worker'

2018-06-21 Thread t oo (JIRA)
t oo created SPARK-24617: Summary: Spark driver not requesting another executor once original executor exits due to 'lost worker' Key: SPARK-24617 URL: https://issues.apache.org/jira/browse/SPARK-24617

[jira] [Commented] (SPARK-14643) Remove overloaded methods which become ambiguous in Scala 2.12

2018-06-21 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519312#comment-16519312 ] Stavros Kontopoulos commented on SPARK-14643: - [~srowen] [~joshrosen] there is a new

[jira] [Commented] (SPARK-14540) Support Scala 2.12 closures and Java 8 lambdas in ClosureCleaner

2018-06-21 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519310#comment-16519310 ] Stavros Kontopoulos commented on SPARK-14540: - [~srowen] There is a new discussion/doc here: 

[jira] [Updated] (SPARK-22911) Migrate structured streaming sources to new DataSourceV2 APIs

2018-06-21 Thread Teng Peng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teng Peng updated SPARK-22911: -- Issue Type: Umbrella (was: Improvement) > Migrate structured streaming sources to new DataSourceV2

[jira] [Commented] (SPARK-17248) Add native Scala enum support to Dataset Encoders

2018-06-21 Thread Angel Cervera Claudio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519214#comment-16519214 ] Angel Cervera Claudio commented on SPARK-17248: --- [~dongjin] suggestion is really good.

[jira] [Commented] (SPARK-24347) df.alias() in python API should not clear metadata by default

2018-06-21 Thread Ruben Berenguel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519191#comment-16519191 ] Ruben Berenguel commented on SPARK-24347: - [~holdenkarau] [~ueshin] (I ping you since you have

[jira] [Commented] (SPARK-21918) HiveClient shouldn't share Hive object between different thread

2018-06-21 Thread fengchaoge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519149#comment-16519149 ] fengchaoge commented on SPARK-21918: Hello Hu Liu, can you share you patch? we are suffering DDL

[jira] [Commented] (SPARK-24607) Distribute by rand() can lead to data inconsistency

2018-06-21 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519119#comment-16519119 ] Liang-Chi Hsieh commented on SPARK-24607: - [~gostop_zlx] If you don't give a seed to {{rand}},

[jira] [Resolved] (SPARK-24614) PySpark - Fix SyntaxWarning on tests.py

2018-06-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24614. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21604

[jira] [Assigned] (SPARK-24614) PySpark - Fix SyntaxWarning on tests.py

2018-06-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-24614: Assignee: Rekha Joshi > PySpark - Fix SyntaxWarning on tests.py >

[jira] [Updated] (SPARK-24615) Accelerator aware task scheduling for Spark

2018-06-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-24615: Labels: SPIP (was: ) > Accelerator aware task scheduling for Spark >

[jira] [Updated] (SPARK-24615) Accelerator aware task scheduling for Spark

2018-06-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-24615: Description: In the machine learning area, accelerator card (GPU, FPGA, TPU) is predominant

[jira] [Commented] (SPARK-24347) df.alias() in python API should not clear metadata by default

2018-06-21 Thread Ruben Berenguel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519074#comment-16519074 ] Ruben Berenguel commented on SPARK-24347: - Looking into this > df.alias() in python API should

[jira] [Updated] (SPARK-24615) Accelerator aware task scheduling for Spark

2018-06-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-24615: Description: In the machine learning area, accelerator card (GPU, FPGA, TPU) is predominant

[jira] [Created] (SPARK-24615) Accelerator aware task scheduling for Spark

2018-06-21 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-24615: --- Summary: Accelerator aware task scheduling for Spark Key: SPARK-24615 URL: https://issues.apache.org/jira/browse/SPARK-24615 Project: Spark Issue Type:

[jira] [Created] (SPARK-24616) Need to retreive free memory on command prompt on DSE cluster

2018-06-21 Thread Kushagra Bindal (JIRA)
Kushagra Bindal created SPARK-24616: --- Summary: Need to retreive free memory on command prompt on DSE cluster Key: SPARK-24616 URL: https://issues.apache.org/jira/browse/SPARK-24616 Project: Spark

[jira] [Resolved] (SPARK-24574) improve array_contains function of the sql component to deal with Column type

2018-06-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24574. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21581

[jira] [Assigned] (SPARK-24574) improve array_contains function of the sql component to deal with Column type

2018-06-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-24574: Assignee: Chongguang LIU > improve array_contains function of the sql component to deal

[jira] [Commented] (SPARK-24612) Running into "Py4JJavaError" while converting list to Dataframe using Pyspark, Jupyter notebook

2018-06-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16518994#comment-16518994 ] Hyukjin Kwon commented on SPARK-24612: -- Please refer this https://spark.apache.org/community.html

[jira] [Assigned] (SPARK-24614) PySpark - Fix SyntaxWarning on tests.py

2018-06-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24614: Assignee: (was: Apache Spark) > PySpark - Fix SyntaxWarning on tests.py >

[jira] [Assigned] (SPARK-24614) PySpark - Fix SyntaxWarning on tests.py

2018-06-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24614: Assignee: Apache Spark > PySpark - Fix SyntaxWarning on tests.py >

[jira] [Updated] (SPARK-24614) PySpark - Fix SyntaxWarning on tests.py

2018-06-21 Thread Rekha Joshi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rekha Joshi updated SPARK-24614: Description: Pyspark - Fix SyntaxWarning on tests.py (was: Pyspark - Fix for SyntaxWarning on

[jira] [Updated] (SPARK-24614) PySpark - Fix SyntaxWarning on tests.py

2018-06-21 Thread Rekha Joshi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rekha Joshi updated SPARK-24614: Summary: PySpark - Fix SyntaxWarning on tests.py (was: Pyspark tests.py codestyle correction) >

[jira] [Updated] (SPARK-24614) PySpark - Fix SyntaxWarning on tests.py

2018-06-21 Thread Rekha Joshi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rekha Joshi updated SPARK-24614: Description: Pyspark - Fix for SyntaxWarning on tests.py (was: Pyspark tests.py codestyle

[jira] [Commented] (SPARK-24614) PySpark - Fix SyntaxWarning on tests.py

2018-06-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16518991#comment-16518991 ] Apache Spark commented on SPARK-24614: -- User 'rekhajoshm' has created a pull request for this

[jira] [Resolved] (SPARK-24571) Support literals with values of the Char type

2018-06-21 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24571. - Resolution: Fixed Assignee: Maxim Gekk Fix Version/s: 2.4.0 > Support literals with

[jira] [Created] (SPARK-24614) Pyspark tests.py codestyle correction

2018-06-21 Thread Rekha Joshi (JIRA)
Rekha Joshi created SPARK-24614: --- Summary: Pyspark tests.py codestyle correction Key: SPARK-24614 URL: https://issues.apache.org/jira/browse/SPARK-24614 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-24612) Running into "Py4JJavaError" while converting list to Dataframe using Pyspark, Jupyter notebook

2018-06-21 Thread A B (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16518982#comment-16518982 ] A B commented on SPARK-24612: - [~hyukjin.kwon]  , What is the procedure to ask dev mailing list? > Running

[jira] [Commented] (SPARK-14410) SessionCatalog needs to check function existence

2018-06-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16518964#comment-16518964 ] Apache Spark commented on SPARK-14410: -- User 'rekhajoshm' has created a pull request for this

[jira] [Resolved] (SPARK-24612) Running into "Py4JJavaError" while converting list to Dataframe using Pyspark, Jupyter notebook

2018-06-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24612. -- Resolution: Invalid Let's ask the question into dev mailing list and file a JIRA when we are