[jira] [Commented] (SPARK-24530) Sphinx doesn't render autodoc_docstring_signature correctly (with Python 2?) and pyspark.ml docs are broken

[ https://issues.apache.org/jira/browse/SPARK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16529352#comment-16529352 ] Saisai Shao commented on SPARK-24530: - [~hyukjin.kwon], Spark 2.1.3 and 2.2.2 are on vote, can you

[jira] [Updated] (SPARK-24530) Sphinx doesn't render autodoc_docstring_signature correctly (with Python 2?) and pyspark.ml docs are broken

[ https://issues.apache.org/jira/browse/SPARK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-24530: Target Version/s: 2.1.3, 2.2.2, 2.3.2, 2.4.0 (was: 2.4.0) > Sphinx doesn't ren

Re: Does Livy Support Basic Auth?

No, it doesn't support, you should change the code to add a specific Filter to make it work. Harsch, Tim 于2018年6月30日周六 上午12:49写道： > I thought I read somewhere that Livy supports basic auth, but don't see > configurations for it. Does it? >

Re: Time for 2.3.2?

+1, like mentioned by Marcelo, these issues seems quite severe. I can work on the release if short of hands :). Thanks Jerry Marcelo Vanzin 于2018年6月28日周四 上午11:40写道： > +1. SPARK-24589 / SPARK-24552 are kinda nasty and we should get fixes > for those out. > > (Those are what delayed 2.2.2 and

[jira] [Commented] (LIVY-479) Enable livy.rsc.launcher.address configuration

[ https://issues.apache.org/jira/browse/LIVY-479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525775#comment-16525775 ] Saisai Shao commented on LIVY-479: -- Yes, I've left the comments on the PR. > Ena

Re: A new link request to my project and one question

, but it > would be better to add finer grained permissions. > > On Mon, Jun 25, 2018 at 6:30 PM, Saisai Shao > wrote: > > Yes, has a configuration "livy.superusers". Here in this case, the sql > > server user should be added as a superuser, who can impersonate other >

[jira] [Resolved] (SPARK-24418) Upgrade to Scala 2.11.12

[ https://issues.apache.org/jira/browse/SPARK-24418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24418. - Resolution: Fixed Issue resolved by pull request 21495 [https://github.com/apache/spark/pull

Re: A new link request to my project and one question

Yes, has a configuration "livy.superusers". Here in this case, the sql server user should be added as a superuser, who can impersonate other different users. Marcelo Vanzin 于2018年6月26日周二 上午9:12写道： > You're talking about another service between the user and the application. > > In that case a

[jira] [Commented] (SPARK-24646) Support wildcard '*' for to spark.yarn.dist.forceDownloadSchemes

[ https://issues.apache.org/jira/browse/SPARK-24646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523022#comment-16523022 ] Saisai Shao commented on SPARK-24646: - Hi [~vanzin], here is a specific example: We have

[jira] [Created] (SPARK-24646) Support wildcard '*' for to spark.yarn.dist.forceDownloadSchemes

Saisai Shao created SPARK-24646: --- Summary: Support wildcard '*' for to spark.yarn.dist.forceDownloadSchemes Key: SPARK-24646 URL: https://issues.apache.org/jira/browse/SPARK-24646 Project: Spark

[jira] [Commented] (SPARK-24374) SPIP: Support Barrier Scheduling in Apache Spark

[ https://issues.apache.org/jira/browse/SPARK-24374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519934#comment-16519934 ] Saisai Shao commented on SPARK-24374: - Hi [~mengxr] [~jiangxb1987] SPARK-24615 would be a part

[jira] [Commented] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

[ https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519931#comment-16519931 ] Saisai Shao commented on HIVE-16391: Gently ping [~hagleitn], would you please help to review

[jira] [Updated] (SPARK-24615) Accelerator aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-24615: Labels: SPIP (was: ) > Accelerator aware task scheduling for Sp

[jira] [Updated] (SPARK-24615) Accelerator aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-24615: Description: In the machine learning area, accelerator card (GPU, FPGA, TPU) is predominant

[jira] [Updated] (SPARK-24615) Accelerator aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-24615: Description: In the machine learning area, accelerator card (GPU, FPGA, TPU) is predominant

[jira] [Created] (SPARK-24615) Accelerator aware task scheduling for Spark

Saisai Shao created SPARK-24615: --- Summary: Accelerator aware task scheduling for Spark Key: SPARK-24615 URL: https://issues.apache.org/jira/browse/SPARK-24615 Project: Spark Issue Type

[jira] [Commented] (SPARK-24493) Kerberos Ticket Renewal is failing in Hadoop 2.8+ and Hadoop 3

[ https://issues.apache.org/jira/browse/SPARK-24493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16517720#comment-16517720 ] Saisai Shao commented on SPARK-24493: - This BUG is fixed in HDFS-12670, Spark should bump its

Re: A new link request to my project and one question

Sure, I will merge the website code, thanks! For proxyUser thing, I think there's no particular reason not adding it, maybe we just forgot to add the proxyUser support. It would be better if you could create a JIRA to track this issue. If you're familiar with Livy code, you can also submit a PR

[jira] [Updated] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

[ https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated HIVE-16391: --- Attachment: HIVE-16391.2.patch > Publish proper Hive 1.2 jars (without including all dependenc

[jira] [Commented] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

[ https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16511851#comment-16511851 ] Saisai Shao commented on HIVE-16391: I see. I can keep the "core" classifier and use an

[jira] [Resolved] (LIVY-477) Upgrade Livy Scala version to 2.11.12

[ https://issues.apache.org/jira/browse/LIVY-477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-477. -- Resolution: Fixed Fix Version/s: 0.6.0 > Upgrade Livy Scala version to 2.11

[jira] [Updated] (SPARK-24518) Using Hadoop credential provider API to store password

[ https://issues.apache.org/jira/browse/SPARK-24518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-24518: Description: Current Spark configs password in a plaintext way, like putting

[jira] [Updated] (SPARK-24518) Using Hadoop credential provider API to store password

[ https://issues.apache.org/jira/browse/SPARK-24518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-24518: Environment: (was: Current Spark configs password in a plaintext way, like putting

[jira] [Created] (SPARK-24518) Using Hadoop credential provider API to store password

Saisai Shao created SPARK-24518: --- Summary: Using Hadoop credential provider API to store password Key: SPARK-24518 URL: https://issues.apache.org/jira/browse/SPARK-24518 Project: Spark Issue

[jira] [Commented] (SPARK-23534) Spark run on Hadoop 3.0.0

[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506023#comment-16506023 ] Saisai Shao commented on SPARK-23534: - Yes, it is a Hadoop 2.8+ issue, but we don't a 2.8 profile

Re: Spark YARN job submission error (code 13)

In Spark on YARN, error code 13 means SparkContext doesn't initialize in time. You can check the yarn application log to get more information. BTW, did you just write a plain python script without creating SparkContext/SparkSession? Aakash Basu 于2018年6月8日周五 下午4:15写道： > Hi, > > I'm trying to

[jira] [Resolved] (SPARK-24487) Add support for RabbitMQ.

[ https://issues.apache.org/jira/browse/SPARK-24487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24487. - Resolution: Won't Fix > Add support for Rabbi

[jira] [Commented] (SPARK-24487) Add support for RabbitMQ.

[ https://issues.apache.org/jira/browse/SPARK-24487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505786#comment-16505786 ] Saisai Shao commented on SPARK-24487: - I'm sure Spark community will not merge this into Spark code

[jira] [Updated] (SPARK-23151) Provide a distribution of Spark with Hadoop 3.0

[ https://issues.apache.org/jira/browse/SPARK-23151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23151: Issue Type: Sub-task (was: Dependency upgrade) Parent: SPARK-23534 > Prov

[jira] [Commented] (SPARK-23151) Provide a distribution of Spark with Hadoop 3.0

[ https://issues.apache.org/jira/browse/SPARK-23151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505783#comment-16505783 ] Saisai Shao commented on SPARK-23151: - I will convert this as a subtask of SPARK-23534. > Prov

[jira] [Commented] (SPARK-23534) Spark run on Hadoop 3.0.0

[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505771#comment-16505771 ] Saisai Shao commented on SPARK-23534: - Spark with Hadoop 3 will be failed in token renew for long

[jira] [Comment Edited] (SPARK-24493) Kerberos Ticket Renewal is failing in Hadoop 2.8+ and Hadoop 3

[ https://issues.apache.org/jira/browse/SPARK-24493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505768#comment-16505768 ] Saisai Shao edited comment on SPARK-24493 at 6/8/18 6:56 AM: - Adding more

[jira] [Commented] (SPARK-24493) Kerberos Ticket Renewal is failing in Hadoop 2.8+ and Hadoop 3

[ https://issues.apache.org/jira/browse/SPARK-24493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505768#comment-16505768 ] Saisai Shao commented on SPARK-24493: - Adding more background, the issue is happened when building

[jira] [Updated] (SPARK-24493) Kerberos Ticket Renewal is failing in Hadoop 2.8+ and Hadoop 3

[ https://issues.apache.org/jira/browse/SPARK-24493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-24493: Summary: Kerberos Ticket Renewal is failing in Hadoop 2.8+ and Hadoop 3 (was: Kerberos Ticket

[jira] [Commented] (SPARK-24487) Add support for RabbitMQ.

[ https://issues.apache.org/jira/browse/SPARK-24487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505760#comment-16505760 ] Saisai Shao commented on SPARK-24487: - You can add it in your own package with DataSource API

[jira] [Updated] (SPARK-24493) Kerberos Ticket Renewal is failing in long running Spark job

[ https://issues.apache.org/jira/browse/SPARK-24493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-24493: Component/s: Spark Core > Kerberos Ticket Renewal is failing in long running Spark

[jira] [Updated] (SPARK-24493) Kerberos Ticket Renewal is failing in long running Spark job

[ https://issues.apache.org/jira/browse/SPARK-24493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-24493: Component/s: (was: Spark Core) YARN > Kerberos Ticket Renewal is fail

[jira] [Updated] (SPARK-24493) Kerberos Ticket Renewal is failing in long running Spark job

[ https://issues.apache.org/jira/browse/SPARK-24493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-24493: Priority: Major (was: Blocker) > Kerberos Ticket Renewal is failing in long running Spark

[jira] [Commented] (SPARK-24487) Add support for RabbitMQ.

[ https://issues.apache.org/jira/browse/SPARK-24487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505608#comment-16505608 ] Saisai Shao commented on SPARK-24487: - What's the usage and purpose to integrate RabbitMQ to Spark

[jira] [Commented] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

[ https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505607#comment-16505607 ] Saisai Shao commented on HIVE-16391: Any comment [~vanzin] [~ste...@apache.org]? > Publish pro

[jira] [Commented] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

[ https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503188#comment-16503188 ] Saisai Shao commented on HIVE-16391: Uploaded a new patch [^HIVE-16391.1.patch]to use the solution

[jira] [Updated] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

[ https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated HIVE-16391: --- Attachment: HIVE-16391.1.patch > Publish proper Hive 1.2 jars (without including all dependenc

[jira] [Commented] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

[ https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502976#comment-16502976 ] Saisai Shao commented on HIVE-16391: [~vanzin] one problem about your proposed solution: hive-exec

[jira] [Commented] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

[ https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502756#comment-16502756 ] Saisai Shao commented on HIVE-16391: {quote}The problem with that is that it changes the meaning

[jira] [Updated] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

[ https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated HIVE-16391: --- Fix Version/s: 1.2.3 Affects Version/s: 1.2.2 Attachment: HIVE-16391.patch

[jira] [Commented] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

[ https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501667#comment-16501667 ] Saisai Shao commented on HIVE-16391: I see, thanks. Will upload the patch. > Publish proper Hive

Re: [Spark Streaming] is spark.streaming.concurrentJobs a per node or a cluster global value ?

t is delayed), which will lead to unexpected results. thomas lavocat 于2018年6月5日周二 下午7:48写道： > > On 05/06/2018 13:44, Saisai Shao wrote: > > You need to read the code, this is an undocumented configuration. > > I'm on it right now, but, Spark is a big piece of software. >

Re: [Spark Streaming] is spark.streaming.concurrentJobs a per node or a cluster global value ?

; > On 05/06/2018 11:24, Saisai Shao wrote: > > spark.streaming.concurrentJobs is a driver side internal configuration, > this means that how many streaming jobs can be submitted concurrently in > one batch. Usually this should not be configured by user, unless you're > familiar

[jira] [Commented] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

[ https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501561#comment-16501561 ] Saisai Shao commented on HIVE-16391: Seems there's no permission for me to upload a file. > Publ

[jira] [Comment Edited] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

[ https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501561#comment-16501561 ] Saisai Shao edited comment on HIVE-16391 at 6/5/18 10:15 AM: - Seems there's

Re: [Spark Streaming] is spark.streaming.concurrentJobs a per node or a cluster global value ?

spark.streaming.concurrentJobs is a driver side internal configuration, this means that how many streaming jobs can be submitted concurrently in one batch. Usually this should not be configured by user, unless you're familiar with Spark Streaming internals, and know the implication of this

[jira] [Commented] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

[ https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501415#comment-16501415 ] Saisai Shao commented on HIVE-16391: I'm not sure if submitting a PR is a right way to review in Hive

[jira] [Commented] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

[ https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501285#comment-16501285 ] Saisai Shao commented on HIVE-16391: Hi [~joshrosen] I'm trying to make the hive changes as you

[jira] [Commented] (SPARK-20202) Remove references to org.spark-project.hive

[ https://issues.apache.org/jira/browse/SPARK-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501225#comment-16501225 ] Saisai Shao commented on SPARK-20202: - OK, for the 1st, I've already started working on it locally

[jira] [Commented] (SPARK-20202) Remove references to org.spark-project.hive

[ https://issues.apache.org/jira/browse/SPARK-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499707#comment-16499707 ] Saisai Shao commented on SPARK-20202: - What is our plan to to fix this issue, are we going to use

Re: Query regarding livy session

This is by design that Livy doesn't support concurrent execution of different requests. Because Livy doesn't know if the subsequent requests are semantically dependent or not. Rajeshkumar R 于2018年6月3日周日 下午12:08写道： > Hello >> I have a livy session which is linked to a spark context. This spark

[jira] [Commented] (SPARK-18673) Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version

[ https://issues.apache.org/jira/browse/SPARK-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497931#comment-16497931 ] Saisai Shao commented on SPARK-18673: - Thanks Steve, looking forward to your inputs. > Datafra

[jira] [Commented] (SPARK-24355) Improve Spark shuffle server responsiveness to non-ChunkFetch requests

[ https://issues.apache.org/jira/browse/SPARK-24355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497626#comment-16497626 ] Saisai Shao commented on SPARK-24355: - Do we have a test result before and after? > Improve Sp

[jira] [Commented] (SPARK-24448) File not found on the address SparkFiles.get returns on standalone cluster

[ https://issues.apache.org/jira/browse/SPARK-24448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497548#comment-16497548 ] Saisai Shao commented on SPARK-24448: - Does it only happen in standalone cluster mode, have you

[jira] [Commented] (SPARK-18673) Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version

[ https://issues.apache.org/jira/browse/SPARK-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496187#comment-16496187 ] Saisai Shao commented on SPARK-18673: - I created a PR for this issue [https://github.com/JoshRosen

[jira] [Resolved] (SPARK-23991) data loss when allocateBlocksToBatch

[ https://issues.apache.org/jira/browse/SPARK-23991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23991. - Resolution: Fixed Fix Version/s: 2.3.1 2.4.0 Issue resolved by pull

[jira] [Assigned] (SPARK-23991) data loss when allocateBlocksToBatch

[ https://issues.apache.org/jira/browse/SPARK-23991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-23991: --- Assignee: Gabor Somogyi > data loss when allocateBlocksToBa

[jira] [Resolved] (LIVY-472) Improve the logs for fail-to-create session

[ https://issues.apache.org/jira/browse/LIVY-472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-472. -- Resolution: Fixed Fix Version/s: 0.6.0 0.5.1 Issue resolved by pull request 96

[jira] [Created] (SPARK-24377) Make --py-files work in non pyspark application

Saisai Shao created SPARK-24377: --- Summary: Make --py-files work in non pyspark application Key: SPARK-24377 URL: https://issues.apache.org/jira/browse/SPARK-24377 Project: Spark Issue Type

[jira] [Commented] (LIVY-471) New session creation API set to support resource uploading

[ https://issues.apache.org/jira/browse/LIVY-471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16481375#comment-16481375 ] Saisai Shao commented on LIVY-471: -- I have a local PR for solution 2, which was simpler and straightforward

Re: [VOTE] Spark 2.3.1 (RC1)

+1, checked new py4j related changes. Marcelo Vanzin 于2018年5月17日周四 上午5:41写道： > This is actually in 2.3, jira is just missing the version. > > https://github.com/apache/spark/pull/20765 > > On Wed, May 16, 2018 at 2:14 PM, kant kodali wrote: > > I am not

[jira] [Commented] (SPARK-24241) Do not fail fast when dynamic resource allocation enabled with 0 executor

[ https://issues.apache.org/jira/browse/SPARK-24241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475480#comment-16475480 ] Saisai Shao commented on SPARK-24241: - Issue resolved by pull request 21290 https://github.com/apache

[jira] [Resolved] (SPARK-24241) Do not fail fast when dynamic resource allocation enabled with 0 executor

[ https://issues.apache.org/jira/browse/SPARK-24241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24241. - Resolution: Fixed > Do not fail fast when dynamic resource allocation enabled with 0 execu

[jira] [Updated] (SPARK-24241) Do not fail fast when dynamic resource allocation enabled with 0 executor

[ https://issues.apache.org/jira/browse/SPARK-24241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-24241: Fix Version/s: 2.4.0 > Do not fail fast when dynamic resource allocation enabled with 0 execu

[jira] [Assigned] (SPARK-24241) Do not fail fast when dynamic resource allocation enabled with 0 executor

[ https://issues.apache.org/jira/browse/SPARK-24241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-24241: --- Assignee: Kent Yao > Do not fail fast when dynamic resource allocation enabled wit

[jira] [Resolved] (LIVY-431) Livy batch mode ignore proxyUser parameter

[ https://issues.apache.org/jira/browse/LIVY-431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-431. -- Resolution: Not A Problem > Livy batch mode ignore proxyUser parame

[jira] [Commented] (LIVY-431) Livy batch mode ignore proxyUser parameter

[ https://issues.apache.org/jira/browse/LIVY-431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475255#comment-16475255 ] Saisai Shao commented on LIVY-431: -- I think you should enable impersonation first in Livy

[jira] [Commented] (LIVY-469) "shared" session kind is undocumented

[ https://issues.apache.org/jira/browse/LIVY-469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475221#comment-16475221 ] Saisai Shao commented on LIVY-469: -- It is by intention. Starting from 0.5 session kind is actually useless

About new API set to create interactive and batch session

Hi Team, In our current API design to create interactive / batch session, we assume end user could upload jars, pyFiles and related dependencies to HDFS before creating the session, and we use one POST request to create session. But usually end user may not have the permission to access the HDFS

[jira] [Assigned] (SPARK-24182) Improve error message for client mode when AM fails

[ https://issues.apache.org/jira/browse/SPARK-24182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-24182: --- Assignee: Marcelo Vanzin > Improve error message for client mode when AM fa

[jira] [Resolved] (SPARK-24182) Improve error message for client mode when AM fails

[ https://issues.apache.org/jira/browse/SPARK-24182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24182. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21243 [https

[jira] [Created] (SPARK-24219) Improve the docker build script to avoid copying everything in example

Saisai Shao created SPARK-24219: --- Summary: Improve the docker build script to avoid copying everything in example Key: SPARK-24219 URL: https://issues.apache.org/jira/browse/SPARK-24219 Project: Spark

[jira] [Resolved] (SPARK-24188) /api/v1/version not working

[ https://issues.apache.org/jira/browse/SPARK-24188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24188. - Resolution: Fixed Fix Version/s: 2.3.1 2.4.0 Issue resolved by pull

[jira] [Assigned] (SPARK-24188) /api/v1/version not working

[ https://issues.apache.org/jira/browse/SPARK-24188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-24188: --- Assignee: Marcelo Vanzin > /api/v1/version not work

[jira] [Commented] (SPARK-24174) Expose Hadoop config as part of /environment API

[ https://issues.apache.org/jira/browse/SPARK-24174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463575#comment-16463575 ] Saisai Shao commented on SPARK-24174: - I believe Hadoop web UI already expose such configurations

[jira] [Assigned] (SPARK-24136) MemoryStreamDataReader.next should skip sleeping if record is available

[ https://issues.apache.org/jira/browse/SPARK-24136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-24136: --- Assignee: Arun Mahadevan > MemoryStreamDataReader.next should skip sleeping if rec

[jira] [Resolved] (SPARK-24136) MemoryStreamDataReader.next should skip sleeping if record is available

[ https://issues.apache.org/jira/browse/SPARK-24136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24136. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21207 [https

[jira] [Resolved] (LIVY-466) RSCDriver throws exception during RPC shutdown

[ https://issues.apache.org/jira/browse/LIVY-466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-466. -- Resolution: Fixed Fix Version/s: 0.6.0 0.5.1 Issue resolved by pull request 90

[jira] [Resolved] (SPARK-24110) Avoid calling UGI loginUserFromKeytab in ThriftServer

[ https://issues.apache.org/jira/browse/SPARK-24110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24110. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21178 [https

[jira] [Assigned] (SPARK-24110) Avoid calling UGI loginUserFromKeytab in ThriftServer

[ https://issues.apache.org/jira/browse/SPARK-24110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-24110: --- Assignee: Saisai Shao > Avoid calling UGI loginUserFromKeytab in ThriftSer

Re: [Spark Streaming]: Does DStream workload run over Spark SQL engine?

No, the underlying of DStream is RDD, so it will not leverage any SparkSQL related feature. I think you should use Structured Streaming instead, which is based on SparkSQL. Khaled Zaouk 于2018年5月2日周三 下午4:51写道： > Hi, > > I have a question regarding the execution engine of

[jira] [Updated] (LIVY-465) Client closed before SASL negotiation finished

[ https://issues.apache.org/jira/browse/LIVY-465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated LIVY-465: - Issue Type: Question (was: Bug) > Client closed before SASL negotiation finis

[jira] [Updated] (LIVY-463) Port for accessing livy server

[ https://issues.apache.org/jira/browse/LIVY-463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated LIVY-463: - Fix Version/s: (was: 0.5.1) > Port for accessing livy ser

[jira] [Assigned] (SPARK-23688) Refactor tests away from rate source

[ https://issues.apache.org/jira/browse/SPARK-23688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-23688: --- Assignee: Jungtaek Lim > Refactor tests away from rate sou

[jira] [Resolved] (SPARK-23688) Refactor tests away from rate source

[ https://issues.apache.org/jira/browse/SPARK-23688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23688. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21152 [https

[jira] [Assigned] (SPARK-23830) Spark on YARN in cluster deploy mode fail with NullPointerException when a Spark application is a Scala class not object

[ https://issues.apache.org/jira/browse/SPARK-23830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-23830: --- Assignee: Eric Maynard > Spark on YARN in cluster deploy mode f

[jira] [Resolved] (SPARK-23830) Spark on YARN in cluster deploy mode fail with NullPointerException when a Spark application is a Scala class not object

[ https://issues.apache.org/jira/browse/SPARK-23830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23830. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21168 [https

[jira] [Created] (SPARK-24110) Avoid calling UGI loginUserFromKeytab in ThriftServer

Saisai Shao created SPARK-24110: --- Summary: Avoid calling UGI loginUserFromKeytab in ThriftServer Key: SPARK-24110 URL: https://issues.apache.org/jira/browse/SPARK-24110 Project: Spark Issue

[jira] [Commented] (SPARK-23830) Spark on YARN in cluster deploy mode fail with NullPointerException when a Spark application is a Scala class not object

[ https://issues.apache.org/jira/browse/SPARK-23830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455621#comment-16455621 ] Saisai Shao commented on SPARK-23830: - I agree [~emaynard]. > Spark on YARN in cluster deploy m

[jira] [Commented] (SPARK-24062) SASL encryption cannot be worked in ThriftServer

[ https://issues.apache.org/jira/browse/SPARK-24062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453521#comment-16453521 ] Saisai Shao commented on SPARK-24062: - Issue resolved by pull request 21138 [https://github.com

[jira] [Resolved] (SPARK-24062) SASL encryption cannot be worked in ThriftServer

[ https://issues.apache.org/jira/browse/SPARK-24062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24062. - Resolution: Fixed Assignee: Saisai Shao Fix Version/s: 2.4.0

[jira] [Resolved] (LIVY-460) when using livy with spark2.3, something wrong with the stdout and stderr

[ https://issues.apache.org/jira/browse/LIVY-460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-460. -- Resolution: Fixed Fix Version/s: 0.6.0 Issue resolved by pull request 89 [https://github.com

[jira] [Commented] (LIVY-460) when using livy with spark2.3, something wrong with the stdout and stderr

[ https://issues.apache.org/jira/browse/LIVY-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451522#comment-16451522 ] Saisai Shao commented on LIVY-460: -- I think the word "stdout" is misleading, actually it stil

[jira] [Created] (SPARK-24062) SASL encryption cannot be worked in ThriftServer

Saisai Shao created SPARK-24062: --- Summary: SASL encryption cannot be worked in ThriftServer Key: SPARK-24062 URL: https://issues.apache.org/jira/browse/SPARK-24062 Project: Spark Issue Type

[jira] [Assigned] (SPARK-22372) Make YARN client extend SparkApplication

[ https://issues.apache.org/jira/browse/SPARK-22372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-22372: --- Assignee: Marcelo Vanzin > Make YARN client extend SparkApplicat

<    1   2   3   4   5   6   7   8   9   10   >