[jira] [Commented] (SPARK-23187) Accumulator object can not be sent from Executor to Driver

[ https://issues.apache.org/jira/browse/SPARK-23187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338737#comment-16338737 ] Saisai Shao commented on SPARK-23187: - Actually heartbeat report is OK according to my investigation

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338553#comment-16338553 ] Saisai Shao commented on SPARK-23206: - I think this Jira duplicates SPARK-9103. Also seems some

Fwd: [apache/incubator-livy] One of your dependencies may have a security vulnerability

Hi Alex, Is it due to your recent changes to add ruby file? Thanks Jerry -- Forwarded message -- From: Greg Stein Date: 2018-01-24 6:14 GMT+08:00 Subject: Fwd: [apache/incubator-livy] One of your dependencies may have a security vulnerability To:

[jira] [Commented] (SPARK-23187) Accumulator object can not be sent from Executor to Driver

[ https://issues.apache.org/jira/browse/SPARK-23187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335428#comment-16335428 ] Saisai Shao commented on SPARK-23187: - I will try to investigate on it. > Accumulator object

[jira] [Updated] (SPARK-23187) Accumulator object can not be sent from Executor to Driver

[ https://issues.apache.org/jira/browse/SPARK-23187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23187: Affects Version/s: (was: 2.3.1) 2.3.0 > Accumulator object

[jira] [Resolved] (SPARK-22838) Avoid unnecessary copying of data

[ https://issues.apache.org/jira/browse/SPARK-22838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-22838. - Resolution: Invalid > Avoid unnecessary copying of d

[jira] [Assigned] (SPARK-22976) Worker cleanup can remove running driver directories

[ https://issues.apache.org/jira/browse/SPARK-22976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-22976: --- Assignee: Russell Spitzer > Worker cleanup can remove running driver director

[jira] [Resolved] (SPARK-22976) Worker cleanup can remove running driver directories

[ https://issues.apache.org/jira/browse/SPARK-22976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-22976. - Resolution: Fixed Fix Version/s: 2.3.0 > Worker cleanup can remove running dri

[jira] [Commented] (SPARK-23097) Migrate text socket source

[ https://issues.apache.org/jira/browse/SPARK-23097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331735#comment-16331735 ] Saisai Shao commented on SPARK-23097: - Thanks, will do. > Migrate text socket sou

[jira] [Commented] (SPARK-23097) Migrate text socket source

[ https://issues.apache.org/jira/browse/SPARK-23097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331715#comment-16331715 ] Saisai Shao commented on SPARK-23097: - Hi [~joseph.torres], would you mind if I take a shot

[jira] [Commented] (LIVY-434) Livy 0.5-incubating Release

[ https://issues.apache.org/jira/browse/LIVY-434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331591#comment-16331591 ] Saisai Shao commented on LIVY-434: -- >From permission side I think it is OK to do it. But I'm not s

[jira] [Commented] (LIVY-434) Livy 0.5-incubating Release

[ https://issues.apache.org/jira/browse/LIVY-434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331580#comment-16331580 ] Saisai Shao commented on LIVY-434: -- LGTM. > Livy 0.5-incubating Rele

[jira] [Commented] (LIVY-434) Livy 0.5-incubating Release

[ https://issues.apache.org/jira/browse/LIVY-434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331559#comment-16331559 ] Saisai Shao commented on LIVY-434: -- Sure. We can test with Spark 2.3. But my concern is that 2.3 is a big

[jira] [Commented] (LIVY-434) Livy 0.5-incubating Release

[ https://issues.apache.org/jira/browse/LIVY-434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331549#comment-16331549 ] Saisai Shao commented on LIVY-434: -- IMO, I think we should have a timeline for 0.5 release, which cannot

[jira] [Commented] (SPARK-23128) A new approach to do adaptive execution in Spark SQL

[ https://issues.apache.org/jira/browse/SPARK-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331542#comment-16331542 ] Saisai Shao commented on SPARK-23128: - I'm wondering if we need a SPIP for such changes? > A

[jira] [Updated] (SPARK-23147) Stage page will throw exception when there's no complete tasks

[ https://issues.apache.org/jira/browse/SPARK-23147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23147: Description: Current Stage page's task table will throw an exception when there's no complete

[jira] [Created] (SPARK-23147) Stage page will throw exception when there's no complete tasks

Saisai Shao created SPARK-23147: --- Summary: Stage page will throw exception when there's no complete tasks Key: SPARK-23147 URL: https://issues.apache.org/jira/browse/SPARK-23147 Project: Spark

[jira] [Updated] (SPARK-23147) Stage page will throw exception when there's no complete tasks

[ https://issues.apache.org/jira/browse/SPARK-23147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23147: Attachment: Screen Shot 2018-01-18 at 8.50.08 PM.png > Stage page will throw exception w

[jira] [Updated] (SPARK-23140) DataSourceV2Strategy is missing in HiveSessionStateBuilder

[ https://issues.apache.org/jira/browse/SPARK-23140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23140: Description: DataSourceV2Strategy is not added into HiveSessionStateBuilder's planner, which

[jira] [Updated] (SPARK-23140) DataSourceV2Strategy is missing in HiveSessionStateBuilder

[ https://issues.apache.org/jira/browse/SPARK-23140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23140: Environment: (was: DataSourceV2Strategy is not added into HiveSessionStateBuilder's planner

[jira] [Created] (SPARK-23140) DataSourceV2Strategy is missing in HiveSessionStateBuilder

Saisai Shao created SPARK-23140: --- Summary: DataSourceV2Strategy is missing in HiveSessionStateBuilder Key: SPARK-23140 URL: https://issues.apache.org/jira/browse/SPARK-23140 Project: Spark

[jira] [Commented] (SPARK-23123) Unable to run Spark Job with Hadoop NameNode Federation using ViewFS

[ https://issues.apache.org/jira/browse/SPARK-23123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328364#comment-16328364 ] Saisai Shao commented on SPARK-23123: - [~ste...@apache.org], any thought on this issue? > Una

[jira] [Commented] (SPARK-23123) Unable to run Spark Job with Hadoop NameNode Federation using ViewFS

[ https://issues.apache.org/jira/browse/SPARK-23123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328361#comment-16328361 ] Saisai Shao commented on SPARK-23123: - >From the stack, looks like the issue is from Y

[jira] [Commented] (SPARK-23123) Unable to run Spark Job with Hadoop NameNode Federation using ViewFS

[ https://issues.apache.org/jira/browse/SPARK-23123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328297#comment-16328297 ] Saisai Shao commented on SPARK-23123: - It looks like a Hadoop issue rather than a Spark issue from

[jira] [Commented] (SPARK-22958) Spark is stuck when the only one executor fails to register with driver

[ https://issues.apache.org/jira/browse/SPARK-22958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325986#comment-16325986 ] Saisai Shao commented on SPARK-22958: - Please see this line: https://github.com/apache/spark/blob

[jira] [Commented] (SPARK-23029) Setting spark.shuffle.file.buffer will make the shuffle fail

[ https://issues.apache.org/jira/browse/SPARK-23029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323872#comment-16323872 ] Saisai Shao commented on SPARK-23029: - There should be no issue unless you want to fix doc issue

[jira] [Commented] (SPARK-23029) Setting spark.shuffle.file.buffer will make the shuffle fail

[ https://issues.apache.org/jira/browse/SPARK-23029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323871#comment-16323871 ] Saisai Shao commented on SPARK-23029: - Total buffer size for one executor is {{32MB * C (core) * R

[jira] [Commented] (SPARK-21475) Change to use NIO's Files API for external shuffle service

[ https://issues.apache.org/jira/browse/SPARK-21475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323590#comment-16323590 ] Saisai Shao commented on SPARK-21475: - [~zsxwing] is 3.0.0 the valid fix version? > Change to

[jira] [Commented] (SPARK-22958) Spark is stuck when the only one executor fails to register with driver

[ https://issues.apache.org/jira/browse/SPARK-22958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323569#comment-16323569 ] Saisai Shao commented on SPARK-22958: - If executor is failed to register itself to driver

[jira] [Resolved] (SPARK-23029) Setting spark.shuffle.file.buffer will make the shuffle fail

[ https://issues.apache.org/jira/browse/SPARK-23029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23029. - Resolution: Not A Problem > Setting spark.shuffle.file.buffer will make the shuffle f

[jira] [Commented] (SPARK-23029) Setting spark.shuffle.file.buffer will make the shuffle fail

[ https://issues.apache.org/jira/browse/SPARK-23029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323496#comment-16323496 ] Saisai Shao commented on SPARK-23029: - Please see the comments in the code

[jira] [Commented] (SPARK-22921) Merge script should prompt for assigning jiras

[ https://issues.apache.org/jira/browse/SPARK-22921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16321669#comment-16321669 ] Saisai Shao commented on SPARK-22921: - Hi [~irashid], looks like the changes will throw an exception

[jira] [Resolved] (SPARK-22587) Spark job fails if fs.defaultFS and application jar are different url

[ https://issues.apache.org/jira/browse/SPARK-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-22587. - Resolution: Fixed > Spark job fails if fs.defaultFS and application jar are different

[jira] [Updated] (SPARK-22587) Spark job fails if fs.defaultFS and application jar are different url

[ https://issues.apache.org/jira/browse/SPARK-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-22587: Fix Version/s: 2.3.0 > Spark job fails if fs.defaultFS and application jar are different

[jira] [Assigned] (SPARK-22587) Spark job fails if fs.defaultFS and application jar are different url

[ https://issues.apache.org/jira/browse/SPARK-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-22587: --- Assignee: Mingjie Tang > Spark job fails if fs.defaultFS and application jar are differ

Re: [DRAFT] Incubator PMC Board Report - January 2018

Sorry I was on paternity leave last two weeks, didn't have time to check it. Maybe you can also help on this next time. Thanks Jerry 2018-01-09 4:32 GMT+08:00 Alex Bozarth : > Saw that we didn't report this month, any reason why, or was everyone just > on holiday? > > >

Re: Is Apache Spark-2.2.1 compatible with Hadoop-3.0.0

AFAIK, there's no large scale test for Hadoop 3.0 in the community. So it is not clear whether it is supported or not (or has some issues). I think in the download page "Pre-Built for Apache Hadoop 2.7 and later" mostly means that it supports Hadoop 2.7+ (2.8...), but not 3.0 (IIUC). Thanks Jerry

[jira] [Commented] (SPARK-22876) spark.yarn.am.attemptFailuresValidityInterval does not work correctly

[ https://issues.apache.org/jira/browse/SPARK-22876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16309363#comment-16309363 ] Saisai Shao commented on SPARK-22876: - This looks like a bug, ideally we should get number

[jira] [Commented] (LIVY-426) Livy stops session quietly without error message

[ https://issues.apache.org/jira/browse/LIVY-426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281417#comment-16281417 ] Saisai Shao commented on LIVY-426: -- Sorry I misunderstood your issue, what I mean is that I also saw

[jira] [Commented] (LIVY-426) Livy stops session quietly without error message

[ https://issues.apache.org/jira/browse/LIVY-426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281335#comment-16281335 ] Saisai Shao commented on LIVY-426: -- I also encountered this exception before. Do you have a fix

Re: How to set driverMemory, driverCores, executorMemory using livy?

Using this API "public LivyClientBuilder setConf(String key, String value)" to set Spark configurations you wanted. 2017-12-06 15:34 GMT+08:00 kant kodali : > Hi All, > > > I do see POST /sessions API where I can pass driverMemory, driverCores, > executorMemory as part of

[jira] [Assigned] (LIVY-19) Add Spark SQL support

[ https://issues.apache.org/jira/browse/LIVY-19?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-19: --- Assignee: Saisai Shao > Add Spark SQL support > - > >

[jira] [Commented] (LIVY-104) Switch to scala 2.11

[ https://issues.apache.org/jira/browse/LIVY-104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276172#comment-16276172 ] Saisai Shao commented on LIVY-104: -- Livy server itself requires Spark artifacts to run unit tests, if we

[jira] [Commented] (LIVY-421) support multi statements run at the same time

[ https://issues.apache.org/jira/browse/LIVY-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16274174#comment-16274174 ] Saisai Shao commented on LIVY-421: -- bq. Adding concurrency for sql make sense for me as each sql statements

[jira] [Commented] (LIVY-104) Switch to scala 2.11

[ https://issues.apache.org/jira/browse/LIVY-104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16274123#comment-16274123 ] Saisai Shao commented on LIVY-104: -- Hi [~ajbozarth] would you please drive this issue if you have time, I'm

[jira] [Created] (LIVY-423) Adding Scala 2.12 support

Saisai Shao created LIVY-423: Summary: Adding Scala 2.12 support Key: LIVY-423 URL: https://issues.apache.org/jira/browse/LIVY-423 Project: Livy Issue Type: New Feature Components

[jira] [Commented] (SPARK-22393) spark-shell can't find imported types in class constructors, extends clause

[ https://issues.apache.org/jira/browse/SPARK-22393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273985#comment-16273985 ] Saisai Shao commented on SPARK-22393: - Shall we wait for this before 2.2.1 is out? > spark-sh

Re: Does Apache Livy support Spark Structured Streaming 2.2.0?

com>: > Nice! so if I submit a streaming query over REST can I get the results > back over REST or Websocket? > > On Tue, Nov 28, 2017 at 10:29 PM, Saisai Shao <sai.sai.s...@gmail.com> > wrote: > >> Livy doesn't add any restriction on how user uses Spark API,

Re: Does anyone know how to build spark with scala12.4?

experimental and for > people willing to make their own build. That's why I wanted it in good > enough shape that the scala-2.12 profile produces something basically > functional. > > On Tue, Nov 28, 2017 at 8:43 PM Saisai Shao <sai.sai.s...@gmail.com> > wrote: > >

Re: Does anyone know how to build spark with scala12.4?

Hi Sean, Two questions about Scala 2.12 for release artifacts. Are we planning to ship 2.12 artifacts for Spark 2.3 release? If not, will we only ship 2.11 artifacts? Thanks Jerry 2017-11-28 21:51 GMT+08:00 Sean Owen : > The Scala 2.12 profile mostly works, but not all

Re: How to access Python script's stdout when running in Yarn cluster mode?

applications but it looks > like I might have no choice about this. > > > > Thanks, Lucas. > > > > *From:* Saisai Shao [mailto:sai.sai.s...@gmail.com] > *Sent:* 27 November 2017 02:19 > *To:* user@livy.incubator.apache.org > *Subject:* EXT: Re: How to access Python

[jira] [Commented] (LIVY-421) support multi statements run at the same time

[ https://issues.apache.org/jira/browse/LIVY-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266535#comment-16266535 ] Saisai Shao commented on LIVY-421: -- I agree with what Jeff mentioned, multi-threading is dangerous

Re: How to access Python script's stdout when running in Yarn cluster mode?

Since you're running with yarn cluster mode, the output from your python script should be part of your yarn application log. you can get it via yarn command like yarn log -applicationId , or others like Yarn UI. Currently there's no Livy api for you to get application log via REST API. Thanks

[jira] [Commented] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265035#comment-16265035 ] Saisai Shao commented on SPARK-2926: So your saying of 12x-30x boosting is only referring to reduce

[jira] [Commented] (SPARK-22579) BlockManager.getRemoteValues and BlockManager.getRemoteBytes should be implemented using streaming

[ https://issues.apache.org/jira/browse/SPARK-22579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265021#comment-16265021 ] Saisai Shao commented on SPARK-22579: - I think this issue should have already been fixed by SPARK

[jira] [Commented] (SPARK-22587) Spark job fails if fs.defaultFS and application jar are different url

[ https://issues.apache.org/jira/browse/SPARK-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264986#comment-16264986 ] Saisai Shao commented on SPARK-22587: - [~Prabhu Joseph], I think it is because the logic of comparing

[jira] [Resolved] (LIVY-299) Only the output of last line is returned

[ https://issues.apache.org/jira/browse/LIVY-299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-299. -- Resolution: Fixed Fix Version/s: 0.5.0 > Only the output of last line is retur

[jira] [Commented] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264904#comment-16264904 ] Saisai Shao commented on SPARK-2926: [~XuanYuan], would you please use spark-perf's micro benchmark

[jira] [Assigned] (LIVY-416) com.fasterxml.jackson.core.JsonGenerationException is sometimes thrown

[ https://issues.apache.org/jira/browse/LIVY-416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-416: Assignee: Keiji Yoshida > com.fasterxml.jackson.core.JsonGenerationException is sometimes thr

[jira] [Resolved] (LIVY-416) com.fasterxml.jackson.core.JsonGenerationException is sometimes thrown

[ https://issues.apache.org/jira/browse/LIVY-416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-416. -- Resolution: Fixed Fix Version/s: 0.5.0 > com.fasterxml.jackson.core.JsonGenerationExcept

[jira] [Assigned] (LIVY-418) Respect spark.pyspark.python for PythonInterpreter

[ https://issues.apache.org/jira/browse/LIVY-418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-418: Assignee: Jeff Zhang > Respect spark.pyspark.python for PythonInterpre

[jira] [Resolved] (LIVY-418) Respect spark.pyspark.python for PythonInterpreter

[ https://issues.apache.org/jira/browse/LIVY-418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-418. -- Resolution: Fixed Fix Version/s: 0.5.0 > Respect spark.pyspark.python for PythonInterpre

[jira] [Assigned] (LIVY-299) Only the output of last line is returned

[ https://issues.apache.org/jira/browse/LIVY-299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-299: Assignee: Saisai Shao > Only the output of last line is retur

[jira] [Commented] (SPARK-22528) History service and non-HDFS filesystems

[ https://issues.apache.org/jira/browse/SPARK-22528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254669#comment-16254669 ] Saisai Shao commented on SPARK-22528: - Hi [~adobe_pmackles] I think this issue might be due

[jira] [Assigned] (LIVY-415) Use objects and abstract classes in for Kind and SessionState.

[ https://issues.apache.org/jira/browse/LIVY-415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-415: Assignee: Arun Allamsetty > Use objects and abstract classes in for Kind and SessionSt

[jira] [Updated] (LIVY-415) Use objects and abstract classes in for Kind and SessionState.

[ https://issues.apache.org/jira/browse/LIVY-415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated LIVY-415: - Fix Version/s: 0.5.0 > Use objects and abstract classes in for Kind and SessionSt

[jira] [Updated] (LIVY-415) Use objects and abstract classes in for Kind and SessionState.

[ https://issues.apache.org/jira/browse/LIVY-415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated LIVY-415: - Component/s: Server REPL > Use objects and abstract classes in for Kind and SessionSt

[jira] [Commented] (SPARK-9104) expose network layer memory usage

[ https://issues.apache.org/jira/browse/SPARK-9104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250675#comment-16250675 ] Saisai Shao commented on SPARK-9104: [~vsr] I think SPARK-21934 already exposed Netty shuffle metrics

[jira] [Created] (LIVY-414) Support setting environment variables during session creation

Saisai Shao created LIVY-414: Summary: Support setting environment variables during session creation Key: LIVY-414 URL: https://issues.apache.org/jira/browse/LIVY-414 Project: Livy Issue Type

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

+1, looking forward to more design details of this feature. Thanks Jerry On Wed, Nov 8, 2017 at 6:40 AM, Shixiong(Ryan) Zhu wrote: > +1 > > On Tue, Nov 7, 2017 at 1:34 PM, Joseph Bradley > wrote: > >> +1 >> >> On Mon, Nov 6, 2017 at 5:11 PM,

[jira] [Assigned] (LIVY-412) CreateSession should be reject when too many child process "spark-submit" is running

[ https://issues.apache.org/jira/browse/LIVY-412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-412: Assignee: Hong Shen > CreateSession should be reject when too many child process "spar

[jira] [Resolved] (LIVY-412) CreateSession should be reject when too many child process "spark-submit" is running

[ https://issues.apache.org/jira/browse/LIVY-412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-412. -- Resolution: Fixed Fix Version/s: 0.5.0 Issue resolved by pull request 58 [https://github.com

[jira] [Commented] (SPARK-22426) Spark AM launching containers on node where External spark shuffle service failed to initialize

[ https://issues.apache.org/jira/browse/SPARK-22426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236939#comment-16236939 ] Saisai Shao commented on SPARK-22426: - This kind of scenario was handled in SPARK-13669 regarding

[jira] [Commented] (SPARK-22405) Enrich the event information and add new event of ExternalCatalogEvent

[ https://issues.apache.org/jira/browse/SPARK-22405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235042#comment-16235042 ] Saisai Shao commented on SPARK-22405: - Thanks [~hvanhovell] for your comment, let me propose

[jira] [Assigned] (SPARK-22172) Worker hangs when the external shuffle service port is already in use

[ https://issues.apache.org/jira/browse/SPARK-22172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-22172: --- Assignee: Devaraj K > Worker hangs when the external shuffle service port is already in

[jira] [Resolved] (SPARK-22172) Worker hangs when the external shuffle service port is already in use

[ https://issues.apache.org/jira/browse/SPARK-22172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-22172. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19396 [https

[jira] [Commented] (SPARK-22405) Enrich the event information and add new event of ExternalCatalogEvent

[ https://issues.apache.org/jira/browse/SPARK-22405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16233569#comment-16233569 ] Saisai Shao commented on SPARK-22405: - Thanks [~hvanhovell] for your comments. bq. When implementing

[jira] [Commented] (SPARK-22405) Enrich the event information and add new event of ExternalCatalogEvent

[ https://issues.apache.org/jira/browse/SPARK-22405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16226724#comment-16226724 ] Saisai Shao commented on SPARK-22405: - Thanks [~cloud_fan] for your comments. Using fake

[jira] [Commented] (SPARK-22405) Enrich the event information and add new event of ExternalCatalogEvent

[ https://issues.apache.org/jira/browse/SPARK-22405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16226474#comment-16226474 ] Saisai Shao commented on SPARK-22405: - CC [~smilegator], what's your opinion about this proposal

[jira] [Commented] (SPARK-22405) Enrich the event information and add new event of ExternalCatalogEvent

[ https://issues.apache.org/jira/browse/SPARK-22405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16226468#comment-16226468 ] Saisai Shao commented on SPARK-22405: - This is the WIP branch (https://github.com/jerryshao/apache

[jira] [Created] (SPARK-22405) Enrich the event information of ExternalCatalogEvent

Saisai Shao created SPARK-22405: --- Summary: Enrich the event information of ExternalCatalogEvent Key: SPARK-22405 URL: https://issues.apache.org/jira/browse/SPARK-22405 Project: Spark Issue

[jira] [Updated] (SPARK-22405) Enrich the event information and add new event of ExternalCatalogEvent

[ https://issues.apache.org/jira/browse/SPARK-22405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-22405: Summary: Enrich the event information and add new event of ExternalCatalogEvent (was: Enrich

Re: ClassNotFoundException on job submit

You can choose to set "livy.spark.master" to "local" and "livy.spark.deploy-mode" to "client" to start Spark with local mode, in such case YARN is not required. Otherwise if you plan to run on YARN, you have to install Hadoop and configure HADOOP_CONF_DIR in livy-env.sh. On Thu, Oct 26, 2017 at

[jira] [Commented] (SPARK-22357) SparkContext.binaryFiles ignore minPartitions parameter

[ https://issues.apache.org/jira/browse/SPARK-22357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220431#comment-16220431 ] Saisai Shao commented on SPARK-22357: - Yes, I know this parameter is ignored, but I'm not sure

[jira] [Commented] (SPARK-22357) SparkContext.binaryFiles ignore minPartitions parameter

[ https://issues.apache.org/jira/browse/SPARK-22357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220395#comment-16220395 ] Saisai Shao commented on SPARK-22357: - bq. In our specific case this was a problem, because we

[jira] [Commented] (SPARK-22357) SparkContext.binaryFiles ignore minPartitions parameter

[ https://issues.apache.org/jira/browse/SPARK-22357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220160#comment-16220160 ] Saisai Shao commented on SPARK-22357: - [~WeichenXu123] would you please format the code in JIRA

[jira] [Assigned] (SPARK-21840) Allow multiple SparkSubmit invocations in same JVM without polluting system properties

[ https://issues.apache.org/jira/browse/SPARK-21840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-21840: --- Assignee: Marcelo Vanzin > Allow multiple SparkSubmit invocations in same JVM with

[jira] [Resolved] (SPARK-21840) Allow multiple SparkSubmit invocations in same JVM without polluting system properties

[ https://issues.apache.org/jira/browse/SPARK-21840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-21840. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19519 [https

[jira] [Updated] (SPARK-22319) SparkSubmit calls getFileStatus before calling loginUserFromKeytab

[ https://issues.apache.org/jira/browse/SPARK-22319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-22319: Fix Version/s: 2.2.1 > SparkSubmit calls getFileStatus before calling loginUserFromKey

[jira] [Closed] (SPARK-22272) killing task may cause the executor progress hang because of the JVM bug

[ https://issues.apache.org/jira/browse/SPARK-22272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-22272. --- > killing task may cause the executor progress hang because of the JVM

[jira] [Resolved] (SPARK-22272) killing task may cause the executor progress hang because of the JVM bug

[ https://issues.apache.org/jira/browse/SPARK-22272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-22272. - Resolution: Won't Fix > killing task may cause the executor progress hang because of the JVM

[jira] [Commented] (SPARK-22272) killing task may cause the executor progress hang because of the JVM bug

[ https://issues.apache.org/jira/browse/SPARK-22272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214569#comment-16214569 ] Saisai Shao commented on SPARK-22272: - Setting "spark.file.transferTo" to "false&qu

[jira] [Assigned] (SPARK-22319) SparkSubmit calls getFileStatus before calling loginUserFromKeytab

[ https://issues.apache.org/jira/browse/SPARK-22319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-22319: --- Assignee: Steven Rand > SparkSubmit calls getFileStatus before calling loginUserFromKey

[jira] [Resolved] (SPARK-22319) SparkSubmit calls getFileStatus before calling loginUserFromKeytab

[ https://issues.apache.org/jira/browse/SPARK-22319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-22319. - Resolution: Fixed Fix Version/s: 2.3.0 > SparkSubmit calls getFileStatus before call

[jira] [Updated] (SPARK-22319) SparkSubmit calls getFileStatus before calling loginUserFromKeytab

[ https://issues.apache.org/jira/browse/SPARK-22319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-22319: Affects Version/s: 2.2.0 > SparkSubmit calls getFileStatus before calling loginUserFromKey

[jira] [Assigned] (LIVY-245) Sharing variables across Scala Jobs in RSC.

[ https://issues.apache.org/jira/browse/LIVY-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-245: Assignee: Saisai Shao (was: Alex Man) > Sharing variables across Scala Jobs in

[jira] [Resolved] (LIVY-245) Sharing variables across Scala Jobs in RSC.

[ https://issues.apache.org/jira/browse/LIVY-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-245. -- Resolution: Fixed Fix Version/s: 0.5.0 Issue resolved by pull request 19 [https://github.com

[jira] [Resolved] (LIVY-411) Session cannot be started when Python or R package is missing

[ https://issues.apache.org/jira/browse/LIVY-411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-411. -- Resolution: Fixed Fix Version/s: 0.5.0 Issue resolved by pull request 56 [https://github.com

[jira] [Commented] (SPARK-22312) Spark job stuck with no executor due to bug in Executor Allocation Manager

[ https://issues.apache.org/jira/browse/SPARK-22312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212071#comment-16212071 ] Saisai Shao commented on SPARK-22312: - I think it is duplicated to SPARK-11334 according to your

[jira] [Assigned] (LIVY-409) Improve User Experience in livy-shell

[ https://issues.apache.org/jira/browse/LIVY-409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-409: Assignee: Eric Perry > Improve User Experience in livy-sh

<    2   3   4   5   6   7   8   9   10   11   >