[jira] [Resolved] (SPARK-24241) Do not fail fast when dynamic resource allocation enabled with 0 executor

[ https://issues.apache.org/jira/browse/SPARK-24241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24241. - Resolution: Fixed > Do not fail fast when dynamic resource allocation enabled with 0 executor >

[jira] [Updated] (SPARK-24241) Do not fail fast when dynamic resource allocation enabled with 0 executor

[ https://issues.apache.org/jira/browse/SPARK-24241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-24241: Fix Version/s: 2.4.0 > Do not fail fast when dynamic resource allocation enabled with 0 executor >

[jira] [Assigned] (SPARK-24241) Do not fail fast when dynamic resource allocation enabled with 0 executor

[ https://issues.apache.org/jira/browse/SPARK-24241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-24241: --- Assignee: Kent Yao > Do not fail fast when dynamic resource allocation enabled with 0

[jira] [Resolved] (LIVY-431) Livy batch mode ignore proxyUser parameter

[ https://issues.apache.org/jira/browse/LIVY-431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-431. -- Resolution: Not A Problem > Livy batch mode ignore proxyUser parameter >

[jira] [Commented] (LIVY-431) Livy batch mode ignore proxyUser parameter

[ https://issues.apache.org/jira/browse/LIVY-431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475255#comment-16475255 ] Saisai Shao commented on LIVY-431: -- I think you should enable impersonation first in Livy

[jira] [Commented] (LIVY-469) "shared" session kind is undocumented

[ https://issues.apache.org/jira/browse/LIVY-469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475221#comment-16475221 ] Saisai Shao commented on LIVY-469: -- It is by intention. Starting from 0.5 session kind is actually useless,

[jira] [Assigned] (SPARK-24182) Improve error message for client mode when AM fails

[ https://issues.apache.org/jira/browse/SPARK-24182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-24182: --- Assignee: Marcelo Vanzin > Improve error message for client mode when AM fails >

[jira] [Resolved] (SPARK-24182) Improve error message for client mode when AM fails

[ https://issues.apache.org/jira/browse/SPARK-24182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24182. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21243

[jira] [Created] (SPARK-24219) Improve the docker build script to avoid copying everything in example

Saisai Shao created SPARK-24219: --- Summary: Improve the docker build script to avoid copying everything in example Key: SPARK-24219 URL: https://issues.apache.org/jira/browse/SPARK-24219 Project: Spark

[jira] [Resolved] (SPARK-24188) /api/v1/version not working

[ https://issues.apache.org/jira/browse/SPARK-24188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24188. - Resolution: Fixed Fix Version/s: 2.3.1 2.4.0 Issue resolved by pull

[jira] [Assigned] (SPARK-24188) /api/v1/version not working

[ https://issues.apache.org/jira/browse/SPARK-24188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-24188: --- Assignee: Marcelo Vanzin > /api/v1/version not working > --- > >

[jira] [Commented] (SPARK-24174) Expose Hadoop config as part of /environment API

[ https://issues.apache.org/jira/browse/SPARK-24174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463575#comment-16463575 ] Saisai Shao commented on SPARK-24174: - I believe Hadoop web UI already expose such configurations. It

[jira] [Assigned] (SPARK-24136) MemoryStreamDataReader.next should skip sleeping if record is available

[ https://issues.apache.org/jira/browse/SPARK-24136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-24136: --- Assignee: Arun Mahadevan > MemoryStreamDataReader.next should skip sleeping if record is

[jira] [Resolved] (SPARK-24136) MemoryStreamDataReader.next should skip sleeping if record is available

[ https://issues.apache.org/jira/browse/SPARK-24136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24136. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21207

[jira] [Resolved] (LIVY-466) RSCDriver throws exception during RPC shutdown

[ https://issues.apache.org/jira/browse/LIVY-466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-466. -- Resolution: Fixed Fix Version/s: 0.6.0 0.5.1 Issue resolved by pull request 90

[jira] [Resolved] (SPARK-24110) Avoid calling UGI loginUserFromKeytab in ThriftServer

[ https://issues.apache.org/jira/browse/SPARK-24110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24110. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21178

[jira] [Assigned] (SPARK-24110) Avoid calling UGI loginUserFromKeytab in ThriftServer

[ https://issues.apache.org/jira/browse/SPARK-24110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-24110: --- Assignee: Saisai Shao > Avoid calling UGI loginUserFromKeytab in ThriftServer >

[jira] [Updated] (LIVY-465) Client closed before SASL negotiation finished

[ https://issues.apache.org/jira/browse/LIVY-465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated LIVY-465: - Issue Type: Question (was: Bug) > Client closed before SASL negotiation finished >

[jira] [Updated] (LIVY-463) Port for accessing livy server

[ https://issues.apache.org/jira/browse/LIVY-463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated LIVY-463: - Fix Version/s: (was: 0.5.1) > Port for accessing livy server > --- > >

[jira] [Assigned] (SPARK-23688) Refactor tests away from rate source

[ https://issues.apache.org/jira/browse/SPARK-23688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-23688: --- Assignee: Jungtaek Lim > Refactor tests away from rate source >

[jira] [Resolved] (SPARK-23688) Refactor tests away from rate source

[ https://issues.apache.org/jira/browse/SPARK-23688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23688. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21152

[jira] [Assigned] (SPARK-23830) Spark on YARN in cluster deploy mode fail with NullPointerException when a Spark application is a Scala class not object

[ https://issues.apache.org/jira/browse/SPARK-23830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-23830: --- Assignee: Eric Maynard > Spark on YARN in cluster deploy mode fail with

[jira] [Resolved] (SPARK-23830) Spark on YARN in cluster deploy mode fail with NullPointerException when a Spark application is a Scala class not object

[ https://issues.apache.org/jira/browse/SPARK-23830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23830. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21168

[jira] [Created] (SPARK-24110) Avoid calling UGI loginUserFromKeytab in ThriftServer

Saisai Shao created SPARK-24110: --- Summary: Avoid calling UGI loginUserFromKeytab in ThriftServer Key: SPARK-24110 URL: https://issues.apache.org/jira/browse/SPARK-24110 Project: Spark Issue

[jira] [Commented] (SPARK-23830) Spark on YARN in cluster deploy mode fail with NullPointerException when a Spark application is a Scala class not object

[ https://issues.apache.org/jira/browse/SPARK-23830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455621#comment-16455621 ] Saisai Shao commented on SPARK-23830: - I agree [~emaynard]. > Spark on YARN in cluster deploy mode

[jira] [Commented] (SPARK-24062) SASL encryption cannot be worked in ThriftServer

[ https://issues.apache.org/jira/browse/SPARK-24062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453521#comment-16453521 ] Saisai Shao commented on SPARK-24062: - Issue resolved by pull request 21138

[jira] [Resolved] (SPARK-24062) SASL encryption cannot be worked in ThriftServer

[ https://issues.apache.org/jira/browse/SPARK-24062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24062. - Resolution: Fixed Assignee: Saisai Shao Fix Version/s: 2.4.0

[jira] [Resolved] (LIVY-460) when using livy with spark2.3, something wrong with the stdout and stderr

[ https://issues.apache.org/jira/browse/LIVY-460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-460. -- Resolution: Fixed Fix Version/s: 0.6.0 Issue resolved by pull request 89

[jira] [Commented] (LIVY-460) when using livy with spark2.3, something wrong with the stdout and stderr

[ https://issues.apache.org/jira/browse/LIVY-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451522#comment-16451522 ] Saisai Shao commented on LIVY-460: -- I think the word "stdout" is misleading, actually it still captured

[jira] [Created] (SPARK-24062) SASL encryption cannot be worked in ThriftServer

Saisai Shao created SPARK-24062: --- Summary: SASL encryption cannot be worked in ThriftServer Key: SPARK-24062 URL: https://issues.apache.org/jira/browse/SPARK-24062 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-22372) Make YARN client extend SparkApplication

[ https://issues.apache.org/jira/browse/SPARK-22372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-22372: --- Assignee: Marcelo Vanzin > Make YARN client extend SparkApplication >

[jira] [Commented] (SPARK-24016) Yarn does not update node blacklist in static allocation

[ https://issues.apache.org/jira/browse/SPARK-24016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16445252#comment-16445252 ] Saisai Shao commented on SPARK-24016: - I think this can be useful if we enabled

[jira] [Commented] (LIVY-459) First submission via livy-python-api fails with KryoException, subsequent retries succeed

[ https://issues.apache.org/jira/browse/LIVY-459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16445239#comment-16445239 ] Saisai Shao commented on LIVY-459: -- Thanks for the report. Are you going to fix this issue, or just report

[jira] [Commented] (LIVY-424) Session memory keeps growing as session persists

[ https://issues.apache.org/jira/browse/LIVY-424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443705#comment-16443705 ] Saisai Shao commented on LIVY-424: -- This seems like a Scala issue, and looks like no way to fix it in the

[jira] [Commented] (SPARK-24008) SQL/Hive Context fails with NullPointerException

[ https://issues.apache.org/jira/browse/SPARK-24008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443495#comment-16443495 ] Saisai Shao commented on SPARK-24008: - The case you provided seems not so valid. You're trying to

[jira] [Commented] (SPARK-24008) SQL/Hive Context fails with NullPointerException

[ https://issues.apache.org/jira/browse/SPARK-24008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443475#comment-16443475 ] Saisai Shao commented on SPARK-24008: - Why do you need to broadcast {{SQLContext}} or

[jira] [Resolved] (SPARK-24014) Add onStreamingStarted method to StreamingListener

[ https://issues.apache.org/jira/browse/SPARK-24014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24014. - Resolution: Fixed Fix Version/s: 2.3.1 2.4.0 Issue resolved by pull

[jira] [Assigned] (SPARK-24014) Add onStreamingStarted method to StreamingListener

[ https://issues.apache.org/jira/browse/SPARK-24014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-24014: --- Assignee: Liang-Chi Hsieh > Add onStreamingStarted method to StreamingListener >

[jira] [Resolved] (LIVY-458) Upgrade jackson version from 2.9.2 to 2.9.5

[ https://issues.apache.org/jira/browse/LIVY-458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-458. -- Resolution: Fixed Fix Version/s: 0.6.0 Issue resolved by pull request 87

[jira] [Commented] (SPARK-23989) When using `SortShuffleWriter`, the data will be overwritten

[ https://issues.apache.org/jira/browse/SPARK-23989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441957#comment-16441957 ] Saisai Shao commented on SPARK-23989: - I've no idea what are you trying to express. What specific

[jira] [Commented] (SPARK-23843) Deploy yarn meets incorrect LOCALIZED_CONF_DIR

[ https://issues.apache.org/jira/browse/SPARK-23843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441930#comment-16441930 ] Saisai Shao commented on SPARK-23843: - I think this issue is due to your "new Hadoop-compatible

[jira] [Commented] (SPARK-23830) Spark on YARN in cluster deploy mode fail with NullPointerException when a Spark application is a Scala class not object

[ https://issues.apache.org/jira/browse/SPARK-23830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441919#comment-16441919 ] Saisai Shao commented on SPARK-23830: - What is the reason to use {{class}} instead of {{object}},

[jira] [Commented] (SPARK-24001) Multinode cluster

[ https://issues.apache.org/jira/browse/SPARK-24001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441918#comment-16441918 ] Saisai Shao commented on SPARK-24001: - Question should go to mail list. > Multinode cluster >

[jira] [Resolved] (SPARK-24001) Multinode cluster

[ https://issues.apache.org/jira/browse/SPARK-24001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24001. - Resolution: Invalid > Multinode cluster > -- > > Key:

[jira] [Commented] (SPARK-23989) When using `SortShuffleWriter`, the data will be overwritten

[ https://issues.apache.org/jira/browse/SPARK-23989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441906#comment-16441906 ] Saisai Shao commented on SPARK-23989: - Please provide a reproducible case. Did you reuse the object

[jira] [Commented] (SPARK-23982) NoSuchMethodException: There is no startCredentialUpdater method in the object YarnSparkHadoopUtil

[ https://issues.apache.org/jira/browse/SPARK-23982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441881#comment-16441881 ] Saisai Shao commented on SPARK-23982: - This method should be existed. Would you please paste your

[jira] [Created] (LIVY-458) Upgrade jackson version from 2.9.2 to 2.9.5

Saisai Shao created LIVY-458: Summary: Upgrade jackson version from 2.9.2 to 2.9.5 Key: LIVY-458 URL: https://issues.apache.org/jira/browse/LIVY-458 Project: Livy Issue Type: Bug

[jira] [Resolved] (LIVY-457) PySpark `sqlContext.sparkSession` incorrect on Spark 2.x

[ https://issues.apache.org/jira/browse/LIVY-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-457. -- Resolution: Fixed Fix Version/s: 0.6.0 0.5.1 Issue resolved by pull request 86

[jira] [Commented] (LIVY-457) PySpark `sqlContext.sparkSession` incorrect on Spark 2.x

[ https://issues.apache.org/jira/browse/LIVY-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440460#comment-16440460 ] Saisai Shao commented on LIVY-457: -- I made a PR about this issue

[jira] [Commented] (LIVY-456) IllegalStateException when interactive session is still starting

[ https://issues.apache.org/jira/browse/LIVY-456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440444#comment-16440444 ] Saisai Shao commented on LIVY-456: -- Why don't you make a PR about the issue? > IllegalStateException when

[jira] [Commented] (LIVY-457) PySpark `sqlContext.sparkSession` incorrect on Spark 2.x

[ https://issues.apache.org/jira/browse/LIVY-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440435#comment-16440435 ] Saisai Shao commented on LIVY-457: -- Yes, we also met this issue today. The signature of creating SQLContext

[jira] [Resolved] (SPARK-23956) Use effective RPC port in AM registration

[ https://issues.apache.org/jira/browse/SPARK-23956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23956. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21047

[jira] [Assigned] (SPARK-23956) Use effective RPC port in AM registration

[ https://issues.apache.org/jira/browse/SPARK-23956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-23956: --- Assignee: Gera Shegalov > Use effective RPC port in AM registration >

[jira] [Resolved] (LIVY-455) Livy Spark SQL "not a primitive class java.math.BigDecimal" when querying decimal columns

[ https://issues.apache.org/jira/browse/LIVY-455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-455. -- Resolution: Fixed Fix Version/s: 0.6.0 0.5.1 Issue resolved by pull request 85

[jira] [Commented] (LIVY-455) Livy Spark SQL "not a primitive class java.math.BigDecimal" when querying decimal columns

[ https://issues.apache.org/jira/browse/LIVY-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429944#comment-16429944 ] Saisai Shao commented on LIVY-455: -- Thanks for the reporting, seems like there's some bug in JSON encoding,

[jira] [Created] (SPARK-23845) Continuous rate source uses different offset format

Saisai Shao created SPARK-23845: --- Summary: Continuous rate source uses different offset format Key: SPARK-23845 URL: https://issues.apache.org/jira/browse/SPARK-23845 Project: Spark Issue

[jira] [Created] (SPARK-23844) Socket Stream recovering from checkpoint will throw exception

Saisai Shao created SPARK-23844: --- Summary: Socket Stream recovering from checkpoint will throw exception Key: SPARK-23844 URL: https://issues.apache.org/jira/browse/SPARK-23844 Project: Spark

[jira] [Commented] (LIVY-454) Creating batch sessions seems to be synchronous. Is there any plan to make it asynchronous ?

[ https://issues.apache.org/jira/browse/LIVY-454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421902#comment-16421902 ] Saisai Shao commented on LIVY-454: -- No, I don't think your problem can be solved  with sync or async launch

[jira] [Resolved] (SPARK-23767) DirectStream is producing the incorrect type of message

[ https://issues.apache.org/jira/browse/SPARK-23767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23767. - Resolution: Not A Problem > DirectStream is producing the incorrect type of message >

[jira] [Commented] (SPARK-23767) DirectStream is producing the incorrect type of message

[ https://issues.apache.org/jira/browse/SPARK-23767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420291#comment-16420291 ] Saisai Shao commented on SPARK-23767: - This seems not a Spark issue, more like a microsoft event-hub

[jira] [Assigned] (SPARK-23743) IsolatedClientLoader.isSharedClass returns an unindented result against `slf4j` keyword

[ https://issues.apache.org/jira/browse/SPARK-23743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-23743: --- Assignee: Jongyoul Lee > IsolatedClientLoader.isSharedClass returns an unindented result

[jira] [Resolved] (SPARK-23743) IsolatedClientLoader.isSharedClass returns an unindented result against `slf4j` keyword

[ https://issues.apache.org/jira/browse/SPARK-23743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23743. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20860

[jira] [Commented] (SPARK-23807) Add Hadoop 3 profile with relevant POM fix ups, cloud-storage artifacts and binding

[ https://issues.apache.org/jira/browse/SPARK-23807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418298#comment-16418298 ] Saisai Shao commented on SPARK-23807: - Hi [~ste...@apache.org], this is dup of SPARK-23534, I think

[jira] [Commented] (SPARK-22876) spark.yarn.am.attemptFailuresValidityInterval does not work correctly

[ https://issues.apache.org/jira/browse/SPARK-22876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16413412#comment-16413412 ] Saisai Shao commented on SPARK-22876: - Yes, that's the key problem. Current YARN doesn't support

[jira] [Resolved] (SPARK-23787) SparkSubmitSuite::"download remote resource if it is not supported by yarn" fails on Hadoop 2.9

[ https://issues.apache.org/jira/browse/SPARK-23787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23787. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20895

[jira] [Assigned] (SPARK-23787) SparkSubmitSuite::"download remote resource if it is not supported by yarn" fails on Hadoop 2.9

[ https://issues.apache.org/jira/browse/SPARK-23787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-23787: --- Assignee: Marcelo Vanzin > SparkSubmitSuite::"download remote resource if it is not

[jira] [Resolved] (SPARK-23361) Driver restart fails if it happens after 7 days from app submission

[ https://issues.apache.org/jira/browse/SPARK-23361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23361. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20657

[jira] [Assigned] (SPARK-23361) Driver restart fails if it happens after 7 days from app submission

[ https://issues.apache.org/jira/browse/SPARK-23361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-23361: --- Assignee: Marcelo Vanzin > Driver restart fails if it happens after 7 days from app

[jira] [Updated] (SPARK-23644) SHS with proxy doesn't show applications

[ https://issues.apache.org/jira/browse/SPARK-23644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23644: Fix Version/s: 2.3.1 > SHS with proxy doesn't show applications >

[jira] [Issue Comment Deleted] (SPARK-23534) Spark run on Hadoop 3.0.0

[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23534: Comment: was deleted (was: I'm curious why there's a library which depends on a such new version

[jira] [Commented] (SPARK-23534) Spark run on Hadoop 3.0.0

[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405659#comment-16405659 ] Saisai Shao commented on SPARK-23534: - I'm curious why there's a library which depends on a such new

[jira] [Commented] (SPARK-23534) Spark run on Hadoop 3.0.0

[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405658#comment-16405658 ] Saisai Shao commented on SPARK-23534: - I'm curious why there's a library which depends on a such new

[jira] [Commented] (SPARK-23534) Spark run on Hadoop 3.0.0

[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405649#comment-16405649 ] Saisai Shao commented on SPARK-23534: - I'm trying to move forward this in the community. But due to

[jira] [Resolved] (SPARK-23708) Comment of ShutdownHookManager.addShutdownHook is error

[ https://issues.apache.org/jira/browse/SPARK-23708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23708. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20845

[jira] [Assigned] (SPARK-23708) Comment of ShutdownHookManager.addShutdownHook is error

[ https://issues.apache.org/jira/browse/SPARK-23708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-23708: --- Assignee: zhoukang > Comment of ShutdownHookManager.addShutdownHook is error >

[jira] [Resolved] (SPARK-23635) Spark executor env variable is overwritten by same name AM env variable

[ https://issues.apache.org/jira/browse/SPARK-23635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23635. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20799

[jira] [Assigned] (SPARK-23635) Spark executor env variable is overwritten by same name AM env variable

[ https://issues.apache.org/jira/browse/SPARK-23635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-23635: --- Assignee: Saisai Shao > Spark executor env variable is overwritten by same name AM env

[jira] [Commented] (SPARK-23644) SHS with proxy doesn't show applications

[ https://issues.apache.org/jira/browse/SPARK-23644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401538#comment-16401538 ] Saisai Shao commented on SPARK-23644: - Fixed in https://github.com/apache/spark/pull/20794. > SHS

[jira] [Resolved] (SPARK-23644) SHS with proxy doesn't show applications

[ https://issues.apache.org/jira/browse/SPARK-23644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23644. - Resolution: Fixed Assignee: Marco Gaido Fix Version/s: 2.4.0 > SHS with proxy

[jira] [Resolved] (LIVY-446) Support Spark 2.3 for Livy

[ https://issues.apache.org/jira/browse/LIVY-446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-446. -- Resolution: Fixed Fix Version/s: 0.6.0 > Support Spark 2.3 for Livy > -- >

[jira] [Created] (SPARK-23635) Spark executor env variable is overwritten by same name AM env variable

Saisai Shao created SPARK-23635: --- Summary: Spark executor env variable is overwritten by same name AM env variable Key: SPARK-23635 URL: https://issues.apache.org/jira/browse/SPARK-23635 Project: Spark

[jira] [Updated] (SPARK-23635) Spark executor env variable is overwritten by same name AM env variable

[ https://issues.apache.org/jira/browse/SPARK-23635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23635: Description: In the current Spark on YARN code, AM always will copy and overwrite its env

[jira] [Resolved] (LIVY-448) Assembly format is duplicated in pom.xml and assembly.xml

[ https://issues.apache.org/jira/browse/LIVY-448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-448. -- Resolution: Fixed > Assembly format is duplicated in pom.xml and assembly.xml >

[jira] [Created] (LIVY-449) Livy to Support Hadoop 3.0.0

Saisai Shao created LIVY-449: Summary: Livy to Support Hadoop 3.0.0 Key: LIVY-449 URL: https://issues.apache.org/jira/browse/LIVY-449 Project: Livy Issue Type: Improvement Affects Versions:

[jira] [Comment Edited] (SPARK-23534) Spark run on Hadoop 3.0.0

[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388761#comment-16388761 ] Saisai Shao edited comment on SPARK-23534 at 3/7/18 12:37 AM: -- I don't think

[jira] [Commented] (SPARK-23534) Spark run on Hadoop 3.0.0

[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388761#comment-16388761 ] Saisai Shao commented on SPARK-23534: - I don't think so. Spark uses its own fork hive version

[jira] [Commented] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

[ https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16385539#comment-16385539 ] Saisai Shao commented on HIVE-16391: Hi all, Do we have any progress on it? Spark currently uses

[jira] [Commented] (SPARK-18673) Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version

[ https://issues.apache.org/jira/browse/SPARK-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16385474#comment-16385474 ] Saisai Shao commented on SPARK-18673: - Thanks for the info [~aihuaxu]. > Dataframes doesn't work on

[jira] [Commented] (SPARK-23534) Spark run on Hadoop 3.0.0

[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382932#comment-16382932 ] Saisai Shao commented on SPARK-23534: - One issue is Hive 1.2.1.spark2 rejects Hadoop 3 (SPARK-18673)

[jira] [Commented] (SPARK-18673) Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version

[ https://issues.apache.org/jira/browse/SPARK-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382929#comment-16382929 ] Saisai Shao commented on SPARK-18673: - Spark itself uses it own hive (1.2.1.spark2), I think we need

[jira] [Updated] (SPARK-23534) Spark run on Hadoop 3.0.0

[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23534: Description: Major Hadoop vendors already/will step in Hadoop 3.0. So we should also make sure

[jira] [Created] (SPARK-23534) Spark run on Hadoop 3.0.0

Saisai Shao created SPARK-23534: --- Summary: Spark run on Hadoop 3.0.0 Key: SPARK-23534 URL: https://issues.apache.org/jira/browse/SPARK-23534 Project: Spark Issue Type: Improvement

[jira] [Resolved] (LIVY-445) Upgrade Netty version to avoid security issue

[ https://issues.apache.org/jira/browse/LIVY-445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-445. -- Resolution: Duplicate > Upgrade Netty version to avoid security issue >

[jira] [Closed] (LIVY-445) Upgrade Netty version to avoid security issue

[ https://issues.apache.org/jira/browse/LIVY-445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed LIVY-445. > Upgrade Netty version to avoid security issue > - > >

[jira] [Created] (LIVY-445) Upgrade Netty version to avoid security issue

Saisai Shao created LIVY-445: Summary: Upgrade Netty version to avoid security issue Key: LIVY-445 URL: https://issues.apache.org/jira/browse/LIVY-445 Project: Livy Issue Type: Improvement

[jira] [Commented] (SPARK-12140) Support Streaming UI in HistoryServer

[ https://issues.apache.org/jira/browse/SPARK-12140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363300#comment-16363300 ] Saisai Shao commented on SPARK-12140: - [~gschiavon] the community has concern about supporting

[jira] [Commented] (SPARK-23096) Migrate rate source to v2

[ https://issues.apache.org/jira/browse/SPARK-23096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358064#comment-16358064 ] Saisai Shao commented on SPARK-23096: - [~joseph.torres] [~tdas] can I take a crack on this if you're

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357824#comment-16357824 ] Saisai Shao commented on SPARK-23206: - [~cltlfcjin] from Ebay also plans to do similar things, I

[jira] [Commented] (SPARK-23235) Add executor Threaddump to api

[ https://issues.apache.org/jira/browse/SPARK-23235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349604#comment-16349604 ] Saisai Shao commented on SPARK-23235: - There's a new similar Jira (SPARK-23206) about adding and

[jira] [Updated] (SPARK-23200) Reset configuration when restarting from checkpoints

[ https://issues.apache.org/jira/browse/SPARK-23200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23200: Fix Version/s: (was: 2.4.0) > Reset configuration when restarting from checkpoints >

<    1   2   3   4   5   6   7   8   9   10   >