[jira] [Created] (SPARK-27664) Performance issue with FileStatusCache, while reading from object stores.

2019-05-09 Thread Prashant Sharma (JIRA)
Prashant Sharma created SPARK-27664: --- Summary: Performance issue with FileStatusCache, while reading from object stores. Key: SPARK-27664 URL: https://issues.apache.org/jira/browse/SPARK-27664

[jira] [Commented] (SPARK-24432) Add support for dynamic resource allocation

2019-02-20 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772875#comment-16772875 ] Prashant Sharma commented on SPARK-24432: - Any update on this work? > Add support for dynamic

[jira] [Commented] (SPARK-26059) Spark standalone mode, does not correctly record a failed Spark Job.

2019-01-16 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743783#comment-16743783 ] Prashant Sharma commented on SPARK-26059: - One work around is to capture the JVM exit code for

[jira] [Commented] (SPARK-26059) Spark standalone mode, does not correctly record a failed Spark Job.

2018-11-28 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702809#comment-16702809 ] Prashant Sharma commented on SPARK-26059: - This will be a won't fix, as to fix it, 1) One

[jira] [Comment Edited] (SPARK-26059) Spark standalone mode, does not correctly record a failed Spark Job.

2018-11-28 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702809#comment-16702809 ] Prashant Sharma edited comment on SPARK-26059 at 11/29/18 7:33 AM: ---

[jira] [Resolved] (SPARK-26059) Spark standalone mode, does not correctly record a failed Spark Job.

2018-11-28 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma resolved SPARK-26059. - Resolution: Won't Fix > Spark standalone mode, does not correctly record a failed Spark

[jira] [Updated] (SPARK-26059) Spark standalone mode, does not correctly record a failed Spark Job.

2018-11-25 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-26059: Description: In order to reproduce submit a failing job to spark standalone master. The

[jira] [Commented] (SPARK-26059) Spark standalone mode, does not correctly record a failed Spark Job.

2018-11-25 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698483#comment-16698483 ] Prashant Sharma commented on SPARK-26059: - To fix this, we need to either create an RPC Env

[jira] [Updated] (SPARK-26059) Spark standalone mode, does not correctly record a failed Spark Job.

2018-11-25 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-26059: Description: In order to reproduce submit a failing job to spark standalone master. The

[jira] [Updated] (SPARK-26059) Spark standalone mode, does not correctly record a failed Spark Job.

2018-11-23 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-26059: Description: In order to reproduce submit a failing job to spark standalone master. The

[jira] [Commented] (SPARK-26059) Spark standalone mode, does not correctly record a failed Spark Job.

2018-11-16 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16689210#comment-16689210 ] Prashant Sharma commented on SPARK-26059: - Well to reproduce it, we need to submit a failing job

[jira] [Created] (SPARK-26059) Spark standalone mode, does not correctly record a failed Spark Job.

2018-11-14 Thread Prashant Sharma (JIRA)
Prashant Sharma created SPARK-26059: --- Summary: Spark standalone mode, does not correctly record a failed Spark Job. Key: SPARK-26059 URL: https://issues.apache.org/jira/browse/SPARK-26059 Project:

[jira] [Created] (SPARK-26058) Incorrect logging class loaded for all the logs.

2018-11-14 Thread Prashant Sharma (JIRA)
Prashant Sharma created SPARK-26058: --- Summary: Incorrect logging class loaded for all the logs. Key: SPARK-26058 URL: https://issues.apache.org/jira/browse/SPARK-26058 Project: Spark Issue

[jira] [Resolved] (SPARK-25796) Enable external shuffle service for kubernetes mode.

2018-10-23 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma resolved SPARK-25796. - Resolution: Duplicate > Enable external shuffle service for kubernetes mode. >

[jira] [Created] (SPARK-25796) Enable external shuffle service for kubernetes mode.

2018-10-21 Thread Prashant Sharma (JIRA)
Prashant Sharma created SPARK-25796: --- Summary: Enable external shuffle service for kubernetes mode. Key: SPARK-25796 URL: https://issues.apache.org/jira/browse/SPARK-25796 Project: Spark

[jira] [Created] (SPARK-25744) Allow kubernetes integration tests to be run against a real cluster.

2018-10-16 Thread Prashant Sharma (JIRA)
Prashant Sharma created SPARK-25744: --- Summary: Allow kubernetes integration tests to be run against a real cluster. Key: SPARK-25744 URL: https://issues.apache.org/jira/browse/SPARK-25744 Project:

[jira] [Resolved] (SPARK-25282) Document custom builds needed for K8S development

2018-10-08 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma resolved SPARK-25282. - Resolution: Not A Problem Fix Version/s: 3.0.0 Looks like this is working

[jira] [Updated] (SPARK-25647) Add spark streaming compatibility suite for kubernetes.

2018-10-05 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-25647: Affects Version/s: (was: 2.5.0) 3.0.0 > Add spark streaming

[jira] [Created] (SPARK-25647) Add spark streaming compatibility suite for kubernetes.

2018-10-05 Thread Prashant Sharma (JIRA)
Prashant Sharma created SPARK-25647: --- Summary: Add spark streaming compatibility suite for kubernetes. Key: SPARK-25647 URL: https://issues.apache.org/jira/browse/SPARK-25647 Project: Spark

[jira] [Commented] (SPARK-25515) Add a config property for disabling auto deletion of PODS for debugging.

2018-09-27 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629857#comment-16629857 ] Prashant Sharma commented on SPARK-25515: - Also looking for an option in Kubernetes to look for

[jira] [Updated] (SPARK-25282) Fix support for spark-shell with K8s

2018-09-27 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-25282: Issue Type: Improvement (was: Bug) > Fix support for spark-shell with K8s >

[jira] [Commented] (SPARK-25282) Fix support for spark-shell with K8s

2018-09-27 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629816#comment-16629816 ] Prashant Sharma commented on SPARK-25282: - The reason, I was running into this, was trying to

[jira] [Created] (SPARK-25543) Confusing log messages at DEBUG level, in K8s mode.

2018-09-26 Thread Prashant Sharma (JIRA)
Prashant Sharma created SPARK-25543: --- Summary: Confusing log messages at DEBUG level, in K8s mode. Key: SPARK-25543 URL: https://issues.apache.org/jira/browse/SPARK-25543 Project: Spark

[jira] [Commented] (SPARK-25515) Add a config property for disabling auto deletion of PODS for debugging.

2018-09-24 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625575#comment-16625575 ] Prashant Sharma commented on SPARK-25515: - Also add a fail fast option, if the pod creation

[jira] [Updated] (SPARK-25515) Add a config property for disabling auto deletion of PODS for debugging.

2018-09-24 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-25515: Description: Currently, if a pod fails to start due to some failure, it gets removed and

[jira] [Created] (SPARK-25515) Add a config property for disabling auto deletion of PODS for debugging.

2018-09-24 Thread Prashant Sharma (JIRA)
Prashant Sharma created SPARK-25515: --- Summary: Add a config property for disabling auto deletion of PODS for debugging. Key: SPARK-25515 URL: https://issues.apache.org/jira/browse/SPARK-25515

[jira] [Updated] (SPARK-25295) Pod names conflicts in client mode, if previous submission was not a clean shutdown.

2018-08-31 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-25295: Description: If the previous job was killed somehow, by disconnecting the client. It

[jira] [Created] (SPARK-25295) Pod names conflicts in client mode, if previous submission was not a clean shutdown.

2018-08-31 Thread Prashant Sharma (JIRA)
Prashant Sharma created SPARK-25295: --- Summary: Pod names conflicts in client mode, if previous submission was not a clean shutdown. Key: SPARK-25295 URL: https://issues.apache.org/jira/browse/SPARK-25295

[jira] [Updated] (SPARK-25282) Fix support for spark-shell with K8s

2018-08-30 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-25282: Description: Spark shell when run with kubernetes master, gives following errors.

[jira] [Updated] (SPARK-25282) Fix support for spark-shell with K8s

2018-08-30 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-25282: Description: Spark shell when run with kubernetes master, gives following errors.

[jira] [Created] (SPARK-25282) Fix support for spark-shell with K8s

2018-08-30 Thread Prashant Sharma (JIRA)
Prashant Sharma created SPARK-25282: --- Summary: Fix support for spark-shell with K8s Key: SPARK-25282 URL: https://issues.apache.org/jira/browse/SPARK-25282 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-25066) Provide Spark R image for deploying Spark on kubernetes.

2018-08-17 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma resolved SPARK-25066. - Resolution: Duplicate Ahh found it, sorry for the mess. > Provide Spark R image for

[jira] [Updated] (SPARK-25065) Provide a way to add a custom logging configuration file.

2018-08-17 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-25065: Issue Type: Bug (was: Improvement) > Provide a way to add a custom logging configuration

[jira] [Updated] (SPARK-25065) Driver and executors pick the wrong logging configuration file.

2018-08-17 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-25065: Summary: Driver and executors pick the wrong logging configuration file. (was: Provide a

[jira] [Commented] (SPARK-25065) Provide a way to add a custom logging configuration file.

2018-08-17 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16583789#comment-16583789 ] Prashant Sharma commented on SPARK-25065: - Since this picks up the wrong logging properties file

[jira] [Commented] (SPARK-25066) Provide Spark R image for deploying Spark on kubernetes.

2018-08-17 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16583788#comment-16583788 ] Prashant Sharma commented on SPARK-25066: - Yes, I was planning to work on R support, is it

[jira] [Updated] (SPARK-25065) Provide a way to add a custom logging configuration file.

2018-08-09 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-25065: Description: Currently, when running in kubernetes mode, it sets necessary configuration

[jira] [Updated] (SPARK-25065) Provide a way to add a custom logging configuration file.

2018-08-09 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-25065: Description: Currently, when running in kubernetes mode, it sets necessary configuration

[jira] [Created] (SPARK-25066) Provide Spark R image for deploying Spark on kubernetes.

2018-08-09 Thread Prashant Sharma (JIRA)
Prashant Sharma created SPARK-25066: --- Summary: Provide Spark R image for deploying Spark on kubernetes. Key: SPARK-25066 URL: https://issues.apache.org/jira/browse/SPARK-25066 Project: Spark

[jira] [Commented] (SPARK-25065) Provide a way to add a custom logging configuration file.

2018-08-09 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574414#comment-16574414 ] Prashant Sharma commented on SPARK-25065: - One work around is to edit Dockerfile and copy

[jira] [Created] (SPARK-25065) Provide a way to add a custom logging configuration file.

2018-08-09 Thread Prashant Sharma (JIRA)
Prashant Sharma created SPARK-25065: --- Summary: Provide a way to add a custom logging configuration file. Key: SPARK-25065 URL: https://issues.apache.org/jira/browse/SPARK-25065 Project: Spark

[jira] [Resolved] (SPARK-24279) Incompatible byte code errors, when using test-jar of spark sql.

2018-06-06 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma resolved SPARK-24279. - Resolution: Invalid > Incompatible byte code errors, when using test-jar of spark sql.

[jira] [Commented] (SPARK-24279) Incompatible byte code errors, when using test-jar of spark sql.

2018-06-06 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504246#comment-16504246 ] Prashant Sharma commented on SPARK-24279: - Thanks a lot, that was the mistake.  > Incompatible

[jira] [Updated] (SPARK-24279) Incompatible byte code errors, when using test-jar of spark sql.

2018-05-15 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-24279: Component/s: Tests > Incompatible byte code errors, when using test-jar of spark sql. >

[jira] [Updated] (SPARK-24279) Incompatible byte code errors, when using test-jar of spark sql.

2018-05-15 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-24279: Description: Using test libraries available with spark sql streaming, produces weird

[jira] [Created] (SPARK-24279) Incompatible byte code errors, when using test-jar of spark sql.

2018-05-15 Thread Prashant Sharma (JIRA)
Prashant Sharma created SPARK-24279: --- Summary: Incompatible byte code errors, when using test-jar of spark sql. Key: SPARK-24279 URL: https://issues.apache.org/jira/browse/SPARK-24279 Project:

[jira] [Commented] (SPARK-21869) A cached Kafka producer should not be closed if any task is using it.

2017-08-30 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146889#comment-16146889 ] Prashant Sharma commented on SPARK-21869: - Yes, looking at it. > A cached Kafka producer should

[jira] [Updated] (SPARK-20919) Simplificaiton of CachedKafkaConsumer.

2017-07-28 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-20919: Description: Using an object pool instead of a cache for recycling objects in Kafka

[jira] [Updated] (SPARK-20919) Simplificaiton of CachedKafkaConsumer.

2017-07-28 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-20919: Summary: Simplificaiton of CachedKafkaConsumer. (was: Simplificaiton of

[jira] [Commented] (SPARK-21177) df.saveAsTable slows down linearly, with number of appends

2017-07-24 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098255#comment-16098255 ] Prashant Sharma commented on SPARK-21177: - Yes. In fact, hive should not be setup. Are you on

[jira] [Updated] (SPARK-21177) df.saveAsTable slows down linearly, with number of appends

2017-07-24 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-21177: Description: In short, please use the following shell transcript for the reproducer.

[jira] [Commented] (SPARK-21177) df.saveAsTable slows down linearly, with number of appends

2017-07-24 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098149#comment-16098149 ] Prashant Sharma commented on SPARK-21177: - Well, I have tried it on three different environment.

[jira] [Updated] (SPARK-21177) df.saveAsTable slows down linearly, with number of appends

2017-07-24 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-21177: Description: In short, please use the following shell transcript for the reproducer.

[jira] [Commented] (SPARK-21177) df.saveAsTable slows down linearly, with number of appends

2017-07-24 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098139#comment-16098139 ] Prashant Sharma commented on SPARK-21177: - It is pretty easy to reproduce, you need to run it

[jira] [Commented] (SPARK-21177) df.saveAsTable slows down linearly, with number of appends

2017-07-19 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093080#comment-16093080 ] Prashant Sharma commented on SPARK-21177: - I can reproduce it on another system with latest

[jira] [Commented] (SPARK-20376) Make StateStoreProvider plugable

2017-07-10 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079900#comment-16079900 ] Prashant Sharma commented on SPARK-20376: - [~tdas] This patch has been merged, but this issue is

[jira] [Commented] (SPARK-20870) Update the output of spark-sql -H

2017-07-07 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16077988#comment-16077988 ] Prashant Sharma commented on SPARK-20870: - Is it okay to use, https://github.com/scopt/scopt

[jira] [Resolved] (SPARK-17689) _temporary files breaks the Spark SQL streaming job.

2017-06-29 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma resolved SPARK-17689. - Resolution: Cannot Reproduce I am unable to reproduce this anymore, looks like this

[jira] [Closed] (SPARK-20738) Option to turn off building docs in sbt build.

2017-06-22 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma closed SPARK-20738. --- Resolution: Won't Fix > Option to turn off building docs in sbt build. >

[jira] [Updated] (SPARK-21177) df.saveAsTable slows down linearly, with number of appends

2017-06-22 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-21177: Summary: df.saveAsTable slows down linearly, with number of appends (was: df.SaveAsTable

[jira] [Updated] (SPARK-21177) df.SaveAsTable slows down linearly, with number of appends.

2017-06-22 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-21177: Summary: df.SaveAsTable slows down linearly, with number of appends. (was: Append to hive

[jira] [Created] (SPARK-21177) Append to hive slows down linearly, with number of appends.

2017-06-22 Thread Prashant Sharma (JIRA)
Prashant Sharma created SPARK-21177: --- Summary: Append to hive slows down linearly, with number of appends. Key: SPARK-21177 URL: https://issues.apache.org/jira/browse/SPARK-21177 Project: Spark

[jira] [Commented] (SPARK-20025) Driver fail over will not work, if SPARK_LOCAL* env is set.

2017-06-03 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16035875#comment-16035875 ] Prashant Sharma commented on SPARK-20025: - Done! > Driver fail over will not work, if

[jira] [Updated] (SPARK-20025) Driver fail over will not work, if SPARK_LOCAL* env is set.

2017-06-03 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-20025: Target Version/s: 2.3.0 (was: 2.2.0) > Driver fail over will not work, if SPARK_LOCAL*

[jira] [Updated] (SPARK-17689) _temporary files breaks the Spark SQL streaming job.

2017-05-30 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-17689: Affects Version/s: (was: 2.3.0) 2.2.0 > _temporary files breaks

[jira] [Updated] (SPARK-17689) _temporary files breaks the Spark SQL streaming job.

2017-05-30 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-17689: Target Version/s: 2.3.0 (was: 2.2.0) > _temporary files breaks the Spark SQL streaming

[jira] [Updated] (SPARK-17689) _temporary files breaks the Spark SQL streaming job.

2017-05-30 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-17689: Affects Version/s: 2.3.0 > _temporary files breaks the Spark SQL streaming job. >

[jira] [Created] (SPARK-20919) Simplificaiton of CachedKafkaConsumer using guava cache.

2017-05-30 Thread Prashant Sharma (JIRA)
Prashant Sharma created SPARK-20919: --- Summary: Simplificaiton of CachedKafkaConsumer using guava cache. Key: SPARK-20919 URL: https://issues.apache.org/jira/browse/SPARK-20919 Project: Spark

[jira] [Updated] (SPARK-20738) Option to turn off building docs in sbt build.

2017-05-15 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-20738: Description: The default behavior is unchanged. If the option is not set, the docs are

[jira] [Updated] (SPARK-20738) Option to turn off building docs in sbt build.

2017-05-15 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-20738: Summary: Option to turn off building docs in sbt build. (was: Hide building of docs in

[jira] [Comment Edited] (SPARK-20738) Hide building of docs in sbt behind an option.

2017-05-15 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16010112#comment-16010112 ] Prashant Sharma edited comment on SPARK-20738 at 5/15/17 7:45 AM: -- Yes,

[jira] [Commented] (SPARK-20738) Hide building of docs in sbt behind an option.

2017-05-15 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16010112#comment-16010112 ] Prashant Sharma commented on SPARK-20738: - Yes, we would like to catch these failures. I am not

[jira] [Commented] (SPARK-20738) Hide building of docs in sbt behind an option.

2017-05-15 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16010103#comment-16010103 ] Prashant Sharma commented on SPARK-20738: - The default behavior is unchanged. If the option is

[jira] [Updated] (SPARK-20738) Hide building of docs in sbt behind an option.

2017-05-15 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-20738: Description: sbt publish-local tries to build the docs along with other artifacts and

[jira] [Commented] (SPARK-20738) Hide building of docs in sbt behind an option.

2017-05-15 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16010079#comment-16010079 ] Prashant Sharma commented on SPARK-20738: - Hi Sean, Occasionally on updating to latest master, I

[jira] [Updated] (SPARK-20738) Hide building of docs in sbt behind an option.

2017-05-15 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-20738: Description: sbt publish-local tries to build the docs along with other artifacts and

[jira] [Created] (SPARK-20738) Hide building of docs in sbt behind an option.

2017-05-15 Thread Prashant Sharma (JIRA)
Prashant Sharma created SPARK-20738: --- Summary: Hide building of docs in sbt behind an option. Key: SPARK-20738 URL: https://issues.apache.org/jira/browse/SPARK-20738 Project: Spark Issue

[jira] [Updated] (SPARK-20738) Hide building of docs in sbt behind an option.

2017-05-15 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-20738: Labels: sbt (was: ) > Hide building of docs in sbt behind an option. >

[jira] [Updated] (SPARK-20737) Mechanism for cleanup hooks, for structured-streaming sinks on executor shutdown.

2017-05-15 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-20737: Issue Type: Improvement (was: Bug) > Mechanism for cleanup hooks, for

[jira] [Updated] (SPARK-20737) Mechanism for cleanup hooks, for structured-streaming sinks on executor shutdown.

2017-05-15 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-20737: Labels: Kafka (was: ) > Mechanism for cleanup hooks, for structured-streaming sinks on

[jira] [Updated] (SPARK-20737) Mechanism for cleanup hooks, for structured-streaming sinks on executor shutdown.

2017-05-15 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-20737: Description: Add a standard way of cleanup during shutdown of executors for structured

[jira] [Created] (SPARK-20737) Mechanism for cleanup hooks, for structured-streaming sinks on executor shutdown.

2017-05-15 Thread Prashant Sharma (JIRA)
Prashant Sharma created SPARK-20737: --- Summary: Mechanism for cleanup hooks, for structured-streaming sinks on executor shutdown. Key: SPARK-20737 URL: https://issues.apache.org/jira/browse/SPARK-20737

[jira] [Assigned] (SPARK-19968) Use a cached instance of KafkaProducer for writing to kafka via KafkaSink.

2017-03-20 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma reassigned SPARK-19968: --- Assignee: (was: Prashant Sharma) > Use a cached instance of KafkaProducer for

[jira] [Created] (SPARK-20027) Compilation fixed in java docs.

2017-03-19 Thread Prashant Sharma (JIRA)
Prashant Sharma created SPARK-20027: --- Summary: Compilation fixed in java docs. Key: SPARK-20027 URL: https://issues.apache.org/jira/browse/SPARK-20027 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-20025) Driver fail over will not work, if SPARK_LOCAL* env is set.

2017-03-19 Thread Prashant Sharma (JIRA)
Prashant Sharma created SPARK-20025: --- Summary: Driver fail over will not work, if SPARK_LOCAL* env is set. Key: SPARK-20025 URL: https://issues.apache.org/jira/browse/SPARK-20025 Project: Spark

[jira] [Updated] (SPARK-19968) Use a cached instance of KafkaProducer for writing to kafka via KafkaSink.

2017-03-19 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-19968: Description: KafkaProducer is thread safe and an instance can be reused for writing every

[jira] [Updated] (SPARK-19968) Use a cached instance of KafkaProducer for writing to kafka via KafkaSink.

2017-03-16 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-19968: Description: KafkaProducer is thread safe and an instance can be reused for writing every

[jira] [Updated] (SPARK-19968) Use a cached instance of KafkaProducer for writing to kafka via KafkaSink.

2017-03-15 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-19968: Description: KafkaProducer is thread safe and an instance can be reused for writing every

[jira] [Assigned] (SPARK-19968) Use a cached instance of KafkaProducer for writing to kafka via KafkaSink.

2017-03-15 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma reassigned SPARK-19968: --- Assignee: Prashant Sharma > Use a cached instance of KafkaProducer for writing to

[jira] [Created] (SPARK-19968) Use a cached instance of KafkaProducer for writing to kafka via KafkaSink.

2017-03-15 Thread Prashant Sharma (JIRA)
Prashant Sharma created SPARK-19968: --- Summary: Use a cached instance of KafkaProducer for writing to kafka via KafkaSink. Key: SPARK-19968 URL: https://issues.apache.org/jira/browse/SPARK-19968

[jira] [Assigned] (SPARK-19719) Structured Streaming write to Kafka

2017-03-13 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma reassigned SPARK-19719: --- Assignee: Tyson Condie > Structured Streaming write to Kafka >

[jira] [Commented] (SPARK-18991) Change ContextCleaner.referenceBuffer to ConcurrentHashMap to make it faster

2017-01-02 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15792707#comment-15792707 ] Prashant Sharma commented on SPARK-18991: - Thank you for the fix ! > Change

[jira] [Closed] (SPARK-18526) [Kafka] The property max.poll.records is set to a very low non overridable default.

2016-11-22 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma closed SPARK-18526. --- Resolution: Invalid > [Kafka] The property max.poll.records is set to a very low non

[jira] [Created] (SPARK-18526) [Kafka] The property max.poll.records is set to a very low non overridable default.

2016-11-21 Thread Prashant Sharma (JIRA)
Prashant Sharma created SPARK-18526: --- Summary: [Kafka] The property max.poll.records is set to a very low non overridable default. Key: SPARK-18526 URL: https://issues.apache.org/jira/browse/SPARK-18526

[jira] [Created] (SPARK-17690) Mini dfs cluster based HDFS tests for FileStreamSourceSuite.

2016-09-27 Thread Prashant Sharma (JIRA)
Prashant Sharma created SPARK-17690: --- Summary: Mini dfs cluster based HDFS tests for FileStreamSourceSuite. Key: SPARK-17690 URL: https://issues.apache.org/jira/browse/SPARK-17690 Project: Spark

[jira] [Created] (SPARK-17689) _temporary files breaks the Spark SQL streaming job.

2016-09-27 Thread Prashant Sharma (JIRA)
Prashant Sharma created SPARK-17689: --- Summary: _temporary files breaks the Spark SQL streaming job. Key: SPARK-17689 URL: https://issues.apache.org/jira/browse/SPARK-17689 Project: Spark

[jira] [Commented] (SPARK-17267) Long running structured streaming requirements

2016-08-29 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15444987#comment-15444987 ] Prashant Sharma commented on SPARK-17267: - I think, SPARK-16963 should be moved under this. >

[jira] [Commented] (SPARK-16963) Change Source API so that sources do not need to keep unbounded state

2016-08-24 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434767#comment-15434767 ] Prashant Sharma commented on SPARK-16963: - Thank you Fred, the API, looks sufficient for the

[jira] [Closed] (SPARK-16974) Handle redelivery of message in MQTT structured streaming source.

2016-08-09 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma closed SPARK-16974. --- Resolution: Invalid Created in error, was supposed to create it in Bahir project. > Handle

[jira] [Created] (SPARK-16974) Handle redelivery of message in MQTT structured streaming source.

2016-08-09 Thread Prashant Sharma (JIRA)
Prashant Sharma created SPARK-16974: --- Summary: Handle redelivery of message in MQTT structured streaming source. Key: SPARK-16974 URL: https://issues.apache.org/jira/browse/SPARK-16974 Project:

<    1   2   3   4   >