[jira] [Commented] (SPARK-18165) Kinesis support in Structured Streaming
[ https://issues.apache.org/jira/browse/SPARK-18165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339812#comment-16339812 ] Swaranga Sarma commented on SPARK-18165: Any updates? > Kinesis support in Structured Streaming > --- > > Key: SPARK-18165 > URL: https://issues.apache.org/jira/browse/SPARK-18165 > Project: Spark > Issue Type: New Feature > Components: DStreams >Reporter: Lauren Moos >Priority: Major > > Implement Kinesis based sources and sinks for Structured Streaming -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors
[
https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902634#comment-15902634
]
Swaranga Sarma commented on SPARK-12009:
The JIRA says that the issue is fixed but I still see this error in Spark 2.1.0
{code}
try (JavaSparkContext sc = new JavaSparkContext(new SparkConf()) {
//run the job
}
{code}
> Avoid re-allocate yarn container while driver want to stop all Executors
>
>
> Key: SPARK-12009
> URL: https://issues.apache.org/jira/browse/SPARK-12009
> Project: Spark
> Issue Type: Bug
> Components: YARN
>Affects Versions: 1.5.2
>Reporter: SuYan
>Assignee: SuYan
>Priority: Minor
> Fix For: 2.0.0
>
>
> Log based 1.4.0
> 2015-11-26,03:05:16,176 WARN
> org.spark-project.jetty.util.thread.QueuedThreadPool: 8 threads could not be
> stopped
> 2015-11-26,03:05:16,177 INFO org.apache.spark.ui.SparkUI: Stopped Spark web
> UI at http://
> 2015-11-26,03:05:16,401 INFO org.apache.spark.scheduler.DAGScheduler:
> Stopping DAGScheduler
> 2015-11-26,03:05:16,450 INFO
> org.apache.spark.scheduler.cluster.YarnClusterSchedulerBackend: Shutting down
> all executors
> 2015-11-26,03:05:16,525 INFO
> org.apache.spark.scheduler.cluster.YarnClusterSchedulerBackend: Asking each
> executor to shut down
> 2015-11-26,03:05:16,791 INFO
> org.apache.spark.deploy.yarn.ApplicationMaster$AMEndpoint: Driver terminated
> or disconnected! Shutting down. XX.XX.XX.XX:38734
> 2015-11-26,03:05:16,847 ERROR org.apache.spark.scheduler.LiveListenerBus:
> SparkListenerBus has already stopped! Dropping event
> SparkListenerExecutorMetricsUpdate(164,WrappedArray())
> 2015-11-26,03:05:27,242 INFO org.apache.spark.deploy.yarn.YarnAllocator: Will
> request 13 executor containers, each with 1 cores and 4608 MB memory
> including 1024 MB overhead
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors
[
https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902634#comment-15902634
]
Swaranga Sarma edited comment on SPARK-12009 at 3/9/17 7:59 AM:
The JIRA says that the issue is fixed but I still see this error in Spark 2.1.0
{code}
try (JavaSparkContext sc = new JavaSparkContext(new SparkConf())) {
//run the job
}
{code}
was (Author: swaranga):
The JIRA says that the issue is fixed but I still see this error in Spark 2.1.0
{code}
try (JavaSparkContext sc = new JavaSparkContext(new SparkConf()) {
//run the job
}
{code}
> Avoid re-allocate yarn container while driver want to stop all Executors
>
>
> Key: SPARK-12009
> URL: https://issues.apache.org/jira/browse/SPARK-12009
> Project: Spark
> Issue Type: Bug
> Components: YARN
>Affects Versions: 1.5.2
>Reporter: SuYan
>Assignee: SuYan
>Priority: Minor
> Fix For: 2.0.0
>
>
> Log based 1.4.0
> 2015-11-26,03:05:16,176 WARN
> org.spark-project.jetty.util.thread.QueuedThreadPool: 8 threads could not be
> stopped
> 2015-11-26,03:05:16,177 INFO org.apache.spark.ui.SparkUI: Stopped Spark web
> UI at http://
> 2015-11-26,03:05:16,401 INFO org.apache.spark.scheduler.DAGScheduler:
> Stopping DAGScheduler
> 2015-11-26,03:05:16,450 INFO
> org.apache.spark.scheduler.cluster.YarnClusterSchedulerBackend: Shutting down
> all executors
> 2015-11-26,03:05:16,525 INFO
> org.apache.spark.scheduler.cluster.YarnClusterSchedulerBackend: Asking each
> executor to shut down
> 2015-11-26,03:05:16,791 INFO
> org.apache.spark.deploy.yarn.ApplicationMaster$AMEndpoint: Driver terminated
> or disconnected! Shutting down. XX.XX.XX.XX:38734
> 2015-11-26,03:05:16,847 ERROR org.apache.spark.scheduler.LiveListenerBus:
> SparkListenerBus has already stopped! Dropping event
> SparkListenerExecutorMetricsUpdate(164,WrappedArray())
> 2015-11-26,03:05:27,242 INFO org.apache.spark.deploy.yarn.YarnAllocator: Will
> request 13 executor containers, each with 1 cores and 4608 MB memory
> including 1024 MB overhead
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11620) parquet.hadoop.ParquetOutputCommitter.commitJob() throws parquet.io.ParquetEncodingException
[ https://issues.apache.org/jira/browse/SPARK-11620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839306#comment-15839306 ] Swaranga Sarma commented on SPARK-11620: I encountered this issue in Spark 2.0.2 > parquet.hadoop.ParquetOutputCommitter.commitJob() throws > parquet.io.ParquetEncodingException > > > Key: SPARK-11620 > URL: https://issues.apache.org/jira/browse/SPARK-11620 > Project: Spark > Issue Type: Bug > Components: SQL >Reporter: swetha k > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-7717) Spark Standalone Web UI showing incorrect total memory, workers and cores
[ https://issues.apache.org/jira/browse/SPARK-7717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swaranga Sarma updated SPARK-7717: -- Description: I launched a Spark master in standalone mode in one of my host and then launched 3 workers on three different hosts. The workers successfully connected to my master and the Web UI showed the correct details. Specifically, the Web UI correctly shows that the total memory and the total cores available for the cluster. However on one of the worker, I did a kill -9 worker process id and restarted the worker again. This time though, the master's Web UI shows incorrect total memory and number of cores. The total memory is shown to be 4*n, where n is the memory in each worker. Also the total workers is shown as 4 and the total number of cores shown is incorrect, it shows 4*c, where c is the number of cores on each worker. was: I launched a Spark master in standalone mode in one of my host and then launched 3 workers on three different hosts. The workers successfully connected to my master and the Web UI showed the correct details. Specifically, the Web UI correctly shows that the total memory and the total cores available for the cluster. However on one of the worker, I did a kill -9 worker process id and restarted the worker again. This time though, the master's Web UI shows incorrect total memory and number of cores. The total memory is shown to be 4*n, where n is the memory in each worker. Also the total cores shown is incorrect, it shows 4*c, where c is the number of cores on each worker. Spark Standalone Web UI showing incorrect total memory, workers and cores - Key: SPARK-7717 URL: https://issues.apache.org/jira/browse/SPARK-7717 Project: Spark Issue Type: Bug Components: Web UI Affects Versions: 1.3.1 Environment: RedHat Reporter: Swaranga Sarma Priority: Minor Labels: web-ui Attachments: JIRA.PNG I launched a Spark master in standalone mode in one of my host and then launched 3 workers on three different hosts. The workers successfully connected to my master and the Web UI showed the correct details. Specifically, the Web UI correctly shows that the total memory and the total cores available for the cluster. However on one of the worker, I did a kill -9 worker process id and restarted the worker again. This time though, the master's Web UI shows incorrect total memory and number of cores. The total memory is shown to be 4*n, where n is the memory in each worker. Also the total workers is shown as 4 and the total number of cores shown is incorrect, it shows 4*c, where c is the number of cores on each worker. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-7717) Spark Standalone Web UI showing incorrect total memory and cores
[ https://issues.apache.org/jira/browse/SPARK-7717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swaranga Sarma updated SPARK-7717: -- Summary: Spark Standalone Web UI showing incorrect total memory and cores (was: Spark Standalone Web UI shown incorrect total memory and cores) Spark Standalone Web UI showing incorrect total memory and cores Key: SPARK-7717 URL: https://issues.apache.org/jira/browse/SPARK-7717 Project: Spark Issue Type: Bug Components: Web UI Affects Versions: 1.3.1 Environment: RedHat Reporter: Swaranga Sarma Priority: Minor Labels: web-ui Attachments: JIRA.PNG I launched a Spark master in standalone mode in one of my host and then launched 3 workers on three different hosts. The workers successfully connected to my master and the Web UI showed the correct details. Specifically, the Web UI correctly shows that the total memory and the total cores available for the cluster. However on one of the worker, I did a kill -9 worker process id and restarted the worker again. This time though, the master's Web UI shows incorrect total memory and number of cores. The total memory is shown to be 4*n, where n is the memory in each worker. Also the total cores shown is incorrect, it shows 4*c, where c is the number of cores on each worker. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-7717) Spark Standalone Web UI showing incorrect total memory, workers and cores
[ https://issues.apache.org/jira/browse/SPARK-7717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swaranga Sarma updated SPARK-7717: -- Summary: Spark Standalone Web UI showing incorrect total memory, workers and cores (was: Spark Standalone Web UI showing incorrect total memory and cores) Spark Standalone Web UI showing incorrect total memory, workers and cores - Key: SPARK-7717 URL: https://issues.apache.org/jira/browse/SPARK-7717 Project: Spark Issue Type: Bug Components: Web UI Affects Versions: 1.3.1 Environment: RedHat Reporter: Swaranga Sarma Priority: Minor Labels: web-ui Attachments: JIRA.PNG I launched a Spark master in standalone mode in one of my host and then launched 3 workers on three different hosts. The workers successfully connected to my master and the Web UI showed the correct details. Specifically, the Web UI correctly shows that the total memory and the total cores available for the cluster. However on one of the worker, I did a kill -9 worker process id and restarted the worker again. This time though, the master's Web UI shows incorrect total memory and number of cores. The total memory is shown to be 4*n, where n is the memory in each worker. Also the total cores shown is incorrect, it shows 4*c, where c is the number of cores on each worker. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-7717) Spark Standalone Web UI shown incorrect total memory and cores
[ https://issues.apache.org/jira/browse/SPARK-7717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swaranga Sarma updated SPARK-7717: -- Summary: Spark Standalone Web UI shown incorrect total memory and cores (was: Spark Standalone Web UI shown incorrect total memory) Spark Standalone Web UI shown incorrect total memory and cores -- Key: SPARK-7717 URL: https://issues.apache.org/jira/browse/SPARK-7717 Project: Spark Issue Type: Bug Components: Web UI Affects Versions: 1.3.1 Environment: RedHat Reporter: Swaranga Sarma Priority: Minor Labels: web-ui I launched a Spark master in standalone mode in one of my host and then launched 3 workers on three different hosts. The workers successfully connected to my master and the Web UI showed the correct details. Specifically, the Web UI correctly shows that the total memory and the total cores available for the cluster. However on one of the worker, I did a kill -9 worker process id and restarted the worker again. This time though, the master's Web UI shows incorrect total memory and number of cores. The total memory is shown to be 4*n, where n is the memory in each worker. Also the total cores shown is incorrect, it shows 4*c, where c is the number of cores on each worker. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-7717) Spark Standalone Web UI shown incorrect total memory
Swaranga Sarma created SPARK-7717: - Summary: Spark Standalone Web UI shown incorrect total memory Key: SPARK-7717 URL: https://issues.apache.org/jira/browse/SPARK-7717 Project: Spark Issue Type: Bug Components: Web UI Affects Versions: 1.3.1 Environment: RedHat Reporter: Swaranga Sarma Priority: Minor I launched a Spark master in standalone mode in one of my host and then launched 3 workers on three different hosts. The workers successfully connected to my master and the Web UI showed the correct details. Specifically, the Web UI correctly shows that the total memory and the total cores available for the cluster. However on one of the worker, I did a kill -9 worker process id and restarted the worker again. This time though, the master's Web UI shows incorrect total memory and number of cores. The total memory is shown to be 4*n, where n is the memory in each worker. Also the total cores shown is incorrect, it shows 4*c, where c is the number of cores on each worker. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-7717) Spark Standalone Web UI shown incorrect total memory and cores
[ https://issues.apache.org/jira/browse/SPARK-7717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swaranga Sarma updated SPARK-7717: -- Attachment: JIRA.PNG Attached screenshot highlighting issue Spark Standalone Web UI shown incorrect total memory and cores -- Key: SPARK-7717 URL: https://issues.apache.org/jira/browse/SPARK-7717 Project: Spark Issue Type: Bug Components: Web UI Affects Versions: 1.3.1 Environment: RedHat Reporter: Swaranga Sarma Priority: Minor Labels: web-ui Attachments: JIRA.PNG I launched a Spark master in standalone mode in one of my host and then launched 3 workers on three different hosts. The workers successfully connected to my master and the Web UI showed the correct details. Specifically, the Web UI correctly shows that the total memory and the total cores available for the cluster. However on one of the worker, I did a kill -9 worker process id and restarted the worker again. This time though, the master's Web UI shows incorrect total memory and number of cores. The total memory is shown to be 4*n, where n is the memory in each worker. Also the total cores shown is incorrect, it shows 4*c, where c is the number of cores on each worker. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2243) Support multiple SparkContexts in the same JVM
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14502326#comment-14502326
]
Swaranga Sarma commented on SPARK-2243:
---
No, they are different physical/virtual machines and they have separate JVMs.
What I meant was that if there were no restrictions on creating spark contexts,
then my web application can simply send the client request to any of the hosts
in my driver application fleet. If it ends up on a host that already has a
streaming context running, it doesn't matter. Driver applications are supposed
to be light weight anyway since most of the heavy lifting is done by the
cluster and to me there doesn't seem to be a good reason (fundamental to Spark)
why we cannot have multiple Spark contexts in the same JVM.
Support multiple SparkContexts in the same JVM
--
Key: SPARK-2243
URL: https://issues.apache.org/jira/browse/SPARK-2243
Project: Spark
Issue Type: New Feature
Components: Block Manager, Spark Core
Affects Versions: 0.7.0, 1.0.0, 1.1.0
Reporter: Miguel Angel Fernandez Diaz
We're developing a platform where we create several Spark contexts for
carrying out different calculations. Is there any restriction when using
several Spark contexts? We have two contexts, one for Spark calculations and
another one for Spark Streaming jobs. The next error arises when we first
execute a Spark calculation and, once the execution is finished, a Spark
Streaming job is launched:
{code}
14/06/23 16:40:08 ERROR executor.Executor: Exception in task ID 0
java.io.FileNotFoundException: http://172.19.0.215:47530/broadcast_0
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1624)
at
org.apache.spark.broadcast.HttpBroadcast$.read(HttpBroadcast.scala:156)
at
org.apache.spark.broadcast.HttpBroadcast.readObject(HttpBroadcast.scala:56)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:40)
at
org.apache.spark.scheduler.ResultTask$.deserializeInfo(ResultTask.scala:63)
at
org.apache.spark.scheduler.ResultTask.readExternal(ResultTask.scala:139)
at
java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1837)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:40)
at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:62)
at
org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:193)
at
org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:45)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:176)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
14/06/23 16:40:08 WARN scheduler.TaskSetManager: Lost TID 0 (task 0.0:0)
14/06/23 16:40:08 WARN scheduler.TaskSetManager: Loss was due to
java.io.FileNotFoundException
java.io.FileNotFoundException: http://172.19.0.215:47530/broadcast_0
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1624)
at
org.apache.spark.broadcast.HttpBroadcast$.read(HttpBroadcast.scala:156)
at
org.apache.spark.broadcast.HttpBroadcast.readObject(HttpBroadcast.scala:56)
at
