[jira] [Commented] (SPARK-2593) Add ability to pass an existing Akka ActorSystem into Spark
[ https://issues.apache.org/jira/browse/SPARK-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14169208#comment-14169208 ] Helena Edelson commented on SPARK-2593: --- [~matei] [~pwendell] Yes I see the pain point here now. I just created a ticket to upgrade Akka and thus Typesafe Config versions because I am now locked into 2.2.3 and have binary incompatibility with using latest Akka 2.3.6 / config 1.2.1. Makes me very sad. I think I would throw in the towel on this one if you can make it completely separate so that a user with it's own AkkaSystem and Config versions are not affected? Tricky because when deploying, spark needs its version (provided?) and the user app needs the other. Add ability to pass an existing Akka ActorSystem into Spark --- Key: SPARK-2593 URL: https://issues.apache.org/jira/browse/SPARK-2593 Project: Spark Issue Type: Improvement Components: Spark Core Reporter: Helena Edelson As a developer I want to pass an existing ActorSystem into StreamingContext in load-time so that I do not have 2 actor systems running on a node in an Akka application. This would mean having spark's actor system on its own named-dispatchers as well as exposing the new private creation of its own actor system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2593) Add ability to pass an existing Akka ActorSystem into Spark
[ https://issues.apache.org/jira/browse/SPARK-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14169545#comment-14169545 ] Evan Chan commented on SPARK-2593: -- Hmmm :( I believe Spark already uses a shaded version of Akka with a different namespace. Unfortunately it still creates some dependency conflicts down the chain, but I don't remember the details. On Mon, Oct 13, 2014 at 4:58 AM, Helena Edelson (JIRA) j...@apache.org -- The fruit of silence is prayer; the fruit of prayer is faith; the fruit of faith is love; the fruit of love is service; the fruit of service is peace. -- Mother Teresa Add ability to pass an existing Akka ActorSystem into Spark --- Key: SPARK-2593 URL: https://issues.apache.org/jira/browse/SPARK-2593 Project: Spark Issue Type: Improvement Components: Spark Core Reporter: Helena Edelson As a developer I want to pass an existing ActorSystem into StreamingContext in load-time so that I do not have 2 actor systems running on a node in an Akka application. This would mean having spark's actor system on its own named-dispatchers as well as exposing the new private creation of its own actor system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2593) Add ability to pass an existing Akka ActorSystem into Spark
[ https://issues.apache.org/jira/browse/SPARK-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14168727#comment-14168727 ] Patrick Wendell commented on SPARK-2593: Yeah for Spark Streaming the API visibility is not an issue because we are explicitly exposing Akka as an API (and it's an add-on connector). I still don't quite understand if the model of akka is that different applications are supposed to share actor systems. We rely heavily on customization of the akka configuration and if a user passes in an ActorSystem it will break assumptions inside of Spark - so I think allowing users to pass in an ActorSystem is going to be difficult. Exposing our ActorSystem in Spark Streaming seems more reasonable though, since the configurations are immutable at that point. Other things like providing better naming, etc, that stuff makes a lot of sense. Add ability to pass an existing Akka ActorSystem into Spark --- Key: SPARK-2593 URL: https://issues.apache.org/jira/browse/SPARK-2593 Project: Spark Issue Type: Improvement Components: Spark Core Reporter: Helena Edelson As a developer I want to pass an existing ActorSystem into StreamingContext in load-time so that I do not have 2 actor systems running on a node in an Akka application. This would mean having spark's actor system on its own named-dispatchers as well as exposing the new private creation of its own actor system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2593) Add ability to pass an existing Akka ActorSystem into Spark
[ https://issues.apache.org/jira/browse/SPARK-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14138906#comment-14138906 ] Helena Edelson commented on SPARK-2593: --- [~matei] +1 for spark streaming, that is a primary concern here. And I understand your concern over support for akka upgrades. However I am more than happy to help WRT that and am sure I can find a few others that feel the same so that time isn't taken away from your team's new features/enhancements bandwidth. I will get more data on have 2 actor systems on a node. Add ability to pass an existing Akka ActorSystem into Spark --- Key: SPARK-2593 URL: https://issues.apache.org/jira/browse/SPARK-2593 Project: Spark Issue Type: Improvement Components: Spark Core Reporter: Helena Edelson As a developer I want to pass an existing ActorSystem into StreamingContext in load-time so that I do not have 2 actor systems running on a node in an Akka application. This would mean having spark's actor system on its own named-dispatchers as well as exposing the new private creation of its own actor system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2593) Add ability to pass an existing Akka ActorSystem into Spark
[ https://issues.apache.org/jira/browse/SPARK-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14139163#comment-14139163 ] Matei Zaharia commented on SPARK-2593: -- Sure, it would be great to do this for streaming. For the other stuff by the way, my concern isn't so much upgrades, it's locking users in to a specific Spark version. For example, suppose we add an API involving classes from Akka 2.2 today, and later 2.3 or 2.4 changes those classes in a way that isn't compatible. Then to meet our API stability promises to our users, we're going to keep our version of Akka at 2.2, and users simply won't be able to use a new Akka in the same application as Spark. With Akka in particular you sometimes have to upgrade in order to use a new Scala version too, which is another reason we can't lock it into our API. Basically in these cases it's just very risky to expose fast-moving third-party classes if you want to have a stable API. We've been bitten by exposing stuff as mundane as Guava or Google Protobufs (!) because of incompatible changes in minor versions. We care a lot about API stability within Spark and in particular about shielding our users from the fast-moving APIs in distributed systems land. Add ability to pass an existing Akka ActorSystem into Spark --- Key: SPARK-2593 URL: https://issues.apache.org/jira/browse/SPARK-2593 Project: Spark Issue Type: Improvement Components: Spark Core Reporter: Helena Edelson As a developer I want to pass an existing ActorSystem into StreamingContext in load-time so that I do not have 2 actor systems running on a node in an Akka application. This would mean having spark's actor system on its own named-dispatchers as well as exposing the new private creation of its own actor system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2593) Add ability to pass an existing Akka ActorSystem into Spark
[ https://issues.apache.org/jira/browse/SPARK-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14136920#comment-14136920 ] Evan Chan commented on SPARK-2593: -- [~pwendell] I'd have to agree with Helena and Tupshin; I'd like to see the usage of Akka increase; it is underutilized. Also note that it's not true that Akka is only used for RPC; there are Actors used in several places. Performance and resilience would likely improve significantly in several areas with more usage of Akka; the current threads spun up by each driver for various things like a file server are quite wasteful. On Tue, Sep 16, 2014 at 5:17 PM, Tupshin Harper (JIRA) j...@apache.org -- The fruit of silence is prayer; the fruit of prayer is faith; the fruit of faith is love; the fruit of love is service; the fruit of service is peace. -- Mother Teresa Add ability to pass an existing Akka ActorSystem into Spark --- Key: SPARK-2593 URL: https://issues.apache.org/jira/browse/SPARK-2593 Project: Spark Issue Type: Improvement Components: Spark Core Reporter: Helena Edelson As a developer I want to pass an existing ActorSystem into StreamingContext in load-time so that I do not have 2 actor systems running on a node in an Akka application. This would mean having spark's actor system on its own named-dispatchers as well as exposing the new private creation of its own actor system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2593) Add ability to pass an existing Akka ActorSystem into Spark
[ https://issues.apache.org/jira/browse/SPARK-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14137338#comment-14137338 ] Helena Edelson commented on SPARK-2593: --- [~pwendell] I forgot to not this on my reply above but I was offering to do the work myself - or at least start it. What would be ideal is to have both of these: - Expose the spark actor system which also requires insuring all spark actors are on specified dispatchers (very important) - Optionally allow spark users to pass in their existing actor system I feel both are incredibly important for users. Add ability to pass an existing Akka ActorSystem into Spark --- Key: SPARK-2593 URL: https://issues.apache.org/jira/browse/SPARK-2593 Project: Spark Issue Type: Improvement Components: Spark Core Reporter: Helena Edelson As a developer I want to pass an existing ActorSystem into StreamingContext in load-time so that I do not have 2 actor systems running on a node in an Akka application. This would mean having spark's actor system on its own named-dispatchers as well as exposing the new private creation of its own actor system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2593) Add ability to pass an existing Akka ActorSystem into Spark
[ https://issues.apache.org/jira/browse/SPARK-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14137529#comment-14137529 ] kannapiran commented on SPARK-2593: --- Is there a way to add akka system in to spark streaming context and write the output to hdfs from spark streaming. I tried ActorReceiver rec = new ActorReceiver(props, TestActor, StorageLevel.MEMORY_AND_DISK_2(), SupervisorStrategy.defaultStrategy(), null); ReceiverInputDStreamObject receiverStream = context.receiverStream(rec, null); receiverStream.print(); But the above code doesnt stream anything from actor system Add ability to pass an existing Akka ActorSystem into Spark --- Key: SPARK-2593 URL: https://issues.apache.org/jira/browse/SPARK-2593 Project: Spark Issue Type: Improvement Components: Spark Core Reporter: Helena Edelson As a developer I want to pass an existing ActorSystem into StreamingContext in load-time so that I do not have 2 actor systems running on a node in an Akka application. This would mean having spark's actor system on its own named-dispatchers as well as exposing the new private creation of its own actor system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2593) Add ability to pass an existing Akka ActorSystem into Spark
[ https://issues.apache.org/jira/browse/SPARK-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14138390#comment-14138390 ] Matei Zaharia commented on SPARK-2593: -- The reason that we don't want to expose Akka classes in public APIs is that Spark has stricter versioning guidelines than Akka: https://cwiki.apache.org/confluence/display/SPARK/Spark+Versioning+Policy. Basically, all of Spark 1.X is going to be backward-compatible, so that means that we might get stuck with an old version of Akka that we have to maintain, which won't link with user's new versions of Akka. We can, however, provide a way to do this through reflection or *maybe* through an API marked @DeveloperApi so it's not guaranteed to work across releases. I'm also curious though, in practice, are there major problems with having 2 ActorSystems in the same app? It will create better fault isolation, and we do use Akka in fairly heavy ways (e.g. sending large messages, configuring our own timeouts, etc). I can see this leading to crashes that would not have happened otherwise. Add ability to pass an existing Akka ActorSystem into Spark --- Key: SPARK-2593 URL: https://issues.apache.org/jira/browse/SPARK-2593 Project: Spark Issue Type: Improvement Components: Spark Core Reporter: Helena Edelson As a developer I want to pass an existing ActorSystem into StreamingContext in load-time so that I do not have 2 actor systems running on a node in an Akka application. This would mean having spark's actor system on its own named-dispatchers as well as exposing the new private creation of its own actor system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2593) Add ability to pass an existing Akka ActorSystem into Spark
[ https://issues.apache.org/jira/browse/SPARK-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14138402#comment-14138402 ] Matei Zaharia commented on SPARK-2593: -- BTW doing this for the ActorReceiver for Spark Streaming is fair game, since that is tied to Akka already. Add ability to pass an existing Akka ActorSystem into Spark --- Key: SPARK-2593 URL: https://issues.apache.org/jira/browse/SPARK-2593 Project: Spark Issue Type: Improvement Components: Spark Core Reporter: Helena Edelson As a developer I want to pass an existing ActorSystem into StreamingContext in load-time so that I do not have 2 actor systems running on a node in an Akka application. This would mean having spark's actor system on its own named-dispatchers as well as exposing the new private creation of its own actor system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2593) Add ability to pass an existing Akka ActorSystem into Spark
[ https://issues.apache.org/jira/browse/SPARK-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135835#comment-14135835 ] Patrick Wendell commented on SPARK-2593: Hey [~helena] before you spend a lot of time on this - in Spark we haven't exposed Akka as a public API and we have a policy in general of exposing third party libraries in our API as little as possible. I could see us re-writing the internal communication to not use Akka in the 1.X time frame, so I think there could be resistance to anything that commits our API to akka. We expect users who build apps on akka to maintain their own actor system. And we are willing to trade off some minor inconveniences (e.g. tests are slower, etc) for not having to support that API. Add ability to pass an existing Akka ActorSystem into Spark --- Key: SPARK-2593 URL: https://issues.apache.org/jira/browse/SPARK-2593 Project: Spark Issue Type: Improvement Components: Spark Core Reporter: Helena Edelson As a developer I want to pass an existing ActorSystem into StreamingContext in load-time so that I do not have 2 actor systems running on a node in an Akka application. This would mean having spark's actor system on its own named-dispatchers as well as exposing the new private creation of its own actor system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2593) Add ability to pass an existing Akka ActorSystem into Spark
[ https://issues.apache.org/jira/browse/SPARK-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135840#comment-14135840 ] Patrick Wendell commented on SPARK-2593: One potential alternative would be for the akka community to provide a utility that uses reflection or some other workaround to expose the actor system inside of Spark. Then users could take advantage of this (those who don't mind about potential incompatible changes down the line). Add ability to pass an existing Akka ActorSystem into Spark --- Key: SPARK-2593 URL: https://issues.apache.org/jira/browse/SPARK-2593 Project: Spark Issue Type: Improvement Components: Spark Core Reporter: Helena Edelson As a developer I want to pass an existing ActorSystem into StreamingContext in load-time so that I do not have 2 actor systems running on a node in an Akka application. This would mean having spark's actor system on its own named-dispatchers as well as exposing the new private creation of its own actor system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2593) Add ability to pass an existing Akka ActorSystem into Spark
[ https://issues.apache.org/jira/browse/SPARK-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135968#comment-14135968 ] Andy Petrella commented on SPARK-2593: -- Couldn't we can translate the needs here to an API which would open communication channels between nodes in the cluster (internally, to the driver, ...?) even from *within* a task. Probable use case is metrics gathering (including fine-grained progress monitoring)? Actually, this feature comes OOTB with the spark akka system, which is already configured to access actors on the nodes, so that it's appealing to reuse it for that. _my .2ยข only_ Add ability to pass an existing Akka ActorSystem into Spark --- Key: SPARK-2593 URL: https://issues.apache.org/jira/browse/SPARK-2593 Project: Spark Issue Type: Improvement Components: Spark Core Reporter: Helena Edelson As a developer I want to pass an existing ActorSystem into StreamingContext in load-time so that I do not have 2 actor systems running on a node in an Akka application. This would mean having spark's actor system on its own named-dispatchers as well as exposing the new private creation of its own actor system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2593) Add ability to pass an existing Akka ActorSystem into Spark
[ https://issues.apache.org/jira/browse/SPARK-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14136463#comment-14136463 ] Helena Edelson commented on SPARK-2593: --- [~pwendell] I've spent no time thus far because yet, the Akka ActorSystem is private, that is the point of my ticket ;) once that can be exposed I'm g2g. - Helena Add ability to pass an existing Akka ActorSystem into Spark --- Key: SPARK-2593 URL: https://issues.apache.org/jira/browse/SPARK-2593 Project: Spark Issue Type: Improvement Components: Spark Core Reporter: Helena Edelson As a developer I want to pass an existing ActorSystem into StreamingContext in load-time so that I do not have 2 actor systems running on a node in an Akka application. This would mean having spark's actor system on its own named-dispatchers as well as exposing the new private creation of its own actor system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2593) Add ability to pass an existing Akka ActorSystem into Spark
[ https://issues.apache.org/jira/browse/SPARK-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14136565#comment-14136565 ] Tupshin Harper commented on SPARK-2593: --- [~pwendell] I'd be curious to hear any of the thinking that might go into rewriting spark to not use akka. From my vantage point, I'd like to see the opposite, and have spark take more advantage of akka features. Add ability to pass an existing Akka ActorSystem into Spark --- Key: SPARK-2593 URL: https://issues.apache.org/jira/browse/SPARK-2593 Project: Spark Issue Type: Improvement Components: Spark Core Reporter: Helena Edelson As a developer I want to pass an existing ActorSystem into StreamingContext in load-time so that I do not have 2 actor systems running on a node in an Akka application. This would mean having spark's actor system on its own named-dispatchers as well as exposing the new private creation of its own actor system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2593) Add ability to pass an existing Akka ActorSystem into Spark
[ https://issues.apache.org/jira/browse/SPARK-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14132797#comment-14132797 ] Helena Edelson commented on SPARK-2593: --- Here is a good example of just one of the issues: it is difficult to locate a remote spark actor to publish data to the stream. Here I have to have the streaming actor get created and in the preStart, publish a custom message with `self`which my actors in my ActorSystem can receive in order to get the ActorRef to send to. This is incredibly clunky. I will try to carve out some time to do this PR this week. Add ability to pass an existing Akka ActorSystem into Spark --- Key: SPARK-2593 URL: https://issues.apache.org/jira/browse/SPARK-2593 Project: Spark Issue Type: Improvement Components: Spark Core Reporter: Helena Edelson As a developer I want to pass an existing ActorSystem into StreamingContext in load-time so that I do not have 2 actor systems running on a node in an Akka application. This would mean having spark's actor system on its own named-dispatchers as well as exposing the new private creation of its own actor system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2593) Add ability to pass an existing Akka ActorSystem into Spark
[ https://issues.apache.org/jira/browse/SPARK-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14131333#comment-14131333 ] Ryan D Braley commented on SPARK-2593: -- This would be quite useful. It is hard to use actorStream with spark streaming where you have remote actors sending to spark because we need two actor systems. Right now it seems the name of the actor system in spark is hardcoded to spark. In order for actors to join an akka cluster we need to have the actor systems share the same name. Thus it is currently difficult to distribute work from an external actor system to the spark cluster without this change. Add ability to pass an existing Akka ActorSystem into Spark --- Key: SPARK-2593 URL: https://issues.apache.org/jira/browse/SPARK-2593 Project: Spark Issue Type: Improvement Components: Spark Core Reporter: Helena Edelson As a developer I want to pass an existing ActorSystem into StreamingContext in load-time so that I do not have 2 actor systems running on a node in an Akka application. This would mean having spark's actor system on its own named-dispatchers as well as exposing the new private creation of its own actor system. I would like to create an Akka Extension that wraps around Spark/Spark Streaming and Cassandra. So the programmatic creation would simply be this for a user val extension = SparkCassandra(system) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2593) Add ability to pass an existing Akka ActorSystem into Spark
[ https://issues.apache.org/jira/browse/SPARK-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14070476#comment-14070476 ] Evan Chan commented on SPARK-2593: -- I would say that the base SparkContext should have this ability, after all it creates multiple actors as well as a base ActorSystem. Sharing a single ActorSystem would also speed up all of Spark's tests, many of which repeatedly create and then tear down ActorSystems (since that's what the base SparkContext does). Add ability to pass an existing Akka ActorSystem into Spark --- Key: SPARK-2593 URL: https://issues.apache.org/jira/browse/SPARK-2593 Project: Spark Issue Type: Improvement Components: Spark Core Reporter: Helena Edelson As a developer I want to pass an existing ActorSystem into StreamingContext in load-time so that I do not have 2 actor systems running on a node in an Akka application. This would mean having spark's actor system on its own named-dispatchers as well as exposing the new private creation of its own actor system. I would like to create an Akka Extension that wraps around Spark/Spark Streaming and Cassandra. So the programmatic creation would simply be this for a user val extension = SparkCassandra(system) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-2593) Add ability to pass an existing Akka ActorSystem into Spark
[ https://issues.apache.org/jira/browse/SPARK-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14067540#comment-14067540 ] Helena Edelson commented on SPARK-2593: --- I should note that I'd be happy to do the changes. I am a committer to Akka Cluster. Add ability to pass an existing Akka ActorSystem into Spark --- Key: SPARK-2593 URL: https://issues.apache.org/jira/browse/SPARK-2593 Project: Spark Issue Type: Brainstorming Components: Spark Core Reporter: Helena Edelson As a developer I want to pass an existing ActorSystem into StreamingContext in load-time so that I do not have 2 actor systems running on a node. This would mean having spark's actor system on its own named-dispatchers as well as exposing the new private creation of its own actor system. If it makes sense... I would like to create an Akka Extension that wraps around Spark/Spark Streaming and Cassandra. So the creation would simply be this for a user val extension = SparkCassandra(system) and using is as easy as: import extension._ spark. // do work or, streaming. // do work and all config comes from reference.conf and user overrides of that. The conf file would pick up settings from the deployed environment first, then fallback to -D with a final fallback to configured settings. -- This message was sent by Atlassian JIRA (v6.2#6252)
