[akka-user] Configuring Akka to run as a job server using Play framework

[Mark Joslin]

Hi guys, I was hoping you could point me in the right direction. I have an 
API server with the Play framework that uses actors. Right now, these are 
just local actors but their workload is growing and since I'm beginning to 
introduce more & more scheduled background jobs, I would like it to be its 
own stand-alone job server.

I believe the difficulty lies in differentiating who is the API server and 
who is the job server and how that relates to actor creation / selection. 
Naturally, I just want the API server(s) to target the actor systems on the 
job server(s). Does anyone have any experience with this kind of job? My 
original thought was just to use environment variables and say "Ok, I'm an 
API server, don't spin up this actor system" OR "Ok, I'm a job server, join 
this actor system cluster". I tried looking around for some resources, but 
didn't find anything and was hoping you guys could help me out. 

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Configuring Akka to run as a job server using Play framework

[Michael Frank]

On 05/18/15 14:09, Mark Joslin wrote:
Hi guys, I was hoping you could point me in the right direction. I 
have an API server with the Play framework that uses actors. Right 
now, these are just local actors but their workload is growing and 
since I'm beginning to introduce more & more scheduled background 
jobs, I would like it to be its own stand-alone job server.


I believe the difficulty lies in differentiating who is the API server 
and who is the job server and how that relates to actor creation / 
selection. Naturally, I just want the API server(s) to target the 
actor systems on the job server(s). Does anyone have any experience 
with this kind of job? My original thought was just to use environment 
variables and say "Ok, I'm an API server, don't spin up this actor 
system" OR "Ok, I'm a job server, join this actor system cluster". I 
tried looking around for some resources, but didn't find anything and 
was hoping you guys could help me out.


it seems like this would fit well with cluster sharding 
(http://doc.akka.io/docs/akka/2.3.11/scala/cluster-usage.html#Cluster_Sharding) 
and persistence 
(http://doc.akka.io/docs/akka/2.3.11/scala/persistence.html).  your jobs 
could be cluster shard entities.  on the API side, you can use sharding 
proxy-only mode to route the requests from the API server to the job server.


alternatively, with some extra plumbing, you can use regular akka 
cluster and give each node in the cluster a role 
(http://doc.akka.io/docs/akka/2.3.11/scala/cluster-usage.html#Node_Roles). 
On the API side, you can listen for cluster membership events, and 
maintain a router that routes requests to the current job servers.


of the two, the first choice would be the easier one.

-Michael

--

 Read the docs: http://akka.io/docs/
 Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
 Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka User List" group.

To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Configuring Akka to run as a job server using Play framework

[Mark Joslin]

Hi Michael, thanks for the tips. I believe that would be a bit overkill for 
this project though it would seem fun. I've decided to take a pretty 
minimalistic approach:

   - Start two seed nodes on AWS 
   - Have all the API servers connect to the cluster via these seed nodes
   - Have all API servers create 1 master actor on the cluster (and 
   maintain a reference to it) which has a router actor underneath it of 
   worker actors

Simply said, every API server has one master that lives on the cluster and 
each master has a router of max 5 worker actors underneath it. I'm under 
the assumption jobs get distribute among the cluster in some sort of 
intelligent fashion. Please let me know if I'm under the wrong assumption.

However, I'm still a bit confused when it comes to splitting the play 
project up. I have a play project with 4 sub-modules underneath it, but I 
only want one of the modules. I imagine I can go about this in a few 
different ways:

   - Turn the one sub-module into a git repo itself and then reference it 
   via git submodules
  - One - Regular backend that contains all the submodules but now 
  contains a git reference
  - Two - Seed server that contains git submodule 
   - Package the entire play backend (with submodules) into a jar and copy 
   over to seed server git repo configuration. Treat it more as an artifact. 
   This seems faster, but I had trouble referencing my class files doing this 
   in a test project

Anyway, I'm sure I'll figure things out but appreciate any tips.


On Tuesday, May 19, 2015 at 1:00:27 PM UTC-4, Michael Frank wrote:
>
>  On 05/18/15 14:09, Mark Joslin wrote:
>  
> Hi guys, I was hoping you could point me in the right direction. I have an 
> API server with the Play framework that uses actors. Right now, these are 
> just local actors but their workload is growing and since I'm beginning to 
> introduce more & more scheduled background jobs, I would like it to be its 
> own stand-alone job server. 
>
>  I believe the difficulty lies in differentiating who is the API server 
> and who is the job server and how that relates to actor creation / 
> selection. Naturally, I just want the API server(s) to target the actor 
> systems on the job server(s). Does anyone have any experience with this 
> kind of job? My original thought was just to use environment variables and 
> say "Ok, I'm an API server, don't spin up this actor system" OR "Ok, I'm a 
> job server, join this actor system cluster". I tried looking around for 
> some resources, but didn't find anything and was hoping you guys could help 
> me out. 
>
>  it seems like this would fit well with cluster sharding (
> http://doc.akka.io/docs/akka/2.3.11/scala/cluster-usage.html#Cluster_Sharding)
>  
> and persistence (
> http://doc.akka.io/docs/akka/2.3.11/scala/persistence.html).  your jobs 
> could be cluster shard entities.  on the API side, you can use sharding 
> proxy-only mode to route the requests from the API server to the job server.
>
> alternatively, with some extra plumbing, you can use regular akka cluster 
> and give each node in the cluster a role (
> http://doc.akka.io/docs/akka/2.3.11/scala/cluster-usage.html#Node_Roles).  
> On the API side, you can listen for cluster membership events, and maintain 
> a router that routes requests to the current job servers.
>
> of the two, the first choice would be the easier one.
>
> -Michael
>  

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Configuring Akka to run as a job server using Play framework

[Michael Frank]

On 05/23/15 11:40, Mark Joslin wrote:
Hi Michael, thanks for the tips. I believe that would be a bit 
overkill for this project though it would seem fun. I've decided to 
take a pretty minimalistic approach:


  * Start two seed nodes on AWS
  * Have all the API servers connect to the cluster via these seed nodes
  * Have all API servers create 1 master actor on the cluster (and
maintain a reference to it) which has a router actor underneath it
of worker actors

Simply said, every API server has one master that lives on the cluster 
and each master has a router of max 5 worker actors underneath it. I'm 
under the assumption jobs get distribute among the cluster in some 
sort of intelligent fashion. Please let me know if I'm under the wrong 
assumption.


this isn't very clear to me, perhaps a some ascii box art would help?

regarding job distribution, i think you have the wrong assumption. if 
you are taking a minimalistic approach, then it is *you* who is building 
the intelligence :)


However, I'm still a bit confused when it comes to splitting the play 
project up. I have a play project with 4 sub-modules underneath it, 
but I only want one of the modules. I imagine I can go about this in a 
few different ways:


  * Turn the one sub-module into a git repo itself and then reference
it via git submodules
  o One - Regular backend that contains all the submodules but now
contains a git reference
  o Two - Seed server that contains git submodule
  * Package the entire play backend (with submodules) into a jar and
copy over to seed server git repo configuration. Treat it more as
an artifact. This seems faster, but I had trouble referencing my
class files doing this in a test project

Anyway, I'm sure I'll figure things out but appreciate any tips.


not knowing anything about your project layout, i would suggest creating 
artifacts and shipping those to servers.  in that way you know that each 
server is bit-for-bit the same.  not a new idea but it has a new popular 
term: 'immutable infrastructure': 
http://chadfowler.com/blog/2013/06/23/immutable-deployments/


-Michael

--

 Read the docs: http://akka.io/docs/
 Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
 Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka User List" group.

To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Configuring Akka to run as a job server using Play framework

[Mark Joslin]

Sure, it's probably not explained well. The cluster will consist of API 
server (s) and the seed nodes. It's this simple:



The API server joins the cluster actor system and then creates a *Master *actor 
that has workers underneath it. From here, I *feel* as if I'm finished with 
the scaling. I want to say "Ok, master actor lives in the cluster and will 
distribute the messages passed to it in some sort of intelligent manner 
that utilizes all the machines." I'm guessing this feeling is a bit of a 
farce, but could you clarify?

Regarding immutable architecture, I like to believe I subscribe to this 
since I need to publish artifacts to elastic beanstalk, but that's not 
really so much the issue. The real issue is this: the seed server & API 
server need identical code. The way I currently package the code for the 
API server (in Play) is I have a script that runs `activator docker:stage` 
and then zips things up and deploys to elasticbeanstalk via docker. Now 
that I have a "worker" server that needs the same code as the API, I'm 
asking myself "Do I somehow force an alternative deployment on the already 
existing code base?" -- e.g., just make a dedicated git branch meant for 
seed deployments that always stays updated with master OR "Do I place the 
API code in a 'seed' container that maintains a reference to existing code 
base?" -- e.g., create new git repo, new build.sbt, and use git submodule 
to pull in API code (this would give me greater modularity at the cost of 
ease of work I think)

Appreciate the tips; I had just recently started working with Akka and I 
find myself going through a mental shift where everyday I wake up and 
things become a little more clear. I guess a lot of my confusion with the 
cluster initially came to me thinking that I was connecting *to *a cluster 
from my API server whereas I was actually *joining *the cluster. Now I see 
it a bit different and things make a little more sense to me. 

On Wednesday, May 27, 2015 at 1:07:28 PM UTC-4, Michael Frank wrote:
>
>  On 05/23/15 11:40, Mark Joslin wrote:
>  
> Hi Michael, thanks for the tips. I believe that would be a bit overkill 
> for this project though it would seem fun. I've decided to take a pretty 
> minimalistic approach: 
>
>- Start two seed nodes on AWS  
>- Have all the API servers connect to the cluster via these seed nodes 
>- Have all API servers create 1 master actor on the cluster (and 
>maintain a reference to it) which has a router actor underneath it of 
>worker actors 
>
> Simply said, every API server has one master that lives on the cluster and 
> each master has a router of max 5 worker actors underneath it. I'm under 
> the assumption jobs get distribute among the cluster in some sort of 
> intelligent fashion. Please let me know if I'm under the wrong assumption.
>  
>
> this isn't very clear to me, perhaps a some ascii box art would help?
>
> regarding job distribution, i think you have the wrong assumption.  if you 
> are taking a minimalistic approach, then it is *you* who is building the 
> intelligence :)
>
>   However, I'm still a bit confused when it comes to splitting the play 
> project up. I have a play project with 4 sub-modules underneath it, but I 
> only want one of the modules. I imagine I can go about this in a few 
> different ways:
>  
>- Turn the one sub-module into a git repo itself and then reference it 
>via git submodules 
>   - One - Regular backend that contains all the submodules but now 
>   contains a git reference 
>   - Two - Seed server that contains git submodule  
>- Package the entire play backend (with submodules) into a jar and 
>copy over to seed server git repo configuration. Treat it more as an 
>artifact. This seems faster, but I had trouble referencing my class files 
>doing this in a test project 
>
> Anyway, I'm sure I'll figure things out but appreciate any tips.
>   
>
> not knowing anything about your project layout, i would suggest creating 
> artifacts and shipping those to servers.  in that way you know that each 
> server is bit-for-bit the same.  not a new idea but it has a new popular 
> term: 'immutable infrastructure': 
> http://chadfowler.com/blog/2013/06/23/immutable-deployments/
>
> -Michael
>  

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group