Jenkins build is back to stable : airavata-dev » Airavata Registry Core #642

[Apache Jenkins Server]

See 

Jenkins build is back to stable : airavata-dev #642

[Apache Jenkins Server]

See 

Jenkins build became unstable: airavata-dev » Airavata Registry Core #641

[Apache Jenkins Server]

See 

Re: Mesos based meta-scheduling for Airavata

[Suresh Marru]

Hi Gourav,

Thank you for the nice informative summaries, posts like these are always 
educational. Keep’em coming. 

Suresh

> On Oct 7, 2016, at 10:56 PM, Shenoy, Gourav Ganesh  
> wrote:
> 
> Hi dev,
>  
> I have been exploring different frameworks for Mesos which would help our 
> use-case of providing Airavata the capability to run jobs in a Mesos based 
> ecosystem. In particular, I have been playing around with Marathon & Chronos 
> and I am now going to be working on Apache Aurora. 
>  
> I have summarized my understanding about Mesos, Marathon & Chronos below. I 
> will send out a separate email about Aurora later.
>  
> Apache Mesos:
>  
> · Apache Mesos is an open-source cluster manager, in the sense that 
> it helps deploy & manage different frameworks (or applications) in a large 
> clustered environment easily.
> · Mesos provides the ability to utilize underlying shared pool of 
> nodes as a single compute unit – That is, it can run many applications on 
> these nodes efficiently.
> · Mesos uses the concept of “offers” for scheduling and running jobs 
> on the underlying nodes. When a framework (application) wants to run 
> computations/jobs on the cluster, Mesos will decide how many resources it 
> will “offer” that framework based on the availability. The framework will 
> then decide which resources to use from the offer, and subsequently run the 
> computation/job on that resource.
> · In a typical cluster, you will have 3 or more Mesos masters & 
> multiple Mesos slaves. Multiple mesos masters help in providing high 
> availability – if one master goes down, Mesos will reelect a new leader 
> (master) – using Zookeeper.
> · The task mentioned above of providing “offers” to frameworks is 
> done by a master, whereas the slaves are the ones who run these computations.
>  
> · Some additional points:
> oI built a Mesos cluster with 3 masters & 2 slaves on EC2.
> oEach master & slave have 1GB of RAM & 1vCPU with 20GB of disk space.
>  
> Marathon:
>  
> · Marathon is considered a framework that runs on top of Mesos. It is 
> a container orchestration platform for Mesos and essentially acts as a 
> service scheduler.
> · It is named “marathon” because it is intended for long running 
> applications. That is, Marathon makes sure that the service it is running 
> never stops – if a service goes down or the slave on which the service is run 
> dies, marathon keeps re-starting it on different slaves. 
> · In some sense Marathon is very good for ensuring high availability 
> of services. That is, instead of running services directly on Mesos, run it 
> in Marathon if you never want it to die.
> Note: You can decide to run a service on multiple slave nodes and if 
> resources on these slaves are available, Mesos will “offer” them to Marathon.
> · It is called a container orchestration platform because it 
> “launches” these services inside a container – either Docker OR Mesos 
> container.
> · In my opinion it is not a suitable “job scheduler” for Airavata 
> because in Airavata we need to run a job and get the output rather than 
> keeping it running always. Instead, we can run other schedulers – 
> chronos/aurora as a service in Marathon.
> 
> Chronos:
>  
> · Chronos is a Cron scheduler for Mesos. It is good for running 
> scheduled jobs – jobs that need to be run for a certain number of times, 
> repeatedly after certain intervals.
> · Chronos also provides the ability to add dependencies between jobs 
> – That is, if a job1 is dependent on another job2 then it will run job1 first 
> and then run job2 after job1 completes. It also builds a Directed Acyclic 
> Graph (DAG) based on these dependencies.
> · Similar to Marathon, Chronos receives “offers” from Mesos master 
> whenever it needs to run a job on Mesos.
> · Again, I found that Chronos does not fit the Airavata use-case 
> since I could not find a way to run one-off jobs via Chronos – you need to 
> specify interval time for Chronos, & Chronos then re-runs the job after that 
> interval is complete (even if you decide to specify num. of repetitions=1).
>  
>  
> Some additional points:
> · Marathon & Chronos both have REST API support – eg: you can submit 
> jobs via APIs along with other interactions such as list jobs, etc.
> · I installed Marathon & Chronos frameworks on the Mesos master 
> nodes. This is how their health looks like on the Mesos dashboard:
>  
> 
> As you can see, there are 3 active tasks running in Chronos & 
> 4 active tasks (long running) in Marathon.
>  
> · I also installed Chronos as a service inside Marathon, and this is 
> how it looks like in the Marathon UI:
> 
> 
> Interestingly, Chronos (as a service in Marathon) was smart enough to 
> identify the jobs submitted via Chronos (as a framework on Mesos) & 
> vice-versa.
>  

Re: Work Stealing is not a good solution for Airavata.

[Shameera Rathnayaka]

It seems I might have missed some important parts in Airavata Architecture.
I think it is time for me to step back and re-evaluate again why I see
these big whole in our design. Thanks Amila for following this thread so
far.

Thanks,
Shameera.

On Fri, Oct 7, 2016 at 12:50 PM Shameera Rathnayaka 
wrote:

> On Fri, Oct 7, 2016 at 12:18 PM Amila Jayasekara 
> wrote:
>
> You misunderstood.
>
> Let me re-phrase what mentioned earlier: (You said) "No, it is still
> valid, thread goes to thread pool doesn't say *worker* is complete that
> request, *it* is waiting until actual hpc job runs on target computer
> resoruces".
> Try to read above statement and explain to me what you refer as "worker"
> and "it" (in bold) ?
>
>
> Worker mean, who consume from the queue (to be specific GFac). "It" means
> Worker.
>
>
> Here is how I understood above statement. You submit a job in a thread and
> then you put that thread to "wait" state until job finishes. Is that
> correct ?
>
> No, this is bad we are not putting thread to wait. we release it untill
> the submitted job finish, next time GFac start to process the same request,
>  it can be same thread or different thread from thread pool.
>
>
> Also, others let me know whether I read the statement incorrectly.
>
> -AJ
>
> On Fri, Oct 7, 2016 at 11:46 AM, Shameera Rathnayaka <
> shameerai...@gmail.com> wrote:
>
> glad you got that, also don't you see that when the message comes(very
> first right arrow) and when gfac send the ack for that message(very last
> left arrow)?  That is the lifetime of a one worker queue message.
>
> On Fri, Oct 7, 2016 at 9:57 AM Amila Jayasekara 
> wrote:
>
> As per this diagram, it seems the thread that submits the job is not the
> same as the thread that handles output.
> At least that is what I understand.
>
> -AJ
>
> On Thu, Oct 6, 2016 at 4:26 PM, Shameera Rathnayaka <
> shameerai...@gmail.com> wrote:
>
> Previous attachment doesn't work.
>
> On Thu, Oct 6, 2016 at 4:24 PM, Shameera Rathnayaka <
> shameerai...@gmail.com> wrote:
>
> [image: Work Queue Message Life time.png]Hi Amila,
>
> Please find work queue message execution sequence diagram below. Hope this
> will help to understand how it works in Airavata.
>
>
>
> On Thu, Oct 6, 2016 at 4:05 PM Suresh Marru  wrote:
>
> Just a quick top post. This is informative discussion, please continue :)
>
> I agree on that Airavata does not do Work Stealing but it implements "Work
> Queues”. Conceptually they are similar to the OS Kernel level work queens,
> but more in a distributed context -
> https://www.kernel.org/doc/Documentation/workqueue.txt
>
> Suresh
>
>
> On Oct 6, 2016, at 3:52 PM, Amila Jayasekara 
> wrote:
>
> On Thu, Oct 6, 2016 at 3:17 PM, Shameera Rathnayaka <
> shameerai...@gmail.com> wrote:
>
>
>
> On Thu, Oct 6, 2016 at 2:50 PM Amila Jayasekara 
> wrote:
>
> On Thu, Oct 6, 2016 at 11:07 AM, Shameera Rathnayaka <
> shameerai...@gmail.com> wrote:
>
> Hi Amila,
>
> -- Please explain how you used "work stealing" in distributed system. That
> would be interesting.
>
>
> Airavata depends on work stealing + amqp for followings,
> Fault Tolerance - This is one of major distributed system problem which
> critical in Airavata, What ever the reason experiment request processing
> shouldn't get any effect  from internal node failure. Even with the node
> failures, Airavata should be capable enough to continue experiment request
> processing or hold it until at least one node appear and then continue. How
> this is handled in Ariavata is, worker only ack for messages only after it
> completely processed it. If the node goes down without sendings  ack for
> the messages it was processing,then rabbitmq put all these un-ack messages
> back to the queue and available to consume again.
>
> Resource Utilization- Another important goal of distributed system to
> effectively use available resources in the system, namely the memory and
> processors of components.  In Airavata this will decide the throughput and
> response time of experiments. Currently, at a given time workers only get
> messages up to a preconfigured limit (the limit is prefetch count) But most
> of these jobs are async jobs. That means after worker gets fixed amount of
> jobs, it won't get any other jobs even worker capable or handling more
> jobs, waste of worker resources.
>
>
>
> You still did not answer my question. I want to know how you used "work
> stealing" in your implementation. In other words how distributed work
> stealing works in your implementation. The details  you gave above is
> unrelated and does not answer my question.
>
>
> I think I have explained, how we use work stealing (work queues). If you
> are finding a more analog solution to parallel computing work strealing
> then that is hard to explain.
>
>
> No, you have not. :-).
> Work stealing != work queues. In a 

Improve File Transfer Component

[Ajinkya Dhamnaskar]

Hello All,

Currently, in Airavata we support only scp file transfer. As we move ahead,
we are planing to support different file transfer protocols.
Instead of separate component, we are planing to integrate it in gfac
itself.

Let's use this thread to discuss the improvements in this regard and please
do share your views and suggestions.

-- 
Thanks and regards,

Ajinkya Dhamnaskar
Student ID : 0003469679
Masters (CS)
+1 (812) 369- 5416

Re: Work Stealing is not a good solution for Airavata.

[Shameera Rathnayaka]

On Fri, Oct 7, 2016 at 12:18 PM Amila Jayasekara 
wrote:

> You misunderstood.
>
> Let me re-phrase what mentioned earlier: (You said) "No, it is still
> valid, thread goes to thread pool doesn't say *worker* is complete that
> request, *it* is waiting until actual hpc job runs on target computer
> resoruces".
> Try to read above statement and explain to me what you refer as "worker"
> and "it" (in bold) ?
>

Worker mean, who consume from the queue (to be specific GFac). "It" means
Worker.

>
> Here is how I understood above statement. You submit a job in a thread and
> then you put that thread to "wait" state until job finishes. Is that
> correct ?
>
No, this is bad we are not putting thread to wait. we release it untill the
submitted job finish, next time GFac start to process the same request,  it
can be same thread or different thread from thread pool.


> Also, others let me know whether I read the statement incorrectly.
>
> -AJ
>
> On Fri, Oct 7, 2016 at 11:46 AM, Shameera Rathnayaka <
> shameerai...@gmail.com> wrote:
>
> glad you got that, also don't you see that when the message comes(very
> first right arrow) and when gfac send the ack for that message(very last
> left arrow)?  That is the lifetime of a one worker queue message.
>
> On Fri, Oct 7, 2016 at 9:57 AM Amila Jayasekara 
> wrote:
>
> As per this diagram, it seems the thread that submits the job is not the
> same as the thread that handles output.
> At least that is what I understand.
>
> -AJ
>
> On Thu, Oct 6, 2016 at 4:26 PM, Shameera Rathnayaka <
> shameerai...@gmail.com> wrote:
>
> Previous attachment doesn't work.
>
> On Thu, Oct 6, 2016 at 4:24 PM, Shameera Rathnayaka <
> shameerai...@gmail.com> wrote:
>
> [image: Work Queue Message Life time.png]Hi Amila,
>
> Please find work queue message execution sequence diagram below. Hope this
> will help to understand how it works in Airavata.
>
>
>
> On Thu, Oct 6, 2016 at 4:05 PM Suresh Marru  wrote:
>
> Just a quick top post. This is informative discussion, please continue :)
>
> I agree on that Airavata does not do Work Stealing but it implements "Work
> Queues”. Conceptually they are similar to the OS Kernel level work queens,
> but more in a distributed context -
> https://www.kernel.org/doc/Documentation/workqueue.txt
>
> Suresh
>
>
> On Oct 6, 2016, at 3:52 PM, Amila Jayasekara 
> wrote:
>
> On Thu, Oct 6, 2016 at 3:17 PM, Shameera Rathnayaka <
> shameerai...@gmail.com> wrote:
>
>
>
> On Thu, Oct 6, 2016 at 2:50 PM Amila Jayasekara 
> wrote:
>
> On Thu, Oct 6, 2016 at 11:07 AM, Shameera Rathnayaka <
> shameerai...@gmail.com> wrote:
>
> Hi Amila,
>
> -- Please explain how you used "work stealing" in distributed system. That
> would be interesting.
>
>
> Airavata depends on work stealing + amqp for followings,
> Fault Tolerance - This is one of major distributed system problem which
> critical in Airavata, What ever the reason experiment request processing
> shouldn't get any effect  from internal node failure. Even with the node
> failures, Airavata should be capable enough to continue experiment request
> processing or hold it until at least one node appear and then continue. How
> this is handled in Ariavata is, worker only ack for messages only after it
> completely processed it. If the node goes down without sendings  ack for
> the messages it was processing,then rabbitmq put all these un-ack messages
> back to the queue and available to consume again.
>
> Resource Utilization- Another important goal of distributed system to
> effectively use available resources in the system, namely the memory and
> processors of components.  In Airavata this will decide the throughput and
> response time of experiments. Currently, at a given time workers only get
> messages up to a preconfigured limit (the limit is prefetch count) But most
> of these jobs are async jobs. That means after worker gets fixed amount of
> jobs, it won't get any other jobs even worker capable or handling more
> jobs, waste of worker resources.
>
>
>
> You still did not answer my question. I want to know how you used "work
> stealing" in your implementation. In other words how distributed work
> stealing works in your implementation. The details  you gave above is
> unrelated and does not answer my question.
>
>
> I think I have explained, how we use work stealing (work queues). If you
> are finding a more analog solution to parallel computing work strealing
> then that is hard to explain.
>
>
> No, you have not. :-).
> Work stealing != work queues. In a distributed setting I would image
> following kind of a work stealing implementation; Every worker
> (orchestrator) maintains a request queue locally and it serve requests
> coming to the local queue. Whenever one worker runs out of more requests to
> serve it will query other distributed workers local queues to see whether
> there are requests that 

Jenkins build is back to stable : airavata-dev #633

[Apache Jenkins Server]

See 

Jenkins build is back to stable : airavata-dev » Airavata Registry Core #633

[Apache Jenkins Server]

See 

Jenkins build became unstable: airavata-dev #632

[Apache Jenkins Server]

See 

Re: KafkaAppender for Airavata

[Shameera Rathnayaka]

How many instances need to deploy your logging solution in productions? How
> Simple/Complex compare to Airavata deployment?
>
> These are completely deployment level details which someone needs to
> experiment with based on the load of Airavata deployment, for the setup I
> tested I used 3 node kafka cluster with 25GB in each and I think its more
> than enough but completely depends on the deployment.
>

Do this solution need only kafka? I am interested in deployment setup for
full solution not just for kafka. As you already worked on similar steup
previously(you mentioned above) i though you might have recommended
deployment on board.


>
>
>
> Regarding why Kafka, I've used similar setup and it never gave me a
> problem and worked with thousands of logs without giving performance
> problems or issue with the scale of the logs.  I did a demo/presentation
> about this to the IU folks few weeks back and attached the presentation to
> Jira[1].
>
>
> Just because Kafka works doesn't mean kafka is the best solution for
> Airavata.
>
> Yes obvious statement.
>
 Yes, you gave that obvious statement for the original quection "why
kafka?". Do you have any other reason than this obvious statement.


> If you explain little bit the setup you have worked on, that would help us
> to get some ideas. What is the other advantage make Kafka is the best one
> here instead of RabbitMQ (Which we already use in Airavata for messaging) ?
>
> Yes, having RabbitMQ appender is also good, so if some user wants to use
> RabbitMQ they can use if someone wants Kafka they can use Kafka. I think we
> should support Kafka for internal communication too.
>
Could you explain more about this?


> Having KafkaAppender will not stop you from implementing more Appenders,
> there's more improvements can be done in this area to provide better
> infrastructure with Airavata.
>
But we should have default solution in Airavata, that is the whole point of
this integration, anyone can write any appender. When i see you solution I
don't see an easy way to plug different appenders.

Thanks,
Shameera.




> Regards,
> Shameera.
>
>
>
> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065
> Regards
> Lahiru
>
>
>
>
> Regards,
> Shameera.
>
> On Tue, Oct 4, 2016 at 1:58 PM Lahiru Ginnaliya Gamathige <
> glah...@gmail.com> wrote:
>
> Hi Suresh,
>
> If logstash see back-pressure then it stops processing the messages so its
> always good to have a persistant storage as the input side of the logstash,
> so logstash can crash anytime and kafka offset works very nicely with the
> cases where logstash crashes and you just have to restart the logstash and
> missed logs will be loaded to ES cluster.
>
> There's nothing like sending the logs *to* logstash because logstash is
> simply a data pipeline and not a data store. The other reason is that if we
> push to kafka we can just handle moving the data to ES with minimum number
> of logstash (theoretically one logstash) where we have the full
> configuration with what topics to listen and where to send those logs.
>
> Regards
> Lahiru
>
>
>
> On Tue, Oct 4, 2016 at 10:13 AM, Suresh Marru  wrote:
>
> Hi Lahiru,
>
> Can you please elaborate on the role of the messaging system (Kafka in
> your example) vs directly using log stash plugins [1] as an example?
>
> Suresh
> [1] -
> https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html
>
> On Oct 4, 2016, at 12:29 PM, Lahiru Ginnaliya Gamathige 
> wrote:
>
> Hi Devs,
>
> I've been working on a feature in Airavata to send its logs to a Kafka
> topic and finally move them to elastic search using logstash. Please find
> all the work[1] and the code[2] and I've added a documentation in to wiki
> [3](since we do not have a development location for wiki).
>
> After finishing the issue, Me and Ajinkya did a setup with Airavata and
> elastic search and Kafka cluster and we were able to visualize the logs in
> kibana.
>
> Please provide your valuable feedback on this feature.
>
> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065
> [2]
> https://github.com/apache/airavata/commit/8eea17fb9c8c073784795dfc56618ec556a12ad1
> [3]
> https://cwiki.apache.org/confluence/display/AIRAVATA/Tutorial+09+-+How+to+setup+centralized+logging+for+Airavata
>
>
> Regards
> Lahiru
>
>
>
> --
> Shameera Rathnayaka
>
> --
> Shameera Rathnayaka
>
> --
Shameera Rathnayaka

Jenkins build became unstable: airavata-dev » Airavata Registry Core #632

[Apache Jenkins Server]

See 

Re: KafkaAppender for Airavata

[Shameera Rathnayaka]

On Fri, Oct 7, 2016 at 10:15 AM Lahiru Ginnaliya Gamathige <
glah...@gmail.com> wrote:

> Created a Jira for RabbitMQ Appender[1], please implement it if you like.
>

It is always good to plug differnt implementations, If I am going to
implement a new logging appender(RabbitMQAppender or any other). How I can
plug it to Airavata? Can we easily plug differnt implementations? It seem
you have code agains the kafka logging implementation instead of an
Interface, correct me if I am wrong.  I think if you can provide better
solution to plug different appenders this task would be more easier.




>
>
> [1]https://issues.apache.org/jira/browse/AIRAVATA-2145
>
> On Fri, Oct 7, 2016 at 7:05 AM, Lahiru Ginnaliya Gamathige <
> glah...@gmail.com> wrote:
>
>
>
> On Thu, Oct 6, 2016 at 6:33 AM, Shameera Rathnayaka <
> shameerai...@gmail.com> wrote:
>
>
>
> On Tue, Oct 4, 2016 at 2:28 PM Lahiru Ginnaliya Gamathige <
> glah...@gmail.com> wrote:
>
> On Tue, Oct 4, 2016 at 11:07 AM, Shameera Rathnayaka <
> shameerai...@gmail.com> wrote:
>
> Hi Lahiru,
>
> I didn't see any architectural mail regarding this hence asking, Can you
> explain your design and deployment strategy here? few follow-up questions,
> how many kafka producers, partitions and consumers in the system?, why you
> choose kafka over other messaging systems?
>
> One producer in-process with Airavata JVM(if we deploy gfac, orchestrator,
> airavata-api separate then one produce per each component), consumer is in
> logstash which reads the logs from preconfigured topics. Once we push to
> Kafka its totally up the the developers who deploy airavata to use it
> however they want. Using logstash and ES is just one approach which me and
> Ajinkya tried and a very common solution people use to visualize their logs
> coming from multiple components.
>
>
> How many instances need to deploy your logging solution in productions?
> How Simple/Complex compare to Airavata deployment?
>
> These are completely deployment level details which someone needs to
> experiment with based on the load of Airavata deployment, for the setup I
> tested I used 3 node kafka cluster with 25GB in each and I think its more
> than enough but completely depends on the deployment.
>
>
>
>
> Regarding why Kafka, I've used similar setup and it never gave me a
> problem and worked with thousands of logs without giving performance
> problems or issue with the scale of the logs.  I did a demo/presentation
> about this to the IU folks few weeks back and attached the presentation to
> Jira[1].
>
>
> Just because Kafka works doesn't mean kafka is the best solution for
> Airavata.
>
> Yes obvious statement.
>
> If you explain little bit the setup you have worked on, that would help us
> to get some ideas. What is the other advantage make Kafka is the best one
> here instead of RabbitMQ (Which we already use in Airavata for messaging) ?
>
> Yes, having RabbitMQ appender is also good, so if some user wants to use
> RabbitMQ they can use if someone wants Kafka they can use Kafka. I think we
> should support Kafka for internal communication too. Having KafkaAppender
> will not stop you from implementing more Appenders, there's more
> improvements can be done in this area to provide better infrastructure with
> Airavata.
>
> Regards,
> Shameera.
>
>
>
> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065
> Regards
> Lahiru
>
>
>
>
> Regards,
> Shameera.
>
> On Tue, Oct 4, 2016 at 1:58 PM Lahiru Ginnaliya Gamathige <
> glah...@gmail.com> wrote:
>
> Hi Suresh,
>
> If logstash see back-pressure then it stops processing the messages so its
> always good to have a persistant storage as the input side of the logstash,
> so logstash can crash anytime and kafka offset works very nicely with the
> cases where logstash crashes and you just have to restart the logstash and
> missed logs will be loaded to ES cluster.
>
> There's nothing like sending the logs *to* logstash because logstash is
> simply a data pipeline and not a data store. The other reason is that if we
> push to kafka we can just handle moving the data to ES with minimum number
> of logstash (theoretically one logstash) where we have the full
> configuration with what topics to listen and where to send those logs.
>
> Regards
> Lahiru
>
>
>
> On Tue, Oct 4, 2016 at 10:13 AM, Suresh Marru  wrote:
>
> Hi Lahiru,
>
> Can you please elaborate on the role of the messaging system (Kafka in
> your example) vs directly using log stash plugins [1] as an example?
>
> Suresh
> [1] -
> https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html
>
> On Oct 4, 2016, at 12:29 PM, Lahiru Ginnaliya Gamathige 
> wrote:
>
> Hi Devs,
>
> I've been working on a feature in Airavata to send its logs to a Kafka
> topic and finally move them to elastic search using logstash. Please find
> all the work[1] and the code[2] and I've added a documentation in to wiki
> [3](since we do not have a development location for wiki).
>

Re: Work Stealing is not a good solution for Airavata.

[Shameera Rathnayaka]

glad you got that, also don't you see that when the message comes(very
first right arrow) and when gfac send the ack for that message(very last
left arrow)?  That is the lifetime of a one worker queue message.

On Fri, Oct 7, 2016 at 9:57 AM Amila Jayasekara 
wrote:

> As per this diagram, it seems the thread that submits the job is not the
> same as the thread that handles output.
> At least that is what I understand.
>
> -AJ
>
> On Thu, Oct 6, 2016 at 4:26 PM, Shameera Rathnayaka <
> shameerai...@gmail.com> wrote:
>
> Previous attachment doesn't work.
>
> On Thu, Oct 6, 2016 at 4:24 PM, Shameera Rathnayaka <
> shameerai...@gmail.com> wrote:
>
> [image: Work Queue Message Life time.png]Hi Amila,
>
> Please find work queue message execution sequence diagram below. Hope this
> will help to understand how it works in Airavata.
>
>
>
> On Thu, Oct 6, 2016 at 4:05 PM Suresh Marru  wrote:
>
> Just a quick top post. This is informative discussion, please continue :)
>
> I agree on that Airavata does not do Work Stealing but it implements "Work
> Queues”. Conceptually they are similar to the OS Kernel level work queens,
> but more in a distributed context -
> https://www.kernel.org/doc/Documentation/workqueue.txt
>
> Suresh
>
>
> On Oct 6, 2016, at 3:52 PM, Amila Jayasekara 
> wrote:
>
> On Thu, Oct 6, 2016 at 3:17 PM, Shameera Rathnayaka <
> shameerai...@gmail.com> wrote:
>
>
>
> On Thu, Oct 6, 2016 at 2:50 PM Amila Jayasekara 
> wrote:
>
> On Thu, Oct 6, 2016 at 11:07 AM, Shameera Rathnayaka <
> shameerai...@gmail.com> wrote:
>
> Hi Amila,
>
> -- Please explain how you used "work stealing" in distributed system. That
> would be interesting.
>
>
> Airavata depends on work stealing + amqp for followings,
> Fault Tolerance - This is one of major distributed system problem which
> critical in Airavata, What ever the reason experiment request processing
> shouldn't get any effect  from internal node failure. Even with the node
> failures, Airavata should be capable enough to continue experiment request
> processing or hold it until at least one node appear and then continue. How
> this is handled in Ariavata is, worker only ack for messages only after it
> completely processed it. If the node goes down without sendings  ack for
> the messages it was processing,then rabbitmq put all these un-ack messages
> back to the queue and available to consume again.
>
> Resource Utilization- Another important goal of distributed system to
> effectively use available resources in the system, namely the memory and
> processors of components.  In Airavata this will decide the throughput and
> response time of experiments. Currently, at a given time workers only get
> messages up to a preconfigured limit (the limit is prefetch count) But most
> of these jobs are async jobs. That means after worker gets fixed amount of
> jobs, it won't get any other jobs even worker capable or handling more
> jobs, waste of worker resources.
>
>
>
> You still did not answer my question. I want to know how you used "work
> stealing" in your implementation. In other words how distributed work
> stealing works in your implementation. The details  you gave above is
> unrelated and does not answer my question.
>
>
> I think I have explained, how we use work stealing (work queues). If you
> are finding a more analog solution to parallel computing work strealing
> then that is hard to explain.
>
>
> No, you have not. :-).
> Work stealing != work queues. In a distributed setting I would image
> following kind of a work stealing implementation; Every worker
> (orchestrator) maintains a request queue locally and it serve requests
> coming to the local queue. Whenever one worker runs out of more requests to
> serve it will query other distributed workers local queues to see whether
> there are requests that it can serve. If there are it can steal requests
> from other workers local queues and process. However, this model of
> computation is in efficient to do in a distributed environment. I guess
> that is the same reason we dont find much distributed work stealing
> implementations.
>
> Anyhow lets stop the discussion about work stealing now. :-)
>
>
>
>
>
>
>
>
> -- I dont see AMQP in the architecture diagram you attached above and I
> dont understand why Airavata has to depend on it. One way to figure this
> out is think about the architecture without AMQP and figure out what
> actually should happend and look for a way to do that using AMQP.
>
>
> Worker Queue is AMQP queue.
>
>
> Does the worker queue needs to be an AMQP queue ? Sorry, I dont know much
> about AMQP but it sounds like limitations you are explaining are because of
> AMQP.
>
>
> It is not, but good to use well-defined protocol instead of custom one.
> Almost all messaging systems have implemented AMQP protocol.
>
>
> Can we figure out whether others have also encountered the same/similar
> problem 

Re: [Airavata] : Local Provider

[Ajinkya Dhamnaskar]

Hello,

This is regarding database changes that we had to make to fix Local
provider.

I have added SECURITY_PROTOCOL column in LOCAL_SUBMISSION table. So, if
someone does not want to build database from scratch use below commands to
add these changes without hampering existing data.

ALTER TABLE LOCAL_SUBMISSION ADD COLUMN SECURITY_PROTOCOL VARCHAR (255);

UPDATE LOCAL_SUBMISSION SET SECURITY_PROTOCOL = 'LOCAL';

ALTER TABLE LOCAL_SUBMISSION MODIFY COLUMN SECURITY_PROTOCOL VARCHAR (255)
NOT NULL;
Please revert if you have any questions.

On Thu, Sep 29, 2016 at 4:18 PM, Shameera Rathnayaka  wrote:

> Hi Ajinkay,
>
> In Airavata we use Java Executor Services comes with java concurrency
> package for thread management, so we have delegate thread creation and
> management to Java instead of doing it all by ourselves. To be specific we
> used Fixed ThreadPool Executor inside Airavata. There are both good and bad
> in Thread vs Processes. If we use processes to run this local jobs then you
> have to manage process creation and deletion all by yourself anyway it is
> not a good practice to write long running synchronous jobs so we can assume
> these synchronous jobs are small in execution time.
>
> Thanks,
> Shameera.
>
> On Thu, Sep 29, 2016 at 2:36 PM Ajinkya Dhamnaskar 
> wrote:
>
>> Hello All,
>>
>> For local provider I was exploring different ways to submit and get the
>> job done. As, it is a local provider we need not to have monitoring system
>> in place, simply forking a job on different thread should serve the
>> purpose.
>>
>> But, to do so we need to keep investigation that thread. In my opinion,
>> ProcessBuilder in java would be handy in this case or if you have any other
>> suggestions please do let me know.
>>
>> Just want to know and use if there is any better way to this.
>>
>> Thanks in anticipation
>> --
>> Thanks and regards,
>>
>> Ajinkya Dhamnaskar
>> Student ID : 0003469679
>> Masters (CS)
>> +1 (812) 369- 5416
>>
> --
> Shameera Rathnayaka
>



-- 
Thanks and regards,

Ajinkya Dhamnaskar
Student ID : 0003469679
Masters (CS)
+1 (812) 369- 5416

Re: KafkaAppender for Airavata

[Lahiru Ginnaliya Gamathige]

Created a Jira for RabbitMQ Appender[1], please implement it if you like.


[1]https://issues.apache.org/jira/browse/AIRAVATA-2145

On Fri, Oct 7, 2016 at 7:05 AM, Lahiru Ginnaliya Gamathige <
glah...@gmail.com> wrote:

>
>
> On Thu, Oct 6, 2016 at 6:33 AM, Shameera Rathnayaka <
> shameerai...@gmail.com> wrote:
>
>>
>>
>> On Tue, Oct 4, 2016 at 2:28 PM Lahiru Ginnaliya Gamathige <
>> glah...@gmail.com> wrote:
>>
>>> On Tue, Oct 4, 2016 at 11:07 AM, Shameera Rathnayaka <
>>> shameerai...@gmail.com> wrote:
>>>
>>> Hi Lahiru,
>>>
>>> I didn't see any architectural mail regarding this hence asking, Can you
>>> explain your design and deployment strategy here? few follow-up questions,
>>> how many kafka producers, partitions and consumers in the system?, why you
>>> choose kafka over other messaging systems?
>>>
>>> One producer in-process with Airavata JVM(if we deploy gfac,
>>> orchestrator, airavata-api separate then one produce per each component),
>>> consumer is in logstash which reads the logs from preconfigured topics.
>>> Once we push to Kafka its totally up the the developers who deploy airavata
>>> to use it however they want. Using logstash and ES is just one approach
>>> which me and Ajinkya tried and a very common solution people use to
>>> visualize their logs coming from multiple components.
>>>
>>
>> How many instances need to deploy your logging solution in productions?
>> How Simple/Complex compare to Airavata deployment?
>>
> These are completely deployment level details which someone needs to
> experiment with based on the load of Airavata deployment, for the setup I
> tested I used 3 node kafka cluster with 25GB in each and I think its more
> than enough but completely depends on the deployment.
>
>>
>>
>
>>> Regarding why Kafka, I've used similar setup and it never gave me a
>>> problem and worked with thousands of logs without giving performance
>>> problems or issue with the scale of the logs.  I did a demo/presentation
>>> about this to the IU folks few weeks back and attached the presentation to
>>> Jira[1].
>>>
>>
>> Just because Kafka works doesn't mean kafka is the best solution for
>> Airavata.
>>
> Yes obvious statement.
>
>> If you explain little bit the setup you have worked on, that would help
>> us to get some ideas. What is the other advantage make Kafka is the best
>> one here instead of RabbitMQ (Which we already use in Airavata for
>> messaging) ?
>>
>> Yes, having RabbitMQ appender is also good, so if some user wants to use
> RabbitMQ they can use if someone wants Kafka they can use Kafka. I think we
> should support Kafka for internal communication too. Having KafkaAppender
> will not stop you from implementing more Appenders, there's more
> improvements can be done in this area to provide better infrastructure with
> Airavata.
>
>> Regards,
>> Shameera.
>>
>>
>>>
>>> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065
>>> Regards
>>> Lahiru
>>>
>>>
>>>
>>>
>>> Regards,
>>> Shameera.
>>>
>>> On Tue, Oct 4, 2016 at 1:58 PM Lahiru Ginnaliya Gamathige <
>>> glah...@gmail.com> wrote:
>>>
>>> Hi Suresh,
>>>
>>> If logstash see back-pressure then it stops processing the messages so
>>> its always good to have a persistant storage as the input side of the
>>> logstash, so logstash can crash anytime and kafka offset works very nicely
>>> with the cases where logstash crashes and you just have to restart the
>>> logstash and missed logs will be loaded to ES cluster.
>>>
>>> There's nothing like sending the logs *to* logstash because logstash is
>>> simply a data pipeline and not a data store. The other reason is that if we
>>> push to kafka we can just handle moving the data to ES with minimum number
>>> of logstash (theoretically one logstash) where we have the full
>>> configuration with what topics to listen and where to send those logs.
>>>
>>> Regards
>>> Lahiru
>>>
>>>
>>>
>>> On Tue, Oct 4, 2016 at 10:13 AM, Suresh Marru  wrote:
>>>
>>> Hi Lahiru,
>>>
>>> Can you please elaborate on the role of the messaging system (Kafka in
>>> your example) vs directly using log stash plugins [1] as an example?
>>>
>>> Suresh
>>> [1] - https://www.elastic.co/guide/en/logstash/current/plugins-
>>> inputs-log4j.html
>>>
>>> On Oct 4, 2016, at 12:29 PM, Lahiru Ginnaliya Gamathige <
>>> glah...@gmail.com> wrote:
>>>
>>> Hi Devs,
>>>
>>> I've been working on a feature in Airavata to send its logs to a Kafka
>>> topic and finally move them to elastic search using logstash. Please find
>>> all the work[1] and the code[2] and I've added a documentation in to wiki
>>> [3](since we do not have a development location for wiki).
>>>
>>> After finishing the issue, Me and Ajinkya did a setup with Airavata and
>>> elastic search and Kafka cluster and we were able to visualize the logs in
>>> kibana.
>>>
>>> Please provide your valuable feedback on this feature.
>>>
>>> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065
>>> 

Re: KafkaAppender for Airavata

[Lahiru Ginnaliya Gamathige]

On Thu, Oct 6, 2016 at 6:33 AM, Shameera Rathnayaka 
wrote:

>
>
> On Tue, Oct 4, 2016 at 2:28 PM Lahiru Ginnaliya Gamathige <
> glah...@gmail.com> wrote:
>
>> On Tue, Oct 4, 2016 at 11:07 AM, Shameera Rathnayaka <
>> shameerai...@gmail.com> wrote:
>>
>> Hi Lahiru,
>>
>> I didn't see any architectural mail regarding this hence asking, Can you
>> explain your design and deployment strategy here? few follow-up questions,
>> how many kafka producers, partitions and consumers in the system?, why you
>> choose kafka over other messaging systems?
>>
>> One producer in-process with Airavata JVM(if we deploy gfac,
>> orchestrator, airavata-api separate then one produce per each component),
>> consumer is in logstash which reads the logs from preconfigured topics.
>> Once we push to Kafka its totally up the the developers who deploy airavata
>> to use it however they want. Using logstash and ES is just one approach
>> which me and Ajinkya tried and a very common solution people use to
>> visualize their logs coming from multiple components.
>>
>
> How many instances need to deploy your logging solution in productions?
> How Simple/Complex compare to Airavata deployment?
>
These are completely deployment level details which someone needs to
experiment with based on the load of Airavata deployment, for the setup I
tested I used 3 node kafka cluster with 25GB in each and I think its more
than enough but completely depends on the deployment.

>
>

>> Regarding why Kafka, I've used similar setup and it never gave me a
>> problem and worked with thousands of logs without giving performance
>> problems or issue with the scale of the logs.  I did a demo/presentation
>> about this to the IU folks few weeks back and attached the presentation to
>> Jira[1].
>>
>
> Just because Kafka works doesn't mean kafka is the best solution for
> Airavata.
>
Yes obvious statement.

> If you explain little bit the setup you have worked on, that would help us
> to get some ideas. What is the other advantage make Kafka is the best one
> here instead of RabbitMQ (Which we already use in Airavata for messaging) ?
>
> Yes, having RabbitMQ appender is also good, so if some user wants to use
RabbitMQ they can use if someone wants Kafka they can use Kafka. I think we
should support Kafka for internal communication too. Having KafkaAppender
will not stop you from implementing more Appenders, there's more
improvements can be done in this area to provide better infrastructure with
Airavata.

> Regards,
> Shameera.
>
>
>>
>> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065
>> Regards
>> Lahiru
>>
>>
>>
>>
>> Regards,
>> Shameera.
>>
>> On Tue, Oct 4, 2016 at 1:58 PM Lahiru Ginnaliya Gamathige <
>> glah...@gmail.com> wrote:
>>
>> Hi Suresh,
>>
>> If logstash see back-pressure then it stops processing the messages so
>> its always good to have a persistant storage as the input side of the
>> logstash, so logstash can crash anytime and kafka offset works very nicely
>> with the cases where logstash crashes and you just have to restart the
>> logstash and missed logs will be loaded to ES cluster.
>>
>> There's nothing like sending the logs *to* logstash because logstash is
>> simply a data pipeline and not a data store. The other reason is that if we
>> push to kafka we can just handle moving the data to ES with minimum number
>> of logstash (theoretically one logstash) where we have the full
>> configuration with what topics to listen and where to send those logs.
>>
>> Regards
>> Lahiru
>>
>>
>>
>> On Tue, Oct 4, 2016 at 10:13 AM, Suresh Marru  wrote:
>>
>> Hi Lahiru,
>>
>> Can you please elaborate on the role of the messaging system (Kafka in
>> your example) vs directly using log stash plugins [1] as an example?
>>
>> Suresh
>> [1] - https://www.elastic.co/guide/en/logstash/current/
>> plugins-inputs-log4j.html
>>
>> On Oct 4, 2016, at 12:29 PM, Lahiru Ginnaliya Gamathige <
>> glah...@gmail.com> wrote:
>>
>> Hi Devs,
>>
>> I've been working on a feature in Airavata to send its logs to a Kafka
>> topic and finally move them to elastic search using logstash. Please find
>> all the work[1] and the code[2] and I've added a documentation in to wiki
>> [3](since we do not have a development location for wiki).
>>
>> After finishing the issue, Me and Ajinkya did a setup with Airavata and
>> elastic search and Kafka cluster and we were able to visualize the logs in
>> kibana.
>>
>> Please provide your valuable feedback on this feature.
>>
>> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065
>> [2]https://github.com/apache/airavata/commit/
>> 8eea17fb9c8c073784795dfc56618ec556a12ad1
>> [3]https://cwiki.apache.org/confluence/display/AIRAVATA/
>> Tutorial+09+-+How+to+setup+centralized+logging+for+Airavata
>>
>>
>> Regards
>> Lahiru
>>
>>
>>
>> --
>> Shameera Rathnayaka
>>
>> --
> Shameera Rathnayaka
>

Jenkins build is back to stable : airavata-dev #629

[Apache Jenkins Server]

See 

Jenkins build is back to stable : airavata-dev » Airavata Registry Core #629

[Apache Jenkins Server]

See