On Tue, Oct 4, 2016 at 11:07 AM, Shameera Rathnayaka <shameerai...@gmail.com > wrote:
> Hi Lahiru, > > I didn't see any architectural mail regarding this hence asking, Can you > explain your design and deployment strategy here? few follow-up questions, > how many kafka producers, partitions and consumers in the system?, why you > choose kafka over other messaging systems? > One producer in-process with Airavata JVM(if we deploy gfac, orchestrator, airavata-api separate then one produce per each component), consumer is in logstash which reads the logs from preconfigured topics. Once we push to Kafka its totally up the the developers who deploy airavata to use it however they want. Using logstash and ES is just one approach which me and Ajinkya tried and a very common solution people use to visualize their logs coming from multiple components. Regarding why Kafka, I've used similar setup and it never gave me a problem and worked with thousands of logs without giving performance problems or issue with the scale of the logs. I did a demo/presentation about this to the IU folks few weeks back and attached the presentation to Jira[1]. [1]https://issues.apache.org/jira/browse/AIRAVATA-2065 Regards Lahiru > > Regards, > Shameera. > > On Tue, Oct 4, 2016 at 1:58 PM Lahiru Ginnaliya Gamathige < > glah...@gmail.com> wrote: > >> Hi Suresh, >> >> If logstash see back-pressure then it stops processing the messages so >> its always good to have a persistant storage as the input side of the >> logstash, so logstash can crash anytime and kafka offset works very nicely >> with the cases where logstash crashes and you just have to restart the >> logstash and missed logs will be loaded to ES cluster. >> >> There's nothing like sending the logs *to* logstash because logstash is >> simply a data pipeline and not a data store. The other reason is that if we >> push to kafka we can just handle moving the data to ES with minimum number >> of logstash (theoretically one logstash) where we have the full >> configuration with what topics to listen and where to send those logs. >> >> Regards >> Lahiru >> >> >> >> On Tue, Oct 4, 2016 at 10:13 AM, Suresh Marru <sma...@apache.org> wrote: >> >> Hi Lahiru, >> >> Can you please elaborate on the role of the messaging system (Kafka in >> your example) vs directly using log stash plugins [1] as an example? >> >> Suresh >> [1] - https://www.elastic.co/guide/en/logstash/current/plugins- >> inputs-log4j.html >> >> On Oct 4, 2016, at 12:29 PM, Lahiru Ginnaliya Gamathige < >> glah...@gmail.com> wrote: >> >> Hi Devs, >> >> I've been working on a feature in Airavata to send its logs to a Kafka >> topic and finally move them to elastic search using logstash. Please find >> all the work[1] and the code[2] and I've added a documentation in to wiki >> [3](since we do not have a development location for wiki). >> >> After finishing the issue, Me and Ajinkya did a setup with Airavata and >> elastic search and Kafka cluster and we were able to visualize the logs in >> kibana. >> >> Please provide your valuable feedback on this feature. >> >> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065 >> [2]https://github.com/apache/airavata/commit/8eea17fb9c8c073 >> 784795dfc56618ec556a12ad1 >> [3]https://cwiki.apache.org/confluence/display/AIRAVATA/Tuto >> rial+09+-+How+to+setup+centralized+logging+for+Airavata >> >> >> Regards >> Lahiru >> >> >> >> -- > Shameera Rathnayaka >
