How many instances need to deploy your logging solution in productions? How > Simple/Complex compare to Airavata deployment? > > These are completely deployment level details which someone needs to > experiment with based on the load of Airavata deployment, for the setup I > tested I used 3 node kafka cluster with 25GB in each and I think its more > than enough but completely depends on the deployment. >
Do this solution need only kafka? I am interested in deployment setup for full solution not just for kafka. As you already worked on similar steup previously(you mentioned above) i though you might have recommended deployment on board. > > > > Regarding why Kafka, I've used similar setup and it never gave me a > problem and worked with thousands of logs without giving performance > problems or issue with the scale of the logs. I did a demo/presentation > about this to the IU folks few weeks back and attached the presentation to > Jira[1]. > > > Just because Kafka works doesn't mean kafka is the best solution for > Airavata. > > Yes obvious statement. > Yes, you gave that obvious statement for the original quection "why kafka?". Do you have any other reason than this obvious statement. > If you explain little bit the setup you have worked on, that would help us > to get some ideas. What is the other advantage make Kafka is the best one > here instead of RabbitMQ (Which we already use in Airavata for messaging) ? > > Yes, having RabbitMQ appender is also good, so if some user wants to use > RabbitMQ they can use if someone wants Kafka they can use Kafka. I think we > should support Kafka for internal communication too. > Could you explain more about this? > Having KafkaAppender will not stop you from implementing more Appenders, > there's more improvements can be done in this area to provide better > infrastructure with Airavata. > But we should have default solution in Airavata, that is the whole point of this integration, anyone can write any appender. When i see you solution I don't see an easy way to plug different appenders. Thanks, Shameera. > Regards, > Shameera. > > > > [1]https://issues.apache.org/jira/browse/AIRAVATA-2065 > Regards > Lahiru > > > > > Regards, > Shameera. > > On Tue, Oct 4, 2016 at 1:58 PM Lahiru Ginnaliya Gamathige < > glah...@gmail.com> wrote: > > Hi Suresh, > > If logstash see back-pressure then it stops processing the messages so its > always good to have a persistant storage as the input side of the logstash, > so logstash can crash anytime and kafka offset works very nicely with the > cases where logstash crashes and you just have to restart the logstash and > missed logs will be loaded to ES cluster. > > There's nothing like sending the logs *to* logstash because logstash is > simply a data pipeline and not a data store. The other reason is that if we > push to kafka we can just handle moving the data to ES with minimum number > of logstash (theoretically one logstash) where we have the full > configuration with what topics to listen and where to send those logs. > > Regards > Lahiru > > > > On Tue, Oct 4, 2016 at 10:13 AM, Suresh Marru <sma...@apache.org> wrote: > > Hi Lahiru, > > Can you please elaborate on the role of the messaging system (Kafka in > your example) vs directly using log stash plugins [1] as an example? > > Suresh > [1] - > https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html > > On Oct 4, 2016, at 12:29 PM, Lahiru Ginnaliya Gamathige <glah...@gmail.com> > wrote: > > Hi Devs, > > I've been working on a feature in Airavata to send its logs to a Kafka > topic and finally move them to elastic search using logstash. Please find > all the work[1] and the code[2] and I've added a documentation in to wiki > [3](since we do not have a development location for wiki). > > After finishing the issue, Me and Ajinkya did a setup with Airavata and > elastic search and Kafka cluster and we were able to visualize the logs in > kibana. > > Please provide your valuable feedback on this feature. > > [1]https://issues.apache.org/jira/browse/AIRAVATA-2065 > [2] > https://github.com/apache/airavata/commit/8eea17fb9c8c073784795dfc56618ec556a12ad1 > [3] > https://cwiki.apache.org/confluence/display/AIRAVATA/Tutorial+09+-+How+to+setup+centralized+logging+for+Airavata > > > Regards > Lahiru > > > > -- > Shameera Rathnayaka > > -- > Shameera Rathnayaka > > -- Shameera Rathnayaka
