Hello Shashank, I understand this won't help with merging all the log outputs into one single file, but you can configure your Storm workers to write logs to an NFS mounted to each of the workers (that's what we do) and than access them all at that one place with whichever tool you prefer.
Cheers Stefan On Fri, Mar 31, 2017 at 9:50 AM, Jungtaek Lim <[email protected]> wrote: > Hi Shashank, > > previously I used Fluentd with some plugins to tail log lines, parse, > transform, and store to ES. I think you can do it similar with Logstash. > > There're some frameworks like Apache NiFi to help ETL and I guess it can > achieve what you'd like to do. > > Thanks, > Jungtaek Lim (HeartSaVioR) > > 2017년 3월 31일 (금) 오후 4:46, Shashank Prasad <[email protected]>님이 작성: > >> Hi folks, >> >> Storm is a great tool but the logs are all over the place. As you >> increase your workers, your log files will increase as well and there is no >> single file it logs to. >> >> This makes it very hard to troubleshoot since you have to tail multiple >> logs. >> >> Ideally, i would like to ship all the logs for a topology to a >> centralized log server where i could use something like Kibana and filter >> the logs on what i am searching for. >> >> Anyone has any suggestion on how to achieve this or has a use case of how >> you currently doing it. >> >> Thanks a lot for your time! >> >> -shashank >> > -- Mgr. Stefan Rakonjac (+420) 774 692 671 [email protected]
