Hi Sergey, What I'm asking is that the feature is implemented in a way that will allow Apex to run with different logger backend. That means that log4j needs to be optional.
Thanks, Thomas On Sat, Apr 15, 2017 at 2:01 PM, Sergey Golovko <ser...@datatorrent.com> wrote: > I agree it would be very nice to use only slf4j interfaces for the > implementation. But unfortunately the interface Appender belongs to > org.apache.log4j package. > > "SLF4J is only a facade, meaning that it does not provide a complete > logging solution. Operations such as configuring appenders or setting > logging levels cannot be performed with SLF4J. Thus, at some point in time, > any non-trivial application will need to directly invoke the underlying > logging system. In other words, complete independence from the API > underlying logging system is not possible for a stand-alone application. > Nevertheless, SLF4J reduces the impact of this dependence to near-painless > levels." > > https://www.slf4j.org/faq.html#when > > Thanks, > Sergey > > > On Thu, Apr 13, 2017 at 7:56 AM, Thomas Weise <t...@apache.org> wrote: > > > +1 > > > > Also the proposed feature would need to be implemented in a way that > avoids > > a hard dependency on log4j. The interface for logging is slf4j and it > > should be possible to use other logger backends. > > > > > > On Mon, Apr 10, 2017 at 9:21 PM, Sergey Golovko <ser...@datatorrent.com> > > wrote: > > > > > I don't think an operator needs a specific appender. An appender can be > > > dynamically assigned to an application designer, application master and > > > container. > > > > > > Thanks, > > > Sergey > > > > > > > > > On Mon, Apr 10, 2017 at 6:26 PM, Munagala Ramanath < > r...@datatorrent.com> > > > wrote: > > > > > > > I don't have one, I thought that was what the intent of the proposal > > was, > > > > but looks like > > > > I misunderstood. After re-reading some of the earlier responses, I > > > > understand the > > > > proposal better. > > > > > > > > Ram > > > > > > > > > > > > > > > > On Mon, Apr 10, 2017 at 5:39 PM, Vlad Rozov <v.ro...@datatorrent.com > > > > > > wrote: > > > > > > > > > I don't see a use case where an individual operators need to > define a > > > > > specific appender, can you provide one? > > > > > > > > > > Thank you, > > > > > > > > > > Vlad > > > > > > > > > > On 4/10/17 16:53, Munagala Ramanath wrote: > > > > > > > > > >> Yes, totally agree, it would be helpful to have a detailed use > case > > > > and/or > > > > >> a detailed spec > > > > >> of the desired capabilities -- not necessarily a complete spec but > > > with > > > > >> enough detail to > > > > >> understand why existing capabilities are inadequate. > > > > >> > > > > >> Ram > > > > >> > > > > >> On Mon, Apr 10, 2017 at 4:43 PM, Vlad Rozov < > > v.ro...@datatorrent.com> > > > > >> wrote: > > > > >> > > > > >> It will be good to understand a use case where an operator needs a > > > > >>> specific appender. > > > > >>> > > > > >>> IMO, an operator designer defines *what* should be logged and > > dev-ops > > > > >>> team > > > > >>> defines *where* to log. > > > > >>> > > > > >>> Thank you, > > > > >>> > > > > >>> Vlad > > > > >>> On 4/10/17 16:27, Munagala Ramanath wrote: > > > > >>> > > > > >>> Yes, I understand, I was just wondering if individual operators > > could > > > > >>>> define the appenders > > > > >>>> they potentially need at compile time and then the operator > > > callbacks > > > > >>>> could > > > > >>>> simply > > > > >>>> check the desired runtime condition and add the appropriate > > > appender. > > > > >>>> > > > > >>>> Or are we saying there are scenarios where we absolutely cannot > > > create > > > > >>>> the > > > > >>>> appender beforehand ? > > > > >>>> > > > > >>>> So broadly speaking, my question is whether the combination of > > > > providing > > > > >>>> predefined appenders > > > > >>>> and the PropertyConfigurator capabilities meets the need. > > > > >>>> > > > > >>>> Ram > > > > >>>> > > > > >>>> On Mon, Apr 10, 2017 at 2:18 PM, Sergey Golovko < > > > > ser...@datatorrent.com > > > > >>>> > > > > > >>>> wrote: > > > > >>>> > > > > >>>> Ram, > > > > >>>> > > > > >>>>> Really the new appender class must extend the abstract class > > > > >>>>> AppenderSkeleton. And in order to add a new appender > > > programmatically > > > > >>>>> in > > > > >>>>> Java, some code in Apex should call the following log4j method: > > > > >>>>> > > > > >>>>> org.apache.log4j.Logger.getRootLogger().addAppender(Appender > > > > >>>>> newAppender) > > > > >>>>> > > > > >>>>> The general idea of my proposal is "*based on some runtime > > > > parameter(s) > > > > >>>>> to > > > > >>>>> provide ability to create an appender instance via reflection > and > > > add > > > > >>>>> it > > > > >>>>> to > > > > >>>>> the list of active log4j appenders*". > > > > >>>>> > > > > >>>>> Thanks, > > > > >>>>> Sergey > > > > >>>>> > > > > >>>>> > > > > >>>>> On Mon, Apr 10, 2017 at 2:04 PM, Vlad Rozov < > > > v.ro...@datatorrent.com > > > > > > > > > >>>>> wrote: > > > > >>>>> > > > > >>>>> It will require application recompilation and repackaging. The > > > > proposed > > > > >>>>> > > > > >>>>>> functionality is for dev-ops to be able to route application > > > logging > > > > >>>>>> to > > > > >>>>>> a > > > > >>>>>> preferred destination without recompiling applications. It is > > > > run-time > > > > >>>>>> configuration vs compile time hardcoded appender. > > > > >>>>>> > > > > >>>>>> Thank you, > > > > >>>>>> > > > > >>>>>> Vlad > > > > >>>>>> > > > > >>>>>> On 4/10/17 11:23, Munagala Ramanath wrote: > > > > >>>>>> > > > > >>>>>> You can do it in a trivial derived class without changing the > > base > > > > >>>>>> class. > > > > >>>>>> Ram > > > > >>>>>> > > > > >>>>>>> On Mon, Apr 10, 2017 at 11:19 AM, Vlad Rozov < > > > > >>>>>>> v.ro...@datatorrent.com> > > > > >>>>>>> wrote: > > > > >>>>>>> > > > > >>>>>>> Does not the proposal to use Logger.addAppender() requires > > > > >>>>>>> modifications > > > > >>>>>>> > > > > >>>>>>> to used operators code? > > > > >>>>>>>> > > > > >>>>>>>> Thank you, > > > > >>>>>>>> > > > > >>>>>>>> Vlad > > > > >>>>>>>> > > > > >>>>>>>> On 4/10/17 10:58, Munagala Ramanath wrote: > > > > >>>>>>>> > > > > >>>>>>>> People can currently do this by simply implementing the > > Appender > > > > >>>>>>>> > > > > >>>>>>>> interface > > > > >>>>>>>>> and adding it > > > > >>>>>>>>> with Logger.addAppender() in the setup method. Why do we > need > > > > >>>>>>>>> > > > > >>>>>>>>> something > > > > >>>>>>>> > > > > >>>>>>> more elaborate ? > > > > >>>>>> > > > > >>>>>>> Ram > > > > >>>>>>>>> > > > > >>>>>>>>> On Mon, Apr 10, 2017 at 10:30 AM, Sergey Golovko < > > > > >>>>>>>>> ser...@datatorrent.com> > > > > >>>>>>>>> wrote: > > > > >>>>>>>>> > > > > >>>>>>>>> The configuration of a log4j appender via log4j > configuration > > > > file > > > > >>>>>>>>> is > > > > >>>>>>>>> > > > > >>>>>>>>> a > > > > >>>>>>>> > > > > >>>>>>> static configuration that cannot be disabled/enabled and > > managed > > > > >>>>>> > > > > >>>>>>> dynamically by an application designer. The programmatic > > approach > > > > >>>>>>>>>> > > > > >>>>>>>>>> will > > > > >>>>>>>>> > > > > >>>>>>>> allow an application designer to specify which of the > > available > > > > >>>>>> > > > > >>>>>>> log4j > > > > >>>>>>>>> > > > > >>>>>>>> appenders should be used for the specific application. > > > > >>>>>> > > > > >>>>>>> It is not necessary Apex should use the predefined log4j > > > appenders > > > > >>>>>>>>>> only. > > > > >>>>>>>>>> The log4j events contain useful but the very limited > number > > of > > > > >>>>>>>>>> properties > > > > >>>>>>>>>> which values can be printed into output log4j sources. But > > > based > > > > >>>>>>>>>> on > > > > >>>>>>>>>> > > > > >>>>>>>>>> the > > > > >>>>>>>>> > > > > >>>>>>>> knowledge of the software product workflow, the custom > defined > > > > log4j > > > > >>>>>> > > > > >>>>>>> appender can extend a list of predefined output log events > > > > >>>>>>>>>> properties > > > > >>>>>>>>>> and, > > > > >>>>>>>>>> for instance for Apex, return: node, user name, > application > > > > name, > > > > >>>>>>>>>> application id, container id, operator name, etc. > > > > >>>>>>>>>> > > > > >>>>>>>>>> Also the output log events that are generated by a custom > > > > defined > > > > >>>>>>>>>> > > > > >>>>>>>>>> log4j > > > > >>>>>>>>> > > > > >>>>>>>> appender can be stored and indexed by any type of a full > text > > > > search > > > > >>>>>> > > > > >>>>>>> database. It will allow the customers and developers to > > simplify > > > > >>>>>>>>>> collection > > > > >>>>>>>>>> of log events statistics and searching/filtering of > specific > > > > >>>>>>>>>> events > > > > >>>>>>>>>> > > > > >>>>>>>>>> for > > > > >>>>>>>>> > > > > >>>>>>>> debugging and investigation. > > > > >>>>>> > > > > >>>>>>> Thanks, > > > > >>>>>>>>>> Sergey > > > > >>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>>>> On Mon, Apr 10, 2017 at 6:34 AM, Vlad Rozov < > > > > >>>>>>>>>> v.ro...@datatorrent.com > > > > >>>>>>>>>> wrote: > > > > >>>>>>>>>> > > > > >>>>>>>>>> +1 Apex engine does not own log4j config file - it is > > provided > > > > >>>>>>>>>> either > > > > >>>>>>>>>> by > > > > >>>>>>>>>> > > > > >>>>>>>>>> Hadoop or an application. Hadoop log4j config does not > > > > necessarily > > > > >>>>>>>>>> > > > > >>>>>>>>>>> meet > > > > >>>>>>>>>>> application logging requirements, but if log4j is > provided > > by > > > > an > > > > >>>>>>>>>>> application designer, who can only specify what to log, > it > > > may > > > > >>>>>>>>>>> not > > > > >>>>>>>>>>> meet > > > > >>>>>>>>>>> operations requirements. Dev-ops should have an ability > to > > > > >>>>>>>>>>> specify > > > > >>>>>>>>>>> where > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> to > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> log depending on the available infrastructure at > run-time. > > > > >>>>>>>>>> > > > > >>>>>>>>>> It will be good to have an ability not only specify extra > > > log4j > > > > >>>>>>>>>>> appenders > > > > >>>>>>>>>>> at lunch time, but also at run-time, the same way how > log4j > > > > >>>>>>>>>>> logger > > > > >>>>>>>>>>> levels > > > > >>>>>>>>>>> may be changed. > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> Thank you, > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> Vlad > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> On 4/9/17 23:14, Priyanka Gugale wrote: > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> We can always write a custom appender and add it by > > changing > > > > root > > > > >>>>>>>>>>> appender > > > > >>>>>>>>>>> in log4j config file. Can you explain how adding appender > > > > >>>>>>>>>>> grammatically > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> would help? > > > > >>>>>>>>>>> > > > > >>>>>>>>>>>> -Priyanka > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> On Sun, Apr 9, 2017 at 11:50 AM, Sanjay Pujare < > > > > >>>>>>>>>>>> san...@datatorrent.com > > > > >>>>>>>>>>>> wrote: > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> Please give some examples and/or use cases of this > > > > programmatic > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> log4j > > > > >>>>>>>>>>> > > > > >>>>>>>>>> appender. > > > > >>>>>> > > > > >>>>>>> On Fri, Apr 7, 2017 at 8:40 PM, Sergey Golovko < > > > > >>>>>>>>>>>>> ser...@datatorrent.com > > > > >>>>>>>>>>>>> wrote: > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> Hi All, > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> I'd like to add supporting of a custom defined log4j > > > appender > > > > >>>>>>>>>>>>> that > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> can > > > > >>>>>>>>>>>>>> be > > > > >>>>>>>>>>>>>> added to Apex Application Master and Containers and be > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> configurable > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>> programmatically. > > > > >>>>>> > > > > >>>>>>> Sometimes it is not trivial to control log4j configuration > via > > > > >>>>>>>>>>>>>> log4j > > > > >>>>>>>>>>>>>> properties. And I think the having of the approach to > > add > > > a > > > > >>>>>>>>>>>>>> log4j > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> appender > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> programmatically will allow the customers and > developers > > > to > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>> plugin > > > > >>>>>>>>>>>>> their > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> own custom defined log4j appenders and be much flexible > > for > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>> streaming > > > > >>>>>>>>>>> > > > > >>>>>>>>>> and > > > > >>>>>> > > > > >>>>>>> collection of Apex log events. > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> I assume to provide generic approach for definition of > > the > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> programmatic > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> log4j appender and to pass all configuration > parameters > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>> including a > > > > >>>>>>>>>>>> name > > > > >>>>>>>>>>>> of > > > > >>>>>>>>>>>> the Java class with implementation of the log4j appender > > via > > > > >>>>>>>>>>>> system > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> and/or > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> command line properties. > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>> Thanks, > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> Sergey > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> > > > > >>>> > > > > >> > > > > > > > > > > > > > > > > > -- > > > > > > > > _______________________________________________________ > > > > > > > > Munagala V. Ramanath > > > > > > > > Software Engineer > > > > > > > > E: r...@datatorrent.com | M: (408) 331-5034 | Twitter: @UnknownRam > > > > > > > > www.datatorrent.com | apex.apache.org > > > > > > > > > >
