I spent some time reading the code of log4j2-core, and posted what I have learned so far on the jira issue.
I could give it a try but I need a mentor. Appreciate if someone in the community could help here. Thanks. 张铎(Duo Zhang) <[email protected]> 于2022年1月15日周六 11:54写道： > Filed https://issues.apache.org/jira/browse/LOG4J2-3341. > > Gary Gregory <[email protected]> 于2022年1月11日周二 21:21写道： > >> On Tue, Jan 11, 2022 at 8:20 AM 张铎(Duo Zhang) <[email protected]> >> wrote: >> >> > So we could open an issue for this feature then? As we have already >> started >> > to discuss the implementation :) >> > >> >> Yes, that sounds like a good idea. Feel free to go ahead :-) >> >> Gary >> >> >> > >> > Thanks~ >> > >> > Volkan Yazıcı <[email protected]> 于2022年1月11日周二 19:42写道： >> > >> > > Giving it a one more try: >> > > >> > > <CompoundProperty> >> > > <Source>${sys:hadoop.root.logger}</Source> >> > > <Split delimiter="\s+" trim="true" blanksExcluded="true"> >> > > <item index="0">hadoop.root.logger.level</item> >> > > <slice startIndex="1">hadoop.root.logger.appender</slice> >> > > </Split> >> > > </CompoundProperty> >> > > >> > > On Mon, Jan 10, 2022 at 3:54 PM Ralph Goers < >> [email protected]> >> > > wrote: >> > > >> > > > Yeah, that syntax is better although it doesn’t solve the multiple >> > > > appender refs case either. >> > > > >> > > > LoggerConfig accepts a List<AppenderRef> so it would seem that if we >> > > > modify LoggerConfig >> > > > to add another attribute that is a string that creates an >> AppenderRef >> > for >> > > > each item then >> > > > one could do >> > > > >> > > > <Logger name=“xyzzy” level=“INFO” appenderRefNames=“a, b, c”/> >> > > > >> > > > >> > > > Ralph >> > > > >> > > > > On Jan 10, 2022, at 5:04 AM, Volkan Yazıcı <[email protected]> >> wrote: >> > > > > >> > > > > I am curious if shall we adopt a more generic approach to >> extraction: >> > > > > >> > > > > <CompoundProperty> >> > > > > <Source>${sys:hadoop.root.logger}</Source> >> > > > > <Targets pattern="^(.+)\s*,\s(.+)$"> >> > > > > <Target>hadoop.root.logger.level</Target> >> > > > > <Target>hadoop.root.logger.appender</Target> >> > > > > </Targets> >> > > > > </CompoundProperty> >> > > > > >> > > > > On Mon, Jan 10, 2022 at 5:53 AM Ralph Goers < >> > > [email protected]> >> > > > > wrote: >> > > > > >> > > > >> OK, I had a suspicion the reason for needing this was something >> like >> > > > that. >> > > > >> >> > > > >> The syntax you propose below would require modifying >> PropertiesUtil, >> > > the >> > > > >> Property >> > > > >> plugin and StrSubstitutor to support arrays. I don’t think I’d >> want >> > to >> > > > go >> > > > >> down that path. >> > > > >> StrSubstitutor is already complicated and the idea of enhancing >> it >> > to >> > > > >> support stuff like >> > > > >> this doesn’t excite me. >> > > > >> >> > > > >> I think we’d be better off creating a new plugin for this. >> Something >> > > > like: >> > > > >> >> > > > >> <CompoundProperty name=“hadoop.root.logger.level” split=“,” >> > > > >> index=“0”>${sys:hadoop.root.logger}</CompoundProperty> >> > > > >> <CompoundProperty name=“hadoop.root.logger.appender” split=“,” >> > > > >> index=“1”>${sys:hadoop.root.logger}</CompoundProperty> >> > > > >> >> > > > >> The only problem here is if you want to provide more than one >> > appender >> > > > >> this won’t >> > > > >> work well. But I am sure we could enhance AppenderRef to accept a >> > list >> > > > of >> > > > >> appenders. >> > > > >> >> > > > >> Ralph >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> > > > >>> On Jan 9, 2022, at 5:29 PM, 张铎(Duo Zhang) < >> [email protected]> >> > > > wrote: >> > > > >>> >> > > > >>> Thank you for the reply. >> > > > >>> >> > > > >>> And yes, it is not only for the root logger, we have tons of >> > > different >> > > > >>> loggers in hadoop, as hadoop is constructed with several >> different >> > > > >> projects >> > > > >>> actually, and they are developed by different groups of >> people... >> > > > >>> >> > > > >>> And on splitting the config in shell, actually this is exactly >> > what I >> > > > >> have >> > > > >>> done in HBase >> > > > >>> >> > > > >>> See >> > > > >>> >> > > > >> >> > > > >> > > >> > >> https://github.com/apache/hbase/blob/3ec3df5887e9271f7e75779eafe2439012cfb2c3/bin/hbase#L829 >> > > > >>> >> > > > >>> And also >> > > > >>> >> > > > >> >> > > > >> > > >> > >> https://github.com/apache/hbase/blob/3ec3df5887e9271f7e75779eafe2439012cfb2c3/bin/hbase.cmd#L336 >> > > > >>> >> > > > >>> For HBase maybe it is acceptable but hadoop is another story. >> > > > >>> Hadoop is widely used almost in every bigdata platform. Among >> the >> > > > >>> companies I've worked with, they all have their own hadoop >> > deployment >> > > > >>> systems which have modified version of start up scripts... >> > > > >>> >> > > > >>> I can imagine that if we do the same trick in HBase, then people >> > will >> > > > ask >> > > > >>> again and again on the mailing list why passing >> > -Dhadoop.root.logger >> > > > does >> > > > >>> not work, as well as other options like -Dyarn.root.logger, >> > > > >>> -Dhadoop.security.logger, etc... >> > > > >>> >> > > > >>> So it will be good if we can support configure level and >> appenders >> > in >> > > > one >> > > > >>> property. >> > > > >>> >> > > > >>> Or maybe another way is to enhance the ability in the Properties >> > > > section, >> > > > >>> so we can split one property into two properties, something like >> > > > >>> >> > > > >>> <Properties> >> > > > >>> <Property name="hadoop.root.logger" >> > > > >>> type="array">${sys:hadoop.root.logger:-INFO,Console}</Property> >> > > > >>> <Property >> > > > >>> >> name="hadoop.root.logger.level">${hadoop.root.logger[0]}</Property> >> > > > >>> <Property name="hadoop.root.logger.appender"> >> > > ${hadoop.root.logger[1]} >> > > > >>> </Property> >> > > > >>> <Properties> >> > > > >>> >> > > > >>> Notice that since we could add multiple appenders here, maybe we >> > need >> > > > to >> > > > >>> support something like ${hadoop.root.logger[1:] to combine all >> the >> > > > >>> appenders... >> > > > >>> This could also solve the problem as people could still pass >> > > > >>> -Dhadoop.root.logger. >> > > > >>> >> > > > >>> But I'm not sure if adding the ability to split a string in >> > > > configuation >> > > > >>> could introduce new possible security concerns... >> > > > >>> >> > > > >>> Thanks. >> > > > >>> >> > > > >>> Ralph Goers <[email protected]> 于2022年1月10日周一 07:14写道： >> > > > >>> >> > > > >>>> We already support this with two variables. But if they only >> want >> > to >> > > > >> pass >> > > > >>>> one then we have two options: >> > > > >>>> 1. Modify PropertiesConfiguration to support a new element that >> > > > allows a >> > > > >>>> Level and AppenderRef which then gets split internally. >> > > > >>>> 2. Create a Lookup that extracts the relevant portion of the >> data. >> > > > >>>> >> > > > >>>> Ralph >> > > > >>>> >> > > > >>>>> On Jan 9, 2022, at 3:08 PM, Gary Gregory < >> [email protected] >> > > >> > > > >> wrote: >> > > > >>>>> >> > > > >>>>> I think it is reasonable to say we can support this through 2 >> > > instead >> > > > >> of >> > > > >>>> 1 >> > > > >>>>> variable. >> > > > >>>>> >> > > > >>>>> Duo? >> > > > >>>>> >> > > > >>>>> Gary >> > > > >>>>> >> > > > >>>>> On Sun, Jan 9, 2022, 16:24 Ralph Goers < >> > [email protected] >> > > > >> > > > >>>> wrote: >> > > > >>>>> >> > > > >>>>>> I’m looking at this and have a couple of concerns. The script >> > has >> > > > >>>>>> >> > > > >>>>>> >> > > > >>>>>> >> > > HADOOP_ROOT_LOGGER=${HADOOP_ROOT_LOGGER:-${HADOOP_LOGLEVEL},console} >> > > > >>>>>> >> > > > >>>>>> >> > > > >>>> >> > > > >> >> > > > >> > > >> > >> HADOOP_DAEMON_ROOT_LOGGER=${HADOOP_DAEMON_ROOT_LOGGER:-${HADOOP_LOGLEVEL},RFA} >> > > > >>>>>> >> > > HADOOP_SECURITY_LOGGER=${HADOOP_SECURITY_LOGGER:-INFO,NullAppender} >> > > > >>>>>> >> > > > >>>>>> So it seems you need this for more than just the root logger. >> > > > >>>>>> >> > > > >>>>>> Second, you are asking us to accept “level, appender” as the >> > > value >> > > > >> and >> > > > >>>>>> under the covers split the >> > > > >>>>>> value and assign the level to the level attribute and the >> > appender >> > > > >> name >> > > > >>>> to >> > > > >>>>>> the AppenderRef. >> > > > >>>>>> >> > > > >>>>>> While this certainly can be done it seems like it would be >> just >> > as >> > > > >> easy >> > > > >>>> to >> > > > >>>>>> do the split in the script >> > > > >>>>>> and create two different variables for the logging >> configuration >> > > to >> > > > >> pick >> > > > >>>>>> up. Is there a reason that >> > > > >>>>>> you can’t do this - for example perhaps users have hacked >> your >> > > > scripts >> > > > >>>> and >> > > > >>>>>> now you can’t >> > > > >>>>>> change them without breaking them? If so, was this >> “supported”? >> > > > >>>>>> >> > > > >>>>>> Ralph >> > > > >>>>>> >> > > > >>>>>> >> > > > >>>>>> >> > > > >>>>>>> On Jan 9, 2022, at 8:01 AM, 张铎(Duo Zhang) < >> > [email protected] >> > > > >> > > > >>>> wrote: >> > > > >>>>>>> >> > > > >>>>>>> I brought this up in the incubator mailing list, and was >> > > suggested >> > > > to >> > > > >>>>>>> report directly to the log4j community. >> > > > >>>>>>> >> > > > >>>>>>> https://issues.apache.org/jira/browse/HADOOP-16206 >> > > > >>>>>>> >> > > > >>>>>>> In hadoop we started to try migrating to log4j2 long ago, >> but >> > it >> > > is >> > > > >> not >> > > > >>>>>>> easy. For now, one of the most blocker issues is the lack of >> > > > support >> > > > >> of >> > > > >>>>>>> 'log4j.rootLogger=INFO,Console' grammar. Notice that we do >> not >> > > want >> > > > >> to >> > > > >>>>>> stay >> > > > >>>>>>> on the bridge api, we want to fully migrate to log4j2 >> finally, >> > so >> > > > >>>>>>> supporting the above grammar in log4j1 bridge is not enough. >> > > > >>>>>>> >> > > > >>>>>>> This is the main log4j configuration file for hadoop >> > > > >>>>>>> >> > > > >>>>>>> >> > > > >>>>>> >> > > > >>>> >> > > > >> >> > > > >> > > >> > >> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties >> > > > >>>>>>> >> > > > >>>>>>> We have this in the file >> > > > >>>>>>> >> > > > >>>>>>> # Define the root logger to the system property >> > > > "hadoop.root.logger". >> > > > >>>>>>>> log4j.rootLogger=${hadoop.root.logger} >> > > > >>>>>>> >> > > > >>>>>>> >> > > > >>>>>>> So our users could simply pass -Dhadoop.root.logger to >> control >> > > the >> > > > >>>> level >> > > > >>>>>>> and appender. >> > > > >>>>>>> >> > > > >>>>>>> >> > > > >>>>>> >> > > > >>>> >> > > > >> >> > > > >> > > >> > >> https://github.com/apache/hadoop/blob/39efbc6b6fe53abed15f5639edcbaaa2a9dda6d2/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh#L904 >> > > > >>>>>>> >> > > > >>>>>>> Here we use an environment variable in our start up scripts. >> > And >> > > I >> > > > >>>>>> believe >> > > > >>>>>>> there are lots of other hadoop deployment systems which will >> > have >> > > > >> their >> > > > >>>>>> own >> > > > >>>>>>> start scripts. >> > > > >>>>>>> >> > > > >>>>>>> So in general, it is just impossible for us to drop the >> > > > >>>>>>> -Dhadoop.root.logger way of configuring our logging system. >> > > > >>>>>>> >> > > > >>>>>>> So here I want to ask if it is possible for log4j2 to still >> > > support >> > > > >> the >> > > > >>>>>>> 'log4j.rootLogger=INFO,Console' grammar. I'm not saying you >> > must >> > > > add >> > > > >> it >> > > > >>>>>>> back with no change, we just need a way to configure the >> level >> > > and >> > > > >>>>>> appeners >> > > > >>>>>>> at once, so something like >> > > > >>>>>>> 'log4j2.rootLogger.levelAndAppender=INFO,Console' is also >> OK. >> > > > >>>>>>> >> > > > >>>>>>> Thanks. >> > > > >>>>>> >> > > > >>>>>> >> > > > >>>> >> > > > >>>> >> > > > >> >> > > > >> >> > > > >> > > > >> > > >> > >> >
