Thanks Ralph, what you proposed is fine by me. I'm not very familiar with the implementations inside log4j2 so in the above example I just want to make clear that enhancing property parsing could also be a direction for solve the problem.
Ralph Goers <[email protected]> 于2022年1月10日周一 12:53写道： > OK, I had a suspicion the reason for needing this was something like that. > > The syntax you propose below would require modifying PropertiesUtil, the > Property > plugin and StrSubstitutor to support arrays. I don’t think I’d want to go > down that path. > StrSubstitutor is already complicated and the idea of enhancing it to > support stuff like > this doesn’t excite me. > > I think we’d be better off creating a new plugin for this. Something like: > > <CompoundProperty name=“hadoop.root.logger.level” split=“,” > index=“0”>${sys:hadoop.root.logger}</CompoundProperty> > <CompoundProperty name=“hadoop.root.logger.appender” split=“,” > index=“1”>${sys:hadoop.root.logger}</CompoundProperty> > The only problem here is if you want to provide more than one appender > this won’t > work well. But I am sure we could enhance AppenderRef to accept a list of > appenders. > > Ralph > > > > > > On Jan 9, 2022, at 5:29 PM, 张铎(Duo Zhang) <[email protected]> wrote: > > > > Thank you for the reply. > > > > And yes, it is not only for the root logger, we have tons of different > > loggers in hadoop, as hadoop is constructed with several different > projects > > actually, and they are developed by different groups of people... > > > > And on splitting the config in shell, actually this is exactly what I > have > > done in HBase > > > > See > > > https://github.com/apache/hbase/blob/3ec3df5887e9271f7e75779eafe2439012cfb2c3/bin/hbase#L829 > > > > And also > > > https://github.com/apache/hbase/blob/3ec3df5887e9271f7e75779eafe2439012cfb2c3/bin/hbase.cmd#L336 > > > > For HBase maybe it is acceptable but hadoop is another story. > > Hadoop is widely used almost in every bigdata platform. Among the > > companies I've worked with, they all have their own hadoop deployment > > systems which have modified version of start up scripts... > > > > I can imagine that if we do the same trick in HBase, then people will ask > > again and again on the mailing list why passing -Dhadoop.root.logger does > > not work, as well as other options like -Dyarn.root.logger, > > -Dhadoop.security.logger, etc... > > > > So it will be good if we can support configure level and appenders in one > > property. > > > > Or maybe another way is to enhance the ability in the Properties section, > > so we can split one property into two properties, something like > > > > <Properties> > > <Property name="hadoop.root.logger" > > type="array">${sys:hadoop.root.logger:-INFO,Console}</Property> > > <Property > > name="hadoop.root.logger.level">${hadoop.root.logger[0]}</Property> > > <Property name="hadoop.root.logger.appender"> ${hadoop.root.logger[1]} > > </Property> > > <Properties> > > > > Notice that since we could add multiple appenders here, maybe we need to > > support something like ${hadoop.root.logger[1:] to combine all the > > appenders... > > This could also solve the problem as people could still pass > > -Dhadoop.root.logger. > > > > But I'm not sure if adding the ability to split a string in configuation > > could introduce new possible security concerns... > > > > Thanks. > > > > Ralph Goers <[email protected]> 于2022年1月10日周一 07:14写道： > > > >> We already support this with two variables. But if they only want to > pass > >> one then we have two options: > >> 1. Modify PropertiesConfiguration to support a new element that allows a > >> Level and AppenderRef which then gets split internally. > >> 2. Create a Lookup that extracts the relevant portion of the data. > >> > >> Ralph > >> > >>> On Jan 9, 2022, at 3:08 PM, Gary Gregory <[email protected]> > wrote: > >>> > >>> I think it is reasonable to say we can support this through 2 instead > of > >> 1 > >>> variable. > >>> > >>> Duo? > >>> > >>> Gary > >>> > >>> On Sun, Jan 9, 2022, 16:24 Ralph Goers <[email protected]> > >> wrote: > >>> > >>>> I’m looking at this and have a couple of concerns. The script has > >>>> > >>>> > >>>> HADOOP_ROOT_LOGGER=${HADOOP_ROOT_LOGGER:-${HADOOP_LOGLEVEL},console} > >>>> > >>>> > >> > HADOOP_DAEMON_ROOT_LOGGER=${HADOOP_DAEMON_ROOT_LOGGER:-${HADOOP_LOGLEVEL},RFA} > >>>> HADOOP_SECURITY_LOGGER=${HADOOP_SECURITY_LOGGER:-INFO,NullAppender} > >>>> > >>>> So it seems you need this for more than just the root logger. > >>>> > >>>> Second, you are asking us to accept “level, appender” as the value > and > >>>> under the covers split the > >>>> value and assign the level to the level attribute and the appender > name > >> to > >>>> the AppenderRef. > >>>> > >>>> While this certainly can be done it seems like it would be just as > easy > >> to > >>>> do the split in the script > >>>> and create two different variables for the logging configuration to > pick > >>>> up. Is there a reason that > >>>> you can’t do this - for example perhaps users have hacked your scripts > >> and > >>>> now you can’t > >>>> change them without breaking them? If so, was this “supported”? > >>>> > >>>> Ralph > >>>> > >>>> > >>>> > >>>>> On Jan 9, 2022, at 8:01 AM, 张铎(Duo Zhang) <[email protected]> > >> wrote: > >>>>> > >>>>> I brought this up in the incubator mailing list, and was suggested to > >>>>> report directly to the log4j community. > >>>>> > >>>>> https://issues.apache.org/jira/browse/HADOOP-16206 > >>>>> > >>>>> In hadoop we started to try migrating to log4j2 long ago, but it is > not > >>>>> easy. For now, one of the most blocker issues is the lack of support > of > >>>>> 'log4j.rootLogger=INFO,Console' grammar. Notice that we do not want > to > >>>> stay > >>>>> on the bridge api, we want to fully migrate to log4j2 finally, so > >>>>> supporting the above grammar in log4j1 bridge is not enough. > >>>>> > >>>>> This is the main log4j configuration file for hadoop > >>>>> > >>>>> > >>>> > >> > https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties > >>>>> > >>>>> We have this in the file > >>>>> > >>>>> # Define the root logger to the system property "hadoop.root.logger". > >>>>>> log4j.rootLogger=${hadoop.root.logger} > >>>>> > >>>>> > >>>>> So our users could simply pass -Dhadoop.root.logger to control the > >> level > >>>>> and appender. > >>>>> > >>>>> > >>>> > >> > https://github.com/apache/hadoop/blob/39efbc6b6fe53abed15f5639edcbaaa2a9dda6d2/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh#L904 > >>>>> > >>>>> Here we use an environment variable in our start up scripts. And I > >>>> believe > >>>>> there are lots of other hadoop deployment systems which will have > their > >>>> own > >>>>> start scripts. > >>>>> > >>>>> So in general, it is just impossible for us to drop the > >>>>> -Dhadoop.root.logger way of configuring our logging system. > >>>>> > >>>>> So here I want to ask if it is possible for log4j2 to still support > the > >>>>> 'log4j.rootLogger=INFO,Console' grammar. I'm not saying you must add > it > >>>>> back with no change, we just need a way to configure the level and > >>>> appeners > >>>>> at once, so something like > >>>>> 'log4j2.rootLogger.levelAndAppender=INFO,Console' is also OK. > >>>>> > >>>>> Thanks. > >>>> > >>>> > >> > >> > >
