OK, I had a suspicion the reason for needing this was something like that.
The syntax you propose below would require modifying PropertiesUtil, the
Property
plugin and StrSubstitutor to support arrays. I don’t think I’d want to go down
that path.
StrSubstitutor is already complicated and the idea of enhancing it to support
stuff like
this doesn’t excite me.
I think we’d be better off creating a new plugin for this. Something like:
<CompoundProperty name=“hadoop.root.logger.level” split=“,”
index=“0”>${sys:hadoop.root.logger}</CompoundProperty>
<CompoundProperty name=“hadoop.root.logger.appender” split=“,”
index=“1”>${sys:hadoop.root.logger}</CompoundProperty>
The only problem here is if you want to provide more than one appender this
won’t
work well. But I am sure we could enhance AppenderRef to accept a list of
appenders.
Ralph
> On Jan 9, 2022, at 5:29 PM, 张铎(Duo Zhang) <[email protected]> wrote:
>
> Thank you for the reply.
>
> And yes, it is not only for the root logger, we have tons of different
> loggers in hadoop, as hadoop is constructed with several different projects
> actually, and they are developed by different groups of people...
>
> And on splitting the config in shell, actually this is exactly what I have
> done in HBase
>
> See
> https://github.com/apache/hbase/blob/3ec3df5887e9271f7e75779eafe2439012cfb2c3/bin/hbase#L829
>
> And also
> https://github.com/apache/hbase/blob/3ec3df5887e9271f7e75779eafe2439012cfb2c3/bin/hbase.cmd#L336
>
> For HBase maybe it is acceptable but hadoop is another story.
> Hadoop is widely used almost in every bigdata platform. Among the
> companies I've worked with, they all have their own hadoop deployment
> systems which have modified version of start up scripts...
>
> I can imagine that if we do the same trick in HBase, then people will ask
> again and again on the mailing list why passing -Dhadoop.root.logger does
> not work, as well as other options like -Dyarn.root.logger,
> -Dhadoop.security.logger, etc...
>
> So it will be good if we can support configure level and appenders in one
> property.
>
> Or maybe another way is to enhance the ability in the Properties section,
> so we can split one property into two properties, something like
>
> <Properties>
> <Property name="hadoop.root.logger"
> type="array">${sys:hadoop.root.logger:-INFO,Console}</Property>
> <Property
> name="hadoop.root.logger.level">${hadoop.root.logger[0]}</Property>
> <Property name="hadoop.root.logger.appender"> ${hadoop.root.logger[1]}
> </Property>
> <Properties>
>
> Notice that since we could add multiple appenders here, maybe we need to
> support something like ${hadoop.root.logger[1:] to combine all the
> appenders...
> This could also solve the problem as people could still pass
> -Dhadoop.root.logger.
>
> But I'm not sure if adding the ability to split a string in configuation
> could introduce new possible security concerns...
>
> Thanks.
>
> Ralph Goers <[email protected]> 于2022年1月10日周一 07:14写道：
>
>> We already support this with two variables. But if they only want to pass
>> one then we have two options:
>> 1. Modify PropertiesConfiguration to support a new element that allows a
>> Level and AppenderRef which then gets split internally.
>> 2. Create a Lookup that extracts the relevant portion of the data.
>>
>> Ralph
>>
>>> On Jan 9, 2022, at 3:08 PM, Gary Gregory <[email protected]> wrote:
>>>
>>> I think it is reasonable to say we can support this through 2 instead of
>> 1
>>> variable.
>>>
>>> Duo?
>>>
>>> Gary
>>>
>>> On Sun, Jan 9, 2022, 16:24 Ralph Goers <[email protected]>
>> wrote:
>>>
>>>> I’m looking at this and have a couple of concerns. The script has
>>>>
>>>>
>>>> HADOOP_ROOT_LOGGER=${HADOOP_ROOT_LOGGER:-${HADOOP_LOGLEVEL},console}
>>>>
>>>>
>> HADOOP_DAEMON_ROOT_LOGGER=${HADOOP_DAEMON_ROOT_LOGGER:-${HADOOP_LOGLEVEL},RFA}
>>>> HADOOP_SECURITY_LOGGER=${HADOOP_SECURITY_LOGGER:-INFO,NullAppender}
>>>>
>>>> So it seems you need this for more than just the root logger.
>>>>
>>>> Second, you are asking us to accept “level, appender” as the value and
>>>> under the covers split the
>>>> value and assign the level to the level attribute and the appender name
>> to
>>>> the AppenderRef.
>>>>
>>>> While this certainly can be done it seems like it would be just as easy
>> to
>>>> do the split in the script
>>>> and create two different variables for the logging configuration to pick
>>>> up. Is there a reason that
>>>> you can’t do this - for example perhaps users have hacked your scripts
>> and
>>>> now you can’t
>>>> change them without breaking them? If so, was this “supported”?
>>>>
>>>> Ralph
>>>>
>>>>
>>>>
>>>>> On Jan 9, 2022, at 8:01 AM, 张铎(Duo Zhang) <[email protected]>
>> wrote:
>>>>>
>>>>> I brought this up in the incubator mailing list, and was suggested to
>>>>> report directly to the log4j community.
>>>>>
>>>>> https://issues.apache.org/jira/browse/HADOOP-16206
>>>>>
>>>>> In hadoop we started to try migrating to log4j2 long ago, but it is not
>>>>> easy. For now, one of the most blocker issues is the lack of support of
>>>>> 'log4j.rootLogger=INFO,Console' grammar. Notice that we do not want to
>>>> stay
>>>>> on the bridge api, we want to fully migrate to log4j2 finally, so
>>>>> supporting the above grammar in log4j1 bridge is not enough.
>>>>>
>>>>> This is the main log4j configuration file for hadoop
>>>>>
>>>>>
>>>>
>> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
>>>>>
>>>>> We have this in the file
>>>>>
>>>>> # Define the root logger to the system property "hadoop.root.logger".
>>>>>> log4j.rootLogger=${hadoop.root.logger}
>>>>>
>>>>>
>>>>> So our users could simply pass -Dhadoop.root.logger to control the
>> level
>>>>> and appender.
>>>>>
>>>>>
>>>>
>> https://github.com/apache/hadoop/blob/39efbc6b6fe53abed15f5639edcbaaa2a9dda6d2/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh#L904
>>>>>
>>>>> Here we use an environment variable in our start up scripts. And I
>>>> believe
>>>>> there are lots of other hadoop deployment systems which will have their
>>>> own
>>>>> start scripts.
>>>>>
>>>>> So in general, it is just impossible for us to drop the
>>>>> -Dhadoop.root.logger way of configuring our logging system.
>>>>>
>>>>> So here I want to ask if it is possible for log4j2 to still support the
>>>>> 'log4j.rootLogger=INFO,Console' grammar. I'm not saying you must add it
>>>>> back with no change, we just need a way to configure the level and
>>>> appeners
>>>>> at once, so something like
>>>>> 'log4j2.rootLogger.levelAndAppender=INFO,Console' is also OK.
>>>>>
>>>>> Thanks.
>>>>
>>>>
>>
>>
