Thanks Chesnay. Will try and report back.
On Fri, Jul 30, 2021, 10:19 Chesnay Schepler <ches...@apache.org> wrote:
> Of course if is finding the file, you are actively pointing it towards it.
> The BashJavaUtils are supposed to use the log4j configuration file *that
> is bundled in the BashJavaUtils.jar, *which you are now interfering with.
> That's also why it doesn't require all of lib/ to be on the classpath; the
> flink-dist jar is enough.
>
> If you configure the log4j configuration file via the env.java.options in
> flink-conf.yaml, then it should not be picked up by the BashJavaUtils.
>
> On 30/07/2021 08:48, Yuval Itzchakov wrote:
>
> It is finding the file though, the problem is that the lib/ might not be
> on the classpath when the file is being parsed, thus the YAML file is not
> recognized as being parsable.
>
> Is there a way to differ the inference from BashJavaUtils to the actual
> bootstrap of the app? or perhaps add the lib to the utils classpath?
>
> On Thu, Jul 29, 2021, 17:55 Chesnay Schepler <ches...@apache.org> wrote:
>
>> That could be...could you try configuring "env.java.opts:
>> -Dlog4j.configurationFile=..."?
>>
>> On 29/07/2021 15:18, Yuval Itzchakov wrote:
>>
>> Perhaps because I'm passing it as an argument to the job with
>> -Dlog4j.configurationFile=... ?
>>
>> On Thu, Jul 29, 2021 at 3:27 PM Chesnay Schepler <ches...@apache.org>
>> wrote:
>>
>>> Looking once more at the stacktrace this happens in the BashJavaUtils,
>>> which only puts flink-dist and not the entire lib/ directory on the
>>> classpath.
>>> What I don't understand is why it would pick up your log4j file. It
>>> should only use the file that is embedded within BashJavaUtils.jar.
>>>
>>> On 29/07/2021 13:11, Yuval Itzchakov wrote:
>>>
>>> Hi Chesnay,
>>>
>>> So I looked into it, and jackson-databind is definitely in my UBER jar
>>> as a transitive dependency. Additionally, this uber JAR is added to the
>>> `lib/` directly for Flink before I start the job, yet it still fails.
>>> Are there any other paths I should explore?
>>>
>>> One thing to add is that I'm using the "classloader.resolve-order" =
>>> "parent-first" flag due to my lib having some deps that otherwise cause
>>> runtime collision errors in Kafka.
>>>
>>>
>>> On Wed, Jul 7, 2021 at 12:25 PM Yuval Itzchakov <yuva...@gmail.com>
>>> wrote:
>>>
>>>> Interesting, I don't have bind explicitly on the classpath, will give
>>>> it a try.
>>>>
>>>> Although locally it is working properly.
>>>>
>>>> On Wed, Jul 7, 2021, 12:19 Chesnay Schepler <ches...@apache.org> wrote:
>>>>
>>>>> According to the log4j documentation you need both jackson-databind
>>>>> and jackson-dataformat-yaml to be on the classpath.
>>>>> Did you make sure that that this is the case in 1.13.1? It could very
>>>>> well be that in 1.9 databind was on the classpath by chance.
>>>>>
>>>>> On 07/07/2021 10:22, Arvid Heise wrote:
>>>>>
>>>>> Hi Yuval,
>>>>>
>>>>> For some reason the YamlConfigurationFactory is not correctly loaded
>>>>> and the fallback XmlConfigurationFactory is used unsuccessfully.
>>>>> You could try to force DEBUG on that YamlConfigurationFactory and
>>>>> check for some output like
>>>>> "Missing dependencies for Yaml support, ConfigurationFactory {} is
>>>>> inactive"
>>>>>
>>>>> In particular, it checks for these classes
>>>>> "com.fasterxml.jackson.databind.ObjectMapper",
>>>>> "com.fasterxml.jackson.databind.JsonNode",
>>>>> "com.fasterxml.jackson.core.JsonParser",
>>>>> "com.fasterxml.jackson.dataformat.yaml.YAMLFactory"
>>>>>
>>>>> I'm assuming that with improved classloader separation, you actually
>>>>> need to put your dependency into `lib/` instead of putting it into your
>>>>> user jar. But I'm pulling in @Chesnay Schepler <ches...@apache.org>
>>>>> who has much more insights.
>>>>>
>>>>> On Sun, Jul 4, 2021 at 9:45 PM Yuval Itzchakov <yuva...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am attempting to upgrade Flink from 1.9 to 1.13.1
>>>>>> I am using a YAML based log4j file. In 1.9, it worked perfectly fine
>>>>>> by adding the following dependency to the classpath (I deploy with an
>>>>>> uber
>>>>>> JAR):
>>>>>>
>>>>>> "com.fasterxml.jackson.dataformat" % "jackson-dataformat-yaml"
>>>>>> % "2.12.3"
>>>>>>
>>>>>> However, with Flink 1.13.1 I get the following error at runtime (only
>>>>>> when running in a Cluster, running locally works fine):
>>>>>>
>>>>>>
>>>>>>
>>>>>> *flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>>> taskmanager [Fatal Error] log4j2.yaml:1:1: Content is not allowed in
>>>>>> prolog. flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>>> taskmanager ERROR StatusLogger Error parsing /opt/foo/conf/log4j2.yaml
>>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>>> taskmanager
>>>>>> org.xml.sax.SAXParseException; systemId:
>>>>>> file:///opt/foo/conf/log4j2.yaml;
>>>>>> lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.*
>>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>>> taskmanager at
>>>>>> java.xml/com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:261)
>>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>>> taskmanager at
>>>>>> java.xml/com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339)
>>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>>> taskmanager at
>>>>>> org.apache.logging.log4j.core.config.xml.XmlConfiguration.<init>(XmlConfiguration.java:92)
>>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>>> taskmanager at
>>>>>> org.apache.logging.log4j.core.config.xml.XmlConfigurationFactory.getConfiguration(XmlConfigurationFactory.java:46)
>>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>>> taskmanager at
>>>>>> org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:458)
>>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>>> taskmanager at
>>>>>> org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:385)
>>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>>> taskmanager at
>>>>>> org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:293)
>>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>>> taskmanager at
>>>>>> org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:647)
>>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>>> taskmanager at
>>>>>> org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
>>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>>> taskmanager at
>>>>>> org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
>>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>>> taskmanager at
>>>>>> org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
>>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>>> taskmanager at
>>>>>> org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
>>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>>> taskmanager at
>>>>>> org.apache.logging.log4j.LogManager.getContext(LogManager.java:155)
>>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>>> taskmanager at
>>>>>> org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
>>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>>> taskmanager at
>>>>>> org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
>>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>>> taskmanager at
>>>>>> org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
>>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>>> taskmanager at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
>>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>>> taskmanager at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
>>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>>> taskmanager at
>>>>>> org.apache.flink.runtime.util.bash.BashJavaUtils.<clinit>(BashJavaUtils.java:45)
>>>>>> flink-aamtr-a11e58230095d9390e8547eccb0290d4-27ef11e7-tm-6r8dfx
>>>>>> taskmanager ERROR StatusLogger No logging configuration
>>>>>>
>>>>>> This indicates that for some reason, the jackson dataformat YAML
>>>>>> library is not getting properly loaded from my uber JAR at runtime.
>>>>>>
>>>>>> Has anyone run into this? Any possible workarounds?
>>>>>> --
>>>>>> Best Regards,
>>>>>> Yuval Itzchakov.
>>>>>>
>>>>>
>>>>>
>>>
>>> --
>>> Best Regards,
>>> Yuval Itzchakov.
>>>
>>>
>>>
>>
>> --
>> Best Regards,
>> Yuval Itzchakov.
>>
>>
>>
>
