Hello,
Please help me upgrade to 1.10 in AWS EMR.
On Fri, May 1, 2020 at 4:05 PM aj <ajainje...@gmail.com> wrote:
> Hi Yang,
>
> I am attaching the logs for your reference, please help me what i am doing
> wrong.
>
> Thanks,
> Anuj
>
> On Wed, Apr 29, 2020 at 9:06 AM Yang Wang <danrtsey...@gmail.com> wrote:
>
>> Hi Anuj,
>>
>> I think the exception you come across still because the hadoop version
>> is 2.4.1. I have checked the hadoop code, the code line are exactly same.
>> For 2.8.1, i also have checked the ruleParse. It could work.
>>
>> /**
>> * A pattern for parsing a auth_to_local rule.
>> */
>> private static final Pattern ruleParser =
>>
>> Pattern.compile("\\s*((DEFAULT)|(RULE:\\[(\\d*):([^\\]]*)](\\(([^)]*)\\))?"+
>> "(s/([^/]*)/([^/]*)/(g)?)?))/?(L)?");
>>
>>
>> Could you share the jobmanager logs so that i could check the classpath
>> and hadoop version?
>>
>> Best,
>> Yang
>>
>> aj <ajainje...@gmail.com> 于2020年4月28日周二 上午1:01写道：
>>
>>> Hello Yang,
>>> My Hadoop version is Hadoop 3.2.1-amzn-0
>>> and I have put in flink/lib. flink-shaded-hadoop-2-uber-2.8.3-10.0.jar
>>>
>>> then I am getting below error :
>>>
>>> SLF4J: Class path contains multiple SLF4J bindings.
>>> SLF4J: Found binding in
>>> [jar:file:/mnt/yarn/usercache/hadoop/appcache/application_1587983834922_0002/filecache/10/slf4j-log4j12-1.7.15.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>> SLF4J: Found binding in
>>> [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>>> explanation.
>>> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>>> Exception in thread "main" java.lang.IllegalArgumentException: Invalid
>>> rule: /L
>>> RULE:[2:$1@$0](.*@)s/@.*///L
>>> DEFAULT
>>> at
>>> org.apache.hadoop.security.authentication.util.KerberosName.parseRules(KerberosName.java:321)
>>> at
>>> org.apache.hadoop.security.authentication.util.KerberosName.setRules(KerberosName.java:386)
>>> at
>>> org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:75)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:247)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:232)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:718)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:703)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:605)
>>> at
>>> org.apache.flink.yarn.entrypoint.YarnEntrypointUtils.logYarnEnvironmentInformation(YarnEntrypointUtils.java:136)
>>> at
>>> org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint.main(YarnJobClusterEntrypoint.java:109)
>>>
>>>
>>> if I remove the flink-shaded-hadoop-2-uber-2.8.3-10.0.jar from lib
>>> then i get below error:
>>>
>>> 2020-04-27 16:59:37,293 INFO org.apache.flink.client.cli.CliFrontend
>>> - Classpath:
>>> /usr/lib/flink/lib/flink-table-blink_2.11-1.10.0.jar:/usr/lib/flink/lib/flink-table_2.11-1.10.0.jar:/usr/lib/flink/lib/log4j-1.2.17.jar:/usr/lib/flink/lib/slf4j-log4j12-1.7.15.jar:/usr/lib/flink/lib/flink-dist_2.11-1.10.0.jar::/etc/hadoop/conf:/etc/hadoop/conf
>>> 2020-04-27 16:59:37,293 INFO org.apache.flink.client.cli.CliFrontend
>>> -
>>> --------------------------------------------------------------------------------
>>> 2020-04-27 16:59:37,300 INFO
>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>> configuration property: jobmanager.heap.size, 1024m
>>> 2020-04-27 16:59:37,300 INFO
>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>> configuration property: taskmanager.memory.process.size, 1568m
>>> 2020-04-27 16:59:37,300 INFO
>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>> configuration property: taskmanager.numberOfTaskSlots, 1
>>> 2020-04-27 16:59:37,300 INFO
>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>> configuration property: parallelism.default, 1
>>> 2020-04-27 16:59:37,300 INFO
>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>> configuration property: env.yarn.conf.dir, /etc/hadoop/conf
>>> 2020-04-27 16:59:37,300 INFO
>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>> configuration property: env.hadoop.conf.dir, /etc/hadoop/conf
>>> 2020-04-27 16:59:37,301 INFO
>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>> configuration property: jobmanager.execution.failover-strategy, region
>>> 2020-04-27 16:59:37,301 INFO
>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>> configuration property: classloader.resolve-order, parent-first
>>> 2020-04-27 16:59:37,301 INFO
>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>> configuration property: s3.access-key, AKIA52DD5QA5FC7HPKXG
>>> 2020-04-27 16:59:37,301 INFO
>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>> configuration property: s3.secret-key, ******
>>> 2020-04-27 16:59:37,305 WARN org.apache.flink.client.cli.CliFrontend
>>> - Could not load CLI class
>>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.
>>> java.lang.NoClassDefFoundError:
>>> org/apache/hadoop/yarn/exceptions/YarnException
>>> at java.lang.Class.forName0(Native Method)
>>> at java.lang.Class.forName(Class.java:264)
>>> at
>>> org.apache.flink.client.cli.CliFrontend.loadCustomCommandLine(CliFrontend.java:1076)
>>> at
>>> org.apache.flink.client.cli.CliFrontend.loadCustomCommandLines(CliFrontend.java:1030)
>>> at
>>> org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:957)
>>> Caused by: java.lang.ClassNotFoundException:
>>> org.apache.hadoop.yarn.exceptions.YarnException
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
>>> ... 5 more
>>> 2020-04-27 16:59:37,406 INFO org.apache.flink.core.fs.FileSystem
>>> - Hadoop is not in the classpath/dependencies. The
>>> extended set of supported File Systems via Hadoop is not available.
>>> 2020-04-27 16:59:37,458 INFO
>>> org.apache.flink.runtime.security.modules.HadoopModuleFactory - Cannot
>>> create Hadoop Security Module because Hadoop cannot be found in the
>>> Classpath.
>>> 2020-04-27 16:59:37,476 INFO
>>> org.apache.flink.runtime.security.modules.JaasModule - Jaas file
>>> will be created as /tmp/jaas-7054453135321774613.conf.
>>> 2020-04-27 16:59:37,480 INFO
>>> org.apache.flink.runtime.security.SecurityUtils - Cannot
>>> install HadoopSecurityContext because Hadoop cannot be found in the
>>> Classpath.
>>> 2020-04-27 16:59:37,481 INFO org.apache.flink.client.cli.CliFrontend
>>> - Running 'run' command.
>>> 2020-04-27 16:59:37,488 INFO org.apache.flink.client.cli.CliFrontend
>>> - Building program from JAR file
>>> 2020-04-27 16:59:37,488 ERROR org.apache.flink.client.cli.CliFrontend
>>> - Invalid command line arguments.
>>> org.apache.flink.client.cli.CliArgsException: Could not build the
>>> program from JAR file.
>>> at
>>> org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:203)
>>> at
>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:895)
>>> at
>>> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:968)
>>> at
>>> org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
>>> at
>>> org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:968)
>>> Caused by: java.io.FileNotFoundException: JAR file does not exist: -ynm
>>> at
>>> org.apache.flink.client.cli.CliFrontend.getJarFile(CliFrontend.java:719)
>>> at
>>> org.apache.flink.client.cli.CliFrontend.buildProgram(CliFrontend.java:695)
>>> at
>>> org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:200)
>>> ... 4 more
>>>
>>>
>>> Can you please help.
>>>
>>> Thanks,
>>> Anuj
>>>
>>>
>>> On Mon, Apr 13, 2020 at 7:43 AM Yang Wang <danrtsey...@gmail.com> wrote:
>>>
>>>> Hi Anuj,
>>>>
>>>> It seems that you are using hadoop version 2.4.1. I think "L" could not
>>>> be supported in
>>>> this version. Could you upgrade your hadoop version to 2.8 and have a
>>>> try? If your
>>>> YARN cluster version is 2.8+, then you could directly remove the
>>>> flink-shaded-hadoop
>>>> in your lib directory. Otherwise, you need to download the
>>>> flink-shaded-hadoop with
>>>> version 2.8 here[1].
>>>>
>>>>
>>>> [1]. https://flink.apache.org/downloads.html#additional-components
>>>>
>>>> Best,
>>>> Yang
>>>>
>>>> aj <ajainje...@gmail.com> 于2020年4月11日周六 上午4:21写道：
>>>>
>>>>> Hi Robert,
>>>>> attached the full application log file.
>>>>>
>>>>> Thanks,
>>>>> Anuj
>>>>> <http://www.cse.iitm.ac.in/%7Eanujjain/>
>>>>>
>>>>
>>>
>>> --
>>> Thanks & Regards,
>>> Anuj Jain
>>> Mob. : +91- 8588817877
>>> Skype : anuj.jain07
>>> <http://www.oracle.com/>
>>>
>>>
>>> <http://www.cse.iitm.ac.in/%7Eanujjain/>
>>>
>>
>
> --
> Thanks & Regards,
> Anuj Jain
> Mob. : +91- 8588817877
> Skype : anuj.jain07
> <http://www.oracle.com/>
>
>
> <http://www.cse.iitm.ac.in/%7Eanujjain/>
>
--
Thanks & Regards,
Anuj Jain
Mob. : +91- 8588817877
Skype : anuj.jain07
<http://www.oracle.com/>
<http://www.cse.iitm.ac.in/%7Eanujjain/>
