I think we should include OOZIE-2888 in the 4.3.1 release as well so that sharelib creation works with newer versions of hadoop. Satish is on vacation for next two weeks. Peter Cseh volunteered to take over as the release manager and do the next RC.
Regards, Rohini On Wed, Jan 31, 2018 at 10:14 AM, Satish Saley <satishsa...@apache.org> wrote: > The vote for rc-3 has been extended till 02/02/2018 11:59 PST as some devs > have asked for more time. > > On Wed, Jan 31, 2018 at 8:16 AM, Artem Ervits <artemerv...@gmail.com> > wrote: > > > @Rohini, yes Oozie-2888 fixed the issue what I meant was to replace > commons > > lib with java.nio just like the Hadoop jira he had referenced. The > > Oozie-2888 bumps commons lib to latest only, it doesn't use built-in > libs. > > > > On Jan 31, 2018 9:41 AM, "Rohini Palaniswamy" <roh...@apache.org> wrote: > > > > > >for 4.3.1 it's probably a non-issue but for 5.0 perhaps we'd like to > > > tackle > > > it > > > Attila mentioned that OOZIE-2888 already fixed the problem in 5.0. > > > > > > > > > On Wed, Jan 31, 2018 at 6:34 AM, Artem Ervits <artemerv...@gmail.com> > > > wrote: > > > > > > > @Attila, would you like me to open jira for: > > > > > > > > - HADOOP-13444 replaced org.apache.commons.io.Charsets with > > > > java.nio.charset.StandardCharsets so it shall not be a problem on > > hadoop > > > > 2.8+ > > > > > > > > for 4.3.1 it's probably a non-issue but for 5.0 perhaps we'd like to > > > tackle > > > > it? I'm happy to dedicate some cycles to it. > > > > > > > > On Tue, Jan 30, 2018 at 11:52 PM, Attila Sasvari < > > asasv...@cloudera.com> > > > > wrote: > > > > > > > > > - I could not reproduce the NoClassDefFoundError Artem reported > with > > > > hadoop > > > > > 2.6.0. I changed bin/oozie-setup.sh so that it prints loaded > classes > > > > > (passed -verbose:class to the JVM). Interestingly, > > > org/apache/commons/io/ > > > > > Charsets was not even loaded. > > > > > > > > > > However, when I tried again with hadoop.version *2.7.5* , it > resulted > > > in > > > > > the error mentioned: > > > > > > > > > > java.lang.NoClassDefFoundError: org/apache/commons/io/Charsets > > > > > at org.apache.hadoop.security.Credentials.<clinit>( > > > Credentials.java:222) > > > > > at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:334) > > > > > at org.apache.oozie.service.HadoopAccessorService.createJobConf( > > > > > HadoopAccessorService.java:349) > > > > > at org.apache.oozie.tools.OozieSharelibCLI.run( > > > > OozieSharelibCLI.java:171) > > > > > at org.apache.oozie.tools.OozieSharelibCLI.main( > > > > OozieSharelibCLI.java:67) > > > > > Caused by: java.lang.ClassNotFoundException: > > > > > org.apache.commons.io.Charsets > > > > > > > > > > It looks like https://issues.apache.org/jira/browse/OOZIE-2396 > > > > > > > > > > Some more info: > > > > > - ./lib/commons-io-2.1.jar does not contain org/apache/commons/io/ > > > > > Charsets. > > > > > > > > > > - related hadoop 2.7.0 jira: > > > > > https://issues.apache.org/jira/browse/HADOOP-11389 > > > > > - related hadoop code: > > > > > https://github.com/apache/hadoop/blame/branch-2.7.0/ > > > > > hadoop-common-project/hadoop-common/src/main/java/org/ > > > > > apache/hadoop/security/Credentials.java#L36 > > > > > - HADOOP-13444 replaced org.apache.commons.io.Charsets with > > > > > java.nio.charset.StandardCharsets so it shall not be a problem on > > > hadoop > > > > > 2.8+ > > > > > > > > > > Note: Oozie 5.0.0-beta1 includes OOZIE-2888 that add updated > > commons-io > > > > to > > > > > 2.4. That is why it is not a problem there. > > > > > > > > > > On Wed, Jan 31, 2018 at 1:18 AM, Satish Saley < > > satishsa...@apache.org> > > > > > wrote: > > > > > > > > > > > 1. However, json2.js is referenced by webapp/index.html and it > > > > > > is not found (not a blocker in my opinion). > > > > > > > > > > > > -> It will show only in firebug/developer tools. Not on UI. > > > > > > > > > > > > 2. [WARNING] Unable to process class com/sun/tools/jdi/ > > > > > ClassTypeImpl.class > > > > > > in > > > > > > JarAnalyzer File > > > > > > > > > > > > -> We can ignore the warning as build is going through > > > > > > > > > > > > 3. /user/oozie/share/lib/lib_20180130164518 > > > > > > Exception in thread "main" java.lang.NoClassDefFoundError: > > > > > > > > > > > > I am not able to reproduce this. > > > > > > > > > > > > saley@easyfamily-lm on Tue Jan 30 at 11:44 AM > > > > > > ~/src/oozie-4.3.1/distro/target/oozie-4.3.1-distro/oozie-4.3.1 > > > > > > $ bin/oozie-setup.sh sharelib create -fs hdfs://localhost > > > -concurrency > > > > 8 > > > > > > -locallib oozie-sharelib-4.3.1.tar.gz > > > > > > setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m" > > > > > > log4j:WARN No appenders could be found for logger > > > > > > (org.apache.hadoop.util.Shell). > > > > > > log4j:WARN Please initialize the log4j system properly. > > > > > > log4j:WARN See http://logging.apache.org/ > > log4j/1.2/faq.html#noconfig > > > > for > > > > > > more info. > > > > > > SLF4J: Class path contains multiple SLF4J bindings. > > > > > > SLF4J: Found binding in > > > > > > [jar:file:/Users/saley/src/oozie-4.3.1/distro/target/ > > > > > > oozie-4.3.1-distro/oozie-4.3.1/lib/slf4j-log4j12-1.6.6.jar! > > > > > > /org/slf4j/impl/StaticLoggerBinder.class] > > > > > > SLF4J: Found binding in > > > > > > [jar:file:/Users/saley/src/oozie-4.3.1/distro/target/ > > > > > > oozie-4.3.1-distro/oozie-4.3.1/lib/slf4j-simple-1.6.6.jar!/ > > > > > org/slf4j/impl/ > > > > > > StaticLoggerBinder.class] > > > > > > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for > > an > > > > > > explanation. > > > > > > SLF4J: Actual binding is of type [org.slf4j.impl. > > Log4jLoggerFactory] > > > > > > the destination path for sharelib is: > > > > > > /user/saley/share/lib/lib_20180130114509 > > > > > > Running 410 copy tasks on 8 threads > > > > > > > > > > > > > > > > > > saley@easyfamily-lm on Tue Jan 30 at 11:45 AM > > > > > > ~/src/oozie-4.3.1/distro/target/oozie-4.3.1-distro/oozie-4.3.1 > > > > > > $ hdfs dfs -ls > > > > > > SLF4J: Class path contains multiple SLF4J bindings. > > > > > > SLF4J: Found binding in > > > > > > [jar:file:/Users/saley/hadoop-2.6.3/share/hadoop/common/lib/ > > > > > > slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder. > class] > > > > > > SLF4J: Found binding in > > > > > > [jar:file:/Users/saley/src/tez/tez-dist/target/tez-0.8.5- > > > > > > minimal/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/ > > > > > > StaticLoggerBinder.class] > > > > > > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for > > an > > > > > > explanation. > > > > > > SLF4J: Actual binding is of type [org.slf4j.impl. > > Log4jLoggerFactory] > > > > > > 18/01/30 11:45:27 WARN util.NativeCodeLoader: Unable to load > > > > > native-hadoop > > > > > > library for your platform... using builtin-java classes where > > > > applicable > > > > > > Found 1 items > > > > > > drwxr-xr-x - saley supergroup 0 2018-01-30 11:45 share > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jan 30, 2018 at 8:52 AM, Artem Ervits < > > artemerv...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > yes I realized that after hitting send, I encountered the > > following > > > > > > issues > > > > > > > but it may just be me > > > > > > > > > > > > > > hitting https://issues.apache.org/jira/browse/BCEL-173 when > > > > compiling > > > > > > with > > > > > > > > > > > > > > bin/mkdistro.sh -Phadoop-2 -DskipTests -Puber > > > > -DtargetJavaVersion=1.8 > > > > > > > -DjavaVersion=1.8 -Dhadoop.version=2.7.5 > > > > > > > > > > > > > > [WARNING] Unable to process class com/sun/tools/jdi/ > > > > > ClassTypeImpl.class > > > > > > in > > > > > > > JarAnalyzer File > > > > > > > /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4. > > > > > > > x86_64/jre/../lib/tools.jar > > > > > > > org.apache.bcel.classfile.ClassFormatException: Invalid byte > tag > > > in > > > > > > > constant pool: 18 > > > > > > > at org.apache.bcel.classfile.Constant.readConstant( > > > > Constant.java:146) > > > > > > > at org.apache.bcel.classfile.ConstantPool.<init>( > > > > ConstantPool.java:67) > > > > > > > at > > > > > > > org.apache.bcel.classfile.ClassParser.readConstantPool( > > > > > > > ClassParser.java:222) > > > > > > > at org.apache.bcel.classfile.ClassParser.parse(ClassParser. > > > java:136) > > > > > > > at > > > > > > > org.apache.maven.shared.jar.classes.JarClassesAnalysis. > > > > > > > analyze(JarClassesAnalysis.java:92) > > > > > > > at > > > > > > > org.apache.maven.report.projectinfo.dependencies.Dependencies. > > > > > > > getJarDependencyDetails(Dependencies.java:276) > > > > > > > at > > > > > > > org.apache.maven.report.projectinfo.dependencies. > > > > > > > renderer.DependenciesRenderer.hasSealed( > > DependenciesRenderer.java: > > > > > 1374) > > > > > > > at > > > > > > > org.apache.maven.report.projectinfo.dependencies. > > > > > > > renderer.DependenciesRenderer.renderSectionDependencyFileDet > > > > > > > ails(DependenciesRenderer.java:544) > > > > > > > at > > > > > > > org.apache.maven.report.projectinfo.dependencies. > > > > > > > renderer.DependenciesRenderer.renderBody( > > > > > DependenciesRenderer.java:271) > > > > > > > at > > > > > > > org.apache.maven.reporting.AbstractMavenReportRenderer.render( > > > > > > > AbstractMavenReportRenderer.java:79) > > > > > > > at > > > > > > > org.apache.maven.report.projectinfo.DependenciesReport. > > > > executeReport( > > > > > > > DependenciesReport.java:206) > > > > > > > at > > > > > > > org.apache.maven.reporting.AbstractMavenReport.generate( > > > > > > > AbstractMavenReport.java:190) > > > > > > > at > > > > > > > org.apache.maven.report.projectinfo.AbstractProjectInfoReport. > > > > execute( > > > > > > > AbstractProjectInfoReport.java:202) > > > > > > > at > > > > > > > org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo( > > > > > > > DefaultBuildPluginManager.java:101) > > > > > > > at > > > > > > > org.apache.maven.lifecycle.internal.MojoExecutor.execute( > > > > > > > MojoExecutor.java:209) > > > > > > > at > > > > > > > org.apache.maven.lifecycle.internal.MojoExecutor.execute( > > > > > > > MojoExecutor.java:153) > > > > > > > at > > > > > > > org.apache.maven.lifecycle.internal.MojoExecutor.execute( > > > > > > > MojoExecutor.java:145) > > > > > > > at > > > > > > > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder. > > > > > buildProject( > > > > > > > LifecycleModuleBuilder.java:84) > > > > > > > at > > > > > > > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder. > > > > > buildProject( > > > > > > > LifecycleModuleBuilder.java:59) > > > > > > > at > > > > > > > org.apache.maven.lifecycle.internal.LifecycleStarter. > > > > > > singleThreadedBuild( > > > > > > > LifecycleStarter.java:183) > > > > > > > at > > > > > > > org.apache.maven.lifecycle.internal.LifecycleStarter. > > > > > > > execute(LifecycleStarter.java:161) > > > > > > > at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java: > > 320) > > > > > > > at org.apache.maven.DefaultMaven. > execute(DefaultMaven.java:156) > > > > > > > at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537) > > > > > > > at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196) > > > > > > > at org.apache.maven.cli.MavenCli.main(MavenCli.java:141) > > > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > > > > at > > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke( > > > > > > NativeMethodAccessorImpl.java: > > > > > > > 62) > > > > > > > at > > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke( > > > > > > > DelegatingMethodAccessorImpl.java:43) > > > > > > > at java.lang.reflect.Method.invoke(Method.java:498) > > > > > > > at > > > > > > > org.codehaus.plexus.classworlds.launcher.Launcher. > > > > > > > launchEnhanced(Launcher.java:290) > > > > > > > at > > > > > > > org.codehaus.plexus.classworlds.launcher.Launcher. > > > > > > > launch(Launcher.java:230) > > > > > > > at > > > > > > > org.codehaus.plexus.classworlds.launcher.Launcher. > > > > > > > mainWithExitCode(Launcher.java:414) > > > > > > > at org.codehaus.plexus.classworlds.launcher.Launcher. > > > > > > > main(Launcher.java:357) > > > > > > > > > > > > > > Then after a successful build getting the following version > > string > > > > > > > > > > > > > > Oozie distro created, DATE[2018.01.30-16:22:57GMT] > > > > VC-REV[unavailable], > > > > > > > available at [/tmp/oozie-4.3.1/distro/target] > > > > > > > > > > > > > > we can now pull VC-REV with https://issues.apache.org/ > > > > > > > jira/browse/OOZIE-3152 > > > > > > > > > > > > > > finally saw the following > > > > > > > > > > > > > > [oozie@hadoop oozie-4.3.1]$ bin/oozie-setup.sh sharelib create > > -fs > > > > > > > hdfs://localhost:9000 -concurrency 8 -locallib > > > > > > oozie-sharelib-4.3.1.tar.gz > > > > > > > setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m" > > > > > > > log4j:WARN No appenders could be found for logger > > > > > > > (org.apache.hadoop.util.Shell). > > > > > > > log4j:WARN Please initialize the log4j system properly. > > > > > > > log4j:WARN See http://logging.apache.org/ > > > log4j/1.2/faq.html#noconfig > > > > > for > > > > > > > more info. > > > > > > > SLF4J: Class path contains multiple SLF4J bindings. > > > > > > > SLF4J: Found binding in > > > > > > > [jar:file:/opt/oozie/oozie-4.3.1/distro/target/oozie-4.3.1- > > > > > > > distro/oozie-4.3.1/lib/slf4j-log4j12-1.6.6.jar!/org/slf4j/ > > > > > > > impl/StaticLoggerBinder.class] > > > > > > > SLF4J: Found binding in > > > > > > > [jar:file:/opt/oozie/oozie-4.3.1/distro/target/oozie-4.3.1- > > > > > > > distro/oozie-4.3.1/lib/slf4j-simple-1.6.6.jar!/org/slf4j/ > > > > > > > impl/StaticLoggerBinder.class] > > > > > > > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings > for > > > an > > > > > > > explanation. > > > > > > > SLF4J: Actual binding is of type [org.slf4j.impl. > > > Log4jLoggerFactory] > > > > > > > the destination path for sharelib is: > > > > > > > /user/oozie/share/lib/lib_20180130164518 > > > > > > > Exception in thread "main" java.lang.NoClassDefFoundError: > > > > > > > org/apache/commons/io/Charsets > > > > > > > at org.apache.hadoop.security.Credentials.<clinit>( > > > > > Credentials.java:222) > > > > > > > at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:334) > > > > > > > at > > > > > > > org.apache.oozie.service.HadoopAccessorService.createJobConf( > > > > > > > HadoopAccessorService.java:349) > > > > > > > at org.apache.oozie.tools.OozieSharelibCLI.run( > > > > > > OozieSharelibCLI.java:171) > > > > > > > at org.apache.oozie.tools.OozieSharelibCLI.main( > > > > > > OozieSharelibCLI.java:67) > > > > > > > Caused by: java.lang.ClassNotFoundException: > > > > > > > org.apache.commons.io.Charsets > > > > > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > > > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > > > > > > > at sun.misc.Launcher$AppClassLoader.loadClass( > Launcher.java:338) > > > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > > > > > > > ... 5 more > > > > > > > > > > > > > > none were blockers as I was able to get the following output > > > > > > > > > > > > > > $ curl http://localhost:11000/oozie/v2/admin/status > > > > > > > {"systemMode":"NORMAL"} > > > > > > > > > > > > > > > > > > > > > On Tue, Jan 30, 2018 at 10:45 AM, Rohini Palaniswamy < > > > > > roh...@apache.org> > > > > > > > wrote: > > > > > > > > > > > > > > > > the server directory is missing altogether now. > > > > > > > > That is as expected. It is supposed to be only in 5.0 > > > > > > > > > > > > > > > > On Mon, Jan 29, 2018 at 12:47 PM, Artem Ervits < > > > > > artemerv...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > the server directory is missing altogether now. > > > > > > > > > > > > > > > > > > On Mon, Jan 29, 2018 at 6:59 AM, Attila Sasvari < > > > > > > asasv...@cloudera.com > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > +1 (non-binding) > > > > > > > > > > > > > > > > > > > > - Verified hash, signature files, rat report > > > > > > > > > > - Built Oozie distro with hadoop.version set to 2.6.0 > using > > > the > > > > > > uber, > > > > > > > > > > hadoop-2, spark-1 profiles > > > > > > > > > > - Submitted example workflows/coordinators/bundles > (except > > > hive > > > > > > > related > > > > > > > > > > ones) on a hadoop 2.6.0 in pseudo distributed mode > > > > > > > > > > - Verified Tomcat version was 6.0.53 > > > > > > > > > > - Clicked through the web UI, log retrieval worked, job > > DAG-s > > > > > were > > > > > > > > > > rendered, version was correct. > > > > > > > > > > - Verified instrumentation, SLA pages; they are now HTML > > > based > > > > as > > > > > > > > > expected > > > > > > > > > > (OOZIE-2533). However, json2.js is referenced by > > > > > webapp/index.html > > > > > > > and > > > > > > > > it > > > > > > > > > > is not found (not a blocker in my opinion). > > > > > > > > > > - Enabled SLA, re-submitted sla coordinator example, and > > > > > retrieved > > > > > > > SLA > > > > > > > > > > details for the "one-op-wf" application, graph was > > displayed > > > > > > > properly. > > > > > > > > > > > > > > > > > > > > I used JDK 1.8.0_121 > > > > > > > > > > > > > > > > > > > > On Sun, Jan 28, 2018 at 7:09 AM, Satish Saley < > > > > > > > satishsa...@apache.org> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > I have created a build for Oozie 4.3.1, candidate 3. > > > > > > > > > > > > > > > > > > > > > > It includes all the changes decided earlier except > > > > OOZIE-3018, > > > > > > > > > OOZIE-2771 > > > > > > > > > > > and > > > > > > > > > > > OOZIE-3072. OOZIE-3018 makes changes to > > > SchemaCheckerService > > > > > > which > > > > > > > we > > > > > > > > > are > > > > > > > > > > > not picking. > > > > > > > > > > > OOZIE-2771 makes changes to file created in OOZIE-2666. > > > > > > > > > > > OOZIE-3072 makes changes to files created in > OOZIE-1770. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Keys to verify the signature of the release artifact > are > > > > > > available > > > > > > > at > > > > > > > > > > > http://www.apache.org/dist/oozie/KEYS > > > > > > > > > > > > > > > > > > > > > > Please download, test, and try it out: > > > > > > > > > > > > > > > > > > > > > > http://people.apache.org/~satishsaley/oozie-4.3.1-rc3/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The release, md5 signature, gpg signature, and rat > report > > > can > > > > > be > > > > > > > > found > > > > > > > > > at > > > > > > > > > > > the above address. > > > > > > > > > > > > > > > > > > > > > > Vote closes on 01/31/2018 11:59 PM PT. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > -- > > > > > > > > > > Attila Sasvari > > > > > > > > > > Software Engineer > > > > > > > > > > <http://www.cloudera.com/> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > -- > > > > > Attila Sasvari > > > > > Software Engineer > > > > > <http://www.cloudera.com/> > > > > > > > > > > > > > > >
