Robert,
Thank you for putting the effort in to debugging this. Narrowing down
the issue to a simple test case is extremely helpful.
The behaviour you describe looks odd to me. I'd expect consistent
behaviour across platforms irrespective of the case sensitivity of the
file system in use.
I'm going to use your test case to investigate this further. I'll report
back here with my findings - hopefully later today.
Mark
On 08/02/2022 03:29, Robert Turner wrote:
Okay, so I have finally narrowed it down the trivial failure case, and I
think I have an explanation as a result:
[1] works (in docker), and [2] fails (in docker) but works outside. The
difference between the two is the import directive being a wildcard
(ugly,
but historical in our app in some places we haven't yet cleaned up).
I am therefore speculating based on the Expression Language
specification
that because the class wasn't explicitly imported, it's not in the list
of
available classes for static class resolution, and thus it fails.
Combine this with MacOS and Windows not caring about filename cases, and
Linux caring, then I suspect it's just matching differently in both
cases.
Workaround/fix would be:
- to ensure we explicitly import the class (instead or in addition to
the
wildcard)
OR
- rename the attribute so it doesn't map directly to the class name.
So I think I can bring my overly-detailed thread to an end...unless my
guess at the reasoning is incorrect and someone has a better
explanation.
Thanks Rob S and Neil for having a look and providing suggestions -- in
part, it was something related to what you both said, but I believe in
different contexts than you expected.
Robert
[1]
$ cat src/main/webapp/index.jsp
<%@page contentType="text/html" pageEncoding="UTF-8"
isELIgnored="false" %>
<%@page import="com.example.rt.Failing" %>
<%
final Failing failing = null;
pageContext.setAttribute("failing", failing);
%>
<html>
<body>
<h2>Hello World!</h2>
<p>field1=${failing.field1}</p>
</body>
[2]
$ cat src/main/webapp/index.jsp
<%@page contentType="text/html" pageEncoding="UTF-8"
isELIgnored="false" %>
<%@page import="com.example.rt.*" %>
<%
final Failing failing = null;
pageContext.setAttribute("failing", failing);
%>
<html>
<body>
<h2>Hello World!</h2>
<p>field1=${failing.field1}</p>
</body>
On Mon, Feb 7, 2022 at 10:14 PM Robert Turner <rtur...@e-djuster.ca>
wrote:
So back to a divide and conquer approach I think.
I just created a trivial "example" [1] and it works as expected (i.e.
no
exception was generated) (on the same servers I was testing the
complex JAR
file) -- so possibly something else modifying the behaviour -- a JAR
on the
classpath, or something in the JSP file...
[1]
$ cat src/main/webapp/index.jsp
<%@page contentType="text/html" pageEncoding="UTF-8"
isELIgnored="false" %>
<%@page import="com.example.rt.Failing"%>
<%
final Failing failing = null;
pageContext.setAttribute("failing", failing);
%>
<html>
<body>
<h2>Hello World!</h2>
<p>field1=${failing.field1}</p>
</body>
</html>
$ cat src/main/java/com/example/rt/Failing.java
package com.example.rt;
public class Failing {
private final String field1 = "field1_value";
public String getField1() {
return field1;
}
}
On Mon, Feb 7, 2022 at 9:51 PM Robert Turner <rtur...@e-djuster.ca>
wrote:
I'm just avoiding sharing product details or things I think only
serves
to confuse the problem. Sorry if you felt I wasn't sharing. It wasn't
my
intention to be obtuse. I didn't believe they added any value for the
diagnostics (of course that assumes I know enough about the problem).
However, since you think they might be useful, here they the name
mapping
from the exception and the Java and JSP code excerpts:
"package1" -> "model"
"Class1" -> "Organization"
"class1" -> "organization"
The class is present in the package (see [1]) -- otherwise it wouldn't
work in one environment and not the other -- I believe I have
confirmed
that carefully. Full class paths (with path-names relativised to
compare
between environments) listed below [2], and the code that emitted the
listing is here [3]. I post-processed the log lines as follows [4].
Docker
file for the Tomcat container provided [5]. JDK details listed in [6].
Tomcat version is 9.0.58 in all cases.
What seems to be different is:
- the way the EL resolver is working
OR
- the behaviour of the class loader differs in the different
environments.
OR
- something else I do not understand is relevant
The working environments are MacOS and Windows, and the failing
environment is either Debian or AWS Linux 2 running in a docker
container.
If the class loaders behaviour differently, then that could explain
the
issues, however, that would surprise me if they differed in any
material
way on the different platforms.
I hope that helps provide more detail that might be useful...
Robert
[1]
$ unzip -l target/app.war | grep "model\/Organization\.class"
66246 02-07-2022 20:17
WEB-INF/classes/model/Organization.class
[2]
java.class.path=/bin/bootstrap.jar:/bin/tomcat-juli.jar
Class loader URLs:
Class loader
name=org.apache.catalina.loader.ParallelWebappClassLoader,
URL count=146
URL=file:/WEB-INF/classes/
URL=file:/WEB-INF/lib/FastInfoset-1.2.18.jar
URL=file:/WEB-INF/lib/SparseBitSet-1.2.jar
URL=file:/WEB-INF/lib/VeracodeAnnotations-1.2.1.jar
URL=file:/WEB-INF/lib/activation-1.1.jar
URL=file:/WEB-INF/lib/animal-sniffer-annotations-1.20.jar
URL=file:/WEB-INF/lib/annotations-4.1.1.4.jar
URL=file:/WEB-INF/lib/api-common-2.0.1.jar
URL=file:/WEB-INF/lib/auto-value-annotations-1.8.2.jar
URL=file:/WEB-INF/lib/avatax-rest-v2-api-java_2.11-21.12.1.jar
URL=file:/WEB-INF/lib/aws-java-sdk-core-1.12.145.jar
URL=file:/WEB-INF/lib/aws-java-sdk-kms-1.12.145.jar
URL=file:/WEB-INF/lib/aws-java-sdk-s3-1.12.145.jar
URL=file:/WEB-INF/lib/bcmail-jdk15on-1.70.jar
URL=file:/WEB-INF/lib/bcpkix-jdk15on-1.70.jar
URL=file:/WEB-INF/lib/bcprov-jdk15on-1.70.jar
URL=file:/WEB-INF/lib/bcutil-jdk15on-1.70.jar
URL=file:/WEB-INF/lib/castor-core-1.4.1.jar
URL=file:/WEB-INF/lib/castor-xml-1.4.1.jar
URL=file:/WEB-INF/lib/checker-qual-3.5.0.jar
URL=file:/WEB-INF/lib/commons-beanutils-1.9.4.jar
URL=file:/WEB-INF/lib/commons-codec-1.15.jar
URL=file:/WEB-INF/lib/commons-collections-3.2.2.jar
URL=file:/WEB-INF/lib/commons-collections4-4.4.jar
URL=file:/WEB-INF/lib/commons-compress-1.21.jar
URL=file:/WEB-INF/lib/commons-digester-2.1.jar
URL=file:/WEB-INF/lib/commons-fileupload-1.4.jar
URL=file:/WEB-INF/lib/commons-io-2.11.0.jar
URL=file:/WEB-INF/lib/commons-lang3-3.12.0.jar
URL=file:/WEB-INF/lib/commons-logging-1.2.jar
URL=file:/WEB-INF/lib/commons-math3-3.6.1.jar
URL=file:/WEB-INF/lib/commons-text-1.9.jar
URL=file:/WEB-INF/lib/conscrypt-openjdk-uber-2.5.1.jar
URL=file:/WEB-INF/lib/curvesapi-1.06.jar
URL=file:/WEB-INF/lib/ecj-3.21.0.jar
URL=file:/WEB-INF/lib/encoder-1.2.3.jar
URL=file:/WEB-INF/lib/encoder-jsp-1.2.3.jar
URL=file:/WEB-INF/lib/error_prone_annotations-2.9.0.jar
URL=file:/WEB-INF/lib/failureaccess-1.0.1.jar
URL=file:/WEB-INF/lib/fontbox-2.0.25.jar
URL=file:/WEB-INF/lib/gax-2.3.0.jar
URL=file:/WEB-INF/lib/gax-grpc-2.3.0.jar
URL=file:/WEB-INF/lib/gmbal-api-only-4.0.3.jar
URL=file:/WEB-INF/lib/google-auth-library-credentials-1.1.0.jar
URL=file:/WEB-INF/lib/google-auth-library-oauth2-http-1.1.0.jar
URL=file:/WEB-INF/lib/google-cloud-speech-1.30.5.jar
URL=file:/WEB-INF/lib/google-http-client-1.39.2.jar
URL=file:/WEB-INF/lib/google-http-client-gson-1.39.2.jar
URL=file:/WEB-INF/lib/googleauth-1.5.0.jar
URL=file:/WEB-INF/lib/grpc-alts-1.40.0.jar
URL=file:/WEB-INF/lib/grpc-api-1.40.0.jar
URL=file:/WEB-INF/lib/grpc-auth-1.40.0.jar
URL=file:/WEB-INF/lib/grpc-context-1.40.0.jar
URL=file:/WEB-INF/lib/grpc-core-1.40.0.jar
URL=file:/WEB-INF/lib/grpc-grpclb-1.40.0.jar
URL=file:/WEB-INF/lib/grpc-netty-shaded-1.40.0.jar
URL=file:/WEB-INF/lib/grpc-protobuf-1.40.0.jar
URL=file:/WEB-INF/lib/grpc-protobuf-lite-1.40.0.jar
URL=file:/WEB-INF/lib/grpc-stub-1.40.0.jar
URL=file:/WEB-INF/lib/gson-2.8.9.jar
URL=file:/WEB-INF/lib/guava-31.0.1-jre.jar
URL=file:/WEB-INF/lib/ha-api-3.1.13.jar
URL=file:/WEB-INF/lib/handlebars-4.3.0.jar
URL=file:/WEB-INF/lib/httpclient-4.5.13.jar
URL=file:/WEB-INF/lib/httpcore-4.4.15.jar
URL=file:/WEB-INF/lib/ion-java-1.0.2.jar
URL=file:/WEB-INF/lib/ipaddress-5.3.3.jar
URL=file:/WEB-INF/lib/itext-2.1.7.js9.jar
URL=file:/WEB-INF/lib/j2objc-annotations-1.3.jar
URL=file:/WEB-INF/lib/jackson-annotations-2.13.1.jar
URL=file:/WEB-INF/lib/jackson-core-2.13.1.jar
URL=file:/WEB-INF/lib/jackson-databind-2.13.1.jar
URL=file:/WEB-INF/lib/jackson-dataformat-cbor-2.13.1.jar
URL=file:/WEB-INF/lib/jai-imageio-core-1.4.0.jar
URL=file:/WEB-INF/lib/jakarta.activation-2.0.1.jar
URL=file:/WEB-INF/lib/jakarta.annotation-api-1.3.5.jar
URL=file:/WEB-INF/lib/jakarta.jws-api-2.1.0.jar
URL=file:/WEB-INF/lib/jakarta.mail-1.6.7.jar
URL=file:/WEB-INF/lib/jakarta.xml.bind-api-2.3.3.jar
URL=file:/WEB-INF/lib/jakarta.xml.soap-api-1.4.2.jar
URL=file:/WEB-INF/lib/jakarta.xml.ws-api-2.3.3.jar
URL=file:/WEB-INF/lib/jasperreports-6.18.1.jar
URL=file:/WEB-INF/lib/jasperreports-fonts-6.18.1.jar
URL=file:/WEB-INF/lib/java-saml-2.8.0.jar
URL=file:/WEB-INF/lib/java-saml-core-2.8.0.jar
URL=file:/WEB-INF/lib/javax.activation-api-1.2.0.jar
URL=file:/WEB-INF/lib/javax.annotation-api-1.3.2.jar
URL=file:/WEB-INF/lib/javax.inject-1.jar
URL=file:/WEB-INF/lib/javax.mail-1.6.2.jar
URL=file:/WEB-INF/lib/javax.xml.soap-api-1.4.0.jar
URL=file:/WEB-INF/lib/jaxb-api-2.3.1.jar
URL=file:/WEB-INF/lib/jaxb-core-2.3.0.1.jar
URL=file:/WEB-INF/lib/jaxb-impl-2.3.5.jar
URL=file:/WEB-INF/lib/jaxws-api-2.3.1.jar
URL=file:/WEB-INF/lib/jaxws-rt-2.3.5.jar
URL=file:/WEB-INF/lib/jbig2-imageio-3.0.3.jar
URL=file:/WEB-INF/lib/jcommon-1.0.24.jar
URL=file:/WEB-INF/lib/jempbox-1.8.16.jar
URL=file:/WEB-INF/lib/jettison-1.4.1.jar
URL=file:/WEB-INF/lib/jfreechart-1.0.19.jar
URL=file:/WEB-INF/lib/jmespath-java-1.12.145.jar
URL=file:/WEB-INF/lib/joda-time-2.10.6.jar
URL=file:/WEB-INF/lib/jsch-0.1.55.jar
URL=file:/WEB-INF/lib/jsoup-1.14.3.jar
URL=file:/WEB-INF/lib/jsr181-api-1.0-MR1.jar
URL=file:/WEB-INF/lib/jsr305-3.0.2.jar
URL=file:/WEB-INF/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
URL=file:/WEB-INF/lib/log4j-api-2.17.1.jar
URL=file:/WEB-INF/lib/log4j-core-2.17.1.jar
URL=file:/WEB-INF/lib/log4j-slf4j-impl-2.17.1.jar
URL=file:/WEB-INF/lib/log4j-web-2.17.1.jar
URL=file:/WEB-INF/lib/management-api-3.2.3.jar
URL=file:/WEB-INF/lib/metadata-extractor-2.16.0.jar
URL=file:/WEB-INF/lib/mimepull-1.9.15.jar
URL=file:/WEB-INF/lib/opencensus-api-0.28.0.jar
URL=file:/WEB-INF/lib/opencensus-contrib-http-util-0.28.0.jar
URL=file:/WEB-INF/lib/owasp-java-html-sanitizer-20211018.2.jar
URL=file:/WEB-INF/lib/pdfbox-2.0.25.jar
URL=file:/WEB-INF/lib/perfmark-api-0.23.0.jar
URL=file:/WEB-INF/lib/poi-4.1.2.jar
URL=file:/WEB-INF/lib/poi-ooxml-4.1.2.jar
URL=file:/WEB-INF/lib/poi-ooxml-schemas-4.1.2.jar
URL=file:/WEB-INF/lib/policy-2.7.10.jar
URL=file:/WEB-INF/lib/postgresql-42.3.1.jar
URL=file:/WEB-INF/lib/proto-google-cloud-speech-v1-1.30.5.jar
URL=file:/WEB-INF/lib/proto-google-cloud-speech-v1p1beta1-0.83.5.jar
URL=file:/WEB-INF/lib/proto-google-common-protos-2.3.2.jar
URL=file:/WEB-INF/lib/protobuf-java-3.17.3.jar
URL=file:/WEB-INF/lib/protobuf-java-util-3.17.3.jar
URL=file:/WEB-INF/lib/saaj-impl-1.5.3.jar
URL=file:/WEB-INF/lib/scala-library-2.11.12.jar
URL=file:/WEB-INF/lib/slf4j-api-1.7.33.jar
URL=file:/WEB-INF/lib/stax-api-1.0.1.jar
URL=file:/WEB-INF/lib/stax-ex-1.8.3.jar
URL=file:/WEB-INF/lib/stax2-api-4.2.1.jar
URL=file:/WEB-INF/lib/streambuffer-2.0.2.jar
URL=file:/WEB-INF/lib/taglibs-standard-impl-1.2.5.jar
URL=file:/WEB-INF/lib/taglibs-standard-spec-1.2.5.jar
URL=file:/WEB-INF/lib/threetenbp-1.3.5.jar
URL=file:/WEB-INF/lib/tika-core-2.2.1.jar
URL=file:/WEB-INF/lib/tika-parser-image-module-2.2.1.jar
URL=file:/WEB-INF/lib/tika-parser-xmp-commons-2.2.1.jar
URL=file:/WEB-INF/lib/woodstox-core-6.2.6.jar
URL=file:/WEB-INF/lib/xmlbeans-3.1.0.jar
URL=file:/WEB-INF/lib/xmlsec-2.2.3.jar
URL=file:/WEB-INF/lib/xmpcore-6.1.11.jar
Class loader name=java.net.URLClassLoader, URL count=33
URL=file:/lib/
URL=file:/lib/annotations-api.jar
URL=file:/lib/catalina-ant.jar
URL=file:/lib/catalina-ha.jar
URL=file:/lib/catalina-ssi.jar
URL=file:/lib/catalina-storeconfig.jar
URL=file:/lib/catalina-tribes.jar
URL=file:/lib/catalina.jar
URL=file:/lib/ecj-4.20.jar
URL=file:/lib/el-api.jar
URL=file:/lib/jasper-el.jar
URL=file:/lib/jasper.jar
URL=file:/lib/jaspic-api.jar
URL=file:/lib/jsp-api.jar
URL=file:/lib/servlet-api.jar
URL=file:/lib/tomcat-api.jar
URL=file:/lib/tomcat-coyote.jar
URL=file:/lib/tomcat-dbcp.jar
URL=file:/lib/tomcat-i18n-cs.jar
URL=file:/lib/tomcat-i18n-de.jar
URL=file:/lib/tomcat-i18n-es.jar
URL=file:/lib/tomcat-i18n-fr.jar
URL=file:/lib/tomcat-i18n-ja.jar
URL=file:/lib/tomcat-i18n-ko.jar
URL=file:/lib/tomcat-i18n-pt-BR.jar
URL=file:/lib/tomcat-i18n-ru.jar
URL=file:/lib/tomcat-i18n-zh-CN.jar
URL=file:/lib/tomcat-jdbc.jar
URL=file:/lib/tomcat-jni.jar
URL=file:/lib/tomcat-util-scan.jar
URL=file:/lib/tomcat-util.jar
URL=file:/lib/tomcat-websocket.jar
URL=file:/lib/websocket-api.jar
Class loader name=jdk.internal.loader.ClassLoaders$AppClassLoader
Class loader
name=jdk.internal.loader.ClassLoaders$PlatformClassLoader
[3]
private static void
logClassLoaderUrls(org.apache.logging.log4j.Level
logLevel, ClassLoader cl) {
if (null == cl) {
return;
}
boolean loaderNameLogged = false;
if (cl instanceof URLClassLoader) {
final URLClassLoader urlCl = (URLClassLoader) cl;
final URL[] urls = urlCl.getURLs();
if (null != urls) {
LOG.log(logLevel, " Class loader name={}, URL
count={}",
cl.getClass().getName(), urls.length);
loaderNameLogged = true;
for (int idx = 0; idx < urls.length; idx++) {
final URL url = urls[idx];
LOG.log(logLevel, " URL[{}]={}", idx, url);
}
}
}
if (!loaderNameLogged) {
LOG.log(logLevel, " Class loader name={}",
cl.getClass().getName());
}
logClassLoaderUrls(logLevel, cl.getParent());
}
..snip..
LOG.info("java.class.path={}",
System.getProperty("java.class.path"));
LOG.debug("Class loader URLs:");
logClassLoaderUrls(org.apache.logging.log4j.Level.DEBUG,
ctx.getClassLoader());
[4]
cat log-extract.txt | cut -c137- | sed -e
"s+/usr/local/tomcat++g" -e
"s+/usr/local/tomcat/webapps/exclaim++g" -e "s+\[[0-9]*\]++"
and I manually sorted the "sections" using vim, with the "sort"
command.
[5]
# Debian with OpenJDK
#FROM tomcat:9.0.58-jre11-openjdk AS tomcat_package
# AWS Linux 2 with Corretto
FROM tomcat:9.0.58-jdk11-corretto AS tomcat_package
# Set the timezone to Canada/Eastern
# For Debian
#RUN ln -sf /usr/share/zoneinfo/Canada/Eastern /etc/localtime && echo
"Canada/Eastern" > /etc/timezone && dpkg-reconfigure -f noninteractive
tzdata
# For AWS Linux
RUN ln -sf /usr/share/zoneinfo/Canada/Eastern /etc/localtime && sed -i
's/\"UTC\"/\"Canada\/Eastern\"/' /etc/sysconfig/clock # and reboot,
but
will skip as container will be created and ran separately
# Install a postgresql client for testing and the required font
packages
needed by JasperReports
# For Debian:
#RUN echo "deb http://deb.debian.org/debian bullseye contrib
non-free"
/etc/apt/sources.list && apt-get update && apt-get install -y
postgresql-client ttf-mscorefonts-installer
# For AWS Linux:
RUN yum -q list installed epel-release.noarch &>/dev/null && echo
'epel-release already installed' || yum install -y
https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
RUN yum install -y cabextract xorg-x11-font-utils fontconfig
RUN rpm -q msttcore-fonts-installer-2.6-1 &>/dev/null && echo
'msttcore-fonts-installer-2.6-1 already installed' || rpm -i
https://iweb.dl.sourceforge.net/project/mscorefonts2/rpms/msttcore-fonts-installer-2.6-1.noarch.rpm
# Add a health-check script to wait for exclaim to be running --
allow at
least 10 minutes for it to start
COPY health_check.sh /usr/local/bin/
HEALTHCHECK --interval=10s --timeout=5s --start-period=10m
--retries=10
CMD /usr/local/bin/health_check.sh
# Set up the resource folders needed by application and make it a
# volume so it persists across container restarts
COPY resources/ /resources/
VOLUME /resources
# Map some of the Tomcat folders to volumes so they persist between
restarts of the container
VOLUME /usr/local/tomcat/logs
VOLUME /usr/local/tomcat/webapps
VOLUME /usr/local/tomcat/work
VOLUME /usr/local/tomcat/conf
# Enable connecting from anywhere for JPDA (req'd for Java 9+)
ENV JPDA_ADDRESS=*:8000
CMD ["catalina.sh","jpda","run"]
[6]
JDK on MacOS is:
openjdk version "11.0.13" 2021-10-19 LTS
OpenJDK Runtime Environment Corretto-11.0.13.8.1 (build 11.0.13+8-LTS)
OpenJDK 64-Bit Server VM Corretto-11.0.13.8.1 (build 11.0.13+8-LTS,
mixed
mode)
JDK on Windows is:
openjdk 11.0.14 2022-01-18 LTS
OpenJDK Runtime Environment Corretto-11.0.14.9.1 (build 11.0.14+9-LTS)
OpenJDK 64-Bit Server VM Corretto-11.0.14.9.1 (build 11.0.14+9-LTS,
mixed
mode)
JDK on Docker with AWS Linux 2 is:
openjdk 11.0.14 2022-01-18 LTS
OpenJDK Runtime Environment Corretto-11.0.14.9.1 (build 11.0.14+9-LTS)
OpenJDK 64-Bit Server VM Corretto-11.0.14.9.1 (build 11.0.14+9-LTS,
mixed
mode)
On Mon, Feb 7, 2022 at 9:24 PM Rob Sargent <rsarg...@xmission.com>
wrote:
On 2/7/22 19:13, Robert Turner wrote:
So, I've gone back and double-checked as much as I can (Tomcat
version, JDK
version), and the classpath, and I have identical classpaths in both
environments (except the sort order of the URLs per "level" of
ClassLoader), and I've re-verified the behaviour:
- fails in the docker environment
- works locally
(Same WAR file on both).
I guess I'm in to one of the following approaches next:
- build a debug version of Tomcat, and step through the code that
breaks
and see if I can figure out why (not that I have time to do this of
course...but might be necessary)
- construct a trivial application reproduction, along with docker
layout,
and see if anyone else can reproduce... (assuming anyone else has
time
to
do that of course...)
Anyone got any suggestions of what to look into next?
On Mon, Feb 7, 2022 at 5:05 PM Rob Sargent <rsarg...@xmission.com>
wrote:
As you appear averse to sharing names of things, seems you will need
to
do a lot of extra clean-up before you can share a docker image. Make
sure you're clear on what NoClassDefFoundError indicates (present at
compile time, absent at runtime) and how that translates to
docker-ness
(of which I know naught).
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
