[jira] [Updated] (OOZIE-2651) Set javax.xml.parsers.DocumentBuilderFactory sys prop to make XML handling faster
[
https://issues.apache.org/jira/browse/OOZIE-2651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated OOZIE-2651:
-
Attachment: OOZIE-2651.005.patch
The 005 patch:
- Adds unit tests
> Set javax.xml.parsers.DocumentBuilderFactory sys prop to make XML handling
> faster
> -
>
> Key: OOZIE-2651
> URL: https://issues.apache.org/jira/browse/OOZIE-2651
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: OOZIE-2651.001.patch, OOZIE-2651.002.patch,
> OOZIE-2651.003.patch, OOZIE-2651.004.patch, OOZIE-2651.005.patch
>
>
> We noticed from some jstacks that Oozie was spending a lot of time looking
> for XML related jars (we do a lot of XML parsing in Oozie and Hadoop code).
> For instance, we'd see hundreds of threads blocked here
> {noformat}
> "pool-5-thread-57" prio=10 tid=0x7f9579923000 nid=0x6b7f waiting for
> monitor entry [0x7f94225e3000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
> - waiting to lock <0x000500026148> (a
> org.apache.catalina.loader.WebappClassLoader)
> at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
> at
> javax.xml.parsers.FactoryFinder.getProviderClass(FactoryFinder.java:117)
> at javax.xml.parsers.FactoryFinder.newInstance(FactoryFinder.java:178)
> at
> javax.xml.parsers.FactoryFinder.findJarServiceProvider(FactoryFinder.java:333)
> at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:255)
> at
> javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:121)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2526)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409)
> - locked <0x0007eac5ea28> (a org.apache.hadoop.conf.Configuration)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1144)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1116)
> at org.apache.oozie.util.XConfiguration.copy(XConfiguration.java:206)
> ...
> {noformat}
> There's nothing we can do about the lock at the {{WebappClassLoader}}, but by
> setting the {{javax.xml.parsers.DocumentBuilderFactory}} system property to
> {{org.apache.xerces.jaxp.DocumentBuilderFactoryImpl}}, we can speed this up a
> lot by getting rid of all of the searching that it's doing to find the
> implementation.
> See
> https://docs.oracle.com/javase/7/docs/api/javax/xml/parsers/DocumentBuilderFactory.html#newInstance()
>
> and
> https://www.databasesandlife.com/java-always-explicitly-specify-which-xml-parser-to-use/
> In our testing, we had a metric around creating a new {{JobClient}}, which
> also calls {{DocumentBuilderFactory.newInstance}}, and it dropped from ~3min
> max to ~2sec max (not an exact test of the savings here). And we also didn't
> see so much blocking on this in the jstacks anymore.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-2651) Set javax.xml.parsers.DocumentBuilderFactory sys prop to make XML handling faster
[
https://issues.apache.org/jira/browse/OOZIE-2651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated OOZIE-2651:
-
Attachment: OOZIE-2651.004.patch
The 004 patch removes the tab.
> Set javax.xml.parsers.DocumentBuilderFactory sys prop to make XML handling
> faster
> -
>
> Key: OOZIE-2651
> URL: https://issues.apache.org/jira/browse/OOZIE-2651
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: OOZIE-2651.001.patch, OOZIE-2651.002.patch,
> OOZIE-2651.003.patch, OOZIE-2651.004.patch
>
>
> We noticed from some jstacks that Oozie was spending a lot of time looking
> for XML related jars (we do a lot of XML parsing in Oozie and Hadoop code).
> For instance, we'd see hundreds of threads blocked here
> {noformat}
> "pool-5-thread-57" prio=10 tid=0x7f9579923000 nid=0x6b7f waiting for
> monitor entry [0x7f94225e3000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
> - waiting to lock <0x000500026148> (a
> org.apache.catalina.loader.WebappClassLoader)
> at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
> at
> javax.xml.parsers.FactoryFinder.getProviderClass(FactoryFinder.java:117)
> at javax.xml.parsers.FactoryFinder.newInstance(FactoryFinder.java:178)
> at
> javax.xml.parsers.FactoryFinder.findJarServiceProvider(FactoryFinder.java:333)
> at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:255)
> at
> javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:121)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2526)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409)
> - locked <0x0007eac5ea28> (a org.apache.hadoop.conf.Configuration)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1144)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1116)
> at org.apache.oozie.util.XConfiguration.copy(XConfiguration.java:206)
> ...
> {noformat}
> There's nothing we can do about the lock at the {{WebappClassLoader}}, but by
> setting the {{javax.xml.parsers.DocumentBuilderFactory}} system property to
> {{org.apache.xerces.jaxp.DocumentBuilderFactoryImpl}}, we can speed this up a
> lot by getting rid of all of the searching that it's doing to find the
> implementation.
> See
> https://docs.oracle.com/javase/7/docs/api/javax/xml/parsers/DocumentBuilderFactory.html#newInstance()
>
> and
> https://www.databasesandlife.com/java-always-explicitly-specify-which-xml-parser-to-use/
> In our testing, we had a metric around creating a new {{JobClient}}, which
> also calls {{DocumentBuilderFactory.newInstance}}, and it dropped from ~3min
> max to ~2sec max (not an exact test of the savings here). And we also didn't
> see so much blocking on this in the jstacks anymore.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OOZIE-2651) Set javax.xml.parsers.DocumentBuilderFactory sys prop to make XML handling faster
[
https://issues.apache.org/jira/browse/OOZIE-2651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated OOZIE-2651:
-
Attachment: OOZIE-2651.003.patch
Rebased the patch again (003) because it didn't apply anymore due to
oozie-default.xml changes. I also fixed the grammar in the config description.
[~gezapeti], [~pbacsko] can one of you review?
> Set javax.xml.parsers.DocumentBuilderFactory sys prop to make XML handling
> faster
> -
>
> Key: OOZIE-2651
> URL: https://issues.apache.org/jira/browse/OOZIE-2651
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: OOZIE-2651.001.patch, OOZIE-2651.002.patch,
> OOZIE-2651.003.patch
>
>
> We noticed from some jstacks that Oozie was spending a lot of time looking
> for XML related jars (we do a lot of XML parsing in Oozie and Hadoop code).
> For instance, we'd see hundreds of threads blocked here
> {noformat}
> "pool-5-thread-57" prio=10 tid=0x7f9579923000 nid=0x6b7f waiting for
> monitor entry [0x7f94225e3000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
> - waiting to lock <0x000500026148> (a
> org.apache.catalina.loader.WebappClassLoader)
> at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
> at
> javax.xml.parsers.FactoryFinder.getProviderClass(FactoryFinder.java:117)
> at javax.xml.parsers.FactoryFinder.newInstance(FactoryFinder.java:178)
> at
> javax.xml.parsers.FactoryFinder.findJarServiceProvider(FactoryFinder.java:333)
> at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:255)
> at
> javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:121)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2526)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409)
> - locked <0x0007eac5ea28> (a org.apache.hadoop.conf.Configuration)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1144)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1116)
> at org.apache.oozie.util.XConfiguration.copy(XConfiguration.java:206)
> ...
> {noformat}
> There's nothing we can do about the lock at the {{WebappClassLoader}}, but by
> setting the {{javax.xml.parsers.DocumentBuilderFactory}} system property to
> {{org.apache.xerces.jaxp.DocumentBuilderFactoryImpl}}, we can speed this up a
> lot by getting rid of all of the searching that it's doing to find the
> implementation.
> See
> https://docs.oracle.com/javase/7/docs/api/javax/xml/parsers/DocumentBuilderFactory.html#newInstance()
>
> and
> https://www.databasesandlife.com/java-always-explicitly-specify-which-xml-parser-to-use/
> In our testing, we had a metric around creating a new {{JobClient}}, which
> also calls {{DocumentBuilderFactory.newInstance}}, and it dropped from ~3min
> max to ~2sec max (not an exact test of the savings here). And we also didn't
> see so much blocking on this in the jstacks anymore.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OOZIE-2651) Set javax.xml.parsers.DocumentBuilderFactory sys prop to make XML handling faster
[
https://issues.apache.org/jira/browse/OOZIE-2651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated OOZIE-2651:
-
Attachment: OOZIE-2651.002.patch
I've just verified that this doesn't seem to be a problem for the Web UI with
Jetty. I'm uploading a 002 patch that's rebased on the latest master, but is
otherwise the same.
> Set javax.xml.parsers.DocumentBuilderFactory sys prop to make XML handling
> faster
> -
>
> Key: OOZIE-2651
> URL: https://issues.apache.org/jira/browse/OOZIE-2651
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: OOZIE-2651.001.patch, OOZIE-2651.002.patch
>
>
> We noticed from some jstacks that Oozie was spending a lot of time looking
> for XML related jars (we do a lot of XML parsing in Oozie and Hadoop code).
> For instance, we'd see hundreds of threads blocked here
> {noformat}
> "pool-5-thread-57" prio=10 tid=0x7f9579923000 nid=0x6b7f waiting for
> monitor entry [0x7f94225e3000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
> - waiting to lock <0x000500026148> (a
> org.apache.catalina.loader.WebappClassLoader)
> at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
> at
> javax.xml.parsers.FactoryFinder.getProviderClass(FactoryFinder.java:117)
> at javax.xml.parsers.FactoryFinder.newInstance(FactoryFinder.java:178)
> at
> javax.xml.parsers.FactoryFinder.findJarServiceProvider(FactoryFinder.java:333)
> at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:255)
> at
> javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:121)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2526)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409)
> - locked <0x0007eac5ea28> (a org.apache.hadoop.conf.Configuration)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1144)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1116)
> at org.apache.oozie.util.XConfiguration.copy(XConfiguration.java:206)
> ...
> {noformat}
> There's nothing we can do about the lock at the {{WebappClassLoader}}, but by
> setting the {{javax.xml.parsers.DocumentBuilderFactory}} system property to
> {{org.apache.xerces.jaxp.DocumentBuilderFactoryImpl}}, we can speed this up a
> lot by getting rid of all of the searching that it's doing to find the
> implementation.
> See
> https://docs.oracle.com/javase/7/docs/api/javax/xml/parsers/DocumentBuilderFactory.html#newInstance()
>
> and
> https://www.databasesandlife.com/java-always-explicitly-specify-which-xml-parser-to-use/
> In our testing, we had a metric around creating a new {{JobClient}}, which
> also calls {{DocumentBuilderFactory.newInstance}}, and it dropped from ~3min
> max to ~2sec max (not an exact test of the savings here). And we also didn't
> see so much blocking on this in the jstacks anymore.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OOZIE-2651) Set javax.xml.parsers.DocumentBuilderFactory sys prop to make XML handling faster
[
https://issues.apache.org/jira/browse/OOZIE-2651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated OOZIE-2651:
-
Fix Version/s: (was: 4.3.0)
> Set javax.xml.parsers.DocumentBuilderFactory sys prop to make XML handling
> faster
> -
>
> Key: OOZIE-2651
> URL: https://issues.apache.org/jira/browse/OOZIE-2651
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: OOZIE-2651.001.patch
>
>
> We noticed from some jstacks that Oozie was spending a lot of time looking
> for XML related jars (we do a lot of XML parsing in Oozie and Hadoop code).
> For instance, we'd see hundreds of threads blocked here
> {noformat}
> "pool-5-thread-57" prio=10 tid=0x7f9579923000 nid=0x6b7f waiting for
> monitor entry [0x7f94225e3000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
> - waiting to lock <0x000500026148> (a
> org.apache.catalina.loader.WebappClassLoader)
> at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
> at
> javax.xml.parsers.FactoryFinder.getProviderClass(FactoryFinder.java:117)
> at javax.xml.parsers.FactoryFinder.newInstance(FactoryFinder.java:178)
> at
> javax.xml.parsers.FactoryFinder.findJarServiceProvider(FactoryFinder.java:333)
> at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:255)
> at
> javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:121)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2526)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409)
> - locked <0x0007eac5ea28> (a org.apache.hadoop.conf.Configuration)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1144)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1116)
> at org.apache.oozie.util.XConfiguration.copy(XConfiguration.java:206)
> ...
> {noformat}
> There's nothing we can do about the lock at the {{WebappClassLoader}}, but by
> setting the {{javax.xml.parsers.DocumentBuilderFactory}} system property to
> {{org.apache.xerces.jaxp.DocumentBuilderFactoryImpl}}, we can speed this up a
> lot by getting rid of all of the searching that it's doing to find the
> implementation.
> See
> https://docs.oracle.com/javase/7/docs/api/javax/xml/parsers/DocumentBuilderFactory.html#newInstance()
>
> and
> https://www.databasesandlife.com/java-always-explicitly-specify-which-xml-parser-to-use/
> In our testing, we had a metric around creating a new {{JobClient}}, which
> also calls {{DocumentBuilderFactory.newInstance}}, and it dropped from ~3min
> max to ~2sec max (not an exact test of the savings here). And we also didn't
> see so much blocking on this in the jstacks anymore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OOZIE-2651) Set javax.xml.parsers.DocumentBuilderFactory sys prop to make XML handling faster
[
https://issues.apache.org/jira/browse/OOZIE-2651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated OOZIE-2651:
-
Attachment: OOZIE-2651.001.patch
The 001 patch adds a new config property,
{{oozie.javax.xml.parsers.DocumentBuilderFactory}}, that lets you specify the
Factory to use (default {{org.apache.xerces.jaxp.DocumentBuilderFactoryImpl}}).
A blank/whitespace value tells Oozie not to set this.
Unfortunately, I'll need to look into this some more because it breaks the
Oozie Web UI (Tomcat can't find
org.apache.xerces.jaxp.DocumentBuilderFactoryImpl). I probably need to add the
Xerces jar to Tomcat somewhere.
> Set javax.xml.parsers.DocumentBuilderFactory sys prop to make XML handling
> faster
> -
>
> Key: OOZIE-2651
> URL: https://issues.apache.org/jira/browse/OOZIE-2651
> Project: Oozie
> Issue Type: Improvement
> Components: core
>Affects Versions: trunk
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: 4.3.0
>
> Attachments: OOZIE-2651.001.patch
>
>
> We noticed from some jstacks that Oozie was spending a lot of time looking
> for XML related jars (we do a lot of XML parsing in Oozie and Hadoop code).
> For instance, we'd see hundreds of threads blocked here
> {noformat}
> "pool-5-thread-57" prio=10 tid=0x7f9579923000 nid=0x6b7f waiting for
> monitor entry [0x7f94225e3000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
> - waiting to lock <0x000500026148> (a
> org.apache.catalina.loader.WebappClassLoader)
> at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
> at
> javax.xml.parsers.FactoryFinder.getProviderClass(FactoryFinder.java:117)
> at javax.xml.parsers.FactoryFinder.newInstance(FactoryFinder.java:178)
> at
> javax.xml.parsers.FactoryFinder.findJarServiceProvider(FactoryFinder.java:333)
> at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:255)
> at
> javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:121)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2526)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409)
> - locked <0x0007eac5ea28> (a org.apache.hadoop.conf.Configuration)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1144)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1116)
> at org.apache.oozie.util.XConfiguration.copy(XConfiguration.java:206)
> ...
> {noformat}
> There's nothing we can do about the lock at the {{WebappClassLoader}}, but by
> setting the {{javax.xml.parsers.DocumentBuilderFactory}} system property to
> {{org.apache.xerces.jaxp.DocumentBuilderFactoryImpl}}, we can speed this up a
> lot by getting rid of all of the searching that it's doing to find the
> implementation.
> See
> https://docs.oracle.com/javase/7/docs/api/javax/xml/parsers/DocumentBuilderFactory.html#newInstance()
>
> and
> https://www.databasesandlife.com/java-always-explicitly-specify-which-xml-parser-to-use/
> In our testing, we had a metric around creating a new {{JobClient}}, which
> also calls {{DocumentBuilderFactory.newInstance}}, and it dropped from ~3min
> max to ~2sec max (not an exact test of the savings here). And we also didn't
> see so much blocking on this in the jstacks anymore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OOZIE-2651) Set javax.xml.parsers.DocumentBuilderFactory sys prop to make XML handling faster
[
https://issues.apache.org/jira/browse/OOZIE-2651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated OOZIE-2651:
-
Description:
We noticed from some jstacks that Oozie was spending a lot of time looking for
XML related jars (we do a lot of XML parsing in Oozie and Hadoop code). For
instance, we'd see hundreds of threads blocked here
{noformat}
"pool-5-thread-57" prio=10 tid=0x7f9579923000 nid=0x6b7f waiting for
monitor entry [0x7f94225e3000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
- waiting to lock <0x000500026148> (a
org.apache.catalina.loader.WebappClassLoader)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
at
javax.xml.parsers.FactoryFinder.getProviderClass(FactoryFinder.java:117)
at javax.xml.parsers.FactoryFinder.newInstance(FactoryFinder.java:178)
at
javax.xml.parsers.FactoryFinder.findJarServiceProvider(FactoryFinder.java:333)
at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:255)
at
javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:121)
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2526)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409)
- locked <0x0007eac5ea28> (a org.apache.hadoop.conf.Configuration)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1144)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1116)
at org.apache.oozie.util.XConfiguration.copy(XConfiguration.java:206)
...
{noformat}
There's nothing we can do about the lock at the {{WebappClassLoader}}, but by
setting the {{javax.xml.parsers.DocumentBuilderFactory}} system property to
{{org.apache.xerces.jaxp.DocumentBuilderFactoryImpl}}, we can speed this up a
lot by getting rid of all of the searching that it's doing to find the
implementation.
See
https://docs.oracle.com/javase/7/docs/api/javax/xml/parsers/DocumentBuilderFactory.html#newInstance()
and
https://www.databasesandlife.com/java-always-explicitly-specify-which-xml-parser-to-use/
In our testing, we had a metric around creating a new {{JobClient}}, which also
calls {{DocumentBuilderFactory.newInstance}}, and it dropped from ~3min max to
~2sec max (not an exact test of the savings here). And we also didn't see so
much blocking on this in the jstacks anymore.
was:
We noticed from some jstacks that Oozie was spending a lot of time looking for
XML related jars (we do a lot of XML parsing in Oozie and Hadoop code). For
instance, we'd see hundreds of threads blocked here
{noformat}
"pool-5-thread-57" prio=10 tid=0x7f9579923000 nid=0x6b7f waiting for
monitor entry [0x7f94225e3000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
- waiting to lock <0x000500026148> (a
org.apache.catalina.loader.WebappClassLoader)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
at
javax.xml.parsers.FactoryFinder.getProviderClass(FactoryFinder.java:117)
at javax.xml.parsers.FactoryFinder.newInstance(FactoryFinder.java:178)
at
javax.xml.parsers.FactoryFinder.findJarServiceProvider(FactoryFinder.java:333)
at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:255)
at
javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:121)
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2526)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409)
- locked <0x0007eac5ea28> (a org.apache.hadoop.conf.Configuration)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1144)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1116)
at org.apache.oozie.util.XConfiguration.copy(XConfiguration.java:206)
...
{noformat}
There's nothing we can do about the lock at the {{WebappClassLoader}}, but by
setting the {{javax.xml.parsers.DocumentBuilderFactory}} system property to
{{org.apache.xerces.jaxp.DocumentBuilderFactoryImpl}}, we can speed this up a
lot by getting rid of all of the searching that it's doing to find the
implementation.
See
https://docs.oracle.com/javase/7/docs/api/javax/xml/parsers/DocumentBuilderFactory.html#newInstance()
and
https://www.databasesandlife.com/java-always-explicitly-specify-which-xml-parser-to-use/
In our testing, we had a metric around creating a
