Bug#567210: doc-available always returns false without network

[Eugene Zhukov]

Hello,

A fresh look/opinion is welcome! Could someone please take a fresh
look at the bug #567210.
After closer look at my last test results (described in my two last
mails to the bug), I tend to think the problem lies in Saxon-{B|HE}.

Any comments are very much appreciated,
Eugene

__
This is the maintainer address of Debian's Java team
<http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-java-maintainers>. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#567210: doc-available always returns false without network

[Eugene Zhukov]

On Thu, Jan 30, 2014 at 1:49 PM, Eugene Zhukov  wrote:
> It fails immediately without network, but hangs and succeeds with
> network. "Saxon does not have a local copy of PUBLIC -//W3C//DTD
> XHTML+RDFa" is not shown anymore! With tcpdump and wireshark I see a
> request:
> GET /MarkUp/DTD/xhtml-rdfa-1.dtd HTTP/1.1
> User-Agent: Java/1.6.0_27
> Host: www.w3.org
> Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
> Connection: keep-alive
I just wanted to amend this test-case with more-or-less full HTTP
request/response chain from tcpdump: http://paste.debian.net/79423/

__
This is the maintainer address of Debian's Java team
. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#567210: doc-available always returns false without network

[Eugene Zhukov]

On Wed, Jan 29, 2014 at 4:25 PM, Michael Kay  wrote:
>
>> Warning: SXXP0005: The source document is in namespace
>> http://www.w3.org/2005/Atom, but none of the
>>  template rules match elements in this namespace
>
> You can ignore that warning for present purposes.
>> [...]
>> Saxon does not have a local copy of PUBLIC -//W3C//DTD XHTML+RDFa
>> 1.0//EN SYSTEM http://www.w3.org/MarkUp/DTD/xhtml-rdfa-1.dtd
>
> Unfortunately there is no complete list of DTDs on the W3C site that might 
> potentially needed, and even if there were, I probably wouldn't want to ship 
> them all with Saxon. So you might have to go back to using catalogs. On the 
> other hand, if you can identify where this was referenced from, I can take a 
> look and see if it ought to be included. It looks as if it comes from one of 
> the XHTML variants, but there seem to be many of these in use.
>

Sorry about the confusion with catalogs. Back to initial test-case
with the latest Saxon-HE from [0]:
$ java -cp /etc/xml/resolver:/usr/share/java/xml-resolver.jar:../saxon9he.jar
-Dxml.catalog.files=/etc/xml/catalog -Dxml.catalog.verbosity=1
net.sf.saxon.Transform -t
-x:org.apache.xml.resolver.tools.ResolvingXMLReader
-y:org.apache.xml.resolver.tools.ResolvingXMLReader
-r:org.apache.xml.resolver.tools.CatalogResolver -s:foo.xml
-xsl:foo.xsl

It fails immediately without network, but hangs and succeeds with
network. "Saxon does not have a local copy of PUBLIC -//W3C//DTD
XHTML+RDFa" is not shown anymore! With tcpdump and wireshark I see a
request:
GET /MarkUp/DTD/xhtml-rdfa-1.dtd HTTP/1.1
User-Agent: Java/1.6.0_27
Host: www.w3.org
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive

As you say in the previous letter there is no complete list of DTDs on
the W3C site that might potentially be needed, and even if there were,
Saxon would not ship them all.
Should we conclude then this is a corner test-case and is not
supported by Saxon offline?

[0] 
http://sourceforge.net/projects/saxon/files/Saxon-HE/9.5/SaxonHE9-5-1-4J.zip/download

Eugene

__
This is the maintainer address of Debian's Java team
. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#567210: doc-available always returns false without network

[Michael Kay]

> Warning: SXXP0005: The source document is in namespace
> http://www.w3.org/2005/Atom, but none of the
>  template rules match elements in this namespace

You can ignore that warning for present purposes.
> [...]
> Saxon does not have a local copy of PUBLIC -//W3C//DTD XHTML+RDFa
> 1.0//EN SYSTEM http://www.w3.org/MarkUp/DTD/xhtml-rdfa-1.dtd

Unfortunately there is no complete list of DTDs on the W3C site that might 
potentially needed, and even if there were, I probably wouldn't want to ship 
them all with Saxon. So you might have to go back to using catalogs. On the 
other hand, if you can identify where this was referenced from, I can take a 
look and see if it ought to be included. It looks as if it comes from one of 
the XHTML variants, but there seem to be many of these in use.

Michael Kay
Saxonica

__
This is the maintainer address of Debian's Java team
. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#567210: doc-available always returns false without network

[Eugene Zhukov]

On Wed, Jan 29, 2014 at 3:01 PM, Michael Kay  wrote:
> OK, so the problem seems to be here:
>
> Cannot read xhtml11/xhtml-inlpres-1.mod file
>
I tried with latest 9.4 and it wasn't the case anymore, but main
problem persisted.

> and the reason would appear to be the absence of the w3c/ prefix on the file 
> name.
>
> This takes us to here:
>
> https://saxonica.plan.io/boards/3/topics/5625
>
> and that in turn leads me to
>
> https://saxonica.plan.io/issues/1813
>
> which I think is fixed in the 9.5 branch but not in 9.4.
>
> The underlying cause is inconsistent use of system IDs and public IDs in the 
> W3C-published DTDs.
>

I tried with the latest Saxon from [0] and it still fails without
network, but now it gives some valuable output:
[...]
Warning: SXXP0005: The source document is in namespace
http://www.w3.org/2005/Atom, but none of the
  template rules match elements in this namespace
[...]
Saxon does not have a local copy of PUBLIC -//W3C//DTD XHTML+RDFa
1.0//EN SYSTEM http://www.w3.org/MarkUp/DTD/xhtml-rdfa-1.dtd
[...]

[0] 
http://sourceforge.net/projects/saxon/files/Saxon-HE/9.5/SaxonHE9-5-1-4J.zip/download

__
This is the maintainer address of Debian's Java team
. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#567210: doc-available always returns false without network

[Michael Kay]

OK, so the problem seems to be here:

Cannot read xhtml11/xhtml-inlpres-1.mod file

and the reason would appear to be the absence of the w3c/ prefix on the file 
name.

This takes us to here:

https://saxonica.plan.io/boards/3/topics/5625

and that in turn leads me to

https://saxonica.plan.io/issues/1813

which I think is fixed in the 9.5 branch but not in 9.4.

The underlying cause is inconsistent use of system IDs and public IDs in the 
W3C-published DTDs.

Michael Kay
Saxonica


On 29 Jan 2014, at 12:41, Eugene Zhukov  wrote:

> On Wed, Jan 29, 2014 at 11:00 AM, Michael Kay  wrote:
>> If you use the -t option on the command line, then attempts to use local 
>> copies of W3C DTDs will be traced on System.err. Hopefully this will shed 
>> more light on why the mechanism isn't working for you.
>> 
>> The EntityResolver that Saxon uses in 9.4 can be found here:
>> 
>> https://dev.saxonica.com/repos/archive/opensource/tags/9.4.0.7/hej/net/sf/saxon/lib/StandardEntityResolver.java
>> 
>> I'm not sure why the data files aren't included under the 9.4.0.7 Subversion 
>> tag, but the files are here:
>> 
>> https://dev.saxonica.com/repos/archive/opensource/latest9.4/data/w3c/
>> 
>> I note that your JAR file has been renamed, so it's possible it has also 
>> been rebuilt. Look inside it with a ZIP utility and check for the directory 
>> named "w3c".
>> 
>> A list of the W3C documents bundled with Saxon for 9.5 can also be found 
>> here:
>> 
>> http://www.saxonica.com/documentation/index.html#!sourcedocs/w3c-dtds
>> 
>> and the corresponding list for 9.4 is at:
>> 
>> http://www.saxonica.com/documentation9.4-demo/index.html#!sourcedocs/w3c-dtds
>> 
> Thanks for the links!
> I downloaded the official release from [0] and tried the test with it.
> Here is the result with network: http://paste.debian.net/78983/
> And here is the result without network: http://paste.debian.net/78984/
> As you can see the test without network still fails. With this -t
> option, when the test succeeds, you can see Saxon fetching a local
> copy, but that doesn't seem to be the case without network.
> 
> [0] 
> http://sourceforge.net/projects/saxon/files/Saxon-HE/9.4/SaxonHE9-4-0-7J.zip/download
> 
> Eugene

__
This is the maintainer address of Debian's Java team
. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#567210: doc-available always returns false without network

[Eugene Zhukov]

On Wed, Jan 29, 2014 at 11:00 AM, Michael Kay  wrote:
> If you use the -t option on the command line, then attempts to use local 
> copies of W3C DTDs will be traced on System.err. Hopefully this will shed 
> more light on why the mechanism isn't working for you.
>
> The EntityResolver that Saxon uses in 9.4 can be found here:
>
> https://dev.saxonica.com/repos/archive/opensource/tags/9.4.0.7/hej/net/sf/saxon/lib/StandardEntityResolver.java
>
> I'm not sure why the data files aren't included under the 9.4.0.7 Subversion 
> tag, but the files are here:
>
> https://dev.saxonica.com/repos/archive/opensource/latest9.4/data/w3c/
>
> I note that your JAR file has been renamed, so it's possible it has also been 
> rebuilt. Look inside it with a ZIP utility and check for the directory named 
> "w3c".
>
> A list of the W3C documents bundled with Saxon for 9.5 can also be found here:
>
> http://www.saxonica.com/documentation/index.html#!sourcedocs/w3c-dtds
>
> and the corresponding list for 9.4 is at:
>
> http://www.saxonica.com/documentation9.4-demo/index.html#!sourcedocs/w3c-dtds
>
Thanks for the links!
I downloaded the official release from [0] and tried the test with it.
Here is the result with network: http://paste.debian.net/78983/
And here is the result without network: http://paste.debian.net/78984/
As you can see the test without network still fails. With this -t
option, when the test succeeds, you can see Saxon fetching a local
copy, but that doesn't seem to be the case without network.

[0] 
http://sourceforge.net/projects/saxon/files/Saxon-HE/9.4/SaxonHE9-4-0-7J.zip/download

Eugene

__
This is the maintainer address of Debian's Java team
. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#567210: doc-available always returns false without network

[Michael Kay]

If you use the -t option on the command line, then attempts to use local copies 
of W3C DTDs will be traced on System.err. Hopefully this will shed more light 
on why the mechanism isn't working for you.

The EntityResolver that Saxon uses in 9.4 can be found here:

https://dev.saxonica.com/repos/archive/opensource/tags/9.4.0.7/hej/net/sf/saxon/lib/StandardEntityResolver.java

I'm not sure why the data files aren't included under the 9.4.0.7 Subversion 
tag, but the files are here:

https://dev.saxonica.com/repos/archive/opensource/latest9.4/data/w3c/

I note that your JAR file has been renamed, so it's possible it has also been 
rebuilt. Look inside it with a ZIP utility and check for the directory named 
"w3c".

A list of the W3C documents bundled with Saxon for 9.5 can also be found here:

http://www.saxonica.com/documentation/index.html#!sourcedocs/w3c-dtds

and the corresponding list for 9.4 is at:

http://www.saxonica.com/documentation9.4-demo/index.html#!sourcedocs/w3c-dtds

Michael Kay
Saxonica


On 29 Jan 2014, at 08:28, Eugene Zhukov  wrote:

> On Tue, Jan 28, 2014 at 4:25 PM, Michael Kay  wrote:
>> Saxon-B 9.1 does not include copies of these resources.
>> 
>> You can always write a URIResolver and direct the request to copies held at 
>> application level, but it can't be done "behind the scenes".
>> 
>> My recommendation would be to move forward to a later Saxon release that 
>> fixes the problem. The current release is 9.5. We have no plans to issue 
>> further maintenance releases for 9.1, although we do appreciate that some 
>> users have been sticking with that release because of the discontinuities 
>> introduced between 9.1 and 9.2.
>> 
> 
> We have Saxon-HE 9.4.0.7 in Debian archive. So I tried the above
> test-case with it:
> $ java -cp 
> /etc/xml/resolver:/usr/share/java/xml-resolver.jar:/usr/share/java/Saxon-HE.jar
> -Dxml.catalog.files=/etc/xml/catalog -Dxml.catalog.verbosity=1
> net.sf.saxon.Transform -s:foo.xml -xsl:foo.xsl
> 
> The result is it still fails without network. With network it works.
> Also, when I look into the source code of Saxon-HE 9.4.0.7 at [0], I
> cannot find the local copies of those resources. So I don't understand
> how it would work without the network. What did I miss?
> 
> [0] https://dev.saxonica.com/repos/archive/opensource/tags/9.4.0.7/
> 
> Eugene

__
This is the maintainer address of Debian's Java team
. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#567210: doc-available always returns false without network

[Eugene Zhukov]

On Tue, Jan 28, 2014 at 4:25 PM, Michael Kay  wrote:
> Saxon-B 9.1 does not include copies of these resources.
>
> You can always write a URIResolver and direct the request to copies held at 
> application level, but it can't be done "behind the scenes".
>
> My recommendation would be to move forward to a later Saxon release that 
> fixes the problem. The current release is 9.5. We have no plans to issue 
> further maintenance releases for 9.1, although we do appreciate that some 
> users have been sticking with that release because of the discontinuities 
> introduced between 9.1 and 9.2.
>

We have Saxon-HE 9.4.0.7 in Debian archive. So I tried the above
test-case with it:
$ java -cp 
/etc/xml/resolver:/usr/share/java/xml-resolver.jar:/usr/share/java/Saxon-HE.jar
-Dxml.catalog.files=/etc/xml/catalog -Dxml.catalog.verbosity=1
net.sf.saxon.Transform -s:foo.xml -xsl:foo.xsl

The result is it still fails without network. With network it works.
Also, when I look into the source code of Saxon-HE 9.4.0.7 at [0], I
cannot find the local copies of those resources. So I don't understand
how it would work without the network. What did I miss?

[0] https://dev.saxonica.com/repos/archive/opensource/tags/9.4.0.7/

Eugene

__
This is the maintainer address of Debian's Java team
. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#567210: libsaxonb-java: doc-available always returns false without network

[tony mancill]

Control: -1 tag +help

Hello,

The patch/different arguments for invoking saxonb-xslt attached to this
bug report doesn't appear to work with the test case provided by Brian
Carlson and current version of libsaxonb-java in the archive in
jessie/sid.  Or at least it's not working for me; it hangs after the
first "Requested document is file:/path/to/index.xhtml" and I can
confirm via wireshark that it is phoning home to www.w3.org.

I also tried specifying the Apache resolver (as in [0], but with the
paths updated to match Debian's libxml-commons-resolver1.1-java JAR,
etc.), and it hangs in the same way.

Can someone demonstrate test cases that show that this bug is addressed?

Thank you,
tony

[0] https://bugs.launchpad.net/ubuntu/+source/saxonb/+bug/400277



signature.asc
Description: OpenPGP digital signature
__
This is the maintainer address of Debian's Java team
. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#567210:

[Mathieu Malaterre]

tags 567210 patch
severity 567210 grave
thanks

Instead of:

$ cat /usr/bin/saxon-xslt
#!/bin/sh

exec java -classpath /usr/share/java/saxon.jar com.icl.saxon.StyleSheet "$@"

it should read:

#!/bin/sh

java -cp 
/etc/xml/resolver:/usr/share/java/xslthl.jar:/usr/share/java/xml-resolver.jar:/usr/share/java/saxonb.jar
net.sf.saxon.Transform -x
org.apache.xml.resolver.tools.ResolvingXMLReader -y
org.apache.xml.resolver.tools.ResolvingXMLReader -r
org.apache.xml.resolver.tools.CatalogResolver "$@"

__
This is the maintainer address of Debian's Java team
. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#567210:

[Mathieu Malaterre]

tags 567210 important
thanks

Technically this render the package unusable if you do not have
internet connection, right ? Should'nt this be marked a 'grave' then ?

__
This is the maintainer address of Debian's Java team
. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#567210: Testcase

[Torsten Werner]

On Mon, Aug 29, 2011 at 8:45 PM, brian m. carlson
 wrote:
> I'm not using saxonb via ant right now, so the ant task is not critical
> for me anymore.  Do you still need someone to fix the underlying bug
> (doc-available returns false for file URIs) or just the ant task?  I'll
> try to look into the former sometime this wekk if that's the case to see
> if I can fix it.

That would be nice.

Thanks,
Torsten



__
This is the maintainer address of Debian's Java team
. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#567210: Testcase

[brian m. carlson]

On Mon, Aug 29, 2011 at 07:35:50PM +0200, Torsten Werner wrote:
> On Thu, Jan 28, 2010 at 2:15 AM, brian m. carlson
>  wrote:
> > I forgot to include the testcase, so it is now attached.  Also, since
> > I'm really only using Saxon-B via Ant, if a new feature is added to
> > resolve this problem, I really need it to be accessible via the Ant
> > task.
> 
> thanks for your report. I am afraid that we need someone to implement
> the feature. Please note that I have just uploaded a newer upstream
> version 9.1.0.8 of saxonb.

I'm not using saxonb via ant right now, so the ant task is not critical
for me anymore.  Do you still need someone to fix the underlying bug
(doc-available returns false for file URIs) or just the ant task?  I'll
try to look into the former sometime this wekk if that's the case to see
if I can fix it.

-- 
brian m. carlson / brian with sandals: Houston, Texas, US
+1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187


signature.asc
Description: Digital signature
__
This is the maintainer address of Debian's Java team
. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#567210: Testcase

[Torsten Werner]

Hi Brian,

On Thu, Jan 28, 2010 at 2:15 AM, brian m. carlson
 wrote:
> I forgot to include the testcase, so it is now attached.  Also, since
> I'm really only using Saxon-B via Ant, if a new feature is added to
> resolve this problem, I really need it to be accessible via the Ant
> task.

thanks for your report. I am afraid that we need someone to implement
the feature. Please note that I have just uploaded a newer upstream
version 9.1.0.8 of saxonb.

Sorry,
Torsten



__
This is the maintainer address of Debian's Java team
. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#567210: Testcase

[brian m. carlson]

I forgot to include the testcase, so it is now attached.  Also, since
I'm really only using Saxon-B via Ant, if a new feature is added to
resolve this problem, I really need it to be accessible via the Ant
task.

-- 
brian m. carlson / brian with sandals: Houston, Texas, US
+1 713 440 7475 | http://crustytoothpaste.ath.cx/~bmc | My opinion only
OpenPGP: RSA v4 4096b 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187


saxonb-testcase.tar.gz
Description: Binary data


signature.asc
Description: Digital signature
___
pkg-java-maintainers mailing list
pkg-java-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/mailman/listinfo/pkg-java-maintainers

Bug#567210: libsaxonb-java: doc-available always returns false without network

[brian m. carlson]

Package: libsaxonb-java
Version: 9.0.0.4+svn20080322-3
Severity: normal

In certain cases, the doc-available function wrongly returns false when
the document does, in fact, exist.  The attached testcase contains three
files.  Running "saxonb-xslt -s:foo.xml -xsl:foo.xsl" produces the
output:

Requested document is file:/tmp/saxonb-testcase/index.xhtml
Missing‽
Requested document is index.xhtml
Missing‽
Requested document is file:/tmp/saxonb-testcase/index.xhtml
Missing‽
Requested document is index.xhtml
Missing‽

Note that if Java networking works, it works correctly:

Requested document is file:/tmp/saxonb-testcase/index.xhtml
It exists!
Requested document is index.xhtml
It exists!
Requested document is file:/tmp/saxonb-testcase/index.xhtml
It exists!
Requested document is index.xhtml
It exists!

Nevertheless, there is no reason to require networking for file: URIs.
Saxon B should work correctly for file: URIs regardless of the state of
networking, period.  There is no legitimate reason to make a network
request when all documents are local.

Upon further examination, it appears that Saxon is attempting to do a
lookup of the DTD on www.w3.org, even when -expand:off and -dtd:off are
specified.  It does not use catalogs to do so, and therefore fails.

-- System Information:
Debian Release: squeeze/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (1, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.32-trunk-amd64 (SMP w/2 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages libsaxonb-java depends on:
ii  default-jre-headless [jav 1.6-34 Standard Java or Java compatible R
ii  gcj-4.4-jre-headless [jav 4.4.3-1Java runtime environment using GIJ
ii  gcj-jre-headless [java2-r 4:4.4.2-3  Java runtime environment using GIJ
ii  gij-4.3 [java2-runtime-he 4.3.4-4The GNU Java bytecode interpreter
ii  libdom4j-java 1.6.1+dfsg.2-2 flexible XML framework for Java
ii  libjdom1-java 1.1.1+dfsg-1   lightweight and fast library using
ii  libxom-java   1.2.1-1A new XML object model for Java
ii  openjdk-6-jre-headless [j 6b17~pre3-1OpenJDK Java runtime, using Hotspo

libsaxonb-java recommends no packages.

Versions of packages libsaxonb-java suggests:
pn  libsaxonb-java-doc (no description available)

-- no debconf information

-- 
brian m. carlson / brian with sandals: Houston, Texas, US
+1 713 440 7475 | http://crustytoothpaste.ath.cx/~bmc | My opinion only
OpenPGP: RSA v4 4096b 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187


signature.asc
Description: Digital signature
___
pkg-java-maintainers mailing list
pkg-java-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/mailman/listinfo/pkg-java-maintainers