Bug#256283: [pylucene-dev] Re: PyLucence Debian Package
Hi, I did a little digging. Package gcj-3.4 was removed from Debian Unstable on Aug 14, 2005 because it was Not Built by Source http://ftp-master.debian.org/removals.txt Since gcj is the critical path, this suggests to me that maybe gcj-4.x is the better place to put effort. Andi, do you and OSAF already have all the communication and clout you need with gcj devs? I suppose we could try getting Debian maintainers to lean on upstream a little bit - not that they responded to my initial inquiry. And Ubuntu might also care. On the other hand, gcj is so high profile they may be immune to a little extra persuasion. Are the bugs rocket sciencish, or can a random C programmer like me dive in and help? Any other way we can help? Other possible TODO items: * Swig sounds like it is progressing just fine. * Figure out Java Lucene / Free Software build issues. Barry told me today on IM that he is interested in this, but he is totally swamped. I'm also willing to look, I'm just very slow. FYI, Java Lucene 1.4.3 only builds via a Free Software toolchain in Debian Testing (etch) and Debian Unstable (sid). Debian Stable (sarge) doesn't have a powerful enough free software toolchain for Java. * Andi, think about - eventually - having a future PyLucene source release that is more sourcetastic. For example, maybe include a tarball of the SVN snapshot and a new build target that untars, patches, and rebuilds the .jar files. The precompiled .jar files can still be included. If that's a real pain, then defer because it is not on the critical path. But this or something like it will eventually be helpful, especially since it sounds like the gcj-3.4 package got kicked out of Debian for similar reasons. PyLucene only works with Python 2.4 so to make use of it ensure you are using /usr/bin/python2.4 and not /usr/bin/python. * Mathew, the PyLucene module will not be found if one tries to import it from python 2.3 right? I don't have the PyLucene package in front of me to double check this. http://www.debian.org/doc/packaging-manuals/python-policy/ch-python.html#s-paths * I guess backports.org is still an open question. Anyway, great job on getting this far with the sarge packages. Matthew, you've already made a significant positive difference, and I'm already hearing positive comments. Most recently from a hacker at PARC (my employer) who is actively migrating an internal application called UpLib from Java Lucene towards PyLucene. -Jeff
Bug#256283: [pylucene-dev] Re: PyLucence Debian Package
Jeff Breidenbach [EMAIL PROTECTED] said: Hi, I did a little digging. Package gcj-3.4 was removed from Debian Unstable on Aug 14, 2005 because it was Not Built by Source I also did some digging. I found the same removal message and then tracked down a gcj maintainer IRC. He simply said: doko moconnor: we only want to have one gcj version So, that's that. Also, all the gcj-3.4 bugs were reassigned to gcj-4.0. * Swig sounds like it is progressing just fine. I was able to build PyLucene with what will be SWIG 1.3.28. However, PyLucene failed to load. So work is needed there but at least it's not generating bad stuff, that's a plus. PyLucene only works with Python 2.4 so to make use of it ensure you are using /usr/bin/python2.4 and not /usr/bin/python. * Mathew, the PyLucene module will not be found if one tries to import it from python 2.3 right? I don't have the PyLucene package in front of me to double check this. Correct. It goes in /usr/lib/python2.4/site-packages/. I figured this was a fine place b/c it is where python2.4-elementtree puts itself. * I guess backports.org is still an open question. Anyway, great job on getting this far with the sarge packages. Matthew, you've already made a significant positive difference, and I'm already hearing positive comments. Most recently from a hacker at PARC (my employer) who is actively migrating an internal application called UpLib from Java Lucene towards PyLucene. Cool. I submitted the repository to http://www.apt-get.org as well. As asked around and it looks like the backports.org path is as you describe: unstable or testing packages which get backported. -matthew -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#256283: [pylucene-dev] Re: PyLucence Debian Package
On Tue, 7 Feb 2006, Jeff Breidenbach wrote: Hi, I did a little digging. Package gcj-3.4 was removed from Debian Unstable on Aug 14, 2005 because it was Not Built by Source http://ftp-master.debian.org/removals.txt Since gcj is the critical path, this suggests to me that maybe gcj-4.x is the better place to put effort. Andi, do you and OSAF already have all the communication and clout you need with gcj devs? I tried gcj 4.x and have had little luck with it. For the longest time, I wasn't even able to build it. I should try it again... gcj 3.4.x is the safe and stable route for the time being. With the arrival of Intel Macs and no Intel OS X gcj available until probably gcj 4.2, this may change sometime in the next 12 months. As for 'clout' with the gcj devs, well, that word would be inappropriate. I suppose we could try getting Debian maintainers to lean on upstream a little bit - not that they responded to my initial inquiry. And Ubuntu might also care. On the other hand, gcj is so high profile they may be immune to a little extra persuasion. Are the bugs rocket sciencish, or can a random C programmer like me dive in and help? Any other way we can help? Definitely not a random C programmer. gcj is actually at least 3 projects put together, with their own interests and schedules: gcc + classpath + boehm-gc. I've had very little luck in getting any issue resolved, most of them are hard to isolate and reproduce or not on the gcj devs' agenda (static linking, for example). The [EMAIL PROTECTED] mailing list is quite active and very responsive and can be very helpful if you're willing to do most of the work since there is a lot more work than gcj devs can handle. I've had better luck working around issues on my own. * Swig sounds like it is progressing just fine. In a way, it is, but distros are very quick to upgrade to the latest swig in spite of the fact that swig keeps making incompatible changes that make it unlikely for a new swig release to work out of the box for pre-existing software packages. * Andi, think about - eventually - having a future PyLucene source release that is more sourcetastic. For example, maybe include a tarball of the SVN snapshot and a new build target that untars, patches, and rebuilds the .jar files. This is how PyLucene is built from scratch. The source release includes the pre-compiled jars because the audience for PyLucene is Python programmers and I didn't want to impose a JDK requirement on them. GCJ is difficult enough to get right - on the Mac, you have to build gcj yourself since Apple doesn't ship it - adding a JDK requirement on top of it (+ ant) would be gratuitous. The precompiled .jar files can still be included. If that's a real pain, then defer because it is not on the critical path. But this or something like it will eventually be helpful, especially since it sounds like the gcj-3.4 package got kicked out of Debian for similar reasons. I have no problems with making a source-only package, it'd add a java compiler requirement to this exercise. Whichever works best in the world of Debian is fine by me. * Mathew, the PyLucene module will not be found if one tries to import it from python 2.3 right? I don't have the PyLucene package in front of me to double check this. It'll be found if it is in the right place to be found. If PyLucene is built against Python 2.4 headers it may very well crash the process when imported from Python 2.3. Anyway, great job on getting this far with the sarge packages. Matthew, you've already made a significant positive difference, and I'm already hearing positive comments. Most recently from a hacker at PARC (my employer) who is actively migrating an internal application called UpLib from Java Lucene towards PyLucene. Excellent ! Andi.. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#256283: [pylucene-dev] Re: PyLucence Debian Package
Matthew, I just built your package on sarge, it went fine. Good job. python2.4-pylucene_1.9-1_i386.deb I noticed you chose not to have a README.Debian file. That's probably ok, since most of the weirdness affects package maintainers and not package users. On the other hand, you might want to create one and include a link to this conversation (bug 256283) in case the end user is wondering why the heck it isn't included in Debian proper. Your call. Andi, how about distributing the binary package (.deb) from the PyLucene website while things are shaking out? Or perhaps Matthew can distribute from canonical.org, and have a link from the PyLucene homepage? This can be done immediately, allows Debian sarge users to try out the package and provide feedback, and will make a few people's lives a little easier. I can't think of any disadvantages. Matthew, I think your list of open issues is spot-on. One good thing about item #3 (building Java Lucene 1.9 with a Free Software toolchain) is this goal is shared by the folks who package Java Lucene. That's Barry, myself, and other members of the Java packaging team. I can assure you there will be no lack of motivation and manpower for the task if Eclipse ever starts depending on Java Lucene 1.9. http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=272295 Cheers, Jeff
Bug#256283: [pylucene-dev] Re: PyLucence Debian Package
Jeff Breidenbach [EMAIL PROTECTED] said: However, it is technically possible to compile on Debian stable then upload the binary package to unstable (a.k.a. sid). This can - somewhat - bypass the swig issue and get a working PyLucene package in Debian for one architecture, presumably i386. Once the swig issue gets straightened out, the autobuilders will be able to handle the other architectures. Okay, so this is what I'd want to do if the GCC folks respond and say GCJ 3.4.x (x = 3) will be in Unstable. Since GCC 3.4.4 is in Unstable I'm mildly optimistic GCJ 3.4.x will be as well. The gcj situation is more serious. No way in hell can we package the gcj runtime inside the PyLucene package. As far as I can tell this is a showstopper, although I'm curious what the gcj package maintainers have to say about the matter. Yes, I agree, without GCJ runtime support there can be no PyLucene package. :( Finally, Matthew I'd like to know your estimated attention span as we consider the possibility of putting a semi-broken package into Debian. My comfort zone for [get it fairly good first] vs [put it in then improve it] shifts a little depending on whether you have short term or long term interest. If things resolve themselves such that a package could actually make it into Debian then I'd certainly take a long-term responsibility for it. I'd like to see it happen. As you suggested, if Andi agrees I think providing a link to a .deb for Debian Stable (Sarge) from the PyLucene website is the best short-term solution. This would take into account the shifting RC status of Lucene 1.9, the GCJ issues in Debian Unstable, the SWIG issues, and it makes something available to folks. -matthew -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#256283: [pylucene-dev] Re: PyLucence Debian Package
Jeff Breidenbach [EMAIL PROTECTED] said: Matthew, I just built your package on sarge, it went fine. Good job. python2.4-pylucene_1.9-1_i386.deb I noticed you chose not to have a README.Debian file. That's probably ok, since most of the weirdness affects package maintainers and not package users. On the other hand, you might want to create one and include a link to this conversation (bug 256283) in case the end user is wondering why the heck it isn't included in Debian proper. Your call. Thank you! I will add a README.Debian, it's a good suggestion. I have access to Debian amd64 and PowerPC architectures. If building for them is identical to building for i386 then I can make those available as well. Andi, how about distributing the binary package (.deb) from the PyLucene website while things are shaking out? Or perhaps Matthew can distribute from canonical.org, and have a link from the PyLucene homepage? This can be done immediately, allows Debian sarge users to try out the package and provide feedback, and will make a few people's lives a little easier. I can't think of any disadvantages. This sounds like a good idea to me. I can rebuild the package for i386, with the README.Debian, and make it available later today. Jeff and Andi, thanks for your time on this so far. Your help is appreciated! -matthew -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#256283: [pylucene-dev] Re: PyLucence Debian Package
On Sun, 5 Feb 2006, Jeff Breidenbach wrote: Andi, how about distributing the binary package (.deb) from the PyLucene website while things are shaking out? Or perhaps Matthew can distribute from canonical.org, and have a link from the PyLucene homepage? This can be done immediately, allows Debian sarge users to try out the package and provide feedback, and will make a few people's lives a little easier. I can't think of any disadvantages. +1 Where can I get the package from ? (I don't have a Debian system available to me to build it myself). Alternatively, I could also just put a link to the package - hosted elsewhere - onto the PyLucene homepage. Either way is fine. Andi.. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#256283: [pylucene-dev] Re: PyLucence Debian Package
Andy, I compiled PyLucene 1.9rc1-7 for the i386, amd64, and powerpc Debian architectures. I created an APT repository because I figured it would be easier to manage. So if you add the follow instructions to your website they should make sense to any Debian user: There are binary PyLucene 1.9rc1-7 Debian Stable (Sarge) packages for the i386, amd64, and powerpc architectures. To install PyLucene add the following apt repository to your /etc/apt/sources.list file: deb http://www.canonical.org/debian/ sarge main then run the following commands: apt-get update apt-get install python2.4-pylucene PyLucene only works with Python 2.4 so to make use of it ensure you are using /usr/bin/python2.4 and not /usr/bin/python. Any questions / concerns, just tell me and I'll change whatever you need. Oh, one last thing, in addition to adding a README.Debian file I added the samples and test cases to the documentation set. The /usr/share/doc/python2.4-pylucene/README.Debian file gives instructions on how to run them. -matthew Andi Vajda [EMAIL PROTECTED] said: On Sun, 5 Feb 2006, Jeff Breidenbach wrote: Andi, how about distributing the binary package (.deb) from the PyLucene website while things are shaking out? Or perhaps Matthew can distribute from canonical.org, and have a link from the PyLucene homepage? This can be done immediately, allows Debian sarge users to try out the package and provide feedback, and will make a few people's lives a little easier. I can't think of any disadvantages. +1 Where can I get the package from ? (I don't have a Debian system available to me to build it myself). Alternatively, I could also just put a link to the package - hosted elsewhere - onto the PyLucene homepage. Either way is fine. Andi.. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#256283: [pylucene-dev] Re: PyLucence Debian Package
On Sun, 5 Feb 2006, Matthew O'Connor wrote: I compiled PyLucene 1.9rc1-7 for the i386, amd64, and powerpc Debian architectures. I created an APT repository because I figured it would be easier to manage. So if you add the follow instructions to your website they should make sense to any Debian user. I added your stuff to the PyLucene homepage under the Linux (Debian) section. http://pylucene.osafoundation.org/ Let me know if this is what you had in mind, Thanks ! Andi.. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#256283: [pylucene-dev] Re: PyLucence Debian Package
Jeff Breidenbach [EMAIL PROTECTED] said: Andi, Debian's infrastructure is designed such that a source package is not allowed to be a build dependency. Matthew, please file a wishlist bug against swig, requesting a version update. I think I confused the issue. Unstable has SWIG 1.3.27. PyLucene requires version 1.3.24, which is older. So I don't think a wishlist bug is appropriate here. Also, PyLucene requires GCJ 3.4.x, x = 3. However, unstable has 4.0.2. This is an even more egregious issue because even providing a .jar in the source package would not be enough since that version of gcj would produce bad results (everything compiles, nothing works). I don't think a build on unstable is currently possible. PyLucene requires versions of GCJ and SWIG which are too old to be in Debian unstable. So, I see the open issues as follows: 1. PyLucene and/or SWIG needs to change so that the version of SWIG in Debian Unstable can be used. 2. PyLucene and/or GCJ 4.0 needs to change so that the version of GCJ in Debian Unstable can be used. 3. PyLucene may or may not compile from source with a free software JDK (e.g. free-java-sdk package). Of course, as time passes, the particular version of GCJ and SWIG in Unstable will change. Unless I'm missing something, I don't think making a package for Debian Unstable is possible right now unless everything PyLucene needs is statically linked into its .so. The way Andi currently distributes binary packages won't fly in Debian; at least I can't imagine having the pylucene package provide /usr/lib/libstdc++.so.5 would be okay with folks. FWIW, it is possible to build a package for Debian Stable (Sarge), but that's just b/c Sarge includes all the old stuff PyLucene needs. So I'm at a loss as to what to do, other than wait. Anyone else? -matthew -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#256283: [pylucene-dev] Re: PyLucence Debian Package
Debian GCC team, We're looking at packaging PyLucene for Debian, and are having some build dependency problems with sid. PyLucene currently requires gcj 3.4.x, x =3. However, sid appears to only have gcj 4.x I'm a little surprised to see this - I had thought it likely multiple versions of gcj would be kicking around sid. So my questions are: a) are we overlooking something? b) is this situation likely to be temporary or permanent? The large CC: list is because PyLucene is a kind of tricky to package and upstream is tightly in the loop. -Jeff
Bug#256283: [pylucene-dev] Re: PyLucence Debian Package
Matthew, Regarding the swig incompatibility, I think you are right. It probably is a matter of poking the right people and waiting. However, it is technically possible to compile on Debian stable then upload the binary package to unstable (a.k.a. sid). This can - somewhat - bypass the swig issue and get a working PyLucene package in Debian for one architecture, presumably i386. Once the swig issue gets straightened out, the autobuilders will be able to handle the other architectures. I know this sounds yucky, but it's not that uncommon. Usually the way it goes is a package builds on unstable, gets added to Debian, some build dependency gets updated, then the build breaks, and the package gets a FTBFS (fails to build from source) bug, then someone fixes it. Debian has been dealing with this type of problem for a long time with reasonable success. The gcj situation is more serious. No way in hell can we package the gcj runtime inside the PyLucene package. As far as I can tell this is a showstopper, although I'm curious what the gcj package maintainers have to say about the matter. As for things to do, here's the best list I can think of: - find out what the real story is for getting into backports.org - help make sure the right people have the right information. Is Andi in good contact with the Swig authors already? Do they need more detailed bug reports? - I noticed the package is only valid for python-2.4. Most Debian python packages support multiple versions of python. Maybe this is something to work on while other paths are blocked? - See if our counterparts in RedHat, Gentoo, Suse, etc. have made progress on any of these issues. (Unlikely, I suspect otherwise Andi would know about it). Finally, Matthew I'd like to know your estimated attention span as we consider the possibility of putting a semi-broken package into Debian. My comfort zone for [get it fairly good first] vs [put it in then improve it] shifts a little depending on whether you have short term or long term interest. Jeff
Bug#256283: [pylucene-dev] Re: PyLucence Debian Package
On Sat, 4 Feb 2006, Jeff Breidenbach wrote: The gcj situation is more serious. No way in hell can we package the gcj runtime inside the PyLucene package. As far as I can tell this is a showstopper,although I'm curious what the gcj package maintainers have to say about the matter. Ah, if only I could statically link _PyLucene.so and libgcj.a, but that I've never been able to do. On Windows, this is the only way to get an executable and _PyLucene.pyd is only 4Mb in size. Lucene only needs a small portion of libgcj. - help make sure the right people have the right information. Is Andi in good contact with the Swig authors already? Do they need more detailed bug reports? I got a set of patches that solve the issues on swig 1.3.27 from Robin Dunn, the wxPython maintainer, another big SWIG user (and contributor). I expect his patches to make it into the SWIG codebase in the near future. From what I've heard, though, the upgrade to 1.3.28 is going to be more difficult as a number of changes are in progress for that release. - I noticed the package is only valid for python-2.4. Most Debian python packages support multiple versions of python. Maybe this is something to work on while other paths are blocked? I have no interest in supporting older versions of python. Python 2.3 has had a threading bug fixed that seems important enough. I believe this bug fix made it into Python 2.3.5, if in 2.3 at all. - See if our counterparts in RedHat, Gentoo, Suse, etc. have made progress on any of these issues. (Unlikely, I suspect otherwise Andi would know about it). I only know of the Debian attempt. I build PyLucene on gentoo regularly, and it works fine, but the problems are the same: requirements are gcj 3.4.x (x = 3), Python 2.4, SWIG 1.3.24. Andi.. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#256283: [pylucene-dev] Re: PyLucence Debian Package
Matthew, Given Andi's comments, one possibility is to put the PyLucene package into Debian, but under the contrib section and marking it with appropriate bug entries. The hope would be people could improve the build situation over time. Another possibility - maybe - is to package an older version of PyLucene that depends on Java Lucene 1.4.3. However, I suspect there are likely to be similar issues and an upatched Java Lucene 1.4.3 will not be a viable build dependency either. A third possibility is to simply wait and hope the situation gets better. Since both Java Lucene 1.9 and PyLucene have an rc in their version numbers, this is not a completely crazy idea. On the other hand, it may be a really long wait. A fourth possibility is to modify the Debian PyLucene package such that it first builds the patched Java Lucene .jar from .java files. That's kind of messy, and redundant, but may not have any blockers. I am putting the Debian Java packaging team on the CC: list in case someone there has a suggestion or comment. Officially, the Java Lucene package is jointly maintained by this team. The extended discussion and context for this email is here: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=256283 The requirement to get into Debian main is a complete build from source, using a Free Software toolchain, contolled by the Debian packaging system. Other sections of Debian relax some of these requirements. In any case, I will look over other apects of the package for possible improvements. Jeff
Bug#256283: [pylucene-dev] Re: PyLucence Debian Package
Jeff Breidenbach [EMAIL PROTECTED] said: Given Andi's comments, one possibility is to put the PyLucene package into Debian, but under the contrib section and marking it with appropriate bug entries. The hope would be people could improve the build situation over time. Personally, I'm fine with it going into contrib. I'd like it to go into main but that's probably not feasible unless there's a DFSG-free JDK that could compile it. Another possibility - maybe - is to package an older version of PyLucene that depends on Java Lucene 1.4.3. However, I suspect there are likely to be similar issues and an upatched Java Lucene 1.4.3 will not be a viable build dependency either. Andi, correct me if I am wrong, but I believe PyLucene has always required patches to Lucene's source code. This is the impression I get from the README: http://svn.osafoundation.org/pylucene/trunk/README A third possibility is to simply wait and hope the situation gets better. Since both Java Lucene 1.9 and PyLucene have an rc in their version numbers, this is not a completely crazy idea. On the other hand, it may be a really long wait. We could wait until Lucene 1.9 is official, that'd be fine. However, I don't expect that will change things much since it's my understanding that the issues are mostly with GCJ. Is that right Andi? Andi would know better if waiting for GCJ to mature is a good idea. In Sept. 2004 Andi seemed to be mildly optimistic that gcj 3.5.0 would eliminate the need for patches. See the bottom half of: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=256283#msg25 However, that doesn't seem to have panned out. See this email: http://lists.osafoundation.org/pipermail/pylucene-dev/2006-February/000815.html (I'm assuming here gcc 3.5 is more or less the same as gcc 4.0). A fourth possibility is to modify the Debian PyLucene package such that it first builds the patched Java Lucene .jar from .java files. That's kind of messy, and redundant, but may not have any blockers. Let me see if I have this suggestion right. The pylucene source package would include a patched fork of the Lucene code. Those sources get compiled with Java Lucene's Ant build using a regular 1.4.2 JDK. This produces a .class file that we then compile with GCJ. If that's right then that sounds fine to me. However, if I understand Debian Policy right, that'd would almost certainly mean it'd have to go in contrib. Which, as I said, is fine with me. ... So I'm a little vague on where to go from here. In my opinion your fourth option is the best. If you think it's wise, I could back off and package the PyLucene 1.0.1 tree which is based on the Lucene 1.4.3 sources. That'd side-step the fact that 1.9 is still a release candidate. What do you say? Thanks for taking time to look over the package and give your comments! -matthew -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#256283: [pylucene-dev] Re: PyLucence Debian Package
On Fri, 3 Feb 2006, Matthew O'Connor wrote: Personally, I'm fine with it going into contrib. I'd like it to go into main but that's probably not feasible unless there's a DFSG-free JDK that could compile it. If Java Lucene exists as a Debian package already, then this problem must have been solved there. The Ant/JDK build of Java Lucene in PyLucene is just a regular Java Lucene build using their build procedures, except that some of the .java source files are patched first. Andi, correct me if I am wrong, but I believe PyLucene has always required patches to Lucene's source code. This is the impression I get from the README: Yes, this is correct. The situation is better now than it has been in the past as some of the patches were included into the Java Lucene sources but the remaining ones have no place in the Java Lucene sources and as long as gcj is bleeding edge, such patches are bound to come and go... We could wait until Lucene 1.9 is official, that'd be fine. However, I don't expect that will change things much since it's my understanding that the issues are mostly with GCJ. Is that right Andi? Correct. The current patches are not due to Java Lucene bugs but are due to gcj, gcjh and javacc bugs or issues. Andi would know better if waiting for GCJ to mature is a good idea. In Sept. 2004 Andi seemed to be mildly optimistic that gcj 3.5.0 would eliminate the need for patches. See the bottom half of: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=256283#msg25 However, that doesn't seem to have panned out. See this email: http://lists.osafoundation.org/pipermail/pylucene-dev/2006-February/000815.html (I'm assuming here gcc 3.5 is more or less the same as gcc 4.0). I'm currently recommending gcj 3.4.x, with x = 3 for PyLucene. I've had little luck with gcj 4.x thus far. Let me see if I have this suggestion right. The pylucene source package would include a patched fork of the Lucene code. Those sources get compiled with Java Lucene's Ant build using a regular 1.4.2 JDK. This produces a .class file that we then compile with GCJ. Once there is a Debian Java Lucene 1.9 source package available, the PyLucene 1.9 package could be made to depend on it, unpack it, apply the patches, build it (and not install it) and then build itself. Seems pretty straightforward to me. So I'm a little vague on where to go from here. In my opinion your fourth option is the best. If you think it's wise, I could back off and package the PyLucene 1.0.1 tree which is based on the Lucene 1.4.3 sources. That'd side-step the fact that 1.9 is still a release candidate. What do you say? Doing this with Java Lucene 1.4.3 instead of 1.9 solves only one problem: Java Lucene 1.9 is not a release yet. The vibes I get on the Java Lucene dev list don't make me think that there is any sense of urgency in making such a release happen even though the Java Lucene 1.9 code as it stands now seems very ready for it. Andi.. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#256283: [pylucene-dev] Re: PyLucence Debian Package
I'm dropping the Debian Java package maintainers from the CC: list. Individuals can join in if they want. Andi is correct, Java Lucene 1.4.3 compiles ok with a Free Software toolchain. Specifically we use kaffe. Getting this to work took a lot of time and effort. It is unknown at this time whether the release candidate for Java Lucene 1.9 can be built using kaffe. Once there is a Debian Java Lucene 1.9 source package available, the PyLucene 1.9 package could be made to depend on it, unpack it, apply the patches, build it (and not install it) and then build itself. Seems pretty straightforward to me. Sorry, that's not how it works. Currently Java Lucene 1.4.3 is in Debian. That does not mean the Java Lucene 1.4.3 source code is available to be used as a build dependency. A more likely scenario is what Matthew described - snapshot Java Lucene source code and make that part of the PyLucene source package. Yet Another Option (what are we on now, 5? 6?) is for Debian to patch it's version of Java Lucene 1.9 to be compatible with PyLucene. This only makes sense if the patches are benign - i.e. do not significantly interfere with other users of Java Lucene. For example, the eclipse people would go berserk if we broke their build. Let's try to boil this down into an immediate todo list: 1) Find out if Java Lucene 1.9 can compile with kaffe [JB? Barry? Anyone?] 2) Look for other issues in the current PyLucene package [JB] 3) ??? In any case, my vote is to pick an approach that gets something in Debian sooner rather than later. Programmer attention span is a rare commodity and we should strike while it is hot. Jeff
Bug#256283: [pylucene-dev] Re: PyLucence Debian Package
On Fri, 3 Feb 2006, Jeff Breidenbach wrote: Once there is a Debian Java Lucene 1.9 source package available, the PyLucene 1.9 package could be made to depend on it, unpack it, apply the patches, build it (and not install it) and then build itself. Seems pretty straightforward to me. Sorry, that's not how it works. Currently Java Lucene 1.4.3 is in Debian. That does not mean the Java Lucene 1.4.3 source code is available to be used as a build dependency. That is why I mentioned a dependency on a Java Lucene *source* package. If such a thing doesn't exist (pardon my ignorance) then this option will not work. A more likely scenario is what Matthew described - snapshot Java Lucene source code and make that part of the PyLucene source package. That'd be fine too. Yet Another Option (what are we on now, 5? 6?) is for Debian to patch it's version of Java Lucene 1.9 to be compatible with PyLucene. This only makes sense if the patches are benign - i.e. do not significantly interfere with other users of Java Lucene. For example, the eclipse people would go berserk if we broke their build. At the moment, the patches are not making any API changes nor incompatible class changes although I'm not so sure about the last patch in patches.lucene adding a new private final int to the class definition. Also, there are other patches for the Java Lucene contrib packages in PyLucene such as the regex contrib package which is patched to replace the dependency on a java-based regex processing package with Python's. In any case, I wouldn't assume that PyLucene patches are going to be that safe always. I've had to make API renames in the source code in the past. This makes this fifth option a non starter. Let's try to boil this down into an immediate todo list: 1) Find out if Java Lucene 1.9 can compile with kaffe [JB? Barry? Anyone?] Would compiling Java Lucene with Jikes qualify, or with Eclipse's ecj ? Are these java compilers compatible with Debian's licensing requirements ? Andi.. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#256283: [pylucene-dev] Re: PyLucence Debian Package
Jeff Breidenbach [EMAIL PROTECTED] said: Andi is correct, Java Lucene 1.4.3 compiles ok with a Free Software toolchain. Specifically we use kaffe. Getting this to work took a lot of time and effort. It is unknown at this time whether the release candidate for Java Lucene 1.9 can be built using kaffe. Oh, yes. I thought it did not and that was why it was in contrib. I've finally upgraded one of my spare computers to unstable so I can test for real. Yet Another Option (what are we on now, 5? 6?) is for Debian to patch it's version of Java Lucene 1.9 to be compatible with PyLucene. This only makes sense if the patches are benign - i.e. do not significantly interfere with other users of Java Lucene. For example, the eclipse people would go berserk if we broke their build. As Andi implied, this option seems kind of sketchy and not very future proof. That seems to just be the nature of the beast. Besides, the Eclipse people scare me :) Let's try to boil this down into an immediate todo list: 1) Find out if Java Lucene 1.9 can compile with kaffe [JB? Barry? Anyone?] If it can be compiled with the software stack that gets installed from the free-java-sdk virtual package, is that good enough? That does not appear to install kaffee. I'll play around with trying to get this to work. Andi, when you normally compile the Lucene code which JDK do you use and on which platform (sorry if you answered this elsewhere)? Before I begin I want to verify I can at least reproduce what Andi is doing before I try to attempt a build with a totally free software Java stack. However, don't let my trying stop anyone else from trying or asking around. 2) Look for other issues in the current PyLucene package [JB] I already noticed that unstable has swig 1.2.27, which Andi says will not work. So, that's another issue. :( In any case, my vote is to pick an approach that gets something in Debian sooner rather than later. Programmer attention span is a rare commodity and we should strike while it is hot. Okay, I'll try to build it with a free software java stack. However, if that proves too difficult is building it with a non-free JDK and putting the package in contrib an option? -matthew -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#256283: [pylucene-dev] Re: PyLucence Debian Package
On Fri, 3 Feb 2006, Matthew O'Connor wrote: I'll play around with trying to get this to work. Andi, when you normally compile the Lucene code which JDK do you use and on which platform (sorry if you answered this elsewhere)? Before I begin I want to verify I can at least reproduce what Andi is doing before I try to attempt a build with a totally free software Java stack. I use Apple's JDK on Mac OS X, my main development platform. Andi.. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#256283: [pylucene-dev] Re: PyLucence Debian Package
On Fri, 3 Feb 2006, Jeff Breidenbach wrote: I use Apple's JDK on Mac OS X, my main development platform. Oh yeah. There is one more technical factor to add into the mix, hopefully people's heads aren't ready to explode. Debian supports a lot of architecture, I think the current number is 11. There are autobuilders for all of these architectures, but they only have access to Free Software. They don't have access to - say - the Apple or Sun JDK. Clearly. That is why I was asking if jikes or the eclipse java compiler were considered free software by Debian. * Matthew's current PyLucene package sidesteps this issue and should build on all architectures. * If we usse an entirely Free Software toolchain, it should also build on all architectures. I don't exactly know what you mean by '11 architectures' but if these are strange combinations of chips, kernels and the like, you should first make sure gcj is functional on these 11 architectures. I routinely build PyLucene on Mac OS X, Windows (with mingw) and Gentoo linux (32 bit). I also know of a user of PyLucene who seems to be able to use it on FreeBSD. Any other architecture such as 64 bit OSs, etc.. are even more bleeding edge Andi.. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#256283: [pylucene-dev] Re: PyLucence Debian Package
By architectures I mean i386, powerpc, sparc, S390, mips, m68k, AMD64, etc. http://www.debian.org/ports/ you should first make sure gcj is functional on these 11 architectures. No real need to check ahead of time - gcj will either work on a given architecture or it won't. We just want to be in a position where autobuilders can at least try. That way if gcj improves on some architectures over time, the architecture-specific package can go through another build attempt - automatically - in the future.
Bug#256283: [pylucene-dev] Re: PyLucence Debian Package
On Thu, 2 Feb 2006, Jeff Breidenbach wrote: Hi Matthew, Good work on the package. However, I don't like that it starts with Java bytecode instead of canonical source code. Do you think it would be possible to have the PyLucene package use the Java Lucene package as a build dependency? With Lucene 1.9 I moved PyLucene to building from a svn checkout of Lucene java sources. gcj is much better at compiling .class files than .java files however, so I use the Java Lucene Ant build and a regular 1.4.2 JDK for the .java to .class step and release a PyLucene 'source' tarball that includes the Java Lucene .class files so that Ant and a JDK are not normally required to build PyLucene unless one wants to build from the very latest sources in the Java Lucene subversion repository. Last time I tried to compile Java Lucene from .java source files with gcj I ended up making 14 patches. There were a number of problems, in particular, with compiling anonymous inner classes. Experience has shown that compiling .class files to .o files with gcj is more likely to succeed. Hence, I chose the path of least resistance since releasing a PyLucene 'source' tarball with Java Lucene .class files seems to be good enough. Using the Java Lucene package as a dependency might work once there are no source patches to Java Lucene anymore. At this time, I still need to apply 4 patches to the Java Lucene .java sources to work around some gcj compilation or runtime issues before I can feed them to Ant and gcj. Here is the detail on the patches, in their order of occurrence in PyLucene's patches.lucene file: - there is a bug in gcj that causes it to produces wrong code when the two local stack variable named 'required' and 'prohibited' are not initialized. According to the logic in the java source code, they don't need to be but their values will swap at some point and lead to errors if they're not. - gcjh cannot deal with a static method and a static field having the same name. I filed a bug last year with javacc which is responsible for generating such unhappy code but no fix has come forward thus far. - the code relying on a NullPointerException in FieldInfos.java causes the runtime to crash. Clearly a bug in libgcj or the resulting object code but the java code is better written to not rely on the exception anyway. - declaring MAX_BBUF that way worked around another code generation bug of gcj's for which I don't remember the details at the moment. These patches were made as needed using gcj 3.4.4 Several other patches were removed after the corresponding changes were made in the Java Lucene source code (for instance, the workaround for gcj infamous bug http://gcc.gnu.org/bugzilla/show_bug.cgi?id=15411). Andi.. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
