Re: [gentoo-dev] OpenLDAP 2.3.x removal on October 27, migrate to 2.4.x
On Tue, Oct 14, 2014 at 07:52:03AM +0800, Patrick Lauer wrote: On 10/14/14 05:22, Robin H. Johnson wrote: For compatibility and migration support, we've kept the old OpenLDAP 2.3.x ebuilds in the tree for nearly 5 years. And you better keep them for a while, because some of us are stuck with 2.3, and mixed operation (e.g. master 2.4, slaves 2.3) is not supported. Since for example CentOS 5 is still around and there's no upgrade path, well, some people like me still have to use 2.3 ... CentOS6/7 don't resolve that issue for you? So far you're one user, but that's enough to warrant a news item. I would really like to get the old stuff out of the tree, it's been around a very long time, has security issues, and the 2.3 ebuilds don't comply with newer EAPIs and QA practices (I suspect that probably won't even compile with newer berkdb and gnutls). -- Robin Hugh Johnson Gentoo Linux: Developer, Infrastructure Lead E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
[gentoo-dev] Re: new virtual: virtual/podofo-build
On 10/13/2014 12:46 PM, Zac Medico wrote: Hi, In order to solve bug #503802 [1], I would like to add a virtual/podofo-build package to pull in app-text/podofo and dev-libs/boost. Then packages like app-text/calibre can put virtual/podofo-build in DEPEND and app-text/podofo in RDEPEND. The advantage of this approach is that it makes it possible to use a command like `emerge --depclean --with-bdeps=n` to remove the build-time only boost package (and virtual/podofo-build), since boost is only needed for build-time headers. There may be some other possible ways to specify the dependency, but this approach is the most attractive one that I've seen. In fact, this approach is basically identical to the Virtual for C++ tr1 type_traits example that's given in the dev-manual [2]. Would anyone like to suggest improvements to this idea, alternatives, or raise any objections? [1] https://bugs.gentoo.org/show_bug.cgi?id=503802 [2] http://devmanual.gentoo.org/general-concepts/virtuals/ The proposed virtual/podofo-build ebuild is attached. -- Thanks, Zac # Copyright 1999-2014 Gentoo Foundation # Distributed under the terms of the GNU General Public License v2 # $Header: $ EAPI=5 DESCRIPTION=Virtual package for building against PoDoFo HOMEPAGE= SRC_URI= LICENSE= SLOT=0 KEYWORDS=amd64 ~arm hppa ppc ppc64 ~sparc x86 IUSE=+boost idn debug test # Pull in boost for build-against header dependency (see bug #503802). RDEPEND= ~app-text/podofo-0.9.2[boost=,idn=,debug=,test=] boost? ( dev-libs/boost ) DEPEND=
Re: [gentoo-dev] new virtual: virtual/podofo-build
On 13/10/14 03:46 PM, Zac Medico wrote: Hi, In order to solve bug #503802 [1], I would like to add a virtual/podofo-build package to pull in app-text/podofo and dev-libs/boost. Then packages like app-text/calibre can put virtual/podofo-build in DEPEND and app-text/podofo in RDEPEND. The advantage of this approach is that it makes it possible to use a command like `emerge --depclean --with-bdeps=n` to remove the build-time only boost package (and virtual/podofo-build), since boost is only needed for build-time headers. There may be some other possible ways to specify the dependency, but this approach is the most attractive one that I've seen. In fact, this approach is basically identical to the Virtual for C++ tr1 type_traits example that's given in the dev-manual [2]. Would anyone like to suggest improvements to this idea, alternatives, or raise any objections? [1] https://bugs.gentoo.org/show_bug.cgi?id=503802 [2] http://devmanual.gentoo.org/general-concepts/virtuals/ I feel like there should be a DEPEND specifier for packages required to build against this package. For example, xproto is required to build against SDL (at least using pkg-config), but not to simply use it at runtime. This applies even if one does not directly use xproto headers. It would not be sufficient to specify that DEPEND includes the DEPENDs and RDEPENDs of the dependencies, since then it would be impossible to specify build dependencies that only require the runtime of the dependee, like most build tools. It seems like a poor solution to have to create virtuals for every package that requires such an arrangement. signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] new virtual: virtual/podofo-build
On 10/14/2014 03:56 PM, Alex Xu wrote: On 13/10/14 03:46 PM, Zac Medico wrote: Hi, In order to solve bug #503802 [1], I would like to add a virtual/podofo-build package to pull in app-text/podofo and dev-libs/boost. Then packages like app-text/calibre can put virtual/podofo-build in DEPEND and app-text/podofo in RDEPEND. The advantage of this approach is that it makes it possible to use a command like `emerge --depclean --with-bdeps=n` to remove the build-time only boost package (and virtual/podofo-build), since boost is only needed for build-time headers. There may be some other possible ways to specify the dependency, but this approach is the most attractive one that I've seen. In fact, this approach is basically identical to the Virtual for C++ tr1 type_traits example that's given in the dev-manual [2]. Would anyone like to suggest improvements to this idea, alternatives, or raise any objections? [1] https://bugs.gentoo.org/show_bug.cgi?id=503802 [2] http://devmanual.gentoo.org/general-concepts/virtuals/ I feel like there should be a DEPEND specifier for packages required to build against this package. For example, xproto is required to build against SDL (at least using pkg-config), but not to simply use it at runtime. This applies even if one does not directly use xproto headers. It would not be sufficient to specify that DEPEND includes the DEPENDs and RDEPENDs of the dependencies, since then it would be impossible to specify build dependencies that only require the runtime of the dependee, like most build tools. We can add that in a future EAPI: https://bugs.gentoo.org/show_bug.cgi?id=392239 It seems like a poor solution to have to create virtuals for every package that requires such an arrangement. Well, it's not quite as convenient as having an EAPI extension that's specifically designed for that purpose. However, using virtuals gives us the desired result without having to wait for a new EAPI. -- Thanks, Zac
Re: [gentoo-dev] Removing a blocker from a stable package
On 10/13/2014 02:41 PM, Mike Gilbert wrote: I agree with Diego and Ralph: Go with d. repoman will generate a warning (not an error) about a dependency which does not exist, but this is safe to ignore. Given that (d) didn't require me to do anything else, I just went ahead and removed app-doc/djbdns-man. Repoman doesn't even warn. Huh.
Re: [gentoo-dev] Removing a blocker from a stable package
On 10/14/14 20:25, Michael Orlitzky wrote: On 10/13/2014 02:41 PM, Mike Gilbert wrote: I agree with Diego and Ralph: Go with d. repoman will generate a warning (not an error) about a dependency which does not exist, but this is safe to ignore. Given that (d) didn't require me to do anything else, I just went ahead and removed app-doc/djbdns-man. Repoman doesn't even warn. Huh. I tested after the last round of emails and found the same. Let us know if something else complains. -- Anthony G. Basile, Ph.D. Gentoo Linux Developer [Hardened] E-Mail: bluen...@gentoo.org GnuPG FP : 1FED FAD9 D82C 52A5 3BAB DC79 9384 FA6E F52D 4BBA GnuPG ID : F52D4BBA
Re: [gentoo-dev] OpenLDAP 2.3.x removal on October 27, migrate to 2.4.x
On Tuesday 14 October 2014 16:02:20 Robin H. Johnson wrote: On Tue, Oct 14, 2014 at 07:52:03AM +0800, Patrick Lauer wrote: On 10/14/14 05:22, Robin H. Johnson wrote: For compatibility and migration support, we've kept the old OpenLDAP 2.3.x ebuilds in the tree for nearly 5 years. And you better keep them for a while, because some of us are stuck with 2.3, and mixed operation (e.g. master 2.4, slaves 2.3) is not supported. Since for example CentOS 5 is still around and there's no upgrade path, well, some people like me still have to use 2.3 ... CentOS6/7 don't resolve that issue for you? Not an upgrade ... And if I have to reinstall from scratch then it'd be Gentoo, but that's a rather brutal amount of work since there's about 8 years of new features, changes in config files and so on (still on my todo list, but there's other fires burning more brightly) So far you're one user, but that's enough to warrant a news item. I would really like to get the old stuff out of the tree, it's been around a very long time, has security issues, and the 2.3 ebuilds don't comply with newer EAPIs and QA practices (I suspect that probably won't even compile with newer berkdb and gnutls). Well, I'm also stuck with Samba 3.5 because upstream managed to break group permissions with LDAP in really hilarious ways. I don't have a strong objection to having to keep oldcruft in an overlay, but (like Samba) it'd be quite acceptable to have these ebuilds masked indefinitely
Re: [gentoo-dev] new virtual: virtual/podofo-build
On Tue, Oct 14, 2014 at 6:56 PM, Alex Xu alex_y...@yahoo.ca wrote: I feel like there should be a DEPEND specifier for packages required to build against this package. For example, xproto is required to build against SDL (at least using pkg-config), but not to simply use it at runtime. This applies even if one does not directly use xproto headers. It would not be sufficient to specify that DEPEND includes the DEPENDs and RDEPENDs of the dependencies, since then it would be impossible to specify build dependencies that only require the runtime of the dependee, like most build tools. I thought that a package could only assume that the direct DEPENDs were installed anyway. Those DEPENDs could only assume that their immediate RDEPENDs are installed when they're used to build something else. So, if the podofo-build virtual listed boost in its RDEPENDs that would be fine. However, it would seem simpler to me to just list boost in the calibre DEPENDs. I'm not really opposed to having a special reverse build DEPEND variable of some kind, but does it really add that much value? If I ever got around to building that portage sandbox extension that blocked read access to anything that wasn't in DEPENDS, or RDEPENDs of DEPENDs, and so on, or @system, then it would detect these kinds of situations. -- Rich
Re: [gentoo-dev] new virtual: virtual/podofo-build
On 10/14/2014 06:20 PM, Rich Freeman wrote: So, if the podofo-build virtual listed boost in its RDEPENDs that would be fine. However, it would seem simpler to me to just list boost in the calibre DEPENDs. Yeah, boost in the calibre DEPENDs satisfies my `emerge --depclean --with-bdeps=n` requirement, so I guess KISS wins here. -- Thanks, Zac
Re: [gentoo-dev] new virtual: virtual/podofo-build
On 10/14/2014 06:42 PM, Zac Medico wrote: On 10/14/2014 06:20 PM, Rich Freeman wrote: So, if the podofo-build virtual listed boost in its RDEPENDs that would be fine. However, it would seem simpler to me to just list boost in the calibre DEPENDs. Yeah, boost in the calibre DEPENDs satisfies my `emerge --depclean --with-bdeps=n` requirement, so I guess KISS wins here. With some investigation, I've realized that podofo has a USE flag which allows the boost dependency to be disabled. So, it would be unfortunate if calibre pulled in boost when it wasn't needed. Given this issue, the virtual seems more appealing. -- Thanks, Zac
[gentoo-portage-dev] [RFC] Package description index file format for faster emerge search actions
Hi,
As we all know, emerge --search/--searchdesc actions are embarrassingly
slow (from most users' perspectives, anyway), especially in comparison
to external tools like eix and esearch.
Wouldn't it be nice if the performance of emerge's search functionality
was more competitive with other offerings? Then, external search tools
might not be seen as an absolute necessity.
In order to solve this problem, I suggest that we add support for a
package description index file format. For example, the attached script
will generate a suitable index formatted as series of lines like this:
sys-apps/sandbox-1.6-r2,2.3-r1,2.4,2.5,2.6-r1: sandbox'd LD_PRELOAD hack
Using this format, the index file for the entire gentoo-x86 repository
consumes approximately 1.5 MB. The whole file can be quickly searched as
a stream (the whole file need not be in memory at once), yielding emerge
--search/--searchdesc performance that will be competitive with
app-portage/esearch.
The index can either be generated on the server side by egencache, or on
the client side by a post emerge --sync hook. It makes sense to support
both modes of operation, so that server side generation is purely optional.
What do others think about this proposal?
--
Thanks,
Zac
#!/usr/bin/env python
import os
import sys
import portage
from portage.versions import _pkg_str
usage = usage: %s repo\n % os.path.basename(sys.argv[0])
def main(args):
if len(args) != 1:
sys.stderr.write(usage)
return 1
repo_name = args[0]
repo_info = portage.settings.repositories.prepos.get(repo_name)
if repo_info is None:
sys.stderr.write(unknown repo: %s\n % repo_name)
return 1
portdb = portage.db[portage.root][porttree].dbapi
portdb.porttrees = [repo_info.location]
f = sys.stdout
if sys.hexversion = 0x300:
f = f.buffer
class duplicates(object):
cp = None
desc = None
pkgs = []
def flush_duplicates():
if duplicates.pkgs:
if len(duplicates.pkgs) == 1:
output = %s: %s\n % (duplicates.pkgs[0],
duplicates.desc)
else:
output = %s,%s: %s\n % (duplicates.pkgs[0],
,.join(pkg.version
for pkg in duplicates.pkgs[1:]), duplicates.desc)
f.write(output.encode('utf_8'))
del duplicates.pkgs[:]
for cp in portdb.cp_all():
for cpv in portdb.cp_list(cp):
desc, = portdb.aux_get(cpv, [DESCRIPTION])
if duplicates.cp != cp or duplicates.desc != desc:
flush_duplicates()
duplicates.cp = cp
duplicates.desc = desc
duplicates.pkgs.append(_pkg_str(cpv))
flush_duplicates()
return os.EX_OK
if __name__ == '__main__':
sys.exit(main(sys.argv[1:]))
[gentoo-portage-dev] Re: [RFC] Package description index file format for faster emerge search actions
Zac Medico posted on Tue, 14 Oct 2014 00:40:21 -0700 as excerpted: I suggest that we add support for a package description index file format. For example, the attached script will generate a suitable index formatted as series of lines like this: sys-apps/sandbox-1.6-r2,2.3-r1,2.4,2.5,2.6-r1: sandbox'd LD_PRELOAD hack What about homepage? An index for it too? (I found myself so frequently esearching for homepage that I hacked up a script that greps package names and homepages from the esearch results.) -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman
Re: [gentoo-portage-dev] Re: [RFC] Package description index file format for faster emerge search actions
On 10/14/2014 02:53 AM, Duncan wrote: Zac Medico posted on Tue, 14 Oct 2014 00:40:21 -0700 as excerpted: I suggest that we add support for a package description index file format. For example, the attached script will generate a suitable index formatted as series of lines like this: sys-apps/sandbox-1.6-r2,2.3-r1,2.4,2.5,2.6-r1: sandbox'd LD_PRELOAD hack What about homepage? An index for it too? (I found myself so frequently esearching for homepage that I hacked up a script that greps package names and homepages from the esearch results.) The intent is to use the package name/version as a foreign key that links an entry in the description index to an entry in one of portage's other caches. For example, the other cache could be the cache located inside the $repository/metadata/md5-cache directory (which contains the HOMEPAGE metadata that you are interested in). So, the description index serves as a fast way to initially locate a package by name or description. Then the foreign key is used to efficiently pull specific metadata from the $repository/metadata/md5-cache directory. -- Thanks, Zac
[gentoo-portage-dev] Re: [RFC] Package description index file format for faster emerge search actions
Zac Medico posted on Tue, 14 Oct 2014 03:05:35 -0700 as excerpted: On 10/14/2014 02:53 AM, Duncan wrote: Zac Medico posted on Tue, 14 Oct 2014 00:40:21 -0700 as excerpted: I suggest that we add support for a package description index file format. What about homepage? An index for it too? The intent is to use the package name/version as a foreign key that links an entry in the description index to an entry in one of portage's other caches. Thanks. Makes sense now. =:^) -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman
Re: [gentoo-portage-dev] Re: [RFC] Package description index file format for faster emerge search actions
On 10/14/2014 02:53 AM, Duncan wrote:
What about homepage? An index for it too?
If we really want to index the homepage, then a more extensible format
might be better. For example, each line of the index could be a JSON
object like this:
{description: sandbox'd LD_PRELOAD hack, homepage:
http://www.gentoo.org/proj/en/portage/sandbox/;, package_versions:
sys-apps/sandbox-1.6-r2,2.3-r1,2.4,2.5,2.6-r1}
The attached script generates this JSON lines format.
--
Thanks,
Zac
#!/usr/bin/env python
import json
import os
import sys
import portage
from portage.versions import _pkg_str
usage = usage: %s repo\n % os.path.basename(sys.argv[0])
def main(args):
if len(args) != 1:
sys.stderr.write(usage)
return 1
repo_name = args[0]
repo_info = portage.settings.repositories.prepos.get(repo_name)
if repo_info is None:
sys.stderr.write(unknown repo: %s\n % repo_name)
return 1
json_write_opts = {
ensure_ascii: False,
sort_keys: True
}
portdb = portage.db[portage.root][porttree].dbapi
portdb.porttrees = [repo_info.location]
f = sys.stdout
if sys.hexversion = 0x300:
f = f.buffer
class duplicates(object):
cp = None
desc = None
homepage = None
pkgs = []
def flush_duplicates():
if duplicates.pkgs:
if len(duplicates.pkgs) == 1:
package_versions = duplicates.pkgs[0]
else:
package_versions = %s,%s % (duplicates.pkgs[0],
,.join(pkg.version
for pkg in duplicates.pkgs[1:]))
obj = {
package_versions: package_versions,
description: duplicates.desc,
homepage: duplicates.homepage
}
f.write(json.dumps(obj, **json_write_opts).encode('utf_8'))
f.write(b'\n')
del duplicates.pkgs[:]
for cp in portdb.cp_all():
for cpv in portdb.cp_list(cp):
desc, homepage = portdb.aux_get(cpv,
[DESCRIPTION, HOMEPAGE])
if duplicates.cp != cp or \
duplicates.desc != desc or \
duplicates.homepage != homepage:
flush_duplicates()
duplicates.cp = cp
duplicates.desc = desc
duplicates.homepage = homepage
duplicates.pkgs.append(_pkg_str(cpv))
flush_duplicates()
return os.EX_OK
if __name__ == '__main__':
sys.exit(main(sys.argv[1:]))
[gentoo-portage-dev] Re: [RFC] Package description index file format for faster emerge search actions
Zac Medico zmed...@gentoo.org wrote:
If we really want to index the homepage, then a more extensible format
might be better. For example, each line of the index could be a JSON
object like this:
{description: sandbox'd LD_PRELOAD hack, homepage:
http://www.gentoo.org/proj/en/portage/sandbox/;, package_versions:
sys-apps/sandbox-1.6-r2,2.3-r1,2.4,2.5,2.6-r1}
...and when you also add some other data like LICENSE, IUSE, KEYWORDS,
SLOT and (optionally) {P,R,}DEPEND, you essentially end up with the
eix database (/var/cache/eix/portage.eix generated by eix-update).
The only difference is that the eix database is even somewhat more
optimized for searching (so that e.g. the homepage and license data
can be skipped without reading when you are looking for the description)
and slightly compressed (common words are in a dictionary).
It would make sense that portage and eix agree on a common database
instead of caching essentially the same information twice.
This appears especially important in view of things like eix-remote
where this data is merged for various overlays without downloading
the overlays.
You can find the description of the current format of the eix database
in eix-db.html when you emerge eix with USE=doc.
Re: [gentoo-portage-dev] Re: [RFC] Package description index file format for faster emerge search actions
On 10/14/2014 09:27 AM, Martin Vaeth wrote:
Zac Medico zmed...@gentoo.org wrote:
If we really want to index the homepage, then a more extensible format
might be better. For example, each line of the index could be a JSON
object like this:
{description: sandbox'd LD_PRELOAD hack, homepage:
http://www.gentoo.org/proj/en/portage/sandbox/;, package_versions:
sys-apps/sandbox-1.6-r2,2.3-r1,2.4,2.5,2.6-r1}
...and when you also add some other data like LICENSE, IUSE, KEYWORDS,
SLOT and (optionally) {P,R,}DEPEND, you essentially end up with the
eix database (/var/cache/eix/portage.eix generated by eix-update).
After some thought, I'd prefer to stick with the simpler non-extensible
format described in my first email. Reasons include:
1) The package names and descriptions are, by far, the most commonly
searched items. So, for general use, emerge --search/--searchdesc
actions should be sufficient for most users. More advanced queries are
better suited to something like eix-db or sqlite, but the majority of
users have negligible interest in performing such advanced queries, so
it's hard to justify distributing a relatively large binary database
inside the package repository (it puts extra load on the rsync servers).
So, I think it's better to generate such databases on the client side,
using $repository/metadata/md5-cache as a source when available.
2) A plain text index, like the one I originally suggested, is small
enough (1.5 MB for current gentoo-x86) so that the additional load it
puts on the rsync servers should be manageable. Also, for repositories
distributed via a vcs such as git, changes to the plain text index will
transfer efficiently (only differences are transferred).
The only difference is that the eix database is even somewhat more
optimized for searching (so that e.g. the homepage and license data
can be skipped without reading when you are looking for the description)
and slightly compressed (common words are in a dictionary).
It would make sense that portage and eix agree on a common database
instead of caching essentially the same information twice.
This appears especially important in view of things like eix-remote
where this data is merged for various overlays without downloading
the overlays.
I agree, but as mentioned above, I think it's better to generate such a
database on the client side, using $repository/metadata/md5-cache as a
source when available.
You can find the description of the current format of the eix database
in eix-db.html when you emerge eix with USE=doc.
Thanks for the info. I've reviewed the specification, and it looks like
a nice format. However, if we're going to create a shared database that
suits the needs of both portage and eix, then I would prefer to use a
general-purpose RDBMS such as sqlite (sqlightning looks interesting,
btw). Why should be go to the trouble of developing and maintaining a
special-purpose RDBMS, even though a general-purpose RDBMS suits our needs?
--
Thanks,
Zac
