Re: [PROPOSAL] Ports using SCM repositories as source instead of distfiles
* Jeremy Messenger (me...@cox.net) wrote: I mean by keep static pkg-plist (not get rid of it). I don't like dynamic plist. I do not need to list my reasons as it's already in the archives by someone and maybe me (not remember). I understand your point, but it's only theoretical research for now. If we manage to get soemthing useful it may be used optionally, may be used only for generating static plists etc. -- Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D amd...@amdmi3.ru ..: jabber: amd...@jabber.ruhttp://www.amdmi3.ru ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: [PROPOSAL] Ports using SCM repositories as source instead of distfiles
* Garrett Cooper (yanef...@gmail.com) wrote: It's useful for me too sometimes, but it's already not 100% reliable, as some ports still generate it dynamically, some use PLIST_FILES, some use PLIST_SUB so it's not apparent under which path and/or name the file mentioned in plist is actually installed. Getting rid of the plist files will force a major rewrite of pkg_install. Are you really comfortable doing that :\? Not getting rid but generating automatically. Many ports already have no pkg-plist. -- Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D amd...@amdmi3.ru ..: jabber: amd...@jabber.ruhttp://www.amdmi3.ru ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: [PROPOSAL] Ports using SCM repositories as source instead of distfiles
On Wed, 17 Dec 2008 12:47:56 -0600, Dmitry Marakasov amd...@amdmi3.ru wrote: * Garrett Cooper (yanef...@gmail.com) wrote: It's useful for me too sometimes, but it's already not 100% reliable, as some ports still generate it dynamically, some use PLIST_FILES, some use PLIST_SUB so it's not apparent under which path and/or name the file mentioned in plist is actually installed. Getting rid of the plist files will force a major rewrite of pkg_install. Are you really comfortable doing that :\? Not getting rid but generating automatically. Many ports already have no pkg-plist. I mean by keep static pkg-plist (not get rid of it). I don't like dynamic plist. I do not need to list my reasons as it's already in the archives by someone and maybe me (not remember). Cheers, Mezz -- me...@cox.net - m...@freebsd.org FreeBSD GNOME Team http://www.FreeBSD.org/gnome/ - gn...@freebsd.org ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: [PROPOSAL] Ports using SCM repositories as source instead of distfiles
In aeac14f0-5f2d-462b-94a9-0f859f87e...@gmail.com, Garrett Cooper wrote: On Dec 9, 2008, at 10:13 AM, Dmitry Marakasov wrote: It's not like your proposal is bad, ports instantaneously tracking upstream changes and not needing maintainers would really be cool, but unfortunately that's practically impossible. Some software projects are switching over to this kind of a model, like glibc. What kind of model ? You mean using repositories instead of release tarballs, hmm..? -- Ashish Shukla pgpWV5NKyVVmz.pgp Description: PGP signature
Re: [PROPOSAL] Ports using SCM repositories as source instead of distfiles
Dmitry Marakasov writes: * Ashish Shukla आशीष शुक्ल (wahjava...@gmail.com) wrote: No. If there's DESTDIR, you don't need to log writes to it, as you can just use `find` to see what's there after installation as in quote below. The idea is that if we log all writes to the fulesystem during port installation, we don't need intermediate directory at all. Okay, so if we go with your approach, then can you tell me how do you plan to go about logging writes to filesystem with your .so which you're planning to preload as you mentioned in one of your mails in this thread ? The idea is to redefine functions that execute syscalls which will modify the filesystem add add loging there. Here's a test code http://www.amdmi3.ru/files/ldpreload.c. Compile it into shared lib and run `LD_PRELOAD=path_to_lib.so someapp`. Cool, this works great :) . It turned out to be not the best solution as it relies on environment heavily, and some apps may clear the environment (scons does this by default), or call syscalls via syscall(), instead of libc wrapper, thus it's not reliable enough. Another way is ptrace, which should intercept syscalls without fail. So, we're going with ptrace way, lets hope this way is foolproof :) . Thanks -- Ashish Shukla pgp4YUYqe8cYg.pgp Description: PGP signature
Re: [PROPOSAL] Ports using SCM repositories as source instead of distfiles
* Ashish Shukla आशीष शुक्ल (wahjava...@gmail.com) wrote: No. If there's DESTDIR, you don't need to log writes to it, as you can just use `find` to see what's there after installation as in quote below. The idea is that if we log all writes to the fulesystem during port installation, we don't need intermediate directory at all. Okay, so if we go with your approach, then can you tell me how do you plan to go about logging writes to filesystem with your .so which you're planning to preload as you mentioned in one of your mails in this thread ? The idea is to redefine functions that execute syscalls which will modify the filesystem add add loging there. Here's a test code http://www.amdmi3.ru/files/ldpreload.c. Compile it into shared lib and run `LD_PRELOAD=path_to_lib.so someapp`. It turned out to be not the best solution as it relies on environment heavily, and some apps may clear the environment (scons does this by default), or call syscalls via syscall(), instead of libc wrapper, thus it's not reliable enough. Another way is ptrace, which should intercept syscalls without fail. -- Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D amd...@amdmi3.ru ..: jabber: amd...@jabber.ruhttp://www.amdmi3.ru ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: [PROPOSAL] Ports using SCM repositories as source instead of distfiles
On Fri, Dec 12, 2008 at 2:29 AM, Dmitry Marakasov amd...@amdmi3.ru wrote: * Jeremy Messenger (me...@cox.net) wrote: So if I understand correctly, you're proposing to only use dynamic plist generation for the ports that support it without modification, i.e. autotools-based? My opinion is that we should support the feature for all ports, or don't support it at all. Only getting rid of ~5k pkg-plists is not a huge accomplishment considering the mess it causes and I doubt it's worth the work on adding the feature to port.mk and then rebuilding and testing all affected ports. Being able to forget about pkg-plists once and forever however would be a huge accomplishment and if that's possible it should be done sooner or later. I object on get rid of pkg-plist. I depend on pkg-plist too much. I think it's important for us to keep on track where the files/directories are. It's useful for me too sometimes, but it's already not 100% reliable, as some ports still generate it dynamically, some use PLIST_FILES, some use PLIST_SUB so it's not apparent under which path and/or name the file mentioned in plist is actually installed. Getting rid of the plist files will force a major rewrite of pkg_install. Are you really comfortable doing that :\? -Garrett ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: [PROPOSAL] Ports using SCM repositories as source instead of distfiles
* Ashish Shukla आशीष शुक्ल (wahjava...@gmail.com) wrote:
Yes, that is why I mentioned having a variable which enables this
behaviour, by default it is disabled. I mean ports which are okay
with providing static plists are fine, but ports which aren't
predictable with what files are going to installed can go with this
dynamic plist support, where ports infrastructure will only help in
generating a plist from an already setup directory tree
(/var/tmp/${portname}), now it is maintainer's responsibility to make
sure that all files will be installed in /var/tmp/${portname} which
{s,}he can do by either using 'make install DESTDIR=/var/tmp/${portname}'
or something similar if supported by port's upstream or {s,}he has to
add installation commands in ports Makefile rather than going with
upstream's way of installing things.
I really don't see any point in supporting this only in individual
ports. If you can do something, you expect (and should be able)
to do it with any and all ports.
See
http://lists.freebsd.org/pipermail/freebsd-ports/2006-August/034745.html
those are some real examples of complexity and resulting confusion,
from first variant of DESTDIR support in ports. Now, when we have
one DESTDIR implementation, adding another will likely make some heads
explode, just think of variable naming.
The DESTDIR issue in above link refers to the DESTDIR support[1] present in
FreeBSD Ports system, and the one which I'm talking about has nothing to
do with that.
Yes, I just meant that that'll bring too much confusion.
I'll remind that what we are talking about is automatic plist generation,
and I think that this can be done without any hacks like installing a
port into intermediate directory before real installation just by
logging all writes to the filesystem.
Yes that intermediate directory is what DESTDIR is. And if you're
capable of logging all writes in the DESTDIR, then its cool, but
No. If there's DESTDIR, you don't need to log writes to it, as you
can just use `find` to see what's there after installation as in
quote below. The idea is that if we log all writes to the fulesystem
during port installation, we don't need intermediate directory at
all.
remember you're also talking about installing port in an intermediate
directory. After the port gets installed in intermediate directory, the
plist can be generated with your filesystem writes logger component or a
well tested version of following simply command line:
% find /var/tmp/${PORTNAME} -type f |sed -e \
s[/var/tmp/${PORTNAME}${PREFIX}/[[g plist.tmp
--
Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D
amd...@amdmi3.ru ..: jabber: amd...@jabber.ruhttp://www.amdmi3.ru
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: [PROPOSAL] Ports using SCM repositories as source instead of distfiles
* Ashish Shukla आशीष शुक्ल ([EMAIL PROTECTED]) wrote: This is what Debian and Gentoo does. Remember we don't have to pass DESTDIR variable to 'make -C /usr/ports/editors/emacs-cvs' instead it will be passed to the 'gmake' process invoked by port's Makefile. If we I understand. But you're implying that there is Makefile and it supports DESTDIR. As I understand, you're referring to autotools-based ports. Remember, those are less than 1/4 of the collection. pass DESTDIR to port's commandline, then it will install all dependencies in that chroot which is not desired, we simply care about the files installed by that port. Since there're already 20,000 ports we can't do it by default, so we've to hack some knob (like REQUIRES_DYNAMIC_INSTALLATION) which if defined will enable this behaviour. So if I understand correctly, you're proposing to only use dynamic plist generation for the ports that support it without modification, i.e. autotools-based? My opinion is that we should support the feature for all ports, or don't support it at all. Only getting rid of ~5k pkg-plists is not a huge accomplishment considering the mess it causes and I doubt it's worth the work on adding the feature to port.mk and then rebuilding and testing all affected ports. Being able to forget about pkg-plists once and forever however would be a huge accomplishment and if that's possible it should be done sooner or later. -- Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D [EMAIL PROTECTED] ..: jabber: [EMAIL PROTECTED]http://www.amdmi3.ru ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: [PROPOSAL] Ports using SCM repositories as source instead of distfiles
On Thu, Dec 11, 2008 at 12:23 AM, Dmitry Marakasov [EMAIL PROTECTED] wrote: * Ashish Shukla आशीष शुक्ल ([EMAIL PROTECTED]) wrote: This is what Debian and Gentoo does. Remember we don't have to pass DESTDIR variable to 'make -C /usr/ports/editors/emacs-cvs' instead it will be passed to the 'gmake' process invoked by port's Makefile. If we I understand. But you're implying that there is Makefile and it supports DESTDIR. As I understand, you're referring to autotools-based ports. Remember, those are less than 1/4 of the collection. pass DESTDIR to port's commandline, then it will install all dependencies in that chroot which is not desired, we simply care about the files installed by that port. Since there're already 20,000 ports we can't do it by default, so we've to hack some knob (like REQUIRES_DYNAMIC_INSTALLATION) which if defined will enable this behaviour. So if I understand correctly, you're proposing to only use dynamic plist generation for the ports that support it without modification, i.e. autotools-based? My opinion is that we should support the feature for all ports, or don't support it at all. Only getting rid of ~5k pkg-plists is not a huge accomplishment considering the mess it causes and I doubt it's worth the work on adding the feature to port.mk and then rebuilding and testing all affected ports. Being able to forget about pkg-plists once and forever however would be a huge accomplishment and if that's possible it should be done sooner or later. -- Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D [EMAIL PROTECTED] ..: jabber: [EMAIL PROTECTED]http://www.amdmi3.ru Agreed. I've come across many ports where folks haven't written Makefiles properly and it results in problems . Most cases you can make the following assumption: if a port can be cross-compiled, it can most likely be installed under another destination directory. The converse is not necessarily true. These are issues which should be documented in a Makefile FAQ for everyone, and this should be noted whenever an issue is found so that folks can be educated and things like this can be fixed. -Garrett ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: [PROPOSAL] Ports using SCM repositories as source instead of distfiles
On Thu, Dec 11, 2008 at 10:23 AM, Dmitry Marakasov [EMAIL PROTECTED] wrote: * Ashish Shukla आशीष शुक्ल ([EMAIL PROTECTED]) wrote: This is what Debian and Gentoo does. Remember we don't have to pass DESTDIR variable to 'make -C /usr/ports/editors/emacs-cvs' instead it will be passed to the 'gmake' process invoked by port's Makefile. If we I understand. But you're implying that there is Makefile and it supports DESTDIR. As I understand, you're referring to autotools-based ports. Remember, those are less than 1/4 of the collection. Excuse me, but he refers not to autotools-based ports, but to ports that follows GNU Coding Standards (section Makefile Conventions if more preciously). Autotools just brings such support out-of-the-box. And, IMO, projects that violates these things heavy, should be fixed upstream. BTW, from my expiriense, there are little amount of project that doesn't support DESTDIR of it's analog. And many of them may be worked around anyway. -- Andrew W. Nosenko [EMAIL PROTECTED] ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: [PROPOSAL] Ports using SCM repositories as source instead of distfiles
* Andrew W. Nosenko ([EMAIL PROTECTED]) wrote: I understand. But you're implying that there is Makefile and it supports DESTDIR. As I understand, you're referring to autotools-based ports. Remember, those are less than 1/4 of the collection. Excuse me, but he refers not to autotools-based ports, but to ports that follows GNU Coding Standards (section Makefile Conventions if more preciously). Autotools just brings such support out-of-the-box. And, IMO, projects that violates these things heavy, should be fixed upstream. BTW, from my expiriense, there are little amount of project that doesn't support DESTDIR of it's analog. And many of them may be worked around anyway. I didn't count or check thoroughfully, but the feeling I've got from my 150+ ports is that no one actually supports it. I may be wrong though. And again, GNU Coding Standards don't cover build systems other than make. Also, it's not even a requirement: So, we strongly recommend GNU packages support DESTDIR, though it is not an absolute requirement. I agree with that you can't require all upstream maintainers to support this feature. Architecturally this should be completely package manager's problem (i.e. upstream should only provide installation into PREFIX, and may optionally support DESTDIR, and if package manager needs features like that staged install or automatic plist generation, and upstream don't provide it, package manager should take care of it by itself. Tecnically, we may support DESTDIR in all ports. But considering the amount of work required, and increased complexety of everything as a result, I'd stick with static plists without hesitation. See http://lists.freebsd.org/pipermail/freebsd-ports/2006-August/034745.html those are some real examples of complexity and resulting confusion, from first variant of DESTDIR support in ports. Now, when we have one DESTDIR implementation, adding another will likely make some heads explode, just think of variable naming. I'll remind that what we are talking about is automatic plist generation, and I think that this can be done without any hacks like installing a port into intermediate directory before real installation just by logging all writes to the filesystem. This will require no modifications to the ports, very minor modifications to Mk, and will (in theory) work without fail whatever the port will install, also ensuring there're no runaway files (that is, files not listed in plist, or files not installed into DESTDIR for some reason). -- Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D [EMAIL PROTECTED] ..: jabber: [EMAIL PROTECTED]http://www.amdmi3.ru ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: [PROPOSAL] Ports using SCM repositories as source instead of distfiles
Dmitry Marakasov writes:
* Andrew W. Nosenko ([EMAIL PROTECTED]) wrote:
I understand. But you're implying that there is Makefile and it supports
DESTDIR. As I understand, you're referring to autotools-based ports.
Remember, those are less than 1/4 of the collection.
Excuse me, but he refers not to autotools-based ports, but to ports
that follows GNU Coding Standards (section Makefile Conventions if
more preciously).
Autotools just brings such support out-of-the-box.
And, IMO, projects that violates these things heavy, should be fixed
upstream.
BTW, from my expiriense, there are little amount of project that
doesn't support DESTDIR of it's analog. And many of them may be
worked around anyway.
Yes, I meant that only.
I didn't count or check thoroughfully, but the feeling I've got
from my 150+ ports is that no one actually supports it. I may be
wrong though.
And again, GNU Coding Standards don't cover build systems other
than make. Also, it's not even a requirement: So, we strongly
recommend GNU packages support DESTDIR, though it is not an absolute
requirement.
I agree with that you can't require all upstream maintainers to
support this feature. Architecturally this should be completely
package manager's problem (i.e. upstream should only provide
installation into PREFIX, and may optionally support DESTDIR, and
if package manager needs features like that staged install or
automatic plist generation, and upstream don't provide it, package
manager should take care of it by itself.
Tecnically, we may support DESTDIR in all ports. But considering
the amount of work required, and increased complexety of everything
as a result, I'd stick with static plists without hesitation.
Yes, that is why I mentioned having a variable which enables this
behaviour, by default it is disabled. I mean ports which are okay
with providing static plists are fine, but ports which aren't
predictable with what files are going to installed can go with this
dynamic plist support, where ports infrastructure will only help in
generating a plist from an already setup directory tree
(/var/tmp/${portname}), now it is maintainer's responsibility to make
sure that all files will be installed in /var/tmp/${portname} which
{s,}he can do by either using 'make install DESTDIR=/var/tmp/${portname}'
or something similar if supported by port's upstream or {s,}he has to
add installation commands in ports Makefile rather than going with
upstream's way of installing things.
See
http://lists.freebsd.org/pipermail/freebsd-ports/2006-August/034745.html
those are some real examples of complexity and resulting confusion,
from first variant of DESTDIR support in ports. Now, when we have
one DESTDIR implementation, adding another will likely make some heads
explode, just think of variable naming.
The DESTDIR issue in above link refers to the DESTDIR support[1] present in
FreeBSD Ports system, and the one which I'm talking about has nothing to
do with that.
I'll remind that what we are talking about is automatic plist generation,
and I think that this can be done without any hacks like installing a
port into intermediate directory before real installation just by
logging all writes to the filesystem.
Yes that intermediate directory is what DESTDIR is. And if you're
capable of logging all writes in the DESTDIR, then its cool, but
remember you're also talking about installing port in an intermediate
directory. After the port gets installed in intermediate directory, the
plist can be generated with your filesystem writes logger component or a
well tested version of following simply command line:
% find /var/tmp/${PORTNAME} -type f |sed -e \
s[/var/tmp/${PORTNAME}${PREFIX}/[[g plist.tmp
HTH
--
Ashish Shukla
pgpNOI51EppJr.pgp
Description: PGP signature
Re: [PROPOSAL] Ports using SCM repositories as source instead of distfiles
On Thu, 11 Dec 2008 02:23:25 -0600, Dmitry Marakasov [EMAIL PROTECTED] wrote: * Ashish Shukla आशीष शुक्ल ([EMAIL PROTECTED]) wrote: This is what Debian and Gentoo does. Remember we don't have to pass DESTDIR variable to 'make -C /usr/ports/editors/emacs-cvs' instead it will be passed to the 'gmake' process invoked by port's Makefile. If we I understand. But you're implying that there is Makefile and it supports DESTDIR. As I understand, you're referring to autotools-based ports. Remember, those are less than 1/4 of the collection. pass DESTDIR to port's commandline, then it will install all dependencies in that chroot which is not desired, we simply care about the files installed by that port. Since there're already 20,000 ports we can't do it by default, so we've to hack some knob (like REQUIRES_DYNAMIC_INSTALLATION) which if defined will enable this behaviour. So if I understand correctly, you're proposing to only use dynamic plist generation for the ports that support it without modification, i.e. autotools-based? My opinion is that we should support the feature for all ports, or don't support it at all. Only getting rid of ~5k pkg-plists is not a huge accomplishment considering the mess it causes and I doubt it's worth the work on adding the feature to port.mk and then rebuilding and testing all affected ports. Being able to forget about pkg-plists once and forever however would be a huge accomplishment and if that's possible it should be done sooner or later. I object on get rid of pkg-plist. I depend on pkg-plist too much. I think it's important for us to keep on track where the files/directories are. Cheers, Mezz -- [EMAIL PROTECTED] - [EMAIL PROTECTED] FreeBSD GNOME Team http://www.FreeBSD.org/gnome/ - [EMAIL PROTECTED] ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: [PROPOSAL] Ports using SCM repositories as source instead of distfiles
Dmitry Marakasov writes:
[...]
Maybe we can introduce a hack in ports system like by adding some
variable like 'USES_DYNAMIC_PLIST=yes' in Makefile, which fill let the
port first installed with DESTDIR=/var/tmp/ports/${PORTNAME} and then a
packing list is generated and then finally whole tree is moved to
${PREFIX}, hmm...? What do you say ?
Current DESTDIR implementation uses chroot and obviously requires
complete system installed in DESTDIR. Also installing a port will
install all dependencies in the chroot as well.
The Debian package building system, also expects the package files in a
directory named ${packageroot}/debian/${packagename}/ which you can
install by doing 'make install DESTDIR=debian/${packagename}' in
${packageroot} directory (the directory where you've extracted the
package tarball, similar to the FreeBSD's ${WRKSRC} directory.
So, my idea is to specify make -C ${WRKSRC} install
DESTDIR=/var/tmp/${PORTNAME}
in the install target of port's Makefile, not to be confused with
upstream package's Makefile. And I think you confused it with passing
'DESTDIR' variable to 'make' commandline to start port building
procedure:
Following is an example illustrating how port is going to be installed
in the way I'm thinking:
,
| % sudo make -C /usr/ports/editors/emacs-cvs install
| ...checkout from CVS...
| ...configuring...
| ./configure --prefix=${LOCALBASE} ...and other configure options ...
| ...configuration messages...
| ...compilation begins...
| make
| ...compiling the files...
| ...compilation over...
| make install DESTDIR=/var/tmp/emacs
| ...files being copied to /var/tmp/emacs directory...
| ...package list creation...
| ...package list finished...
| ...copying files into ${LOCALBASE}...
| ...end copying files...
| ...package installed successfully...
`
The package list can be created via he automated package-list creation
method[1] documented in porters-handbook with a 'mtree' command executed
in 'pre-install' to prepare the '/var/tmp/${PORTNAME}' prior to
installing package. And to assure safety to this procedure, ports should
be built with a separate unprivileged user 'portbuilder' whose job is
to build ports and install them in '/var/tmp/${PORTNAME}', and create a
temporary package list. And to perform the actual installation, the port
process should switch to 'root' user.
There was proposal of another implementation that would behave as
you describe without chroot, but it would require too much work,
as most ports will need hacks so they install software into
${DESTDIR}/${PREFIX}, but the software would think that it's installed
into ${PREFIX}. That is not even always possible, so the idea was
dropped.
Thus, the only reliable way to generate plist with standart tools
is using fat chroot.
My idea however is monitor all filesystem writes by port's make and all
descendant processes. That may be done with monitoring or replacing
syscalls and may be done with LD_PRELOAD or some *trace kernel
facilities. The former is what I'm currently experimenting with.
This is also a cool way if something like this can be hacked ? How about
using truss or other syscall tracing facilities, hmm..?
Thanks for the links and replies.
References:
[1] - http://www.freebsd.org/doc/en/books/porters-handbook/plist-autoplist.html
--
Ashish Shukla
pgphIObrzBn48.pgp
Description: PGP signature
Re: [PROPOSAL] Ports using SCM repositories as source instead of distfiles
* Ashish Shukla आशीष शुक्ल ([EMAIL PROTECTED]) wrote:
Current DESTDIR implementation uses chroot and obviously requires
complete system installed in DESTDIR. Also installing a port will
install all dependencies in the chroot as well.
The Debian package building system, also expects the package files in a
directory named ${packageroot}/debian/${packagename}/ which you can
install by doing 'make install DESTDIR=debian/${packagename}' in
${packageroot} directory (the directory where you've extracted the
package tarball, similar to the FreeBSD's ${WRKSRC} directory.
So, my idea is to specify make -C ${WRKSRC} install
DESTDIR=/var/tmp/${PORTNAME}
in the install target of port's Makefile, not to be confused with
upstream package's Makefile. And I think you confused it with passing
'DESTDIR' variable to 'make' commandline to start port building
procedure:
How do you expect all ports to respect DESTDIR in their Makefiles,
while many ports don't even use make? As I've said, that will require
tremendous amount of hacking and is not even possible sometimes.
My idea however is monitor all filesystem writes by port's make and all
descendant processes. That may be done with monitoring or replacing
syscalls and may be done with LD_PRELOAD or some *trace kernel
facilities. The former is what I'm currently experimenting with.
This is also a cool way if something like this can be hacked ? How about
using truss or other syscall tracing facilities, hmm..?
Truss uses ptrace(2). There are other facilities, but they are less
useable - ktrace (produces too much output, needs to be enabled in
kernel) and dtrace (only available in current for now, also should
be enabled in kernel).
--
Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D
[EMAIL PROTECTED] ..: jabber: [EMAIL PROTECTED]http://www.amdmi3.ru
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: [PROPOSAL] Ports using SCM repositories as source instead of distfiles
Dmitry Marakasov writes:
* Ashish Shukla आशीष शुक्ल ([EMAIL PROTECTED]) wrote:
Current DESTDIR implementation uses chroot and obviously requires
complete system installed in DESTDIR. Also installing a port will
install all dependencies in the chroot as well.
The Debian package building system, also expects the package files in a
directory named ${packageroot}/debian/${packagename}/ which you can
install by doing 'make install DESTDIR=debian/${packagename}' in
${packageroot} directory (the directory where you've extracted the
package tarball, similar to the FreeBSD's ${WRKSRC} directory.
So, my idea is to specify make -C ${WRKSRC} install
DESTDIR=/var/tmp/${PORTNAME}
in the install target of port's Makefile, not to be confused with
upstream package's Makefile. And I think you confused it with passing
'DESTDIR' variable to 'make' commandline to start port building
procedure:
How do you expect all ports to respect DESTDIR in their Makefiles,
while many ports don't even use make? As I've said, that will require
tremendous amount of hacking and is not even possible sometimes.
How about adding a variable like REQUIRES_DYNAMIC_INSTALLATION=yes,
which will then generate a plist on the basis of /var/tmp/${PORTNAME},
so now it is the port's responsibility to install all contents to
/var/tmp/${PORTNAME} either by make install DESTDIR=... or by some
other way. Having that REQUIRES_DYNAMIC_INSTALLATION also includes a
'post-install' target which will copy all files from
/var/tmp/${PORTNAME} to ${PREFIX}.
--
Ashish Shukla
pgplWmDn8BjDr.pgp
Description: PGP signature
Re: [PROPOSAL] Ports using SCM repositories as source instead of distfiles
* Ashish Shukla आशीष शुक्ल ([EMAIL PROTECTED]) wrote:
How do you expect all ports to respect DESTDIR in their Makefiles,
while many ports don't even use make? As I've said, that will require
tremendous amount of hacking and is not even possible sometimes.
How about adding a variable like REQUIRES_DYNAMIC_INSTALLATION=yes,
which will then generate a plist on the basis of /var/tmp/${PORTNAME},
so now it is the port's responsibility to install all contents to
/var/tmp/${PORTNAME} either by make install DESTDIR=... or by some
other way. Having that REQUIRES_DYNAMIC_INSTALLATION also includes a
'post-install' target which will copy all files from
/var/tmp/${PORTNAME} to ${PREFIX}.
You do not understand the basic thing. The PROBLEM is to make all ports
install into one dir, but be runnable from another. As simple as that.
Imagine a software like that:
--- foo.c
#include stdlib.h
int main() {
return system(cat DATAFILE);
}
--- datafile.dat
Hello, world!
--- Makefile
PREFIX?=/usr/local
DATADIR?= ${PREFIX}/share/foo
all: foo
foo: foo.c
cc -DDATAFILE=\${DATADIR}/datafile.dat\ foo.c -o foo
install: foo datafile.dat
install -s foo ${PREFIX}/bin
install -d ${DATADIR}
install datafile.dat ${DATADIR}
---
As you can see, if this is installed with PREFIX=/var/tmp/foobar,
then moved to /usr and ran, it won't work, as it has DATAFILE
compiled in as /var/tmp/foobar/share/foo/datafile.dat instead of
/usr/local/share/foo/datafile.dat.
To make it behave as we indend, you'll have to either hack Makefile
to distinguish DESTDIR and PREFIX, or do all installation in port's
Makefile by yourself.
We have around 20k ports, and it will be extremely hard task to
convert them all to such scheme (though afaik configure-based ones
should support this without modifications). The example above is
quite simple, the real life can be much more ugly, I assure. The
worst thing is that we won't be able to test ports automatically,
as the fact the port builds and installs successfully won't mean
that it will run without problems.
I wonder how debian and gentoo get around this problem...
--
Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D
[EMAIL PROTECTED] ..: jabber: [EMAIL PROTECTED]http://www.amdmi3.ru
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: [PROPOSAL] Ports using SCM repositories as source instead of distfiles
Dmitry Marakasov writes:
[snip]
--- Makefile
PREFIX?= /usr/local
DATADIR?= ${PREFIX}/share/foo
all: foo
foo: foo.c
cc -DDATAFILE=\${DATADIR}/datafile.dat\ foo.c -o foo
install: foo datafile.dat
install -s foo ${PREFIX}/bin
install -d ${DATADIR}
install datafile.dat ${DATADIR}
# Above statements should be changed to:
install -s foo ${DESTDIR}${PREFIX}/bin
install -d ${DESTDIR}${DATADIR}
install datafile.dat ${DESTDIR}${DATADIR}
---
In above case the package list can be created with:
$ find ${DESTDIR} -type f |sed -e s[${DESTDIR}${PREFIX}/[[g
As you can see, if this is installed with PREFIX=/var/tmp/foobar,
then moved to /usr and ran, it won't work, as it has DATAFILE
compiled in as /var/tmp/foobar/share/foo/datafile.dat instead of
/usr/local/share/foo/datafile.dat.
To make it behave as we indend, you'll have to either hack Makefile
to distinguish DESTDIR and PREFIX, or do all installation in port's
Makefile by yourself.
We have around 20k ports, and it will be extremely hard task to
convert them all to such scheme (though afaik configure-based ones
should support this without modifications). The example above is
quite simple, the real life can be much more ugly, I assure. The
worst thing is that we won't be able to test ports automatically,
as the fact the port builds and installs successfully won't mean
that it will run without problems.
I wonder how debian and gentoo get around this problem...
I understood what you're trying to say, but you're missing my point, so
I'm explaining it in the following example:
, Quoting from my earlier example
| % sudo make -C /usr/ports/editors/emacs-cvs install
| ...checkout from CVS...
| ...configuring...
| ./configure --prefix=${LOCALBASE} ...and other configure options ...
That will compile Emacs with PREFIX=${LOCALBASE} (which is /usr/local
by default)
| ...configuration messages...
| ...compilation begins...
| make
| ...compiling the files...
| ...compilation over...
| make install DESTDIR=/var/tmp/emacs
This will install all files into DESTDIR=/var/tmp/emacs, with directory
structure similar to following:
/var/tmp/emacs
/usr/local/bin/emacs
/usr/local/share/emacs/23.0.60/...
etc.
| ...files being copied to /var/tmp/emacs directory...
| ...package list creation...
| ...package list finished...
| ...copying files into ${LOCALBASE}...
And finally this will copy all files present in /var/tmp/emacs to
${LOCALBASE} for final installation with package we recorded earlier.
| ...end copying files...
| ...package installed successfully...
`
This is what Debian and Gentoo does. Remember we don't have to pass
DESTDIR variable to 'make -C /usr/ports/editors/emacs-cvs' instead it
will be passed to the 'gmake' process invoked by port's Makefile. If we
pass DESTDIR to port's commandline, then it will install all
dependencies in that chroot which is not desired, we simply care about
the files installed by that port. Since there're already 20,000 ports we
can't do it by default, so we've to hack some knob (like
REQUIRES_DYNAMIC_INSTALLATION) which if defined will enable this
behaviour.
Following are some links:
Debian's New Maintainer's Guide
http://www.debian.org/doc/maint-guide/ch-modify.en.html#s-destdir
Gentoo's Ebuild Quickstart Guide
http://devmanual.gentoo.org/quickstart/index.html
I hope I'm clear now.
Thanks
--
Ashish Shukla
pgpANuTUI7juX.pgp
Description: PGP signature
[PROPOSAL] Ports using SCM repositories as source instead of distfiles
Hi all,
I'm to proposing an enhancement to existing FreeBSD ports system. I
think it'll be great if ports can use SCM (source code management)
repositories like CVS, Subversion, Git, etc. as their sources instead of
distfiles. Following are some of the {dis,}advantages of this approach:
* Anyone interested in tracking the development branch of any project
can do so with installation/deinstallation of the project integrated
in the ports management system. e.g. anyone interested in trying out
GNU Emacs CVS version, can simply use editors/emacs-cvs
(not-implemented atm), which will checkout from the Emacs's CVS
repository and build and install the port with all installed files
recorded in 'plist' of the port. And if anyone wants to uninstall the
Emacs CVS port, {s,}he can simply deinstall it in their preferred
ports/package way.
* ATM, development versions of ports are packaged as snapshots, and
maintainer has to keep updating ports snapshots. And sometimes,
it is not possible for maintainer (due to lack of time and other
issues) to update snapshots timely. So going proposed way can ease the
work for them, and beneficial for users who are interested in latest
bits.
* As far as PR related to such ports are concerned I think one should
directly submit them to the upstream rather than maintainer, unless
that PR has anything to do with its packaging, in which case it should
be submitted to FreeBSD PR system.
* I've not played (or worked) with dynamic packaging lists on FreeBSD,
so I'm not sure if it is possible to properly track all installed
files dynamically, e.g. if a new commit in the upstream causes 3 new
files to be installed, then is it possible for FreeBSD ports
management system to include those 3 files also in the packing list,
in the next installation of the port, hmm...?
BtW, I've implemented a basic proof-of-concept and hacked a port
audio/scrobby (for scrobby[1] application, not submitted to PR system
yet) based on proof-of-concept. The shar can be downloaded from following URL:
http://wahjava.googlepages.com/scm-ports.shar
(md5: 7f8392e7c9e31fe5842279e1730a2051, size: 4710 bytes)
Some existing tools like patchtool also needs to be modified which rely
for the presence of 'distinfo' file in the port directory, which in this
case will never be present.
References:
[1] - http://unkart.ovh.org/scrobby/
Thanks for trying it.
--
Ashish Shukla
pgpQJU0pd4l3k.pgp
Description: PGP signature
Re: [PROPOSAL] Ports using SCM repositories as source instead of distfiles
* Ashish Shukla आशीष शुक्ल ([EMAIL PROTECTED]) wrote:
I'm to proposing an enhancement to existing FreeBSD ports system. I
think it'll be great if ports can use SCM (source code management)
repositories like CVS, Subversion, Git, etc. as their sources instead of
distfiles. Following are some of the {dis,}advantages of this approach:
This was discussed before. The summary: this won't work with ports.
Reasons:
- No way to track port updates.
- No way to save distfile. Source will be redownloaded on every rebuild.
- Security. No one can guarantee that malicious source won't be cheched
into the repository at some random time. Fixed versions can be at
least minimally checked and it is possible to not update the port to
new version if it contains problems and/or tell users to not install
version XXX. Impossible for SCM-based ports.
* ATM, development versions of ports are packaged as snapshots, and
maintainer has to keep updating ports snapshots. And sometimes,
it is not possible for maintainer (due to lack of time and other
issues) to update snapshots timely. So going proposed way can ease the
work for them, and beneficial for users who are interested in latest
bits.
- _Much_ more (instead of less) work for maintainer, as he won't be able
to test the port before committing it and will have to deal with all
the problems post factum, under extra pressure.
- Actually, any SCM-based port will become broken rather soon than
later with no ability to prevent it.
The port uses patches? Due to mutable source it'll become broken.
Any structural change upstream? Port broken. Changed build system?
Broken. Changed paths? Broken. Changed depends? Broken. Changed
options/configure args? Broken. Etc.
* I've not played (or worked) with dynamic packaging lists on FreeBSD,
so I'm not sure if it is possible to properly track all installed
files dynamically, e.g. if a new commit in the upstream causes 3 new
files to be installed, then is it possible for FreeBSD ports
management system to include those 3 files also in the packing list,
in the next installation of the port, hmm...?
- Generic dynamic plist generation is impossible unless the port
is installed into some clean chrooted environemnt (for example,
using DESTDIR). The latter, however takes extra space and time,
as you need the whole system and all dependent packages installed
there as well. Simply building the port will become more more like
producing package in a tinderbox:
- unpack the system image
- mount all required filesystems - devfs, ports, distfiles, packages
- install all required packages
- take list of all files in the chroot
- chroot and install the port
- take list of all files in the chroot, compare with previous one and
make a plist out of it
- umount and remove everything
- now you have package and may install in normally
* As far as PR related to such ports are concerned I think one should
directly submit them to the upstream rather than maintainer, unless
that PR has anything to do with its packaging, in which case it should
be submitted to FreeBSD PR system.
Sometimes it's hard to tell whether the problem is FreeBSD-specific.
Also, upstream is unlikely to have FreeBSD box for testing, so again
it'll be more work for maintainer.
--
Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D
[EMAIL PROTECTED] ..: jabber: [EMAIL PROTECTED]http://www.amdmi3.ru
___
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: [PROPOSAL] Ports using SCM repositories as source instead of distfiles
* Ashish Shukla आशीष शुक्ल ([EMAIL PROTECTED]) wrote: - _Much_ more (instead of less) work for maintainer, as he won't be able to test the port before committing it and will have to deal with all the problems post factum, under extra pressure. - Actually, any SCM-based port will become broken rather soon than later with no ability to prevent it. The port uses patches? Due to mutable source it'll become broken. Any structural change upstream? Port broken. Changed build system? Broken. Changed paths? Broken. Changed depends? Broken. Changed options/configure args? Broken. Etc. These are the problems already expected with this but the only suggestion is to have PRs filed if anything breaks during compilation and investigate what caused it. No, those problems will not arise as long as the maintainer tests the port before submitting an update. And the tested port of fixed version will be usable for a long time, unlike SCM-based one which may break every second. - Generic dynamic plist generation is impossible unless the port is installed into some clean chrooted environemnt (for example, using DESTDIR). The latter, however takes extra space and time, as you need the whole system and all dependent packages installed there as well. So, this is the main reason which prevents this :( . I'd say it's the least significant reason. The main reasons are the first three which can be shortened as `the port will be unuseable and sometimes dangerous'. What's for automatic plist generation, I've given it some thought, and it seems like there could be a more or less reliable way after all. I'm currently doing some experiments. Sometimes it's hard to tell whether the problem is FreeBSD-specific. Also, upstream is unlikely to have FreeBSD box for testing, so again it'll be more work for maintainer. True, so either have all PRs should be submitted to FreeBSD PR system, where maintainer will decide if its porting issue or upstream related issue. My opinion is that if you can diagnose the problem by yourself and come with a proper fix, you should submit it directly upstream. If you think that the problem is serious enough, you can send a port PR as well. If you cannot do it all by yourself though, you should submit a PR, in which case port's maintainer will take care of it. So with all the problems you mentioned above, I guess, I'll take my proposal back :) . It's not like your proposal is bad, ports instantaneously tracking upstream changes and not needing maintainers would really be cool, but unfortunately that's practically impossible. -- Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D [EMAIL PROTECTED] ..: jabber: [EMAIL PROTECTED]http://www.amdmi3.ru ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: [PROPOSAL] Ports using SCM repositories as source instead of distfiles
Dmitry Marakasov writes:
* Ashish Shukla आशीष शुक्ल ([EMAIL PROTECTED]) wrote:
- _Much_ more (instead of less) work for maintainer, as he won't be able
to test the port before committing it and will have to deal with all
the problems post factum, under extra pressure.
- Actually, any SCM-based port will become broken rather soon than
later with no ability to prevent it.
The port uses patches? Due to mutable source it'll become broken.
Any structural change upstream? Port broken. Changed build system?
Broken. Changed paths? Broken. Changed depends? Broken. Changed
options/configure args? Broken. Etc.
These are the problems already expected with this but the only
suggestion is to have PRs filed if anything breaks during compilation
and investigate what caused it.
No, those problems will not arise as long as the maintainer tests the
port before submitting an update. And the tested port of fixed version
will be usable for a long time, unlike SCM-based one which may break
every second.
That is true, but the only problem I see with snapshots is, if
maintainer is busy you can't do anything except maintaining your own local
port version on your box. And anyone using these SCM ports is the one
who knows things can go wrong often, and this is only for development
use.
- Generic dynamic plist generation is impossible unless the port
is installed into some clean chrooted environemnt (for example,
using DESTDIR). The latter, however takes extra space and time,
as you need the whole system and all dependent packages installed
there as well.
So, this is the main reason which prevents this :( .
I'd say it's the least significant reason. The main reasons are the
first three which can be shortened as `the port will be unuseable and
sometimes dangerous'.
What's for automatic plist generation, I've given it some thought,
and it seems like there could be a more or less reliable way after all.
I'm currently doing some experiments.
Cool, would like to see the results of your experiments :).
Sometimes it's hard to tell whether the problem is FreeBSD-specific.
Also, upstream is unlikely to have FreeBSD box for testing, so again
it'll be more work for maintainer.
True, so either have all PRs should be submitted to FreeBSD PR system,
where maintainer will decide if its porting issue or upstream related
issue.
My opinion is that if you can diagnose the problem by yourself and come
with a proper fix, you should submit it directly upstream. If you think
that the problem is serious enough, you can send a port PR as well. If
you cannot do it all by yourself though, you should submit a PR, in
which case port's maintainer will take care of it.
I agree.
So with all the problems you mentioned above, I guess, I'll take my
proposal back :) .
It's not like your proposal is bad, ports instantaneously tracking
upstream changes and not needing maintainers would really be cool,
but unfortunately that's practically impossible.
Gentoo GNU/Linux which is a source-based GNU/Linux distribution has this
feature available, what is different in that is, that it uses a separate
root for recording the packing list port and optionally creating a
package.
Maybe we can introduce a hack in ports system like by adding some
variable like 'USES_DYNAMIC_PLIST=yes' in Makefile, which fill let the
port first installed with DESTDIR=/var/tmp/ports/${PORTNAME} and then a
packing list is generated and then finally whole tree is moved to
${PREFIX}, hmm...? What do you say ?
Thanks for you replies.
Ashish Shukla
P.S. do you've any ideas about when a discussion on this subject took
place on this list, hmm..?
pgpYjAw2dFrgM.pgp
Description: PGP signature
