Re: Detecting dependencies
On 09/17/11 18:09, b. f. wrote: On 09/15/11 07:06, chukharev at mail.ru wrote: Hi, There have been a discussion about finding interdependencies of ports. I have a relatively simple Python script for that. There is a pr ports/160007 to add its early version. Unfortunately, I missed a reply to it, so there is an issue which I have not yet addressed... Since that time, I added reverse dependencies with full ports tree scanning (1 h on my 2.5GHz notebook) and saving the tree (directed graph, actually) to a file, so that rescanning all ports tree is not needed. See http://code.google.com/p/porttree/ If there will be interest, scanning packages interdependencies could also be added. On a related subtopic, we also need a tool that identifies implicit dependencies not captured in the ports Makefiles. I hacked the following together earlier this year to smooth over the updating process when key libraries get bumped (e.g. the gettext update at the time I wrote the script was a nightmare). There were a tonne of ports which needed to be updated even though they didn't explicitly record a dependency on gettext. https://lauren.room52.net/hg/scripts/raw-file/tip/libdepend/libdepend.sh It's still quite rough and manually driven and is tied to portmaster at the moment, but I use it routinely after a "portmaster -ad" to check that no libs are missing dependencies. It works pretty well most of the time, but definitely needs more finessing. I share it mostly to prove the feasibility of the approach and in case anyone is curious. What, no check to see if the libraries listed in the DT_NEEDED tags are actually needed? And no kitchen sink? ;) err... look, over there! A dog with a puffy tail chasing a kitchen sink! *runs* There are scripts in ports/Tools/scripts that were intended to perform similar tasks, although they may be rougher than your script. Yes they provide various bits and pieces of the puzzle. I haven't thought the following ideas through a great deal and welcome feedback, but I think the basic functionality/premise of this script could be integrated into the ports framework so that at package registration time, implicit deps are identified and marked in the package database. A warning could also be generated that the port is using deps not identified in the Makefile, and perhaps trigger a send-pr to the port maintainer to let them know. ... A script like this could also be integrated/called somehow from a tool like portmaster during an update to ensure ports with implicit dependencies on another port which has been updated are identified and recompiled too so that we avoid the nasty problems that crop up with missing library dependencies. Just as in the other *_DEPENDS lists, it was a conscious policy decision, for the sake of brevity and efficiency, that if port B requires port C, and port A requires port B, then libraries from port C will not be listed in the LIB_DEPENDS of port A, even if port A links directly to those libraries. But because Right, which is fine (and more desirable - simpler is better) if we have a way built into the system to actually derive them at upgrade/install time and ensure we don't leave the system broken. Given that the information can be derived, it seems sensible not to have ports' Makefiles define all deps explicitly, and instead use tools at install/upgrade time to do the heavy lifting automatically. Going for brevity without the infrastructure in place to automagically compensate for the information not being explicitly codified in the makefiles means certain brokeness, which is not cool. of recurring problems with partial port updates, this policy has been criticized. I think that the last time the matter was raised, the consensus seemed to lean toward listing all needed libraries, but the amount of work involved in, and the likely disruption arising from, refactoring all LIB_DEPENDS in the tree dissuaded anyone doing so. I can't see a reason why the policy can't stay as it is if the smarts can be added to generate the implicit dependency info when needed, and more importantly use that generated information to avoid leaving the system broken. Whether we argue the smarts belong solely in a tool like portmaster or should be integrated into the ports infrastructure itself is fair game. My gut feeling is that the deps list stored on disk by the ports system at registration time should be complete with explicit and implicit deps, even though the port's Makefile only lists those which are explicit. Tools like portmaster then only need to use the information as is to do their part of the job and the system should be left intact post upgrade cycle, at least from a broken lib deps perspective. If my gut feeling is valid, then that implies the ports infrastructure itself should do a step post install but pre registration where implicit deps are identified and added to the port's +CONTENTS file. I'm very unfamiliar with the
Re: Detecting dependencies
> On 09/15/11 07:06, chukharev at mail.ru wrote: > > Hi, > > > > There have been a discussion about finding interdependencies of ports. > > I have a relatively simple Python script for that. There is a pr > > ports/160007 > > to add its early version. Unfortunately, I missed a reply to it, so > > there is > > an issue which I have not yet addressed... > > > > Since that time, I added reverse dependencies with full ports tree scanning > > (1 h on my 2.5GHz notebook) and saving the tree (directed graph, actually) > > to a file, so that rescanning all ports tree is not needed. > > > > See http://code.google.com/p/porttree/ > > > > If there will be interest, scanning packages interdependencies could > > also be added. > > > > On a related subtopic, we also need a tool that identifies implicit > dependencies not captured in the ports Makefiles. I hacked the following > together earlier this year to smooth over the updating process when key > libraries get bumped (e.g. the gettext update at the time I wrote the > script was a nightmare). There were a tonne of ports which needed to be > updated even though they didn't explicitly record a dependency on gettext. > > https://lauren.room52.net/hg/scripts/raw-file/tip/libdepend/libdepend.sh > > It's still quite rough and manually driven and is tied to portmaster at > the moment, but I use it routinely after a "portmaster -ad" to check > that no libs are missing dependencies. It works pretty well most of the > time, but definitely needs more finessing. I share it mostly to prove > the feasibility of the approach and in case anyone is curious. What, no check to see if the libraries listed in the DT_NEEDED tags are actually needed? And no kitchen sink? ;) There are scripts in ports/Tools/scripts that were intended to perform similar tasks, although they may be rougher than your script. > > I haven't thought the following ideas through a great deal and welcome > feedback, but I think the basic functionality/premise of this script > could be integrated into the ports framework so that at package > registration time, implicit deps are identified and marked in the > package database. A warning could also be generated that the port is > using deps not identified in the Makefile, and perhaps trigger a send-pr > to the port maintainer to let them know. ... > A script like this could also be integrated/called somehow from a tool > like portmaster during an update to ensure ports with implicit > dependencies on another port which has been updated are identified and > recompiled too so that we avoid the nasty problems that crop up with > missing library dependencies. Just as in the other *_DEPENDS lists, it was a conscious policy decision, for the sake of brevity and efficiency, that if port B requires port C, and port A requires port B, then libraries from port C will not be listed in the LIB_DEPENDS of port A, even if port A links directly to those libraries. But because of recurring problems with partial port updates, this policy has been criticized. I think that the last time the matter was raised, the consensus seemed to lean toward listing all needed libraries, but the amount of work involved in, and the likely disruption arising from, refactoring all LIB_DEPENDS in the tree dissuaded anyone doing so. b. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: Detecting dependencies
On 09/15/11 07:06, chukha...@mail.ru wrote: Hi, There have been a discussion about finding interdependencies of ports. I have a relatively simple Python script for that. There is a pr ports/160007 to add its early version. Unfortunately, I missed a reply to it, so there is an issue which I have not yet addressed... Since that time, I added reverse dependencies with full ports tree scanning (1 h on my 2.5GHz notebook) and saving the tree (directed graph, actually) to a file, so that rescanning all ports tree is not needed. See http://code.google.com/p/porttree/ If there will be interest, scanning packages interdependencies could also be added. On a related subtopic, we also need a tool that identifies implicit dependencies not captured in the ports Makefiles. I hacked the following together earlier this year to smooth over the updating process when key libraries get bumped (e.g. the gettext update at the time I wrote the script was a nightmare). There were a tonne of ports which needed to be updated even though they didn't explicitly record a dependency on gettext. https://lauren.room52.net/hg/scripts/raw-file/tip/libdepend/libdepend.sh It's still quite rough and manually driven and is tied to portmaster at the moment, but I use it routinely after a "portmaster -ad" to check that no libs are missing dependencies. It works pretty well most of the time, but definitely needs more finessing. I share it mostly to prove the feasibility of the approach and in case anyone is curious. I haven't thought the following ideas through a great deal and welcome feedback, but I think the basic functionality/premise of this script could be integrated into the ports framework so that at package registration time, implicit deps are identified and marked in the package database. A warning could also be generated that the port is using deps not identified in the Makefile, and perhaps trigger a send-pr to the port maintainer to let them know. That way when we update ports using a tool like portmaster, it will know to update all the relevant ports and avoid leaving your system broken (yes, I'm aware of the -w switch, but I prefer not to use it as you can get into nasty situations if the compat and non-compat libs get mixed at run-time). A script like this could also be integrated/called somehow from a tool like portmaster during an update to ensure ports with implicit dependencies on another port which has been updated are identified and recompiled too so that we avoid the nasty problems that crop up with missing library dependencies. Cheers, Lawrence ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: Detecting dependencies
Michel Talon wrote: > Oliver Fromme wrote: > > > > That's what a script of mine does (it's also in Python): > > > > http://www.secnetix.de/olli/scripts/pkg_dep_view > > Waooh! this is very cute. > > While we are in python i have something which draws > graphviz dependency graphs for ports here > http://www.lpthe.jussieu.fr/~talon/pkg_check.py Very nice! Your script seems to do several jobs at once, while I have separate scripts for those tasks. E.g. I have a separate script for checking origins and the consistency of dependencies. > Seeing things like your script, the perl script port-easy by des, etc. > i really wonder why people write stuff in C or worse shell for ports. > One could write ten times smarter and ten times shorter things in real > languages like python, lisp, etc. This argument of being "included in > base system" is so completely bogus ... I think "high-level" languages like Python are very well suited for managing data structures like dependency graphs. These structures are native to the language, so reading the +REQUIRED_BY files into a tree structure is absolutely trivial and requires just a few lines, wereas in C it gets a lot more complicated and a lot more error-prone. In Python you don't have to care about pointers and memory allocation (the same is true for Ruby, Perl and others, of course, but I think that Perl's syntax is horrible). As for "it's not in the base system": I don't care much. The Python port is easy enough to install, and in fact it's installed on most machines anyway because quite a lot of ports depend on it. Actually, some of my scripts that deal with package began as awk scripts, precisely for the reason that it's in the base system. But while awk supports dynamic arrays and associative arrays, it doesn't support nesting them, so you can't directly make a tree-like structure (well, you can, but it gets ugly very quickly). So I started porting them to my favourite scripting language, which is Python. Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "Being really good at C++ is like being really good at using rocks to sharpen sticks." -- Thant Tessman ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: Detecting dependencies
Oliver Fromme wrote: That's what a script of mine does (it's also in Python): http://www.secnetix.de/olli/scripts/pkg_dep_view Waooh! this is very cute. While we are in python i have something which draws graphviz dependency graphs for ports here http://www.lpthe.jussieu.fr/~talon/pkg_check.py Unfortunately for many ports the diagram becomes completely unreadable. Your display is wonderful. Seeing things like your script, the perl script port-easy by des, etc. i really wonder why people write stuff in C or worse shell for ports. One could write ten times smarter and ten times shorter things in real languages like python, lisp, etc. This argument of being "included in base system" is so completely bogus ... -- Michel TALON ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: Detecting dependencies
Jason Hellenthal wrote: > On Thu, Sep 15, 2011 at 12:06:03AM +0300, chukha...@mail.ru wrote: > > There have been a discussion about finding interdependencies of ports. > > I have a relatively simple Python script for that. There is a pr > > ports/160007 > > to add its early version. Unfortunately, I missed a reply to it, so there > > is > > an issue which I have not yet addressed... > [...] > > 1. Would be cool if this would work on already installed ports or packages > too! just a thought. That's what a script of mine does (it's also in Python): http://www.secnetix.de/olli/scripts/pkg_dep_view > 2. If it would detect the presence of UTF-8 in the LANG or LC_ALL > environment vars and use the appropriate drawing method for each rather > than a default to UTF-8. "I am happy with it as is though" Maybe have a look at my script, it might help. It uses Python's curses module to display graphical line characters using the ACS feature (alternate character set). This is independent from the local (UTF-8, ISO8859, ASCII) and works very well with terminals that support line drawing characters, such as xterm, vt100 and so on. When there is no such support (dumb terminal, or not supported by termcap, or stdout is a pipe), normal ASCII characters are used instead. Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "I have stopped reading Stephen King novels. Now I just read C code instead." -- Richard A. O'Keefe ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: Detecting dependencies
On Thu, Sep 15, 2011 at 12:06:03AM +0300, chukha...@mail.ru wrote:
> There have been a discussion about finding interdependencies of ports.
> I have a relatively simple Python script for that. There is a pr ports/160007
> to add its early version. Unfortunately, I missed a reply to it, so there is
> an issue which I have not yet addressed...
>
> Since that time, I added reverse dependencies with full ports tree scanning
> (1 h on my 2.5GHz notebook) and saving the tree (directed graph, actually)
> to a file, so that rescanning all ports tree is not needed.
>
> See http://code.google.com/p/porttree/
>
> If there will be interest, scanning packages interdependencies could
> also be added.
This looks like a useful tool. However, as Doug pointed out in another
thread ("Detecting dependencies"), its method is inadequate when it
comes to finding dependencies for shared library bumps. Specifically, it
won't find dependencies hidden by disabled OPTIONS knobs.
--
Shaun Amott // PGP: 0x6B387A9A
"A foolish consistency is the hobgoblin
of little minds." - Ralph Waldo Emerson
pgpzMX1cHyEGf.pgp
Description: PGP signature
Detecting dependencies
Hi, There have been a discussion about finding interdependencies of ports. I have a relatively simple Python script for that. There is a pr ports/160007 to add its early version. Unfortunately, I missed a reply to it, so there is an issue which I have not yet addressed... Since that time, I added reverse dependencies with full ports tree scanning (1 h on my 2.5GHz notebook) and saving the tree (directed graph, actually) to a file, so that rescanning all ports tree is not needed. See http://code.google.com/p/porttree/ If there will be interest, scanning packages interdependencies could also be added. -- Vladimir Chukharev Tampere University of Technology ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Detecting dependencies
Howdy, A couple of recent threads have mentioned clever ways to search for dependencies. One problem ... there aren't any. :) The *only* safe way to make sure you have found all possible references to a dependency is to grep the entire ports tree (grep -r category/portname /usr/ports/*). There are just too many clever things that people do in various files that aren't given conventional names that you will likely miss some references if you don't. Particularly useless (by itself) is using the INDEX to find dependencies because it will miss any optional dependencies that are not enabled by default. It is usually useful to cross-check the outcome of your grep command to make sure that you have captured at least the known on-by-default ones. hth, Doug -- Nothin' ever doesn't change, but nothin' changes much. -- OK Go Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
