Re: Dependency discovery

[Guillem Jover]

Hi!

On Fri, 2011-04-01 at 16:41:57 -0500, Donnie Berkholz wrote:
> On 17:04 Fri 01 Apr , Joseph Tate wrote:
> > Conary has some really nifty tools for detecting dependencies.  Some 
> > of it runs on source code, like Python deps.  And some runs on 
> > compiled code, like for shared object dependencies, etc.  I only know 
> > about it as a user/packager though.
> 
> Compiled code is easy. =)

Well, I'd reword that to “shared object dependencies are easy”.

Compiled code can have additional dependencies which might be optional
or required depending if it will gracefully fallback or degrade which
can be tricky to detect and which are not needed at build time, from
the top of my head those could be:

  * Programs being execve()ed.
  * Data files.
  * Unix sockets, named pipes.
  * System V/POSIX message queues, shared memory and semaphores.
  * D-BUS (or other IPC system) interfaces.
  * ...

regards,
guillem
___
Distributions mailing list
Distributions@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/distributions

Re: Dependency discovery

[Stanislav Brabec]

Donnie Berkholz wrote:
> Do any of you have useful software that scans source code to determine 
> its dependencies? We're considering a Google Summer of Code project that 
> would include this as one aspect of a semi-automated package generator, 
> but I'd hate to see us duplicating effort.
> 
> The basic idea I had in mind is a set of plugins, one per language, with 
> a sufficient understanding of that language's syntax to determine the 
> external files a given piece of code depends on. We'd then couple that 
> with a tool like distromatch to figure out which packages the files came 
> from.

I wrote a small utility pkg-buildrequires, a post-mortem analyzer of the
autoconf-style configure script checks that were called. It was capable
to collect most dependencies in several steps. I tried to build with
base system in the build root, then analyzed and updated list of
required packages. It was tested on ~200 packages (GNOME and related
packages) and succeeded for ~90% of packages in ~4 iterations.
http://ftp.suse.com/pub/people/sbrabec/pkg-buildrequires/

Problems of that idea:

1. There is not yet a solution for the situation, when building for more
platforms or more distro versions will end with a different requirement
list.

2. Thinking about it, ideally such tool should be integrated directly
into autoconf macros - configure script could optionally write a machine
readable .pc and library check list. Package system tools can then
assign these checks to packages or symbols.

My idea was following:

Packaging system or package creating tool would call:
./configure --generate-check-log ...

And then parse config-check.log, update new requirements for build and
try again until all dependencies are checked and reviewed by the
packager.

The log should contain following information:

- What type of dependency it is (.pc file check, library check)
Easy to implement.


- whether it is mandatory check (configure fails if not found), optional
check (configure continues and the package is limited in features), pick
one check (one of several checks must succeed)

This would probably require autoconf syntax change; we really need to
know it only for failed checks - and there it is easy to detect
mandatory check.

- whether it is sequential version check (more versions is checked, one
check passes, other don't; can be mandatory, optional or pick-one).
This would probably require autoconf syntax change.

-- 
Best Regards / S pozdravem,

Stanislav Brabec
software developer
-
SUSE LINUX, s. r. o.  e-mail: sbra...@suse.cz
Lihovarská 1060/12tel: +49 911 7405384547
190 00 Praha 9  fax: +420 284 028 951
Czech Republichttp://www.suse.cz/

___
Distributions mailing list
Distributions@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/distributions

Re: Dependency discovery

[Matthias Klumpp]

Hi!

> Do any of you have useful software that scans source code to determine 
> its dependencies?
Currently, I am creating such a tool for Listaller. (cross-distro
app-installer & manager)
At time we have a tool called "Visual-LDD" to display a
library-dependency-tree from binary files. Someone also planned to
implement this for Python, which already has something like a dependency
finder in its standard-library. (But I don't know if there has been some
more work...)
My plan is to create a tool which first find all dependencies of a
software automatically, no matter if it is a compiled binary app or Python
or Perl, or XYZ and then uses distromatch to determine which packages will
match the dependency. In a third step the tool will check in which
distro-releases the package is present and give information like "your tool
will run on Fedora 12-15". If a dependency cannot be found in a
distribution, the tool will use 3rd-party sources. (It's pupose is to
generate a bit of Listaller-XML which is used by the setup to fetch
dependencies)
If this is successful, I think the "depscan" tool can easily be used
without Listaller. I haven't followed the whole thread, for which purpose
do you want this tool? Or is it just a "what if..." thought?
Regards
   Matthias Klumpp


___
Distributions mailing list
Distributions@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/distributions

Re: Dependency discovery

[Éric Araujo]

Hi,

> Do any of you have useful software that scans source code to determine 
> its dependencies?

Python has the modulefinder module/script in its standard library.  You
can hook it into another tool to map Python module names to OS package
names.

Projects that use setuptools or distribute can also specify their
dependencies (at the project level, not module level) in their setup
script, and this information is preserved after installation.  In Python
3.3, this will be standardized (see PEP 376 and the distutils2 project).

Some projects use other formats: pip freeze files, buildout config
files, fabric fabfiles, etc.

Regards
___
Distributions mailing list
Distributions@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/distributions

Re: Dependency discovery

[Jos Poortvliet]

On Fri, Apr 1, 2011 at 6:41 PM, Donnie Berkholz  wrote:
> On 17:04 Fri 01 Apr     , Joseph Tate wrote:
>> Conary has some really nifty tools for detecting dependencies.  Some
>> of it runs on source code, like Python deps.  And some runs on
>> compiled code, like for shared object dependencies, etc.  I only know
>> about it as a user/packager though.
>
> Compiled code is easy. =)
>
> Picture this use case, though — there's a web interface where users can
> upload tarballs to a server, and the interface automatically detects
> dependencies, deciphers the build process for common build systems, and
> creates a basic packaging script. It would be a lot cheaper on the
> resources, not to mention probably more secure, if the package never
> even needed to get built.

Read up on project Bretzn, this is exactly what they are working on.
Lubos Lunak has created a tool which can do this, at least for C/C++
afaik. They use it for their whole "build packages for all major
distro's directly from your IDE" thing.

Bretzn website here:
http://www.socialdesktop.org/bretzn/
Lubos Lunak's generator (old blog, it has progressed and you'd have to
ask on the bretzn mailinglist for the latest status)
http://www.kdedevelopers.org/node/4177


>
> --
> Thanks,
> Donnie
>
> Donnie Berkholz
> Sr. Developer, Gentoo Linux
> Blog: http://dberkholz.com
>
> ___
> Distributions mailing list
> Distributions@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/distributions
>
>
___
Distributions mailing list
Distributions@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/distributions

Re: Dependency discovery

[Enrico Zini]

On Fri, Apr 01, 2011 at 01:46:40PM -0700, Toshio Kuratomi wrote:

> Oh!  Regarding distromatch -- at PyCon, Allison Randall and I were figuring
> out how to match packages in Fedora, packages in Ubuntu, and packages on
> pypi (upstream python code hosting) together.  We came up with sonmething
> pretty interesting.  tarballs.  The name portion of a tarballs/zip file/etc
> can be used to match a large number of source packages together.  Even for
> packages which generate multiple binary subpackages, this should help narrow
> down which binary packages are not even remotely related despite having
> similar names.

Thanks, I agree it's a good idea. In distromatch I'm already collecting
"source package name" information from all the distributions supported
so far.  "Source package name" just matches the tarball name everywhere
as far as I know.

So far, however, I still don't make use of that information. One
intention is, in some future tweaking iteration, to use it to validate
approximate matches: given a list of approximate matches, one can try to
trim it to only those that come from a source package (or tarball) with
the same name.

[end of the reply to Toshio Kuratomi, start of a little preaching to the
distributions crowd]

However, I think that distromatch is already good enough that it's time
to build some prototype on top of it that addresses some real problem:
for example, we have categories and screenshots in Debian that it's
really high time others started to steal.

Those are very, very low hanging fruits now: I'd like to see people make
that little bit of effort and just grab them.

I confess that maintaining distromatch as a tool "per se" does not
motivate me: I would like further tweaking of the matching algorithms to
be driven by the need of perfecting its applications.

Until distromatch gets really used, of course, if anybody else is
intrigued by the idea of tweaking distromatch for the pure sake of
tweaking distromatch, they are more than welcome to do so.

I also wouldn't mind to be able to say "in distromatch, *we* do this",
instead of "in distromatch, *I* do this": I am staying next to an
*extremely frequent* bus route[1], after all :P


Ciao,

Enrico

[1] http://en.wikipedia.org/wiki/Greater_Manchester_bus_route_192#Fame
-- 
GPG key: 4096R/E7AD5568 2009-05-08 Enrico Zini 


signature.asc
Description: Digital signature
___
Distributions mailing list
Distributions@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/distributions

Re: Dependency discovery

[Donnie Berkholz]

On 17:04 Fri 01 Apr , Joseph Tate wrote:
> Conary has some really nifty tools for detecting dependencies.  Some 
> of it runs on source code, like Python deps.  And some runs on 
> compiled code, like for shared object dependencies, etc.  I only know 
> about it as a user/packager though.

Compiled code is easy. =)

Picture this use case, though — there's a web interface where users can 
upload tarballs to a server, and the interface automatically detects 
dependencies, deciphers the build process for common build systems, and 
creates a basic packaging script. It would be a lot cheaper on the 
resources, not to mention probably more secure, if the package never 
even needed to get built.


-- 
Thanks,
Donnie

Donnie Berkholz
Sr. Developer, Gentoo Linux
Blog: http://dberkholz.com


pgpgzTn9GqeqJ.pgp
Description: PGP signature
___
Distributions mailing list
Distributions@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/distributions

Re: Dependency discovery

[Andrey Ponomarenko]

02.04.2011 0:11, Donnie Berkholz пишет:

Do any of you have useful software that scans source code to determine
its dependencies? We're considering a Google Summer of Code project that
would include this as one aspect of a semi-automated package generator,
but I'd hate to see us duplicating effort.

The basic idea I had in mind is a set of plugins, one per language, with
a sufficient understanding of that language's syntax to determine the
external files a given piece of code depends on. We'd then couple that
with a tool like distromatch to figure out which packages the files came
from.



Try the Linux Application Checker [1-3].

[1] Check Your App 
[2] All About the Linux Application Checker 

[3] About Linux Application Checker 



--
Andrey Ponomarenko
Department for Operating Systems at ISPRAS
 web:http://www.LinuxTesting.org
 mail:   aponomare...@ispras.ru

___
Distributions mailing list
Distributions@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/distributions

Re: Dependency discovery

[Per Øyvind Karlsen]

2011/4/1 Donnie Berkholz :
> Do any of you have useful software that scans source code to determine
> its dependencies? We're considering a Google Summer of Code project that
> would include this as one aspect of a semi-automated package generator,
> but I'd hate to see us duplicating effort.
>
> The basic idea I had in mind is a set of plugins, one per language, with
> a sufficient understanding of that language's syntax to determine the
> external files a given piece of code depends on. We'd then couple that
> with a tool like distromatch to figure out which packages the files came
> from.
You'll find several such scripts at:
http://rpm5.org/cvs/dir?d=rpm/scripts

Of recent additions, you'll probably find these two the most interesting:
I've recently written a dependency generator for ruby & ruby gems (ruby code
quality is awful though, not my cup of tea for what conding languages go:p
patches cleaning it up are more than welcome!):
http://rpm5.org/cvs/rlog?f=rpm/scripts/rubygems.rb

And also one for python & python eggs:
http://rpm5.org/cvs/rlog?f=rpm/scripts/pythoneggs.py

--
Regards,
Per Øyvind
___
Distributions mailing list
Distributions@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/distributions

Re: Dependency discovery

[Joseph Tate]

On 04/01/2011 04:11 PM, Donnie Berkholz wrote:
> Do any of you have useful software that scans source code to determine 
> its dependencies? We're considering a Google Summer of Code project that 
> would include this as one aspect of a semi-automated package generator, 
> but I'd hate to see us duplicating effort.
>
> The basic idea I had in mind is a set of plugins, one per language, with 
> a sufficient understanding of that language's syntax to determine the 
> external files a given piece of code depends on. We'd then couple that 
> with a tool like distromatch to figure out which packages the files came 
> from.
>
Conary has some really nifty tools for detecting dependencies.  Some of
it runs on source code, like Python deps.  And some runs on compiled
code, like for shared object dependencies, etc.  I only know about it as
a user/packager though.
___
Distributions mailing list
Distributions@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/distributions

Re: Dependency discovery

[Toshio Kuratomi]

On Fri, Apr 01, 2011 at 03:11:30PM -0500, Donnie Berkholz wrote:
> Do any of you have useful software that scans source code to determine 
> its dependencies? We're considering a Google Summer of Code project that 
> would include this as one aspect of a semi-automated package generator, 
> but I'd hate to see us duplicating effort.
> 
> The basic idea I had in mind is a set of plugins, one per language, with 
> a sufficient understanding of that language's syntax to determine the 
> external files a given piece of code depends on. We'd then couple that 
> with a tool like distromatch to figure out which packages the files came 
> from.
> 
rpm has one for perl.

We've bandied about ideas and sample code for python -- but there's two
forms of python deps; those that are created via egginfo and those that are
a product of importing modules.

I think that we have dep generators for ocaml and erlang as well (these are
shipped in some base erlang and ocaml package in Fedora).

Oh!  Regarding distromatch -- at PyCon, Allison Randall and I were figuring
out how to match packages in Fedora, packages in Ubuntu, and packages on
pypi (upstream python code hosting) together.  We came up with sonmething
pretty interesting.  tarballs.  The name portion of a tarballs/zip file/etc
can be used to match a large number of source packages together.  Even for
packages which generate multiple binary subpackages, this should help narrow
down which binary packages are not even remotely related despite having
similar names.

-Toshio


pgpaZDilAPsdz.pgp
Description: PGP signature
___
Distributions mailing list
Distributions@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/distributions