Re: CPANDB - was: Module::Dependency 1.84

[brian d foy]

In article <[EMAIL PROTECTED]>, Adam
Kennedy <[EMAIL PROTECTED]> wrote:


> Nobody would care about dependencies if they never failed (except for 
> the issue of installation time).

I have a couple of clients that are very skittish about outside
dependencies in general. They have to get thrid-party code legally
approved, etc, and since they can already use Perl they don't need any
special permission to use those modules. One client specifically wasn't
allowed from CPAN (although we solved that with a minicpan on a disk).

I don't make up these rules, but they are out there.

For the CPANDB, as long as I can read the data it in my own program, I
don't much care what CPAN::Index uses.

Re: CPANDB - was: Module::Dependency 1.84

[brian d foy]

In article <[EMAIL PROTECTED]>, Tels
<[EMAIL PROTECTED]> wrote:

>> I am not sure what "stupid" consists, but my system wouldn't have
problems 
> handling 512 MB of memory.

I'd prefer that you not decide that my system needs 512Mb to use this,
even if you can handle that on your side.

Re: CPANDB - was: Module::Dependency 1.84

[brian d foy]

In article <[EMAIL PROTECTED]>, Tels
<[EMAIL PROTECTED]> wrote:


> My real-grand-plan was always to have a CPANDB module that does exactly the 
> following:

I think the latest version of my cpan(1) script does everything you
show, although it doesn't use a local database. It would be nice to
have all of that locally so any program could use it.

Re: CPANDB - was: Module::Dependency 1.84

[Adam Kennedy]

A. Pagaltzis wrote:

* Tels <[EMAIL PROTECTED]> [2006-07-14 02:40]:

announced to early - now everybody tells me how I have to
implement it and why my way wont work


You were saying Adam’s way is inferior to your way. I disagreed.
That’s about the size of it.


I agree my way is inferior, especially on a couple of axis like lightness.

The most value you can get from these lists is to treat them as an 
expert panel. If you put up come idea, people will try to tear it apart, 
and present you with problems you may not have considered.


Despite the particular wording of any given conversation, that is almost 
never to say you SHOULDN'T do it. Just that if you want to do some 
problem, there's people that know the problems you are going to face.


And you want to know these things now, before you get going, so you can 
address them YOURSELF and prove to yourself if they are or not a problem 
for you.


Adam K

Re: CPANDB - was: Module::Dependency 1.84

[Adam Kennedy]

Plus, I planned to

use YAML because it creates a _much_ less heavy overhead and dependency
chain. Using SQLite or similiar is what really creates the problems
with CPANTS - you cant just access the raw database without the
front-end.

Erm, I'm not sure I get you here. The main problem with all the things
that loads the index currently is that they tend to consume 50-100 meg
of ram, and that's just the active index. Add in extra stuff and you're
up into stupid amounts of ram quite quickly.


I am not sure what "stupid" consists, but my system wouldn't have problems 
handling 512 MB of memory.


OTOH, if the raw data in memory is like 100 Megabyte, then where does all 
the data come from? The CPAN index is surely not 100 Megabyte, right?


Raw data inflates into memory, once you factor in Perl's overheads. 
CPAN.pm uses about 50meg to do it, CPANPLUS used to use about 80meg, 
although kane has pulled that back a bit now I believe.


And while doing this temporarily for the purposes of installing a module 
might be fine, I have use cases in mind like running a CPAN index inside 
a Perl editor.


And it would most certainly be "stupid" if your editor uses 250 meg of RAM.


It's already necesary, based on the memory load. That's not to say you
have to use SQLite to DISTRIBUTE or publish the data, just that when you
access and manipulate it computationally, you do it via SQLite.


Well, I guess you have more experience on that than I have. I would have 
tried without it first. 


This method has worked spectacularly well in the JSAN code, which


What is JSAN?

implements something very similar to what I'm heading towards. 


Er, so why aren't you using what JSAN is? *confused*


JSAN is the JavaScript Archive Network, the port of CPAN for JavaScript.

The implementation was a nice chance to blow away all the CPAN cruft and 
try some new ideas out, without having to care about back-compatibility.


Although I'm reusing the same techniques (SQLite + ORM layer) for 
CPAN::Index, I can't reuse the code directly, because there are 
different language concepts involved, so the XPAN client for each needs 
to work differently as a result.


To do 
dependencies and so on, it just links an Algorithm::Dependency object


Another dependency to find out what the dependency is.


I know you don't like dependencies, but your choices come down to 
dependency or reimplementation, if you want some feature. And as long as 
the dependency has 100% platform support and never fails to install, it 
isn't a problem.


The reason CPAN is in the current situation of "dependency == bad" is 
primarily (but not exclusively) that many modules fail on various edge 
cases, and those edge cases have combined effect. Throw in dependency 
recursion and unless you are paranoid about the things you depend on 
being clean, you run into trouble.


Nobody would care about dependencies if they never failed (except for 
the issue of installation time).



The bit that isn't finished is the complete "grab your CPAN.pm config
and stream down all the files and populate the database".


I see you have don muchmuch more work and given it much much more though,
so I better just shut my mouth, throw my silly alittle attempt at 
yet-another-wheel into the dustbin and go away. :-/


Good luck with CPAN::Index.


I'm all for more wheels, I'm just pointing out some issues you might run 
into. And if yours exists and mine doesn't, well yours is far better 
than mine.


Adam K

Re: CPANDB - was: Module::Dependency 1.84

[A. Pagaltzis]

* Tels <[EMAIL PROTECTED]> [2006-07-14 02:40]:
> announced to early - now everybody tells me how I have to
> implement it and why my way wont work

You were saying Adam’s way is inferior to your way. I disagreed.
That’s about the size of it.

Regards,
-- 
Aristotle Pagaltzis // 

Re: CPANDB - was: Module::Dependency 1.84

[Leon Brocard]

On 7/14/06, Tels <[EMAIL PROTECTED]> wrote:


afte seeing the recent discussion about CPANDB and CPAN::Index, I don't
think I want to work on that project anymore for two reasons:


Hey Tels,

Please ignore the other people in the thread. Instead spend a little
bit of time to do something simple that works.

My preferred development methodology is to underpromise and
overdeliver. Mailing lists generally do not help in this regard.

To the other people in this thread: please ignore this mail ;-)

Leon

Re: CPANDB - was: Module::Dependency 1.84

[Tels]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Heya Tim,

afte seeing the recent discussion about CPANDB and CPAN::Index, I don't 
think I want to work on that project anymore for two reasons:

* announced to early - now everybody tells me how I have to implement it and
  why my way wont work
* another project (yet-another-database-for-cpan called CPAN::Index) is on
  the way and much further along - this would just mean nobody will use
  mywork and I don't want to "waste" my time re-inventing the wheel.

I am sorry.

Best wishes,

Tels

- -- 
 Signed on Fri Jul 14 02:33:57 2006 with key 0x93B84C15.
 Visit my photo gallery at http://bloodgate.com/photos/
 PGP key on http://bloodgate.com/tels.asc or per email.

 STOP! We have run out of virgins!

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.2 (GNU/Linux)

iQEVAwUBRLbm+XcLPEOTuEwVAQJYEgf/Q/hyEDlL4XuN9OSC/Y2DSg1LWfqbkBMo
sux8Rh+3ckRvJvEzsmtI8Gprv2h64RjEgoYbyQ8AUhT70csEdZY92/3EisQ3LrJs
LZybg0ROLo3nrizy/kZ6+/WMBIpWVjUroLeky2r2wqGdHCJkP76jDIMR3WLMkKJ2
Pa2MhSzbNnK+qAXSQ8unOspxIqhvOtmFuF7z1zT01uf/F8cj8czrzs5E5DxS4k5/
J/D0Jf+wShjICqP2eq63mM8bhLmsMWviFELSJRSdbDmCnOltiVCegHCLEX37my5I
xo0TMYz39+lUPH7ja1wYGCprWY/rIm6OWD45kB1thBSKPjix4bACPw==
=0z1I
-END PGP SIGNATURE-

Re: CPANDB - was: Module::Dependency 1.84

[A. Pagaltzis]

* Tels <[EMAIL PROTECTED]> [2006-07-13 21:50]:
> On Thursday 13 July 2006 19:40, A. Pagaltzis wrote:
> > * Tels <[EMAIL PROTECTED]> [2006-07-12 19:00]:
> > > Using SQLite or similiar is what really creates the
> > > problems with CPANTS - you cant just access the raw
> > > database without the front-end.
> >
> > All you need is one binary from www.sqlite.org – I don’t
> > follow why this is a problem. And I certainly prefer talking
> > SQL at a database than writing throwaway helper script code
> > to manipulate serialised data structures.
> 
> But what if the DB isn't SQL?

I have no idea how that question relates to anything I said.

> And it can change at a whim?

I have no idea how you can accomodate every possible form of
storage backend unless you build an internal model of your data
and then write mappers to serialise it to all these storage
formats. Now, you are welcome to do that, but I’d prefer to get
some work done instead of spending my time boiling the ocean.

Premature generalisation if the root of all bloat. If you really
do need this level of genericity, you can always refactor later,
with practice telling you just which abstractions you need. But
until then?

YAGNI.

> > > Of course for performance reasons it might not be good, but
> > > since premature optimization is the root of all evil, I'd
> > > say use YAML now, change when nec.
> >
> > Who cares about performance – I don’t want to have to write
> > code to walk data structures just to ask questions about a
> > heap of records. Sometimes with an in-memory data structure I
> > resort to DBD::RAM instead of writing endless chains of maps,
> > greps and foreach loops just because SQL is so much lazier.
> >
> > SQLite is a godsend. (drh++)
> 
> But how can you then switch the database backend if the user
> doesn't want to use helper scripts or the OO interface?

Are you reading the same message that I wrote? Because your
questions aren’t making any sense to me. Where I did I say
anything about requiring the use of helper scripts or an OO
interface? Wasn’t that’s exactly what I said I’d prefer be
avoided?

> I see I wont ever be able to please people that prefer their
> access raw and without a layer in between then.

I have no idea what “raw access” means. I cannot figure out how
you can have data that has no particular representation. Please
explain.

Regards,
-- 
Aristotle Pagaltzis // 

Re: CPANDB - was: Module::Dependency 1.84

[Tels]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Moin,

On Thursday 13 July 2006 19:40, A. Pagaltzis wrote:
> * Tels <[EMAIL PROTECTED]> [2006-07-12 19:00]:
> > Using SQLite or similiar is what really creates the problems
> > with CPANTS - you cant just access the raw database without the
> > front-end.
>
> All you need is one binary from www.sqlite.org – I don’t follow
> why this is a problem. And I certainly prefer talking SQL at a
> database than writing throwaway helper script code to manipulate
> serialised data structures.

But what if the DB isn't SQL? And it can change at a whim?

> > Of course for performance reasons it might not be good, but
> > since premature optimization is the root of all evil, I'd say
> > use YAML now, change when nec.
>
> Who cares about performance – I don’t want to have to write code
> to walk data structures just to ask questions about a heap of
> records. Sometimes with an in-memory data structure I resort to
> DBD::RAM instead of writing endless chains of maps, greps and
> foreach loops just because SQL is so much lazier.
>
> SQLite is a godsend. (drh++)

But how can you then switch the database backend if the user doesn't want to 
use helper scripts or the OO interface?

I see I wont ever be able to please people that prefer their access raw and 
without a layer in between then.

Best wishes,

Tels

- -- 
 Signed on Thu Jul 13 21:41:02 2006 with key 0x93B84C15.
 Visit my photo gallery at http://bloodgate.com/photos/
 PGP key on http://bloodgate.com/tels.asc or per email.

 "Those are my principles, and if you don't like them... well, I have
 others." -- Groucho Marx

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.2 (GNU/Linux)

iQEVAwUBRLaiKHcLPEOTuEwVAQL2XQf+M8HAR4yC3KIIK1B3nW9bxGxPV/vX8eeR
/6v8p+/14veYWIxjwvf7OVXos0RjA0LzIvOnOzjNV1JBDmRebkhU4ago4ZGpMMkL
L6uVM1P9C4aWgTjBxKiQ3jw3+fBGBRlDBd7Z7c4nqY4wQpZg2aS85ol/u8k+39vO
gbeqzf958xy2o62g2bg7FR3l2FjR5bYM8mj+qWNzrdq2PJJOzXb7H56j+yRG9yWK
k7oH9Ce01estiigSWwCKW7UqNExk9HseC48U3QeZ8s0qrqvQBzUp8fZPX6hqgeot
uklbniT+zrw3zcexodzwzYoRJAP3PpXb84qmUmYIcliVcDJLMqrztA==
=bbQk
-END PGP SIGNATURE-

Re: CPANDB - was: Module::Dependency 1.84

[A. Pagaltzis]

* Michael Peters <[EMAIL PROTECTED]> [2006-07-13 19:50]:
> And if you use DBD::SQLite you don't even need that. I has
> everything you'd need.

Actually that’s more of a hassle. If you install DBI::Shell you
get close, but having to specify a DSN is still a hassle. With
the SQLite binary you just say `sqlite mp3.db` and you’re at
cruising speed.

Regards,
-- 
Aristotle Pagaltzis // 

Re: CPANDB - was: Module::Dependency 1.84

[Michael Peters]

A. Pagaltzis wrote:
> * Tels <[EMAIL PROTECTED]> [2006-07-12 19:00]:
>> Using SQLite or similiar is what really creates the problems
>> with CPANTS - you cant just access the raw database without the
>> front-end. 
> 
> All you need is one binary from www.sqlite.org – I don’t follow
> why this is a problem. And I certainly prefer talking SQL at a
> database than writing throwaway helper script code to manipulate
> serialised data structures.

And if you use DBD::SQLite you don't even need that. I has everything you'd 
need.

-- 
Michael Peters
Developer
Plus Three, LP

Re: CPANDB - was: Module::Dependency 1.84

[A. Pagaltzis]

* Tels <[EMAIL PROTECTED]> [2006-07-12 19:00]:
> Using SQLite or similiar is what really creates the problems
> with CPANTS - you cant just access the raw database without the
> front-end. 

All you need is one binary from www.sqlite.org – I don’t follow
why this is a problem. And I certainly prefer talking SQL at a
database than writing throwaway helper script code to manipulate
serialised data structures.

> Of course for performance reasons it might not be good, but
> since premature optimization is the root of all evil, I'd say
> use YAML now, change when nec.

Who cares about performance – I don’t want to have to write code
to walk data structures just to ask questions about a heap of
records. Sometimes with an in-memory data structure I resort to
DBD::RAM instead of writing endless chains of maps, greps and
foreach loops just because SQL is so much lazier.

SQLite is a godsend. (drh++)

Regards,
-- 
#Aristotle
*AUTOLOAD=*_;sub _{s/(.*)::(.*)/print$2,(",$\/"," ")[defined wantarray]/e;$1};
&Just->another->Perl->hacker;

Re: CPANDB - was: Module::Dependency 1.84

[Tels]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Moin,

On Thursday 13 July 2006 08:26, Adam Kennedy wrote:
> Tels wrote:
> > The DB backend shouldn't matter at all, it should be transparent and be
> > switchable without any noticable change at the front.
>
> Yep, right with you. Hence DBIx::Class.

*sigh* Another dependecy...

> Plus, I planned to
> > use YAML because it creates a _much_ less heavy overhead and dependency
> > chain. Using SQLite or similiar is what really creates the problems
> > with CPANTS - you cant just access the raw database without the
> > front-end.
>
> Erm, I'm not sure I get you here. The main problem with all the things
> that loads the index currently is that they tend to consume 50-100 meg
> of ram, and that's just the active index. Add in extra stuff and you're
> up into stupid amounts of ram quite quickly.

I am not sure what "stupid" consists, but my system wouldn't have problems 
handling 512 MB of memory.

OTOH, if the raw data in memory is like 100 Megabyte, then where does all 
the data come from? The CPAN index is surely not 100 Megabyte, right?

I didn't intend to read all of it into ram and keep it there, thats for 
sure.

But thats why I also wrote that YAML might be replaced by something else.

> > WithYAML, at least you could get at the data by other means. Of course
> > for performance reasons it might not be good, but since premature
> > optimization is the root of all evil, I'd say use YAML now, change when
> > nec.
>
> It's already necesary, based on the memory load. That's not to say you
> have to use SQLite to DISTRIBUTE or publish the data, just that when you
> access and manipulate it computationally, you do it via SQLite.

Well, I guess you have more experience on that than I have. I would have 
tried without it first. 

> This method has worked spectacularly well in the JSAN code, which

What is JSAN?

> implements something very similar to what I'm heading towards. 

Er, so why aren't you using what JSAN is? *confused*

> To do 
> dependencies and so on, it just links an Algorithm::Dependency object

Another dependency to find out what the dependency is.

> into the DBIx::Class code, and there you have your full range of
> graph-math things, like dep chains (up and downstream) weights, etc.

That sounds like quite a bit overkill.

Well, I guess I just wanted to keep the dependecies to a very minimum. I 
rally _DONT_ want to install 20 different modules just to find out what 
Foo::Bar has on dependencies. Thats why CPANTS never worked for me, I never 
could find the patience to hunt down and install everything just to get it 
working. And I guess every user of Graph::Dependency doesn't want to do 
that, either.

However, at the current stage I would settle for a working method, almost 
any method at all.

> > Otherwise, the projects seem similiar in scope, except that I focus on
> > the DB and let things like "download stuff" be done outside.
> >
> > Whether that works out, uh I don't know. In any event I have quite a
> > few ideas I'd like to try out and this proves to be fun to me. I hope
> > it doesn't end upI have to implement other people ideas - thats too
> > much work like :D
>
> Have a look at just the database parts, which I've finished already from
> memory.
>
> The bit that isn't finished is the complete "grab your CPAN.pm config
> and stream down all the files and populate the database".

I see you have don muchmuch more work and given it much much more though,
so I better just shut my mouth, throw my silly alittle attempt at 
yet-another-wheel into the dustbin and go away. :-/

Good luck with CPAN::Index.

Best wishes,

Tels

- -- 
 Signed on Thu Jul 13 18:29:40 2006 with key 0x93B84C15.
 Visit my photo gallery at http://bloodgate.com/photos/
 PGP key on http://bloodgate.com/tels.asc or per email.

 "In my opiniation, "burglarize" is a perfectionally validative
 wordification. How else would reportization of the securitial/policial
 forceship appearize to be importantive enoughly to be respectative by
 the massmediaship and influentate the societyness?" SharpFang (651121)
 on 2004-12-13 at /. about "burgle" vs. "burglarize"

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.2 (GNU/Linux)

iQEVAwUBRLZ2x3cLPEOTuEwVAQLOIAf/QITUcv+L68ekV/4TfpUVM0akC9slGCLM
0FoUdAkCIvd8PbwZTgGenc9aYOU3/2h41PB5rV5ub+LNAQYT2DyjUs+kMZPHkoAo
LO20UQZMtAwQKl65gI11XBSh5bIJEbVh5UVnSubv9uKcokYTDHEgiQzppI52/BT1
jKXW9/dX2jbn74+AjhtnpIcvANfs/DeYcJeCOoqacQCTo7Wer0HZEmxKmlOppgJL
YjCK6X9DY+HYstIO4bZXn1NUxqXIKsvjK38I7HlspODyCMqlLzyj0O+lubcGT/A4
EyKJHh6nSDgbb7X+eM1104uvrgeRp49F/yNAceSDRSssWuLGpxoOmA==
=I7Ex
-END PGP SIGNATURE-

Re: CPANDB - was: Module::Dependency 1.84

[Adam Kennedy]

Tels wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Moin,

On Wednesday 12 July 2006 03:13, David Golden wrote:

Tels wrote:

My idea was to build _only_ the database, and do it right, simple and
easy to use and then get everyone else to just use the DB instead of
fiddling with their own. (simple by having the database being superior
to every other hack thats in existance now :-)

I even got so far as to do a mockup v0.02 - but then went back to
playing Guildwars.

Is this a project that would be of general interest?

At YAPC::NA, Adam Kennedy mention that he wanted to try to make some
headway on CPAN::Index, which sounds very similar in intent.  While it's
not released, you can see the formative project at his public repository:

http://tinyurl.com/g888h

Perhaps you can join forces with him and help push some collective
project towards a release.


Quoted:


B provides object-oriented access to the CPAN index,
using a collection of relatively common modules, and automates
entire process of fetching and accessing the index.


Uhm, no the DB should maybe be able to have a front-end that fills it from 
the CPAN index, but you should also be able to build a DB from a local 
file, if you so wish.


Well of course, and that's what it does.

If has a database structure with a DBIx::Class layer to the schema for a 
CPAN index, and it handily comes by default with modules to download the 
main CPAN index and stream it into the database.


If we wanted to add a different mechanism to populate the same database, 
that would be just fine. In fact, part of what I want is to create an 
equivalent of CPAN::Inject, to be able to build out a custom "modified 
CPAN" that also includes commercial modules.



The index is stored in a L database file, with an object
model implemented around it using L. To update the index,
the L class implements the logic to flush and reset
the database, fetch the index files, parse them, and repopulate the
database.


The DB backend shouldn't matter at all, it should be transparent and be 
switchable without any noticable change at the front.


Yep, right with you. Hence DBIx::Class.

Plus, I planned to
use YAML because it creates a _much_ less heavy overhead and dependency 
chain. Using SQLite or similiar is what really creates the problems with 
CPANTS - you cant just access the raw database without the front-end.


Erm, I'm not sure I get you here. The main problem with all the things 
that loads the index currently is that they tend to consume 50-100 meg 
of ram, and that's just the active index. Add in extra stuff and you're 
up into stupid amounts of ram quite quickly.


WithYAML, at least you could get at the data by other means. Of course for 
performance reasons it might not be good, but since premature optimization 
is the root of all evil, I'd say use YAML now, change when nec.


It's already necesary, based on the memory load. That's not to say you 
have to use SQLite to DISTRIBUTE or publish the data, just that when you 
access and manipulate it computationally, you do it via SQLite.


This method has worked spectacularly well in the JSAN code, which 
implements something very similar to what I'm heading towards. To do 
dependencies and so on, it just links an Algorithm::Dependency object 
into the DBIx::Class code, and there you have your full range of 
graph-math things, like dep chains (up and downstream) weights, etc.


Otherwise, the projects seem similiar in scope, except that I focus on the 
DB and let things like "download stuff" be done outside.


Whether that works out, uh I don't know. In any event I have quite a few 
ideas I'd like to try out and this proves to be fun to me. I hope it 
doesn't end upI have to implement other people ideas - thats too much work 
like :D


Have a look at just the database parts, which I've finished already from 
memory.


The bit that isn't finished is the complete "grab your CPAN.pm config 
and stream down all the files and populate the database".


Adam K

Re: CPANDB - was: Module::Dependency 1.84

[Tim Bunce]

On Wed, Jul 12, 2006 at 03:03:14AM +0200, Tels wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
> Moin Tim,
> 
> On Tuesday 11 July 2006 18:34, Tim Bunce wrote:
> > I needed some code to trawl through a directory tree parsing perl
> > modules and scripts to determine their dependencies.
> >
> > The closest existing CPAN code was Module::Dependency but it fell short
> > of what I needed. The original author (P Kent) has passed over
> > maintenance to me. My latest release is:
> >
> >   file: $CPAN/authors/id/T/TI/TIMB/Module-Dependency-1.84.tar.gz
> >   size: 52161 bytes
> >md5: 90a83b2aee39f5d25060ebdb6cc3105d
> 
> Thats sort of cool, although I havent looked at it yet :-D
> 
> My real-grand-plan was always to have a CPANDB module that does exactly the 
> following:
> 
> maintains a database with:
> 
> * every CPAN author with all details (ID, email, name)
> * every package with all details (author id, version, name etc etc)

> This database could then be used by all the following modules:
> 
>   Module::Dependency
>   Graph::Dependency
>   CPANPLUS
>   CPANTS
>   CPAN
>   Module::Author
> 
> and a few others I forgot. Basically by every module out there that 
> re-invents the wheel over and over again just to be able to query stuff 
> about CPAN modules. (some of them do really horrible stuff like accessing 
> search.cpan.org - I know I wrote one of them :D

A key point about those modules (other than Module::Dependency) is that they
don't work for private modules (the so called "Dark CPAN") nor for scripts.

Module::Dependency handles both (details changed to protect the guilty):

$ pmd_dump.pl adnetwork/www/cron/cv.pl
adnetwork/www/cron/cv.pl depended_upon_by: 
adnetwork/www/cron/cv.pl depends_on: strict File::Basename lib Env wConfig 
vConfig Churl::Log Churl::MyDBI Churl::Util

$ pmd_dump.pl Churl::MyDBI 
Churl::MyDBI depended_upon_by: www/cron/cv.pl www/cron/pub_login.pl
Churl::MyDBI depends_on: Exporter strict warnings Carp DBIx::DWIW 
File::Basename Sys::Syslog vars
Churl::MyDBI filename: clcyt/Churl/lib/Churl/MyDBI.pm


> And up until today it is still not possible to get easily the answer "what 
> modules do I need install for Foo::Bar 1.23 when using Perl 5.8.x".

Or what module may be affected if I upgrade Foo::Bar.

> My idea was to build _only_ the database, and do it right, simple and easy 
> to use and then get everyone else to just use the DB instead of fiddling 
> with their own. (simple by having the database being superior to every 
> other hack thats in existance now :-)
> 
> I even got so far as to do a mockup v0.02 - but then went back to playing 
> Guildwars.
> 
> Is this a project that would be of general interest?

Yes! But widen your horizon, and schema, beyond just CPAN.
I'd be interested in helping out and, if it works out, perhaps migrate
Module::Dependency to use it.

Tim.

Re: CPANDB - was: Module::Dependency 1.84

[Tels]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Moin,

On Wednesday 12 July 2006 03:13, David Golden wrote:
> Tels wrote:
> > My idea was to build _only_ the database, and do it right, simple and
> > easy to use and then get everyone else to just use the DB instead of
> > fiddling with their own. (simple by having the database being superior
> > to every other hack thats in existance now :-)
> >
> > I even got so far as to do a mockup v0.02 - but then went back to
> > playing Guildwars.
> >
> > Is this a project that would be of general interest?
>
> At YAPC::NA, Adam Kennedy mention that he wanted to try to make some
> headway on CPAN::Index, which sounds very similar in intent.  While it's
> not released, you can see the formative project at his public repository:
>
> http://tinyurl.com/g888h
>
> Perhaps you can join forces with him and help push some collective
> project towards a release.

Quoted:

>B provides object-oriented access to the CPAN index,
>using a collection of relatively common modules, and automates
>entire process of fetching and accessing the index.

Uhm, no the DB should maybe be able to have a front-end that fills it from 
the CPAN index, but you should also be able to build a DB from a local 
file, if you so wish.

>The index is stored in a L database file, with an object
>model implemented around it using L. To update the index,
>the L class implements the logic to flush and reset
>the database, fetch the index files, parse them, and repopulate the
>database.

The DB backend shouldn't matter at all, it should be transparent and be 
switchable without any noticable change at the front. Plus, I planned to 
use YAML because it creates a _much_ less heavy overhead and dependency 
chain. Using SQLite or similiar is what really creates the problems with 
CPANTS - you cant just access the raw database without the front-end. 
WithYAML, at least you could get at the data by other means. Of course for 
performance reasons it might not be good, but since premature optimization 
is the root of all evil, I'd say use YAML now, change when nec.

Otherwise, the projects seem similiar in scope, except that I focus on the 
DB and let things like "download stuff" be done outside.

Whether that works out, uh I don't know. In any event I have quite a few 
ideas I'd like to try out and this proves to be fun to me. I hope it 
doesn't end upI have to implement other people ideas - thats too much work 
like :D

Best wishes,

Tels

- -- 
 Signed on Wed Jul 12 18:54:01 2006 with key 0x93B84C15.
 Visit my photo gallery at http://bloodgate.com/photos/
 PGP key on http://bloodgate.com/tels.asc or per email.

 "When it's done in 2001." - 2000 Christmas card about Duke Nukem Forever
 (http://tinyurl.com/6m8nh)

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.2 (GNU/Linux)

iQEVAwUBRLUqL3cLPEOTuEwVAQK3DAf/VEIV3fCdHTMinHZqL9N3JTxAHBIVZSlw
4grWxod0u3/8DiAhHkSjNu4y0XjE2WV1LrI6JWFD+4/qoElUhvMFydLYXP8cG31X
CnXGuAFvq6TVKXCQbHWWucSlcHrILc648i5sCkmi+OTJYUP1AP0/IWdyEbgevWp3
7daHed1UvCetKYyWwTNwgxerTwHAyziFd8kCR2pWzsXUve3zcHhEudm+KF6q6mOI
K069eG99V0R5Ix5nisYAdJdei1+TQ8SSXmNZ30VKju8YFTRKcWQNMmWcdxqErydB
19E5C8HDpXAiafUizpmvG8/tChioFjOS+bbsV/AInid6BcQDBDXONg==
=2qra
-END PGP SIGNATURE-

Re: CPANDB - was: Module::Dependency 1.84

[David Golden]

Tels wrote:
My idea was to build _only_ the database, and do it right, simple and easy 
to use and then get everyone else to just use the DB instead of fiddling 
with their own. (simple by having the database being superior to every 
other hack thats in existance now :-)


I even got so far as to do a mockup v0.02 - but then went back to playing 
Guildwars.


Is this a project that would be of general interest?


At YAPC::NA, Adam Kennedy mention that he wanted to try to make some 
headway on CPAN::Index, which sounds very similar in intent.  While it's 
not released, you can see the formative project at his public repository:


http://tinyurl.com/g888h

Perhaps you can join forces with him and help push some collective 
project towards a release.


Regards,
David Golden

CPANDB - was: Module::Dependency 1.84

[Tels]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Moin Tim,

On Tuesday 11 July 2006 18:34, Tim Bunce wrote:
> I needed some code to trawl through a directory tree parsing perl
> modules and scripts to determine their dependencies.
>
> The closest existing CPAN code was Module::Dependency but it fell short
> of what I needed. The original author (P Kent) has passed over
> maintenance to me. My latest release is:
>
>   file: $CPAN/authors/id/T/TI/TIMB/Module-Dependency-1.84.tar.gz
>   size: 52161 bytes
>md5: 90a83b2aee39f5d25060ebdb6cc3105d

Thats sort of cool, although I havent looked at it yet :-D

My real-grand-plan was always to have a CPANDB module that does exactly the 
following:

maintains a database with:

* every CPAN author with all details (ID, email, name)
* every package with all details (author id, version, name etc etc)

You would get current packages and old, for performance reasons in tow Dbs 
but this would be transparent for the user. There would be two interfaces:

  # cpandb --query Math::BigInt
  name: Math::Bigint
  author: TELS
  version: v1.77
  latest: v1.77
  

  # cpandb --update Math-BigInt-1.78.tar.gz
  package Math::Bigint updated.
  New entry:
  name: Math::Bigint
  author: TELS
  version: v1.78
  latest: v1.78
  

  # cpandb --query Math-BigInt-1.77
  name: Math::Bigint
  author: TELS
  version: v1.77
  latest: v1.78
  

and so on. This script would also update, add etc to the DB, much like RPM. 
(it only maintains the DB, no code no install etc).

The other interface is Perl OO like so:

use CPANDB;

my $db = CPANDB->open();# default location

print $db->author('TELS');
my $m = $db->latest_package'(Math::BigInt');

print $m->depends_on();

etc.

The Db would be empty, but with a suitale wrapper around cpandb could be 
filled, or you could download a pre-made one. Pause could even maintain on 
on their own, because then it would be always update.

This database could then be used by all the following modules:

Module::Dependency
Graph::Dependency
CPANPLUS
CPANTS
CPAN
Module::Author

and a few others I forgot. Basically by every module out there that 
re-invents the wheel over and over again just to be able to query stuff 
about CPAN modules. (some of them do really horrible stuff like accessing 
search.cpan.org - I know I wrote one of them :D

And up until today it is still not possible to get easily the answer "what 
modules do I need install for Foo::Bar 1.23 when using Perl 5.8.x".

(Why not use CPANTs database? Easy, because the interface module is as far 
as I know broken since a few years, the database changed formats a few 
times (breaking any interfacing module again), its undocumented (that is 
not the documentation you are looking for) and has a few other small 
problems, like no command line interface etc. And it is tightly integrated 
into CPANTS and it has a lot of dependency like sql or postgres or 
whatever. All the other solutions out there have some of the same problems 
or other problems of the same kind.

The most common theme seems to be that everyone wants what the DB does, plus 
something more (website generator, graph generator, installator and a shell 
etc). And since no real DB exists yet, everyone invents their own DB 
backend again :)

My idea was to build _only_ the database, and do it right, simple and easy 
to use and then get everyone else to just use the DB instead of fiddling 
with their own. (simple by having the database being superior to every 
other hack thats in existance now :-)

I even got so far as to do a mockup v0.02 - but then went back to playing 
Guildwars.

Is this a project that would be of general interest?

best wishes,

Tels

PS: Sorry for the rambling on, its 3:00 am local time and I should be asleep 
since 3  or 4 hours. And this email should have been written like 2 months 
ago. I hope someone understand what I wanted to say :-D

- -- 
 Signed on Wed Jul 12 02:58:38 2006 with key 0x93B84C15.
 Visit my photo gallery at http://bloodgate.com/photos/
 PGP key on http://bloodgate.com/tels.asc or per email.

 I am looking for a Perl-related job (no, Python is not Perl-related :-).
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.2 (GNU/Linux)

iQEVAwUBRLRKWncLPEOTuEwVAQI9TQf+I9xoOMGGEj6V5poS4VUX9fVN8FcPyYlP
jdT6PGCfurG7kGs2zuhyPYBM+W6rlk5wvrfI30KoDtxzDuCyCdogNIWekpjtnWB+
Ei72dZdVtiWEADxLopGG2gAMjhE4OPKo7qkmiox3sZQj/9jpU87nHCDpVHxrC1as
QlVJUaA3MGl75mYepiMoyoBcGfjyWsVzd59rouymwp4/NVpSfU+z4/cZU4KRFNb+
RuKmYXQKSfSsQe4Jn1o69ZCkzPOlZ2NC0HfVN6ZXCZ/iz5/CZTq3nT6iM/JBOIOE
HOAqwJzz3fkdS1z/AlKaoDbKNv+tBaH+6+wyX15dzr5f3UKR/2pb2g==
=lDv9
-END PGP SIGNATURE-