Re: wget scripting?

[Aaron S. Hawley]

Such a design would be nice.  It would also be great to make an extensible
API (like Guile scheme) available along the way for allowing specific
program behaviors (for example: obscure accept/reject rules) to be created
by users.

That seemed like an application design suggestion, but an actual Wget
"scripting"  project was being developed, but it seems to have fallen off
the face of the earth:

wgsgen - Wget Script generator
see: http://directory.fsf.org/devel/specific/wgetsg.html

Cheers.
/a

On Mon, 4 Oct 2004, Juhana Sadeharju wrote:

> Hello.
>
> I have slightly thought how to make wget more better, possibly.
> We would need a scripting system so that features can be programmed
> more easily. One way how to incorporate the scripting to wget would
> be to re-write wget as a data flow system. Much similar way than
> OpenGL (www.opengl.org) is a data flow for graphics. The scripts
> would be executed in specific places in the data flow graph.
> Much similar way than vertex and fragment programs are executed
> in OpenGL in the specific places of the graph.
>
> So, the urls would enter the data flow and the routines in the
> graph would do something to them. I don't know yet what kind of
> graph we would have but here is a simple one:
>
>   url input --> url processing --> site exclusion --> dir path exclusion
>   --> get file -->
>
> Then the graph goes deeper in parsing the html.
>
> Example: I could add a script just after the "get file". The script
> would uncompress the downloaded file to a new file and change the
> local_filename variable to the name of the new file.
>
> The graph would make it possible to use different granularity.
> Details can be added by splitting the large graph nodes later.
>
> Regards,
> Juhana

-- 
PINE 4.58 Mailer - www.washington.edu/pine/
source-included, proprietary, gratis, text-based, console email client

Re: 2 giga file size limit ?

[Leonid]

Hallo, Noel,
  I would suggest you to take 
http://download.fedora.redhat.com/pub/fedora/linux/core/development/SRPMS/wget-1.9.1-17.src.rpm
then look carefully at redhat patches.

  I took wget-cvs as a basic, because I had an impression there were
important bug fixes there. FC folks took wget 1.9.1, applied a battery
of their patches, then tried to create their own LFS patch using my
patch as a basis. This required an extra work which unfortunately
(see https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=123524 )
did not go smoothly, but eventually it has succeeded. I do not know new 
bugs in FC wget-1.9.1-17.src.rpm

Leonid

wget scripting?

[Juhana Sadeharju]

Hello.

I have slightly thought how to make wget more better, possibly.
We would need a scripting system so that features can be programmed
more easily. One way how to incorporate the scripting to wget would
be to re-write wget as a data flow system. Much similar way than
OpenGL (www.opengl.org) is a data flow for graphics. The scripts
would be executed in specific places in the data flow graph.
Much similar way than vertex and fragment programs are executed
in OpenGL in the specific places of the graph.

So, the urls would enter the data flow and the routines in the
graph would do something to them. I don't know yet what kind of
graph we would have but here is a simple one:

  url input --> url processing --> site exclusion --> dir path exclusion
  --> get file -->

Then the graph goes deeper in parsing the html.

Example: I could add a script just after the "get file". The script
would uncompress the downloaded file to a new file and change the
local_filename variable to the name of the new file.

The graph would make it possible to use different granularity.
Details can be added by splitting the large graph nodes later.

Regards,
Juhana