[Bug-wget] [GSOC 2017] Design and implementation of a statistics module

[Avinash Sonawane]

Hello!

I am Avinash Sonawane, studying at Pune Institute of Computer
Technology in final year of engineering. I know I'm late as I was busy
with mid-sem exams but now I am eager to make the best of whatever
time is left.

I have a considerable experience in C. And I'm interested in "Design
and implementation of a statistics module" as the subject line
suggests.

I have cloned the github wget2 repo and have installed it on my
machine and I just can't wait to get my hands dirty with the code.

I found http://wget.addictivecode.org/NavigatingTheSource.html but
it's outdated (I guess it's applicable to wget1.x) Where can I find
the similar doc for wget2 to get myself acquainted with the overall
architecture including but not limited to call graph and such?

Also can I please get few low hanging fruits to try my hands on?

Thank you.

Regards,
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

[Bug-wget] libwget documentaion

[Avinash Sonawane]

Hello!

I can't seem to find any documentation on libwget. Is it there?

I hope there is a doc somewhere as we want to use libwget APIs
wherever possible instead of reinventing the wheel and it's quite
cumbersome to go through the whole libwget each time you want to
change/add something.

BTW this is in reference to this issue
https://github.com/rockdaboot/wget2/issues/143

-- 
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

Re: [Bug-wget] [GSOC 2017] Design and implementation of a statistics module

[Avinash Sonawane]

On Mon, Mar 20, 2017 at 9:39 PM, Darshit Shah  wrote:

> As for some low hanging fruits, have you tried looking into the bug tracker?
> DO you see something interesting?

Yes. Currently I'm working on https://github.com/rockdaboot/wget2/issues/143

> Apart from that, try using Wget2 as much as possible. Try to find cases,
> where it breaks compatibility with Wget 1.x, or corner cases that break
> expectations / cause wget2 to crash. File issues about those and see if you
> would like to solve those issues.

Ok. Good idea. Thank you!

>
> Another simple low hanging fruit is to improve the test coverage of libwget
> and wget2. Try to write new unit tests and integration tests to ensure that
> regressions are not introduced in the future.

Haven't written unit tests before. Let me read more up on how to write
unit tests in C. I don't think it'll be that difficult though.

-- 
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

Re: [Bug-wget] [GSOC 2017] Design and implementation of a statistics module

[Avinash Sonawane]

On Mon, Mar 20, 2017 at 9:39 PM, Tim Ruehsen  wrote:

> There is not yet a similar doc, but it is time to write it. This is low
> hanging fruit #1 :-) If you find some time to start a document, maybe during
> the process of diving into the code, that would be highly appreciated.

Sure. I'll add the docs to wiki. But first let me fix few hangs. :)

> - issue #143 (you already started that)
> - issue #44 (HTML entities)
> - issue #43 (Cookie extensions)
> - improve test code coverage (see 'make check-coverage')
> - add missing cli options from wget1.x (ignore ftp and WARC options), see
> https://github.com/rockdaboot/wget2/wiki for a list.
> - docs

Thanks Tim! Just what I was looking for.

-- 
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

Re: [Bug-wget] libwget documentaion

[Avinash Sonawane]

On Mon, Mar 20, 2017 at 8:24 PM, Tim Ruehsen  wrote:

> Currently the HTML docs are built in doc/html.

Oops! I missed that. Thank you.

> I just added Github Pages, which are under construction now and should soon be
> available at https://rockdaboot.github.io/wget2/.

It's perfect.

-- 
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

[Bug-wget] [GSOC 2017] make check fails almost all the tests on fresh clone of wget2

[Avinash Sonawane]

Hello!

As I have mentioned earlier I have built and installed wget2 on my
system. But I hadn't performed `male check` since I wasn't about to
write unit tests.

But now as I am extending tests/test-parse-html.c to include new unit
tests I did `make check`.

Now there are two problems:
1) make check is utterly slow. I can see my CPU usage going up to
1-2%. That's it! It's almost an hour or and it's still running with
this output:
FAIL: test
FAIL: test-wget-1
FAIL: test-restrict-ascii
FAIL: test-i-http
FAIL: test-i-https
FAIL: test-np
FAIL: test--spider-r
FAIL: test-meta-robots
FAIL: test-idn-robots
FAIL: test-idn-meta
FAIL: test-idn-cmd
FAIL: test-iri
FAIL: test-iri-percent
FAIL: test-iri-list
FAIL: test-iri-forced-remote
FAIL: test-auth-basic
PASS: test-parse-html
...

How much time does it take usually? I haven't configured to use Valgrind yet.

2) All the tests before test-parse-html have failed and it's still
running so have no idea how many more will fail. Let me remind you I'm
performing `make check` on fresh clone and haven't modified it in any
manner. Is this an expected behavior? What does FAIL tests mean? Can
we do something about them?

Any help will be deeply appreciated.

Thank you.

Regards,
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

Re: [Bug-wget] [GSOC 2017] make check fails almost all the tests on fresh clone of wget2

[Avinash Sonawane]

On Wed, Mar 22, 2017 at 1:10 PM, Ander Juaristi  wrote:
> Did you pass any flags or env vars to configure?
No. Nothing. I just did `./bootstrap` then `./configure` then `make`
and then `make check`

BTW those tests finished running.

Testsuite summary for wget2 1.0.0

# TOTAL: 30
# PASS:  1
# SKIP:  0
# XFAIL: 0
# FAIL:  29
# XPASS: 0
# ERROR: 0

Also I see many "Failed to resolve 'http:80 (Name or service not
known)" in tests/test-suite.log

I think I should mention that I'm using institute proxy server to
access Internet. I don't think it should matter as tests are run on
localhost.
-- 
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

Re: [Bug-wget] [GSOC 2017] make check fails almost all the tests on fresh clone of wget2

[Avinash Sonawane]

On Wed, Mar 22, 2017 at 1:31 PM, Tim Ruehsen  wrote:

> Did you install all prerequisites ?(see output from ./configure)

h I didn't install anything explicitly as I thought if there's
something missing ./configure will complain. But it didn't so I went
ahead with make. But now let me explicitly check if all the required
dependencies have been installed or not.

> Send us your config.log and tests/test.log.

Oh I just did `make check TESTS=test-parse-html` to check the test
file I'm interested in and it screwed the test-suit.log as earlier log
for all tests have been overwritten. Let me run `make check` again.


-- 
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

Re: [Bug-wget] [GSOC 2017] make check fails almost all the tests on fresh clone of wget2

[Avinash Sonawane]

On Wed, Mar 22, 2017 at 1:49 PM, Tim Ruehsen  wrote:

>> Also I see many "Failed to resolve 'http:80 (Name or service not
>> known)" in tests/test-suite.log
>
> Should be http://localhost:80 ...

As I don't have earlier test-suite.log which contained the log for all
31 tests. And also since I hadn't installed autogen earlier I'm
installing all the dependencies and wget2 from scratch. I'll then run
the tests again and will get back to you.

> what does this output:
> $ set|grep -i prox

It shows all my environment variables with proxy set to them.
Here's complete output http://pastebin.com/UNPcgRHC

-- 
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

Re: [Bug-wget] [GSOC 2017] make check fails almost all the tests on fresh clone of wget2

[Avinash Sonawane]

On Wed, Mar 22, 2017 at 2:27 PM, Tim Ruehsen  wrote:

> Yup, that's it.
>
> $ cd tests
> $ http_proxy='http://192.168.8.253:3128/' ./test-i-http
>
> And it hangs...

I confirm that it's indeed a proxy issue. As I reclonned the repo,
installed all the dependencies by hand (it turn out that I already had
all the required non-optional packages installed!) and it still hangs
for `make check`

>
> I'll create one or two issues... if you like to, work on these. They are
> blockers for you, you have the best test environment. And they are easy to 
> fix.

I'm all game!

-- 
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

[Bug-wget] `make check` fails 4 tests on fresh wget2 clone

[Avinash Sonawane]

Hello!

I just cloned the fresh wget2 repo and did `make check` but it failed
for 4 tests. I haven't configured to use valgrind.

Is this an expected behavior?

Here is the tests/test-suite.log http://pastebin.com/g1crJBJ7

PS - No. I'm not using proxy connection. Also I haven't set any proxy variables.
-- 
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

Re: [Bug-wget] `make check` fails 4 tests on fresh wget2 clone

[Avinash Sonawane]

On Thu, Mar 23, 2017 at 2:34 PM, Tim Ruehsen  wrote:

> We could:
> 1. Skip those tests if IDNA is not available
> 2. Make IDNA 200x a hard build prerequisite
>
> WDYT ?

I think let's go with skipping those 4 tests if IDNA is not available.
Making IDNA 200x a hard build prerequisite is totally unnecessary as
an ordinary user is not building `wget2` to perform `wget2` tests :P

-- 
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

Re: [Bug-wget] `make check` fails 4 tests on fresh wget2 clone

[Avinash Sonawane]

On Thu, Mar 23, 2017 at 2:34 PM, Tim Ruehsen  wrote:

> You should have libidn2/libidn2-dev (IDNA 2008 + TR46) installed.
I had installed libidn2 on my system already. it turned out that I
needed to install libidn2-dev instead. Now all tests pass
successfully.

BTW https://github.com/rockdaboot/wget2 lists libidn2 as requirement.
I think we should update it to libidn2-dev

-- 
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

Re: [Bug-wget] `make check` fails 4 tests on fresh wget2 clone

[Avinash Sonawane]

On Thu, Mar 23, 2017 at 3:13 PM, Avinash Sonawane  wrote:
> On Thu, Mar 23, 2017 at 2:34 PM, Tim Ruehsen  wrote:
>
>> We could:
>> 1. Skip those tests if IDNA is not available
>> 2. Make IDNA 200x a hard build prerequisite
>>
>> WDYT ?
>
> I think let's go with skipping those 4 tests if IDNA is not available.
> Making IDNA 200x a hard build prerequisite is totally unnecessary as
> an ordinary user is not building `wget2` to perform `wget2` tests :P

On a second thought not having libidn2 also means that the wget2 will
not work with non-ascii domain names. Now that's a bummer!

I retract my earlier view. Let's make libidn2 as a hard prerequisite.

>
> --
> Avinash Sonawane (rootKea)
> PICT, Pune
> https://rootkea.wordpress.com



-- 
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

Re: [Bug-wget] `make check` fails 4 tests on fresh wget2 clone

[Avinash Sonawane]

On Thu, Mar 23, 2017 at 2:34 PM, Tim Ruehsen  wrote:

> We could:
> 1. Skip those tests if IDNA is not available
> 2. Make IDNA 200x a hard build prerequisite

Oh wait. Were you talking about making idna2 a hard prerequisite
similar to what curl do? It seems I misunderstood you. I should take a
time off!

BTW if that's indeed what you asked then I'm in favor of dropping idna
and moving to idna2.

-- 
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

Re: [Bug-wget] GSoC 2017 "GNU Wget" by SL

[Avinash Sonawane]

On Thu, Mar 23, 2017 at 5:13 PM, Sumit Lakra  wrote:
> Hello there,

Hello Sumit! Welcome to wget!

> I am still going through the
> wget2 code, and I would be extremely grateful if you could guide me to any
> particular resource or tell me how to proceed with this task.. like which
> part of code do I need to get really acquainted with and other stuff.

First clone the wget2 repo from https://github.com/rockdaboot/wget2
and build it on your system. Then to get acquainted with the code-base
you can pick an issue from https://github.com/rockdaboot/wget2/issues
and try to fix it.

You can find the required documentation in docs/ directory after you
build the project. Docs are not complete per se. We are working on
them.

Regards,
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

[Bug-wget] [GSoC] Git & Github workflow

[Avinash Sonawane]

Hello!

I am familiar with git. But currently I'm feeling pretty overwhelmed
with git operations. There are so many of them!

Also we are coordinating wget2 development on Github, which is though
a front-end to git but has its own work-flow. You have fork, pull
request etc.

I have read sufficient git documentation from https://git-scm.com/doc
and manpages so I'm very quite familiar with the git data model and
its 3 states etc. I have also gone through the considerable Github
documentation at https://help.github.com/ I even went through
https://try.github.io

Now, I'm sure other students appearing for GSoC must have gone through
the docs like me. But the problem is to figure out the efficient
minimal work-flow while dealing with the git and github with respect
to wget2 development.

For starters my current (flawed) work-flow is this :
1) fork the upstream repo. Let's call it origin (using Github)
2) clone the origin/master (using git)
3) Pickup an issue (using Github)
4) For a said issue create new branch (git checkout -b)
5) `add` and `commit` changes to branch (using Git)
6) Push changes to origin (git push origin foo)
7) Send a pull request to merge branch foo in rockdaboot/wget (using Github)

Periodically keep synching the fork which involves:
1) add remote upstream <- just once
2) fetch upstream
3) checkout master
4) merge upstream/master

Now I am sure there's an efficient minimal operation workflow to deal
with Git and Github with respect to wget2 development. I am interested
in knowing what's your Git-Github workflow? Please no tutorial.
There's tons of tutorials/docs on the Internet. Just the workflow you
use.

I am quite sure this will be very helpful to current and future GSoC
students who are most often new to distributed version control
systems, particularly Git and Github.

If we get helpful responses, I will add this "contributing guide for
GSoC students" to our wiki.

Thank you.

Regards,
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

Re: [Bug-wget] [GSoC] Git & Github workflow

[Avinash Sonawane]

On Thu, Mar 23, 2017 at 9:08 PM, Avinash Sonawane  wrote:

> Periodically keep synching the fork which involves:
> 1) add remote upstream <- just once
> 2) fetch upstream
> 3) checkout master
> 4) merge upstream/master

5) checkout foo
6) merge upstream/master

-- 
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

Re: [Bug-wget] [GSoC] Git & Github workflow

[Avinash Sonawane]

On Thu, Mar 23, 2017 at 9:43 PM, Tim Ruehsen  wrote:

> A beginner should read 'Getting Started' and 'Git Basics' to get an idea of
> how Git works at http://git-scm.com/doc.

Just to make it more clear, Tim meant first 2 chapters of Pro Git
https://git-scm.com/book/en/v2

> - when your work has been accepted (merged into upstream master), your are
> done and start the next issue on another branch.

And delete the old branch (the one which got merged) either through
Github or Git (git branch -d)

> we should set up a command sequence example for the above (tomorrow).

I'm in!

-- 
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

On Thu, Mar 23, 2017 at 9:43 PM, Tim Ruehsen  wrote:
> On Thursday, March 23, 2017 9:08:45 PM CET Avinash Sonawane wrote:
>> Hello!
>>
>> I am familiar with git. But currently I'm feeling pretty overwhelmed
>> with git operations. There are so many of them!
>>
>> Also we are coordinating wget2 development on Github, which is though
>> a front-end to git but has its own work-flow. You have fork, pull
>> request etc.
>>
>> I have read sufficient git documentation from https://git-scm.com/doc
>> and manpages so I'm very quite familiar with the git data model and
>> its 3 states etc. I have also gone through the considerable Github
>> documentation at https://help.github.com/ I even went through
>> https://try.github.io
>>
>> Now, I'm sure other students appearing for GSoC must have gone through
>> the docs like me. But the problem is to figure out the efficient
>> minimal work-flow while dealing with the git and github with respect
>> to wget2 development.
>>
>> For starters my current (flawed) work-flow is this :
>> 1) fork the upstream repo. Let's call it origin (using Github)
>> 2) clone the origin/master (using git)
>> 3) Pickup an issue (using Github)
>> 4) For a said issue create new branch (git checkout -b)
>> 5) `add` and `commit` changes to branch (using Git)
>> 6) Push changes to origin (git push origin foo)
>> 7) Send a pull request to merge branch foo in rockdaboot/wget (using Github)
>>
>> Periodically keep synching the fork which involves:
>> 1) add remote upstream <- just once
>> 2) fetch upstream
>> 3) checkout master
>> 4) merge upstream/master
>>
>> Now I am sure there's an efficient minimal operation workflow to deal
>> with Git and Github with respect to wget2 development. I am interested
>> in knowing what's your Git-Github workflow? Please no tutorial.
>> There's tons of tutorials/docs on the Internet. Just the workflow you
>> use.
>>
>> I am quite sure this will be very helpful to current and future GSoC
>> students who are most often new to distributed version control
>> systems, particularly Git and Github.
>>
>> If we get helpful responses, I will add this "contributing guide for
>> GSoC students" to our wiki.
>
> Very good thing !
>
> A beginner should read 'Getting Started' and 'Git Basics' to get an idea of
> how Git works at http://git-scm.com/doc.
>
> When it comes to branching (maybe later), here is a good article (also
> explains some git basics): 
> http://nvie.com/posts/a-successful-git-branching-model/
>
> Well, also whenever you have a specific question, search the internet. You 
> find
> a good solution for 99% of your questions.
>
> My general workflow when I want to work on a Github project that I have no
> write access (this is mostly the case):
> - fork the repo via Github interface
> - clone your repo to disk
> - for each issue you work on: create an new branch
> - switch to that branch and try to commit as often as possible (e.g. whenever
> you have one step completed)
> - when think you are finished: push and create a pull request (via Github
> interface)
> - ... now wait for review
> - if you have edit your code, do that test, commit and push again
> - ... now wait for next review (might repeat)
> - if you are asked to just change the commit message: 'git commit --amend' and
> 'git push -f'
> - when your work has been accepted (merged into upstream master), your are
> done and start the next issue on another branch.
>
> No more time now... we should set up a command sequence example for the above
> (tomorrow).
>
> More questions:
> - how to cleanup old branches
> - how to sync with 'upstream' (setting up remotes, merge, fast-forward merge,
> pull, push, ...)
>
> Regards, Tim



-- 
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

[Bug-wget] [GSoC] Proposal : Design and implementation of a statistics module

[Avinash Sonawane]

Hello!

As I have mentioned before I am applying for Google Summer of Code
under GNU organization. I'm contributing to wget project and planning
to continue the same under GSoC.

The title of my project is "Design and implementation of a statistics
module". The aim of my project is to collect useful statistics and
present it to user when demanded.

Here's my proposal[1]. If you have any suggestions or feedback then
please feel free to comment.

Thanks.

[1]https://docs.google.com/document/d/18vdGqP2P7vnqTLbbRxZCbjCHVvjT48x4VXl1nnSej1k/edit?usp=sharing

-- 
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

Re: [Bug-wget] [GSoC] Git & Github workflow

[Avinash Sonawane]

On Thu, Mar 23, 2017 at 9:08 PM, Avinash Sonawane  wrote:

> If we get helpful responses, I will add this "contributing guide for
> GSoC students" to our wiki.

As promised earlier, I just added a "contributing guide for GSoC
students" to our wiki.[0]

I hope this helps GSoC aspiring students to not hold back just because
they weren't comfortable with git/Github. :)

It's still far from being perfect. So I invite your contributions.

Finally, I would like to thank Tim for his inputs. Thank you!

[0]https://github.com/rockdaboot/wget2/wiki/Contributing-guide-for-GSoC-students

Regards,
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

Re: [Bug-wget] [GSoC] Git & Github workflow

[Avinash Sonawane]

On Sun, Apr 2, 2017 at 8:20 PM, Ander Juaristi  wrote:

> I've made a couple of revisions [0].

Wow! Didn't know that we were following Linux kernel coding style. I
remember asking about this one. Thank you!

Regards,
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

[Bug-wget] [GSoC 2017] Design and implementation of statistics module

[Avinash Sonawane]

Hello all!

I have been working on Wget2 for couple of weeks now and had submitted
the proposal to design and implement a statistics module for Wget2[0].

I'm very delighted to share this with you that the proposal has been
accepted and I have been selected for the GSoC 2017!

I would like to thank the Wget community for being supportive this
whole time. Thank you all!

I would like to explicitly thank Ander Juristi for helping me out when
I got stuck. I would also like to thank Darshit Shah for code reviews,
for making me conform my code to the coding style and for patiently
pointing out silly mistakes ;). I would like to thank Tim for guiding
me at each and every step of development starting from setting up an
editor (yes, literally!) to rebasing a branch, writing tests etc.
Thank you so much!

I would also like to take this opportunity to congratulate fellow
students Didik Setiawan and Akash Rawal for making it to GSoC 2017.
Congratulations guys!

Meanwhile, I have provided the link to proposal. Please feel free to
suggest/comment. All contributions are highly welcomed!

Starting from 17th, I'll be busy with university exams till 24th May.
So I'll join you guys after 24th.

Looking forward to an awesome summer to code away!

[0] 
https://docs.google.com/document/d/18vdGqP2P7vnqTLbbRxZCbjCHVvjT48x4VXl1nnSej1k/edit?usp=sharing

Regards,
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

Re: [Bug-wget] How to disable reading or settings in .wgetrc?

[Avinash Sonawane]

On Thu, Jun 1, 2017 at 2:47 AM, L A Walsh  wrote:

> But that is only referring to 1 switch.  How do you clear other
> .wgetrc settings (or have it ignored?).

$ man wget

"Affirmative options can be negated by prepending the --no- to the
option name; negative options can be negated by omitting the --no-
prefix.
...
For instance, using "follow_ftp = on" in .wgetrc makes Wget follow FTP
links by default, and using --no-follow-ftp is the only way to restore
the factory default from the command line."

The words *For instance* are very important here. They mean "For example" ;)

> How do I turn off convert-links from the
> command line (so it will override the .wgetrc value)?

So from manpage it follows that you need --no-convert-links on the command line.

BTW if you want to ignore .wgetrc altogether (which seems to be
indicated by the subject line) then you have:

$ man wget

"--config=FILE
   Specify the location of a startup file you wish to use."

So in that case you need to use --config (without any value) on the
command line.

Hope that helps!

-- 
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

[Bug-wget] Cloning Wget2 (Was Re: ANN: Wget2 development shifting to GitLab)

[Avinash Sonawane]

On Thu, Jun 1, 2017 at 3:03 AM, L Walsh  wrote:

> Is the permission denied normal?

Yes, if you haven't generated SSH keypair on your computer and haven't
added the public key to your GitLab account.

> Is that the right syntax?

Yes. That's a correct syntax.

> I don't see a download option for another format...

Currently, GitLab provides 3 main options to download the source code

1) git clone 
The one which you tried. This needs you public key to be added to you
GitLab account.

2) git clone 
This is the recommended method to clone the repo.
For wget2 the command will be:
$ git clone https://gitlab.com/gnuwget/wget2.git

3) compressed archive downloads
Apparently this is only available when you are logged in to GitLab.
Weird! Didn't know that!

Regards,
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

[Bug-wget] Cloning Wget2 (Was Re: ANN: Wget2 development shifting to GitLab)

[Avinash Sonawane]

On Thu, Jun 1, 2017 at 3:10 AM, L A Walsh  wrote:

> What's the point of a tar that needs online access?

While building, Wget2 fetches the latest current version of Gnulib
from Savannah. Bundling the Gnulib with Wget2 compressed archive might
get you a stale, outdated version of Gnulib. :)

> Looks like wget2 is "infected" with the gnulib virus.

No. Gnulib is a very cool library which keeps the code shared by many
GNU packages at one central location which helps to make the packages
more modular.

Additionally, Gnulib makes porting a package much easier and that's
how you can use Wget2 on Windows platform even if the primary dev
machines are Linux boxes. :)

> Hthis is not very easy to build...
>
> (sigh).

No. Building Wget2 is very easy. It just requires 4-5 commands. That's it!

Please take a look at "Building from git" section in the README.md[0].
And just remember to use https url to clone the repo if you don't want
to deal with SSH keys. I'm reiterating it here just for the
completeness sake:
$ git clone https://gitlab.com/gnuwget/wget2.git

[0] https://gitlab.com/gnuwget/wget2/blob/master/README.md
-- 
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

Re: [Bug-wget] How to disable reading or settings in .wgetrc?

[Avinash Sonawane]

On Fri, Jun 2, 2017 at 6:03 AM, Dale R. Worley  wrote:
> Avinash Sonawane  writes:
>> So in that case you need to use --config (without any value) on the
>> command line.

This is incorrect. Pardon me for misleading you.
Option '--config' compulsorily requires an argument.

> Actually, it means you could use "--config=/dev/null".

Yes, you are right! Currently, this is the only sane way to ignore
.wgetrc. Though --config= and --config="" works too but as Linda
pointed out, wget screams about not being able to read the file and
*then* ignores .wgetrc.

> The documentation does not suggest that --config is valid without a value.

Yes. Because that's not the case.

> So we really should add:
>
> --config
> Do not use any startup file.

I don't think we should. ;)

-- 
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

Re: [Bug-wget] How to disable reading or settings in .wgetrc?

[Avinash Sonawane]

On Fri, Jun 2, 2017 at 8:00 AM, L A Walsh  wrote:

> Tried that, but I used the 'no' prefix, as in --noconvert-links.

Then you didn't exactly try that!

> "no-" isn't really an english prefix.

Well, wget isn't really an English word either ;)

> What would happen if wget allowed specifying 'no' in addition to its current 
> behavior?

Think about a hypothetical future option "nocturnal". Now is that a
"no cturnal"? Or is it an option which can be negated as
"nonocturnal"?

Of course, any technical challenge can be handled by forming an
appropriate logic. But bringing in an additional complexity to the
code just to save one key press ("no-" vs "no") isn't really worth it
IMHO. :)

> BTW, --noconfig doesn't work, (--config is affirming you want a
> specific config file, no)?
[...]
> Why not --noconfig?

Because again, $ man wget says:
"Most options that do not accept arguments are boolean options, so
named because their state can be captured with a yes-or-no ("boolean")
variable.
...
--config=FILE
   Specify the location of a startup file you wish to use."

Clearly, --config is not a boolean option. So can't be "negated".

> I tried it with --config="" and it didn't like that
> (file not found).
> Seems like that should have cleared the config file as well,
> since "" is hardly a valid filename.

This seems like a good idea actually! And if we implement this for --config
then I'd love to see this implemented for other relevant options as
well!

> I would have thought /dev/zero to
> be about as likely.

No, you shouldn't use /dev/zero as the file is infinite in length and
wget will never finish reading it which will lead to all memory being
eaten up!

> I may have looked
> at the section on switches, and not all the other, "unrelated" text(if
> it was related to one of the switches, wouldn't it be
> mentioned w/that switch?)... (*sigh*)

That "unrelated" text is related to multiple switches so it has kept
under the heading "Option Syntax" right before where all the options
are explained. I'm sorry but I can't think of more logical place to
present that "unrelated" text ;)

-- 
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

Re: [Bug-wget] How to disable reading or settings in .wgetrc?

[Avinash Sonawane]

On Fri, Jun 2, 2017 at 1:17 PM, Tim Rühsen  wrote:

> 3. Tell wget not to read a config file at all with --no-config.

Thanks! Didn't know that!

But it seems this is specific to --config only and not for other
similar options.

For example,
I have `logfile = wget.log` in my .wgetrc along with other commands.
And now I want to override this specific .wgetrc configuration and
would like wget to show log on terminal (default behavior). How do I
do that?

1. There is no `--no-output-file` to achieve this.

2. --output-file="" makes wget exit without downloading the document
and spitting out an error about there being no such file or directory.

Of course, as you suggested I can create an another rc file for this
specific use case containing all of .wgetrc but this specific command.
But then there as other similar options as well and so this doesn't
look scalable.

May be we should accommodate 1. or 2. or may be both?

-- 
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

Re: [Bug-wget] How to disable reading or settings in .wgetrc?

[Avinash Sonawane]

On Fri, Jun 2, 2017 at 10:41 PM, Tim Rühsen  wrote:

> Instead of wasting time with Wget1.x we should use that time to make Wget2
> release ready (IMO).

Agreed! :)

-- 
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

[Bug-wget] [bug #51250] support for html5 video element

[Avinash Sonawane]

Follow-up Comment #1, bug #51250 (project wget):

Looking at the source I think you are right.  element is not supported
but it should be trivial to implement it.

___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/

Re: [Bug-wget] wget2 missing -X?

[Avinash Sonawane]

On Mon, Jun 26, 2017 at 3:25 PM, chris  wrote:
> Hi,

Hello Chris!

> wget2, seems to be missing the -X parameter as far as I can see,

Yes. We don't have -X.

> although it seems to be mentioned existing in the reference?

Where? Can you please point out the reference?

$ wget2 --help | grep X
$

Regards,
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

Re: [Bug-wget] wget2 fails to build if spaces in directory path

[Avinash Sonawane]

On Mon, Jun 26, 2017 at 3:24 PM, chris  wrote:
> Hi,

Hello!

>
> When wget2 was cloned to:
> /home/chris/Other Repositories/wget2
>
> It fails to build:
>
> """
> gcc: error: Repositories/wget2/data": No such file or directory
> gcc: error: Repositories/wget2/tests": No such file or directory
> """
>
> You can see how it's missed out Other from the dir path.
>
> Builds fine if in:
>
> /home/chris/wget2

Thank you for reporting! Can you please share the build log? If
possible use pastebin.

Regards,
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

Re: [Bug-wget] wget2 missing -X?

[Avinash Sonawane]

On Mon, Jun 26, 2017 at 3:45 PM, chris  wrote:

> It's mentioned in the online reference sorry:
>
> https://gnuwget.gitlab.io/wget2/reference/md_wget2_manual.html

Oh! Looks like it has been carried over from wget. Thanks for pointing
it out though! :)

I'm not really sure about the plans regarding -X in wget2 but we gotta
edit the docs for sure!

Regards,
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

Re: [Bug-wget] wget2 missing -X?

[Avinash Sonawane]

On Mon, Jun 26, 2017 at 3:59 PM, chris  wrote:
> The reason I was using it, because I was using the command line from:
> https://raywoodcockslatest.wordpress.com/2014/02/14/configuring-wget/
> which is useful for backing up wordpress sites.

That's good to know. The thing is wget2 is under heavy development
right now. We are adding new options, porting useful ones from
wget1.x, adding docs etc. In the midst of these rapid changes I'll
advise you to check `wget2 --help` to know which options wget2
supports as of now.

If you want something to be implemented in wget2 or would like to be
part of wget2 development you are very much welcomed at
https://gitlab.com/gnuwget/wget2 :)

Regards,
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

Re: [Bug-wget] wget2 fails to build if spaces in directory path

[Avinash Sonawane]

On Mon, Jun 26, 2017 at 3:49 PM, chris  wrote:
> Sure:
>
> https://gist.github.com/anfractuosity/0a9c12f81ebbb1033883bbec28b62dcd

This is happening because absolute paths have been used in Makefiles.
Please find the attached patch which fixes the issue. Sorry, haven't
sent the merge request to upstream as `make check` still fails. but
you can do 'make' and 'sudo make install' the wget2.

Apply the patch to your local git wget2 repo (git apply) if you are
keen on building the wget2 at a location with spaces in it's absolute
path.

Regards,
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com
From 2d41ed3b6a526e7f3e0552b3219fafcaae3368f2 Mon Sep 17 00:00:00 2001
From: Avinash Sonawane 
Date: Mon, 26 Jun 2017 22:16:28 +0530
Subject: [PATCH] use srcdir and top_srcdir instead of abs_srcdir and
 abs_top_srcdir

---
 fuzz/Makefile.am   | 2 +-
 tests/Makefile.am  | 2 +-
 unit-tests/Makefile.am | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/fuzz/Makefile.am b/fuzz/Makefile.am
index 59de23e..265bef6 100644
--- a/fuzz/Makefile.am
+++ b/fuzz/Makefile.am
@@ -1,6 +1,6 @@
 AM_CFLAGS = $(WERROR_CFLAGS) $(WARN_CFLAGS) -Wno-unused-parameter
 AM_CPPFLAGS = -I$(top_srcdir)/include/wget -I$(srcdir) \
- -DWGETVER_FILE=\"$(top_builddir)/include/wget/wgetver.h\" -DSRCDIR=\"$(abs_srcdir)\" -DTEST_RUN
+ -DWGETVER_FILE=\"$(top_builddir)/include/wget/wgetver.h\" -DSRCDIR=\"$(srcdir)\" -DTEST_RUN
 AM_LDFLAGS = -static
 LDADD = ../lib/libgnu.la ../libwget/libwget.la \
  $(LIBOBJS) $(GETADDRINFO_LIB) $(HOSTENT_LIB) $(INET_NTOP_LIB)\
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 36d6d87..e56b3e9 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -1,4 +1,4 @@
-DEFS = @DEFS@ -DDATADIR=\"$(abs_top_srcdir)/data\" -DSRCDIR=\"$(abs_srcdir)\" -DEXEEXT=\"$(EXEEXT)\"
+DEFS = @DEFS@ -DDATADIR=\"$(top_srcdir)/data\" -DSRCDIR=\"$(srcdir)\" -DEXEEXT=\"$(EXEEXT)\"
 
 AM_CFLAGS = $(WERROR_CFLAGS) $(WARN_CFLAGS)
 AM_CPPFLAGS = -Wno-missing-field-initializers -I$(top_srcdir)/include/wget -I$(srcdir) -I$(top_builddir)/lib -I$(top_srcdir)/lib \
diff --git a/unit-tests/Makefile.am b/unit-tests/Makefile.am
index e0c982e..da206b1 100644
--- a/unit-tests/Makefile.am
+++ b/unit-tests/Makefile.am
@@ -1,4 +1,4 @@
-DEFS = @DEFS@ -DDATADIR=\"$(abs_top_srcdir)/data\" -DSRCDIR=\"$(abs_srcdir)\" -DEXEEXT=\"$(EXEEXT)\"
+DEFS = @DEFS@ -DDATADIR=\"$(top_srcdir)/data\" -DSRCDIR=\"$(srcdir)\" -DEXEEXT=\"$(EXEEXT)\"
 
 AM_CFLAGS = $(WERROR_CFLAGS) $(WARN_CFLAGS)
 AM_CPPFLAGS = -Wno-missing-field-initializers -I$(top_srcdir)/include/wget -I$(srcdir) -I$(top_builddir)/lib -I$(top_srcdir)/lib \
-- 
2.11.0

Re: [Bug-wget] wget2 fails to build if spaces in directory path

[Avinash Sonawane]

On Mon, Jun 26, 2017 at 10:46 PM, Akash Rawal  wrote:

> We use absolute paths because relative paths break tests when build
> tree is separate from source tree.

Thank you! Didn't know about VPATH builds before and I just loved the idea!

> The reason the make check fails is that test program cd's to a
> subdirectory to run tests. After changing directory, the
> relative path is no longer correct. To further complicate the issue,
> srcdir and top_srcdir are not guaranteed to be relative paths.

Oh! Then, may be it's better to not build wget2 at a location with
space characters in it's path as make doesn't handle well the paths
with space characters in it. And that's not a wget2 issue. ;)

-- 
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

[Bug-wget] Compiling against libwget built with ubsan

[Avinash Sonawane]

Hello!

$ cat foo.c
#include 
#include 

int main(int argc, char **argv)
{
char foo[] = "FOO";
printf("%s\n", wget_strtolower(foo));

return 0;
}

$ gcc -lwget foo.c -o foo
//usr/local/lib/libwget.so: undefined reference to
`__ubsan_handle_divrem_overflow_abort'
//usr/local/lib/libwget.so: undefined reference to
`__ubsan_handle_add_overflow_abort'
//usr/local/lib/libwget.so: undefined reference to
`__ubsan_handle_negate_overflow_abort'
//usr/local/lib/libwget.so: undefined reference to
`__ubsan_handle_out_of_bounds_abort'
//usr/local/lib/libwget.so: undefined reference to
`__ubsan_handle_mul_overflow_abort'
//usr/local/lib/libwget.so: undefined reference to
`__ubsan_handle_vla_bound_not_positive_abort'
//usr/local/lib/libwget.so: undefined reference to
`__ubsan_handle_load_invalid_value_abort'
//usr/local/lib/libwget.so: undefined reference to
`__ubsan_handle_float_cast_overflow_abort'
//usr/local/lib/libwget.so: undefined reference to
`__ubsan_handle_type_mismatch_abort'
//usr/local/lib/libwget.so: undefined reference to
`__ubsan_handle_sub_overflow_abort'
//usr/local/lib/libwget.so: undefined reference to
`__ubsan_handle_nonnull_arg_abort'
//usr/local/lib/libwget.so: undefined reference to
`__ubsan_handle_shift_out_of_bounds_abort'
collect2: error: ld returned 1 exit status

If I compile libwget without --fsanitize-ubsan then the object files
get linked as expected. But how to use libwget when it's compiled with
ubsan?

PS - I know this is not a libwget question per-se. But a library
linker issue. Nevertheless, any pointers will be highly appreciated!

Regards,
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

Re: [Bug-wget] Compiling against libwget built with ubsan

[Avinash Sonawane]

On Tue, Aug 29, 2017 at 8:42 PM, Tim Rühsen  wrote:

> If you compile wget2/libwget with -fsanitize=undefined, then you have to
> use this flag later as well, so that the right libubsan gets linked.
>
> gcc -fsanitize=undefined -lwget foo.c -o foo

Thank you! That was it!

Regards,
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

Re: [Bug-wget] Fwd: patch proposition

[Avinash Sonawane]

On Thu, Sep 14, 2017 at 1:28 PM, kalle  wrote:
> hello,

Hi Kalle!

> i hereby repeat my request (in standard english for Darshit):

Please don't mind but there are people from all around the globe on
bug-wget so it'll be great if we could extend the courtesy of not
wasting others time and helping each other understand better by
communicating in common language (de facto English) instead of
regional ones!

Think this way, what if I had written this email in Hindi? Would it
have been more helpful? Or a plain nuisance? :)

Regards,
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

Re: [Bug-wget] http server responding with 416 but file was not transferred completely

[Avinash Sonawane]

On Thu, Sep 14, 2017 at 8:36 PM, Tim Rühsen  wrote:

>> So I humbly propose the attached patch.
>> I tried to create a pull request, but got a 403.
>
> Thanks for the patch - I'll test it in the next days.
> BTW, we currently work on Wget2 where we have a related issue, if you
> like to take a look at it: https://gitlab.com/gnuwget/wget2/issues/278

It's https://gitlab.com/gnuwget/wget2/issues/268 :)

-- 
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

Re: [Bug-wget] lex compile problem on AIX 7.1

[Avinash Sonawane]

On Mon, Oct 2, 2017 at 8:56 PM, Tim Rühsen  wrote:
> On 10/02/2017 10:00 AM, l...@langs.se wrote:
>> Hi!
>>
>> I get the following error when compiling wget 1.19.1 on AIX 7.1:
>>
>> make all-am
>> CC connect.o
>> CC convert.o
>> CC cookies.o
>> CC ftp.o
>> lex -ocss.c
>> 0: Warning: 1285-300 The o flag is not valid.
>> 0: Warning: 1285-300 The s flag is not valid.
>> 0: Warning: 1285-300 The s flag is not valid.
>> 0: Warning: 1285-300 The . flag is not valid.
>>
>> Seems the LEX arguments are not valid?
>>
>> Any suggestions?
>
> Hi,
>
> some 'lex' versions *must have* a space after -o, some others *must not
> have* a space there. We decided not to use a space since this covers
> most build environments.

I am concerned about the `-o` option itself. As per the POSIX.1-2008
[0], lex only supports -t, -n and -v. Though agreed flex is the de
facto implementation of lex which supports `-o`, I think we should
stick to the POSIX standard and shouldn't use `-o`.

I see 2 possible approaches:
1. use lex.yy.c, default file generated or
2. use `-t >` to have the desired custom filename.

[0] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/lex.html

Regards,
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

Re: [Bug-wget] [bug #55229] memory exhausted even though stating: Omitting download

[Avinash Sonawane]

On Sun, 16 Dec 2018 16:44:58 -0500 (EST)
Fenevad Kan  wrote:

> Follow-up Comment #7, bug #55229 (project wget):

> checking for GNUTLS... no
> configure: error: Package requirements (gnutls) were not met:
> 
> No package 'gnutls' found

I believe you need libgnutls-dev.
Also, please consider going through
https://git.savannah.gnu.org/cgit/wget.git/tree/README.checkout

Regards,
Avinash Sonawane (rootKea)
PICT, Pune
https://rootkea.wordpress.com

[Bug-wget] [bug #55229] memory exhausted even though stating: Omitting download

[Avinash Sonawane]

Follow-up Comment #8, bug #55229 (project wget):

Copy-pasting the reply posted on ML:

> checking for GNUTLS... no
> configure: error: Package requirements (gnutls) were not met:
> 
> No package 'gnutls' found  

I believe you need libgnutls-dev.
Also, please consider going through
https://git.savannah.gnu.org/cgit/wget.git/tree/README.checkout

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/