Re: Broken links continue to exist on major pages on dlang.org

[cym13 via Digitalmars-d]

On Thursday, 2 June 2016 at 20:09:57 UTC, Andrei Alexandrescu 
wrote:

On 06/02/2016 03:27 PM, Adam D. Ruppe wrote:
Fixing links is, in theory, one of the easiest tasks people 
can do, but
in practice the plethora of bizarre macros makes it far harder 
than it

should be.


We'd be greatly helped by some automation here (i.e. pinpoint 
the wrong links). -- Andrei


I did a (quite long actually) “wget --spider -o ~/wget.log -e 
robots=off -w 1 -r -p http://dlang.org/” to get a list of unused 
links. There are 690 of them, you can find the list there [1]. If 
you need the full log of the scan ask for it by mail, it's about 
35 lines long so I don't feel like pasting it.


[1]: http://paste.devys.org/mydalona/raw

Re: Broken links continue to exist on major pages on dlang.org

[Sönke Ludwig via Digitalmars-d]

Am 03.06.2016 um 14:16 schrieb Andrei Alexandrescu:

On 6/3/16 7:59 AM, Sönke Ludwig wrote:

Wait a minute... MYREF = $(D $1) in std-ddox-override.ddoc should
actually do the job just fine. Those links should have actually worked
before until I removed full-text automatic cross-references from DDOX.


This seems good, please try it out and post a PR if it works. -- Andrei


I'm already on it: https://github.com/dlang/dlang.org/pull/1324

Re: Broken links continue to exist on major pages on dlang.org

[ag0aep6g via Digitalmars-d]

On 06/03/2016 02:15 PM, Vladimir Panteleev wrote:

Only in code blocks (i.e. $(D ...) or `...`).


Makes sense :)

Re: Broken links continue to exist on major pages on dlang.org

[Andrei Alexandrescu via Digitalmars-d]

On 6/3/16 7:59 AM, Sönke Ludwig wrote:

Wait a minute... MYREF = $(D $1) in std-ddox-override.ddoc should
actually do the job just fine. Those links should have actually worked
before until I removed full-text automatic cross-references from DDOX.


This seems good, please try it out and post a PR if it works. -- Andrei

Re: Broken links continue to exist on major pages on dlang.org

[Vladimir Panteleev via Digitalmars-d]

On Friday, 3 June 2016 at 12:08:33 UTC, ag0aep6g wrote:

On 06/03/2016 01:59 PM, Sönke Ludwig wrote:
Wait a minute... MYREF = $(D $1) in std-ddox-override.ddoc 
should
actually do the job just fine. Those links should have 
actually worked
before until I removed full-text automatic cross-references 
from DDOX.


So DDOX is supposed to link any symbol just by its name, 
without being fully qualified?


Only in code blocks (i.e. $(D ...) or `...`).

Re: Broken links continue to exist on major pages on dlang.org

[ag0aep6g via Digitalmars-d]

On 06/03/2016 01:59 PM, Sönke Ludwig wrote:

Wait a minute... MYREF = $(D $1) in std-ddox-override.ddoc should
actually do the job just fine. Those links should have actually worked
before until I removed full-text automatic cross-references from DDOX.


So DDOX is supposed to link any symbol just by its name, without being 
fully qualified?


I'm not sure if that's good. We've had a couple problems with Ddoc being 
annoyingly eager in highlighting parameter names and such. This goes 
even further, doesn't it?

Re: Broken links continue to exist on major pages on dlang.org

[Sönke Ludwig via Digitalmars-d]

Am 03.06.2016 um 13:47 schrieb Sönke Ludwig:

Am 03.06.2016 um 11:49 schrieb ag0aep6g:

On 06/02/2016 09:09 PM, Sönke Ludwig wrote:

Which are the remaining issues that you know of?


Next up: MYREF.

Example page with missing links:
http://dlang.org/library/std/string.html

In std.ddoc, MYREF is defined as:

  MYREF = $(TT $1)$(NBSP)

In std-ddox-override.ddoc it's:

  MYREF = $1

That's no good, of course. But I also don't see a way to define it
properly. It would have to be something like this:

 MYREF = $(TT $1)$(NBSP)

where $(MODULE) is the current module name without packages.

But:

1) Something like $(MODULE) doesn't seem to exist. (Loosely related:
DDOX doesn't emit $(TITLE) properly.)

2) camelCase strikes again. $1 would have to be transformed from fooBar
to foo_bar to match DDOX's naming scheme. No can do with macros alone. I
think those underscore names may have been a mistakee.


Since only 12 modules are affected, probably the easiest way is to use
the same approach as std.algorithm and define MYREF in each module
separately as:

 MYREF = $(REF_ALTTEXT $(TT $1), $1, std, xxx)$(NBSP)

And then remove the definition from std.ddoc


Wait a minute... MYREF = $(D $1) in std-ddox-override.ddoc should 
actually do the job just fine. Those links should have actually worked 
before until I removed full-text automatic cross-references from DDOX.

Re: Broken links continue to exist on major pages on dlang.org

[Sönke Ludwig via Digitalmars-d]

Am 03.06.2016 um 11:49 schrieb ag0aep6g:

On 06/02/2016 09:09 PM, Sönke Ludwig wrote:

Which are the remaining issues that you know of?


Next up: MYREF.

Example page with missing links:
http://dlang.org/library/std/string.html

In std.ddoc, MYREF is defined as:

  MYREF = $(TT $1)$(NBSP)

In std-ddox-override.ddoc it's:

  MYREF = $1

That's no good, of course. But I also don't see a way to define it
properly. It would have to be something like this:

 MYREF = $(TT $1)$(NBSP)

where $(MODULE) is the current module name without packages.

But:

1) Something like $(MODULE) doesn't seem to exist. (Loosely related:
DDOX doesn't emit $(TITLE) properly.)

2) camelCase strikes again. $1 would have to be transformed from fooBar
to foo_bar to match DDOX's naming scheme. No can do with macros alone. I
think those underscore names may have been a mistakee.


Since only 12 modules are affected, probably the easiest way is to use 
the same approach as std.algorithm and define MYREF in each module 
separately as:


MYREF = $(REF_ALTTEXT $(TT $1), $1, std, xxx)$(NBSP)

And then remove the definition from std.ddoc

Re: Broken links continue to exist on major pages on dlang.org

[ag0aep6g via Digitalmars-d]

On 06/02/2016 09:09 PM, Sönke Ludwig wrote:

Which are the remaining issues that you know of?


Next up: MYREF.

Example page with missing links:
http://dlang.org/library/std/string.html

In std.ddoc, MYREF is defined as:

 MYREF = $(TT $1)$(NBSP)

In std-ddox-override.ddoc it's:

 MYREF = $1

That's no good, of course. But I also don't see a way to define it 
properly. It would have to be something like this:


MYREF = $(TT $1)$(NBSP)

where $(MODULE) is the current module name without packages.

But:

1) Something like $(MODULE) doesn't seem to exist. (Loosely related: 
DDOX doesn't emit $(TITLE) properly.)


2) camelCase strikes again. $1 would have to be transformed from fooBar 
to foo_bar to match DDOX's naming scheme. No can do with macros alone. I 
think those underscore names may have been a mistakee.

Re: Broken links continue to exist on major pages on dlang.org

[Adam D. Ruppe via Digitalmars-d]

On Thursday, 2 June 2016 at 20:34:24 UTC, Andrei Alexandrescu 
wrote:
Interestingly it came as encouraging and empowering some 
fledgling work that had compelling things going for it 
(including but not limited to enthusiastic receipt in this 
forum), which ironically is exactly what you just asked for.


Yes, indeed, it was a good first (and second) step. But further 
steps are necessary too in order to finish a project.


Here's what would have been ideal to me:

1) Someone writes a cool thing.

2) We encourage further exploration and see interest.

3) After deciding there's serious potential, we decide on the end 
goal, a timeframe, and set the conditions of success. For 
example: ddox becomes the official documentation generator at the 
end of the year if there are no major bugs remaining open.


4) We put it on the website and work toward the goal, with all 
the teams - Phobos, dlang.org, RejectedSoftware, etc., 
understanding their role.


5) When the goal deadline arrives, if it passes the major bug 
test, it goes live and we are committed to it going forward.




Why this order? First, someone writing the cool thing means we 
actually have something to sink our teeth into and a de facto 
champion in the original author.


Second, we need to incubate this work and not discourage the 
author.


ddox got a decent go up to here.

But then we need to decide what's next - a clear goal, including 
a due date, gets us all aligned and removes a lot of the 
uncertainty on the author's side; it is some reassurance that 
they aren't wasting their time, and encourages outside teams to 
get onboard.


That leads directly into step four, and then step five actually 
proves that the others were not in vain.

Re: Broken links continue to exist on major pages on dlang.org

[Andrei Alexandrescu via Digitalmars-d]

On 06/02/2016 04:27 PM, Adam D. Ruppe wrote:

It got added to the official website without a real support plan: a
clear commitment was never made to fix Phobos issues, and nobody knew if
we were actually going to transition from ddoc, or live side-by-side, or
what. Certainly, nobody knew when, so there was no sense of urgency to
fix the Phobos source regardless.

It just kinda sat in a dark corner of the website for years with little
attention from the Phobos crew. I doubt many of them use it.


Interestingly it came as encouraging and empowering some fledgling work 
that had compelling things going for it (including but not limited to 
enthusiastic receipt in this forum), which ironically is exactly what 
you just asked for.


Yes, I do wish /library/ were better and got more TLC. But I'm not sorry 
for approving it.



Andrei

Re: Broken links continue to exist on major pages on dlang.org

[Adam D. Ruppe via Digitalmars-d]

On Thursday, 2 June 2016 at 19:53:29 UTC, Sönke Ludwig wrote:

So you actually display the documentation for your own
Phobos fork, right?


Yeah.

This hasn't been an option for the DDOX based documentation for 
obvious reasons, though, so no reason to laugh at it..


Well, I think ddox is a competent implementation (of an iffy 
design, I don't love ddoc itself), and the fact that it is ahead 
on search engine rankings shows the page breakdown is more 
SEO-friendly than stock ddoc, among other advantages.


But what I find laughable is the surrounding ecosystem. It got 
added to the official website without a real support plan: a 
clear commitment was never made to fix Phobos issues, and nobody 
knew if we were actually going to transition from ddoc, or live 
side-by-side, or what. Certainly, nobody knew when, so there was 
no sense of urgency to fix the Phobos source regardless.


It just kinda sat in a dark corner of the website for years with 
little attention from the Phobos crew. I doubt many of them use 
it.



I feel the same thing is happening with dub by the way: we're 
told that it is the D package manager and there's 
code.dlang.org... but where's the follow-up action to back it up? 
It isn't included with the dmd zip and has very little marketing 
on the website. There's still a drive to get stuff into Phobos 
proper, or at least std.experimental, instead of dogfooding the 
package manager.


Sure, there's a link to it on the website, but there seems to be 
little personal use of it - and thus little drive to improve it - 
from among the Phobos core themselves.

Re: Broken links continue to exist on major pages on dlang.org

[ag0aep6g via Digitalmars-d]

On 06/02/2016 09:57 PM, Sönke Ludwig wrote:

Okay, that should be solved in a minute!


Nice :D

Re: Broken links continue to exist on major pages on dlang.org

[Andrei Alexandrescu via Digitalmars-d]

On 06/02/2016 03:27 PM, Adam D. Ruppe wrote:

Fixing links is, in theory, one of the easiest tasks people can do, but
in practice the plethora of bizarre macros makes it far harder than it
should be.


We'd be greatly helped by some automation here (i.e. pinpoint the wrong 
links). -- Andrei

Re: Broken links continue to exist on major pages on dlang.org

[Sönke Ludwig via Digitalmars-d]

Am 02.06.2016 um 21:29 schrieb ag0aep6g:

On 06/02/2016 09:09 PM, Sönke Ludwig wrote:

Which are the remaining issues that you know of? I've searched bugzilla
earlier today and couldn't find anything apart from broken "Improve this
page" links for package.d modules.


The big one I know of is (M)REF_ALTTEXT. We need something better than
the ugly parentheses thing we've got.

Other than that I don't have any blockers on my mind. We've made good
progress. I haven't really looked for issues, though.


Okay, that should be solved in a minute!



Maybe there is no point in hiding the pages from Google now. But they
should have been hidden before. Then again, we would have had less
pressure to fix things then.


Agreed, unlisting from search engines + making the link to the docs more 
prominent to keep the pressure would probably have been the best choice.

Re: Broken links continue to exist on major pages on dlang.org

[Sönke Ludwig via Digitalmars-d]

Am 02.06.2016 um 21:27 schrieb Adam D. Ruppe:

On Thursday, 2 June 2016 at 19:06:42 UTC, Sönke Ludwig wrote:

But I'd hate to implement some kind of special case here instead of
simply fixing the Phobos sources to use macros that carry proper
semantical meaning.


That's actually exactly what I did:

https://github.com/dlang/dlang.org/pull/1177

The foundation even got merged into upstream dlang.org, six months ago!

Sadly, the follow-up, just about two weeks after I laid the foundation*
got stalled out  until
recently, now a PR based on that one
 got merged ten days ago.

* Cross-linking in ddoc has been remarkably non-trivial until recently,
and even so, MREF, REF, and REF_ALTTEXT, while much better than it was
before, still aren't ideal.

Fixing links is, in theory, one of the easiest tasks people can do, but
in practice the plethora of bizarre macros makes it far harder than it
should be. Even the REF macro family, which aims to unify them, has hit
hurdles:

https://github.com/dlang/dlang.org/pull/1257


I know and I think I misunderstood your last message. So you actually 
display the documentation for your own Phobos fork, right? That would 
make a lot more sense then. This hasn't been an option for the DDOX 
based documentation for obvious reasons, though, so no reason to laugh 
at it..

Re: Broken links continue to exist on major pages on dlang.org

[ag0aep6g via Digitalmars-d]

On 06/02/2016 09:09 PM, Sönke Ludwig wrote:

Which are the remaining issues that you know of? I've searched bugzilla
earlier today and couldn't find anything apart from broken "Improve this
page" links for package.d modules.


The big one I know of is (M)REF_ALTTEXT. We need something better than 
the ugly parentheses thing we've got.


Other than that I don't have any blockers on my mind. We've made good 
progress. I haven't really looked for issues, though.


Maybe there is no point in hiding the pages from Google now. But they 
should have been hidden before. Then again, we would have had less 
pressure to fix things then.

Re: Broken links continue to exist on major pages on dlang.org

[Adam D. Ruppe via Digitalmars-d]

On Thursday, 2 June 2016 at 19:06:42 UTC, Sönke Ludwig wrote:
But I'd hate to implement some kind of special case here 
instead of simply fixing the Phobos sources to use macros that 
carry proper semantical meaning.


That's actually exactly what I did:

https://github.com/dlang/dlang.org/pull/1177

The foundation even got merged into upstream dlang.org, six 
months ago!


Sadly, the follow-up, just about two weeks after I laid the 
foundation* got stalled out 
 until recently, now a 
PR based on that one  
got merged ten days ago.


* Cross-linking in ddoc has been remarkably non-trivial until 
recently, and even so, MREF, REF, and REF_ALTTEXT, while much 
better than it was before, still aren't ideal.


Fixing links is, in theory, one of the easiest tasks people can 
do, but in practice the plethora of bizarre macros makes it far 
harder than it should be. Even the REF macro family, which aims 
to unify them, has hit hurdles:


https://github.com/dlang/dlang.org/pull/1257

Re: Broken links continue to exist on major pages on dlang.org

[Andrei Alexandrescu via Digitalmars-d]

On 06/02/2016 02:44 PM, ag0aep6g wrote:

I guess no one really feels responsible for this stuff.


Vladimir is responsible (I cc'd him with the post and he was quick to 
reply). He can of course delegate but he's the entry point. All - if you 
find any issues please report to him. As has become apparent in this 
thread, Sönke is a great resource as well. Let's keep the band rolling. 
Thanks! -- Andrei

Re: Broken links continue to exist on major pages on dlang.org

[Sönke Ludwig via Digitalmars-d]

Am 02.06.2016 um 20:44 schrieb ag0aep6g:

/library/ pages are DDOX pages. That is the "NEW Library Reference
Preview". It's not ready for prime time. It should be hidden from Google
until it's ready. Here's a quick PR to do that:
https://github.com/dlang/dlang.org/pull/1320

I guess no one really feels responsible for this stuff.


Which are the remaining issues that you know of? I've searched bugzilla 
earlier today and couldn't find anything apart from broken "Improve this 
page" links for package.d modules.

Re: Broken links continue to exist on major pages on dlang.org

[Sönke Ludwig via Digitalmars-d]

Am 02.06.2016 um 20:44 schrieb Adam D. Ruppe:

On Thursday, 2 June 2016 at 18:24:30 UTC, Andrei Alexandrescu wrote:

Within that page, the top table has links to various constituents such
as std.algorithm.searching, std.algorithm.comparison etc. None of
those links work.


I fixed it months ago on my fork
http://dpldocs.info/std.algorithm

(I also removed the "this is a submodule of std.algorithm" since that's
a useless sentence anyway, given my autogenerated breadcrumb navigation
that makes the submodule relationship clear outside the text)

The continued existence of such bugs is a reason I laugh at the idea
that ddox is a serious contender btw.


I know you are a hacker by heart and of course you can patch and work 
around everything. But I'd hate to implement some kind of special case 
here instead of simply fixing the Phobos sources to use macros that 
carry proper semantical meaning. Those changes now simply have to get 
into the latest stable version to appear on the doc pages.


Apart from that, this is really the only major known issue left (links 
to GitHub for package.d modules are still broken and that's about it).

Re: Broken links continue to exist on major pages on dlang.org

[Steven Schveighoffer via Digitalmars-d]

On 6/2/16 2:44 PM, Adam D. Ruppe wrote:

(I also removed the "this is a submodule of std.algorithm" since that's
a useless sentence anyway, given my autogenerated breadcrumb navigation
that makes the submodule relationship clear outside the text)


http://dpldocs.info/experimental-docs/std.algorithm.html

The breadcrumb for std is missing on that page.

-Steve

Re: Broken links continue to exist on major pages on dlang.org

[Adam D. Ruppe via Digitalmars-d]

On Thursday, 2 June 2016 at 18:24:30 UTC, Andrei Alexandrescu 
wrote:
Within that page, the top table has links to various 
constituents such as std.algorithm.searching, 
std.algorithm.comparison etc. None of those links work.


I fixed it months ago on my fork
http://dpldocs.info/std.algorithm

(I also removed the "this is a submodule of std.algorithm" since 
that's a useless sentence anyway, given my autogenerated 
breadcrumb navigation that makes the submodule relationship clear 
outside the text)



The continued existence of such bugs is a reason I laugh at the 
idea that ddox is a serious contender btw.

Re: Broken links continue to exist on major pages on dlang.org

[ag0aep6g via Digitalmars-d]

On Thursday, 2 June 2016 at 18:24:30 UTC, Andrei Alexandrescu 
wrote:

The first D-related hit on searching for

std.algorithm

on Google is https://dlang.org/library/std/algorithm.html. That 
is the 255th most accessed file on dlang.org (including the 
.ico, css, .js etc many of which are accessed from most other 
pages).


/library/ pages are DDOX pages. That is the "NEW Library 
Reference Preview". It's not ready for prime time. It should be 
hidden from Google until it's ready. Here's a quick PR to do 
that: https://github.com/dlang/dlang.org/pull/1320


I guess no one really feels responsible for this stuff.

I recall I reported this a while ago. It either has never been 
fixed, or it became a problem again. Folks, I can't babysit all 
this stuff. I can't babysit all this stuff.


Meh.

Re: Broken links continue to exist on major pages on dlang.org

[Andrei Alexandrescu via Digitalmars-d]

On 06/02/2016 02:36 PM, Seb wrote:

Imho we should only host on documentation on dlang.org and mark the
other as experimental by a separate subdomain.


We're fine as we are, many documentations offer distinct formats. -- Andrei

Re: Broken links continue to exist on major pages on dlang.org

[Seb via Digitalmars-d]

On Thursday, 2 June 2016 at 18:24:30 UTC, Andrei Alexandrescu 
wrote:

The first D-related hit on searching for

std.algorithm

on Google is https://dlang.org/library/std/algorithm.html. That 
is the 255th most accessed file on dlang.org (including the 
.ico, css, .js etc many of which are accessed from most other 
pages).


Within that page, the top table has links to various 
constituents such as std.algorithm.searching, 
std.algorithm.comparison etc. None of those links work.


I recall I reported this a while ago. It either has never been 
fixed, or it became a problem again. Folks, I can't babysit all 
this stuff. I can't babysit all this stuff.


This is important and urgent. Who can get on this and stay on 
this until it's done?


For the future, who can work on a simple tool (or hook us into 
one of the many existing ones) that detects bad links?



Thanks,

Andrei


There seems to some confusion between "phobos" (ddoc) and 
"library" (ddox).

Both are publicly available.

Afaict the plan is to move over to ddox at some point, because 
ddoc is way too limited, but (if i am not mistaken) this is 
blocked by existing bugs in ddoc.


Imho we should only host on documentation on dlang.org and mark 
the other as experimental by a separate subdomain.


Within that page, the top table has links to various 
constituents such as std.algorithm.searching, 
std.algorithm.comparison etc. None of those links work.


The PR already has been merged, but you have to wait for the next 
release:


https://github.com/dlang/phobos/pull/4303

Broken links continue to exist on major pages on dlang.org

[Andrei Alexandrescu via Digitalmars-d]

The first D-related hit on searching for

std.algorithm

on Google is https://dlang.org/library/std/algorithm.html. That is the 
255th most accessed file on dlang.org (including the .ico, css, .js etc 
many of which are accessed from most other pages).


Within that page, the top table has links to various constituents such 
as std.algorithm.searching, std.algorithm.comparison etc. None of those 
links work.


I recall I reported this a while ago. It either has never been fixed, or 
it became a problem again. Folks, I can't babysit all this stuff. I 
can't babysit all this stuff.


This is important and urgent. Who can get on this and stay on this until 
it's done?


For the future, who can work on a simple tool (or hook us into one of 
the many existing ones) that detects bad links?



Thanks,

Andrei