Re: [Wikitech-l] Search Flow using ES/CirrusSearch in MW 1.25?

2015-12-06 Thread Matthias Mullie 
I have no idea.
Flow’s search API should be able to provide all the data that is needed, but I 
haven’t
looked into how it could be integrated into normal searches yet (if at all)

Matthias


> On 03 Dec 2015, at 18:09, James Montalvo  wrote:
> 
> Any estimate on the amount of work required to integrate Flow searches into
> normal content searches?
> 
> On Thu, Dec 3, 2015 at 10:45 AM, Matthias Mullie 
> wrote:
> 
>>> So it sounds like there's no way to search Flow content through
>>> Special:Search (or any other interface a normal user would have). Is that
>>> right?
>> Correct, there is currently no UI.
>> 
>>> 
>>> It looks like ApiFlowSearch.php was added in REL1_26 [1] (I don't see it
>> in
>>> REL1_25 [2]). I'll tinker around with a 1.26 setup and see if I can
>> figure
>>> out how to use ApiFlowSearch.php.
>> Yeah, all of that search-related code happened around the same time.
>> Come to think of it, there were related changes in Elastica & CirrusSearch
>> extensions,
>> but I think most (or all) of those are in 1.25 already.
>> 
>> Matthias
>> 
>>> On 03 Dec 2015, at 16:41, Daren Welsh  wrote:
>>> 
>>> So it sounds like there's no way to search Flow content through
>>> Special:Search (or any other interface a normal user would have). Is that
>>> right?
>>> 
>>> It looks like ApiFlowSearch.php was added in REL1_26 [1] (I don't see it
>> in
>>> REL1_25 [2]). I'll tinker around with a 1.26 setup and see if I can
>> figure
>>> out how to use ApiFlowSearch.php.
>>> 
>>> Daren
>>> 
>>> [1]
>>> 
>> http://git.wikimedia.org/blob/mediawiki%2Fextensions%2FFlow.git/REL1_26/includes%2FApi%2FApiFlowSearch.php
>>> [2]
>>> 
>> http://git.wikimedia.org/tree/mediawiki%2Fextensions%2FFlow.git/REL1_25/includes%2FApi
>>> 
>>> On Thu, Dec 3, 2015 at 4:53 AM, Matthias Mullie 
>>> wrote:
>>> 
 Hi Daren,
 
 I don’t think the Flow search index scripts rely on any specific 1.26
>> code,
 Running it on 1.25 will be fine, I guess (though I haven’t tried it)
 
 I’m not sure those scripts do what you’d expect them to do, though.
 Running the Flow index scripts won’t add Flow’s data into the existing
 search
 index and won’t make Flow results appear in Special:Search.
 
 The first goal was having a Flow-specific in-board search (to filter
 results
 in a busy board)
 The existing search code was mostly written with that in mind: the data
>> is
 indexed in a separate index and only available via ApiFlowSearch.
 
 That said, ApiFlowSearch will let you search for discussions across
 multiple/all pages, namespaces, …
 If you don’t mind building that integration/frontend yourself, that is,
>> it
 is
 not currently being worked on.
 
 There is no other documentation available, but let me know if there’s
 anything
 in particular you’d like to know more about.
 
 Matthias
 
 
> On 03 Dec 2015, at 00:20, Daren Welsh  wrote:
> 
> I'm referencing [1] in an attempt to get Cirrus Search (Elasticsearch)
>> to
> search and return results from Flow boards in MW 1.25. Some of the
 scripts
> like FlowSearchConfig.php don't seem to be in REL1_25, but they do seem
 to
> be in REL1_26. Does this mean I won't be able to get this to work in
>> 1.25
> or is there another (older) set of instructions on how to configure so
>> ES
> will search Flow content?
> 
> Daren
> 
> [1] https://www.mediawiki.org/wiki/Flow/Architecture/Search
> 
> --
> __
> http://enterprisemediawiki.org
> http://mixcloud.com/darenwelsh
> http://www.beatportfolio.com
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
 
 
 ___
 Wikitech-l mailing list
 Wikitech-l@lists.wikimedia.org
 https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>> 
>>> 
>>> 
>>> 
>>> --
>>> __
>>> http://enterprisemediawiki.org
>>> http://mixcloud.com/darenwelsh
>>> http://www.beatportfolio.com
>>> ___
>>> Wikitech-l mailing list
>>> Wikitech-l@lists.wikimedia.org
>>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>> 
>> 
>> ___
>> Wikitech-l mailing list
>> Wikitech-l@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>> 
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l


___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] API Sandbox (was: 2015-12-02 Scrum of Scrums meeting notes)

2015-12-06 Thread Bartosz Dziewoński 

On 2015-12-04 18:59, Adam Baso wrote:

I do wonder though if we've spent much time studying the ease of getting at
least some part of oojs-ui split out or making it so that new stuff going
forward is part of the oojs-ui family but it's not as monolithich?


Not any more than what is written at 
, as far as I know.


OOjs UI doesn't have a dedicated team (or even a dedicated person) 
supporting it. The changes are mostly driven by whatever the 
VisualEditor or Design teams need, and what they are willing to 
implement themselves (and to a lesser extent, Multimedia team and random 
people like Brad or Florian). This makes it difficult to make any big 
changes happen.


--
Bartosz Dziewoński

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Ifexists across wikis

2015-12-06 Thread Tim Starling 
On 07/12/15 06:29, Bartosz Dziewoński wrote:
> To add to what Alex and Florian said, the simple database lookup to
> check page existence is not actually that simple. When parsing a page,
> the query to determine link color (and to mark links to non-existent,
> redirect or disambig pages) is done in batches of 1000 links, after
> the whole page has been parsed and we know all the pages it links to.
> Special pages that have lists of links use a similar method.

Also, when you make a red link, and then someone creates the page,
people expect the link to turn blue straight away. That's implemented
using the pagelinks table -- when a page is created, we use pagelinks
to find all pages with red links to that page, update all their
page_touched fields, and purge them from Varnish, so that all the
links will turn blue in under a second.

It's possible to do that for interwiki links, but it increases the
amount of time it would take to implement such a feature. We currently
don't have a way to efficiently find all interwiki links to a page, so
one would have to be added.

-- Tim Starling



___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Ifexists across wikis

2015-12-06 Thread John Erling Blad 
Use Q-ids and get the links from Wikidata.

On Sun, Dec 6, 2015 at 10:49 PM, Purodha Blissenbach <
puro...@blissenbach.org> wrote:

> How about using the API on the targe side?
> Purodha
>
>
> On 06.12.2015 18:04, Alex Monk wrote:
>
>> I don't think there is a way to get a database name from an interwiki
>> prefix.
>>
>> Also, whether a page is known or not does not just depend on a simple
>> database lookup. Extensions can add arbitrary rules about which titles
>> should be considered known or not. EducationProgram, GlobalUserPage, and
>> WikimediaIncubator all do this.
>>
>> On 6 December 2015 at 16:26, Lars Aronsson  wrote:
>>
>> If I write a [[link]] it will be blue if the page exists and red
>>> otherwise.
>>> But if I write [[:sw:link]] that will be an external or cross-wiki link,
>>> that is never red, as if it were impossible to know whether that page
>>> existed in Swahili Wikipedia.
>>>
>>> But determining the existence of a page is just a quick database table
>>> lookup, and all databases run on WMF's servers, so it shouldn't be more
>>> expensive to look up a cross-wiki link, as long as it is one of WMF's
>>> wikis.
>>>
>>> In Wiktionary, it is common to link to entries in foreign languages both
>>> on the local wiki and to the native wiki for that language. For example,
>>> in English Wikitionary the entry for "blue" links to the Swahili word
>>> "bluu"
>>> both on en.wiktionary and on sw.wiktionary, using the template
>>> {{t+|sw|bluu}}.
>>>
>>> https://en.wiktionary.org/wiki/blue#Translations
>>>
>>> But since the Afrikaans translation "blou" doesn't have an entry on the
>>> Afrikaans Wiktionary, another template is used: {{t|af|blou}}. And it is
>>> a pain to know which one of these two templates to use. If it was
>>> possible
>>> in {{#ifexists}} to determine the existence of a page in another wiki,
>>> only one template would be needed, and the bot job to change to the right
>>> template would not be needed.
>>>
>>> #ifexist already works across namespaces (well, of course), so is there
>>> any
>>> good reason it shouldn't work across wikis?
>>>
>>> Oddly, the documentation says #ifexist is an "expensive" parser function.
>>> That doesn't make much sense to me. It's as if red/blue links were
>>> expensive, and most of our list pages should be banned.
>>>
>>> https://www.mediawiki.org/wiki/Help:Extension:ParserFunctions#.23ifexist
>>>
>>>
>>> --
>>>   Lars Aronsson (l...@aronsson.se)
>>>   Aronsson Datateknik - http://aronsson.se
>>>
>>>
>>>
>>> ___
>>> Wikitech-l mailing list
>>> Wikitech-l@lists.wikimedia.org
>>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>>
>> ___
>> Wikitech-l mailing list
>> Wikitech-l@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>
>
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Ifexists across wikis

2015-12-06 Thread Purodha Blissenbach 

How about using the API on the targe side?
Purodha

On 06.12.2015 18:04, Alex Monk wrote:

I don't think there is a way to get a database name from an interwiki
prefix.

Also, whether a page is known or not does not just depend on a simple
database lookup. Extensions can add arbitrary rules about which 
titles
should be considered known or not. EducationProgram, GlobalUserPage, 
and

WikimediaIncubator all do this.

On 6 December 2015 at 16:26, Lars Aronsson  wrote:

If I write a [[link]] it will be blue if the page exists and red 
otherwise.
But if I write [[:sw:link]] that will be an external or cross-wiki 
link,
that is never red, as if it were impossible to know whether that 
page

existed in Swahili Wikipedia.

But determining the existence of a page is just a quick database 
table
lookup, and all databases run on WMF's servers, so it shouldn't be 
more
expensive to look up a cross-wiki link, as long as it is one of 
WMF's

wikis.

In Wiktionary, it is common to link to entries in foreign languages 
both
on the local wiki and to the native wiki for that language. For 
example,
in English Wikitionary the entry for "blue" links to the Swahili 
word

"bluu"
both on en.wiktionary and on sw.wiktionary, using the template
{{t+|sw|bluu}}.

https://en.wiktionary.org/wiki/blue#Translations

But since the Afrikaans translation "blou" doesn't have an entry on 
the
Afrikaans Wiktionary, another template is used: {{t|af|blou}}. And 
it is
a pain to know which one of these two templates to use. If it was 
possible
in {{#ifexists}} to determine the existence of a page in another 
wiki,
only one template would be needed, and the bot job to change to the 
right

template would not be needed.

#ifexist already works across namespaces (well, of course), so is 
there any

good reason it shouldn't work across wikis?

Oddly, the documentation says #ifexist is an "expensive" parser 
function.

That doesn't make much sense to me. It's as if red/blue links were
expensive, and most of our list pages should be banned.

https://www.mediawiki.org/wiki/Help:Extension:ParserFunctions#.23ifexist


--
  Lars Aronsson (l...@aronsson.se)
  Aronsson Datateknik - http://aronsson.se



___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l



___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Ifexists across wikis

2015-12-06 Thread Bartosz Dziewoński 

On 2015-12-06 17:26, Lars Aronsson wrote:

If I write a [[link]] it will be blue if the page exists and red otherwise.
But if I write [[:sw:link]] that will be an external or cross-wiki link,
that is never red, as if it were impossible to know whether that page
existed in Swahili Wikipedia.

But determining the existence of a page is just a quick database table
lookup, and all databases run on WMF's servers, so it shouldn't be more
expensive to look up a cross-wiki link, as long as it is one of WMF's
wikis.


> (...)


#ifexist already works across namespaces (well, of course), so is there any
good reason it shouldn't work across wikis?

Oddly, the documentation says #ifexist is an "expensive" parser function.
That doesn't make much sense to me. It's as if red/blue links were
expensive, and most of our list pages should be banned.
https://www.mediawiki.org/wiki/Help:Extension:ParserFunctions#.23ifexist


To add to what Alex and Florian said, the simple database lookup to 
check page existence is not actually that simple. When parsing a page, 
the query to determine link color (and to mark links to non-existent, 
redirect or disambig pages) is done in batches of 1000 links, after the 
whole page has been parsed and we know all the pages it links to.

Special pages that have lists of links use a similar method.

This wouldn't be possible if we needed to query a different database for 
each link (at best, perhaps we could batch them per-database, which 
doesn't help the Wiktionary use case of links to various sites).


It's also why #ifexist is expensive: it needs a separate database query 
for each time it's used, to check for a single page, because it's 
impossible to determine the list of pages to check in advance.


--
Bartosz Dziewoński

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Is there some non-ssl mirror of wikipedia?

2015-12-06 Thread Andre Klapper 
On Sun, 2015-12-06 at 17:07 +0100, Petr Bena wrote:
> I noticed that since we enforce SSL on Wikipedia for everyone,
> Wikipedia is much more restricted in some countries, such as China,
> where it's entirely blocked (I think only SSL is blocked, but users
> now have no option to fall back to non-ssl version).
> 
> Is there some non-ssl mirror for people in these countries? Or did we
> just gave up on "free knowledge for everyone"?

https://meta.wikimedia.org/wiki/HTTPS#Links 
links to a "Discussions" subpage which lists past discussions.

Cheers,
andre
-- 
Andre Klapper | Wikimedia Bugwrangler
http://blogs.gnome.org/aklapper/



___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Ifexists across wikis

2015-12-06 Thread Florian Schmidt 
I'm not very familar with this, but wouldn't this need a bigger change in 
LinksUpdate? Or the question: how would a wiki know, if a page get's created 
after it was linked and mark it blue instead of red?

Gesendet mit meinem HTC

- Nachricht beantworten -
Von: "Alex Monk" 
An: "Wikimedia developers" 
Betreff: [Wikitech-l] Ifexists across wikis
Datum: So., Dez. 6, 2015 18:04

I don't think there is a way to get a database name from an interwiki
prefix.

Also, whether a page is known or not does not just depend on a simple
database lookup. Extensions can add arbitrary rules about which titles
should be considered known or not. EducationProgram, GlobalUserPage, and
WikimediaIncubator all do this.

On 6 December 2015 at 16:26, Lars Aronsson  wrote:

> If I write a [[link]] it will be blue if the page exists and red otherwise.
> But if I write [[:sw:link]] that will be an external or cross-wiki link,
> that is never red, as if it were impossible to know whether that page
> existed in Swahili Wikipedia.
>
> But determining the existence of a page is just a quick database table
> lookup, and all databases run on WMF's servers, so it shouldn't be more
> expensive to look up a cross-wiki link, as long as it is one of WMF's
> wikis.
>
> In Wiktionary, it is common to link to entries in foreign languages both
> on the local wiki and to the native wiki for that language. For example,
> in English Wikitionary the entry for "blue" links to the Swahili word
> "bluu"
> both on en.wiktionary and on sw.wiktionary, using the template
> {{t+|sw|bluu}}.
>
> https://en.wiktionary.org/wiki/blue#Translations
>
> But since the Afrikaans translation "blou" doesn't have an entry on the
> Afrikaans Wiktionary, another template is used: {{t|af|blou}}. And it is
> a pain to know which one of these two templates to use. If it was possible
> in {{#ifexists}} to determine the existence of a page in another wiki,
> only one template would be needed, and the bot job to change to the right
> template would not be needed.
>
> #ifexist already works across namespaces (well, of course), so is there any
> good reason it shouldn't work across wikis?
>
> Oddly, the documentation says #ifexist is an "expensive" parser function.
> That doesn't make much sense to me. It's as if red/blue links were
> expensive, and most of our list pages should be banned.
> https://www.mediawiki.org/wiki/Help:Extension:ParserFunctions#.23ifexist
>
>
> --
>   Lars Aronsson (l...@aronsson.se)
>   Aronsson Datateknik - http://aronsson.se
>
>
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Ifexists across wikis

2015-12-06 Thread Alex Monk 
I don't think there is a way to get a database name from an interwiki
prefix.

Also, whether a page is known or not does not just depend on a simple
database lookup. Extensions can add arbitrary rules about which titles
should be considered known or not. EducationProgram, GlobalUserPage, and
WikimediaIncubator all do this.

On 6 December 2015 at 16:26, Lars Aronsson  wrote:

> If I write a [[link]] it will be blue if the page exists and red otherwise.
> But if I write [[:sw:link]] that will be an external or cross-wiki link,
> that is never red, as if it were impossible to know whether that page
> existed in Swahili Wikipedia.
>
> But determining the existence of a page is just a quick database table
> lookup, and all databases run on WMF's servers, so it shouldn't be more
> expensive to look up a cross-wiki link, as long as it is one of WMF's
> wikis.
>
> In Wiktionary, it is common to link to entries in foreign languages both
> on the local wiki and to the native wiki for that language. For example,
> in English Wikitionary the entry for "blue" links to the Swahili word
> "bluu"
> both on en.wiktionary and on sw.wiktionary, using the template
> {{t+|sw|bluu}}.
>
> https://en.wiktionary.org/wiki/blue#Translations
>
> But since the Afrikaans translation "blou" doesn't have an entry on the
> Afrikaans Wiktionary, another template is used: {{t|af|blou}}. And it is
> a pain to know which one of these two templates to use. If it was possible
> in {{#ifexists}} to determine the existence of a page in another wiki,
> only one template would be needed, and the bot job to change to the right
> template would not be needed.
>
> #ifexist already works across namespaces (well, of course), so is there any
> good reason it shouldn't work across wikis?
>
> Oddly, the documentation says #ifexist is an "expensive" parser function.
> That doesn't make much sense to me. It's as if red/blue links were
> expensive, and most of our list pages should be banned.
> https://www.mediawiki.org/wiki/Help:Extension:ParserFunctions#.23ifexist
>
>
> --
>   Lars Aronsson (l...@aronsson.se)
>   Aronsson Datateknik - http://aronsson.se
>
>
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Ifexists across wikis

2015-12-06 Thread Lars Aronsson 

If I write a [[link]] it will be blue if the page exists and red otherwise.
But if I write [[:sw:link]] that will be an external or cross-wiki link,
that is never red, as if it were impossible to know whether that page
existed in Swahili Wikipedia.

But determining the existence of a page is just a quick database table
lookup, and all databases run on WMF's servers, so it shouldn't be more
expensive to look up a cross-wiki link, as long as it is one of WMF's wikis.

In Wiktionary, it is common to link to entries in foreign languages both
on the local wiki and to the native wiki for that language. For example,
in English Wikitionary the entry for "blue" links to the Swahili word "bluu"
both on en.wiktionary and on sw.wiktionary, using the template 
{{t+|sw|bluu}}.


https://en.wiktionary.org/wiki/blue#Translations

But since the Afrikaans translation "blou" doesn't have an entry on the
Afrikaans Wiktionary, another template is used: {{t|af|blou}}. And it is
a pain to know which one of these two templates to use. If it was possible
in {{#ifexists}} to determine the existence of a page in another wiki,
only one template would be needed, and the bot job to change to the right
template would not be needed.

#ifexist already works across namespaces (well, of course), so is there any
good reason it shouldn't work across wikis?

Oddly, the documentation says #ifexist is an "expensive" parser function.
That doesn't make much sense to me. It's as if red/blue links were
expensive, and most of our list pages should be banned.
https://www.mediawiki.org/wiki/Help:Extension:ParserFunctions#.23ifexist


--
  Lars Aronsson (l...@aronsson.se)
  Aronsson Datateknik - http://aronsson.se



___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Is there some non-ssl mirror of wikipedia?

2015-12-06 Thread Petr Bena 
I noticed that since we enforce SSL on Wikipedia for everyone,
Wikipedia is much more restricted in some countries, such as China,
where it's entirely blocked (I think only SSL is blocked, but users
now have no option to fall back to non-ssl version).

Is there some non-ssl mirror for people in these countries? Or did we
just gave up on "free knowledge for everyone"?

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l