Re: [Wiki-research-l] Sampling new editors in English Wikipedia

fn Wed, 13 Mar 2019 09:02:36 -0700

Haifeng,


On 13/03/2019 15:56, Haifeng Zhang wrote:

Thanks for pointing me to Quarray, Finn.

I tried a couple queries, but not sure why all took forever to get result.

I am not familiar with Quarry. It might have a timeout. The user tableassociated with the English Wikipedia is quite large, so any operationon that may take long time.

You might be able to get "timein" with a simplified SQL. For instance,the query below takes 52.35 seconds:


USE enwiki_p;

SELECT user_id, user_name, user_registration, user_editcount
        FROM user
LIMIT 1000
OFFSET 32000000

Is it possible to download relevant Media Wiki database tables (e.g., user, 
user_groups, logging) and run SQL in my local machine?

There are SQL files available herehttps://dumps.wikimedia.org/enwiki/20190301/ but I do not think the usertable is there, - at least I cannot identify it. Perhaps other peoplewould know.

You might be able try the Toolforge https://tools.wmflabs.org/ Youshould be able to access the tables via mysql on the prompt.


Login to dev.tools.wmflabs.org
Then do "sql enwiki"

Read more about Toolforge here:https://wikitech.wikimedia.org/wiki/Help:Toolforge



/Finn


Thanks,

Haifeng Zhang
________________________________
From: Wiki-research-l <wiki-research-l-boun...@lists.wikimedia.org> on behalf of 
f...@imm.dtu.dk <f...@imm.dtu.dk>
Sent: Tuesday, March 12, 2019 7:25:53 PM
To: wiki-research-l@lists.wikimedia.org
Subject: Re: [Wiki-research-l] Sampling new editors in English Wikipedia

Haifeng ,


While some suggests the dumps or notice boards, my immediate thought was
a database query, e.g., through Quarry. It just happens that Jonathan T.
Morgan has created a query there:

https://quarry.wmflabs.org/query/310

SELECT user_id, user_name, user_registration, user_editcount
         FROM enwiki_p.user
         WHERE user_registration > DATE_FORMAT(DATE_SUB(NOW(),INTERVAL 1
DAY),'%Y%m%d%H%i%s')
         AND user_editcount > 10
         AND user_id NOT IN (SELECT ug_user FROM enwiki_p.user_groups WHERE
ug_group = 'bot')
         AND user_name not in (SELECT REPLACE(log_title,"_"," ") from
enwiki_p.logging
                 where log_type = "block" and log_action = "block"
                 and log_timestamp >  DATE_FORMAT(DATE_SUB(NOW(),INTERVAL 2
DAY),'%Y%m%d%H%i%s'));


You may fork from that query. There is R. Stuart Geiger (Staeiou)'s fork
here https://quarry.wmflabs.org/query/34256 querying for month, - as
another example.



Finn Årup Nielsen
http://people.compute.dtu.dk/faan/


On 12/03/2019 19:18, Haifeng Zhang wrote:

Hi folks,

My work needs to randomly sample new editors in each month, e.g., 100 editors 
per month.

Do any of you have good suggestions for how to do this efficiently?

I could think of using the dump files, but wonder are there other options?


Thanks,

Haifeng Zhang
_______________________________________________
Wiki-research-l mailing list
Wiki-research-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l


_______________________________________________
Wiki-research-l mailing list
Wiki-research-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
_______________________________________________
Wiki-research-l mailing list
Wiki-research-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l


_______________________________________________
Wiki-research-l mailing list
Wiki-research-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

Re: [Wiki-research-l] Sampling new editors in English Wikipedia

Reply via email to