Re: [Toolserver-l] Is the wikidata replication out of sync?

[Kolossos]

Thanks for your answer.
I was connect with sql-s3 by using for connecting to db by:
mysql -h wikidatawiki-p.db.toolserver.org

On sql-s3 I still miss the entry but on sql-s5 it works.

Greetings Tim


Am 02.03.2014 14:32, schrieb Merlissimo:
> Am 02.03.2014 12:58, schrieb Kolossos:
>> Is there any idea for a workaround to get from 300.000 Wikipedia
>> articles the Wikidata Q-Number?
>>
>> [1]   SELECT `ips_item_id` FROM `wb_items_per_site`
>> WHERE `ips_site_id` = 'dewiki'
>> AND `ips_site_page` = 'Bundesanstalt_für_Verwaltungsdienstleistungen';
> 
> dewiki and wikidatawiki are on the same database s5, so there is not
> difference. And replication is ok. Only commonswiki is missing on s5
> since two days.
> Your query should not return any result on both databases because
> ips_site_page is using spaces instead of underscores. Because of the "ü"
> you could also use a wrong character encoding on your connection.
> For me
> 
>  SELECT @@hostname, `ips_item_id` FROM
> wikidatawiki_p.`wb_items_per_site` WHERE `ips_site_id` = 'dewiki' AND
> `ips_site_page` = 'Bundesanstalt für Verwaltungsdienstleistungen';
> 
> returns the correct result on toolserver and labs.
> 
> ++-+
> | @@hostname | ips_item_id |
> ++-+
> | z-dat-s5-b |15793045 |
> ++-+
> 1 row in set (0.00 sec)
> ++-+
> | @@hostname | ips_item_id |
> ++-+
> | labsdb1002 |15793045 |
> ++-+
> 1 row in set (0.03 sec)
> 
> 
> But you could also rewrite your query and request dewiki instead of
> wikidatawiki:
> 
> SELECT TRIM(LEADING 'Q' FROM TRIM(LEADING 'q' FROM pp_value)) AS ips_item_id
>  FROM dewiki_p.page
>   INNER JOIN dewiki_p.page_props ON page_id=pp_page
>  WHERE page_namespace=0 AND
> page_title='Bundesanstalt_für_Verwaltungsdienstleistungen'
>  AND pp_propname='wikibase_item';
> 
> +-+
> | ips_item_id |
> +-+
> | 15793045|
> +-+
> 1 row in set (0.04 sec)
> 
> 
> ___
> Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
> Posting guidelines for this list: 
> https://wiki.toolserver.org/view/Mailing_list_etiquette
> 



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

[Toolserver-l] Is the wikidata replication out of sync?

[Kolossos]

Hello,
if I test the query [1] on Toolserver I got an empty result, on labs I
got an correct answer. With other articles this query works fine. Could
somebody check this?

This query is part of project WIWOSM, so unfortunately I can not move to
Labs in the moment, because this project need a Postgresql installation
that there still not exist.

Is there any idea for a workaround to get from 300.000 Wikipedia
articles the Wikidata Q-Number?

Greetings Tim

[1]   SELECT `ips_item_id` FROM `wb_items_per_site`
WHERE `ips_site_id` = 'dewiki'
AND `ips_site_page` = 'Bundesanstalt_für_Verwaltungsdienstleistungen';


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] [Maps-l] Making Toolserver work - rate limited OSM

[Kolossos]

Hello,
the second line in the log had the referrer:
http://toolserver.org/~kolossos/openlayers/embed.html
This script is included inside geohack, so it's no wonder if a lot of
requests coming with this referrer from different IP's. But if a lot of
requests are coming only from one IP than perhaps a kid want to create
the next big search engine after Google. So it's correct to throttle
them. Perhaps we should also check our robot.txt.

Gpsie.com is a special topic, as they have many users they can have a
positive effect for OSM. They still use cached hillshading generated by
us, but there will be no updates necessary in the next time.
The tiles from hikebike-styles can change each minute, and it depends on
the user which delay is acceptable. For a mapper the hikbike map is the
only chance to see own modifications on hiking routes, so a delay of 10
minutes is the maximum. For a normal user a delay of a week or a month
should be no problem.

Greetings Tim alias Kolossos


Am 04.10.2013 23:39, schrieb Marlen Caemmerer:
> Hello,
> 
> Kai, thank you for your prompt and informative response.
> 
> On Thu, 3 Oct 2013, Kai Krueger wrote:
> 
>>
>> do you have any more details of which tile layers are getting hit? Is it
>> low or high zoom tiles? What referes / user-agents do they come from? Is
>> it the tiles that get served through mod_tile, the hillshading tiles or
>> the tiles for the wiki mini atlas?
> 
> I can send you some example log lines of the throttled IPs:
> 
> 2013/10/04 08:11:30 [error] 28822#0: *53658093 limiting requests,
> excess: 55.240 by zone "hikebike", client: 213.73.96.44, server:
> toolserver.org, request: "GET /tiles/hikebike/15/17169/11177.png
> HTTP/1.1", host: "toolserver.org", referrer:
> "http://www.gpsies.com/map.do?fileId=gbcojbhrdfqlglbc";
> 
> 2013/10/04 08:11:02 [error] 28822#0: *53650597 limiting requests,
> excess: 55.120 by zone "hikebike", client: 85.0.37.63, server:
> toolserver.org, request: "GET /tiles/
> osm/3/4/0.png HTTP/1.1", host: "c.www.toolserver.org", referrer:
> "http://toolserver.org/~kolossos/openlayers/embed.html?layer=mapnik&bbox=39.68865498340449,43.5524339
> 
> 5214844,39.778011016595514,43.614232047851566&marker=43.58,39.73
> 
> If you want to get more logs I'd send them to you in private.
> 
> It seems to relate especially hikebike/cmarq tools.
> 
>>
>> Too high load from individual clients has been an issue on many other
>> tileservers as well. Mostly it comes from various mobile apps, that
>> offer their users to download large areas (e.g. Germany) for offline
>> use. These areas then cover potentially millions of tiles, that the
>> clients then try and download as fast as the connection allows.
> 
> Sounds plausible.
> 
>>
>> For that reason, the tileservers on osm.org have a significant list of
>> user-agents that they block completely and in addition they also have an
>> automatic rate limiting per IP. There is also a specific tile usage
>> policy ( https://wiki.openstreetmap.org/wiki/Tile_usage_policy ) that
>> gouverns how you are allowed to technically access the tile servers
>> (once you have it downloaded, the use is freely gouverned by the
>> CC-BY-SA licence)
>>
>> Other tileservers like the opencyclemap, equally have restrictions and
>> mod_tile (the apache module used to deliver tiles) has a number of
>> features available to limit traffic. mod_tile also has a complex system
>> to try and ensure maximum cachability of tiles while still ensuring
>> up-to-dateness. This system can furthermore be tuned either towards
>> fresshness or cacheability as needed.
>>
>> My impression was so far this has never been an issue with the
>> toolserver and I wasn't aware of any explicit policies of how the
>> toolserver tiles are allowed to be accessed, so I never activated any of
>> the limiting features. But if it is becoming an issue we can see how
>> best to compat the issue.
> 
> gpsies.com stated they use the cache-control header which is sometimes
> not set reasonably probably as far as I tried to see - i had a look at
> these hikebike URL delivered from cmarq.
> The will look  at the problem closer on their side so I expect some more
> details in the next days.
> 
> I could set cache control headers in the nginx which acts as load
> balancer for TS for tiles where it makes sense. Do you have any advices
> on this? Which tiles dont change for what time about?
> 
> 
>>
>> At least on the munin graphs for ptolemy, I don't see much increased
>> load. But if it is the hillshading tiles, or the WMA tiles, 

Re: [Toolserver-l] My fading out

[Kolossos]

Am 31.05.2013 02:20, schrieb DaB.:

The Toolserver is not just a place where you can put a program and run it or
host a website. It’s a living community creating stuff in a anarchic way that
works only in praxis but not in theory; it’s like Wikipedia. WikiLabs is more
like Nupedia – in theory it is better, but in praxis it is empty and cold. The
difference is that for Nupedia Jimbo accepted that it can not work and stopped
it, and forced not Wikipedia user to switch to the_better_  platform. In our
case it is just the way around: After the WMF noticed that nobody needed
WikiLabs that started to look for a problem for their solution, and found the
Toolserver.


I take a look at Toollabs, also to be able as a WMDE member to defend 
Toolserver. What I found was the opposite of cold and empty.
It cost me only 1 minute to transform my project to a multi-maintainer 
project with a co-maintainer. I don't need to change the URL or need to 
ask an admin. So anarchic cooperation is really easy. Other things are 
very similar to the toolserver so that the community can further riding 
on an other horse.


@Dab: If you see it as a fight against WMF, you had never a chance to 
win and I can understand your personal tragedy. But in OpenSource field 
you can not avoid that somebody takes your idea and copy it. But I say 
we can also learn from Toolslabs and should think about after some time 
to create a Toolserver 2.0.


Special topic OpenStreetMap:
On toolserver I had the problem that some of my OSM tools "fading out" 
over the years because the database become incredible slow as the data 
volume grow by factor 5. A solution on Toolserver seems not possible, 
also if it would cost (with a SSD) under 1000€. So there was a level of 
frustrating and some people leave the project. And so I'm looking for an 
other solution to bring my work, my projects and the projects of others 
back to life and develop new stuff. I hope now that we will have on labs 
a fast OSM database for tools and experiment. (Rendering of popular 
styles should run in production environment, not in Labs.)


Greeting Tim alias Kolossos


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] My fading out

[Kolossos]

Hello,
so many wonderful things were created at Toolserver, so many nice 
cooperation’s with other users happen here and so many things I learn on 
Toolserver. I don't want look back in anger.


I know that all of these was only possible with your work Dab. So thank 
you very much.


A modification/restart of Toolserver seems for me after the years so or 
so necessary and perhaps we can find in the next 6 months a concept for 
a leaner, powerful Toolserver that gives the WMDE flexibility and 
independence back to develop ideas and support free projects. Hardware 
support is for me still a very good way of spending money if we want to 
support free knowledge, but it makes in my eyes no sense to fight 
against the majority of WMDE.


Greetings Tim alias Kolossos

Am 29.05.2013 15:09, schrieb DaB.:

Hello guys,

I just extended my personal account until 5. January of 2014 – it is the last
time I do this. At this day I will also remove my access as root of the
Toolserver. Beginning of 1. July I will start my fade out, doing less and less
work for the Toolserver until I am not longer visible. I announce that this
early because I think it is fair for you to know that will happen and I like
not just to vanish like some roots before.

There are 4 main factors why I decided to not continue my work until the end
of the Toolserver in December 2014.

Reason 1 is that the Toolserver now has a second paid root and 6 months will
be enough to teach amette and nosy what I know about the Toolserver.

Reason 2 is that there was no real investment in the Toolserver in the first 6
months of 2013 and I very doubt that there will ever be any in the second half
or beyond.

Reason 3 is that I learned during the last weekend that the support of the
Toolserver in the board of WMDE reached its minimum.
One board-member announced publicly during the general meeting of WMDE that it
is good that there is a timetable for the Toolserver now – I know only 1
timetable for the Toolserver and that’s Silke’s plan of destruction
roadmap for migration [1].
Another board-member told me during a chatting in the halls that ToolLabs (or
the move to) is "klasse" (~great).
It is impossible to improve the Toolserver against the CEO *and* the board of
WMDE.

Reason 4 are you, the tool-authors.
The participation in my survey [2] was pitiful low and the majority of these
few who voted, voted to leave the Toolserver as soon as possible or this year
– a trend that was already visible on the mailing-list before. So I conclude
that the most of you don’t care and whose care will leave this year.
While I asked for documentation (or at least correction) in the toolserver-
wikis for years, nearly nothing ever happened. But now that ToolLabs is on the
horizon you write documentation for THAT – freely.
And it is really a joke to compare the empty new database-servers of ToolLabs
with our old and heavy loaded servers for performance. Let’s see how fast they
are if 10 slow queries, which had run for hours, run in parallel.
With very few exceptions none of you helped to protect the Toolserver against
ToolLabs; all you were interested in was that ToolLabs provides the same
environment so your tools can continue to run there. When I read such phrases
like "we have to stabilize the Toolserver until Labs is ready" or now "we need
the Toolserver for redirects to ToolLabs" I could vomit!

I promised in November 2012 that I will stay for another year and I will
fulfill that promise – but not a day longer. There is no point in fighting for
something if the something has already surrendered and no support is there
(not from you, the toolusers, the board of WMDE, the CEO of WMDE or the
general meeting of WMDE).

These of you who are able to move to ToolLabs I wish luck. Let’s hope that the
WMF does not decide to "re-focus" again too soon. Let’s hope that the WMF does
not disable tools just because there are a little slow. Let’s hope that the
WMF does not restrict the database-tables even more. Let’s hope that the WMF
does not kick the volunteers out completely some days like they did with the
WMF-wiki-admins some weeks ago. And hoping is all we can do, because the WMF
is a undemocratic construct and ToolLabs is lead by paid roots, so whatever
the WMF staff decides will happen.
Maybe if one of these things happen you will remember the tiny, slow,
unstable, but free Toolserver — but it will not be there anymore.

Sincerely,
DaB.


[1] http://www.mediawiki.org/wiki/Tool_Labs/Roadmap_en
[2] https://wiki.toolserver.org/view/Labs-Moving-Survey




___
Toolserver-announce mailing list
toolserver-annou...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/toolserver-announce





___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for

Re: [Toolserver-l] Migrating to labs from the toolserver

[Kolossos]

For links in Wikimedia project I hope there will be a bot-system where I 
as developer can say tool xy is now at labs with name xyz and a bot 
change all links.


Redirects will not work after shut down, but will help for a long time.

Greetings Tim

Am 04.05.2013 15:13, schrieb Magnus Manske:

.htaccess redirects?


On Sat, May 4, 2013 at 2:10 PM, rupert THURNER mailto:rupert.thur...@gmail.com>> wrote:

What happens to the zillions of published toolserver links when a
tool switches to labs?

Rupert

Am 04.05.2013 14:18 schrieb "Magnus Manske"
mailto:magnusman...@googlemail.com>>:

As I boldly ssh in where few toolserver users have ssh'ed in
yet, I try to keep a log of what I'm doing, what issues arise,
and where I had to stop and scratch my head. Behold:


https://wikitech.wikimedia.org/wiki/User:Magnus_Manske/Migrating_from_toolserver

Feel free to improve the page as you see fit.

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org
)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list:
https://wiki.toolserver.org/view/Mailing_list_etiquette


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org
)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list:
https://wiki.toolserver.org/view/Mailing_list_etiquette




___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette





___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Thoughts and questions to WMDE members about TS and expectations

[Kolossos]

Hello,
If I think positive, I have the hope that we can move relatively shortly 
the 20% of tools to labs that are producing 80% of our load. So than we 
should come back in a stable situation on toolserver with acceptabel 
performance. This "stable time" we can use than to run tools until they 
are obsolete or moved to labs.


For now you should analyse the system and predict what we will need in 
such or an other scenario. It can be that we need so or so some new 
database servers, some more storage or whatever. If you need place and 
power in the rig, replace old stuff.


In the moment it can also be helpful to be hard and kill some bots/tools 
with a relative bad cost-benefit-relation (Perhaps every user can also 
check this for it's own, because it's hard to decide from the outside.). 
A statistic with the loads of tools would be nice for this.


The actual situation of Toolserver is a shame for Wikimedia Germany!
One actual example is the zoom-viewer[1], it's a tool that is "in 
production" on commons. Yesterday I want to use it, but it was unusable 
slow. I was so frustrated, but after months with tons of these problems 
on the server I feel that it makes no sense to fight against the 
situation, if WMDE decide to invest nothing. It's nearly impossible to 
develop on this server in the moment, because you get no felling for the 
performance of your changes in simple request can take 30sec or so. 
Again, only a shame!


For the long term I (as WMDE member) would support Toolserver to win 
flexibility and in-dependency. I learn that decisions from WMF can take 
months or years. Projects sponsert by WMDE (like my "Multilingual 
Maps"[2]) die perhaps because there is no server available.


WMDE has money and it's relatively cheap to invest money into techniques 
compared to pay persons in high-wages Germany. I don't believe that we 
lose only one donor if we spend money for a server system.


It would be nice if someone could explain me why it should be useful and 
secure for free knowledge to collect everything on one system. Yes, I 
understand the financial aspect but I also know the fire in the Library 
of Alexandria and the Nazi book burnings.


With more than one system in the world the developers can decide where 
the best environment and best support is for there work. This option 
will motivate all to increase the quality of service. With only one 
system we have to live with some decisions and also WMFlabs will become 
old over the years and better systems will come.


Greetings Tim alias Kolossos


[1] 
http://toolserver.org/~dschwen/iip/wip.php?f=Dresden-Neumarkt-Kulti-2013-04.jpg

[2] http://wiki.openstreetmap.org/wiki/Multilingual_maps_Wikipedia_project




___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Wikidata tables

[Kolossos]

Sounds fine, but will it be possible to join the data with data from 
other tables and other projects? This joins are the base for a lot of 
tools on toolserver and I#m not sure how good joins on application level 
will work.


BTW: With the project Templatetiger we already handle tons of 
informations of infoboxes in MYSQL on Toolserver, all these data are 
highly redundant because we support a lot of languages. So there should 
be enough space on toolserver in midterm. But I also think that labs are 
the right place to start such a project.


Greetings Tim

Am 19.04.2013 10:37, schrieb Magnus Manske:

FWIW, I have implemented a query-able stand-alone web server that keeps
all of the wikidata property-item-links in memory. This uses the
wikidata dumps which appear to be rather frequent. I'll try do deploy a
test version on wikilabs (once I figure out how all that works); it
seems to be more favourable to such services than the toolserver.




___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Wikidata tables

[Kolossos]

Hello,
Postrgesql starts now to support also JSON, so we should try to find a 
way to bring Wikidata available for us and I would prefer to use 
furthermore SQL.


One way could be minutely diff-files, that's the way OpenStreetMap use.
Alternatively we could use API for each updated article.

Every central service is better than let's fighting everyone with the 
problem alone.


I think the support of hierarchical informations was the key to use JSON 
instead of a key-value store. A point that I can understand.


Greetings Tim


Am 19.04.2013 00:43, schrieb Daniel Schwen:

if these JSON-data is stored where the normal wiki-text is, it is imposable

To my understanding it is.


for us to replicate it: Because we have no access to these wmf-servers, there

IMO that was a questionable design decision. JSON plaintext storage in
SQL is shoehorning a do-it-yourself object store onto a classical
RDBMS.
Postgres at least has hstore. This may be even a genuine usecase for
one of those hipster databases (noSQL like mongdb etc.). But who knows
what points were taken into consideration when making this decision.


would be no way to separate Wikidata from the rest

I don't understand why separating plaintext storage between different
projects would be an issue. Is it all lumped into one storage
"namespace"?
I'm sure nobody at Wikimedia would be the least bit motivated to make
this data available to the toolserver, but maybe it will be usable in
labs. Otherwise it would be quite a waste of a great opportunity.


and/or we have not enough disc-space.

If you can separate it out I seriously doubt that wikidata would
require storage any where near as large in magnitude as the other
wikimedia projects (at least in the mid-term)

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette





___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] IP Renumbering of the complete Toolserver cluster

[Kolossos]

Hello,
in the moment it's not possible to access ptolemy database via ssh.

Please repair it:
https://jira.toolserver.org/browse/OSM-22
(Jira allows no automatic assign in the moment, that's why this double 
posting.)


Greeting Tim

Am 04.04.2013 16:25, schrieb Marlen Caemmerer:


Hello,

On Thu, 4 Apr 2013, Artem Korzhimanov wrote:



I'm not sure that this is a problem of a toolserver-side, but I am unable
to login to (and even ping) submit.toolserver.org

Probably, it is the problem of DNS because it returns an
IP-address 81.15.59.215 which is not in a range given by DaB earlier.

There are no problems with any other hosts maintained yesterday.


This was a typo in DNS. Thanks for reporting. I fixed it and it might
take an hour to propagate.
The IP really is 185.15.59.215.

Cheers
 Marlen/nosy

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list:
https://wiki.toolserver.org/view/Mailing_list_etiquette




___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Reasons for not migrating to Tool Lab-OSM

[Kolossos]

Am 27.09.2012 17:21, schrieb Andrei Cipu:


Where can we find more information about these projects, especially OSM and WLM?


For OSM you can look at:
https://labsconsole.wikimedia.org/wiki/Nova_Resource:Maps

None of that instances has a public IP or an planet import in the 
moment. So it's far away from being usable, but that's also what Ryan said.


We don't know how efficient a PostgreSQL server runs in a 
virtualisation, but after talk with some PostgreSQL experts I'm a little 
bit sceptical.
We also don't know if the foundation will maintain an external 
high-performance db-server for OSM rendering or if we are as a 
non-wikimedia project at the end of the priority list or out-of-scope.


Especially we don't know when will labs be ready for production. We 
could need it at begining of next year or sooner to roll-out a 
CPB-project (sponsored by German chapter) and to do other map things.


Greetings Tim alias Kolossos



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] External authentication?

[Kolossos]

Am 09.02.2012 14:08, schrieb Victor Vasiliev:

On Thu, Feb 9, 2012 at 3:37 PM, Magnus Manske
  wrote:

Hi all,

on my talk page, [[User:Pathoschild]] raised the idea of allowing
OpenID authentication to operate toolserver tools that currently rely
on TUSC. While I'd rather go for browserID [1] (not mutually
exclusive), it raised the point of which authentication is "good
enough" for using some toolserver tools, especially those that edit or
upload on Wiki(m|p)edia projects.

Would these non-TUSC accounts need to be linked to Wiki(m|p)edia user
names? If so, how would this be done? If Wiki(m|p)edia were to provide
openID/browserID authentication, it would be a non-issue, but as it
stands, this would need to be done on the toolserver in some form,
which would most likely be more cumbersome than the current TUSC
account creation.

Ideas?


Magnus



Well, Wikimedia should become OpenID/OAuth provider for that.

--vvv

We had a talk on maps-l[1] to allow Wikipedians to edit easily 
OpenStreetMap. So Wikimedia as OpenID/OAuth provider would be really nice.


Greetings Tim

[1]http://lists.wikimedia.org/pipermail/maps-l/2012-January/001113.html


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Google Art Project ripping

[Kolossos]

You can make your own pictures if you have
a 7 Gigapixel camera ;-)
In this video they show how google works:
http://www.youtube.com/watch?v=D1EOJr11bvo

Also if highres-cameras from gigapan.org are cheap available you need 
time and the right light in the museum.

We still have some technics on commons to get high resolution images 
from websites:
http://commons.wikimedia.org/wiki/Help:Zoomable_images
This seems the right place for documentation.

But I'm not sure perhaps better to ask friendly or to wait until more 
images from museums are online. Not sure if the museums are happy if we 
take the pictures and present it in a format that you can easily print 
out

Greetings

Am 04.02.2011 14:12, schrieb Magnus Manske:
> I see "NPG, reloaded" coming our way...
>
> Note that Commons already has plenty of images from at least some of
> these museums, e.g.:
> http://commons.wikimedia.org/wiki/Van_Gogh_Museum
>
> For the rest, we could just ask them "now that Google has put images
> online, can we too?", then go and take our own pictures.
>
> Too radical? ;-)
>
> Magnus
>
>
> On Fri, Feb 4, 2011 at 7:15 AM, Beao at Toolserver.org
>   wrote:
>> Hello everybody! I've been working on a bash script to rip the high quality
>> images from the Google Art Project.
>>  From what I understand, all depictions of the original works go under
>> PD-Art, even though Google claims otherwise in their FAQ. What I'm wondering
>> is whether anyone is interested in helping me rip all these images. There
>> are a lot of images to rip, and each rip takes at least an hour. All you
>> need is a GNU/Linux OS and some basic knowledge on using the terminal.
>>
>> --
>> Beao
>>
>> ___
>> Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
>> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
>> Posting guidelines for this list:
>> https://wiki.toolserver.org/view/Mailing_list_etiquette
>>
>
> ___
> Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
> Posting guidelines for this list: 
> https://wiki.toolserver.org/view/Mailing_list_etiquette



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] MySQL problem - works

[Kolossos]

Thanks MZMcBride for help.


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

[Toolserver-l] MySQL problem

[Kolossos]

Hello,
If I want to load a file into MYSQL with the command[1] I get an error 
1148 [2].
I did this many times before without problems. So I want to ask if there 
was a change in toolserver configuration and what the solution can be 
for me.

Greetings Kolossos


[1]
LOAD /* SLOW_OK */ DATA LOCAL INFILE 
'/home/sk/data/templatetiger/dewiki/dewiki_templatetiger.txt' IGNORE 
INTO TABLE `dewiki_neu` FIELDS TERMINATED BY '\t' ESCAPED BY '\\' LINES 
TERMINATED BY '\n' ;

[2]
ERROR 1148 (42000): The used command is not allowed with this MySQL version


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Monthly pageviews

[Kolossos]

A text file would be a good step to put the data in a database.
I did this in past with my public database u_kolossos_wp_logs_p  as 
there was http://wikistics.falsikon.de/dumps.htm with such text files,
but the last update was 2009.

Greetings Kolossos


Frédéric Schütz schrieb:
> Would text files (similar to the current page views, but summarised over 
> each month) be ok ? I have a few scripts (some from Erik Zachte, and 
> some of mine) that could be adapted to do this.
> 
> It's not the most efficient way to do it if you need random access (the 
> API from stats.grok.se is probably better for this), but it would still 
> be quite straightforward to parse.
> 
> Frédéric
> 
> On 17.11.2010 22:08, Kolossos wrote:
>> Hello, I'm also very interested to get easily montly statistics, that I
>> can use as criteria for importance of articles to show them on a map[1].
>> So, I hope we get it.
>>
>> Greetings Kolossos
>> [1] http://de.wikipedia.org/wiki/Hilfe:OpenStreetMap/en
>>
>> Magnus Manske schrieb:
>>> On Mon, Nov 15, 2010 at 10:35 PM, MZMcBride  wrote:
>>>> Magnus Manske wrote:
>>>>> I know there are lots'o'files for daily (hourly?) pageview stats on
>>>>> the toolserver.
>>>>>
>>>>> Are there aggregated counts for the whole month? So I only have to
>>>>> check 1 file instead of hundreds (the aggregated file would, of
>>>>> course, be smaller than the concatenated hourly ones).
>>>>> Or maybe even as a database? (Onecan dream...)
>>>>>
>>>>> If not, does anyone volunteer to generate them? They'd really help
>>>>> with my GLAM tools, increase Wikimedia outreach etc.
>>>> Pageview stats are still a mess and there's no centralized or clean
>>>> database, as far as I'm aware. Henrik's tool (stats.grok.se) has an API you
>>>> can hit for monthly stats: http://stats.grok.se/json/en/201006/Barack_Obama
>>>>
>>>> That's probably your best bet right now.
>>> And that's what I'm doing, but I need to look for tens of thousands of
>>> pages, and it's very slow, not to mention traffic.
>>>
>>>>  From what I understand, Wikimedia is devoting resources to setting up Open
>>>> Web Analytics. The first test run is supposed to be this week, I think.
>>> That sounds good. Was that announced anywhere?
>>>
>>> Thanks,
>>> Magnus
>>>
>>
>> ___
>> Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
>> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
>> Posting guidelines for this list: 
>> https://wiki.toolserver.org/view/Mailing_list_etiquette
> 
> 


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Monthly pageviews

[Kolossos]

Hello, I'm also very interested to get easily montly statistics, that I 
can use as criteria for importance of articles to show them on a map[1]. 
So, I hope we get it.

Greetings Kolossos
[1] http://de.wikipedia.org/wiki/Hilfe:OpenStreetMap/en

Magnus Manske schrieb:
> On Mon, Nov 15, 2010 at 10:35 PM, MZMcBride  wrote:
>> Magnus Manske wrote:
>>> I know there are lots'o'files for daily (hourly?) pageview stats on
>>> the toolserver.
>>>
>>> Are there aggregated counts for the whole month? So I only have to
>>> check 1 file instead of hundreds (the aggregated file would, of
>>> course, be smaller than the concatenated hourly ones).
>>> Or maybe even as a database? (Onecan dream...)
>>>
>>> If not, does anyone volunteer to generate them? They'd really help
>>> with my GLAM tools, increase Wikimedia outreach etc.
>> Pageview stats are still a mess and there's no centralized or clean
>> database, as far as I'm aware. Henrik's tool (stats.grok.se) has an API you
>> can hit for monthly stats: http://stats.grok.se/json/en/201006/Barack_Obama
>>
>> That's probably your best bet right now.
> 
> And that's what I'm doing, but I need to look for tens of thousands of
> pages, and it's very slow, not to mention traffic.
> 
>> From what I understand, Wikimedia is devoting resources to setting up Open
>> Web Analytics. The first test run is supposed to be this week, I think.
> 
> That sounds good. Was that announced anywhere?
> 
> Thanks,
> Magnus
> 


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Fwd: [Tagging] Download huge file via HTTP

[Kolossos]

Try gzip/zip perhaps it has a positive effect. 1.5 GB should be than no 
problem.

Greetings Kolossos

Peter Körner schrieb:
> No comments on that?
> 
> Peter
> 
> Am 06.11.2010 20:18, schrieb Peter Körner:
>> Hi
>>
>> I tried to provide a download link to a complete OSM Render Stack in a
>> Virtual Machine on the toolserver. The files are located at
>> /mnt/user-store and symlinked into my public_html.
>>
>> This works for the small files (md5sums) but the 3.1G VMDK File is not
>> accessible: http://toolserver.org/~mazder/osm_vm/
>>
>> Is it possible to serve such a large file via http?
>>
>> Peter
> 


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] google analytics (and similar services)

[Kolossos]

I would also like to see piwik or webstat or something else.
Such statistics are mostly the only feedback for developer and so IMO an 
important
way of developer motivation. Some tools are important for quality improvements 
in WP but not so
often used that they would be inside existing statistics[1].

A monthly usage statistic for all tools would be enough for me, with more 
analytics I would be carefull.
Not sure what we need to detect misusage of tools by automatic request, that we 
had also in past.

Greetings Kolossos

[1] http://toolserver.org/~daniel/stats/


Daniel Kinzler schrieb:
> DaB. schrieb:
>> Hello all,
>>
>> I was noticed by a user today that a few web-tools on the toolserver (at 
>> least 
>> 3) uses google analytics and violates our rules with that. No user is 
>> allowed 
>> to "publish connection data, especially IP addresses, of other people " [1]; 
>> "publish" means also "give data away to (other) companies or webpages".
> 
> As a free and safe alternative, i can recommend piwik. maybe we could set that
> up globally, even?
> 
> -- daniel
> 


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette