Re: [Wikitech-l] database encoding for field with mathematical expressions
On 05/23/2013 11:31 PM, phy...@physikerwelt.de wrote: > Hi, > > I'm a testing a new rendering option for the element and had > problems to store MathML elements in the database field > math_mathml which is of type text. > The MathML elements contain a wide range of Unicode characters like the > INVISIBLE TIMES that is encoded as 0xE2 0x81 0xA2 in UTF-8 or even 4 byte > chars like MATHEMATICAL BOLD CAPITAL A 0xF0 0x9D 0x90 0x80 . > In some rar cases I had problem to retrieve the stored value correctly from > MySQL. > To fix that problem I'm now using the PHP functions utf8_encode /decode to > which is not a very intuitive solution. > Do you know a better method to solve this issue without to change the > database layout. > > Best > Physikerwelt If you use MySQL, when you installed MediaWiki (or created the table), did you choose the "UTF-8" option instead of "binary"? The underlying MySQL character set is "utf8"[1], which does not support characters above U+ (four-byte characters). This is mentioned in the web installer (message 'config-charset-help'): > In binary mode, MediaWiki stores UTF-8 text to the database in binary > fields. This is more efficient than MySQL's UTF-8 mode, and allows > you to use the full range of Unicode characters. In UTF-8 mode, MySQL > will know what character set your data is in, and can present and > convert it appropriately, but it will not let you store characters > above the Basic Multilingual Plane[2]." MySQL 5.5 did introduce a new "utf8mb4" character set, which does support four-byte characters; however, MediaWiki does not currently support that option (now filed as bug 48767). The WMF of course has to use the 'binary' option (actually, UTF-8 stored in latin1 columns, as mentioned in bug 32217) to allow storage of all sorts of obscure characters from different languages. utf8_encode()/utf8_decode() work around the problem because they replace byte values 80 to FF with two-byte characters from U+0080 to U+00FF, (encoded as C2 80 to C3 BF) and the 'utf8' option does allow those characters. [1]: https://dev.mysql.com/doc/refman/5.5/en/charset-unicode-utf8.html [2]: http://en.wikipedia.org/wiki/Mapping_of_Unicode_character_planes -- Wikipedia user PleaseStand http://en.wikipedia.org/wiki/User:PleaseStand ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] Publish-staying-in-editmode feature for WikiEditor
Hello! I have implemented an idea for WikiEditor extension: replace "step-by-step publish" feature with another one - "publish staying in edit mode via AJAX". You can see a demo at http://wiki.4intra.net/ if you want. It works simply by sending an API save article request while NOT closing the article being edited. Also it handles section edits correctly via re-requesting section content after editing, so you'll stay with consistent edit form even if you add sections. The idea is to give authors the ability to save intermediate results. My question is - does anyone really need "step-by-step publishing" feature that is in WikiEditor? I think it's useless because it just duplicates the existing functionality, just submits the form using normal POST request, and makes editing harder as you have to do more clicks. I would submit a patch to Gerrit if you're interested in replacing it with "publish-staying-in-editmode". -- With best regards, Vitaliy Filippov ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Recursive merging enabled for Gerrit repositories
awesome! On Thu, May 9, 2013 at 11:42 PM, Chad wrote: > Hi, > > We've enabled JGit's recursive merger for all repositories using content > merge > strategies (basically any not using fast forwarding, which is most). > The intention > is to lower the number of trivial conflicts that people are having to > resolve locally. > > This is considered experimental by Gerrit, but is now the default for > JGit itself, so > I believe it's stable enough for us to use. That being said, it's > still new-ish so there's > always a chance we'll hit some bug. So, if you see anything, ANYTHING > related to > merge problems then I'd like to know about it so we can either get it > fixed or turn the > feature back off (if it's patently broken). > > Thanks! > > -Chad > > ___ > Wikitech-l mailing list > Wikitech-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Hackathon idea: How about a live bug triage and code review hour ?
On Tue, 2013-05-21 at 22:28 +0200, Derk-Jan Hartman wrote: > We'd need to find a timeslot though, that's probably gonna be the > hardest part. I'm guessing that many WMF folks will have quite a few > meetings again. I'd be in and available, if somebody defines a room & time for it. (I'm also in for any Bugzilla or bug report questions in general.) andre -- Andre Klapper | Wikimedia Bugwrangler http://blogs.gnome.org/aklapper/ ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] database encoding for field with mathematical expressions
On 05/23/2013 11:31 PM, phy...@physikerwelt.de wrote: > Hi, > > I'm a testing a new rendering option for the element and had > problems to store MathML elements in the database field > math_mathml which is of type text. The Gerrit for this is https://gerrit.wikimedia.org/r/#/c/61987/ Matt Flaschen ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] Information on MW and Redis?
Hello all, I keep seeing references in WMF documents to using Redis for session storage and as the storage for the job queue going forward. However, LocalSettings doesn't have any references to Redis. It looks like the session changes for Redis were in 1.20, and I thought that the change for the job queue to be backed by Redis was coming in 1.21. It also seems that Notifications (Echo) may require Redis, but that's not real clear. I'm wondering what the general status of this is? As a 3rd party mediawiki admin, I'm eager to get redis going primarily for the job queue improvements, but with nothing noted in local settings documentation I'm wondering what the plan is for these redis based features? Will $wgMainCacheType be getting a CACHE_REDIS option? Jamie Thingelstad ja...@thingelstad.com mobile: 612-810-3699 find me on AIM Twitter Facebook LinkedIn ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Information on MW and Redis?
To use redis as a cache you can have something like: // requires phpredis extension for PHP $wgObjectCaches['pecl-redis'] = array( 'class' => 'RedisBagOStuff', 'servers' => array( '127.0.0.1:6379' ), ); $wgMainCacheType = 'pecl-redis'; This would also require that the redis server would have allkeys-lru for its eviction policy in redis.conf. To use redis for a jobqueue, one can have something like: // requires phpredis extension for PHP $wgJobTypeConf['default'] = array( 'class' => 'JobQueueRedis', 'redisServer'=> '127.0.0.1:6379', 'redisConfig'=> array(), 'claimTTL' => 3600 ); This works best if the redis server uses rdb snapshots and/or append-only-file logging in redis.conf so that jobs are lost with power outages or restarts. -- View this message in context: http://wikimedia.7.x6.nabble.com/Information-on-MW-and-Redis-tp5005659p5005660.html Sent from the Wikipedia Developers mailing list archive at Nabble.com. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Information on MW and Redis?
Awesome! Thank you! Do you know the version required for these? Thinking maybe I could writeup a page on mw.o documenting the process of putting redis in place for these. Jamie Thingelstad ja...@thingelstad.com mobile: 612-810-3699 find me on AIM Twitter Facebook LinkedIn On May 24, 2013, at 2:44 PM, Aaron Schulz wrote: > To use redis as a cache you can have something like: > > // requires phpredis extension for PHP > $wgObjectCaches['pecl-redis'] = array( > 'class' => 'RedisBagOStuff', > 'servers' => array( '127.0.0.1:6379' ), > ); > $wgMainCacheType = 'pecl-redis'; > > This would also require that the redis server would have allkeys-lru for its > eviction policy in redis.conf. > > To use redis for a jobqueue, one can have something like: > > // requires phpredis extension for PHP > $wgJobTypeConf['default'] = array( > 'class' => 'JobQueueRedis', > 'redisServer'=> '127.0.0.1:6379', > 'redisConfig'=> array(), > 'claimTTL' => 3600 > ); > > This works best if the redis server uses rdb snapshots and/or > append-only-file logging in redis.conf so that jobs are lost with power > outages or restarts. > > > > -- > View this message in context: > http://wikimedia.7.x6.nabble.com/Information-on-MW-and-Redis-tp5005659p5005660.html > Sent from the Wikipedia Developers mailing list archive at Nabble.com. > > ___ > Wikitech-l mailing list > Wikitech-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Information on MW and Redis?
If you implement this, is there any use for memcached still? I assume PHP also needs to be told that sessions are in redis? Jamie Thingelstad ja...@thingelstad.com mobile: 612-810-3699 find me on AIM Twitter Facebook LinkedIn On May 24, 2013, at 2:44 PM, Aaron Schulz wrote: > To use redis as a cache you can have something like: > > // requires phpredis extension for PHP > $wgObjectCaches['pecl-redis'] = array( > 'class' => 'RedisBagOStuff', > 'servers' => array( '127.0.0.1:6379' ), > ); > $wgMainCacheType = 'pecl-redis'; > > This would also require that the redis server would have allkeys-lru for its > eviction policy in redis.conf. > > To use redis for a jobqueue, one can have something like: > > // requires phpredis extension for PHP > $wgJobTypeConf['default'] = array( > 'class' => 'JobQueueRedis', > 'redisServer'=> '127.0.0.1:6379', > 'redisConfig'=> array(), > 'claimTTL' => 3600 > ); > > This works best if the redis server uses rdb snapshots and/or > append-only-file logging in redis.conf so that jobs are lost with power > outages or restarts. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Information on MW and Redis?
2.2.2 of the extensions works for me. I downloaded it from source and compiled it. The redis server itself will need to be 2.6 or higher for the job queue. Looking around, I forgot to mention that JobQueueRedis was actually removed from 1.21 (though it's in master and will be in 1.22). -- View this message in context: http://wikimedia.7.x6.nabble.com/Information-on-MW-and-Redis-tp5005659p5005664.html Sent from the Wikipedia Developers mailing list archive at Nabble.com. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Information on MW and Redis?
Note that if you already use memcached for the main cache, there isn't really any reason to switch to redis unless you need replication or persistence. Anyway, to use it for sessions, if you had $wgSessionCacheType explicitly set to something, then you'd need to change that too (like to 'pecl-redis'). In any case, it doesn't hurt to be explicit. This all assumes that $wgSessionsInObjectCache = true as well. -- View this message in context: http://wikimedia.7.x6.nabble.com/Information-on-MW-and-Redis-tp5005659p5005665.html Sent from the Wikipedia Developers mailing list archive at Nabble.com. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Information on MW and Redis?
I can see how memcached and redis would have similar performance for sessions. Is that also true for job queue? I was assuming that job queue could move from database to redis, but not to memcached. Is that a correct assumption? The main reason I could see moving sessions into redis would be to not manage both redis and memcached, assuming that redis is the key to making the job queue faster. -- Jamie Thingelstad ja...@thingelstad.com mobile: 612-810-3699 On May 24, 2013, at 3:07 PM, Aaron Schulz wrote: > Note that if you already use memcached for the main cache, there isn't really > any reason to switch to redis unless you need replication or persistence. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Information on MW and Redis?
Indeed, the queue cannot use memcached. Redis will trivialize the time spent on actually queue operations, which could help if that is a bottleneck for job runners. If the actual jobs themselves are slow, of course it won't help too much. Have you already tried setting the job run rate to 0 and is a background script instead? -- View this message in context: http://wikimedia.7.x6.nabble.com/Information-on-MW-and-Redis-tp5005659p5005693.html Sent from the Wikipedia Developers mailing list archive at Nabble.com. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
