Re: [Maria-developers] Incorrect format description event skipping in queue_event()

2013-09-03 Thread Kristian Nielsen
Pavel Ivanov writes: > Why not change Format_description_log_event::write()? It seems > actually that it will be more correct if we change it. I made the > following change (line numbers may be off) and it worked: Agree, that sounds like a good idea, thanks! >> If you are interested in helping

Re: [Maria-developers] Incorrect format description event skipping in queue_event()

2013-09-04 Thread Kristian Nielsen
Pavel Ivanov writes: > Thanks. This patch works for me. Ok, I'll push it then. > Note that there's also Query_log_event::peek() that should have the > same dependency on common_header_len. It's hard (if at all possible) > for me to experience that code path in my tests, but for the sake of > ge

Re: [Maria-developers] Rollback causes assertion with Galera 10.0

2013-09-09 Thread Kristian Nielsen
Jan Lindström writes: > #3 0x7f1ff9877eb2 in __GI___assert_fail (assertion=0x109b4ff "empty()", > file=0x109b4d8 "/home/jan/mysql/galera-10.0/sql/log.cc", line=294, function= > 0x10a01e0 "void > binlog_cache_data::reset()") at assert.c:101 > #4 0x009497d4 in binlog_cache_data::rese

Re: [Maria-developers] Rollback causes assertion with Galera 10.0

2013-09-10 Thread Kristian Nielsen
Jan Lindström writes: > (gdb) p cache_log > $1 = {pos_in_file = 11936128518282651045, end_of_file = 18446744073709547520, > (gdb) p *cache_log.pos_in_file > Cannot access memory at address 0xa5a5a5a5a5a5a5a5 Ouch. That looks like memory corruption :-( Seems something has overwritten the start

Re: [Maria-developers] Replicating same server_id problem

2013-09-11 Thread Kristian Nielsen
Pavel Ivanov writes: > Kristian Nielsen writes: >> So perhaps it is better to just say that --replicate-same-server-id does not >> apply to GTID mode at all. Instead, in GTID mode, if we receive an event with >> our own server_id and smaller seq_no than what we alread

Re: [Maria-developers] [Commits] Rev 3810: MDEV-4994: Crash in dynamic column API in file:///home/bell/maria/bzr/work-maria-10.0-MDEV-4994/

2013-09-13 Thread Kristian Nielsen
sa...@askmonty.org writes: > At file:///home/bell/maria/bzr/work-maria-10.0-MDEV-4994/ > > > revno: 3810 > revision-id: sa...@askmonty.org-20130911151436-rjz6pw6fh6rtj83e > parent: s...@mariadb.org-20130904072837-u3zj89fq4xvb4cnp > commi

Re: [Maria-developers] [Commits] Rev 3810: MDEV-4994: Crash in dynamic column API in file:///home/bell/maria/bzr/work-maria-10.0-MDEV-4994/

2013-09-13 Thread Kristian Nielsen
Sorry about the empty reply. Slip of my fingers, Emacs provides very nice facilities for working quickly, and just a bit too quickly in this case ;-) - Kristian. ___ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@l

Re: [Maria-developers] Review of MDEV-4506, parallel replication, part 1

2013-09-13 Thread Kristian Nielsen
Monty, Thanks for the comments so far. I have pushed fixes to 10.0-knielsen, and some specific replies are below. - Kristian. Michael Widenius writes: > + --slave-parallel-threads=# > + If non-zero, number of threads to spawn to apply in > + parallel events on the slave that were group-comm

Re: [Maria-developers] Review of patch for MDEV-4820

2013-09-15 Thread Kristian Nielsen
Pavel Ivanov writes: > 3. Set gtid_binlog_state and gtid_slave_pos to '0-3-10' on S1 and to > '0-1-1' on S2. Try to start slave on S2. Now I get error "slave has > diverged". What gives? It's not diverged, it's just behind. Yes, it is diverged. S1 has binlog state '0-3-10'. This means that the

Re: [Maria-developers] Review of MDEV-4506, parallel replication, part 1

2013-09-17 Thread Kristian Nielsen
Michael Widenius writes: > + if (list->wakeup_subsequent_commits_running) > + { > +mysql_mutex_lock(&list->LOCK_wait_commit); > +list->wakeup_subsequent_commits_running= false; > +mysql_mutex_unlock(&list->LOCK_wait_commit); > > Why do we need the mutex above? I

Re: [Maria-developers] [All] MariaDB's best (download) month ever

2013-09-17 Thread Kristian Nielsen
Colin Charles writes: > It's 17.09 today, and we've hit 50,160 total downloads (this number is > considered company-internal, we don't share this number externally). "company-internal"? What have you been smoking? Colin, you are supposed to be the community guy for crying out lout :-( This is *

Re: [Maria-developers] Galera-10.0 still does not work with rollbacks

2013-09-19 Thread Kristian Nielsen
Jan Lindström writes: > I need help. After merge galera-10.0 with 10.0, rollback asserts. I seem not > to > be able to find the actual reason. I added some extra output on log.cc to dump > out the cache_log data from both trx_cache and stmt_cache, but to me these > values do not really mean anyt

Re: [Maria-developers] Galera-10.0 still does not work with rollbacks

2013-09-20 Thread Kristian Nielsen
Jan Lindström writes: > Yes, I did run using valgrind, while it reported some uninitialized variables, > not really any memory corruptions.  Attached, the log from node0. Similarly, Well, uninitialised variables are exactly what I would expect could lead to things looking like you have shown so

Re: [Maria-developers] Galera-10.0 still does not work with rollbacks

2013-09-20 Thread Kristian Nielsen
Jan Lindström writes: > Correct log file attached. > ==00:00:02:33.708 25289== Use of uninitialised value of size 8 > ==00:00:02:33.708 25289==at 0x5EF865B: _itoa_word (_itoa.c:179) > ==00:00:02:33.708 25289==by 0x5EFCB91: vfprintf (vfprintf.c:1654) > ==00:00:02:33.708 25289==by 0x5F

Re: [Maria-developers] Review of MDEV-4506, parallel replication, part 1

2013-09-23 Thread Kristian Nielsen
Michael Widenius writes: > Kristian> We _do_ need a full memory barrier here (memory barrier is implied > in taking a > Kristian> mutex). Otherwise the compiler or CPU could re-order the setting of > the > Kristian> wakeup_subsequent_commits_running flag with the reads and writes > done in the

Re: [Maria-developers] Review of MDEV-4506, parallel replication, part 1

2013-09-23 Thread Kristian Nielsen
Michael Widenius writes: > What should happen if you kill a replication thread is that > replication should stop for that master. > > Kristian> This needs more thought, I think ... certainly something looks not > right. > > After looking at the full code, I think that the logical way things > sh

Re: [Maria-developers] Galera-10.0 still does not work with rollbacks

2013-09-24 Thread Kristian Nielsen
Jan Lindström writes: > Full unedited log using --valgrind and --mysqld=--debug=+d at location: > https:/ > /www.dropbox.com/s/dcufp7l1cch8dfs/mysql.err.gz Ok, thanks. Hm, this log looks different from what I am used to seeing when I use mysql-test-run.pl --debug. From looking at the mysql-tes

Re: [Maria-developers] Galera-10.0 still does not work with rollbacks

2013-09-24 Thread Kristian Nielsen
tuff for that thread id only. Hope this helps, - Kristian. Kristian Nielsen writes: > Jan Lindström writes: > >> Full unedited log using --valgrind and --mysqld=--debug=+d at location: >> https:/ >> /www.dropbox.com/s/dcufp7l1cch8dfs/mysql.err.gz > > Ok, thank

Re: [Maria-developers] Galera-10.0 still does not work with rollbacks

2013-09-25 Thread Kristian Nielsen
Jan Lindström writes: > static int binlog_savepoint_set(handlerton *hton, THD *thd, void *sv) > { > #ifdef WITH_WSREP >   if (wsrep_emulate_bin_log) DBUG_RETURN(0); > #endif /* WITH_WSREP */ > > wsrep_emulate_bin_log = 1, thus no savepoint is written to bin log Ah, I missed that. Right, then I

Re: [Maria-developers] Galera-10.0 still does not work with rollbacks

2013-09-25 Thread Kristian Nielsen
Jan Lindström writes: > The point I do not understant is at ha_savepoint > > we have again: > > if ((err= ht->savepoint_set(ht, thd, (uchar *)(sv+1)+ht->savepoint_offset))) > > (gdb) p *sv > $3 = {prev = 0xa5a5a5a5a5a5a5a5, name = 0x7f505c007390 "A", length = 1, > ha_list > = 0xa5a5a5a5a5a5a5a5,

Re: [Maria-developers] Review of MDEV-4506, parallel replication, part 1

2013-09-25 Thread Kristian Nielsen
Hi Monty, So as promised, I took a look at the existing code for STOP SLAVE, and came up with some ideas for how to extend this to handle parallel replication. In existing code, STOP SLAVE ends up in terminate_slave_threads(). The interesting part here is the SQL thread; stopping the IO thread sh

Re: [Maria-developers] Galera-10.0 still does not work with rollbacks

2013-09-25 Thread Kristian Nielsen
Kristian Nielsen writes: >> if ((err= ht->savepoint_set(ht, thd, (uchar *)(sv+1)+ht->savepoint_offset))) > Note that it is not "sv+1". It is ((uchar *)sv+1) + ht->savepoint_offset. Sorry, I messed up that explanation. The point is: sv points to a SAVEPOINT object.

Re: [Maria-developers] Review of MDEV-4506, parallel replication, part 1

2013-09-25 Thread Kristian Nielsen
Kristian Nielsen writes: > So we should introduce >rgi->abort_slave, and set it during STOP SLAVE for all queued rgi entries. Actually, we do not need to introduce another flag. We can just check rgi->rli->abort_slave. - Kristian.

Re: [Maria-developers] Galera-10.0 still does not work with rollbacks

2013-09-27 Thread Kristian Nielsen
seppo.jaak...@codership.com writes: > Did I understand it right that the problem is with "ROLLBACK TO > SAVEPOINT...", where the designated savepoint does not exist? Did you No. It is a very simple issue. The beginning of binlog_savepoint_set() reads: #ifdef WITH_WSREP if (wsrep_emula

[Maria-developers] Comments for parallel replication patch (MDEV-4506)

2013-10-07 Thread Kristian Nielsen
Hi Monty, I read through your latest patch for the MDEV-4506 task, parallel replication. Just to re-cap, this is a work-in-progress patch, so these are just some comments along the way that I hope will be useful. > -static bool sql_slave_killed(THD* thd, Relay_log_info* rli) > +static bool sql_s

Re: [Maria-developers] Wrong SQL thread position reporting after IO thread restart

2013-10-11 Thread Kristian Nielsen
Pavel Ivanov writes: > Here's my approach to fixing both of these problems if you are interested. Thanks. My patch does basically the similar thing, though I changed some details (MDEV-5130). - Kristian. ___ Mailing list: https://launchpad.net/~mari

Re: [Maria-developers] Comments for parallel replication patch (MDEV-4506)

2013-10-18 Thread Kristian Nielsen
Michael Widenius writes: >> What about Rotate_log_event::do_update_pos() and >> Stop_log_event::do_update_pos()? >> >> They currently access the flag in rli. I am not sure that is right, as > this >> could be far ahead of the event they are actually executing? > > Aren't the above events handled

[Maria-developers] Preparing MDEV-4506, Parallel replication, for 10.0.5 release

2013-11-01 Thread Kristian Nielsen
Hi Monty, I was asked to prepare MDEV-4506, parallel replication, for release of 10.0.5 on Monday. For this, I did merge to 10.0-base and then to 10.0. I have pushed the results here: lp:~maria-captains/maria/mdev-4506-base # for 10.0-base merge lp:~maria-captains/maria/mdev-4506

Re: [Maria-developers] Preparing MDEV-4506, Parallel replication, for 10.0.5 release

2013-11-03 Thread Kristian Nielsen
Michael Widenius writes: > Did you forget to push the latest changes into mdev-4506-base ? Yes, sorry about that. > If all tests passes in your 10.0-base, please push the changes into > 10.0-base tree. Done. > I have now one failing test in suite/rpl in mdev-4506 that I am trying > to fix bef

Re: [Maria-developers] Semisync plugin incompatibility

2013-11-11 Thread Kristian Nielsen
Pavel Ivanov writes: > We've noticed recently that semisync_master plugin in MariaDB (which > apparently was fully inherited from MySQL) is seriously incompatible > with our understanding of the purpose of semi-sync replication. This > incompatibility was apparently introduced as a fix for > http

Re: [Maria-developers] Semisync plugin incompatibility

2013-11-14 Thread Kristian Nielsen
Pavel Ivanov writes: > So basically my question is: if I prepare a patch that will restore > the original behavior of semi-sync replication (and remove the tests > added for Bug#45672) will that be acceptable for MariaDB? I don't have anything against it, as I said I do not have much opinion on

Re: [Maria-developers] Slave can take a very long time to start replication

2013-11-14 Thread Kristian Nielsen
Pavel Ivanov writes: > start replicating it passes GTID to start from, master finds binlog > file where the earliest GTID is located and then scans through that > file to find the exact binlog position to start sending binlog events > from. If this binlog file is pretty big then scanning can take

Re: [Maria-developers] parallel replication monitoring

2013-11-14 Thread Kristian Nielsen
Giuseppe Maxia writes: > I am looking for: > * how many workers are used; The number of workers created is determined by the value of @@slave_parallel_threads. That many worker threads are in the replication worker thread pool, shared among all multi-source master connections. Workers are sched

Re: [Maria-developers] MDEV-5277 - Ensure that all MySQL 5.6 options are supported by the MariaDB 10.0 server

2013-11-15 Thread Kristian Nielsen
Sergey Vojtovich writes: > * binlog-max-flush-queue-time This is not needed (it is about MySQL 5.6 group commit, MariaDB uses a different implementation. I suppose one could say it is always set to 0). > * binlog-row-image This is not merged to 10.0 yet (as far as I know). It looks like it wou

Re: [Maria-developers] parallel replication monitoring

2013-11-15 Thread Kristian Nielsen
Giuseppe Maxia writes: > * which database they are running is important even if the parallel > replication is not split by schema. As a DBA, I need to know at a glance > where the action is occurring. SHOW PROCESSLIST gives me that information, > but it doesn't tell me the GTID, which is the info

Re: [Maria-developers] MDEV-5277 - Ensure that all MySQL 5.6 options are supported by the MariaDB 10.0 server

2013-11-15 Thread Kristian Nielsen
Sergey Vojtovich writes: >> > * enforce-gtid-consistency >> > * gtid-mode >> >> This is for MySQL 5.6 GTID, which will not be in MariaDB (we have a >> completely >> different GTID implementation). So they should not be implemented, nor should >> we have dummy options for compatibility. > You me

Re: [Maria-developers] Accessing rpl_global_gtid_binlog_state is not consistently protected

2013-11-18 Thread Kristian Nielsen
Pavel Ivanov writes: > sql/log.cc has highly inconsistent pattern of accessing > rpl_global_gtid_binlog_state. In some places it's protected by mutex > rpl_global_gtid_binlog_state.LOCK_binlog_state, in some places it's > protected by global mutex LOCK_rpl_gtid_state, and in some places it's > no

Re: [Maria-developers] MariaDB-10.0-beta sysbench results

2013-11-22 Thread Kristian Nielsen
MARK CALLAGHAN writes: > Resources are finite, but it would be great if MariaDB also did something > about single-thread performance regressions. MySQL has not done/said much Do you have some pointers about where to start with this? I am thinking about which types of queries / benchmarks to chec

[Maria-developers] MDEV-5115 review

2013-12-04 Thread Kristian Nielsen
Hi Serg, Here is my review of your MDEV-5115 patch. First, a general comment: Another way to fix this would be a smaller patch. It would not attempt to implement anything related to the new event types. Instead, it would just have a small bit of code that allows to read the binary format of the

[Maria-developers] Need Review: MDEV-5509: Seconds_behind_master incorrect in parallel replication in http://bazaar.launchpad.net/~maria-captains/maria/10.0

2014-01-08 Thread Kristian Nielsen
Could someone review this patch for MDEV-5509? I am in particular worried that this might change some subtle behaviour of Seconds_Behind_Master in unexpected ways. Of course I tried my best to make sure such could not occur, but the semantics is really tricky and poorly documented, and I do not th

Re: [Maria-developers] benchmarking next steps

2014-01-20 Thread Kristian Nielsen
MARK CALLAGHAN writes: > Are there other goals for single-thread performance regressions? There are > many that have nothing to do with replication code although almost every > single-threaded replication hurts replication. I am planning to do a general in-depth analysis of single-threaded perf

Re: [Maria-developers] [Commits] Rev 3964: MDEV-4816: rpl.rpl_trunc_temp fails in 10.0-serg in file:///home/psergey/dev2/10.0-fix1/

2014-01-20 Thread Kristian Nielsen
Sergey Petrunya writes: > > revno: 3964 > revision-id: pser...@askmonty.org-20140120123657-g1z4g11xmsdiw2dc > committer: Sergey Petrunya > message: > MDEV-4816: rpl.rpl_trunc_temp fails in 10.0-serg > Undo the previous band-aid fix

Re: [Maria-developers] New message from Yoshinori Matsunobu

2014-01-20 Thread Kristian Nielsen
"Facebook" writes: > I read your latest blog post about profile-guided optisation, which is really > interesting. > > I have one question.. What perf options/arguments did you use to get > INST_RETIRED.ANY and ICACHE.MISSES ? perf stat -e cache-misses? I'm not sure > what options give these nu

Re: [Maria-developers] uint6korr optimization

2014-01-23 Thread Kristian Nielsen
Alexey Botchkov writes: > Only question i have with this is if there's a possibility to make the > hf_mi_uint6korr(dest,src) macro a function. > So we can write as usual > dest= hf_mi_uint6korr(src) > > Didn't find how that can nicely be done with the assembler code. Do it like this: static

Re: [Maria-developers] uint6korr optimization

2014-01-23 Thread Kristian Nielsen
Kristian Nielsen writes: > Do it like this: > static inline ulonglong > mi_uint6korr(const void *p) > { > uint32 a= *(uint32 *)p; > uint16 b= *(uint16 *)(4+(char *)p); > ulonglong v= ((ulonglong)a | ((ulonglong)b << 32)) << 16; > asm ("bswap

Re: [Maria-developers] [Commits] Rev 3965: MDEV-5403 - Reduce usage of LOCK_open: tc_count in lp:maria/10.0

2014-01-24 Thread Kristian Nielsen
Sergey Vojtovich writes: > +#ifdef __cplusplus > +template class Atomic_type_triat; > + > + > +template class Atomic_type_triat > +template class Atomic_type_triat > +template > class > Atomic > + T operator++() { return add(1) + 1; } [lots and lots of more template code snipped] I am

Re: [Maria-developers] [Commits] Rev 3965: MDEV-5403 - Reduce usage of LOCK_open: tc_count in lp:maria/10.0

2014-01-24 Thread Kristian Nielsen
I really do not like hiding the type. If you did not think carefully about the type of the underlying variable, you _really_ should not be using something as complex and subtle as an atomic operation. And I _really_ do not like that we would now have two _different_ ways to do atomic operations, o

[Maria-developers] Next steps in improving single-threaded performance

2014-01-24 Thread Kristian Nielsen
I have been analysing CPU bottlenecks in single-threaded sysbench read-only load. I found that icache misses is the main bottleneck, and that profile-guided compiler optimisation (PGO) with GCC gives a large speedup, 25% or more. (More details in my blog posts: http://kristiannielsen.livejour

Re: [Maria-developers] Next steps in improving single-threaded performance

2014-01-27 Thread Kristian Nielsen
Sergey Vojtovich writes: > just out of curiosity: is it possible to find out which functions cause > highest > amount of icache misses? Yes, see the second post, the profiles marked "Icache misses (ICACHE.MISSES), before PGO" and "Icache misses (ICACHE.MISSES), after PGO". These are level 1 cac

Re: [Maria-developers] Next steps in improving single-threaded performance

2014-01-28 Thread Kristian Nielsen
Sergey Vojtovich writes: > Still a questions mostly to educate myself. According to proc mysqld > executable > size is something like: > VmExe: 12228 kB > VmLib:6272 kB > > I assume the above refers to overall instructions. Level 1 instruction cache > size is like 32Kb, rig

Re: [Maria-developers] Next steps in improving single-threaded performance

2014-01-28 Thread Kristian Nielsen
Axel Schwenke writes: > Wow. 25% is a lot. Have you also tried compiling MySQL 5.6 with PGO? No. > Because if that gets the same improvement, we haven't won anything in the > comparison. On the contrary, if the same works for MySQL 5.6 (and it seems likely it will), then we have won double - b

Re: [Maria-developers] buildbot for non ~maria-captains branches on launchpad

2014-01-30 Thread Kristian Nielsen
Daniel Bartholomew writes: > On Wed, Jan 29, 2014 at 4:01 AM, Alexander Barkov wrote: >> Would it be possible to have buildbot watch branches >> which are on Launchpad, but which are outside of >> the /~maria-captains/ directory? > Sure. It can be configured to do so. Branches are defined using

Re: [Maria-developers] MariaDB

2014-01-30 Thread Kristian Nielsen
Otto Kekäläinen writes: > 2013-10-01 Otto Kekäläinen : >> 2013/10/1 Ondřej Surý : >>> - it would be better to repack the tarball and kill the upstream debian/ >>> directory to not mess with our packaging >> >> I hope to merge my work upstream and that is easier if my changes are >> a made on top

Re: [Maria-developers] Extra copying in debian/rules causes troubles

2014-01-30 Thread Kristian Nielsen
Elena Stepanova writes: > ifeq ($(findstring nocheck,$(DEB_BUILD_OPTIONS)),) > # Don't know why the following is necessary... > cp unittest/unit.pl $(builddir)/unittest/ > cp -r mysql-test/* $(builddir)/mysql-test/ > cp -r sql/share/* $(builddir)/sql/share/ >

Re: [Maria-developers] MariaDB build dependencies

2014-02-03 Thread Kristian Nielsen
Otto Kekäläinen writes: > Can anybody tell me how in particular these are supposed to affect the > build process? > - libevent-dev I think libevent was at some point used for the thread pool. But that the current thread pool implementation does not use libevent due to performance reason. So it s

Re: [Maria-developers] MariaDB

2014-02-04 Thread Kristian Nielsen
Otto Kekäläinen writes: > 2014-01-31 Kristian Nielsen : >> Wouldn't it be better to discard the existing debian/ stuff in the upstream >> sources, and work from the native Debian packaging as the base? >> >> The existing stuff has _so_ much legacy and crap, and

Re: [Maria-developers] Extra copying in debian/rules causes troubles

2014-02-04 Thread Kristian Nielsen
Elena Stepanova writes: > I'm ready to do so, but Otto says that he's currently working on the > contents of the debian folder and is getting ready to push it, and > thus I should probably not change anything in there. > On the other hand, it might be nice to have the change in 10.0.8, to > let i

[Maria-developers] GCC native thread local storage (Was: [Maria-discuss] xtradb in 10.0)

2014-02-07 Thread Kristian Nielsen
Laurynas Biveinis writes: > We have another platform-specific addition: thread-local storage > implemented by __thread GCC keyword, which is GNU specific. This is Hm, this is interesting, I was not aware of the __thread feature. This looks to be much more efficient than pthread_getspecific(), wh

Re: [Maria-developers] MariaDB config for Akonadi - suggestions?

2014-02-12 Thread Kristian Nielsen
Sergey Petrunia writes: > * innodb_flush_log_at_trx_commit. Currently, Akonadi uses > innodb_flush_log_at_trx_commit=2 which as far as I understand can cause up to > 1 > second of changes to be lost. Using value of 1 (always flush on commit) > should > be safer, but maybe it would put an unnec

Re: [Maria-developers] Next steps in improving single-threaded performance

2014-02-12 Thread Kristian Nielsen
Axel Schwenke writes: > I propose the following change to make it work with MariaDB releases before > 10.0.4 (and MySQL) that lack the SHUTDOWN statement: Nice, thanks Axel! - Kristian. ___ Mailing list: https://launchpad.net/~maria-developers Post

Re: [Maria-developers] Next steps in improving single-threaded performance

2014-02-13 Thread Kristian Nielsen
Axel Schwenke writes: > Benchmark 1 is good old sysbench OLTP. I tested 10.0.7 vs. 10.0.7-pgo. With > low concurrency there is about 10% win by PGO; however this is completely > reversed at higher concurrency by mutex contention (the test was with > performance schema disabled, so cannot say whic

Re: [Maria-developers] How handle_rpl_parallel_thread rollback partial transaction when STOP SLAVE In parallel replication

2014-02-25 Thread Kristian Nielsen
"nanyi607rao" writes: > hi Kristian, Hi nanyi607rao, >In handle_rpl_parallel_thread(), a worker thread has got a whole >transaction events, but it only apply partial events. this thread wouldl >skip left events when do STOP SLAVE(is that rigth?) ,because >sql_worker_killed() ret

Re: [Maria-developers] How handle_rpl_parallel_thread rollback partial transaction when STOP SLAVE In parallel replication

2014-02-25 Thread Kristian Nielsen
Kristian Nielsen writes: > The idea to fix this isas follows: > > - We record which transactions have started to commit > > - When we do STOP SLAVE we remember the transaction that last started to >commit at the point at which we stopped. > > - In handle_rpl_pa

[Maria-developers] Problem with debug_sync, THD::enter_cond(), and kill ...

2014-02-26 Thread Kristian Nielsen
I got a crash in my parallel replication test case, and this one turned out to be caused by a fundamental problem in debug_sync. I am not sure how to solve it, so I wanted to explain the issue in case others have some ideas. The crash happens in THD::awake() when a thread is killed: mysql_mu

Re: [Maria-developers] Problem with debug_sync, THD::enter_cond(), and kill ...

2014-02-26 Thread Kristian Nielsen
Pavel Ivanov writes: > Looking at the code it looks like most places (including THD::awake) > assume that thd->mysys_var->current_mutex and > thd->mysys_var->current_cond can be changed only when > thd->mysys_var->mutex is locked. debug_sync.cc is breaking this > assumption. I've found that Item_

Re: [Maria-developers] How handle_rpl_parallel_thread rollback partial transaction when STOP SLAVE In parallel replication

2014-02-26 Thread Kristian Nielsen
Kristian Nielsen writes: >> I put my current patch here: >> >> http://knielsen-hq.org/parallel_replication_patch_intermediate.diff > > Hm, that patch is not sufficient, the bug is still there :-( > Sorry about that. At least I did say that the patch was incomplete.

Re: [Maria-developers] MariaDB Master Replication To MySQL

2014-03-03 Thread Kristian Nielsen
> In this particular case, we have someone that wants to use MariaDB 10.0 > master with a MySQL 5.6 slave. But they are running into a problem with GTID. Generally, the recommended setup is to only replicate from old version on master to new (or same) version on slave. Replicating from new master

Re: [Maria-developers] two places maybe need adjusted in rpl_parallel.cc

2014-03-04 Thread Kristian Nielsen
"nanyi607rao" writes: > First is in rpl_parallel_thread::get_rgi() > adjusted like this > ... > else > { > if(!(rgi= new rpl_group_info(rli))) > { > my_error(ER_OUTOFMEMORY, MYF(0), (int)sizeof(*rgi)); > return NULL; > } > rgi->is_parallel_exec = true; > -if ((

Re: [Maria-developers] MariaDB slave_parallel_threads and MySQL slave_parallel_workers

2014-03-05 Thread Kristian Nielsen
Rasmus Johansson writes: > I received a question that I wanted to share here to get some insight on. > > MariaDB 10.0 has a global setting named "slave_parallel_threads" and > MySQL 5.7 has the same parameter named "slave_parallel_workers". > > Should slave_parallel_workers be an alias to slave_p

Re: [Maria-developers] gsoc replication tasks

2014-03-05 Thread Kristian Nielsen
Sergei Golubchik writes: > I'd like to discuss what replication tasks we might want to be > implemented in this year's google summer of code. I don't really have the bandwidth for handling any GSoC at the moment. But here are a bunch of replication tasks that should be suitable for a newcomer, i

Re: [Maria-developers] A problem of DDL apply in parallel replication

2014-03-06 Thread Kristian Nielsen
"nanyi607rao" writes: > Maybe you have encountered this problem and fixed it, that is a DDL > statement Query event has a Gtid event ahead, but has no end_event (like > xid_event, commit or rollback) behind. so in the latest codes of parallel > replication, count_queued_event_groups will increase

Re: [Maria-developers] New MariaDB builds: regular trusty and special kvm-deb-sid-pbuilder-amd and (Re: MariaDB)

2014-03-06 Thread Kristian Nielsen
Otto Kekäläinen writes: > I assumed this meant that Kristian would finish the setup. Kristian, > are you following this? I am. Unfortunately, I am extremely busy at the moment with several tasks. I will try to get something done within a week or so, if possible... meanwhile, if you can make a l

[Maria-developers] Parallel replication benchmarks

2014-03-06 Thread Kristian Nielsen
I've done a set of benchmarks for parallel replication on the same machine I used previously for my group commit benchmarks, http://kristiannielsen.livejournal.com/16382.html The code tested is the newest code in the bzr repository and what will be in 10.0.9 (this is significantly improved from wh

Re: [Maria-developers] Replication consistancy in MaxScale

2014-03-06 Thread Kristian Nielsen
Mark Riddoch writes: > We have been looking for a cheap way to find this out and came up with an > idea for MariaDB 10 that might be good to discuss. If we use the global > transaction ID we can get the current master ID and the current GTID form > each slave and use them to compare and discover

Re: [Maria-developers] two places maybe need adjusted in rpl_parallel.cc

2014-03-07 Thread Kristian Nielsen
"nanyi607rao" writes: > First is in rpl_parallel_thread::get_rgi() > adjusted like this > ... > else > { > if(!(rgi= new rpl_group_info(rli))) > { > my_error(ER_OUTOFMEMORY, MYF(0), (int)sizeof(*rgi)); > return NULL; > } > rgi->is_parallel_exec = true; > -if ((

Re: [Maria-developers] New MariaDB builds: regular trusty and special kvm-deb-sid-pbuilder-amd and (Re: MariaDB)

2014-03-14 Thread Kristian Nielsen
Hi Otto, Ok, so I setup something in Buildbot. Whenever something is pushed to git://github.com/ottok/mariadb-5.5.git, Buildbot will do a build in two VMs: trusty amd64 and sid amd64. You need to configure your repository to send notifications to Buildbot for this to take effect (I tested with m

Re: [Maria-developers] New MariaDB builds: regular trusty and special kvm-deb-sid-pbuilder-amd and (Re: MariaDB)

2014-03-14 Thread Kristian Nielsen
Kristian Nielsen writes: > But that is not going to work in Buildbot, we do not have 9+ GB of memory > available for builds. I think to fix this requires changes in your packaging > files? Or alternatively - do you have a machine available? Then we could set that up as a buildslav

[Maria-developers] Group commit id in mysqlbinlog output

2014-03-14 Thread Kristian Nielsen
There was a question on the output of mysqlbinlog related to group commit. In 10.0, if two transactions group commit together on the master, their GTID event contains a "commit id" - a 64-bit number. This is used by parallel replication; if two transactions have the same commit id, they can be exe

Re: [Maria-developers] New MariaDB builds: regular trusty and special kvm-deb-sid-pbuilder-amd and (Re: MariaDB)

2014-03-14 Thread Kristian Nielsen
Otto Kekäläinen writes: > Change the line > 3902 time git-buildpackage --git-pristine-tar --git-notify=false > --git-pbuilder --git-dist=trusty --git-arch=amd64 > to > 3902 time git-buildpackage --git-pristine-tar --git-notify=false Ok, I did that, we will see how it goes. Buildbot is suffering

Re: [Maria-developers] Group commit id in mysqlbinlog output

2014-03-14 Thread Kristian Nielsen
Robert Hodges writes: > a.) It seems logical that transactions within a group commit should appear > together in the binlog and should be serialized before and after other > transactions in the binlog. Is there *any* way this ordering could be > violated, for example to mix in a non-grouped tran

Re: [Maria-developers] New MariaDB builds: regular trusty and special kvm-deb-sid-pbuilder-amd and (Re: MariaDB)

2014-03-17 Thread Kristian Nielsen
Kristian Nielsen writes: >> 3902 time git-buildpackage --git-pristine-tar --git-notify=false > > Ok, I did that, we will see how it goes. Seems to work now, except that it fails with this: gpg: /tmp/debsign.vybBjDWq/mariadb-5.5_5.5.36-1.dsc: clearsign failed: secret key not

Re: [Maria-developers] redundant compilation of targets in mariadb 10.0

2014-03-17 Thread Kristian Nielsen
Rohit Kalhans writes: > Also if it is relevant in some way the bug i was trying to fix is > MDEV-5873. i have changed some function signatures in the header and > ~5 lines of change in the cc file. Oh, you're looking into that? I do hope you will share any findings :-) I was rather annoyed that

Re: [Maria-developers] [Maria-discuss] MariaDB vs. MySQL error codes (1900+)

2014-03-17 Thread Kristian Nielsen
Felipe Gasper writes: > Hi everyone, > > I notice that MariaDB is using the 1900+ range for its own > error codes; however, MySQL 5.7, at least, has a few errors in that > range as well. > > Are error codes just going to be an area of increasing > conflict between the two, or is there

Re: [Maria-developers] MaxScale as a binlog server

2014-03-18 Thread Kristian Nielsen
Mark Riddoch writes: > we have been approached by a user with a suggestion for a project that we > might do with them. They have a replication environment with a large fan out > (100 slaves per master) and are looking at doing something to cut one the > load on the master. the thought they had wa

Re: [Maria-developers] New MariaDB builds: regular trusty and special kvm-deb-sid-pbuilder-amd and (Re: MariaDB)

2014-03-18 Thread Kristian Nielsen
Otto Kekäläinen writes: > 2014-03-14 12:59 GMT+02:00 Kristian Nielsen : > >> Create a new hook ("Add webhook" button), Make it post to this url: >> >> >> https://buildbot.askmonty.org/buildbot/change_hook/github?project=MariaDB > These are in place, a

Re: [Maria-developers] New MariaDB builds: regular trusty and special kvm-deb-sid-pbuilder-amd and (Re: MariaDB)

2014-03-18 Thread Kristian Nielsen
Otto Kekäläinen writes: > I assume Github.com changed how they query dates from git, as there is > an option for time format output in git. I can't find any time input > format options, so that is not anything we can change AFAIK. The thing is, just today I pushed a test commit to my own repo, a

Re: [Maria-developers] New MariaDB builds: regular trusty and special kvm-deb-sid-pbuilder-amd and (Re: MariaDB)

2014-03-18 Thread Kristian Nielsen
Otto Kekäläinen writes: > You're right. This thing seems to be that in this time format there is > no + time zone, instead it's called just 'Z'. So all commits done > by Danes and Finns will work, but commits by James Page from England > will fail. We can live with that and don't need to do a

Re: [Maria-developers] Transactions behind a failed transaction could commit in parallel replication

2014-03-19 Thread Kristian Nielsen
"nanyi607rao" writes: > if (unlikely(entry->stop_on_error_sub_id <= rgi->wait_commit_sub_id)) > skip_event_group= true; > > this codes can tell latter transactions to skip but can't tell them rollback. > because if a transaction started commiting before a former transaction fai

Re: [Maria-developers] MDEV-5914

2014-03-21 Thread Kristian Nielsen
Jan Lindström writes: > Here is necessary changes. I have run whole mysql-test-run and I did not see > any failures, thus is is mostly safe. I say mostly, > because we really do not have much test cases that would use multimaster setup > and try to get conflicting transactions > generated from lo

Re: [Maria-developers] Transactions behind a failed transaction could commit in parallel replication

2014-03-21 Thread Kristian Nielsen
"nanyi607rao" writes: > if (!rgi->is_error && !skip_event_group) > err= rpt_handle_event(events, rpt); > else > err= thd->wait_for_prior_commit(); > ... ... > finish_event_group(thd, err, event_gtid_sub_id, entry, rgi); > > if the failed t

Re: [Maria-developers] Patch for MDEV 5873

2014-03-25 Thread Kristian Nielsen
Rohit Kalhans writes: > As mentioned earlier i have a patch for MDEV 5873 and i am attaching here > in the bundle of the patch. I have run tests on this patch and all of them > have passed. > Kindly review the patch. Also i am not sure which is the suitable mailing > list for sending patches, s

Re: [Maria-developers] parallel replication performance

2014-03-25 Thread Kristian Nielsen
Axel Schwenke writes: > The poor performance of MySQL with 1 database is expected. However MySQL is > much better with 32 databases. Even better than MariaDB :( I haven't had time to go through all the numbers and details yet, but just a quick check: In this test, are you sure that you are runn

Re: [Maria-developers] parallel replication performance

2014-03-25 Thread Kristian Nielsen
Axel Schwenke writes: >> In this test, are you sure that you are running MariaDB in GTID mode? >> >> CHANGE MASTER TO master_use_gtid=slave_pos > > Yes. This is how the shell code looks: > > mysql -S ${SLAVE_SOCKET} -u root -e "SET GLOBAL gtid_slave_pos=''" > mysql -S ${SLAVE_SOCKET} -u root

Re: [Maria-developers] parallel replication performance

2014-03-26 Thread Kristian Nielsen
Axel Schwenke writes: > The poor performance of MySQL with 1 database is expected. However MySQL is > much better with 32 databases. Even better than MariaDB :( Ok, so I looked closer at the numbers (of which there were quite a few, thanks for doing this work :-). I have some idea of what could

Re: [Maria-developers] Buildbot machines

2014-04-07 Thread Kristian Nielsen
Rasmus Johansson writes: > To sum it up, there seems to be two aspects to it; 1) more machines, i.e. > slaves could/should be added immediately and 2) some development work Right. We have a setup where we can more or less plug in an empty machine with Linux, Buildbot and KVM, and it will downloa

Re: [Maria-developers] Buildbot machines

2014-04-09 Thread Kristian Nielsen
Daniel Bartholomew writes: > Kristian (or others): Any preferences on Linux distro for the box? > Should I go with OpenSuse, which is what is on the terriers, or does > it matter? Heh, I'd prefer anything _but_ Suse, I think we've had only grief from that. But in the end I don't really have any

Re: [Maria-developers] MariaDB 10.x GTID

2014-04-11 Thread Kristian Nielsen
Jan Lindström writes: > some background, lets assume we have two galera clusters that are > geographically distributed (lets say for example that cluster A is on Europe > and cluster B on USA). Inside a cluster all nodes are set up so that server_id > and domain_id are the same and we will use st

Re: [Maria-developers] [GSoC] Accepted student ready to work : )

2014-04-22 Thread Kristian Nielsen
Sergei Golubchik writes: > as for (1) - I think the tables where test failures are recorded is our > own addition, not part of the buildbot. if that's true, then buildbot > documentation won't help much. It is our own addition, but it has been included in upstream Buildbot, including documentati

Re: [Maria-developers] Next steps in improving single-threaded performance

2014-04-22 Thread Kristian Nielsen
Kristian Nielsen writes: > Axel Schwenke writes: > >> Benchmark 1 is good old sysbench OLTP. I tested 10.0.7 vs. 10.0.7-pgo. With >> low concurrency there is about 10% win by PGO; however this is completely >> reversed at higher concurrency by mutex contention (the test

Re: [Maria-developers] a Query_log_event charset bug in parallel replication

2014-04-23 Thread Kristian Nielsen
Hi nanyi607rao, sorry for the delayed response, in part due to the Easter holidays. "nanyi607rao" writes: > If character_set in different Query_log_events changed, worker threads may > apply them with wrong character_set. the codes leading this problem is in > Query_log_event::do_apply_event,

<    2   3   4   5   6   7   8   9   10   11   >