Re: [Maria-developers] GSoC 2016:Unique indexes for blobs

[Shubham Barai]

Hi Sergei,
   Sorry! I made a mistake. Actually, when I changed my storage
engine to MyISAM and executed  create table statement(with unique blobs and
no prefix length),mysqld crashed. After reconnecting to the server, the
storage engine got changed to default (InnoDB).I didn't know about that and
I executed all queries on InnoDB. I thought my patch was working for MyISAM
but there are some issues.I am working on it but it might require some time
to get it working. Since application deadline for GSoC is 25 March, I will
try to submit my proposal as early as possible.

Thanks,
Shubham

On 19 March 2016 at 14:22, Shubham Barai  wrote:

> Hello Sergei,
>
> As you suggested, I tried  to develop a prototype  just for
> blobs.
> So far it is working fine but I am still verifying  it for different
>  cases.
> I will provide you the link to my Github repository shortly.
>
> Thanks,
> Shubham
>
> On 16 March 2016 at 17:55, Sergei Golubchik  wrote:
>
>> Hi, Shubham!
>>
>> On Mar 16, Shubham Barai wrote:
>> > Hello, Sergie!
>> >  I  tried to explore the source code from
>> > mi_create.c,ha_myisam.cc,sql/sql_table.cc,include/myisam.h,.and some
>> other
>> > files.
>> > The main task is to create  MI_UNIQUEDEF "uniques" for long unique
>> > constraints.
>> > We have to consider all the cases where we need to create MI_UNIQUEDEF
>> > instead of MI_KEYDEF.
>> >
>> > It will include queries like
>> >create table table1 (blob_column blob,unique(blob_column) );
>> >create table table1 (a int,blob_column
>> > blob,unique(a,blob_column) );
>> >create table table1 (a int,blob_column1 blob,blob_column2
>> > blob,unique(blob_column1(300),blob_column2) );
>> > (key with multiple blob columns and one of the blob column specified
>> with
>> > prefix length).
>> >
>> > I think we have to create  MI_UNIQUEDEF if any one of the columns in a
>> key
>> > is a blob field without  prefix length.
>>
>> Yes. And also for any other UNIQUE constraint that is too long for a
>> normal index. For example, many long VARCHAR columns.
>>
>> But, of course, for a prototype one can start just with blobs.
>>
>> > In sql/sql_table, mysql_prepare_create_table is the function which
>> > prepares the table and key structures for table creation in mi_create.
>> > It generates an error if any one of the blob fields in a key is
>> > specified without length.  Currently, this task is limited to MyISAM,
>> > so if any other storage engine is selected, we have to generate the
>> > same error in mysql_prepare_create_table.
>>
>> There's a generic check whether the key is too long. It can be used
>> here, it doesn't depend on the storage engine.
>>
>> > In storage/myisam/ha_myisam.cc, table2myisam is a function which
>> allocates
>> > and initializes myisam key and column definitions.The current function
>> > prototype of table2myisam is
>> >table2myisam(TABLE  *table_arg, MI_KEYDEF  **keydef_out,
>> > MI_COLUMNDEF **recinfo_out, uint records_out)
>> > We have to change it to
>> >table2myisam(TABLE  *table_arg,MI_KEYDEF  **keydef_out,MI_UNIQUEDEF
>> **
>> > uniquedef_out,MI_COLUMNDEF  **recinfo_out,uint records_out)
>>
>> Right.
>>
>> > table2myisam initializes all the key definitions from
>> table_arg->keyinfo.
>> > So we can  set  a new flag  (say uniquedef) in a keyinfo struct in
>> > mysql_prepare_create_table if any one of the key_part consists of blob
>> > field without prefix length.
>>
>> There's a field 'algorithm' already. Because MI_UNIQUEDEF in MyISAM is,
>> basically, an index of hashed column values, it is kind of a hash index.
>> So you can use algorithm=HA_KEY_ALG_HASH to mark such columns. And the
>> user will be able to create these indexes explicitly with
>>
>>   create table table1 (blob_column blob,unique(blob_column) using hash);
>>
>> > Later we can check the flag in table2myisam to see  if we want to
>> > create MI_KEYDEF or MI_UNIQUEDEF.
>> >
>> > Thanks,
>> > Shubham.
>>
>> Very good!
>>
>> Regards,
>> Sergei
>> Chief Architect MariaDB
>> and secur...@mariadb.org
>>
>
>
___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] NO PAD Collation for MariaDB

[Melissa Rose]

Thanks for the kind response Alexander Barkov: I have already build the Server 
from source and I am looking at some character sets in string/*.c files and how 
they are implemented. I have also written and submitted a draft proposal on the 
summerofcode  website.
Please I would appreciate if anyone could go through it and leave comments so 
that I can make any  necessary corrections.

Thanks, 
Melisa.
___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 72b709a: MDEV-9701: CREATE VIEW with GROUP BY or ORDER BY and constant produces invalid definition

[Sergei Golubchik]

Hi, Oleksandr!

On Mar 16, Oleksandr Byelkin wrote:
> > On Mar 16, OleksandrByelkin wrote:
> >> revision-id: 72b709ac7503ae6dd2b5e9049322fefb90b0ebbe 
> >> (mariadb-10.1.12-16-g72b709a)
> >> parent(s): 9b53d84d14a9b031d193f6beae382a232aa738e3
> >> committer: Oleksandr Byelkin
> >> timestamp: 2016-03-16 19:49:17 +0100
> >> message:
> >>
> >> MDEV-9701: CREATE VIEW with GROUP BY or ORDER BY and constant produces 
> >> invalid definition
> >>
> >> Fixed printing integer constant in the ORDER clause (MySQL solution)
> >> Removed workaround for double resolving counter in the ORDER.
> >>
> >> diff --git a/sql/sql_select.cc b/sql/sql_select.cc
> >> index fff24b9..f14e8b1 100644
> >> --- a/sql/sql_select.cc
> >> +++ b/sql/sql_select.cc
> >> @@ -21874,7 +21874,11 @@ find_order_in_list(THD *thd, Item 
> >> **ref_pointer_array, TABLE_LIST *tables,
> >> */
> >> if (order_item->type() == Item::INT_ITEM && 
> >> order_item->basic_const_item())
> >> {  /* Order by position */
> >> -uint count= (uint) order_item->val_int();
> >> +uint count;
> >> +if (order->counter_used)
> > how can this happen?
> we merge view and so its ORDER BY first we resolve it in  view then in 
> outer query. It actually works (I am not sure if I should include also 
> following test:
> 
> create table t1 (a int, b int);
> insert into t1 values (1,2), (2,1);
> create view v1 as select a, b from t1 order by 1;
> select b,a from v1;
> ba
> 21
> 12

Did it work before this your change? If yes - how?

> Actually I can make above even more efficient, just exit in case if 
> resolving already done. Try it?

If you do - do it in a separate commit, please

> >> +  count= order->counter; // counter was once resolved
> >> +else
> >> +  count= (uint) order_item->val_int();
> >>   if (!count || count > fields.elements)
> >>   {
> >> my_error(ER_BAD_FIELD_ERROR, MYF(0),
> >> diff --git a/sql/sql_union.cc b/sql/sql_union.cc
> >> index c835083..8145cec 100644
> >> --- a/sql/sql_union.cc
> >> +++ b/sql/sql_union.cc
> >>   bool st_select_lex::cleanup()
> >>   {
> >> bool error= FALSE;
> >> DBUG_ENTER("st_select_lex::cleanup()");
> >>   
> >> +  cleanup_order(order_list.first);
> >> +  cleanup_order(group_list.first);
> > why is it needed?
> For PS and exactly for PS  with "ORDER BY ?" to make new resolving for 
> re-execution.

Why would one need to redo it? The number will always resolve to the
same Item, won't it? One can safely do it only once, it seems.

Regards,
Sergei
Chief Architect MariaDB
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Maria-discuss] Collations, trailing spaces and unique indexes

[Binarus]

Alexander,

I'd like to continue our conversation from Maria-discuss.

>> - I have found my_strnxfrm_unicode in the same file and will need more time 
>> to make my opinion of how difficult it will be
>> (I don't know what a weight is, so I currently try to understand what the 
>> function does at all).
> 
> This function is used to create sort keys for non-indexed ORDER BY,
> for these cases:
> 
> - ORDER BY on an expression
> - ORDER BY on a column that does not have an index
> 
> 
> The idea is exactly the same with the C function strxfrm.
> See "man strxfrm".
> 
> The code implements non-indexed ORDER BY in filesort.cc
> as follows:
> 
> 1. It calls *_strnxfrm_* functions for all records and converts
> CHAR/VARCHAR/TEXT values into their fixed length binary sortable keys.
> 
> 2 . Then executes binary sorting on these keys.
> 
> By the way, fixing this function might be tricky.
> 
> 
> Currently my_strnxfrm_unicode() pads the tail using weights of the SPACE
> character.
> The NO PAD version will need to pad the tail using a weight which is
> less than the weight of the smallest possible character.
> 
> This should be easy for UCA bases collations (e.g.
> utf8_unicode_nopad_ci), because the smallest possible
> character in UCA collations is "U+0009 HORIZONTAL TABULATION",
> and its weight is 0x0201. So we can just pad the sort key
> using a smaller value 0x0200.
> 
> 
> But I'm not sure yet what to do with 8-bit collations,
> which usually use 0x00 as weight for the smallest character.
> So we don't have a smaller value.
> There are two options here:
> 
> 1. Pad with 0x00. But this will mean that 'aaa' and just 'aaa'
> will  have unpredictable order when doing ORDER BY without an index
> (where  is the smallest possible character in the collation).
> 
> As the smallest character in non-UCA collations is usually
> "U+ NULL", this will mean that 'aaa\0' and just 'aaa'
> will have unpredictable order.
> 
> 2. Reserve extra bytes at the end of the key, to store the true length, so
> - 'aaa\0' will have the key '4141410004'
> - 'aaa'   will have the key '4141410003', and therefore will always
>be sorted before 'aaa\0'.
> 
> I'm inclined towards #2, to have consistent ORDER BY behavior
> with and without indexes.

Wow, thanks for explaining. I now have learned that I probably can't assist you 
in solving that problem (I haven't been in unicode, weights, sorting and 
character sets very deep until now).

But if it helps, you eventually could break the problem into two steps:

1) Fix the problem with unique indexes

2) Fix ordering for any possible situation

The problems we have encountered in real life so far are all related to 1), 
i.e. we would be happy if unique indexes would consider 'a' and 'a ' being 
different values. In contrast, ordering is not that important for us. I don't 
know what others are thinking about that, though.

>>
>> - My main problem: I did not find my_strnncollsp_utf8_general_ci anywhere 
>> (nor in the same neither in any other file). Where is it?
>
> The function name is just "my_strnncollsp_utf8".

OK, thank you very much. I must have misunderstood something.

>> Furthermore, studying the code has led to some questions; for example, there 
>> already seems to be a #define which controls the padding-when-comparing 
>> mode, but only for the _cs collations?
> 
> Can you please clarify which lines do you mean?

I am referring to the file ctype-utf8.c, line 5698. There is:

#ifndef VARCHAR_WITH_DIFF_ENDSPACE_ARE_DIFFERENT_FOR_UNIQUE
  diff_if_only_endspace_difference= 0;
#endif

I haven't been able yet to thoroughly analyze where and how this is used, but 
at least somebody obviously already had some thoughts whether or not trailing 
spaces should make a difference in UNIQUEs :-)

Regards,

Binarus

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] Please review MDEV-9653 Assertion `length || !scale' failed in uint my_decimal_length_to_precision(uint, uint, bool)

[Sergei Golubchik]

Hi, Alexander!

Looks very good!

Just one comment - instead of a special treatment for MYSQL_TYPE_NULL, I
would try to set decimal=0 in Item_null. It already sets max_length=0,
and I cannot think of any reason why we might want decimal to be 31 for it.

On Mar 17, Alexander Barkov wrote:
> Hi Sergei,
> 
> It appeared to be very easy to fix all hybrid functions to use the same
> attribute aggregation code, and therefore get rid of duplicate
> implementations for Item_func_case_abbreviation2 and
> Item_func_case.
> 
> This patch fixes:
> - The original problem reported in MDEV-9653
> 
> - An additional problem, see new comments about COALESCE and IF
>   returning 1.100 instead of just 1.1
> 
> - A new problem that I found:
>   MDEV-9752 Wrong data type for COALEASCE(?,1) in prepared statements
> 
> 
> Note, the patch does not cover LEAST/GREATEST. They have another
> version of their own aggregation code. I don't want to touch them
> in this fix. Let's do them separately.
> 
> Note, I also removed Item_func_case::agg_str_lengths().
> It was a dead code.

Regards,
Sergei
Chief Architect MariaDB
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] Please review MDEV-9653 Assertion `length || !scale' failed in uint my_decimal_length_to_precision(uint, uint, bool)

[Sergei Golubchik]

Hi, Alexander!

On Mar 17, Alexander Barkov wrote:
> > 
> > why is it NOT_FIXED_DEC? it seems to me that it has to be 0 here.
> 
> Perhaps my example in the comments made you think that
> it happens in a very special case with CAST AS UNSIGNED.
> It also happens in simpler cases:
> 
> SELECT CASE WHEN TRUE
> THEN COALESCE(NULL)
> ELSE 40
> END;
> 
> I guess I should fix the example to this ^^^.

Yes, please.

More questions:

1. Perhaps my_decimal_length_to_precision() then? It's used not only in
   Item_func_case::agg_num_lengths. Quick grepping finds more
   potentially dangerous places. For example
   Item_decimal_typecast::print.

2. Why my_decimal_length_to_precision is only used in Item_func_case?
   How do other functions aggregate types? Perhaps CASE needs
   to be changed to work like other similar functions do?

Regards,
Sergei
Chief Architect MariaDB
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] GSoC 2016:Unique indexes for blobs

[Sergei Golubchik]

Hi, Shubham!

On Mar 16, Shubham Barai wrote:
> Hello, Sergie!
>  I  tried to explore the source code from
> mi_create.c,ha_myisam.cc,sql/sql_table.cc,include/myisam.h,.and some other
> files.
> The main task is to create  MI_UNIQUEDEF "uniques" for long unique
> constraints.
> We have to consider all the cases where we need to create MI_UNIQUEDEF
> instead of MI_KEYDEF.
> 
> It will include queries like
>create table table1 (blob_column blob,unique(blob_column) );
>create table table1 (a int,blob_column
> blob,unique(a,blob_column) );
>create table table1 (a int,blob_column1 blob,blob_column2
> blob,unique(blob_column1(300),blob_column2) );
> (key with multiple blob columns and one of the blob column specified with
> prefix length).
> 
> I think we have to create  MI_UNIQUEDEF if any one of the columns in a key
> is a blob field without  prefix length.

Yes. And also for any other UNIQUE constraint that is too long for a
normal index. For example, many long VARCHAR columns.

But, of course, for a prototype one can start just with blobs.

> In sql/sql_table, mysql_prepare_create_table is the function which
> prepares the table and key structures for table creation in mi_create.
> It generates an error if any one of the blob fields in a key is
> specified without length.  Currently, this task is limited to MyISAM,
> so if any other storage engine is selected, we have to generate the
> same error in mysql_prepare_create_table.

There's a generic check whether the key is too long. It can be used
here, it doesn't depend on the storage engine.

> In storage/myisam/ha_myisam.cc, table2myisam is a function which allocates
> and initializes myisam key and column definitions.The current function
> prototype of table2myisam is
>table2myisam(TABLE  *table_arg, MI_KEYDEF  **keydef_out,
> MI_COLUMNDEF **recinfo_out, uint records_out)
> We have to change it to
>table2myisam(TABLE  *table_arg,MI_KEYDEF  **keydef_out,MI_UNIQUEDEF **
> uniquedef_out,MI_COLUMNDEF  **recinfo_out,uint records_out)

Right.

> table2myisam initializes all the key definitions from table_arg->keyinfo.
> So we can  set  a new flag  (say uniquedef) in a keyinfo struct in
> mysql_prepare_create_table if any one of the key_part consists of blob
> field without prefix length.

There's a field 'algorithm' already. Because MI_UNIQUEDEF in MyISAM is,
basically, an index of hashed column values, it is kind of a hash index.
So you can use algorithm=HA_KEY_ALG_HASH to mark such columns. And the
user will be able to create these indexes explicitly with

  create table table1 (blob_column blob,unique(blob_column) using hash);

> Later we can check the flag in table2myisam to see  if we want to
> create MI_KEYDEF or MI_UNIQUEDEF.
> 
> Thanks,
> Shubham.

Very good!

Regards,
Sergei
Chief Architect MariaDB
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] Gsoc 2016 Mdev 371 Unique index for blob

[Sergei Golubchik]

Hi, Sachin!

On Mar 16, Sachin Setia wrote:
> Thanks sir for your reply
> I have done two thing
> 1. First commenting some code to remove the error we get when we try to
> create unique blob
> file=sql/sql_table.cc
> line no=3877

It'd be easier to read, if you'd sent a patch instead.

> /*
> * Gsoc 2016
> * I am implementing this so comment this stuff out
>  */
> //  if (!column->length)
> //  {
> //my_error(ER_BLOB_KEY_WITHOUT_LENGTH, MYF(0), column->field_name.str);
> //DBUG_RETURN(TRUE);
> //  }
> 
> 2. here I am assuming this thing for for just prototyping
> Assumption My table will just contain two column first will be primary key
> second will be unique blob
> So basically what i am doing is create a custom unique key and passing it
> to mi_create
> Of course in real patching i will replace this code with logic like if i
> have m unique blobs (i will find this using key_length ==0)
> create m unique key array and pass it
> 
> file=storage/myisam/ha_myisam.cc
> line no=2067
> //some tweak in share for prototype
> share->keys--;
> share->uniques=1;
> MI_UNIQUEDEF uniquedef;
> MI_KEYDEF keydef_blob=*(keydef+1);
> bzero((char*) ,sizeof(uniquedef));
> uniquedef.keysegs=1;
> uniquedef.seg=keydef_blob.seg;
> uniquedef.null_are_equal=1;
> 
>   /* TODO: Check that the following fn_format is really needed */
>   error= mi_create(fn_format(buff, name, "", "",
>  MY_UNPACK_FILENAME|MY_APPEND_EXT),
>share->keys, keydef,
>record_count, recinfo,
>1, ,
>_info, create_flags);
> 
> If i am doing it wrongly please let me know

Looks fine for a prototype.
Very good!

Regards,
Sergei
Chief Architect MariaDB
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] b1a1e5b: MDEV-6058 MySQL Bug #11766693: LOG-SLOW-ADMIN-STATEMENTS AND LOG-SLOW-SLAVE-STATEMENTS NOT DISPLAYED.

[Sergei Golubchik]

Hi, Alexey!

On Mar 18, Alexey Botchkov wrote:
> revision-id: b1a1e5b3d3d5ab95c596580de07daf38a3efa039 
> (mariadb-10.1.12-19-gb1a1e5b)
> parent(s): 4aac51db9a55a11574cbfa60484d4329d35b6f2c
> committer: Alexey Botchkov
> timestamp: 2016-03-18 11:47:05 +0400
> message:
> 
> MDEV-6058 MySQL Bug #11766693: LOG-SLOW-ADMIN-STATEMENTS AND 
> LOG-SLOW-SLAVE-STATEMENTS NOT DISPLAYED.
> These parameters were moved from the command line options to the
> system variables section. Treatment of the
> opt_log_slow_slave_statements changed to let the dynamic change
> of the variable.
> 
> ---
>  mysql-test/r/variables.result  |  10 ++
>  mysql-test/suite/rpl/r/rpl_slow_query_log.result   |  10 ++
>  mysql-test/suite/rpl/t/rpl_slow_query_log.test |  43 
>  .../r/log_slow_admin_statements_basic.result   |  76 ++
>  .../r/log_slow_admin_statements_func.result|  46 +
>  .../r/log_slow_slave_statements_basic.result   |  76 ++
>  .../t/log_slow_admin_statements_basic.test | 110 
> +
>  .../sys_vars/t/log_slow_admin_statements_func.test |  61 
>  .../t/log_slow_slave_statements_basic.test | 110 
> +
>  mysql-test/t/variables.test|  11 +++

I suppose you didn't run the full test suite. Otherwise you would've
noticed that sysvar_* results were changed.

Also, because we have sysvar_* tests, we no longer create
_basic tests for every new variable, they are no longer useful.
Starting from 10.1 you only need to test the functionality of new
variables, not their basic properties.

>  sql/log_event.cc   |  10 ++
>  sql/mysqld.cc  |   8 --
>  sql/sys_vars.cc|  13 +++
>  13 files changed, 576 insertions(+), 8 deletions(-)
> 
> diff --git a/mysql-test/r/variables.result b/mysql-test/r/variables.result
> index fef3e4a..e46e65f 100644
> --- a/mysql-test/r/variables.result
> +++ b/mysql-test/r/variables.result
> @@ -1797,3 +1797,13 @@ select * from information_schema.session_variables 
> where variable_name='sql_mode
>  VARIABLE_NAMEVARIABLE_VALUE
>  SQL_MODE ANSI_QUOTES
>  End of 5.5 tests
> +show variables like 'log_slow%statements';
> +Variable_nameValue
> +log_slow_admin_statementsOFF
> +log_slow_slave_statementsOFF
> +select * from information_schema.global_variables where variable_name like 
> 'log_slow_admin_statements';
> +VARIABLE_NAMEVARIABLE_VALUE
> +LOG_SLOW_ADMIN_STATEMENTSOFF
> +select * from information_schema.global_variables where variable_name like 
> 'log_slow_slave_statements';
> +VARIABLE_NAMEVARIABLE_VALUE
> +LOG_SLOW_SLAVE_STATEMENTSOFF

You can remove this test.

> diff --git a/mysql-test/suite/rpl/t/rpl_slow_query_log.test 
> b/mysql-test/suite/rpl/t/rpl_slow_query_log.test
> index faf037a..de336fc 100644
> --- a/mysql-test/suite/rpl/t/rpl_slow_query_log.test
> +++ b/mysql-test/suite/rpl/t/rpl_slow_query_log.test
> @@ -299,6 +299,49 @@ if ($master_slow_query == $slave_slow_query)
>-- echo ### same number of queries in slow log: $master_slow_query
>  }
>  
> +-- echo 
> +-- echo  TRUNCATE the slow log then check whether runtime changes of
> +-- echo  log_slow_slave_statements work without slave restart.
> +-- echo 
> +
> +SET @old_log_slow_slave_statements= @@global.log_slow_slave_statements;
> +SET @@global.log_slow_slave_statements = off;
> +TRUNCATE mysql.slow_log;
> +
> +-- connection master
> +
> +--disable_warnings
> +-- eval $slow_query;
> +--enable_warnings
> +sync_slave_with_master;
> +
> +-- connection slave
> +
> +let $slave_slow_query= `SELECT count(*) = 1 FROM mysql.slow_log WHERE 
> sql_text like '$slow_query'`;
> +if ($slave_slow_query)
> +{
> +  SELECT * FROM mysql.slow_log;
> +  die "Assertion failed! Slow query FOUND in slow query log. Bailing out!";
> +}

please, NO!
If you want to test whether a certain row is present in a certain table
you do

  SELECT columns FROM table;

may be with a WHERE clause. But neverif (`SELECT`) { die; }

> +
> +SET @@global.log_slow_slave_statements = on;
> +
> +-- connection master
> +
> +--disable_warnings
> +-- eval $slow_query;
> +--enable_warnings
> +sync_slave_with_master;
> +
> +-- connection slave
> +
> +let $slave_slow_query= `SELECT count(*) = 1 FROM mysql.slow_log WHERE 
> sql_text like '$slow_query'`;
> +if (!$slave_slow_query)
> +{
> +  SELECT * FROM mysql.slow_log;
> +  die "Assertion failed! Slow query NOT FOUND in slow query log. Bailing 
> out!";
> +}

Same.

> +
>  -- connection master
>  SET @@global.log_output= @old_log_output;
>  SET @@global.long_query_time= @old_long_query_time;
> diff --git 
> 

Re: [Maria-developers] bf2eaa7: MDEV-9382: After updating mariadb server apt-configure fails

[Jean Weisbuch]

I forgot to create the PR for upstream as i did for Otto's repo : 
https://github.com/ottok/mariadb-10.0/pull/26 and 
https://github.com/ottok/mariadb-10.0/pull/27
These changes are already in Debian testing packages, should i create 
the PR or you will handle it on your side?


Le 16/03/2016 20:02, Sergei Golubchik a écrit :

This doesn't fix much. Next time somebody will need 61 second and you'll
adjust to 120?  SST might take many minutes and it's not a good idea to
have every user wait this long.

Instead you should do what MDEV-8509 says. Add a line like

   [ -r /etc/default/mariadb ] && . /etc/default/mariadb

then one can have a very personal setting of MYSQLD_STARTUP_TIMEOUT
in /etc/default/mariadb


___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] Please review MDEV-9653 Assertion `length || !scale' failed in uint my_decimal_length_to_precision(uint, uint, bool)

[Alexander Barkov]

Hi Sergei,


On 03/17/2016 01:16 AM, Sergei Golubchik wrote:
> Hi, Alexander!
> 
> On Mar 16, Alexander Barkov wrote:
>> Hello Sergei,
>>
>> Please review a patch for:
>>
>> MDEV-9653 Assertion `length || !scale' failed in uint
>> my_decimal_length_to_precision(uint, uint, bool)
>>
>> Thanks!
> 
>> diff --git a/sql/item_cmpfunc.cc b/sql/item_cmpfunc.cc
>> index 2783a05..73a47ac 100644
>> --- a/sql/item_cmpfunc.cc
>> +++ b/sql/item_cmpfunc.cc
>> @@ -3086,10 +3086,28 @@ void Item_func_case::agg_str_lengths(Item* arg)
>>  
>>  void Item_func_case::agg_num_lengths(Item *arg)
>>  {
>> -  uint len= my_decimal_length_to_precision(arg->max_length, arg->decimals,
>> -   arg->unsigned_flag) - 
>> arg->decimals;
>> -  set_if_bigger(max_length, len); 
>> -  set_if_bigger(decimals, arg->decimals);
>> +  /*
>> +In a query like this:
>> +SELECT CASE WHEN TRUE
>> + THEN COALESCE(CAST(NULL AS UNSIGNED))
>> + ELSE 40
>> +   END;
>> +"arg" corresponding to Item_func_coalesce has:
>> +  arg->max_length==0
>> +  arg->decimals==31 (NOT_FIXED_DEC).
> 
> why is it NOT_FIXED_DEC? it seems to me that it has to be 0 here.

Perhaps my example in the comments made you think that
it happens in a very special case with CAST AS UNSIGNED.
It also happens in simpler cases:

SELECT CASE WHEN TRUE
THEN COALESCE(NULL)
ELSE 40
END;

I guess I should fix the example to this ^^^.


Explicit NULL has STRING_RESULT as cmp_type() and result_type().
So it behaves like strings. It's all around the code.


For COALESCE, IF, IFULL, CASE with explicit NULL input
decimals is set to NOT_FIXED_DEC in here:


bool Item_func::count_string_result_length(enum_field_types field_type_arg,
   Item **items, uint nitems)
{
  if (agg_arg_charsets_for_string_result(collation, items, nitems, 1))
return true;
  if (is_temporal_type(field_type_arg))
count_datetime_length(items, nitems);
  else
  {
decimals= NOT_FIXED_DEC;
count_only_length(items, nitems);
  }
  return false;
}


But the problem is that it happens not only here.
It's also done that way in all string functions:

For example:

MariaDB [test]> SELECT LEFT(null,1);
Field   1:  `LEFT(null,1)`
Type:   VAR_STRING
Collation:  binary (63)
Length: 0
Max_length: 0
Decimals:   31  -- NOT_FIXED_DEC
Flags:  BINARY


MariaDB [test]> SELECT CONCAT(null);
Field   1:  `CONCAT(null)`
Type:   VAR_STRING
Collation:  binary (63)
Length: 0
Max_length: 0
Decimals:   31  -- NOT_FIXED_DEC
Flags:  BINARY

It has always been that way, for years.


I guess other non-function Item types do the same for an explicit NULL
input.


Fixing all Items to have decimals==0 in case
of explicit NULL input looks too dangerous in 10.1.
(btw, in 10.2 it should be easier).


So in 10,1 I decided to fix Item_func_case::agg_num_lengths.

> 
> Regards,
> Sergei
> Chief Architect MariaDB
> and secur...@mariadb.org
> 

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

[Maria-developers] [GSoC 2016] NO PAD Collations

[Даниил Медведев]

Hi, I am Daniil Medvedev from Lomonosov Moscow State University, a 3rd year
student, pursuing my bachelor’s degree in Computer Science.
I was interested in project idea NO PAD Collations, and I wish to
contribute to MariaDB using my skills and ideas.
Technologies that i know:  C/C++, git, linux, shell, cmake, some databases.
My experience:
Some projects on github: https://github.com/medvdanil
And I contributed one open source NoSQL DBMS on github last summer.

Also I am ready to perform some opening task or do commit in your repo, if
it helps me to join GSoC.

Thank you,
Daniil Medvedev
___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] bf2eaa7: MDEV-9382: After updating mariadb server apt-configure fails

[Nirbhay Choubey]

Hi Serg,

On Wed, Mar 16, 2016 at 3:02 PM, Sergei Golubchik  wrote:
.. cut..

> diff --git a/debian/mariadb-server-10.1.mysql.init
> b/debian/mariadb-server-10.1.mysql.init
> > index 9e098b4..005d0e8 100644
> > --- a/debian/mariadb-server-10.1.mysql.init
> > +++ b/debian/mariadb-server-10.1.mysql.init
> > @@ -109,7 +109,7 @@ case "${1:-''}" in
> >   /usr/bin/mysqld_safe "${@:2}" > /dev/null 2>&1 &
> >
> >   # 6s was reported in #352070 to be too little
> > - for i in $(seq 1 "${MYSQLD_STARTUP_TIMEOUT:-30}"); do
> > + for i in $(seq 1 "${MYSQLD_STARTUP_TIMEOUT:-60}"); do
> >  sleep 1
>
> This doesn't fix much. Next time somebody will need 61 second and you'll
> adjust to 120?  SST might take many minutes and it's not a good idea to
> have every user wait this long.
>
> Instead you should do what MDEV-8509 says. Add a line like
>
>   [ -r /etc/default/mariadb ] && . /etc/default/mariadb
>
> then one can have a very personal setting of MYSQLD_STARTUP_TIMEOUT
> in /etc/default/mariadb
>

I was also keen on changing the default 30 secs limit to some higher value
like
60 or 90 secs, so that it works for more installations by default.

I will re-commit one.

Best,
Nirbhay


>
> Regards,
> Sergei
> Chief Architect MariaDB
> and secur...@mariadb.org
>
___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 72b709a: MDEV-9701: CREATE VIEW with GROUP BY or ORDER BY and constant produces invalid definition

[Oleksandr Byelkin]

On 17.03.2016 10:21, Sergei Golubchik wrote:

Hi, Oleksandr!

On Mar 16, Oleksandr Byelkin wrote:

On Mar 16, OleksandrByelkin wrote:

revision-id: 72b709ac7503ae6dd2b5e9049322fefb90b0ebbe 
(mariadb-10.1.12-16-g72b709a)
parent(s): 9b53d84d14a9b031d193f6beae382a232aa738e3
committer: Oleksandr Byelkin
timestamp: 2016-03-16 19:49:17 +0100
message:

MDEV-9701: CREATE VIEW with GROUP BY or ORDER BY and constant produces invalid 
definition

Fixed printing integer constant in the ORDER clause (MySQL solution)
Removed workaround for double resolving counter in the ORDER.

diff --git a/sql/sql_select.cc b/sql/sql_select.cc
index fff24b9..f14e8b1 100644
--- a/sql/sql_select.cc
+++ b/sql/sql_select.cc
@@ -21874,7 +21874,11 @@ find_order_in_list(THD *thd, Item **ref_pointer_array, 
TABLE_LIST *tables,
 */
 if (order_item->type() == Item::INT_ITEM && order_item->basic_const_item())
 {  /* Order by position */
-uint count= (uint) order_item->val_int();
+uint count;
+if (order->counter_used)

how can this happen?

we merge view and so its ORDER BY first we resolve it in  view then in
outer query. It actually works (I am not sure if I should include also
following test:

create table t1 (a int, b int);
insert into t1 values (1,2), (2,1);
create view v1 as select a, b from t1 order by 1;
select b,a from v1;
ba
21
12

Did it work before this your change? If yes - how?
It worked because of IF which avoid printing order number in case of 
printing for VIEW (expression was printed) so not 1 but a (or even 
constant in the case of the bug)





Actually I can make above even more efficient, just exit in case if
resolving already done. Try it?

If you do - do it in a separate commit, please

OK



+  count= order->counter; // counter was once resolved
+else
+  count= (uint) order_item->val_int();
   if (!count || count > fields.elements)
   {
 my_error(ER_BAD_FIELD_ERROR, MYF(0),
diff --git a/sql/sql_union.cc b/sql/sql_union.cc
index c835083..8145cec 100644
--- a/sql/sql_union.cc
+++ b/sql/sql_union.cc
   bool st_select_lex::cleanup()
   {
 bool error= FALSE;
 DBUG_ENTER("st_select_lex::cleanup()");
   
+  cleanup_order(order_list.first);

+  cleanup_order(group_list.first);

why is it needed?

For PS and exactly for PS  with "ORDER BY ?" to make new resolving for
re-execution.

Why would one need to redo it? The number will always resolve to the
same Item, won't it? One can safely do it only once, it seems.
According to the test case we have in case of integer  parameter 'ORDER 
BY ?' interpret as "by which column we will order", so result order 
differs depends on execution parameter.




Regards,
Sergei
Chief Architect MariaDB
and secur...@mariadb.org



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] bf2eaa7: MDEV-9382: After updating mariadb server apt-configure fails

[Sergei Golubchik]

Hi, Nirbhay!

On Mar 02, Nirbhay Choubey wrote:
> revision-id: bf2eaa7c1548f07ce5de13a406ff9183f082b8b3 
> (mariadb-10.1.12-4-gbf2eaa7)
> parent(s): fd741c1a12e78035dcef9a5684b530fe09cbfdde
> author: Nirbhay Choubey
> committer: Nirbhay Choubey
> timestamp: 2016-03-02 09:41:03 -0500
> message:
> 
> MDEV-9382: After updating mariadb server apt-configure fails
> 
> When acting as a Galera receiver node, server startup may take
> more than 30 secs (the current default) as it has to wait for
> SST/IST operation to complete besides spending some time doing
> wsrep recovery.
> 
> In specific case of this MDEV, the server startup timedout as
> it took ~48 secs for the node to fully start.
> 
> A viable workaround would be to adjust the MYSQLD_STARTUP_TIMEOUT
> env to a higher value, but raising the default to 60 secs should
> fix scenarios like these.
> 
> ---
>  debian/mariadb-server-10.1.mysql.init | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/debian/mariadb-server-10.1.mysql.init 
> b/debian/mariadb-server-10.1.mysql.init
> index 9e098b4..005d0e8 100644
> --- a/debian/mariadb-server-10.1.mysql.init
> +++ b/debian/mariadb-server-10.1.mysql.init
> @@ -109,7 +109,7 @@ case "${1:-''}" in
>   /usr/bin/mysqld_safe "${@:2}" > /dev/null 2>&1 &
>  
>   # 6s was reported in #352070 to be too little
> - for i in $(seq 1 "${MYSQLD_STARTUP_TIMEOUT:-30}"); do
> + for i in $(seq 1 "${MYSQLD_STARTUP_TIMEOUT:-60}"); do
>  sleep 1

This doesn't fix much. Next time somebody will need 61 second and you'll
adjust to 120?  SST might take many minutes and it's not a good idea to
have every user wait this long.

Instead you should do what MDEV-8509 says. Add a line like

  [ -r /etc/default/mariadb ] && . /etc/default/mariadb

then one can have a very personal setting of MYSQLD_STARTUP_TIMEOUT
in /etc/default/mariadb

Regards,
Sergei
Chief Architect MariaDB
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] failing tests

[Elena Stepanova]

Hi Alexander,

It's https://jira.mariadb.org/browse/MDEV-9739

Regards,
Elena

On 18.03.2016 15:46, Alexander Barkov wrote:

Hello Olivier,

a few tests are failing in the main 10.1 tree on Linux:

connect.xml
connect.xml_grant
connect.xml_html
connect.xml_mdev5261
connect.xml_mult

Can you please have a look?

Thanks!

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] Gsoc 2016 Mdev 371 Unique index for blob

[Sergei Golubchik]

Hi, Sachin!

On Mar 18, Sachin Setia wrote:
> 
> ERROR 1030 (HY000): Got error 190 "Incompatible key or row definition
> between the MariaDB .frm file and the information in the storage engine.
> You have to dump an" from storage engine MyISAM
> 
> We are getting this becuase in mi_create for each unique_def it creates one
> keydef and writes it.And this creates two  problem
> 1. frm keydef algorithm is different from saved kefdef algorithm(always
> zero) for the time I have solved this problem .
> 
> 2.Mismatch between keydef's keysegs the reason for this is when mi_create
> creates keyseg for unique_def it did not keeps the orignal uniquedef's
> keyseg parameters in mind like language start length which creates problem
> in check_definition function in ha_myisam.cc.I am currently working on it
> Once again sorry for this foolish mistake.
> Regars
> sachin

No problem, everyone makes mistakes :)

It's a prototype, after all. It's much more important that you
understand how the code works and why it doesn't work.

Regards,
Sergei
Chief Architect MariaDB
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] MDEV-9369 IN operator with ( num, NULL ) gives inconsistent result

[Sergei Golubchik]

Hi, Alexander!

On Feb 08, Alexander Barkov wrote:
> Hello Sergei,
> 
> Please review a patch for MDEV-9369.
> 
> Briefly, it fixes cmp_item::cmp() to return three possible values: 
> FALSE, TRUE, UNKNOWN. These values are then recursively pulled to
> the very top level, to the owner Item_xxx.
> 
> The owner Item_func_in (as well as Item_func_case and Item_func_equal)
> then further "translate" these tree-values logic into the traditional
> val_int()+null_value combination using some additional information.
> 
> For example, Item_func_in::val_int() uses "negated" as an additional
> information.
> 
> The patch looks Ok for me.

Looks ok to me too. A couple of stylistic comments, see below.
ok to push afterwards.

> diff --git a/sql/item_cmpfunc.cc b/sql/item_cmpfunc.cc
> index d5f5087..39003de 100644
> --- a/sql/item_cmpfunc.cc
> +++ b/sql/item_cmpfunc.cc
> @@ -3837,15 +3843,15 @@ void cmp_item_decimal::store_value(Item *item)
>/* val may be zero if item is nnull */
>if (val && val != )
>  my_decimal2decimal(val, );
> +  set_null_value(item->null_value);
>  }
>  
>  
>  int cmp_item_decimal::cmp(Item *arg)
>  {
>my_decimal tmp_buf, *tmp= arg->val_decimal(_buf);
> -  if (arg->null_value)
> -return 1;
> -  return my_decimal_cmp(, tmp);
> +  return (m_null_value || arg->null_value) ?

if you access m_null_value directly, you don't need a setter for it.
it'll be less confusing if you replace set_null_value(X) with m_null_value=X.

alternatively, you can add a getter is_null_value(). But I personally
wouldn't do it, cmp_xxx is a small set of simple classes, no need to get
too enterprisey there.

> +UNKNOWN : (my_decimal_cmp(, tmp) != 0);
>  }
>  
>  
> @@ -3892,10 +3900,10 @@ cmp_item *cmp_item_datetime::make_same()
>  }
>  
>  
> -bool Item_func_in::nulls_in_row()
> +bool Item_func_in::list_contains_null()

I found the old name clear enough. But ok, whatever...

>  {
>Item **arg,**arg_end;
> -  for (arg= args+1, arg_end= args+arg_count; arg != arg_end ; arg++)
> +  for (arg= args + 1, arg_end= args+arg_count; arg != arg_end ; arg++)
>{
>  if ((*arg)->null_inside())
>return 1;
> @@ -4010,6 +4018,32 @@ void Item_func_in::fix_length_and_dec()
>  }
>}
>  
> +  /*
> +First conditions for bisection to be possible:
> + 1. All types are similar, and
> + 2. All expressions in  are const
> +  */
> +  bool bisection_possible=
> +type_cnt == 1 &&   // 1
> +const_itm; // 2
> +  if (bisection_possible)
> +  {
> +/*
> +  In the presence of NULLs, the correct result of evaluating this item
> +  must be UNKNOWN or FALSE. To achieve that:
> +  - If type is scalar, we can use bisection and the "have_null" boolean.
> +  - If type is ROW, we will need to scan all of  when
> +searching, so bisection is impossible. Unless:
> +3. UNKNOWN and FALSE are equivalent results
> +4. Neither left expression nor  contain any NULL value
> +  */
> +
> +if (cmp_type == ROW_RESULT &&
> +!((is_top_level_item() && !negated) ||  // 3
> +  (!list_contains_null() && !args[0]->maybe_null))) // 4
> +  bisection_possible= false;

I think the condition will be easier to understand if you remove the
negation of that complex condition in parentheses:

if (cmp_type == ROW_RESULT &&
((!is_top_level_item() || negated) && // 3
  (list_contains_null() || args[0]->maybe_null))) // 4

I find this ^^^ much easier to understand.

> +  }
> +
>if (type_cnt == 1)
>{
>  if (cmp_type == STRING_RESULT && 

Regards,
Sergei
Chief Architect MariaDB
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

[Maria-developers] failing tests

[Alexander Barkov]

Hello Olivier,

a few tests are failing in the main 10.1 tree on Linux:

connect.xml
connect.xml_grant
connect.xml_html
connect.xml_mdev5261
connect.xml_mult

Can you please have a look?

Thanks!

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] Please review MDEV-9653 Assertion `length || !scale' failed in uint my_decimal_length_to_precision(uint, uint, bool)

[Alexander Barkov]

Hi Sergei,

On 03/17/2016 01:14 PM, Sergei Golubchik wrote:
> Hi, Alexander!
> 
> On Mar 17, Alexander Barkov wrote:
>>>
>>> why is it NOT_FIXED_DEC? it seems to me that it has to be 0 here.
>>
>> Perhaps my example in the comments made you think that
>> it happens in a very special case with CAST AS UNSIGNED.
>> It also happens in simpler cases:
>>
>> SELECT CASE WHEN TRUE
>> THEN COALESCE(NULL)
>> ELSE 40
>> END;
>>
>> I guess I should fix the example to this ^^^.
> 
> Yes, please.
> 
> More questions:
> 
> 1. Perhaps my_decimal_length_to_precision() then? It's used not only in
>Item_func_case::agg_num_lengths. Quick grepping finds more
>potentially dangerous places. For example
>Item_decimal_typecast::print.

I could not find places where my_decimal_length_to_precision() can be
called with (0,NOT_FIXED_DEC).

Item_decimal_typecast::print() is safe.
Item_decimal_typecast::max_length and Item_decimal_typecast::decimals
are set to valid values in constructor. The latter cannot be NOT_FIXED_DEC.


> 
> 2. Why my_decimal_length_to_precision is only used in Item_func_case?
>How do other functions aggregate types? Perhaps CASE needs
>to be changed to work like other similar functions do?

There is some code duplication in hybrid type functions:

- Item_func_case::agg_num_lengths()

- Item_func::count_decimal_length(), which is called from
  Item_func_coalesce::fix_length_and_dec()

- Item_func_case_abbreviation2::fix_length_and_dec()

I earlier created a task for this for 10.2:
https://jira.mariadb.org/browse/MDEV-9406


I found new bugs when replying to this letter:

SELECT COALESCE(COALESCE(NULL), 1.1);
SELECT IF(0, COALESCE(NULL), 1.1);

return 1.100, which is wrong.

I'll come up with a new patch soon.

Thanks.


> 
> Regards,
> Sergei
> Chief Architect MariaDB
> and secur...@mariadb.org
> 

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] GSoC 2016 CONC-125

[Christian Massey]

Hi Oleksandr, MariaDB Mentors,

I have submitted a draft proposal for this project on the summer of code
website. Please kindly review it and leave comments so I can make any
necessary corrections.

Cheers,
Christian

2016-03-06 15:53 GMT+01:00 Oleksandr Byelkin :

> Hi, Christian!
>
> On 06.03.2016 12:23, Christian Massey wrote:
>
>> Hi everyone,
>>
>> I am Christian Massey a computer science student at York University and I
>> wish to work on Providing import and export functions for popular data
>> formats like JSON, XML , PHP, ... for Connector/C and MariaDB Server. I
>> have been inspecting the code to understand how the dynamic column packed
>> string is encoded and I see kind on big endian encoding( Correct me if I am
>> wrong please).
>>
>> Also I see that the old format already work like arrays and it uses 3
>> bits for encoding type which is the main reason why not all types  in the
>> enum are supported in this old format.
>>
>>  DBUG_ASSERT(((type - 1) & (~7)) == 0); /* fit in 3 bits */
>>
>> (Again please correct me if I am wrong. The encoding is not easy to
>> understand)
>>
>
> You are right.
>
>
>> My Question is do we need to create a completely new type to support
>> arrays or do we change the old format which already work like arrays in a
>> way but lacks support for some types(I don't know which of the enums yet).
>>
>
> How it is done internally not so important. But "externally" it should be
> new type because:
> 1) we should read old types
> 2) we can't use old "as is" because it lack types
>
> But "internally" code can be of course reused.
>
>
>> Please, I wish to write a proposal for this project and hope to discuss
>> more with mentors for more clarification on how we are going to implement
>> this as it seems to be a very important if not the most critical part of
>> this project.
>>
>>
>> Cheers,
>> Kris.
>>
>>
>> ___
>> Mailing list: https://launchpad.net/~maria-developers
>> Post to : maria-developers@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~maria-developers
>> More help   : https://help.launchpad.net/ListHelp
>>
>
>
___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [GSoC 2016] NO PAD Collations

[Sergei Golubchik]

Hi, Daniil!

On Mar 16, Даниил Медведев wrote:
> Hi, I am Daniil Medvedev from Lomonosov Moscow State University, a 3rd year
> student, pursuing my bachelor’s degree in Computer Science.
> I was interested in project idea NO PAD Collations, and I wish to
> contribute to MariaDB using my skills and ideas.
> Technologies that i know:  C/C++, git, linux, shell, cmake, some databases.
> My experience:
> Some projects on github: https://github.com/medvdanil
> And I contributed one open source NoSQL DBMS on github last summer.
> 
> Also I am ready to perform some opening task or do commit in your repo, if
> it helps me to join GSoC.

That's up to you. We don't require students to do commits or even look
at the code until after they are accepted.

But, of course, if you're familiar with the code you might be able to
write a much better proposal and have better chances of being accepted
:)

Regards,
Sergei
Chief Architect MariaDB
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] 1e542ec: MDEV-9382: After updating mariadb server apt-configure fails

[Sergei Golubchik]

Hi, Nirbhay!

Ok to push.
And, please, check whether /etc/default/mariadb needs to be mentioned
in the corresponding (mariadb on debian/ubuntu) knowledgebase pages.

On Mar 16, Nirbhay Choubey wrote:
> revision-id: 1e542ecd8af6f78db767a9b3da016b38d1ec3970 
> (mariadb-10.1.12-6-g1e542ec)
> parent(s): b3bde93149e7135507fd7abbca2786d77552953d
> author: Nirbhay Choubey
> committer: Nirbhay Choubey
> timestamp: 2016-03-16 19:39:19 -0400
> message:
> 
> MDEV-9382: After updating mariadb server apt-configure fails
> 
> When acting as a Galera receiver node, server startup may take
> more than 30 secs (the current default) as it has to wait for
> SST/IST operation to complete besides spending some time doing
> wsrep recovery.
> 
> Fixed by raising the default value of MYSQLD_STARTUP_TIMEOUT
> to 60 secs. Also sourced /etc/default/mariadb into the init
> script so that it can be used to set MYSQLD_STARTUP_TIMEOUT.
> 
Regards,
Sergei
Chief Architect MariaDB
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] GSoC 2016:Unique indexes for blobs

[Shubham Barai]

Hello Sergei,

As you suggested, I tried  to develop a prototype  just for
blobs.
So far it is working fine but I am still verifying  it for different  cases.
I will provide you the link to my Github repository shortly.

Thanks,
Shubham

On 16 March 2016 at 17:55, Sergei Golubchik  wrote:

> Hi, Shubham!
>
> On Mar 16, Shubham Barai wrote:
> > Hello, Sergie!
> >  I  tried to explore the source code from
> > mi_create.c,ha_myisam.cc,sql/sql_table.cc,include/myisam.h,.and some
> other
> > files.
> > The main task is to create  MI_UNIQUEDEF "uniques" for long unique
> > constraints.
> > We have to consider all the cases where we need to create MI_UNIQUEDEF
> > instead of MI_KEYDEF.
> >
> > It will include queries like
> >create table table1 (blob_column blob,unique(blob_column) );
> >create table table1 (a int,blob_column
> > blob,unique(a,blob_column) );
> >create table table1 (a int,blob_column1 blob,blob_column2
> > blob,unique(blob_column1(300),blob_column2) );
> > (key with multiple blob columns and one of the blob column specified with
> > prefix length).
> >
> > I think we have to create  MI_UNIQUEDEF if any one of the columns in a
> key
> > is a blob field without  prefix length.
>
> Yes. And also for any other UNIQUE constraint that is too long for a
> normal index. For example, many long VARCHAR columns.
>
> But, of course, for a prototype one can start just with blobs.
>
> > In sql/sql_table, mysql_prepare_create_table is the function which
> > prepares the table and key structures for table creation in mi_create.
> > It generates an error if any one of the blob fields in a key is
> > specified without length.  Currently, this task is limited to MyISAM,
> > so if any other storage engine is selected, we have to generate the
> > same error in mysql_prepare_create_table.
>
> There's a generic check whether the key is too long. It can be used
> here, it doesn't depend on the storage engine.
>
> > In storage/myisam/ha_myisam.cc, table2myisam is a function which
> allocates
> > and initializes myisam key and column definitions.The current function
> > prototype of table2myisam is
> >table2myisam(TABLE  *table_arg, MI_KEYDEF  **keydef_out,
> > MI_COLUMNDEF **recinfo_out, uint records_out)
> > We have to change it to
> >table2myisam(TABLE  *table_arg,MI_KEYDEF  **keydef_out,MI_UNIQUEDEF **
> > uniquedef_out,MI_COLUMNDEF  **recinfo_out,uint records_out)
>
> Right.
>
> > table2myisam initializes all the key definitions from table_arg->keyinfo.
> > So we can  set  a new flag  (say uniquedef) in a keyinfo struct in
> > mysql_prepare_create_table if any one of the key_part consists of blob
> > field without prefix length.
>
> There's a field 'algorithm' already. Because MI_UNIQUEDEF in MyISAM is,
> basically, an index of hashed column values, it is kind of a hash index.
> So you can use algorithm=HA_KEY_ALG_HASH to mark such columns. And the
> user will be able to create these indexes explicitly with
>
>   create table table1 (blob_column blob,unique(blob_column) using hash);
>
> > Later we can check the flag in table2myisam to see  if we want to
> > create MI_KEYDEF or MI_UNIQUEDEF.
> >
> > Thanks,
> > Shubham.
>
> Very good!
>
> Regards,
> Sergei
> Chief Architect MariaDB
> and secur...@mariadb.org
>
___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

[Maria-developers] Randomness in TokuDB builds

[Otto Kekäläinen]

Hello TokuDB experts!

There is some randomness, maybe some unique build id in every TokuDB
build that breaks the reproducable builds of it in Debian.

Check out:
https://tests.reproducible-builds.org/dbd/unstable/amd64/mariadb-10.0_10.0.24-5.diffoscope.html

I can't find any mentions of a build id or similar in the sources. Can
somebody help me figure out where the randomness stems from, and how I
should patch the source to make the build deterministic?

- Otto

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

[Maria-developers] Please review MDEV-9653 Assertion `length || !scale' failed in uint my_decimal_length_to_precision(uint, uint, bool)

[Alexander Barkov]

Hello Sergei,

Please review a patch for:

MDEV-9653 Assertion `length || !scale' failed in uint
my_decimal_length_to_precision(uint, uint, bool)

Thanks!
diff --git a/mysql-test/r/func_hybrid_type.result b/mysql-test/r/func_hybrid_type.result
index 95a8a82..29f9f46 100644
--- a/mysql-test/r/func_hybrid_type.result
+++ b/mysql-test/r/func_hybrid_type.result
@@ -3396,5 +3396,17 @@ c1
 DROP TABLE t2;
 DROP TABLE t1;
 #
+# MDEV-9653 Assertion `length || !scale' failed in uint my_decimal_length_to_precision(uint, uint, bool)
+#
+SELECT CASE 0 WHEN 1 THEN (CASE 2 WHEN 3 THEN NULL END) WHEN 4 THEN 5 END;
+CASE 0 WHEN 1 THEN (CASE 2 WHEN 3 THEN NULL END) WHEN 4 THEN 5 END
+NULL
+SELECT CASE 0 WHEN 1 THEN (COALESCE(NULL)) WHEN 4 THEN 5 END;
+CASE 0 WHEN 1 THEN (COALESCE(NULL)) WHEN 4 THEN 5 END
+NULL
+SELECT CASE WHEN TRUE THEN COALESCE(NULL) ELSE 4 END;
+CASE WHEN TRUE THEN COALESCE(NULL) ELSE 4 END
+NULL
+#
 # End of 10.1 tests
 #
diff --git a/mysql-test/t/func_hybrid_type.test b/mysql-test/t/func_hybrid_type.test
index 047e5f7..cc5d067 100644
--- a/mysql-test/t/func_hybrid_type.test
+++ b/mysql-test/t/func_hybrid_type.test
@@ -434,5 +434,13 @@ DROP TABLE t1;
 
 
 --echo #
+--echo # MDEV-9653 Assertion `length || !scale' failed in uint my_decimal_length_to_precision(uint, uint, bool)
+--echo #
+SELECT CASE 0 WHEN 1 THEN (CASE 2 WHEN 3 THEN NULL END) WHEN 4 THEN 5 END;
+SELECT CASE 0 WHEN 1 THEN (COALESCE(NULL)) WHEN 4 THEN 5 END;
+SELECT CASE WHEN TRUE THEN COALESCE(NULL) ELSE 4 END;
+
+
+--echo #
 --echo # End of 10.1 tests
 --echo #
diff --git a/sql/item_cmpfunc.cc b/sql/item_cmpfunc.cc
index 2783a05..73a47ac 100644
--- a/sql/item_cmpfunc.cc
+++ b/sql/item_cmpfunc.cc
@@ -3086,10 +3086,28 @@ void Item_func_case::agg_str_lengths(Item* arg)
 
 void Item_func_case::agg_num_lengths(Item *arg)
 {
-  uint len= my_decimal_length_to_precision(arg->max_length, arg->decimals,
-   arg->unsigned_flag) - arg->decimals;
-  set_if_bigger(max_length, len); 
-  set_if_bigger(decimals, arg->decimals);
+  /*
+In a query like this:
+SELECT CASE WHEN TRUE
+ THEN COALESCE(CAST(NULL AS UNSIGNED))
+ ELSE 40
+   END;
+"arg" corresponding to Item_func_coalesce has:
+  arg->max_length==0
+  arg->decimals==31 (NOT_FIXED_DEC).
+my_decimal_length_to_precision() does not expect this kind of input
+and would crash on DBUG_ASSERT. See MDEV-9653.
+So let's skip length calculation for explicit NULLs and
+for expressions derived from explicit NULLs.
+  */
+  if (arg->field_type() != MYSQL_TYPE_NULL)
+  {
+uint len= my_decimal_length_to_precision(arg->max_length, arg->decimals,
+ arg->unsigned_flag) -
+ arg->decimals;
+set_if_bigger(max_length, len);
+set_if_bigger(decimals, arg->decimals);
+  }
   unsigned_flag= unsigned_flag && arg->unsigned_flag; 
 }
 
___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Maria-discuss] Collations, trailing spaces and unique indexes

[Alexander Barkov]

Hi Binarus,


On 03/16/2016 07:02 PM, Binarus wrote:
> Alexander,
> 
> I'd like to continue our conversation from Maria-discuss.

Sure.

> 
>>> - I have found my_strnxfrm_unicode in the same file and will need more time 
>>> to make my opinion of how difficult it will be
>>> (I don't know what a weight is, so I currently try to understand what the 
>>> function does at all).
>>
>> This function is used to create sort keys for non-indexed ORDER BY,
>> for these cases:
>>
>> - ORDER BY on an expression
>> - ORDER BY on a column that does not have an index
>>
>>
>> The idea is exactly the same with the C function strxfrm.
>> See "man strxfrm".
>>
>> The code implements non-indexed ORDER BY in filesort.cc
>> as follows:
>>
>> 1. It calls *_strnxfrm_* functions for all records and converts
>> CHAR/VARCHAR/TEXT values into their fixed length binary sortable keys.
>>
>> 2 . Then executes binary sorting on these keys.
>>
>> By the way, fixing this function might be tricky.
>>
>>
>> Currently my_strnxfrm_unicode() pads the tail using weights of the SPACE
>> character.
>> The NO PAD version will need to pad the tail using a weight which is
>> less than the weight of the smallest possible character.
>>
>> This should be easy for UCA bases collations (e.g.
>> utf8_unicode_nopad_ci), because the smallest possible
>> character in UCA collations is "U+0009 HORIZONTAL TABULATION",
>> and its weight is 0x0201. So we can just pad the sort key
>> using a smaller value 0x0200.
>>
>>
>> But I'm not sure yet what to do with 8-bit collations,
>> which usually use 0x00 as weight for the smallest character.
>> So we don't have a smaller value.
>> There are two options here:
>>
>> 1. Pad with 0x00. But this will mean that 'aaa' and just 'aaa'
>> will  have unpredictable order when doing ORDER BY without an index
>> (where  is the smallest possible character in the collation).
>>
>> As the smallest character in non-UCA collations is usually
>> "U+ NULL", this will mean that 'aaa\0' and just 'aaa'
>> will have unpredictable order.
>>
>> 2. Reserve extra bytes at the end of the key, to store the true length, so
>> - 'aaa\0' will have the key '4141410004'
>> - 'aaa'   will have the key '4141410003', and therefore will always
>>be sorted before 'aaa\0'.
>>
>> I'm inclined towards #2, to have consistent ORDER BY behavior
>> with and without indexes.
> 
> Wow, thanks for explaining. I now have learned that I probably can't assist 
> you in solving that problem (I haven't been in unicode, weights, sorting and 
> character sets very deep until now).
> 
> But if it helps, you eventually could break the problem into two steps:
> 
> 1) Fix the problem with unique indexes
> 
> 2) Fix ordering for any possible situation
> 
> The problems we have encountered in real life so far are all related to 1), 
> i.e. we would be happy if unique indexes would consider 'a' and 'a ' being 
> different values. In contrast, ordering is not that important for us. I don't 
> know what others are thinking about that, though.
> 

Thanks for the explanation. That makes understand your needs better.

When we add new collations into the main tree, I prefer to have
both steps already addressed, to have consistent collation behavior.
But you're right that in a development tree it can be done in two
separate steps, for simplicity.


>>>
>>> - My main problem: I did not find my_strnncollsp_utf8_general_ci anywhere 
>>> (nor in the same neither in any other file). Where is it?
>>
>> The function name is just "my_strnncollsp_utf8".
> 
> OK, thank you very much. I must have misunderstood something.
> 
>>> Furthermore, studying the code has led to some questions; for example, 
>>> there already seems to be a #define which controls the 
>>> padding-when-comparing mode, but only for the _cs collations?
>>
>> Can you please clarify which lines do you mean?
> 
> I am referring to the file ctype-utf8.c, line 5698. There is:
> 
> #ifndef VARCHAR_WITH_DIFF_ENDSPACE_ARE_DIFFERENT_FOR_UNIQUE
>   diff_if_only_endspace_difference= 0;
> #endif
> 
> I haven't been able yet to thoroughly analyze where and how this is used, but 
> at least somebody obviously already had some thoughts whether or not trailing 
> spaces should make a difference in UNIQUEs :-)


Thanks. This is a kind of dead code. You correctly guessed that it was
supposed to switch between NO PAD and PAD SPACE, but the exact behavior
was supposed to be chosen and compile time. Then we understood that
we want NO PAD and PAD SPACE collations to co-exist, so this approach
with conditional compilation appeared not to be a way to go.

So this code has never been used in the NO PAD mode actually.
We should refactor it when working on NO PAD collations,
by removing #ifndef and #else and moving the relevant pieces into
different functions, for PAD SPACE and NO PAD handlers.


> 
> Regards,
> 
> Binarus
> 
> ___
> Mailing list: https://launchpad.net/~maria-developers
> Post to 

Re: [Maria-developers] [Maria-discuss] Collations, trailing spaces and unique indexes

[Alexander Barkov]

Hi Binarus,

On 03/16/2016 06:13 PM, Binarus wrote:
>> I just wanted to split the task into these steps:
>>
>> 1. Default and _bin collations.
>>This is what MDEV-9711 is about.
>>
>> 2. Unicode Collation Algorithm (UCA) based collations
>>(e.g. utf8_unicode_nopad_ci)
>>
>>
>> 3. The rest (not covered by the previous steps)
>>This will include latin1_general_nopad_cs, latin2_czech_nopad_cs
>>
>>
>> This is why asked in the previous letter if utf8_unicode_nopad_ci would
>> work for you.
>> Also, which MariaDB version do you need this for? Will 10.1 work?
>>
> 
> Thank you very much for this explanation.
> 
> Regarding the version: Our O/S usually is Debian which generally uses 
> software which is quite old. This means that I will have to compile MariaDB 
> myself in every case.
> The current project where this new collations are needed will probably take 
> about three months before being finished. It would be very nice if we could 
> have the new collations until then.
> In the meantime, we'll continue development with MySQL and switch to MariaDB 
> as soon as the new collations are available.

Excellent. The fact that you compile from sources makes the things
easier. I guess we can add new collations officially into 10.2.

Then after 3 month if you'll feel that 10.2 is not mature enough
for you, we can make a custom non-official 10.1 tree with the new
collations back-ported (it should be trivial to do).

> 
> ci collations would be sufficient as a starting point. Nevertheless, I think 
> there must be a nopad collation for every collation which already exists.
> Personally, since we are using utf8 (and no other encoding) in every part of 
> every project, I would prefer to have *all utf8* nopad collations before
>having any other nopad collation. But that's just my personal opinion.
>We will be grateful for any progress regardless of the order of things
being implemented.

Thanks. Then we're flexible to implement #2 first, and wait for #1
to be done under terms of Google Summer Of Code 2016.


> 
> Binarus
> 
> ___
> Mailing list: https://launchpad.net/~maria-discuss
> Post to : maria-disc...@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~maria-discuss
> More help   : https://help.launchpad.net/ListHelp
> 

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] Aggregate stored functions [MDEV-7773]

[Sergei Golubchik]

Hi, Varun!

On Mar 19, Varun Gupta wrote:
> Hi Sergei,
> I have gone through the code you suggested. I have got a good understanding
> of what the code actually does. So now I wanted to start writing the
> proposal for the idea. Can you tell me what all is needed to written in the
> proposal. I am asking for a template or something that could guide me to
> write the proposal.

We don't have a template, but other organizations too, google "gsoc
student template". Also there are many tutorials and guides about
writing a proposal. And, of course, there's the official GSoC student manual:
http://write.flossmanuals.net/gsocstudentguide/what-is-google-summer-of-code/
(it's also linked from the main GSoC site, link "HELP" at the bottom).

Regards,
Sergei
Chief Architect MariaDB
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp