Re: Postal code searching
On 24/04/2012 17:24, Tompkins Neil wrote: How about if I want to only return postal codes that are like W1U 8JE not W13 0SU. Because in this example I have W1 as the postal code and W13 is the other postal code No, you don't. In this example you have W1U as one outbound code and W13 as the other. W1U postcodes are not a subset of W1 postcodes, any more than IP27 postcodes are a subset of IP2 postcodes. The fact that in W1U the district segment is in the form of NA rather than NN doesn't change the fact that it's an indivisible two-character code. So I think the first question has to be, why do you want to get W1 as a particular substring from the postcode W1U 8JE? British postcodes have a structure which is easy for humans to understand, although (unfortunately) rather hard to parse automatically. Essentially, every full postcode contains four elements: Area code: one or two alpha characters, either A or AA District code: one or two alphanumeric characters the first of which is always numeric, either N, NN or NA Sector code: single numeric character, always N Walk code: two alpha characters, always AA It's customary, but not part of the formal specification, to insert whitespace between the District and Sector codes. So, given the postcode WC1H 8EJ, we have: Area: WC District: 1H Sector: 8 Walk: EJ Taken together, the first two sections form the outbound part of the postcode, and the second two form the inbound. (That is, the first two identify the destination sorting depot that the originating depot will send the post to, and the second two are used by the destination depot to make the actual delivery). The reason for mentioning this is that postcodes, having a wide range of possible formats, are not easy to handle with simple substring searches if you're trying to extract outbound codes from a full postcode. It can be done with regular expressions, but you have to be wary of assuming that the space between District and Sector will always be present as, particularly if you're getting data from user input, it might not be. In my own experience (which is quite extensive, as I've done a lot of work with systems, such as online retail, which use postcodes as a key part of the data), I've always found it simpler to pre-process the postcodes prior to inserting them into the database in order to ensure they have a consistent format (eg, inserting a space if none exists). That then makes it easy to select an outbound code, as you can use the space as a boundary. But if you want to be able to go further up the tree and select area codes (eg, distinguishing between EC, WC and W) then it's harder, as you have to account for the fact that some are two characters and some are only one. You can do it with a regular expression, taking everything prior to the first digit, but it's a lot easier in this case to extract the area code prior to inserting the data into the database and store the area code in a separate column. Mark -- Sent from my ZX Spectrum HD http://mark.goodge.co.uk -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/mysql
RE: problems with INNODB tables
Thanks for your answer. I read http://mysql.rjweb.org/doc.php/memory where it tells you to do one thing if using MYIASM tables and another if using INNODB tables. We are using both. Any suggestions? Thanks for any help. Malki Cymbalista Webmaster, Weizmann Institute of Science malki.cymbali...@weizmann.ac.il 08-9343036 -Original Message- From: Rick James [mailto:rja...@yahoo-inc.com] Sent: Monday, April 23, 2012 9:42 PM To: Andrés Tello; Malka Cymbalista Cc: mysql@lists.mysql.com; Shlomit Afgin; Ronen Hayun Subject: RE: problems with INNODB tables Check your memory usage according to http://mysql.rjweb.org/doc.php/memory -Original Message- From: Andrés Tello [mailto:mr.crip...@gmail.com] Sent: Monday, April 23, 2012 9:00 AM To: Malka Cymbalista Cc: mysql@lists.mysql.com; Shlomit Afgin; Ronen Hayun Subject: Re: problems with INNODB tables Weird, I use a lot Innodb, and no issue, I even kill bravely the mysql process with pkill -9 -f mysql Y suppose the way drupal is being programed. PHP open and closes database connections each time a webpage with db access is issued. When a php exceution ends and the apache webserver have fullfilled the http request, again, php memory is freed and connections closed... UNLESS:.. you are using a mem cached db connection, wich I doubt it since drupal doens't requiere one, or using persistent connections, again, I doubt it, because persistante database connections aren't recommended to innodb tables... Mysql server by default can handles 100 conections, if you get to thata limit you need to fine tune the number of connections allowed. show full processlist can give you a better idea of what is going on, connections with the sleep status, are open connections with no currently no transacctions... I never use script based stop, I always use mysqladmin -u root -p -h localhost shutdown which properly tells mysql to flush tables and terminate. I can almost bet that you are using Ubuntu... ubuntu had given me sometimes very hard times because of the edgy code they use to use, ext4 last version, and so on... what can you tell us about that? How much amount of memory you have? How much concurrent apache/php users you have? Can you provide more cuantitive data please? Hardware, php version, distro, kernel... Cheers... To start, 100 process is quite a lot, something isn't fine. Each time On Mon, Apr 23, 2012 at 9:10 AM, Malka Cymbalista malki.cymbali...@weizmann.ac.il wrote: We are running MySQL version 5.0.45 on a Linux machine. Over the past few months we have been having several problems: 1. Our mysql processes have increased the memory used from about .3% per process to 8% per process 2. We sometimes can have over 100 processes running which brings the machine to its knees and we have to stop and start MySQL in order to kill all the processes. We think that maybe the processes are not finishing normally and are just hanging around. 3. The machine is a web server and in the last few months we are moving over to drupal 7 to build our sites and Drupal 7 requires INNODB tables. Sometimes, when we restart MySQL using the commands /etc/init.d/mysql stop and /etc/init.d/mysql start our sites that were built in drupal 7 do not come up. In order for the INNODB tables to work, we have to stop mysql, rename the ibdata1 file, copy it back to ibdata1 and then restart mysql. Otherwise the INNODB tables are not accessable. In the past all our tables were MYIASM. Our problems started as we started using more and more INNODB tables. Is there anything special that has to be done to configure MySQL when using INNODB tables? We clearly have a problem but we have no idea where to start looking. Our error logs don't show anything. If anyone has any suggestions, we will be happy to hear them. We are considering hiring a consultant who is an expert in MySQL. We are in Israel and we are open to suggestions. Thanks for any help. Malki Cymbalista Webmaster, Weizmann Institute of Science malki.cymbali...@weizmann.ac.ilmailto:malki.cymbali...@weizmann.ac.il 08-9343036 -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/mysql
Re: Postal code searching
If nothing else a great intro to the UK postcode. I find this very interesting/useful. Thanks Mark. On Wed, Apr 25, 2012 at 10:14 AM, Mark Goodge m...@good-stuff.co.uk wrote: On 24/04/2012 17:24, Tompkins Neil wrote: How about if I want to only return postal codes that are like W1U 8JE not W13 0SU. Because in this example I have W1 as the postal code and W13 is the other postal code No, you don't. In this example you have W1U as one outbound code and W13 as the other. W1U postcodes are not a subset of W1 postcodes, any more than IP27 postcodes are a subset of IP2 postcodes. The fact that in W1U the district segment is in the form of NA rather than NN doesn't change the fact that it's an indivisible two-character code. So I think the first question has to be, why do you want to get W1 as a particular substring from the postcode W1U 8JE? British postcodes have a structure which is easy for humans to understand, although (unfortunately) rather hard to parse automatically. Essentially, every full postcode contains four elements: Area code: one or two alpha characters, either A or AA District code: one or two alphanumeric characters the first of which is always numeric, either N, NN or NA Sector code: single numeric character, always N Walk code: two alpha characters, always AA It's customary, but not part of the formal specification, to insert whitespace between the District and Sector codes. So, given the postcode WC1H 8EJ, we have: Area: WC District: 1H Sector: 8 Walk: EJ Taken together, the first two sections form the outbound part of the postcode, and the second two form the inbound. (That is, the first two identify the destination sorting depot that the originating depot will send the post to, and the second two are used by the destination depot to make the actual delivery). The reason for mentioning this is that postcodes, having a wide range of possible formats, are not easy to handle with simple substring searches if you're trying to extract outbound codes from a full postcode. It can be done with regular expressions, but you have to be wary of assuming that the space between District and Sector will always be present as, particularly if you're getting data from user input, it might not be. In my own experience (which is quite extensive, as I've done a lot of work with systems, such as online retail, which use postcodes as a key part of the data), I've always found it simpler to pre-process the postcodes prior to inserting them into the database in order to ensure they have a consistent format (eg, inserting a space if none exists). That then makes it easy to select an outbound code, as you can use the space as a boundary. But if you want to be able to go further up the tree and select area codes (eg, distinguishing between EC, WC and W) then it's harder, as you have to account for the fact that some are two characters and some are only one. You can do it with a regular expression, taking everything prior to the first digit, but it's a lot easier in this case to extract the area code prior to inserting the data into the database and store the area code in a separate column. Mark -- Sent from my ZX Spectrum HD http://mark.goodge.co.uk -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/mysql
Re: Postal code searching
Thanks for your very detailed response Mark. Most helpful. On Wed, Apr 25, 2012 at 10:14 AM, Mark Goodge m...@good-stuff.co.uk wrote: On 24/04/2012 17:24, Tompkins Neil wrote: How about if I want to only return postal codes that are like W1U 8JE not W13 0SU. Because in this example I have W1 as the postal code and W13 is the other postal code No, you don't. In this example you have W1U as one outbound code and W13 as the other. W1U postcodes are not a subset of W1 postcodes, any more than IP27 postcodes are a subset of IP2 postcodes. The fact that in W1U the district segment is in the form of NA rather than NN doesn't change the fact that it's an indivisible two-character code. So I think the first question has to be, why do you want to get W1 as a particular substring from the postcode W1U 8JE? British postcodes have a structure which is easy for humans to understand, although (unfortunately) rather hard to parse automatically. Essentially, every full postcode contains four elements: Area code: one or two alpha characters, either A or AA District code: one or two alphanumeric characters the first of which is always numeric, either N, NN or NA Sector code: single numeric character, always N Walk code: two alpha characters, always AA It's customary, but not part of the formal specification, to insert whitespace between the District and Sector codes. So, given the postcode WC1H 8EJ, we have: Area: WC District: 1H Sector: 8 Walk: EJ Taken together, the first two sections form the outbound part of the postcode, and the second two form the inbound. (That is, the first two identify the destination sorting depot that the originating depot will send the post to, and the second two are used by the destination depot to make the actual delivery). The reason for mentioning this is that postcodes, having a wide range of possible formats, are not easy to handle with simple substring searches if you're trying to extract outbound codes from a full postcode. It can be done with regular expressions, but you have to be wary of assuming that the space between District and Sector will always be present as, particularly if you're getting data from user input, it might not be. In my own experience (which is quite extensive, as I've done a lot of work with systems, such as online retail, which use postcodes as a key part of the data), I've always found it simpler to pre-process the postcodes prior to inserting them into the database in order to ensure they have a consistent format (eg, inserting a space if none exists). That then makes it easy to select an outbound code, as you can use the space as a boundary. But if you want to be able to go further up the tree and select area codes (eg, distinguishing between EC, WC and W) then it's harder, as you have to account for the fact that some are two characters and some are only one. You can do it with a regular expression, taking everything prior to the first digit, but it's a lot easier in this case to extract the area code prior to inserting the data into the database and store the area code in a separate column. Mark -- Sent from my ZX Spectrum HD http://mark.goodge.co.uk -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/mysql
RFE: Allow to use version-specific my.cnf files
Hi,
PostgreSQL allows to use version-specific configuration files, which
allows to change some settings only for particular version of DB.
I think a similar enhancement would be nice and usable for
administrators of MySQL as well.
Please, consider the attached patch as a simple proposal. Any comments
are welcome.
Cheers,
Honza
diff -up mysql-5.5.22/mysys/default.c.versionedcnf mysql-5.5.22/mysys/default.c
--- mysql-5.5.22/mysys/default.c.versionedcnf 2012-03-02 20:44:47.0 +0100
+++ mysql-5.5.22/mysys/default.c 2012-04-25 14:51:32.824181063 +0200
@@ -37,6 +37,7 @@
#include m_string.h
#include m_ctype.h
#include my_dir.h
+#include mysql_version.h
#ifdef __WIN__
#include winbase.h
#endif
@@ -94,10 +95,10 @@ static my_bool defaults_already_read= FA
static const char **default_directories = NULL;
#ifdef __WIN__
-static const char *f_extensions[]= { .ini, .cnf, 0 };
+static const char *f_extensions[]= { .ini, .cnf, .ini- MYSQL_SERVER_VERSION, .cnf- MYSQL_SERVER_VERSION, 0 };
#define NEWLINE \r\n
#else
-static const char *f_extensions[]= { .cnf, 0 };
+static const char *f_extensions[]= { .cnf, .cnf- MYSQL_SERVER_VERSION, 0 };
#define NEWLINE \n
#endif
--
MySQL General Mailing List
For list archives: http://lists.mysql.com/mysql
To unsubscribe:http://lists.mysql.com/mysql
Re: problems with INNODB tables
switch to innodb... and use one_file_per_table I use both, but I try to use myisam for cataloges. Innodb and myisam are truly different engines, they do things completely different, for example, with myisam you have parameters to configure the size of the memory for the indexes, and several others, meanwhile most of innodb performase is bound to innodb buffer pools, and with the newerst mysql version, yo u can have several innodb buffer pools lowering your mutex wait a lot... Also you can switch from myisam to innodb quickly, for that type of tasks, I do a mysqldump with tab formatted texts because it gives 2 files per table, 1 file with the sql query to create de database and other, tab delimited file with all the data of that table, to be used with mysqlimport For a properly recommendation, we would need to know much more about the system using the database, some statistics... What is the database used for? On Wed, Apr 25, 2012 at 5:14 AM, Malka Cymbalista malki.cymbali...@weizmann.ac.il wrote: Thanks for your answer. I read http://mysql.rjweb.org/doc.php/memorywhere it tells you to do one thing if using MYIASM tables and another if using INNODB tables. We are using both. Any suggestions? Thanks for any help. Malki Cymbalista Webmaster, Weizmann Institute of Science malki.cymbali...@weizmann.ac.il 08-9343036 -Original Message- From: Rick James [mailto:rja...@yahoo-inc.com] Sent: Monday, April 23, 2012 9:42 PM To: Andrés Tello; Malka Cymbalista Cc: mysql@lists.mysql.com; Shlomit Afgin; Ronen Hayun Subject: RE: problems with INNODB tables Check your memory usage according to http://mysql.rjweb.org/doc.php/memory -Original Message- From: Andrés Tello [mailto:mr.crip...@gmail.com] Sent: Monday, April 23, 2012 9:00 AM To: Malka Cymbalista Cc: mysql@lists.mysql.com; Shlomit Afgin; Ronen Hayun Subject: Re: problems with INNODB tables Weird, I use a lot Innodb, and no issue, I even kill bravely the mysql process with pkill -9 -f mysql Y suppose the way drupal is being programed. PHP open and closes database connections each time a webpage with db access is issued. When a php exceution ends and the apache webserver have fullfilled the http request, again, php memory is freed and connections closed... UNLESS:.. you are using a mem cached db connection, wich I doubt it since drupal doens't requiere one, or using persistent connections, again, I doubt it, because persistante database connections aren't recommended to innodb tables... Mysql server by default can handles 100 conections, if you get to thata limit you need to fine tune the number of connections allowed. show full processlist can give you a better idea of what is going on, connections with the sleep status, are open connections with no currently no transacctions... I never use script based stop, I always use mysqladmin -u root -p -h localhost shutdown which properly tells mysql to flush tables and terminate. I can almost bet that you are using Ubuntu... ubuntu had given me sometimes very hard times because of the edgy code they use to use, ext4 last version, and so on... what can you tell us about that? How much amount of memory you have? How much concurrent apache/php users you have? Can you provide more cuantitive data please? Hardware, php version, distro, kernel... Cheers... To start, 100 process is quite a lot, something isn't fine. Each time On Mon, Apr 23, 2012 at 9:10 AM, Malka Cymbalista malki.cymbali...@weizmann.ac.il wrote: We are running MySQL version 5.0.45 on a Linux machine. Over the past few months we have been having several problems: 1. Our mysql processes have increased the memory used from about .3% per process to 8% per process 2. We sometimes can have over 100 processes running which brings the machine to its knees and we have to stop and start MySQL in order to kill all the processes. We think that maybe the processes are not finishing normally and are just hanging around. 3. The machine is a web server and in the last few months we are moving over to drupal 7 to build our sites and Drupal 7 requires INNODB tables. Sometimes, when we restart MySQL using the commands /etc/init.d/mysql stop and /etc/init.d/mysql start our sites that were built in drupal 7 do not come up. In order for the INNODB tables to work, we have to stop mysql, rename the ibdata1 file, copy it back to ibdata1 and then restart mysql. Otherwise the INNODB tables are not accessable. In the past all our tables were MYIASM. Our problems started as we started using more and more INNODB tables. Is there anything special that has to be done to configure MySQL when using INNODB tables? We clearly have a problem but we have no idea where to start looking. Our error logs don't show anything. If anyone has
Re: RFE: Allow to use version-specific my.cnf files
Reads interesting, but... Why would you need that? I mean... If I run several databases in the same hardware, I use completely diferent paths for evertying, so I can have atomic, clean and specific files for each instance/version of the database I think is much more easy to migrato to another hardware that way, just copy the instance and you are set... On Wed, Apr 25, 2012 at 9:23 AM, Honza Horak hho...@redhat.com wrote: Hi, PostgreSQL allows to use version-specific configuration files, which allows to change some settings only for particular version of DB. I think a similar enhancement would be nice and usable for administrators of MySQL as well. Please, consider the attached patch as a simple proposal. Any comments are welcome. Cheers, Honza -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/mysql
Re: Postal code searching
2012/04/25 10:14 +0100, Mark Goodge On 24/04/2012 17:24, Tompkins Neil wrote: How about if I want to only return postal codes that are like W1U 8JE not W13 0SU. Because in this example I have W1 as the postal code and W13 is the other postal code No, you don't. In this example you have W1U as one outbound code and W13 as the other. W1U postcodes are not a subset of W1 postcodes, any more than IP27 postcodes are a subset of IP2 postcodes. The fact that in W1U the district segment is in the form of NA rather than NN doesn't change the fact that it's an indivisible two-character code. So I think the first question has to be, why do you want to get W1 as a particular substring from the postcode W1U 8JE? British postcodes have a structure which is easy for humans to understand, although (unfortunately) rather hard to parse automatically. Essentially, every full postcode contains four elements: Area code: one or two alpha characters, either A or AA District code: one or two alphanumeric characters the first of which is always numeric, either N, NN or NA Sector code: single numeric character, always N Walk code: two alpha characters, always AA It's customary, but not part of the formal specification, to insert whitespace between the District and Sector codes. So, given the postcode WC1H 8EJ, we have: Area: WC District: 1H Sector: 8 Walk: EJ Taken together, the first two sections form the outbound part of the postcode, and the second two form the inbound. (That is, the first two identify the destination sorting depot that the originating depot will send the post to, and the second two are used by the destination depot to make the actual delivery). The reason for mentioning this is that postcodes, having a wide range of possible formats, are not easy to handle with simple substring searches if you're trying to extract outbound codes from a full postcode. It can be done with regular expressions, but you have to be wary of assuming that the space between District and Sector will always be present as, particularly if you're getting data from user input, it might not be. In my own experience (which is quite extensive, as I've done a lot of work with systems, such as online retail, which use postcodes as a key part of the data), I've always found it simpler to pre-process the postcodes prior to inserting them into the database in order to ensure they have a consistent format (eg, inserting a space if none exists). That then makes it easy to select an outbound code, as you can use the space as a boundary. But if you want to be able to go further up the tree and select area codes (eg, distinguishing between EC, WC and W) then it's harder, as you have to account for the fact that some are two characters and some are only one. You can do it with a regular expression, taking everything prior to the first digit, but it's a lot easier in this case to extract the area code prior to inserting the data into the database and store the area code in a separate column. It seems to me that sector walk taken together always make up three characters; therefore, blanks aside, the outbound part from a good postcode is LEFT(pc, CHAR_LENGTH(pc)-3) , and with REPLACE it is trivial to drop all blanks. If Neil Tompkins wanted only to get the outbound part, that is enough. As for the area, if it is only one or twain characters long, to get that this is enough: LEFT(pc, IF(SUBSTR(pc, 2, 1) '9', 2, 1)) . Extremely crude coding, but if the postcode is right, This much one can do within an SQL function with no regular-expression handling --and MySQL s regular-expression handling yields only 'yes' or 'no'-- , but, of course, if one wishes to verify that it is right, that is another matter. Are there any rules for that, or is the best recourse to get a file of good outbound codes from the post office? As for the string-matching question, matching 'W1' and 'W13' against 'W13 0SU', one rule to consider is that the longest match is the right one. This problem or like is often given in SQL classes: SELECT * FROM pc JOIN shortpc ON LEFT(pc.c, CHAR_LENGTH(shortpc.c)) = shortpc.c WHERE (SELECT MAX(CHAR_LENGTH(shortpc.c)) FROM pc AS a JOIN shortpc ON LEFT(a.c, CHAR_LENGTH(shortpc.c)) = shortpc.c WHERE a.c = pc.c) = LENGTH(shortpc.c) I hope this is not a class problem. And after this I ask, is any outbound code a leading part of any other outbound code? If not, this twist is not needed. I also consider it good design if not. I was glad to see this lesson in British postcodes, something that I never pursued because I had no need of it. -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe:http://lists.mysql.com/mysql
