The reality we have to live in is that decisions about software for large projects are made several years prior to turning the system on line, so inevitably many guesses have to be made based on what is known today. Although (obviously) no one knows how the database world will look like in 2012, your answers did give me useful hints to consider/think about. Thank you all who responded.
Jacek
Rhino wrote:
I hope that the original poster notes that none of the comments in this thread actually answer the question about how scaleable MySQL will be in 2012. No one has talked about that for one simple reason: no one knows.
Frankly, I think the entire question was naive from the start. I can't think of any software developers who announce their plans for 8 or 10 years down the line. Software is a very competitive and ever-evolving thing. Changes in software, like increased scalability, are driven primarily by a few factors: a) Customer demand - if no customers are switching to other databases because they have greater scalability, MySQL has relatively little incentive to invest time and money on improving scalability unless perhaps they want to do it for prestige ("bragging rights"). b) Hardware innovation - the advent of new and faster devices and networks may make scalability easier to accomplish at a price that is attractive to customers.
While we can guess at what hardware innovations are coming along by reading announcements about new research findings, some research in hardware is proprietary and is therefore not necessarily published. That means it is difficult to know about some new innovations until someone has released a device that uses the new innovation; then, there may be months or years of delay before competitors can acquire that innovation (or make their own version of it) and modify their software to use it. Even publicized research is not necessarily readily available to software developers: even if the new Fizzbin UltraHardDrive is very hot, it might be so expensive that very few people will ever buy one and therefore it isn't economic to write software for it because the installed base of these things will be tiny until the price of the device is cheaper. Or maybe the Fizzbin drive will be used by elite customers who will spare no expense to achieve scalability and MySQL will write a special version that addresses that hardware.
As for customer demand, marketing gurus study this subject endlessly but don't really know what customers will do or want in the future; they can only offer guesses.
MySQL has a "to do" list in the MySQL manual but the items in that list are mostly things that they plan to work on in the next year or two. I doubt many of the developers are thinking much about the shape of MySQL in 2012 yet. There may be an architect or chief developer of some kind who is fantasizing about major increases in scalability somewhere down the road but I'd bet that those plans are vague at best and certainly nothing that would be quantifiable. Even if they were, I doubt that MySQL would announce specific plans this far ahead of time because it would give their competitors too much information that the competitors might use against them. Besides, if they did make such an announcement, they would have to phrase it as either: 1) a solid committment, in which case they would probably *have* to live up to the promise for fear of legal action if they failed to do so. 2) a "statement of intent" that they could wiggle out of if the marketplace changed in such a way that it didn't make sense to achieve the stated scalability of the goals.
The only type of announcement that a system planner should probably take seriously is the solid commitment; anything else is just vapourware and should be ignored. I doubt you will see many solid commitments for 8 years in the future, unless you consider platitudes like "we will be the best" as a solid commitment.
I hope this doesn't come across as cynical. I have seen the software industry do tremendous things in the past 20 odd years. I'm expecting many more stunningly impressive innovations in the coming years. I just don't think anyone can predict with any precision what capabilities any given program, like MySQL, will have in 2012.
Rhino
----- Original Message ----- From: "Mark Papadakis" <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>; <[EMAIL PROTECTED]>
Sent: Saturday, November 13, 2004 8:05 AM
Subject: Re: scalability of MySQL - future plans?
allowsSmart Software is the key to many problems hardware with 'standard' software cannot solve or handle.
I believe that mySQL will, by year 2012, be able to handle it gracefully. It will be able to do so much more by then (easy and robust clustering / HA, for example) and even incorporate technologies and ideas that we (as in, people of our time) have not thought of as yet. It might even become the dominant database on the market.. 7 years is a century's worth of time in our world.
It is the evolution of software. Regarding hardware, you should be certain technological advances would make it more than possible. Just look back at what was thought possible 7 years ago, and compare it with today's standards.
On Fri, 12 Nov 2004 10:42:32 -0800, [EMAIL PROTECTED]
<[EMAIL PROTECTED]> wrote:
Adequate data warehouse performance requires more than just hardware. 2 crucial make-or-break software features are partitioning and parallel query.
On very large tables - accessing a large slice of the data via index is
completely unfeasible. Table scan is the only option. Partitioning
tableyou to scan only the necessary segments instead of reading the whole
soand rejecting massive numbers of rows. Parallel query breaks the job up
largethat multiple processes of the OS can participate and speed up the process.
These features are an absolute necessity if we wanted to migrate our
todatabases from Oracle to MySQL. We are eager for MySQL to make them priority features. MySQL's market appeal would just explode. We will do our best to contribute to the effort if we can. I'd like to urge others who plan to use MySQL with large databases to consider doing the same.
Thanks,
Udi
"Heikki Tuuri" <[EMAIL PROTECTED]> 11/12/2004 06:57 AM
To: <[EMAIL PROTECTED]> cc: Subject: Re: scalability of MySQL - future plans?
Jacek,
----- Original Message ----- From: "Jacek Becla" <[EMAIL PROTECTED]> Newsgroups: mailing.database.myodbc Sent: Friday, November 12, 2004 2:30 AM Subject: scalability of MySQL - future plans?
Hello,
What are the plans regarding improving scalability of MySQL? We are
currently trying to decide what technology/product to use for a large
project that will generate ~600TB/year starting in 2012. Any pointers
ablerelated articles or hints how safe is to assume that MySQL will be
speedto handle petabyte-scale dataset in 8-10 years would be greatly
appreciated.
hmm... this mostly depends on hardware. With the innodb_file_per_table option, a single InnoDB table can be 64 TB in size, and you can have 4 billion such tables.
With current PC hardware, the speed of a single CPU allows you to insert
10
000 rows per second, if the load is not disk-bound. Let us assume that a
single row in 100 bytes. That makes 1 MB/s, which is 30 TB/year. CPU
ifwill probably double every 4 years or so. Thus, CPU speed will suffice
size.you use a multiprocessor.
Normally, a database server has main memory at least 1 % of the data
everyIs 6000 GB RAM realistic in 2012? Memory sizes will probably double
combination2 to 3 years. If a high-end server today has 32 GB of RAM, in year 2012 it might have 512 GB of RAM. You will need a huge server.
The worst problem is the disk seek time. If your tables have secondary
indexes where the insertion order is random, a modern disk, in
notwith the InnoDB insert buffer, can insert maybe 200 random records per second. That is 100 rows/s for a typical table. You are going to insert 200 000 rows/s. You may need a disk farm of 4000 physical disks. Such disk farms exist today, but they are expensive, and we have no experience how Linux performs on them. Probably by 2012, Linux is good enough, if not yet today.
If you insert rows in large batches to tables smaller than your main
memory,
or if you insert in the prder of the primary key, and you do not have
secondary keys, then there are no random accesses to disks, and you do
TBneed a disk farm.
A typical disk in 2012 may store 1 TB. Thus, you will need at least 600 disks anyway.
How long does it take to build an index to a 64 TB table if you have 6
3of
memory? If the index completely fits in the memory, then this is
sequential
disk I/O. With today's high end disks, you can read 60 MB/s. Building an
index with a single disk would take 2 weeks. In 2012, it might take only
indays.
Conclusion: MySQL/InnoDB is able to handle that workload of 600 TB/year
ayear 2012. But you will need a huge server which has 10 x the memory of
diskhigh-end server, and 600 - 4000 physical disk drives.
The following link describes a system with 512 GB of memory, and 2000
drives: http://www.tpc.org/results/individual_results/IBM/IBM_690_040217_es.pdf The system costs 5.6 million US dollars.
Best regards,
Jacek Becla
Stanford University
Best regards,
Heikki Tuuri Innobase Oy Foreign keys, transactions, and row level locking for MySQL InnoDB Hot Backup - a hot backup tool for InnoDB which also backs up MyISAM tables http://www.innodb.com/order.php
Order MySQL technical support from https://order.mysql.com/
-- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe: http://lists.mysql.com/[EMAIL PROTECTED]
-- Mark Papadakis Head of R&D Phaistos Networks, S.A
-- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe: http://lists.mysql.com/[EMAIL PROTECTED]
-- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe: http://lists.mysql.com/[EMAIL PROTECTED]
