Very, very large .dat files
We've seen .dat files in seg0 directory grow to several hundreds of gigabytes. While everything works ok such a file becomes unmanageable from an OS point of view. Is there a way to control when Derby starts a new conglomerate? ... so that there are more .dat files but each of them of less size ? Thanks.
Re: Very, very large .dat files
I guess the question is if Apache Derby supports table partitioning (the term used in Oracle RDBMS). I understand it doesn't. This means that if I have an incredible large Derby table then that will mean an incredible large single .dat file too. (a table is always stored in a single .dat file, right?) Any suggestions? Any props that may control this? On 05-Dec-17 13:28, Peter Hansson wrote: We've seen .dat files in seg0 directory grow to several hundreds of gigabytes. While everything works ok such a file becomes unmanageable from an OS point of view. Is there a way to control when Derby starts a new conglomerate? ... so that there are more .dat files but each of them of less size ? Thanks.
Re: Very, very large .dat files
You are correct, Derby does not provide table partitioning features such as those provided by Oracle. And you are correct, a single Derby table is a single .dat file. But modern filesystems handle very large files without problems. Is there a particular reason that you think a very large file will be a problem for you? Or that you think a larger number of smaller .dat files would have some benefit? I believe there are a variety of library/framework approaches to splitting up a logical table into multiple physical tables. People often do this with data that is oriented around "time", and becomes of less interest as it ages. I've seen applications which, e.g., create a new table each day, named appropriately, and "rotate" these tables over time so that the old data ages out and is dropped (by dropping the older no-longer-wanted tables). Careful use of CREATE VIEW can shield your application from most of the impact of such techniques. thanks, bryan
Re: Very, very large .dat files
Thank for taking time to answer. Appreciated. You basically confirmed my understanding: I need to go for a DIU approach if I want table partitioning. On 05-Dec-17 17:30, Bryan Pendleton wrote: You are correct, Derby does not provide table partitioning features such as those provided by Oracle. And you are correct, a single Derby table is a single .dat file. But modern filesystems handle very large files without problems. Is there a particular reason that you think a very large file will be a problem for you? Or that you think a larger number of smaller .dat files would have some benefit? I believe there are a variety of library/framework approaches to splitting up a logical table into multiple physical tables. People often do this with data that is oriented around "time", and becomes of less interest as it ages. I've seen applications which, e.g., create a new table each day, named appropriately, and "rotate" these tables over time so that the old data ages out and is dropped (by dropping the older no-longer-wanted tables). Careful use of CREATE VIEW can shield your application from most of the impact of such techniques. thanks, bryan