Re: auto-growing data size

[Dave Horsfall]

On Fri, 13 Apr 2018, Howard Chu wrote:

I suppose so. Nobody says you have to use UFS for everything, just use 
it for a data partition.


Yep, but I wanted the archives to show that using UFS, whilst solving the 
problem with sparse files, could break other things on the Mac side, so 
use it at your own risk.


--
Dave Horsfall DTM (VK2KFU)  "Those who don't understand security will suffer."

Re: auto-growing data size

[mark diener]

Howard:

So you are saying that on Android, Ios, and Windows 10, the
mdb_env_set_mapsize() will grow only as needed
and not pre-allocate the file space?

Only on MacOSX is there a funky behavior where I need to switch the
partition to UFS?

>From a commercial standpoint, that is not feasible.  How do I create
an installation program
that automatically creates a UFS partition?

Looks like OpenLDAP has some issues on on MacOSX.  What do you guys do
for MacOSX? Or do you even ship software for it?

Thanks,

Marco

On Wed, Apr 11, 2018 at 11:12 PM, Howard Chu  wrote:
> mark diener wrote:
>>
>> I came across a python write up about lmdb:
>>
>> http://lmdb.readthedocs.io/en/release/#environment-class
>>
>> map_size:
>>
>> Maximum size database may grow to; used to size the memory mapping. If
>> database grows larger than map_size, an exception will be raised and
>> the user must close and reopen Environment. On 64-bit there is no
>> penalty for making this huge (say 1TB). Must be <2GB on 32-bit.
>>
>> Well, it will be a penalty on MacOSX, it will create a 1TB file on
>> disk upon initialization!
>
>
> If you have a 1TB or larger filesystem, then there's obviously no problem.
> If the filesystem is smaller than that, obviously you can't choose a size
> bigger than the filesystem has space for.
>>
>>
>> What am I missing here?
>
>
> Most POSIX filesystems don't preallocate space unless you explicitly request
> it. I.e., setting the size of a file (using ftruncate()) merely stores that
> number in an inode. You have to do an explicit fallocate() to make the
> filesystem allocate all of the space.
>
> Since you're on MacOSX, things are different. The default filesystem (HFS or
> HFS+) doesn't support sparse files, so when you set the size of a file to a
> particular number, the filesystem allocates the space at that moment.
>
> If you want to get the behavior you're looking for, you need to switch to
> UFS, which supports sparse files like most other POSIX filesystems.
>
> Meanwhile - even if you're using HFS, you should still just set the map size
> to the largest value you ever expect to use. Preallocating the space will
> give better runtime performance, and will allow the FS to optimize the
> layout of the allocated blocks.
>>
>>
>> Marco
>>
>>
>> On Wed, Apr 11, 2018 at 10:27 AM, mark diener  wrote:
>>>
>>> Anybody have any idea on how to have the database file
>>> grow until you get to the mapsize instead of pre-allocating the entire
>>> mapsize (4GB) upon initialization?
>>>
>>> mdb_env_set_mapsize() ;
>>>
>>> I don't mind having an upper limit on the map size, but I think it
>>> would be helpful
>>> to have it grow to the upper limit, not preallocate the space on storage.
>
>
> Your thought here is invalid.
>
>>> There are various areas in the documentation where it talks about
>>> having the database not grow without bounds.
>>>
>>> Comments?
>>>
>>>
>>> Marco
>>
>>
>>
>
>
> --
>   -- Howard Chu
>   CTO, Symas Corp.   http://www.symas.com
>   Director, Highland Sun http://highlandsun.com/hyc/
>   Chief Architect, OpenLDAP  http://www.openldap.org/project/

Re: auto-growing data size

[Howard Chu]

Dave Horsfall wrote:

On Thu, 12 Apr 2018, Howard Chu wrote:

[ Sparse files ]

If you want to get the behavior you're looking for, you need to switch to 
UFS, which supports sparse files like most other POSIX filesystems.


But does that not also remove case-insensitivity,


Yes

leading to failures in Mac 
applications that do not bother to get the case right?


I suppose so. Nobody says you have to use UFS for everything, just use it for 
a data partition.


Coming from a "real" Unix background, it's quite irritating having (say) 
"Makefile" and "makefile" the same object...  Yes, I've seen cases (!) where 
both are used.


Yeah, I've seen that too. Not in quite a while tho.

--
  -- Howard Chu
  CTO, Symas Corp.   http://www.symas.com
  Director, Highland Sun http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

Re: auto-growing data size

[Dave Horsfall]

On Thu, 12 Apr 2018, Howard Chu wrote:

[ Sparse files ]

If you want to get the behavior you're looking for, you need to switch 
to UFS, which supports sparse files like most other POSIX filesystems.


But does that not also remove case-insensitivity, leading to failures in 
Mac applications that do not bother to get the case right?


Coming from a "real" Unix background, it's quite irritating having (say) 
"Makefile" and "makefile" the same object...  Yes, I've seen cases (!) 
where both are used.


--
Dave Horsfall DTM (VK2KFU)  "Those who don't understand security will suffer."

Re: auto-growing data size

[Howard Chu]

mark diener wrote:

I came across a python write up about lmdb:

http://lmdb.readthedocs.io/en/release/#environment-class

map_size:

Maximum size database may grow to; used to size the memory mapping. If
database grows larger than map_size, an exception will be raised and
the user must close and reopen Environment. On 64-bit there is no
penalty for making this huge (say 1TB). Must be <2GB on 32-bit.

Well, it will be a penalty on MacOSX, it will create a 1TB file on
disk upon initialization!


If you have a 1TB or larger filesystem, then there's obviously no problem. If 
the filesystem is smaller than that, obviously you can't choose a size bigger 
than the filesystem has space for.


What am I missing here?


Most POSIX filesystems don't preallocate space unless you explicitly request 
it. I.e., setting the size of a file (using ftruncate()) merely stores that 
number in an inode. You have to do an explicit fallocate() to make the 
filesystem allocate all of the space.


Since you're on MacOSX, things are different. The default filesystem (HFS or 
HFS+) doesn't support sparse files, so when you set the size of a file to a 
particular number, the filesystem allocates the space at that moment.


If you want to get the behavior you're looking for, you need to switch to UFS, 
which supports sparse files like most other POSIX filesystems.


Meanwhile - even if you're using HFS, you should still just set the map size 
to the largest value you ever expect to use. Preallocating the space will give 
better runtime performance, and will allow the FS to optimize the layout of 
the allocated blocks.


Marco


On Wed, Apr 11, 2018 at 10:27 AM, mark diener  wrote:

Anybody have any idea on how to have the database file
grow until you get to the mapsize instead of pre-allocating the entire
mapsize (4GB) upon initialization?

mdb_env_set_mapsize() ;

I don't mind having an upper limit on the map size, but I think it
would be helpful
to have it grow to the upper limit, not preallocate the space on storage.


Your thought here is invalid.


There are various areas in the documentation where it talks about
having the database not grow without bounds.

Comments?


Marco






--
  -- Howard Chu
  CTO, Symas Corp.   http://www.symas.com
  Director, Highland Sun http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

Re: auto-growing data size

[mark diener]

I came across a python write up about lmdb:

http://lmdb.readthedocs.io/en/release/#environment-class

map_size:

Maximum size database may grow to; used to size the memory mapping. If
database grows larger than map_size, an exception will be raised and
the user must close and reopen Environment. On 64-bit there is no
penalty for making this huge (say 1TB). Must be <2GB on 32-bit.

Well, it will be a penalty on MacOSX, it will create a 1TB file on
disk upon initialization!

What am I missing here?

Marco


On Wed, Apr 11, 2018 at 10:27 AM, mark diener  wrote:
> Anybody have any idea on how to have the database file
> grow until you get to the mapsize instead of pre-allocating the entire
> mapsize (4GB) upon initialization?
>
> mdb_env_set_mapsize() ;
>
> I don't mind having an upper limit on the map size, but I think it
> would be helpful
> to have it grow to the upper limit, not preallocate the space on storage.
>
> There are various areas in the documentation where it talks about
> having the database not grow without bounds.
>
> Comments?
>
>
> Marco