Re: [sqlite] sqlite4 lsm storage engine

2013-02-20 Thread Dan Kennedy 

On 02/21/2013 12:31 AM, Rob Turpin wrote:

Dan,

I'm getting an LSM_OK on lsm_close.  I attached the writer and reader test
case.  If you comment out the lsm_config call that turns off logging, all
the writes get in.


I think the mailing list stripped the attachment. Can you send them
to me directly?

Thanks,
Dan.





Thanks,
Rob

On Wed, Feb 20, 2013 at 9:43 AM, Dan Kennedy  wrote:


On 02/20/2013 11:37 PM, Rob Turpin wrote:


Yes.



Is it succeeding? Returning LSM_OK?

Dan.





On Wed, Feb 20, 2013 at 3:14 AM, Dan Kennedy 
wrote:

  On 02/20/2013 05:07 PM, Rob Turpin wrote:


  I'm running some performance tests on the lsm storage engine, and an

issue
has cropped up for me.  I retrieved the sqlite4 code from the repository
about a week ago.

I'm doing a simple single threaded test to see what kind of performance
I
can get on write transactions.  After completing the write test, I run
another program to read the DB, and all the writes are not there.

Here's are the config options I set before calling lsm_open.

int iVal = 0;
lsm_config(db, LSM_CONFIG_MULTIPLE_PROCESSES, );
lsm_config(db, LSM_CONFIG_USE_LOG, );
iVal = 4096;
lsm_config(db, LSM_CONFIG_AUTOFLUSH, );
iVal = 8192;
lsm_config(db, LSM_CONFIG_AUTOCHECKPOINT, );


If logging is on, there is no issue, all writes are in the DB.  But
since
I'm trying to increase performance I turn this off.

   From reading the documentation I was assuming the
LSM_CONFIG_AUTOCHECKPOINT
setting should sync the writes in memory to disk at 8MB (per my setting,
default is 2 MB).  I'm performing no explicit commits or checkpoints.
  I'm
simply calling lsm_insert to write data to the DB.  Perhaps this is
where
I'm confused about the expected behavior.

Could anyone correct me where I'm wrong on my assumptions, or could
there
be an issue with this?



Are you calling lsm_close() at the end of the write test?

Dan.




___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users





  __**_

sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-**bin/mailman/listinfo/sqlite-**users



__**_
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-**bin/mailman/listinfo/sqlite-**users



___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] sqlite4 lsm storage engine

2013-02-20 Thread Rob Turpin 
Dan,

I'm getting an LSM_OK on lsm_close.  I attached the writer and reader test
case.  If you comment out the lsm_config call that turns off logging, all
the writes get in.

Thanks,
Rob

On Wed, Feb 20, 2013 at 9:43 AM, Dan Kennedy  wrote:

> On 02/20/2013 11:37 PM, Rob Turpin wrote:
>
>> Yes.
>>
>
> Is it succeeding? Returning LSM_OK?
>
> Dan.
>
>
>
>
>> On Wed, Feb 20, 2013 at 3:14 AM, Dan Kennedy 
>> wrote:
>>
>>  On 02/20/2013 05:07 PM, Rob Turpin wrote:
>>>
>>>  I'm running some performance tests on the lsm storage engine, and an
 issue
 has cropped up for me.  I retrieved the sqlite4 code from the repository
 about a week ago.

 I'm doing a simple single threaded test to see what kind of performance
 I
 can get on write transactions.  After completing the write test, I run
 another program to read the DB, and all the writes are not there.

 Here's are the config options I set before calling lsm_open.

 int iVal = 0;
 lsm_config(db, LSM_CONFIG_MULTIPLE_PROCESSES, );
 lsm_config(db, LSM_CONFIG_USE_LOG, );
 iVal = 4096;
 lsm_config(db, LSM_CONFIG_AUTOFLUSH, );
 iVal = 8192;
 lsm_config(db, LSM_CONFIG_AUTOCHECKPOINT, );


 If logging is on, there is no issue, all writes are in the DB.  But
 since
 I'm trying to increase performance I turn this off.

   From reading the documentation I was assuming the
 LSM_CONFIG_AUTOCHECKPOINT
 setting should sync the writes in memory to disk at 8MB (per my setting,
 default is 2 MB).  I'm performing no explicit commits or checkpoints.
  I'm
 simply calling lsm_insert to write data to the DB.  Perhaps this is
 where
 I'm confused about the expected behavior.

 Could anyone correct me where I'm wrong on my assumptions, or could
 there
 be an issue with this?


>>> Are you calling lsm_close() at the end of the write test?
>>>
>>> Dan.
>>>
>>>
>>>
>>>
>>> ___
>>> sqlite-users mailing list
>>> sqlite-users@sqlite.org
>>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>> 
>>> >
>>>
>>>  __**_
>> sqlite-users mailing list
>> sqlite-users@sqlite.org
>> http://sqlite.org:8080/cgi-**bin/mailman/listinfo/sqlite-**users
>>
>>
> __**_
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-**bin/mailman/listinfo/sqlite-**users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] sqlite4 lsm storage engine

2013-02-20 Thread Dan Kennedy 

On 02/20/2013 11:37 PM, Rob Turpin wrote:

Yes.


Is it succeeding? Returning LSM_OK?

Dan.





On Wed, Feb 20, 2013 at 3:14 AM, Dan Kennedy  wrote:


On 02/20/2013 05:07 PM, Rob Turpin wrote:


I'm running some performance tests on the lsm storage engine, and an issue
has cropped up for me.  I retrieved the sqlite4 code from the repository
about a week ago.

I'm doing a simple single threaded test to see what kind of performance I
can get on write transactions.  After completing the write test, I run
another program to read the DB, and all the writes are not there.

Here's are the config options I set before calling lsm_open.

int iVal = 0;
lsm_config(db, LSM_CONFIG_MULTIPLE_PROCESSES, );
lsm_config(db, LSM_CONFIG_USE_LOG, );
iVal = 4096;
lsm_config(db, LSM_CONFIG_AUTOFLUSH, );
iVal = 8192;
lsm_config(db, LSM_CONFIG_AUTOCHECKPOINT, );


If logging is on, there is no issue, all writes are in the DB.  But since
I'm trying to increase performance I turn this off.

  From reading the documentation I was assuming the
LSM_CONFIG_AUTOCHECKPOINT
setting should sync the writes in memory to disk at 8MB (per my setting,
default is 2 MB).  I'm performing no explicit commits or checkpoints.  I'm
simply calling lsm_insert to write data to the DB.  Perhaps this is where
I'm confused about the expected behavior.

Could anyone correct me where I'm wrong on my assumptions, or could there
be an issue with this?



Are you calling lsm_close() at the end of the write test?

Dan.




__**_
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-**bin/mailman/listinfo/sqlite-**users


___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users



___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] sqlite4 lsm storage engine

2013-02-20 Thread Rob Turpin 
Yes.

On Wed, Feb 20, 2013 at 3:14 AM, Dan Kennedy  wrote:

> On 02/20/2013 05:07 PM, Rob Turpin wrote:
>
>> I'm running some performance tests on the lsm storage engine, and an issue
>> has cropped up for me.  I retrieved the sqlite4 code from the repository
>> about a week ago.
>>
>> I'm doing a simple single threaded test to see what kind of performance I
>> can get on write transactions.  After completing the write test, I run
>> another program to read the DB, and all the writes are not there.
>>
>> Here's are the config options I set before calling lsm_open.
>>
>> int iVal = 0;
>> lsm_config(db, LSM_CONFIG_MULTIPLE_PROCESSES, );
>> lsm_config(db, LSM_CONFIG_USE_LOG, );
>> iVal = 4096;
>> lsm_config(db, LSM_CONFIG_AUTOFLUSH, );
>> iVal = 8192;
>> lsm_config(db, LSM_CONFIG_AUTOCHECKPOINT, );
>>
>>
>> If logging is on, there is no issue, all writes are in the DB.  But since
>> I'm trying to increase performance I turn this off.
>>
>>  From reading the documentation I was assuming the
>> LSM_CONFIG_AUTOCHECKPOINT
>> setting should sync the writes in memory to disk at 8MB (per my setting,
>> default is 2 MB).  I'm performing no explicit commits or checkpoints.  I'm
>> simply calling lsm_insert to write data to the DB.  Perhaps this is where
>> I'm confused about the expected behavior.
>>
>> Could anyone correct me where I'm wrong on my assumptions, or could there
>> be an issue with this?
>>
>
> Are you calling lsm_close() at the end of the write test?
>
> Dan.
>
>
>
>
> __**_
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-**bin/mailman/listinfo/sqlite-**users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] sqlite4 lsm storage engine

2013-02-20 Thread Dan Kennedy 

On 02/20/2013 05:07 PM, Rob Turpin wrote:

I'm running some performance tests on the lsm storage engine, and an issue
has cropped up for me.  I retrieved the sqlite4 code from the repository
about a week ago.

I'm doing a simple single threaded test to see what kind of performance I
can get on write transactions.  After completing the write test, I run
another program to read the DB, and all the writes are not there.

Here's are the config options I set before calling lsm_open.

int iVal = 0;
lsm_config(db, LSM_CONFIG_MULTIPLE_PROCESSES, );
lsm_config(db, LSM_CONFIG_USE_LOG, );
iVal = 4096;
lsm_config(db, LSM_CONFIG_AUTOFLUSH, );
iVal = 8192;
lsm_config(db, LSM_CONFIG_AUTOCHECKPOINT, );


If logging is on, there is no issue, all writes are in the DB.  But since
I'm trying to increase performance I turn this off.

 From reading the documentation I was assuming the LSM_CONFIG_AUTOCHECKPOINT
setting should sync the writes in memory to disk at 8MB (per my setting,
default is 2 MB).  I'm performing no explicit commits or checkpoints.  I'm
simply calling lsm_insert to write data to the DB.  Perhaps this is where
I'm confused about the expected behavior.

Could anyone correct me where I'm wrong on my assumptions, or could there
be an issue with this?


Are you calling lsm_close() at the end of the write test?

Dan.




___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users