Re: [sqlite] Opening an in-memory database in two connections, one read-write, one read-only.

2013-05-02 Thread Scott Hess
No comment? Do I need to recast the problem or something? Thanks, scott On Fri, Apr 26, 2013 at 9:54 PM, Scott Hess sh...@google.com wrote: Someone over here was trying to use a shared-cache in-memory database with two connections, one a read-write connection for purposes of populating

[sqlite] Opening an in-memory database in two connections, one read-write, one read-only.

2013-04-26 Thread Scott Hess
Someone over here was trying to use a shared-cache in-memory database with two connections, one a read-write connection for purposes of populating things, the other a read-only connection for purposes of letting a (trusted) user make queries. They were surprised to find out that they could run

Re: [sqlite] update record in contentless FTS4

2013-04-16 Thread Scott Hess
On Fri, Apr 12, 2013 at 9:28 AM, Lukas Gebauer gebyl...@mlp.cz wrote: I have contentless FTS4 index for searching some my external data paired by docid. What I can do, when my existing previously indexed document was changed? I need to update existing FTS4 index too. But documentation says:

[sqlite] secure_delete versus journal_mode persist.

2013-01-29 Thread Scott Hess
Given a 3.7.15.2 OSX binary fresh from the website, the following scenario seems suspect: ./sqlite3 trial.db sqlite pragma secure_delete = 1; 1 sqlite pragma journal_mode = persist; persist sqlite pragma journal_size_limit = 4096; 4096 sqlite pragma page_size; 1024 sqlite create table x (x);

Re: [sqlite] secure_delete versus journal_mode persist.

2013-01-29 Thread Scott Hess
On Tue, Jan 29, 2013 at 7:52 AM, Yuriy Kaminskiy yum...@gmail.com wrote: User-space database encryption and all safe data erase tools only provide you with warm feeling of safety without any real security. I agree. I am merely pointing out that I think that this result may be surprising in

Re: [sqlite] How to know what terms were created using FTS

2012-08-27 Thread Scott Hess
On Sat, Aug 18, 2012 at 10:00 AM, Mohit Sindhwani m...@onghu.com wrote: On 17/8/2012 7:14 PM, Dominique Pellé wrote: This gives the tokens: sqlite CREATE VIRTUAL TABLE ft USING fts4(x); sqlite INSERT INTO ft VALUES(hello world); sqlite INSERT INTO ft VALUES(hello there); sqlite CREATE

Re: [sqlite] VACUUMing large DBs

2012-03-22 Thread Scott Hess
On Tue, Mar 20, 2012 at 8:25 PM, Jay A. Kreibich j...@kreibi.ch wrote: On Tue, Mar 20, 2012 at 01:59:59PM -0700, Udi Karni scratched on the wall: Is there a way to go directory from original to journal/final - skipping the creation of the Temp version?  No, it requires all three copies. ...  

Re: [sqlite] SIGBUS error in case of disk full with WAL mode

2012-03-08 Thread Scott Hess
On Thu, Mar 8, 2012 at 9:28 AM, Pavel Ivanov paiva...@gmail.com wrote: Question:  Does anybody know of a better way to get memory shared among processes other than to create a fake file and mmap() it?  Are there some magic options to mmap() (perhaps Linux-only options) that prevent it from

Re: [sqlite] Tuning a SQLite database for best compression

2012-03-05 Thread Scott Hess
Note that importing the .dump file does effectively strip the indices and re-create them. In fact, there's a good chance that if you drop the indices, VACUUM, and then compress the database file, you'll find that it comes close to what you get with the .dump-then-compress values. -scott On

Re: [sqlite] Prepare SQL for Read-Only Database with Journal File

2012-02-17 Thread Scott Hess
On Fri, Feb 17, 2012 at 1:02 PM, Richard Hipp d...@sqlite.org wrote: On Fri, Feb 17, 2012 at 3:52 PM, Marc L. Allen mlal...@outsitenetworks.comwrote: My concept of always consistent is a.. for lack of a better term... virtual concept.  That is, anytime anything accesses the database, as a

Re: [sqlite] Function context

2012-02-13 Thread Scott Hess
On Mon, Feb 13, 2012 at 9:24 AM, Steinar Midtskogen stei...@latinitas.org wrote: [Peter Aronson] (2) You can associate data with an argument to a regular user-defined function using sqlite3_set_auxdata() and sqlite3_get_auxdata() as long as the value of the argument is static.  If you don't

Re: [sqlite] Function context

2012-02-13 Thread Scott Hess
On Mon, Feb 13, 2012 at 11:51 AM, Steinar Midtskogen stei...@latinitas.org wrote: [Scott Hess] Unfortunately, I can't offhand think of a reasonable solution for you, I think I'd just use the SELECT to generate the data, while calculating the moving average in my application code. Yes

Re: [sqlite] Function context

2012-02-13 Thread Scott Hess
On Mon, Feb 13, 2012 at 12:28 PM, Simon Slavin slav...@bigfraud.org wrote: On 13 Feb 2012, at 7:51pm, Steinar Midtskogen wrote: One should think that the ability to calculate a moving average would be a pretty common request.  But people do it in their application code instead? Actually, my

Re: [sqlite] Segment merging in FTS and updates-deletes

2011-11-11 Thread Scott Hess
On Fri, Nov 11, 2011 at 9:58 AM, nobre rafael.ro...@novaprolink.com.br wrote: Hi, I'm studying the indexing mechanism of FTS3/4, I can pretty much understand how doclists, terms, segments are created and stored, but one thing I can't grasp is about updating and deleting docs and keeping up the

Re: [sqlite] Readonly error when copying to mem-based db

2011-11-07 Thread Scott Hess
On Mon, Nov 7, 2011 at 9:02 AM, Mills, Steve smi...@makemusic.com wrote:                if(pageSize 0) {                        SqliteStatement  cmd(PRAGMA page_size = ?, *this);                        cmd.BindToInt(1, syz);                        cmd.Step();                } Can't prepare

Re: [sqlite] cache size and insert-only

2011-10-28 Thread Scott Hess
On Fri, Oct 28, 2011 at 5:36 AM, Jay A. Kreibich j...@kreibi.ch wrote: On Fri, Oct 28, 2011 at 01:33:31PM +0200, Gert Corthout scratched on the wall: we have a database that only performs insert statements on a table (database contains only this table), is it in this case a good idea to reduce

Re: [sqlite] FTS vs INDEX

2011-10-19 Thread Scott Hess
On Wed, Oct 19, 2011 at 7:56 AM, Fabian fabianpi...@gmail.com wrote: 2011/10/19 Alexey Pechnikov pechni...@mobigroup.ru FTS use index multi-tree and de-facto has _no_ insert speed degradation. Thanks, that's good to hear! It makes me wonder why SQLite doesn't use that same multi-tree

Re: [sqlite] FTS vs INDEX

2011-10-19 Thread Scott Hess
On Wed, Oct 19, 2011 at 12:50 PM, Fabian fabianpi...@gmail.com wrote: 2011/10/19 Scott Hess sh...@google.com To be clear, how it works is that new insertions are batched into a new index tree, with index trees periodically aggregated to keep selection efficient and to keep the size contained

Re: [sqlite] feed endless data into sqlite, thru a shell script

2011-09-27 Thread Scott Hess
On Tue, Sep 27, 2011 at 11:18 AM, Gabor Grothendieck ggrothendi...@gmail.com wrote: On Tue, Sep 27, 2011 at 2:14 PM, David Garfield garfi...@irving.iisd.sra.com wrote: Any entry in a pipe could be buffering. In a quick test here, awk is buffering. To find the buffering, try using the

Re: [sqlite] feed endless data into sqlite, thru a shell script

2011-09-27 Thread Scott Hess
On Tue, Sep 27, 2011 at 2:07 PM, Patrick Proniewski pat...@patpro.netwrote: On 27 sept. 2011, at 20:18, Gabor Grothendieck wrote: gawk has fflush() none of them is available out of the box on Mac OS X, or FreeBSD. gawk can be installed, but I'd rather use my while true loop instead of

Re: [sqlite] How to reindex an FTS3 table after changing the tokenizer

2011-08-22 Thread Scott Hess
On Fri, Aug 12, 2011 at 11:27 AM, john Papier johnpap...@gmail.com wrote: I have a FTS3 table that was created with the simple tokenizer. I want to change the tokenizer and reindex the table. Is there a way to change the tokenizer in place and have it reindex with minimal code? Else the

Re: [sqlite] Fwd: Case Sensitive FTS searches.

2011-08-11 Thread Scott Hess
On Thu, Aug 11, 2011 at 8:57 AM, Dennis Suehr den...@suehr.me.uk wrote: After some digging through the sqlite3 source code, I came across the code for the ICU tokenizer.  After enabling that and then commenting out the one line where u_foldCase() is called, i.e. icuOpen(), I retested and

Re: [sqlite] How to convert a database with FTS2 to FTS3/4?

2011-07-31 Thread Scott Hess
On Fri, Jul 29, 2011 at 10:33 PM, Ray Van Dolson ra...@bludgeon.org wrote: sqlite newbie here.  I have a sqlite3 database that was built using the FTS2 module.  As such I get errors about the non-existent fts2 module when attempting to query the data. Is there a way to convert/migrate the

[sqlite] Bug with fts3 when using a restrictive authorizer.

2011-06-22 Thread Scott Hess
If the authorizer prevents PRAGMA, then fts3 fails with an SQLITE_AUTH error from fts3DatabasePageSize(). http://crbug.com/85522 is the Chromium bug where this originated (I upgraded the Chromium SQLite version a few weeks back). http://codereview.chromium.org/7230021 is the short-term patch I'm

Re: [sqlite] Handling of IEEE-754 nan and +/-inf in SQLite?

2011-06-06 Thread Scott Hess
On Mon, Jun 6, 2011 at 1:46 PM, Nico Williams n...@cryptonector.com wrote: I can see two ways to add proper IEEE754 support in a backwards-compatible way: a) via a pragma to enable raising exceptions or returning +-inf/NaNs, b) by adding a callback via which to report such exceptions, with

Re: [sqlite] Page Size with NAND FLASH

2011-05-04 Thread Scott Hess
On Tue, May 3, 2011 at 4:44 PM, Sugathan, Rupesh r...@carriercomm.com wrote: I am planning to use sqlite on a Linux system with JFFS2 file system on NAND flash. NAND device that I am using has page size of 2048 bytes and a erase sector size of 128K. I would like to take advantage of sqlite

[sqlite] WAL checkpoints versus fsync.

2011-03-30 Thread Scott Hess
[From a thread about WAL checkpoints...] On Wed, Mar 30, 2011 at 6:28 AM, Richard Hipp d...@sqlite.org wrote: On Wed, Mar 30, 2011 at 8:46 AM, Mikael mikael@gmail.com wrote: Therefore I now wish to check with you, is SQLite implemented to somehow make checkpoints not be able to corrupt the

Re: [sqlite] Optimizing list retrieval with a FTS3 table

2011-03-16 Thread Scott Hess
On Wed, Mar 16, 2011 at 12:15 PM, Travis Orr t...@ivl.com wrote: Can someone enlighten me about this. To me a lot of the details appear to be hidden since my main SongTable is a FTS3 virtual table. You don't provide your schema, but based on your queries, I'll make unwarranted assumptions :-).

Re: [sqlite] Fwd: Fwd: fts virtual table questions

2011-03-01 Thread Scott Hess
On Tue, Mar 1, 2011 at 4:54 PM, Paul Shaffer sqli...@cyberplasm.com wrote: I had an API problem, and I now think that the virtual table does not need to be created each time. I can't find documentation that covers this. I don't think there is documentation which covers this, because it's how

Re: [sqlite] apostrophes in strings...

2011-02-23 Thread Scott Hess
I believe the key is used for attaching to encrypted databases. -scott On Wed, Feb 23, 2011 at 7:15 AM, Sam Carleton scarle...@miltonstreet.com wrote: Kevin, Thank you, that is what I needed.  Now to statisfy my curiosity...  What exactly is the KEY value? On Tue, Feb 22, 2011 at 10:39

Re: [sqlite] apostrophes in strings...

2011-02-23 Thread Scott Hess
On Wed, Feb 23, 2011 at 12:50 PM, BareFeetWare list@barefeetware.com wrote: On 21/02/2011, at 8:11 AM, Scott Hess wrote: You can also convert:  ATTACH DATABASE x AS y KEY z to:  SELECT sqlite_attach(x, y, z) where the parameters can be turned into bind arguments.  Then embedded quotes

Re: [sqlite] apostrophes in strings...

2011-02-20 Thread Scott Hess
You can also convert: ATTACH DATABASE x AS y KEY z to: SELECT sqlite_attach(x, y, z) where the parameters can be turned into bind arguments. Then embedded quotes won't be an issue. -scott On Sun, Feb 20, 2011 at 11:31 AM, Pavel Ivanov paiva...@gmail.com wrote: I believe doubling the

Re: [sqlite] will FTS speed up [LIKE '%foo%'] searches? or: how does iTunes do it?

2011-02-18 Thread Scott Hess
On Fri, Feb 18, 2011 at 12:08 AM, David M. Cotter m...@davecotter.com wrote: so i am still left wondering if searching substrings is really any faster using FTS. You may want to search the archives, as this has come up before. I don't recall if anyone had an inspired solution. You could

Re: [sqlite] integer or string in fts3 table

2011-01-31 Thread Scott Hess
On Mon, Jan 31, 2011 at 9:15 AM, Samuel Adam a...@certifound.com wrote: On Mon, 31 Jan 2011 11:46:21 -0500, Samuel Adam a...@certifound.com wrote: On Mon, 31 Jan 2011 11:05:13 -0500, Simon Slavin slav...@bigfraud.org wrote: In the definition given in the original post, which I quoted, the

Re: [sqlite] Deleting the sqlite journal file?

2011-01-21 Thread Scott Hess
On Fri, Jan 21, 2011 at 5:33 PM, Pavel Ivanov paiva...@gmail.com wrote:        (2) Are there any changes that can/should be made to sqlite3 so that it can identify the bogus journal in this scenario and discard it? Define the word bogus. How should SQLite understand that the journal and the

Re: [sqlite] WAL on a separate filesystem?

2011-01-19 Thread Scott Hess
On Wed, Jan 19, 2011 at 12:15 PM, Dustin Sallings dus...@spy.net wrote: This isn't a question so much about value judgment (I've already argued that some, though mentioning maintenance tools is helpful there, too).  It comes down to whether reliability of SQLite itself would be reduced if a

Re: [sqlite] pragma vs select for introspection

2010-12-13 Thread Scott Hess
On Mon, Dec 13, 2010 at 1:27 PM, Puneet Kishor punk.k...@gmail.com wrote: Wols Lists wrote: On 13/12/10 01:38, Darren Duncan wrote: Darren Duncan wrote: Wols Lists wrote: Dunno how well that approach translates into a relational engine, because Pick has several very non-relational quirks

Re: [sqlite] Increase the datafile file size to limit the file fragmentation

2010-12-10 Thread Scott Hess
On Fri, Dec 10, 2010 at 12:20 PM, Vander Clock Stephane svandercl...@yahoo.fr wrote: to limit the file fragmentation i want to increase the size of the database file (with the windows API function). Consider http://www.sqlite.org/c3ref/c_fcntl_chunk_size.html . -scott

Re: [sqlite] Stored procedures (was: Question about SQLite features.)

2010-11-12 Thread Scott Hess
On Fri, Nov 12, 2010 at 5:50 AM, BareFeetWare list@barefeetware.com wrote: IMO, if you're implementing database logic (ie constraints and triggers) in application code, then you're reinventing the wheel, making your package unnecessarily complex and grossly inefficient. If you're just using

Re: [sqlite] Full text search FTS3 of files

2010-10-18 Thread Scott Hess
On Sun, Oct 17, 2010 at 11:13 PM, Dami Laurent (PJ) laurent.d...@justice.ge.ch wrote: Is it possible to use FTS3 for search without storing the actual file contents/search terms/keywords in a row. In other words, create a FTS3 tables with rows that only contains an ID and populate the B-Tree with

Re: [sqlite] Many many tables... performance wise?

2010-10-15 Thread Scott Hess
Having a table with an owner_id, key, and value, with a unique index on (owner_id, key) will probably be more efficient than having a separate table per owner. Also, it will be easier to code safely, because bind parameters don't work on table names (I'm assuming you're using dynamic table names

Re: [sqlite] Speed up DELETE of a lot of records

2010-10-12 Thread Scott Hess
On Mon, Oct 11, 2010 at 8:13 AM, Jay A. Kreibich j...@kreibi.ch wrote: On Mon, Oct 11, 2010 at 02:08:54PM +0200, Michele Pradella scratched on the wall:   Ok so the main idea it's always the same: split the DELETE to make the operation on less records, but do it more often.  Another thought

Re: [sqlite] Registering a custom tokenizer per database rather than per connection

2010-10-12 Thread Scott Hess
On Tue, Oct 12, 2010 at 8:40 AM, Drake Wilson dr...@begriffli.ch wrote: Just to clarify, a trigger isn't useful for this in general, because you still have nowhere to get the function pointer from.  Above I was worrying that the current SQLite code might be less safe than it could be as a

Re: [sqlite] Feature request: copying vacuum

2010-09-01 Thread Scott Hess
On Wed, Sep 1, 2010 at 12:46 PM, Jay A. Kreibich j...@kreibi.ch wrote: On Wed, Sep 01, 2010 at 11:41:00AM -0700, Taras Glek scratched on the wall: Currently VACUUM takes care of sqlite-level fragmentation. Unfortunately it does little for fs-level fragmentation since the same file is being

Re: [sqlite] Feature request: copying vacuum

2010-09-01 Thread Scott Hess
On Wed, Sep 1, 2010 at 10:09 PM, Max Vlasov max.vla...@gmail.com wrote: I agree with Jay - while it is tempting to have SQLite bite off optimizing this kind of thing, it's pretty far out of scope.  Next we'll be talking about running SQLite on raw partitions! Scott, thought about it,

Re: [sqlite] upstreaming Chromium patches for file handle passing support

2010-09-01 Thread Scott Hess
On Wed, Sep 1, 2010 at 10:20 PM, Max Vlasov max.vla...@gmail.com wrote: I wonder whether it would be possible to extend the VFS in a way that would make our use case possible (transferring a file handle over process boundary). Please note that we do it on all platforms. On POSIX we pass an

Re: [sqlite] [PATCH] Fix locale-unsafe usage of tolower (upstreaming Chromium change)

2010-08-18 Thread Scott Hess
I'm not sure Chromium has any fts1 databases, I think the original patch was applied there for completeness. The change from fts2 to fts3 has been made in the history system, but it only applies to new data, and hasn't yet rolled out to stable. So we wouldn't be able to even start to cease using

Re: [sqlite] [PATCH] cache preloading

2010-08-17 Thread Scott Hess
On Mon, Aug 16, 2010 at 7:58 PM, Shawn Wilsher m...@shawnwilsher.com wrote: On Mon, Aug 16, 2010 at 5:13 PM, Paweł Hajdan, Jr. phajdan...@chromium.org wrote: Is it something you'd like to include in SQLite? If so, does the patch need any adjustments before that's possible? I'm slightly

Re: [sqlite] [PATCH] Fix locale-unsafe usage of tolower (upstreaming Chromium change)

2010-08-06 Thread Scott Hess
This bug comment describes the problem: http://code.google.com/p/chromium/issues/detail?id=15261#c20 excerpt: Apparently the problem is caused by tolower(), whose behavior is affected by current locale. Under locale tr_TR.UTF-8, tolower('I') returns 'I' rather than 'i', because lower

Re: [sqlite] Substring (LIKE %key%) searches, would FTS3 with suffix-tree tokenizer be the fast way?

2010-08-06 Thread Scott Hess
On Thu, Aug 5, 2010 at 12:42 PM, Sam Roberts vieuxt...@gmail.com wrote: FTS3 only searches full terms/words by default, but I think if I built a custom tokenizer that returned all the suffix trees for a name: FTS3 can do prefix searches, MATCH 'a*'. Also, it aimed to support multiple hits at

Re: [sqlite] [PATCH] Fix locale-unsafe usage of tolower (upstreaming Chromium change)

2010-08-06 Thread Scott Hess
Hipp d...@sqlite.org wrote: FTS3 updated here:  http://www.sqlite.org/src/ci/b8b465ed2c On Fri, Aug 6, 2010 at 2:24 PM, Scott Hess sh...@google.com wrote: This bug comment describes the problem:   http://code.google.com/p/chromium/issues/detail?id=15261#c20 excerpt: Apparently the problem

Re: [sqlite] Substring (LIKE %key%) searches, would FTS3 with suffix-tree tokenizer be the fast way?

2010-08-06 Thread Scott Hess
On Fri, Aug 6, 2010 at 6:08 PM, Sam Roberts vieuxt...@gmail.com wrote: On Fri, Aug 6, 2010 at 11:32 AM, Scott Hess sh...@google.com wrote: On Thu, Aug 5, 2010 at 12:42 PM, Sam Roberts vieuxt...@gmail.com wrote: FTS3 only searches full terms/words by default, but I think if I built a custom

Re: [sqlite] [PATCH] Fix locale-unsafe usage of tolower (upstreaming Chromium change)

2010-08-04 Thread Scott Hess
Pawel, You should probably pull the current SQLite code and make sure the patch even applies, and if not, check to make sure that the problem hasn't already been fixed. ext/fts3 should no longer have the flaw in question, as that code was heavily rewritten. Chromium's SQLite was last synced

Re: [sqlite] Optimizing Songbird

2010-08-02 Thread Scott Hess
On Mon, Aug 2, 2010 at 11:11 AM, Simon Slavin slav...@bigfraud.org wrote: On 2 Aug 2010, at 7:06pm, Richard Hipp wrote: Most of the slow queries seem to be SELECT count(*) FROM .  Such queries have to visit every row in the table (in order to count the rows) and thus get progressively

Re: [sqlite] Optimizing Songbird

2010-08-02 Thread Scott Hess
[Sorry for the mis-fire.] On Mon, Aug 2, 2010 at 11:11 AM, Simon Slavin slav...@bigfraud.org wrote: On 2 Aug 2010, at 7:06pm, Richard Hipp wrote: Most of the slow queries seem to be SELECT count(*) FROM .  Such queries have to visit every row in the table (in order to count the rows) and

Re: [sqlite] [PATCH] Verify number of arguments in icuRegexpFunc

2010-07-30 Thread Scott Hess
PM, Scott Hess wrote: On Thu, Jul 29, 2010 at 10:05 PM, Dan Kennedy danielk1...@gmail.com wrote: On Jul 30, 2010, at 8:39 AM, Paweł Hajdan, Jr. wrote: I'm attaching a suggested patch to verify number of arguments in icuRegexpFunc. Please review it. This is upstreaming of http

Re: [sqlite] [PATCH] Verify number of arguments in icuRegexpFunc

2010-07-29 Thread Scott Hess
On Thu, Jul 29, 2010 at 10:05 PM, Dan Kennedy danielk1...@gmail.com wrote: On Jul 30, 2010, at 8:39 AM, Paweł Hajdan, Jr. wrote: I'm attaching a suggested patch to verify number of arguments in icuRegexpFunc. Please review it. This is upstreaming of

Re: [sqlite] Database corruption on Linux ext3

2010-07-13 Thread Scott Hess
On Tue, Jul 13, 2010 at 8:24 PM, Simon Slavin slav...@bigfraud.org wrote: It might be useful to figure out whether we're aiming for detection or correction. By 'correction' I don't mean recovery of all information, I mean restoring the database to some state it was in just after a COMMIT took

Re: [sqlite] Database corruption on Linux ext3

2010-07-13 Thread Scott Hess
On Tue, Jul 13, 2010 at 10:03 PM, Scott Hess sh...@google.com wrote: On Tue, Jul 13, 2010 at 8:24 PM, Simon Slavin slav...@bigfraud.org wrote: It might be useful to figure out whether we're aiming for detection or correction.  By 'correction' I don't mean recovery of all information, I mean

Re: [sqlite] linear behavior on INSERT (was Re: unexpected large journal file)

2010-06-22 Thread Scott Hess
Eric, You should reduce your demonstration case to something you'd be willing to post the code for. Probably using synthetic data (preferably generated data). There's something missing in the thread right now, and it's unlikely to be exposed by random shots in the dark. -scott On Tue, Jun

Re: [sqlite] unexpected large journal file

2010-06-18 Thread Scott Hess
On Fri, Jun 18, 2010 at 5:24 PM, Eric Smith eas@gmail.com wrote: Jay A.  Kreibich wrote: Yes.  Hence the and this is the important part comment.  Most of the time when people are building billion-row files, they're building a new DB by importing a static source of data.  If things go

Re: [sqlite] Is it possible to generate tf-idf matrix from the FTS3 table?

2010-06-08 Thread Scott Hess
On Sat, Jun 5, 2010 at 10:29 AM, Han-Teng Liao hant...@gmail.com wrote:    I intend to use my existing datasets stored in sqlite3 database for some linguistic analysis for Chinese language. After I have successfully installed and run the FTS3 Extension and ICU Extension, I am curious whether

Re: [sqlite] MySQL vs. SQLite

2010-06-01 Thread Scott Hess
On Tue, Jun 1, 2010 at 2:37 PM, Simon Slavin slav...@bigfraud.org wrote: On 1 Jun 2010, at 7:11pm, Israel Lins Albuquerque wrote: About that future release functionality. Will be possible to know whats temporary index are created? Using that information will be easy to know what indexes we

Re: [sqlite] create virtual table if not exists table_id???

2010-05-12 Thread Scott Hess
On Wed, May 12, 2010 at 10:40 AM, Roger Binns rog...@rogerbinns.com wrote: On 05/11/2010 11:50 AM, Matt Young wrote: sqlite create virtual table if not exists words using fts3  (f1 ); Error: near not: syntax error  http://www.sqlite.org/cvstrac/tktview?tn=2604 To fix it requires code

Re: [sqlite] FTS3 Appropriate usage

2010-04-16 Thread Scott Hess
On Fri, Apr 16, 2010 at 3:24 AM, Alexey Pechnikov pechni...@mobigroup.ru wrote: And you can use my patches for zlib-compression for FTS3. I'm planning to make the fts3z extension because I want to use as original FTS3 as FTS3 with compression together. Back when I was working up fts1, I

Re: [sqlite] VACUUM journal size

2010-03-15 Thread Scott Hess
On Sun, Mar 14, 2010 at 5:18 PM, Jay A. Kreibich j...@kreibi.ch wrote: On Sun, Mar 14, 2010 at 07:19:59PM -0400, Matthew L. Creech scratched on the wall: I have a SQLite database with one large table, and I'd like to shrink the size of that table to free up space in the filesystem.  My

Re: [sqlite] VACUUM journal size

2010-03-15 Thread Scott Hess
On Mon, Mar 15, 2010 at 11:18 AM, Scott Hess sh...@google.com wrote: AFAICT, the operation to copy the pages back _is_ journaled, and the journal will get any pages which are overwritten in the front of the main database.  If the initial database has half of the pages used, it seems like

Re: [sqlite] Min(null) ; was: Re: if exist

2010-03-09 Thread Scott Hess
On Tue, Mar 9, 2010 at 7:15 AM, Tim Romano tim.rom...@yahoo.com wrote: Of these three: select c from T where 1=2                 // returns 0 rows select min(c) from T where 1=2         // returns 1 row select min(88,99) from T where 1=2  // returns 0 rows the only case that threw me is the

Re: [sqlite] Min(null) ; was: Re: if exist

2010-03-09 Thread Scott Hess
On Tue, Mar 9, 2010 at 10:46 AM, Tim Romano tim.rom...@yahoo.com wrote: On 3/9/2010 10:56 AM, Scott Hess wrote: On Tue, Mar 9, 2010 at 7:15 AM, Tim Romanotim.rom...@yahoo.com  wrote: Of these three: select c from T where 1=2                 // returns 0 rows select min(c) from T where 1=2

Re: [sqlite] FTS3 bug with MATCH plus OR

2010-03-03 Thread Scott Hess
I can't speak to the question of whether it's a real problem, but I bet you can work around with a sub-select. Something like: select * from myfts where rowid = 1 OR rowid IN (select rowid from myfts where (myfts MATCH 'one')); -scott On Wed, Mar 3, 2010 at 3:26 AM, Ralf Junker

Re: [sqlite] why is underscore like dash?

2010-03-02 Thread Scott Hess
On Tue, Mar 2, 2010 at 9:41 AM, Wilson, Ronald rwils...@harris.com wrote: sqlite select * from test where text like '_'; from http://www.sqlite.org/lang_expr.html An underscore (_) in the LIKE pattern matches any single character in the string. ___

Re: [sqlite] Is there an optimization for IS NULL operator in a SELECT query ?

2010-03-01 Thread Scott Hess
On Mon, Mar 1, 2010 at 2:44 PM, Jean-Christophe Deschamps j...@q-e-d.org wrote: The actual reason for the way NOT works as for now may be due to the fact that negating a condition may cause the resulting set to be in fact itself the union of two subsets. Say the where condition K is col =

Re: [sqlite] Is there an optimization for IS NULL operator in a SELECT query ?

2010-03-01 Thread Scott Hess
On Mon, Mar 1, 2010 at 3:12 PM, Jean-Christophe Deschamps j...@q-e-d.org wrote: NULL = 12345 is NULL, NOT NULL is NULL, so subset N is not part of NOT (col = 12345). You're right of course!  (and I was even saying about nulls treated apart) But, in your view, that the set can be

Re: [sqlite] Bug in porter stemmer

2010-02-24 Thread Scott Hess
Actually, I think a new version of the tokenizer would have to be a distinct tokenizer (ie, porter versus porter1 versus porter2, whatever). fts4 should not interpret the meaning of an explicit tokenizer differently from fts3, but it could use a different default tokenizer. [Don't take this as

Re: [sqlite] Interrupt first sqlite3_step in FTS3 query

2010-02-20 Thread Scott Hess
On Sat, Feb 20, 2010 at 4:28 AM, Max Vlasov max.vla...@gmail.com wrote: you mentioned full-text search. I just tried to search for mentioning of sqlite3_interrupt in the sqlite sources The main is the implemention of the function itself that just sets the isInterrupted variable: void

Re: [sqlite] Is there a way to predict autoincrement rowid range

2010-01-07 Thread Scott Hess
As mentioned, don't use raw rowid, define a my_id INTEGER PRIMARY KEY AUTOINCREMENT. This currently acts as an alias to rowid, but if the implementation changes in the future, it will continue to work as documented. So things will be correct, but they might not be as efficient as before.

Re: [sqlite] Unique column constrained to 0 .. 2^32?

2009-12-30 Thread Scott Hess
Since you're asking the question, it seems safe to assume that you expect the table to have on the order of 2^32 values? And that you'll be freeing up old values by deleting rows from the table? First off, don't use rowid. Use my_id INTEGER PRIMARY KEY or something like that. rowids are an

Re: [sqlite] SELECT from fts3 tokens, is it possible?

2009-12-28 Thread Scott Hess
. As the tokens are not columns of the virtual table, this syntax have to be different from general Select. On Wed, Dec 23, 2009 at 8:51 PM, Scott Hess sh...@google.com wrote: Doing this in the tokenizer is probably not great because you can't determine which of the tokens actually

Re: [sqlite] SELECT from fts3 tokens, is it possible?

2009-12-28 Thread Scott Hess
On Mon, Dec 28, 2009 at 8:19 PM, Dan Kennedy danielk1...@gmail.com wrote: If you want the data for other purposes, you could almost implement a hack to get is.  For instance, SELECT count(docid) FROM fts_table WHERE fts_table MATCH 'x'; is kind of close, and you could change the match to

Re: [sqlite] SELECT from fts3 tokens, is it possible?

2009-12-23 Thread Scott Hess
The tokens are all there, so it is theoretically possible. But the performance would be very bad because it would require iterating over all the segment data, basically a worst-case query. Even then, you wouldn't have information about hit counts. To do it reasonably would require redesigning

Re: [sqlite] SELECT from fts3 tokens, is it possible?

2009-12-23 Thread Scott Hess
, non-exising words (but ironically it would be helpful for collecting hit data). If we know inside xNext that the call from a real data appending then at least we will be able to maintain a dictionary of words used at least once. Max On Wed, Dec 23, 2009 at 5:54 PM, Scott Hess sh

[sqlite] Is the dbFileVers written to journal files?

2009-11-10 Thread Scott Hess
Someone was asking me a question about what happens if the wrong journal file gets applied to a database. My answer was that terrible things happen. I found myself wondering if the File change counter value is stored in the journal (*). It would seem like a cheap(ish) way to give some

Re: [sqlite] FTS3 IGNORE OR REPLACE????

2009-11-04 Thread Scott Hess
On Tue, Nov 3, 2009 at 9:12 PM, sorka sorka95...@gmail.com wrote: Is there any way to have an intsert into an FTS3 table ignore a row if the ROWID being inserted already exists? This is turning out to be quite troublesome because I'm inserting thousands of records where just a few like 3 or 4

Re: [sqlite] FTS3 IGNORE OR REPLACE????

2009-11-04 Thread Scott Hess
duplicates out of the hundreds of records so it will always fail. For whatever reason, the delete, even though it's just 2 or 3 records is taking 10 times longer than just the insert alone where I can artificially make a case where there are no duplicates. Scott Hess wrote: My experience

Re: [sqlite] FTS3 IGNORE OR REPLACE????

2009-11-04 Thread Scott Hess
fts3 columns are all implicitly TEXT, no matter how you dress them up. There's already docid as a primary-key alias for rowid. -scott On Wed, Nov 4, 2009 at 11:03 AM, Simon Slavin slav...@bigfraud.org wrote: On 4 Nov 2009, at 5:05pm, sorka wrote: Hmm. Have you actually tried this yourself?

Re: [sqlite] Roadmap for SQLite

2009-11-02 Thread Scott Hess
On Mon, Nov 2, 2009 at 3:56 PM, Shaun Seckman (Firaxis) shaun.seck...@firaxis.com wrote: The SQLite Consortium membership would be great but so far this newsgroup has answered every single one of my questions within a 2 hour window of posting..for free!  (you guys frickin rock.)  So SQLite

Re: [sqlite] The zlib-compression patch for FTS3

2009-11-01 Thread Scott Hess
I did some tests with compression back when writing fts. I don't recall if it was the fts1 time-frame or fts2 (which is mostly the same as fts3). It didn't end up integrated into fts because zlib isn't always available to SQLite. Metadata is sorted and the deltas are encoded, so it's already

Re: [sqlite] feature proposal - strong but dynamic typing

2009-10-30 Thread Scott Hess
On Fri, Oct 30, 2009 at 7:14 AM, P Kishor punk.k...@gmail.com wrote: Actually, there can be one bad effect of Darren's suggestion, now that I think of it, and that would be for those who don't care for strong typing. They will end up getting strong typing for all non-UNIVERSAL columns whether

Re: [sqlite] Grammar of X is Y

2009-10-28 Thread Scott Hess
On Tue, Oct 27, 2009 at 9:35 PM, John Crenshaw johncrens...@priacta.com wrote: Meh, I don't want it THAT badly. I'm just saying that's how it should have been in the original design of the SQL language. In fact though, it probably wouldn't have mattered. Every different RDBMS seems to treat

Re: [sqlite] FTS3 - closed ticket 3950 - bug still exists in 3.6.19

2009-10-26 Thread Scott Hess
The bug was closed with cannot reproduce. So IMHO either the bug should be re-opened with the provided repro case, or a new bug should be opened with the provided repro case. Then, if the team decides that this is works-as-designed, the team can close the bug with that resolution. When I

Re: [sqlite] Null-values in NATURAL JOIN

2009-10-23 Thread Scott Hess
NULL is the logical lack of a value, and is not the same as some other value EVERY SINGLE TIME. NULL is not equal to 1, NULL is not equal to 'one', NULL is also not equal to NULL. It's like NaN, except not-a-value. When you join using equality (or comparison) on a column which can have NULL

Re: [sqlite] Null-values in NATURAL JOIN

2009-10-23 Thread Scott Hess
Sigh. That last sentence sounds confusing, I meant it literally, in your where clause you tell the system how to deal with NULL cases using IS NOT and IS NOT NULL. -scott On Fri, Oct 23, 2009 at 7:37 AM, Scott Hess sh...@google.com wrote: NULL is the logical lack of a value

Re: [sqlite] Why FTS3 has the limitations it does

2009-10-22 Thread Scott Hess
On Mon, Oct 19, 2009 at 11:25 AM, John Crenshaw johncrens...@priacta.com wrote: On the other hand, we could add a tree inside each segment to index the doclist. The term would be looked up as normal at a cost of O(log nT). After that though, if the docid is known, it could be looked up at an

Re: [sqlite] Why FTS3 has the limitations it does

2009-10-19 Thread Scott Hess
Here's a long-ago thread on this: http://www.mail-archive.com/sqlite-users@sqlite.org/msg30540.html Looks like it hasn't been addressed, and I've yet to come up for air on it. There's a ticket out there which looks like the same thing: http://www.sqlite.org/cvstrac/tktview?tn=3338 which is

Re: [sqlite] Why FTS3 has the limitations it does

2009-10-19 Thread Scott Hess
On Sat, Oct 17, 2009 at 1:25 PM, John Crenshaw johncrens...@priacta.com wrote: Agreed, HUGE thanks for FTS. Hopefully my original post didn't come off ungrateful. I was just confused by limitations that looked like they could have been removed during the initial design (at least more easily

Re: [sqlite] FTS3 performance oddity

2009-10-09 Thread Scott Hess
Your analysis is correct. One way this is often fixed in full-text-search systems is to keep index stats so that the more specific bits of the query can come first. In this case, nicky AND nigel gives you a small enough result set that it would be more efficient to scan the matched documents for

Re: [sqlite] Stored procedures performance issue

2009-10-05 Thread Scott Hess
Not to be flippant, but how could SQLite optimize for the output of a custom function you define outside of SQLite? It can't possibly know anything about how IS_MY_BIRTHDAY() works. Your function could return 1 for every single row, or for no rows at all, without regard to whether you're passing

Re: [sqlite] multiple prepare statements

2009-09-30 Thread Scott Hess
Since SQLite is an embedded database, it generally does not pay to count statements, unless they add additional disk I/O. You can code like this: BEGIN SELECT ... if (select results A) INSERT ... else UPDATE ... END and it will be about as fast as either the INSERT

Re: [sqlite] multiple prepare statements

2009-09-30 Thread Scott Hess
On Wed, Sep 30, 2009 at 7:13 AM, Pavel Ivanov paiva...@gmail.com wrote: Well, assuming that your SELECT is selecting the rows you mean to UPDATE or INSERT ... Also assuming that - all data necessary for these statements can fit into sqlite's cache; Generally the desire to

Re: [sqlite] multiple prepare statements

2009-09-30 Thread Scott Hess
On Wed, Sep 30, 2009 at 11:58 AM, Sam Carleton scarle...@gmail.com wrote: On Wed, Sep 30, 2009 at 10:02 AM, Scott Hess sh...@google.com wrote: Since SQLite is an embedded database, it generally does not pay to count statements, unless they add additional disk I/O.  You can code like

  1   2   3   >