Re: [sqlite] Please help me fix the SQLite Git mirror

2019-04-22 Thread Rowan Worth
Richard Hipp wrote (quoting from several emails): > The problem is that Git now thinks that 9b888fcc is the HEAD of master > and that the true continuation of master (check-in 4f35b3b7 and > beyond) are disconnected check-ins > Because from the git perspective it _is_ still the HEAD -- there's

[sqlite] 1st Call For Papers - 26th Annual Tcl/Tk Conference (Tcl'2019)

2019-04-22 Thread conference
Hello SQLite Users, fyi ... 26th Annual Tcl/Tk Conference (Tcl'2019) https://www.tcl-lang.org/community/tcl2019/ November 04 - 08, 2019 Crowne Plaza Houston River Oaks 2712 Southwest Freeway, 77098 Houston, Texas, USA [ NEWS * [Submission is

[sqlite] Best way to ALTER COLUMN ADD multiple tables if the columns don't already exist

2019-04-22 Thread Tommy Lane
Hi all, I'm still working on this journal app and ran across a need to update our table schema, I want my entries to be _essentially_ immutable. My solution to this problem is a linked list type dependency where each entry has a parent and a child, which corresponds to an entry's past and

Re: [sqlite] Multiple Independent Database Instances

2019-04-22 Thread Keith Medcalf
Interesting. If you can guarantee that you will only have a single thread accessing a single database only from one single thread, give it a try with SQLITE_OPEN_NOMUTEX | SQLITE_OPEN_READONLY in the flags parameter of sqlite3_open_v2 ... Don't know if it will make a difference, but it

Re: [sqlite] Multiple Independent Database Instances

2019-04-22 Thread Simon Slavin
On 23 Apr 2019, at 12:32am, Lee, Jason wrote: > The current code is effectively just an sqlite3_open_v2 followed by an > sqlite3_close Then either your code is faulty, and doesn't actually do this, or your problem has nothing to do with SQLite. SQLite doesn't open a database file when you

Re: [sqlite] Multiple Independent Database Instances

2019-04-22 Thread Lee, Jason
> How does each thread know whether the file has been "previously processed" or > not? The paths are pushed onto a queue and each thread pops the top off. I am also looking into the queuing code to see if there are issues > In other words, if you "get rid of" all the sqlite3 processing and

Re: [sqlite] Multiple Independent Database Instances

2019-04-22 Thread James K. Lowden
On Mon, 22 Apr 2019 21:25:31 + "Lee, Jason" wrote: > I have a set of several million database files sitting on my > filesystem. Each thread will open a previously unprocessed database > file, do some queries, close the database, and move on to the next > unprocessed database file.

Re: [sqlite] Multiple Independent Database Instances

2019-04-22 Thread Keith Medcalf
Interesting ... How does each thread know whether the file has been "previously processed" or not? In other words, if you "get rid of" all the sqlite3 processing and replace it with a 5 ms sleep, does increasing the number of threads exhibit the same symptom? That is if your thread code

Re: [sqlite] Multiple Independent Database Instances

2019-04-22 Thread Simon Slavin
On 22 Apr 2019, at 10:25pm, Lee, Jason wrote: > I have a set of several million database files sitting on my filesystem. Each > thread will open a previously unprocessed database file, do some queries, > close the database, and move on to the next unprocessed database file. If this process is

Re: [sqlite] Multiple Independent Database Instances

2019-04-22 Thread Lee, Jason
I have a set of several million database files sitting on my filesystem. Each thread will open a previously unprocessed database file, do some queries, close the database, and move on to the next unprocessed database file. Jason Lee From: sqlite-users on

Re: [sqlite] Multiple Independent Database Instances

2019-04-22 Thread Keith Medcalf
This is somewhat unclear. You make two conflicting statements: "I have been testing with 16, 32, and 48 threads/databases at once ..." and "time it takes for all of the threads to just open all (millions) of the databases" So, are you: (a) opening one independently and uniquely named database

Re: [sqlite] Multiple Independent Database Instances

2019-04-22 Thread Simon Slavin
On 22 Apr 2019, at 9:08pm, Lee, Jason wrote: > the cumulative time it takes for all of the threads to just open all > (millions) of the databases goes from 1200 seconds to 2200 seconds to 3300 > seconds. I'm guessing that it's the number of file handles which increases. Most OSes maintain a

Re: [sqlite] Please help me fix the SQLite Git mirror

2019-04-22 Thread Carl Edquist
I would like to better understand how rewiring the refs this way constitutes "changing history". The refs/heads entries are all ephemeral - they are constantly changing on their own, and no historical record of their past values is retained. The key bit here is that in git, every commit

Re: [sqlite] Multiple Independent Database Instances

2019-04-22 Thread Lee, Jason
Thanks for the quick responses! I am on a machine with many many cores, 500GB RAM, and lots of NVMe drives raided together, so the system should not be the issue. I have been testing with 16, 32, and 48 threads/databases at once, and the cumulative time it takes for all of the threads to just

Re: [sqlite] Please help me fix the SQLite Git mirror

2019-04-22 Thread Carl Edquist
Hi Richard, As Jonathan mentioned, in git land, if you have already published a "mistake" commit publicly, the proper way to revert it is to make another commit to reverse/undo the change. By removing a commit from the public history of the published 'master' branch, it forces everyone

Re: [sqlite] Please help me fix the SQLite Git mirror

2019-04-22 Thread Jonathan Brandmeyer
On Mon, Apr 22, 2019 at 12:22 PM Richard Hipp wrote: > But before I proceed, I would like to better understand how rewiring > the refs this way constitutes "changing history". The refs/heads > entries are all ephemeral - they are constantly changing on their own, > and no historical record of

Re: [sqlite] Multiple Independent Database Instances

2019-04-22 Thread Jens Alfke
> On Apr 22, 2019, at 11:39 AM, Lee, Jason wrote: > > Hi. Are there any gotchas when opening multiple independent databases from > within one process using the C API? Do you mean different database files, or multiple connections to the same file? > I am opening one database per thread in my

Re: [sqlite] Multiple Independent Database Instances

2019-04-22 Thread Simon Slavin
On 22 Apr 2019, at 7:39pm, Lee, Jason wrote: > Hi. Are there any gotchas when opening multiple independent databases from > within one process using the C API? I am opening one database per thread in > my code, and noticed that sqlite3_open_v2 and sqlite3_close slow down as the > number of

Re: [sqlite] SQLITE_MAX_MMAP_SIZE 2GB default

2019-04-22 Thread Jens Alfke
> On Apr 19, 2019, at 12:46 PM, Carl Edquist wrote: > > For instance - if you have a 30GB db file on a 64bit system with <= 2GB ram, > you can still mmap the whole file, and benefit from that mmap. If the > portion of the db that gets used for a query fits within the available > pagecache

[sqlite] Multiple Independent Database Instances

2019-04-22 Thread Lee, Jason
Hi. Are there any gotchas when opening multiple independent databases from within one process using the C API? I am opening one database per thread in my code, and noticed that sqlite3_open_v2 and sqlite3_close slow down as the number of threads increase, indicating there might be some resource

Re: [sqlite] Please help me fix the SQLite Git mirror

2019-04-22 Thread Richard Hipp
Thanks for the help. See additional questions and remarks below On 4/22/19, Jonathan Brandmeyer wrote: > ``` > # Construct the matching branch name > git branch mistake 9b888fc > # Push the name alone to the remote > git push -u origin mistake > # Move the name of master > git checkout

Re: [sqlite] Please help me fix the SQLite Git mirror

2019-04-22 Thread Jonathan Brandmeyer
``` # Construct the matching branch name git branch mistake 9b888fc # Push the name alone to the remote git push -u origin mistake # Move the name of master git checkout master && git reset --hard # Push the new name of master git push --force ``` Git reset --hard will move the name of the

Re: [sqlite] Please help me fix the SQLite Git mirror

2019-04-22 Thread Richard Hipp
On 4/22/19, Jeffrey Schiller wrote: > So if I understand you correctly, you just want to make "master" point to a > particular known commit. To do this, you can issue the commands (in a local > copy): > > git branch -m master oldmaster # Move it out of the way > git branch master 4f35b3b7 > This

Re: [sqlite] Please help me fix the SQLite Git mirror

2019-04-22 Thread Jeffrey Schiller
So if I understand you correctly, you just want to make "master" point to a particular known commit. To do this, you can issue the commands (in a local copy): git branch -m master oldmaster # Move it out of the way git branch master 4f35b3b7 Then do a "git push -f origin master" (assuming that

[sqlite] Please help me fix the SQLite Git mirror

2019-04-22 Thread Richard Hipp
The Git mirror of SQLite found at https://github.com/sqlite/sqlite is busted. I don't know how to fix it and would appreciate advice from people who have more experience with Git internals. To describe the problem, consider this excerpt from the check-in sequence for SQLite:

Re: [sqlite] SQLite error while fetching the data from a table

2019-04-22 Thread Tommy Lane
Hi All, Hi Ananta Need quick help to resolve one issue i am getting now. I am a new user of SQLite. my code: connection = DriverManager.getConnection("jdbc:sqlite:C:\\sqllite\\sqlite-tools-win32-x86-328\\Stories.db"); Statement st = connection.createStatement(); ResultSet b =

Re: [sqlite] "Table Not Found" when multiple reader processes written in C++ accessing the same DB file in Ubuntu

2019-04-22 Thread Simon Slavin
On 22 Apr 2019, at 3:19pm, Polly Tang wrote: > I have an urgent issue with multiple reader processes in C++ accessing the > same DB file in Ubuntu and all reader experience "Table Not Found". /All/ say "Table Not Found" ? Including if you open just one reader process ? Are you sure your

[sqlite] "Table Not Found" when multiple reader processes written in C++ accessing the same DB file in Ubuntu

2019-04-22 Thread Polly Tang
Hi : I have an urgent issue with multiple reader processes in C++ accessing the same DB file in Ubuntu and all reader experience "Table Not Found". Do I need special configuration in Sqlite3 to make it accessible by different processes ? This is a DB file created and stored inside a file

Re: [sqlite] SQLite error while fetching the data from a table

2019-04-22 Thread Luuk
On 22-4-2019 14:03, Ananta Jena wrote: Hi All, Need quick help to resolve one issue i am getting now. I am a new user of SQLite. my code: connection = DriverManager.getConnection("jdbc:sqlite:C:\\sqllite\\sqlite-tools-win32-x86-328\\Stories.db"); Statement st =

[sqlite] SQLite error while fetching the data from a table

2019-04-22 Thread Ananta Jena
Hi All, Need quick help to resolve one issue i am getting now. I am a new user of SQLite. my code: connection = DriverManager.getConnection("jdbc:sqlite:C:\\sqllite\\sqlite-tools-win32-x86-328\\Stories.db"); Statement st = connection.createStatement(); ResultSet b = st.executeQuery("select