Re: [sqlite] A kind help-request.

Hi Simon, Olaf , Garry, Great news to you all,now i can able to insert 65k docs in second, only i had done as per Simon and Garry suggestion 1. i freed the array as per suggestion 2. remove 'cmd.Parameters.Add(myparam) as per Garry suggestion. *here are summary code :inserting 65k in 4-5 sec* *

Re: [sqlite] A kind help-request.

Alok Singh wrote on 1/7/2011 : > Hi Guys , > > I again stuck with this. kindly give your suggestion. > Its taking time for to insert 65k of docs around 20 min. > kindly have a look and help me on my basic code.hope this will help you all > over my problem. > i think something going to have huge me

Re: [sqlite] A kind help-request.

On 7 Jan 2011, at 9:55pm, Alok Singh wrote: > i think something going to have huge memory or becoming lengthy here, so > after going > 1 in few sec > next 10001 to 2 in 1min > next 20001 to 3 in more than 1.5 min > next 30001 to 4 in more than 3 min > next 40001 to 5 in more t

Re: [sqlite] A kind help-request.

Hi Guys , I again stuck with this. kindly give your suggestion. Its taking time for to insert 65k of docs around 20 min. kindly have a look and help me on my basic code.hope this will help you all over my problem. i think something going to have huge memory or becoming lengthy here, so after goin

Re: [sqlite] A kind help-request.

Hi Guys, whatever i did , it was very simple as Simon suggested, my project something which will have 50k to millions of record lines, in 2 -3 or 4 text files, each having same no. of records i.e each will have 50k to millions of record lines ,there is one common filed such as Doc-id which work a

Re: [sqlite] A kind help-request.

Simon Slavin explained on 1/6/2011 : > On 7 Jan 2011, at 3:55am, GS wrote: > >> The irony is that my data files don't use commas or tabs as delimiters >> (for good reasons) and so I can't use SQLite's commandline import >> feature. > > You can change the separator it uses. Look for the '.separat

Re: [sqlite] A kind help-request.

On 7 Jan 2011, at 3:55am, GS wrote: > The irony is that my data files don't use commas or tabs as delimiters > (for good reasons) and so I can't use SQLite's commandline import > feature. You can change the separator it uses. Look for the '.separator' command in

Re: [sqlite] A kind help-request.

Simon Slavin explained on 1/6/2011 : > http://www.tech-archive.net/Archive/Data/microsoft.public.data.ado/2006-11/msg00033.html Thanks again. I appreciate you taking the time to find this code. This offering from RB writes one record at a time, which is what I'd do currently either from an array

Re: [sqlite] A kind help-request.

Thanks, Simon, for the enlightenment! Sorry for assuming you were familiar with Micro$oft db terminology. Olaf has tuned me up on the fact that SQLite is used with many other languages (not necessarily Micro$oft languages). Olaf was also able to steer me toward a solution via his SQLite wrapper

Re: [sqlite] A kind help-request.

Thanks, Olaf! That's very helpful. I will certainly look at your demo. As for the enlightening experience of browsing this forum, I connect to it back when I first downloaded your RichClient package. While I haven't read them all, to date I have over 62K posts to use for learning. That said, I

Re: [sqlite] A kind help-request.

"GS" schrieb im Newsbeitrag news:ig582d$go...@dough.gmane.org... > > That's memory-intensive - but "Ok" (and fast) for Testfiles > > with that RowCount (filesize of your 10.5K-Rows > > testfiles around 4-6MB I'd guess). > > > I just checked a 21,000 line x 30 column delimited file > and it is

Re: [sqlite] A kind help-request.

On 7 Jan 2011, at 1:08am, GS wrote: > Recordsets are common db programming objects, and so I'm a bit awe > struck that you wouldn't know what I mean by the term being as you seem > to be very knowledgeable in SQLite. I'm assuming (rightly or wrongly) > then that you're familiar with db termino

Re: [sqlite] A kind help-request.

Simon Slavin wrote : > Ah, okay. First, I don't understand what you mean by 'recordset'. Hi Simon, Thanks for the info. I understand about the command line feature tool and CSV/TSV file import. Recordsets are common db programming objects, and so I'm a bit awe struck that you wouldn't know wha

Re: [sqlite] A kind help-request.

On 6 Jan 2011, at 11:59pm, GS wrote: > I'm not worried about anything. I merely wish to know if we can create > a new table in SQLite and 'dump' a recordset into it. So if data was > 'dumped' from a database into a delimited file I'm guessing you mean a file in comma-delimited or tab-delimited

Re: [sqlite] A kind help-request.

Simon Slavin formulated the question : > On 6 Jan 2011, at 8:18pm, GS wrote: > >> I just checked a 21,000 line x 30 column delimited file and it is >> 817KB. I draw the line (for performance and/or convenience working with >> the data) at about 50K lines before I'd use ADO to load the entire file

Re: [sqlite] A kind help-request.

On 6 Jan 2011, at 8:18pm, GS wrote: > I just checked a 21,000 line x 30 column delimited file and it is > 817KB. I draw the line (for performance and/or convenience working with > the data) at about 50K lines before I'd use ADO to load the entire file > into a recordset, OR criteria-specific r

Re: [sqlite] A kind help-request.

Olaf Schmidt pretended : > "Alok Singh" schrieb > im Newsbeitrag > news:aanlktikhcyfsuybpjtv=+cd4asrddt-9+f7qx_qpq...@mail.gmail.com... > >> yeah that's correct Simon, its in 0.6 sec to insert >> for 10.5K rows with 20 columns (2 files both >> having 10.5k rows) > > That's the timing I would expec

Re: [sqlite] A kind help-request.

"Alok Singh" schrieb im Newsbeitrag news:aanlktikhcyfsuybpjtv=+cd4asrddt-9+f7qx_qpq...@mail.gmail.com... > yeah that's correct Simon, its in 0.6 sec to insert > for 10.5K rows with 20 columns (2 files both > having 10.5k rows) That's the timing I would expect, if you'd have used Garrys recommen

Re: [sqlite] A kind help-request.

"Simon Slavin" schrieb im Newsbeitrag news:8e8a9b9d-fd6c-4e4f-b43e-21f5fd846...@bigfraud.org... > Alok is writing in a language which doesn't compile very well. Nah, even with VB.NET it should be possible, to bring the import "up to speed" ... at least a factor 20 improvement should be achievabl

Re: [sqlite] A kind help-request.

yeah that's correct Simon, its in 0.6 sec to insert for 10.5K rows with 20 columns (2 files both having 10.5k rows) Thanks alot :-) That was my Typo.. ;-) Perfect. On 6 January 2011 19:57, Olaf Schmidt wrote: > > "Alok Singh" schrieb > im Newsbeitrag > news:aanlktinopo0+3pzk77d=ov0kbgmd2j7p3ah3

Re: [sqlite] A kind help-request.

On 6 Jan 2011, at 2:27pm, Olaf Schmidt wrote: > "Alok Singh" schrieb > im Newsbeitrag > news:aanlktinopo0+3pzk77d=ov0kbgmd2j7p3ah3-jjfg...@mail.gmail.com... > > [Importing from Text-Files] >> ... that's *PERFECT* , much faster than my expectation >> 0.6 min can able to access 10.5k row records.

Re: [sqlite] A kind help-request.

"Alok Singh" schrieb im Newsbeitrag news:aanlktinopo0+3pzk77d=ov0kbgmd2j7p3ah3-jjfg...@mail.gmail.com... [Importing from Text-Files] > ... that's *PERFECT* , much faster than my expectation > 0.6 min can able to access 10.5k row records. If that's not a typo (and you mean 0.6 seconds instead) -

Re: [sqlite] A kind help-request.

Hi Simon, Perfect, i followed as whatever you advised me, that's *PERFECT* , much faster than my expectation 0.6 min can able to access 10.5k row records. superb i like that. again many thanks,and i appreciate your kind support. * Regards, Alok * On 4 January 2011 04:32, GS wrote: > Simon Slavi

Re: [sqlite] A kind help-request.

Simon Slavin was thinking very hard : > On 3 Jan 2011, at 9:08pm, GS wrote: > >> Looping through the array gives me access to each record if I start the >> loop at vaDataArray(1), thus the loop parameters of '1 To >> UBound(vaDataArray)'. > > I have twice suggest you do not do this. Holding the

Re: [sqlite] A kind help-request.

On 3 Jan 2011, at 9:08pm, GS wrote: > Looping through the array gives me access to each record if I start the > loop at vaDataArray(1), thus the loop parameters of '1 To > UBound(vaDataArray)'. I have twice suggest you do not do this. Holding the entire data in memory at the same time will t

Re: [sqlite] A kind help-request.

It happens that Alok Singh formulated : > Hi Garry, > > can you show me with code how You are inserting with maintaining 1st row as > to make header of table ,and next other inserting fastest way into db... > please show me your code view for that. little bit i m confuse over its > inserting proces

Re: [sqlite] A kind help-request.

Hi Garry, can you show me with code how You are inserting with maintaining 1st row as to make header of table ,and next other inserting fastest way into db... please show me your code view for that. little bit i m confuse over its inserting process into db directly into db. mine still its slow

Re: [sqlite] A kind help-request.

Simon Slavin wrote : > Also you are still using a 2D array to store your values. This is slow and > requires a lot of memory. Instead, use one array to read in the first row > (with the names of the columns) and then use a second 1D array for each row > of values: read one line in then write t

Re: [sqlite] A kind help-request.

Thanks alot Simon , let me do as you advised me, will update you if that help me. have a nice day again many thanks :-), Regards, Alok On 3 January 2011 04:18, Simon Slavin wrote: > > On 2 Jan 2011, at 10:33pm, Alok Singh wrote: > > > i am doing the same inserting line by line using Transaction

Re: [sqlite] A kind help-request.

On 2 Jan 2011, at 10:33pm, Alok Singh wrote: > i am doing the same inserting line by line using Transaction but its like > dead code as its very slow where as my system config is quite heavy then > also. Sorry, I don't use your language so I can't help with your code. However, I do notice that

Re: [sqlite] A kind help-request.

Hi Simon, Thanks alot for your kind response. i am doing the same inserting line by line using Transaction but its like dead code as its very slow where as my system config is quite heavy then also. look into my code please : acc. to me i am doing same as you mean : Public Function insert_sqli

Re: [sqlite] A kind help-request.

On 2 Jan 2011, at 10:11pm, Simon Slavin wrote: > It will take less memory and be faster if you do it line by line: read one > line of the .csv file, then write one row to your SQLite table, then read the > next line. Sorry, I forgot: SQLite will make changes faster if you bundle them all toge

Re: [sqlite] A kind help-request.

On 2 Jan 2011, at 3:44pm, Alok Singh wrote: > My question is: Can i will able to Insert /Import Tab delimited files into > data base directly using VB.NET with ADONET connectivity, as right now i am > doing through reading files in 2-D array and inserting them line by line. > but its taking huge

[sqlite] A kind help-request.

Hi Sir, I am just a software Graduate , need your help over SQLite functionality. My question is: Can i will able to Insert /Import Tab delimited files into data base directly using VB.NET with ADONET connectivity, as right now i am doing through reading files in 2-D array and inserting them line