What do you do when you have to import a text file that equates to a
cursor/table too big (beyond 2GB)? I'm using MySQL's import which is
fine for such huge beasts, but the import isn't liking some of the field
formats (related to my last post but not exactly that) and so I was
going to use VF
On 2/21/2012 8:07 PM, MB Software Solutions, LLC wrote:
> What do you do when you have to import a text file that equates to a
> cursor/table too big (beyond 2GB)? I'm using MySQL's import which is
> fine for such huge beasts, but the import isn't liking some of the field
> formats (related to my
That's how I would do it, cut it in half or in quarters.
- Original Message -
From: "MB Software Solutions, LLC"
To: ProFox Email List
Cc:
Sent: Tuesday, February 21, 2012 11:37 PM
Subject: Re: Importing files > 2GB
On 2/21/2012 8:07 PM, MB Software Solutions, LL
On 2/22/2012 12:34 AM, Michael Madigan wrote:
> That's how I would do it, cut it in half or in quarters.
Yes, but is there another way or is that the ONLY option? How messy
would going the LLFF route be I wonder?
--
Mike Babcock, MCP
MB Software Solutions, LLC
President, Chief Software Archi
When dealing with files of that size, I've always done just that. Split the
file into reasonable chunks that I then process for whatever backend it is
destined for.
Michael Oke, II
661-349-6221
Contents of this and all messages are intended for their designated recipient.
On Feb 21, 2012, a
I guess it all depends whether this is a one-time thing or will be done
multiple times.
- Original Message -
From: "MB Software Solutions, LLC"
To: ProFox Email List
Cc:
Sent: Wednesday, February 22, 2012 12:40 AM
Subject: Re: Importing files > 2GB
On 2/22/2012 12:3
On 2/22/2012 12:42 AM, Michael Oke, II wrote:
> When dealing with files of that size, I've always done just that. Split the
> file into reasonable chunks that I then process for whatever backend it is
> destined for.
How are you inserting the rows into the final destination backend?
Let's say
On 2/22/2012 12:55 AM, MB Software Solutions, LLC wrote:
> On 2/22/2012 12:42 AM, Michael Oke, II wrote:
>> When dealing with files of that size, I've always done just that. Split the
>> file into reasonable chunks that I then process for whatever backend it is
>> destined for.
>
>
> How are you
> I was thinking about saving to a CSV file again and then inserting into
> the MySQL table from that, with the fields fixed that had issue prior.
>
If wasting time is not a problem, you can use a loop to split the input file
into multiple cursors, then use them your way out :
[VFP]
*** creati
I don't know how long, but.
Using fgets, you can get each line. Parse that, and place into a vfp
table/cursor.
Once you have a large group,. under the 2G barrier, post to mySQL.
Zap the VFP table/cursor.
Pick up your low level file read, and repeat.
Might work?
Mike
On 21/02/2012 11:37 PM, MB So
On 2/22/2012 2:24 AM, AndyD wrote:
> llfs in Fox is *fast* - and very flexible - that's the way I'd go. Read
> in a chunk, take a substring up to the last whole record, work with that
> then read on from that offset, rinse& repeat.
>
> "Here's a potential gotcha...there's no clearly defined PK for
You could alter your vfp table and add a column called PK, fill it with
recno() and index it. Now you have your PK.
You could also split the big file in reasonably sized chunks as
suggested and do the above. It would be much faster
Rafael Copquin
El 22/02/2012 02:30 p.m., MB Software Solutions
On 2/22/2012 3:51 PM, Rafael Copquin wrote:
> You could alter your vfp table and add a column called PK, fill it with
> recno() and index it. Now you have your PK.
> You could also split the big file in reasonably sized chunks as
> suggested and do the above. It would be much faster
Good idea, Ra
Or there's a premature EOF character?
Mike
Original Message
Subject: Re: Importing files > 2GB
From: MB Software Solutions, LLC
To: profoxt...@leafe.com
Date: 2/22/2012 3:18 PM
On 2/22/2012 3:51 PM, Rafael Copquin wrote:
> You could alter your vfp table and a
I don't think the VFP LLFF work beyond 2GB.
Yep, it's in the Visual FoxPro System Capacities under miscellaneous. 2GB
max.
Fred
On Wed, Feb 22, 2012 at 2:18 PM, MB Software Solutions, LLC <
mbsoftwaresoluti...@mbsoftwaresolutions.com> wrote:
> On 2/22/2012 3:51 PM, Rafael Copquin wrote:
> > Y
On 2/22/2012 4:37 PM, Fred Taylor wrote:
> I don't think the VFP LLFF work beyond 2GB.
>
> Yep, it's in the Visual FoxPro System Capacities under miscellaneous. 2GB
> max.
I figured as such. So what now, Mr. Peabody? Do we jump into way-back
machine? :-) (Nearly everyone here is old enough
On Wed, Feb 22, 2012 at 4:08 PM, MB Software Solutions, LLC
wrote:
> On 2/22/2012 4:37 PM, Fred Taylor wrote:
>> I don't think the VFP LLFF work beyond 2GB.
>>
>> Yep, it's in the Visual FoxPro System Capacities under miscellaneous. 2GB
>> max.
>
>
> I figured as such. So what now, Mr. Peabody?
On Feb 22, 2012, at 4:08 PM, MB Software Solutions, LLC wrote:
>> I don't think the VFP LLFF work beyond 2GB.
>>
>> Yep, it's in the Visual FoxPro System Capacities under miscellaneous. 2GB
>> max.
>
> I figured as such. So what now, Mr. Peabody? Do we jump into way-back
> machine? :-) (Ne
Sounds like you'll need to find a tool that can break up the file into <2GB
chunks and process the pieces. Or you may be able to use a language like
Python to parse the file and use ODBC to add the values to a VFP table(s).
Fred
On Wed, Feb 22, 2012 at 3:08 PM, MB Software Solutions, LLC <
mbso
ssage
Subject: Re: Importing files > 2GB
From: Fred Taylor
To: profoxt...@leafe.com
Date: 2/22/2012 5:57 PM
Sounds like you'll need to find a tool that can break up the file into<2GB
chunks and process the pieces. Or you may be able to use a language like
Python to parse the file and use
On 2/22/2012 6:57 PM, Fred Taylor wrote:
> Sounds like you'll need to find a tool that can break up the file into<2GB
> chunks and process the pieces. Or you may be able to use a language like
> Python to parse the file and use ODBC to add the values to a VFP table(s).
I might as well just go ri
On 2/22/2012 7:03 PM, Mike Copeland wrote:
> I would recommend a great program (free for personal use, too) called
> EditPad.
> www.editpadlite.com
>
> It will open files as big as the OS will store. It also doesn't gag and
> get sluggish when dealing with large files. Edits ASCII as well as
On Wed, Feb 22, 2012 at 4:55 PM, Ed Leafe wrote:
> On Feb 22, 2012, at 4:08 PM, MB Software Solutions, LLC wrote:
>
>>> I don't think the VFP LLFF work beyond 2GB.
>>>
>>> Yep, it's in the Visual FoxPro System Capacities under miscellaneous. 2GB
>>> max.
>>
>> I figured as such. So what now, Mr.
Using a modern tool, split the file. Using old VFP low-level functions do your
thing.
Just my (what was that ?), E.
>>> I don't think the VFP LLFF work beyond 2GB.
Yep, it's in the Visual FoxPro System Capacities under miscellaneous. 2GB
max.
>>>
>>> I figured as such. So what
llfs in Fox is *fast* - and very flexible - that's the way I'd go. Read
in a chunk, take a substring up to the last whole record, work with that
then read on from that offset, rinse & repeat.
"Here's a potential gotcha...there's no clearly defined PK for these rows."
Write-only storage then?
A
You could add an auto-increment field?
AndyD 8-)₹
On 22/02/2012 17:30, MB Software Solutions, LLC wrote:
> On 2/22/2012 2:24 AM, AndyD wrote:
>> llfs in Fox is *fast* - and very flexible - that's the way I'd go. Read
>> in a chunk, take a substring up to the last whole record, work with that
>> th
I guess it's not the Fox but Windows; 2Gb being the limit of signed
32-bit addressing - if I counted properly.
Don't think they're going to give us a 64-bit Foxpro.
It does raise the question as to what tools *can* handle an address
space over 32-bits wide on 32-bit architecture!
AndyD 8-)₹
On
27 matches
Mail list logo
