Re: [sqlite] Producing RFC4180-compliant CSV output

2014-08-06 Thread Peter Waller
On 4 August 2014 16:39, Edward Lau elau1...@aim.com wrote: Specifically, if you are going to use SQL Server Integration Services (SSIS) you can specify LF as the record delimiter. It defaults to {CR}{LF} but you can change it in the FlatFile Connection Manager under the Advance tab. Thanks

Re: [sqlite] Producing RFC4180-compliant CSV output

2014-08-04 Thread Edward Lau
...@scraperwiki.com To: sqlite-users sqlite-users@sqlite.org Cc: developers develop...@scraperwiki.com Sent: Thu, Jul 24, 2014 2:27 am Subject: [sqlite] Producing RFC4180-compliant CSV output Hi All, We're using `sqlite -csv` to generate CSV files because it is the best performing thing available to us

Re: [sqlite] Producing RFC4180-compliant CSV output

2014-07-25 Thread Peter Waller
That was quick, thanks! :) On 24 July 2014 13:21, Richard Hipp d...@sqlite.org wrote: On Thu, Jul 24, 2014 at 7:47 AM, Richard Hipp d...@sqlite.org wrote: On Thu, Jul 24, 2014 at 6:15 AM, Peter Waller pe...@scraperwiki.com wrote: Unfortunately, we already tried this, sed, and

[sqlite] Producing RFC4180-compliant CSV output

2014-07-24 Thread Peter Waller
Hi All, We're using `sqlite -csv` to generate CSV files because it is the best performing thing available to us. It turns out however that the CSV spec demands CRLF line endings [1]. There is software in the wild that barfs on non-CRLF-line-ending files, such as SQL Server Integration Services

Re: [sqlite] Producing RFC4180-compliant CSV output

2014-07-24 Thread Hick Gunter
How about piping your csv file through unix2dos? -Ursprüngliche Nachricht- Von: Peter Waller [mailto:pe...@scraperwiki.com] Gesendet: Donnerstag, 24. Juli 2014 11:27 An: sqlite-users@sqlite.org Cc: developers Betreff: [sqlite] Producing RFC4180-compliant CSV output Hi All, We're using

Re: [sqlite] Producing RFC4180-compliant CSV output

2014-07-24 Thread Richard Hipp
On Thu, Jul 24, 2014 at 5:46 AM, Hick Gunter h...@scigames.at wrote: How about piping your csv file through unix2dos? Yeah. Having a unix program generate \r\n line endings just seems wrong. Standard or no standard. -- D. Richard Hipp d...@sqlite.org

Re: [sqlite] Producing RFC4180-compliant CSV output

2014-07-24 Thread Peter Waller
Unfortunately, we already tried this, sed, and writing our own translation program in C. In the best case it doubled (!) the CPU-time usage and ate more than one core at a time, which isn't acceptable in our environment. Making the change in place in the sqlite source has no observable impact.

Re: [sqlite] Producing RFC4180-compliant CSV output

2014-07-24 Thread Stephen Chrzanowski
Of course, I've not seen the code, but, I'd throw in that a command line option be added that'll force a particular mode. By default, it is 'as is' right now being platform dependent, but, if a CLO is added, either mode is forced, or toggled. I'd ask for separate toggles for forced or toggle. :]

Re: [sqlite] Producing RFC4180-compliant CSV output

2014-07-24 Thread Richard Hipp
On Thu, Jul 24, 2014 at 6:15 AM, Peter Waller pe...@scraperwiki.com wrote: Unfortunately, we already tried this, sed, and writing our own translation program in C. The other problem is that it adds \r in front of *ever* \n, including those that are part of data and which are not record

Re: [sqlite] Producing RFC4180-compliant CSV output

2014-07-24 Thread David Empson
On 24/07/2014, at 10:15 pm, Peter Waller pe...@scraperwiki.com wrote: I too am sad that CRLF is mandated in the specification and still in wide use because of Windows. It may be nothing to do with Windows. CRLF is the Internet standard for the end of a line, as mentioned in RFC2234 and

Re: [sqlite] Producing RFC4180-compliant CSV output

2014-07-24 Thread Richard Hipp
On Thu, Jul 24, 2014 at 7:47 AM, Richard Hipp d...@sqlite.org wrote: On Thu, Jul 24, 2014 at 6:15 AM, Peter Waller pe...@scraperwiki.com wrote: Unfortunately, we already tried this, sed, and writing our own translation program in C. The other problem is that it adds \r in front of

Re: [sqlite] Producing RFC4180-compliant CSV output

2014-07-24 Thread Simon Slavin
On 24 Jul 2014, at 1:21pm, Richard Hipp d...@sqlite.org wrote: www.sqlite.org/src/info/16c8ce10e15 If I might suggest, for maximum flexibility, that the end-line-separator accept the following values: CR NL CRNL r n rn and maybe even \r \n \r\n . That should future-proof it. Simon.