On 2015-08-20, at 08:33, James Vincent wrote: > > Does anyone have a suggested good way to use CSV formatted data in a PIPE, > especially to reference the fields? > > I have a bazillion data records in CSV format. All of the fields are > quoted and separated by a comma, like; "field1","field2","field3",... and > so on. > > I would like to build a PIPE to grab different fields as needed. I can > 'un-CSV' the data but that takes quite a bit of time and processing > considering the amount of data. > > Blue-sky would be to have FS/Fieldsep work something like SPEC FS CSV F12 > 1 ... but I know that is not likely. > Confronting a similar problem where my source data were in a .xlsx file I considered options and chose to export as .html rather than as .csv:
o HTML has a more orderly scheme for protecting delimiters in fields. I know that any "<" introduces an HTML delimiter; "<" in the data appears as "<". o It does not rely on the assumption that any particular character is guaranteed to be absent from the data. But I had the luxury of being able to choose the format to process; I was not constrained to CSV. And I was not working on CMS; I used entirely procedural code. My data appeared as a table; I could not rely on each rendered row's appearing as a distinct record in the HTML. (How does CSV handle a field where the datum contains a newline?) (Charlotte (originally) relied on a massive HTML->FULLSCREEN Pipeline filter. It was my introduction by immersion to Pipelines.) -- gil
