attached On Sat, Jul 22, 2017 at 02:12:10PM +0200, Peter Uhnak wrote: > Hi, > > this is a continuation of an older thread about quoting fields only when > necessary. ( > http://forum.world.st/NeoCSVWriter-automatic-quotes-td4924781.html ) > > I've attached fileouts of NeoCSV packages with the addition (I don't know if > Metacello can file-out only changesets). > > The change doesn't affect the default behavior, only when explicitly > requested. > > Peter
Object subclass: #NeoCSVReader instanceVariableNames: 'readStream charBuffer separator stringStream fieldCount recordClass recordClassIsIndexable fieldAccessors emptyFieldValue' classVariableNames: '' poolDictionaries: '' category: 'Neo-CSV-Core'! !NeoCSVReader commentStamp: '<historical>' prior: 0! I am NeoCSVReader.
I read a format that - is text based (ASCII, Latin1, Unicode) - consists of records, 1 per line (any line ending convention) - where records consist of fields separated by a delimiter (comma, tab, semicolon) - where every record has the same number of fields - where fields can be quoted should they contain separators or line endings Without further configuration, records will become Arrays of Strings. By specifiying a recordClass and fields with optional converters most objects can be read and instanciated correctly. MIT License. ! !NeoCSVReader methodsFor: 'accessing' stamp: 'SvenVanCaekenberghe 5/13/2014 15:34'! readHeader "Read a record, presumably a header and return the header field names. This should normally be called only at the beginning and only once. This sets the fieldCount (but fieldAccessors overrides fieldCount)." | names | names := Array streamContents: [ :out | [ self atEnd or: [ self readEndOfLine ] ] whileFalse: [ out nextPut: self readField. (self readSeparator and: [ self atEnd or: [ self peekEndOfLine ] ]) ifTrue: [ out nextPut: emptyFieldValue ] ] ]. fieldCount := names size. ^ names! ! !NeoCSVReader methodsFor: 'accessing' stamp: 'SvenVanCaekenberghe 10/6/2014 17:34'! select: filter "Read and collect records that satisfy filter into an Array until there are none left. Return the array." ^ Array streamContents: [ :stream | self select: filter thenDo: [ :each | stream nextPut: each ] ]! ! !NeoCSVReader methodsFor: 'accessing' stamp: 'SvenVanCaekenberghe 6/21/2012 22:30'! next "Read the next record. I will return an instance of recordClass." ^ recordClassIsIndexable ifTrue: [ self readNextRecordAsArray ] ifFalse: [ self readNextRecordAsObject ]! ! !NeoCSVReader methodsFor: 'accessing' stamp: 'SvenVanCaekenberghe 10/6/2014 17:33'! select: filter thenDo: block "Execute block for each record that satisfies filter until I am at end." [ self atEnd ] whileFalse: [ | record | record := self next. (filter value: record) ifTrue: [ block value: record ] ]! ! !NeoCSVReader methodsFor: 'accessing' stamp: 'SvenVanCaekenberghe 6/25/2012 14:45'! upToEnd "Read and collect records into an Array until there are none left. Return the array." ^ Array streamContents: [ :stream | self do: [ :each | stream nextPut: each ] ]! ! !NeoCSVReader methodsFor: 'accessing' stamp: 'SvenVanCaekenberghe 6/25/2012 14:45'! do: block "Execute block for each record until I am at end." [ self atEnd ] whileFalse: [ block value: self next ]! ! !NeoCSVReader methodsFor: 'private - reading' stamp: 'SvenVanCaekenberghe 6/14/2012 17:24'! readField ^ self peekQuote ifTrue: [ self readQuotedField ] ifFalse: [ self readUnquotedField ]! ! !NeoCSVReader methodsFor: 'private - reading' stamp: 'SvenVanCaekenberghe 1/15/2014 09:55'! readNextRecord | record | record := recordClass new: fieldCount. fieldAccessors ifNil: [ self readNextRecordWithoutFieldAccessors: record ] ifNotNil: [ self readNextRecordWithFieldAccessors: record ]. self readAtEndOrEndOfLine. ^ record! ! !NeoCSVReader methodsFor: 'private - reading' stamp: 'SvenVanCaekenberghe 5/13/2014 15:35'! readNextRecordAsObject | object | object := recordClass new. fieldAccessors do: [ :each | | rawValue | rawValue := self readFieldAndSeparator. "Note that empty/missing fields are not set" (rawValue = emptyFieldValue or: [ each isNil ]) ifFalse: [ each value: object value: rawValue ] ]. self readAtEndOrEndOfLine. ^ object! ! !NeoCSVReader methodsFor: 'private - reading' stamp: 'SvenVanCaekenberghe 5/5/2013 13:17'! readNextRecordAsArray fieldAccessors ifNotNil: [ fieldCount := fieldAccessors count: [ :each | each notNil ] ]. ^ fieldCount ifNil: [ | record | record := self readFirstRecord. fieldCount := record size. record ] ifNotNil: [ self readNextRecord ]! ! !NeoCSVReader methodsFor: 'private - reading' stamp: 'SvenVanCaekenberghe 5/13/2014 15:36'! readUnquotedField (self atEnd or: [ self peekSeparator or: [ self peekEndOfLine ] ]) ifTrue: [ ^ emptyFieldValue ]. ^ self stringStreamContents: [ :stream | [ self atEnd or: [ self peekSeparator or: [ self peekEndOfLine ] ] ] whileFalse: [ stream nextPut: self nextChar ] ]! ! !NeoCSVReader methodsFor: 'private - reading' stamp: 'SvenVanCaekenberghe 5/13/2014 15:34'! readFirstRecord "This is only used for array based records when there are no field accessors or when there is no field count, to obtain a field count based on the first record" ^ recordClass streamContents: [ :stream | [ self atEnd or: [ self readEndOfLine ] ] whileFalse: [ stream nextPut: self readField. (self readSeparator and: [ self atEnd or: [ self peekEndOfLine ] ]) ifTrue: [ stream nextPut: emptyFieldValue ] ] ]! ! !NeoCSVReader methodsFor: 'private - reading' stamp: 'SvenVanCaekenberghe 9/18/2014 09:50'! readNextRecordWithFieldAccessors: record | fieldIndex | fieldIndex := 1. fieldAccessors do: [ :each | | rawValue | rawValue := self readFieldAndSeparator. "nil field accessors are used to ignore fields" each ifNotNil: [ rawValue = emptyFieldValue ifTrue: [ record at: fieldIndex put: emptyFieldValue ] ifFalse: [ record at: fieldIndex put: (each value: rawValue) ]. fieldIndex := fieldIndex + 1 ] ]! ! !NeoCSVReader methodsFor: 'private - reading' stamp: 'SvenVanCaekenberghe 9/18/2014 09:42'! readNextRecordWithoutFieldAccessors: record 1 to: fieldCount do: [ :each | record at: each put: self readFieldAndSeparator ]! ! !NeoCSVReader methodsFor: 'private - reading' stamp: 'SvenVanCaekenberghe 1/15/2014 10:11'! readFieldAndSeparator | field | field := self readField. self readSeparator. ^ field! ! !NeoCSVReader methodsFor: 'private - reading' stamp: 'SvenVanCaekenberghe 5/13/2014 15:36'! readQuotedField | field | self readQuote. field := self stringStreamContents: [ :stream | [ self atEnd or: [ self readEndOfQuotedField ] ] whileFalse: [ stream nextPut: self nextChar ] ]. ^ field isEmpty ifTrue: [ emptyFieldValue ] ifFalse: [ field ]! ! !NeoCSVReader methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 6/17/2012 16:24'! separator: character "Set the field separator character to use, defaults to comma" self assert: character isCharacter. separator := character ! ! !NeoCSVReader methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 6/26/2012 20:49'! addFieldAt: key "Add a field that will be stored under key in recordClass as String" self recordClassIsIndexable: false; addFieldAccessor: [ :object :string | object at: key put: string ]! ! !NeoCSVReader methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 6/26/2012 20:50'! addFieldAt: key converter: converter "Add a field that will be stored under key in recordClass as the result of applying the converter block on the field String read as argument" self recordClassIsIndexable: false; addFieldAccessor: [ :object :string | object at: key put: (converter value: string) ]! ! !NeoCSVReader methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 6/21/2012 22:29'! initialize super initialize. recordClass := Array. recordClassIsIndexable := true. separator := $,! ! !NeoCSVReader methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 6/21/2012 22:35'! addField "Add the next indexable field with a pass through converter" self recordClassIsIndexable: true; addFieldAccessor: [ :string | string ]! ! !NeoCSVReader methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 5/13/2013 13:09'! addField: accessor converter: converter "Add a field based on a mutator accessor accepting the result of applying the converter block on the field String read as argument to be sent to an instance of recordClass. Accessor can be a Symbol or a Block" self recordClassIsIndexable: false; addFieldAccessor: [ :object :string | self applyAccessor: accessor on: object with: (converter value: string) ]! ! !NeoCSVReader methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 6/25/2012 21:15'! on: aReadStream "Initialize on aReadStream, which should be a character stream that implements #next, #atEnd and (optionally) #close." readStream := aReadStream! ! !NeoCSVReader methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 6/14/2012 12:32'! close readStream ifNotNil: [ readStream close. readStream := nil ]! ! !NeoCSVReader methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 5/5/2013 14:26'! recordClass: anObject "Set the object class to instanciate while reading records. Unless the objets are integer indexable, you have to specify fields as well." recordClass := anObject! ! !NeoCSVReader methodsFor: 'initialize-release' stamp: 'EstebanMaringolo 10/3/2014 12:14'! addIgnoredFields: count "Add a count of consecutive ignored fields to receiver." count timesRepeat: [ self addIgnoredField ]! ! !NeoCSVReader methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 5/5/2013 11:06'! addIgnoredField "Add a field that should be ignored, should not become part of the record" self addFieldAccessor: nil! ! !NeoCSVReader methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 6/21/2012 23:01'! fieldCount: anObject "Set the field count up front. This will be used when reading records as Arrays. This instance variable will be set and used automatically based on the first record seen. If set, the fieldAccessors collection defines (overrides) the fieldCount." fieldCount := anObject! ! !NeoCSVReader methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 5/13/2014 15:32'! recordClassIsIndexable: boolean "Set whether recordClass should be treated as an indexable sequenceable collection class that implements #new: and #streamContents and whose instances implement #at:put: If false, fields accessors have to be provided. The default is true. Will be set automatically when field accessors or converters are set." recordClassIsIndexable := boolean ! ! !NeoCSVReader methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 6/21/2012 23:00'! addFieldConverter: converter "Add the next indexable field with converter block, accepting a String and returning a specific object" self recordClassIsIndexable: true; addFieldAccessor: converter! ! !NeoCSVReader methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 5/13/2013 13:09'! addField: accessor "Add a field based on a mutator accessor accepting a field String as argument to be sent to an instance of recordClass. Accessor can be a Symbol or a Block" self recordClassIsIndexable: false; addFieldAccessor: [ :object :string | self applyAccessor: accessor on: object with: string ]! ! !NeoCSVReader methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 5/13/2014 15:32'! emptyFieldValue: object "Set the value to be used when reading empty or missing fields. The default is nil. Empty or missing fields are never set when the record class is non-indexeabe." emptyFieldValue := object! ! !NeoCSVReader methodsFor: 'private' stamp: 'SvenVanCaekenberghe 1/14/2014 23:46'! peekFor: character self peekChar == character ifTrue: [ self nextChar. ^ true ]. ^ false! ! !NeoCSVReader methodsFor: 'private' stamp: 'SvenVanCaekenberghe 3/25/2016 14:00'! readEndOfQuotedField "A double quote inside a quoted field is an embedded quote (escaped)" ^ self readQuote and: [ self peekQuote not ]! ! !NeoCSVReader methodsFor: 'private' stamp: 'SvenVanCaekenberghe 1/14/2014 23:47'! readSeparator ^ self peekFor: separator! ! !NeoCSVReader methodsFor: 'private' stamp: 'SvenVanCaekenberghe 6/25/2012 20:56'! peekEndOfLine | char | char := self peekChar codePoint. ^ (char == 10 "Character lf" ) or: [ char == 13 "Character cr" ]! ! !NeoCSVReader methodsFor: 'private' stamp: 'SvenVanCaekenberghe 1/14/2014 23:47'! readQuote ^ self peekFor: $"! ! !NeoCSVReader methodsFor: 'private' stamp: 'SvenVanCaekenberghe 5/13/2013 10:56'! applyAccessor: accessor on: object with: value "Use accessor to assign value on a property of object. Accessor can be a block or mutator symbol." "If Symbol implemented #value:value: this could be implemented more elegantly." accessor isBlock ifTrue: [ accessor value: object value: value ] ifFalse: [ object perform: accessor with: value ]! ! !NeoCSVReader methodsFor: 'private' stamp: 'SvenVanCaekenberghe 1/15/2014 09:58'! peekSeparator ^ self peekChar == separator! ! !NeoCSVReader methodsFor: 'private' stamp: 'SvenVanCaekenberghe 6/16/2012 13:33'! addFieldAccessor: block fieldAccessors ifNil: [ fieldAccessors := Array with: block ] ifNotNil: [ fieldAccessors := fieldAccessors copyWith: block ]! ! !NeoCSVReader methodsFor: 'private' stamp: 'SvenVanCaekenberghe 6/25/2012 20:28'! nextChar ^ charBuffer ifNil: [ readStream next ] ifNotNil: [ | char | char := charBuffer. charBuffer := nil. ^ char ]! ! !NeoCSVReader methodsFor: 'private' stamp: 'SvenVanCaekenberghe 6/25/2012 20:56'! readEndOfLine | char | char := self peekChar codePoint. char == 10 "Character lf" ifTrue: [ self nextChar. ^ true ]. char == 13 "Character cr" ifTrue: [ self nextChar. (self atEnd not and: [ self peekChar codePoint == 10 "Character lf" ]) ifTrue: [ self nextChar ]. ^ true ]. ^ false ! ! !NeoCSVReader methodsFor: 'private' stamp: 'SvenVanCaekenberghe 6/14/2012 12:32'! stringStreamContents: block "Like String streamContents: block but reusing the underlying buffer for improved efficiency" stringStream ifNil: [ stringStream := (String new: 32) writeStream ]. stringStream reset. block value: stringStream. ^ stringStream contents! ! !NeoCSVReader methodsFor: 'private' stamp: 'SvenVanCaekenberghe 6/25/2012 20:56'! peekQuote ^ self peekChar == $"! ! !NeoCSVReader methodsFor: 'private' stamp: 'SvenVanCaekenberghe 6/25/2012 20:23'! peekChar ^ charBuffer ifNil: [ charBuffer := readStream next ]! ! !NeoCSVReader methodsFor: 'private' stamp: 'SvenVanCaekenberghe 1/15/2014 09:54'! readAtEndOrEndOfLine ^ self atEnd or: [ self readEndOfLine ] ! ! !NeoCSVReader methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 6/25/2012 20:47'! atEnd ^ charBuffer == nil and: [ readStream atEnd ]! ! !NeoCSVReader methodsFor: 'convenience' stamp: 'SvenVanCaekenberghe 5/5/2013 14:30'! addFields: accessors "Add fields based on a collection of accessors, not doing any conversions." accessors do: [ :each | self addField: each ]! ! !NeoCSVReader methodsFor: 'convenience' stamp: 'SvenVanCaekenberghe 5/5/2013 14:48'! addSymbolField "Add a field for indexable records read as Symbol" self addFieldConverter: [ :string | string asSymbol ]! ! !NeoCSVReader methodsFor: 'convenience' stamp: 'SvenVanCaekenberghe 5/5/2013 14:49'! addIntegerFieldAt: key "Add a field for key for #at:put: parsed as Integer" self addFieldAt: key converter: [ :string | NeoNumberParser parse: string ]! ! !NeoCSVReader methodsFor: 'convenience' stamp: 'SvenVanCaekenberghe 5/5/2013 14:34'! addFloatField "Add a field for indexable records parsed as Float" self addFieldConverter: [ :string | NeoNumberParser parse: string ]! ! !NeoCSVReader methodsFor: 'convenience' stamp: 'SvenVanCaekenberghe 5/5/2013 14:49'! addFloatFieldAt: key "Add a field for key for #at:put: parsed as Float" self addFieldAt: key converter: [ :string | NeoNumberParser parse: string ]! ! !NeoCSVReader methodsFor: 'convenience' stamp: 'SvenVanCaekenberghe 5/5/2013 14:47'! addIntegerField "Add a field for indexable records parsed as Integer" self addFieldConverter: [ :string | NeoNumberParser parse: string ]! ! !NeoCSVReader methodsFor: 'convenience' stamp: 'SvenVanCaekenberghe 5/5/2013 14:30'! addFieldsAt: keys "Add fields based on a collection of keys for #at:put: not doing any conversions" keys do: [ :each | self addFieldAt: each ]! ! !NeoCSVReader methodsFor: 'convenience' stamp: 'SvenVanCaekenberghe 5/5/2013 14:35'! addFloatField: accessor "Add a field with accessor parsed as Float" self addField: accessor converter: [ :string | NeoNumberParser parse: string ]! ! !NeoCSVReader methodsFor: 'convenience' stamp: 'SvenVanCaekenberghe 5/10/2015 21:59'! namedColumnsConfiguration "Assuming there is a header row that has not yet been read, configure the receiver to read each row as a Dictionary where each field is stored under a column name. Note that each field is read as a string." self recordClass: Dictionary. self addFieldsAt: (self readHeader collect: [ :each | each asSymbol ])! ! !NeoCSVReader methodsFor: 'convenience' stamp: 'SvenVanCaekenberghe 5/5/2013 14:48'! addIntegerField: accessor "Add a field with accessor parsed as Integer" self addField: accessor converter: [ :string | NeoNumberParser parse: string ]! ! !NeoCSVReader methodsFor: 'convenience' stamp: 'SvenVanCaekenberghe 5/13/2014 11:44'! skipHeader "Read a record, presumably a header, with the intention of skipping it. This should normally be called only at the beginning and only once. This sets the fieldCount (but fieldAccessors overrides fieldCount)." self readHeader! ! !NeoCSVReader methodsFor: 'convenience' stamp: 'SvenVanCaekenberghe 5/5/2013 14:48'! addSymbolField: accessor "Add a field with accessor read as Symbol" self addField: accessor converter: [ :string | string asSymbol ]! ! !NeoCSVReader methodsFor: 'convenience' stamp: 'SvenVanCaekenberghe 5/5/2013 14:49'! addSymbolFieldAt: key "Add a field for key for #at:put: read as Symbol" self addFieldAt: key converter: [ :string | string asSymbol ]! ! "-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- "! NeoCSVReader class slots: { }! !NeoCSVReader class methodsFor: 'instance creation' stamp: 'SvenVanCaekenberghe 6/25/2012 21:15'! on: readStream "Initialize on readStream, which should be a character stream that implements #next, #atEnd and (optionally) #close." ^ self new on: readStream; yourself! ! Object subclass: #NeoCSVWriter instanceVariableNames: 'writeStream separator fieldWriter lineEnd fieldAccessors emptyFieldValue' classVariableNames: '' poolDictionaries: '' category: 'Neo-CSV-Core'! !NeoCSVWriter commentStamp: '<historical>' prior: 0! I am NeoCSVWriter. I write a format that - is text based (ASCII, Latin1, Unicode) - consists of records, 1 per line (any line ending convention) - where records consist of fields separated by a delimiter (comma, tab, semicolon) - where every record has the same number of fields - where fields can be quoted should they contain separators or line endings Without further configuration, I write record objects whose fields can be enumerated using #do: such as SequenceableCollections By specifiying fields any object can be written converting and/or quoting each field as needed. MIT License.! !NeoCSVWriter methodsFor: 'private' stamp: 'SvenVanCaekenberghe 6/15/2012 12:47'! writeField: object self perform: fieldWriter with: object! ! !NeoCSVWriter methodsFor: 'private' stamp: 'SvenVanCaekenberghe 5/13/2014 15:53'! writeQuotedField: object object = emptyFieldValue ifTrue: [ writeStream nextPut: $" ; nextPut: $" ] ifFalse: [ | string | string := object asString. writeStream nextPut: $". string do: [ :each | each == $" ifTrue: [ writeStream nextPut: $"; nextPut: $" ] ifFalse: [ writeStream nextPut: each ] ]. writeStream nextPut: $" ]! ! !NeoCSVWriter methodsFor: 'private' stamp: 'SvenVanCaekenberghe 5/13/2014 15:51'! writeObjectField: object object = emptyFieldValue ifFalse: [ object printOn: writeStream ]! ! !NeoCSVWriter methodsFor: 'private' stamp: 'SvenVanCaekenberghe 6/14/2012 10:03'! writeSeparator writeStream nextPut: separator ! ! !NeoCSVWriter methodsFor: 'private' stamp: 'SvenVanCaekenberghe 6/15/2012 23:03'! addFieldAccessor: block fieldAccessors ifNil: [ fieldAccessors := Array with: block ] ifNotNil: [ fieldAccessors := fieldAccessors copyWith: block ]! ! !NeoCSVWriter methodsFor: 'private' stamp: 'SvenVanCaekenberghe 6/15/2012 19:16'! writeEndOfLine writeStream nextPutAll: lineEnd ! ! !NeoCSVWriter methodsFor: 'private' stamp: 'SvenVanCaekenberghe 6/15/2012 22:37'! writeFieldsUsingAccessors: anObject | first | first := true. fieldAccessors do: [ :each | | fieldValue | first ifTrue: [ first := false ] ifFalse: [ self writeSeparator ]. fieldValue := each value: anObject ]! ! !NeoCSVWriter methodsFor: 'private' stamp: 'SvenVanCaekenberghe 9/18/2014 09:20'! writeRawField: object object = emptyFieldValue ifFalse: [ writeStream nextPutAll: object asString ]! ! !NeoCSVWriter methodsFor: 'private' stamp: 'SvenVanCaekenberghe 6/15/2012 19:45'! writeFieldsUsingDo: anObject | first | first := true. anObject do: [ :each | first ifTrue: [ first := false ] ifFalse: [ self writeSeparator ]. self writeField: each ]! ! !NeoCSVWriter methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 5/13/2013 13:10'! addRawField: accessor "Add a field based on an accessor to be written as a #raw field. Accessor can be a Symbol or a Block" self addFieldAccessor: [ :object | self writeRawField: (accessor value: object) ]! ! !NeoCSVWriter methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 5/13/2013 13:10'! addQuotedField: accessor "Add a field based on an accessor to be written as a #quoted field. Accessor can be a Symbol or a Block" self addFieldAccessor: [ :object | self writeQuotedField: (accessor value: object) ]! ! !NeoCSVWriter methodsFor: 'initialize-release' stamp: 'PeterUhnak 7/22/2017 13:47'! addOptionalQuotedFieldAt: key "Add a field based on a key to be written as a #optionalQuoted field" self addFieldAccessor: [ :object | self writeOptionalQuotedField: (object at: key) ]! ! !NeoCSVWriter methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 6/23/2014 11:19'! addFieldAt: key "Add a field based on a key to be written using fieldWriter" self addFieldAccessor: [ :object | self writeField: (object at: key ifAbsent: [ '' ]) ]! ! !NeoCSVWriter methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 5/13/2013 13:10'! addObjectField: accessor "Add a field based on an accessor to be written as an #object field. Accessor can be a Symbol or a Block" self addFieldAccessor: [ :object | self writeObjectField: (accessor value: object) ]! ! !NeoCSVWriter methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 5/13/2013 13:10'! addField: accessor "Add a field based on an accessor to be written using fieldWriter. Accessor can be a Symbol or a Block" self addFieldAccessor: [ :object | self writeField: (accessor value: object) ]! ! !NeoCSVWriter methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 6/26/2012 20:31'! addRawFieldAt: key "Add a field based on a key to be written as a #raw field" self addFieldAccessor: [ :object | self writeRawField: (object at: key) ]! ! !NeoCSVWriter methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 6/17/2012 16:25'! lineEndConvention: symbol "Set the end of line convention to be used. Either #cr, #lf or #crlf (the default)." self assert: (#(cr lf crlf) includes: symbol). lineEnd := String perform: symbol! ! !NeoCSVWriter methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 5/13/2013 11:09'! addConstantField: string "Add a constant field to be written using fieldWriter" self addFieldAccessor: [ :object | self writeField: string ]! ! !NeoCSVWriter methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 6/14/2012 09:48'! on: aWriteStream "Initialize on aWriteStream, which should be a character stream that implements #nextPut:, #nextPutAll:, #space and (optionally) #close." writeStream := aWriteStream ! ! !NeoCSVWriter methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 9/18/2014 09:21'! emptyFieldValue: object "Set the empty field value to object. When reading fields from records to be written out, if the field value equals the emptyFieldValue, it will be considered an empty field and written as such." emptyFieldValue := object! ! !NeoCSVWriter methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 6/26/2012 20:30'! addQuotedFieldAt: key "Add a field based on a key to be written as a #quoted field" self addFieldAccessor: [ :object | self writeQuotedField: (object at: key) ]! ! !NeoCSVWriter methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 6/26/2012 20:30'! addObjectFieldAt: key "Add a field based on a key to be written as an #object field" self addFieldAccessor: [ :object | self writeObjectField: (object at: key) ]! ! !NeoCSVWriter methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 5/13/2013 11:09'! addEmptyField "Add an empty field to be written using fieldWriter" self addFieldAccessor: [ :object | self writeField: '' ]! ! !NeoCSVWriter methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 6/14/2012 10:10'! close writeStream ifNotNil: [ writeStream close. writeStream := nil ]! ! !NeoCSVWriter methodsFor: 'initialize-release' stamp: 'PeterUhnak 7/22/2017 13:46'! addOptionalQuotedField: accessor "Add a field based on an accessor to be written as a #optionalQuoted field. Accessor can be a Symbol or a Block" self addFieldAccessor: [ :object | self writeOptionalQuotedField: (accessor value: object) ]! ! !NeoCSVWriter methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 6/15/2012 19:16'! initialize super initialize. lineEnd := String crlf. separator := $, . fieldWriter := #writeQuotedField: ! ! !NeoCSVWriter methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 6/17/2012 16:24'! separator: character "Set character to be used as separator" self assert: character isCharacter. separator := character ! ! !NeoCSVWriter methodsFor: 'convenience' stamp: 'SvenVanCaekenberghe 6/16/2012 13:34'! addFields: accessors accessors do: [ :each | self addField: each ]! ! !NeoCSVWriter methodsFor: 'convenience' stamp: 'SvenVanCaekenberghe 6/26/2012 20:31'! addObjectFieldsAt: keys keys do: [ :each | self addObjectFieldAt: each ]! ! !NeoCSVWriter methodsFor: 'convenience' stamp: 'SvenVanCaekenberghe 6/26/2012 20:31'! addFieldsAt: keys keys do: [ :each | self addFieldAt: each ]! ! !NeoCSVWriter methodsFor: 'convenience' stamp: 'SvenVanCaekenberghe 6/26/2012 20:32'! addRawFieldsAt: keys keys do: [ :each | self addRawFieldAt: each ]! ! !NeoCSVWriter methodsFor: 'convenience' stamp: 'SvenVanCaekenberghe 5/10/2015 21:51'! namedColumnsConfiguration: columns "Configure the receiver to output the named columns as keyed properties. The objects to be written should respond to #at: like a Dictionary. Writes a header first. Uses the configured field writer." self writeHeader: columns. self addFieldsAt: columns! ! !NeoCSVWriter methodsFor: 'convenience' stamp: 'SvenVanCaekenberghe 6/15/2012 22:38'! addRawFields: accessors accessors do: [ :each | self addRawField: each ]! ! !NeoCSVWriter methodsFor: 'convenience' stamp: 'SvenVanCaekenberghe 6/15/2012 22:38'! addObjectFields: accessors accessors do: [ :each | self addObjectField: each ]! ! !NeoCSVWriter methodsFor: 'convenience' stamp: 'SvenVanCaekenberghe 6/26/2012 20:32'! addQuotedFieldsAt: keys keys do: [ :each | self addQuotedFieldAt: each ]! ! !NeoCSVWriter methodsFor: 'convenience' stamp: 'SvenVanCaekenberghe 6/15/2012 22:39'! addQuotedFields: accessors accessors do: [ :each | self addQuotedField: each ]! ! !NeoCSVWriter methodsFor: 'convenience' stamp: 'PeterUhnak 7/22/2017 13:45'! addOptionalQuotedFieldsAt: keys keys do: [ :each | self addOptionalQuotedFieldAt: each ]! ! !NeoCSVWriter methodsFor: 'convenience' stamp: 'PeterUhnak 7/22/2017 13:45'! addOptionalQuotedFields: accessors accessors do: [ :each | self addOptionalQuotedField: each ]! ! !NeoCSVWriter methodsFor: 'writing' stamp: 'PeterUhnak 7/22/2017 13:55'! writeOptionalQuotedField: object | string | object = emptyFieldValue ifTrue: [ ^ self ]. string := object asString. ({lineEnd asString. separator asString. '"'} anySatisfy: [ :each | string includesSubstring: each ]) ifTrue: [ self writeQuotedField: object ] ifFalse: [ self writeRawField: object ]! ! !NeoCSVWriter methodsFor: 'accessing' stamp: 'PeterUhnak 7/22/2017 14:06'! fieldWriter: symbol "Set the field write to be used, either #quoted (the default), #raw or #object. This determines how field values will be written in the general case. #quoted will wrap fields #asString in double quotes and escape embedded double quotes #raw will write fields #asString as such (no separator, double quote or end of line chars allowed) #optionalQuoted will write fields using #raw if possible (no separators, ...), and #quoted otherwise #object will #print: fields (no separator, double quote or end of line chars allowed)" self assert: (#(quoted raw object optionalQuoted) includes: symbol). fieldWriter := ('write', symbol capitalized, 'Field:') asSymbol ! ! !NeoCSVWriter methodsFor: 'accessing' stamp: 'SvenVanCaekenberghe 6/17/2012 16:23'! nextPut: anObject "Write anObject as single record. Depending on configuration fieldAccessors or a #do: enumeration will be used." fieldAccessors ifNil: [ self writeFieldsUsingDo: anObject ] ifNotNil: [ self writeFieldsUsingAccessors: anObject ]. self writeEndOfLine ! ! !NeoCSVWriter methodsFor: 'accessing' stamp: 'SvenVanCaekenberghe 6/17/2012 16:23'! nextPutAll: collection "Write a collection of objects as records" collection do: [ :each | self nextPut: each ]! ! !NeoCSVWriter methodsFor: 'accessing' stamp: 'SvenVanCaekenberghe 6/17/2012 19:39'! flush writeStream flush! ! !NeoCSVWriter methodsFor: 'accessing' stamp: 'SvenVanCaekenberghe 12/11/2013 21:47'! writeHeader: fieldNames "Write the header, a collection of field names. This should normally be called only at the beginning and only once." fieldNames do: [ :each | self writeQuotedField: each ] separatedBy: [ self writeSeparator ]. self writeEndOfLine! ! "-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- "! NeoCSVWriter class slots: { }! !NeoCSVWriter class methodsFor: 'instance creation' stamp: 'SvenVanCaekenberghe 6/14/2012 09:47'! on: writeStream "Initialize on writeStream, which should be a character stream that implements #nextPut:, #nextPutAll:, #space and (optionally) #close." ^ self new on: writeStream; yourself! ! Object subclass: #NeoNumberParser instanceVariableNames: 'stream base' classVariableNames: '' poolDictionaries: '' category: 'Neo-CSV-Core'! !NeoNumberParser commentStamp: '<historical>' prior: 0! I am NeoNumberParser, an alternative number parser that needs only a minimal read stream protocol. I accept the following syntax: number int int frac int exp int frac exp int digits - digits frac . digits exp e digits digits digit digit digits e e e+ e- E E+ E- where digit depends on the base (2 to 36), 0 .. 9, A-Z, a-z.! !NeoNumberParser methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 12/2/2012 13:49'! initialize super initialize. self base: 10! ! !NeoNumberParser methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 12/2/2012 13:44'! base: integer self assert: (integer between: 2 and: 36) description: 'Number base must be between 2 and 36'. base := integer! ! !NeoNumberParser methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 12/2/2012 13:43'! on: readStream stream := readStream ! ! !NeoNumberParser methodsFor: 'parsing' stamp: 'SvenVanCaekenberghe 12/2/2012 17:30'! consumeWhitespace "Strip whitespaces from the input stream." [ stream atEnd not and: [ stream peek isSeparator ] ] whileTrue: [ stream next ] ! ! !NeoNumberParser methodsFor: 'parsing' stamp: 'SvenVanCaekenberghe 12/2/2012 20:06'! parseNumberInteger | number | number := nil. [ stream atEnd not and: [ stream peek digitValue between: 0 and: base - 1 ] ] whileTrue: [ number := base * (number ifNil: [ 0 ]) + stream next digitValue ]. number ifNil: [ self error: 'Integer digit expected' ]. ^ number! ! !NeoNumberParser methodsFor: 'parsing' stamp: 'SvenVanCaekenberghe 12/2/2012 20:06'! parseNumberFraction | number power | number := 0. power := 1.0. [ stream atEnd not and: [ stream peek digitValue between: 0 and: base - 1 ] ] whileTrue: [ number := base * number + stream next digitValue. power := power * base ]. ^ number / power! ! !NeoNumberParser methodsFor: 'parsing' stamp: 'SvenVanCaekenberghe 12/3/2012 11:00'! parseNumber | negated number | negated := stream peekFor: $-. number := self parseNumberInteger. (stream peekFor: $.) ifTrue: [ number := number + self parseNumberFraction ]. ((stream peekFor: $e) or: [ stream peekFor: $E ]) ifTrue: [ number := number * self parseNumberExponent ]. negated ifTrue: [ number := number negated ]. ^ number! ! !NeoNumberParser methodsFor: 'parsing' stamp: 'SvenVanCaekenberghe 12/2/2012 20:06'! parseNumberExponent | number negated | number := 0. (negated := stream peekFor: $-) ifFalse: [ stream peekFor: $+ ]. [ stream atEnd not and: [ stream peek digitValue between: 0 and: base - 1 ] ] whileTrue: [ number := base * number + stream next digitValue ]. negated ifTrue: [ number := number negated ]. ^ base raisedTo: number! ! !NeoNumberParser methodsFor: 'accessing' stamp: 'SvenVanCaekenberghe 12/2/2012 13:31'! next ^ self parseNumber! ! "-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- "! NeoNumberParser class slots: { }! !NeoNumberParser class methodsFor: 'queries' stamp: 'SvenVanCaekenberghe 12/2/2012 13:35'! parse: stringOrStream | stream | stream := stringOrStream isString ifTrue: [ stringOrStream readStream ] ifFalse: [ stringOrStream ]. ^ (self on: stream) next! ! !NeoNumberParser class methodsFor: 'queries' stamp: 'SvenVanCaekenberghe 12/2/2012 17:43'! parse: stringOrStream base: base ifFail: block ^ [ self parse: stringOrStream base: base ] on: Error do: block! ! !NeoNumberParser class methodsFor: 'queries' stamp: 'SvenVanCaekenberghe 12/2/2012 17:43'! parse: stringOrStream ifFail: block ^ [ self parse: stringOrStream ] on: Error do: block! ! !NeoNumberParser class methodsFor: 'queries' stamp: 'SvenVanCaekenberghe 12/2/2012 13:51'! parse: stringOrStream base: base | stream | stream := stringOrStream isString ifTrue: [ stringOrStream readStream ] ifFalse: [ stringOrStream ]. ^ (self on: stream) base: base; next! ! !NeoNumberParser class methodsFor: 'instance creation' stamp: 'SvenVanCaekenberghe 12/2/2012 13:31'! on: readStream ^ self new on: readStream; yourself! !
Object subclass: #NeoCSVBenchmark instanceVariableNames: 'data' classVariableNames: '' poolDictionaries: '' category: 'Neo-CSV-Tests'! !NeoCSVBenchmark commentStamp: '<historical>' prior: 0! I am NeoCSVBenchmark. | benchmark | benchmark := NeoCSVBenchmark new. benchmark cleanup. [ benchmark write1 ] timeToRun. | benchmark | benchmark := NeoCSVBenchmark new. [ benchmark read0 ] timeToRun. | benchmark | benchmark := NeoCSVBenchmark new. [ benchmark read1 ] timeToRun. | benchmark | benchmark := NeoCSVBenchmark new. benchmark cleanup. [ benchmark write2 ] timeToRun. | benchmark | benchmark := NeoCSVBenchmark new. benchmark cleanup. [ benchmark write3 ] timeToRun. | benchmark | benchmark := NeoCSVBenchmark new. benchmark cleanup. [ benchmark write4 ] timeToRun. | benchmark | benchmark := NeoCSVBenchmark new. benchmark cleanup. [ benchmark write5 ] timeToRun. | benchmark | benchmark := NeoCSVBenchmark new. [ benchmark read2 ] timeToRun. | benchmark | benchmark := NeoCSVBenchmark new. [ benchmark read3 ] timeToRun. ! !NeoCSVBenchmark methodsFor: 'accessing' stamp: 'SvenVanCaekenberghe 9/27/2012 20:35'! filename ^ 'NeoCSVBenchmark.csv' asFileReference! ! !NeoCSVBenchmark methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 6/17/2012 17:22'! initialize data := Array new: 100000 streamContents: [ :stream | 1 to: 100000 do: [ :each | stream nextPut: (Array with: each with: each negated with: (100000 - each)) ] ]! ! !NeoCSVBenchmark methodsFor: 'public' stamp: 'SvenVanCaekenberghe 9/27/2012 20:35'! cleanup self filename ensureDeleted ! ! !NeoCSVBenchmark methodsFor: 'public' stamp: 'SvenVanCaekenberghe 11/30/2012 22:35'! read1 self filename readStreamDo: [ :stream | (NeoCSVReader on: (ZnBufferedReadStream on: stream)) upToEnd ]! ! !NeoCSVBenchmark methodsFor: 'public' stamp: 'SvenVanCaekenberghe 9/27/2012 20:37'! write2 self filename writeStreamDo: [ :stream | (NeoCSVWriter on: (ZnBufferedWriteStream on: stream)) addRawFields: #(first second third); nextPutAll: data; flush ]! ! !NeoCSVBenchmark methodsFor: 'public' stamp: 'SvenVanCaekenberghe 9/27/2012 20:36'! write0 self filename writeStreamDo: [ :stream | (NeoCSVWriter on: stream) nextPutAll: data ]! ! !NeoCSVBenchmark methodsFor: 'public' stamp: 'SvenVanCaekenberghe 9/27/2012 20:36'! read2 self filename readStreamDo: [ :stream | (NeoCSVReader on: stream) recordClass: NeoCSVTestObject; addIntegerField: #x: ; addIntegerField: #y: ; addIntegerField: #z: ; upToEnd ]! ! !NeoCSVBenchmark methodsFor: 'public' stamp: 'SvenVanCaekenberghe 9/27/2012 20:37'! write5 self filename writeStreamDo: [ :stream | (NeoCSVWriter on: (ZnBufferedWriteStream on: stream)) fieldWriter: #object; nextPutAll: data; flush ]! ! !NeoCSVBenchmark methodsFor: 'public' stamp: 'SvenVanCaekenberghe 9/27/2012 20:37'! write3 self filename writeStreamDo: [ :stream | (NeoCSVWriter on: (ZnBufferedWriteStream on: stream)) addObjectFields: #(first second third); nextPutAll: data; flush ]! ! !NeoCSVBenchmark methodsFor: 'public' stamp: 'SvenVanCaekenberghe 11/30/2012 22:34'! read0 self filename readStreamDo: [ :stream | (NeoCSVReader on: stream) upToEnd ]! ! !NeoCSVBenchmark methodsFor: 'public' stamp: 'SvenVanCaekenberghe 9/27/2012 20:37'! write4 self filename writeStreamDo: [ :stream | (NeoCSVWriter on: (ZnBufferedWriteStream on: stream)) fieldWriter: #raw; nextPutAll: data; flush ]! ! !NeoCSVBenchmark methodsFor: 'public' stamp: 'SvenVanCaekenberghe 9/27/2012 20:37'! write1 self filename writeStreamDo: [ :stream | (NeoCSVWriter on: (ZnBufferedWriteStream on: stream)) nextPutAll: data; flush ]! ! !NeoCSVBenchmark methodsFor: 'public' stamp: 'SvenVanCaekenberghe 11/30/2012 22:35'! read3 self filename readStreamDo: [ :stream | (NeoCSVReader on: (ZnBufferedReadStream on: stream)) recordClass: NeoCSVTestObject; addIntegerField: #x: ; addIntegerField: #y: ; addIntegerField: #z: ; upToEnd ]! ! TestCase subclass: #NeoCSVReaderTests instanceVariableNames: '' classVariableNames: '' poolDictionaries: '' category: 'Neo-CSV-Tests'! !NeoCSVReaderTests commentStamp: '<historical>' prior: 0! I am NeoCSVReaderTests, a suite of unit tests for NeoCSVReader. ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 11/4/2012 19:54'! testEmptyConversionsTestObject | input | input := (String crlf join: #( '1,2.5,foo' ',,' )). self assert: ((NeoCSVReader on: input readStream) recordClass: NeoCSVTestObject; addIntegerField: #x: ; addFloatField: #y: ; addField: #z: ; upToEnd) equals: { NeoCSVTestObject x: 1 y: 2.5 z: 'foo'. NeoCSVTestObject new }! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 5/13/2014 10:57'! testEmptyLastFieldUnquoted self assert: (NeoCSVReader on: '1,2,' readStream) upToEnd equals: #(('1' '2' nil))! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 6/14/2012 19:56'! testOneLineOneFieldQuoted self assert: (NeoCSVReader on: '"1"' readStream) upToEnd equals: #(('1'))! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 5/13/2014 14:47'! testEmptyFieldUnquoted self assert: (NeoCSVReader on: '1,,3' readStream) upToEnd equals: #(('1' nil '3'))! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 1/15/2014 12:10'! testEmptyFieldSecondRecordUnquoted self assert: (NeoCSVReader on: 'foo,bar\100,' withCRs readStream) upToEnd equals: #(('foo' 'bar')('100' nil))! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 6/14/2012 18:57'! testOneLineUnquoted self assert: (NeoCSVReader on: '1,2,3' readStream) upToEnd equals: #(('1' '2' '3'))! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 5/13/2014 16:31'! testReadHeader | input | input := (String crlf join: #( '"x","y","z"' '100,200,300' '100,200,300' '100,200,300' '')). self assert: (NeoCSVReader on: input readStream) readHeader equals: #('x' 'y' 'z')! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 1/14/2014 23:20'! testEmbeddedQuotes self assert: (NeoCSVReader on: '1,"x""y""z",3' readStream) upToEnd equals: #(('1' 'x"y"z' '3'))! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 11/4/2012 19:54'! testEmptyConversions | input | input := (String crlf join: #( '1,2.5,foo' ',,' )). self assert: ((NeoCSVReader on: input readStream) addIntegerField; addFloatField; addField; upToEnd) equals: { #( 1 2.5 'foo' ). #( nil nil nil ) }! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 6/21/2012 22:57'! testReadAsByteArrays | input | input := (String crlf join: #( '1,2,3' '1,2,3' '1,2,3' '')). self assert: ((NeoCSVReader on: input readStream) recordClass: ByteArray; addIntegerField; addIntegerField ; addIntegerField; upToEnd) equals: { #[1 2 3]. #[1 2 3]. #[1 2 3].}! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 6/17/2012 13:59'! testSimpleTabDelimited | input | input := (String crlf join: #('1 2 3' '4 5 6' '7 8 9' '')). self assert: ((NeoCSVReader on: input readStream) separator: Character tab ; upToEnd) equals: #(('1' '2' '3')('4' '5' '6')('7' '8' '9'))! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 5/5/2013 12:15'! testReadDictionaries | input | input := (String crlf join: #( '"x","y","z"' '100,200,300' '100,200,300' '100,200,300' '')). self assert: ((NeoCSVReader on: input readStream) skipHeader; recordClass: Dictionary; addIntegerFieldAt: #x ; addIntegerFieldAt: #y ; addIntegerFieldAt: #z ; upToEnd) equals: { Dictionary newFromPairs: #(x 100 y 200 z 300). Dictionary newFromPairs: #(x 100 y 200 z 300). Dictionary newFromPairs: #(x 100 y 200 z 300) }! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 10/6/2014 17:46'! testSkippingEmptyRecords | input output | input := '1,2,3\\4,5,6\,,\7,8,9' withCRs. output := (NeoCSVReader on: input readStream) select: [ :each | each notEmpty and: [ (each allSatisfy: #isNil) not ] ]. self assert: output equals: #(#('1' '2' '3') #('4' '5' '6') #('7' '8' '9')). output := (NeoCSVReader on: input readStream) emptyFieldValue: ''; select: [ :each | each notEmpty and: [ (each allSatisfy: #isEmpty) not ] ]. self assert: output equals: #(#('1' '2' '3') #('4' '5' '6') #('7' '8' '9'))! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 6/17/2012 13:56'! testSimpleLfUnquoted | input | input := (String lf join: #('1,2,3' '4,5,6' '7,8,9' '')). self assert: (NeoCSVReader on: input readStream) upToEnd equals: #(('1' '2' '3')('4' '5' '6')('7' '8' '9'))! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 6/21/2012 22:56'! testReadAsIntegerArrays | input | input := (String crlf join: #( '100,200,300' '100,200,300' '100,200,300' '')). self assert: ((NeoCSVReader on: input readStream) recordClass: IntegerArray; addIntegerField; addIntegerField ; addIntegerField; upToEnd) equals: { #(100 200 300) asIntegerArray. #(100 200 300) asIntegerArray. #(100 200 300) asIntegerArray }! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 5/5/2013 12:15'! testReadIntegers | input | input := (String crlf join: #( '"x","y","z"' '100,200,300' '100,200,300' '100,200,300' '')). self assert: ((NeoCSVReader on: input readStream) skipHeader; addIntegerField; addIntegerField ; addIntegerField; upToEnd) equals: #((100 200 300)(100 200 300)(100 200 300))! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 5/13/2014 14:47'! testEmptyLastFieldQuoted self assert: (NeoCSVReader on: '"1","2",""' readStream) upToEnd equals: #(('1' '2' nil))! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 5/5/2013 14:20'! testReadWithIgnoredField | input | input := (String crlf join: #( '1,2,a,3' '1,2,b,3' '1,2,c,3' '')). self assert: ((NeoCSVReader on: input readStream) addIntegerField; addIntegerField; addIgnoredField; addIntegerField; upToEnd) equals: { #(1 2 3). #(1 2 3). #(1 2 3).}! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 6/17/2012 13:58'! testSimpleSemiColonDelimited | input | input := (String crlf join: #('1;2;3' '4;5;6' '7;8;9' '')). self assert: ((NeoCSVReader on: input readStream) separator: $; ; upToEnd) equals: #(('1' '2' '3')('4' '5' '6')('7' '8' '9'))! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 6/14/2012 19:55'! testOneLineOneFieldUnquoted self assert: (NeoCSVReader on: '1' readStream) upToEnd equals: #(('1'))! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 1/15/2014 12:10'! testEmptyFieldSecondRecordQuoted self assert: (NeoCSVReader on: '"foo","bar"\"100",' withCRs readStream) upToEnd equals: #(('foo' 'bar')('100' nil))! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 11/21/2016 11:37'! testReadTestsObjectsWithEmptyFieldValue | input | input := (String crlf join: #( '"x","y","z"' '100,200,300' '1,,3' '100,200,300' '')). self assert: ((NeoCSVReader on: input readStream) skipHeader; recordClass: NeoCSVTestObject2; emptyFieldValue: #empty; addIntegerField: #x: ; addIntegerField: #y: ; addIntegerField: #z: ; upToEnd) equals: { NeoCSVTestObject2 example. NeoCSVTestObject2 new x: 1; z: 3; yourself. "Note that y contains #y from #initialize and was NOT touched" NeoCSVTestObject2 example }. self assert: ((NeoCSVReader on: input readStream) skipHeader; recordClass: NeoCSVTestObject2; addIntegerField: #x: ; addIntegerField: #y: ; addIntegerField: #z: ; upToEnd) equals: { NeoCSVTestObject2 example. NeoCSVTestObject2 new x: 1; z: 3; yourself. "Note that y contains #y from #initialize and was NOT touched" NeoCSVTestObject2 example }! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 5/5/2013 12:15'! testReadTestsObjects | input | input := (String crlf join: #( '"x","y","z"' '100,200,300' '100,200,300' '100,200,300' '')). self assert: ((NeoCSVReader on: input readStream) skipHeader; recordClass: NeoCSVTestObject; addIntegerField: #x: ; addIntegerField: #y: ; addIntegerField: #z: ; upToEnd) equals: { NeoCSVTestObject example. NeoCSVTestObject example. NeoCSVTestObject example }! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 5/5/2013 13:09'! testReadTestsObjectsWithIgnoredField | input | input := (String crlf join: #( '"x","y",''-'',"z"' '100,200,a,300' '100,200,b,300' '100,200,c,300' '')). self assert: ((NeoCSVReader on: input readStream) skipHeader; recordClass: NeoCSVTestObject; addIntegerField: #x: ; addIntegerField: #y: ; addIgnoredField; addIntegerField: #z: ; upToEnd) equals: { NeoCSVTestObject example. NeoCSVTestObject example. NeoCSVTestObject example }! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 6/17/2012 13:56'! testSimpleCrUnquoted | input | input := (String cr join: #('1,2,3' '4,5,6' '7,8,9' '')). self assert: (NeoCSVReader on: input readStream) upToEnd equals: #(('1' '2' '3')('4' '5' '6')('7' '8' '9'))! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 5/13/2014 14:46'! testEmptyFieldQuoted self assert: (NeoCSVReader on: '"1",,"3"' readStream) upToEnd equals: #(('1' nil '3'))! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 12/11/2013 20:43'! testReadIntegersReadingHeaderAfterFieldDefinitions | input | input := (String crlf join: #( '"x","y","z"' '100,200,300' '100,200,300' '100,200,300' '')). self assert: ((NeoCSVReader on: input readStream) addIntegerField; addIntegerField ; addIntegerField; skipHeader; upToEnd) equals: #((100 200 300)(100 200 300)(100 200 300))! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 5/13/2013 12:51'! testReadTestsObjectsUsingBlockAccessors | input | input := (String crlf join: #( '"x","y","z"' '100,200,300' '100,200,300' '100,200,300' '')). self assert: ((NeoCSVReader on: input readStream) skipHeader; recordClass: NeoCSVTestObject; addIntegerField: [ :object :value | object x: value ]; addIntegerField: [ :object :value | object y: value ]; addIntegerField: [ :object :value | object z: value ]; upToEnd) equals: { NeoCSVTestObject example. NeoCSVTestObject example. NeoCSVTestObject example }! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 6/17/2012 13:53'! testSimpleCrLfQuoted | input | input := (String crlf join: #('"1","2","3"' '"4","5","6"' '"7","8","9"' '')). self assert: (NeoCSVReader on: input readStream) upToEnd equals: #(('1' '2' '3')('4' '5' '6')('7' '8' '9'))! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 6/14/2012 18:58'! testOneLineQuoted self assert: (NeoCSVReader on: '"1","2","3"' readStream) upToEnd equals: #(('1' '2' '3'))! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 6/14/2012 19:57'! testOneLineEmpty self assert: (NeoCSVReader on: '' readStream) upToEnd equals: #()! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 6/17/2012 13:56'! testSimpleCrLfUnquoted | input | input := (String crlf join: #('1,2,3' '4,5,6' '7,8,9' '')). self assert: (NeoCSVReader on: input readStream) upToEnd equals: #(('1' '2' '3')('4' '5' '6')('7' '8' '9'))! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 6/17/2012 13:54'! testSimpleCrQuoted | input | input := (String cr join: #('"1","2","3"' '"4","5","6"' '"7","8","9"' '')). self assert: (NeoCSVReader on: input readStream) upToEnd equals: #(('1' '2' '3')('4' '5' '6')('7' '8' '9'))! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 6/17/2012 13:54'! testSimpleLfQuoted | input | input := (String lf join: #('"1","2","3"' '"4","5","6"' '"7","8","9"' '')). self assert: (NeoCSVReader on: input readStream) upToEnd equals: #(('1' '2' '3')('4' '5' '6')('7' '8' '9'))! ! !NeoCSVReaderTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 9/18/2014 09:24'! testEmptyFieldValue self assert: ((NeoCSVReader on: '"1",,3,"","5"' readStream) emptyFieldValue: #empty; upToEnd) equals: #(('1' empty '3' empty '5')). self assert: ((NeoCSVReader on: '"1",,3,"","5"' readStream) emptyFieldValue: ''; upToEnd) equals: #(('1' '' '3' '' '5')). self assert: ((NeoCSVReader on: 'a,b,c\,,\"","",""\1,2,3\' withCRs readStream) emptyFieldValue: #empty; upToEnd) equals: #(('a' 'b' 'c')(empty empty empty)(empty empty empty)('1' '2' '3'))! ! Object subclass: #NeoCSVTestObject instanceVariableNames: 'x y z' classVariableNames: '' poolDictionaries: '' category: 'Neo-CSV-Tests'! !NeoCSVTestObject commentStamp: '<historical>' prior: 0! I am NeoCSVTestObject.! !NeoCSVTestObject methodsFor: 'accessing' stamp: 'SvenVanCaekenberghe 6/15/2012 22:40'! x ^ x! ! !NeoCSVTestObject methodsFor: 'accessing' stamp: 'SvenVanCaekenberghe 6/15/2012 22:40'! x: anObject x := anObject! ! !NeoCSVTestObject methodsFor: 'accessing' stamp: 'SvenVanCaekenberghe 6/15/2012 22:40'! y: anObject y := anObject! ! !NeoCSVTestObject methodsFor: 'accessing' stamp: 'SvenVanCaekenberghe 6/15/2012 22:40'! z: anObject z := anObject! ! !NeoCSVTestObject methodsFor: 'accessing' stamp: 'SvenVanCaekenberghe 6/15/2012 22:40'! z ^ z! ! !NeoCSVTestObject methodsFor: 'accessing' stamp: 'SvenVanCaekenberghe 6/15/2012 22:40'! y ^ y! ! !NeoCSVTestObject methodsFor: 'comparing' stamp: 'SvenVanCaekenberghe 6/16/2012 18:58'! = anObject self == anObject ifTrue: [ ^ true ]. self class = anObject class ifFalse: [ ^ false ]. ^ x = anObject x and: [ y = anObject y and: [ z = anObject z ] ]! ! !NeoCSVTestObject methodsFor: 'comparing' stamp: 'SvenVanCaekenberghe 6/16/2012 18:58'! hash ^ x hash bitXor: (y hash bitXor: z)! ! "-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- "! NeoCSVTestObject class slots: { }! !NeoCSVTestObject class methodsFor: 'instance creation' stamp: 'SvenVanCaekenberghe 11/4/2012 19:39'! x: x y: y z: z ^ self new x: x; y: y; z: z; yourself! ! !NeoCSVTestObject class methodsFor: 'instance creation' stamp: 'SvenVanCaekenberghe 11/4/2012 19:40'! example ^ self x: 100 y: 200 z: 300! ! NeoCSVTestObject subclass: #NeoCSVTestObject2 instanceVariableNames: '' classVariableNames: '' poolDictionaries: '' category: 'Neo-CSV-Tests'! !NeoCSVTestObject2 commentStamp: 'SvenVanCaekenberghe 11/21/2016 11:28' prior: 0! I am NeoCSVTestObject2. I am a NeoCSVTestObject. I initialize my fields to specific values.! !NeoCSVTestObject2 methodsFor: 'initalize' stamp: 'SvenVanCaekenberghe 11/21/2016 11:28'! initialize super initialize. x := #x. y := #y. z := #z! ! TestCase subclass: #NeoCSVWriterTests instanceVariableNames: '' classVariableNames: '' poolDictionaries: '' category: 'Neo-CSV-Tests'! !NeoCSVWriterTests commentStamp: '<historical>' prior: 0! I am NeoCSVWriterTests, a suite of unit tests for NeoCSVWriter. ! !NeoCSVWriterTests methodsFor: 'testing' stamp: 'PeterUhnak 7/22/2017 14:02'! testOptionalQuotedFields self assert: (String streamContents: [ :stream | (NeoCSVWriter on: stream) fieldWriter: #optionalQuoted; nextPut: {'one'. 't,wo'. 't"hree'. 'fo' , String crlf , 'ur'} ]) equals: 'one,"t,wo","t""hree","fo' , String crlf , 'ur"', String crlf! ! !NeoCSVWriterTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 12/11/2013 21:50'! testWriteHeader | header | header := String streamContents: [ :out | (NeoCSVWriter on: out) writeHeader: #(foo bar) ]. self assert: header equals: '"foo","bar"', String crlf! ! !NeoCSVWriterTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 6/15/2012 12:45'! testSimpleRaw self assert: (String streamContents: [ :stream | (NeoCSVWriter on: stream) fieldWriter: #raw; nextPut: #(one two three); nextPutAll: #( (1 2 3) (4 5 6) (7 8 9)) ]) equals: (String crlf join: #( 'one,two,three' '1,2,3' '4,5,6' '7,8,9' ''))! ! !NeoCSVWriterTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 5/13/2013 13:07'! testObjectFieldsTestObjectsExtra self assert: (String streamContents: [ :stream | (NeoCSVWriter on: stream) fieldWriter: #raw; nextPut: #(x empty y constant z); addObjectField: #x; addEmptyField; addObjectField: #y; addConstantField: 'X'; addObjectField: #z; nextPutAll: { NeoCSVTestObject example. NeoCSVTestObject example. NeoCSVTestObject example } ]) equals: (String crlf join: #( 'x,empty,y,constant,z' '100,,200,X,300' '100,,200,X,300' '100,,200,X,300' ''))! ! !NeoCSVWriterTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 6/15/2012 23:06'! testObjectFieldsTestObjects self assert: (String streamContents: [ :stream | (NeoCSVWriter on: stream) nextPut: #(x y z); addObjectFields: #(x y z); nextPutAll: { NeoCSVTestObject example. NeoCSVTestObject example. NeoCSVTestObject example } ]) equals: (String crlf join: #( '"x","y","z"' '100,200,300' '100,200,300' '100,200,300' ''))! ! !NeoCSVWriterTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 5/13/2013 12:53'! testObjectFieldsTestObjectsUsingBlockAccessors self assert: (String streamContents: [ :stream | (NeoCSVWriter on: stream) nextPut: #(x y z); addObjectFields: { [ :object | object x ]. [ :object | object y ]. [ :object | object z ] }; nextPutAll: { NeoCSVTestObject example. NeoCSVTestObject example. NeoCSVTestObject example } ]) equals: (String crlf join: #( '"x","y","z"' '100,200,300' '100,200,300' '100,200,300' ''))! ! !NeoCSVWriterTests methodsFor: 'testing' stamp: 'PeterUhnak 7/22/2017 14:09'! testSimpleOptionalQuoted self assert: (String streamContents: [ :stream | (NeoCSVWriter on: stream) fieldWriter: #optionalQuoted; nextPut: #(one two 'thr,ee'); nextPutAll: #( (1 2 3) (4 5 6) (7 8 9)) ]) equals: (String crlf join: #( 'one,two,"thr,ee"' '1,2,3' '4,5,6' '7,8,9' ''))! ! !NeoCSVWriterTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 1/14/2014 23:37'! testWriteEmbeddedQuote | header | header := String streamContents: [ :out | (NeoCSVWriter on: out) nextPut: #(foo 'x"y"z') ]. self assert: header equals: '"foo","x""y""z"', String crlf! ! !NeoCSVWriterTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 5/13/2014 15:53'! testEmptyFieldValue self assert: (String streamContents: [ :stream | (NeoCSVWriter on: stream) nextPut: #(one two three); nextPutAll: #( (1 2 nil) (4 nil 6) (nil 8 9)) ]) equals: (String crlf join: #( '"one","two","three"' '"1","2",""' '"4","","6"' '"","8","9"' '')). self assert: (String streamContents: [ :stream | (NeoCSVWriter on: stream) emptyFieldValue: #empty; nextPut: #(one two three); nextPutAll: #( (1 2 empty) (4 empty 6) (empty 8 9)) ]) equals: (String crlf join: #( '"one","two","three"' '"1","2",""' '"4","","6"' '"","8","9"' '')). self assert: (String streamContents: [ :stream | (NeoCSVWriter on: stream) emptyFieldValue: Object new; nextPut: #(one two three); nextPutAll: #( (1 2 nil) (4 nil 6) (nil 8 9)) ]) equals: (String crlf join: #( '"one","two","three"' '"1","2","nil"' '"4","nil","6"' '"nil","8","9"' ''))! ! !NeoCSVWriterTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 6/15/2012 23:06'! testRawFieldsTestObjects self assert: (String streamContents: [ :stream | (NeoCSVWriter on: stream) nextPut: #(x y z); addRawFields: #(x y z); nextPutAll: { NeoCSVTestObject example. NeoCSVTestObject example. NeoCSVTestObject example } ]) equals: (String crlf join: #( '"x","y","z"' '100,200,300' '100,200,300' '100,200,300' ''))! ! !NeoCSVWriterTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 6/26/2012 20:39'! testRawFieldsDictionaries self assert: (String streamContents: [ :stream | (NeoCSVWriter on: stream) nextPut: #(x y z); addRawFieldsAt: #(x y z); nextPutAll: { Dictionary newFromPairs: #(x 100 y 200 z 300). Dictionary newFromPairs: #(x 400 y 500 z 600). Dictionary newFromPairs: #(x 700 y 800 z 900) } ]) equals: (String crlf join: #( '"x","y","z"' '100,200,300' '400,500,600' '700,800,900' ''))! ! !NeoCSVWriterTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 6/14/2012 20:58'! testSimple self assert: (String streamContents: [ :stream | (NeoCSVWriter on: stream) nextPut: #(one two three); nextPutAll: #( (1 2 3) (4 5 6) (7 8 9)) ]) equals: (String crlf join: #( '"one","two","three"' '"1","2","3"' '"4","5","6"' '"7","8","9"' ''))! ! TestCase subclass: #NeoNumberParserTests instanceVariableNames: '' classVariableNames: '' poolDictionaries: '' category: 'Neo-CSV-Tests'! !NeoNumberParserTests commentStamp: '<historical>' prior: 0! I am NeoNumberParserTests the unit test suite for NeoNumberParser.! !NeoNumberParserTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 12/2/2012 17:42'! testErrors self should: [ NeoNumberParser parse: nil ] raise: Error. self should: [ NeoNumberParser parse: '' ] raise: Error. self should: [ NeoNumberParser parse: '.5' ] raise: Error.! ! !NeoNumberParserTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 12/2/2012 17:02'! testOctalIntegers self assert: (NeoNumberParser parse: '173' base: 8) equals: 123. self assert: (NeoNumberParser parse: '-173' base: 8) equals: -123. self assert: (NeoNumberParser parse: '0' base: 8) equals: 0. ! ! !NeoNumberParserTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 12/2/2012 17:02'! testBinaryIntegers self assert: (NeoNumberParser parse: '1111011' base: 2) equals: 123. self assert: (NeoNumberParser parse: '-1111011' base: 2) equals: -123. self assert: (NeoNumberParser parse: '0' base: 2) equals: 0. ! ! !NeoNumberParserTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 12/2/2012 17:08'! testFloats #('1.5' 1.5 '-1.5' -1.5 '0.0' 0.0 '3.14159' 3.14159 '1e3' 1000.0 '1e-2' 0.01) pairsDo: [ :string :float | self assert: ((NeoNumberParser parse: string) closeTo: float) ]! ! !NeoNumberParserTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 5/14/2015 13:27'! testHexadecimalIntegers self assert: (NeoNumberParser parse: '7B' base: 16) equals: 123. self assert: (NeoNumberParser parse: '-7B' base: 16) equals: -123. self assert: (NeoNumberParser parse: '0' base: 16) equals: 0. "On some platforms Character>>#digitValue only handles uppercase, then NeoNumberParser cannot deal with lowercase hex characters" $a digitValue = 10 ifFalse: [ ^ self ]. self assert: (NeoNumberParser parse: '7b' base: 16) equals: 123. self assert: (NeoNumberParser parse: '-7b' base: 16) equals: -123 ! ! !NeoNumberParserTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 12/2/2012 17:03'! testDecimalIntegers self assert: (NeoNumberParser parse: '123') equals: 123. self assert: (NeoNumberParser parse: '-123') equals: -123. self assert: (NeoNumberParser parse: '0') equals: 0. self assert: (NeoNumberParser parse: '12345678901234567890') equals: 12345678901234567890. self assert: (NeoNumberParser parse: '00123ABC') equals: 123. self assert: (NeoNumberParser parse: '-0') equals: 0. ! !