Hi Przemek,
> The original behavior was intentional and fully consistence.
Sorry I don't understand how can we have a Tokenizer return 1 token when
tokenizing an empty string, regardless of the delimiter. IMO it is not
expected nor consistent with my understanding of tokenizing .
> I can understand that you enabled above exception for situations when no
> delimiter
> is passed and the default space stripping mode is enabled. But when user
> enables
> his own delimiter then for me it's odd that in current xHarbour code it's
> hardcoded
> that empty token is ignored when only one token exists.
>
> ? len( hb_aTokens( "" , "|" ) )
> ? len( hb_aTokens( "|" , "|" ) )
> ? len( hb_aTokens( "||" , "|" ) )
>
> do you find above results natural?
Sorry I really don't understand. I expect 0, 2, 3 or 0,1,2 depending on
whether terminator delimiter is expected or not (IMO there should be an
additional argument to control such behavior). What results would you
expect, and more importantly WHY?
IMO there is a clear difference between the following 2 situations:
1. DELIMITER is DEFINED and MATCHED, and therefore brings to existence an
unavoidable VIRTUAL TOKEN after each instance of a FOUND DELIMITER, and such
virtual token is therefore EMPTY when it does not have any physical
presence.
2 DELIMITER is DEFINED but NOT MATCHED, and therefore no virtual token is
implied, and the token could only exist EXPLICTLTLY.
> Please note that now it's not possible to serialize (convert to text) such
> array
> { "" }
> so later it can be restored from string by hb_aTokens() function.
Sorry, I don't understand why? What logic do you use to de-serialize such
array?
I would assume:
//-------------------------------------------------------------------------//
PROCEDURE Main()
LOCAL aArray := {""}
? Serialize( aArray )
? Serialize( DeSerialize( Serialize( aArray ) ) )
RETURN
FUNCTION Serialize( xValue )
SWITCH ValType( xValue )
CASE 'A'
RETURN SerializeArray( xValue )
CASE 'C'
RETURN '"' + xValue + '"'
CASE 'N'
RETURN Str( xValue )
CASE 'D'
RETURN "STOD(" + DTOS( xValue ) + ")"
CASE 'L'
RETURN IF( xValue, ".T.", ".F." )
END
Alert( "Unsupported type!" )
RETURN ""
FUNCTION SerializeArray( aArray )
LOCAL sList := "{", xValue
FOR EACH xValue IN aArray
sList += Serialize( xValue ) + ","
NEXT
sList[-1] := "}"
RETURN sList
FUNCTION DeSerialize( sValue )
LOCAL sList, sToken, aArray
IF sValue[1] == '{'
sList := SubStr( sValue, 2, Len( sValue ) - 2 )
aArray := {}
FOR EACH sToken IN hb_aTokens( sList, ",", .T. )
aAdd( aArray, &( sToken ) )
NEXT
RETURN aArray
ENDIF
RETURN &( sValue )
//-------------------------------------------------------------------------//
Ron
------------------------------------------------------------------------------
What Every C/C++ and Fortran developer Should Know!
Read this article and learn how Intel has extended the reach of its
next-generation tools to help Windows* and Linux* C/C++ and Fortran
developers boost performance applications - including clusters.
http://p.sf.net/sfu/intel-dev2devmay
_______________________________________________
xHarbour-developers mailing list
xHarbour-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xharbour-developers
