[ https://issues.apache.org/jira/browse/HDFS-13744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16582703#comment-16582703 ]
Zsolt Venczel commented on HDFS-13744: -------------------------------------- After doing some more analysis it turns out that very few CSV and XML clients are following the LF character encoding specifications. This can have the following impact: * For the XML processor: Escaping the LF character following the specification can distort an XML parser to correctly reproduce a file name. It can also modify filenames when using the ReverseXML processor. *I would not recommend escaping here.* * For the Delimited processor: The output of the Delimited processor is handy for report creation and grepping where a wrongly displayed filename or directory name having LF can cause more problems than the appearance of an escaped LF character therefore *I would recommend escaping in this scenario*. In my uploaded patch I added escaping for the Delimited processor only. > OIV tool should better handle control characters present in file or directory > names > ----------------------------------------------------------------------------------- > > Key: HDFS-13744 > URL: https://issues.apache.org/jira/browse/HDFS-13744 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, tools > Affects Versions: 2.6.5, 2.9.1, 2.8.4, 2.7.6, 3.0.3 > Reporter: Zsolt Venczel > Assignee: Zsolt Venczel > Priority: Critical > Attachments: HDFS-13744.01.patch > > > In certain cases when control characters or white space is present in file or > directory names OIV tool processors can export data in a misleading format. > In the below examples we have EXAMPLE_NAME as a file and a directory name > where the directory has a line feed character at the end (the actual > production case has multiple line feeds and multiple spaces) > * CSV processor case: > ** misleading example: > {code:java} > /user/data/EXAMPLE_NAME > ,0,2017-04-24 04:34,1969-12-31 16:00,0,0,0,-1,-1,drwxrwxr-x+,user,group > /user/data/EXAMPLE_NAME,2016-08-26 03:00,2017-05-16 > 10:05,134217728,1,520,0,0,-rw-rwxr--+,user,group > {code} > ** expected example as suggested by > [https://tools.ietf.org/html/rfc4180#section-2]: > {code:java} > "/user/data/EXAMPLE_NAME%x0A",0,2017-04-24 04:34,1969-12-31 > 16:00,0,0,0,-1,-1,drwxrwxr-x+,user,group > "/user/data/EXAMPLE_NAME",2016-08-26 03:00,2017-05-16 > 10:05,134217728,1,520,0,0,-rw-rwxr--+,user,group > {code} > * XML processor case: > ** misleading example: > {code:java} > <inode><id>479867791</id><type>DIRECTORY</type><name>EXAMPLE_NAME > </name><mtime>1493033668294</mtime><permission>user:group:0775</permission></inode> > <inode><id>113632535</id><type>FILE</type><name>EXAMPLE_NAME</name><replication>3</replication><mtime>1472205657504</mtime><atime>1494954320141</atime><preferredBlockSize>134217728</preferredBlockSize><permission>user:group:0674</permission></inode> > {code} > ** expected example as specified in > [https://www.w3.org/TR/REC-xml/#sec-line-ends]: > {code:java} > <inode><id>479867791</id><type>DIRECTORY</type><name>EXAMPLE_NAME#xA</name><mtime>1493033668294</mtime><permission>user:group:0775</permission></inode> > <inode><id>113632535</id><type>FILE</type><name>EXAMPLE_NAME</name><replication>3</replication><mtime>1472205657504</mtime><atime>1494954320141</atime><preferredBlockSize>134217728</preferredBlockSize><permission>user:group:0674</permission></inode> > {code} > * JSON: > The OIV Web Processor behaves correctly and produces the following: > {code:java} > { > "FileStatuses": { > "FileStatus": [ > { > "fileId": 113632535, > "accessTime": 1494954320141, > "replication": 3, > "owner": "user", > "length": 520, > "permission": "674", > "blockSize": 134217728, > "modificationTime": 1472205657504, > "type": "FILE", > "group": "group", > "childrenNum": 0, > "pathSuffix": "EXAMPLE_NAME" > }, > { > "fileId": 479867791, > "accessTime": 0, > "replication": 0, > "owner": "user", > "length": 0, > "permission": "775", > "blockSize": 0, > "modificationTime": 1493033668294, > "type": "DIRECTORY", > "group": "group", > "childrenNum": 0, > "pathSuffix": "EXAMPLE_NAME\n" > } > ] > } > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org