[ https://issues.apache.org/jira/browse/IMPALA-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510383#comment-16510383 ]
Tim Armstrong commented on IMPALA-7167: --------------------------------------- I noticed this was wonky too while working on IMPALA-6941. The patch solved this by accepting them at the impalad and then failing during query execution http://gerrit.cloudera.org:8080/10165. The previous behaviour was very weird because it actually allowed unrecognised suffixes (like .foobar) by interpreting them as uncompressed text. > Catalogd should reject files with unsupported formats > ----------------------------------------------------- > > Key: IMPALA-7167 > URL: https://issues.apache.org/jira/browse/IMPALA-7167 > Project: IMPALA > Issue Type: Bug > Affects Versions: Impala 2.13.0, Impala 3.1.0 > Reporter: Tianyi Wang > Priority: Minor > > Currently when a file format is unsupported, impalad will log an error: > {noformat} > E0611 13:42:47.428393 22076 ImpaladCatalog.java:201] Error adding catalog > object: Expected compressed text file with {.lzo,.gzip,.snappy,.bz2} suffix: > 000000_0.deflate > Java exception follows: > java.lang.RuntimeException: Expected compressed text file with > {.lzo,.gzip,.snappy,.bz2} suffix: 000000_0.deflate > at > org.apache.impala.catalog.HdfsPartition.<init>(HdfsPartition.java:772) > at > org.apache.impala.catalog.HdfsPartition.fromThrift(HdfsPartition.java:884) > at > org.apache.impala.catalog.HdfsTable.loadFromThrift(HdfsTable.java:1678) > at org.apache.impala.catalog.Table.fromThrift(Table.java:311) > at > org.apache.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:403) > at > org.apache.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:292) > at > org.apache.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:199) > at > org.apache.impala.service.Frontend.updateCatalogCache(Frontend.java:228) > at > org.apache.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:174) > {noformat} > Catalogd should filter out unsupported files instead of letting every impalad > log the error. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org