Hi, I have a stream of data from Flume landing in HDFS in files of a set size. I can query these files individually just fine, and across multiple ones too - except if the wildcard encompasses the *currently open HDFS file that Flume is writing to*. When this happens, Drill understandably barfs.
0: jdbc:drill:drillbit=localhost> show files in `hdfs`.`/user/flume/incoming/twitter/2016/09/21/`; +------------------------------+--------------+---------+----------+--------+-------------+--------------+--------------------------+--------------------------+ | name | isDirectory | isFile | length | owner | group | permissions | accessTime | modificationTime | +------------------------------+--------------+---------+----------+--------+-------------+--------------+--------------------------+--------------------------+ [...] | FlumeData.1474467815652 | false | true | 1055490 | flume | supergroup | rw-r--r-- | 2016-09-21 21:52:07.219 | 2016-09-21 21:58:58.28 | | FlumeData.1474467815653 | false | true | 1050470 | flume | supergroup | rw-r--r-- | 2016-09-21 21:58:58.556 | 2016-09-21 22:06:28.636 | | FlumeData.1474467815654 | false | true | 1051043 | flume | supergroup | rw-r--r-- | 2016-09-21 22:06:29.564 | 2016-09-21 22:13:40.808 | | FlumeData.1474467815655 | false | true | 1052657 | flume | supergroup | rw-r--r-- | 2016-09-21 22:13:40.978 | 2016-09-21 22:23:00.409 | | FlumeData.1474467815656.tmp | false | true | 9447 | flume | supergroup | rw-r--r-- | 2016-09-21 22:23:00.788 | 2016-09-21 22:23:00.788 | +------------------------------+--------------+---------+----------+--------+-------------+--------------+--------------------------+--------------------------+ 59 rows selected (0.265 seconds) Note the .tmp file as the last one in the folder Querying a single file works : 0: jdbc:drill:drillbit=localhost> SELECT count(*) FROM table(`hdfs`.`/user/flume/incoming/twitter/2016/09/21/FlumeData.1474467815655`(type => 'json')); +---------+ | EXPR$0 | +---------+ | 221 | +---------+ 1 row selected (0.685 seconds) As does across multiple files where the wildcard pattern would exclude the .tmp file: 0: jdbc:drill:drillbit=localhost> SELECT count(*) FROM table(`hdfs`.`/user/flume/incoming/twitter/2016/09/21/FlumeData.147446781564*`(type => 'json')); +---------+ | EXPR$0 | +---------+ | 2178 | +---------+ 1 row selected (1.24 seconds) But if I try to query all the files, Drill includes the .tmp file and errors: 0: jdbc:drill:drillbit=localhost> SELECT count(*) FROM table(`hdfs`.`/user/flume/incoming/twitter/2016/09/21/*`(type => 'json')); Error: DATA_READ ERROR: Failure reading JSON file - Cannot obtain block length for LocatedBlock{BP-478416316-192.168.10.112-1466151126376:blk_1074004983_264343; getBlockSize()=9447; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[ 192.168.10.116:50010,DS-39bf5e74-3eec-4447-9cd2-f17b5cc259b8,DISK], DatanodeInfoWithStorage[192.168.10.113:50010,DS-845945e7-0bc8-44aa-945c-a140ad1f55ab,DISK], DatanodeInfoWithStorage[192.168.10.115:50010 ,DS-a0e97909-3d40-4f49-b67f-636e9f10928a,DISK]]} File /user/flume/incoming/twitter/2016/09/21/FlumeData.1474467815656.tmp Record 1 Fragment 0:0 [Error Id: d3f322cb-c64d-43c8-9231-fb2c96e8589d on cdh57-01-node-01.moffatt.me:31010] (state=,code=0) 0: jdbc:drill:drillbit=localhost> Is there a way around this with Drill? For example, can I use a regex in the path? I've tried, but just hit Error: VALIDATION ERROR: null thanks, Robin.