alamb commented on PR #3976: URL: https://github.com/apache/arrow-datafusion/pull/3976#issuecomment-1302199781
> I could be mistaken but I believe static destructors are not run, so this will leak the file. This is likely not ideal... Particularly on linux machines where /tmp is typically backed by memory I think you are correct . I added [be746ac](https://github.com/apache/arrow-datafusion/pull/3976/commits/be746ac879fac0c12fcda33bb5dfe37e781d5ee4) to drop the temp file but yet still only created it once. Can you give it a look? I tested this code by running ``` cargo test -p datafusion --test parquet_filter_pushdown -- --nocapture ``` And you can see the file is created once : ``` ... Writing test data to "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" ... ``` And then cleaned up afterwards ```shell $ ls /var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet ls: /var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet: No such file or directory ``` <details> <summary>Full logs:</summary> ``` running 12 tests Writing test data to "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" Generated test dataset with 53819 rows Completed generating test data in 2.498437474s Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: false, reorder_filters: false, enable_page_index: false } reading with filter container != Utf8("backend_container_0") Using pre-existing file Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: false, reorder_filters: false, enable_page_index: false } reading with filter container = Utf8("backend_container_0") OR pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") Using pre-existing file Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: false, reorder_filters: false, enable_page_index: false } reading with filter container = Utf8("backend_container_0") Using pre-existing file Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: false, reorder_filters: false, enable_page_index: false } reading with filter response_status = UInt16(429) Using pre-existing file Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: false, reorder_filters: false, enable_page_index: false } reading with filter container = Utf8("backend_container_0") AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") Using pre-existing file Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: false, reorder_filters: false, enable_page_index: false } reading with filter request_bytes > Int32(2000000000) AND container = Utf8("backend_container_0") AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") Using pre-existing file Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: false, reorder_filters: false, enable_page_index: false } reading with filter request_method = Utf8("POST") AND response_status = UInt16(503) Using pre-existing file Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: false, reorder_filters: false, enable_page_index: false } reading with filter request_bytes > Int32(2000000000) AND container = Utf8("backend_container_0") AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") Using pre-existing file Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: false, reorder_filters: false, enable_page_index: false } reading with filter response_status > UInt16(0) Using pre-existing file Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: false, reorder_filters: false, enable_page_index: false } Using pre-existing file reading with filter request_method = Utf8("GET") Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: false, reorder_filters: false, enable_page_index: false } reading with filter request_method != Utf8("GET") OR response_status = UInt16(400) OR service = Utf8("backend") Using pre-existing file Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: false, reorder_filters: false, enable_page_index: false } reading with filter request_method != Utf8("GET") Filter: container != Utf8("backend_container_0"), total records: 53819, after filter: 15963, selectivty: 0.29660528809528236 pushdown_rows_filtered: 0 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: false, enable_page_index: false } reading with filter container != Utf8("backend_container_0") Filter: container = Utf8("backend_container_0") OR pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg"), total records: 53819, after filter: 39982, selectivty: 0.7428974897341087 pushdown_rows_filtered: 0 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: false, enable_page_index: false } reading with filter container = Utf8("backend_container_0") OR pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") Filter: container = Utf8("backend_container_0"), total records: 53819, after filter: 37856, selectivty: 0.7033947119047177 pushdown_rows_filtered: 0 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: false, enable_page_index: false } reading with filter container = Utf8("backend_container_0") Filter: request_bytes > Int32(2000000000) AND container = Utf8("backend_container_0") AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg"), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 Filter: container = Utf8("backend_container_0") AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg"), total records: 53819, after filter: 3052, selectivty: 0.05670859733551348 Filter: response_status = UInt16(429), total records: 53819, after filter: 0, selectivty: 0 pushdown_rows_filtered: 0 pushdown_rows_filtered: 0 pushdown_rows_filtered: 0 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: false, enable_page_index: false } reading with filter response_status = UInt16(429) Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: false, enable_page_index: false } reading with filter request_bytes > Int32(2000000000) AND container = Utf8("backend_container_0") AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") Filter: request_method = Utf8("POST") AND response_status = UInt16(503), total records: 53819, after filter: 1729, selectivty: 0.0321262007841097 Filter: response_status = UInt16(429), total records: 53819, after filter: 0, selectivty: 0 Filter: response_status > UInt16(0), total records: 53819, after filter: 53819, selectivty: 1 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: false, enable_page_index: false } reading with filter container = Utf8("backend_container_0") AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") pushdown_rows_filtered: 0 pushdown_rows_filtered: 0 pushdown_rows_filtered: 53819 Filter: request_bytes > Int32(2000000000) AND container = Utf8("backend_container_0") AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg"), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: true, enable_page_index: false } reading with filter response_status = UInt16(429) Filter: request_method = Utf8("GET"), total records: 53819, after filter: 8886, selectivty: 0.16510897638380498 Filter: request_method != Utf8("GET") OR response_status = UInt16(400) OR service = Utf8("backend"), total records: 53819, after filter: 53819, selectivty: 1 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" pushdown_rows_filtered: 0 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: false, enable_page_index: false } reading with filter response_status > UInt16(0) scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: false, enable_page_index: false } reading with filter request_method = Utf8("POST") AND response_status = UInt16(503) pushdown_rows_filtered: 0 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: false, enable_page_index: false } reading with filter request_method = Utf8("GET") pushdown_rows_filtered: 0 Filter: request_method != Utf8("GET"), total records: 53819, after filter: 44933, selectivty: 0.834891023616195 Filter: response_status = UInt16(429), total records: 53819, after filter: 0, selectivty: 0 pushdown_rows_filtered: 0 pushdown_rows_filtered: 53819 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: false, enable_page_index: false } reading with filter request_method != Utf8("GET") OR response_status = UInt16(400) OR service = Utf8("backend") Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: false, enable_page_index: false } reading with filter request_bytes > Int32(2000000000) AND container = Utf8("backend_container_0") AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") Filter: container != Utf8("backend_container_0"), total records: 53819, after filter: 15963, selectivty: 0.29660528809528236 Filter: request_bytes > Int32(2000000000) AND container = Utf8("backend_container_0") AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg"), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 Filter: container = Utf8("backend_container_0") AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg"), total records: 53819, after filter: 3052, selectivty: 0.05670859733551348 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: false, enable_page_index: false } reading with filter request_method != Utf8("GET") pushdown_rows_filtered: 53731 pushdown_rows_filtered: 50767 test everything ... pushdown_rows_filtered: 37856 ok Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: true, enable_page_index: false } reading with filter request_bytes > Int32(2000000000) AND container = Utf8("backend_container_0") AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: true, enable_page_index: false } reading with filter container = Utf8("backend_container_0") AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") Filter: container = Utf8("backend_container_0"), total records: 53819, after filter: 37856, selectivty: 0.7033947119047177 Filter: container = Utf8("backend_container_0") OR pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg"), total records: 53819, after filter: 39982, selectivty: 0.7428974897341087 pushdown_rows_filtered: 15963 pushdown_rows_filtered: 13837 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: true, enable_page_index: false } reading with filter container != Utf8("backend_container_0") Filter: request_bytes > Int32(2000000000) AND container = Utf8("backend_container_0") AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg"), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 53731 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: true, enable_page_index: false } reading with filter request_bytes > Int32(2000000000) AND container = Utf8("backend_container_0") AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") Filter: request_bytes > Int32(2000000000) AND container = Utf8("backend_container_0") AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg"), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 53731 Using pre-existing file Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: false, reorder_filters: false, enable_page_index: false } reading with filter request_bytes > Int32(2000000000) AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND container = Utf8("backend_container_0") Filter: container = Utf8("backend_container_0") AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg"), total records: 53819, after filter: 3052, selectivty: 0.05670859733551348 pushdown_rows_filtered: 50767 Filter: request_method = Utf8("POST") AND response_status = UInt16(503), total records: 53819, after filter: 1729, selectivty: 0.0321262007841097 pushdown_rows_filtered: 52090 Using pre-existing file Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: false, reorder_filters: false, enable_page_index: false } reading with filter pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND container = Utf8("backend_container_0") Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: true, enable_page_index: false } reading with filter request_method = Utf8("POST") AND response_status = UInt16(503) Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: true, enable_page_index: false } reading with filter container = Utf8("backend_container_0") Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: true, enable_page_index: false } reading with filter container = Utf8("backend_container_0") OR pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") Filter: request_bytes > Int32(2000000000) AND container = Utf8("backend_container_0") AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg"), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 53731 Using pre-existing file Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: false, reorder_filters: false, enable_page_index: false } reading with filter request_bytes > Int32(2000000000) AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND container = Utf8("backend_container_0") Filter: response_status > UInt16(0), total records: 53819, after filter: 53819, selectivty: 1 pushdown_rows_filtered: 0 Filter: container != Utf8("backend_container_0"), total records: 53819, after filter: 15963, selectivty: 0.29660528809528236 pushdown_rows_filtered: 37856 Filter: request_method = Utf8("GET"), total records: 53819, after filter: 8886, selectivty: 0.16510897638380498 pushdown_rows_filtered: 44933 test dict_non_selective ... ok Filter: request_method != Utf8("GET") OR response_status = UInt16(400) OR service = Utf8("backend"), total records: 53819, after filter: 53819, selectivty: 1 pushdown_rows_filtered: 0 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: true, enable_page_index: false } reading with filter request_method = Utf8("GET") Filter: request_method = Utf8("POST") AND response_status = UInt16(503), total records: 53819, after filter: 1729, selectivty: 0.0321262007841097 pushdown_rows_filtered: 52090 Using pre-existing file Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: false, reorder_filters: false, enable_page_index: false } reading with filter response_status = UInt16(503) AND request_method = Utf8("POST") Filter: request_bytes > Int32(2000000000) AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND container = Utf8("backend_container_0"), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 0 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: false, enable_page_index: false } reading with filter request_bytes > Int32(2000000000) AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND container = Utf8("backend_container_0") Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: true, enable_page_index: false } reading with filter response_status > UInt16(0) Filter: pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND container = Utf8("backend_container_0"), total records: 53819, after filter: 3052, selectivty: 0.05670859733551348 pushdown_rows_filtered: 0 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: false, enable_page_index: false } reading with filter pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND container = Utf8("backend_container_0") Filter: container = Utf8("backend_container_0"), total records: 53819, after filter: 37856, selectivty: 0.7033947119047177 pushdown_rows_filtered: 15963 Filter: request_bytes > Int32(2000000000) AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND container = Utf8("backend_container_0"), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 0 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: false, enable_page_index: false } reading with filter request_bytes > Int32(2000000000) AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND container = Utf8("backend_container_0") Filter: container = Utf8("backend_container_0") OR pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg"), total records: 53819, after filter: 39982, selectivty: 0.7428974897341087 pushdown_rows_filtered: 13837 Filter: request_bytes > Int32(2000000000) AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND container = Utf8("backend_container_0"), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 53731 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: true, enable_page_index: false } reading with filter request_bytes > Int32(2000000000) AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND container = Utf8("backend_container_0") Filter: request_method != Utf8("GET"), total records: 53819, after filter: 44933, selectivty: 0.834891023616195 pushdown_rows_filtered: 8886 Filter: pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND container = Utf8("backend_container_0"), total records: 53819, after filter: 3052, selectivty: 0.05670859733551348 pushdown_rows_filtered: 50767 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: true, enable_page_index: false } reading with filter request_method != Utf8("GET") OR response_status = UInt16(400) OR service = Utf8("backend") Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: true, enable_page_index: false } reading with filter pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND container = Utf8("backend_container_0") Filter: request_bytes > Int32(2000000000) AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND container = Utf8("backend_container_0"), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 53731 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: true, enable_page_index: false } reading with filter request_bytes > Int32(2000000000) AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND container = Utf8("backend_container_0") test dict_selective ... ok Filter: request_bytes > Int32(2000000000) AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND container = Utf8("backend_container_0"), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 53731 Using pre-existing file Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: false, reorder_filters: false, enable_page_index: false } reading with filter container = Utf8("backend_container_0") AND request_bytes > Int32(2000000000) AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") test dict_disjunction ... ok Filter: pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND container = Utf8("backend_container_0"), total records: 53819, after filter: 3052, selectivty: 0.05670859733551348 pushdown_rows_filtered: 50767 test dict_conjunction ... ok Filter: response_status = UInt16(503) AND request_method = Utf8("POST"), total records: 53819, after filter: 1729, selectivty: 0.0321262007841097 pushdown_rows_filtered: 0 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: false, enable_page_index: false } reading with filter response_status = UInt16(503) AND request_method = Utf8("POST") Filter: request_bytes > Int32(2000000000) AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND container = Utf8("backend_container_0"), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 53731 Using pre-existing file Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: false, reorder_filters: false, enable_page_index: false } reading with filter container = Utf8("backend_container_0") AND request_bytes > Int32(2000000000) AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: true, enable_page_index: false } reading with filter request_method != Utf8("GET") Filter: request_method = Utf8("GET"), total records: 53819, after filter: 8886, selectivty: 0.16510897638380498 pushdown_rows_filtered: 44933 Filter: response_status > UInt16(0), total records: 53819, after filter: 53819, selectivty: 1 pushdown_rows_filtered: 0 test selective ... ok Filter: container = Utf8("backend_container_0") AND request_bytes > Int32(2000000000) AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg"), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 0 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: false, enable_page_index: false } reading with filter container = Utf8("backend_container_0") AND request_bytes > Int32(2000000000) AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") Filter: request_method != Utf8("GET") OR response_status = UInt16(400) OR service = Utf8("backend"), total records: 53819, after filter: 53819, selectivty: 1 pushdown_rows_filtered: 0 Filter: response_status = UInt16(503) AND request_method = Utf8("POST"), total records: 53819, after filter: 1729, selectivty: 0.0321262007841097 pushdown_rows_filtered: 52090 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: true, enable_page_index: false } reading with filter response_status = UInt16(503) AND request_method = Utf8("POST") test nothing ... ok Filter: container = Utf8("backend_container_0") AND request_bytes > Int32(2000000000) AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg"), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 0 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: false, enable_page_index: false } reading with filter container = Utf8("backend_container_0") AND request_bytes > Int32(2000000000) AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") Filter: container = Utf8("backend_container_0") AND request_bytes > Int32(2000000000) AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg"), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 53731 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: true, enable_page_index: false } reading with filter container = Utf8("backend_container_0") AND request_bytes > Int32(2000000000) AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") Filter: container = Utf8("backend_container_0") AND request_bytes > Int32(2000000000) AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg"), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 53731 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: true, enable_page_index: false } reading with filter container = Utf8("backend_container_0") AND request_bytes > Int32(2000000000) AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") test dict_disjunction3 ... ok Filter: container = Utf8("backend_container_0") AND request_bytes > Int32(2000000000) AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg"), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 53731 Using pre-existing file Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: false, reorder_filters: false, enable_page_index: false } reading with filter container = Utf8("backend_container_0") AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND request_bytes > Int32(2000000000) Filter: response_status = UInt16(503) AND request_method = Utf8("POST"), total records: 53819, after filter: 1729, selectivty: 0.0321262007841097 pushdown_rows_filtered: 52090 test basic_conjunction ... ok Filter: container = Utf8("backend_container_0") AND request_bytes > Int32(2000000000) AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg"), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 53731 Using pre-existing file Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: false, reorder_filters: false, enable_page_index: false } reading with filter container = Utf8("backend_container_0") AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND request_bytes > Int32(2000000000) Filter: request_method != Utf8("GET"), total records: 53819, after filter: 44933, selectivty: 0.834891023616195 pushdown_rows_filtered: 8886 test non_selective ... ok Filter: container = Utf8("backend_container_0") AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND request_bytes > Int32(2000000000), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 0 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: false, enable_page_index: false } reading with filter container = Utf8("backend_container_0") AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND request_bytes > Int32(2000000000) Filter: container = Utf8("backend_container_0") AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND request_bytes > Int32(2000000000), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 0 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: false, enable_page_index: false } reading with filter container = Utf8("backend_container_0") AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND request_bytes > Int32(2000000000) Filter: container = Utf8("backend_container_0") AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND request_bytes > Int32(2000000000), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 53731 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: true, enable_page_index: false } reading with filter container = Utf8("backend_container_0") AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND request_bytes > Int32(2000000000) Filter: container = Utf8("backend_container_0") AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND request_bytes > Int32(2000000000), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 53731 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: true, enable_page_index: false } reading with filter container = Utf8("backend_container_0") AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND request_bytes > Int32(2000000000) Filter: container = Utf8("backend_container_0") AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND request_bytes > Int32(2000000000), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 53731 Using pre-existing file Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: false, reorder_filters: false, enable_page_index: false } reading with filter pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND request_bytes > Int32(2000000000) AND container = Utf8("backend_container_0") Filter: container = Utf8("backend_container_0") AND pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND request_bytes > Int32(2000000000), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 53731 Using pre-existing file Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: false, reorder_filters: false, enable_page_index: false } reading with filter pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND request_bytes > Int32(2000000000) AND container = Utf8("backend_container_0") Filter: pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND request_bytes > Int32(2000000000) AND container = Utf8("backend_container_0"), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 0 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: false, enable_page_index: false } reading with filter pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND request_bytes > Int32(2000000000) AND container = Utf8("backend_container_0") Filter: pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND request_bytes > Int32(2000000000) AND container = Utf8("backend_container_0"), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 0 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: false, enable_page_index: false } reading with filter pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND request_bytes > Int32(2000000000) AND container = Utf8("backend_container_0") Filter: pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND request_bytes > Int32(2000000000) AND container = Utf8("backend_container_0"), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 53731 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: true, enable_page_index: false } reading with filter pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND request_bytes > Int32(2000000000) AND container = Utf8("backend_container_0") Filter: pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND request_bytes > Int32(2000000000) AND container = Utf8("backend_container_0"), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 53731 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: true, enable_page_index: false } reading with filter pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND request_bytes > Int32(2000000000) AND container = Utf8("backend_container_0") Filter: pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND request_bytes > Int32(2000000000) AND container = Utf8("backend_container_0"), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 53731 Using pre-existing file Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: false, reorder_filters: false, enable_page_index: false } reading with filter pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND container = Utf8("backend_container_0") AND request_bytes > Int32(2000000000) Filter: pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND request_bytes > Int32(2000000000) AND container = Utf8("backend_container_0"), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 53731 Using pre-existing file Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: false, reorder_filters: false, enable_page_index: false } reading with filter pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND container = Utf8("backend_container_0") AND request_bytes > Int32(2000000000) Filter: pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND container = Utf8("backend_container_0") AND request_bytes > Int32(2000000000), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 0 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: false, enable_page_index: false } reading with filter pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND container = Utf8("backend_container_0") AND request_bytes > Int32(2000000000) Filter: pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND container = Utf8("backend_container_0") AND request_bytes > Int32(2000000000), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 0 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: false, enable_page_index: false } reading with filter pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND container = Utf8("backend_container_0") AND request_bytes > Int32(2000000000) Filter: pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND container = Utf8("backend_container_0") AND request_bytes > Int32(2000000000), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 53731 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: true, enable_page_index: false } reading with filter pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND container = Utf8("backend_container_0") AND request_bytes > Int32(2000000000) Filter: pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND container = Utf8("backend_container_0") AND request_bytes > Int32(2000000000), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 53731 Querying "/var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/.tmpeaNruZ/data.parquet" scan options: ParquetScanOptions { pushdown_filters: true, reorder_filters: true, enable_page_index: false } reading with filter pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND container = Utf8("backend_container_0") AND request_bytes > Int32(2000000000) Filter: pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND container = Utf8("backend_container_0") AND request_bytes > Int32(2000000000), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 53731 test dict_very_selective ... ok Filter: pod = Utf8("aqcathnxqsphdhgjtgvxsfyiwbmhlmg") AND container = Utf8("backend_container_0") AND request_bytes > Int32(2000000000), total records: 53819, after filter: 88, selectivty: 0.001635110277039707 pushdown_rows_filtered: 53731 test dict_very_selective2 ... ok test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.65s ``` </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
