Hi JAmes,
My apologies for the delayed reply. The drill version is 1.20.1 and below
is JSON profile
{
"id": {
"part1": 2089923342900841500,
"part2": -6273738056146546000
},
"type": 1,
"start": 1660885450907,
"end": 1660885532000,
"query": "select * from parquetstore.table1 where dir0='2018' and
dir1>='1' and dir1<='3' and gid='01U000041'",
"plan": "00-00 Screen : rowType = RecordType(DYNAMIC_STAR **):
rowcount = 1000.0, cumulative cost = {3.01451237E8 rows, 1.3135629269775E9
cpu, 4.01925516E8 io, 1.6384E7 network, 0.0 memory}, id = 935\n00-01
Project(**=[$0]) : rowType = RecordType(DYNAMIC_STAR **): rowcount =
1000.0, cumulative cost = {3.01451137E8 rows, 1.3135628269775E9 cpu,
4.01925516E8 io, 1.6384E7 network, 0.0 memory}, id = 934\n00-02
Project(T2¦¦**=[$0]) : rowType = RecordType(DYNAMIC_STAR T2¦¦**): rowcount
= 1000.0, cumulative cost = {3.01450137E8 rows, 1.3135618269775E9 cpu,
4.01925516E8 io, 1.6384E7 network, 0.0 memory}, id = 931\n00-03
SelectionVectorRemover : rowType = RecordType(DYNAMIC_STAR T2¦¦**, ANY
dir0, ANY dir1, ANY gid): rowcount = 1000.0, cumulative cost =
{3.01449137E8 rows, 1.3135608269775E9 cpu, 4.01925516E8 io, 1.6384E7
network, 0.0 memory}, id = 930\n00-04 Limit(fetch=[1000]) :
rowType = RecordType(DYNAMIC_STAR T2¦¦**, ANY dir0, ANY dir1, ANY gid):
rowcount = 1000.0, cumulative cost = {3.01448137E8 rows, 1.3135598269775E9
cpu, 4.01925516E8 io, 1.6384E7 network, 0.0 memory}, id = 929\n00-05
UnionExchange : rowType = RecordType(DYNAMIC_STAR T2¦¦**, ANY dir0,
ANY dir1, ANY gid): rowcount = 1000.0, cumulative cost = {3.01447137E8
rows, 1.3135558269775E9 cpu, 4.01925516E8 io, 1.6384E7 network, 0.0
memory}, id = 928\n01-01 SelectionVectorRemover : rowType =
RecordType(DYNAMIC_STAR T2¦¦**, ANY dir0, ANY dir1, ANY gid): rowcount =
1000.0, cumulative cost = {3.01446137E8 rows, 1.3135478269775E9 cpu,
4.01925516E8 io, 0.0 network, 0.0 memory}, id = 927\n01-02
Limit(fetch=[1000]) : rowType = RecordType(DYNAMIC_STAR T2¦¦**, ANY dir0,
ANY dir1, ANY gid): rowcount = 1000.0, cumulative cost = {3.01445137E8
rows, 1.3135468269775E9 cpu, 4.01925516E8 io, 0.0 network, 0.0 memory}, id
= 926\n01-03 Filter(condition=[AND(=($1, '2018'), >=($2,
'1'), <=($2, '3'), =($3, '01U000041'))]) : rowType =
RecordType(DYNAMIC_STAR T2¦¦**, ANY dir0, ANY dir1, ANY gid): rowcount =
565207.756875, cumulative cost = {3.01444137E8 rows, 1.3135428269775E9 cpu,
4.01925516E8 io, 0.0 network, 0.0 memory}, id = 925\n01-04
Project(T2¦¦**=[$0], dir0=[$1], dir1=[$2], gid=[$3]) : rowType =
RecordType(DYNAMIC_STAR T2¦¦**, ANY dir0, ANY dir1, ANY gid): rowcount =
1.00481379E8, cumulative cost = {2.00962758E8 rows, 8.03851032E8 cpu,
4.01925516E8 io, 0.0 network, 0.0 memory}, id = 924\n01-05
Scan(table=[[]], groupscan=[ParquetGroupScan
[entries=[ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/4/table1
~155~3558~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/4/1/table1
~155~3587~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/3/table1
~155~3599~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/2/table1
~155~3465~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/1/table1
~155~3603~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/1/1/table1
~155~3600~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/2/table1
~155~3613~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/4/1/table1
~155~3599~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/4/1/table1
~155~3558~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/3/1/table1
~155~3600~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/2/1/table1
~155~3613~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/1/table1
~155~3462~2018~JANUARY~1/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/2/1/table1
~155~3587~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/4/table1
~155~3537~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/4/1/table1
~155~3603~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/1/table1
~155~3613~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/1/table1
~155~3598~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/1/table1
~155~3537~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/2/1/table1
~155~3599~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/1/table1
~155~3465~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/4/table1
~155~3465~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/4/table1
~155~3613~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/4/table1
~155~3600~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/3/table1
~155~3603~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/2/table1
~155~3599~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/1/table1
~155~3459~2018~JANUARY~1/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/3/table1
~155~3537~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/3/1/table1
~155~3603~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/1/table1
~155~3587~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/2/table1
~155~3600~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/4/table1
~155~3599~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/3/table1
~155~3603~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/3/table1
~155~3598~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/3/1/table1
~155~3558~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/1/table1
~155~3459~2018~JANUARY~1~col_added/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/4/1/table1
~155~3613~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/3/table1
~155~3537~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/2/table1
~155~3603~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/4/1/table1
~155~3600~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/3/table1
~155~3465~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/2/table1
~155~3603~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/4/1/table1
~155~3465~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/4/table1
~155~3599~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/3/table1
~155~3598~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/4/table1
~155~3587~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/1/table1
~155~3599~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/1/1/table1
~155~3465~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/2/table1
~155~3600~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/3/1/table1
~155~3599~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/3/table1
~155~3613~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/1/1/table1
~155~3613~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/2/1/table1
~155~3598~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/3/table1
~155~3558~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/1/table1
~155~3587~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/3/table1
~155~3465~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/3/1/table1
~155~3587~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/3/1/table1
~155~3537~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/3/table1
~155~3558~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/3/table1
~155~3613~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/1/table1
~155~3459~2018~JANUARY~1~col_deleted/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/1/1/table1
~155~3537~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/4/table1
~155~3587~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/4/table1
~155~3600~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/1/1/table1
~155~3598~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/1/table1
~155~3598~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/2/table1
~155~3613~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/4/table1
~155~3558~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/2/1/table1
~155~3537~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/2/table1
~155~3598~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/2/table1
~155~3465~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/3/table1
~155~3600~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/1/table1
~155~3599~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/4/table1
~155~3465~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/1/1/table1
~155~3558~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/3/1/table1
~155~3598~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/1/table1
~155~3613~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/1/table1
~155~3600~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/1/1/table1
~155~3603~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/1/table1
~155~3465~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/4/table1
~155~3613~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/2/table1
~155~3599~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/4/table1
~155~3598~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/1/table1
~155~3537~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/4/table1
~155~3598~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/4/table1
~155~3537~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/2/table1
~155~3587~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/2/table1
~155~3587~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/2/table1
~155~3537~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/4/1/table1
~155~3537~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/1/table1
~155~3600~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/2/1/table1
~155~3465~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/3/1/table1
~155~3613~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/4/table1
~155~3603~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/2/table1
~155~3558~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/1/table1
~155~3558~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/1/1/table1
~155~3587~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/3/1/table1
~155~3465~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/3/table1
~155~3600~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/1/1/table1
~155~3599~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/1/table1
~155~3558~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/2/1/table1
~155~3603~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/2/table1
~155~3537~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/2/1/table1
~155~3558~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/4/1/table1
~155~3598~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/2/table1
~155~3558~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/3/3/table1
~155~3587~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/1/table1
~155~3603~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/4/table1
~155~3603~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/1/table1
~155~3459~2018~JANUARY~1~col_added_and_deleted/0_0_0.parquet],
ReadEntryWithPath [path=/remote_t1/archived_files1/parquetstore/table1
/2018/2/3/table1 ~155~3599~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/2/table1
~155~3598~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/3/table1
~155~3587~2018~MAY~7/0_0_0.parquet], ReadEntryWithPath
[path=/remote_t1/archived_files1/parquetstore/table1 /2018/2/2/1/table1
~155~3600~2018~MAY~7/0_0_0.parquet]],
selectionRoot=file:/remote_t1/archived_files1/parquetstore/table1 ,
numFiles=113, numRowGroups=113, usedMetadataFile=false,
usedMetastore=false, filter=booleanAnd(equal(`dir0`, '2018') ,
greater_than_or_equal_to(`dir1`, '1') , less_than_or_equal_to(`dir1`, '3')
, equal(`gid`, '01U000041') ) , columns=[`**`, `dir0`, `dir1`, `gid`]]]) :
rowType = RecordType(DYNAMIC_STAR **, ANY dir0, ANY dir1, ANY gid):
rowcount = 1.00481379E8, cumulative cost = {1.00481379E8 rows, 4.01925516E8
cpu, 4.01925516E8 io, 0.0 network, 0.0 memory}, id = 923\n",
"foreman": {
"address": xxx.xx.xxx.xxx",
"userPort": 31010,
"controlPort": 31011,
"dataPort": 31012,
"version": "1.20.1",
"state": 0
},
"state": 4,
"totalFragments": 4,
"finishedFragments": 0,
"fragmentProfile": [
{
"majorFragmentId": 0,
"minorFragmentProfile": [
{
"state": 2,
"minorFragmentId": 0,
"operatorProfile": [
{
"inputProfile": [
{
"records": 3,
"batches": 4,
"schemas": 2
}
],
"operatorId": 5,
"operatorType": 11,
"setupNanos": 0,
"processNanos": 5961213,
"peakLocalMemoryAllocated": 16384,
"metric": [
{
"metricId": 0,
"longValue": 0
},
{
"metricId": 1,
"longValue": 3
}
],
"waitNanos": 33600908972,
"operatorTypeName": "UNORDERED_RECEIVER"
},
{
"inputProfile": [
{
"records": 3,
"batches": 4,
"schemas": 2
}
],
"operatorId": 4,
"operatorType": 7,
"setupNanos": 386455,
"processNanos": 1632941,
"peakLocalMemoryAllocated": 4911104,
"waitNanos": 0,
"operatorTypeName": "LIMIT"
},
{
"inputProfile": [
{
"records": 3,
"batches": 4,
"schemas": 2
}
],
"operatorId": 3,
"operatorType": 14,
"setupNanos": 422660,
"processNanos": 1535024,
"peakLocalMemoryAllocated": 4911104,
"waitNanos": 0,
"operatorTypeName": "SELECTION_VECTOR_REMOVER"
},
{
"inputProfile": [
{
"records": 3,
"batches": 4,
"schemas": 2
}
],
"operatorId": 2,
"operatorType": 10,
"setupNanos": 4409001,
"processNanos": 1856811,
"peakLocalMemoryAllocated": 4751360,
"waitNanos": 0,
"operatorTypeName": "PROJECT"
},
{
"inputProfile": [
{
"records": 3,
"batches": 4,
"schemas": 2
}
],
"operatorId": 1,
"operatorType": 10,
"setupNanos": 3799823,
"processNanos": 1649806,
"peakLocalMemoryAllocated": 4751360,
"waitNanos": 0,
"operatorTypeName": "PROJECT"
},
{
"inputProfile": [
{
"records": 3,
"batches": 4,
"schemas": 2
}
],
"operatorId": 0,
"operatorType": 13,
"setupNanos": 0,
"processNanos": 8094781,
"peakLocalMemoryAllocated": 0,
"waitNanos": 18112145,
"operatorTypeName": "SCREEN"
}
],
"startTime": 1660885473038,
"endTime": 1660885528771,
"memoryUsed": 6000000,
"maxMemoryUsed": 13822208,
"endpoint": {
"address": xxx.xx.xxx.xxx",
"userPort": 31010,
"controlPort": 31011,
"dataPort": 31012,
"version": "1.20.1",
"state": 0
},
"lastUpdate": 1660885528771,
"lastProgress": 1660885508760
}
]
},
{
"majorFragmentId": 1,
"minorFragmentProfile": [
{
"state": 6,
"minorFragmentId": 0,
"operatorProfile": [
{
"inputProfile": [
{
"records": 16324784,
"batches": 1105,
"schemas": 1
}
],
"operatorId": 5,
"operatorType": 21,
"setupNanos": 0,
"processNanos": 50741966229,
"peakLocalMemoryAllocated": 160948224,
"metric": [
{
"metricId": 0,
"longValue": 38
},
{
"metricId": 2,
"longValue": 0
},
{
"metricId": 7,
"longValue": 0
},
{
"metricId": 19,
"longValue": 7214523883
},
{
"metricId": 3,
"longValue": 69513
},
{
"metricId": 13,
"longValue": 94710129
},
{
"metricId": 21,
"longValue": 28718488617
},
{
"metricId": 11,
"longValue": 0
},
{
"metricId": 5,
"longValue": 0
},
{
"metricId": 15,
"longValue": 0
},
{
"metricId": 12,
"longValue": 4308563531
},
{
"metricId": 16,
"longValue": 0
},
{
"metricId": 8,
"longValue": 0
},
{
"metricId": 1,
"longValue": 0
},
{
"metricId": 10,
"longValue": 0
},
{
"metricId": 14,
"longValue": 0
},
{
"metricId": 9,
"longValue": 0
},
{
"metricId": 18,
"longValue": 0
},
{
"metricId": 4,
"longValue": 69513
},
{
"metricId": 6,
"longValue": 0
},
{
"metricId": 20,
"longValue": 21492489764
},
{
"metricId": 17,
"longValue": 4308563531
}
],
"waitNanos": 4428905716,
"operatorTypeName": "PARQUET_ROW_GROUP_SCAN"
},
{
"inputProfile": [
{
"records": 16324784,
"batches": 1088,
"schemas": 1
}
],
"operatorId": 4,
"operatorType": 10,
"setupNanos": 5386231,
"processNanos": 2930485601,
"peakLocalMemoryAllocated": 15335424,
"waitNanos": 0,
"operatorTypeName": "PROJECT"
},
{
"inputProfile": [
{
"records": 16324784,
"batches": 1088,
"schemas": 1
}
],
"operatorId": 3,
"operatorType": 2,
"setupNanos": 7852924,
"processNanos": 420310428,
"peakLocalMemoryAllocated": 15368192,
"waitNanos": 0,
"operatorTypeName": "FILTER"
},
{
"inputProfile": [
{
"records": 0,
"batches": 1088,
"schemas": 1
}
],
"operatorId": 2,
"operatorType": 7,
"setupNanos": 183244,
"processNanos": 221612188,
"peakLocalMemoryAllocated": 4911104,
"waitNanos": 0,
"operatorTypeName": "LIMIT"
},
{
"inputProfile": [
{
"records": 0,
"batches": 1,
"schemas": 1
}
],
"operatorId": 1,
"operatorType": 14,
"setupNanos": 216571,
"processNanos": 1964317,
"peakLocalMemoryAllocated": 4911104,
"waitNanos": 0,
"operatorTypeName": "SELECTION_VECTOR_REMOVER"
},
{
"inputProfile": [
{
"records": 0,
"batches": 1,
"schemas": 1
}
],
"operatorId": 0,
"operatorType": 0,
"setupNanos": 0,
"processNanos": 323036,
"peakLocalMemoryAllocated": 0,
"metric": [
{
"metricId": 0,
"longValue": 0
}
],
"waitNanos": 150967,
"operatorTypeName": "SINGLE_SENDER"
}
],
"startTime": 1660885473040,
"endTime": 1660885531983,
"memoryUsed": 138800960,
"maxMemoryUsed": 180263616,
"endpoint": {
"address": xxx.xx.xxx.xxx",
"userPort": 31010,
"controlPort": 31011,
"dataPort": 31012,
"version": "1.20.1",
"state": 1,
"httpPort": 8047
},
"lastUpdate": 1660885531983,
"lastProgress": 1660885531983
},
{
"state": 6,
"minorFragmentId": 2,
"operatorProfile": [
{
"inputProfile": [
{
"records": 16753632,
"batches": 1139,
"schemas": 1
}
],
"operatorId": 5,
"operatorType": 21,
"setupNanos": 0,
"processNanos": 50425825887,
"peakLocalMemoryAllocated": 160948224,
"metric": [
{
"metricId": 0,
"longValue": 37
},
{
"metricId": 15,
"longValue": 0
},
{
"metricId": 17,
"longValue": 4615057992
},
{
"metricId": 12,
"longValue": 4615207172
},
{
"metricId": 19,
"longValue": 7355052180
},
{
"metricId": 18,
"longValue": 0
},
{
"metricId": 11,
"longValue": 0
},
{
"metricId": 5,
"longValue": 0
},
{
"metricId": 16,
"longValue": 2200333
},
{
"metricId": 10,
"longValue": 2481
},
{
"metricId": 1,
"longValue": 0
},
{
"metricId": 13,
"longValue": 106356162
},
{
"metricId": 8,
"longValue": 217
},
{
"metricId": 14,
"longValue": 0
},
{
"metricId": 6,
"longValue": 79
},
{
"metricId": 7,
"longValue": 0
},
{
"metricId": 2,
"longValue": 0
},
{
"metricId": 21,
"longValue": 29729954304
},
{
"metricId": 4,
"longValue": 73775
},
{
"metricId": 3,
"longValue": 73782
},
{
"metricId": 9,
"longValue": 0
},
{
"metricId": 20,
"longValue": 22344329397
}
],
"waitNanos": 4589770743,
"operatorTypeName": "PARQUET_ROW_GROUP_SCAN"
},
{
"inputProfile": [
{
"records": 16753632,
"batches": 1119,
"schemas": 1
}
],
"operatorId": 4,
"operatorType": 10,
"setupNanos": 24115600,
"processNanos": 2807894311,
"peakLocalMemoryAllocated": 15335424,
"waitNanos": 0,
"operatorTypeName": "PROJECT"
},
{
"inputProfile": [
{
"records": 16753632,
"batches": 1119,
"schemas": 1
}
],
"operatorId": 3,
"operatorType": 2,
"setupNanos": 32059642,
"processNanos": 443301805,
"peakLocalMemoryAllocated": 15368192,
"waitNanos": 0,
"operatorTypeName": "FILTER"
},
{
"inputProfile": [
{
"records": 2,
"batches": 1119,
"schemas": 1
}
],
"operatorId": 2,
"operatorType": 7,
"setupNanos": 224627,
"processNanos": 241532317,
"peakLocalMemoryAllocated": 15065090,
"waitNanos": 0,
"operatorTypeName": "LIMIT"
},
{
"inputProfile": [
{
"records": 2,
"batches": 3,
"schemas": 1
}
],
"operatorId": 1,
"operatorType": 14,
"setupNanos": 191232,
"processNanos": 3674631,
"peakLocalMemoryAllocated": 4911104,
"waitNanos": 0,
"operatorTypeName": "SELECTION_VECTOR_REMOVER"
},
{
"inputProfile": [
{
"records": 2,
"batches": 3,
"schemas": 1
}
],
"operatorId": 0,
"operatorType": 0,
"setupNanos": 0,
"processNanos": 13941708,
"peakLocalMemoryAllocated": 4579409,
"metric": [
{
"metricId": 0,
"longValue": 2230
}
],
"waitNanos": 626878,
"operatorTypeName": "SINGLE_SENDER"
}
],
"startTime": 1660885473040,
"endTime": 1660885531999,
"memoryUsed": 6057956,
"maxMemoryUsed": 180263616,
"endpoint": {
"address": xxx.xx.xxx.xxx",
"userPort": 31010,
"controlPort": 31011,
"dataPort": 31012,
"version": "1.20.1",
"state": 1,
"httpPort": 8047
},
"lastUpdate": 1660885531999,
"lastProgress": 1660885531999
},
{
"state": 6,
"minorFragmentId": 1,
"operatorProfile": [
{
"inputProfile": [
{
"records": 15919365,
"batches": 1083,
"schemas": 2
}
],
"operatorId": 5,
"operatorType": 21,
"setupNanos": 0,
"processNanos": 50087585055,
"peakLocalMemoryAllocated": 161144832,
"metric": [
{
"metricId": 0,
"longValue": 38
},
{
"metricId": 11,
"longValue": 0
},
{
"metricId": 13,
"longValue": 115441724
},
{
"metricId": 2,
"longValue": 0
},
{
"metricId": 18,
"longValue": 0
},
{
"metricId": 5,
"longValue": 0
},
{
"metricId": 6,
"longValue": 89
},
{
"metricId": 16,
"longValue": 3661859
},
{
"metricId": 9,
"longValue": 0
},
{
"metricId": 10,
"longValue": 2783
},
{
"metricId": 20,
"longValue": 22308111595
},
{
"metricId": 15,
"longValue": 0
},
{
"metricId": 17,
"longValue": 5154432183
},
{
"metricId": 4,
"longValue": 69689
},
{
"metricId": 14,
"longValue": 0
},
{
"metricId": 8,
"longValue": 0
},
{
"metricId": 19,
"longValue": 6806062135
},
{
"metricId": 12,
"longValue": 5154432183
},
{
"metricId": 7,
"longValue": 0
},
{
"metricId": 21,
"longValue": 29126131786
},
{
"metricId": 1,
"longValue": 0
},
{
"metricId": 3,
"longValue": 69689
}
],
"waitNanos": 5170645651,
"operatorTypeName": "PARQUET_ROW_GROUP_SCAN"
},
{
"inputProfile": [
{
"records": 15919365,
"batches": 1064,
"schemas": 2
}
],
"operatorId": 4,
"operatorType": 10,
"setupNanos": 38354442,
"processNanos": 2796770682,
"peakLocalMemoryAllocated": 15515648,
"waitNanos": 0,
"operatorTypeName": "PROJECT"
},
{
"inputProfile": [
{
"records": 15919365,
"batches": 1064,
"schemas": 2
}
],
"operatorId": 3,
"operatorType": 2,
"setupNanos": 6053453,
"processNanos": 409967112,
"peakLocalMemoryAllocated": 15548416,
"waitNanos": 0,
"operatorTypeName": "FILTER"
},
{
"inputProfile": [
{
"records": 2,
"batches": 1064,
"schemas": 2
}
],
"operatorId": 2,
"operatorType": 7,
"setupNanos": 399696,
"processNanos": 239682814,
"peakLocalMemoryAllocated": 15065090,
"waitNanos": 0,
"operatorTypeName": "LIMIT"
},
{
"inputProfile": [
{
"records": 2,
"batches": 3,
"schemas": 2
}
],
"operatorId": 1,
"operatorType": 14,
"setupNanos": 405527,
"processNanos": 4460402,
"peakLocalMemoryAllocated": 4911104,
"waitNanos": 0,
"operatorTypeName": "SELECTION_VECTOR_REMOVER"
},
{
"inputProfile": [
{
"records": 2,
"batches": 3,
"schemas": 2
}
],
"operatorId": 0,
"operatorType": 0,
"setupNanos": 0,
"processNanos": 3042274,
"peakLocalMemoryAllocated": 4685905,
"metric": [
{
"metricId": 0,
"longValue": 2256
}
],
"waitNanos": 1271429,
"operatorTypeName": "SINGLE_SENDER"
}
],
"startTime": 1660885473040,
"endTime": 1660885531984,
"memoryUsed": 142144320,
"maxMemoryUsed": 180263616,
"endpoint": {
"address": xxx.xx.xxx.xxx",
"userPort": 31010,
"controlPort": 31011,
"dataPort": 31012,
"version": "1.20.1",
"state": 1,
"httpPort": 8047
},
"lastUpdate": 1660885531984,
"lastProgress": 1660885531984
}
]
}
],
"user": "anonymous",
"error": "SYSTEM ERROR: Drill Remote Exception\n\n\nPlease, refer to
logs for more information.\n",
"verboseError": "SYSTEM ERROR: Drill Remote Exception\n\n\nPlease,
refer to logs for more information.\n\n\n
(java.lang.IllegalStateException) java.lang.IndexOutOfBoundsException:
readerIndex: 0, writerIndex: 2013265920 (expected: 0 <= readerIndex <=
writerIndex <= capacity(0))\n
org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.doWork():79\n
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():103\n
org.apache.drill.exec.record.AbstractRecordBatch.next():170\n
org.apache.drill.exec.record.AbstractRecordBatch.next():119\n
org.apache.drill.exec.record.AbstractRecordBatch.next():111\n
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():59\n
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():85\n
org.apache.drill.exec.record.AbstractRecordBatch.next():170\n
org.apache.drill.exec.record.AbstractRecordBatch.next():119\n
org.apache.drill.exec.record.AbstractRecordBatch.next():111\n
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():59\n
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():85\n
org.apache.drill.exec.record.AbstractRecordBatch.next():170\n
org.apache.drill.exec.physical.impl.BaseRootExec.next():103\n
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():81\n
org.apache.drill.exec.physical.impl.BaseRootExec.next():93\n
org.apache.drill.exec.work.fragment.FragmentExecutor.lambda$run$0():321\n
.......():0\n
org.apache.hadoop.security.UserGroupInformation.doAs():1762\n
org.apache.drill.exec.work.fragment.FragmentExecutor.run():310\n
org.apache.drill.common.SelfCleaningRunnable.run():38\n .......():0\n
Caused By (java.lang.IndexOutOfBoundsException) readerIndex: 0,
writerIndex: 2013265920 (expected: 0 <= readerIndex <= writerIndex <=
capacity(0))\n io.netty.buffer.AbstractByteBuf.checkIndexBounds():112\n
io.netty.buffer.AbstractByteBuf.writerIndex():135\n
org.apache.drill.exec.vector.VarCharVector$Mutator.setValueCount():854\n
org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setValueCount():679\n
org.apache.drill.exec.record.VectorAccessibleUtilities.setValueCount():49\n
org.apache.drill.exec.record.VectorContainer.setValueCount():468\n
org.apache.drill.exec.physical.impl.svremover.AbstractCopier.updateCounts():76\n
org.apache.drill.exec.physical.impl.svremover.AbstractSV2Copier.copyRecords():60\n
org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.doWork():76\n
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():103\n
org.apache.drill.exec.record.AbstractRecordBatch.next():170\n
org.apache.drill.exec.record.AbstractRecordBatch.next():119\n
org.apache.drill.exec.record.AbstractRecordBatch.next():111\n
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():59\n
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():85\n
org.apache.drill.exec.record.AbstractRecordBatch.next():170\n
org.apache.drill.exec.record.AbstractRecordBatch.next():119\n
org.apache.drill.exec.record.AbstractRecordBatch.next():111\n
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():59\n
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():85\n
org.apache.drill.exec.record.AbstractRecordBatch.next():170\n
org.apache.drill.exec.physical.impl.BaseRootExec.next():103\n
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():81\n
org.apache.drill.exec.physical.impl.BaseRootExec.next():93\n
org.apache.drill.exec.work.fragment.FragmentExecutor.lambda$run$0():321\n
.......():0\n
org.apache.hadoop.security.UserGroupInformation.doAs():1762\n
org.apache.drill.exec.work.fragment.FragmentExecutor.run():310\n
org.apache.drill.common.SelfCleaningRunnable.run():38\n .......():0\n",
"errorId": "42f43fca-75c9-4e58-a306-c2d3e36ba482",
"optionsJson": "[ {\n \"kind\" : \"LONG\",\n \"accessibleScopes\" :
\"ALL\",\n \"name\" : \"exec.query.max_rows\",\n \"num_val\" : 1000,\n
\"scope\" : \"SESSION\"\n} ]",
"planEnd": 1660885473038,
"queueWaitEnd": 1660885473038,
"totalCost": 201535965.756875,
"queueName": "Unknown",
"queryId": "1d00e635-1d66-cb86-a8ef-347a45891445",
"autoLimit": 1000
}
On Wed, Aug 17, 2022 at 1:17 PM James Turton <[email protected]> wrote:
Please attach the query profile JSON disclose the version of Drill.
On 2022/08/17 08:51, Prabhakar Bhosale wrote:
Hi Team,
Any pointers on the below error? thx
Regards
Prabhakar
On Tue, Aug 16, 2022 at 9:05 AM Prabhakar Bhosale <[email protected]
wrote:
Hi Team,
I am querying parquet files as per below query and getting the error.
Please advise
select * from parquetstore.table1 where dir0='2018' and dir1>='1' and
dir1<='3' and gid='01U000041'
Getting error
org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
IndexOutOfBoundsException: readerIndex: 0, writerIndex: 1946157056
(expected: 0 <= readerIndex <= writerIndex <= capacity(0)) Fragment: 0:0
Please, refer to logs for more information. [Error Id:
b335690a-a814-41dc-ba5f-a3c4d8e3ccc2
