I do also see mapr15 reporting a *lot* of data in swap. It looks like
the RAM on that box could be over allocated. I haven't tested this but
I'm pretty sure it's going to be better to let Drill intentionally spill
when it thinks RAM is constrained than to tell it can have more RAM than
there is available and let it thrash the swap partition. Swapping
during query processing could also be consistent with your point 1
(performance degradation when two queries are run) if the two queries
are competing for seeks from the storage device hosting the swap. I
recommend running `smem -atk` to see which processes have been swapped
out and watching a monitoring tool that can show swapping counters like
major page faults per second during query execution.
On 2021/09/28 15:52, James Turton wrote:
Plenty of direct memory there, 50Gb on every Drillbit, far in excess
of the modest 134Mb that Drill says it can't get. Can you share a
JSON profile from an execution of this query (Web UI -> Profiles ->
Completed queries -> (choose query) -> Full JSON Profile?
Have you made any changes to Drill memory options such as
|planner.memory.max_query_memory_per_node|?
On 2021/09/28 13:55, nawfal osman wrote:
Hi
Please notice we have 10 nodes (mapr11-mapr15 ) ( mapr31-35)
please find below the output of the command : free -h of the node
mapr15 :
[mapr@mapr15 conf]$ free -h
total used free shared buff/cache
available
Mem: 125G 97G 3.1G 1.3G 25G
24G
Swap: 117G 54G
Find below the output of the command svc :
[mapr@mapr15 conf]$ maprcli node list -columns svc
2021-09-28 12:42:30,6130 INFO:Unable to set F_SETPIPE_SZ vmsplice
406 err = 1 10.148.0.73:7222
hostname service ip
mapr11
cldb,data-access-gateway,mastgateway,nodemanager,kibana,drill-bits,collectd,hoststats,fluentd,fileserver,nfs,gateway,apiserver
10.148.0.44,10.32.243.44
mapr12
data-access-gateway,fileserver,elasticsearch,mastgateway,nodemanager,kibana,drill-bits,collectd,hoststats,fluentd,grafana,resourcemanager,historyserver,nfs,opentsdb,gateway,apiserver
10.148.0.45,10.32.243.45
mapr13
httpfs,hs2,data-access-gateway,fileserver,mastgateway,nodemanager,kibana,hcat,drill-bits,hoststats,fluentd,hivemeta,grafana,hue,nfs,gateway,apiserver
10.148.0.46,10.32.243.46
mapr14
cldb,mastgateway,nodemanager,drill-bits,hbasethrift,oozie,collectd,hoststats,fluentd,hbaserest,fileserver,nfs,apiserver
10.148.0.47
mapr15
fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver
10.148.0.48,10.32.243.48
mapr31
fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver
10.148.0.70
mapr32
fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver
10.148.0.71
mapr33
fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver
10.148.0.72
mapr34
cldb,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,fileserver,nfs,apiserver
10.148.0.73
mapr35
data-access-gateway,fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver
For the memory setting of drill ; please find attached the conf of
all nodes
Kind Regards;
