Re: HDFS datanode used space is increasing without any writes
What is the size of your Hbase table? A copy of snapshot will be stored in archive directory. hadoop fs -du -s -h /apps/hbase/data/data/default/table-name hadoop fs -du -s -h /apps/hbase/data/archive/data/default/table-name Check this directory size. Thanks Sandeep Nemuri ᐧ On Thu, Jul 30, 2015 at 3:36 PM, Akmal Abbasov akmal.abba...@icloud.com wrote: I am running HBase snapshot exporting, but I stopped it, and still the capacity used is increasing. Here you can see that it is increased to 60 GB, and mostly it is because of 1 detanode. Moreover I am checking the directories sizes using bin/hdfs dfs -du -h / and while the HDFS used capacity is increasing, the directories sizes in HDFS is not changing. Any ideas? p.s. I started a hdfs balancer several days ago, then stopped it after running for about 10 minutes, after reading that it is not a good idea to run it on the same cluster with HBase. Could it be because of this? Thank you. On 28 Jul 2015, at 19:08, Harshit Mathur mathursh...@gmail.com wrote: Are there any map reduce jobs running? On Jul 28, 2015 10:11 PM, Akmal Abbasov akmal.abba...@icloud.com wrote: Hi, I’m observing strange behaviour in HDFS/HBase cluster. The disk space of one of datanodes is increasing very fast even when there are no write requests. It is 8GB per hour in average. Here is the graph which shows it. screenshot.png I am using hbase-0.98.7-hadoop2 and hadoop-2.5.1. And this is logs from the node 2015-07-28 15:40:38,795 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: / 10.32.1.12:50010, dest: /10.32.0.140:38699, bytes: 1071, op: HDFS_READ, cliID: DFSClient_NONMAPREDUCE_-689748537_1, offset: 0, srvID: 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: BP-439084760-10.32.0.180-1387281790961:blk_1074784244_1045663, duration: 17759797 2015-07-28 15:41:15,111 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-439084760-10.32.0.180-1387281790961:blk_1075311714_1574450 src: / 10.0.0.21:60540 dest: /10.32.1.12:50010 2015-07-28 15:41:15,304 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: / 10.0.0.21:59054, dest: /10.32.1.12:50010, bytes: 124121, op: HDFS_WRITE, cliID: DFSClient_hb_rs_hbase-rs4,60020,1438094355024_530940245_35, offset: 0, srvID: 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: BP-439084760-10.32.0.180-1387281790961:blk_1075311536_1574238, duration: 3600203675041 2015-07-28 15:41:15,304 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-439084760-10.32.0.180-1387281790961:blk_1075311536_1574238, type=HAS_DOWNSTREAM_IN_PIPELINE terminating 2015-07-28 15:50:40,745 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:6099ms (threshold=300ms) 2015-07-28 15:59:21,130 INFO org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool BP-439084760-10.32.0.180-1387281790961 Total blocks: 65856, missing metadata files:0, missing block files:0, missing blocks in memory:0, mismatched blocks:0 2015-07-28 16:00:16,770 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-439084760-10.32.0.180-1387281790961:blk_1075311715_1574451 src: / 10.32.1.12:36998 dest: /10.32.1.12:50010 2015-07-28 16:00:17,469 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: / 10.32.1.12:36150, dest: /10.32.1.12:50010, bytes: 32688, op: HDFS_WRITE, cliID: DFSClient_hb_rs_hbase-rs5,60020,1438088401479_1146354759_35, offset: 0, srvID: 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: BP-439084760-10.32.0.180-1387281790961:blk_1075311706_1574442, duration: 3601152263901 2015-07-28 16:00:17,472 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-439084760-10.32.0.180-1387281790961:blk_1075311706_1574442, type=HAS_DOWNSTREAM_IN_PIPELINE terminating 2015-07-28 16:03:44,011 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-439084760-10.32.0.180-1387281790961:blk_1075311716_1574452 src: / 10.0.0.19:35851 dest: /10.32.1.12:50010 2015-07-28 16:03:44,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: / 10.0.0.20:40176, dest: /10.32.1.12:50010, bytes: 316062, op: HDFS_WRITE, cliID: DFSClient_hb_rs_hbase-rs1,60020,1438092204868_-99326843_35, offset: 0, srvID: 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: BP-439084760-10.32.0.180-1387281790961:blk_1075311707_1574443, duration: 3600482062810 2015-07-28 16:03:44,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-439084760-10.32.0.180-1387281790961:blk_1075311707_1574443, type=LAST_IN_PIPELINE, downstreams=0:[] terminating 2015-07-28 16:11:10,961 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:7884ms (threshold=300ms) 2015-07-28 16:11:14,122 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:4362ms (threshold=300ms) 2015-07-28 16:11:14,123
Re: HDFS datanode used space is increasing without any writes
Please take a look at HDFS-6133 which aims to help with hbase data locality. It was integrated to hadoop 2.7.0 release. FYI On Thu, Jul 30, 2015 at 3:06 AM, Akmal Abbasov akmal.abba...@icloud.com wrote: I am running HBase snapshot exporting, but I stopped it, and still the capacity used is increasing. Here you can see that it is increased to 60 GB, and mostly it is because of 1 detanode. Moreover I am checking the directories sizes using bin/hdfs dfs -du -h / and while the HDFS used capacity is increasing, the directories sizes in HDFS is not changing. Any ideas? p.s. I started a hdfs balancer several days ago, then stopped it after running for about 10 minutes, after reading that it is not a good idea to run it on the same cluster with HBase. Could it be because of this? Thank you. On 28 Jul 2015, at 19:08, Harshit Mathur mathursh...@gmail.com wrote: Are there any map reduce jobs running? On Jul 28, 2015 10:11 PM, Akmal Abbasov akmal.abba...@icloud.com wrote: Hi, I’m observing strange behaviour in HDFS/HBase cluster. The disk space of one of datanodes is increasing very fast even when there are no write requests. It is 8GB per hour in average. Here is the graph which shows it. screenshot.png I am using hbase-0.98.7-hadoop2 and hadoop-2.5.1. And this is logs from the node 2015-07-28 15:40:38,795 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: / 10.32.1.12:50010, dest: /10.32.0.140:38699, bytes: 1071, op: HDFS_READ, cliID: DFSClient_NONMAPREDUCE_-689748537_1, offset: 0, srvID: 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: BP-439084760-10.32.0.180-1387281790961:blk_1074784244_1045663, duration: 17759797 2015-07-28 15:41:15,111 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-439084760-10.32.0.180-1387281790961:blk_1075311714_1574450 src: / 10.0.0.21:60540 dest: /10.32.1.12:50010 2015-07-28 15:41:15,304 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: / 10.0.0.21:59054, dest: /10.32.1.12:50010, bytes: 124121, op: HDFS_WRITE, cliID: DFSClient_hb_rs_hbase-rs4,60020,1438094355024_530940245_35, offset: 0, srvID: 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: BP-439084760-10.32.0.180-1387281790961:blk_1075311536_1574238, duration: 3600203675041 2015-07-28 15:41:15,304 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-439084760-10.32.0.180-1387281790961:blk_1075311536_1574238, type=HAS_DOWNSTREAM_IN_PIPELINE terminating 2015-07-28 15:50:40,745 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:6099ms (threshold=300ms) 2015-07-28 15:59:21,130 INFO org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool BP-439084760-10.32.0.180-1387281790961 Total blocks: 65856, missing metadata files:0, missing block files:0, missing blocks in memory:0, mismatched blocks:0 2015-07-28 16:00:16,770 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-439084760-10.32.0.180-1387281790961:blk_1075311715_1574451 src: / 10.32.1.12:36998 dest: /10.32.1.12:50010 2015-07-28 16:00:17,469 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: / 10.32.1.12:36150, dest: /10.32.1.12:50010, bytes: 32688, op: HDFS_WRITE, cliID: DFSClient_hb_rs_hbase-rs5,60020,1438088401479_1146354759_35, offset: 0, srvID: 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: BP-439084760-10.32.0.180-1387281790961:blk_1075311706_1574442, duration: 3601152263901 2015-07-28 16:00:17,472 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-439084760-10.32.0.180-1387281790961:blk_1075311706_1574442, type=HAS_DOWNSTREAM_IN_PIPELINE terminating 2015-07-28 16:03:44,011 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-439084760-10.32.0.180-1387281790961:blk_1075311716_1574452 src: / 10.0.0.19:35851 dest: /10.32.1.12:50010 2015-07-28 16:03:44,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: / 10.0.0.20:40176, dest: /10.32.1.12:50010, bytes: 316062, op: HDFS_WRITE, cliID: DFSClient_hb_rs_hbase-rs1,60020,1438092204868_-99326843_35, offset: 0, srvID: 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: BP-439084760-10.32.0.180-1387281790961:blk_1075311707_1574443, duration: 3600482062810 2015-07-28 16:03:44,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-439084760-10.32.0.180-1387281790961:blk_1075311707_1574443, type=LAST_IN_PIPELINE, downstreams=0:[] terminating 2015-07-28 16:11:10,961 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:7884ms (threshold=300ms) 2015-07-28 16:11:14,122 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:4362ms (threshold=300ms) 2015-07-28 16:11:14,123 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow flushOrSync took 3160ms (threshold=300ms), isSync:false, flushTotalNanos=3160364203ns
Re: HDFS datanode used space is increasing without any writes
From log below, hbase-rs4 was writing to the datanode. Can you take a look at region server log and see if there is some clue ? Thanks On Jul 28, 2015, at 9:41 AM, Akmal Abbasov akmal.abba...@icloud.com wrote: Hi, I’m observing strange behaviour in HDFS/HBase cluster. The disk space of one of datanodes is increasing very fast even when there are no write requests. It is 8GB per hour in average. Here is the graph which shows it. screenshot.png I am using hbase-0.98.7-hadoop2 and hadoop-2.5.1. And this is logs from the node 2015-07-28 15:40:38,795 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.32.1.12:50010, dest: /10.32.0.140:38699, bytes: 1071, op: HDFS_READ, cliID: DFSClient_NONMAPREDUCE_-689748537_1, offset: 0, srvID: 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: BP-439084760-10.32.0.180-1387281790961:blk_1074784244_1045663, duration: 17759797 2015-07-28 15:41:15,111 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-439084760-10.32.0.180-1387281790961:blk_1075311714_1574450 src: /10.0.0.21:60540 dest: /10.32.1.12:50010 2015-07-28 15:41:15,304 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.0.0.21:59054, dest: /10.32.1.12:50010, bytes: 124121, op: HDFS_WRITE, cliID: DFSClient_hb_rs_hbase-rs4,60020,1438094355024_530940245_35, offset: 0, srvID: 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: BP-439084760-10.32.0.180-1387281790961:blk_1075311536_1574238, duration: 3600203675041 2015-07-28 15:41:15,304 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-439084760-10.32.0.180-1387281790961:blk_1075311536_1574238, type=HAS_DOWNSTREAM_IN_PIPELINE terminating 2015-07-28 15:50:40,745 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:6099ms (threshold=300ms) 2015-07-28 15:59:21,130 INFO org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool BP-439084760-10.32.0.180-1387281790961 Total blocks: 65856, missing metadata files:0, missing block files:0, missing blocks in memory:0, mismatched blocks:0 2015-07-28 16:00:16,770 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-439084760-10.32.0.180-1387281790961:blk_1075311715_1574451 src: /10.32.1.12:36998 dest: /10.32.1.12:50010 2015-07-28 16:00:17,469 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.32.1.12:36150, dest: /10.32.1.12:50010, bytes: 32688, op: HDFS_WRITE, cliID: DFSClient_hb_rs_hbase-rs5,60020,1438088401479_1146354759_35, offset: 0, srvID: 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: BP-439084760-10.32.0.180-1387281790961:blk_1075311706_1574442, duration: 3601152263901 2015-07-28 16:00:17,472 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-439084760-10.32.0.180-1387281790961:blk_1075311706_1574442, type=HAS_DOWNSTREAM_IN_PIPELINE terminating 2015-07-28 16:03:44,011 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-439084760-10.32.0.180-1387281790961:blk_1075311716_1574452 src: /10.0.0.19:35851 dest: /10.32.1.12:50010 2015-07-28 16:03:44,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.0.0.20:40176, dest: /10.32.1.12:50010, bytes: 316062, op: HDFS_WRITE, cliID: DFSClient_hb_rs_hbase-rs1,60020,1438092204868_-99326843_35, offset: 0, srvID: 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: BP-439084760-10.32.0.180-1387281790961:blk_1075311707_1574443, duration: 3600482062810 2015-07-28 16:03:44,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-439084760-10.32.0.180-1387281790961:blk_1075311707_1574443, type=LAST_IN_PIPELINE, downstreams=0:[] terminating 2015-07-28 16:11:10,961 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:7884ms (threshold=300ms) 2015-07-28 16:11:14,122 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:4362ms (threshold=300ms) 2015-07-28 16:11:14,123 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow flushOrSync took 3160ms (threshold=300ms), isSync:false, flushTotalNanos=3160364203ns 2015-07-28 16:13:29,968 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:659ms (threshold=300ms) 2015-07-28 16:18:33,336 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-439084760-10.32.0.180-1387281790961:blk_1075311717_1574453 src: /10.0.0.20:41527 dest: /10.32.1.12:50010 2015-07-28 16:18:38,926 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:1767ms (threshold=300ms) 2015-07-28 16:28:40,580 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:4099ms (threshold=300ms) What could be the cause of this? Thank you.
Re: HDFS datanode used space is increasing without any writes
Are there any map reduce jobs running? On Jul 28, 2015 10:11 PM, Akmal Abbasov akmal.abba...@icloud.com wrote: Hi, I’m observing strange behaviour in HDFS/HBase cluster. The disk space of one of datanodes is increasing very fast even when there are no write requests. It is 8GB per hour in average. Here is the graph which shows it. I am using hbase-0.98.7-hadoop2 and hadoop-2.5.1. And this is logs from the node 2015-07-28 15:40:38,795 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: / 10.32.1.12:50010, dest: /10.32.0.140:38699, bytes: 1071, op: HDFS_READ, cliID: DFSClient_NONMAPREDUCE_-689748537_1, offset: 0, srvID: 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: BP-439084760-10.32.0.180-1387281790961:blk_1074784244_1045663, duration: 17759797 2015-07-28 15:41:15,111 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-439084760-10.32.0.180-1387281790961:blk_1075311714_1574450 src: / 10.0.0.21:60540 dest: /10.32.1.12:50010 2015-07-28 15:41:15,304 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: / 10.0.0.21:59054, dest: /10.32.1.12:50010, bytes: 124121, op: HDFS_WRITE, cliID: DFSClient_hb_rs_hbase-rs4,60020,1438094355024_530940245_35, offset: 0, srvID: 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: BP-439084760-10.32.0.180-1387281790961:blk_1075311536_1574238, duration: 3600203675041 2015-07-28 15:41:15,304 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-439084760-10.32.0.180-1387281790961:blk_1075311536_1574238, type=HAS_DOWNSTREAM_IN_PIPELINE terminating 2015-07-28 15:50:40,745 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:6099ms (threshold=300ms) 2015-07-28 15:59:21,130 INFO org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool BP-439084760-10.32.0.180-1387281790961 Total blocks: 65856, missing metadata files:0, missing block files:0, missing blocks in memory:0, mismatched blocks:0 2015-07-28 16:00:16,770 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-439084760-10.32.0.180-1387281790961:blk_1075311715_1574451 src: / 10.32.1.12:36998 dest: /10.32.1.12:50010 2015-07-28 16:00:17,469 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: / 10.32.1.12:36150, dest: /10.32.1.12:50010, bytes: 32688, op: HDFS_WRITE, cliID: DFSClient_hb_rs_hbase-rs5,60020,1438088401479_1146354759_35, offset: 0, srvID: 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: BP-439084760-10.32.0.180-1387281790961:blk_1075311706_1574442, duration: 3601152263901 2015-07-28 16:00:17,472 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-439084760-10.32.0.180-1387281790961:blk_1075311706_1574442, type=HAS_DOWNSTREAM_IN_PIPELINE terminating 2015-07-28 16:03:44,011 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-439084760-10.32.0.180-1387281790961:blk_1075311716_1574452 src: / 10.0.0.19:35851 dest: /10.32.1.12:50010 2015-07-28 16:03:44,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: / 10.0.0.20:40176, dest: /10.32.1.12:50010, bytes: 316062, op: HDFS_WRITE, cliID: DFSClient_hb_rs_hbase-rs1,60020,1438092204868_-99326843_35, offset: 0, srvID: 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: BP-439084760-10.32.0.180-1387281790961:blk_1075311707_1574443, duration: 3600482062810 2015-07-28 16:03:44,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-439084760-10.32.0.180-1387281790961:blk_1075311707_1574443, type=LAST_IN_PIPELINE, downstreams=0:[] terminating 2015-07-28 16:11:10,961 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:7884ms (threshold=300ms) 2015-07-28 16:11:14,122 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:4362ms (threshold=300ms) 2015-07-28 16:11:14,123 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow flushOrSync took 3160ms (threshold=300ms), isSync:false, flushTotalNanos=3160364203ns 2015-07-28 16:13:29,968 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:659ms (threshold=300ms) 2015-07-28 16:18:33,336 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-439084760-10.32.0.180-1387281790961:blk_1075311717_1574453 src: / 10.0.0.20:41527 dest: /10.32.1.12:50010 2015-07-28 16:18:38,926 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:1767ms (threshold=300ms) 2015-07-28 16:28:40,580 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:4099ms (threshold=300ms) What could be the cause of this? Thank you.
