You don't mention the version, but here are some general suggestions
- 2 GB heap is very small for a node, especially with 1 TB+ of data. What is the physical RAM on the host? In general, you want ½ of physical RAM for the JVM. (Look in jvm.options or cassandra-env.sh) - You can change the streaming throughput from the existing nodes, if it looks like the new node can handle it. Look at nodetool setstreamthroughput. Default is 200 (MB/sec). - You might want to check for a streaming_socket_timeout_in_ms. This has changed over the versions. Some details are at: https://issues.apache.org/jira/browse/CASSANDRA-11839. 24 hours is good recommendation. - If your new node can't compact fast enough to keep disk usage down, look at compactionthroughput on that node - nodetool netstats | grep -v "100%" is a good way to see what is happening/if anything is stuck. Newer versions give a bit more info on progress. - Don't forget to run cleanup on existing nodes after the new nodes are added. Sean Durity From: qf zhou [mailto:zhouqf2...@gmail.com] Sent: Tuesday, January 02, 2018 10:30 PM To: user@cassandra.apache.org Subject: [EXTERNAL] Cassandra cluster add new node slowly The cluster has 3 nodes, and the data in each node is about 1.2 T. I want to add two new nodes to expand the cluster. Following the instructions from the datastax website, ie, (http://docs.datastax.com/en/archived/cassandra/3.x/cassandra/operations/opsAddNodeToCluster.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.datastax.com_en_archived_cassandra_3.x_cassandra_operations_opsAddNodeToCluster.html&d=DwMFoQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=AfP0BxfTG1o_-2nRtGSjngs45YndH060h39eVlypVso&s=AaiOypurMtWA470038fHMbaXJPIZeYtLzv3Yt9vf1Sc&e=>), I try to add one node to the cluster. However, it is too slow and time cost too much. After about 24 hours, it still didn't success. I run the command: nodetool netstats on the new node, it shows that: (tb1fullwithstate2 is a big table and 90% of the cluster data is in it. Here I use CompactionStrategy: TimeWindowCompactionStrategy). /*.*.*.3 Receiving 136 files, 328573794609 bytes total. Already received 8 files, 5774621188 bytes total tb1/tb1fullwithneweststatetest 3758271/3758271 bytes(100%) received from idx:0/*.*.*.3 system_distributed/repair_history 57534/57534 bytes(100%) received from idx:0/*.*.*.3 system_distributed/parent_repair_history 507660/507660 bytes(100%) received from idx:0/*.*.*.3 tb1/tb1_device_last_state_eachday 15754096/15754096 bytes(100%) received from idx:0/*.*.*.3 mytest1/tb1_test1 8143775/8143775 bytes(100%) received from idx:0/*.*.*.3 tb1/tb1fullwithstate 2251191007/2251191007 bytes(100%) received from idx:0/*.*.*.3 applocationinfo/weiyirong_app 2760/2760 bytes(100%) received from idx:0/*.*.*.3 tb1/tb1fullwithstate2 3490748006/4909554503 bytes(71%) received from idx:0/*.*.*.3 tb1/tb1fullwithneweststate 4458079/4458079 bytes(100%) received from idx:0/*.*.*.3 /*.*.*.2 Receiving 136 files, 336762487360 bytes total. Already received 3 files, 5695770181 bytes total system_distributed/repair_history 31684/31684 bytes(100%) received from idx:0/*.*.*.2 tb1/tb1fullwithstate 908260516/908260516 bytes(100%) received from idx:0/*.*.*.2 tb1/tb1fullwithstate2 4783622958/4990450588 bytes(95%) received from idx:0/*.*.*.2 tb1/tb1fullwithneweststate 3855023/3855023 bytes(100%) received from idx:0/*.*.*.2 /*.*.*.4 Receiving 132 files, 236250553620 bytes total. Already received 10 files, 3117465128 bytes total mytest1/wordstest2 46/46 bytes(100%) received from idx:0/*.*.*.4 tb1/tb1fullwithneweststatetest 3416891/3416891 bytes(100%) received from idx:0/*.*.*.4 system_distributed/repair_history 39720/39720 bytes(100%) received from idx:0/*.*.*.4 system_distributed/parent_repair_history 452250/452250 bytes(100%) received from idx:0/*.*.*.4 mytest1/weblogs 104/104 bytes(100%) received from idx:0/*.*.*.4 tb1/tb1_device_last_state_eachday 12670998/12670998 bytes(100%) received from idx:0/*.*.*.4 mytest1/tb1_test1 3257952/3257952 bytes(100%) received from idx:0/*.*.*.4 tb1/tb1fullwithstate 647702056/647702056 bytes(100%) received from idx:0/*.*.*.4 applocationinfo/weiyirong_app 3509/3509 bytes(100%) received from idx:0/*.*.*.4 tb1/tb1fullwithstate2 2446436305/3566562762 bytes(68%) received from idx:0/*.*.*.4 tb1/tb1fullwithneweststate 3485297/3485297 bytes(100%) received from idx:0/*.*.*.4 check the log in the logs/system.log, it shows that: INFO 06:09:33 Updating topology for /*.*.*.2 INFO 06:09:33 Updating topology for /*.*.*.2 INFO 06:09:33 Node /*.*.*.3 has restarted, now UP INFO 06:09:33 InetAddress /*.*.*.3 is now UP INFO 06:09:33 Node /*.*.*.3 state jump to NORMAL INFO 06:09:33 Updating topology for /*.*.*.3 INFO 06:09:33 Updating topology for /*.*.*.3 INFO 06:09:33 Handshaking version with /*.*.*.2 INFO 06:09:33 Handshaking version with /*.*.*.3 INFO 06:09:36 Handshaking version with /*.*.*.4 INFO 06:10:03 JOINING: Starting to bootstrap... INFO 06:10:04 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75] Executing streaming plan for Bootstrap INFO 06:10:04 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75] Starting streaming to /*.*.*.4 INFO 06:10:04 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75, ID#0] Beginning stream session with /*.*.*.4 INFO 06:10:04 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527790874-1.hints INFO 06:10:04 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527790874-1.hints to endpoint c5271e74-19a1-4cee-98d7-dc169cf87 INFO 06:10:11 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527800873-1.hints INFO 06:10:11 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527800873-1.hints to endpoint c5271e74-19a1-4cee-98d7-dc169cf87 INFO 06:10:12 Deleted hint file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527790881-1.hints INFO 06:10:12 Finished hinted handoff of file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527790881-1.hints to endpoint c555e44c-9590-4f45-aea4-f5eca6818 INFO 06:10:20 Deleted hint file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527800880-1.hints INFO 06:10:20 Finished hinted handoff of file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527800880-1.hints to endpoint c555e44c-9590-4f45-aea4-f5eca6818 INFO 06:10:21 Deleted hint file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527790884-1.hints INFO 06:10:21 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527790884-1.hints to endpoint c623bbc0-9839-4d2d-8ff3-db7115719 INFO 06:10:28 Deleted hint file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527800887-1.hints INFO 06:10:28 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527800887-1.hints to endpoint c623bbc0-9839-4d2d-8ff3-db7115719 INFO 06:10:33 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75 ID#0] Prepare completed. Receiving 132 files(220.025GiB), sending 0 files(0.000KiB) INFO 06:10:33 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75] Starting streaming to /*.*.*.2 INFO 06:10:33 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75, ID#0] Beginning stream session with /*.*.*.2 INFO 06:10:34 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527810873-1.hints INFO 06:10:34 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527810873-1.hints to endpoint c5271e74-19a1-4cee-98d7-dc169cf87 INFO 06:10:42 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527820873-1.hints INFO 06:10:42 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527820873-1.hints to endpoint c5271e74-19a1-4cee-98d7-dc169cf87 INFO 06:10:58 Deleted hint file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527810881-1.hints INFO 06:10:58 Finished hinted handoff of file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527810881-1.hints to endpoint c555e44c-9590-4f45-aea4-f5eca6818 INFO 06:11:03 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75 ID#0] Prepare completed. Receiving 136 files(313.635GiB), sending 0 files(0.000KiB) INFO 06:11:03 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75] Starting streaming to /*.*.*.3 INFO 06:11:03 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75, ID#0] Beginning stream session with /*.*.*.3 INFO 06:11:06 Deleted hint file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527830880-1.hints INFO 06:11:06 Finished hinted handoff of file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527830880-1.hints to endpoint c555e44c-9590-4f45-aea4-f5eca6818 INFO 06:11:29 Deleted hint file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527810889-1.hints INFO 06:11:29 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527810889-1.hints to endpoint c623bbc0-9839-4d2d-8ff3-db7115719 INFO 06:11:33 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75 ID#0] Prepare completed. Receiving 136 files(306.008GiB), sending 0 files(0.000KiB) INFO 06:11:36 Deleted hint file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527840887-1.hints INFO 06:11:36 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527840887-1.hints to endpoint c623bbc0-9839-4d2d-8ff3-db7115719 INFO 06:11:49 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527830873-1.hints INFO 06:11:49 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527830873-1.hints to endpoint c5271e74-19a1-4cee-98d7-dc169cf87 INFO 06:11:56 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527850874-1.hints INFO 06:11:56 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527850874-1.hints to endpoint c5271e74-19a1-4cee-98d7-dc169cf87 INFO 06:12:26 Deleted hint file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527840880-1.hints INFO 06:12:26 Finished hinted handoff of file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527840880-1.hints to endpoint c555e44c-9590-4f45-aea4-f5eca6818 INFO 06:12:45 Deleted hint file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527850883-1.hints INFO 06:12:45 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527850883-1.hints to endpoint c623bbc0-9839-4d2d-8ff3-db7115719 INFO 06:12:56 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527860874-1.hints INFO 06:12:56 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527860874-1.hints to endpoint c5271e74-19a1-4cee-98d7-dc169cf87 INFO 06:24:03 G1 Young Generation GC in 242ms. G1 Eden Space: 18924699648 -> 0; G1 Old Gen: 4360876536 -> 5095617016; G1 Survivor Space: 1677721600 - INFO 06:26:06 G1 Young Generation GC in 215ms. G1 Eden Space: 18220056576 -> 0; G1 Old Gen: 5981077496 -> 6597610488; G1 Survivor Space: 1493172224 - INFO 06:29:04 G1 Young Generation GC in 225ms. G1 Eden Space: 18538823680 -> 0; G1 Old Gen: 7910457840 -> 8860639728; G1 Survivor Space: 1895825408 - INFO 06:33:22 G1 Young Generation GC in 214ms. G1 Eden Space: 16559112192 -> 0; G1 Old Gen: 11868321784 -> 11941600752; G1 Survivor Space: 1241513984 INFO 06:33:33 G1 Young Generation GC in 205ms. G1 Eden Space: 8069840896 -> 0; G1 Old Gen: 11941600752 -> 13683606520; G1 Survivor Space: 1895825408 INFO 06:41:01 G1 Young Generation GC in 383ms. G1 Eden Space: 18874368000 -> 0; G1 Old Gen: 4770399736 -> 5551887344; G1 Survivor Space: 1728053248 - INFO 06:45:07 G1 Young Generation GC in 226ms. G1 Eden Space: 19025362944 -> 0; G1 Old Gen: 8036286448 -> 8739698672; G1 Survivor Space: 1577058304 - INFO 06:47:11 G1 Young Generation GC in 203ms. G1 Eden Space: 18538823680 -> 0; G1 Survivor Space: 1191182336 -> 1778384896; INFO 06:47:22 G1 Young Generation GC in 241ms. G1 Eden Space: 8086618112 -> 0; G1 Old Gen: 9999220720 -> 11591876088; G1 Survivor Space: 1778384896 - INFO 06:49:43 G1 Young Generation GC in 201ms. G1 Eden Space: 16022241280 -> 0; G1 Old Gen: 11591876088 -> 12084766712; G1 Survivor Space: 1644167168 INFO 06:53:21 G1 Young Generation GC in 322ms. G1 Eden Space: 13237223424 -> 0; G1 Old Gen: 14755561464 -> 15445886448; G1 Survivor Space: 1459617792 INFO 06:55:34 G1 Young Generation GC in 343ms. G1 Eden Space: 16525557760 -> 0; G1 Old Gen: 2939907056 -> 3636770792; G1 Survivor Space: 1610612736 - INFO 07:09:30 Redistributing index summaries INFO 07:14:25 G1 Young Generation GC in 270ms. G1 Eden Space: 15602810880 -> 0; G1 Old Gen: 4360518136 -> 5243274224; G1 Survivor Space: 1610612736 - INFO 07:16:51 G1 Young Generation GC in 229ms. G1 Eden Space: 19008585728 -> 0; G1 Old Gen: 6132072440 -> 6815145976; G1 Survivor Space: 1593835520 - INFO 07:19:34 G1 Young Generation GC in 215ms. G1 Eden Space: 18991808512 -> 0; G1 Old Gen: 8689337336 -> 9071161328; G1 Survivor Space: 1610612736 - INFO 07:23:35 G1 Young Generation GC in 293ms. G1 Eden Space: 15753805824 -> 0; G1 Old Gen: 11777605616 -> 12289649640; G1 Survivor Space: 1694498816 INFO 07:29:42 G1 Young Generation GC in 362ms. G1 Eden Space: 15955132416 -> 0; G1 Old Gen: 3742637560 -> 4410361824; G1 Survivor Space: 1644167168 - INFO 07:35:12 G1 Young Generation GC in 208ms. G1 Eden Space: 19478347776 -> 0; G1 Old Gen: 7199737832 -> 7208126432; G1 Survivor Space: 1124073472 - INFO 07:35:33 G1 Young Generation GC in 219ms. G1 Eden Space: 18740150272 -> 0; G1 Old Gen: 7208126432 -> 8072478184; G1 Survivor Space: 1862270976 - INFO 07:45:19 G1 Young Generation GC in 213ms. G1 Eden Space: 13153337344 -> 0; G1 Old Gen: 15418261472 -> 16123805152; G1 Survivor Space: 1174405120 INFO 07:48:08 G1 Young Generation GC in 335ms. G1 Eden Space: 18387828736 -> 0; G1 Old Gen: 3852453368 -> 4464797168; G1 Survivor Space: 1644167168 - INFO 07:48:08 G1 Young Generation GC in 209ms. G1 Eden Space: 352321536 -> 0; G1 Old Gen: 4464797168 -> 5989466096; G1 Survivor Space: 1476395008 -> INFO 07:50:25 G1 Young Generation GC in 221ms. G1 Eden Space: 17498636288 -> 0; G1 Old Gen: 5981077496 -> 6488668656; G1 Survivor Space: 1593835520 - INFO 08:03:31 G1 Young Generation GC in 748ms. G1 Eden Space: 8673820672 -> 0; G1 Old Gen: 4386367472 -> 5874747352; G1 Survivor Space: 1627389952 -> INFO 08:09:30 Redistributing index summaries INFO 08:10:05 G1 Young Generation GC in 213ms. G1 Eden Space: 18220056576 -> 0; G1 Old Gen: 9276024272 -> 9969348048; G1 Survivor Space: 1711276032 - INFO 08:18:59 G1 Young Generation GC in 228ms. G1 Eden Space: 17347641344 -> 0; G1 Old Gen: 4546626040 -> 5180788216; G1 Survivor Space: 1593835520 - INFO 08:33:12 G1 Young Generation GC in 207ms. G1 Eden Space: 19411238912 -> 0; G1 Survivor Space: 1191182336 -> 1828716544; INFO 08:33:13 G1 Young Generation GC in 273ms. G1 Eden Space: 452984832 -> 0; G1 Old Gen: 4668340728 -> 6320276984; G1 Survivor Space: 1828716544 -> INFO 08:41:59 G1 Young Generation GC in 202ms. G1 Eden Space: 19025362944 -> 0; G1 Old Gen: 8701765104 -> 8927144944; G1 Survivor Space: 1577058304 - INFO 08:56:31 G1 Young Generation GC in 205ms. G1 Eden Space: 11072962560 -> 0; G1 Old Gen: 9563013104 -> 11068430328; G1 Survivor Space: 1660944384 INFO 09:05:49 G1 Young Generation GC in 367ms. G1 Eden Space: 18522046464 -> 0; G1 Old Gen: 4009754608 -> 4732347872; G1 Survivor Space: 1778384896 - INFO 09:09:30 Redistributing index summaries INFO 09:14:52 G1 Young Generation GC in 213ms. G1 Eden Space: 15267266560 -> 0; G1 Old Gen: 12155092968 -> 12906235360; G1 Survivor Space: 1744830464 INFO 09:17:23 G1 Young Generation GC in 203ms. G1 Eden Space: 14159970304 -> 0; G1 Old Gen: 14779568608 -> 14771180008; G1 Survivor Space: 989855744 INFO 09:17:45 G1 Young Generation GC in 204ms. G1 Eden Space: 12616466432 -> 0; G1 Old Gen: 14771180008 -> 15500684768; G1 Survivor Space: 1761607680 INFO 09:20:52 G1 Young Generation GC in 361ms. G1 Eden Space: 19025362944 -> 0; G1 Old Gen: 4351990768 -> 5086824424; G1 Survivor Space: 1577058304 - INFO 09:23:51 G1 Young Generation GC in 207ms. G1 Eden Space: 18891145216 -> 0; G1 Old Gen: 7001879024 -> 7716560880; G1 Survivor Space: 1711276032 - INFO 09:29:20 G1 Young Generation GC in 202ms. G1 Eden Space: 17028874240 -> 0; G1 Survivor Space: 1040187392 -> 1828716544; INFO 09:29:40 G1 Young Generation GC in 219ms. G1 Eden Space: 15451815936 -> 0; G1 Old Gen: 11802771944 -> 12574988776; G1 Survivor Space: 1828716544 INFO 09:44:51 G1 Young Generation GC in 202ms. G1 Eden Space: 18001952768 -> 0; G1 Old Gen: 9325481456 -> 10085596664; G1 Survivor Space: 1795162112 INFO 09:45:23 G1 Young Generation GC in 209ms. G1 Eden Space: 9177137152 -> 0; G1 Old Gen: 10093985264 -> 12309118960; G1 Survivor Space: 1677721600 INFO 09:50:33 G1 Young Generation GC in 201ms. G1 Eden Space: 12901679104 -> 0; G1 Old Gen: 14487498224 -> 15268560880; G1 Survivor Space: 1761607680 INFO 10:08:09 Saved KeyCache (2989 items) in 92 ms INFO 10:09:30 Redistributing index summaries INFO 10:22:17 G1 Young Generation GC in 216ms. G1 Eden Space: 19495124992 -> 0; G1 Old Gen: 7742571504 -> 7734182904; G1 Survivor Space: 1107296256 - INFO 10:25:31 G1 Young Generation GC in 206ms. G1 Eden Space: 18622709760 -> 0; G1 Survivor Space: 1040187392 -> 1711276032; INFO 10:34:54 G1 Young Generation GC in 251ms. G1 Eden Space: 18991808512 -> 0; G1 Old Gen: 5066719736 -> 5691775992; G1 Survivor Space: 1610612736 - INFO 10:40:08 G1 Young Generation GC in 209ms. G1 Eden Space: 19025362944 -> 0; G1 Old Gen: 9000976880 -> 9090653168; G1 Survivor Space: 1442840576 - INFO 10:40:08 G1 Young Generation GC in 223ms. G1 Eden Space: 469762048 -> 0; G1 Old Gen: 9090653168 -> 11140124152; G1 Survivor Space: 1862270976 -> INFO 10:52:15 G1 Young Generation GC in 219ms. G1 Eden Space: 15820914688 -> 0; G1 Old Gen: 5331537912 -> 5828939768; G1 Survivor Space: 1275068416 - INFO 10:55:30 G1 Young Generation GC in 207ms. G1 Eden Space: 15804137472 -> 0; G1 Old Gen: 7501364216 -> 8001282032; G1 Survivor Space: 1459617792 - INFO 11:07:57 G1 Young Generation GC in 295ms. G1 Eden Space: 19159580672 -> 0; G1 Old Gen: 4301379064 -> 4919709168; G1 Survivor Space: 1442840576 - INFO 11:09:30 Redistributing index summaries INFO 11:13:06 G1 Young Generation GC in 204ms. G1 Eden Space: 19126026240 -> 0; G1 Old Gen: 8044675064 -> 8659737080; G1 Survivor Space: 1476395008 - INFO 11:27:07 G1 Young Generation GC in 202ms. G1 Eden Space: 19159580672 -> 0; G1 Old Gen: 4328521712 -> 4905718776; G1 Survivor Space: 1442840576 - INFO 11:30:08 G1 Young Generation GC in 209ms. G1 Eden Space: 19243466752 -> 0; G1 Survivor Space: 1358954496 -> 1828716544; INFO 11:30:25 G1 Young Generation GC in 226ms. G1 Eden Space: 14277410816 -> 0; G1 Old Gen: 6349296624 -> 7552639472; G1 Survivor Space: 1828716544 - INFO 11:32:30 G1 Young Generation GC in 215ms. G1 Eden Space: 19142803456 -> 0; G1 Survivor Space: 1459617792 -> 2046820352; INFO 11:33:01 G1 Young Generation GC in 247ms. G1 Eden Space: 17263755264 -> 0; G1 Old Gen: 7544250872 -> 8698295280; G1 Survivor Space: 2046820352 - INFO 11:46:29 G1 Young Generation GC in 355ms. G1 Eden Space: 17230200832 -> 0; G1 Old Gen: 4769358328 -> 5362416624; G1 Survivor Space: 1761607680 - INFO 11:49:18 G1 Young Generation GC in 233ms. G1 Eden Space: 17867735040 -> 0; G1 Old Gen: 6752829432 -> 6944805360; G1 Survivor Space: 1509949440 - INFO 11:52:24 G1 Young Generation GC in 209ms. G1 Eden Space: 19193135104 -> 0; G1 Survivor Space: 1409286144 -> 1946157056; INFO 11:52:47 G1 Young Generation GC in 338ms. G1 Eden Space: 18287165440 -> 0; G1 Old Gen: 8740881904 -> 9340426216; G1 Survivor Space: 1946157056 - INFO 12:04:08 G1 Young Generation GC in 810ms. G1 Eden Space: 18119393280 -> 0; G1 Old Gen: 3471267832 -> 5003518440; G1 Survivor Space: 1325400064 - INFO 12:06:28 G1 Young Generation GC in 345ms. G1 Eden Space: 8673820672 -> 0; G1 Old Gen: 5318377952 -> 6577762784; G1 Survivor Space: 1409286144 -> INFO 12:09:31 G1 Young Generation GC in 204ms. G1 Eden Space: 19159580672 -> 0; G1 Old Gen: 6577762784 -> 6586151384; G1 Survivor Space: 1442840576 - INFO 12:09:31 Redistributing index summaries INFO 12:10:01 G1 Young Generation GC in 298ms. G1 Eden Space: 17230200832 -> 0; G1 Old Gen: 6586151384 -> 7710758368; G1 Survivor Space: 1946157056 - INFO 12:12:37 G1 Young Generation GC in 214ms. G1 Eden Space: 18773704704 -> 0; G1 Old Gen: 8380219864 -> 8919123416; G1 Survivor Space: 1728053248 - INFO 12:24:54 G1 Young Generation GC in 299ms. G1 Eden Space: 16055795712 -> 0; G1 Old Gen: 4375908856 -> 5009812976; G1 Survivor Space: 1392508928 - INFO 12:28:34 G1 Young Generation GC in 485ms. G1 Eden Space: 13304332288 -> 0; G1 Old Gen: 6481607672 -> 7506585072; G1 Survivor Space: 1979711488 - INFO 12:44:52 G1 Young Generation GC in 261ms. G1 Eden Space: 18773704704 -> 0; G1 Old Gen: 4239590896 -> 4692442608; G1 Survivor Space: 1828716544 - INFO 12:48:37 G1 Young Generation GC in 251ms. G1 Eden Space: 13656653824 -> 0; G1 Old Gen: 6719275504 -> 7776254960; G1 Survivor Space: 1862270976 - INFO 12:53:47 G1 Young Generation GC in 217ms. G1 Eden Space: 17548967936 -> 0; G1 Old Gen: 9667142640 -> 10202159608; G1 Survivor Space: 1845493760 INFO 13:05:40 G1 Young Generation GC in 287ms. G1 Eden Space: 18320719872 -> 0; G1 Old Gen: 5298861040 -> 5826686456; G1 Survivor Space: 1862270976 - INFO 13:09:31 Redistributing index summaries INFO 13:11:44 G1 Young Generation GC in 268ms. G1 Eden Space: 10871635968 -> 0; G1 Old Gen: 8799649784 -> 9797878264; G1 Survivor Space: 1811939328 - INFO 13:14:22 G1 Young Generation GC in 202ms. G1 Eden Space: 17750294528 -> 0; G1 Survivor Space: 1325400064 -> 1795162112; INFO 13:14:40 G1 Young Generation GC in 254ms. G1 Eden Space: 14092861440 -> 0; G1 Old Gen: 10509551608 -> 11563311096; G1 Survivor Space: 1795162112 INFO 13:16:33 G1 Young Generation GC in 213ms. G1 Eden Space: 16039018496 -> 0; G1 Old Gen: 11619767792 -> 12115354104; G1 Survivor Space: 1627389952 INFO 13:25:40 G1 Young Generation GC in 245ms. G1 Eden Space: 18790481920 -> 0; G1 Old Gen: 4253024752 -> 4761733616; G1 Survivor Space: 1811939328 - INFO 13:27:38 G1 Young Generation GC in 208ms. G1 Eden Space: 13958643712 -> 0; G1 Old Gen: 5377097720 -> 6427987960; G1 Survivor Space: 1912602624 - INFO 13:29:30 G1 Young Generation GC in 207ms. G1 Eden Space: 12264144896 -> 0; G1 Old Gen: 6427987960 -> 7337681912; G1 Survivor Space: 1795162112 - INFO 13:33:32 G1 Young Generation GC in 202ms. G1 Eden Space: 18589155328 -> 0; G1 Old Gen: 8506316280 -> 8514704880; G1 Survivor Space: 1308622848 - INFO 13:35:09 G1 Young Generation GC in 203ms. G1 Eden Space: 17532190720 -> 0; G1 Survivor Space: 1258291200 -> 1761607680; INFO 13:38:33 G1 Young Generation GC in 216ms. G1 Eden Space: 11106516992 -> 0; G1 Old Gen: 11571635704 -> 12698973168; G1 Survivor Space: 1795162112 INFO 13:47:36 G1 Young Generation GC in 229ms. G1 Eden Space: 19075694592 -> 0; G1 Old Gen: 3825205752 -> 4303660528; G1 Survivor Space: 1526726656 - INFO 13:53:43 G1 Young Generation GC in 207ms. G1 Eden Space: 14042529792 -> 0; G1 Old Gen: 7324985848 -> 8363582960; G1 Survivor Space: 1660944384 - INFO 13:55:40 G1 Young Generation GC in 201ms. G1 Eden Space: 18874368000 -> 0; G1 Old Gen: 8355194360 -> 8820888560; G1 Survivor Space: 1728053248 - INFO 13:56:22 G1 Young Generation GC in 206ms. G1 Eden Space: 19394461696 -> 0; G1 Survivor Space: 1207959552 -> 1711276032; INFO 14:08:09 Saved KeyCache (4563 items) in 189 ms INFO 14:09:31 Redistributing index summaries INFO 14:12:47 G1 Young Generation GC in 271ms. G1 Eden Space: 18874368000 -> 0; G1 Old Gen: 4345299448 -> 4814148600; G1 Survivor Space: 1728053248 - INFO 14:18:05 G1 Young Generation GC in 224ms. G1 Eden Space: 18907922432 -> 0; G1 Old Gen: 7021265392 -> 7483297784; G1 Survivor Space: 1627389952 - INFO 14:36:05 G1 Young Generation GC in 304ms. G1 Eden Space: 18958254080 -> 0; G1 Old Gen: 3976200696 -> 4437455344; G1 Survivor Space: 1644167168 - INFO 14:36:07 G1 Young Generation GC in 212ms. G1 Eden Space: 1711276032 -> 0; G1 Old Gen: 4437455344 -> 5830082552; G1 Survivor Space: 1459617792 -> INFO 15:00:30 G1 Young Generation GC in 233ms. G1 Eden Space: 19159580672 -> 0; G1 Old Gen: 4040399864 -> 4343952368; G1 Survivor Space: 1442840576 - INFO 15:09:31 Redistributing index summaries INFO 15:26:28 G1 Young Generation GC in 234ms. G1 Eden Space: 19075694592 -> 0; G1 Old Gen: 4604513264 -> 5064485872; G1 Survivor Space: 1526726656 - INFO 15:52:08 G1 Young Generation GC in 222ms. G1 Eden Space: 19126026240 -> 0; G1 Old Gen: 4118806520 -> 4391313912; G1 Survivor Space: 1476395008 - INFO 15:55:06 G1 Young Generation GC in 202ms. G1 Eden Space: 19142803456 -> 0; G1 Old Gen: 5267292152 -> 5559551472; INFO 16:09:31 Redistributing index summaries INFO 16:20:48 G1 Young Generation GC in 241ms. G1 Eden Space: 19126026240 -> 0; G1 Old Gen: 3854587896 -> 4287600112; G1 Survivor Space: 1476395008 - INFO 16:31:07 G1 Young Generation GC in 403ms. G1 Eden Space: 822083584 -> 0; G1 Old Gen: 9219080688 -> 9994162168; G1 Survivor Space: 889192448 -> 2 INFO 16:33:41 G1 Young Generation GC in 426ms. G1 Eden Space: 620756992 -> 0; G1 Old Gen: 10147301880 -> 11065889792; G1 Survivor Space: 1090519040 - INFO 16:38:17 G1 Young Generation GC in 835ms. G1 Eden Space: 251658240 -> 0; G1 Old Gen: 12062818288 -> 13404581312; G1 Survivor Space: 1459617792 - INFO 16:43:08 G1 Young Generation GC in 569ms. G1 Eden Space: 3154116608 -> 0; G1 Old Gen: 14361297352 -> 15461740984; G1 Survivor Space: 1258291200 INFO 16:46:08 G1 Young Generation GC in 432ms. G1 Eden Space: 18706595840 -> 0; G1 Old Gen: 3256274416 -> 4355653104; G1 Survivor Space: 1023410176 - INFO 17:09:31 Redistributing index summaries INFO 17:26:10 G1 Young Generation GC in 744ms. G1 Eden Space: 201326592 -> 0; G1 Old Gen: 6969713640 -> 8462801376; G1 Survivor Space: 1577058304 -> INFO 17:29:04 G1 Young Generation GC in 204ms. G1 Eden Space: 603979776 -> 0; G1 Old Gen: 8454412776 -> 9439609320; G1 Survivor Space: 1107296256 -> INFO 17:41:09 G1 Young Generation GC in 279ms. G1 Eden Space: 352321536 -> 0; G1 Old Gen: 13615267816 -> 14860613592; G1 Survivor Space: 1358954496 - INFO 17:46:51 G1 Young Generation GC in 493ms. G1 Eden Space: 754974720 -> 0; G1 Old Gen: 3560839664 -> 4350490576; G1 Survivor Space: 956301312 -> 2 INFO 17:49:09 G1 Young Generation GC in 290ms. G1 Eden Space: 335544320 -> 0; G1 Old Gen: 4350490576 -> 5603827664; G1 Survivor Space: 1375731712 -> INFO 17:55:17 G1 Young Generation GC in 803ms. G1 Eden Space: 201326592 -> 0; G1 Old Gen: 6543114704 -> 8036691400; G1 Survivor Space: 1577058304 -> INFO 17:58:07 G1 Young Generation GC in 206ms. G1 Eden Space: 4764729344 -> 0; G1 Old Gen: 8036691400 -> 8934491600; G1 Survivor Space: 973078528 -> INFO 18:07:29 G1 Young Generation GC in 306ms. G1 Eden Space: 704643072 -> 0; G1 Old Gen: 12163481536 -> 13045510592; G1 Survivor Space: 1006632960 - INFO 18:08:09 Saved KeyCache (5307 items) in 85 ms INFO 18:09:31 Redistributing index summaries INFO 18:10:11 G1 Young Generation GC in 277ms. G1 Eden Space: 402653184 -> 0; G1 Old Gen: 13045510592 -> 14252493256; G1 Survivor Space: 1308622848 - INFO 18:15:39 G1 Young Generation GC in 238ms. G1 Eden Space: 13052674048 -> 0; G1 Old Gen: 15141437888 -> 15520873416; G1 Survivor Space: 1358954496 INFO 18:20:16 G1 Young Generation GC in 540ms. G1 Eden Space: 7616856064 -> 0; G1 Old Gen: 3909694456 -> 4886048752; G1 Survivor Space: 1509949440 -> INFO 18:23:04 G1 Young Generation GC in 221ms. G1 Eden Space: 16894656512 -> 0; G1 Old Gen: 5420089848 -> 6187322368; G1 Survivor Space: 939524096 -> INFO 18:42:08 G1 Young Generation GC in 708ms. G1 Eden Space: 352321536 -> 0; G1 Old Gen: 12840221688 -> 14085112816; G1 Survivor Space: 1358954496 - INFO 18:50:13 G1 Young Generation GC in 318ms. G1 Eden Space: 218103808 -> 0; G1 Old Gen: 3631963120 -> 5033289720; G1 Survivor Space: 1493172224 -> INFO 19:09:31 Redistributing index summaries INFO 19:27:02 G1 Young Generation GC in 384ms. G1 Eden Space: 12901679104 -> 0; G1 Old Gen: 7034014192 -> 7759489528; G1 Survivor Space: 889192448 -> INFO 19:31:25 G1 Young Generation GC in 770ms. G1 Eden Space: 218103808 -> 0; G1 Old Gen: 8506048488 -> 9921951720; G1 Survivor Space: 1493172224 -> INFO 20:09:31 Redistributing index summaries INFO 21:09:31 Redistributing index summaries INFO 21:19:39 G1 Young Generation GC in 201ms. G1 Eden Space: 19142803456 -> 0; G1 Old Gen: 3917480432 -> 4124623344; G1 Survivor Space: 1459617792 - INFO 22:04:05 G1 Young Generation GC in 209ms. G1 Eden Space: 19142803456 -> 0; G1 Old Gen: 4420796912 -> 4618141680; G1 Survivor Space: 1459617792 - INFO 22:07:49 G1 Young Generation GC in 215ms. G1 Eden Space: 14713618432 -> 0; G1 Old Gen: 5970255352 -> 6555802608; G1 Survivor Space: 1493172224 - INFO 22:08:09 Saved KeyCache (5845 items) in 87 ms INFO 22:09:31 Redistributing index summaries INFO 22:38:04 G1 Young Generation GC in 201ms. G1 Eden Space: 19075694592 -> 0; G1 Old Gen: 4248291312 -> 4459621368; G1 Survivor Space: 1526726656 - INFO 22:44:12 G1 Young Generation GC in 203ms. G1 Eden Space: 10770972672 -> 0; G1 Old Gen: 6703617016 -> 7549747184; G1 Survivor Space: 1325400064 - INFO 22:47:05 G1 Young Generation GC in 207ms. G1 Eden Space: 19025362944 -> 0; G1 Old Gen: 7549747184 -> 7771343352; G1 Survivor Space: 1577058304 - INFO 23:08:59 G1 Young Generation GC in 369ms. G1 Eden Space: 15820914688 -> 0; G1 Old Gen: 4823449592 -> 5478502888; G1 Survivor Space: 1660944384 - INFO 23:09:31 Redistributing index summaries INFO 23:45:21 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB INFO 23:58:40 G1 Young Generation GC in 283ms. G1 Eden Space: 18941476864 -> 0; G1 Old Gen: 4831838192 -> 5297926640; G1 Survivor Space: 1660944384 - INFO 00:01:54 G1 Young Generation GC in 211ms. G1 Eden Space: 18891145216 -> 0; G1 Old Gen: 6425673712 -> 6912948208; G1 Survivor Space: 1711276032 - INFO 00:05:37 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB INFO 00:09:31 Redistributing index summaries INFO 00:14:33 G1 Young Generation GC in 282ms. G1 Eden Space: 14344519680 -> 0; G1 Old Gen: 14159246832 -> 14677156848; G1 Survivor Space: 1207959552 INFO 00:21:32 G1 Young Generation GC in 225ms. G1 Eden Space: 19042140160 -> 0; G1 Old Gen: 4346926576 -> 4865982456; G1 Survivor Space: 1560281088 - INFO 00:22:09 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB INFO 00:24:31 G1 Young Generation GC in 266ms. G1 Eden Space: 19008585728 -> 0; G1 Old Gen: 6106907128 -> 6613907952; G1 Survivor Space: 1593835520 - INFO 00:29:46 G1 Young Generation GC in 233ms. G1 Eden Space: 18236833792 -> 0; G1 Old Gen: 9076474360 -> 9498544624; G1 Survivor Space: 1795162112 - INFO 00:32:35 G1 Young Generation GC in 226ms. G1 Eden Space: 16492003328 -> 0; G1 Old Gen: 10804527088 -> 11351712752; G1 Survivor Space: 1811939328 INFO 00:37:09 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB INFO 00:41:24 G1 Young Generation GC in 330ms. G1 Eden Space: 19142803456 -> 0; G1 Old Gen: 3426117112 -> 3961507312; G1 Survivor Space: 1459617792 - INFO 00:44:26 G1 Young Generation GC in 325ms. G1 Eden Space: 19042140160 -> 0; G1 Old Gen: 5268046320 -> 5788210120; G1 Survivor Space: 1560281088 - INFO 00:47:13 G1 Young Generation GC in 209ms. G1 Eden Space: 19243466752 -> 0; G1 Old Gen: 7147093960 -> 7138705360; G1 Survivor Space: 1358954496 - INFO 00:47:28 G1 Young Generation GC in 312ms. G1 Eden Space: 18605932544 -> 0; G1 Old Gen: 7138705360 -> 7710094280; G1 Survivor Space: 1996488704 - INFO 00:50:35 G1 Young Generation GC in 330ms. G1 Eden Space: 17934843904 -> 0; G1 Old Gen: 9118416840 -> 9688820680; G1 Survivor Space: 1929379840 - INFO 00:52:52 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB INFO 00:56:29 G1 Young Generation GC in 251ms. G1 Eden Space: 14797504512 -> 0; G1 Old Gen: 13404995520 -> 13787049928; G1 Survivor Space: 1358954496 INFO 00:58:14 G1 Young Generation GC in 662ms. G1 Eden Space: 369098752 -> 0; G1 Old Gen: 14730395584 -> 15984921520; G1 Survivor Space: 1342177280 - INFO 01:01:25 G1 Young Generation GC in 288ms. G1 Eden Space: 19042140160 -> 0; G1 Old Gen: 2618580968 -> 3127466464; G1 Survivor Space: 1560281088 - INFO 01:04:17 G1 Young Generation GC in 201ms. G1 Eden Space: 19327352832 -> 0; G1 Old Gen: 4521459688 -> 4529848288; G1 Survivor Space: 1275068416 - INFO 01:04:39 G1 Young Generation GC in 283ms. G1 Eden Space: 18723373056 -> 0; G1 Old Gen: 4529848288 -> 5024210408; G1 Survivor Space: 1879048192 - INFO 01:07:23 G1 Young Generation GC in 294ms. G1 Eden Space: 18740150272 -> 0; G1 Old Gen: 6308233704 -> 6875745256; G1 Survivor Space: 1862270976 - INFO 01:07:53 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB INFO 01:09:31 Redistributing index summaries INFO 01:10:12 G1 Young Generation GC in 213ms. G1 Eden Space: 19243466752 -> 0; G1 Old Gen: 8262779360 -> 8254390760; G1 Survivor Space: 1358954496 - INFO 01:10:33 G1 Young Generation GC in 319ms. G1 Eden Space: 18639486976 -> 0; G1 Old Gen: 8254390760 -> 8821123552; G1 Survivor Space: 1962934272 - INFO 01:22:55 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB INFO 01:24:25 G1 Young Generation GC in 227ms. G1 Eden Space: 19226689536 -> 0; G1 Old Gen: 4245307376 -> 4236918776; G1 Survivor Space: 1375731712 - INFO 01:24:48 G1 Young Generation GC in 219ms. G1 Eden Space: 15401484288 -> 0; G1 Old Gen: 4236918776 -> 5231503864; G1 Survivor Space: 2097152000 - INFO 01:42:01 G1 Young Generation GC in 212ms. G1 Eden Space: 16005464064 -> 0; G1 Old Gen: 4320133112 -> 5100782576; G1 Survivor Space: 1627389952 - INFO 01:48:28 G1 Young Generation GC in 209ms. G1 Eden Space: 14730395648 -> 0; G1 Old Gen: 8901994480 -> 9402393592; G1 Survivor Space: 1291845632 - INFO 02:04:18 G1 Young Generation GC in 241ms. G1 Eden Space: 19042140160 -> 0; G1 Old Gen: 7825270264 -> 8327173624; G1 Survivor Space: 1560281088 - INFO 02:08:09 Saved KeyCache (8634 items) in 94 ms INFO 02:09:31 Redistributing index summaries INFO 02:14:50 G1 Young Generation GC in 314ms. G1 Eden Space: 19075694592 -> 0; G1 Old Gen: 3505807352 -> 4140375528; G1 Survivor Space: 1526726656 - INFO 02:48:35 G1 Young Generation GC in 321ms. G1 Eden Space: 11576279040 -> 0; G1 Old Gen: 3865890800 -> 4482588664; G1 Survivor Space: 1543503872 - INFO 02:53:54 G1 Young Generation GC in 203ms. G1 Eden Space: 17901289472 -> 0; G1 Old Gen: 7927235056 -> 8552357368; G1 Survivor Space: 1593835520 - INFO 03:04:33 G1 Young Generation GC in 345ms. G1 Eden Space: 13505658880 -> 0; G1 Old Gen: 3765058032 -> 4443454952; G1 Survivor Space: 1442840576 - INFO 03:09:31 Redistributing index summaries INFO 03:22:56 G1 Young Generation GC in 250ms. G1 Eden Space: 18991808512 -> 0; G1 Old Gen: 4806672880 -> 5340134904; G1 Survivor Space: 1610612736 - INFO 03:35:13 G1 Young Generation GC in 210ms. G1 Eden Space: 14445182976 -> 0; G1 Survivor Space: 1073741824 -> 1744830464; INFO 03:38:42 G1 Young Generation GC in 214ms. G1 Eden Space: 18303942656 -> 0; G1 Old Gen: 4786540536 -> 5432116720; G1 Survivor Space: 1375731712 - INFO 03:52:11 G1 Young Generation GC in 207ms. G1 Eden Space: 117440512 -> 0; G1 Old Gen: 12054831088 -> 10154180600; G1 Survivor Space: 1593835520 - INFO 03:54:48 G1 Young Generation GC in 224ms. G1 Eden Space: 16156459008 -> 0; G1 Old Gen: 4303355896 -> 4801558520; G1 Survivor Space: 1560281088 - INFO 04:09:31 Redistributing index summaries INFO 04:16:09 G1 Young Generation GC in 216ms. G1 Eden Space: 19142803456 -> 0; G1 Old Gen: 5853311984 -> 6366188536; G1 Survivor Space: 1459617792 - INFO 04:19:13 G1 Young Generation GC in 229ms. G1 Eden Space: 19042140160 -> 0; G1 Old Gen: 7826571256 -> 8457920504; G1 Survivor Space: 1560281088 - INFO 04:27:41 G1 Young Generation GC in 311ms. G1 Eden Space: 13723762688 -> 0; G1 Old Gen: 14176747504 -> 14808230392; G1 Survivor Space: 1509949440 INFO 04:29:56 G1 Young Generation GC in 274ms. G1 Eden Space: 18740150272 -> 0; G1 Old Gen: 3376452080 -> 3818638328; G1 Survivor Space: 1459617792 - INFO 04:33:40 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB INFO 04:49:07 G1 Young Generation GC in 319ms. G1 Eden Space: 16005464064 -> 0; G1 Old Gen: 4435714040 -> 5023449072; G1 Survivor Space: 1442840576 - INFO 04:54:06 G1 Young Generation GC in 208ms. G1 Eden Space: 18706595840 -> 0; G1 Old Gen: 7843348472 -> 8466889208; G1 Survivor Space: 1476395008 - INFO 05:09:31 Redistributing index summaries I found INFO Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB, does it mean the memory is not enough ? I want to know why adding a new node spent so much time and what need I do to complete a new node joining the cluster ? I need some help and really need some suggestions. Thanks a lot! ________________________________ The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
