[jira] [Comment Edited] (KUDU-2131) Tablet copy session may expire before completion for large tablet
[
https://issues.apache.org/jira/browse/KUDU-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151564#comment-16151564
]
Hao Hao edited comment on KUDU-2131 at 9/2/17 9:05 PM:
---
[~adar] Yes, that is the issue.
I did two tablet copy tests, the first one passed without problem, though the
second failed.
1) in a fresh tablet server with singe disk, and no container opened. The size
of the copied tablet is 37.2GB.
2) in a large cluster that has being running with several kudu release bits.
The tablet server has 14 disks, and 19199 containers only 97 of them are full.
The size of the copied tablet is 26.81 GB.
I suspect the issue could be related to the number of opened containers.
Looking at the code, in LBM when we {{GetNextDataDir}}, we always do pop_front
of the opened container queue, while when {{MakeContainerAvailable}}, we
push_back the used container to the end of the opened container queue.
I just did an equivalent experiment as test 2) I mentioned above, with the
change in {{MakeContainerAvailable}} to always add the container back at the
front of the opened container queue. This time DownloadBlocks finished quickly
and the table copy session complete successfully.
was (Author: hahao):
[~adar] Yes, that is the issue.
I did two tests:
1) in a fresh tablet server with singe disk, and no container opened. The size
of the copied tablet is 37.2GB.
2) in a large cluster that has being running with several kudu release bits.
The tablet server has 14 disks, and 19199 containers only 97 of them are full.
The size of the copied tablet is 26.81 GB.
I suspect the issue could be related to the number of opened containers.
Looking at the code, in LBM when we {{GetNextDataDir}}, we always do pop_front
of the opened container queue, while when {{MakeContainerAvailable}}, we
push_back the used container to the end of the opened container queue.
I just did an equivalent experiment as test 2) I mentioned above, with the
change in {{MakeContainerAvailable}} to always add the container back at the
front of the opened container queue. This time DownloadBlocks finished quickly
and the table copy session complete successfully.
> Tablet copy session may expire before completion for large tablet
> -
>
> Key: KUDU-2131
> URL: https://issues.apache.org/jira/browse/KUDU-2131
> Project: Kudu
> Issue Type: Bug
>Reporter: Hao Hao
>Priority: Blocker
> Fix For: 1.5.0
>
>
> KUDU-1726 introduced an optimization to do a bulk sync-to-disk once the
> tablet copy operation is complete. However, when tested in a large cluster, I
> found disk synchronization in batches can result in the tablet session
> expires before the synchronization complete. There is a flag
> '--tablet_copy_idle_timeout_ms' to control the amount of time without
> activity before a tablet copy session expires, but it is tagged as hidden(not
> user-facing).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KUDU-2131) Tablet copy session may expire before completion for large tablet
[ https://issues.apache.org/jira/browse/KUDU-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151591#comment-16151591 ] Hao Hao commented on KUDU-2131: --- Yes, #1 first test passed but the #2 failed without the quick fix I mentioned. Sorry for not being clear. If you think the fix makes sense, I will put up a CR for it. Meanwhile, do you think we should still add a user facing flag to control the tablet copy expiration time, similar to "--tablet_copy_idle_timeout_ms", to at least have a workaround if encounter case like this again? Even though I would image it would be very rare given "log_container_max_size" to be 10GB. > Tablet copy session may expire before completion for large tablet > - > > Key: KUDU-2131 > URL: https://issues.apache.org/jira/browse/KUDU-2131 > Project: Kudu > Issue Type: Bug >Reporter: Hao Hao >Priority: Blocker > Fix For: 1.5.0 > > > KUDU-1726 introduced an optimization to do a bulk sync-to-disk once the > tablet copy operation is complete. However, when tested in a large cluster, I > found disk synchronization in batches can result in the tablet session > expires before the synchronization complete. There is a flag > '--tablet_copy_idle_timeout_ms' to control the amount of time without > activity before a tablet copy session expires, but it is tagged as hidden(not > user-facing). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KUDU-2131) Tablet copy session may expire before completion for large tablet
[ https://issues.apache.org/jira/browse/KUDU-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151567#comment-16151567 ] Adar Dembo commented on KUDU-2131: -- Interesting. So although the FIFO container retrieval serves to "spread the load" across all available containers, that actually hurts us by increasing the total number of containers (and thus files) that need to be fsynced. The change you suggested seems like an easy fix; I don't see any reason why FIFO would be a _good_ thing here. Just to be clear, your first test didn't trigger the issue, right? On a fresh tserver doing a copy I'd expect one non-full container per disk at a time. But I can definitely see how test #2 would trigger it. > Tablet copy session may expire before completion for large tablet > - > > Key: KUDU-2131 > URL: https://issues.apache.org/jira/browse/KUDU-2131 > Project: Kudu > Issue Type: Bug >Reporter: Hao Hao >Priority: Blocker > Fix For: 1.5.0 > > > KUDU-1726 introduced an optimization to do a bulk sync-to-disk once the > tablet copy operation is complete. However, when tested in a large cluster, I > found disk synchronization in batches can result in the tablet session > expires before the synchronization complete. There is a flag > '--tablet_copy_idle_timeout_ms' to control the amount of time without > activity before a tablet copy session expires, but it is tagged as hidden(not > user-facing). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KUDU-2131) Tablet copy session may expire before completion for large tablet
[
https://issues.apache.org/jira/browse/KUDU-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151564#comment-16151564
]
Hao Hao commented on KUDU-2131:
---
[~adar] Yes, that is the issue.
I did two tests:
1) in a fresh tablet server with singe disk, and no container opened. The size
of the copied tablet is 37.2GB.
2) in a large cluster that has being running with several kudu release bits.
The tablet server has 14 disks, and 19199 containers only 97 of them are full.
The size of the copied tablet is 26.81 GB.
I suspect the issue could be related to the number of opened containers.
Looking at the code, in LBM when we {{GetNextDataDir}}, we always do pop_front
of the opened container queue, while when {{MakeContainerAvailable}}, we
push_back the used container to the end of the opened container queue.
I just did an equivalent experiment as test 2) I mentioned above, with the
change in {{MakeContainerAvailable}} to always add back at the front of the
opened container queue. This time DownloadBlocks finished quickly and the table
copy session complete successfully.
> Tablet copy session may expire before completion for large tablet
> -
>
> Key: KUDU-2131
> URL: https://issues.apache.org/jira/browse/KUDU-2131
> Project: Kudu
> Issue Type: Bug
>Reporter: Hao Hao
>Priority: Blocker
> Fix For: 1.5.0
>
>
> KUDU-1726 introduced an optimization to do a bulk sync-to-disk once the
> tablet copy operation is complete. However, when tested in a large cluster, I
> found disk synchronization in batches can result in the tablet session
> expires before the synchronization complete. There is a flag
> '--tablet_copy_idle_timeout_ms' to control the amount of time without
> activity before a tablet copy session expires, but it is tagged as hidden(not
> user-facing).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (KUDU-2132) alter_table-randomized-test can drop primary key columns
Adar Dembo created KUDU-2132: Summary: alter_table-randomized-test can drop primary key columns Key: KUDU-2132 URL: https://issues.apache.org/jira/browse/KUDU-2132 Project: Kudu Issue Type: Bug Components: test Reporter: Adar Dembo Attachments: alter_table-randomized-test.txt There's code in the test to prevent this from happening, so I'm not quite sure why it happened. I've attached the test log. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KUDU-2131) Tablet copy session may expire before completion for large tablet
[ https://issues.apache.org/jira/browse/KUDU-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151536#comment-16151536 ] Adar Dembo commented on KUDU-2131: -- [~hahao] can you elaborate? Is the issue that DownloadBlocks can take >3m (default idle timeout) to commit the new blocks such that the session is terminated before DownloadWALs? How large of a tablet is necessary to trigger this effect? > Tablet copy session may expire before completion for large tablet > - > > Key: KUDU-2131 > URL: https://issues.apache.org/jira/browse/KUDU-2131 > Project: Kudu > Issue Type: Bug >Reporter: Hao Hao >Priority: Blocker > Fix For: 1.5.0 > > > KUDU-1726 introduced an optimization to do a bulk sync-to-disk once the > tablet copy operation is complete. However, when tested in a large cluster, I > found disk synchronization in batches can result in the tablet session > expires before the synchronization complete. There is a flag > '--tablet_copy_idle_timeout_ms' to control the amount of time without > activity before a tablet copy session expires, but it is tagged as hidden(not > user-facing). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KUDU-2128) 'move replica' tool should cancel tablet copy when exit with timeout
[ https://issues.apache.org/jira/browse/KUDU-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151427#comment-16151427 ] Hao Hao commented on KUDU-2128: --- Found out the tablet copy failure I encountered is caused by KUDU-2131. But I still think we may want to cancel the tablet copy in this case, as that is the intention of the flag? Or we can just remove the '--move_copy_timeout_sec' flag? > 'move replica' tool should cancel tablet copy when exit with timeout > > > Key: KUDU-2128 > URL: https://issues.apache.org/jira/browse/KUDU-2128 > Project: Kudu > Issue Type: Improvement >Affects Versions: 1.5.0 >Reporter: Hao Hao > > Currently, 'kudu move replica' tool, which moves tablet replica from one to > another, will time out based on the setting of flag --move_copy_timeout_sec. > When the request timeout, it will just exit without aborting the ongoing > tablet copy. > 1) It would be a better experience to cancel the tablet copy while exit. > 2) It also would be nice to show the timeout setting in the help content for > users to be aware of this config. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
