[ https://issues.apache.org/jira/browse/IMPALA-7138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tim Armstrong updated IMPALA-7138: ---------------------------------- Summary: Fix detection and handling of Device Mapper volumes (was: Impala gets confused by multiple scratch directories on Device Mapper volumes) > Fix detection and handling of Device Mapper volumes > --------------------------------------------------- > > Key: IMPALA-7138 > URL: https://issues.apache.org/jira/browse/IMPALA-7138 > Project: IMPALA > Issue Type: Bug > Components: Backend > Affects Versions: Impala 3.0, Impala 2.12.0 > Reporter: Tim Armstrong > Assignee: Tim Armstrong > Priority: Major > Labels: resource-management > > I've heard a couple of instances of Impala doing the wrong thing when > multiple scratch directories are present on a single DeviceMapper logical > volume. E.g. in the following setup, we only use one directory as a disk. > {noformat} > >> mount list > /dev/mapper/xx01-lv_data01 on /data01 type ext4 (rw,relatime,data=ordered) > /dev/mapper/xx02-lv_data02 on /data02 type ext4 (rw,relatime,data=ordered) > /dev/mapper/xx03-lv_data03 on /data03 type ext4 (rw,relatime,data=ordered) > /dev/mapper/xx04-lv_data04 on /data04 type ext4 (rw,relatime,data=ordered) > /dev/mapper/xx05-lv_data05 on /data05 type ext4 (rw,relatime,data=ordered) > /dev/mapper/xx06-lv_data06 on /data06 type ext4 (rw,relatime,data=ordered) > /dev/mapper/xx07-lv_data07 on /data07 type ext4 (rw,relatime,data=ordered) > /dev/mapper/xx08-lv_data08 on /data08 type ext4 (rw,relatime,data=ordered) > /dev/mapper/xx09-lv_data09 on /data09 type ext4 (rw,relatime,data=ordered) > /dev/mapper/xx10-lv_data10 on /data10 type ext4 (rw,relatime,data=ordered) > /dev/mapper/xx11-lv_data11 on /data11 type ext4 (rw,relatime,data=ordered) > /dev/mapper/xx12-lv_data12 on /data12 type ext4 (rw,relatime,data=ordered) > Scratch dirs are: > /data01/impala/impalad,/data02/impala/impalad,/data03/impala/impalad,/data04/impala/impalad,/data05/impala/impalad,/data06/impala/impalad,/data07/impala/impalad,/data08/impala/impalad,/data09/impala/impalad,/data10/impala/impalad,/data11/impala/impalad,/data12/impala/impalad > ~~~ tmp-file-mgr.cc:122] Using scratch directory > /data01/impala/impalad/impala-scratch on disk13. > {noformat} > A workaround is to set --allow_multiple_scratch_dirs_per_device=true > There are a few questions here: > # Does the deduplication logic even make sense to have? > # Do we need to fix how we treat Device mapper volumes? -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org