[jira] [Updated] (HIVE-20515) Empty query results when using results cache and query temp dir, results cache dir in different filesystems
[ https://issues.apache.org/jira/browse/HIVE-20515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-20515: - Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) > Empty query results when using results cache and query temp dir, results > cache dir in different filesystems > --- > > Key: HIVE-20515 > URL: https://issues.apache.org/jira/browse/HIVE-20515 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20515.1.patch, HIVE-20515.2.patch > > > If the scratchdir for temporary query results and the results cache dir are > in different filesystems, moving the query from the temp directory to results > cache will fail. > Looking at the moveResultsToCacheDirectory() logic in QueryResultsCache.java, > I see the following issues: > - FileSystem.rename() is used, which only works if the files are on the same > filesystem. Need to use something like Hive.mvFile or something similar which > can work between different filesystems. > - The return code from rename() was not checked which might possibly have > caught the error here. This may not be applicable if a different method from > FS.rename() is used in the proper fix. > With some filesystems (noticed this with WASB), if FileSystem.rename() > returns false on failure rather than throwing an exception, then this results > in empty results showing up for the query because the return code was not > checked properly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20515) Empty query results when using results cache and query temp dir, results cache dir in different filesystems
[ https://issues.apache.org/jira/browse/HIVE-20515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20515: -- Attachment: HIVE-20515.2.patch > Empty query results when using results cache and query temp dir, results > cache dir in different filesystems > --- > > Key: HIVE-20515 > URL: https://issues.apache.org/jira/browse/HIVE-20515 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20515.1.patch, HIVE-20515.2.patch > > > If the scratchdir for temporary query results and the results cache dir are > in different filesystems, moving the query from the temp directory to results > cache will fail. > Looking at the moveResultsToCacheDirectory() logic in QueryResultsCache.java, > I see the following issues: > - FileSystem.rename() is used, which only works if the files are on the same > filesystem. Need to use something like Hive.mvFile or something similar which > can work between different filesystems. > - The return code from rename() was not checked which might possibly have > caught the error here. This may not be applicable if a different method from > FS.rename() is used in the proper fix. > With some filesystems (noticed this with WASB), if FileSystem.rename() > returns false on failure rather than throwing an exception, then this results > in empty results showing up for the query because the return code was not > checked properly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20515) Empty query results when using results cache and query temp dir, results cache dir in different filesystems
[ https://issues.apache.org/jira/browse/HIVE-20515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20515: -- Attachment: HIVE-20515.1.patch Status: Patch Available (was: Open) > Empty query results when using results cache and query temp dir, results > cache dir in different filesystems > --- > > Key: HIVE-20515 > URL: https://issues.apache.org/jira/browse/HIVE-20515 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20515.1.patch > > > If the scratchdir for temporary query results and the results cache dir are > in different filesystems, moving the query from the temp directory to results > cache will fail. > Looking at the moveResultsToCacheDirectory() logic in QueryResultsCache.java, > I see the following issues: > - FileSystem.rename() is used, which only works if the files are on the same > filesystem. Need to use something like Hive.mvFile or something similar which > can work between different filesystems. > - The return code from rename() was not checked which might possibly have > caught the error here. This may not be applicable if a different method from > FS.rename() is used in the proper fix. > With some filesystems (noticed this with WASB), if FileSystem.rename() > returns false on failure rather than throwing an exception, then this results > in empty results showing up for the query because the return code was not > checked properly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
