[jira] [Updated] (HIVE-17482) External LLAP client: acquire locks for tables queried directly by LLAP
[ https://issues.apache.org/jira/browse/HIVE-17482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-17482: -- Component/s: Transactions > External LLAP client: acquire locks for tables queried directly by LLAP > --- > > Key: HIVE-17482 > URL: https://issues.apache.org/jira/browse/HIVE-17482 > Project: Hive > Issue Type: Sub-task > Components: llap, Transactions >Reporter: Jason Dere >Assignee: Jason Dere > Fix For: 3.0.0 > > Attachments: HIVE-17482.1.patch, HIVE-17482.2.patch, > HIVE-17482.3.patch, HIVE-17482.4.patch, HIVE-17482.5.patch, HIVE-17482.6.patch > > > When using the LLAP external client with simple queries (filter/project of > single table), the appropriate locks should be taken on the table being read > like they are for normal Hive queries. This is important in the case of > transactional tables being queried, since the compactor relies on the > presence of table locks to determine whether it can safely delete old > versions of compacted files without affecting currently running queries. > This does not have to happen in the complex query case, since a query is used > (with the appropriate locking mechanisms) to create/populate the temp table > holding the results to the complex query. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17482) External LLAP client: acquire locks for tables queried directly by LLAP
[ https://issues.apache.org/jira/browse/HIVE-17482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-17482: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to master > External LLAP client: acquire locks for tables queried directly by LLAP > --- > > Key: HIVE-17482 > URL: https://issues.apache.org/jira/browse/HIVE-17482 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Fix For: 3.0.0 > > Attachments: HIVE-17482.1.patch, HIVE-17482.2.patch, > HIVE-17482.3.patch, HIVE-17482.4.patch, HIVE-17482.5.patch, HIVE-17482.6.patch > > > When using the LLAP external client with simple queries (filter/project of > single table), the appropriate locks should be taken on the table being read > like they are for normal Hive queries. This is important in the case of > transactional tables being queried, since the compactor relies on the > presence of table locks to determine whether it can safely delete old > versions of compacted files without affecting currently running queries. > This does not have to happen in the complex query case, since a query is used > (with the appropriate locking mechanisms) to create/populate the temp table > holding the results to the complex query. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17482) External LLAP client: acquire locks for tables queried directly by LLAP
[ https://issues.apache.org/jira/browse/HIVE-17482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-17482: -- Attachment: HIVE-17482.6.patch Updated test based on feedback from [~ekoifman] > External LLAP client: acquire locks for tables queried directly by LLAP > --- > > Key: HIVE-17482 > URL: https://issues.apache.org/jira/browse/HIVE-17482 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-17482.1.patch, HIVE-17482.2.patch, > HIVE-17482.3.patch, HIVE-17482.4.patch, HIVE-17482.5.patch, HIVE-17482.6.patch > > > When using the LLAP external client with simple queries (filter/project of > single table), the appropriate locks should be taken on the table being read > like they are for normal Hive queries. This is important in the case of > transactional tables being queried, since the compactor relies on the > presence of table locks to determine whether it can safely delete old > versions of compacted files without affecting currently running queries. > This does not have to happen in the complex query case, since a query is used > (with the appropriate locking mechanisms) to create/populate the temp table > holding the results to the complex query. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17482) External LLAP client: acquire locks for tables queried directly by LLAP
[ https://issues.apache.org/jira/browse/HIVE-17482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-17482: -- Attachment: HIVE-17482.5.patch Added a unit test to TestAcidOnTez, will post update to RB > External LLAP client: acquire locks for tables queried directly by LLAP > --- > > Key: HIVE-17482 > URL: https://issues.apache.org/jira/browse/HIVE-17482 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-17482.1.patch, HIVE-17482.2.patch, > HIVE-17482.3.patch, HIVE-17482.4.patch, HIVE-17482.5.patch > > > When using the LLAP external client with simple queries (filter/project of > single table), the appropriate locks should be taken on the table being read > like they are for normal Hive queries. This is important in the case of > transactional tables being queried, since the compactor relies on the > presence of table locks to determine whether it can safely delete old > versions of compacted files without affecting currently running queries. > This does not have to happen in the complex query case, since a query is used > (with the appropriate locking mechanisms) to create/populate the temp table > holding the results to the complex query. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17482) External LLAP client: acquire locks for tables queried directly by LLAP
[ https://issues.apache.org/jira/browse/HIVE-17482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-17482: -- Status: Patch Available (was: Open) > External LLAP client: acquire locks for tables queried directly by LLAP > --- > > Key: HIVE-17482 > URL: https://issues.apache.org/jira/browse/HIVE-17482 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-17482.1.patch, HIVE-17482.2.patch, > HIVE-17482.3.patch, HIVE-17482.4.patch > > > When using the LLAP external client with simple queries (filter/project of > single table), the appropriate locks should be taken on the table being read > like they are for normal Hive queries. This is important in the case of > transactional tables being queried, since the compactor relies on the > presence of table locks to determine whether it can safely delete old > versions of compacted files without affecting currently running queries. > This does not have to happen in the complex query case, since a query is used > (with the appropriate locking mechanisms) to create/populate the temp table > holding the results to the complex query. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17482) External LLAP client: acquire locks for tables queried directly by LLAP
[ https://issues.apache.org/jira/browse/HIVE-17482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-17482: -- Attachment: HIVE-17482.4.patch Updating patch after review feedback > External LLAP client: acquire locks for tables queried directly by LLAP > --- > > Key: HIVE-17482 > URL: https://issues.apache.org/jira/browse/HIVE-17482 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-17482.1.patch, HIVE-17482.2.patch, > HIVE-17482.3.patch, HIVE-17482.4.patch > > > When using the LLAP external client with simple queries (filter/project of > single table), the appropriate locks should be taken on the table being read > like they are for normal Hive queries. This is important in the case of > transactional tables being queried, since the compactor relies on the > presence of table locks to determine whether it can safely delete old > versions of compacted files without affecting currently running queries. > This does not have to happen in the complex query case, since a query is used > (with the appropriate locking mechanisms) to create/populate the temp table > holding the results to the complex query. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17482) External LLAP client: acquire locks for tables queried directly by LLAP
[ https://issues.apache.org/jira/browse/HIVE-17482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-17482: -- Status: Open (was: Patch Available) > External LLAP client: acquire locks for tables queried directly by LLAP > --- > > Key: HIVE-17482 > URL: https://issues.apache.org/jira/browse/HIVE-17482 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-17482.1.patch, HIVE-17482.2.patch, > HIVE-17482.3.patch > > > When using the LLAP external client with simple queries (filter/project of > single table), the appropriate locks should be taken on the table being read > like they are for normal Hive queries. This is important in the case of > transactional tables being queried, since the compactor relies on the > presence of table locks to determine whether it can safely delete old > versions of compacted files without affecting currently running queries. > This does not have to happen in the complex query case, since a query is used > (with the appropriate locking mechanisms) to create/populate the temp table > holding the results to the complex query. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17482) External LLAP client: acquire locks for tables queried directly by LLAP
[ https://issues.apache.org/jira/browse/HIVE-17482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-17482: -- Attachment: HIVE-17482.3.patch Precommit build never ran .. re-attaching patch. > External LLAP client: acquire locks for tables queried directly by LLAP > --- > > Key: HIVE-17482 > URL: https://issues.apache.org/jira/browse/HIVE-17482 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-17482.1.patch, HIVE-17482.2.patch, > HIVE-17482.3.patch > > > When using the LLAP external client with simple queries (filter/project of > single table), the appropriate locks should be taken on the table being read > like they are for normal Hive queries. This is important in the case of > transactional tables being queried, since the compactor relies on the > presence of table locks to determine whether it can safely delete old > versions of compacted files without affecting currently running queries. > This does not have to happen in the complex query case, since a query is used > (with the appropriate locking mechanisms) to create/populate the temp table > holding the results to the complex query. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17482) External LLAP client: acquire locks for tables queried directly by LLAP
[ https://issues.apache.org/jira/browse/HIVE-17482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-17482: -- Attachment: HIVE-17482.2.patch > External LLAP client: acquire locks for tables queried directly by LLAP > --- > > Key: HIVE-17482 > URL: https://issues.apache.org/jira/browse/HIVE-17482 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-17482.1.patch, HIVE-17482.2.patch > > > When using the LLAP external client with simple queries (filter/project of > single table), the appropriate locks should be taken on the table being read > like they are for normal Hive queries. This is important in the case of > transactional tables being queried, since the compactor relies on the > presence of table locks to determine whether it can safely delete old > versions of compacted files without affecting currently running queries. > This does not have to happen in the complex query case, since a query is used > (with the appropriate locking mechanisms) to create/populate the temp table > holding the results to the complex query. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17482) External LLAP client: acquire locks for tables queried directly by LLAP
[ https://issues.apache.org/jira/browse/HIVE-17482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-17482: -- Attachment: (was: HIVE-17482.2.patch) > External LLAP client: acquire locks for tables queried directly by LLAP > --- > > Key: HIVE-17482 > URL: https://issues.apache.org/jira/browse/HIVE-17482 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-17482.1.patch > > > When using the LLAP external client with simple queries (filter/project of > single table), the appropriate locks should be taken on the table being read > like they are for normal Hive queries. This is important in the case of > transactional tables being queried, since the compactor relies on the > presence of table locks to determine whether it can safely delete old > versions of compacted files without affecting currently running queries. > This does not have to happen in the complex query case, since a query is used > (with the appropriate locking mechanisms) to create/populate the temp table > holding the results to the complex query. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17482) External LLAP client: acquire locks for tables queried directly by LLAP
[ https://issues.apache.org/jira/browse/HIVE-17482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-17482: -- Attachment: HIVE-17482.2.patch > External LLAP client: acquire locks for tables queried directly by LLAP > --- > > Key: HIVE-17482 > URL: https://issues.apache.org/jira/browse/HIVE-17482 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-17482.1.patch, HIVE-17482.2.patch > > > When using the LLAP external client with simple queries (filter/project of > single table), the appropriate locks should be taken on the table being read > like they are for normal Hive queries. This is important in the case of > transactional tables being queried, since the compactor relies on the > presence of table locks to determine whether it can safely delete old > versions of compacted files without affecting currently running queries. > This does not have to happen in the complex query case, since a query is used > (with the appropriate locking mechanisms) to create/populate the temp table > holding the results to the complex query. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17482) External LLAP client: acquire locks for tables queried directly by LLAP
[ https://issues.apache.org/jira/browse/HIVE-17482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-17482: -- Status: Patch Available (was: Open) [~ekoifman] [~sershe] would you guys be able to take a look? > External LLAP client: acquire locks for tables queried directly by LLAP > --- > > Key: HIVE-17482 > URL: https://issues.apache.org/jira/browse/HIVE-17482 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-17482.1.patch > > > When using the LLAP external client with simple queries (filter/project of > single table), the appropriate locks should be taken on the table being read > like they are for normal Hive queries. This is important in the case of > transactional tables being queried, since the compactor relies on the > presence of table locks to determine whether it can safely delete old > versions of compacted files without affecting currently running queries. > This does not have to happen in the complex query case, since a query is used > (with the appropriate locking mechanisms) to create/populate the temp table > holding the results to the complex query. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17482) External LLAP client: acquire locks for tables queried directly by LLAP
[ https://issues.apache.org/jira/browse/HIVE-17482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-17482: -- Attachment: HIVE-17482.1.patch - Add call to Driver to perform necessary locks for compiled query - Tie the cleanup of the locks to the connection used by the LLAP client to get the LLAP splits, like what was done for the temp tables generated for complex queries (HIVE-17455). When the LLAP splits for the external client are generated, call to acquire locks will be done, with a Closeable added to the SessionState so that the lock will be released when the session is cleaned up. > External LLAP client: acquire locks for tables queried directly by LLAP > --- > > Key: HIVE-17482 > URL: https://issues.apache.org/jira/browse/HIVE-17482 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-17482.1.patch > > > When using the LLAP external client with simple queries (filter/project of > single table), the appropriate locks should be taken on the table being read > like they are for normal Hive queries. This is important in the case of > transactional tables being queried, since the compactor relies on the > presence of table locks to determine whether it can safely delete old > versions of compacted files without affecting currently running queries. > This does not have to happen in the complex query case, since a query is used > (with the appropriate locking mechanisms) to create/populate the temp table > holding the results to the complex query. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
