[GitHub] incubator-hawq pull request #1370: HAWQ-1508 fix travis broken
Github user weinan003 closed the pull request at: https://github.com/apache/incubator-hawq/pull/1370 ---
[jira] [Assigned] (HAWQ-1622) Cache PXF proxy UGI so that cleanup of FileSystem cache does't have to be done on each request
[ https://issues.apache.org/jira/browse/HAWQ-1622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Denissov reassigned HAWQ-1622: Assignee: Divya Bhargov (was: Radar Lei) > Cache PXF proxy UGI so that cleanup of FileSystem cache does't have to be > done on each request > -- > > Key: HAWQ-1622 > URL: https://issues.apache.org/jira/browse/HAWQ-1622 > Project: Apache HAWQ > Issue Type: Improvement > Components: PXF >Reporter: Alexander Denissov >Assignee: Divya Bhargov >Priority: Major > > Closing PXF proxy UGIs on each request (implemented in HAWQ-1621) slows down > PXF request response time significantly when several threads work > concurrently as it locks FileSystem cache and holds the lock while the > cleanup of DFSClients is completed. > This can be avoided by caching the proxy UGI for a given proxy user between > requests. Care must be taken to remove the cached entry after some > pre-defined TTL if and only if there are no current threads using any > FileSystem entries held by the cache. A combination of TTL-based cache with > ref-counting might be utilized to achieve this. > > For some example of this, see: > https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/service/UserGroupInformationService.java > Caching UGIs might be tricky when Kerberos support is implemented later, see: > https://issues.apache.org/jira/browse/HIVE-3098?focusedCommentId=13398979&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-13398979 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HAWQ-1622) Cache PXF proxy UGI so that cleanup of FileSystem cache doesn't have to be done on each request
[ https://issues.apache.org/jira/browse/HAWQ-1622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Denissov updated HAWQ-1622: - Summary: Cache PXF proxy UGI so that cleanup of FileSystem cache doesn't have to be done on each request (was: Cache PXF proxy UGI so that cleanup of FileSystem cache does't have to be done on each request) > Cache PXF proxy UGI so that cleanup of FileSystem cache doesn't have to be > done on each request > --- > > Key: HAWQ-1622 > URL: https://issues.apache.org/jira/browse/HAWQ-1622 > Project: Apache HAWQ > Issue Type: Improvement > Components: PXF >Reporter: Alexander Denissov >Assignee: Divya Bhargov >Priority: Major > > Closing PXF proxy UGIs on each request (implemented in HAWQ-1621) slows down > PXF request response time significantly when several threads work > concurrently as it locks FileSystem cache and holds the lock while the > cleanup of DFSClients is completed. > This can be avoided by caching the proxy UGI for a given proxy user between > requests. Care must be taken to remove the cached entry after some > pre-defined TTL if and only if there are no current threads using any > FileSystem entries held by the cache. A combination of TTL-based cache with > ref-counting might be utilized to achieve this. > > For some example of this, see: > https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/service/UserGroupInformationService.java > Caching UGIs might be tricky when Kerberos support is implemented later, see: > https://issues.apache.org/jira/browse/HIVE-3098?focusedCommentId=13398979&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-13398979 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HAWQ-1622) Cache PXF proxy UGI so that cleanup of FileSystem cache does't have to be done on each request
Alexander Denissov created HAWQ-1622: Summary: Cache PXF proxy UGI so that cleanup of FileSystem cache does't have to be done on each request Key: HAWQ-1622 URL: https://issues.apache.org/jira/browse/HAWQ-1622 Project: Apache HAWQ Issue Type: Improvement Reporter: Alexander Denissov Assignee: Radar Lei Closing PXF proxy UGIs on each request (implemented in HAWQ-1621) slows down PXF request response time significantly when several threads work concurrently as it locks FileSystem cache and holds the lock while the cleanup of DFSClients is completed. This can be avoided by caching the proxy UGI for a given proxy user between requests. Care must be taken to remove the cached entry after some pre-defined TTL if and only if there are no current threads using any FileSystem entries held by the cache. A combination of TTL-based cache with ref-counting might be utilized to achieve this. For some example of this, see: https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/service/UserGroupInformationService.java Caching UGIs might be tricky when Kerberos support is implemented later, see: https://issues.apache.org/jira/browse/HIVE-3098?focusedCommentId=13398979&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-13398979 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HAWQ-1622) Cache PXF proxy UGI so that cleanup of FileSystem cache does't have to be done on each request
[ https://issues.apache.org/jira/browse/HAWQ-1622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Denissov updated HAWQ-1622: - Component/s: PXF > Cache PXF proxy UGI so that cleanup of FileSystem cache does't have to be > done on each request > -- > > Key: HAWQ-1622 > URL: https://issues.apache.org/jira/browse/HAWQ-1622 > Project: Apache HAWQ > Issue Type: Improvement > Components: PXF >Reporter: Alexander Denissov >Assignee: Radar Lei >Priority: Major > > Closing PXF proxy UGIs on each request (implemented in HAWQ-1621) slows down > PXF request response time significantly when several threads work > concurrently as it locks FileSystem cache and holds the lock while the > cleanup of DFSClients is completed. > This can be avoided by caching the proxy UGI for a given proxy user between > requests. Care must be taken to remove the cached entry after some > pre-defined TTL if and only if there are no current threads using any > FileSystem entries held by the cache. A combination of TTL-based cache with > ref-counting might be utilized to achieve this. > > For some example of this, see: > https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/service/UserGroupInformationService.java > Caching UGIs might be tricky when Kerberos support is implemented later, see: > https://issues.apache.org/jira/browse/HIVE-3098?focusedCommentId=13398979&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-13398979 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HAWQ-1621) Call FileSystem.closeAllForUGI at the end of request when PXF impersonation is on
[ https://issues.apache.org/jira/browse/HAWQ-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Denissov resolved HAWQ-1621. -- Resolution: Fixed Fix Version/s: 2.4.0.0-incubating > Call FileSystem.closeAllForUGI at the end of request when PXF impersonation > is on > - > > Key: HAWQ-1621 > URL: https://issues.apache.org/jira/browse/HAWQ-1621 > Project: Apache HAWQ > Issue Type: Improvement > Components: PXF >Reporter: Divya Bhargov >Assignee: Alexander Denissov >Priority: Major > Fix For: 2.4.0.0-incubating > > > Due to open Hadoop JIRA: https://issues.apache.org/jira/browse/HDFS-3545 , > when PXF impersonation is on, proxy UGIs created for each request produce > FileSystem cache entries that are not removed when the request is over. This > causes OutOfMemory errors since the entries are never removed. To remove the > entries and clean the cache, we should call FileSystem.closeAllForUGI at the > end of request when PXF impersonation is on. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] incubator-hawq pull request #1374: HAWQ-1621. Call FileSystem.closeAllForUGI...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-hawq/pull/1374 ---
[jira] [Assigned] (HAWQ-1621) Call FileSystem.closeAllForUGI at the end of request when PXF impersonation is on
[ https://issues.apache.org/jira/browse/HAWQ-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Denissov reassigned HAWQ-1621: Assignee: Alexander Denissov (was: Ed Espino) > Call FileSystem.closeAllForUGI at the end of request when PXF impersonation > is on > - > > Key: HAWQ-1621 > URL: https://issues.apache.org/jira/browse/HAWQ-1621 > Project: Apache HAWQ > Issue Type: Improvement > Components: PXF >Reporter: Divya Bhargov >Assignee: Alexander Denissov >Priority: Major > > Due to open Hadoop JIRA: https://issues.apache.org/jira/browse/HDFS-3545 , > when PXF impersonation is on, proxy UGIs created for each request produce > FileSystem cache entries that are not removed when the request is over. This > causes OutOfMemory errors since the entries are never removed. To remove the > entries and clean the cache, we should call FileSystem.closeAllForUGI at the > end of request when PXF impersonation is on. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HAWQ-1621) Call FileSystem.closeAllForUGI at the end of request when PXF impersonation is on
[ https://issues.apache.org/jira/browse/HAWQ-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497230#comment-16497230 ] Alexander Denissov commented on HAWQ-1621: -- The fix from [~dbhargov] is available here: https://github.com/apache/incubator-hawq/pull/1374 > Call FileSystem.closeAllForUGI at the end of request when PXF impersonation > is on > - > > Key: HAWQ-1621 > URL: https://issues.apache.org/jira/browse/HAWQ-1621 > Project: Apache HAWQ > Issue Type: Improvement > Components: PXF >Reporter: Divya Bhargov >Assignee: Alexander Denissov >Priority: Major > > Due to open Hadoop JIRA: https://issues.apache.org/jira/browse/HDFS-3545 , > when PXF impersonation is on, proxy UGIs created for each request produce > FileSystem cache entries that are not removed when the request is over. This > causes OutOfMemory errors since the entries are never removed. To remove the > entries and clean the cache, we should call FileSystem.closeAllForUGI at the > end of request when PXF impersonation is on. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] incubator-hawq pull request #1374: HAWQ-1621. Call FileSystem.closeAllForUGI...
GitHub user divyabhargov opened a pull request: https://github.com/apache/incubator-hawq/pull/1374 HAWQ-1621. Call FileSystem.closeAllForUGI at the end of request when ⦠â¦PXF impersonation is on Authored-by: Divya Bhargov You can merge this pull request into a Git repository by running: $ git pull https://github.com/divyabhargov/incubator-hawq closeUGI Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1374.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1374 commit eb7ade4e0070571f409446070c2560e5ee504746 Author: Divya Bhargov Date: 2018-05-31T18:20:27Z HAWQ-1621. Call FileSystem.closeAllForUGI at the end of request when PXF impersonation is on Authored-by: Divya Bhargov ---
[jira] [Created] (HAWQ-1621) Call FileSystem.closeAllForUGI at the end of request when PXF impersonation is on
Divya Bhargov created HAWQ-1621: --- Summary: Call FileSystem.closeAllForUGI at the end of request when PXF impersonation is on Key: HAWQ-1621 URL: https://issues.apache.org/jira/browse/HAWQ-1621 Project: Apache HAWQ Issue Type: Improvement Components: PXF Reporter: Divya Bhargov Assignee: Ed Espino Due to open Hadoop JIRA: https://issues.apache.org/jira/browse/HDFS-3545 , when PXF impersonation is on, proxy UGIs created for each request produce FileSystem cache entries that are not removed when the request is over. This causes OutOfMemory errors since the entries are never removed. To remove the entries and clean the cache, we should call FileSystem.closeAllForUGI at the end of request when PXF impersonation is on. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HAWQ-1620) Push Down Target List Information To Parquet Scan For Bloomfilter
[ https://issues.apache.org/jira/browse/HAWQ-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lin Wen resolved HAWQ-1620. --- Resolution: Fixed > Push Down Target List Information To Parquet Scan For Bloomfilter > - > > Key: HAWQ-1620 > URL: https://issues.apache.org/jira/browse/HAWQ-1620 > Project: Apache HAWQ > Issue Type: Bug > Components: Query Execution >Reporter: Lin Wen >Assignee: Lin Wen >Priority: Major > Fix For: 2.4.0.0-incubating > > > In function CreateRuntimeFilterState(), only simple Var information is pushed > down to parquet scan, target list information(pi_targetlist in structure > ProjectionInfo) should be pushed down too. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HAWQ-1620) Push Down Target List Information To Parquet Scan For Bloomfilter
[ https://issues.apache.org/jira/browse/HAWQ-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lin Wen reassigned HAWQ-1620: - Assignee: Lin Wen (was: Lei Chang) > Push Down Target List Information To Parquet Scan For Bloomfilter > - > > Key: HAWQ-1620 > URL: https://issues.apache.org/jira/browse/HAWQ-1620 > Project: Apache HAWQ > Issue Type: Bug > Components: Query Execution >Reporter: Lin Wen >Assignee: Lin Wen >Priority: Major > Fix For: 2.4.0.0-incubating > > > In function CreateRuntimeFilterState(), only simple Var information is pushed > down to parquet scan, target list information(pi_targetlist in structure > ProjectionInfo) should be pushed down too. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] incubator-hawq issue #1371: HAWQ-1620. Push down target list information(pi_...
Github user linwen commented on the issue: https://github.com/apache/incubator-hawq/pull/1371 merged to master. ---
[GitHub] incubator-hawq pull request #1371: HAWQ-1620. Push down target list informat...
Github user linwen closed the pull request at: https://github.com/apache/incubator-hawq/pull/1371 ---
