[GitHub] incubator-hawq pull request #1284: HAWQ-1524 - Fix travis ci build failure c...

2017-08-31 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/incubator-hawq/pull/1284


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #1284: HAWQ-1524 - Fix travis ci build failure caused a...

2017-08-31 Thread radarwave 
Github user radarwave commented on the issue:

https://github.com/apache/incubator-hawq/pull/1284
  
+1
Thanks @outofmem0ry for your activated contribution.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #1267: HAWQ-1504 - Fixed namenode hang during docker co...

2017-08-31 Thread radarwave 
Github user radarwave commented on the issue:

https://github.com/apache/incubator-hawq/pull/1267
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (HAWQ-1522) Provide ability to create a single PXF Bundle containing all PXF artifacts

2017-08-31 Thread Lav Jain (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HAWQ-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lav Jain resolved HAWQ-1522.

   Resolution: Duplicate
Fix Version/s: backlog

Duplicate of HAWQ-1523

> Provide ability to create a single PXF Bundle containing all PXF artifacts
> --
>
> Key: HAWQ-1522
> URL: https://issues.apache.org/jira/browse/HAWQ-1522
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: PXF
>Reporter: Lav Jain
>Assignee: Vineet Goel
> Fix For: backlog
>
>
> make bundle should invoke the gradle task bundle. This task should aggregate 
> all the pxf jars and apache-tomcat into a single pxf tarball so that any 
> external client can consume PXF without having to rely on rpms



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] incubator-hawq issue #1263: HAWQ-1495 Corrected answer file to match insert ...

2017-08-31 Thread outofmem0ry 
Github user outofmem0ry commented on the issue:

https://github.com/apache/incubator-hawq/pull/1263
  
@paul-guo- @radarwave do you need any other inputs needed from my side ? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1284: HAWQ-1524 - Fix travis ci build failure c...

2017-08-31 Thread outofmem0ry 
GitHub user outofmem0ry opened a pull request:

https://github.com/apache/incubator-hawq/pull/1284

HAWQ-1524 - Fix travis ci build failure caused after protobuf upgrade to 3.4



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/outofmem0ry/incubator-hawq feature/HAWQ-1524

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/1284.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1284


commit a4357501040f2e1a25356976d1de3d5fbfef8af1
Author: Shubham Sharma 
Date:   2017-08-31T15:52:21Z

HAWQ-1524 - Fix travis ci build failure after protobuf upgrade to 3.4




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (HAWQ-1520) gpcheckhdfs should skip hdfs trash directory

2017-08-31 Thread Hongxu Ma (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HAWQ-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hongxu Ma updated HAWQ-1520:

Description: 
When enable hdfs trash feature, there is a *Trash* directory under the 
encryption zone.
Example:

{code}
[gpadmin@test1 hawq]$ sudo -u hdfs hdfs crypto -listZones
/hawq/hawq-1503886333  tdekey
[gpadmin@test1 hawq]$ hdfs dfs -ls /hawq/hawq-1503886333/
Found 1 items
drwxrwxrwt   - hdfs hdfs  0 2017-08-27 23:59 
/hawq/hawq-1503886333/.Trash
{code}

But gpcheckhdfs scrpit doesn't consider it(/hawq/hawq-1503886333) as an empty 
directory.
We should fix it to skip this trash directory.

Note:
If you just enable trash without encryption zone, the trash directory is 
*/user//.Trash/*, so it doesn't influence gpcheckhdfs in the 
condition.

  was:
When enable hdfs trash feature, there is a *Trash* directory under the 
encryption zone.
Example:

{code}
[gpadmin@test1 hawq]$ sudo -u hdfs hdfs crypto -listZones
/hawq/hawq-1503886333  tdekey
[gpadmin@test1 hawq]$ hdfs dfs -ls /hawq/hawq-1503886333/
Found 1 items
drwxrwxrwt   - hdfs hdfs  0 2017-08-27 23:59 
/hawq/hawq-1503886333/.Trash
{code}

But gpcheckhdfs scrpit doesn't consider it(/hawq/hawq-1503886333) as an empty 
directory.
We should fix it to skip this trash directory.

Note:



> gpcheckhdfs should skip hdfs trash directory
> 
>
> Key: HAWQ-1520
> URL: https://issues.apache.org/jira/browse/HAWQ-1520
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Command Line Tools
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
> Fix For: 2.3.0.0-incubating
>
>
> When enable hdfs trash feature, there is a *Trash* directory under the 
> encryption zone.
> Example:
> {code}
> [gpadmin@test1 hawq]$ sudo -u hdfs hdfs crypto -listZones
> /hawq/hawq-1503886333  tdekey
> [gpadmin@test1 hawq]$ hdfs dfs -ls /hawq/hawq-1503886333/
> Found 1 items
> drwxrwxrwt   - hdfs hdfs  0 2017-08-27 23:59 
> /hawq/hawq-1503886333/.Trash
> {code}
> But gpcheckhdfs scrpit doesn't consider it(/hawq/hawq-1503886333) as an empty 
> directory.
> We should fix it to skip this trash directory.
> Note:
> If you just enable trash without encryption zone, the trash directory is 
> */user//.Trash/*, so it doesn't influence gpcheckhdfs in the 
> condition.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HAWQ-1520) gpcheckhdfs should skip hdfs trash directory

2017-08-31 Thread Hongxu Ma (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HAWQ-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hongxu Ma updated HAWQ-1520:

Description: 
When enable hdfs trash feature, there is a *Trash* directory under the 
encryption zone.
Example:

{code}
[gpadmin@test1 hawq]$ sudo -u hdfs hdfs crypto -listZones
/hawq/hawq-1503886333  tdekey
[gpadmin@test1 hawq]$ hdfs dfs -ls /hawq/hawq-1503886333/
Found 1 items
drwxrwxrwt   - hdfs hdfs  0 2017-08-27 23:59 
/hawq/hawq-1503886333/.Trash
{code}

But gpcheckhdfs scrpit doesn't consider it(/hawq/hawq-1503886333) as an empty 
directory.
We should fix it to skip this trash directory.

Note:


  was:
When enable hdfs trash feature, there is a *Trash* directory under the 
encryption zone.
Example:

{code}
[gpadmin@test1 hawq]$ sudo -u hdfs hdfs crypto -listZones
/hawq/hawq-1503886333  tdekey
[gpadmin@test1 hawq]$ hdfs dfs -ls /hawq/hawq-1503886333/
Found 1 items
drwxrwxrwt   - hdfs hdfs  0 2017-08-27 23:59 
/hawq/hawq-1503886333/.Trash
{code}

But gpcheckhdfs scrpit doesn't consider it(/hawq/hawq-1503886333) as an empty 
directory.
We should fix it to skip this trash directory.


> gpcheckhdfs should skip hdfs trash directory
> 
>
> Key: HAWQ-1520
> URL: https://issues.apache.org/jira/browse/HAWQ-1520
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Command Line Tools
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
> Fix For: 2.3.0.0-incubating
>
>
> When enable hdfs trash feature, there is a *Trash* directory under the 
> encryption zone.
> Example:
> {code}
> [gpadmin@test1 hawq]$ sudo -u hdfs hdfs crypto -listZones
> /hawq/hawq-1503886333  tdekey
> [gpadmin@test1 hawq]$ hdfs dfs -ls /hawq/hawq-1503886333/
> Found 1 items
> drwxrwxrwt   - hdfs hdfs  0 2017-08-27 23:59 
> /hawq/hawq-1503886333/.Trash
> {code}
> But gpcheckhdfs scrpit doesn't consider it(/hawq/hawq-1503886333) as an empty 
> directory.
> We should fix it to skip this trash directory.
> Note:



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (HAWQ-1521) Idle QE Processes Can't Quit After An Interval

2017-08-31 Thread Hubert Zhang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HAWQ-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148589#comment-16148589
 ] 

Hubert Zhang edited comment on HAWQ-1521 at 8/31/17 7:48 AM:
-

HAWQ remove allocateGang to record gang information in the following variables.
{code}
static List *allocatedReaderGangsN = NIL;
static List *availableReaderGangsN = NIL;
static List *allocatedReaderGangs1 = NIL;
static List *availableReaderGangs1 = NIL;
static Gang *primaryWriterGang = NULL;
{code}
As a result, calling gangsExist will always return false.
But the cached gang information, to be more specific the cached QE information 
could be found in other places at QD side. 
The following struct ExecutorCache(the two pools) contains the all the cached 
QE by a session. We could use this data structure to quit idle QE when session 
is idled for a long time.
{code}
typedef struct ExecutorCache {
boolinit;
MemoryContext   ctx;
struct PoolMgrState *pool;
struct PoolMgrState *entrydb_pool; // pool for entry db connection
int cached_num;
int allocated_num;
int takeover_num;
} ExecutorCache;

static ExecutorCacheexecutor_cache;
{code}


was (Author: hubertzhang):
HAWQ remove allocateGang to record gang information in the following variables.
static List *allocatedReaderGangsN = NIL;
static List *availableReaderGangsN = NIL;
static List *allocatedReaderGangs1 = NIL;
static List *availableReaderGangs1 = NIL;
static Gang *primaryWriterGang = NULL;
As a result, calling gangsExist will always return false.
But the cached gang information, to be more specific the cached QE information 
could be found in other places at QD side. 
The following struct ExecutorCache(the two pools) contains the all the cached 
QE by a session. We could use this data structure to quit idle QE when session 
is idled for a long time.
typedef struct ExecutorCache {
boolinit;
MemoryContext   ctx;
struct PoolMgrState *pool;
struct PoolMgrState *entrydb_pool; // pool for entry db connection
int cached_num;
int allocated_num;
int takeover_num;
} ExecutorCache;

static ExecutorCacheexecutor_cache;

> Idle QE Processes Can't Quit After An Interval
> --
>
> Key: HAWQ-1521
> URL: https://issues.apache.org/jira/browse/HAWQ-1521
> Project: Apache HAWQ
>  Issue Type: Bug
>Reporter: Lin Wen
>Assignee: Radar Lei
>
> After a query is finished, there are some idle QE processes on segments. 
> These QE processes are expected to quit after a time interval, this interval 
> is controlled by a GUC gp_vmem_idle_resource_timeout, the default value is 18 
> seconds.
> However, this does't act as expected. Idle QE processes on segments always 
> exist there, unless the QD process quit. 
> The reason is in postgres.c, the codes to enable this timer can't get 
> executed. function gangsExist() always return false, since gang related 
> structures are all NULL.
>   if (IdleSessionGangTimeout > 0 && gangsExist())
>   if (!enable_sig_alarm( IdleSessionGangTimeout /* ms */, false))
>   elog(FATAL, "could not set timer for client wait 
> timeout");



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HAWQ-1521) Idle QE Processes Can't Quit After An Interval

2017-08-31 Thread Hubert Zhang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HAWQ-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148589#comment-16148589
 ] 

Hubert Zhang commented on HAWQ-1521:


HAWQ remove allocateGang to record gang information in the following variables.
static List *allocatedReaderGangsN = NIL;
static List *availableReaderGangsN = NIL;
static List *allocatedReaderGangs1 = NIL;
static List *availableReaderGangs1 = NIL;
static Gang *primaryWriterGang = NULL;
As a result, calling gangsExist will always return false.
But the cached gang information, to be more specific the cached QE information 
could be found in other places at QD side. 
The following struct ExecutorCache(the two pools) contains the all the cached 
QE by a session. We could use this data structure to quit idle QE when session 
is idled for a long time.
typedef struct ExecutorCache {
boolinit;
MemoryContext   ctx;
struct PoolMgrState *pool;
struct PoolMgrState *entrydb_pool; // pool for entry db connection
int cached_num;
int allocated_num;
int takeover_num;
} ExecutorCache;

static ExecutorCacheexecutor_cache;

> Idle QE Processes Can't Quit After An Interval
> --
>
> Key: HAWQ-1521
> URL: https://issues.apache.org/jira/browse/HAWQ-1521
> Project: Apache HAWQ
>  Issue Type: Bug
>Reporter: Lin Wen
>Assignee: Radar Lei
>
> After a query is finished, there are some idle QE processes on segments. 
> These QE processes are expected to quit after a time interval, this interval 
> is controlled by a GUC gp_vmem_idle_resource_timeout, the default value is 18 
> seconds.
> However, this does't act as expected. Idle QE processes on segments always 
> exist there, unless the QD process quit. 
> The reason is in postgres.c, the codes to enable this timer can't get 
> executed. function gangsExist() always return false, since gang related 
> structures are all NULL.
>   if (IdleSessionGangTimeout > 0 && gangsExist())
>   if (!enable_sig_alarm( IdleSessionGangTimeout /* ms */, false))
>   elog(FATAL, "could not set timer for client wait 
> timeout");



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)