[jira] [Commented] (HAWQ-1473) document ranger plug-in service high availability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-1473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16027146#comment-16027146
 ] 

ASF GitHub Bot commented on HAWQ-1473:
--

Github user ictmalili commented on a diff in the pull request:

https://github.com/apache/incubator-hawq-docs/pull/120#discussion_r118810915
  
--- Diff: markdown/admin/MasterMirroring.html.md.erb ---
@@ -103,6 +103,8 @@ Upon activation of the standby master, HAWQ 
reconstructs the state of the master
```

The newly-activated master's status should be **Active**. If you 
configured a new standby master, its status is **Passive**. When a standby 
master is not configured, the command displays `-No entries found`, the message 
indicating that no standby master instance is configured.
+   
+6. If you have enabled HAWQ Ranger Authentication in high availability 
mode in your cluster, you must manually update the Ranger HAWQ service 
definition to identify the new master node. Refer to [Failover to HAWQ Standby 
Master](../ranger/ranger-ha.html#rps_ha_cfg_masterfailover) in the HAWQ Ranger 
documentation for additional information.
 
 6. Query the `gp_segment_configuration` table to verify that segments have 
registered themselves to the new master:
--- End diff --

The index number for next item should be incremented.


> document ranger plug-in service high availability
> -
>
> Key: HAWQ-1473
> URL: https://issues.apache.org/jira/browse/HAWQ-1473
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Lisa Owen
>Assignee: David Yozie
>
> add RPS high availability information to the docs.  include config info as 
> well as failover scenarios.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HAWQ-1473) document ranger plug-in service high availability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-1473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16027143#comment-16027143
 ] 

ASF GitHub Bot commented on HAWQ-1473:
--

Github user ictmalili commented on a diff in the pull request:

https://github.com/apache/incubator-hawq-docs/pull/120#discussion_r118811043
  
--- Diff: markdown/ranger/ranger-integration-config.html.md.erb ---
@@ -129,6 +129,62 @@ Once the connection between HAWQ and Ranger is 
configured, you may choose to set
 5. Click **Save** to save your changes.
 6. Select **Service Actions > Restart All** and confirm that you want to 
restart the HAWQ cluster.
 
+## Step 3: (Optional) Register a Standby Ranger 
Plug-in Service
+
+The HAWQ Ranger Plug-in Service runs on the HAWQ master node. If this 
service goes down, all HAWQ database operations will fail. Configure a highly 
available HAWQ Ranger Plug-in Service to eliminate possible downtime should 
this situation occur.
+
+### Prerequisites 
+
+Before you configure HAWQ Ranger authentication in high availability mode, 
ensure that you have:
+
+- Installed or upgraded to a version of HAWQ that includes support for 
HAWQ Ranger Authentication.
+
+- Configured Ranger Admin for high availability.
+
+- Configured a HAWQ standby master node for your HAWQ cluster.
+
+You must configure a standby master for your HAWQ deployment before 
enabling HAWQ Ranger high availability mode. If you have not configured your 
HAWQ standby master, follow the instructions in [Adding a HAWQ Standby 
Master](../admin/ambari-admin.html#amb-add-standby) (if you manage your HAWQ 
cluster with Ambari) or [Configuring Master 
Mirroring](../admin/MasterMirroring.html#standby_master_configure) (for a 
command-line-managed HAWQ cluster).
+
+- Registered the HAWQ Ranger Plug-in Service on your HAWQ master node.
+
+The HAWQ Ranger Plug-in Service runs on the HAWQ master node. If you 
have not yet enabled the Ranger Plug-in Service, refer to [Install Ranger 
Connectivity to HAWQ](ranger-integration-config.html#jar) for registration 
instructions. Make sure to identify the Ranger Admin HA proxy when you enable 
the plug-in.
--- End diff --

This is not required. 


> document ranger plug-in service high availability
> -
>
> Key: HAWQ-1473
> URL: https://issues.apache.org/jira/browse/HAWQ-1473
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Lisa Owen
>Assignee: David Yozie
>
> add RPS high availability information to the docs.  include config info as 
> well as failover scenarios.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HAWQ-1473) document ranger plug-in service high availability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-1473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16027144#comment-16027144
 ] 

ASF GitHub Bot commented on HAWQ-1473:
--

Github user ictmalili commented on a diff in the pull request:

https://github.com/apache/incubator-hawq-docs/pull/120#discussion_r118811032
  
--- Diff: markdown/ranger/ranger-integration-config.html.md.erb ---
@@ -129,6 +129,62 @@ Once the connection between HAWQ and Ranger is 
configured, you may choose to set
 5. Click **Save** to save your changes.
 6. Select **Service Actions > Restart All** and confirm that you want to 
restart the HAWQ cluster.
 
+## Step 3: (Optional) Register a Standby Ranger 
Plug-in Service
+
+The HAWQ Ranger Plug-in Service runs on the HAWQ master node. If this 
service goes down, all HAWQ database operations will fail. Configure a highly 
available HAWQ Ranger Plug-in Service to eliminate possible downtime should 
this situation occur.
+
+### Prerequisites 
+
+Before you configure HAWQ Ranger authentication in high availability mode, 
ensure that you have:
+
+- Installed or upgraded to a version of HAWQ that includes support for 
HAWQ Ranger Authentication.
+
+- Configured Ranger Admin for high availability.
--- End diff --

Actually Ranger HA and RPS HA are two separated and independent parts. If 
the user does not configure Ranger HA, he can also configure RPS HA in HAWQ 
standby master.


> document ranger plug-in service high availability
> -
>
> Key: HAWQ-1473
> URL: https://issues.apache.org/jira/browse/HAWQ-1473
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Lisa Owen
>Assignee: David Yozie
>
> add RPS high availability information to the docs.  include config info as 
> well as failover scenarios.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] incubator-hawq pull request #1225: HAWQ-1446: Introduce vectorized profile f...

[sansanichfb]

Github user sansanichfb commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1225#discussion_r118808410
  
--- Diff: 
pxf/pxf-hive/src/test/java/org/apache/hawq/pxf/plugins/hive/utilities/ProfileFactoryTest.java
 ---
@@ -34,31 +34,31 @@
 public void get() throws Exception {
 
 // For TextInputFormat when table has no complex types, HiveText 
profile should be used
-String profileName = ProfileFactory.get(new TextInputFormat(), 
false);
+String profileName = ProfileFactory.get(new TextInputFormat(), 
false, null);
--- End diff --

Sure


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1225: HAWQ-1446: Introduce vectorized profile f...

[denalex]

Github user denalex commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1225#discussion_r118807815
  
--- Diff: 
pxf/pxf-hive/src/test/java/org/apache/hawq/pxf/plugins/hive/utilities/ProfileFactoryTest.java
 ---
@@ -34,31 +34,31 @@
 public void get() throws Exception {
 
 // For TextInputFormat when table has no complex types, HiveText 
profile should be used
-String profileName = ProfileFactory.get(new TextInputFormat(), 
false);
+String profileName = ProfileFactory.get(new TextInputFormat(), 
false, null);
--- End diff --

can revert back these changes now that the function with 2 arguments is 
back, right ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1225: HAWQ-1446: Introduce vectorized profile f...

[denalex]

Github user denalex commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1225#discussion_r118807905
  
--- Diff: 
pxf/pxf-service/src/main/java/org/apache/hawq/pxf/service/BridgeOutputBuilder.java
 ---
@@ -137,6 +137,18 @@ public Writable getErrorOutput(Exception ex) throws 
Exception {
 return outputList;
 }
 
+public LinkedList makeVectorizedOutput(List 
recordsBatch) throws BadRecordException {
+outputList.clear();
+for (List record : recordsBatch) {
--- End diff --

no null checks necessary for recordsBatch and record ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1244: HAWQ-1443. Implement Ranger lookup for HA...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-hawq/pull/1244


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1225: HAWQ-1446: Introduce vectorized profile f...

[sansanichfb]

Github user sansanichfb commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1225#discussion_r118788222
  
--- Diff: pxf/pxf-service/src/main/resources/pxf-profiles-default.xml ---
@@ -101,6 +101,17 @@ under the License.
 
org.apache.hawq.pxf.service.io.GPDBWritable
 
 
+
+HiveVectorizedORC
--- End diff --

Renamed all classes to use "vectorized"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1225: HAWQ-1446: Introduce vectorized profile f...

[sansanichfb]

Github user sansanichfb commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1225#discussion_r118766404
  
--- Diff: 
pxf/pxf-service/src/main/java/org/apache/hawq/pxf/service/ReadVectorizedBridge.java
 ---
@@ -0,0 +1,126 @@
+package org.apache.hawq.pxf.service;
--- End diff --

Makes sense, extended.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (HAWQ-1474) docs - instructions to create/use a minimal psql client pkg

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-1474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026519#comment-16026519
 ] 

ASF GitHub Bot commented on HAWQ-1474:
--

GitHub user lisakowen opened a pull request:

https://github.com/apache/incubator-hawq-docs/pull/121

HAWQ-1474 - document minimal psql client pkg create/install/run

document how to create a minimal psql client "package" including 
instructions on installing and using the package.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lisakowen/incubator-hawq-docs 
feature/hawq-psql-client

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq-docs/pull/121.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #121


commit d8a2c5f4ff1bc3c56a239f90e3fb02dd577c73af
Author: Lisa Owen 
Date:   2017-05-23T22:33:33Z

document minimal psql client pkg create/install/run




> docs - instructions to create/use a minimal psql client pkg
> ---
>
> Key: HAWQ-1474
> URL: https://issues.apache.org/jira/browse/HAWQ-1474
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Lisa Owen
>Assignee: David Yozie
>
> add instructions to create, install, and use a minimal psql client pkg.  one 
> would install this package on a like linux client system outside of the hawq 
> cluster.
> this will basically provide instructions to:
> - package up the psql binary and libraries and an auto-generated 
> environment-setting file
> - install this on a client system
> - run the client



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HAWQ-1474) docs - instructions to create/use a minimal psql client pkg

[Lisa Owen (JIRA)]

Lisa Owen created HAWQ-1474:
---

 Summary: docs - instructions to create/use a minimal psql client 
pkg
 Key: HAWQ-1474
 URL: https://issues.apache.org/jira/browse/HAWQ-1474
 Project: Apache HAWQ
  Issue Type: Improvement
  Components: Documentation
Reporter: Lisa Owen
Assignee: David Yozie


add instructions to create, install, and use a minimal psql client pkg.  one 
would install this package on a like linux client system outside of the hawq 
cluster.

this will basically provide instructions to:
- package up the psql binary and libraries and an auto-generated 
environment-setting file
- install this on a client system
- run the client



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HAWQ-1473) document ranger plug-in service high availability

[Lisa Owen (JIRA)]

Lisa Owen created HAWQ-1473:
---

 Summary: document ranger plug-in service high availability
 Key: HAWQ-1473
 URL: https://issues.apache.org/jira/browse/HAWQ-1473
 Project: Apache HAWQ
  Issue Type: Improvement
  Components: Documentation
Reporter: Lisa Owen
Assignee: David Yozie


add RPS high availability information to the docs.  include config info as well 
as failover scenarios.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] incubator-hawq issue #1244: HAWQ-1443. Implement Ranger lookup for HAWQ with...

[linwen]

Github user linwen commented on the issue:

https://github.com/apache/incubator-hawq/pull/1244
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1244: HAWQ-1443. Implement Ranger lookup for HA...

[zhangh43]

GitHub user zhangh43 opened a pull request:

https://github.com/apache/incubator-hawq/pull/1244

HAWQ-1443. Implement Ranger lookup for HAWQ with Kerberos enabled.

This PR implements the HAWQ client in ranger to lookup HAWQ resource in 
kerberos environment.
With Kerberos authentication, the jdbc URL looks like:  
jdbc:postgresql://host:port/db?kerberosServerName=HawqKerberosServiceName=pgjdbc=LookupPricipal
With Simple authentication,  the jdbc URL just looks like:  
jdbc:postgresql://host:port/db and with user name and password as parameter.

In Kerberos env, we also do kinit with principal name and principal 
password. Keytab file is not supported yet.


Signed-off-by: xiang sheng 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zhangh43/incubator-hawq m2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/1244.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1244


commit 9775424ba11546248ac9010ed6f1cc331545
Author: Hubert Zhang 
Date:   2017-05-26T12:17:53Z

HAWQ-1443. Implement Ranger lookup for HAWQ with Kerberos enabled.

Signed-off-by: xiang sheng 




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1243: HAWQ-1458. Fix share input scan bug for w...

[amyrazz44]

Github user amyrazz44 commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1243#discussion_r118670052
  
--- Diff: src/backend/executor/nodeShareInputScan.c ---
@@ -793,15 +877,72 @@ shareinput_reader_waitready(int share_id, 
PlanGenerator planGen)
}
else if(n==0)
{
-   elog(DEBUG1, "SISC READER (shareid=%d, slice=%d): Wait 
ready time out once",
-   share_id, currentSliceId);
+   file_exists = access(writer_lock_file, F_OK);   
+   if(file_exists != 0)
+   {
+   elog(DEBUG3, "Wait lock file for writer time 
out interval is %d", timeout_interval);
+   if(timeout_interval >= 
share_input_scan_wait_lockfile_timeout || flag == true) //If lock file never 
exists or disappeared, reader will no longer waiting for writer
+   {
+   elog(LOG, "SISC READER (shareid=%d, 
slice=%d): Wait ready time out and break",
+   share_id, currentSliceId);
+   pfree(writer_lock_file);
+   break;
+   }
+   timeout_interval += tval.tv_sec;
+   }
+   else
+   {
+   elog(LOG, "writer lock file of 
shareinput_reader_waitready() is %s", writer_lock_file);
+   flag = true;
+   fd_lock = open(writer_lock_file, O_RDONLY);
+   if(fd_lock < 0)
+   {
+   elog(DEBUG3, "Open writer's lock file 
%s failed!, error number is %d", writer_lock_file, errno);
+   }
+   lock = flock(fd_lock, LOCK_EX | LOCK_NB);
+   if(lock == -1)
+   {
+   elog(DEBUG3, "Lock writer's lock file 
%s failed!, error number is %d", writer_lock_file, errno);
+   }
+   else if(lock == 0)
+   {
+   /*
+* There is one situation to consider 
about.
+* Writer need a time interval to lock 
the lock file after the lock file has been created.
+* So, if reader lock the lock file 
ahead of writer, we should unlock it.
+* If reader lock the lock file after 
writer, it means that writer process has abort.
+* We should break the loop to make 
sure reader no longer wait for writer.
+*/  
+   if(lock_count == 0)  
+   {
+   lock = flock(fd_lock, LOCK_UN); 
+   lock_count++;
--- End diff --

Because writer needs a time interval to lock the lock file. During the time 
interval, if reader lock the lock file ahead of writer , reader should unlock 
the lock file. In the code logic, lock_count==0 indicates the above logic.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1243: HAWQ-1458. Fix share input scan bug for w...

[amyrazz44]

Github user amyrazz44 commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1243#discussion_r118668389
  
--- Diff: src/backend/executor/nodeShareInputScan.c ---
@@ -793,15 +877,72 @@ shareinput_reader_waitready(int share_id, 
PlanGenerator planGen)
}
else if(n==0)
{
-   elog(DEBUG1, "SISC READER (shareid=%d, slice=%d): Wait 
ready time out once",
-   share_id, currentSliceId);
+   file_exists = access(writer_lock_file, F_OK);   
+   if(file_exists != 0)
+   {
+   elog(DEBUG3, "Wait lock file for writer time 
out interval is %d", timeout_interval);
+   if(timeout_interval >= 
share_input_scan_wait_lockfile_timeout || flag == true) //If lock file never 
exists or disappeared, reader will no longer waiting for writer
+   {
+   elog(LOG, "SISC READER (shareid=%d, 
slice=%d): Wait ready time out and break",
+   share_id, currentSliceId);
+   pfree(writer_lock_file);
+   break;
+   }
+   timeout_interval += tval.tv_sec;
+   }
+   else
+   {
+   elog(LOG, "writer lock file of 
shareinput_reader_waitready() is %s", writer_lock_file);
+   flag = true;
+   fd_lock = open(writer_lock_file, O_RDONLY);
+   if(fd_lock < 0)
+   {
+   elog(DEBUG3, "Open writer's lock file 
%s failed!, error number is %d", writer_lock_file, errno);
--- End diff --

Writer needs time to create the lock file.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1243: HAWQ-1458. Fix share input scan bug for w...

[amyrazz44]

Github user amyrazz44 commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1243#discussion_r118668142
  
--- Diff: src/backend/executor/nodeShareInputScan.c ---
@@ -793,15 +877,72 @@ shareinput_reader_waitready(int share_id, 
PlanGenerator planGen)
}
else if(n==0)
{
-   elog(DEBUG1, "SISC READER (shareid=%d, slice=%d): Wait 
ready time out once",
-   share_id, currentSliceId);
+   file_exists = access(writer_lock_file, F_OK);   
+   if(file_exists != 0)
+   {
+   elog(DEBUG3, "Wait lock file for writer time 
out interval is %d", timeout_interval);
+   if(timeout_interval >= 
share_input_scan_wait_lockfile_timeout || flag == true) //If lock file never 
exists or disappeared, reader will no longer waiting for writer
+   {
+   elog(LOG, "SISC READER (shareid=%d, 
slice=%d): Wait ready time out and break",
+   share_id, currentSliceId);
+   pfree(writer_lock_file);
+   break;
+   }
+   timeout_interval += tval.tv_sec;
+   }
+   else
+   {
+   elog(LOG, "writer lock file of 
shareinput_reader_waitready() is %s", writer_lock_file);
+   flag = true;
+   fd_lock = open(writer_lock_file, O_RDONLY);
+   if(fd_lock < 0)
+   {
+   elog(DEBUG3, "Open writer's lock file 
%s failed!, error number is %d", writer_lock_file, errno);
--- End diff --

Reader try to lock the lock file which writer created until locked the lock 
file successfully which means that writer process quits. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---