from:"kavinderd"

[GitHub] incubator-hawq pull request #1224: HAWQ-1440. Support ANALYZE for all Hive E...

2017-04-26 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1224#discussion_r113578883
  
--- Diff: 
pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveDataFragmenter.java
 ---
@@ -466,7 +466,14 @@ private boolean buildSingleFilter(Object filter,
  */
 @Override
 public FragmentsStats getFragmentsStats() throws Exception {
-throw new UnsupportedOperationException(
-"ANALYZE for Hive plugin is not supported");
+Metadata.Item tblDesc = 
HiveUtilities.extractTableFromName(inputData.getDataSource());
+Table tbl = HiveUtilities.getHiveTable(client, tblDesc);
+Metadata metadata = new Metadata(tblDesc);
+HiveUtilities.getSchema(tbl, metadata);
+
+long split_count = 
Long.parseLong(tbl.getParameters().get("numFiles"));
--- End diff --

Hmm, ok I'll use that metadata stat instead


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1224: HAWQ-1440. Support ANALYZE for all Hive E...

2017-04-26 Thread kavinderd

GitHub user kavinderd opened a pull request:

https://github.com/apache/incubator-hawq/pull/1224

HAWQ-1440. Support ANALYZE for all Hive External Tables



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kavinderd/incubator-hawq HAWQ-1440

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/1224.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1224


commit fb67dcbc5e98f50639c3d9f732d0b9cf260f8b1d
Author: Kavinder Dhaliwal <kavind...@gmail.com>
Date:   2017-04-25T22:14:47Z

HAWQ-1440. Support ANALYZE for all Hive External Tables




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1189: HAWQ-1409. Send AGG-TYPE header to PXF

2017-04-21 Thread kavinderd

Github user kavinderd closed the pull request at:

https://github.com/apache/incubator-hawq/pull/1189


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1216: HAWQ-1429. Do not use AggBridge when WHER...

2017-04-12 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1216#discussion_r90746
  
--- Diff: 
pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/utilities/Utilities.java ---
@@ -209,7 +209,8 @@ public static boolean useAggBridge(InputData inputData) 
{
 LOG.error("Unable to load accessor class: " + e.getMessage());
 return false;
 }
-return (inputData != null) && (inputData.getAggType() != null)
+return (inputData != null) && !inputData.hasFilter()
--- End diff --

Maybe add a comment for why so many checks are required


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1189: HAWQ-1409. Send AGG-TYPE header to PXF

2017-03-28 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1189#discussion_r108538619
  
--- Diff: src/backend/executor/nodeAgg.c ---
@@ -1950,6 +1950,21 @@ ExecInitAgg(Agg *node, EState *estate, int eflags)
 * initialize child nodes
 */
outerPlan = outerPlan(node);
+   if (outerPlan->type == T_ExternalScan) {
+   /*
+* Hack to indicate to PXF when there is an external scan
+*/
+   if (aggstate->aggs && list_length(aggstate->aggs) == 1) {
--- End diff --

Oh ok, I thought `list_length()` on a null value would cause an error


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1189: HAWQ-1409. Send AGG-TYPE header to PXF

2017-03-24 Thread kavinderd

GitHub user kavinderd opened a pull request:

https://github.com/apache/incubator-hawq/pull/1189

HAWQ-1409. Send AGG-TYPE header to PXF

This change is mean to be a proof of concept that pushing down
aggregate function information from HAWQ to the underlying external
storage layer does indeed improve performance

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kavinderd/incubator-hawq HAWQ-1409

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/1189.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1189


commit 7f14691d427a4d4669da8c9127d8f1d0800dcdd8
Author: Kavinder Dhaliwal <kavind...@gmail.com>
Date:   2017-03-03T23:27:05Z

HAWQ-1409. Send AGG-TYPE header to PXF

This change is mean to be a proof of concept that pushing down
aggregate function information from HAWQ to the underlying external
storage layer does indeed improve performance




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #1104: HAWQ-1306. Removed links in Javadoc referencing ...

2017-02-01 Thread kavinderd

Github user kavinderd commented on the issue:

https://github.com/apache/incubator-hawq/pull/1104
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #808: HAWQ-944. Implement new pg_ltoa function as per p...

2017-02-01 Thread kavinderd

Github user kavinderd commented on the issue:

https://github.com/apache/incubator-hawq/pull/808
  
@huor @paul-guo- Can you look at this PR when you get a chance. I need one 
more +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1076: HAWQ-1228. Use profile based on file form...

2017-01-27 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1076#discussion_r98277093
  
--- Diff: 
pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/utilities/HiveUtilities.java
 ---
@@ -432,32 +447,144 @@ public static String 
serializePartitionKeys(HiveTablePartition partData) throws
 @SuppressWarnings("unchecked")
 public static byte[] makeUserData(String fragmenterClassName, 
HiveTablePartition partData, boolean filterInFragmenter) throws Exception {
 
-String userData = null;
+HiveUserData hiveUserData = null;
 
 if (fragmenterClassName == null) {
 throw new IllegalArgumentException("No fragmenter provided.");
 }
 
 Class fragmenterClass = Class.forName(fragmenterClassName);
 
+String inputFormatName = partData.storageDesc.getInputFormat();
+String serdeClassName = 
partData.storageDesc.getSerdeInfo().getSerializationLib();
+String propertiesString = serializeProperties(partData.properties);
+String partitionKeys = serializePartitionKeys(partData);
+String delimiter = 
getDelimiterCode(partData.storageDesc).toString();
+
 if 
(HiveInputFormatFragmenter.class.isAssignableFrom(fragmenterClass)) {
-String inputFormatName = partData.storageDesc.getInputFormat();
-String serdeName = 
partData.storageDesc.getSerdeInfo().getSerializationLib();
-String partitionKeys = serializePartitionKeys(partData);
 assertFileType(inputFormatName, partData);
-userData = assertSerde(serdeName, partData) + 
HiveDataFragmenter.HIVE_UD_DELIM
-+ partitionKeys + HiveDataFragmenter.HIVE_UD_DELIM + 
filterInFragmenter;
-} else if 
(HiveDataFragmenter.class.isAssignableFrom(fragmenterClass)){
-String inputFormatName = partData.storageDesc.getInputFormat();
-String serdeName = 
partData.storageDesc.getSerdeInfo().getSerializationLib();
-String propertiesString = 
serializeProperties(partData.properties);
-String partitionKeys = serializePartitionKeys(partData);
-userData = inputFormatName + HiveDataFragmenter.HIVE_UD_DELIM 
+ serdeName
-+ HiveDataFragmenter.HIVE_UD_DELIM + propertiesString 
+ HiveDataFragmenter.HIVE_UD_DELIM
-+ partitionKeys + HiveDataFragmenter.HIVE_UD_DELIM + 
filterInFragmenter;
+}
+
+hiveUserData = new HiveUserData(inputFormatName, serdeClassName, 
propertiesString, partitionKeys, filterInFragmenter, delimiter);
+
+return hiveUserData.toString().getBytes();
+}
+
+public static HiveUserData parseHiveUserData(InputData input, 
PXF_HIVE_SERDES... supportedSerdes) throws UserDataException{
+String userData = new String(input.getFragmentUserData());
+String[] toks = userData.split(HiveUserData.HIVE_UD_DELIM, 
EXPECTED_NUM_OF_TOKS);
+
+if (toks.length != (EXPECTED_NUM_OF_TOKS)) {
+throw new UserDataException("HiveInputFormatFragmenter 
expected "
++ EXPECTED_NUM_OF_TOKS + " tokens, but got " + 
toks.length);
+}
+
+HiveUserData hiveUserData = new HiveUserData(toks[0], toks[1], 
toks[2], toks[3], Boolean.valueOf(toks[4]), toks[5]);
+
+if (supportedSerdes.length > 0) {
--- End diff --

Indentation


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1076: HAWQ-1228. Use profile based on file form...

2017-01-26 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1076#discussion_r98115361
  
--- Diff: src/bin/gpfusion/gpbridgeapi.c ---
@@ -68,7 +68,6 @@ void  free_token_resources(PxfInputData *inputData);
 Datum gpbridge_import(PG_FUNCTION_ARGS)
 {
gpbridge_check_inside_extproto(fcinfo, "gpbridge_import");
-// ExternalSelectDesc desc = EXTPROTOCOL_GET_SELECTDESC(fcinfo);
--- End diff --

remove instead of commenting out


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1076: HAWQ-1228. Use profile based on file form...

2017-01-26 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1076#discussion_r98115518
  
--- Diff: src/backend/catalog/external/externalmd.c ---
@@ -631,3 +630,79 @@ static int ComputeTypeMod(Oid typeOid, const char 
*colname, int *typemod, int nT
return VARHDRSZ + result;
 }
 
+static Datum GetFormatTypeForProfile(const List *outputFormats)
+{
+
+   /* if table is homogeneous and output format is text - use text*/
+   if (list_length(outputFormats) == 1 && 
strcmp(lfirst(list_head(outputFormats)), TextFormatName) == 0)
+   {
+   return CharGetDatum(TextFormatType);
+   } else
+   {
+   return CharGetDatum(CustomFormatType);
+   }
+}
+
+static Datum GetFormatOptionsForProfile(const List *outputFormats, int 
delimiter)
+{
+   StringInfoData formatStr;
+   initStringInfo();
+
+   /* "delimiter 'delimiter' null '\\N' escape '\\'"*/
+   char formatArr[35] = { 0x64, 0x65, 0x6c, 0x69, 0x6d, 0x69, 0x74, 0x65,
+   0x72, 0x20, 0x27, delimiter, 0x27, 0x20, 0x6e, 0x75, 
0x6c, 0x6c,
+   0x20, 0x27, 0x5c, 0x4e, 0x27, 0x20, 0x65, 0x73, 0x63, 
0x61, 0x70,
+   0x65, 0x20, 0x27, 0x5c, 0x27, 0x00 };
+
+   if (list_length(outputFormats) == 1 && 
strcmp(lfirst(list_head(outputFormats)),TextFormatName) == 0)
+   {
+   appendStringInfo(, "%s", formatArr);
+   } else {
+   appendStringInfo(, "formatter 'pxfwritable_import'");
+   }
+   Datum format_opts = DirectFunctionCall1(textin, 
CStringGetDatum(formatStr.data));
+   pfree(formatStr.data);
+   return format_opts;
+}
+
+/* location - should be an array of text with one element:
+ * 
pxf:///.?Profile=profileName=delimiterCode
 */
+static Datum GetLocationForFormat(char *profile, List *outputFormats, char 
*pxf_service_address, char *path, char *name, int delimiter)
--- End diff --

Maybe rename path to `db` and name to `path`, the comment and the actual 
variable names are confusing


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1075: HAWQ-762. Login to kerberos if credential...

2017-01-26 Thread kavinderd

Github user kavinderd closed the pull request at:

https://github.com/apache/incubator-hawq/pull/1075


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1102: HAWQ-1297. Make PXF install ready from so...

2017-01-26 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1102#discussion_r98055597
  
--- Diff: pxf/pxf-service/src/scripts/pxf-service ---
@@ -40,6 +34,29 @@ else
source $env_script
 fi
 
+pxf_user=$PXF_USER
+instance_port=$PXF_PORT
+instance_name=pxf-service
+
+if [ -z $PXF_HOME ]; then
+# RPM based setup
+pxf_root=/usr/lib/pxf
+tomcat_root=/opt/apache-tomcat
+tomcat_templates=/opt/pxf/tomcat-templates
+instance_root=/var/pxf
+instance_owner=$pxf_user:$pxf_user
+else
+# OSS/Source code based setup
+pxf_root=$PXF_HOME/lib
+tomcat_root=$PXF_HOME/apache-tomcat
+tomcat_templates=$PXF_HOME/tomcat-templates
+instance_root=$PXF_HOME
+instance_owner=$pxf_user
+fi
+
+curl=`which curl`
--- End diff --

Maybe check that this command returns a non-null value


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1102: HAWQ-1297. Make PXF install ready from so...

2017-01-26 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1102#discussion_r98056223
  
--- Diff: pxf/pxf-service/src/scripts/pxf-service ---
@@ -193,39 +210,83 @@ function doInit()
return 0
fi
 
-   createInstance || return 1
+createInstance || return 1
configureInstance || return 1
deployWebapp || return 1
 }
 
+#
+# patchWebapp patches the webapp config files
+# patch applied only if PXF_HOME is defined
+#
+function patchWebapp()
+{
+if [ -z $PXF_HOME ]; then
+# webapp doesn't require patch
+return 0
+fi
+pushd $instance_root/$instance_name/webapps || return 1
+rm -rf pxf
+mkdir pxf
+cd pxf
+unzip -q ../pxf.war
+popd
+
+
context_file=$instance_root/$instance_name/webapps/pxf/META-INF/context.xml
+cat $context_file | \
+sed  -e 
"s:classpathFiles=\"[a-zA-Z0-9\/\;.-]*\":classpathFiles=\"$PXF_HOME\/conf\/pxf-private.classpath\":"
 \
+-e 
"s:secondaryClasspathFiles=\"[a-zA-Z0-9\/\;.-]*\":secondaryClasspathFiles=\"$PXF_HOME\/conf\/pxf-public.classpath\":"
 > context.xml.tmp
+mv context.xml.tmp $context_file
+
+web_file=$instance_root/$instance_name/webapps/pxf/WEB-INF/web.xml
+cat $web_file | \
+sed 
"s:.*pxf-log4j.properties<\/param-value>:$PXF_HOME\/conf\/pxf-log4j.properties<\/param-value>:"
 > web.xml.tmp
+mv web.xml.tmp $web_file
+}
+
+function commandWebapp()
+{
+command=$1
+pushd $instance_root
+su $pxf_user -c "$instance_root/$instance_name/bin/catalina.sh 
$command"
+if [ $? -ne 0 ]; then
+return 1
+fi
+popd
+}
+
 # 
-# doStartStop handles start/stop commands
-# commands are executed as the user $pxf_user
+# doStart handles start command
+# command is executed as the user $pxf_user
 #
 # after start, uses checkWebapp to verify the PXF webapp was loaded
 # successfully
 #
-function doStartStop()
+function doStart()
 {
-   command=$1
-
instanceExists
if [ $? -ne 0 ]; then
echo ERROR: cant find PXF instance, maybe call init?
return 1
fi
+   patchWebapp || return 1
+commandWebapp start || return 1
+checkWebapp 300 || return 1
--- End diff --

Indentation of line 273, 274 doesn't match rest of the function


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1102: HAWQ-1297. Make PXF install ready from so...

2017-01-26 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1102#discussion_r98055816
  
--- Diff: pxf/pxf-service/src/scripts/pxf-service ---
@@ -193,39 +210,83 @@ function doInit()
return 0
fi
 
-   createInstance || return 1
+createInstance || return 1
--- End diff --

indentation


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1102: HAWQ-1297. Make PXF install ready from so...

2017-01-26 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1102#discussion_r98058834
  
--- Diff: pxf/pxf-service/src/scripts/pxf-service ---
@@ -193,39 +210,83 @@ function doInit()
return 0
fi
 
-   createInstance || return 1
+createInstance || return 1
configureInstance || return 1
deployWebapp || return 1
 }
 
+#
+# patchWebapp patches the webapp config files
+# patch applied only if PXF_HOME is defined
+#
+function patchWebapp()
+{
+if [ -z $PXF_HOME ]; then
+# webapp doesn't require patch
+return 0
+fi
+pushd $instance_root/$instance_name/webapps || return 1
+rm -rf pxf
+mkdir pxf
+cd pxf
+unzip -q ../pxf.war
+popd
+
+
context_file=$instance_root/$instance_name/webapps/pxf/META-INF/context.xml
+cat $context_file | \
+sed  -e 
"s:classpathFiles=\"[a-zA-Z0-9\/\;.-]*\":classpathFiles=\"$PXF_HOME\/conf\/pxf-private.classpath\":"
 \
+-e 
"s:secondaryClasspathFiles=\"[a-zA-Z0-9\/\;.-]*\":secondaryClasspathFiles=\"$PXF_HOME\/conf\/pxf-public.classpath\":"
 > context.xml.tmp
+mv context.xml.tmp $context_file
+
+web_file=$instance_root/$instance_name/webapps/pxf/WEB-INF/web.xml
+cat $web_file | \
+sed 
"s:.*pxf-log4j.properties<\/param-value>:$PXF_HOME\/conf\/pxf-log4j.properties<\/param-value>:"
 > web.xml.tmp
+mv web.xml.tmp $web_file
+}
+
+function commandWebapp()
+{
+command=$1
+pushd $instance_root
+su $pxf_user -c "$instance_root/$instance_name/bin/catalina.sh 
$command"
+if [ $? -ne 0 ]; then
+return 1
+fi
+popd
+}
+
 # 
-# doStartStop handles start/stop commands
-# commands are executed as the user $pxf_user
+# doStart handles start command
+# command is executed as the user $pxf_user
 #
 # after start, uses checkWebapp to verify the PXF webapp was loaded
 # successfully
 #
-function doStartStop()
+function doStart()
 {
-   command=$1
-
instanceExists
if [ $? -ne 0 ]; then
echo ERROR: cant find PXF instance, maybe call init?
return 1
fi
+   patchWebapp || return 1
+commandWebapp start || return 1
+checkWebapp 300 || return 1
+}
 
-   pushd $instance_root
-   su $pxf_user -c "$instance_root/$instance_name/bin/catalina.sh $command"
+#
+# doStart handles stop command
+# command is executed as the user $pxf_user
+#
+#
+function doStop()
+{
+   instanceExists
if [ $? -ne 0 ]; then
+   echo ERROR: cant find PXF instance, maybe call init?
--- End diff --

quote the string just to be safe


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1064: HAWQ-1215. Support Complextypes with Hive...

2017-01-11 Thread kavinderd

Github user kavinderd closed the pull request at:

https://github.com/apache/incubator-hawq/pull/1064


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1075: HAWQ-762. Login to kerberos if credential...

2017-01-06 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1075#discussion_r95034235
  
--- Diff: 
pxf/pxf-service/src/main/java/org/apache/hawq/pxf/service/utilities/SecuredHDFS.java
 ---
@@ -53,6 +53,10 @@
 public static void verifyToken(ProtocolData protData, ServletContext 
context) {
 try {
 if (UserGroupInformation.isSecurityEnabled()) {
+UserGroupInformation loginUser = 
UserGroupInformation.getLoginUser();
+if (!loginUser.hasKerberosCredentials()) {
+SecureLogin.login();
--- End diff --

Instead of `SecureLogin` we can just directly use 
`UserGroupInformation.getLoginUser().reloginFromKeytab()` seems more 
lightweight since it leverages all the kerberos information that's already in 
memory


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1075: HAWQ-762. Login to kerberos if credential...

2017-01-06 Thread kavinderd

GitHub user kavinderd opened a pull request:

https://github.com/apache/incubator-hawq/pull/1075

HAWQ-762. Login to kerberos if credentials are no longer valid



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kavinderd/incubator-hawq HAWQ-762

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/1075.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1075


commit b6f8a9a0c4c1e9b6f85971f8dfe2e32bed6a6c45
Author: Kavinder Dhaliwal <kavind...@gmail.com>
Date:   2017-01-06T19:56:29Z

HAWQ-762. Login to kerberos if credentials are no longer valid




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1075: HAWQ-762. Login to kerberos if credential...

2017-01-06 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1075#discussion_r95010426
  
--- Diff: 
pxf/pxf-service/src/main/java/org/apache/hawq/pxf/service/utilities/SecuredHDFS.java
 ---
@@ -53,6 +53,10 @@
 public static void verifyToken(ProtocolData protData, ServletContext 
context) {
 try {
 if (UserGroupInformation.isSecurityEnabled()) {
+UserGroupInformation loginUser = 
UserGroupInformation.getLoginUser();
+if (!loginUser.hasKerberosCredentials()) {
--- End diff --

I don't think so if we pass the above check 
`UserGroupInformation.isSecurityEnabled()` but I can add a null check to be safe


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1064: HAWQ-1215. Support Complextypes with Hive...

2017-01-04 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1064#discussion_r94677406
  
--- Diff: 
pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveORCSerdeResolver.java
 ---
@@ -102,11 +104,19 @@ void initSerde(InputData input) throws Exception {
 
 StringBuilder columnNames = new StringBuilder(numberOfDataColumns 
* 2); // column + delimiter
 StringBuilder columnTypes = new StringBuilder(numberOfDataColumns 
* 2); // column + delimiter
+String[] tmp = typesString.split(":");
+String[] hiveColTypes = new String[numberOfDataColumns];
+parseColTypes(tmp, hiveColTypes);
+
 String delim = ",";
 for (int i = 0; i < numberOfDataColumns; i++) {
 ColumnDescriptor column = input.getColumn(i);
 String columnName = column.columnName();
 String columnType = 
HiveUtilities.toCompatibleHiveType(DataType.get(column.columnTypeCode()), 
column.columnTypeModifiers());
+//Complex Types will have a mismatch between Hive and Hawq type
--- End diff --

The only issue I have with that approach is that whether a column is a 
complex type is table specific so HiveUtilities.toCompatibleHiveType would need 
to also take the Hive column information as an additional parameter since Hawq 
type text could either be a Hive string or any Hive complex type.

I think it does make sense to put the below logic into a utility method


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1064: HAWQ-1215. Support Complextypes with Hive...

2016-12-23 Thread kavinderd

GitHub user kavinderd opened a pull request:

https://github.com/apache/incubator-hawq/pull/1064

HAWQ-1215. Support Complextypes with HiveORC

Preliminary PR for complex type support in Hive

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kavinderd/incubator-hawq HAWQ-1215

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/1064.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1064


commit dfa186d7eed43e0437654075b25481f0623e8254
Author: Kavinder Dhaliwal <kavind...@gmail.com>
Date:   2016-12-24T00:27:58Z

HAWQ-1215. Support Complextypes with HiveORC




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1039: HAWQ-1191. Decomissioned DELIMITER from H...

2016-12-07 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1039#discussion_r91353048
  
--- Diff: 
pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveResolver.java 
---
@@ -79,7 +79,7 @@
 private String serdeName;
 private String propsString;
 String partitionKeys;
-char delimiter;
+char delimiter = 1;
--- End diff --

Why do we even need this value anymore?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #1021: HAWQ-1115. Implement filter-push down for IN on ...

2016-11-22 Thread kavinderd

Github user kavinderd commented on the issue:

https://github.com/apache/incubator-hawq/pull/1021
  
Nice work


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1002: HAWQ-1130. Make HCatalog integration work...

2016-11-16 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1002#discussion_r88318669
  
--- Diff: src/backend/access/transam/varsup.c ---
@@ -474,74 +483,128 @@ ResetExternalObjectId(void)
 
 /*
  * master_highest_used_oid
- * Query the database to find the highest used Oid by
+ * Uses CAQL and SPI to find the highest used Oid among 
user and catalog tables
+ *
+ * Uses CAQL to query catalog tables
+ * Uses SPI to query user tables, because CAQL supports 
tables from CatCoreRelation array only
  * 1) Find all the relations that has Oids
  * 2) Find max oid from those relations
  */
 Oid
 master_highest_used_oid(void)
 {
+   Oid oidMaxCatalog = InvalidOid;
+   Oid oidMaxUser = InvalidOid;
Oid oidMax = InvalidOid;
+   Oid currentOid;
+   Form_pg_class classForm;
+   cqContext *pcqOuterCtx;
+   cqContext *pcqInnerCtx;
+   HeapTuple outerTuple;
+   HeapTuple innerTuple;
+   /* number of user tables having oids*/
+   int userTablesNum = 0;
+   int ret;
+
+   pcqOuterCtx = caql_beginscan(NULL, cql("SELECT * FROM pg_class WHERE 
relhasoids = :1", BoolGetDatum(true)));
 
-   if (SPI_OK_CONNECT != SPI_connect())
+   outerTuple = caql_getnext(pcqOuterCtx);
+
+   if (!HeapTupleIsValid(outerTuple))
{
-   ereport(ERROR, (errcode(ERRCODE_CDB_INTERNAL_ERROR),
-   errmsg("Unable to connect to execute internal 
query for HCatalog.")));
+   caql_endscan(pcqOuterCtx);
+   elog(DEBUG1, "Unable to get list of tables having oids");
+   return oidMax;
}
 
-   int ret = SPI_execute("SELECT relname FROM pg_class where 
relhasoids=true", true, 0);
+   /* construct query to get max oid from all tables with oids */
+   StringInfo sqlstrCatalog = makeStringInfo();
+   StringInfo sqlstrUser = makeStringInfo();
+   appendStringInfo(sqlstrUser, "SELECT max(oid) FROM (");
+   while (HeapTupleIsValid(outerTuple))
+   {
+   classForm = (Form_pg_class) GETSTRUCT(outerTuple);
 
-   int rows = SPI_processed;
+   /* use CAQL for accessing catalog tables*/
+   if (classForm->relnamespace == PG_CATALOG_NAMESPACE)
+   {
+   appendStringInfo(sqlstrCatalog,
+   "SELECT oid FROM %s WHERE oid > :1 
ORDER BY oid",
+   classForm->relname.data);
 
-   char *tableNames[rows];
+   pcqInnerCtx = caql_beginscan(NULL,
+   cql1(sqlstrCatalog->data, __FILE__, 
__LINE__,
+   
ObjectIdGetDatum(oidMaxCatalog)));
 
-   if (rows == 0 || ret <= 0 || NULL == SPI_tuptable)
-   {
-   SPI_finish();
-   return oidMax;
-   }
+   innerTuple = caql_getnext(pcqInnerCtx);
 
-   TupleDesc tupdesc = SPI_tuptable->tupdesc;
-   SPITupleTable *tuptable = SPI_tuptable;
+   currentOid = InvalidOid;
 
-   for (int i = 0; i < rows; i++)
-   {
-   HeapTuple tuple = tuptable->vals[i];
-   tableNames[i] = SPI_getvalue(tuple, tupdesc, 1);
-   }
+   while (HeapTupleIsValid(innerTuple))
+   {
+   currentOid = HeapTupleGetOid(innerTuple);
+   innerTuple = caql_getnext(pcqInnerCtx);
--- End diff --

I see, that sucks


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1002: HAWQ-1130. Make HCatalog integration work...

2016-11-16 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1002#discussion_r88310416
  
--- Diff: src/backend/access/transam/varsup.c ---
@@ -474,74 +483,128 @@ ResetExternalObjectId(void)
 
 /*
  * master_highest_used_oid
- * Query the database to find the highest used Oid by
+ * Uses CAQL and SPI to find the highest used Oid among 
user and catalog tables
+ *
+ * Uses CAQL to query catalog tables
+ * Uses SPI to query user tables, because CAQL supports 
tables from CatCoreRelation array only
  * 1) Find all the relations that has Oids
  * 2) Find max oid from those relations
  */
 Oid
 master_highest_used_oid(void)
 {
+   Oid oidMaxCatalog = InvalidOid;
+   Oid oidMaxUser = InvalidOid;
Oid oidMax = InvalidOid;
+   Oid currentOid;
+   Form_pg_class classForm;
+   cqContext *pcqOuterCtx;
+   cqContext *pcqInnerCtx;
+   HeapTuple outerTuple;
+   HeapTuple innerTuple;
+   /* number of user tables having oids*/
+   int userTablesNum = 0;
+   int ret;
+
+   pcqOuterCtx = caql_beginscan(NULL, cql("SELECT * FROM pg_class WHERE 
relhasoids = :1", BoolGetDatum(true)));
 
-   if (SPI_OK_CONNECT != SPI_connect())
+   outerTuple = caql_getnext(pcqOuterCtx);
+
+   if (!HeapTupleIsValid(outerTuple))
{
-   ereport(ERROR, (errcode(ERRCODE_CDB_INTERNAL_ERROR),
-   errmsg("Unable to connect to execute internal 
query for HCatalog.")));
+   caql_endscan(pcqOuterCtx);
+   elog(DEBUG1, "Unable to get list of tables having oids");
+   return oidMax;
}
 
-   int ret = SPI_execute("SELECT relname FROM pg_class where 
relhasoids=true", true, 0);
+   /* construct query to get max oid from all tables with oids */
+   StringInfo sqlstrCatalog = makeStringInfo();
+   StringInfo sqlstrUser = makeStringInfo();
+   appendStringInfo(sqlstrUser, "SELECT max(oid) FROM (");
+   while (HeapTupleIsValid(outerTuple))
+   {
+   classForm = (Form_pg_class) GETSTRUCT(outerTuple);
 
-   int rows = SPI_processed;
+   /* use CAQL for accessing catalog tables*/
+   if (classForm->relnamespace == PG_CATALOG_NAMESPACE)
+   {
+   appendStringInfo(sqlstrCatalog,
+   "SELECT oid FROM %s WHERE oid > :1 
ORDER BY oid",
+   classForm->relname.data);
 
-   char *tableNames[rows];
+   pcqInnerCtx = caql_beginscan(NULL,
+   cql1(sqlstrCatalog->data, __FILE__, 
__LINE__,
+   
ObjectIdGetDatum(oidMaxCatalog)));
 
-   if (rows == 0 || ret <= 0 || NULL == SPI_tuptable)
-   {
-   SPI_finish();
-   return oidMax;
-   }
+   innerTuple = caql_getnext(pcqInnerCtx);
 
-   TupleDesc tupdesc = SPI_tuptable->tupdesc;
-   SPITupleTable *tuptable = SPI_tuptable;
+   currentOid = InvalidOid;
 
-   for (int i = 0; i < rows; i++)
-   {
-   HeapTuple tuple = tuptable->vals[i];
-   tableNames[i] = SPI_getvalue(tuple, tupdesc, 1);
-   }
+   while (HeapTupleIsValid(innerTuple))
+   {
+   currentOid = HeapTupleGetOid(innerTuple);
+   innerTuple = caql_getnext(pcqInnerCtx);
--- End diff --

If the caql query orderd by oid value, why do you need to call caql_getnext 
repeatedly?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #981: HAWQ-1114. Implement filter-push down for ...

2016-11-15 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/981#discussion_r88100166
  
--- Diff: src/backend/access/external/pxffilters.c ---
@@ -496,23 +629,116 @@ opexpr_to_pxffilter(OpExpr *expr, PxfFilterDesc 
*filter)
return false;
}
 
-   /* is operator supported? if so, set the corresponding PXFOP */
-   for (i = 0; i < nargs; i++)
+   return true;
+}
+
+static bool
+scalar_array_op_expr_to_pxffilter(ScalarArrayOpExpr *expr, PxfFilterDesc 
*filter)
+{
+
+   Node*leftop = NULL;
+   Node*rightop= NULL;
+
+   leftop = (Node *) linitial(expr->args);
+   rightop = (Node *) lsecond(expr->args);
+   Oid  leftop_type = exprType(leftop);
+   Oid  rightop_type = exprType(rightop);
+
+   /*
+* check if supported type -
+*/
+   if (!supported_filter_type(rightop_type) || 
!supported_filter_type(leftop_type))
+   return false;
+
+   /*
+* check if supported operator -
+*/
+   if(!supported_operator_type(expr->opno, filter))
+   return false;
+
+   if (IsA(leftop, Var) && IsA(rightop, Const))
{
-   /* NOTE: switch to hash table lookup if   */
-   /* array grows. for now it's cheap enough */
-   if(expr->opno == pxf_supported_opr[i].dbop)
-   {
-   filter->op = pxf_supported_opr[i].pxfop;
-   return true; /* filter qualifies! */
-   }
+   filter->l.opcode = PXF_ATTR_CODE;
+   filter->l.attnum = ((Var *) leftop)->varattno;
+   filter->l.consttype = InvalidOid;
+   if (filter->l.attnum <= InvalidAttrNumber)
+   return false; /* system attr not supported */
+
+   filter->r.opcode = PXF_LIST_CONST_CODE;
+   filter->r.attnum = InvalidAttrNumber;
+   filter->r.conststr = makeStringInfo();
+   list_const_to_str((Const *)rightop, filter->r.conststr);
+   filter->r.consttype = ((Const *)rightop)->consttype;
+   }
+   else if (IsA(leftop, Const) && IsA(rightop, Var))
+   {
+   filter->l.opcode = PXF_LIST_CONST_CODE;
+   filter->l.attnum = InvalidAttrNumber;
+   filter->l.conststr = makeStringInfo();
+   list_const_to_str((Const *)leftop, filter->l.conststr);
+   filter->l.consttype = ((Const *)leftop)->consttype;
+
+   filter->r.opcode = PXF_ATTR_CODE;
+   filter->r.attnum = ((Var *) rightop)->varattno;
+   filter->r.consttype = InvalidOid;
+   if (filter->r.attnum <= InvalidAttrNumber)
+   return false; /* system attr not supported */
+   }
+   else
+   {
+   elog(DEBUG1, "pxf_serialize_filter_list: expression is not a 
Var+Const");
+   return false;
}
 
-   elog(DEBUG1, "opexpr_to_pxffilter: operator is not supported, operator 
code: %d", expr->opno);
 
-   /* NOTE: if more validation needed, add it before the operators test
-* or alternatively change it to use a false flag and return true below 
*/
-   return false;
+
+   return true;
+}
+
+static bool
+var_to_pxffilter(Var *var, PxfFilterDesc *filter)
+{
+   Oid var_type = InvalidOid;
+
+   if ((!var) || (!filter))
+   return false;
+
+   var_type = exprType(var);
+
+   /*
+* check if supported type -
+*/
+   if (!supported_filter_type(var_type))
+   return false;
+
+   /*
+* check if supported operator -
+*/
+   if (!supported_operator_type(BooleanEqualOperator, filter))
+   return false;
+
+   /* arguments must be VAR and CONST */
+   if (IsA(var,  Var))
+   {
+   filter->l.opcode = PXF_ATTR_CODE;
+   filter->l.attnum = var->varattno;
+   filter->l.consttype = InvalidOid;
--- End diff --

Why is this `InvalidOid` is that just a default value?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #969: HAWQ-1100. Support for new filter string

2016-10-24 Thread kavinderd

Github user kavinderd closed the pull request at:

https://github.com/apache/incubator-hawq/pull/969


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #969: HAWQ-1100. Support for new filter string

2016-10-24 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/969#discussion_r84760743
  
--- Diff: 
pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveFilterBuilder.java
 ---
@@ -61,8 +61,11 @@ public HiveFilterBuilder(InputData input) {
  * filter or list of basic filters
  */
 public Object getFilterObject(String filterString) throws Exception {
+if (filterString == null)
+return null;
+
 FilterParser parser = new FilterParser(this);
-Object result = parser.parse(filterString);
+Object result = parser.parse(filterString.getBytes());
--- End diff --

ok, makes sense


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #969: HAWQ-1100. Support for new filter string

2016-10-24 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/969#discussion_r84744935
  
--- Diff: 
pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/FilterParser.java ---
@@ -279,25 +296,110 @@ int safeToInt(Long value) throws 
FilterStringSyntaxException {
 return value.intValue();
 }
 
+private int parseConstDataType() throws Exception {
+if (!Character.isDigit((char) filterByteArr[index])) {
+throw new FilterStringSyntaxException("datatype OID should 
follow at " + index);
+}
+
+String digits = parseDigits();
+
+try {
+return Integer.parseInt(digits);
+} catch (NumberFormatException e) {
+throw new FilterStringSyntaxException("invalid numeric 
argument at " + digits);
+}
+}
+
+private int parseDataLength() throws Exception {
+if (((char) filterByteArr[index]) != CONST_LEN) {
+throw new FilterStringSyntaxException("data length delimiter 
's' expected at " +  index);
+}
+
+index++;
+return parseInt();
+}
+
+private int parseInt() throws Exception {
+if (index == filterByteArr.length) {
+throw new FilterStringSyntaxException("numeric argument 
expected at " + index);
+}
+
+String digits = parseDigits();
+
+try {
+return Integer.parseInt(digits);
+} catch (NumberFormatException e) {
+throw new FilterStringSyntaxException("invalid numeric 
argument " + digits);
+}
+}
+
+private Object convertDataType(byte[] byteData, int start, int end, 
DataType dataType) throws Exception {
+String data = new String(byteData, start, end-start);
--- End diff --

Sure, I was under the assumption that the default charset is UTF-8 but it 
makes sense to be explicit


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #969: HAWQ-1100. Support for new filter string

2016-10-21 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/969#discussion_r84528951
  
--- Diff: 
pxf/pxf-hbase/src/main/java/org/apache/hawq/pxf/plugins/hbase/HBaseFilterBuilder.java
 ---
@@ -229,8 +233,20 @@ private ByteArrayComparable getComparator(int type, 
Object data) throws Exceptio
 break;
 case SMALLINT:
 case INTEGER:
+result = new HBaseIntegerComparator(((Integer) 
data).longValue());
+break;
 case BIGINT:
-result = new HBaseIntegerComparator((Long) data);
+if (data instanceof Long) {
+result = new HBaseIntegerComparator((Long) data);
+} else if (data instanceof Integer) {
+result = new HBaseIntegerComparator(((Integer) 
data).longValue());
+} else {
+result = null;
+}
+break;
+case FLOAT8:
--- End diff --

Good point


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #968: HAWQ 1116

2016-10-20 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/968#discussion_r84334739
  
--- Diff: src/backend/access/external/pxffilters.c ---
@@ -406,6 +406,22 @@ pxf_serialize_filter_list(List *expressionItems)
appendStringInfo(resbuf, "%c%d", 
PXF_LOGICAL_OPERATOR_CODE, boolType);
break;
}
+   case T_NullTest:
+   {
+   elog(DEBUG1, "pxf_serialize_filter_list: node 
tag %d (T_NullTest)", tag);
+   NullTest *expr = (NullTest *) node;
+
+   /* filter expression for T_NullTest will not 
have any constant value */
+   if (expr->nulltesttype == IS_NULL)
--- End diff --

Maybe put this into a conditional statement instead of an if/else block. 
But it's fine as is too

```
op = expr->nulltestype == IS_NULL ? PXFOP_IS_NULL :  PXFOP_IS_NOTNULL;
appendStringInfo(resbuf, "%c%d%c%d", PXF_ATTR_CODE, ((Var *) 
expr->arg)->varattno - 1, PXF_OPERATOR_CODE,  op);
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #968: HAWQ 1116

2016-10-20 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/968#discussion_r84333580
  
--- Diff: src/backend/access/external/test/pxffilters_test.c ---
@@ -507,6 +542,23 @@ void test__pxf_serialize_filter_list__oneFilter(void 
**state) {
 
 }
 
+void test__pxf_serialize_fillter_list__nullFilter(void **state) {
+
--- End diff --

brace on newline


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #968: HAWQ 1116

2016-10-20 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/968#discussion_r8414
  
--- Diff: src/backend/access/external/test/pxffilters_test.c ---
@@ -323,6 +323,30 @@ OpExpr* build_op_expr(void* left, void* right, int op)
return expr;
 }
 
+NullTest* build_null_expr(void* arg, NullTestType nullType)
+{
+   NullTest *expr = (NullTest*) palloc0(sizeof(NullTest));
+   expr->nulltesttype = nullType;
+   expr->arg = arg;
+
+   expr->xpr.type = T_NullTest;
+   return expr;
+}
+
+ExpressionItem* build_null_expression_item(int attnum, Oid attrtype, 
NullTestType nullType) {
+
--- End diff --

style: the '{' should be on a new line


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #969: HAWQ-1100. Support for new filter string

2016-10-20 Thread kavinderd

GitHub user kavinderd opened a pull request:

https://github.com/apache/incubator-hawq/pull/969

HAWQ-1100. Support for new filter string

- constant data type is now of the form "csd"

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kavinderd/incubator-hawq HAWQ-1100

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/969.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #969


commit ff189586841f27813f2b94f06d708dcbe12197b4
Author: Kavinder Dhaliwal <kavind...@gmail.com>
Date:   2016-10-13T22:53:39Z

HAWQ-1100. Support for new filter string

- constant data type is now of the form "csd"




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #958: HAWQ-1048. Support OR, NOT logical operato...

2016-10-12 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/958#discussion_r83111789
  
--- Diff: src/backend/access/external/pxffilters.c ---
@@ -31,14 +31,14 @@
 #include "utils/guc.h"
 #include "utils/lsyscache.h"
 
-static List* pxf_make_filter_list(List* quals);
+static List* pxf_make_expression_items_list(List *quals, Node *parent, 
bool *logicalOpsNum);
 static void pxf_free_filter(PxfFilterDesc* filter);
-static void pxf_free_filter_list(List *filters);
 static char* pxf_serialize_filter_list(List *filters);
 static bool opexpr_to_pxffilter(OpExpr *expr, PxfFilterDesc *filter);
 static bool supported_filter_type(Oid type);
 static void const_to_str(Const *constval, StringInfo buf);
 static List* append_attr_from_var(Var* var, List* attrs);
+static void enrichTrivialExpression(List *expressionItems);
--- End diff --

change function signature to match style with underscores 
`enrich_trivial_expression`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #959: HAWQ-1084. Fixed memory allocation for table prin...

2016-10-12 Thread kavinderd

Github user kavinderd commented on the issue:

https://github.com/apache/incubator-hawq/pull/959
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #959: HAWQ-1084. Fixed memory allocation for tab...

2016-10-12 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/959#discussion_r83110560
  
--- Diff: src/bin/psql/describe.c ---
@@ -4316,10 +4316,10 @@ describePxfTable(const char *profile, const char 
*pattern, bool verbose)
if (verbose)
{
sourcefieldtype = PQgetvalue(res, i, 4);
-   total_fields = PQgetvalue(res, i, 5);
+   total_fields = atoi(PQgetvalue(res, i, 5));
--- End diff --

Ah, good catch


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #959: HAWQ-1084. Fixed memory allocation for tab...

2016-10-12 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/959#discussion_r83109712
  
--- Diff: src/bin/psql/describe.c ---
@@ -4316,10 +4316,10 @@ describePxfTable(const char *profile, const char 
*pattern, bool verbose)
if (verbose)
{
sourcefieldtype = PQgetvalue(res, i, 4);
-   total_fields = PQgetvalue(res, i, 5);
+   total_fields = atoi(PQgetvalue(res, i, 5));
--- End diff --

Why was an OOM being thrown without `atoi`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #958: HAWQ-1048. Support OR, NOT logical operato...

2016-10-12 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/958#discussion_r83108994
  
--- Diff: src/backend/access/external/pxffilters.c ---
@@ -135,7 +135,40 @@ dbop_pxfop_map pxf_supported_opr[] =
{1097 /* date_gt */, PXFOP_GT},
{1096 /* date_le */, PXFOP_LE},
{1098 /* date_ge */, PXFOP_GE},
-   {1094 /* date_ne */, PXFOP_NE}
+   {1094 /* date_ne */, PXFOP_NE},
+
+   /* float8 */
+   {Float8EqualOperator  /* float8eq */, PXFOP_EQ},
+   {672 /* float8lt */, PXFOP_LT},
+   {674 /* float8gt */, PXFOP_GT},
+   {673 /* float8le */, PXFOP_LE},
+   {675 /* float8ge */, PXFOP_GE},
+   {671 /* float8ne */, PXFOP_NE},
+
+   /* float48 */
+   {1120 /* float48eq */, PXFOP_EQ},
+   {1122 /* float48lt */, PXFOP_LT},
+   {1123 /* float48gt */, PXFOP_GT},
+   {1124 /* float48le */, PXFOP_LE},
+   {1125 /* float48ge */, PXFOP_GE},
+   {1121 /* float48ne */, PXFOP_NE},
+
+   /* bpchar */
+   {BPCharEqualOperator  /* bpchareq */, PXFOP_EQ},
+   {1058  /* bpcharlt */, PXFOP_LT},
+   {1060 /* bpchargt */, PXFOP_GT},
+   {1059 /* bpcharle */, PXFOP_LE},
+   {1061 /* bpcharge */, PXFOP_GE},
+   {1057 /* bpcharne */, PXFOP_NE}
+
+   /* bytea */
+   // TODO: uncomment ocne HAWQ-1085 is done
--- End diff --

typo 'once'


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #906: HAWQ-964. Support for OR and NOT Logical O...

2016-10-12 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/906#discussion_r83103736
  
--- Diff: 
pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveDataFragmenter.java
 ---
@@ -404,24 +411,41 @@ private String buildFilterStringForHive() throws 
Exception {
 HiveFilterBuilder eval = new HiveFilterBuilder(inputData);
 Object filter = eval.getFilterObject(filterInput);
 
-String prefix = "";
-
-if (filter instanceof List) {
-
-for (Object f : (List) filter) {
-if (buildSingleFilter(f, filtersString, prefix)) {
-// Set 'and' operator between each matched partition 
filter.
-prefix = " and ";
-}
-}
-
+if (filter instanceof LogicalFilter) {
+buildCompoundFilter((LogicalFilter) filter, filtersString);
 } else {
-buildSingleFilter(filter, filtersString, prefix);
+buildSingleFilter(filter, filtersString, "");
 }
 
 return filtersString.toString();
 }
 
+private void buildCompoundFilter(LogicalFilter filter, StringBuilder 
filterString) throws Exception{
+String prefix;
+switch(filter.getOperator()) {
--- End diff --

Since LogicalOperation is generic, do you think we should put a very Hive 
specific string in that enum?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #906: HAWQ-964. Support for OR and NOT Logical O...

2016-10-12 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/906#discussion_r83100399
  
--- Diff: 
pxf/pxf-hbase/src/main/java/org/apache/hawq/pxf/plugins/hbase/HBaseFilterBuilder.java
 ---
@@ -52,12 +52,15 @@
  */
 public class HBaseFilterBuilder implements FilterParser.FilterBuilder {
 private Map<FilterParser.Operation, CompareFilter.CompareOp> 
operatorsMap;
+private Map<FilterParser.LogicalOperation, FilterList.Operator> 
logicalOperatorsMap;
 private byte[] startKey;
 private byte[] endKey;
 private HBaseTupleDescription tupleDescription;
+private static final String NOT_OP = "l2";
--- End diff --

Because HBase doesn't support NOT pushdown so we ignore any filters with 
NOT only in HBase


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #906: HAWQ-964. Support for OR and NOT Logical O...

2016-10-12 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/906#discussion_r83098043
  
--- Diff: 
pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/FilterParser.java ---
@@ -396,4 +387,12 @@ private Operation reverseOp(Operation operation) {
 operatorTranslationMap.put(8, Operation.HDOP_LIKE);
 return operatorTranslationMap;
 }
+
+static private Map<Integer, LogicalOperation> 
initLogicalOperatorTransMap() {
+Map<Integer, LogicalOperation> integerLogicalOperationMap = new 
HashMap<>();
--- End diff --

@denalex Thanks. I had stumbled on the same thing


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #906: HAWQ-964. Support for OR and NOT Logical O...

2016-10-12 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/906#discussion_r83097937
  
--- Diff: 
pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/FilterParser.java ---
@@ -396,4 +387,12 @@ private Operation reverseOp(Operation operation) {
 operatorTranslationMap.put(8, Operation.HDOP_LIKE);
 return operatorTranslationMap;
 }
+
+static private Map<Integer, LogicalOperation> 
initLogicalOperatorTransMap() {
+Map<Integer, LogicalOperation> integerLogicalOperationMap = new 
HashMap<>();
--- End diff --

Nvm. Enum.values() returns an array of values


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #906: HAWQ-964. Support for OR and NOT Logical O...

2016-10-12 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/906#discussion_r83097391
  
--- Diff: 
pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/FilterParser.java ---
@@ -396,4 +387,12 @@ private Operation reverseOp(Operation operation) {
 operatorTranslationMap.put(8, Operation.HDOP_LIKE);
 return operatorTranslationMap;
 }
+
+static private Map<Integer, LogicalOperation> 
initLogicalOperatorTransMap() {
+Map<Integer, LogicalOperation> integerLogicalOperationMap = new 
HashMap<>();
--- End diff --

I think we'll need to go with an array. I haven't found a way to do a 
reverse lookup of a Enum based on `int` values. Enum.valueOf() takes a String.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #906: HAWQ-964. Support for OR and NOT Logical O...

2016-10-05 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/906#discussion_r82033272
  
--- Diff: 
pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/FilterParser.java ---
@@ -396,4 +387,12 @@ private Operation reverseOp(Operation operation) {
 operatorTranslationMap.put(8, Operation.HDOP_LIKE);
 return operatorTranslationMap;
 }
+
+static private Map<Integer, LogicalOperation> 
initLogicalOperatorTransMap() {
+Map<Integer, LogicalOperation> integerLogicalOperationMap = new 
HashMap<>();
--- End diff --

That's true. I like that idea. @shivzone any objections to @denalex 
suggestion?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #906: HAWQ-964. Support for OR and NOT Logical O...

2016-10-05 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/906#discussion_r82033359
  
--- Diff: 
pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/FilterParser.java ---
@@ -220,6 +193,24 @@ public Object parse(String filter) throws Exception {
 // Store result on stack
 operandsStack.push(result);
 break;
+case 'l':
+LogicalOperation logicalOperation = 
logicalOperationTranslationMap.get(safeToInt(parseNumber()));
+
+if (logicalOperation == null) {
+throw new FilterStringSyntaxException("unknown op 
ending at " + index);
+}
+
+if (logicalOperation == LogicalOperation.HDOP_NOT) {
+Object exp = operandsStack.pop();
+result = filterBuilder.build(logicalOperation, 
exp);
+} else {
--- End diff --

Ok, I'll add the check


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #906: HAWQ-964. Support for OR and NOT Logical O...

2016-10-05 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/906#discussion_r82032792
  
--- Diff: 
pxf/pxf-hbase/src/main/java/org/apache/hawq/pxf/plugins/hbase/HBaseFilterBuilder.java
 ---
@@ -71,14 +74,19 @@ public HBaseFilterBuilder(HBaseTupleDescription 
tupleDescription) {
  * @throws Exception if parsing failed
  */
 public Filter getFilterObject(String filterString) throws Exception {
-FilterParser parser = new FilterParser(this);
-Object result = parser.parse(filterString);
+// First check for NOT, HBase does not support this
+if (filterString.contains(NOT_OP)) {
--- End diff --

Hmm, that's possible. I'll test and fix if needed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #906: HAWQ-964. Support for OR and NOT Logical O...

2016-10-05 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/906#discussion_r82032692
  
--- Diff: 
pxf/pxf-hbase/src/main/java/org/apache/hawq/pxf/plugins/hbase/HBaseFilterBuilder.java
 ---
@@ -71,14 +74,19 @@ public HBaseFilterBuilder(HBaseTupleDescription 
tupleDescription) {
  * @throws Exception if parsing failed
  */
 public Filter getFilterObject(String filterString) throws Exception {
-FilterParser parser = new FilterParser(this);
-Object result = parser.parse(filterString);
+// First check for NOT, HBase does not support this
+if (filterString.contains(NOT_OP)) {
+return null;
+} else {
--- End diff --

Ok, sounds good


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #949: HAWQ-1080. Fixed unit-tests on local environment.

2016-10-04 Thread kavinderd

Github user kavinderd commented on the issue:

https://github.com/apache/incubator-hawq/pull/949
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #949: HAWQ-1080. Fixed unit-tests on local environment.

2016-10-04 Thread kavinderd

Github user kavinderd commented on the issue:

https://github.com/apache/incubator-hawq/pull/949
  
The issue was a bunch of missing function prototypes?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #951: HAWQ-1083

2016-10-04 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/951#discussion_r81869294
  
--- Diff: src/backend/access/external/libchurl.c ---
@@ -312,16 +313,29 @@ CHURL_HANDLE churl_init(const char* url, 
CHURL_HEADERS headers)
 
/* needed to resolve localhost */
if (strstr(url, LocalhostIpV4) != NULL) {
-   struct curl_slist *resolve_hosts = NULL;
-   char *pxf_host_entry = (char *) 
palloc0(strlen(pxf_service_address) + strlen(LocalhostIpV4Entry) + 1);
-   strcat(pxf_host_entry, pxf_service_address);
-   strcat(pxf_host_entry, LocalhostIpV4Entry);
-   resolve_hosts = curl_slist_append(NULL, pxf_host_entry);
-   set_curl_option(context, CURLOPT_RESOLVE, resolve_hosts);
-   pfree(pxf_host_entry);
-   }
+   //get loopback interface ip address
+   char* loopback_addr = get_loopback_ip_addr();
+   elog(DEBUG1, "Loopback interface IP address: %s", 
loopback_addr);
+   //find host start position in url
+   char* start = strstr(url, LocalhostIpV4);
+   //allocate new url with replaced host
+   char* replaced_url = palloc(strlen(url) + strlen(loopback_addr) 
- strlen(LocalhostIpV4) + 1);
+   //token before host
+   char* before_host = pnstrdup(url, start - url);
+   //token after host
+   char* after_host = pstrdup(url + (start - url) + 
strlen(LocalhostIpV4));
+   //construct replaced url using loopback interface ip
+   sprintf(replaced_url, "%s%s%s", before_host, loopback_addr, 
after_host);
--- End diff --

Sounds good


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #951: HAWQ-1083

2016-10-04 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/951#discussion_r81863084
  
--- Diff: src/backend/access/external/pxfutils.c ---
@@ -133,3 +137,60 @@ static void process_request(ClientContext* 
client_context, char *uri)
 
 
 }
+
+/*
+ * Finds ip address of any available loopback interface(ipv4/ipv6).
+ * Returns ip for ipv4 addresses and [ip] for ipv6 addresses.
+ *
+ */
+char* get_loopback_ip_addr() {
+   struct ifaddrs *ifaddr, *ifa;
+   int family, s, n;
+   char host[NI_MAXHOST];
+   char *loopback_addr;
+
+   if (getifaddrs() == -1) {
+   elog(ERROR, "Unable to obtain list of network interfaces.");
+   }
+
+   for (ifa = ifaddr, n = 0; ifa != NULL; ifa = ifa->ifa_next, n++) {
+   if (ifa->ifa_addr == NULL)
+   continue;
+
+   family = ifa->ifa_addr->sa_family;
+
+   if (family == AF_INET || family == AF_INET6) {
+   s = getnameinfo(ifa->ifa_addr,
+   (family == AF_INET) ?
+   sizeof(struct 
sockaddr_in) :
+   sizeof(struct 
sockaddr_in6), host, NI_MAXHOST,
+   NULL, 0, NI_NUMERICHOST);
+   if (s != 0) {
+   elog(WARNING, "Unable to get name information 
for interface, getnameinfo() failed: %s\n", gai_strerror(s));
+   }
+
+   //get loopback interface
+   if (ifa->ifa_flags & IFF_LOOPBACK) {
+   if (family == AF_INET)
+   {
+   loopback_addr = palloc(strlen(host) + 
1);
+   sprintf(loopback_addr, "%s", host);
--- End diff --

Ok, makes sense. I overlooked the previous line


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #906: HAWQ-964. Support for OR and NOT Logical O...

2016-10-04 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/906#discussion_r81858427
  
--- Diff: 
pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/FilterParser.java ---
@@ -81,6 +88,8 @@
  * @throws Exception if building the filter failed
  */
 public Object build(Operation operation, Object left, Object 
right) throws Exception;
+public Object build(LogicalOperation operation, Object left, 
Object right) throws Exception;
+public Object build(LogicalOperation operation, Object filter) 
throws Exception;
--- End diff --

The reason for having a separate LogicalOperator enum is that fundamentally 
a Logical Operator and a Comparison Operator are different. The grammar for a 
Logical Operators is the following

Logical Operation -> Expr1 (Logical Operator) Expr2
OR
Logical Operation -> (Logical Operator) Expr1  [ This is the NOT case]

Here "Expr" will stand for either a Logical Operation or a Conditional One

A Conditional Operator will have the following form: (Note: they can be in 
reverse form)

Conditional Operation -> Column (Conditional Operator) Constant

So when it comes to parsing a Filter, a Logical Operator will always have 
1/2 children that are themselves an Expression. Whereas a Conditional Operation 
will always have 2 (primitive or constant) children and those children will 
always be leaf nodes.

Thus the reason for having separate op codes is to have a clear way of 
distinguishing Logical Operations from Conditional Ones and internally the 
changes of this Pull Request introduce two Java Objects `Basic Filter` and 
`Logical Filter`. A `BasicFilter` can only have a `column` and `constant` as 
fields but a `LogicalFilter` has a list of subexpressions (`filterList`) as a 
field.

All of the above doesn't explain why we couldn't have a unified enum, but I 
hope it better explains why we felt that the two cases are different enough to 
warrant a separate code. Additionally, when we build the filter string in HAWQ 
itself HAWQ also has two separate Expr types `OpExpr` and `BoolExpr` so our 
approach follows this convention.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #915: HAWQ-1052. Fixed extraction logic for bool...

2016-09-20 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/915#discussion_r79740844
  
--- Diff: src/backend/access/external/pxfheaders.c ---
@@ -66,8 +66,13 @@ void build_http_header(PxfInputData *input)
if (proj_info != NULL && proj_info->pi_isVarList)
{
List* qualsAttributes = extractPxfAttributes(input->quals);
-
-   add_projection_desc_httpheader(headers, proj_info, 
qualsAttributes);
+   /* projection information is incomplete if columns from WHERE 
clause wasn't extracted */
+   if (qualsAttributes !=  NIL || list_length(input->quals) == 0)
--- End diff --

Ok, got it


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #915: HAWQ-1052. Fixed extraction logic for boolean dat...

2016-09-20 Thread kavinderd

Github user kavinderd commented on the issue:

https://github.com/apache/incubator-hawq/pull/915
  
Aside from my confusion with my comment, +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #915: HAWQ-1052. Fixed extraction logic for bool...

2016-09-20 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/915#discussion_r79740167
  
--- Diff: src/backend/access/external/pxfheaders.c ---
@@ -66,8 +66,13 @@ void build_http_header(PxfInputData *input)
if (proj_info != NULL && proj_info->pi_isVarList)
{
List* qualsAttributes = extractPxfAttributes(input->quals);
-
-   add_projection_desc_httpheader(headers, proj_info, 
qualsAttributes);
+   /* projection information is incomplete if columns from WHERE 
clause wasn't extracted */
+   if (qualsAttributes !=  NIL || list_length(input->quals) == 0)
--- End diff --

Why is the check for `list_length(input->quals)` with `0`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #906: HAWQ-964. Support for OR and NOT Logical O...

2016-09-16 Thread kavinderd

GitHub user kavinderd opened a pull request:

https://github.com/apache/incubator-hawq/pull/906

HAWQ-964. Support for OR and NOT Logical Operators

Signed-off-by: Leslie Chang <hi.lesliech...@gmail.com>

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kavinderd/incubator-hawq HAWQ-964

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/906.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #906


commit fec6e3055f6b42b5932cbbe34672282d93790e6c
Author: Kavinder Dhaliwal <kavind...@gmail.com>
Date:   2016-09-15T17:56:20Z

HAWQ-964. Support for OR and NOT Logical Operators

Signed-off-by: Leslie Chang <hi.lesliech...@gmail.com>




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #905: HAWQ-964. Support for OR and NOT Logical O...

2016-09-16 Thread kavinderd

Github user kavinderd closed the pull request at:

https://github.com/apache/incubator-hawq/pull/905


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #905: HAWQ-964. Support for OR and NOT Logical O...

2016-09-16 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/905#discussion_r79261601
  
--- Diff: 
pxf/pxf-hbase/src/main/java/org/apache/hawq/pxf/plugins/hbase/HBaseFilterBuilder.java
 ---
@@ -71,14 +74,19 @@ public HBaseFilterBuilder(HBaseTupleDescription 
tupleDescription) {
  * @throws Exception if parsing failed
  */
 public Filter getFilterObject(String filterString) throws Exception {
-FilterParser parser = new FilterParser(this);
-Object result = parser.parse(filterString);
+// First check for NOT, HBase does not support this
+if (filterString.contains(OR_OP)) {
--- End diff --

Change to NOT_OP


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #905: HAWQ-964. Support for OR and NOT Logical O...

2016-09-16 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/905#discussion_r79261531
  
--- Diff: 
pxf/pxf-hive/src/test/java/org/apache/hawq/pxf/plugins/hive/HiveORCSearchArgumentExample.java
 ---
@@ -0,0 +1,87 @@
+package org.apache.hawq.pxf.plugins.hive;
+
+import org.apache.hadoop.hive.ql.io.sarg.SearchArgument;
+import org.apache.hadoop.hive.ql.io.sarg.SearchArgumentFactory;
+import org.apache.hawq.pxf.api.BasicFilter;
+import org.apache.hawq.pxf.api.LogicalFilter;
+import org.apache.hawq.pxf.api.utilities.ColumnDescriptor;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.List;
+
+
--- End diff --

Whitespace


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #905: HAWQ-964. Support for OR and NOT Logical O...

2016-09-16 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/905#discussion_r79261543
  
--- Diff: 
pxf/pxf-hive/src/test/java/org/apache/hawq/pxf/plugins/hive/HiveFilterBuilderTest.java
 ---
@@ -20,11 +20,14 @@
  */
 
 
+import org.apache.hawq.pxf.api.FilterParser.LogicalOperation;
+import org.apache.hawq.pxf.api.LogicalFilter;
 import org.junit.Test;
 
 import java.util.List;
 
-import static org.apache.hawq.pxf.api.FilterParser.BasicFilter;
+import org.apache.hawq.pxf.api.BasicFilter;
+
--- End diff --

Whitespace


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #905: HAWQ-964. Support for OR and NOT Logical O...

2016-09-16 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/905#discussion_r79261518
  
--- Diff: 
pxf/pxf-hive/src/test/java/org/apache/hawq/pxf/plugins/hive/HiveORCSearchArgumentExample.java
 ---
@@ -0,0 +1,87 @@
+package org.apache.hawq.pxf.plugins.hive;
+
+import org.apache.hadoop.hive.ql.io.sarg.SearchArgument;
+import org.apache.hadoop.hive.ql.io.sarg.SearchArgumentFactory;
+import org.apache.hawq.pxf.api.BasicFilter;
+import org.apache.hawq.pxf.api.LogicalFilter;
+import org.apache.hawq.pxf.api.utilities.ColumnDescriptor;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.List;
+
+
+public class HiveORCSearchArgumentExample {
+
+@Test
+public void buildLogicalOperationTree() throws Exception {
+
+/* Predicate pushdown configuration */
+String filterStr = "a2c1o2a3c3o3l0a4c5o1l1";
+HiveFilterBuilder eval = new HiveFilterBuilder(null);
+Object filter = eval.getFilterObject(filterStr);
+
+Object current = filter;
+SearchArgument.Builder filterBuilder = 
SearchArgumentFactory.newBuilder();
+buildExpression(filterBuilder, Arrays.asList(filter));
+SearchArgument sarg = filterBuilder.build();
+Assert.assertEquals("and(or(lt(col1, 5), not(lteq(col1, 1))), 
or(lt(col1, 5), lteq(col1, 3)))", sarg.toFilterPredicate().toString());
+}
+
+private void buildExpression(SearchArgument.Builder builder, 
List filterList) {
+for (Object f : filterList) {
+if (f instanceof LogicalFilter) {
+switch(((LogicalFilter) f).getOperator()) {
+case HDOP_OR:
+builder.startOr();
+break;
+case HDOP_AND:
+builder.startAnd();
+break;
+case HDOP_NOT:
+builder.startNot();
+break;
+}
+buildExpression(builder, ((LogicalFilter) 
f).getFilterList());
+builder.end();
+} else {
+buildArgument(builder, f);
+}
+}
+}
+
+private void buildArgument(SearchArgument.Builder builder, Object 
filterObj) {
+/* The below functions will not be compatible and requires update  
with Hive 2.0 APIs */
+BasicFilter filter = (BasicFilter) filterObj;
+int filterColumnIndex = filter.getColumn().index();
+Object filterValue = filter.getConstant().constant();
+Integer[] arr = {};
+ColumnDescriptor filterColumn = new ColumnDescriptor("col1", 1,1, 
"Integer", arr);
+String filterColumnName = filterColumn.columnName();
+
+switch(filter.getOperation()) {
+case HDOP_LT:
+builder.lessThan(filterColumnName, filterValue);
+break;
+case HDOP_GT:
+builder.startNot().lessThanEquals(filterColumnName, 
filterValue).end();
+break;
+case HDOP_LE:
+builder.lessThanEquals(filterColumnName, filterValue);
+break;
+case HDOP_GE:
+builder.startNot().lessThanEquals(filterColumnName, 
filterValue).end();
+break;
+case HDOP_EQ:
+builder.equals(filterColumnName, filterValue);
+break;
+case HDOP_NE:
+builder.startNot().equals(filterColumnName, 
filterValue).end();
+break;
+}
+return;
+}
+
--- End diff --

Whitespace


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #905: HAWQ-964. Support for OR and NOT Logical O...

2016-09-16 Thread kavinderd

GitHub user kavinderd opened a pull request:

https://github.com/apache/incubator-hawq/pull/905

HAWQ-964. Support for OR and NOT Logical Operators

Signed-off-by: Leslie Chang <hi.lesliech...@gmail.com>

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kavinderd/incubator-hawq HAWQ-964

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/905.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #905


commit 5100ceb178f38c121b2f51ba7a6c2389c976d0fc
Author: Kavinder Dhaliwal <kavind...@gmail.com>
Date:   2016-09-15T17:56:20Z

HAWQ-964. Support for OR and NOT Logical Operators

Signed-off-by: Leslie Chang <hi.lesliech...@gmail.com>




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #902: HAWQ-1053. Fixed extraction logic for varchar typ...

2016-09-16 Thread kavinderd

Github user kavinderd commented on the issue:

https://github.com/apache/incubator-hawq/pull/902
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #895: HAWQ 971

2016-09-09 Thread kavinderd

Github user kavinderd commented on the issue:

https://github.com/apache/incubator-hawq/pull/895
  
Can you remove all references to `Vectorized` Hive classes since they are 
not yet being used


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #895: HAWQ 971

2016-09-09 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/895#discussion_r78207962
  
--- Diff: 
pxf/pxf-hive/src/test/java/org/apache/hawq/pxf/plugins/hive/HiveDataFragmenterTest.java
 ---
@@ -52,55 +52,55 @@
 HiveMetaStoreClient hiveClient;
 HiveDataFragmenter fragmenter;
 
-@Test
-public void construction() throws Exception {
-prepareConstruction();
-fragmenter = new HiveDataFragmenter(inputData);
-
PowerMockito.verifyNew(JobConf.class).withArguments(hadoopConfiguration, 
HiveDataFragmenter.class);
-
PowerMockito.verifyNew(HiveMetaStoreClient.class).withArguments(hiveConfiguration);
-}
-
-@Test
-public void constructorCantAccessMetaStore() throws Exception {
-prepareConstruction();
-
PowerMockito.whenNew(HiveMetaStoreClient.class).withArguments(hiveConfiguration).thenThrow(new
 MetaException("which way to albuquerque"));
-
-try {
-fragmenter = new HiveDataFragmenter(inputData);
-fail("Expected a RuntimeException");
-} catch (RuntimeException ex) {
-assertEquals(ex.getMessage(), "Failed connecting to Hive 
MetaStore service: which way to albuquerque");
-}
-}
-
-@Test
-public void invalidTableName() throws Exception {
-prepareConstruction();
-fragmenter = new HiveDataFragmenter(inputData);
-
-
when(inputData.getDataSource()).thenReturn("t.r.o.u.b.l.e.m.a.k.e.r");
-
-try {
-fragmenter.getFragments();
-fail("Expected an IllegalArgumentException");
-} catch (IllegalArgumentException ex) {
-assertEquals(ex.getMessage(), "\"t.r.o.u.b.l.e.m.a.k.e.r\" is 
not a valid Hive table name. Should be either  or 
");
-}
-}
-
-private void prepareConstruction() throws Exception {
-inputData = mock(InputData.class);
-
-hadoopConfiguration = mock(Configuration.class);
-
PowerMockito.whenNew(Configuration.class).withNoArguments().thenReturn(hadoopConfiguration);
-
-jobConf = mock(JobConf.class);
-
PowerMockito.whenNew(JobConf.class).withArguments(hadoopConfiguration, 
HiveDataFragmenter.class).thenReturn(jobConf);
-
-hiveConfiguration = mock(HiveConf.class);
-
PowerMockito.whenNew(HiveConf.class).withNoArguments().thenReturn(hiveConfiguration);
-
-hiveClient = mock(HiveMetaStoreClient.class);
-
PowerMockito.whenNew(HiveMetaStoreClient.class).withArguments(hiveConfiguration).thenReturn(hiveClient);
-}
+//@Test
--- End diff --

Can you just delete all this commented code?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #894: HAWQ-1042. Fixed NPE.

2016-09-08 Thread kavinderd

Github user kavinderd commented on the issue:

https://github.com/apache/incubator-hawq/pull/894
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #873: HAWQ-992. PXF Hive data type check in Fragmenter ...

2016-08-31 Thread kavinderd

Github user kavinderd commented on the issue:

https://github.com/apache/incubator-hawq/pull/873
  
Overall LGTM, but I'd like to spend more time reviewing. This is a lot of 
code


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #873: HAWQ-992. PXF Hive data type check in Frag...

2016-08-31 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/873#discussion_r77096574
  
--- Diff: 
pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/utilities/EnumHiveToHawqType.java
 ---
@@ -110,4 +122,68 @@ public static EnumHiveToHawqType 
getHiveToHawqType(String hiveType) {
 + hiveType + " to HAWQ's type");
 }
 
+
+/**
+ * 
+ * @param dataType Hawq data type
+ * @return compatible Hive type to given Hawq type, if there are more 
than one compatible types, it returns one with bigger size
+ * @throws UnsupportedTypeException if there is no corresponding Hive 
type for given Hawq type
+ */
+public static EnumHiveToHawqType getCompatibleHawqToHiveType(DataType 
dataType) {
+
+SortedSet types = new 
TreeSet(
+new Comparator() {
+public int compare(EnumHiveToHawqType a,
+EnumHiveToHawqType b) {
+return Byte.compare(a.getSize(), b.getSize());
+}
+});
--- End diff --

Can this `types` variable be instantiated outside the method? It doesn't 
changed based on input right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #873: HAWQ-992. PXF Hive data type check in Frag...

2016-08-31 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/873#discussion_r77082531
  
--- Diff: 
pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/utilities/EnumHawqType.java 
---
@@ -43,37 +45,40 @@ public void serialize(EnumHawqType value, JsonGenerator 
generator,
  */
 @JsonSerialize(using = EnumHawqTypeSerializer.class)
 public enum EnumHawqType {
-Int2Type("int2"),
-Int4Type("int4"),
-Int8Type("int8"),
-Float4Type("float4"),
-Float8Type("float8"),
-TextType("text"),
-VarcharType("varchar", (byte) 1, true),
-ByteaType("bytea"),
-DateType("date"),
-TimestampType("timestamp"),
-BoolType("bool"),
-NumericType("numeric", (byte) 2, true),
-BpcharType("bpchar", (byte) 1, true);
+Int2Type("int2", DataType.SMALLINT),
+Int4Type("int4", DataType.INTEGER),
+Int8Type("int8", DataType.BIGINT),
+Float4Type("float4", DataType.REAL),
+Float8Type("float8", DataType.FLOAT8),
+TextType("text", DataType.TEXT),
+VarcharType("varchar", DataType.VARCHAR, (byte) 1, false),
+ByteaType("bytea", DataType.BYTEA),
+DateType("date", DataType.DATE),
+TimestampType("timestamp", DataType.TIMESTAMP),
+BoolType("bool", DataType.BOOLEAN),
+NumericType("numeric", DataType.NUMERIC, (byte) 2, false),
+BpcharType("bpchar", DataType.BPCHAR, (byte) 1, false),
--- End diff --

Nvm, I understand why because the modifiers are not mandatory right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #873: HAWQ-992. PXF Hive data type check in Frag...

2016-08-31 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/873#discussion_r77082166
  
--- Diff: 
pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/utilities/EnumHawqType.java 
---
@@ -43,37 +45,40 @@ public void serialize(EnumHawqType value, JsonGenerator 
generator,
  */
 @JsonSerialize(using = EnumHawqTypeSerializer.class)
 public enum EnumHawqType {
-Int2Type("int2"),
-Int4Type("int4"),
-Int8Type("int8"),
-Float4Type("float4"),
-Float8Type("float8"),
-TextType("text"),
-VarcharType("varchar", (byte) 1, true),
-ByteaType("bytea"),
-DateType("date"),
-TimestampType("timestamp"),
-BoolType("bool"),
-NumericType("numeric", (byte) 2, true),
-BpcharType("bpchar", (byte) 1, true);
+Int2Type("int2", DataType.SMALLINT),
+Int4Type("int4", DataType.INTEGER),
+Int8Type("int8", DataType.BIGINT),
+Float4Type("float4", DataType.REAL),
+Float8Type("float8", DataType.FLOAT8),
+TextType("text", DataType.TEXT),
+VarcharType("varchar", DataType.VARCHAR, (byte) 1, false),
+ByteaType("bytea", DataType.BYTEA),
+DateType("date", DataType.DATE),
+TimestampType("timestamp", DataType.TIMESTAMP),
+BoolType("bool", DataType.BOOLEAN),
+NumericType("numeric", DataType.NUMERIC, (byte) 2, false),
+BpcharType("bpchar", DataType.BPCHAR, (byte) 1, false),
--- End diff --

Can you explain why the last parameter is changed to `false`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #873: HAWQ-992. PXF Hive data type check in Frag...

2016-08-31 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/873#discussion_r77047856
  
--- Diff: 
pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/utilities/ColumnDescriptor.java
 ---
@@ -26,10 +26,11 @@
  */
 public class ColumnDescriptor {
 
-   int gpdbColumnTypeCode;
-String gpdbColumnName;
-String gpdbColumnTypeName;
-int gpdbColumnIndex;
+   int dbColumnTypeCode;
--- End diff --

Indent


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...

2016-08-31 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/837#discussion_r77047594
  
--- Diff: 
pxf/pxf-hbase/src/main/java/org/apache/hawq/pxf/plugins/hbase/HBaseFilterBuilder.java
 ---
@@ -165,6 +165,14 @@ private Filter 
handleSimpleOperations(FilterParser.Operation opId,
 ByteArrayComparable comparator = 
getComparator(hbaseColumn.columnTypeCode(),
 constant.constant());
 
+if(operatorsMap.get(opId) == null){
+//HBase not support HDOP_LIKE, use 'NOT NULL' Comarator
--- End diff --

No, @hsyuan comment was that the comment should read "//HBase does not 
support..." HBase does not support the LIKE Filter as far as I am aware


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #854: HAWQ-997. HAWQ doesn't send PXF data type with pr...

2016-08-24 Thread kavinderd

Github user kavinderd commented on the issue:

https://github.com/apache/incubator-hawq/pull/854
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #853: HAWQ-967. Extend Projection info to include filte...

2016-08-24 Thread kavinderd

Github user kavinderd commented on the issue:

https://github.com/apache/incubator-hawq/pull/853
  
Nice work! LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #853: HAWQ-967. Extend Projection info to includ...

2016-08-24 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/853#discussion_r76100388
  
--- Diff: src/backend/access/external/pxffilters.c ---
@@ -451,6 +452,51 @@ opexpr_to_pxffilter(OpExpr *expr, PxfFilterDesc 
*filter)
return false;
 }
 
+static List*
+append_attr_from_var(Var* var, List* attrs)
+{
+   AttrNumber varattno = var->varattno;
+   /* system attr not supported */
+   if (varattno > InvalidAttrNumber)
--- End diff --

So the `varattno` is only valid when greater than `InvalidAttrNumber`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...

2016-08-24 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/837#discussion_r76098154
  
--- Diff: src/backend/optimizer/plan/createplan.c ---
@@ -1146,7 +1146,7 @@ static char** create_pxf_plan(char **segdb_file_map, 
RelOptInfo *rel, int total_
Relation relation = RelationIdGetRelation(planner_rt_fetch(scan_relid, 
ctx->root)->relid);
segdb_work_map = map_hddata_2gp_segments(uri_str, 

 total_segs, segs_participating,
-   
 relation, NULL);
+   
 relation, ctx->root->parse->jointree->quals);
--- End diff --

Can you ensure that the filters are only passed when the guc 
`pxf_enable_filter_pushdown` is enabled.  Such as is done here: 
https://github.com/apache/incubator-hawq/blob/316c5048f303513c40e8194b15024df12a3d76ad/src/backend/access/external/pxffilters.c#L554
 

Currently if this guc is disabled filters should not be passed to PXF


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #848: HAWQ-1006. Fix RPM compliance in Redhat Satellite...

2016-08-16 Thread kavinderd

Github user kavinderd commented on the issue:

https://github.com/apache/incubator-hawq/pull/848
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...

2016-08-10 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/837#discussion_r74286229
  
--- Diff: src/backend/access/external/test/pxffilters_test.c ---
@@ -61,7 +62,7 @@ test__supported_filter_type(void **state)
 
/* go over pxf_supported_types array */
int nargs = sizeof(pxf_supported_types) / sizeof(Oid);
-   assert_int_equal(nargs, 12);
+   assert_int_equal(nargs, 13);
--- End diff --

Maybe have this value derived instead of hard-coded number


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #830: HAWQ-973. Check pi_varList before adding p...

2016-08-02 Thread kavinderd

Github user kavinderd closed the pull request at:

https://github.com/apache/incubator-hawq/pull/830


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #830: HAWQ-973. Check pi_varList before adding p...

2016-08-01 Thread kavinderd

GitHub user kavinderd opened a pull request:

https://github.com/apache/incubator-hawq/pull/830

HAWQ-973. Check pi_varList before adding proj headers



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kavinderd/incubator-hawq HAWQ-973

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/830.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #830


commit 1f30d4139914f692cd42fc32915193edc0fa6505
Author: Kavinder Dhaliwal <kavind...@gmail.com>
Date:   2016-08-01T22:53:40Z

HAWQ-973. Check pi_varList before adding proj headers




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #829: HAWQ-972.Setup multi handle after setting ...

2016-08-01 Thread kavinderd

GitHub user kavinderd opened a pull request:

https://github.com/apache/incubator-hawq/pull/829

HAWQ-972.Setup multi handle after setting all conf



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kavinderd/incubator-hawq HAWQ-972

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/829.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #829


commit 117bb5bf6607e35046622e4cc79b4b8cf3e0392a
Author: Kavinder Dhaliwal <kavind...@gmail.com>
Date:   2016-08-01T22:24:19Z

HAWQ-972.Setup multi handle after setting all conf




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #812: HAWQ-949. Revert serializing floats in pxf string...

2016-07-28 Thread kavinderd

Github user kavinderd commented on the issue:

https://github.com/apache/incubator-hawq/pull/812
  
Already reverted


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #817: HAWQ-954. Check that ExternalSelectDesc reference...

2016-07-28 Thread kavinderd

Github user kavinderd commented on the issue:

https://github.com/apache/incubator-hawq/pull/817
  
Merged


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #812: HAWQ-949. Revert serializing floats in pxf...

2016-07-28 Thread kavinderd

Github user kavinderd closed the pull request at:

https://github.com/apache/incubator-hawq/pull/812


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #817: HAWQ-954. Check that ExternalSelectDesc re...

2016-07-28 Thread kavinderd

Github user kavinderd closed the pull request at:

https://github.com/apache/incubator-hawq/pull/817


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #808: HAWQ-944. Implement new pg_ltoa function as per p...

2016-07-28 Thread kavinderd

Github user kavinderd commented on the issue:

https://github.com/apache/incubator-hawq/pull/808
  
Added `INT32_CHAR_SIZE` to `configure.in`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #820: HAWQ-953 hawq pxf-hive support partition c...

2016-07-28 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/820#discussion_r72680191
  
--- Diff: 
pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveDataFragmenter.java
 ---
@@ -470,3 +476,4 @@ public FragmentsStats getFragmentsStats() throws 
Exception {
 "ANALYZE for Hive plugin is not supported");
 }
 }
+
--- End diff --

Remove whitespace


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #820: HAWQ-953 hawq pxf-hive support partition c...

2016-07-28 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/820#discussion_r72680060
  
--- Diff: 
pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveDataFragmenter.java
 ---
@@ -331,7 +331,7 @@ String serializePartitionKeys(HiveTablePartition 
partData) throws Exception {
 if (partData.partition == null) /*
  * this is a simple hive table - 
there
  * are no partitions
- */{
+ */ {
--- End diff --

Whitespace, please remove


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #821: HAWQ-931. ORC optimized profile for PPD/CP

2016-07-28 Thread kavinderd

Github user kavinderd commented on the issue:

https://github.com/apache/incubator-hawq/pull/821
  
LGTM, nice job


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #821: HAWQ-931. ORC optimized profile for PPD/CP

2016-07-28 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/821#discussion_r72679394
  
--- Diff: 
pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveORCAccessor.java
 ---
@@ -0,0 +1,170 @@
+package org.apache.hawq.pxf.plugins.hive;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+import org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector;
+import org.apache.hadoop.hive.ql.exec.vector.ColumnVector;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch;
+import org.apache.hadoop.hive.ql.io.orc.OrcInputFormat;
+import org.apache.hadoop.hive.ql.io.sarg.SearchArgument;
+import org.apache.hadoop.hive.ql.io.sarg.SearchArgumentFactory;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.InputSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.Reporter;
+import org.apache.hawq.pxf.api.FilterParser;
+import org.apache.hawq.pxf.api.OneRow;
+import org.apache.hawq.pxf.api.utilities.ColumnDescriptor;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import org.apache.orc.Reader;
+import org.apache.orc.RecordReader;
+import org.apache.orc.TypeDescription;
+import org.apache.commons.lang.StringUtils;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import static 
org.apache.hawq.pxf.plugins.hive.HiveInputFormatFragmenter.PXF_HIVE_SERDES;
+
+/**
+ * Specialization of HiveAccessor for a Hive table that stores only ORC 
files.
+ * This class replaces the generic HiveAccessor for a case where a table 
is stored entirely as ORC files.
+ * Use together with {@link HiveInputFormatFragmenter}/{@link 
HiveColumnarSerdeResolver}
+ */
+public class HiveORCAccessor extends HiveAccessor {
+
+private RecordReader batchReader = null;
+private Reader reader = null;
+private VectorizedRowBatch batch = null;
+
+private final String READ_COLUMN_IDS_CONF_STR = 
"hive.io.file.readcolumn.ids";
+private final String READ_ALL_COLUMNS = 
"hive.io.file.read.all.columns";
+private final String READ_COLUMN_NAMES_CONF_STR = 
"hive.io.file.readcolumn.names";
+private final String SARG_PUSHDOWN = "sarg.pushdown";
+
+/**
+ * Constructs a HiveRCFileAccessor.
+ *
+ * @param input input containing user data
+ * @throws Exception if user data was wrong
+ */
+public HiveORCAccessor(InputData input) throws Exception {
+super(input, new OrcInputFormat());
+String[] toks = HiveInputFormatFragmenter.parseToks(input, 
PXF_HIVE_SERDES.COLUMNAR_SERDE.name(), 
PXF_HIVE_SERDES.LAZY_BINARY_COLUMNAR_SERDE.name(), 
PXF_HIVE_SERDES.ORC_SERDE.name(), PXF_HIVE_SERDES.VECTORIZED_ORC_SERDE.name());
+initPartitionFields(toks[HiveInputFormatFragmenter.TOK_KEYS]);
+filterInFragmenter = new 
Boolean(toks[HiveInputFormatFragmenter.TOK_FILTER_DONE]);
+}
+
+@Override
+public boolean openForRead() throws Exception {
+addColumns();
+addFilters();
+return super.openForRead();
+}
+
+@Override
+protected Object getReader(JobConf jobConf, InputSplit split)
+throws IOException {
+return inputFormat.getRecordReader(split, jobConf, Reporter.NULL);
+}
+
+/**
+ * Adds the table tuple description to JobConf ojbect
+ * so only these columns will be returned.
+ */
+private void addColumns() throws Exception {
+
+List colIds = new ArrayList();
+List colNames = new ArrayList();
+for(ColumnDescriptor col: inputData.getTupleDescription()) {
+if(col.isProjected()) {
+colIds.add(String.valueOf(col.columnIndex()));
--- End diff --

Do we need to convert

[GitHub] incubator-hawq issue #821: HAWQ-931. ORC optimized profile for PPD/CP

2016-07-28 Thread kavinderd

Github user kavinderd commented on the issue:

https://github.com/apache/incubator-hawq/pull/821
  
@shivzone Can you explain why we need a separate Resolver for ORC?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #821: HAWQ-931. ORC optimized profile for PPD/CP

2016-07-28 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/821#discussion_r72675808
  
--- Diff: 
pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveORCAccessor.java
 ---
@@ -0,0 +1,170 @@
+package org.apache.hawq.pxf.plugins.hive;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+import org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector;
+import org.apache.hadoop.hive.ql.exec.vector.ColumnVector;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch;
+import org.apache.hadoop.hive.ql.io.orc.OrcInputFormat;
+import org.apache.hadoop.hive.ql.io.sarg.SearchArgument;
+import org.apache.hadoop.hive.ql.io.sarg.SearchArgumentFactory;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.InputSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.Reporter;
+import org.apache.hawq.pxf.api.FilterParser;
+import org.apache.hawq.pxf.api.OneRow;
+import org.apache.hawq.pxf.api.utilities.ColumnDescriptor;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import org.apache.orc.Reader;
+import org.apache.orc.RecordReader;
+import org.apache.orc.TypeDescription;
+import org.apache.commons.lang.StringUtils;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import static 
org.apache.hawq.pxf.plugins.hive.HiveInputFormatFragmenter.PXF_HIVE_SERDES;
+
+/**
+ * Specialization of HiveAccessor for a Hive table that stores only ORC 
files.
+ * This class replaces the generic HiveAccessor for a case where a table 
is stored entirely as ORC files.
+ * Use together with {@link HiveInputFormatFragmenter}/{@link 
HiveColumnarSerdeResolver}
+ */
+public class HiveORCAccessor extends HiveAccessor {
+
+private RecordReader batchReader = null;
+private Reader reader = null;
+private VectorizedRowBatch batch = null;
+
+private final String READ_COLUMN_IDS_CONF_STR = 
"hive.io.file.readcolumn.ids";
+private final String READ_ALL_COLUMNS = 
"hive.io.file.read.all.columns";
+private final String READ_COLUMN_NAMES_CONF_STR = 
"hive.io.file.readcolumn.names";
+private final String SARG_PUSHDOWN = "sarg.pushdown";
+
+/**
+ * Constructs a HiveRCFileAccessor.
+ *
+ * @param input input containing user data
+ * @throws Exception if user data was wrong
+ */
+public HiveORCAccessor(InputData input) throws Exception {
+super(input, new OrcInputFormat());
+String[] toks = HiveInputFormatFragmenter.parseToks(input, 
PXF_HIVE_SERDES.COLUMNAR_SERDE.name(), 
PXF_HIVE_SERDES.LAZY_BINARY_COLUMNAR_SERDE.name(), 
PXF_HIVE_SERDES.ORC_SERDE.name(), PXF_HIVE_SERDES.VECTORIZED_ORC_SERDE.name());
+initPartitionFields(toks[HiveInputFormatFragmenter.TOK_KEYS]);
+filterInFragmenter = new 
Boolean(toks[HiveInputFormatFragmenter.TOK_FILTER_DONE]);
+}
+
+@Override
+public boolean openForRead() throws Exception {
+addColumns();
+addFilters();
+return super.openForRead();
+}
+
+@Override
+protected Object getReader(JobConf jobConf, InputSplit split)
+throws IOException {
+return inputFormat.getRecordReader(split, jobConf, Reporter.NULL);
+}
+
+/**
+ * Adds the table tuple description to JobConf ojbect
+ * so only these columns will be returned.
+ */
+private void addColumns() throws Exception {
+
+List colIds = new ArrayList();
+List colNames = new ArrayList();
+for(ColumnDescriptor col: inputData.getTupleDescription()) {
+if(col.isProjected()) {
+colIds.add(String.valueOf(col.columnIndex()));
+colNames.add(col.columnName())

[GitHub] incubator-hawq pull request #821: HAWQ-931. ORC optimized profile for PPD/CP

2016-07-28 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/821#discussion_r72674048
  
--- Diff: 
pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveORCAccessor.java
 ---
@@ -0,0 +1,170 @@
+package org.apache.hawq.pxf.plugins.hive;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+import org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector;
+import org.apache.hadoop.hive.ql.exec.vector.ColumnVector;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch;
+import org.apache.hadoop.hive.ql.io.orc.OrcInputFormat;
+import org.apache.hadoop.hive.ql.io.sarg.SearchArgument;
+import org.apache.hadoop.hive.ql.io.sarg.SearchArgumentFactory;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.InputSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.Reporter;
+import org.apache.hawq.pxf.api.FilterParser;
+import org.apache.hawq.pxf.api.OneRow;
+import org.apache.hawq.pxf.api.utilities.ColumnDescriptor;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import org.apache.orc.Reader;
+import org.apache.orc.RecordReader;
+import org.apache.orc.TypeDescription;
+import org.apache.commons.lang.StringUtils;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import static 
org.apache.hawq.pxf.plugins.hive.HiveInputFormatFragmenter.PXF_HIVE_SERDES;
+
+/**
+ * Specialization of HiveAccessor for a Hive table that stores only ORC 
files.
+ * This class replaces the generic HiveAccessor for a case where a table 
is stored entirely as ORC files.
+ * Use together with {@link HiveInputFormatFragmenter}/{@link 
HiveColumnarSerdeResolver}
+ */
+public class HiveORCAccessor extends HiveAccessor {
+
+private RecordReader batchReader = null;
+private Reader reader = null;
+private VectorizedRowBatch batch = null;
+
+private final String READ_COLUMN_IDS_CONF_STR = 
"hive.io.file.readcolumn.ids";
+private final String READ_ALL_COLUMNS = 
"hive.io.file.read.all.columns";
+private final String READ_COLUMN_NAMES_CONF_STR = 
"hive.io.file.readcolumn.names";
+private final String SARG_PUSHDOWN = "sarg.pushdown";
+
+/**
+ * Constructs a HiveRCFileAccessor.
--- End diff --

Spelling `HiveORCAcessor`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #821: HAWQ-931. ORC optimized profile for PPD/CP

2016-07-28 Thread kavinderd

Github user kavinderd commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/821#discussion_r72673496
  
--- Diff: 
pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/utilities/ColumnDescriptor.java
 ---
@@ -89,11 +106,20 @@ public boolean isKeyColumn() {
 return RECORD_KEY_NAME.equalsIgnoreCase(gpdbColumnName);
 }
 
+public boolean isProjected() {
+return isProjected;
+}
+
+public void setProjected(boolean projected) {
+isProjected = projected;
+}
+
 @Override
public String toString() {
return "ColumnDescriptor [gpdbColumnTypeCode=" + 
gpdbColumnTypeCode
+ ", gpdbColumnName=" + gpdbColumnName
+ ", gpdbColumnTypeName=" + gpdbColumnTypeName
-   + ", gpdbColumnIndex=" + gpdbColumnIndex + "]";
+   + ", gpdbColumnIndex=" + gpdbColumnIndex
++ ", isProjected=" + isProjected + "]";
--- End diff --

Indent


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #817: HAWQ-954. Check that ExternalSelectDesc re...

2016-07-26 Thread kavinderd

GitHub user kavinderd opened a pull request:

https://github.com/apache/incubator-hawq/pull/817

HAWQ-954. Check that ExternalSelectDesc reference exists



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kavinderd/incubator-hawq HAWQ-954

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/817.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #817


commit 633f398d330fe5581951e0fa3773bdfa02a39fea
Author: Kavinder Dhaliwal <kavind...@gmail.com>
Date:   2016-07-26T23:09:38Z

HAWQ-954. Check that ExternalSelectDesc reference exists




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #812: HAWQ-949. Revert serializing floats in pxf...

2016-07-25 Thread kavinderd

GitHub user kavinderd opened a pull request:

https://github.com/apache/incubator-hawq/pull/812

HAWQ-949. Revert serializing floats in pxf string filter



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kavinderd/incubator-hawq HAWQ-949

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/812.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #812


commit 5387f1c93afc286bba628c66b9cc2015f4b7474e
Author: Kavinder Dhaliwal <kavind...@gmail.com>
Date:   2016-07-25T23:50:50Z

HAWQ-949. Revert serializing floats in pxf string filter




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #808: HAWQ-944. Implement new pg_ltoa function as per p...

2016-07-22 Thread kavinderd

Github user kavinderd commented on the issue:

https://github.com/apache/incubator-hawq/pull/808
  
@xunzhang No rush, take your time


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #808: HAWQ-944. Implement new pg_ltoa function as per p...

2016-07-21 Thread kavinderd

Github user kavinderd commented on the issue:

https://github.com/apache/incubator-hawq/pull/808
  
@paul-guo- @xunzhang Please review, I think I covered all invocations of 
the two functions.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

1 2 >

1 - 100 of 114 matches

Mail list logo