[GitHub] incubator-hawq issue #1203: HAWQ-1420: Enable build of Ranger plugin open so...
Github user huor commented on the issue: https://github.com/apache/incubator-hawq/pull/1203 +1 for building ranger rpm for apache hawq --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1208: HAWQ-1421. PXF rpm updates to make it oss friend...
Github user huor commented on the issue: https://github.com/apache/incubator-hawq/pull/1208 +1 for great effort to building hawq with apache hadoop --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1203: HAWQ-1420: Enable build of Ranger plugin open so...
Github user radarwave commented on the issue: https://github.com/apache/incubator-hawq/pull/1203 LGTM. +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1208: HAWQ-1421. PXF rpm updates to make it oss friend...
Github user radarwave commented on the issue: https://github.com/apache/incubator-hawq/pull/1208 Thanks @shivzone .The virtual packages works to me. +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1203: HAWQ-1420: Enable build of Ranger plugin open so...
Github user ljainpivotalio commented on the issue:
https://github.com/apache/incubator-hawq/pull/1203
```
mvn ${MVN_OPTS} -N ${BUILD_OPTS} install
[INFO] Creating spec file
/root/ranger-plugin/target/rpm/hawq-ranger-plugin/SPECS/hawq-ranger-plugin.spec
[INFO] Building target platforms: noarch-redhat-linux
[INFO] Building for target noarch-redhat-linux
[INFO] Processing files: hawq-ranger-plugin-2.2.0.0-1.el6.noarch
[INFO] Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1
[INFO] Checking for unpackaged file(s): /usr/lib/rpm/check-files
/root/ranger-plugin/target/rpm/hawq-ranger-plugin/buildroot
[INFO] Wrote:
/root/ranger-plugin/target/rpm/hawq-ranger-plugin/RPMS/noarch/hawq-ranger-plugin-2.2.0.0-1.el6.noarch.rpm
[INFO] Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.EplE5i
[WARNING] + umask 022
[WARNING] + cd /root/ranger-plugin/target/rpm/hawq-ranger-plugin/BUILD
[WARNING] + /bin/rm -rf
/root/ranger-plugin/target/rpm/hawq-ranger-plugin/buildroot
[WARNING] + exit 0
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 30.102 s
[INFO] Finished at: 2017-03-31T23:37:12+00:00
[INFO] Final Memory: 14M/95M
[INFO]
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1209: HAWQ-1404. PXF to leverage file-level sta...
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1209#discussion_r109272018
--- Diff:
pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/utilities/EnumAggregationType.java
---
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hawq.pxf.api.utilities;
+
+public enum EnumAggregationType {
+
+COUNT("count", true),
+UNKNOWN("unknown", false);
--- End diff --
Deleted.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1209: HAWQ-1404. PXF to leverage file-level sta...
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1209#discussion_r109272012
--- Diff:
pxf/pxf-service/src/main/java/org/apache/hawq/pxf/service/AggBridge.java ---
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hawq.pxf.service;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.hawq.pxf.api.BadRecordException;
+import org.apache.hawq.pxf.api.OneField;
+import org.apache.hawq.pxf.api.OneRow;
+import org.apache.hawq.pxf.api.StatsAccessor;
+import org.apache.hawq.pxf.service.io.Writable;
+import org.apache.hawq.pxf.service.utilities.ProtocolData;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.commons.collections.map.LRUMap;
+
+/**
+ * Bridge class optimized for aggregate queries.
+ *
+ */
+public class AggBridge extends ReadBridge implements Bridge {
+private static final Log LOG = LogFactory.getLog(AggBridge.class);
+/* Avoid resolving rows with the same key twice */
+private LRUMap resolvedFieldsCache;
+
+public AggBridge(ProtocolData protData) throws Exception {
+super(protData);
+}
+
+@Override
+public boolean beginIteration() throws Exception {
+/* Initialize LRU cache with 100 items*/
+resolvedFieldsCache = new LRUMap();
+return super.fileAccessor.openForRead();
+}
+
+@Override
+@SuppressWarnings("unchecked")
+public Writable getNext() throws Exception {
+Writable output = null;
+List resolvedFields = null;
+OneRow onerow = null;
+
+if (!outputQueue.isEmpty()) {
+return outputQueue.pop();
+}
+
+try {
+while (outputQueue.isEmpty()) {
+onerow = ((StatsAccessor) fileAccessor).emitAggObject();
+if (onerow == null) {
+break;
+}
+resolvedFields = (List)
resolvedFieldsCache.get(onerow.getKey());
+if (resolvedFields == null) {
+resolvedFields = fieldsResolver.getFields(onerow);
+resolvedFieldsCache.put(onerow.getKey(),
resolvedFields);
+}
+outputQueue = outputBuilder.makeOutput(resolvedFields);
+if (!outputQueue.isEmpty()) {
+output = outputQueue.pop();
+break;
+}
+}
+} catch (Exception ex) {
+throw ex;
--- End diff --
Added logging of message before re-throwing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1209: HAWQ-1404. PXF to leverage file-level sta...
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1209#discussion_r109271522
--- Diff:
pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveORCAccessor.java
---
@@ -65,13 +78,24 @@ public HiveORCAccessor(InputData input) throws
Exception {
HiveUserData hiveUserData = HiveUtilities.parseHiveUserData(input,
PXF_HIVE_SERDES.ORC_SERDE);
initPartitionFields(hiveUserData.getPartitionKeys());
filterInFragmenter = hiveUserData.isFilterInFragmenter();
+useStats = Utilities.useStats(this, inputData);
}
@Override
public boolean openForRead() throws Exception {
-addColumns();
-addFilters();
-return super.openForRead();
+if (useStats) {
+orcReader = HiveUtilities.getOrcReader(inputData);
+if (orcReader == null) {
+return false;
+}
+retrieveStats();
+objectsEmitted = 0;
+return super.openForRead();
--- End diff --
Sure, updated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1209: HAWQ-1404. PXF to leverage file-level sta...
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1209#discussion_r109271460
--- Diff:
pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/utilities/Utilities.java ---
@@ -151,4 +157,78 @@ public static String maskNonPrintables(String input) {
}
return input.replaceAll("[^a-zA-Z0-9_:/-]", ".");
}
+
+/**
+ * Parses input data and returns fragment metadata.
+ *
+ * @param inputData input data which has protocol information
+ * @return fragment metadata
+ * @throws IllegalArgumentException if fragment metadata information
wasn't found in input data
+ * @throws Exception
+ */
+public static FragmentMetadata parseFragmentMetadata(InputData
inputData) throws Exception {
+byte[] serializedLocation = inputData.getFragmentMetadata();
+if (serializedLocation == null) {
+throw new IllegalArgumentException("Missing fragment location
information");
+}
+try (ByteArrayInputStream bytesStream = new
ByteArrayInputStream(serializedLocation);
--- End diff --
Thanks, makes sense.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1209: HAWQ-1404. PXF to leverage file-level sta...
Github user denalex commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1209#discussion_r109264668
--- Diff:
pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/utilities/Utilities.java ---
@@ -151,4 +157,78 @@ public static String maskNonPrintables(String input) {
}
return input.replaceAll("[^a-zA-Z0-9_:/-]", ".");
}
+
+/**
+ * Parses input data and returns fragment metadata.
+ *
+ * @param inputData input data which has protocol information
+ * @return fragment metadata
+ * @throws IllegalArgumentException if fragment metadata information
wasn't found in input data
+ * @throws Exception
+ */
+public static FragmentMetadata parseFragmentMetadata(InputData
inputData) throws Exception {
+byte[] serializedLocation = inputData.getFragmentMetadata();
+if (serializedLocation == null) {
+throw new IllegalArgumentException("Missing fragment location
information");
+}
+try (ByteArrayInputStream bytesStream = new
ByteArrayInputStream(serializedLocation);
--- End diff --
since you're not using bytesStream directly, can collapse to
ObjectInputStream objectStream = new ObjectInputStream( new
ByteArrayInputStream(serializedLocation))
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1209: HAWQ-1404. PXF to leverage file-level sta...
Github user denalex commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1209#discussion_r109265306
--- Diff:
pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveORCAccessor.java
---
@@ -213,4 +237,40 @@ private boolean buildArgument(SearchArgument.Builder
builder, Object filterObj)
return true;
}
+/**
+ * Fetches file-level statistics from an ORC file.
+ */
+@Override
+public void retrieveStats() throws Exception {
+FragmentMetadata fragmentMetadata =
Utilities.parseFragmentMetadata(inputData);
+/*
+ * We are using file-level stats therefore if file has multiple
splits,
+ * it's enough to return count for a first split in file.
+ * In case file has multiple splits - we don't want to duplicate
counts.
+ */
+if (inputData.getFragmentIndex() == 0)
+this.count = this.orcReader.getNumberOfRows();
+}
+
+/**
+ * Emits tuple without reading from disk, currently supports COUNT
+ */
+@Override
+public OneRow emitAggObject() {
+OneRow row = null;
+switch (inputData.getAggType()) {
+case COUNT:
+if (objectsEmitted < count) {
+objectsEmitted++;
+row = new OneRow(key, data);
--- End diff --
as this is likely a dummy object, can you reuse a single object and return
it multiple times rather than creating multiple dummy objects ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1209: HAWQ-1404. PXF to leverage file-level sta...
Github user denalex commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1209#discussion_r109265434
--- Diff:
pxf/pxf-service/src/main/java/org/apache/hawq/pxf/service/AggBridge.java ---
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hawq.pxf.service;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.hawq.pxf.api.BadRecordException;
+import org.apache.hawq.pxf.api.OneField;
+import org.apache.hawq.pxf.api.OneRow;
+import org.apache.hawq.pxf.api.StatsAccessor;
+import org.apache.hawq.pxf.service.io.Writable;
+import org.apache.hawq.pxf.service.utilities.ProtocolData;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.commons.collections.map.LRUMap;
+
+/**
+ * Bridge class optimized for aggregate queries.
+ *
+ */
+public class AggBridge extends ReadBridge implements Bridge {
+private static final Log LOG = LogFactory.getLog(AggBridge.class);
+/* Avoid resolving rows with the same key twice */
+private LRUMap resolvedFieldsCache;
+
+public AggBridge(ProtocolData protData) throws Exception {
+super(protData);
+}
+
+@Override
+public boolean beginIteration() throws Exception {
+/* Initialize LRU cache with 100 items*/
+resolvedFieldsCache = new LRUMap();
+return super.fileAccessor.openForRead();
+}
+
+@Override
+@SuppressWarnings("unchecked")
+public Writable getNext() throws Exception {
+Writable output = null;
+List resolvedFields = null;
+OneRow onerow = null;
+
+if (!outputQueue.isEmpty()) {
+return outputQueue.pop();
+}
+
+try {
+while (outputQueue.isEmpty()) {
+onerow = ((StatsAccessor) fileAccessor).emitAggObject();
+if (onerow == null) {
+break;
+}
+resolvedFields = (List)
resolvedFieldsCache.get(onerow.getKey());
+if (resolvedFields == null) {
+resolvedFields = fieldsResolver.getFields(onerow);
+resolvedFieldsCache.put(onerow.getKey(),
resolvedFields);
+}
+outputQueue = outputBuilder.makeOutput(resolvedFields);
+if (!outputQueue.isEmpty()) {
+output = outputQueue.pop();
+break;
+}
+}
+} catch (Exception ex) {
+throw ex;
--- End diff --
what's the point of this ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1209: HAWQ-1404. PXF to leverage file-level sta...
Github user denalex commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1209#discussion_r109265088
--- Diff:
pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveORCAccessor.java
---
@@ -65,13 +78,24 @@ public HiveORCAccessor(InputData input) throws
Exception {
HiveUserData hiveUserData = HiveUtilities.parseHiveUserData(input,
PXF_HIVE_SERDES.ORC_SERDE);
initPartitionFields(hiveUserData.getPartitionKeys());
filterInFragmenter = hiveUserData.isFilterInFragmenter();
+useStats = Utilities.useStats(this, inputData);
}
@Override
public boolean openForRead() throws Exception {
-addColumns();
-addFilters();
-return super.openForRead();
+if (useStats) {
+orcReader = HiveUtilities.getOrcReader(inputData);
+if (orcReader == null) {
+return false;
+}
+retrieveStats();
+objectsEmitted = 0;
+return super.openForRead();
--- End diff --
should carry this common line outside if / else ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (HAWQ-1421) Improve PXF rpm package name format and dependencies
[
https://issues.apache.org/jira/browse/HAWQ-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951674#comment-15951674
]
Shivram Mani commented on HAWQ-1421:
Just to summarize the discussion within the pull request.
An apache complaint rpm will be the default result of running make rpm
If you need it to be hdp distribution compliant, make sure the HDP rpms (along
with the virtual rpms) are installed or made available in a repo.
This is to keep the dependancy generic across apache and every other hadoop
distribution.
> Improve PXF rpm package name format and dependencies
>
>
> Key: HAWQ-1421
> URL: https://issues.apache.org/jira/browse/HAWQ-1421
> Project: Apache HAWQ
> Issue Type: Improvement
> Components: Build, PXF
>Reporter: Radar Lei
>Assignee: Shivram Mani
> Fix For: 2.2.0.0-incubating
>
>
> If we build pxf rpm package by 'make rpm', we will get below pxf packages:
> {quote}
> apache-tomcat-7.0.62-el6.noarch.rpm
> pxf-3.2.1.0-root.el6.noarch.rpm
> pxf-hbase_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-hdfs_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-hive_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-jdbc_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-json_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-service_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> {quote}
> These rpm packages have dependencies on Apache Hadoop components only, some
> other Hadoop distributes can't satisfy it. E.g. :
> {quote}
> rpm -ivh pxf-hdfs_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> error: Failed dependencies:
> pxf-service_3_2_1_0 >= 3.2.1.0 is needed by
> pxf-hdfs_3_2_1_0-0:3.2.1.0-root.el6.noarch
> hadoop >= 2.7.1 is needed by pxf-hdfs_3_2_1_0-0:3.2.1.0-root.el6.noarch
> hadoop-mapreduce >= 2.7.1 is needed by
> pxf-hdfs_3_2_1_0-0:3.2.1.0-root.el6.noarch
> {quote}
> We'd better make the rpm package name format and dependencies better.
> 1. Remove the version string like '3_2_1_0'.
> 2. Remove the user name from the build environment.
> 3. Consider do we need to include the apache-tomcat rpm package into HAWQ
> rpm release tarball.
> 4. Improve the hard code 'el6' string. (This might be optinal)
> 5. Improve the dependencies, including the dependencies between these pxf
> rpm packages.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] incubator-hawq issue #1209: HAWQ-1404. PXF to leverage file-level stats of O...
Github user shivzone commented on the issue: https://github.com/apache/incubator-hawq/pull/1209 Do you plan to add any unit tests ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1208: HAWQ-1421. PXF rpm updates to make it oss friend...
Github user shivzone commented on the issue: https://github.com/apache/incubator-hawq/pull/1208 @radarwave looks like you have installed the HDP rpms without using the virtual rpms. Products that depend on hdp will only have a dependancy on their virtual rpms as it is close to impossible to keep track of the exact build number of hadoop as part of the dependancy. hadoop virutal rpm will be hadoop-hdfs which will have a dependancy on the physical rpm hadoop_2_5_0_0_1245-hdfs. Hope this fixes your issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (HAWQ-1415) Set JAVA_HOME / JRE_HOME for RPS in the scripts
[ https://issues.apache.org/jira/browse/HAWQ-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Denissov resolved HAWQ-1415. -- Resolution: Fixed Fix Version/s: (was: backlog) 2.2.0.0-incubating > Set JAVA_HOME / JRE_HOME for RPS in the scripts > --- > > Key: HAWQ-1415 > URL: https://issues.apache.org/jira/browse/HAWQ-1415 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Security >Reporter: Alexander Denissov >Assignee: Alexander Denissov > Fix For: 2.2.0.0-incubating > > > JAVA_HOME or JRE_HOME need to be set for RPS to run. Ambari installs JDK > under /usr/jdk64 by default. Until Ambari starts managing RPS, we should do > the following: > - discover the JDK under /usr/jdk64 > - if found, set this value as default value for JAVA_HOME property in > rps.properties > - export JAVA_HOME in rps.sh > If the JDK is not available in /usr/jdk64, warn the user that JAVA_HOME must > be setup. The logic can go into enable-ranger-plugin.sh script. > Once Ambari starts managing the RPS configuration, the default can be changed > to /usr/java/default targeted for non-Ambari installations. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HAWQ-1415) Set JAVA_HOME / JRE_HOME for RPS in the scripts
[ https://issues.apache.org/jira/browse/HAWQ-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951390#comment-15951390 ] Alexander Denissov commented on HAWQ-1415: -- Work has been done by [~lavjain] > Set JAVA_HOME / JRE_HOME for RPS in the scripts > --- > > Key: HAWQ-1415 > URL: https://issues.apache.org/jira/browse/HAWQ-1415 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Security >Reporter: Alexander Denissov >Assignee: Alexander Denissov > Fix For: backlog > > > JAVA_HOME or JRE_HOME need to be set for RPS to run. Ambari installs JDK > under /usr/jdk64 by default. Until Ambari starts managing RPS, we should do > the following: > - discover the JDK under /usr/jdk64 > - if found, set this value as default value for JAVA_HOME property in > rps.properties > - export JAVA_HOME in rps.sh > If the JDK is not available in /usr/jdk64, warn the user that JAVA_HOME must > be setup. The logic can go into enable-ranger-plugin.sh script. > Once Ambari starts managing the RPS configuration, the default can be changed > to /usr/java/default targeted for non-Ambari installations. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HAWQ-1415) Set JAVA_HOME / JRE_HOME for RPS in the scripts
[ https://issues.apache.org/jira/browse/HAWQ-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Denissov reassigned HAWQ-1415: Assignee: Alexander Denissov (was: Ed Espino) > Set JAVA_HOME / JRE_HOME for RPS in the scripts > --- > > Key: HAWQ-1415 > URL: https://issues.apache.org/jira/browse/HAWQ-1415 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Security >Reporter: Alexander Denissov >Assignee: Alexander Denissov > Fix For: backlog > > > JAVA_HOME or JRE_HOME need to be set for RPS to run. Ambari installs JDK > under /usr/jdk64 by default. Until Ambari starts managing RPS, we should do > the following: > - discover the JDK under /usr/jdk64 > - if found, set this value as default value for JAVA_HOME property in > rps.properties > - export JAVA_HOME in rps.sh > If the JDK is not available in /usr/jdk64, warn the user that JAVA_HOME must > be setup. The logic can go into enable-ranger-plugin.sh script. > Once Ambari starts managing the RPS configuration, the default can be changed > to /usr/java/default targeted for non-Ambari installations. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HAWQ-1422) Provide user-group membership in RPS when requesting access
[ https://issues.apache.org/jira/browse/HAWQ-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Denissov reassigned HAWQ-1422: Assignee: Alexander Denissov (was: Ed Espino) > Provide user-group membership in RPS when requesting access > --- > > Key: HAWQ-1422 > URL: https://issues.apache.org/jira/browse/HAWQ-1422 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Security >Reporter: Alexander Denissov >Assignee: Alexander Denissov > Fix For: backlog > > > Currently user group membership is not provided to RangerBasePlugin when > access request is made, so group policies will not be applied. > We need to lookup user group membership, one way to do it is using > HadoopCommon library. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HAWQ-1422) Provide user-group membership in RPS when requesting access
Alexander Denissov created HAWQ-1422: Summary: Provide user-group membership in RPS when requesting access Key: HAWQ-1422 URL: https://issues.apache.org/jira/browse/HAWQ-1422 Project: Apache HAWQ Issue Type: Sub-task Components: Security Reporter: Alexander Denissov Assignee: Ed Espino Currently user group membership is not provided to RangerBasePlugin when access request is made, so group policies will not be applied. We need to lookup user group membership, one way to do it is using HadoopCommon library. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HAWQ-1421) Improve PXF rpm package name format and dependencies
[
https://issues.apache.org/jira/browse/HAWQ-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15950605#comment-15950605
]
Radar Lei commented on HAWQ-1421:
-
[~shivram] I'm fine with 'export HD=hdp'.
Please check the comments in PR, let's test and verify to make sure 'export
HD=hdp' works.
> Improve PXF rpm package name format and dependencies
>
>
> Key: HAWQ-1421
> URL: https://issues.apache.org/jira/browse/HAWQ-1421
> Project: Apache HAWQ
> Issue Type: Improvement
> Components: Build, PXF
>Reporter: Radar Lei
>Assignee: Shivram Mani
> Fix For: 2.2.0.0-incubating
>
>
> If we build pxf rpm package by 'make rpm', we will get below pxf packages:
> {quote}
> apache-tomcat-7.0.62-el6.noarch.rpm
> pxf-3.2.1.0-root.el6.noarch.rpm
> pxf-hbase_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-hdfs_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-hive_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-jdbc_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-json_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-service_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> {quote}
> These rpm packages have dependencies on Apache Hadoop components only, some
> other Hadoop distributes can't satisfy it. E.g. :
> {quote}
> rpm -ivh pxf-hdfs_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> error: Failed dependencies:
> pxf-service_3_2_1_0 >= 3.2.1.0 is needed by
> pxf-hdfs_3_2_1_0-0:3.2.1.0-root.el6.noarch
> hadoop >= 2.7.1 is needed by pxf-hdfs_3_2_1_0-0:3.2.1.0-root.el6.noarch
> hadoop-mapreduce >= 2.7.1 is needed by
> pxf-hdfs_3_2_1_0-0:3.2.1.0-root.el6.noarch
> {quote}
> We'd better make the rpm package name format and dependencies better.
> 1. Remove the version string like '3_2_1_0'.
> 2. Remove the user name from the build environment.
> 3. Consider do we need to include the apache-tomcat rpm package into HAWQ
> rpm release tarball.
> 4. Improve the hard code 'el6' string. (This might be optinal)
> 5. Improve the dependencies, including the dependencies between these pxf
> rpm packages.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HAWQ-1391) s390x support for HWCRC32c
[
https://issues.apache.org/jira/browse/HAWQ-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15950520#comment-15950520
]
Namrata Bhave commented on HAWQ-1391:
-
while further debugging the GPHD_HA_load_nodes tests mentioned above, I
observed that the root cause of failure is "size" not getting correctly set in
file
"https://github.com/apache/incubator-hawq/blob/master/src/backend/access/external/test/ha_config_mock.c;
in hdfsGetHANamenodes function.
Below is the code snippet where I had printed the size on s390x as well as x86
and found it to be different for above tests.
Namenode * hdfsGetHANamenodes(const char *nameservice, int *size)
{
optional_assignment(size);
// printed the value of *size here and it always shows zero!
return (Namenode *)mock();
}
The 'optional_assignment' is added from cmockery.h/.c files.
Anyone has faced such issue before?
I could see code related to memory assignment in cmockery files. However
couldn't find anywhere whether cmockery is platform specific or doesn't support
big endian/s390x.
> s390x support for HWCRC32c
> --
>
> Key: HAWQ-1391
> URL: https://issues.apache.org/jira/browse/HAWQ-1391
> Project: Apache HAWQ
> Issue Type: Bug
> Components: libhdfs
>Reporter: ketan
>Assignee: Ed Espino
>
> Hi ,
> I am in progress building Apache - HAWQ on s390x
> following instruction on
> https://cwiki.apache.org/confluence/display/HAWQ/Build+and+Install
> I am in the build stage i notice that during the build i encounter
> undefined reference to vtable for Hdfs::Internal::HWCrc32c
> On further debugging i observed that libhdfs3/src/common/HWCRC32c.cpp has
> not support for s390x.
> My questions are as follows.
> 1) I want to confirm whether does this check happens as part of unit testing
> of libhdfs3?
> 2) if yes to 1 whether this test is specific to SSE based platforms ?
> 3) can we exactly get some information on what this check does>?
> 4) Is HAWQ source supported on SSE based platforms only ?
> Help would be appreciated.
> Adding Log for reference.
> **
> make[3]: Leaving directory `//incubator-hawq/src/backend/cdb'
> g++ -O3 -std=gnu99 -Wall -Wmissing-prototypes -Wpointer-arith
> -Wendif-labels -Wformat-security -fno-strict-aliasing -fwrapv
> -fno-aggressive-loop-optimizations -I/usr/include/libxml2 -L../../src/port
> -L../../src/port -Wl,--as-needed
> -L/scratch/ecos0013/ketan/incubator-hawq/depends/libhdfs3/build/install/usr/local/hawq/lib
>
> -L/scratch/ecos0013/ketan/incubator-hawq/depends/libyarn/build/install/usr/local/hawq/lib
> -Wl,-rpath,'/usr/local/hawq/lib',--enable-new-dtags -Wl,-E access/SUBSYS.o
> bootstrap/SUBSYS.o catalog/SUBSYS.o parser/SUBSYS.o commands/SUBSYS.o
> executor/SUBSYS.o foreign/SUBSYS.o lib/SUBSYS.o libpq/SUBSYS.o
> gp_libpq_fe/SUBSYS.o main/SUBSYS.o nodes/SUBSYS.o optimizer/SUBSYS.o
> port/SUBSYS.o postmaster/SUBSYS.o regex/SUBSYS.o rewrite/SUBSYS.o
> storage/SUBSYS.o tcop/SUBSYS.o utils/SUBSYS.o resourcemanager/SUBSYS.o
> ../../src/timezone/SUBSYS.o cdb/SUBSYS.o ../../src/port/libpgport_srv.a
> -lprotobuf -lboost_system -lboost_date_time -lstdc++ -lhdfs3 -lgsasl -lxml2
> -ljson-c -levent -lyaml -lsnappy -lbz2 -lrt -lz -lcrypt -ldl -lm -lcurl
> -lyarn -lkrb5 -lpthread -lthrift -lsnappy -o postgres
> /scratch/ecos0013/ketan/incubator-hawq/depends/libhdfs3/build/install/usr/local/hawq/lib/libhdfs3.so:
> undefined reference to `Hdfs::Internal::HWCrc32c::available()'
> /scratch/ecos0013/ketan/incubator-hawq/depends/libhdfs3/build/install/usr/local/hawq/lib/libhdfs3.so:
> undefined reference to `vtable for Hdfs::Internal::HWCrc32c'
> collect2: error: ld returned 1 exit status
> make[2]: *** [postgres] Error 1
> make[2]: Leaving directory `incubator-hawq/src/backend'
> make[1]: *** [all] Error 2
> make[1]: Leaving directory `/incubator-hawq/src'
> make: *** [all] Error 2
> **
> Regards
> Ketan
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] incubator-hawq issue #1208: HAWQ-1421. PXF rpm updates to make it oss friend...
Github user radarwave commented on the issue: https://github.com/apache/incubator-hawq/pull/1208 sudo yum list installed | grep hadoop hadoop-conf-pseudo.noarch2.7.3.2.5.0.0-1245.el6 @HDP-2.5.0.0 hadoop_2_5_0_0_1245.x86_64 2.7.3.2.5.0.0-1245.el6 @HDP-2.5.0.0 hadoop_2_5_0_0_1245-conf-pseudo.x86_64 hadoop_2_5_0_0_1245-hdfs.x86_64 2.7.3.2.5.0.0-1245.el6 @HDP-2.5.0.0 hadoop_2_5_0_0_1245-hdfs-datanode.x86_64 hadoop_2_5_0_0_1245-hdfs-namenode.x86_64 hadoop_2_5_0_0_1245-hdfs-secondarynamenode.x86_64 hadoop_2_5_0_0_1245-mapreduce.x86_64 hadoop_2_5_0_0_1245-mapreduce-historyserver.x86_64 hadoop_2_5_0_0_1245-yarn.x86_64 2.7.3.2.5.0.0-1245.el6 @HDP-2.5.0.0 hadoop_2_5_0_0_1245-yarn-nodemanager.x86_64 hadoop_2_5_0_0_1245-yarn-resourcemanager.x86_64 rpm -qR hadoop-hdfs.noarch package hadoop-hdfs.noarch is not installed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
