[jira] [Commented] (HAWQ-1512) Check Apache HAWQ mandatory libraries to match LC20, LC30 license criteria
[ https://issues.apache.org/jira/browse/HAWQ-1512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352435#comment-16352435 ] Radar Lei commented on HAWQ-1512: - Thanks [~yjin] AND [~huor] 's review. Already added googlemock. For Ranger dependences, as Ruilong's comment, they are not mandatory so they will not be added. For libhdfs3 I think the previous open source place is retired and the latest code only exist in HAWQ, so we do not need to add it. Libyarn is similar case. Another one I want to mention here is "libgsasl" which is using LGPL, I did not add it because it can be treat as a system dependence. See previous discuss email: https://lists.apache.org/thread.html/5ae122b59529de58c5c668fa0e703a53ad9efb0fddb0fb26ecbcace8@%3Cdev.hawq.apache.org%3E > Check Apache HAWQ mandatory libraries to match LC20, LC30 license criteria > -- > > Key: HAWQ-1512 > URL: https://issues.apache.org/jira/browse/HAWQ-1512 > Project: Apache HAWQ > Issue Type: Task > Components: Build >Reporter: Yi Jin >Assignee: Radar Lei >Priority: Major > Fix For: 2.3.0.0-incubating > > Attachments: HAWQ Ranger Pluggin Service Dependencies.xlsx > > > Check Apache HAWQ mandatory libraries to match LC20, LC30 license criteria > Check the following page for the criteria > https://cwiki.apache.org/confluence/display/HAWQ/ASF+Maturity+Evaluation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] incubator-hawq issue #1335: Add setting for cloudera manager
Github user lavjain commented on the issue: https://github.com/apache/incubator-hawq/pull/1335 @yk-st Your change looks good. However, it might be cleaner to create a separate template for CDH manager (eg `pxf-private-cdh-manager.classpath.template`) rather than substituting the token values. This will also ensure that PXF pipelines for GPDB are not affected. ---
[GitHub] incubator-hawq pull request #1334: HAWQ-1584. Don't ignore exceptions during...
Github user denalex commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1334#discussion_r166099466
--- Diff:
pxf/pxf-service/src/main/java/org/apache/hawq/pxf/service/rest/WritableResource.java
---
@@ -143,36 +142,38 @@ private static synchronized Response
synchronizedWriteResponse(Bridge bridge,
private static Response writeResponse(Bridge bridge,
String path,
- InputStream inputStream) throws
Exception {
-
-String returnMsg;
-
+ InputStream inputStream)
+throws Exception {
// Open the output file
bridge.beginIteration();
-
long totalWritten = 0;
+Exception ex = null;
// dataStream will close automatically in the end of the try.
// inputStream is closed by dataStream.close().
try (DataInputStream dataStream = new
DataInputStream(inputStream)) {
while (bridge.setNext(dataStream)) {
++totalWritten;
}
-} catch (ClientAbortException e) {
-LOG.debug("Remote connection closed by HAWQ", e);
-} catch (Exception ex) {
-LOG.debug("totalWritten so far " + totalWritten + " to " +
path);
-throw ex;
+} catch (ClientAbortException cae) {
+LOG.error("Remote connection closed by HAWQ", cae);
+} catch (Exception e) {
+LOG.error("Exception: totalWritten so far " + totalWritten + "
to " + path, e);
+ex = e;
} finally {
try {
bridge.endIteration();
} catch (Exception e) {
-// ignore ... any significant errors should already have
been handled
+if (ex == null)
+ex = e;
--- End diff --
Another way would be to preserve throwing the exception inside original
catch block (line 162), then here you would say
```
if (ex == null)
throw e;
else
throw ex;
```
and you would not need the block below (lines 170-172) as the original will
still be thrown if endIterations() completes without an error.
---
[GitHub] incubator-hawq pull request #1334: HAWQ-1584. Don't ignore exceptions during...
Github user denalex commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1334#discussion_r166099856
--- Diff:
pxf/pxf-service/src/main/java/org/apache/hawq/pxf/service/rest/WritableResource.java
---
@@ -143,36 +142,38 @@ private static synchronized Response
synchronizedWriteResponse(Bridge bridge,
private static Response writeResponse(Bridge bridge,
String path,
- InputStream inputStream) throws
Exception {
-
-String returnMsg;
-
+ InputStream inputStream)
+throws Exception {
// Open the output file
bridge.beginIteration();
-
long totalWritten = 0;
+Exception ex = null;
// dataStream will close automatically in the end of the try.
// inputStream is closed by dataStream.close().
try (DataInputStream dataStream = new
DataInputStream(inputStream)) {
while (bridge.setNext(dataStream)) {
++totalWritten;
}
-} catch (ClientAbortException e) {
-LOG.debug("Remote connection closed by HAWQ", e);
-} catch (Exception ex) {
-LOG.debug("totalWritten so far " + totalWritten + " to " +
path);
-throw ex;
+} catch (ClientAbortException cae) {
+LOG.error("Remote connection closed by HAWQ", cae);
+} catch (Exception e) {
+LOG.error("Exception: totalWritten so far " + totalWritten + "
to " + path, e);
+ex = e;
} finally {
try {
bridge.endIteration();
} catch (Exception e) {
-// ignore ... any significant errors should already have
been handled
+if (ex == null)
+ex = e;
}
+// propagate any exceptions
+if (ex != null)
+throw ex;
}
String censuredPath = Utilities.maskNonPrintables(path);
-returnMsg = "wrote " + totalWritten + " bulks to " + censuredPath;
+String returnMsg = "wrote " + totalWritten + " bulks to " +
censuredPath;
--- End diff --
move string concat inside LOG.isDebugEnabled()
---
[GitHub] incubator-hawq pull request #1334: HAWQ-1584. Don't ignore exceptions during...
Github user lavjain commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1334#discussion_r166122917
--- Diff:
pxf/pxf-service/src/main/java/org/apache/hawq/pxf/service/rest/WritableResource.java
---
@@ -143,36 +142,38 @@ private static synchronized Response
synchronizedWriteResponse(Bridge bridge,
private static Response writeResponse(Bridge bridge,
String path,
- InputStream inputStream) throws
Exception {
-
-String returnMsg;
-
+ InputStream inputStream)
+throws Exception {
// Open the output file
bridge.beginIteration();
-
long totalWritten = 0;
+Exception ex = null;
// dataStream will close automatically in the end of the try.
// inputStream is closed by dataStream.close().
try (DataInputStream dataStream = new
DataInputStream(inputStream)) {
while (bridge.setNext(dataStream)) {
++totalWritten;
}
-} catch (ClientAbortException e) {
-LOG.debug("Remote connection closed by HAWQ", e);
-} catch (Exception ex) {
-LOG.debug("totalWritten so far " + totalWritten + " to " +
path);
-throw ex;
+} catch (ClientAbortException cae) {
+LOG.error("Remote connection closed by HAWQ", cae);
+} catch (Exception e) {
+LOG.error("Exception: totalWritten so far " + totalWritten + "
to " + path, e);
+ex = e;
} finally {
try {
bridge.endIteration();
} catch (Exception e) {
-// ignore ... any significant errors should already have
been handled
+if (ex == null)
+ex = e;
}
+// propagate any exceptions
+if (ex != null)
+throw ex;
}
String censuredPath = Utilities.maskNonPrintables(path);
-returnMsg = "wrote " + totalWritten + " bulks to " + censuredPath;
+String returnMsg = "wrote " + totalWritten + " bulks to " +
censuredPath;
--- End diff --
returnMsg is also being used in the response.
---
[GitHub] incubator-hawq pull request #1326: HAWQ-1575. Implemented readable Parquet p...
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1326#discussion_r166173537
--- Diff:
pxf/pxf-hdfs/src/main/java/org/apache/hawq/pxf/plugins/hdfs/utilities/HdfsUtilities.java
---
@@ -151,18 +153,42 @@ public static boolean isThreadSafe(String dataDir,
String compCodec) {
* @param fsp file split to be serialized
* @return byte serialization of fsp
* @throws IOException if I/O errors occur while writing to the
underlying
- * stream
+ * stream
*/
public static byte[] prepareFragmentMetadata(FileSplit fsp)
throws IOException {
-ByteArrayOutputStream byteArrayStream = new
ByteArrayOutputStream();
-ObjectOutputStream objectStream = new ObjectOutputStream(
-byteArrayStream);
-objectStream.writeLong(fsp.getStart());
-objectStream.writeLong(fsp.getLength());
-objectStream.writeObject(fsp.getLocations());
+
+return prepareFragmentMetadata(fsp.getStart(), fsp.getLength(),
fsp.getLocations());
+
+}
+
+public static byte[] prepareFragmentMetadata(long start, long length,
String[] locations)
--- End diff --
Both functions are used so I would rather keep them both for sake of
compatibility.
---
[GitHub] incubator-hawq pull request #1326: HAWQ-1575. Implemented readable Parquet p...
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1326#discussion_r166173757
--- Diff:
pxf/pxf-hdfs/src/main/java/org/apache/hawq/pxf/plugins/hdfs/ParquetFileAccessor.java
---
@@ -0,0 +1,168 @@
+package org.apache.hawq.pxf.plugins.hdfs;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hawq.pxf.api.OneRow;
+import org.apache.hawq.pxf.api.ReadAccessor;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import org.apache.hawq.pxf.api.utilities.Plugin;
+import org.apache.hawq.pxf.plugins.hdfs.utilities.HdfsUtilities;
+
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.example.data.Group;
+import org.apache.parquet.example.data.simple.convert.GroupRecordConverter;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.io.ColumnIOFactory;
+import org.apache.parquet.io.MessageColumnIO;
+import org.apache.parquet.io.RecordReader;
+import org.apache.parquet.schema.MessageType;
+
+import java.io.IOException;
+import java.util.Iterator;
+
+/**
+ * Parquet file accessor.
+ */
+public class ParquetFileAccessor extends Plugin implements ReadAccessor {
+private ParquetFileReader reader;
+private MessageColumnIO columnIO;
+private RecordIterator recordIterator;
+private MessageType schema;
+
+
+private class RecordIterator implements Iterator {
+
+private final ParquetFileReader reader;
+private PageReadStore currentRowGroup;
+private RecordReader recordReader;
+private long rowsRemainedInRowGroup;
+
+public RecordIterator(ParquetFileReader reader) {
+this.reader = reader;
+readNextRowGroup();
+}
+
+@Override
+public boolean hasNext() {
+return rowsRemainedInRowGroup > 0;
+}
+
+@Override
+public OneRow next() {
+return new OneRow(null, readNextGroup());
+}
+
+@Override
+public void remove() {
+throw new UnsupportedOperationException();
+}
+
+private void readNextRowGroup() {
+try {
+currentRowGroup = reader.readNextRowGroup();
+} catch (IOException e) {
+throw new RuntimeException("Error occurred during reading
new row group", e);
+}
+if (currentRowGroup == null)
+return;
+rowsRemainedInRowGroup = currentRowGroup.getRowCount();
+recordReader = columnIO.getRecordReader(currentRowGroup, new
GroupRecordConverter(schema));
+}
+
+private Group readNextGroup() {
+Group g = null;
+if (rowsRemainedInRowGroup == 0) {
+readNextRowGroup();
+if (currentRowGroup != null) {
+g = recordReader.read();
+}
+} else {
+g = recordReader.read();
+if (g == null) {
--- End diff --
Even though could looks slightly more complex - we are saving on invoking
that method for every single record.
---
[GitHub] incubator-hawq pull request #1326: HAWQ-1575. Implemented readable Parquet p...
Github user sansanichfb commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1326#discussion_r166175619 --- Diff: pxf/pxf-hdfs/src/main/java/org/apache/hawq/pxf/plugins/hdfs/ParquetFileAccessor.java --- @@ -0,0 +1,168 @@ +package org.apache.hawq.pxf.plugins.hdfs; + +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.mapred.FileSplit; +import org.apache.hawq.pxf.api.OneRow; +import org.apache.hawq.pxf.api.ReadAccessor; +import org.apache.hawq.pxf.api.utilities.InputData; +import org.apache.hawq.pxf.api.utilities.Plugin; +import org.apache.hawq.pxf.plugins.hdfs.utilities.HdfsUtilities; + +import org.apache.parquet.column.page.PageReadStore; +import org.apache.parquet.example.data.Group; +import org.apache.parquet.example.data.simple.convert.GroupRecordConverter; +import org.apache.parquet.format.converter.ParquetMetadataConverter; +import org.apache.parquet.hadoop.ParquetFileReader; +import org.apache.parquet.io.ColumnIOFactory; +import org.apache.parquet.io.MessageColumnIO; +import org.apache.parquet.io.RecordReader; +import org.apache.parquet.schema.MessageType; + +import java.io.IOException; +import java.util.Iterator; + +/** + * Parquet file accessor. --- End diff -- Added ---
[jira] [Updated] (HAWQ-1575) Implement readable Parquet profile
[ https://issues.apache.org/jira/browse/HAWQ-1575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksandr Diachenko updated HAWQ-1575: -- Fix Version/s: (was: 2.4.0.0-incubating) 2.3.0.0-incubating > Implement readable Parquet profile > -- > > Key: HAWQ-1575 > URL: https://issues.apache.org/jira/browse/HAWQ-1575 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF >Reporter: Oleksandr Diachenko >Assignee: Ed Espino >Priority: Major > Fix For: 2.3.0.0-incubating > > > PXF should be able to read data from Parquet files stored in HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HAWQ-1575) Implement readable Parquet profile
[ https://issues.apache.org/jira/browse/HAWQ-1575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353297#comment-16353297 ] Oleksandr Diachenko commented on HAWQ-1575: --- [~yjin] updated version. > Implement readable Parquet profile > -- > > Key: HAWQ-1575 > URL: https://issues.apache.org/jira/browse/HAWQ-1575 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF >Reporter: Oleksandr Diachenko >Assignee: Ed Espino >Priority: Major > Fix For: 2.3.0.0-incubating > > > PXF should be able to read data from Parquet files stored in HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HAWQ-1575) Implement readable Parquet profile
[ https://issues.apache.org/jira/browse/HAWQ-1575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksandr Diachenko resolved HAWQ-1575. --- Resolution: Fixed > Implement readable Parquet profile > -- > > Key: HAWQ-1575 > URL: https://issues.apache.org/jira/browse/HAWQ-1575 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF >Reporter: Oleksandr Diachenko >Assignee: Ed Espino >Priority: Major > Fix For: 2.3.0.0-incubating > > > PXF should be able to read data from Parquet files stored in HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] incubator-hawq pull request #1326: HAWQ-1575. Implemented readable Parquet p...
Github user sansanichfb closed the pull request at: https://github.com/apache/incubator-hawq/pull/1326 ---
[GitHub] incubator-hawq issue #1287: HAWQ-1527 Enabled partition filtering for integr...
Github user radarwave commented on the issue: https://github.com/apache/incubator-hawq/pull/1287 @outofmem0ry The PR is merged, please close this PR. Thanks. ---
