[GitHub] incubator-hawq issue #989: HAWQ-1129. Install PLR into hawq home directory
Github user paul-guo- commented on the issue: https://github.com/apache/incubator-hawq/pull/989 Seems that hawq_version is not needed, so it could be removed safely in Makefile. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (HAWQ-1130) Make HCatalog integration work with non-superusers
[ https://issues.apache.org/jira/browse/HAWQ-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15624380#comment-15624380 ] Goden Yao commented on HAWQ-1130: - [~nhorn] [~jimmida] may know the history and rationale behind that. > Make HCatalog integration work with non-superusers > -- > > Key: HAWQ-1130 > URL: https://issues.apache.org/jira/browse/HAWQ-1130 > Project: Apache HAWQ > Issue Type: Improvement > Components: PXF >Reporter: Oleksandr Diachenko >Assignee: Oleksandr Diachenko > Fix For: 2.0.1.0-incubating > > > According to current implementation user who uses HCatalog integration > feature should have SELECT privileges for pg_authid, pg_user_mapping tables. > It's fine for superusers but we shouldn't expose them to non-superusers > because they store hashed user passwords. > Basically, the problem is how to determine max oid among all oid-having > tables. > Possible solutions: > * Creating view returning max oid and grant select privilege to public. > ** Cons: > *** Requires catalog upgrade; > * Reading current oid from shared memory. > ** Pros: > *** No catalog upgrade needed. > ** Cons: > *** Additional exclusive locks needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1130) Make HCatalog integration work with non-superusers
[ https://issues.apache.org/jira/browse/HAWQ-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Goden Yao updated HAWQ-1130: Assignee: Oleksandr Diachenko (was: Lei Chang) > Make HCatalog integration work with non-superusers > -- > > Key: HAWQ-1130 > URL: https://issues.apache.org/jira/browse/HAWQ-1130 > Project: Apache HAWQ > Issue Type: Improvement > Components: PXF >Reporter: Oleksandr Diachenko >Assignee: Oleksandr Diachenko > Fix For: 2.0.1.0-incubating > > > According to current implementation user who uses HCatalog integration > feature should have SELECT privileges for pg_authid, pg_user_mapping tables. > It's fine for superusers but we shouldn't expose them to non-superusers > because they store hashed user passwords. > Basically, the problem is how to determine max oid among all oid-having > tables. > Possible solutions: > * Creating view returning max oid and grant select privilege to public. > ** Cons: > *** Requires catalog upgrade; > * Reading current oid from shared memory. > ** Pros: > *** No catalog upgrade needed. > ** Cons: > *** Additional exclusive locks needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1035) support partition table register
[ https://issues.apache.org/jira/browse/HAWQ-1035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lili Ma updated HAWQ-1035: -- Assignee: Chunling Wang (was: Hubert Zhang) > support partition table register > > > Key: HAWQ-1035 > URL: https://issues.apache.org/jira/browse/HAWQ-1035 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Command Line Tools >Reporter: Lili Ma >Assignee: Chunling Wang > Fix For: 2.0.1.0-incubating > > > Support partition table register, limited to 1 level partition table, since > hawq extract only supports 1-level partition table. > Expected behavior: > 1. Create a partition table in HAWQ, then extract the information out to .yml > file > 2. Call hawq register and specify identified .yml file and a new table name, > the files should be registered into the new table. > Work can be detailed down to implement partition table register: > 1. modify .yml configuration file parsing function, add content for partition > table. > 2. construct partition table DDL regards to .yml configuration file > 3. map sub partition table name to the table list in .yml configuration file > 4. register the subpartition table one by one -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HAWQ-1034) add --repair option for hawq register
[ https://issues.apache.org/jira/browse/HAWQ-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15624133#comment-15624133 ] Lili Ma edited comment on HAWQ-1034 at 11/1/16 2:44 AM: Repair mode can be thought of particular case of force mode. 1) Force mode registers the files according to yaml configuration file, erase all the records in catalog (pg_aoseg.pg_aoseg(paqseg)_$relid) and re-implement catalog insert. It requires HDFS files for the table be included in yaml configuation file. 2) Repair mode also registers files according to yaml configuration file, erase the catalog records and re-insert. But it doesn't require all the HDFS files for the table be included in yaml configuration file. It will directly delete those files which are under the table directory but not included in yaml configuration file. Since repair mode may directly deleting HDFS files, say, if user uses repair mode by mistake, his/her data may be deleted, it may bring some risks. We can allow them to use force mode, and throw error for files under the directory but not included in yaml configuration file. If user does think the files are unnecessary, he/she can delete the files by himself/herself. The workaround for supporting repair mode use --force option: 1) If there is no added files since last checkpoint where the yaml configuration file is generated, force mode can directly handle it. 2) If there are some added files since last checkpoint which the user does want to delete, we can output those file information in force mode so that users can delete those files by themselves and then do register force mode again. Since we can use force mode to implement repair feature, we will remove existing code for repair mode and close this JIRA. Thanks was (Author: lilima): Repair mode can be thought of particular case of force mode. 1) Force mode registers the files according to yaml configuration file, erase all the records in catalog (pg_aoseg.pg_aoseg(paqseg)_$relid) and re-implement catalog insert. It requires HDFS files for the table be included in yaml configuation file. 2) Repair mode also registers files according to yaml configuration file, erase the catalog records and re-insert. But it doesn't require all the HDFS files for the table be included in yaml configuration file. It will directly delete those files which are under the table directory but not included in yaml configuration file. I'm a little concerned about directly deleting HDFS files, say, if user uses repair mode by mistake, his/her data may be deleted. So, what if we just allow them to use force mode, and throw error for files under the directory but not included in yaml configuration file. If user does think the files are unnecessary, he/she can delete the files by himself/herself. The workaround for supporting repair mode use --force option: 1) If there is no added files since last checkpoint where the yaml configuration file is generated, force mode can directly handle it. 2) If there are some added files since last checkpoint which the user does want to delete, we can output those file information in force mode so that users can delete those files by themselves and then do register force mode again. Since we can use force mode to implement repair feature, we will remove existing code for repair mode and close this JIRA. Thanks > add --repair option for hawq register > - > > Key: HAWQ-1034 > URL: https://issues.apache.org/jira/browse/HAWQ-1034 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Command Line Tools >Affects Versions: 2.0.1.0-incubating >Reporter: Lili Ma >Assignee: Chunling Wang > Fix For: 2.0.1.0-incubating > > > add --repair option for hawq register > Will change both file folder and catalog table pg_aoseg.pg_paqseg_$relid to > the state which .yml file configures. Note may some new generated files since > the checkpoint may be deleted here. Also note the all the files in .yml file > should all under the table folder on HDFS. Limitation: Do not support cases > for hash table redistribution, table truncate and table drop. This is for > scenario rollback of table: Do checkpoints somewhere, and need to rollback to > previous checkpoint. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HAWQ-1034) add --repair option for hawq register
[ https://issues.apache.org/jira/browse/HAWQ-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lili Ma resolved HAWQ-1034. --- Resolution: Done > add --repair option for hawq register > - > > Key: HAWQ-1034 > URL: https://issues.apache.org/jira/browse/HAWQ-1034 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Command Line Tools >Affects Versions: 2.0.1.0-incubating >Reporter: Lili Ma >Assignee: Chunling Wang > Fix For: 2.0.1.0-incubating > > > add --repair option for hawq register > Will change both file folder and catalog table pg_aoseg.pg_paqseg_$relid to > the state which .yml file configures. Note may some new generated files since > the checkpoint may be deleted here. Also note the all the files in .yml file > should all under the table folder on HDFS. Limitation: Do not support cases > for hash table redistribution, table truncate and table drop. This is for > scenario rollback of table: Do checkpoints somewhere, and need to rollback to > previous checkpoint. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-1034) add --repair option for hawq register
[ https://issues.apache.org/jira/browse/HAWQ-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15624133#comment-15624133 ] Lili Ma commented on HAWQ-1034: --- Repair mode can be thought of particular case of force mode. 1) Force mode registers the files according to yaml configuration file, erase all the records in catalog (pg_aoseg.pg_aoseg(paqseg)_$relid) and re-implement catalog insert. It requires HDFS files for the table be included in yaml configuation file. 2) Repair mode also registers files according to yaml configuration file, erase the catalog records and re-insert. But it doesn't require all the HDFS files for the table be included in yaml configuration file. It will directly delete those files which are under the table directory but not included in yaml configuration file. I'm a little concerned about directly deleting HDFS files, say, if user uses repair mode by mistake, his/her data may be deleted. So, what if we just allow them to use force mode, and throw error for files under the directory but not included in yaml configuration file. If user does think the files are unnecessary, he/she can delete the files by himself/herself. The workaround for supporting repair mode use --force option: 1) If there is no added files since last checkpoint where the yaml configuration file is generated, force mode can directly handle it. 2) If there are some added files since last checkpoint which the user does want to delete, we can output those file information in force mode so that users can delete those files by themselves and then do register force mode again. Since we can use force mode to implement repair feature, we will remove existing code for repair mode and close this JIRA. Thanks > add --repair option for hawq register > - > > Key: HAWQ-1034 > URL: https://issues.apache.org/jira/browse/HAWQ-1034 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Command Line Tools >Affects Versions: 2.0.1.0-incubating >Reporter: Lili Ma >Assignee: Chunling Wang > Fix For: 2.0.1.0-incubating > > > add --repair option for hawq register > Will change both file folder and catalog table pg_aoseg.pg_paqseg_$relid to > the state which .yml file configures. Note may some new generated files since > the checkpoint may be deleted here. Also note the all the files in .yml file > should all under the table folder on HDFS. Limitation: Do not support cases > for hash table redistribution, table truncate and table drop. This is for > scenario rollback of table: Do checkpoints somewhere, and need to rollback to > previous checkpoint. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin
Github user jiadexin commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/972#discussion_r85867311
--- Diff:
pxf/pxf-jdbc/src/main/java/org/apache/hawq/pxf/plugins/jdbc/utils/ByteUtil.java
---
@@ -0,0 +1,86 @@
+package org.apache.hawq.pxf.plugins.jdbc.utils;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+/**
+ * A tool class, used to deal with byte array merging, split and other
methods.
+ */
+public class ByteUtil {
+
+public static byte[] mergeBytes(byte[] b1, byte[] b2) {
--- End diff --
This method is simple, I do not want to import a dependency.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #989: HAWQ-1129. Install PLR into hawq home dire...
Github user radarwave closed the pull request at: https://github.com/apache/incubator-hawq/pull/989 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin
Github user jiadexin commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/972#discussion_r85867986
--- Diff:
pxf/pxf-jdbc/src/main/java/org/apache/hawq/pxf/plugins/jdbc/utils/ByteUtil.java
---
@@ -0,0 +1,86 @@
+package org.apache.hawq.pxf.plugins.jdbc.utils;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+/**
+ * A tool class, used to deal with byte array merging, split and other
methods.
+ */
+public class ByteUtil {
+
+public static byte[] mergeBytes(byte[] b1, byte[] b2) {
--- End diff --
This method is simple, I do not want to import a dependency.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Updated] (HAWQ-1130) Make HCatalog integration work with non-superusers
[ https://issues.apache.org/jira/browse/HAWQ-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksandr Diachenko updated HAWQ-1130: -- Description: According to current implementation user who uses HCatalog integration feature should have SELECT privileges for pg_authid, pg_user_mapping tables. It's fine for superusers but we shouldn't expose them to non-superusers because they store hashed user passwords. Basically, the problem is how to determine max oid among all oid-having tables. Possible solutions: * Creating view returning max oid and grant select privilege to public. ** Cons: *** Requires catalog upgrade; * Reading current oid from shared memory. ** Pros: *** No catalog upgrade needed. ** Cons: *** Additional exclusive locks needed. was: According to current implementation user who uses HCatalog integration feature should have SELECT privileges for pg_authid, pg_user_mapping tables. It's fine for superusers but we shouldn't expose them to non-superusers because they store hashed user passwords. Basically, the problem is how to determine max oid among all oid-having tables. Possible solutions: * Creating view returning max oid and grant select privilege to public. ** Cons: *** Requires catalog upgrade; > Make HCatalog integration work with non-superusers > -- > > Key: HAWQ-1130 > URL: https://issues.apache.org/jira/browse/HAWQ-1130 > Project: Apache HAWQ > Issue Type: Improvement > Components: PXF >Reporter: Oleksandr Diachenko >Assignee: Lei Chang > Fix For: 2.0.1.0-incubating > > > According to current implementation user who uses HCatalog integration > feature should have SELECT privileges for pg_authid, pg_user_mapping tables. > It's fine for superusers but we shouldn't expose them to non-superusers > because they store hashed user passwords. > Basically, the problem is how to determine max oid among all oid-having > tables. > Possible solutions: > * Creating view returning max oid and grant select privilege to public. > ** Cons: > *** Requires catalog upgrade; > * Reading current oid from shared memory. > ** Pros: > *** No catalog upgrade needed. > ** Cons: > *** Additional exclusive locks needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1130) Make HCatalog integration work with non-superusers
[ https://issues.apache.org/jira/browse/HAWQ-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksandr Diachenko updated HAWQ-1130: -- Description: According to current implementation user who uses HCatalog integration feature should have SELECT privileges for pg_authid, pg_user_mapping tables. It's fine for superusers but we shouldn't expose them to non-superusers because they store hashed user passwords. Basically, the problem is how to determine max oid among all oid-having tables. Possible solutions: * Creating view returning max oid and grant select privilege to public. ** Cons: *** Requires catalog upgrade; was: According to current implementation user who uses HCatalog integration feature should have SELECT privileges for pg_authid, pg_user_mapping tables. It's fine for superusers but we shouldn't expose them to non-superusers because they store hashed user passwords. > Make HCatalog integration work with non-superusers > -- > > Key: HAWQ-1130 > URL: https://issues.apache.org/jira/browse/HAWQ-1130 > Project: Apache HAWQ > Issue Type: Improvement > Components: PXF >Reporter: Oleksandr Diachenko >Assignee: Lei Chang > Fix For: 2.0.1.0-incubating > > > According to current implementation user who uses HCatalog integration > feature should have SELECT privileges for pg_authid, pg_user_mapping tables. > It's fine for superusers but we shouldn't expose them to non-superusers > because they store hashed user passwords. > Basically, the problem is how to determine max oid among all oid-having > tables. > Possible solutions: > * Creating view returning max oid and grant select privilege to public. > ** Cons: > *** Requires catalog upgrade; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1130) Make HCatalog integration work with non-superusers
[ https://issues.apache.org/jira/browse/HAWQ-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksandr Diachenko updated HAWQ-1130: -- Description: According to current implementation user who uses HCatalog integration feature should have SELECT privileges for pg_authid, pg_user_mapping tables. It's fine for superusers but we shouldn't expose them to non-superusers because they store hashed user passwords. > Make HCatalog integration work with non-superusers > -- > > Key: HAWQ-1130 > URL: https://issues.apache.org/jira/browse/HAWQ-1130 > Project: Apache HAWQ > Issue Type: Improvement > Components: PXF >Reporter: Oleksandr Diachenko >Assignee: Lei Chang > Fix For: 2.0.1.0-incubating > > > According to current implementation user who uses HCatalog integration > feature should have SELECT privileges for pg_authid, pg_user_mapping tables. > It's fine for superusers but we shouldn't expose them to non-superusers > because they store hashed user passwords. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HAWQ-1130) Make HCatalog integration work with non-superusers
Oleksandr Diachenko created HAWQ-1130: - Summary: Make HCatalog integration work with non-superusers Key: HAWQ-1130 URL: https://issues.apache.org/jira/browse/HAWQ-1130 Project: Apache HAWQ Issue Type: Improvement Components: PXF Reporter: Oleksandr Diachenko Assignee: Lei Chang -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1130) Make HCatalog integration work with non-superusers
[ https://issues.apache.org/jira/browse/HAWQ-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksandr Diachenko updated HAWQ-1130: -- Fix Version/s: 2.0.1.0-incubating > Make HCatalog integration work with non-superusers > -- > > Key: HAWQ-1130 > URL: https://issues.apache.org/jira/browse/HAWQ-1130 > Project: Apache HAWQ > Issue Type: Improvement > Components: PXF >Reporter: Oleksandr Diachenko >Assignee: Lei Chang > Fix For: 2.0.1.0-incubating > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-1119) create new documentation topic for PXF writable profiles
[
https://issues.apache.org/jira/browse/HAWQ-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15623629#comment-15623629
]
ASF GitHub Bot commented on HAWQ-1119:
--
Github user dyozie commented on a diff in the pull request:
https://github.com/apache/incubator-hawq-docs/pull/46#discussion_r85845296
--- Diff: pxf/HDFSWritablePXF.html.md.erb ---
@@ -0,0 +1,410 @@
+---
+title: Writing Data to HDFS
+---
+
+The PXF HDFS plug-in supports writable external tables using the
`HdfsTextSimple` and `SequenceWritable` profiles. You might create a writable
table to export data from a HAWQ internal table to HDFS.
+
+This section describes how to use these PXF profiles to create writable
external tables.
+
+**Note**: You cannot directly query data in a HAWQ writable table. After
creating the external writable table, you must create a HAWQ readable external
table accessing the HDFS file, then query that table. ??You can also create a
Hive table to access the HDFS file.??
+
+## Prerequisites
+
+Before working with HDFS file data using HAWQ and PXF, ensure that:
+
+- The HDFS plug-in is installed on all cluster nodes. See [Installing
PXF Plug-ins](InstallPXFPlugins.html) for PXF plug-in installation information.
+- All HDFS users have read permissions to HDFS services and that write
permissions have been restricted to specific users.
+
+## Writing to PXF External Tables
+The PXF HDFS plug-in supports writable two profiles: `HdfsTextSimple` and
`SequenceWritable`.
+
+Use the following syntax to create a HAWQ external writable table
representing HDFS data:
+
+``` sql
+CREATE EXTERNAL WRITABLE TABLE
+( [, ...] | LIKE )
+LOCATION ('pxf://[:]/
+
?PROFILE=HdfsTextSimple|SequenceWritable[&=[...]]')
+FORMAT '[TEXT|CSV|CUSTOM]' ();
+```
+
+HDFS-plug-in-specific keywords and values used in the [CREATE EXTERNAL
TABLE](../reference/sql/CREATE-EXTERNAL-TABLE.html) call are described in the
table below.
+
+| Keyword | Value |
+|---|-|
+| \[:\]| The HDFS NameNode and port. |
+| \| The path to the file in the HDFS data store. |
+| PROFILE| The `PROFILE` keyword must specify one of the values
`HdfsTextSimple` or `SequenceWritable`. |
+| \ | \ is profile-specific. These
options are discussed in the next topic.|
+| FORMAT 'TEXT' | Use '`TEXT`' `FORMAT` with the `HdfsTextSimple` profile
when \ will reference a plain text delimited file. The
`HdfsTextSimple` '`TEXT`' `FORMAT` supports only the built-in
`(delimiter=)` \. |
+| FORMAT 'CSV' | Use '`CSV`' `FORMAT` with `HdfsTextSimple` when
\ will reference a comma-separated value file. |
+| FORMAT 'CUSTOM' | Use the `'CUSTOM'` `FORMAT` with the
`SequenceWritable` profile. The `SequenceWritable` '`CUSTOM`' `FORMAT` supports
only the built-in `(formatter='pxfwritable_export)` (write) and
`(formatter='pxfwritable_import)` (read) \.
+
+**Note**: When creating PXF external tables, you cannot use the `HEADER`
option in your `FORMAT` specification.
+
+## Custom Options
+
+The `HdfsTextSimple` and `SequenceWritable` profiles support the following
\:
+
+| Keyword | Value Description |
+|---|-|
+| COMPRESSION_CODEC| The compression codec Java class name. If this
option is not provided, no data compression is performed. Supported compression
codecs include: `org.apache.hadoop.io.compress.DefaultCodec`,
`org.apache.hadoop.io.compress.BZip2Codec`, and
`org.apache.hadoop.io.compress.GzipCodec` (`HdfsTextSimple` profile only) |
+| COMPRESSION_TYPE| The compression type to employ; supported values
are `RECORD` (the default) or `BLOCK`. |
+| DATA-SCHEMA| (`SequenceWritable` profile only) The name of the
writer serialization/deserialization class. The jar file in which this class
resides must be in the PXF class path. This option has no default value. |
+| THREAD-SAFE | Boolean value determining if a table query can run in
multi-thread mode. Default value is `TRUE`, requests run in multi-threaded
mode. When set to `FALSE`, requests will be handled in a single thread.
`THREAD-SAFE` should be set appropriately when operations that are not
thread-safe are performed (i.e. compression). |
+
+## HdfsTextSimple Profile
+
+Use the `HdfsTextSimple` profile when writing delimited data to a plain
text file where each row is a single record.
+
+Writable tables created using the `HdfsTextSimple` profile can use no,
record, or block compression. When compression is used, the default, gzip, and
bzip2 Hadoop compression codecs are supported:
+
+- org.apache.hadoop.io.compress.DefaultCodec
+- org.apache.hadoop.io.compres
[jira] [Commented] (HAWQ-1119) create new documentation topic for PXF writable profiles
[ https://issues.apache.org/jira/browse/HAWQ-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15623627#comment-15623627 ] ASF GitHub Bot commented on HAWQ-1119: -- Github user dyozie commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/46#discussion_r85791234 --- Diff: pxf/HDFSWritablePXF.html.md.erb --- @@ -0,0 +1,410 @@ +--- +title: Writing Data to HDFS +--- + +The PXF HDFS plug-in supports writable external tables using the `HdfsTextSimple` and `SequenceWritable` profiles. You might create a writable table to export data from a HAWQ internal table to HDFS. + +This section describes how to use these PXF profiles to create writable external tables. + +**Note**: You cannot directly query data in a HAWQ writable table. After creating the external writable table, you must create a HAWQ readable external table accessing the HDFS file, then query that table. ??You can also create a Hive table to access the HDFS file.?? + +## Prerequisites + +Before working with HDFS file data using HAWQ and PXF, ensure that: + +- The HDFS plug-in is installed on all cluster nodes. See [Installing PXF Plug-ins](InstallPXFPlugins.html) for PXF plug-in installation information. +- All HDFS users have read permissions to HDFS services and that write permissions have been restricted to specific users. --- End diff -- Break the last requirement into a separate bullet: Write permissions should be provided to a restricted set of user. > create new documentation topic for PXF writable profiles > > > Key: HAWQ-1119 > URL: https://issues.apache.org/jira/browse/HAWQ-1119 > Project: Apache HAWQ > Issue Type: Improvement > Components: Documentation >Reporter: Lisa Owen >Assignee: David Yozie > Fix For: 2.0.1.0-incubating > > > certain profiles supported by the existing PXF plug-ins support writable > tables. create some documentation content for these profiles. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-1119) create new documentation topic for PXF writable profiles
[ https://issues.apache.org/jira/browse/HAWQ-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15623620#comment-15623620 ] ASF GitHub Bot commented on HAWQ-1119: -- Github user dyozie commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/46#discussion_r85790837 --- Diff: pxf/HDFSWritablePXF.html.md.erb --- @@ -0,0 +1,410 @@ +--- +title: Writing Data to HDFS +--- + +The PXF HDFS plug-in supports writable external tables using the `HdfsTextSimple` and `SequenceWritable` profiles. You might create a writable table to export data from a HAWQ internal table to HDFS. + +This section describes how to use these PXF profiles to create writable external tables. + +**Note**: You cannot directly query data in a HAWQ writable table. After creating the external writable table, you must create a HAWQ readable external table accessing the HDFS file, then query that table. ??You can also create a Hive table to access the HDFS file.?? --- End diff -- This note could use some editing. Something like: Tables that you create with the SequenceWritable profile can only be used for INSERT operations. If you want to query the inserted data, you must define a separate external table that references the new HDFS file, using the or ??? profiles. > create new documentation topic for PXF writable profiles > > > Key: HAWQ-1119 > URL: https://issues.apache.org/jira/browse/HAWQ-1119 > Project: Apache HAWQ > Issue Type: Improvement > Components: Documentation >Reporter: Lisa Owen >Assignee: David Yozie > Fix For: 2.0.1.0-incubating > > > certain profiles supported by the existing PXF plug-ins support writable > tables. create some documentation content for these profiles. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-1119) create new documentation topic for PXF writable profiles
[
https://issues.apache.org/jira/browse/HAWQ-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15623613#comment-15623613
]
ASF GitHub Bot commented on HAWQ-1119:
--
Github user dyozie commented on a diff in the pull request:
https://github.com/apache/incubator-hawq-docs/pull/46#discussion_r85813059
--- Diff: pxf/HDFSWritablePXF.html.md.erb ---
@@ -0,0 +1,410 @@
+---
+title: Writing Data to HDFS
+---
+
+The PXF HDFS plug-in supports writable external tables using the
`HdfsTextSimple` and `SequenceWritable` profiles. You might create a writable
table to export data from a HAWQ internal table to HDFS.
+
+This section describes how to use these PXF profiles to create writable
external tables.
+
+**Note**: You cannot directly query data in a HAWQ writable table. After
creating the external writable table, you must create a HAWQ readable external
table accessing the HDFS file, then query that table. ??You can also create a
Hive table to access the HDFS file.??
+
+## Prerequisites
+
+Before working with HDFS file data using HAWQ and PXF, ensure that:
+
+- The HDFS plug-in is installed on all cluster nodes. See [Installing
PXF Plug-ins](InstallPXFPlugins.html) for PXF plug-in installation information.
+- All HDFS users have read permissions to HDFS services and that write
permissions have been restricted to specific users.
+
+## Writing to PXF External Tables
+The PXF HDFS plug-in supports writable two profiles: `HdfsTextSimple` and
`SequenceWritable`.
+
+Use the following syntax to create a HAWQ external writable table
representing HDFS data:
+
+``` sql
+CREATE EXTERNAL WRITABLE TABLE
+( [, ...] | LIKE )
+LOCATION ('pxf://[:]/
+
?PROFILE=HdfsTextSimple|SequenceWritable[&=[...]]')
+FORMAT '[TEXT|CSV|CUSTOM]' ();
+```
+
+HDFS-plug-in-specific keywords and values used in the [CREATE EXTERNAL
TABLE](../reference/sql/CREATE-EXTERNAL-TABLE.html) call are described in the
table below.
+
+| Keyword | Value |
+|---|-|
+| \[:\]| The HDFS NameNode and port. |
+| \| The path to the file in the HDFS data store. |
+| PROFILE| The `PROFILE` keyword must specify one of the values
`HdfsTextSimple` or `SequenceWritable`. |
+| \ | \ is profile-specific. These
options are discussed in the next topic.|
+| FORMAT 'TEXT' | Use '`TEXT`' `FORMAT` with the `HdfsTextSimple` profile
when \ will reference a plain text delimited file. The
`HdfsTextSimple` '`TEXT`' `FORMAT` supports only the built-in
`(delimiter=)` \. |
+| FORMAT 'CSV' | Use '`CSV`' `FORMAT` with `HdfsTextSimple` when
\ will reference a comma-separated value file. |
+| FORMAT 'CUSTOM' | Use the `'CUSTOM'` `FORMAT` with the
`SequenceWritable` profile. The `SequenceWritable` '`CUSTOM`' `FORMAT` supports
only the built-in `(formatter='pxfwritable_export)` (write) and
`(formatter='pxfwritable_import)` (read) \.
+
+**Note**: When creating PXF external tables, you cannot use the `HEADER`
option in your `FORMAT` specification.
+
+## Custom Options
+
+The `HdfsTextSimple` and `SequenceWritable` profiles support the following
\:
+
+| Keyword | Value Description |
+|---|-|
+| COMPRESSION_CODEC| The compression codec Java class name. If this
option is not provided, no data compression is performed. Supported compression
codecs include: `org.apache.hadoop.io.compress.DefaultCodec`,
`org.apache.hadoop.io.compress.BZip2Codec`, and
`org.apache.hadoop.io.compress.GzipCodec` (`HdfsTextSimple` profile only) |
+| COMPRESSION_TYPE| The compression type to employ; supported values
are `RECORD` (the default) or `BLOCK`. |
+| DATA-SCHEMA| (`SequenceWritable` profile only) The name of the
writer serialization/deserialization class. The jar file in which this class
resides must be in the PXF class path. This option has no default value. |
+| THREAD-SAFE | Boolean value determining if a table query can run in
multi-thread mode. Default value is `TRUE`, requests run in multi-threaded
mode. When set to `FALSE`, requests will be handled in a single thread.
`THREAD-SAFE` should be set appropriately when operations that are not
thread-safe are performed (i.e. compression). |
+
+## HdfsTextSimple Profile
+
+Use the `HdfsTextSimple` profile when writing delimited data to a plain
text file where each row is a single record.
+
+Writable tables created using the `HdfsTextSimple` profile can use no,
record, or block compression. When compression is used, the default, gzip, and
bzip2 Hadoop compression codecs are supported:
--- End diff --
Small edit: Writable tables created using the HdfsTextSimple
[jira] [Commented] (HAWQ-1119) create new documentation topic for PXF writable profiles
[ https://issues.apache.org/jira/browse/HAWQ-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15623611#comment-15623611 ] ASF GitHub Bot commented on HAWQ-1119: -- Github user dyozie commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/46#discussion_r85789703 --- Diff: pxf/HDFSWritablePXF.html.md.erb --- @@ -0,0 +1,410 @@ +--- +title: Writing Data to HDFS +--- + +The PXF HDFS plug-in supports writable external tables using the `HdfsTextSimple` and `SequenceWritable` profiles. You might create a writable table to export data from a HAWQ internal table to HDFS. --- End diff -- Seems like this last sentence should say something other than "to HDFS", since internal tables are already stored in HDFS but in HAWQ proprietary format. Not sure what the best wording is - HDFS text files/flat files? > create new documentation topic for PXF writable profiles > > > Key: HAWQ-1119 > URL: https://issues.apache.org/jira/browse/HAWQ-1119 > Project: Apache HAWQ > Issue Type: Improvement > Components: Documentation >Reporter: Lisa Owen >Assignee: David Yozie > Fix For: 2.0.1.0-incubating > > > certain profiles supported by the existing PXF plug-ins support writable > tables. create some documentation content for these profiles. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-1119) create new documentation topic for PXF writable profiles
[
https://issues.apache.org/jira/browse/HAWQ-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15623625#comment-15623625
]
ASF GitHub Bot commented on HAWQ-1119:
--
Github user dyozie commented on a diff in the pull request:
https://github.com/apache/incubator-hawq-docs/pull/46#discussion_r85793349
--- Diff: pxf/HDFSWritablePXF.html.md.erb ---
@@ -0,0 +1,410 @@
+---
+title: Writing Data to HDFS
+---
+
+The PXF HDFS plug-in supports writable external tables using the
`HdfsTextSimple` and `SequenceWritable` profiles. You might create a writable
table to export data from a HAWQ internal table to HDFS.
+
+This section describes how to use these PXF profiles to create writable
external tables.
+
+**Note**: You cannot directly query data in a HAWQ writable table. After
creating the external writable table, you must create a HAWQ readable external
table accessing the HDFS file, then query that table. ??You can also create a
Hive table to access the HDFS file.??
+
+## Prerequisites
+
+Before working with HDFS file data using HAWQ and PXF, ensure that:
+
+- The HDFS plug-in is installed on all cluster nodes. See [Installing
PXF Plug-ins](InstallPXFPlugins.html) for PXF plug-in installation information.
+- All HDFS users have read permissions to HDFS services and that write
permissions have been restricted to specific users.
+
+## Writing to PXF External Tables
+The PXF HDFS plug-in supports writable two profiles: `HdfsTextSimple` and
`SequenceWritable`.
+
+Use the following syntax to create a HAWQ external writable table
representing HDFS data:
+
+``` sql
+CREATE EXTERNAL WRITABLE TABLE
+( [, ...] | LIKE )
+LOCATION ('pxf://[:]/
+
?PROFILE=HdfsTextSimple|SequenceWritable[&=[...]]')
+FORMAT '[TEXT|CSV|CUSTOM]' ();
+```
+
+HDFS-plug-in-specific keywords and values used in the [CREATE EXTERNAL
TABLE](../reference/sql/CREATE-EXTERNAL-TABLE.html) call are described in the
table below.
+
+| Keyword | Value |
+|---|-|
+| \[:\]| The HDFS NameNode and port. |
+| \| The path to the file in the HDFS data store. |
+| PROFILE| The `PROFILE` keyword must specify one of the values
`HdfsTextSimple` or `SequenceWritable`. |
+| \ | \ is profile-specific. These
options are discussed in the next topic.|
+| FORMAT 'TEXT' | Use '`TEXT`' `FORMAT` with the `HdfsTextSimple` profile
when \ will reference a plain text delimited file. The
`HdfsTextSimple` '`TEXT`' `FORMAT` supports only the built-in
`(delimiter=)` \. |
--- End diff --
Change the wording of this row and the next to something like: Use
'`TEXT`' `FORMAT` with the `HdfsTextSimple` profile to create a
plain-text-delimited file at the specified \.
(same for CSV)
> create new documentation topic for PXF writable profiles
>
>
> Key: HAWQ-1119
> URL: https://issues.apache.org/jira/browse/HAWQ-1119
> Project: Apache HAWQ
> Issue Type: Improvement
> Components: Documentation
>Reporter: Lisa Owen
>Assignee: David Yozie
> Fix For: 2.0.1.0-incubating
>
>
> certain profiles supported by the existing PXF plug-ins support writable
> tables. create some documentation content for these profiles.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HAWQ-1119) create new documentation topic for PXF writable profiles
[
https://issues.apache.org/jira/browse/HAWQ-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15623621#comment-15623621
]
ASF GitHub Bot commented on HAWQ-1119:
--
Github user dyozie commented on a diff in the pull request:
https://github.com/apache/incubator-hawq-docs/pull/46#discussion_r85794550
--- Diff: pxf/HDFSWritablePXF.html.md.erb ---
@@ -0,0 +1,410 @@
+---
+title: Writing Data to HDFS
+---
+
+The PXF HDFS plug-in supports writable external tables using the
`HdfsTextSimple` and `SequenceWritable` profiles. You might create a writable
table to export data from a HAWQ internal table to HDFS.
+
+This section describes how to use these PXF profiles to create writable
external tables.
+
+**Note**: You cannot directly query data in a HAWQ writable table. After
creating the external writable table, you must create a HAWQ readable external
table accessing the HDFS file, then query that table. ??You can also create a
Hive table to access the HDFS file.??
+
+## Prerequisites
+
+Before working with HDFS file data using HAWQ and PXF, ensure that:
+
+- The HDFS plug-in is installed on all cluster nodes. See [Installing
PXF Plug-ins](InstallPXFPlugins.html) for PXF plug-in installation information.
+- All HDFS users have read permissions to HDFS services and that write
permissions have been restricted to specific users.
+
+## Writing to PXF External Tables
+The PXF HDFS plug-in supports writable two profiles: `HdfsTextSimple` and
`SequenceWritable`.
+
+Use the following syntax to create a HAWQ external writable table
representing HDFS data:
+
+``` sql
+CREATE EXTERNAL WRITABLE TABLE
+( [, ...] | LIKE )
+LOCATION ('pxf://[:]/
+
?PROFILE=HdfsTextSimple|SequenceWritable[&=[...]]')
+FORMAT '[TEXT|CSV|CUSTOM]' ();
+```
+
+HDFS-plug-in-specific keywords and values used in the [CREATE EXTERNAL
TABLE](../reference/sql/CREATE-EXTERNAL-TABLE.html) call are described in the
table below.
+
+| Keyword | Value |
+|---|-|
+| \[:\]| The HDFS NameNode and port. |
+| \| The path to the file in the HDFS data store. |
+| PROFILE| The `PROFILE` keyword must specify one of the values
`HdfsTextSimple` or `SequenceWritable`. |
+| \ | \ is profile-specific. These
options are discussed in the next topic.|
+| FORMAT 'TEXT' | Use '`TEXT`' `FORMAT` with the `HdfsTextSimple` profile
when \ will reference a plain text delimited file. The
`HdfsTextSimple` '`TEXT`' `FORMAT` supports only the built-in
`(delimiter=)` \. |
+| FORMAT 'CSV' | Use '`CSV`' `FORMAT` with `HdfsTextSimple` when
\ will reference a comma-separated value file. |
+| FORMAT 'CUSTOM' | Use the `'CUSTOM'` `FORMAT` with the
`SequenceWritable` profile. The `SequenceWritable` '`CUSTOM`' `FORMAT` supports
only the built-in `(formatter='pxfwritable_export)` (write) and
`(formatter='pxfwritable_import)` (read) \.
+
+**Note**: When creating PXF external tables, you cannot use the `HEADER`
option in your `FORMAT` specification.
+
+## Custom Options
+
+The `HdfsTextSimple` and `SequenceWritable` profiles support the following
\:
+
+| Keyword | Value Description |
+|---|-|
+| COMPRESSION_CODEC| The compression codec Java class name. If this
option is not provided, no data compression is performed. Supported compression
codecs include: `org.apache.hadoop.io.compress.DefaultCodec`,
`org.apache.hadoop.io.compress.BZip2Codec`, and
`org.apache.hadoop.io.compress.GzipCodec` (`HdfsTextSimple` profile only) |
+| COMPRESSION_TYPE| The compression type to employ; supported values
are `RECORD` (the default) or `BLOCK`. |
+| DATA-SCHEMA| (`SequenceWritable` profile only) The name of the
writer serialization/deserialization class. The jar file in which this class
resides must be in the PXF class path. This option has no default value. |
--- End diff --
Is DATA-SCHEMA an option, or is it required?
> create new documentation topic for PXF writable profiles
>
>
> Key: HAWQ-1119
> URL: https://issues.apache.org/jira/browse/HAWQ-1119
> Project: Apache HAWQ
> Issue Type: Improvement
> Components: Documentation
>Reporter: Lisa Owen
>Assignee: David Yozie
> Fix For: 2.0.1.0-incubating
>
>
> certain profiles supported by the existing PXF plug-ins support writable
> tables. create some documentation content for these profiles.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HAWQ-1119) create new documentation topic for PXF writable profiles
[
https://issues.apache.org/jira/browse/HAWQ-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15623624#comment-15623624
]
ASF GitHub Bot commented on HAWQ-1119:
--
Github user dyozie commented on a diff in the pull request:
https://github.com/apache/incubator-hawq-docs/pull/46#discussion_r85793887
--- Diff: pxf/HDFSWritablePXF.html.md.erb ---
@@ -0,0 +1,410 @@
+---
+title: Writing Data to HDFS
+---
+
+The PXF HDFS plug-in supports writable external tables using the
`HdfsTextSimple` and `SequenceWritable` profiles. You might create a writable
table to export data from a HAWQ internal table to HDFS.
+
+This section describes how to use these PXF profiles to create writable
external tables.
+
+**Note**: You cannot directly query data in a HAWQ writable table. After
creating the external writable table, you must create a HAWQ readable external
table accessing the HDFS file, then query that table. ??You can also create a
Hive table to access the HDFS file.??
+
+## Prerequisites
+
+Before working with HDFS file data using HAWQ and PXF, ensure that:
+
+- The HDFS plug-in is installed on all cluster nodes. See [Installing
PXF Plug-ins](InstallPXFPlugins.html) for PXF plug-in installation information.
+- All HDFS users have read permissions to HDFS services and that write
permissions have been restricted to specific users.
+
+## Writing to PXF External Tables
+The PXF HDFS plug-in supports writable two profiles: `HdfsTextSimple` and
`SequenceWritable`.
+
+Use the following syntax to create a HAWQ external writable table
representing HDFS data:
+
+``` sql
+CREATE EXTERNAL WRITABLE TABLE
+( [, ...] | LIKE )
+LOCATION ('pxf://[:]/
+
?PROFILE=HdfsTextSimple|SequenceWritable[&=[...]]')
+FORMAT '[TEXT|CSV|CUSTOM]' ();
+```
+
+HDFS-plug-in-specific keywords and values used in the [CREATE EXTERNAL
TABLE](../reference/sql/CREATE-EXTERNAL-TABLE.html) call are described in the
table below.
+
+| Keyword | Value |
+|---|-|
+| \[:\]| The HDFS NameNode and port. |
+| \| The path to the file in the HDFS data store. |
+| PROFILE| The `PROFILE` keyword must specify one of the values
`HdfsTextSimple` or `SequenceWritable`. |
+| \ | \ is profile-specific. These
options are discussed in the next topic.|
+| FORMAT 'TEXT' | Use '`TEXT`' `FORMAT` with the `HdfsTextSimple` profile
when \ will reference a plain text delimited file. The
`HdfsTextSimple` '`TEXT`' `FORMAT` supports only the built-in
`(delimiter=)` \. |
+| FORMAT 'CSV' | Use '`CSV`' `FORMAT` with `HdfsTextSimple` when
\ will reference a comma-separated value file. |
+| FORMAT 'CUSTOM' | Use the `'CUSTOM'` `FORMAT` with the
`SequenceWritable` profile. The `SequenceWritable` '`CUSTOM`' `FORMAT` supports
only the built-in `(formatter='pxfwritable_export)` (write) and
`(formatter='pxfwritable_import)` (read) \.
+
+**Note**: When creating PXF external tables, you cannot use the `HEADER`
option in your `FORMAT` specification.
+
+## Custom Options
+
+The `HdfsTextSimple` and `SequenceWritable` profiles support the following
\:
+
+| Keyword | Value Description |
+|---|-|
+| COMPRESSION_CODEC| The compression codec Java class name. If this
option is not provided, no data compression is performed. Supported compression
codecs include: `org.apache.hadoop.io.compress.DefaultCodec`,
`org.apache.hadoop.io.compress.BZip2Codec`, and
`org.apache.hadoop.io.compress.GzipCodec` (`HdfsTextSimple` profile only) |
--- End diff --
Instead of including parentheticals here (`HdfsTextSimple` profile only),
add a third column to indicate which profile(s) the option applies to.
> create new documentation topic for PXF writable profiles
>
>
> Key: HAWQ-1119
> URL: https://issues.apache.org/jira/browse/HAWQ-1119
> Project: Apache HAWQ
> Issue Type: Improvement
> Components: Documentation
>Reporter: Lisa Owen
>Assignee: David Yozie
> Fix For: 2.0.1.0-incubating
>
>
> certain profiles supported by the existing PXF plug-ins support writable
> tables. create some documentation content for these profiles.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HAWQ-1119) create new documentation topic for PXF writable profiles
[
https://issues.apache.org/jira/browse/HAWQ-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15623615#comment-15623615
]
ASF GitHub Bot commented on HAWQ-1119:
--
Github user dyozie commented on a diff in the pull request:
https://github.com/apache/incubator-hawq-docs/pull/46#discussion_r85845000
--- Diff: pxf/HDFSWritablePXF.html.md.erb ---
@@ -0,0 +1,410 @@
+---
+title: Writing Data to HDFS
+---
+
+The PXF HDFS plug-in supports writable external tables using the
`HdfsTextSimple` and `SequenceWritable` profiles. You might create a writable
table to export data from a HAWQ internal table to HDFS.
+
+This section describes how to use these PXF profiles to create writable
external tables.
+
+**Note**: You cannot directly query data in a HAWQ writable table. After
creating the external writable table, you must create a HAWQ readable external
table accessing the HDFS file, then query that table. ??You can also create a
Hive table to access the HDFS file.??
+
+## Prerequisites
+
+Before working with HDFS file data using HAWQ and PXF, ensure that:
+
+- The HDFS plug-in is installed on all cluster nodes. See [Installing
PXF Plug-ins](InstallPXFPlugins.html) for PXF plug-in installation information.
+- All HDFS users have read permissions to HDFS services and that write
permissions have been restricted to specific users.
+
+## Writing to PXF External Tables
+The PXF HDFS plug-in supports writable two profiles: `HdfsTextSimple` and
`SequenceWritable`.
+
+Use the following syntax to create a HAWQ external writable table
representing HDFS data:
+
+``` sql
+CREATE EXTERNAL WRITABLE TABLE
+( [, ...] | LIKE )
+LOCATION ('pxf://[:]/
+
?PROFILE=HdfsTextSimple|SequenceWritable[&=[...]]')
+FORMAT '[TEXT|CSV|CUSTOM]' ();
+```
+
+HDFS-plug-in-specific keywords and values used in the [CREATE EXTERNAL
TABLE](../reference/sql/CREATE-EXTERNAL-TABLE.html) call are described in the
table below.
+
+| Keyword | Value |
+|---|-|
+| \[:\]| The HDFS NameNode and port. |
+| \| The path to the file in the HDFS data store. |
+| PROFILE| The `PROFILE` keyword must specify one of the values
`HdfsTextSimple` or `SequenceWritable`. |
+| \ | \ is profile-specific. These
options are discussed in the next topic.|
+| FORMAT 'TEXT' | Use '`TEXT`' `FORMAT` with the `HdfsTextSimple` profile
when \ will reference a plain text delimited file. The
`HdfsTextSimple` '`TEXT`' `FORMAT` supports only the built-in
`(delimiter=)` \. |
+| FORMAT 'CSV' | Use '`CSV`' `FORMAT` with `HdfsTextSimple` when
\ will reference a comma-separated value file. |
+| FORMAT 'CUSTOM' | Use the `'CUSTOM'` `FORMAT` with the
`SequenceWritable` profile. The `SequenceWritable` '`CUSTOM`' `FORMAT` supports
only the built-in `(formatter='pxfwritable_export)` (write) and
`(formatter='pxfwritable_import)` (read) \.
+
+**Note**: When creating PXF external tables, you cannot use the `HEADER`
option in your `FORMAT` specification.
+
+## Custom Options
+
+The `HdfsTextSimple` and `SequenceWritable` profiles support the following
\:
+
+| Keyword | Value Description |
+|---|-|
+| COMPRESSION_CODEC| The compression codec Java class name. If this
option is not provided, no data compression is performed. Supported compression
codecs include: `org.apache.hadoop.io.compress.DefaultCodec`,
`org.apache.hadoop.io.compress.BZip2Codec`, and
`org.apache.hadoop.io.compress.GzipCodec` (`HdfsTextSimple` profile only) |
+| COMPRESSION_TYPE| The compression type to employ; supported values
are `RECORD` (the default) or `BLOCK`. |
+| DATA-SCHEMA| (`SequenceWritable` profile only) The name of the
writer serialization/deserialization class. The jar file in which this class
resides must be in the PXF class path. This option has no default value. |
+| THREAD-SAFE | Boolean value determining if a table query can run in
multi-thread mode. Default value is `TRUE`, requests run in multi-threaded
mode. When set to `FALSE`, requests will be handled in a single thread.
`THREAD-SAFE` should be set appropriately when operations that are not
thread-safe are performed (i.e. compression). |
+
+## HdfsTextSimple Profile
+
+Use the `HdfsTextSimple` profile when writing delimited data to a plain
text file where each row is a single record.
+
+Writable tables created using the `HdfsTextSimple` profile can use no,
record, or block compression. When compression is used, the default, gzip, and
bzip2 Hadoop compression codecs are supported:
+
+- org.apache.hadoop.io.compress.DefaultCodec
+- org.apache.hadoop.io.compres
[jira] [Commented] (HAWQ-1119) create new documentation topic for PXF writable profiles
[
https://issues.apache.org/jira/browse/HAWQ-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15623616#comment-15623616
]
ASF GitHub Bot commented on HAWQ-1119:
--
Github user dyozie commented on a diff in the pull request:
https://github.com/apache/incubator-hawq-docs/pull/46#discussion_r85845526
--- Diff: pxf/HDFSWritablePXF.html.md.erb ---
@@ -0,0 +1,410 @@
+---
+title: Writing Data to HDFS
+---
+
+The PXF HDFS plug-in supports writable external tables using the
`HdfsTextSimple` and `SequenceWritable` profiles. You might create a writable
table to export data from a HAWQ internal table to HDFS.
+
+This section describes how to use these PXF profiles to create writable
external tables.
+
+**Note**: You cannot directly query data in a HAWQ writable table. After
creating the external writable table, you must create a HAWQ readable external
table accessing the HDFS file, then query that table. ??You can also create a
Hive table to access the HDFS file.??
+
+## Prerequisites
+
+Before working with HDFS file data using HAWQ and PXF, ensure that:
+
+- The HDFS plug-in is installed on all cluster nodes. See [Installing
PXF Plug-ins](InstallPXFPlugins.html) for PXF plug-in installation information.
+- All HDFS users have read permissions to HDFS services and that write
permissions have been restricted to specific users.
+
+## Writing to PXF External Tables
+The PXF HDFS plug-in supports writable two profiles: `HdfsTextSimple` and
`SequenceWritable`.
+
+Use the following syntax to create a HAWQ external writable table
representing HDFS data:
+
+``` sql
+CREATE EXTERNAL WRITABLE TABLE
+( [, ...] | LIKE )
+LOCATION ('pxf://[:]/
+
?PROFILE=HdfsTextSimple|SequenceWritable[&=[...]]')
+FORMAT '[TEXT|CSV|CUSTOM]' ();
+```
+
+HDFS-plug-in-specific keywords and values used in the [CREATE EXTERNAL
TABLE](../reference/sql/CREATE-EXTERNAL-TABLE.html) call are described in the
table below.
+
+| Keyword | Value |
+|---|-|
+| \[:\]| The HDFS NameNode and port. |
+| \| The path to the file in the HDFS data store. |
+| PROFILE| The `PROFILE` keyword must specify one of the values
`HdfsTextSimple` or `SequenceWritable`. |
+| \ | \ is profile-specific. These
options are discussed in the next topic.|
+| FORMAT 'TEXT' | Use '`TEXT`' `FORMAT` with the `HdfsTextSimple` profile
when \ will reference a plain text delimited file. The
`HdfsTextSimple` '`TEXT`' `FORMAT` supports only the built-in
`(delimiter=)` \. |
+| FORMAT 'CSV' | Use '`CSV`' `FORMAT` with `HdfsTextSimple` when
\ will reference a comma-separated value file. |
+| FORMAT 'CUSTOM' | Use the `'CUSTOM'` `FORMAT` with the
`SequenceWritable` profile. The `SequenceWritable` '`CUSTOM`' `FORMAT` supports
only the built-in `(formatter='pxfwritable_export)` (write) and
`(formatter='pxfwritable_import)` (read) \.
+
+**Note**: When creating PXF external tables, you cannot use the `HEADER`
option in your `FORMAT` specification.
+
+## Custom Options
+
+The `HdfsTextSimple` and `SequenceWritable` profiles support the following
\:
+
+| Keyword | Value Description |
+|---|-|
+| COMPRESSION_CODEC| The compression codec Java class name. If this
option is not provided, no data compression is performed. Supported compression
codecs include: `org.apache.hadoop.io.compress.DefaultCodec`,
`org.apache.hadoop.io.compress.BZip2Codec`, and
`org.apache.hadoop.io.compress.GzipCodec` (`HdfsTextSimple` profile only) |
+| COMPRESSION_TYPE| The compression type to employ; supported values
are `RECORD` (the default) or `BLOCK`. |
+| DATA-SCHEMA| (`SequenceWritable` profile only) The name of the
writer serialization/deserialization class. The jar file in which this class
resides must be in the PXF class path. This option has no default value. |
+| THREAD-SAFE | Boolean value determining if a table query can run in
multi-thread mode. Default value is `TRUE`, requests run in multi-threaded
mode. When set to `FALSE`, requests will be handled in a single thread.
`THREAD-SAFE` should be set appropriately when operations that are not
thread-safe are performed (i.e. compression). |
+
+## HdfsTextSimple Profile
+
+Use the `HdfsTextSimple` profile when writing delimited data to a plain
text file where each row is a single record.
+
+Writable tables created using the `HdfsTextSimple` profile can use no,
record, or block compression. When compression is used, the default, gzip, and
bzip2 Hadoop compression codecs are supported:
+
+- org.apache.hadoop.io.compress.DefaultCodec
+- org.apache.hadoop.io.compres
[jira] [Commented] (HAWQ-1119) create new documentation topic for PXF writable profiles
[ https://issues.apache.org/jira/browse/HAWQ-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15623626#comment-15623626 ] ASF GitHub Bot commented on HAWQ-1119: -- Github user dyozie commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/46#discussion_r85791950 --- Diff: pxf/HDFSWritablePXF.html.md.erb --- @@ -0,0 +1,410 @@ +--- +title: Writing Data to HDFS +--- + +The PXF HDFS plug-in supports writable external tables using the `HdfsTextSimple` and `SequenceWritable` profiles. You might create a writable table to export data from a HAWQ internal table to HDFS. + +This section describes how to use these PXF profiles to create writable external tables. + +**Note**: You cannot directly query data in a HAWQ writable table. After creating the external writable table, you must create a HAWQ readable external table accessing the HDFS file, then query that table. ??You can also create a Hive table to access the HDFS file.?? + +## Prerequisites + +Before working with HDFS file data using HAWQ and PXF, ensure that: + +- The HDFS plug-in is installed on all cluster nodes. See [Installing PXF Plug-ins](InstallPXFPlugins.html) for PXF plug-in installation information. +- All HDFS users have read permissions to HDFS services and that write permissions have been restricted to specific users. + +## Writing to PXF External Tables +The PXF HDFS plug-in supports writable two profiles: `HdfsTextSimple` and `SequenceWritable`. + +Use the following syntax to create a HAWQ external writable table representing HDFS data: + +``` sql +CREATE EXTERNAL WRITABLE TABLE --- End diff -- That syntax is unfortunate. GPDB uses CREATE WRITABLE EXTERNAL instead of CREATE EXTERNAL WRITABLE :( > create new documentation topic for PXF writable profiles > > > Key: HAWQ-1119 > URL: https://issues.apache.org/jira/browse/HAWQ-1119 > Project: Apache HAWQ > Issue Type: Improvement > Components: Documentation >Reporter: Lisa Owen >Assignee: David Yozie > Fix For: 2.0.1.0-incubating > > > certain profiles supported by the existing PXF plug-ins support writable > tables. create some documentation content for these profiles. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-1119) create new documentation topic for PXF writable profiles
[
https://issues.apache.org/jira/browse/HAWQ-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15623623#comment-15623623
]
ASF GitHub Bot commented on HAWQ-1119:
--
Github user dyozie commented on a diff in the pull request:
https://github.com/apache/incubator-hawq-docs/pull/46#discussion_r85812589
--- Diff: pxf/HDFSWritablePXF.html.md.erb ---
@@ -0,0 +1,410 @@
+---
+title: Writing Data to HDFS
+---
+
+The PXF HDFS plug-in supports writable external tables using the
`HdfsTextSimple` and `SequenceWritable` profiles. You might create a writable
table to export data from a HAWQ internal table to HDFS.
+
+This section describes how to use these PXF profiles to create writable
external tables.
+
+**Note**: You cannot directly query data in a HAWQ writable table. After
creating the external writable table, you must create a HAWQ readable external
table accessing the HDFS file, then query that table. ??You can also create a
Hive table to access the HDFS file.??
+
+## Prerequisites
+
+Before working with HDFS file data using HAWQ and PXF, ensure that:
+
+- The HDFS plug-in is installed on all cluster nodes. See [Installing
PXF Plug-ins](InstallPXFPlugins.html) for PXF plug-in installation information.
+- All HDFS users have read permissions to HDFS services and that write
permissions have been restricted to specific users.
+
+## Writing to PXF External Tables
+The PXF HDFS plug-in supports writable two profiles: `HdfsTextSimple` and
`SequenceWritable`.
+
+Use the following syntax to create a HAWQ external writable table
representing HDFS data:
+
+``` sql
+CREATE EXTERNAL WRITABLE TABLE
+( [, ...] | LIKE )
+LOCATION ('pxf://[:]/
+
?PROFILE=HdfsTextSimple|SequenceWritable[&=[...]]')
+FORMAT '[TEXT|CSV|CUSTOM]' ();
+```
+
+HDFS-plug-in-specific keywords and values used in the [CREATE EXTERNAL
TABLE](../reference/sql/CREATE-EXTERNAL-TABLE.html) call are described in the
table below.
+
+| Keyword | Value |
+|---|-|
+| \[:\]| The HDFS NameNode and port. |
+| \| The path to the file in the HDFS data store. |
+| PROFILE| The `PROFILE` keyword must specify one of the values
`HdfsTextSimple` or `SequenceWritable`. |
+| \ | \ is profile-specific. These
options are discussed in the next topic.|
+| FORMAT 'TEXT' | Use '`TEXT`' `FORMAT` with the `HdfsTextSimple` profile
when \ will reference a plain text delimited file. The
`HdfsTextSimple` '`TEXT`' `FORMAT` supports only the built-in
`(delimiter=)` \. |
+| FORMAT 'CSV' | Use '`CSV`' `FORMAT` with `HdfsTextSimple` when
\ will reference a comma-separated value file. |
+| FORMAT 'CUSTOM' | Use the `'CUSTOM'` `FORMAT` with the
`SequenceWritable` profile. The `SequenceWritable` '`CUSTOM`' `FORMAT` supports
only the built-in `(formatter='pxfwritable_export)` (write) and
`(formatter='pxfwritable_import)` (read) \.
+
+**Note**: When creating PXF external tables, you cannot use the `HEADER`
option in your `FORMAT` specification.
+
+## Custom Options
+
+The `HdfsTextSimple` and `SequenceWritable` profiles support the following
\:
+
+| Keyword | Value Description |
+|---|-|
+| COMPRESSION_CODEC| The compression codec Java class name. If this
option is not provided, no data compression is performed. Supported compression
codecs include: `org.apache.hadoop.io.compress.DefaultCodec`,
`org.apache.hadoop.io.compress.BZip2Codec`, and
`org.apache.hadoop.io.compress.GzipCodec` (`HdfsTextSimple` profile only) |
+| COMPRESSION_TYPE| The compression type to employ; supported values
are `RECORD` (the default) or `BLOCK`. |
+| DATA-SCHEMA| (`SequenceWritable` profile only) The name of the
writer serialization/deserialization class. The jar file in which this class
resides must be in the PXF class path. This option has no default value. |
+| THREAD-SAFE | Boolean value determining if a table query can run in
multi-thread mode. Default value is `TRUE`, requests run in multi-threaded
mode. When set to `FALSE`, requests will be handled in a single thread.
`THREAD-SAFE` should be set appropriately when operations that are not
thread-safe are performed (i.e. compression). |
--- End diff --
multi-thread -> multi-threaded. Also, the rest some edits:
The default value is true. Set this option to `FALSE` to handle all
requests in a single thread for operations that are not thread-safe (for
example, compression).
> create new documentation topic for PXF writable profiles
>
>
> Key: HAWQ-1119
> URL: https://issues.apache.or
[jira] [Commented] (HAWQ-1119) create new documentation topic for PXF writable profiles
[
https://issues.apache.org/jira/browse/HAWQ-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15623614#comment-15623614
]
ASF GitHub Bot commented on HAWQ-1119:
--
Github user dyozie commented on a diff in the pull request:
https://github.com/apache/incubator-hawq-docs/pull/46#discussion_r85794041
--- Diff: pxf/HDFSWritablePXF.html.md.erb ---
@@ -0,0 +1,410 @@
+---
+title: Writing Data to HDFS
+---
+
+The PXF HDFS plug-in supports writable external tables using the
`HdfsTextSimple` and `SequenceWritable` profiles. You might create a writable
table to export data from a HAWQ internal table to HDFS.
+
+This section describes how to use these PXF profiles to create writable
external tables.
+
+**Note**: You cannot directly query data in a HAWQ writable table. After
creating the external writable table, you must create a HAWQ readable external
table accessing the HDFS file, then query that table. ??You can also create a
Hive table to access the HDFS file.??
+
+## Prerequisites
+
+Before working with HDFS file data using HAWQ and PXF, ensure that:
+
+- The HDFS plug-in is installed on all cluster nodes. See [Installing
PXF Plug-ins](InstallPXFPlugins.html) for PXF plug-in installation information.
+- All HDFS users have read permissions to HDFS services and that write
permissions have been restricted to specific users.
+
+## Writing to PXF External Tables
+The PXF HDFS plug-in supports writable two profiles: `HdfsTextSimple` and
`SequenceWritable`.
+
+Use the following syntax to create a HAWQ external writable table
representing HDFS data:
+
+``` sql
+CREATE EXTERNAL WRITABLE TABLE
+( [, ...] | LIKE )
+LOCATION ('pxf://[:]/
+
?PROFILE=HdfsTextSimple|SequenceWritable[&=[...]]')
+FORMAT '[TEXT|CSV|CUSTOM]' ();
+```
+
+HDFS-plug-in-specific keywords and values used in the [CREATE EXTERNAL
TABLE](../reference/sql/CREATE-EXTERNAL-TABLE.html) call are described in the
table below.
+
+| Keyword | Value |
+|---|-|
+| \[:\]| The HDFS NameNode and port. |
+| \| The path to the file in the HDFS data store. |
+| PROFILE| The `PROFILE` keyword must specify one of the values
`HdfsTextSimple` or `SequenceWritable`. |
+| \ | \ is profile-specific. These
options are discussed in the next topic.|
+| FORMAT 'TEXT' | Use '`TEXT`' `FORMAT` with the `HdfsTextSimple` profile
when \ will reference a plain text delimited file. The
`HdfsTextSimple` '`TEXT`' `FORMAT` supports only the built-in
`(delimiter=)` \. |
+| FORMAT 'CSV' | Use '`CSV`' `FORMAT` with `HdfsTextSimple` when
\ will reference a comma-separated value file. |
+| FORMAT 'CUSTOM' | Use the `'CUSTOM'` `FORMAT` with the
`SequenceWritable` profile. The `SequenceWritable` '`CUSTOM`' `FORMAT` supports
only the built-in `(formatter='pxfwritable_export)` (write) and
`(formatter='pxfwritable_import)` (read) \.
+
+**Note**: When creating PXF external tables, you cannot use the `HEADER`
option in your `FORMAT` specification.
+
+## Custom Options
+
+The `HdfsTextSimple` and `SequenceWritable` profiles support the following
\:
+
+| Keyword | Value Description |
--- End diff --
Change "Keyword" to "Option"?
> create new documentation topic for PXF writable profiles
>
>
> Key: HAWQ-1119
> URL: https://issues.apache.org/jira/browse/HAWQ-1119
> Project: Apache HAWQ
> Issue Type: Improvement
> Components: Documentation
>Reporter: Lisa Owen
>Assignee: David Yozie
> Fix For: 2.0.1.0-incubating
>
>
> certain profiles supported by the existing PXF plug-ins support writable
> tables. create some documentation content for these profiles.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HAWQ-1119) create new documentation topic for PXF writable profiles
[
https://issues.apache.org/jira/browse/HAWQ-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15623618#comment-15623618
]
ASF GitHub Bot commented on HAWQ-1119:
--
Github user dyozie commented on a diff in the pull request:
https://github.com/apache/incubator-hawq-docs/pull/46#discussion_r85794241
--- Diff: pxf/HDFSWritablePXF.html.md.erb ---
@@ -0,0 +1,410 @@
+---
+title: Writing Data to HDFS
+---
+
+The PXF HDFS plug-in supports writable external tables using the
`HdfsTextSimple` and `SequenceWritable` profiles. You might create a writable
table to export data from a HAWQ internal table to HDFS.
+
+This section describes how to use these PXF profiles to create writable
external tables.
+
+**Note**: You cannot directly query data in a HAWQ writable table. After
creating the external writable table, you must create a HAWQ readable external
table accessing the HDFS file, then query that table. ??You can also create a
Hive table to access the HDFS file.??
+
+## Prerequisites
+
+Before working with HDFS file data using HAWQ and PXF, ensure that:
+
+- The HDFS plug-in is installed on all cluster nodes. See [Installing
PXF Plug-ins](InstallPXFPlugins.html) for PXF plug-in installation information.
+- All HDFS users have read permissions to HDFS services and that write
permissions have been restricted to specific users.
+
+## Writing to PXF External Tables
+The PXF HDFS plug-in supports writable two profiles: `HdfsTextSimple` and
`SequenceWritable`.
+
+Use the following syntax to create a HAWQ external writable table
representing HDFS data:
+
+``` sql
+CREATE EXTERNAL WRITABLE TABLE
+( [, ...] | LIKE )
+LOCATION ('pxf://[:]/
+
?PROFILE=HdfsTextSimple|SequenceWritable[&=[...]]')
+FORMAT '[TEXT|CSV|CUSTOM]' ();
+```
+
+HDFS-plug-in-specific keywords and values used in the [CREATE EXTERNAL
TABLE](../reference/sql/CREATE-EXTERNAL-TABLE.html) call are described in the
table below.
+
+| Keyword | Value |
+|---|-|
+| \[:\]| The HDFS NameNode and port. |
+| \| The path to the file in the HDFS data store. |
+| PROFILE| The `PROFILE` keyword must specify one of the values
`HdfsTextSimple` or `SequenceWritable`. |
+| \ | \ is profile-specific. These
options are discussed in the next topic.|
+| FORMAT 'TEXT' | Use '`TEXT`' `FORMAT` with the `HdfsTextSimple` profile
when \ will reference a plain text delimited file. The
`HdfsTextSimple` '`TEXT`' `FORMAT` supports only the built-in
`(delimiter=)` \. |
+| FORMAT 'CSV' | Use '`CSV`' `FORMAT` with `HdfsTextSimple` when
\ will reference a comma-separated value file. |
+| FORMAT 'CUSTOM' | Use the `'CUSTOM'` `FORMAT` with the
`SequenceWritable` profile. The `SequenceWritable` '`CUSTOM`' `FORMAT` supports
only the built-in `(formatter='pxfwritable_export)` (write) and
`(formatter='pxfwritable_import)` (read) \.
+
+**Note**: When creating PXF external tables, you cannot use the `HEADER`
option in your `FORMAT` specification.
+
+## Custom Options
+
+The `HdfsTextSimple` and `SequenceWritable` profiles support the following
\:
--- End diff --
Change to " values"?
> create new documentation topic for PXF writable profiles
>
>
> Key: HAWQ-1119
> URL: https://issues.apache.org/jira/browse/HAWQ-1119
> Project: Apache HAWQ
> Issue Type: Improvement
> Components: Documentation
>Reporter: Lisa Owen
>Assignee: David Yozie
> Fix For: 2.0.1.0-incubating
>
>
> certain profiles supported by the existing PXF plug-ins support writable
> tables. create some documentation content for these profiles.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HAWQ-1119) create new documentation topic for PXF writable profiles
[
https://issues.apache.org/jira/browse/HAWQ-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15623622#comment-15623622
]
ASF GitHub Bot commented on HAWQ-1119:
--
Github user dyozie commented on a diff in the pull request:
https://github.com/apache/incubator-hawq-docs/pull/46#discussion_r85814629
--- Diff: pxf/HDFSWritablePXF.html.md.erb ---
@@ -0,0 +1,410 @@
+---
+title: Writing Data to HDFS
+---
+
+The PXF HDFS plug-in supports writable external tables using the
`HdfsTextSimple` and `SequenceWritable` profiles. You might create a writable
table to export data from a HAWQ internal table to HDFS.
+
+This section describes how to use these PXF profiles to create writable
external tables.
+
+**Note**: You cannot directly query data in a HAWQ writable table. After
creating the external writable table, you must create a HAWQ readable external
table accessing the HDFS file, then query that table. ??You can also create a
Hive table to access the HDFS file.??
+
+## Prerequisites
+
+Before working with HDFS file data using HAWQ and PXF, ensure that:
+
+- The HDFS plug-in is installed on all cluster nodes. See [Installing
PXF Plug-ins](InstallPXFPlugins.html) for PXF plug-in installation information.
+- All HDFS users have read permissions to HDFS services and that write
permissions have been restricted to specific users.
+
+## Writing to PXF External Tables
+The PXF HDFS plug-in supports writable two profiles: `HdfsTextSimple` and
`SequenceWritable`.
+
+Use the following syntax to create a HAWQ external writable table
representing HDFS data:
+
+``` sql
+CREATE EXTERNAL WRITABLE TABLE
+( [, ...] | LIKE )
+LOCATION ('pxf://[:]/
+
?PROFILE=HdfsTextSimple|SequenceWritable[&=[...]]')
+FORMAT '[TEXT|CSV|CUSTOM]' ();
+```
+
+HDFS-plug-in-specific keywords and values used in the [CREATE EXTERNAL
TABLE](../reference/sql/CREATE-EXTERNAL-TABLE.html) call are described in the
table below.
+
+| Keyword | Value |
+|---|-|
+| \[:\]| The HDFS NameNode and port. |
+| \| The path to the file in the HDFS data store. |
+| PROFILE| The `PROFILE` keyword must specify one of the values
`HdfsTextSimple` or `SequenceWritable`. |
+| \ | \ is profile-specific. These
options are discussed in the next topic.|
+| FORMAT 'TEXT' | Use '`TEXT`' `FORMAT` with the `HdfsTextSimple` profile
when \ will reference a plain text delimited file. The
`HdfsTextSimple` '`TEXT`' `FORMAT` supports only the built-in
`(delimiter=)` \. |
+| FORMAT 'CSV' | Use '`CSV`' `FORMAT` with `HdfsTextSimple` when
\ will reference a comma-separated value file. |
+| FORMAT 'CUSTOM' | Use the `'CUSTOM'` `FORMAT` with the
`SequenceWritable` profile. The `SequenceWritable` '`CUSTOM`' `FORMAT` supports
only the built-in `(formatter='pxfwritable_export)` (write) and
`(formatter='pxfwritable_import)` (read) \.
+
+**Note**: When creating PXF external tables, you cannot use the `HEADER`
option in your `FORMAT` specification.
+
+## Custom Options
+
+The `HdfsTextSimple` and `SequenceWritable` profiles support the following
\:
+
+| Keyword | Value Description |
+|---|-|
+| COMPRESSION_CODEC| The compression codec Java class name. If this
option is not provided, no data compression is performed. Supported compression
codecs include: `org.apache.hadoop.io.compress.DefaultCodec`,
`org.apache.hadoop.io.compress.BZip2Codec`, and
`org.apache.hadoop.io.compress.GzipCodec` (`HdfsTextSimple` profile only) |
+| COMPRESSION_TYPE| The compression type to employ; supported values
are `RECORD` (the default) or `BLOCK`. |
+| DATA-SCHEMA| (`SequenceWritable` profile only) The name of the
writer serialization/deserialization class. The jar file in which this class
resides must be in the PXF class path. This option has no default value. |
+| THREAD-SAFE | Boolean value determining if a table query can run in
multi-thread mode. Default value is `TRUE`, requests run in multi-threaded
mode. When set to `FALSE`, requests will be handled in a single thread.
`THREAD-SAFE` should be set appropriately when operations that are not
thread-safe are performed (i.e. compression). |
+
+## HdfsTextSimple Profile
+
+Use the `HdfsTextSimple` profile when writing delimited data to a plain
text file where each row is a single record.
+
+Writable tables created using the `HdfsTextSimple` profile can use no,
record, or block compression. When compression is used, the default, gzip, and
bzip2 Hadoop compression codecs are supported:
+
+- org.apache.hadoop.io.compress.DefaultCodec
+- org.apache.hadoop.io.compres
[jira] [Commented] (HAWQ-1119) create new documentation topic for PXF writable profiles
[ https://issues.apache.org/jira/browse/HAWQ-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15623619#comment-15623619 ] ASF GitHub Bot commented on HAWQ-1119: -- Github user dyozie commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/46#discussion_r85792301 --- Diff: pxf/HDFSWritablePXF.html.md.erb --- @@ -0,0 +1,410 @@ +--- +title: Writing Data to HDFS +--- + +The PXF HDFS plug-in supports writable external tables using the `HdfsTextSimple` and `SequenceWritable` profiles. You might create a writable table to export data from a HAWQ internal table to HDFS. + +This section describes how to use these PXF profiles to create writable external tables. + +**Note**: You cannot directly query data in a HAWQ writable table. After creating the external writable table, you must create a HAWQ readable external table accessing the HDFS file, then query that table. ??You can also create a Hive table to access the HDFS file.?? + +## Prerequisites + +Before working with HDFS file data using HAWQ and PXF, ensure that: + +- The HDFS plug-in is installed on all cluster nodes. See [Installing PXF Plug-ins](InstallPXFPlugins.html) for PXF plug-in installation information. +- All HDFS users have read permissions to HDFS services and that write permissions have been restricted to specific users. + +## Writing to PXF External Tables +The PXF HDFS plug-in supports writable two profiles: `HdfsTextSimple` and `SequenceWritable`. --- End diff -- writable two -> two writable Also, seems like there should be some mention of the difference between these profiles by now. > create new documentation topic for PXF writable profiles > > > Key: HAWQ-1119 > URL: https://issues.apache.org/jira/browse/HAWQ-1119 > Project: Apache HAWQ > Issue Type: Improvement > Components: Documentation >Reporter: Lisa Owen >Assignee: David Yozie > Fix For: 2.0.1.0-incubating > > > certain profiles supported by the existing PXF plug-ins support writable > tables. create some documentation content for these profiles. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-1119) create new documentation topic for PXF writable profiles
[
https://issues.apache.org/jira/browse/HAWQ-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15623617#comment-15623617
]
ASF GitHub Bot commented on HAWQ-1119:
--
Github user dyozie commented on a diff in the pull request:
https://github.com/apache/incubator-hawq-docs/pull/46#discussion_r85792675
--- Diff: pxf/HDFSWritablePXF.html.md.erb ---
@@ -0,0 +1,410 @@
+---
+title: Writing Data to HDFS
+---
+
+The PXF HDFS plug-in supports writable external tables using the
`HdfsTextSimple` and `SequenceWritable` profiles. You might create a writable
table to export data from a HAWQ internal table to HDFS.
+
+This section describes how to use these PXF profiles to create writable
external tables.
+
+**Note**: You cannot directly query data in a HAWQ writable table. After
creating the external writable table, you must create a HAWQ readable external
table accessing the HDFS file, then query that table. ??You can also create a
Hive table to access the HDFS file.??
+
+## Prerequisites
+
+Before working with HDFS file data using HAWQ and PXF, ensure that:
+
+- The HDFS plug-in is installed on all cluster nodes. See [Installing
PXF Plug-ins](InstallPXFPlugins.html) for PXF plug-in installation information.
+- All HDFS users have read permissions to HDFS services and that write
permissions have been restricted to specific users.
+
+## Writing to PXF External Tables
+The PXF HDFS plug-in supports writable two profiles: `HdfsTextSimple` and
`SequenceWritable`.
+
+Use the following syntax to create a HAWQ external writable table
representing HDFS data:
+
+``` sql
+CREATE EXTERNAL WRITABLE TABLE
+( [, ...] | LIKE )
+LOCATION ('pxf://[:]/
+
?PROFILE=HdfsTextSimple|SequenceWritable[&=[...]]')
+FORMAT '[TEXT|CSV|CUSTOM]' ();
+```
+
+HDFS-plug-in-specific keywords and values used in the [CREATE EXTERNAL
TABLE](../reference/sql/CREATE-EXTERNAL-TABLE.html) call are described in the
table below.
+
+| Keyword | Value |
+|---|-|
+| \[:\]| The HDFS NameNode and port. |
+| \| The path to the file in the HDFS data store. |
+| PROFILE| The `PROFILE` keyword must specify one of the values
`HdfsTextSimple` or `SequenceWritable`. |
+| \ | \ is profile-specific. These
options are discussed in the next topic.|
--- End diff --
Maybe change this to ?
> create new documentation topic for PXF writable profiles
>
>
> Key: HAWQ-1119
> URL: https://issues.apache.org/jira/browse/HAWQ-1119
> Project: Apache HAWQ
> Issue Type: Improvement
> Components: Documentation
>Reporter: Lisa Owen
>Assignee: David Yozie
> Fix For: 2.0.1.0-incubating
>
>
> certain profiles supported by the existing PXF plug-ins support writable
> tables. create some documentation content for these profiles.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HAWQ-1119) create new documentation topic for PXF writable profiles
[
https://issues.apache.org/jira/browse/HAWQ-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15623628#comment-15623628
]
ASF GitHub Bot commented on HAWQ-1119:
--
Github user dyozie commented on a diff in the pull request:
https://github.com/apache/incubator-hawq-docs/pull/46#discussion_r85813812
--- Diff: pxf/HDFSWritablePXF.html.md.erb ---
@@ -0,0 +1,410 @@
+---
+title: Writing Data to HDFS
+---
+
+The PXF HDFS plug-in supports writable external tables using the
`HdfsTextSimple` and `SequenceWritable` profiles. You might create a writable
table to export data from a HAWQ internal table to HDFS.
+
+This section describes how to use these PXF profiles to create writable
external tables.
+
+**Note**: You cannot directly query data in a HAWQ writable table. After
creating the external writable table, you must create a HAWQ readable external
table accessing the HDFS file, then query that table. ??You can also create a
Hive table to access the HDFS file.??
+
+## Prerequisites
+
+Before working with HDFS file data using HAWQ and PXF, ensure that:
+
+- The HDFS plug-in is installed on all cluster nodes. See [Installing
PXF Plug-ins](InstallPXFPlugins.html) for PXF plug-in installation information.
+- All HDFS users have read permissions to HDFS services and that write
permissions have been restricted to specific users.
+
+## Writing to PXF External Tables
+The PXF HDFS plug-in supports writable two profiles: `HdfsTextSimple` and
`SequenceWritable`.
+
+Use the following syntax to create a HAWQ external writable table
representing HDFS data:
+
+``` sql
+CREATE EXTERNAL WRITABLE TABLE
+( [, ...] | LIKE )
+LOCATION ('pxf://[:]/
+
?PROFILE=HdfsTextSimple|SequenceWritable[&=[...]]')
+FORMAT '[TEXT|CSV|CUSTOM]' ();
+```
+
+HDFS-plug-in-specific keywords and values used in the [CREATE EXTERNAL
TABLE](../reference/sql/CREATE-EXTERNAL-TABLE.html) call are described in the
table below.
+
+| Keyword | Value |
+|---|-|
+| \[:\]| The HDFS NameNode and port. |
+| \| The path to the file in the HDFS data store. |
+| PROFILE| The `PROFILE` keyword must specify one of the values
`HdfsTextSimple` or `SequenceWritable`. |
+| \ | \ is profile-specific. These
options are discussed in the next topic.|
+| FORMAT 'TEXT' | Use '`TEXT`' `FORMAT` with the `HdfsTextSimple` profile
when \ will reference a plain text delimited file. The
`HdfsTextSimple` '`TEXT`' `FORMAT` supports only the built-in
`(delimiter=)` \. |
+| FORMAT 'CSV' | Use '`CSV`' `FORMAT` with `HdfsTextSimple` when
\ will reference a comma-separated value file. |
+| FORMAT 'CUSTOM' | Use the `'CUSTOM'` `FORMAT` with the
`SequenceWritable` profile. The `SequenceWritable` '`CUSTOM`' `FORMAT` supports
only the built-in `(formatter='pxfwritable_export)` (write) and
`(formatter='pxfwritable_import)` (read) \.
+
+**Note**: When creating PXF external tables, you cannot use the `HEADER`
option in your `FORMAT` specification.
+
+## Custom Options
+
+The `HdfsTextSimple` and `SequenceWritable` profiles support the following
\:
+
+| Keyword | Value Description |
+|---|-|
+| COMPRESSION_CODEC| The compression codec Java class name. If this
option is not provided, no data compression is performed. Supported compression
codecs include: `org.apache.hadoop.io.compress.DefaultCodec`,
`org.apache.hadoop.io.compress.BZip2Codec`, and
`org.apache.hadoop.io.compress.GzipCodec` (`HdfsTextSimple` profile only) |
+| COMPRESSION_TYPE| The compression type to employ; supported values
are `RECORD` (the default) or `BLOCK`. |
+| DATA-SCHEMA| (`SequenceWritable` profile only) The name of the
writer serialization/deserialization class. The jar file in which this class
resides must be in the PXF class path. This option has no default value. |
+| THREAD-SAFE | Boolean value determining if a table query can run in
multi-thread mode. Default value is `TRUE`, requests run in multi-threaded
mode. When set to `FALSE`, requests will be handled in a single thread.
`THREAD-SAFE` should be set appropriately when operations that are not
thread-safe are performed (i.e. compression). |
+
+## HdfsTextSimple Profile
+
+Use the `HdfsTextSimple` profile when writing delimited data to a plain
text file where each row is a single record.
+
+Writable tables created using the `HdfsTextSimple` profile can use no,
record, or block compression. When compression is used, the default, gzip, and
bzip2 Hadoop compression codecs are supported:
+
+- org.apache.hadoop.io.compress.DefaultCodec
+- org.apache.hadoop.io.compres
[jira] [Commented] (HAWQ-1119) create new documentation topic for PXF writable profiles
[
https://issues.apache.org/jira/browse/HAWQ-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15623612#comment-15623612
]
ASF GitHub Bot commented on HAWQ-1119:
--
Github user dyozie commented on a diff in the pull request:
https://github.com/apache/incubator-hawq-docs/pull/46#discussion_r85814125
--- Diff: pxf/HDFSWritablePXF.html.md.erb ---
@@ -0,0 +1,410 @@
+---
+title: Writing Data to HDFS
+---
+
+The PXF HDFS plug-in supports writable external tables using the
`HdfsTextSimple` and `SequenceWritable` profiles. You might create a writable
table to export data from a HAWQ internal table to HDFS.
+
+This section describes how to use these PXF profiles to create writable
external tables.
+
+**Note**: You cannot directly query data in a HAWQ writable table. After
creating the external writable table, you must create a HAWQ readable external
table accessing the HDFS file, then query that table. ??You can also create a
Hive table to access the HDFS file.??
+
+## Prerequisites
+
+Before working with HDFS file data using HAWQ and PXF, ensure that:
+
+- The HDFS plug-in is installed on all cluster nodes. See [Installing
PXF Plug-ins](InstallPXFPlugins.html) for PXF plug-in installation information.
+- All HDFS users have read permissions to HDFS services and that write
permissions have been restricted to specific users.
+
+## Writing to PXF External Tables
+The PXF HDFS plug-in supports writable two profiles: `HdfsTextSimple` and
`SequenceWritable`.
+
+Use the following syntax to create a HAWQ external writable table
representing HDFS data:
+
+``` sql
+CREATE EXTERNAL WRITABLE TABLE
+( [, ...] | LIKE )
+LOCATION ('pxf://[:]/
+
?PROFILE=HdfsTextSimple|SequenceWritable[&=[...]]')
+FORMAT '[TEXT|CSV|CUSTOM]' ();
+```
+
+HDFS-plug-in-specific keywords and values used in the [CREATE EXTERNAL
TABLE](../reference/sql/CREATE-EXTERNAL-TABLE.html) call are described in the
table below.
+
+| Keyword | Value |
+|---|-|
+| \[:\]| The HDFS NameNode and port. |
+| \| The path to the file in the HDFS data store. |
+| PROFILE| The `PROFILE` keyword must specify one of the values
`HdfsTextSimple` or `SequenceWritable`. |
+| \ | \ is profile-specific. These
options are discussed in the next topic.|
+| FORMAT 'TEXT' | Use '`TEXT`' `FORMAT` with the `HdfsTextSimple` profile
when \ will reference a plain text delimited file. The
`HdfsTextSimple` '`TEXT`' `FORMAT` supports only the built-in
`(delimiter=)` \. |
+| FORMAT 'CSV' | Use '`CSV`' `FORMAT` with `HdfsTextSimple` when
\ will reference a comma-separated value file. |
+| FORMAT 'CUSTOM' | Use the `'CUSTOM'` `FORMAT` with the
`SequenceWritable` profile. The `SequenceWritable` '`CUSTOM`' `FORMAT` supports
only the built-in `(formatter='pxfwritable_export)` (write) and
`(formatter='pxfwritable_import)` (read) \.
+
+**Note**: When creating PXF external tables, you cannot use the `HEADER`
option in your `FORMAT` specification.
+
+## Custom Options
+
+The `HdfsTextSimple` and `SequenceWritable` profiles support the following
\:
+
+| Keyword | Value Description |
+|---|-|
+| COMPRESSION_CODEC| The compression codec Java class name. If this
option is not provided, no data compression is performed. Supported compression
codecs include: `org.apache.hadoop.io.compress.DefaultCodec`,
`org.apache.hadoop.io.compress.BZip2Codec`, and
`org.apache.hadoop.io.compress.GzipCodec` (`HdfsTextSimple` profile only) |
+| COMPRESSION_TYPE| The compression type to employ; supported values
are `RECORD` (the default) or `BLOCK`. |
+| DATA-SCHEMA| (`SequenceWritable` profile only) The name of the
writer serialization/deserialization class. The jar file in which this class
resides must be in the PXF class path. This option has no default value. |
+| THREAD-SAFE | Boolean value determining if a table query can run in
multi-thread mode. Default value is `TRUE`, requests run in multi-threaded
mode. When set to `FALSE`, requests will be handled in a single thread.
`THREAD-SAFE` should be set appropriately when operations that are not
thread-safe are performed (i.e. compression). |
+
+## HdfsTextSimple Profile
+
+Use the `HdfsTextSimple` profile when writing delimited data to a plain
text file where each row is a single record.
+
+Writable tables created using the `HdfsTextSimple` profile can use no,
record, or block compression. When compression is used, the default, gzip, and
bzip2 Hadoop compression codecs are supported:
+
+- org.apache.hadoop.io.compress.DefaultCodec
+- org.apache.hadoop.io.compres
[GitHub] incubator-hawq pull request #972: HAWQ-1108 Add JDBC PXF Plugin
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/972#discussion_r85825494
--- Diff:
pxf/pxf-jdbc/src/main/java/org/apache/hawq/pxf/plugins/jdbc/JdbcPartitionFragmenter.java
---
@@ -0,0 +1,298 @@
+package org.apache.hawq.pxf.plugins.jdbc;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.hawq.pxf.api.Fragmenter;
+import org.apache.hawq.pxf.api.FragmentsStats;
+import org.apache.hawq.pxf.api.UserDataException;
+import org.apache.hawq.pxf.plugins.jdbc.utils.DbProduct;
+import org.apache.hawq.pxf.plugins.jdbc.utils.ByteUtil;
+import org.apache.hawq.pxf.api.Fragment;
+import org.apache.hawq.pxf.api.utilities.InputData;
+
+import java.net.InetAddress;
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+/**
+ * Fragmenter class for JDBC data resources.
+ *
+ * Extends the {@link Fragmenter} abstract class, with the purpose of
transforming
+ * an input data path (an JDBC Database table name and user request
parameters) into a list of regions
+ * that belong to this table.
+ *
+ * The parameter Patterns
+ * There are three parameters, the format is as follows:
+ *
+ *
PARTITION_BY=column_name:column_type&RANGE=start_value[:end_value]&INTERVAL=interval_num[:interval_unit]
+ *
+ * The PARTITION_BY parameter can be split by colon(':'),the
column_type current supported : date,int,enum .
+ * The Date format is '-MM-dd'.
+ * The RANGE parameter can be split by colon(':') ,used to
identify the starting range of each fragment.
+ * The range is left-closed, ie: '>= start_value AND < end_value'
.If the column_type is int,
+ * the end_value can be empty. If the
column_typeis enum,the parameter RANGE
can be empty.
+ * The INTERVAL parameter can be split by colon(':'),
indicate the interval value of one fragment.
+ * When column_type is date,this parameter must
be split by colon, and interval_unit can be
year,month,day.
+ * When column_type is int, the
interval_unit can be empty.
+ * When column_type is enum,the
INTERVAL parameter can be empty.
+ *
+ *
+ * The syntax examples is :
+ *
PARTITION_BY=createdate:date&RANGE=2008-01-01:2010-01-01&INTERVAL=1:month'
+ * PARTITION_BY=year:int&RANGE=2008:2010&INTERVAL=1
+ * PARTITION_BY=grade:enum&RANGE=excellent:good:general:bad
+ *
+ *
+ */
+public class JdbcPartitionFragmenter extends Fragmenter {
+String[] partitionBy = null;
+String[] range = null;
+String[] interval = null;
+PartitionType partitionType = null;
+String partitionColumn = null;
+IntervalType intervalType = null;
+int intervalNum = 1;
+
+enum PartitionType {
+DATE,
+INT,
+ENUM;
+
+public static PartitionType getType(String str) {
+return valueOf(str.toUpperCase());
+}
+}
+
+enum IntervalType {
+DAY,
+MONTH,
+YEAR;
+
+public static IntervalType type(String str) {
+return valueOf(str.toUpperCase());
+}
+}
+
+//The unit interval, in milliseconds, that is used to estimate the
number of slices for the date partition type
+static Map intervals = new HashMap();
+
+static {
+intervals.put(IntervalType.DAY, (long) 24 * 60 * 60 * 1000);
+//30 days
+intervals.put(IntervalType.MONTH, (long) 30 * 24 * 60 * 60 * 1000);
+//365 days
+intervals.put(IntervalType.YEAR, (long) 365 * 30 * 24 * 60 * 60 *
1000);
+}
+
+/**
+ * Constructor for JdbcPartitionFragmenter.
+ *
+ * @param inConf input data such as which Jdbc table to scan
+ * @throws UserDataException
+ */
+public JdbcPartitionFragmenter(InputData inConf) throws
UserDataException {
+super(inConf);
+
[GitHub] incubator-hawq issue #720: HAWQ-823. Initial Amazon S3 External Table Suppor...
Github user kdunn-pivotal commented on the issue: https://github.com/apache/incubator-hawq/pull/720 Anything else needed here? This is a critical feature both Hive and Impala already offer. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #990: HAWQ-1128. Support HAWQ register tables with same...
Github user xunzhang commented on the issue: https://github.com/apache/incubator-hawq/pull/990 cc @wcl14 @wengyanqing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #990: HAWQ-1128. Support HAWQ register tables wi...
GitHub user xunzhang opened a pull request: https://github.com/apache/incubator-hawq/pull/990 HAWQ-1128. Support HAWQ register tables with same file name in different schema. You can merge this pull request into a Git repository by running: $ git pull https://github.com/xunzhang/incubator-hawq HAWQ-1128 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/990.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #990 commit f4dc792f50d7107728693a5195ee4d8207d16866 Author: xunzhang Date: 2016-10-31T13:34:11Z HAWQ-1128. Support HAWQ register tables with same file name in different schema. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Assigned] (HAWQ-1128) Support HAWQ register tables with same file name in different schema
[ https://issues.apache.org/jira/browse/HAWQ-1128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongwu reassigned HAWQ-1128: Assignee: hongwu (was: Chunling Wang) > Support HAWQ register tables with same file name in different schema > > > Key: HAWQ-1128 > URL: https://issues.apache.org/jira/browse/HAWQ-1128 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Command Line Tools >Reporter: Chunling Wang >Assignee: hongwu > Fix For: backlog > > > Now, in HAWQ Register, it can not distinguish tables with same file name but > in different schema, which are regarded as same table. We should save and use > schema information for HAWQ register. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-hawq issue #989: HAWQ-1129. Install PLR into hawq home directory
Github user wengyanqing commented on the issue: https://github.com/apache/incubator-hawq/pull/989 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #989: HAWQ-1129. Install PLR into hawq home dire...
GitHub user radarwave opened a pull request: https://github.com/apache/incubator-hawq/pull/989 HAWQ-1129. Install PLR into hawq home directory You can merge this pull request into a Git repository by running: $ git pull https://github.com/radarwave/incubator-hawq pplr Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/989.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #989 commit 551544b5d309c027232045713477e4906280c924 Author: rlei Date: 2016-10-31T07:36:04Z HAWQ-1129. Install PLR into hawq home directory --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Assigned] (HAWQ-1129) plr-hawq default install directory should be the same as HAWQ install directory
[ https://issues.apache.org/jira/browse/HAWQ-1129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radar Lei reassigned HAWQ-1129: --- Assignee: Radar Lei (was: Lei Chang) > plr-hawq default install directory should be the same as HAWQ install > directory > --- > > Key: HAWQ-1129 > URL: https://issues.apache.org/jira/browse/HAWQ-1129 > Project: Apache HAWQ > Issue Type: Bug > Components: Command Line Tools >Reporter: Radar Lei >Assignee: Radar Lei > Fix For: 2.0.1.0-incubating > > > Currently plr-hawq default install path is '/usr/local/hawq_2_0_1_0-2.0.1.0', > this is different from the hawq default install path > '/usr/local/hawq_2_0_1_0'. > We should keep them the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HAWQ-1129) plr-hawq default install directory should be the same as HAWQ install directory
Radar Lei created HAWQ-1129: --- Summary: plr-hawq default install directory should be the same as HAWQ install directory Key: HAWQ-1129 URL: https://issues.apache.org/jira/browse/HAWQ-1129 Project: Apache HAWQ Issue Type: Bug Components: Command Line Tools Reporter: Radar Lei Assignee: Lei Chang Fix For: 2.0.1.0-incubating Currently plr-hawq default install path is '/usr/local/hawq_2_0_1_0-2.0.1.0', this is different from the hawq default install path '/usr/local/hawq_2_0_1_0'. We should keep them the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HAWQ-1128) Support HAWQ register tables with same file name in different schema
[ https://issues.apache.org/jira/browse/HAWQ-1128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chunling Wang reassigned HAWQ-1128: --- Assignee: Chunling Wang (was: Lei Chang) > Support HAWQ register tables with same file name in different schema > > > Key: HAWQ-1128 > URL: https://issues.apache.org/jira/browse/HAWQ-1128 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Command Line Tools >Reporter: Chunling Wang >Assignee: Chunling Wang > Fix For: backlog > > > Now, in HAWQ Register, it can not distinguish tables with same file name but > in different schema, which are regarded as same table. We should save and use > schema information for HAWQ register. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HAWQ-1028) Add '-d' option for hawq state to be compatible with Ambari
[ https://issues.apache.org/jira/browse/HAWQ-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radar Lei resolved HAWQ-1028. - Resolution: Fixed > Add '-d' option for hawq state to be compatible with Ambari > --- > > Key: HAWQ-1028 > URL: https://issues.apache.org/jira/browse/HAWQ-1028 > Project: Apache HAWQ > Issue Type: Task > Components: Command Line Tools >Reporter: Radar Lei >Assignee: Radar Lei > Fix For: 2.0.1.0-incubating > > > Previously we removed the legacy option '-d' '--datadir' from 'hawq state' > command. This option is used to specify the master data directory, but we > never used it in our command line tools. > Now we found this unused option is used by current version Ambari, and will > cause Ambari check HAWQ status failed if we removed it. So to be compatible > with Ambari, we need to add it back until Ambari do not use this option. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HAWQ-1067) Append hawq version number to plr-hawq rpm pakcage name
[ https://issues.apache.org/jira/browse/HAWQ-1067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radar Lei resolved HAWQ-1067. - Resolution: Fixed > Append hawq version number to plr-hawq rpm pakcage name > --- > > Key: HAWQ-1067 > URL: https://issues.apache.org/jira/browse/HAWQ-1067 > Project: Apache HAWQ > Issue Type: Task > Components: Build >Reporter: Radar Lei >Assignee: Radar Lei > Fix For: 2.0.1.0-incubating > > > Now plr-hawq using plr vesion. It's not easy to align with HAWQ version. > So we'd better make below changes: > 1. Add hawq version string to plr-hawq rpm package name. > 2. Create a virtual rpm package to make sure user have same install interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HAWQ-1128) Support HAWQ register tables with same file name in different schema
Chunling Wang created HAWQ-1128: --- Summary: Support HAWQ register tables with same file name in different schema Key: HAWQ-1128 URL: https://issues.apache.org/jira/browse/HAWQ-1128 Project: Apache HAWQ Issue Type: Sub-task Components: Command Line Tools Reporter: Chunling Wang Assignee: Lei Chang Now, in HAWQ Register, it can not distinguish tables with same file name but in different schema, which are regarded as same table. We should save and use schema information for HAWQ register. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HAWQ-1127) HAWQ should print error message instead of python function stack when yaml file is invalid.
[ https://issues.apache.org/jira/browse/HAWQ-1127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lin Wen resolved HAWQ-1127. --- Resolution: Fixed Fix Version/s: 2.0.1.0-incubating > HAWQ should print error message instead of python function stack when yaml > file is invalid. > --- > > Key: HAWQ-1127 > URL: https://issues.apache.org/jira/browse/HAWQ-1127 > Project: Apache HAWQ > Issue Type: Bug > Components: Command Line Tools >Reporter: Lin Wen >Assignee: Lin Wen > Fix For: 2.0.1.0-incubating > > Attachments: force_mode_normal_tpl.yml > > > when use a invalid yaml file to register, hawq prints python stack: > [linw@linw-rhel feature]$ hawq register --force -d hawq_feature_test -c > /home/linw/workspace/hawq_working/apache-hawq/src/test/feature/ManagementTool/partition/force_mode_normal.yml > testhawqregister_testpartitionforcemodenormal.nt > 20161031:12:48:49:557022 hawqregister:linw-rhel:linw-[INFO]:-try to connect > database localhost:5432 hawq_feature_test > Traceback (most recent call last): > File "/home/linw/hawq-bin/bin/hawqregister", line 1137, in > main(options, args) > File "/home/linw/hawq-bin/bin/hawqregister", line 1093, in main > ins.prepare() > File "/home/linw/hawq-bin/bin/hawqregister", line 1021, in prepare > self._option_parser_yml(options.yml_config) > File "/home/linw/hawq-bin/bin/hawqregister", line 475, in _option_parser_yml > partitions_constraint = [d['Constraint'] for d in > params[Format_FileLocations]['Partitions']] > KeyError: 'Constraint' > Instead, hawq should print an error message. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-hawq pull request #988: HAWQ should print error message instead of...
Github user linwen closed the pull request at: https://github.com/apache/incubator-hawq/pull/988 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
