[jira] [Created] (HAWQ-1623) Automatic master node HA

[Lei Chang (JIRA)]

Lei Chang created HAWQ-1623:
---

 Summary: Automatic master node HA
 Key: HAWQ-1623
 URL: https://issues.apache.org/jira/browse/HAWQ-1623
 Project: Apache HAWQ
  Issue Type: New Feature
Reporter: Lei Chang
Assignee: Radar Lei


 

In current HAWQ, when master node dies, it needs manual switch from master to 
standby. It is not convenient for end users. Let's add an automatic failover 
mechanism.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HAWQ-1547) Increase default table name length from 64 to 128 to match Hive

[Lei Chang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252770#comment-16252770
 ] 

Lei Chang commented on HAWQ-1547:
-

Looks a good improvement. 

> Increase default table name length from 64 to 128 to match Hive
> ---
>
> Key: HAWQ-1547
> URL: https://issues.apache.org/jira/browse/HAWQ-1547
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Unknown
>Reporter: Grant Krieger
>Assignee: Radar Lei
>
> Hi,
> Would it be possible to increase 
> incubator-hawq/src/include/pg_config_manual.h property default NAMEDATALEN 
> from 64 to 128. 
> This hopefully will allow one to be able to read Hive tables from Hawq larger 
> than 64 by default without having to change this setting when compiling for 
> downstream systems. It will also allow for equivalently name Hawq tables.
> Does anyone foresee performance challenges with the increase? 
> See problem below:
> In hive
> CREATE TABLE 
> default.test123456789123456789123456789123456789123456789123456789123456789123456789123456789test
>  ( 
>   rtlymth int, 
>   rtlyint)
> STORED AS ORC
> in Hawq
> select * from 
> hcatalog.default.test123456789123456789123456789123456789123456789123456789123456789123456789123456789test
>  ERROR: remote component error (500) from '127.0.0.1:51200':  type  Exception 
> report   message   
> NoSuchObjectException(message:default.test12345678912345678912345678912345678912345678912345678912345
>  table not found)description   The server encountered an internal error 
> that prevented it from fulfilling this request.exception   
> javax.servlet.ServletException: 
> NoSuchObjectException(message:default.test12345678912345678912345678912345678912345678912345678912345
>  table not found) (libchurl.c:897)
>   Position: 15
>  Line: 1 
> Thanks
> Grant



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HAWQ-1530) Illegally killing a JDBC select query causes locking problems

[Lei Chang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241629#comment-16241629
 ] 

Lei Chang commented on HAWQ-1530:
-

Thanks [~kuien] .

I think [~yjin] fixed a bug before, it is quite similar to this bug. 

> Illegally killing a JDBC select query causes locking problems
> -
>
> Key: HAWQ-1530
> URL: https://issues.apache.org/jira/browse/HAWQ-1530
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Transaction
>Reporter: Grant Krieger
>Assignee: Radar Lei
>
> Hi,
> When you perform a long running select statement on 2 hawq tables (join) from 
> JDBC and illegally kill the JDBC client (CTRL ALT DEL) before completion of 
> the query the 2 tables remained locked even when the query completes on the 
> server. 
> The lock is visible via PG_locks. One cannot kill the query via SELECT 
> pg_terminate_backend(393937). The only way to get rid of it is to kill -9 
> from linux or restart hawq but this can kill other things as well.
> The JDBC client I am using is Aqua Data Studio.
> I can provide exact steps to reproduce if required
> Thank you
> Grant 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HAWQ-786) Framework to support pluggable formats and file systems

[Lei Chang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103037#comment-16103037
 ] 

Lei Chang commented on HAWQ-786:


[~rlei] This feature has been implemented in Oushu commercial hawq version. 
currently we are refactoring the code and making it work on oss hawq. After it 
is ready, we will update this JIRA. 

> Framework to support pluggable formats and file systems
> ---
>
> Key: HAWQ-786
> URL: https://issues.apache.org/jira/browse/HAWQ-786
> Project: Apache HAWQ
>  Issue Type: Wish
>  Components: Storage
>Reporter: Lei Chang
>Assignee: Lei Chang
> Fix For: backlog
>
> Attachments: ORCdesign-v0.1-2016-06-17.pdf
>
>
> In current HAWQ, two native formats are supported: AO and parquet. Now we 
> want to support ORC. A framework to support naive c/c++ pluggable format is 
> needed to support ORC more easily. And it can also be potentially used for 
> fast external data access.
> And there are a lot of requests for supporting S3, Ceph and other file 
> systems, and this is closely related to pluggable formats, so this JIRA is 
> proposing a framework to support both.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HAWQ-786) Framework to support pluggable formats and file systems

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang reassigned HAWQ-786:
--

Assignee: Lei Chang  (was: hongwu)

> Framework to support pluggable formats and file systems
> ---
>
> Key: HAWQ-786
> URL: https://issues.apache.org/jira/browse/HAWQ-786
> Project: Apache HAWQ
>  Issue Type: Wish
>  Components: Storage
>Reporter: Lei Chang
>Assignee: Lei Chang
> Fix For: backlog
>
> Attachments: ORCdesign-v0.1-2016-06-17.pdf
>
>
> In current HAWQ, two native formats are supported: AO and parquet. Now we 
> want to support ORC. A framework to support naive c/c++ pluggable format is 
> needed to support ORC more easily. And it can also be potentially used for 
> fast external data access.
> And there are a lot of requests for supporting S3, Ceph and other file 
> systems, and this is closely related to pluggable formats, so this JIRA is 
> proposing a framework to support both.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HAWQ-786) Framework to support pluggable formats and file systems

[Lei Chang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065952#comment-16065952
 ] 

Lei Chang commented on HAWQ-786:


This feature has been not active for some time. A lot of hawq users are asking 
this feature. currently, we are working on this feature. I am assigning this 
issue to me. Hope that we can release this feature in next release.





> Framework to support pluggable formats and file systems
> ---
>
> Key: HAWQ-786
> URL: https://issues.apache.org/jira/browse/HAWQ-786
> Project: Apache HAWQ
>  Issue Type: Wish
>  Components: Storage
>Reporter: Lei Chang
>Assignee: hongwu
> Fix For: backlog
>
> Attachments: ORCdesign-v0.1-2016-06-17.pdf
>
>
> In current HAWQ, two native formats are supported: AO and parquet. Now we 
> want to support ORC. A framework to support naive c/c++ pluggable format is 
> needed to support ORC more easily. And it can also be potentially used for 
> fast external data access.
> And there are a lot of requests for supporting S3, Ceph and other file 
> systems, and this is closely related to pluggable formats, so this JIRA is 
> proposing a framework to support both.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HAWQ-1450) New HAWQ executor with vectorization & possible code generation

[Lei Chang (JIRA)]

Lei Chang created HAWQ-1450:
---

 Summary: New HAWQ executor with vectorization & possible code 
generation
 Key: HAWQ-1450
 URL: https://issues.apache.org/jira/browse/HAWQ-1450
 Project: Apache HAWQ
  Issue Type: New Feature
  Components: Query Execution
Reporter: Lei Chang
Assignee: Lei Chang
 Fix For: backlog



Most HAWQ executor code is inherited from postgres & gpdb. Let's discuss how to 
build a new hawq executor with vectorization and possibly code generation. 
These optimization may potentially improve the query performance a lot.





--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HAWQ-1436) Implement RPS High availability on HAWQ

[Lei Chang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-1436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15983898#comment-15983898
 ] 

Lei Chang commented on HAWQ-1436:
-

Nice doc. Can we treat RPS process similar to a resource manager process? 
postmaster can restart the process automatically. It might potentially simplify 
the design a lot.



> Implement RPS High availability on HAWQ
> ---
>
> Key: HAWQ-1436
> URL: https://issues.apache.org/jira/browse/HAWQ-1436
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Security
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
> Fix For: backlog
>
> Attachments: RPSHADesign_v0.1.pdf
>
>
> Once Ranger is configured, HAWQ will rely on RPS to connect to Ranger. A 
> single point RPS may influence the robustness of HAWQ. 
> Thus We need to investigate and design out the way to implement RPS High 
> availability. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] (HAWQ-1300) hawq cannot compile with Bison 3.x.

[Lei Chang (JIRA)]

Title: Message Title
 
 
 
 
 
 
 
 
 
 
  
 
 Lei Chang created an issue 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
 Apache HAWQ /  HAWQ-1300 
 
 
 
  hawq cannot compile with Bison 3.x.  
 
 
 
 
 
 
 
 
 

Issue Type:
 
  Bug 
 
 
 

Assignee:
 
 Ed Espino 
 
 
 

Components:
 

 Build 
 
 
 

Created:
 

 31/Jan/17 01:24 
 
 
 

Fix Versions:
 

 backlog 
 
 
 

Priority:
 
  Major 
 
 
 

Reporter:
 
 Lei Chang 
 
 
 
 
 
 
 
 
 
 
Yes, I met similar issue, Bison 3.x does not work for HAWQ now. 
On Mon, Jan 30, 2017 at 12:37 PM, Dmitry Bouzolin < 
dbouzo...@yahoo.com.invalid> wrote: 
> Hi Lei, > I use Bison 3.0.2. And looks actually like a bug in gram.c source for this > Bison version.The function refers yyscanner which is not defined. I will > reach out Bison bug list.Thanks for reply! > > On Sunday, January 29, 2017 8:09 PM, Lei Chang  > wrote: > > > Hi Dmitry, > > Which bison version do you use? Looks this is a known issue when compiling > hawq on latest bison (3.x) version. Bison 2.x version should work. > > Thanks > Lei > > > > > On Mon, Jan 30, 2017 at 3:41 AM, Dmitry Bouzolin < > dbouzo...@yahoo.com.invalid> wrote: > > > Hi All, > > Yes, I know arch linux is not supported, however I appreciate any clues > on > > why the build would fail like so: > > > > make -C caql allmake[4]: Entering directory > '/data/src/incubator-hawq/src/ > > backend/catalog/caql' > > gcc -O3 -std=gnu99 -Wall -Wmissing-prototypes 

[jira] [Created] (HAWQ-1255) Looks "segment size with penalty" number in "explain analyze" not correct

[Lei Chang (JIRA)]

Lei Chang created HAWQ-1255:
---

 Summary: Looks "segment size with penalty" number in "explain 
analyze" not correct
 Key: HAWQ-1255
 URL: https://issues.apache.org/jira/browse/HAWQ-1255
 Project: Apache HAWQ
  Issue Type: Bug
  Components: Query Execution
Reporter: Lei Chang
Assignee: Lei Chang



"segment size" is about 500MB, while "segment size with penalty" is about 
100MB. Looks not reasonable.

How to reproduce:
on laptop, 1G tpch data, lineitem table is created as hash distributed with 2 
buckets, and orders table is randomly.


```
postgres=# explain analyze SELECT l_orderkey, count(l_quantity) 
 FROM 
lineitem_b2, orders 
WHERE 
l_orderkey = o_orderkey 
   GROUP BY 
l_orderkey;


QUERY PLAN  


  
--
 Gather Motion 2:1  (slice2; segments: 2)  (cost=291580.96..318527.67 
rows=1230576 width=16)
   Rows out:  Avg 150.0 rows x 1 workers at destination.  
Max/Last(seg-1:changlei.local/seg-1:changlei.local) 150/150 rows with 
2209/2209 ms to first row, 2577/2577 ms to end, start offset by 1.429/1.429 ms.
   ->  HashAggregate  (cost=291580.96..318527.67 rows=615288 width=16)
 Group By: lineitem_b2.l_orderkey
 Rows out:  Avg 75.0 rows x 2 workers.  
Max/Last(seg1:changlei.local/seg1:changlei.local) 75/75 rows with 
2243/2243 ms to first row, 2498/2498 ms to end, start offset by 2.615/2.615 ms.
 Executor memory:  56282K bytes avg, 56282K bytes max 
(seg1:changlei.local).
 ->  Hash Join  (cost=70069.00..250010.38 rows=3000608 width=15)
   Hash Cond: lineitem_b2.l_orderkey = orders.o_orderkey
   Rows out:  Avg 3000607.5 rows x 2 workers.  
Max/Last(seg0:changlei.local/seg1:changlei.local) 3001300/215 rows with 
350/350 ms to first row, 1611/1645 ms to end, start offset by 3.819/3.816 ms.
   Executor memory:  49153K bytes avg, 49153K bytes max 
(seg1:changlei.local).
   Work_mem used:  23438K bytes avg, 23438K bytes max 
(seg1:changlei.local). Workfile: (0 spilling, 0 reused)
   (seg0)   Hash chain length 1.7 avg, 3 max, using 434205 of 
524341 buckets.
   ->  Append-only Scan on lineitem_b2  (cost=0.00..89923.15 
rows=3000608 width=15)
 Rows out:  Avg 3000607.5 rows x 2 workers.  
Max/Last(seg0:changlei.local/seg1:changlei.local) 3001300/215 rows with 
4.460/4.757 ms to first row, 546/581 ms to end, start offset by 350/349 ms.
   ->  Hash  (cost=51319.00..51319.00 rows=75 width=8)
 Rows in:  Avg 75.0 rows x 2 workers.  
Max/Last(seg1:changlei.local/seg0:changlei.local) 75/75 rows with 
341/344 ms to end, start offset by 8.114/5.610 ms.
 ->  Redistribute Motion 2:2  (slice1; segments: 2)  
(cost=0.00..51319.00 rows=75 width=8)
   Hash Key: orders.o_orderkey
   Rows out:  Avg 75.0 rows x 2 workers at 
destination.  Max/Last(seg1:changlei.local/seg0:changlei.local) 75/75 
rows with 0.052/2.461 ms to first row, 207/207 ms to end, start offset by 
8.114/5.611 ms.
   ->  Append-only Scan on orders  (cost=0.00..21319.00 
rows=75 width=8)
 Rows out:  Avg 75.0 rows x 2 workers.  
Max/Last(seg1:changlei.local/seg0:changlei.local) 75/75 rows with 
4.773/4.987 ms to first row, 166/171 ms to end, start offset by 2.911/2.697 ms.
 Slice statistics:
   (slice0)Executor memory: 281K bytes.
   (slice1)Executor memory: 319K bytes avg x 2 workers, 319K bytes max 
(seg1:changlei.local).
   (slice2)Executor memory: 105773K bytes avg x 2 workers, 105773K bytes 
max 

[jira] [Updated] (HAWQ-1255) Looks "segment size with penalty" number in "explain analyze" not correct

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-1255:

Assignee: Hubert Zhang  (was: Lei Chang)

> Looks "segment size with penalty" number in "explain analyze" not correct
> -
>
> Key: HAWQ-1255
> URL: https://issues.apache.org/jira/browse/HAWQ-1255
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Query Execution
>Reporter: Lei Chang
>Assignee: Hubert Zhang
>
> "segment size" is about 500MB, while "segment size with penalty" is about 
> 100MB. Looks not reasonable.
> How to reproduce:
> on laptop, 1G tpch data, lineitem table is created as hash distributed with 2 
> buckets, and orders table is randomly.
> ```
> postgres=# explain analyze SELECT l_orderkey, count(l_quantity)   
>
> FROM lineitem_b2, orders  
>
> WHERE l_orderkey = o_orderkey 
>
> GROUP BY l_orderkey;
>   
>   
> QUERY PLAN
>   
>   
> 
> --
>  Gather Motion 2:1  (slice2; segments: 2)  (cost=291580.96..318527.67 
> rows=1230576 width=16)
>Rows out:  Avg 150.0 rows x 1 workers at destination.  
> Max/Last(seg-1:changlei.local/seg-1:changlei.local) 150/150 rows with 
> 2209/2209 ms to first row, 2577/2577 ms to end, start offset by 1.429/1.429 
> ms.
>->  HashAggregate  (cost=291580.96..318527.67 rows=615288 width=16)
>  Group By: lineitem_b2.l_orderkey
>  Rows out:  Avg 75.0 rows x 2 workers.  
> Max/Last(seg1:changlei.local/seg1:changlei.local) 75/75 rows with 
> 2243/2243 ms to first row, 2498/2498 ms to end, start offset by 2.615/2.615 
> ms.
>  Executor memory:  56282K bytes avg, 56282K bytes max 
> (seg1:changlei.local).
>  ->  Hash Join  (cost=70069.00..250010.38 rows=3000608 width=15)
>Hash Cond: lineitem_b2.l_orderkey = orders.o_orderkey
>Rows out:  Avg 3000607.5 rows x 2 workers.  
> Max/Last(seg0:changlei.local/seg1:changlei.local) 3001300/215 rows with 
> 350/350 ms to first row, 1611/1645 ms to end, start offset by 3.819/3.816 ms.
>Executor memory:  49153K bytes avg, 49153K bytes max 
> (seg1:changlei.local).
>Work_mem used:  23438K bytes avg, 23438K bytes max 
> (seg1:changlei.local). Workfile: (0 spilling, 0 reused)
>(seg0)   Hash chain length 1.7 avg, 3 max, using 434205 of 
> 524341 buckets.
>->  Append-only Scan on lineitem_b2  (cost=0.00..89923.15 
> rows=3000608 width=15)
>  Rows out:  Avg 3000607.5 rows x 2 workers.  
> Max/Last(seg0:changlei.local/seg1:changlei.local) 3001300/215 rows with 
> 4.460/4.757 ms to first row, 546/581 ms to end, start offset by 350/349 ms.
>->  Hash  (cost=51319.00..51319.00 rows=75 width=8)
>  Rows in:  Avg 75.0 rows x 2 workers.  
> Max/Last(seg1:changlei.local/seg0:changlei.local) 75/75 rows with 
> 341/344 ms to end, start offset by 8.114/5.610 ms.
>  ->  Redistribute Motion 2:2  (slice1; segments: 2)  
> (cost=0.00..51319.00 rows=75 width=8)
>Hash Key: orders.o_orderkey
>Rows out:  Avg 75.0 rows x 2 workers at 
> destination.  Max/Last(seg1:changlei.local/seg0:changlei.local) 75/75 
> rows with 0.052/2.461 ms to first row, 207/207 ms to end, start offset by 
> 8.114/5.611 ms.
>->  Append-only Scan on orders  
> (cost=0.00..21319.00 rows=75 width=8)
>  Rows out:  Avg 75.0 rows x 2 workers.  
> 

[jira] [Updated] (HAWQ-982) PL/Python with psycopg2 cannot connect to remote postgres

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-982:
---
Fix Version/s: backlog

> PL/Python with psycopg2 cannot connect to remote postgres
> -
>
> Key: HAWQ-982
> URL: https://issues.apache.org/jira/browse/HAWQ-982
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Core
>Reporter: Lei Chang
>Assignee: Ruilong Huo
> Fix For: backlog
>
>
> For one use case I want to connect to external postgreSQL database from HAWQ 
> PL/Python procedure.
> I use python psycopg2 library.
> Remote postgreSQL server reject connecion from HAWQ  with 
> this error :  FATAL  unsupported frontend protocol 28675.0: server supports 
> 1.0 to 3.0.
> The same python code is running well from OS level.
> I wonder if  it is  HAWQ or PostgreSQL PL/Python interpreter related issiue.
> Any help or pointers would be great.
> ---
> my code below:
> CREATE OR REPLACE FUNCTION dchoma.connection_test( ) RETURNS text AS
> $$
> import psycopg2
> try:
> conn = psycopg2.connect("dbname='database_name' user='user' 
> host='remote_host' password='pass' port=5432")
> return "Connection successful "
> except Exception , msg :
> return "Exception: {m}".format(m=msg)
> $$
> LANGUAGE 'plpythonu' VOLATILE;
> select * from dchoma.connection_test();
> HAWQ version 2.0.1.0 build dev ( compiled from github)
> Remote database version:  PostgreSQL 9.2.15 on x86_64-redhat-linux-gnu, 
> compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-4), 64-bit
> OS: CentOS 7-1511
> i found similar issiue here, but the problem is not solved.
> https://discuss.zendesk.com/hc/en-us/community/posts/200793368-greenplum-dblink-postgresql-remote-is-error



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-982) PL/Python with psycopg2 cannot connect to remote postgres

[Lei Chang (JIRA)]

Lei Chang created HAWQ-982:
--

 Summary: PL/Python with psycopg2 cannot connect to remote postgres
 Key: HAWQ-982
 URL: https://issues.apache.org/jira/browse/HAWQ-982
 Project: Apache HAWQ
  Issue Type: Bug
  Components: Core
Reporter: Lei Chang
Assignee: Lei Chang



For one use case I want to connect to external postgreSQL database from HAWQ 
PL/Python procedure.
I use python psycopg2 library.
Remote postgreSQL server reject connecion from HAWQ  with 
this error :  FATAL  unsupported frontend protocol 28675.0: server supports 1.0 
to 3.0.
The same python code is running well from OS level.

I wonder if  it is  HAWQ or PostgreSQL PL/Python interpreter related issiue.
Any help or pointers would be great.

---
my code below:

CREATE OR REPLACE FUNCTION dchoma.connection_test( ) RETURNS text AS
$$
import psycopg2

try:
conn = psycopg2.connect("dbname='database_name' user='user' 
host='remote_host' password='pass' port=5432")
return "Connection successful "
except Exception , msg :
return "Exception: {m}".format(m=msg)
$$
LANGUAGE 'plpythonu' VOLATILE;

select * from dchoma.connection_test();


HAWQ version 2.0.1.0 build dev ( compiled from github)
Remote database version:  PostgreSQL 9.2.15 on x86_64-redhat-linux-gnu, 
compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-4), 64-bit
OS: CentOS 7-1511

i found similar issiue here, but the problem is not solved.
https://discuss.zendesk.com/hc/en-us/community/posts/200793368-greenplum-dblink-postgresql-remote-is-error




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-982) PL/Python with psycopg2 cannot connect to remote postgres

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-982:
---
Assignee: Ruilong Huo  (was: Lei Chang)

> PL/Python with psycopg2 cannot connect to remote postgres
> -
>
> Key: HAWQ-982
> URL: https://issues.apache.org/jira/browse/HAWQ-982
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Core
>Reporter: Lei Chang
>Assignee: Ruilong Huo
> Fix For: backlog
>
>
> For one use case I want to connect to external postgreSQL database from HAWQ 
> PL/Python procedure.
> I use python psycopg2 library.
> Remote postgreSQL server reject connecion from HAWQ  with 
> this error :  FATAL  unsupported frontend protocol 28675.0: server supports 
> 1.0 to 3.0.
> The same python code is running well from OS level.
> I wonder if  it is  HAWQ or PostgreSQL PL/Python interpreter related issiue.
> Any help or pointers would be great.
> ---
> my code below:
> CREATE OR REPLACE FUNCTION dchoma.connection_test( ) RETURNS text AS
> $$
> import psycopg2
> try:
> conn = psycopg2.connect("dbname='database_name' user='user' 
> host='remote_host' password='pass' port=5432")
> return "Connection successful "
> except Exception , msg :
> return "Exception: {m}".format(m=msg)
> $$
> LANGUAGE 'plpythonu' VOLATILE;
> select * from dchoma.connection_test();
> HAWQ version 2.0.1.0 build dev ( compiled from github)
> Remote database version:  PostgreSQL 9.2.15 on x86_64-redhat-linux-gnu, 
> compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-4), 64-bit
> OS: CentOS 7-1511
> i found similar issiue here, but the problem is not solved.
> https://discuss.zendesk.com/hc/en-us/community/posts/200793368-greenplum-dblink-postgresql-remote-is-error



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-947) set work_mem cannot work

[Lei Chang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397169#comment-15397169
 ] 

Lei Chang commented on HAWQ-947:


we should not configure the work_mem anymore, instead, we can use resource 
queues to configure the memory used.

> set work_mem cannot work
> 
>
> Key: HAWQ-947
> URL: https://issues.apache.org/jira/browse/HAWQ-947
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 2.0.1.0-incubating
>Reporter: Biao Wu
>Assignee: Lei Chang
> Fix For: 2.0.1.0-incubating
>
>
> HAWQ version is 2.0.1.0 build dev.
> EXPLAIN ANALYZE:
> Work_mem: 9554K bytes max, 63834K bytes wanted。
> then set work_mem to '512MB',but not work
> {code:sql}
> test=# EXPLAIN ANALYZE SELECT count(DISTINCT item_sku_id)
> test-# FROM gdm_m03_item_sku_da
> test-# WHERE item_origin ='中国大陆';
>   
>   
>QUERY PLAN
> 
> 
>  Aggregate  (cost=54177150.69..54177150.70 rows=1 width=8)
>Rows out:  Avg 1.0 rows x 1 workers.  
> Max/Last(seg-1:BJHC-HEBE-9014.hadoop.jd.local/seg-1:BJHC-HEBE-9014.hadoop.jd.local)
>  1/1 rows with 532498/532498 ms to end, start offset by 201/201 ms.
>->  Gather Motion 306:1  (slice2; segments: 306)  
> (cost=54177147.60..54177150.68 rows=1 width=8)
>  Rows out:  Avg 306.0 rows x 1 workers at destination.  
> Max/Last(seg-1:BJHC-HEBE-9014.hadoop.jd.local/seg-1:BJHC-HEBE-9014.hadoop.jd.local)
>  306/306 rows with 529394/529394 ms to first row, 532498/532498 ms to end, 
> start offset b
> y 201/201 ms.
>  ->  Aggregate  (cost=54177147.60..54177147.61 rows=1 width=8)
>Rows out:  Avg 1.0 rows x 306 workers.  
> Max/Last(seg305:BJHC-HEBE-9031.hadoop.jd.local/seg258:BJHC-HEBE-9029.hadoop.jd.local)
>  1/1 rows with 530367/532274 ms to end, start offset by 396/246 ms.
>Executor memory:  9554K bytes avg, 9554K bytes max 
> (seg305:BJHC-HEBE-9031.hadoop.jd.local).
>Work_mem used:  9554K bytes avg, 9554K bytes max 
> (seg305:BJHC-HEBE-9031.hadoop.jd.local).
>Work_mem wanted: 63695K bytes avg, 63834K bytes max 
> (seg296:BJHC-HEBE-9031.hadoop.jd.local) to lessen workfile I/O affecting 306 
> workers.
>->  Redistribute Motion 306:306  (slice1; segments: 306)  
> (cost=0.00..53550018.97 rows=819776 width=11)
>  Hash Key: gdm_m03_item_sku_da.item_sku_id
>  Rows out:  Avg 820083.0 rows x 306 workers at 
> destination.  
> Max/Last(seg296:BJHC-HEBE-9031.hadoop.jd.local/seg20:BJHC-HEBE-9016.hadoop.jd.local)
>  821880/818660 rows with 769/771 ms to first row, 524681/525063 ms to e
> nd, start offset by 352/307 ms.
>  ->  Append-only Scan on gdm_m03_item_sku_da  
> (cost=0.00..48532990.00 rows=819776 width=11)
>Filter: item_origin::text = '中国大陆'::text
>Rows out:  Avg 820083.0 rows x 306 workers.  
> Max/Last(seg46:BJHC-HEBE-9017.hadoop.jd.local/seg5:BJHC-HEBE-9015.hadoop.jd.local)
>  893390/810582 rows with 28/127 ms to first row, 73062/526318 ms to end, 
> start off
> set by 354/458 ms.
>  Slice statistics:
>(slice0)Executor memory: 1670K bytes.
>(slice1)Executor memory: 3578K bytes avg x 306 workers, 4711K bytes 
> max (seg172:BJHC-HEBE-9024.hadoop.jd.local).
>(slice2)  * Executor memory: 10056K bytes avg x 306 workers, 10056K bytes 
> max (seg305:BJHC-HEBE-9031.hadoop.jd.local).  Work_mem: 9554K bytes max, 
> 63834K bytes wanted.
>  Statement statistics:
>Memory used: 262144K bytes
>Memory wanted: 64233K bytes
>  Settings:  default_hash_table_bucket_number=6
>  Dispatcher statistics:
>executors used(total/cached/new connection): (612/0/612); dispatcher 
> time(total/connection/dispatch data): (489.036 ms/192.741 ms/293.357 ms).
>dispatch data time(max/min/avg): (37.798 ms/0.011 ms/3.504 ms); consume 
> executor data time(max/min/avg): (0.016 ms/0.002 ms/0.005 ms); free executor 
> time(max/min/avg): (0.000 ms/0.000 ms/0.000 ms).
>  Data locality statistics:
>data locality ratio: 0.864; virtual segment number: 306; different host 
> number: 17; virtual segment number per 

[jira] [Updated] (HAWQ-928) Improve user experience of command-line help for hawq CLI utilities

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-928:
---
Assignee: Radar Lei  (was: Lei Chang)

> Improve user experience of command-line help for hawq CLI utilities
> ---
>
> Key: HAWQ-928
> URL: https://issues.apache.org/jira/browse/HAWQ-928
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Command Line Tools, Documentation
>Reporter: Severine Tymon
>Assignee: Radar Lei
>Priority: Minor
> Fix For: backlog
>
>
> This is an umbrella jira that could be broken into smaller pieces if any of 
> these improvements are deemed worthwhile/useful. While working with the HAWQ 
> CLI, I've noticed some issues with the command help documentation:
> 1) The option used to obtain help is not consistent. Some commands (like hawq 
> checkperf) only take the "-?" option, while others do not take "-?" but take 
> "-h" or "--help". Some do not take "-h" because there is already an "-h" 
> option, typically to supply a hostname.
> 2) Large variations in style and depth of command line help-- some commands 
> (like hawq check) are long, man page style help, while others only provide 
> brief command usage (hawq config). 
> 3) Representations of command-line syntax are inconsistent. Some use <>, some 
> use all caps to indicate user-provided values.
> 4) Indicating the default value would be helpful in some cases. For example, 
> the --locale, --lc* options of hawq init. (We do provide detailed explanation 
> in product docs, but would help if cli informed users where the defaults come 
> from if these options are not specified.)
> 5) Outdated external references ("Report bugs to  postgresql.org>", EMC download links, Greenplum and gpssh, gpssh) 
> 6) Very minor-- Inconsistent word choice, punctuation and spelling 
> (capitalization).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-923) More data skipping optimization for IO intensive performance enhancement

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-923:
---
Summary:  More data skipping optimization for IO intensive performance 
enhancement  (was:  More data skipping technology for IO intensive performance 
enhancement)

>  More data skipping optimization for IO intensive performance enhancement
> -
>
> Key: HAWQ-923
> URL: https://issues.apache.org/jira/browse/HAWQ-923
> Project: Apache HAWQ
>  Issue Type: Wish
>  Components: Query Execution
>Reporter: Ming LI
>Assignee: Lei Chang
> Fix For: backlog
>
>
> see email discussion here: 
> http://mail-archives.apache.org/mod_mbox/hawq-dev/201607.mbox/%3CCA+F1uf=tjcioezkvphsppaog-jg0-0azqtusgr7rv+esv44...@mail.gmail.com%3E
> Data skipping technology can extremely avoiding unnecessary IO,  so it can
> extremely enhance performance for IO intensive query. Including eliminating
> query on unnecessary table partition according to the partition key range ,
> I think more options are available now:
> (1) Parquet / ORC format introduce a lightweight meta data info like
> Min/Max/Bloom filter for each block, such meta data can be exploited when
> predicate/filter info can be fetched before executing scan.
> However now in HAWQ, all data in parquet need to be scanned into memory
> before processing predicate/filter. We don't generate the meta info when
> INSERT into parquet table, the scan executor doesn't utilize the meta info
> neither. Maybe some scan API need to be refactored so that we can get
> predicate/filter
> info before executing base relation scan.
> (2) Base on (1) technology,  especially with Bloom filter, more optimizer
> technology can be explored furthur. E.g. Impala implemented Runtime
> filtering(*https://www.cloudera.com/documentation/enterprise/latest/topics/impala_runtime_filtering.html
> *
> ),  which can be used at
> - dynamic partition pruning
> - converting join predicate to base relation predicate
> It tell the executor to wait for one moment(the interval time can be set in
> guc) before executing base relation scan, if the interested values(e.g. the
> column in join predicate only have very small set) arrived in time, it can
> use these value to filter this scan, if doesn't arrived in time, it scan
> without this filter, which doesn't impact result correctness.
> Unlike (1) technology, this technology cannot be used in any case, it only
> outperform in some cases. So it just add some more query plan
> choices/paths, and the optimizer need based on statistics info to calculate
> the cost, and apply it when cost down.
> All in one, maybe more similar technology can be adoptable for HAWQ now,
> let's start to think about performance related technology, moreover we need
> to instigate how these technology can be implemented in HAWQ.
> Any ideas or suggestions are welcomed? Thanks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-923) More data skipping technology for IO intensive performance enhancement

[Lei Chang (JIRA)]

Lei Chang created HAWQ-923:
--

 Summary:  More data skipping technology for IO intensive 
performance enhancement
 Key: HAWQ-923
 URL: https://issues.apache.org/jira/browse/HAWQ-923
 Project: Apache HAWQ
  Issue Type: Bug
  Components: Query Execution
Reporter: Ming LI
Assignee: Lei Chang


see email discussion here: 
http://mail-archives.apache.org/mod_mbox/hawq-dev/201607.mbox/%3CCA+F1uf=tjcioezkvphsppaog-jg0-0azqtusgr7rv+esv44...@mail.gmail.com%3E

Data skipping technology can extremely avoiding unnecessary IO,  so it can
extremely enhance performance for IO intensive query. Including eliminating
query on unnecessary table partition according to the partition key range ,
I think more options are available now:

(1) Parquet / ORC format introduce a lightweight meta data info like
Min/Max/Bloom filter for each block, such meta data can be exploited when
predicate/filter info can be fetched before executing scan.

However now in HAWQ, all data in parquet need to be scanned into memory
before processing predicate/filter. We don't generate the meta info when
INSERT into parquet table, the scan executor doesn't utilize the meta info
neither. Maybe some scan API need to be refactored so that we can get
predicate/filter
info before executing base relation scan.

(2) Base on (1) technology,  especially with Bloom filter, more optimizer
technology can be explored furthur. E.g. Impala implemented Runtime
filtering(*https://www.cloudera.com/documentation/enterprise/latest/topics/impala_runtime_filtering.html
*
),  which can be used at
- dynamic partition pruning
- converting join predicate to base relation predicate

It tell the executor to wait for one moment(the interval time can be set in
guc) before executing base relation scan, if the interested values(e.g. the
column in join predicate only have very small set) arrived in time, it can
use these value to filter this scan, if doesn't arrived in time, it scan
without this filter, which doesn't impact result correctness.

Unlike (1) technology, this technology cannot be used in any case, it only
outperform in some cases. So it just add some more query plan
choices/paths, and the optimizer need based on statistics info to calculate
the cost, and apply it when cost down.

All in one, maybe more similar technology can be adoptable for HAWQ now,
let's start to think about performance related technology, moreover we need
to instigate how these technology can be implemented in HAWQ.

Any ideas or suggestions are welcomed? Thanks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-923) More data skipping technology for IO intensive performance enhancement

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-923:
---
Fix Version/s: backlog

>  More data skipping technology for IO intensive performance enhancement
> ---
>
> Key: HAWQ-923
> URL: https://issues.apache.org/jira/browse/HAWQ-923
> Project: Apache HAWQ
>  Issue Type: Wish
>  Components: Query Execution
>Reporter: Ming LI
>Assignee: Lei Chang
> Fix For: backlog
>
>
> see email discussion here: 
> http://mail-archives.apache.org/mod_mbox/hawq-dev/201607.mbox/%3CCA+F1uf=tjcioezkvphsppaog-jg0-0azqtusgr7rv+esv44...@mail.gmail.com%3E
> Data skipping technology can extremely avoiding unnecessary IO,  so it can
> extremely enhance performance for IO intensive query. Including eliminating
> query on unnecessary table partition according to the partition key range ,
> I think more options are available now:
> (1) Parquet / ORC format introduce a lightweight meta data info like
> Min/Max/Bloom filter for each block, such meta data can be exploited when
> predicate/filter info can be fetched before executing scan.
> However now in HAWQ, all data in parquet need to be scanned into memory
> before processing predicate/filter. We don't generate the meta info when
> INSERT into parquet table, the scan executor doesn't utilize the meta info
> neither. Maybe some scan API need to be refactored so that we can get
> predicate/filter
> info before executing base relation scan.
> (2) Base on (1) technology,  especially with Bloom filter, more optimizer
> technology can be explored furthur. E.g. Impala implemented Runtime
> filtering(*https://www.cloudera.com/documentation/enterprise/latest/topics/impala_runtime_filtering.html
> *
> ),  which can be used at
> - dynamic partition pruning
> - converting join predicate to base relation predicate
> It tell the executor to wait for one moment(the interval time can be set in
> guc) before executing base relation scan, if the interested values(e.g. the
> column in join predicate only have very small set) arrived in time, it can
> use these value to filter this scan, if doesn't arrived in time, it scan
> without this filter, which doesn't impact result correctness.
> Unlike (1) technology, this technology cannot be used in any case, it only
> outperform in some cases. So it just add some more query plan
> choices/paths, and the optimizer need based on statistics info to calculate
> the cost, and apply it when cost down.
> All in one, maybe more similar technology can be adoptable for HAWQ now,
> let's start to think about performance related technology, moreover we need
> to instigate how these technology can be implemented in HAWQ.
> Any ideas or suggestions are welcomed? Thanks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-923) More data skipping technology for IO intensive performance enhancement

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-923:
---
Issue Type: Wish  (was: Bug)

>  More data skipping technology for IO intensive performance enhancement
> ---
>
> Key: HAWQ-923
> URL: https://issues.apache.org/jira/browse/HAWQ-923
> Project: Apache HAWQ
>  Issue Type: Wish
>  Components: Query Execution
>Reporter: Ming LI
>Assignee: Lei Chang
> Fix For: backlog
>
>
> see email discussion here: 
> http://mail-archives.apache.org/mod_mbox/hawq-dev/201607.mbox/%3CCA+F1uf=tjcioezkvphsppaog-jg0-0azqtusgr7rv+esv44...@mail.gmail.com%3E
> Data skipping technology can extremely avoiding unnecessary IO,  so it can
> extremely enhance performance for IO intensive query. Including eliminating
> query on unnecessary table partition according to the partition key range ,
> I think more options are available now:
> (1) Parquet / ORC format introduce a lightweight meta data info like
> Min/Max/Bloom filter for each block, such meta data can be exploited when
> predicate/filter info can be fetched before executing scan.
> However now in HAWQ, all data in parquet need to be scanned into memory
> before processing predicate/filter. We don't generate the meta info when
> INSERT into parquet table, the scan executor doesn't utilize the meta info
> neither. Maybe some scan API need to be refactored so that we can get
> predicate/filter
> info before executing base relation scan.
> (2) Base on (1) technology,  especially with Bloom filter, more optimizer
> technology can be explored furthur. E.g. Impala implemented Runtime
> filtering(*https://www.cloudera.com/documentation/enterprise/latest/topics/impala_runtime_filtering.html
> *
> ),  which can be used at
> - dynamic partition pruning
> - converting join predicate to base relation predicate
> It tell the executor to wait for one moment(the interval time can be set in
> guc) before executing base relation scan, if the interested values(e.g. the
> column in join predicate only have very small set) arrived in time, it can
> use these value to filter this scan, if doesn't arrived in time, it scan
> without this filter, which doesn't impact result correctness.
> Unlike (1) technology, this technology cannot be used in any case, it only
> outperform in some cases. So it just add some more query plan
> choices/paths, and the optimizer need based on statistics info to calculate
> the cost, and apply it when cost down.
> All in one, maybe more similar technology can be adoptable for HAWQ now,
> let's start to think about performance related technology, moreover we need
> to instigate how these technology can be implemented in HAWQ.
> Any ideas or suggestions are welcomed? Thanks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (HAWQ-19) Money type overflow

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-19?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang closed HAWQ-19.
-
Resolution: Fixed

It is already in.

> Money type overflow
> ---
>
> Key: HAWQ-19
> URL: https://issues.apache.org/jira/browse/HAWQ-19
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Catalog
>Reporter: Feng Tian
>Assignee: Ming LI
> Fix For: 2.0.0.0-incubating
>
>
> Use tpch schema, but change l_extendedprice to use MONEY type, run Q1, you 
> should see negative amounts.   
> I believe this is due to overflow.
> Side mark, postgres 9 money type use 8 bytes and will return correct result.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-19) Money type overflow

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-19?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-19:
--
Fix Version/s: (was: 2.0.0)
   2.0.0.0-incubating

> Money type overflow
> ---
>
> Key: HAWQ-19
> URL: https://issues.apache.org/jira/browse/HAWQ-19
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Catalog
>Reporter: Feng Tian
>Assignee: Ming LI
> Fix For: 2.0.0.0-incubating
>
>
> Use tpch schema, but change l_extendedprice to use MONEY type, run Q1, you 
> should see negative amounts.   
> I believe this is due to overflow.
> Side mark, postgres 9 money type use 8 bytes and will return correct result.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-919) pxf has some issues on RAT check

[Lei Chang (JIRA)]

Lei Chang created HAWQ-919:
--

 Summary: pxf has some issues on RAT check
 Key: HAWQ-919
 URL: https://issues.apache.org/jira/browse/HAWQ-919
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: PXF
Reporter: Lei Chang
Assignee: Goden Yao






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-919) pxf has some issues on RAT check

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-919:
---
Priority: Blocker  (was: Major)

> pxf has some issues on RAT check
> 
>
> Key: HAWQ-919
> URL: https://issues.apache.org/jira/browse/HAWQ-919
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: PXF
>Reporter: Lei Chang
>Assignee: Goden Yao
>Priority: Blocker
> Fix For: 2.0.0.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-751) Add plr, pgcrypto, gporca into Apache HAWQ

[Lei Chang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15372550#comment-15372550
 ] 

Lei Chang commented on HAWQ-751:


didn't see issues from the email thread (see paul's answer), let's close this.

> Add plr, pgcrypto, gporca into Apache HAWQ
> --
>
> Key: HAWQ-751
> URL: https://issues.apache.org/jira/browse/HAWQ-751
> Project: Apache HAWQ
>  Issue Type: Bug
>Reporter: Paul Guo
>Assignee: Lei Chang
> Fix For: 2.0.0.0-incubating
>
>
> Due to license conflict, we can not put them into HAWQ, instead, we use "git 
> submodule".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (HAWQ-751) Add plr, pgcrypto, gporca into Apache HAWQ

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang closed HAWQ-751.
--
Resolution: Fixed

> Add plr, pgcrypto, gporca into Apache HAWQ
> --
>
> Key: HAWQ-751
> URL: https://issues.apache.org/jira/browse/HAWQ-751
> Project: Apache HAWQ
>  Issue Type: Bug
>Reporter: Paul Guo
>Assignee: Lei Chang
> Fix For: 2.0.0.0-incubating
>
>
> Due to license conflict, we can not put them into HAWQ, instead, we use "git 
> submodule".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (HAWQ-867) Replace the git-submobule mechanism with git-clone

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang closed HAWQ-867.
--
Resolution: Fixed

> Replace the git-submobule mechanism with git-clone
> --
>
> Key: HAWQ-867
> URL: https://issues.apache.org/jira/browse/HAWQ-867
> Project: Apache HAWQ
>  Issue Type: Bug
>Reporter: Paul Guo
>Assignee: Paul Guo
>Priority: Blocker
> Fix For: 2.0.0.0-incubating
>
>
> Currently some features/extensions (e.g. pycrytpo, gporca, etc) are embedded 
> into hawq code via the git submodule mechanism which is similar to a link. In 
> hawq git repo, during hawq build, it will download the submodule repos and 
> build those functionality, however if we decide to generate a source tarball 
> which is independent on the git repo, this will be an issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-838) include an alternative for paramiko python module

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-838:
---
Fix Version/s: (was: 2.0.0.0-incubating)
   backlog

> include an alternative for paramiko python module
> -
>
> Key: HAWQ-838
> URL: https://issues.apache.org/jira/browse/HAWQ-838
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Command Line Tools
>Reporter: Radar Lei
>Assignee: Radar Lei
>Priority: Minor
> Fix For: backlog
>
>
> Since paramiko's license is conflict with Apach HAWQ's license, so we can't 
> deliver it with HAWQ package. So users need to install it before installing 
> hawq.
> To save user's time to install paramiko manually, we can find a good 
> replacement which don't have license conflict.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-838) include an alternative for paramiko python module

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-838:
---
Summary: include an alternative for paramiko python module  (was: Replace 
python module paramiko)

> include an alternative for paramiko python module
> -
>
> Key: HAWQ-838
> URL: https://issues.apache.org/jira/browse/HAWQ-838
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Command Line Tools
>Reporter: Radar Lei
>Assignee: Radar Lei
>Priority: Minor
> Fix For: 2.0.0.0-incubating
>
>
> Since paramiko's license is conflict with Apach HAWQ's license, so we can't 
> deliver it with HAWQ package. So users need to install it before installing 
> hawq.
> To save user's time to install paramiko manually, we can find a good 
> replacement which don't have license conflict.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-838) Replace python module paramiko

[Lei Chang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15372098#comment-15372098
 ] 

Lei Chang commented on HAWQ-838:


please see the revised description, this is not in apache hawq, so it is not a 
blocker for this release. This task is just an enhancement for user 
installation experience.

> Replace python module paramiko
> --
>
> Key: HAWQ-838
> URL: https://issues.apache.org/jira/browse/HAWQ-838
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Command Line Tools
>Reporter: Radar Lei
>Assignee: Radar Lei
>Priority: Blocker
> Fix For: 2.0.0.0-incubating
>
>
> Since paramiko's license is conflict with Apach HAWQ's license, so we can't 
> deliver it with HAWQ package. So users need to install it before installing 
> hawq.
> To save user's time to install paramiko manually, we can find a good 
> replacement which don't have license conflict.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-838) Replace python module paramiko

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-838:
---
Priority: Minor  (was: Blocker)

> Replace python module paramiko
> --
>
> Key: HAWQ-838
> URL: https://issues.apache.org/jira/browse/HAWQ-838
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Command Line Tools
>Reporter: Radar Lei
>Assignee: Radar Lei
>Priority: Minor
> Fix For: 2.0.0.0-incubating
>
>
> Since paramiko's license is conflict with Apach HAWQ's license, so we can't 
> deliver it with HAWQ package. So users need to install it before installing 
> hawq.
> To save user's time to install paramiko manually, we can find a good 
> replacement which don't have license conflict.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-838) Replace python module paramiko

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-838:
---
Description: 
Since paramiko's license is conflict with Apach HAWQ's license, so we can't 
deliver it with HAWQ package. So users need to install it before installing 
hawq.

To save user's time to install paramiko manually, we can find a good 
replacement which don't have license conflict.

  was:
Since paramiko's license is conflict with Apach HAWQ's license, so we can't 
deliver it with HAWQ package.

To save user's time to install paramiko manually, we should remove paramiko 
from HAWQ code, and find a good replacement which don't have license conflict.


> Replace python module paramiko
> --
>
> Key: HAWQ-838
> URL: https://issues.apache.org/jira/browse/HAWQ-838
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Command Line Tools
>Reporter: Radar Lei
>Assignee: Radar Lei
>Priority: Blocker
> Fix For: 2.0.0.0-incubating
>
>
> Since paramiko's license is conflict with Apach HAWQ's license, so we can't 
> deliver it with HAWQ package. So users need to install it before installing 
> hawq.
> To save user's time to install paramiko manually, we can find a good 
> replacement which don't have license conflict.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-916) Replace com.pivotal.hawq package name to org.apache.hawq

[Lei Chang (JIRA)]

Lei Chang created HAWQ-916:
--

 Summary: Replace com.pivotal.hawq package name to org.apache.hawq
 Key: HAWQ-916
 URL: https://issues.apache.org/jira/browse/HAWQ-916
 Project: Apache HAWQ
  Issue Type: Bug
  Components: Storage
Reporter: Lei Chang
Assignee: Lei Chang


com.pivotal.hawq.mapreduce types are referenced in at least the following 
apache hawq (incubating) directories, master branch:
contrib/hawq-hadoop
contrib/hawq-hadoop/hawq-mapreduce-tool
contrib/hawq-hadoop/hawq-mapreduce-parquet
contrib/hawq-hadoop/hawq-mapreduce-common
contrib/hawq-hadoop/hawq-mapreduce-ao
contrib/hawq-hadoop/target/apidocs




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-914) Improve user experience of HAWQ's build infrastructure

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-914:
---
Assignee: Paul Guo  (was: Lei Chang)

> Improve user experience of HAWQ's build infrastructure
> --
>
> Key: HAWQ-914
> URL: https://issues.apache.org/jira/browse/HAWQ-914
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Build
>Affects Versions: 2.0.0.0-incubating
>Reporter: Roman Shaposhnik
>Assignee: Paul Guo
>
> This is likely to end up being an umbrella JIRA so feel free to fork off 
> sub-tasks whenever it makes sense.
> As an end-user of HAWQ's build system, I'd like to see the default of the 
> build system (running configure/etc. with no arguments) to be:
>  # treating optional missing dependencies with a WARNING similar to what 
> PostrgreSQL configure does in the following example:
> {noformat}
> checking for bison... no
> configure: WARNING:
> *** Without Bison you will not be able to build PostgreSQL from CVS nor
> *** change any of the parser definition files.  You can obtain Bison from
> *** a GNU mirror site.  (If you are using the official distribution of
> *** PostgreSQL then you do not need to worry about this, because the Bison
> *** output is pre-generated.)  To use a different yacc program (possible,
> *** but not recommended), set the environment variable YACC before running
> *** 'configure'.
> {noformat}
> # treating all the missing suggested dependencies by failing the build and 
> suggesting how to point at binary copies of these missing dependencies 
> similar to what PostrgreSQL configure does in the following example:
> {noformat}
> checking for -ledit... no
> configure: error: readline library not found
> If you have readline already installed, see config.log for details on the
> failure.  It is possible the compiler isn't looking in the proper directory.
> Use --without-readline to disable readline support.
> {noformat}
> # treating the core dependencies same as suggested dependencies, but 
> obviously about the option of continuing the build without them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-892) Need "incubating" in Apache HAWQ release name

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-892:
---
Assignee: Radar Lei  (was: Lei Chang)

> Need "incubating" in Apache HAWQ release name
> -
>
> Key: HAWQ-892
> URL: https://issues.apache.org/jira/browse/HAWQ-892
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Build
>Reporter: Goden Yao
>Assignee: Radar Lei
>Priority: Blocker
> Fix For: 2.0.0-incubating
>
>
> our version file in 2.0.0-incubating branch should be "2.0.0-incubating".
> Master should be "2.0.1-incubating" (or whatever the next version)
> Hawq commandline --version should also show "incubating" in the version 
> number before we graduate.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-864) Support ORC as a native file format

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-864:
---
Description: 
ORC (Optimized Row Columnar) is a very popular open source format adopted in 
some major
components in Hadoop eco­system. It is also used by a lot of users. The 
advantages of
supporting ORC storage in HAWQ are in two folds: firstly, it makes HAWQ more 
Hadoop native
which interacts with other components more easily; secondly, ORC stores some 
meta info for
query optimization, thus, it might potentially outperform two native formats 
(i.e., AO, Parquet) if it
is available.

The implementation can be based on the framework proposed in HAWQ-786.

  was:
ORC (Optimized Row Columnar) is a very popular open source format adopted in 
some major
components in Hadoop eco­system. It is also used by a lot of users. The 
advantages of
supporting ORC storage in HAWQ are in two folds: firstly, it makes HAWQ more 
Hadoop native
which interacts with other components more easily; secondly, ORC stores some 
meta info for
query optimization, thus, it might potentially outperform two native formats 
(i.e., AO, Parquet) if it
is available.


> Support ORC as a native file format
> ---
>
> Key: HAWQ-864
> URL: https://issues.apache.org/jira/browse/HAWQ-864
> Project: Apache HAWQ
>  Issue Type: Wish
>  Components: Storage
>Reporter: Lei Chang
>Assignee: Lei Chang
>
> ORC (Optimized Row Columnar) is a very popular open source format adopted in 
> some major
> components in Hadoop eco­system. It is also used by a lot of users. The 
> advantages of
> supporting ORC storage in HAWQ are in two folds: firstly, it makes HAWQ more 
> Hadoop native
> which interacts with other components more easily; secondly, ORC stores some 
> meta info for
> query optimization, thus, it might potentially outperform two native formats 
> (i.e., AO, Parquet) if it
> is available.
> The implementation can be based on the framework proposed in HAWQ-786.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-864) Support ORC as a native file format

[Lei Chang (JIRA)]

Lei Chang created HAWQ-864:
--

 Summary: Support ORC as a native file format
 Key: HAWQ-864
 URL: https://issues.apache.org/jira/browse/HAWQ-864
 Project: Apache HAWQ
  Issue Type: Wish
  Components: Storage
Reporter: Lei Chang
Assignee: Lei Chang






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-786) Framework to support pluggable formats and file systems

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-786:
---
Description: 
In current HAWQ, two native formats are supported: AO and parquet. Now we want 
to support ORC. A framework to support naive c/c++ pluggable format is needed 
to support ORC more easily. And it can also be potentially used for fast 
external data access.

And there are a lot of requests for supporting S3, Ceph and other file systems, 
and this is closely related to pluggable formats, so this JIRA is proposing a 
framework to support both.

  was:In current HAWQ, two native formats are supported: AO and parquet. Now we 
want to support ORC. A framework to support naive c/c++ pluggable format is 
needed to support ORC more easily. And it can also be potentially used for fast 
external data access.


> Framework to support pluggable formats and file systems
> ---
>
> Key: HAWQ-786
> URL: https://issues.apache.org/jira/browse/HAWQ-786
> Project: Apache HAWQ
>  Issue Type: Wish
>  Components: Storage
>Reporter: Lei Chang
>Assignee: hongwu
> Attachments: ORCdesign-v0.1-2016-06-17.pdf
>
>
> In current HAWQ, two native formats are supported: AO and parquet. Now we 
> want to support ORC. A framework to support naive c/c++ pluggable format is 
> needed to support ORC more easily. And it can also be potentially used for 
> fast external data access.
> And there are a lot of requests for supporting S3, Ceph and other file 
> systems, and this is closely related to pluggable formats, so this JIRA is 
> proposing a framework to support both.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-786) Framework to support pluggable formats and file systems

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-786:
---
Summary: Framework to support pluggable formats and file systems  (was: 
HAWQ supports ORC as a native storage format)

> Framework to support pluggable formats and file systems
> ---
>
> Key: HAWQ-786
> URL: https://issues.apache.org/jira/browse/HAWQ-786
> Project: Apache HAWQ
>  Issue Type: Wish
>  Components: Storage
>Reporter: Lei Chang
>Assignee: hongwu
> Attachments: ORCdesign-v0.1-2016-06-17.pdf
>
>
> In current HAWQ, two native formats are supported: AO and parquet. Now we 
> want to support ORC. A framework to support naive c/c++ pluggable format is 
> needed to support ORC more easily. And it can also be potentially used for 
> fast external data access.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-850) Planner supports refineCachedPlan interface.

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-850:
---
Issue Type: Sub-task  (was: Improvement)
Parent: HAWQ-800

> Planner supports refineCachedPlan interface.
> 
>
> Key: HAWQ-850
> URL: https://issues.apache.org/jira/browse/HAWQ-850
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Core
>Reporter: Hubert Zhang
>Assignee: Hubert Zhang
>
> in PBE(prepare bind execution) cases, plan will be cached, and only be 
> generated once in postgres.
> But in HAWQ since resource is dynamic or elastic, the plan maybe changed 
> during the different execution of PBE. Also, the file split allocation result 
> which is stored in plan is dynamically too, so the split-vseg mapping may 
> also be changed.
> For example, We call a plpgsql function which do "insert into t select * from 
> t".
> in Prepare we cache the plan, with fixed split-vseg mapping(e.g. 10 blocks), 
> but after insert, there would be 20 blocks so the old  split-vseg mapping is 
> invalid. We need to update it. Also when we call this function again and 
> again, the data size of t may request more resource to handle it than the 
> first call, so we should recalculate the entire plan with the new vseg number.
> This function is implemented in a interface called refineCachedPlan.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HAWQ-823) Amazon S3 External Table Support

[Lei Chang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341068#comment-15341068
 ] 

Lei Chang edited comment on HAWQ-823 at 6/21/16 8:44 AM:
-

the hawq architecture for supporting this feature will be somewhat different 
from gpdb. And the new framework to support pluggable file system has been 
proposed here: https://issues.apache.org/jira/browse/HAWQ-786 so let's see how 
we can combine the work together.


was (Author: lei_chang):
the hawq architecture for supporting this feature will be somewhat different 
from gpdb. And the new framework to support pluggable file system has been 
proposed here: https://issues.apache.org/jira/browse/HAWQ-786 so we can see we 
can combine the work together.

> Amazon S3 External Table Support
> 
>
> Key: HAWQ-823
> URL: https://issues.apache.org/jira/browse/HAWQ-823
> Project: Apache HAWQ
>  Issue Type: Wish
>  Components: External Tables
>Reporter: Kyle R Dunn
>Assignee: Lei Chang
>
> As a cloud user, I'd like to be able to create readable external tables with 
> data in Amazon S3.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-823) Amazon S3 External Table Support

[Lei Chang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341068#comment-15341068
 ] 

Lei Chang commented on HAWQ-823:


the hawq architecture for supporting this feature will be somewhat different 
from gpdb. And the new framework to support pluggable file system has been 
proposed here: https://issues.apache.org/jira/browse/HAWQ-786 so we can see we 
can combine the work together.

> Amazon S3 External Table Support
> 
>
> Key: HAWQ-823
> URL: https://issues.apache.org/jira/browse/HAWQ-823
> Project: Apache HAWQ
>  Issue Type: Wish
>  Components: External Tables
>Reporter: Kyle R Dunn
>Assignee: Lei Chang
>
> As a cloud user, I'd like to be able to create readable external tables with 
> data in Amazon S3.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-842) Failed to acquire resource from resource manager

[Lei Chang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15340869#comment-15340869
 ] 

Lei Chang commented on HAWQ-842:


can you give more information on the version you used, and is there a core dump 
generated, and what is the scenario when this issue happens?

> Failed to acquire resource from resource manager
> 
>
> Key: HAWQ-842
> URL: https://issues.apache.org/jira/browse/HAWQ-842
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Resource Manager
>Reporter: Bill Wailliam
>Assignee: Lei Chang
>
> This is the pg_log:
> 2016-06-20 17:56:03.864644 
> CST,,,p526096,th5032941760,,,seg-1,"LOG","0","database system 
> was shut down at 2016-06-20 17:54:32 CST",,,0,,"xlog.c",6205,
> 2016-06-20 17:56:03.864908 
> CST,,,p526096,th5032941760,,,seg-1,"LOG","0","checkpoint 
> record is at 0/2672EF8",,,0,,"xlog.c",6304,
> 2016-06-20 17:56:03.864923 
> CST,,,p526096,th5032941760,,,seg-1,"LOG","0","redo record is 
> at 0/2672EF8; undo record is at 0/0; shutdown TRUE",,,0,,"xlog.c",6338,
> 2016-06-20 17:56:03.864933 
> CST,,,p526096,th5032941760,,,seg-1,"LOG","0","next 
> transaction ID: 0/1284; next OID: 16514",,,0,,"xlog.c",6342,
> 2016-06-20 17:56:03.864942 
> CST,,,p526096,th5032941760,,,seg-1,"LOG","0","next 
> MultiXactId: 1; next MultiXactOffset: 0",,,0,,"xlog.c",6345,
> 2016-06-20 17:56:03.864951 
> CST,,,p526096,th5032941760,,,seg-1,"LOG","0","end of 
> transaction log location is 0/2672F48",,,0,,"xlog.c",6582,
> 2016-06-20 17:56:03.865750 
> CST,,,p526096,th5032941760,,,seg-1,"LOG","0","Oldest active 
> transaction from prepared transactions 1284",,,0,,"xlog.c",5996,
> 2016-06-20 17:56:03.867372 
> CST,,,p526096,th5032941760,,,seg-1,"LOG","0","database system 
> is ready",,,0,,"xlog.c",6022,
> 2016-06-20 17:56:03.867394 
> CST,,,p526096,th5032941760,,,seg-1,"LOG","0","PostgreSQL 
> 8.2.15 (Greenplum Database 4.2.0 build 1) (HAWQ 2.0.0.0 build dev) on 
> x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.8.0 compiled on Jun 19 
> 2016 03:02:01",,,0,,"xlog.c",6032,
> 2016-06-20 17:56:03.868503 
> CST,,,p526096,th5032941760,,,seg-1,"LOG","0","Finished normal 
> startup for clean shutdown case",,,0,,"xlog.c",6810,
> 2016-06-20 17:56:03.876213 
> CST,,,p526097,th5032941760,,,seg-1,"LOG","0","Finished 
> startup integrity checking",,,0,,"xlog.c",7159,
> 2016-06-20 17:56:03.879998 
> CST,,,p526104,th5032941760,,,seg-1,"LOG","0","HAWQ Segment RM 
> :: Temporary directory /data1/hawq/tmp",,,0,,"resourcemanager.c",1055,
> 2016-06-20 17:56:03.880039 
> CST,,,p526104,th5032941760,,,seg-1,"LOG","0","checkAndBuildFailedTmpDirList
>  finished checking temporary directory, which costs 41 
> us",,,0,,"resourcemanager_RMSEG.c",274,
> 2016-06-20 17:56:03.883958 
> CST,,,p526104,th5032941760,con4,,seg-1,"LOG","0","YARN mode 
> resource broker created resource broker process 
> PID=526105.",,,0,,"resourcebroker_LIBYARN.c",158,
> 2016-06-20 17:56:03.884155 
> CST,,,p526105,th5032941760,con4,,seg-1,"LOG","0","YARN mode 
> resource broker accepted YARN connection arguments : YARN Server 
> RM_IP_XX:8032 Scheduler server RM_IP_XX:8030 Queue hawq Application 
> name hawq, by user:postgres",,,0,,"resourcebroker_LIBYARN_proc.c",501,
> 2016-06-20 17:56:03.884283 
> CST,,,p526104,th5032941760,con4,,seg-1,"LOG","0","Resource 
> manager starts accepting resource request. Listening normal socket port 5437. 
> Total listened 1 FDs.",,,0,,"resourcemanager.c",2492,
> 2016-06-20 17:56:03.884378 
> CST,,,p526094,th5032941760,,,seg-1,"LOG","0","Wait for HAWQ 
> RM -1",,,0,,"resourcemanager.c",421,
> 2016-06-20 17:56:03.884409 
> CST,,,p526094,th5032941760,,,seg-1,"LOG","0","HAWQ :: 
> Received signal notification that HAWQ RM works 
> now.",,,0,,"resourcemanager.c",429,
> 2016-06-20 17:56:03.884424 
> CST,,,p526094,th5032941760,,,seg-1,"LOG","0","PostgreSQL 
> 8.2.15 (Greenplum Database 4.2.0 build 1) (HAWQ 2.0.0.0 build dev) on 
> x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.8.0 compiled on Jun 19 
> 2016 03:02:03",,,0,,"postmaster.c",3694,
> 2016-06-20 17:56:03.884441 
> CST,,,p526094,th5032941760,,,seg-1,"LOG","0","database system 
> is ready to accept connections","PostgreSQL 8.2.15 (Greenplum Database 4.2.0 
> build 1) (HAWQ 2.0.0.0 build dev) on x86_64-unknown-linux-gnu, compiled by 
> GCC gcc (GCC) 4.8.0 compiled on Jun 19 2016 
> 03:02:03",,0,,"postmaster.c",3701,
> 2016-06-20 

[jira] [Created] (HAWQ-840) Partition Sort Support

[Lei Chang (JIRA)]

Lei Chang created HAWQ-840:
--

 Summary: Partition Sort Support
 Key: HAWQ-840
 URL: https://issues.apache.org/jira/browse/HAWQ-840
 Project: Apache HAWQ
  Issue Type: Wish
Reporter: Lei Chang
Assignee: Lei Chang


Support the partition sort as a new method for sorting.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-101) Remove postgres version number from HAWQ

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-101:
---
Issue Type: Wish  (was: Task)

> Remove postgres version number from HAWQ 
> -
>
> Key: HAWQ-101
> URL: https://issues.apache.org/jira/browse/HAWQ-101
> Project: Apache HAWQ
>  Issue Type: Wish
>  Components: Command Line Tools
>Reporter: Goden Yao
>Assignee: Lei Chang
>Priority: Minor
>  Labels: OSS
> Fix For: backlog
>
>
> Some version numbers showing in command line is due to historical reason that 
> HAWQ is derived from Greenplum and postgres.
> It doesn't make sense to have these version numbers which would confuse users 
> after open source.
> {code:actionscript}
> gpadmin@rhel65-1 ~]$ psql --version
> psql (PostgreSQL) 8.2.15
> contains support for command-line editing
> {code}
> {code:actionscript}
> [gpadmin@rhel65-1 ~]$ postgres --version
> postgres (HAWQ) 8.2.15
> {code}
> {code:actionscript}
> [gpadmin@rhel65-1 ~]$ postgres --hawq-version
> postgres (HAWQ) 2.0.0.0 build dev
> {code}
> {code:actionscript}
> [gpadmin@rhel65-1 ~]$ postgres --gp-version
> postgres (HAWQ) 4.2.0 build 1
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (HAWQ-490) Add version and compatibility detection for HAWQ + GPORCA

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang closed HAWQ-490.
--
Resolution: Won't Fix

> Add version and compatibility detection for HAWQ + GPORCA
> -
>
> Key: HAWQ-490
> URL: https://issues.apache.org/jira/browse/HAWQ-490
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Build
>Reporter: Jacob Max Frank
>Assignee: Lei Chang
>Priority: Minor
>
> Autoconf should be able to detect GPORCA version information and check for 
> compatibility with the version of HAWQ being built.  Additionally, running 
> {{select gp_opt_version();}} on HAWQ compiled with GPORCA should output 
> correct version information for its components (GPOPT, GPOS, and GPXerces).
> [~rvs] suggested using {{pkgutil}}.  Alternate potential strategies include 
> using a {{git submodule}} or pulling values from a {{version.h}} file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-98) Moving HAWQ docker file into code base

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-98?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-98:
--
Issue Type: Wish  (was: Task)

> Moving HAWQ docker file into code base
> --
>
> Key: HAWQ-98
> URL: https://issues.apache.org/jira/browse/HAWQ-98
> Project: Apache HAWQ
>  Issue Type: Wish
>Reporter: Goden Yao
>Assignee: Roman Shaposhnik
> Fix For: 2.1.0
>
>
> We have a pre-built docker image (check [HAWQ build & 
> install|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61320026]
>  sitting outside the codebase.
> It should be incorporated in the Apache git and maintained by the community.
> Proposed location is to create a  folder under root



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-99) OpenSSL 0.9.x to 1.x upgrade

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-99?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-99:
--
Issue Type: New Feature  (was: Wish)

> OpenSSL 0.9.x to 1.x upgrade
> 
>
> Key: HAWQ-99
> URL: https://issues.apache.org/jira/browse/HAWQ-99
> Project: Apache HAWQ
>  Issue Type: New Feature
>Reporter: Goden Yao
>Assignee: Lei Chang
> Fix For: backlog
>
>
> 0.9.x product line will be deprecated by end of 2015.
> We need to move to the new 1.x product line.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-99) OpenSSL 0.9.x to 1.x upgrade

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-99?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-99:
--
Issue Type: Wish  (was: Improvement)

> OpenSSL 0.9.x to 1.x upgrade
> 
>
> Key: HAWQ-99
> URL: https://issues.apache.org/jira/browse/HAWQ-99
> Project: Apache HAWQ
>  Issue Type: Wish
>Reporter: Goden Yao
>Assignee: Lei Chang
> Fix For: backlog
>
>
> 0.9.x product line will be deprecated by end of 2015.
> We need to move to the new 1.x product line.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-122) FILESPACE URL in pg_filespace_entry is confusing

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-122:
---
Issue Type: Wish  (was: Improvement)

> FILESPACE URL in pg_filespace_entry is confusing
> 
>
> Key: HAWQ-122
> URL: https://issues.apache.org/jira/browse/HAWQ-122
> Project: Apache HAWQ
>  Issue Type: Wish
>  Components: Storage
>Reporter: Lirong Jian
>Assignee: Lirong Jian
> Fix For: backlog
>
>
> postgres=# SELECT * FROM pg_filespace_entry;
>  fsefsoid | fsedbid | fselocation  
> --+-+--
> 16384 |   0 | hdfs://Lirong-MBP:9000/gpsql
> (1 row)
> postgres=# CREATE FILESPACE fs_1 ON HDFS (
> postgres(# 'Lirong-MBP:9000/hawq/fs_1');
> CREATE FILESPACE
> postgres=# CREATE FILESPACE fs_2 ON HDFS (
>   
>
> 'Lirong-MBP:9000/hawq/fs_2') WITH (NUMREPLICA=2);
> CREATE FILESPACE
> postgres=# CREATE FILESPACE fs_3 ON HDFS (
>   
>
> 'Lirong-MBP:9000/hawq/fs_3') WITH (NUMREPLICA=3);
> CREATE FILESPACE
> postgres=# SELECT * FROM pg_filespace_entry;
>  fsefsoid | fsedbid | fselocation 
> --+-+-
> 16384 |   0 | hdfs://Lirong-MBP:9000/gpsql
> 16532 |   0 | hdfs://Lirong-MBP:9000/hawq/fs_1
> 16533 |   0 | hdfs://{replica=2}Lirong-MBP:9000/hawq/fs_2
> 16534 |   0 | hdfs://{replica=3}Lirong-MBP:9000/hawq/fs_3
> (4 rows)
> The {replica=3} option is very confusing. It is not a valid HDSF url, which 
> means we can only access that URL through external tools such as hdfs, 
> although it is valid inside HAWQ (we exclude that part when we are really 
> going to access that path).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (HAWQ-144) Build HAWQ on MacOS

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang closed HAWQ-144.
--
   Resolution: Fixed
Fix Version/s: (was: 2.1.0)
   2.0.0

> Build HAWQ on MacOS
> ---
>
> Key: HAWQ-144
> URL: https://issues.apache.org/jira/browse/HAWQ-144
> Project: Apache HAWQ
>  Issue Type: New Feature
>  Components: Build
>Reporter: Lei Chang
>Assignee: Lei Chang
> Fix For: 2.0.0
>
>
> Currently, the only tested build platform for HAWQ is redhat 6.x. It will be 
> very nice if it can work on Mac with clang. This will make new contributions 
> much easier.
> Instructions on building HAWQ on linux is at: 
> https://github.com/apache/incubator-hawq/blob/master/BUILD_INSTRUCTIONS.md



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-124) Create Project Maturity Model summary file

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-124:
---
Issue Type: Improvement  (was: Task)

> Create Project Maturity Model summary file
> --
>
> Key: HAWQ-124
> URL: https://issues.apache.org/jira/browse/HAWQ-124
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Core
>Reporter: Caleb Welton
>Assignee: Lei Chang
> Fix For: 2.1.0
>
>
> Graduating from an Apache Incubator project requires showing the Apache 
> Incubator IPMC that we have reached a level of maturity as an incubator 
> project.  One tool that can be used to assess our maturity is the [Apache 
> Project Maturity Model 
> Document|https://community.apache.org/apache-way/apache-project-maturity-model.html].
>   
> I propose we do something similar to what Groovy did and include a Project 
> Maturity Self assessment in our source code and evaluate ourselves with 
> respect to project maturity with each of our reports.  
> To do:
> 1. Create a MATURITY.adoc file in our root project directory containing our 
> self assessment.
> See 
> https://github.com/apache/groovy/blob/67b87a3592f13a6281f5b20081c37a66c80079b9/MATURITY.adoc
>  as an example document in the Groovy project.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-150) External tables can be designated for both READ and WRITE

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-150:
---
Issue Type: Wish  (was: New Feature)

> External tables can be designated for both READ and WRITE
> -
>
> Key: HAWQ-150
> URL: https://issues.apache.org/jira/browse/HAWQ-150
> Project: Apache HAWQ
>  Issue Type: Wish
>  Components: External Tables
>Reporter: C.J. Jameson
>Assignee: Lei Chang
> Fix For: 3.0.0
>
>
> Currently, external tables are either read-only or write-only when they are 
> created. We could support an external table with the capability for both 
> reads and writes.
> As pointed out by hawqst...@163.com



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-231) Alter table by drop all columns of it, then it has some interesting problems

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-231:
---
Issue Type: Bug  (was: Improvement)

> Alter table by drop all columns of it, then it has some interesting problems
> 
>
> Key: HAWQ-231
> URL: https://issues.apache.org/jira/browse/HAWQ-231
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Storage
>Reporter: Dong Li
>Assignee: Lei Chang
> Fix For: backlog
>
>
> It is a design behavior problem. 
> When we drop all the columns, should it truncate the table?
> Otherwise if the count of invisible rows has meaning.
> You can not see anything, but it shows that there are 1000 rows here.
> I know that in storage view and design view it is ok.
> But in  user view, it may be puzzled.
> {code}
> intern=# create table alterall (i int, j int);
> CREATE TABLE
> intern=# insert into alterall VALUES 
> (generate_series(1,1000),generate_series(1,2));
> INSERT 0 1000
> intern=# alter table alterall drop COLUMN i;
> ALTER TABLE
> intern=# alter TABLE alterall drop COLUMN j;
> ALTER TABLE
> intern=# select * from alterall ;
> --
> (1000 rows)
> intern=# alter TABLE alterall add column k int default 3;
> ALTER TABLE
> intern=# select * from alterall;
>  k
> ---
>  3
>  3
>  3
>  3
>  3
>  3
>  3
> ...
> (1000 rows)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-304) Support update and delete on non-heap tables

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-304:
---
Issue Type: Wish  (was: New Feature)

> Support update and delete on non-heap tables
> 
>
> Key: HAWQ-304
> URL: https://issues.apache.org/jira/browse/HAWQ-304
> Project: Apache HAWQ
>  Issue Type: Wish
>  Components: Storage
>Reporter: Lei Chang
> Fix For: 3.0.0
>
> Attachments: mutable_table.sql
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HAWQ-305) Support PostGIS

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang resolved HAWQ-305.

   Resolution: Duplicate
Fix Version/s: (was: 2.1.0)

> Support PostGIS
> ---
>
> Key: HAWQ-305
> URL: https://issues.apache.org/jira/browse/HAWQ-305
> Project: Apache HAWQ
>  Issue Type: New Feature
>  Components: Unknown
>Reporter: Lei Chang
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (HAWQ-306) Direct hdfs data file drop & load to native HAWQ tables

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang closed HAWQ-306.
--

> Direct hdfs data file drop & load to native HAWQ tables
> ---
>
> Key: HAWQ-306
> URL: https://issues.apache.org/jira/browse/HAWQ-306
> Project: Apache HAWQ
>  Issue Type: New Feature
>  Components: Storage
>Reporter: Lei Chang
> Fix For: 2.0.0
>
>
> Sometimes, users want to add existing hdfs files to native HAWQ tables. 
> Current method is to load data into the native tables. A lightweight method 
> is to directly move the files to the table directory. 
> With HAWQ 2.0 per table directory feature, this becomes less difficult.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (HAWQ-307) Ubuntu Support

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang closed HAWQ-307.
--

> Ubuntu Support
> --
>
> Key: HAWQ-307
> URL: https://issues.apache.org/jira/browse/HAWQ-307
> Project: Apache HAWQ
>  Issue Type: New Feature
>  Components: Build
>Reporter: Lei Chang
>Assignee: Clay B.
> Fix For: 2.0.0
>
>
> To support HAWQ running on Ubuntu OS 14.04.3



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HAWQ-308) S3 Integration

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang resolved HAWQ-308.

Resolution: Duplicate

> S3 Integration
> --
>
> Key: HAWQ-308
> URL: https://issues.apache.org/jira/browse/HAWQ-308
> Project: Apache HAWQ
>  Issue Type: New Feature
>  Components: Storage
>Reporter: Lei Chang
> Fix For: 2.1.0
>
>
> There are two methods to support S3.
> 1. Support S3 as a native file system, so storing data on S3 is just like 
> storing data on HDFS.
> 2. Support S3 as an external storage. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (HAWQ-308) S3 Integration

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang reopened HAWQ-308:


> S3 Integration
> --
>
> Key: HAWQ-308
> URL: https://issues.apache.org/jira/browse/HAWQ-308
> Project: Apache HAWQ
>  Issue Type: New Feature
>  Components: Storage
>Reporter: Lei Chang
> Fix For: 2.1.0
>
>
> There are two methods to support S3.
> 1. Support S3 as a native file system, so storing data on S3 is just like 
> storing data on HDFS.
> 2. Support S3 as an external storage. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-309) Support Centos/RHEL 7

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-309:
---
Issue Type: Wish  (was: New Feature)

> Support Centos/RHEL 7 
> --
>
> Key: HAWQ-309
> URL: https://issues.apache.org/jira/browse/HAWQ-309
> Project: Apache HAWQ
>  Issue Type: Wish
>  Components: Build
>Reporter: Lei Chang
> Fix For: 2.1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-319) REST API for HAWQ

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-319:
---
Issue Type: Wish  (was: New Feature)

> REST API for HAWQ
> -
>
> Key: HAWQ-319
> URL: https://issues.apache.org/jira/browse/HAWQ-319
> Project: Apache HAWQ
>  Issue Type: Wish
>Reporter: Lei Chang
> Fix For: backlog
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-401) json type support

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-401:
---
Issue Type: Wish  (was: New Feature)

> json type support
> -
>
> Key: HAWQ-401
> URL: https://issues.apache.org/jira/browse/HAWQ-401
> Project: Apache HAWQ
>  Issue Type: Wish
>  Components: Core
>Reporter: Lei Chang
>Assignee: Lei Chang
> Fix For: backlog
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HAWQ-337) Support Label Based Scheduling in Libyarn and RM

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang resolved HAWQ-337.

Resolution: Duplicate

> Support Label Based Scheduling in Libyarn and RM
> 
>
> Key: HAWQ-337
> URL: https://issues.apache.org/jira/browse/HAWQ-337
> Project: Apache HAWQ
>  Issue Type: New Feature
>  Components: libyarn, Resource Manager
>Reporter: Lin Wen
>Assignee: Lei Chang
> Fix For: backlog
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-470) Add IDEA specific files to .gitignore

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-470:
---
Issue Type: Wish  (was: Task)

> Add IDEA specific files to .gitignore
> -
>
> Key: HAWQ-470
> URL: https://issues.apache.org/jira/browse/HAWQ-470
> Project: Apache HAWQ
>  Issue Type: Wish
>  Components: Build
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Konstantin Boudnik
>Assignee: Konstantin Boudnik
> Attachments: 
> 0001-HAWQ-470.-Add-IDEA-specific-files-to-.gitignore.patch
>
>
> If IntelliJ IDEA is used to dev HAWQ, it looks quite ugly because all the 
> service files aren't ignored by git.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (HAWQ-479) Upgrade JSON-C dependency from 0.6 to 0.12

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang closed HAWQ-479.
--
Resolution: Fixed

> Upgrade JSON-C dependency from 0.6 to 0.12
> --
>
> Key: HAWQ-479
> URL: https://issues.apache.org/jira/browse/HAWQ-479
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Build
>Reporter: xin zhang
>Assignee: Lei Chang
>
> In order to support building HAWQ with GPORCA, we have to bump the JSON-C 
> version from 0.6 to 0.12.
> This requires changing the path of JSON from *json/json.h* to *json-c/json.h*.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-565) libhdfs3's support for async IO

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-565:
---
Issue Type: Wish  (was: Improvement)

> libhdfs3's support for async IO
> ---
>
> Key: HAWQ-565
> URL: https://issues.apache.org/jira/browse/HAWQ-565
> Project: Apache HAWQ
>  Issue Type: Wish
>  Components: libhdfs
>Reporter: Glen Cao
>Assignee: Lei Chang
>
> Q1: Does libhdfs3 support async IO operations? I.e., when libhdfs3 writes 
> large files to HDFS, it doesn't need to block to wait for writes done. While 
> files are being written to HDFS, apps using libhdfs3 could do other things 
> meanwhile without using explicit multi-theading.
> Q2: Any schedule to support ORC files?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (HAWQ-708) Consume latest GPORCA features up to version 1.630

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang closed HAWQ-708.
--
Resolution: Fixed

Code has been merged

> Consume latest GPORCA features up to version 1.630
> --
>
> Key: HAWQ-708
> URL: https://issues.apache.org/jira/browse/HAWQ-708
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Optimizer
>Reporter: xin zhang
>Assignee: Amr El-Helw
>
> There are additional GPORCA features benefiting the HAWQ community, and this 
> ticket is to bump the GPORCA from version 1.627 to 1.630. Here is list of new 
> features in GPORCA:
> -- 1.628 and rename EatMalloc to EatTracker
>Better memory tracking.
> -- 1.629 to include new GUC for Join Exploration
> New GUC optimizer_join_arity_for_associativity_commutativity is added to
> control Join exploration, hence speed up optimization time [#116333899]
> For example, `set optimizer_join_arity_for_associativity_commutativity=x` 
> will
> hint at the optimizer to stop exploring join associativity and join
> commutativity transformations when an n-ary join operator has more than
> x children during optimization, pruning quite a bit of the search space.
> -- 1.630 to fix CTE Predicate pushdown
> This fix will prevent predicate pushdown with volatile function like
> random(), which caused wrong result if predicates were duplicated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (HAWQ-718) wiki of "Build and Install" for CentOS 7.x has some little issues.

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang closed HAWQ-718.
--
Resolution: Fixed

> wiki of "Build and Install" for CentOS 7.x has some little issues.
> --
>
> Key: HAWQ-718
> URL: https://issues.apache.org/jira/browse/HAWQ-718
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Rafael Gu
>Assignee: Paul Guo
>
> Followed https://cwiki.apache.org/confluence/display/HAWQ/Build+and+Install 
> several days ago, I built and installed HAWQ on my two PCs both with CentOS 7 
> installed.  One is i7/16G/native centos 7, another is i7/8G/vmware centos 7.  
> I found some issues(both machines) as follows:
> 1. There is another lib need to be yum installed when I met some compiling 
> errors.  Sorry, I forgot the lib's name:(, but it's easy to find if you 
> recompile HAWQ on a cleaned CentOS 7.
> 2. I met "library initialization failed - unable to allocate file descriptor 
> table - out of memory ".  I cannot find suggested hardware(memory) pages, 
> looks 16G memory is too small to compile HAWQ.  I adjust vm.overcommit_memory 
> from 2 to 1(mentioned by Lei Chang) and compiling process passed:)
> 3. It's confused when I saw [# Please follow the above steps to install 
> libyarn. You may need to run "ldconfig -p " after libyarn is 
> installed.], I cannot find what's "the above steps to install libyarn".  I 
> just compiled and ldconfig the libyarn like other lib sourcecode I did, and 
> it works well.
> 4. I also met the errors as follows, when I want to run HAWQ:
> 20160429:18:15:24:071396 hawq_init:localhost:rafael-[ERROR]:-sync 
> hawq-site.xml failed.
> 20160429:18:15:24:071396 hawq_init:localhost:rafael-[ERROR]:-Set 
> default_hash_table_bucket_number failed   
> Lei Change helped me again to solve this issue by using password-less ssh.  
> No corresponding words found in the wiki document.
> Hope above descriptions helps to improve the WiKi document.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-761) Provide brew packaging for HAWQ

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-761:
---
Issue Type: Wish  (was: Improvement)

> Provide brew packaging for HAWQ
> ---
>
> Key: HAWQ-761
> URL: https://issues.apache.org/jira/browse/HAWQ-761
> Project: Apache HAWQ
>  Issue Type: Wish
>  Components: Build
>Reporter: Roman Shaposhnik
>Assignee: Lei Chang
>
> Now that HAWQ is getting really close to rolling its first official Incubator 
> release it would be great to provide brew packaging for it so that more folks 
> can take it for a spin.
> Here's what it takes to add a formula to brew (ask on this JIRA if you have 
> further questions):
>  
> https://github.com/Homebrew/brew/blob/master/share/doc/homebrew/Formula-Cookbook.md
> I propose that for the brew packaging HAWQ is configured to use local 
> filesystem as HDFS (more correctly HCFS) and runs without YARN in a 
> standalone mode.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-779) support more pxf filter pushdwon

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-779:
---
Issue Type: New Feature  (was: Wish)

>  support more pxf filter pushdwon
> -
>
> Key: HAWQ-779
> URL: https://issues.apache.org/jira/browse/HAWQ-779
> Project: Apache HAWQ
>  Issue Type: New Feature
>  Components: PXF
>Reporter: Devin Jia
>Assignee: Goden Yao
> Fix For: backlog
>
>
> When I use the pxf hawq, I need to read a traditional relational database 
> systems and solr by way of the external table. The project 
> :https://github.com/Pivotal-Field-Engineering/pxf-field/tree/master/jdbc-pxf-ext,
>  only "WriteAccessor ",so I developed 2 plug-ins, the projects: 
> https://github.com/inspur-insight/pxf-plugin , But these two plug-ins need to 
> modified HAWQ:
> 1. When get a list of fragment from pxf services, push down the 
> 'filterString'. modify the backend / optimizer / plan / createplan.c of 
> create_pxf_plan methods:
> segdb_work_map = map_hddata_2gp_segments (uri_str,
> total_segs, segs_participating,
> relation, ctx-> root-> parse-> jointree-> quals);
> 2. modify pxffilters.h and pxffilters.c, support TEXT types LIKE operation, 
> Date type data operator, Float type operator.
> 3. Modify org.apache.hawq.pxf.api.FilterParser.java, support the LIKE 
> operator.
> I already created a feature branch in my local ,and tested.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-786) HAWQ supports ORC as a native storage format

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-786:
---
Issue Type: Wish  (was: New Feature)

> HAWQ supports ORC as a native storage format
> 
>
> Key: HAWQ-786
> URL: https://issues.apache.org/jira/browse/HAWQ-786
> Project: Apache HAWQ
>  Issue Type: Wish
>  Components: Storage
>Reporter: Lei Chang
>Assignee: hongwu
> Attachments: ORCdesign-v0.1-2016-06-17.pdf
>
>
> In current HAWQ, two native formats are supported: AO and parquet. Now we 
> want to support ORC. A framework to support naive c/c++ pluggable format is 
> needed to support ORC more easily. And it can also be potentially used for 
> fast external data access.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-788) Explicitly initialize GPOPT and its dependencies

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-788:
---
Issue Type: Improvement  (was: New Feature)

> Explicitly initialize GPOPT and its dependencies
> 
>
> Key: HAWQ-788
> URL: https://issues.apache.org/jira/browse/HAWQ-788
> Project: Apache HAWQ
>  Issue Type: Improvement
>Reporter: Ivan Weng
>Assignee: Lei Chang
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-790) Remove CTranslatorPlStmtToDXL Deadcode

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-790:
---
Summary: Remove CTranslatorPlStmtToDXL Deadcode  (was: Remove 
CTranslatorPlStmtToDXL Deadcode [#119102697])

> Remove CTranslatorPlStmtToDXL Deadcode
> --
>
> Key: HAWQ-790
> URL: https://issues.apache.org/jira/browse/HAWQ-790
> Project: Apache HAWQ
>  Issue Type: Improvement
>Reporter: Ivan Weng
>Assignee: Lei Chang
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-790) Remove CTranslatorPlStmtToDXL Deadcode [#119102697]

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-790:
---
Issue Type: Improvement  (was: Task)

> Remove CTranslatorPlStmtToDXL Deadcode [#119102697]
> ---
>
> Key: HAWQ-790
> URL: https://issues.apache.org/jira/browse/HAWQ-790
> Project: Apache HAWQ
>  Issue Type: Improvement
>Reporter: Ivan Weng
>Assignee: Lei Chang
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HAWQ-817) Remove deprecated test cases from checkinstall-good

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang resolved HAWQ-817.

Resolution: Fixed

> Remove deprecated test cases from checkinstall-good
> ---
>
> Key: HAWQ-817
> URL: https://issues.apache.org/jira/browse/HAWQ-817
> Project: Apache HAWQ
>  Issue Type: New Feature
>  Components: Core
>Reporter: Yi Jin
>Assignee: Yi Jin
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-831) Re-implementation for some data types of Parquet in HAWQ

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-831:
---
Issue Type: New Feature  (was: Task)

> Re-implementation for some data types of Parquet in HAWQ
> 
>
> Key: HAWQ-831
> URL: https://issues.apache.org/jira/browse/HAWQ-831
> Project: Apache HAWQ
>  Issue Type: New Feature
>  Components: Storage
>Reporter: Lili Ma
>Assignee: Lei Chang
>
> Re-implement some HAWQ data type mapping to Parquet, so that HAWQ can be 
> compatible with Hive.
> 1. Currently HAWQ converts data type decimal to byte array, we can modify it 
> to fixed length byte array, so that HAWQ can be compatible with Hive.
> 2. HAWQ itself can convert the data type char to int32 instead of byte array, 
> so that the space for char storage can be saved.
> 3. To be compatible with Hive, HAWQ can change its type mapping for decimal 
> to int96, but this need discussion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-825) HAWQ component reload shows misleading message on stdout

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-825:
---
Assignee: Radar Lei  (was: Lei Chang)

> HAWQ component reload shows misleading message on stdout
> 
>
> Key: HAWQ-825
> URL: https://issues.apache.org/jira/browse/HAWQ-825
> Project: Apache HAWQ
>  Issue Type: Bug
>Reporter: Matt
>Assignee: Radar Lei
> Attachments: stdout.png
>
>
> After running {code}hawq stop master -u{code}, the message on the stdout says 
> the master was successfully stopped.
> The -u option does not stop the master, it only reloads the configurations. 
> The stdout message should be changed for this operation, so that it does not 
> mislead the user.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-786) HAWQ supports ORC as a native storage format

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-786:
---
Description: In current HAWQ, two native formats are supported: AO and 
parquet. Now we want to support ORC. A framework to support naive c/c++ 
pluggable format is needed to support ORC more easily. And it can also be 
potentially used for fast external data access.  (was: In current HAWQ, two 
native formats are supported: AO and parquet. Now we want to support ORC. A 
framework to support naive c/c++ pluggable format is needed. And it can also be 
potentially used for fast external data access.)

> HAWQ supports ORC as a native storage format
> 
>
> Key: HAWQ-786
> URL: https://issues.apache.org/jira/browse/HAWQ-786
> Project: Apache HAWQ
>  Issue Type: New Feature
>  Components: Storage
>Reporter: Lei Chang
>Assignee: hongwu
>
> In current HAWQ, two native formats are supported: AO and parquet. Now we 
> want to support ORC. A framework to support naive c/c++ pluggable format is 
> needed to support ORC more easily. And it can also be potentially used for 
> fast external data access.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Deleted] (HAWQ-639) HAWQ support ORC as a native storage format

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang deleted HAWQ-639:
---


> HAWQ support ORC as a native storage format
> ---
>
> Key: HAWQ-639
> URL: https://issues.apache.org/jira/browse/HAWQ-639
> Project: Apache HAWQ
>  Issue Type: Sub-task
>Reporter: Goden Yao
>Assignee: hongwu
>
> As a user, I want to be able to store my table as ORC format in HAWQ



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HAWQ-639) HAWQ support ORC as a native storage format

[Lei Chang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328762#comment-15328762
 ] 

Lei Chang edited comment on HAWQ-639 at 6/14/16 1:52 AM:
-

consolidate this with HAWQ-786


was (Author: lei_chang):
consolidate this with HAWQ-796

> HAWQ support ORC as a native storage format
> ---
>
> Key: HAWQ-639
> URL: https://issues.apache.org/jira/browse/HAWQ-639
> Project: Apache HAWQ
>  Issue Type: Sub-task
>Reporter: Goden Yao
>Assignee: hongwu
>
> As a user, I want to be able to store my table as ORC format in HAWQ



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-639) HAWQ support ORC as a native storage format

[Lei Chang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328762#comment-15328762
 ] 

Lei Chang commented on HAWQ-639:


consolidate this with HAWQ-796

> HAWQ support ORC as a native storage format
> ---
>
> Key: HAWQ-639
> URL: https://issues.apache.org/jira/browse/HAWQ-639
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Storage
>Reporter: Goden Yao
>Assignee: hongwu
>
> As a user, I want to be able to store my table as ORC format in HAWQ



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-786) HAWQ supports ORC as a native storage format

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-786:
---
Summary: HAWQ supports ORC as a native storage format  (was: Framework to 
support native C/C++ pluggable formats (ORC et al))

> HAWQ supports ORC as a native storage format
> 
>
> Key: HAWQ-786
> URL: https://issues.apache.org/jira/browse/HAWQ-786
> Project: Apache HAWQ
>  Issue Type: New Feature
>  Components: Storage
>Reporter: Lei Chang
>Assignee: hongwu
>
> In current HAWQ, two native formats are supported: AO and parquet. Now we 
> want to support ORC. A framework to support naive c/c++ pluggable format is 
> needed. And it can also be potentially used for fast external data access.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-786) Framework to support native C/C++ pluggable formats (ORC et al)

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-786:
---
Summary: Framework to support native C/C++ pluggable formats (ORC et al)  
(was: Framework to support native C/C++ pluggable format (ORC et al))

> Framework to support native C/C++ pluggable formats (ORC et al)
> ---
>
> Key: HAWQ-786
> URL: https://issues.apache.org/jira/browse/HAWQ-786
> Project: Apache HAWQ
>  Issue Type: New Feature
>  Components: Storage
>Reporter: Lei Chang
>Assignee: hongwu
>
> In current HAWQ, two native formats are supported: AO and parquet. Now we 
> want to support ORC. A framework to support naive c/c++ pluggable format is 
> needed. And it can also be potentially used for fast external data access.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-786) Framework to support native C/C++ pluggable format (ORC et al)

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-786:
---
Description: In current HAWQ, two native formats are supported: AO and 
parquet. Now we want to support ORC. A framework to support naive c/c++ 
pluggable format is needed. And it can also be potentially used for fast 
external data access.  (was: 
In current HAWQ, two native formats are supported: AO and parquet. Now we want 
to support ORC. A framework to support pluggable format with C/C++ scanner is 
needed. And it can also be potentially used for fast external data access.)

> Framework to support native C/C++ pluggable format (ORC et al)
> --
>
> Key: HAWQ-786
> URL: https://issues.apache.org/jira/browse/HAWQ-786
> Project: Apache HAWQ
>  Issue Type: New Feature
>  Components: Storage
>Reporter: Lei Chang
>Assignee: hongwu
>
> In current HAWQ, two native formats are supported: AO and parquet. Now we 
> want to support ORC. A framework to support naive c/c++ pluggable format is 
> needed. And it can also be potentially used for fast external data access.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-786) Framework to support native C/C++ pluggable format

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-786:
---
Summary: Framework to support native C/C++ pluggable format  (was: 
Framework to support pluggable format with C/C++ scanner)

> Framework to support native C/C++ pluggable format
> --
>
> Key: HAWQ-786
> URL: https://issues.apache.org/jira/browse/HAWQ-786
> Project: Apache HAWQ
>  Issue Type: New Feature
>  Components: Storage
>Reporter: Lei Chang
>Assignee: hongwu
>
> In current HAWQ, two native formats are supported: AO and parquet. Now we 
> want to support ORC. A framework to support pluggable format with C/C++ 
> scanner is needed. And it can also be potentially used for fast external data 
> access.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-786) Framework to support native C/C++ pluggable format (ORC et al)

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-786:
---
Summary: Framework to support native C/C++ pluggable format (ORC et al)  
(was: Framework to support native C/C++ pluggable format)

> Framework to support native C/C++ pluggable format (ORC et al)
> --
>
> Key: HAWQ-786
> URL: https://issues.apache.org/jira/browse/HAWQ-786
> Project: Apache HAWQ
>  Issue Type: New Feature
>  Components: Storage
>Reporter: Lei Chang
>Assignee: hongwu
>
> In current HAWQ, two native formats are supported: AO and parquet. Now we 
> want to support ORC. A framework to support pluggable format with C/C++ 
> scanner is needed. And it can also be potentially used for fast external data 
> access.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-639) HAWQ support ORC as a native storage format

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-639:
---
Issue Type: Sub-task  (was: New Feature)
Parent: HAWQ-786

> HAWQ support ORC as a native storage format
> ---
>
> Key: HAWQ-639
> URL: https://issues.apache.org/jira/browse/HAWQ-639
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Storage
>Reporter: Goden Yao
>Assignee: hongwu
>
> As a user, I want to be able to store my table as ORC format in HAWQ



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-786) Framework to support pluggable format with C/C++ scanner

[Lei Chang (JIRA)]

Lei Chang created HAWQ-786:
--

 Summary: Framework to support pluggable format with C/C++ scanner
 Key: HAWQ-786
 URL: https://issues.apache.org/jira/browse/HAWQ-786
 Project: Apache HAWQ
  Issue Type: New Feature
  Components: Storage
Reporter: Lei Chang
Assignee: Lei Chang



In current HAWQ, two native formats are supported: AO and parquet. Now we want 
to support ORC. A framework to support pluggable format with C/C++ scanner is 
needed. And it can also be potentially used for fast external data access.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-752) build pxf compatible with Apache Hadoop

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-752:
---
Fix Version/s: backlog

>  build pxf compatible with Apache Hadoop
> 
>
> Key: HAWQ-752
> URL: https://issues.apache.org/jira/browse/HAWQ-752
> Project: Apache HAWQ
>  Issue Type: Wish
>  Components: Build
>Reporter: louis liu
>Assignee: Goden Yao
> Fix For: backlog
>
>
> How to build pxf compatible with Apache Hadoop？ 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-752) build pxf compatible with Apache Hadoop

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-752:
---
Assignee: Goden Yao  (was: Lei Chang)

>  build pxf compatible with Apache Hadoop
> 
>
> Key: HAWQ-752
> URL: https://issues.apache.org/jira/browse/HAWQ-752
> Project: Apache HAWQ
>  Issue Type: Wish
>  Components: Build
>Reporter: louis liu
>Assignee: Goden Yao
> Fix For: backlog
>
>
> How to build pxf compatible with Apache Hadoop？ 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-744) Add plperl code

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-744:
---
Assignee: Paul Guo  (was: Lei Chang)

> Add plperl code
> ---
>
> Key: HAWQ-744
> URL: https://issues.apache.org/jira/browse/HAWQ-744
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Build
>Reporter: Paul Guo
>Assignee: Paul Guo
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-746) ssh_utils.py:addHostNameAlternatives() should not use short hostname

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-746:
---
Assignee: Paul Guo  (was: Lei Chang)

> ssh_utils.py:addHostNameAlternatives() should not use short hostname
> 
>
> Key: HAWQ-746
> URL: https://issues.apache.org/jira/browse/HAWQ-746
> Project: Apache HAWQ
>  Issue Type: Bug
>Reporter: Paul Guo
>Assignee: Paul Guo
>
> It is possible short hostname (usually means internal hostname) could not be 
> resolved by naming service. Using fqdn is enough. We've seen "hawq 
> ssh-exkeys" failure due to this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-748) Fix platform issue in test framework

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-748:
---
Assignee: zhenglin tao  (was: Jiali Yao)

> Fix platform issue in test framework
> 
>
> Key: HAWQ-748
> URL: https://issues.apache.org/jira/browse/HAWQ-748
> Project: Apache HAWQ
>  Issue Type: Test
>  Components: Tests
>Reporter: zhenglin tao
>Assignee: zhenglin tao
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-742) support "YARN label based scheduling" in HAWQ

[Lei Chang (JIRA)]

Lei Chang created HAWQ-742:
--

 Summary: support "YARN label based scheduling" in HAWQ
 Key: HAWQ-742
 URL: https://issues.apache.org/jira/browse/HAWQ-742
 Project: Apache HAWQ
  Issue Type: New Feature
  Components: Resource Manager
Reporter: Lei Chang
Assignee: Lei Chang


Some customers want to run HAWQ in a subset of nodes by using YARN to configure 
the subset.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-304) Support update and delete on non-heap tables

[Lei Chang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288018#comment-15288018
 ] 

Lei Chang commented on HAWQ-304:


it is possible and the item is currently under investigation. and after 
investigation, the design will be proposed...

> Support update and delete on non-heap tables
> 
>
> Key: HAWQ-304
> URL: https://issues.apache.org/jira/browse/HAWQ-304
> Project: Apache HAWQ
>  Issue Type: New Feature
>  Components: Storage
>Reporter: Lei Chang
> Fix For: 3.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-736) can change ssh port in "hawq ssh"

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-736:
---
Fix Version/s: backlog

> can change ssh port in "hawq ssh"
> -
>
> Key: HAWQ-736
> URL: https://issues.apache.org/jira/browse/HAWQ-736
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Command Line Tools
>Reporter: Lei Chang
>Assignee: Lei Chang
> Fix For: backlog
>
>
> we should be able to change the default port in "hawq ssh"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-736) can change ssh port in "hawq ssh"

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang updated HAWQ-736:
---
Assignee: Radar Lei  (was: Lei Chang)

> can change ssh port in "hawq ssh"
> -
>
> Key: HAWQ-736
> URL: https://issues.apache.org/jira/browse/HAWQ-736
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Command Line Tools
>Reporter: Lei Chang
>Assignee: Radar Lei
> Fix For: backlog
>
>
> we should be able to change the default port in "hawq ssh"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-736) can change ssh port in "hawq ssh"

[Lei Chang (JIRA)]

Lei Chang created HAWQ-736:
--

 Summary: can change ssh port in "hawq ssh"
 Key: HAWQ-736
 URL: https://issues.apache.org/jira/browse/HAWQ-736
 Project: Apache HAWQ
  Issue Type: Improvement
  Components: Command Line Tools
Reporter: Lei Chang
Assignee: Lei Chang


we should be able to change the default port in "hawq ssh"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (HAWQ-355) order by problem: sorting varchar column with space is not correct.

[Lei Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei Chang reopened HAWQ-355:


> order by problem: sorting varchar column with space is not correct.
> ---
>
> Key: HAWQ-355
> URL: https://issues.apache.org/jira/browse/HAWQ-355
> Project: Apache HAWQ
>  Issue Type: Bug
>Reporter: SuperJDC
>Assignee: Lei Chang
> Fix For: 2.0.0
>
>
> Firstly, my hawq download from official website:  
> https://network.pivotal.io/products/pivotal-hdb, a released stable version. 
> My steps:
> DROP TABLE IF EXISTS testorder;
> CREATE TABLE testorder(
>   ss VARCHAR(10)
> ) distributed randomly;
> INSERT INTO testorder 
> VALUES ('cc'), ('c c'), ('cc'), 
> ('aa'), ('a a'), ('ac'), 
> ('b c'), ('bc'), ('bb');
> SELECT ss FROM testorder 
> ORDER BY ss;
> The result:
> aa
> a a
> ac
> bb
> bc
> b c
> cc
> cc
> c c
> It seems that when a colum has a space char, the sorted result would been not 
> correct.
> I followed the document steps and successfully integrated with the ambari. 
> All of hawq configurations are the default.
> Thanks!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)