[
https://issues.apache.org/jira/browse/HAWQ-53?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shivram Mani updated HAWQ-53:
-----------------------------
Description:
*Current Behavior*
Create pxf external table with an avro schema which contains complex nullable
type with null as default value would fail.
*Expected Behavior*
The complex avro types should be properly mapped to HAWQ types and create table
should have no error with data loaded.
*Analysis*
>From code review
>[AvroResolver.java|https://github.com/apache/incubator-hawq/blob/master/pxf/pxf-hdfs/src/main/java/com/pivotal/pxf/plugins/hdfs/AvroResolver.java],
> we didn't specify null situation for complex types:
- Array
- Map
- Record
when using union type , which contains a nullable type(union) and use null as
default - I think complex type with nullable type in value schema and with null
as default is the problem.
{code:javascript}
...
"name" : "meta__kvpairs",
"type" : [ "null", {
"type" : "map",
"values" : [ "null", "string" ],
"default" : null
} ],
"default" : null
},
...
{code}
Create external table from pxf to read it:
{code:sql}
"CREATE EXTERNAL TABLE meetmeevent_hdfs(
sourcetimestamp bigint,
sourceagent varchar(200),
sourceip_address varchar(20),
meetmeuser_id bigint,
meetmeinterested_uid bigint,
meetmevote varchar(4),
meetmeis_match int,
meetmenetwork_score bigint,
meetmeresponsiveness_score bigint,
meetmesession_id varchar(32),
meetmefriends int,
meetmeprevious_view varchar(32),
meetmemodel bigint,
meetmescore float8,
meetmemethod int,
meetmecontributions varchar(120),
meetmeclicksource varchar(20),
meetmeclickaction varchar(20),
meetmeprofileview_ts bigint,
meetmeplatform varchar(20),
metatopic_name varchar(100),
metarequest_user_agent varchar(200),
metarequest_session_id varchar(50),
metarequest_id text,
metakvpairs varchar(1000),
meta_handlers varchar(1000)
)
LOCATION
('pxf://dahdp2nn01.tag-dev.com:50070/data/ramblas/event_log/meetme/20150824/12/s_meetme.20.0.3492572.11600054147.1440442800000.avro?PROFILE=Avro&DATA-SCHEMA=/data/ramblas/schema/meetme')
FORMAT 'CUSTOM' (formatter='pxfwritable_import');
{code}
was:
*Current Behavior*
Create pxf external table with an avro schema which contains complex nullable
type with null as default value would fail.
*Expected Behavior*
The complex avro types should be properly mapped to HAWQ types and create table
should have no error with data loaded.
*Analysis*
>From code review
>[AvroResolver.java|https://github.com/apache/incubator-hawq/blob/master/pxf/pxf-hdfs/src/main/java/com/pivotal/pxf/plugins/hdfs/AvroResolver.java],
> we didn't specify null situation for complex types:
- Array
- Map
- Record
when using union type , which contains a nullable type(union) and use null as
default - I think complex type with nullable type in value schema and with null
as default is the problem.
{code:javascript}
...
"name" : "meta__kvpairs",
"type" : [ "null", {
"type" : "map",
"values" : [ "null", "string" ],
"default" : null
} ],
"default" : null
},
...
{code}
Create external table from pxf to read it:
{code:sql}
"CREATE EXTERNAL TABLE meetmeevent_hdfs(
sourcetimestamp bigint,
sourceagent varchar(200),
sourceip_address varchar(20),
meetmeuser_id numeric(15,0),
meetmeinterested_uid numeric(15,0),
meetmevote varchar(4),
meetmeis_match numeric(3,0),
meetmenetwork_score numeric (10,0),
meetmeresponsiveness_score numeric (10,0),
meetmesession_id varchar(32),
meetmefriends numeric(1,0),
meetmeprevious_view varchar(32),
meetmemodel numeric(15,0),
meetmescore numeric(15,0),
meetmemethod numeric(1,0),
meetmecontributions varchar(120),
meetmeclicksource varchar(20),
meetmeclickaction varchar(20),
meetmeprofileview_ts bigint,
meetmeplatform varchar(20),
metatopic_name varchar(100),
metarequest_user_agent varchar(200),
metarequest_session_id varchar(50),
metarequest_id bigint,
metakvpairs varchar(1000),
meta_handlers varchar(1000),
dt varchar(6),
hour varchar(2)
)
LOCATION
('pxf://dahdp2nn01.tag-dev.com:50070/data/ramblas/event_log/meetme/20150824/12/s_meetme.20.0.3492572.11600054147.1440442800000.avro?PROFILE=Avro&DATA-SCHEMA=/data/ramblas/schema/meetme')
FORMAT 'CUSTOM' (formatter='pxfwritable_import');
{code}
> Avro Union(complex) type containing nullable type and using null as default
> failed
> ----------------------------------------------------------------------------------
>
> Key: HAWQ-53
> URL: https://issues.apache.org/jira/browse/HAWQ-53
> Project: Apache HAWQ
> Issue Type: Bug
> Components: PXF
> Reporter: Goden Yao
> Assignee: Shivram Mani
> Attachments: meetme.avsc, meetme_example.json
>
>
> *Current Behavior*
> Create pxf external table with an avro schema which contains complex nullable
> type with null as default value would fail.
> *Expected Behavior*
> The complex avro types should be properly mapped to HAWQ types and create
> table should have no error with data loaded.
> *Analysis*
> From code review
> [AvroResolver.java|https://github.com/apache/incubator-hawq/blob/master/pxf/pxf-hdfs/src/main/java/com/pivotal/pxf/plugins/hdfs/AvroResolver.java],
> we didn't specify null situation for complex types:
> - Array
> - Map
> - Record
> when using union type , which contains a nullable type(union) and use null as
> default - I think complex type with nullable type in value schema and with
> null as default is the problem.
> {code:javascript}
> ...
> "name" : "meta__kvpairs",
> "type" : [ "null", {
> "type" : "map",
> "values" : [ "null", "string" ],
> "default" : null
> } ],
> "default" : null
> },
> ...
> {code}
> Create external table from pxf to read it:
> {code:sql}
> "CREATE EXTERNAL TABLE meetmeevent_hdfs(
> sourcetimestamp bigint,
> sourceagent varchar(200),
> sourceip_address varchar(20),
> meetmeuser_id bigint,
> meetmeinterested_uid bigint,
> meetmevote varchar(4),
> meetmeis_match int,
> meetmenetwork_score bigint,
> meetmeresponsiveness_score bigint,
> meetmesession_id varchar(32),
> meetmefriends int,
> meetmeprevious_view varchar(32),
> meetmemodel bigint,
> meetmescore float8,
> meetmemethod int,
> meetmecontributions varchar(120),
> meetmeclicksource varchar(20),
> meetmeclickaction varchar(20),
> meetmeprofileview_ts bigint,
> meetmeplatform varchar(20),
> metatopic_name varchar(100),
> metarequest_user_agent varchar(200),
> metarequest_session_id varchar(50),
> metarequest_id text,
> metakvpairs varchar(1000),
> meta_handlers varchar(1000)
> )
> LOCATION
> ('pxf://dahdp2nn01.tag-dev.com:50070/data/ramblas/event_log/meetme/20150824/12/s_meetme.20.0.3492572.11600054147.1440442800000.avro?PROFILE=Avro&DATA-SCHEMA=/data/ramblas/schema/meetme')
> FORMAT 'CUSTOM' (formatter='pxfwritable_import');
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
