[
https://issues.apache.org/jira/browse/HAWQ-459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shivram Mani updated HAWQ-459:
------------------------------
Description:
Enhance the metadata endpoint using /Metdata/getMetadata to support retrieving
metadata for generic datasources, i.e. Hive, HBase, HDFS, etc and support
retrieving metadata for multiple objects based on wildcard pattern
h1.Method supported
||*name*||
|GET|
h1.Path mapped
pxf/<protocolVersion>/Metadata/getMetadata
h1.Input parameters:
||Name||Description||Is mandatory?||Could be a wildcard?||Valid input values||
|profile|Type of source to fetch data from, should support Hive.|Yes|No|hive|
|pattern|File or table name or pattern. Patterns are supported using wildcard
*. |Yes|Yes|db.table or file path|
In case of Hive, patterns corresponds to schema.table
In case of HBase, pattern=schema.table
In case of HDFS, pattern=absolute path to file
h1. HTTP return codes:
||*Code*||*Meaning*||
|200|found some result set|
|400|not all required parameters passed|
|404|didn't find any results for given parameters, empty result set|
|405|called method other than GET|
|500|unable to access underlying metastore(hive, hbase, hdfs etc), datasource
not supported, some other exceptions|
h1. Return data format for 200 code:
{code}
{
"PXFMetadata":[
{
"item":{
"path":"default",
"name":"abc"
},
"fields":[
{
"name":"a",
"type":"int"
},
{
"name":"b",
"type":"float"
}
]
},
{
"item":{
"path":"default",
"name":"abcdef"
},
"fields":[
{
"name":"a",
"type":"int"
},
{
"name":"b",
"type":"float"
}
]
},
{
"item":{
"path":"default",
"name":"abcdefghk"
},
"fields":[
{
"name":"a",
"type":"int"
},
{
"name":"b",
"type":"float"
}
]
}
]
}
{code}
h1. Return data format for 404 code:
{code}
{
"PXFMetadata":[]
}
{code}
h1. Return data format for 500 code:
{code}
{
"PXFError":{"code" : "XXX1", "message" : "Unable to reach Hive metastore"}
}
{code}
h1. Examples
Request:
{code}
curl -i
"http://localhost:51200/pxf/v14/Metadata/getMetadata?profile=hive&pattern=default.abc*";
{code}
Response:
{code}
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: application/json
Transfer-Encoding: chunked
Date: Fri, 26 Feb 2016 03:33:53 GMT
{
"PXFMetadata":[
{
"item":{
"path":"default",
"name":"abc"
},
"fields":[
{
"name":"a",
"type":"int"
},
{
"name":"b",
"type":"float"
}
]
},
{
"item":{
"path":"default",
"name":"abcdef"
},
"fields":[
{
"name":"a",
"type":"int"
},
{
"name":"b",
"type":"float"
}
]
},
{
"item":{
"path":"default",
"item":"abcdefghk"
},
"fields":[
{
"name":"a",
"type":"int"
},
{
"name":"b",
"type":"float"
}
]
}
]
}
{code}
was:
Currently we have *pxf/v14/Metadata/getTableMetadata* endpoint, which takes
*table* as a path parameter and returns metadata for one given table.
Idea is to enhance this endpoint to support retrieving metadata for generic
datasources, i.e. Hive, HBase, HDFS, etc and support retrieving metadata for
multiple objects based on wildcard pattern
h1.Method supported
||*name*||
|GET|
h1.Path mapped
pxf/<protocolVersion>/Metadata/getMetadata
h1.Input parameters:
||Name||Description||Is mandatory?||Could be a wildcard?||Valid input values||
|profile|Type of source to fetch data from, should support Hive.|Yes|No|hive|
|pattern|File or table name or pattern. Patterns are supported using wildcard
*. |Yes|Yes|db.table or file path|
In case of Hive, patterns corresponds to schema.table
In case of HBase, pattern=schema.table
In case of HDFS, pattern=absolute path to file
h1. HTTP return codes:
||*Code*||*Meaning*||
|200|found some result set|
|400|not all required parameters passed|
|404|didn't find any results for given parameters, empty result set|
|405|called method other than GET|
|500|unable to access underlying metastore(hive, hbase, hdfs etc), datasource
not supported, some other exceptions|
h1. Return data format for 200 code:
{code}
{
"PXFMetadata":[
{
"item":{
"path":"default",
"name":"abc"
},
"fields":[
{
"name":"a",
"type":"int"
},
{
"name":"b",
"type":"float"
}
]
},
{
"item":{
"path":"default",
"name":"abcdef"
},
"fields":[
{
"name":"a",
"type":"int"
},
{
"name":"b",
"type":"float"
}
]
},
{
"item":{
"path":"default",
"name":"abcdefghk"
},
"fields":[
{
"name":"a",
"type":"int"
},
{
"name":"b",
"type":"float"
}
]
}
]
}
{code}
h1. Return data format for 404 code:
{code}
{
"PXFMetadata":[]
}
{code}
h1. Return data format for 500 code:
{code}
{
"PXFError":{"code" : "XXX1", "message" : "Unable to reach Hive metastore"}
}
{code}
h1. Examples
Request:
{code}
curl -i
"http://localhost:51200/pxf/v14/Metadata/getMetadata?profile=hive&pattern=default.abc*";
{code}
Response:
{code}
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: application/json
Transfer-Encoding: chunked
Date: Fri, 26 Feb 2016 03:33:53 GMT
{
"PXFMetadata":[
{
"item":{
"path":"default",
"name":"abc"
},
"fields":[
{
"name":"a",
"type":"int"
},
{
"name":"b",
"type":"float"
}
]
},
{
"item":{
"path":"default",
"name":"abcdef"
},
"fields":[
{
"name":"a",
"type":"int"
},
{
"name":"b",
"type":"float"
}
]
},
{
"item":{
"path":"default",
"item":"abcdefghk"
},
"fields":[
{
"name":"a",
"type":"int"
},
{
"name":"b",
"type":"float"
}
]
}
]
}
{code}
> Enhance Metadata Rest API to support table(s)
> ---------------------------------------------
>
> Key: HAWQ-459
> URL: https://issues.apache.org/jira/browse/HAWQ-459
> Project: Apache HAWQ
> Issue Type: Sub-task
> Components: Hcatalog, PXF
> Reporter: Shivram Mani
> Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> Enhance the metadata endpoint using /Metdata/getMetadata to support
> retrieving metadata for generic datasources, i.e. Hive, HBase, HDFS, etc and
> support retrieving metadata for multiple objects based on wildcard pattern
> h1.Method supported
> ||*name*||
> |GET|
> h1.Path mapped
> pxf/<protocolVersion>/Metadata/getMetadata
> h1.Input parameters:
> ||Name||Description||Is mandatory?||Could be a wildcard?||Valid input values||
> |profile|Type of source to fetch data from, should support Hive.|Yes|No|hive|
> |pattern|File or table name or pattern. Patterns are supported using wildcard
> *. |Yes|Yes|db.table or file path|
> In case of Hive, patterns corresponds to schema.table
> In case of HBase, pattern=schema.table
> In case of HDFS, pattern=absolute path to file
> h1. HTTP return codes:
> ||*Code*||*Meaning*||
> |200|found some result set|
> |400|not all required parameters passed|
> |404|didn't find any results for given parameters, empty result set|
> |405|called method other than GET|
> |500|unable to access underlying metastore(hive, hbase, hdfs etc), datasource
> not supported, some other exceptions|
> h1. Return data format for 200 code:
> {code}
> {
> "PXFMetadata":[
> {
> "item":{
> "path":"default",
> "name":"abc"
> },
> "fields":[
> {
> "name":"a",
> "type":"int"
> },
> {
> "name":"b",
> "type":"float"
> }
> ]
> },
> {
> "item":{
> "path":"default",
> "name":"abcdef"
> },
> "fields":[
> {
> "name":"a",
> "type":"int"
> },
> {
> "name":"b",
> "type":"float"
> }
> ]
> },
> {
> "item":{
> "path":"default",
> "name":"abcdefghk"
> },
> "fields":[
> {
> "name":"a",
> "type":"int"
> },
> {
> "name":"b",
> "type":"float"
> }
> ]
> }
> ]
> }
> {code}
> h1. Return data format for 404 code:
> {code}
> {
> "PXFMetadata":[]
> }
> {code}
> h1. Return data format for 500 code:
> {code}
> {
> "PXFError":{"code" : "XXX1", "message" : "Unable to reach Hive metastore"}
> }
> {code}
> h1. Examples
> Request:
> {code}
> curl -i
> "http://localhost:51200/pxf/v14/Metadata/getMetadata?profile=hive&pattern=default.abc*";
>
> {code}
> Response:
> {code}
> HTTP/1.1 200 OK
> Server: Apache-Coyote/1.1
> Content-Type: application/json
> Transfer-Encoding: chunked
> Date: Fri, 26 Feb 2016 03:33:53 GMT
> {
> "PXFMetadata":[
> {
> "item":{
> "path":"default",
> "name":"abc"
> },
> "fields":[
> {
> "name":"a",
> "type":"int"
> },
> {
> "name":"b",
> "type":"float"
> }
> ]
> },
> {
> "item":{
> "path":"default",
> "name":"abcdef"
> },
> "fields":[
> {
> "name":"a",
> "type":"int"
> },
> {
> "name":"b",
> "type":"float"
> }
> ]
> },
> {
> "item":{
> "path":"default",
> "item":"abcdefghk"
> },
> "fields":[
> {
> "name":"a",
> "type":"int"
> },
> {
> "name":"b",
> "type":"float"
> }
> ]
> }
> ]
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
