[jira] [Commented] (FLINK-10729) Create a Hive connector for Hive data access in Flink
[
https://issues.apache.org/jira/browse/FLINK-10729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16690840#comment-16690840
]
xuqianjin commented on FLINK-10729:
---
This function is so great, I hope to see task and design, I also want to join
you.
> Create a Hive connector for Hive data access in Flink
> -
>
> Key: FLINK-10729
> URL: https://issues.apache.org/jira/browse/FLINK-10729
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Affects Versions: 1.6.2
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
>Priority: Major
>
> As part of Flink-Hive integration effort, it's important for Flink to access
> (read/write) Hive data, which is the responsibility of Hive connector. While
> there is a HCatalog data connector in the code base, it's not complete (i.e.
> missing all connector related classes such as validators, etc.). Further,
> HCatalog interface has many limitations such as accessing a subset of Hive
> data, supporting a subset of Hive data types, etc. In addition, it's not
> actively maintained. In fact, it's now only a sub-project in Hive.
> Therefore, here we propose a complete connector set for Hive tables, not via
> HCatalog, but via direct Hive interface. HCatalog connector will be
> deprecated.
> Please note that connector on Hive metadata is already covered in other
> JIRAs, as {{HiveExternalCatalog}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10729) Create a Hive connector for Hive data access in Flink
[
https://issues.apache.org/jira/browse/FLINK-10729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16670867#comment-16670867
]
Xuefu Zhang commented on FLINK-10729:
-
Hi [~ZhenqiuHuang], Sure. Thanks for your interest. Please note this only
highlights direction we are going. We may need a design doc for this.
I was thinking of implement a connector in Flink that utilizes Hive's generic
InputFormats and OutputFormats. This solves the problem of all data formats in
Hive, as compared with solving the data formats one at a time (orc, parquet,
etc.). However, my thought is pretty primitive and I need to take a closer look.
Please also share your thoughts on this. Thanks.
> Create a Hive connector for Hive data access in Flink
> -
>
> Key: FLINK-10729
> URL: https://issues.apache.org/jira/browse/FLINK-10729
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Affects Versions: 1.6.2
>Reporter: Xuefu Zhang
>Assignee: Zhenqiu Huang
>Priority: Major
>
> As part of Flink-Hive integration effort, it's important for Flink to access
> (read/write) Hive data, which is the responsibility of Hive connector. While
> there is a HCatalog data connector in the code base, it's not complete (i.e.
> missing all connector related classes such as validators, etc.). Further,
> HCatalog interface has many limitations such as accessing a subset of Hive
> data, supporting a subset of Hive data types, etc. In addition, it's not
> actively maintained. In fact, it's now only a sub-project in Hive.
> Therefore, here we propose a complete connector set for Hive tables, not via
> HCatalog, but via direct Hive interface. HCatalog connector will be
> deprecated.
> Please note that connector on Hive metadata is already covered in other
> JIRAs, as {{HiveExternalCatalog}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10729) Create a Hive connector for Hive data access in Flink
[
https://issues.apache.org/jira/browse/FLINK-10729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16672525#comment-16672525
]
Xuefu Zhang commented on FLINK-10729:
-
Hi [~ZhenqiuHuang], I did some initial research and found this is much involved
than on the surface. It seems that the complexity comes mostly from Hive side,
and there will probably need a prototype and design. I will spend more time on
this. Thus, if you don't mind I'm going to assign this back to myself.
Once the design is completed, I'm sure there will be many subtasks to be
created. I'd appreciate if you could help at that time.
Thanks.
> Create a Hive connector for Hive data access in Flink
> -
>
> Key: FLINK-10729
> URL: https://issues.apache.org/jira/browse/FLINK-10729
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Affects Versions: 1.6.2
>Reporter: Xuefu Zhang
>Assignee: Zhenqiu Huang
>Priority: Major
>
> As part of Flink-Hive integration effort, it's important for Flink to access
> (read/write) Hive data, which is the responsibility of Hive connector. While
> there is a HCatalog data connector in the code base, it's not complete (i.e.
> missing all connector related classes such as validators, etc.). Further,
> HCatalog interface has many limitations such as accessing a subset of Hive
> data, supporting a subset of Hive data types, etc. In addition, it's not
> actively maintained. In fact, it's now only a sub-project in Hive.
> Therefore, here we propose a complete connector set for Hive tables, not via
> HCatalog, but via direct Hive interface. HCatalog connector will be
> deprecated.
> Please note that connector on Hive metadata is already covered in other
> JIRAs, as {{HiveExternalCatalog}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10729) Create a Hive connector for Hive data access in Flink
[
https://issues.apache.org/jira/browse/FLINK-10729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16672530#comment-16672530
]
Zhenqiu Huang commented on FLINK-10729:
---
[~xuefuz]
Agree. If you want to use native formats in Hive, I feel an initial prototype
is definitely needed for me to follow. For the direction of implementation,
probably we can align with [~fhueske] also. Looking forward to the design.
> Create a Hive connector for Hive data access in Flink
> -
>
> Key: FLINK-10729
> URL: https://issues.apache.org/jira/browse/FLINK-10729
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Affects Versions: 1.6.2
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
>Priority: Major
>
> As part of Flink-Hive integration effort, it's important for Flink to access
> (read/write) Hive data, which is the responsibility of Hive connector. While
> there is a HCatalog data connector in the code base, it's not complete (i.e.
> missing all connector related classes such as validators, etc.). Further,
> HCatalog interface has many limitations such as accessing a subset of Hive
> data, supporting a subset of Hive data types, etc. In addition, it's not
> actively maintained. In fact, it's now only a sub-project in Hive.
> Therefore, here we propose a complete connector set for Hive tables, not via
> HCatalog, but via direct Hive interface. HCatalog connector will be
> deprecated.
> Please note that connector on Hive metadata is already covered in other
> JIRAs, as {{HiveExternalCatalog}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10729) Create a Hive connector for Hive data access in Flink
[
https://issues.apache.org/jira/browse/FLINK-10729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673015#comment-16673015
]
liwei commented on FLINK-10729:
---
I'd like to join you. And this feature is really needed now.
> Create a Hive connector for Hive data access in Flink
> -
>
> Key: FLINK-10729
> URL: https://issues.apache.org/jira/browse/FLINK-10729
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Affects Versions: 1.6.2
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
>Priority: Major
>
> As part of Flink-Hive integration effort, it's important for Flink to access
> (read/write) Hive data, which is the responsibility of Hive connector. While
> there is a HCatalog data connector in the code base, it's not complete (i.e.
> missing all connector related classes such as validators, etc.). Further,
> HCatalog interface has many limitations such as accessing a subset of Hive
> data, supporting a subset of Hive data types, etc. In addition, it's not
> actively maintained. In fact, it's now only a sub-project in Hive.
> Therefore, here we propose a complete connector set for Hive tables, not via
> HCatalog, but via direct Hive interface. HCatalog connector will be
> deprecated.
> Please note that connector on Hive metadata is already covered in other
> JIRAs, as {{HiveExternalCatalog}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10729) Create a Hive connector for Hive data access in Flink
[
https://issues.apache.org/jira/browse/FLINK-10729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673382#comment-16673382
]
Xuefu Zhang commented on FLINK-10729:
-
@li Thanks for your interest. Yes, metadata integration is in progress and
tracked by FLINK-10744. This one covers data aspect. We can only declare
integration is is completed after the two are in place. Your contribution is
certainly welcome.
BTW, I'm currently coming up with a rough design and will be shared soon. I can
image that a prototype is needed.
> Create a Hive connector for Hive data access in Flink
> -
>
> Key: FLINK-10729
> URL: https://issues.apache.org/jira/browse/FLINK-10729
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Affects Versions: 1.6.2
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
>Priority: Major
>
> As part of Flink-Hive integration effort, it's important for Flink to access
> (read/write) Hive data, which is the responsibility of Hive connector. While
> there is a HCatalog data connector in the code base, it's not complete (i.e.
> missing all connector related classes such as validators, etc.). Further,
> HCatalog interface has many limitations such as accessing a subset of Hive
> data, supporting a subset of Hive data types, etc. In addition, it's not
> actively maintained. In fact, it's now only a sub-project in Hive.
> Therefore, here we propose a complete connector set for Hive tables, not via
> HCatalog, but via direct Hive interface. HCatalog connector will be
> deprecated.
> Please note that connector on Hive metadata is already covered in other
> JIRAs, as {{HiveExternalCatalog}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10729) Create a Hive connector for Hive data access in Flink
[
https://issues.apache.org/jira/browse/FLINK-10729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16674863#comment-16674863
]
Fabian Hueske commented on FLINK-10729:
---
Sure, I'd be happy to have a look at the design document once it is ready :-)
> Create a Hive connector for Hive data access in Flink
> -
>
> Key: FLINK-10729
> URL: https://issues.apache.org/jira/browse/FLINK-10729
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Affects Versions: 1.6.2
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
>Priority: Major
>
> As part of Flink-Hive integration effort, it's important for Flink to access
> (read/write) Hive data, which is the responsibility of Hive connector. While
> there is a HCatalog data connector in the code base, it's not complete (i.e.
> missing all connector related classes such as validators, etc.). Further,
> HCatalog interface has many limitations such as accessing a subset of Hive
> data, supporting a subset of Hive data types, etc. In addition, it's not
> actively maintained. In fact, it's now only a sub-project in Hive.
> Therefore, here we propose a complete connector set for Hive tables, not via
> HCatalog, but via direct Hive interface. HCatalog connector will be
> deprecated.
> Please note that connector on Hive metadata is already covered in other
> JIRAs, as {{HiveExternalCatalog}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10729) Create a Hive connector for Hive data access in Flink
[
https://issues.apache.org/jira/browse/FLINK-10729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16680204#comment-16680204
]
Bowen Li commented on FLINK-10729:
--
Glad to see the community has this much interest and help that we can leverage
for this work.
I look forward to your design doc, Xuefu
> Create a Hive connector for Hive data access in Flink
> -
>
> Key: FLINK-10729
> URL: https://issues.apache.org/jira/browse/FLINK-10729
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Affects Versions: 1.6.2
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
>Priority: Major
>
> As part of Flink-Hive integration effort, it's important for Flink to access
> (read/write) Hive data, which is the responsibility of Hive connector. While
> there is a HCatalog data connector in the code base, it's not complete (i.e.
> missing all connector related classes such as validators, etc.). Further,
> HCatalog interface has many limitations such as accessing a subset of Hive
> data, supporting a subset of Hive data types, etc. In addition, it's not
> actively maintained. In fact, it's now only a sub-project in Hive.
> Therefore, here we propose a complete connector set for Hive tables, not via
> HCatalog, but via direct Hive interface. HCatalog connector will be
> deprecated.
> Please note that connector on Hive metadata is already covered in other
> JIRAs, as {{HiveExternalCatalog}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10729) Create a Hive connector for Hive data access in Flink
[
https://issues.apache.org/jira/browse/FLINK-10729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064584#comment-17064584
]
Jingsong Lee commented on FLINK-10729:
--
Close, Thanks for the great work [~xuefuz]
> Create a Hive connector for Hive data access in Flink
> -
>
> Key: FLINK-10729
> URL: https://issues.apache.org/jira/browse/FLINK-10729
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / Hive, Table SQL / Ecosystem
>Affects Versions: 1.6.2
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
>Priority: Major
>
> As part of Flink-Hive integration effort, it's important for Flink to access
> (read/write) Hive data, which is the responsibility of Hive connector. While
> there is a HCatalog data connector in the code base, it's not complete (i.e.
> missing all connector related classes such as validators, etc.). Further,
> HCatalog interface has many limitations such as accessing a subset of Hive
> data, supporting a subset of Hive data types, etc. In addition, it's not
> actively maintained. In fact, it's now only a sub-project in Hive.
> Therefore, here we propose a complete connector set for Hive tables, not via
> HCatalog, but via direct Hive interface. HCatalog connector will be
> deprecated.
> Please note that connector on Hive metadata is already covered in other
> JIRAs, as {{HiveExternalCatalog}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
