Re: HiveIO
Hi Tim, I pushed the HBaseIO on github, I will do the same later today for HiveIO. I will let you know. Thanks ! Regards JB On 12/15/2016 02:39 AM, Tim Taschke wrote: Great to see that there is progress on this! Like written on the user mailing list, I would also be interested in contributing to this. On Thu, Dec 15, 2016 at 4:44 AM, Ismaël Mejía <ieme...@gmail.com> wrote: For ref, I just created a JIRA so people can track the progress/contribute to the progress of HiveIO. https://issues.apache.org/jira/browse/BEAM-1158 On Wed, Dec 7, 2016 at 5:39 PM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Yes that's the first idea ;) Regards JB On Dec 7, 2016, 17:27, at 17:27, Vinoth Chandar <vin...@uber.com> wrote: Interesting. So all the planning & execution is done by Hive, and Beam will process the results of the query? On Wed, Dec 7, 2016 at 8:24 AM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi The HiveIO will directly use the native API and HiveQL. That's the plan on which we are working right now. Regards JB On Dec 7, 2016, 17:18, at 17:18, Vinoth Chandar <vin...@uber.com> wrote: Hi, I am not looking for a way to actually execute the query on Hive. I would like to do something similar to Spark SQL/HiveContext, but with Beam. Just have a HiveIO that reads metadata from Hive metastore, and then later use a Spark runner to execute the query. So, HiveJDBC is not an option I would like to pursue. Thanks for the pointer, though! And does the HiveIO that is being planned, work similarly as above? Thanks Vinoth On Tue, Dec 6, 2016 at 4:55 AM, Ismaël Mejía <ieme...@gmail.com> wrote: Hello, If you really need to read/write via Hive, remember that you can use the Hive Jdbc driver, and achieve this with Beam using the JdbcIO (this is probably less efficient for the streaming case but still a valid solution). Ismaël On Tue, Dec 6, 2016 at 12:04 PM, Vinoth Chandar <vin...@uber.com> wrote: Great. Thanks! Thanks, Vinoth On Dec 6, 2016, at 2:06 AM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: Hi, Ismaël and I started HiveIO. I have several IOs ready to propose as PR, but, in order to limit the number of open PRs, I would like to merge the pending ones. I will let you know when the branches/PRs will be available. Regards JB On 12/05/2016 11:40 PM, Vinoth Chandar wrote: Hi guys, Saw a post around HiveIO on the users list with a PR followup. I am interested in this too and can pitch in on developement and testing.. Who & where is this work happening? Thanks VInoth -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com
Re: HiveIO
For ref, I just created a JIRA so people can track the progress/contribute to the progress of HiveIO. https://issues.apache.org/jira/browse/BEAM-1158 On Wed, Dec 7, 2016 at 5:39 PM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > Yes that's the first idea ;) > > Regards > JB > > On Dec 7, 2016, 17:27, at 17:27, Vinoth Chandar <vin...@uber.com> wrote: > >Interesting. So all the planning & execution is done by Hive, and Beam > >will > >process the results of the query? > > > >On Wed, Dec 7, 2016 at 8:24 AM, Jean-Baptiste Onofré <j...@nanthrax.net> > >wrote: > > > >> Hi > >> > >> The HiveIO will directly use the native API and HiveQL. That's the > >plan on > >> which we are working right now. > >> > >> Regards > >> JB > >> > >> On Dec 7, 2016, 17:18, at 17:18, Vinoth Chandar <vin...@uber.com> > >wrote: > >> >Hi, > >> > > >> >I am not looking for a way to actually execute the query on Hive. I > >> >would > >> >like to do something similar to Spark SQL/HiveContext, but with > >Beam. > >> >Just > >> >have a HiveIO that reads metadata from Hive metastore, and then > >later > >> >use a > >> >Spark runner to execute the query. So, HiveJDBC is not an option I > >> >would > >> >like to pursue. Thanks for the pointer, though! > >> > > >> >And does the HiveIO that is being planned, work similarly as above? > >> > > >> > > >> >Thanks > >> >Vinoth > >> > > >> > > >> > > >> >On Tue, Dec 6, 2016 at 4:55 AM, Ismaël Mejía <ieme...@gmail.com> > >wrote: > >> > > >> >> Hello, > >> >> > >> >> If you really need to read/write via Hive, remember that you can > >use > >> >the > >> >> Hive Jdbc driver, and achieve this with Beam using the JdbcIO > >(this > >> >is > >> >> probably less efficient for the streaming case but still a valid > >> >solution). > >> >> > >> >> Ismaël > >> >> > >> >> > >> >> On Tue, Dec 6, 2016 at 12:04 PM, Vinoth Chandar <vin...@uber.com> > >> >wrote: > >> >> > >> >> > Great. Thanks! > >> >> > > >> >> > Thanks, > >> >> > Vinoth > >> >> > > >> >> > > On Dec 6, 2016, at 2:06 AM, Jean-Baptiste Onofré > >> ><j...@nanthrax.net> > >> >> > wrote: > >> >> > > > >> >> > > Hi, > >> >> > > > >> >> > > Ismaël and I started HiveIO. > >> >> > > > >> >> > > I have several IOs ready to propose as PR, but, in order to > >limit > >> >the > >> >> > number of open PRs, I would like to merge the pending ones. > >> >> > > > >> >> > > I will let you know when the branches/PRs will be available. > >> >> > > > >> >> > > Regards > >> >> > > JB > >> >> > > > >> >> > >> On 12/05/2016 11:40 PM, Vinoth Chandar wrote: > >> >> > >> Hi guys, > >> >> > >> > >> >> > >> Saw a post around HiveIO on the users list with a PR > >followup. I > >> >am > >> >> > >> interested in this too and can pitch in on developement and > >> >testing.. > >> >> > >> > >> >> > >> Who & where is this work happening? > >> >> > >> > >> >> > >> Thanks > >> >> > >> VInoth > >> >> > >> > >> >> > > > >> >> > > -- > >> >> > > Jean-Baptiste Onofré > >> >> > > jbono...@apache.org > >> >> > > http://blog.nanthrax.net > >> >> > > Talend - http://www.talend.com > >> >> > > >> >> > >> >
Re: HiveIO
Yes that's the first idea ;) Regards JB On Dec 7, 2016, 17:27, at 17:27, Vinoth Chandar <vin...@uber.com> wrote: >Interesting. So all the planning & execution is done by Hive, and Beam >will >process the results of the query? > >On Wed, Dec 7, 2016 at 8:24 AM, Jean-Baptiste Onofré <j...@nanthrax.net> >wrote: > >> Hi >> >> The HiveIO will directly use the native API and HiveQL. That's the >plan on >> which we are working right now. >> >> Regards >> JB >> >> On Dec 7, 2016, 17:18, at 17:18, Vinoth Chandar <vin...@uber.com> >wrote: >> >Hi, >> > >> >I am not looking for a way to actually execute the query on Hive. I >> >would >> >like to do something similar to Spark SQL/HiveContext, but with >Beam. >> >Just >> >have a HiveIO that reads metadata from Hive metastore, and then >later >> >use a >> >Spark runner to execute the query. So, HiveJDBC is not an option I >> >would >> >like to pursue. Thanks for the pointer, though! >> > >> >And does the HiveIO that is being planned, work similarly as above? >> > >> > >> >Thanks >> >Vinoth >> > >> > >> > >> >On Tue, Dec 6, 2016 at 4:55 AM, Ismaël Mejía <ieme...@gmail.com> >wrote: >> > >> >> Hello, >> >> >> >> If you really need to read/write via Hive, remember that you can >use >> >the >> >> Hive Jdbc driver, and achieve this with Beam using the JdbcIO >(this >> >is >> >> probably less efficient for the streaming case but still a valid >> >solution). >> >> >> >> Ismaël >> >> >> >> >> >> On Tue, Dec 6, 2016 at 12:04 PM, Vinoth Chandar <vin...@uber.com> >> >wrote: >> >> >> >> > Great. Thanks! >> >> > >> >> > Thanks, >> >> > Vinoth >> >> > >> >> > > On Dec 6, 2016, at 2:06 AM, Jean-Baptiste Onofré >> ><j...@nanthrax.net> >> >> > wrote: >> >> > > >> >> > > Hi, >> >> > > >> >> > > Ismaël and I started HiveIO. >> >> > > >> >> > > I have several IOs ready to propose as PR, but, in order to >limit >> >the >> >> > number of open PRs, I would like to merge the pending ones. >> >> > > >> >> > > I will let you know when the branches/PRs will be available. >> >> > > >> >> > > Regards >> >> > > JB >> >> > > >> >> > >> On 12/05/2016 11:40 PM, Vinoth Chandar wrote: >> >> > >> Hi guys, >> >> > >> >> >> > >> Saw a post around HiveIO on the users list with a PR >followup. I >> >am >> >> > >> interested in this too and can pitch in on developement and >> >testing.. >> >> > >> >> >> > >> Who & where is this work happening? >> >> > >> >> >> > >> Thanks >> >> > >> VInoth >> >> > >> >> >> > > >> >> > > -- >> >> > > Jean-Baptiste Onofré >> >> > > jbono...@apache.org >> >> > > http://blog.nanthrax.net >> >> > > Talend - http://www.talend.com >> >> > >> >> >>
Re: HiveIO
Interesting. So all the planning & execution is done by Hive, and Beam will process the results of the query? On Wed, Dec 7, 2016 at 8:24 AM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > Hi > > The HiveIO will directly use the native API and HiveQL. That's the plan on > which we are working right now. > > Regards > JB > > On Dec 7, 2016, 17:18, at 17:18, Vinoth Chandar <vin...@uber.com> wrote: > >Hi, > > > >I am not looking for a way to actually execute the query on Hive. I > >would > >like to do something similar to Spark SQL/HiveContext, but with Beam. > >Just > >have a HiveIO that reads metadata from Hive metastore, and then later > >use a > >Spark runner to execute the query. So, HiveJDBC is not an option I > >would > >like to pursue. Thanks for the pointer, though! > > > >And does the HiveIO that is being planned, work similarly as above? > > > > > >Thanks > >Vinoth > > > > > > > >On Tue, Dec 6, 2016 at 4:55 AM, Ismaël Mejía <ieme...@gmail.com> wrote: > > > >> Hello, > >> > >> If you really need to read/write via Hive, remember that you can use > >the > >> Hive Jdbc driver, and achieve this with Beam using the JdbcIO (this > >is > >> probably less efficient for the streaming case but still a valid > >solution). > >> > >> Ismaël > >> > >> > >> On Tue, Dec 6, 2016 at 12:04 PM, Vinoth Chandar <vin...@uber.com> > >wrote: > >> > >> > Great. Thanks! > >> > > >> > Thanks, > >> > Vinoth > >> > > >> > > On Dec 6, 2016, at 2:06 AM, Jean-Baptiste Onofré > ><j...@nanthrax.net> > >> > wrote: > >> > > > >> > > Hi, > >> > > > >> > > Ismaël and I started HiveIO. > >> > > > >> > > I have several IOs ready to propose as PR, but, in order to limit > >the > >> > number of open PRs, I would like to merge the pending ones. > >> > > > >> > > I will let you know when the branches/PRs will be available. > >> > > > >> > > Regards > >> > > JB > >> > > > >> > >> On 12/05/2016 11:40 PM, Vinoth Chandar wrote: > >> > >> Hi guys, > >> > >> > >> > >> Saw a post around HiveIO on the users list with a PR followup. I > >am > >> > >> interested in this too and can pitch in on developement and > >testing.. > >> > >> > >> > >> Who & where is this work happening? > >> > >> > >> > >> Thanks > >> > >> VInoth > >> > >> > >> > > > >> > > -- > >> > > Jean-Baptiste Onofré > >> > > jbono...@apache.org > >> > > http://blog.nanthrax.net > >> > > Talend - http://www.talend.com > >> > > >> >
Re: HiveIO
Hi The HiveIO will directly use the native API and HiveQL. That's the plan on which we are working right now. Regards JB On Dec 7, 2016, 17:18, at 17:18, Vinoth Chandar <vin...@uber.com> wrote: >Hi, > >I am not looking for a way to actually execute the query on Hive. I >would >like to do something similar to Spark SQL/HiveContext, but with Beam. >Just >have a HiveIO that reads metadata from Hive metastore, and then later >use a >Spark runner to execute the query. So, HiveJDBC is not an option I >would >like to pursue. Thanks for the pointer, though! > >And does the HiveIO that is being planned, work similarly as above? > > >Thanks >Vinoth > > > >On Tue, Dec 6, 2016 at 4:55 AM, Ismaël Mejía <ieme...@gmail.com> wrote: > >> Hello, >> >> If you really need to read/write via Hive, remember that you can use >the >> Hive Jdbc driver, and achieve this with Beam using the JdbcIO (this >is >> probably less efficient for the streaming case but still a valid >solution). >> >> Ismaël >> >> >> On Tue, Dec 6, 2016 at 12:04 PM, Vinoth Chandar <vin...@uber.com> >wrote: >> >> > Great. Thanks! >> > >> > Thanks, >> > Vinoth >> > >> > > On Dec 6, 2016, at 2:06 AM, Jean-Baptiste Onofré ><j...@nanthrax.net> >> > wrote: >> > > >> > > Hi, >> > > >> > > Ismaël and I started HiveIO. >> > > >> > > I have several IOs ready to propose as PR, but, in order to limit >the >> > number of open PRs, I would like to merge the pending ones. >> > > >> > > I will let you know when the branches/PRs will be available. >> > > >> > > Regards >> > > JB >> > > >> > >> On 12/05/2016 11:40 PM, Vinoth Chandar wrote: >> > >> Hi guys, >> > >> >> > >> Saw a post around HiveIO on the users list with a PR followup. I >am >> > >> interested in this too and can pitch in on developement and >testing.. >> > >> >> > >> Who & where is this work happening? >> > >> >> > >> Thanks >> > >> VInoth >> > >> >> > > >> > > -- >> > > Jean-Baptiste Onofré >> > > jbono...@apache.org >> > > http://blog.nanthrax.net >> > > Talend - http://www.talend.com >> > >>
Re: HiveIO
Hi, I am not looking for a way to actually execute the query on Hive. I would like to do something similar to Spark SQL/HiveContext, but with Beam. Just have a HiveIO that reads metadata from Hive metastore, and then later use a Spark runner to execute the query. So, HiveJDBC is not an option I would like to pursue. Thanks for the pointer, though! And does the HiveIO that is being planned, work similarly as above? Thanks Vinoth On Tue, Dec 6, 2016 at 4:55 AM, Ismaël Mejía <ieme...@gmail.com> wrote: > Hello, > > If you really need to read/write via Hive, remember that you can use the > Hive Jdbc driver, and achieve this with Beam using the JdbcIO (this is > probably less efficient for the streaming case but still a valid solution). > > Ismaël > > > On Tue, Dec 6, 2016 at 12:04 PM, Vinoth Chandar <vin...@uber.com> wrote: > > > Great. Thanks! > > > > Thanks, > > Vinoth > > > > > On Dec 6, 2016, at 2:06 AM, Jean-Baptiste Onofré <j...@nanthrax.net> > > wrote: > > > > > > Hi, > > > > > > Ismaël and I started HiveIO. > > > > > > I have several IOs ready to propose as PR, but, in order to limit the > > number of open PRs, I would like to merge the pending ones. > > > > > > I will let you know when the branches/PRs will be available. > > > > > > Regards > > > JB > > > > > >> On 12/05/2016 11:40 PM, Vinoth Chandar wrote: > > >> Hi guys, > > >> > > >> Saw a post around HiveIO on the users list with a PR followup. I am > > >> interested in this too and can pitch in on developement and testing.. > > >> > > >> Who & where is this work happening? > > >> > > >> Thanks > > >> VInoth > > >> > > > > > > -- > > > Jean-Baptiste Onofré > > > jbono...@apache.org > > > http://blog.nanthrax.net > > > Talend - http://www.talend.com > > >
Re: HiveIO
Hello, If you really need to read/write via Hive, remember that you can use the Hive Jdbc driver, and achieve this with Beam using the JdbcIO (this is probably less efficient for the streaming case but still a valid solution). Ismaël On Tue, Dec 6, 2016 at 12:04 PM, Vinoth Chandar <vin...@uber.com> wrote: > Great. Thanks! > > Thanks, > Vinoth > > > On Dec 6, 2016, at 2:06 AM, Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > > > > Hi, > > > > Ismaël and I started HiveIO. > > > > I have several IOs ready to propose as PR, but, in order to limit the > number of open PRs, I would like to merge the pending ones. > > > > I will let you know when the branches/PRs will be available. > > > > Regards > > JB > > > >> On 12/05/2016 11:40 PM, Vinoth Chandar wrote: > >> Hi guys, > >> > >> Saw a post around HiveIO on the users list with a PR followup. I am > >> interested in this too and can pitch in on developement and testing.. > >> > >> Who & where is this work happening? > >> > >> Thanks > >> VInoth > >> > > > > -- > > Jean-Baptiste Onofré > > jbono...@apache.org > > http://blog.nanthrax.net > > Talend - http://www.talend.com >
Re: HiveIO
Great. Thanks! Thanks, Vinoth > On Dec 6, 2016, at 2:06 AM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > > Hi, > > Ismaël and I started HiveIO. > > I have several IOs ready to propose as PR, but, in order to limit the number > of open PRs, I would like to merge the pending ones. > > I will let you know when the branches/PRs will be available. > > Regards > JB > >> On 12/05/2016 11:40 PM, Vinoth Chandar wrote: >> Hi guys, >> >> Saw a post around HiveIO on the users list with a PR followup. I am >> interested in this too and can pitch in on developement and testing.. >> >> Who & where is this work happening? >> >> Thanks >> VInoth >> > > -- > Jean-Baptiste Onofré > jbono...@apache.org > http://blog.nanthrax.net > Talend - http://www.talend.com
Re: HiveIO
Hi, Ismaël and I started HiveIO. I have several IOs ready to propose as PR, but, in order to limit the number of open PRs, I would like to merge the pending ones. I will let you know when the branches/PRs will be available. Regards JB On 12/05/2016 11:40 PM, Vinoth Chandar wrote: Hi guys, Saw a post around HiveIO on the users list with a PR followup. I am interested in this too and can pitch in on developement and testing.. Who & where is this work happening? Thanks VInoth -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com
