Re: [Architecture] RFC: Could we support Hive executions on top of Hazelcast?
Srinath, related to this topic .. (via Paul) GridGain just offered to donate their in-memory computing stuff to Apache: https://wiki.apache.org/incubator/SilkProposal Can you take a look and see? Maybe we should actively join that effort as well .. Sanjiva. On Fri, Sep 5, 2014 at 2:27 PM, Srinath Perera srin...@wso2.com wrote: Hi All, Hazelcast now also have a MapReduce implementation. May be we can change hive so it can also submit jobs to Hazelcast in addition to MapReduce. Then users can load lot of data to hazelcast and process them with HiveSQL. 1. It will enable users to also do in memory computing using Hive SQL 2. It will enable ad-hoc queries that return fast results over a large dataset using HiveSQL IMO it will be a pretty tool In-memory computing scenario. WDYT? --Srinath p.s. One concern is when we support Spark, it will also have much similar support via .cache() in RDDs. So this might be a redundant feature at the time. -- Director, Research, WSO2 Inc. Visiting Faculty, University of Moratuwa Member, Apache Software Foundation Research Scientist, Lanka Software Foundation Blog: http://srinathsview.blogspot.com twitter:@srinath_perera Site: http://people.apache.org/~hemapani/ Photos: http://www.flickr.com/photos/hemapani/ Phone: 0772360902 -- Sanjiva Weerawarana, Ph.D. Founder, Chairman CEO; WSO2, Inc.; http://wso2.com/ email: sanj...@wso2.com; office: (+1 650 745 4499 | +94 11 214 5345) x5700; cell: +94 77 787 6880 | +1 408 466 5099; voip: +1 650 265 8311 blog: http://sanjiva.weerawarana.org/; twitter: @sanjiva Lean . Enterprise . Middleware ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] RFC: Could we support Hive executions on top of Hazelcast?
Hi Srinath, another minor point that came to me: Google announced recently the successor of MapReduce, called Google Cloud Dataflow ( http://googlecloudplatform.blogspot.in/2014/06/reimagining-developer-productivity-and-data-analytics-in-the-cloud-news-from-google-io.html). Because you are mentioning MapReduce sometimes you may get questions about Google Cloud Dataflow :-) Best regards, Frank 2014-09-05 12:07 GMT+02:00 Srinath Perera srin...@wso2.com: Hi Paul, I have been playing with it a bit, but have lot of figuring out to do. I think we should do Spark anyway in near future. --Srinath On Fri, Sep 5, 2014 at 2:40 PM, Paul Fremantle p...@wso2.com wrote: What about our idea of using the Siddhi language for the long-running map reduce side as well? Paul On 5 September 2014 09:57, Srinath Perera srin...@wso2.com wrote: Hi All, Hazelcast now also have a MapReduce implementation. May be we can change hive so it can also submit jobs to Hazelcast in addition to MapReduce. Then users can load lot of data to hazelcast and process them with HiveSQL. 1. It will enable users to also do in memory computing using Hive SQL 2. It will enable ad-hoc queries that return fast results over a large dataset using HiveSQL IMO it will be a pretty tool In-memory computing scenario. WDYT? --Srinath p.s. One concern is when we support Spark, it will also have much similar support via .cache() in RDDs. So this might be a redundant feature at the time. -- Director, Research, WSO2 Inc. Visiting Faculty, University of Moratuwa Member, Apache Software Foundation Research Scientist, Lanka Software Foundation Blog: http://srinathsview.blogspot.com twitter:@srinath_perera Site: http://people.apache.org/~hemapani/ Photos: http://www.flickr.com/photos/hemapani/ Phone: 0772360902 -- Paul Fremantle CTO and Co-Founder, WSO2 OASIS WS-RX TC Co-chair, Apache Member UK: +44 207 096 0336 blog: http://pzf.fremantle.org twitter.com/pzfreo p...@wso2.com wso2.com Lean Enterprise Middleware Disclaimer: This communication may contain privileged or other confidential information and is intended exclusively for the addressee/s. If you are not the intended recipient/s, or believe that you may have received this communication in error, please reply to the sender indicating that fact and delete the copy you received and in addition, you should not print, copy, retransmit, disseminate, or otherwise use the information contained in this communication. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions. -- Director, Research, WSO2 Inc. Visiting Faculty, University of Moratuwa Member, Apache Software Foundation Research Scientist, Lanka Software Foundation Blog: http://srinathsview.blogspot.com twitter:@srinath_perera Site: http://people.apache.org/~hemapani/ Photos: http://www.flickr.com/photos/hemapani/ Phone: 0772360902 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] RFC: Could we support Hive executions on top of Hazelcast?
Sorry - my comment is on the wrong thread :-( Best regards, Frank 2014-09-07 15:26 GMT+02:00 Frank Leymann fr...@wso2.com: Hi Srinath, another minor point that came to me: Google announced recently the successor of MapReduce, called Google Cloud Dataflow ( http://googlecloudplatform.blogspot.in/2014/06/reimagining-developer-productivity-and-data-analytics-in-the-cloud-news-from-google-io.html). Because you are mentioning MapReduce sometimes you may get questions about Google Cloud Dataflow :-) Best regards, Frank 2014-09-05 12:07 GMT+02:00 Srinath Perera srin...@wso2.com: Hi Paul, I have been playing with it a bit, but have lot of figuring out to do. I think we should do Spark anyway in near future. --Srinath On Fri, Sep 5, 2014 at 2:40 PM, Paul Fremantle p...@wso2.com wrote: What about our idea of using the Siddhi language for the long-running map reduce side as well? Paul On 5 September 2014 09:57, Srinath Perera srin...@wso2.com wrote: Hi All, Hazelcast now also have a MapReduce implementation. May be we can change hive so it can also submit jobs to Hazelcast in addition to MapReduce. Then users can load lot of data to hazelcast and process them with HiveSQL. 1. It will enable users to also do in memory computing using Hive SQL 2. It will enable ad-hoc queries that return fast results over a large dataset using HiveSQL IMO it will be a pretty tool In-memory computing scenario. WDYT? --Srinath p.s. One concern is when we support Spark, it will also have much similar support via .cache() in RDDs. So this might be a redundant feature at the time. -- Director, Research, WSO2 Inc. Visiting Faculty, University of Moratuwa Member, Apache Software Foundation Research Scientist, Lanka Software Foundation Blog: http://srinathsview.blogspot.com twitter:@srinath_perera Site: http://people.apache.org/~hemapani/ Photos: http://www.flickr.com/photos/hemapani/ Phone: 0772360902 -- Paul Fremantle CTO and Co-Founder, WSO2 OASIS WS-RX TC Co-chair, Apache Member UK: +44 207 096 0336 blog: http://pzf.fremantle.org twitter.com/pzfreo p...@wso2.com wso2.com Lean Enterprise Middleware Disclaimer: This communication may contain privileged or other confidential information and is intended exclusively for the addressee/s. If you are not the intended recipient/s, or believe that you may have received this communication in error, please reply to the sender indicating that fact and delete the copy you received and in addition, you should not print, copy, retransmit, disseminate, or otherwise use the information contained in this communication. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions. -- Director, Research, WSO2 Inc. Visiting Faculty, University of Moratuwa Member, Apache Software Foundation Research Scientist, Lanka Software Foundation Blog: http://srinathsview.blogspot.com twitter:@srinath_perera Site: http://people.apache.org/~hemapani/ Photos: http://www.flickr.com/photos/hemapani/ Phone: 0772360902 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
[Architecture] RFC: Could we support Hive executions on top of Hazelcast?
Hi All, Hazelcast now also have a MapReduce implementation. May be we can change hive so it can also submit jobs to Hazelcast in addition to MapReduce. Then users can load lot of data to hazelcast and process them with HiveSQL. 1. It will enable users to also do in memory computing using Hive SQL 2. It will enable ad-hoc queries that return fast results over a large dataset using HiveSQL IMO it will be a pretty tool In-memory computing scenario. WDYT? --Srinath p.s. One concern is when we support Spark, it will also have much similar support via .cache() in RDDs. So this might be a redundant feature at the time. -- Director, Research, WSO2 Inc. Visiting Faculty, University of Moratuwa Member, Apache Software Foundation Research Scientist, Lanka Software Foundation Blog: http://srinathsview.blogspot.com twitter:@srinath_perera Site: http://people.apache.org/~hemapani/ Photos: http://www.flickr.com/photos/hemapani/ Phone: 0772360902 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] RFC: Could we support Hive executions on top of Hazelcast?
What about our idea of using the Siddhi language for the long-running map reduce side as well? Paul On 5 September 2014 09:57, Srinath Perera srin...@wso2.com wrote: Hi All, Hazelcast now also have a MapReduce implementation. May be we can change hive so it can also submit jobs to Hazelcast in addition to MapReduce. Then users can load lot of data to hazelcast and process them with HiveSQL. 1. It will enable users to also do in memory computing using Hive SQL 2. It will enable ad-hoc queries that return fast results over a large dataset using HiveSQL IMO it will be a pretty tool In-memory computing scenario. WDYT? --Srinath p.s. One concern is when we support Spark, it will also have much similar support via .cache() in RDDs. So this might be a redundant feature at the time. -- Director, Research, WSO2 Inc. Visiting Faculty, University of Moratuwa Member, Apache Software Foundation Research Scientist, Lanka Software Foundation Blog: http://srinathsview.blogspot.com twitter:@srinath_perera Site: http://people.apache.org/~hemapani/ Photos: http://www.flickr.com/photos/hemapani/ Phone: 0772360902 -- Paul Fremantle CTO and Co-Founder, WSO2 OASIS WS-RX TC Co-chair, Apache Member UK: +44 207 096 0336 blog: http://pzf.fremantle.org twitter.com/pzfreo p...@wso2.com wso2.com Lean Enterprise Middleware Disclaimer: This communication may contain privileged or other confidential information and is intended exclusively for the addressee/s. If you are not the intended recipient/s, or believe that you may have received this communication in error, please reply to the sender indicating that fact and delete the copy you received and in addition, you should not print, copy, retransmit, disseminate, or otherwise use the information contained in this communication. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions. ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] RFC: Could we support Hive executions on top of Hazelcast?
Hi Paul, I have been playing with it a bit, but have lot of figuring out to do. I think we should do Spark anyway in near future. --Srinath On Fri, Sep 5, 2014 at 2:40 PM, Paul Fremantle p...@wso2.com wrote: What about our idea of using the Siddhi language for the long-running map reduce side as well? Paul On 5 September 2014 09:57, Srinath Perera srin...@wso2.com wrote: Hi All, Hazelcast now also have a MapReduce implementation. May be we can change hive so it can also submit jobs to Hazelcast in addition to MapReduce. Then users can load lot of data to hazelcast and process them with HiveSQL. 1. It will enable users to also do in memory computing using Hive SQL 2. It will enable ad-hoc queries that return fast results over a large dataset using HiveSQL IMO it will be a pretty tool In-memory computing scenario. WDYT? --Srinath p.s. One concern is when we support Spark, it will also have much similar support via .cache() in RDDs. So this might be a redundant feature at the time. -- Director, Research, WSO2 Inc. Visiting Faculty, University of Moratuwa Member, Apache Software Foundation Research Scientist, Lanka Software Foundation Blog: http://srinathsview.blogspot.com twitter:@srinath_perera Site: http://people.apache.org/~hemapani/ Photos: http://www.flickr.com/photos/hemapani/ Phone: 0772360902 -- Paul Fremantle CTO and Co-Founder, WSO2 OASIS WS-RX TC Co-chair, Apache Member UK: +44 207 096 0336 blog: http://pzf.fremantle.org twitter.com/pzfreo p...@wso2.com wso2.com Lean Enterprise Middleware Disclaimer: This communication may contain privileged or other confidential information and is intended exclusively for the addressee/s. If you are not the intended recipient/s, or believe that you may have received this communication in error, please reply to the sender indicating that fact and delete the copy you received and in addition, you should not print, copy, retransmit, disseminate, or otherwise use the information contained in this communication. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions. -- Director, Research, WSO2 Inc. Visiting Faculty, University of Moratuwa Member, Apache Software Foundation Research Scientist, Lanka Software Foundation Blog: http://srinathsview.blogspot.com twitter:@srinath_perera Site: http://people.apache.org/~hemapani/ Photos: http://www.flickr.com/photos/hemapani/ Phone: 0772360902 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
