Re: [VOTE] Accept Concerted into the Apache Incubator
Resending with correct subject. Hi All, This vote passed with the following result: +1(binding):Henry Saputra, Julian Hyde, Chris Nauroth, Lars Hofhansl, Roman Shaposhnik, Jake Farrell, Amareshwari Sriramdasu, Alan D. Cabrera, Bertrand Delacretaz +0(binding):Sergio Fernandez +1(non binding):Amol Kekre, Pavel Stehule, Ayrton Gomesz, Luke Han, Timothy Chen -1: None Thanks everyone for participating and voting. We are excited to be part of ASF Incubator. Regards, Atri On Tue, Oct 13, 2015 at 2:19 PM, Atri Sharma wrote: > Hi All, > > This vote passed with the following result: > > +1(binding):Henry Saputra, Julian Hyde, Chris Nauroth, Lars Hofhansl, > Roman Shaposhnik, Jake Farrell, Amareshwari Sriramdasu, Alan D. Cabrera, > Bertrand Delacretaz > +0(binding):Sergio Fernandez > +1(non binding):Amol Kekre, Pavel Stehule, Ayrton Gomesz, Luke Han, > Timothy Chen > -1: None > > Thanks everyone for participating and voting. > > We are excited to be part of ASF Incubator. > > Regards, > Atri > > On Mon, Oct 12, 2015 at 12:30 PM, Bertrand Delacretaz < > bdelacre...@apache.org> wrote: > >> On Fri, Oct 9, 2015 at 5:55 PM, Atri Sharma wrote: >> > ...Following the discussion about Concerted I would like to call a vote >> for >> > accepting Concerted as a new incubator project... >> >> +1 >> >> -Bertrand >> >> - >> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org >> For additional commands, e-mail: general-h...@incubator.apache.org >> >> >
Re: [VOTE] Accept Concerted into the Apache Incubator
Hi All, This vote passed with the following result: +1(binding):Henry Saputra, Julian Hyde, Chris Nauroth, Lars Hofhansl, Roman Shaposhnik, Jake Farrell, Amareshwari Sriramdasu, Alan D. Cabrera, Bertrand Delacretaz +0(binding):Sergio Fernandez +1(non binding):Amol Kekre, Pavel Stehule, Ayrton Gomesz, Luke Han, Timothy Chen -1: None Thanks everyone for participating and voting. We are excited to be part of ASF Incubator. Regards, Atri On Mon, Oct 12, 2015 at 12:30 PM, Bertrand Delacretaz < bdelacre...@apache.org> wrote: > On Fri, Oct 9, 2015 at 5:55 PM, Atri Sharma wrote: > > ...Following the discussion about Concerted I would like to call a vote > for > > accepting Concerted as a new incubator project... > > +1 > > -Bertrand > > - > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > >
Re: [VOTE] Accept Concerted into the Apache Incubator
On Fri, Oct 9, 2015 at 5:55 PM, Atri Sharma wrote: > ...Following the discussion about Concerted I would like to call a vote for > accepting Concerted as a new incubator project... +1 -Bertrand - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] Accept Concerted into the Apache Incubator
+0 (binding) As we discussed in a previous thread, I have concerns about the current status of the project, both regarding implementation and community, and if incubating it now is the best for the goals of the project. On Fri, Oct 9, 2015 at 5:55 PM, Atri Sharma wrote: > Hi all, > > Following the discussion about Concerted I would like to call a vote for > accepting Concerted as a new incubator project. > > The proposal text is included below, and available on the wiki: > > https://wiki.apache.org/incubator/ConcertedProposal > > The vote is open for 72 hours: > > [ ] +1 accept Concerted in the Incubator > [ ] ±0 > [ ] -1 (please give reason) > > Regards, > > Atri > > = Abstract = > > Concerted is an in memory write less read more engine aimed to provide > extreme read performance with very high degree of concurrency and > scalability and focus on minimizing own resource footprint. > > = Proposal = > Concerted is built on the principal that a new type of workload is > dominating the scene and is now needed to be supported. These are the large > data set analytical workloads being analyzed or used on large clusters or > high power machines. Large analytical workloads depend on the ability to > query large data sets efficiently and in high concurrency while maintaining > semantics such as immediate consistency. An in memory engine designed to > support extreme read queries while providing support for aggregation > through various features (such as multidimensional representation of > tuples) will accelerate many usecases around large scale analytics. > > Concerted believes that best understanding of user application lies with > user application developer. The need for massive read scaling should be on > demand and should be flexible to the level that user can decide as to which > representation and access of data suits his/her current requirements. > Hence, Concerted is not built in a traditional client/server model. > Concerted provides users with an API which can be used to load, read, > update and delete data. User chooses which data structure has to be used > for his current requirements. All API access is covered by Concerted's > internal systems like lock manager, transaction manager and cache manager > which ensure that reads scale to high level in every API call. > > Concerted is a Do It Yourself in memory platform for making in memory > supporting engines. The use case we think of is supporting big data > warehouses like Hive, but there are endless use cases for a custom, highly > scalable in memory platform. > > The goal of this proposal is to leverage an existing code base available on > Github and licensed under the Apache License 2.0 to build a community > around the project. Currently the community consists of existing hackers of > Concerted as well as people who have been following and associated with the > project since a while as well as database experts who are excited about > building a project like this. We are hoping that entering into Apache would > help us attract more contributors as well as connect with existing big data > projects like Apache Hive, Apache HAWQ, Apache Storm, Apache Tajo, Apache > Spark, Apache Geode to leverage their community base while assisting in > their use cases with Concerted. We had a discussion with founders of Apache > Tajo and they showed interest in using Concerted for some of their use > cases. > = Background = > Relational databases were built with the cost of physical memory in mind. > The cost is no longer very relevant and physical memory is now available on > demand. Another driving factor behind Concerted is that there is a paradigm > shift with big data coming into picture. Disk IO speeds are more of a > bottleneck than ever before. Combining the read dominance of analytical > workload with the speed of in memory structures, Concerted fits the current > scene. Also, supporting OLAP workloads with in memory support for faster > read constant queries and joins will be useful. > > = Rationale = > As explained above, large analytical workloads need an in memory > lightweight engine which supports massive read concurrency, ground level > support for aggregations and analytics, extreme scalability and high read > performance, along with the engine being very light itself. Concerted aims > to solve these needs. Concerted is designed and built with three goals as > objectives: > > > Performance > To provide high performance access to data from a large number of rows, > Concerted uses efficient representation and in memory indexing of data > coupled with high performance transactions, custom transactions and > lightweight locking and lockless techniques and an intelligent locking > manager. > > Scalability > Concerted is built with extreme concurrency and scalability in mind. > > Efficiency > Concerted aims to give expected performance under vast variety of > workloads and aims to have as low footprint as possible. > > = Initial Goals = >
Re: [VOTE] Accept Concerted into the Apache Incubator
+1 non binding Tim > On Oct 11, 2015, at 4:59 PM, Luke Han wrote: > > +1 (non-binding) > > > Best Regards! > - > > Luke Han > > On Mon, Oct 12, 2015 at 4:33 AM, Alan D. Cabrera > wrote: > >> +1 - binding >> >> >> Regards, >> Alan >> >>> On Oct 9, 2015, at 8:55 AM, Atri Sharma wrote: >>> >>> Hi all, >>> >>> Following the discussion about Concerted I would like to call a vote for >>> accepting Concerted as a new incubator project. >>> >>> The proposal text is included below, and available on the wiki: >>> >>> https://wiki.apache.org/incubator/ConcertedProposal >>> >>> The vote is open for 72 hours: >>> >>> [ ] +1 accept Concerted in the Incubator >>> [ ] ±0 >>> [ ] -1 (please give reason) >>> >>> Regards, >>> >>> Atri >>> >>> = Abstract = >>> >>> Concerted is an in memory write less read more engine aimed to provide >>> extreme read performance with very high degree of concurrency and >>> scalability and focus on minimizing own resource footprint. >>> >>> = Proposal = >>> Concerted is built on the principal that a new type of workload is >>> dominating the scene and is now needed to be supported. These are the >> large >>> data set analytical workloads being analyzed or used on large clusters or >>> high power machines. Large analytical workloads depend on the ability to >>> query large data sets efficiently and in high concurrency while >> maintaining >>> semantics such as immediate consistency. An in memory engine designed to >>> support extreme read queries while providing support for aggregation >>> through various features (such as multidimensional representation of >>> tuples) will accelerate many usecases around large scale analytics. >>> >>> Concerted believes that best understanding of user application lies with >>> user application developer. The need for massive read scaling should be >> on >>> demand and should be flexible to the level that user can decide as to >> which >>> representation and access of data suits his/her current requirements. >>> Hence, Concerted is not built in a traditional client/server model. >>> Concerted provides users with an API which can be used to load, read, >>> update and delete data. User chooses which data structure has to be used >>> for his current requirements. All API access is covered by Concerted's >>> internal systems like lock manager, transaction manager and cache manager >>> which ensure that reads scale to high level in every API call. >>> >>> Concerted is a Do It Yourself in memory platform for making in memory >>> supporting engines. The use case we think of is supporting big data >>> warehouses like Hive, but there are endless use cases for a custom, >> highly >>> scalable in memory platform. >>> >>> The goal of this proposal is to leverage an existing code base available >> on >>> Github and licensed under the Apache License 2.0 to build a community >>> around the project. Currently the community consists of existing hackers >> of >>> Concerted as well as people who have been following and associated with >> the >>> project since a while as well as database experts who are excited about >>> building a project like this. We are hoping that entering into Apache >> would >>> help us attract more contributors as well as connect with existing big >> data >>> projects like Apache Hive, Apache HAWQ, Apache Storm, Apache Tajo, Apache >>> Spark, Apache Geode to leverage their community base while assisting in >>> their use cases with Concerted. We had a discussion with founders of >> Apache >>> Tajo and they showed interest in using Concerted for some of their use >>> cases. >>> = Background = >>> Relational databases were built with the cost of physical memory in mind. >>> The cost is no longer very relevant and physical memory is now available >> on >>> demand. Another driving factor behind Concerted is that there is a >> paradigm >>> shift with big data coming into picture. Disk IO speeds are more of a >>> bottleneck than ever before. Combining the read dominance of analytical >>> workload with the speed of in memory structures, Concerted fits the >> current >>> scene. Also, supporting OLAP workloads with in memory support for faster >>> read constant queries and joins will be useful. >>> >>> = Rationale = >>> As explained above, large analytical workloads need an in memory >>> lightweight engine which supports massive read concurrency, ground level >>> support for aggregations and analytics, extreme scalability and high read >>> performance, along with the engine being very light itself. Concerted >> aims >>> to solve these needs. Concerted is designed and built with three goals as >>> objectives: >>> >>> >>> Performance >>> To provide high performance access to data from a large number of >> rows, >>> Concerted uses efficient representation and in memory indexing of data >>> coupled with high performance transactions, custom transactions and >>> lightweight locking and lockless techniques and an inte
Re: [VOTE] Accept Concerted into the Apache Incubator
+1 (non-binding) Best Regards! - Luke Han On Mon, Oct 12, 2015 at 4:33 AM, Alan D. Cabrera wrote: > +1 - binding > > > Regards, > Alan > > > On Oct 9, 2015, at 8:55 AM, Atri Sharma wrote: > > > > Hi all, > > > > Following the discussion about Concerted I would like to call a vote for > > accepting Concerted as a new incubator project. > > > > The proposal text is included below, and available on the wiki: > > > > https://wiki.apache.org/incubator/ConcertedProposal > > > > The vote is open for 72 hours: > > > > [ ] +1 accept Concerted in the Incubator > > [ ] ±0 > > [ ] -1 (please give reason) > > > > Regards, > > > > Atri > > > > = Abstract = > > > > Concerted is an in memory write less read more engine aimed to provide > > extreme read performance with very high degree of concurrency and > > scalability and focus on minimizing own resource footprint. > > > > = Proposal = > > Concerted is built on the principal that a new type of workload is > > dominating the scene and is now needed to be supported. These are the > large > > data set analytical workloads being analyzed or used on large clusters or > > high power machines. Large analytical workloads depend on the ability to > > query large data sets efficiently and in high concurrency while > maintaining > > semantics such as immediate consistency. An in memory engine designed to > > support extreme read queries while providing support for aggregation > > through various features (such as multidimensional representation of > > tuples) will accelerate many usecases around large scale analytics. > > > > Concerted believes that best understanding of user application lies with > > user application developer. The need for massive read scaling should be > on > > demand and should be flexible to the level that user can decide as to > which > > representation and access of data suits his/her current requirements. > > Hence, Concerted is not built in a traditional client/server model. > > Concerted provides users with an API which can be used to load, read, > > update and delete data. User chooses which data structure has to be used > > for his current requirements. All API access is covered by Concerted's > > internal systems like lock manager, transaction manager and cache manager > > which ensure that reads scale to high level in every API call. > > > > Concerted is a Do It Yourself in memory platform for making in memory > > supporting engines. The use case we think of is supporting big data > > warehouses like Hive, but there are endless use cases for a custom, > highly > > scalable in memory platform. > > > > The goal of this proposal is to leverage an existing code base available > on > > Github and licensed under the Apache License 2.0 to build a community > > around the project. Currently the community consists of existing hackers > of > > Concerted as well as people who have been following and associated with > the > > project since a while as well as database experts who are excited about > > building a project like this. We are hoping that entering into Apache > would > > help us attract more contributors as well as connect with existing big > data > > projects like Apache Hive, Apache HAWQ, Apache Storm, Apache Tajo, Apache > > Spark, Apache Geode to leverage their community base while assisting in > > their use cases with Concerted. We had a discussion with founders of > Apache > > Tajo and they showed interest in using Concerted for some of their use > > cases. > > = Background = > > Relational databases were built with the cost of physical memory in mind. > > The cost is no longer very relevant and physical memory is now available > on > > demand. Another driving factor behind Concerted is that there is a > paradigm > > shift with big data coming into picture. Disk IO speeds are more of a > > bottleneck than ever before. Combining the read dominance of analytical > > workload with the speed of in memory structures, Concerted fits the > current > > scene. Also, supporting OLAP workloads with in memory support for faster > > read constant queries and joins will be useful. > > > > = Rationale = > > As explained above, large analytical workloads need an in memory > > lightweight engine which supports massive read concurrency, ground level > > support for aggregations and analytics, extreme scalability and high read > > performance, along with the engine being very light itself. Concerted > aims > > to solve these needs. Concerted is designed and built with three goals as > > objectives: > > > > > > Performance > >To provide high performance access to data from a large number of > rows, > > Concerted uses efficient representation and in memory indexing of data > > coupled with high performance transactions, custom transactions and > > lightweight locking and lockless techniques and an intelligent locking > > manager. > > > > Scalability > >Concerted is built with extreme concurrency and scalability in mind. > > > > Effi
Re: [VOTE] Accept Concerted into the Apache Incubator
I’m not sure this needs to be resolved before the polling can be accepted into the Incubator. Regards, Alan > On Oct 9, 2015, at 2:01 PM, Julian Hyde wrote: > > I have agreed to be a mentor to Concerted and I think it is an > interesting idea. I am inclined to vote for it entering the incubator. > > However since the project has not released any source code yet, there > are a couple of questions I'd like to get answered for the record: > > 1. How many lines of existing code are there? What is their approximate age? > > 2. Concerted is in C/C++ but you mention interfacing with JVM-based > products like Hive. How you would interface with other languages? Is > it a goal of the project to create APIs to other languages such as > Java? Would access from those languages be as efficient as native > access? > > I apologize that I didn't bring these up in the discussion thread. > > Julian > > > On Fri, Oct 9, 2015 at 11:53 AM, Ayrton Gomesz wrote: >> +1 >> @henry.saputra thanks man >> On Oct 9, 2015 5:50 PM, "Henry Saputra" wrote: >> >>> +1 (binding) >>> Good luck guys! >>> >>> On Fri, Oct 9, 2015 at 8:55 AM, Atri Sharma wrote: Hi all, Following the discussion about Concerted I would like to call a vote for accepting Concerted as a new incubator project. The proposal text is included below, and available on the wiki: https://wiki.apache.org/incubator/ConcertedProposal The vote is open for 72 hours: [ ] +1 accept Concerted in the Incubator [ ] ±0 [ ] -1 (please give reason) Regards, Atri = Abstract = Concerted is an in memory write less read more engine aimed to provide extreme read performance with very high degree of concurrency and scalability and focus on minimizing own resource footprint. = Proposal = Concerted is built on the principal that a new type of workload is dominating the scene and is now needed to be supported. These are the >>> large data set analytical workloads being analyzed or used on large clusters or high power machines. Large analytical workloads depend on the ability to query large data sets efficiently and in high concurrency while >>> maintaining semantics such as immediate consistency. An in memory engine designed to support extreme read queries while providing support for aggregation through various features (such as multidimensional representation of tuples) will accelerate many usecases around large scale analytics. Concerted believes that best understanding of user application lies with user application developer. The need for massive read scaling should be >>> on demand and should be flexible to the level that user can decide as to >>> which representation and access of data suits his/her current requirements. Hence, Concerted is not built in a traditional client/server model. Concerted provides users with an API which can be used to load, read, update and delete data. User chooses which data structure has to be used for his current requirements. All API access is covered by Concerted's internal systems like lock manager, transaction manager and cache manager which ensure that reads scale to high level in every API call. Concerted is a Do It Yourself in memory platform for making in memory supporting engines. The use case we think of is supporting big data warehouses like Hive, but there are endless use cases for a custom, >>> highly scalable in memory platform. The goal of this proposal is to leverage an existing code base available >>> on Github and licensed under the Apache License 2.0 to build a community around the project. Currently the community consists of existing hackers >>> of Concerted as well as people who have been following and associated with >>> the project since a while as well as database experts who are excited about building a project like this. We are hoping that entering into Apache >>> would help us attract more contributors as well as connect with existing big >>> data projects like Apache Hive, Apache HAWQ, Apache Storm, Apache Tajo, Apache Spark, Apache Geode to leverage their community base while assisting in their use cases with Concerted. We had a discussion with founders of >>> Apache Tajo and they showed interest in using Concerted for some of their use cases. = Background = Relational databases were built with the cost of physical memory in mind. The cost is no longer very relevant and physical memory is now available >>> on demand. Another driving factor behind Concerted is that there is a >>> paradigm shift with big data coming into picture. Disk IO speeds are more of a bottleneck than ever before. Combining the read dominance of analytical workload with the speed of in
Re: [VOTE] Accept Concerted into the Apache Incubator
+1 - binding Regards, Alan > On Oct 9, 2015, at 8:55 AM, Atri Sharma wrote: > > Hi all, > > Following the discussion about Concerted I would like to call a vote for > accepting Concerted as a new incubator project. > > The proposal text is included below, and available on the wiki: > > https://wiki.apache.org/incubator/ConcertedProposal > > The vote is open for 72 hours: > > [ ] +1 accept Concerted in the Incubator > [ ] ±0 > [ ] -1 (please give reason) > > Regards, > > Atri > > = Abstract = > > Concerted is an in memory write less read more engine aimed to provide > extreme read performance with very high degree of concurrency and > scalability and focus on minimizing own resource footprint. > > = Proposal = > Concerted is built on the principal that a new type of workload is > dominating the scene and is now needed to be supported. These are the large > data set analytical workloads being analyzed or used on large clusters or > high power machines. Large analytical workloads depend on the ability to > query large data sets efficiently and in high concurrency while maintaining > semantics such as immediate consistency. An in memory engine designed to > support extreme read queries while providing support for aggregation > through various features (such as multidimensional representation of > tuples) will accelerate many usecases around large scale analytics. > > Concerted believes that best understanding of user application lies with > user application developer. The need for massive read scaling should be on > demand and should be flexible to the level that user can decide as to which > representation and access of data suits his/her current requirements. > Hence, Concerted is not built in a traditional client/server model. > Concerted provides users with an API which can be used to load, read, > update and delete data. User chooses which data structure has to be used > for his current requirements. All API access is covered by Concerted's > internal systems like lock manager, transaction manager and cache manager > which ensure that reads scale to high level in every API call. > > Concerted is a Do It Yourself in memory platform for making in memory > supporting engines. The use case we think of is supporting big data > warehouses like Hive, but there are endless use cases for a custom, highly > scalable in memory platform. > > The goal of this proposal is to leverage an existing code base available on > Github and licensed under the Apache License 2.0 to build a community > around the project. Currently the community consists of existing hackers of > Concerted as well as people who have been following and associated with the > project since a while as well as database experts who are excited about > building a project like this. We are hoping that entering into Apache would > help us attract more contributors as well as connect with existing big data > projects like Apache Hive, Apache HAWQ, Apache Storm, Apache Tajo, Apache > Spark, Apache Geode to leverage their community base while assisting in > their use cases with Concerted. We had a discussion with founders of Apache > Tajo and they showed interest in using Concerted for some of their use > cases. > = Background = > Relational databases were built with the cost of physical memory in mind. > The cost is no longer very relevant and physical memory is now available on > demand. Another driving factor behind Concerted is that there is a paradigm > shift with big data coming into picture. Disk IO speeds are more of a > bottleneck than ever before. Combining the read dominance of analytical > workload with the speed of in memory structures, Concerted fits the current > scene. Also, supporting OLAP workloads with in memory support for faster > read constant queries and joins will be useful. > > = Rationale = > As explained above, large analytical workloads need an in memory > lightweight engine which supports massive read concurrency, ground level > support for aggregations and analytics, extreme scalability and high read > performance, along with the engine being very light itself. Concerted aims > to solve these needs. Concerted is designed and built with three goals as > objectives: > > > Performance >To provide high performance access to data from a large number of rows, > Concerted uses efficient representation and in memory indexing of data > coupled with high performance transactions, custom transactions and > lightweight locking and lockless techniques and an intelligent locking > manager. > > Scalability >Concerted is built with extreme concurrency and scalability in mind. > > Efficiency >Concerted aims to give expected performance under vast variety of > workloads and aims to have as low footprint as possible. > > = Initial Goals = > The initial goal is to leverage an existing code base and invest in > building a community around the project. We anticipate a lot of initial > restructuring of the existing co
Re: [VOTE] Accept Concerted into the Apache Incubator
+1 (binding) On Fri, Oct 9, 2015 at 5:55 PM, Atri Sharma wrote: > Hi all, > > Following the discussion about Concerted I would like to call a vote for > accepting Concerted as a new incubator project. > > The proposal text is included below, and available on the wiki: > > https://wiki.apache.org/incubator/ConcertedProposal > > The vote is open for 72 hours: > > [ ] +1 accept Concerted in the Incubator > [ ] ±0 > [ ] -1 (please give reason) > > Regards, > > Atri > > = Abstract = > > Concerted is an in memory write less read more engine aimed to provide > extreme read performance with very high degree of concurrency and > scalability and focus on minimizing own resource footprint. > > = Proposal = > Concerted is built on the principal that a new type of workload is > dominating the scene and is now needed to be supported. These are the large > data set analytical workloads being analyzed or used on large clusters or > high power machines. Large analytical workloads depend on the ability to > query large data sets efficiently and in high concurrency while maintaining > semantics such as immediate consistency. An in memory engine designed to > support extreme read queries while providing support for aggregation > through various features (such as multidimensional representation of > tuples) will accelerate many usecases around large scale analytics. > > Concerted believes that best understanding of user application lies with > user application developer. The need for massive read scaling should be on > demand and should be flexible to the level that user can decide as to which > representation and access of data suits his/her current requirements. > Hence, Concerted is not built in a traditional client/server model. > Concerted provides users with an API which can be used to load, read, > update and delete data. User chooses which data structure has to be used > for his current requirements. All API access is covered by Concerted's > internal systems like lock manager, transaction manager and cache manager > which ensure that reads scale to high level in every API call. > > Concerted is a Do It Yourself in memory platform for making in memory > supporting engines. The use case we think of is supporting big data > warehouses like Hive, but there are endless use cases for a custom, highly > scalable in memory platform. > > The goal of this proposal is to leverage an existing code base available on > Github and licensed under the Apache License 2.0 to build a community > around the project. Currently the community consists of existing hackers of > Concerted as well as people who have been following and associated with the > project since a while as well as database experts who are excited about > building a project like this. We are hoping that entering into Apache would > help us attract more contributors as well as connect with existing big data > projects like Apache Hive, Apache HAWQ, Apache Storm, Apache Tajo, Apache > Spark, Apache Geode to leverage their community base while assisting in > their use cases with Concerted. We had a discussion with founders of Apache > Tajo and they showed interest in using Concerted for some of their use > cases. > = Background = > Relational databases were built with the cost of physical memory in mind. > The cost is no longer very relevant and physical memory is now available on > demand. Another driving factor behind Concerted is that there is a paradigm > shift with big data coming into picture. Disk IO speeds are more of a > bottleneck than ever before. Combining the read dominance of analytical > workload with the speed of in memory structures, Concerted fits the current > scene. Also, supporting OLAP workloads with in memory support for faster > read constant queries and joins will be useful. > > = Rationale = > As explained above, large analytical workloads need an in memory > lightweight engine which supports massive read concurrency, ground level > support for aggregations and analytics, extreme scalability and high read > performance, along with the engine being very light itself. Concerted aims > to solve these needs. Concerted is designed and built with three goals as > objectives: > > > Performance > To provide high performance access to data from a large number of rows, > Concerted uses efficient representation and in memory indexing of data > coupled with high performance transactions, custom transactions and > lightweight locking and lockless techniques and an intelligent locking > manager. > > Scalability > Concerted is built with extreme concurrency and scalability in mind. > > Efficiency > Concerted aims to give expected performance under vast variety of > workloads and aims to have as low footprint as possible. > > = Initial Goals = > The initial goal is to leverage an existing code base and invest in > building a community around the project. We anticipate a lot of initial > restructuring of the existing code so that it becomes easier to
Re: [VOTE] Accept Concerted into the Apache Incubator
+1 binding -Jake On Fri, Oct 9, 2015 at 11:55 AM, Atri Sharma wrote: > Hi all, > > Following the discussion about Concerted I would like to call a vote for > accepting Concerted as a new incubator project. > > The proposal text is included below, and available on the wiki: > > https://wiki.apache.org/incubator/ConcertedProposal > > The vote is open for 72 hours: > > [ ] +1 accept Concerted in the Incubator > [ ] ±0 > [ ] -1 (please give reason) > > Regards, > > Atri > > = Abstract = > > Concerted is an in memory write less read more engine aimed to provide > extreme read performance with very high degree of concurrency and > scalability and focus on minimizing own resource footprint. > > = Proposal = > Concerted is built on the principal that a new type of workload is > dominating the scene and is now needed to be supported. These are the large > data set analytical workloads being analyzed or used on large clusters or > high power machines. Large analytical workloads depend on the ability to > query large data sets efficiently and in high concurrency while maintaining > semantics such as immediate consistency. An in memory engine designed to > support extreme read queries while providing support for aggregation > through various features (such as multidimensional representation of > tuples) will accelerate many usecases around large scale analytics. > > Concerted believes that best understanding of user application lies with > user application developer. The need for massive read scaling should be on > demand and should be flexible to the level that user can decide as to which > representation and access of data suits his/her current requirements. > Hence, Concerted is not built in a traditional client/server model. > Concerted provides users with an API which can be used to load, read, > update and delete data. User chooses which data structure has to be used > for his current requirements. All API access is covered by Concerted's > internal systems like lock manager, transaction manager and cache manager > which ensure that reads scale to high level in every API call. > > Concerted is a Do It Yourself in memory platform for making in memory > supporting engines. The use case we think of is supporting big data > warehouses like Hive, but there are endless use cases for a custom, highly > scalable in memory platform. > > The goal of this proposal is to leverage an existing code base available on > Github and licensed under the Apache License 2.0 to build a community > around the project. Currently the community consists of existing hackers of > Concerted as well as people who have been following and associated with the > project since a while as well as database experts who are excited about > building a project like this. We are hoping that entering into Apache would > help us attract more contributors as well as connect with existing big data > projects like Apache Hive, Apache HAWQ, Apache Storm, Apache Tajo, Apache > Spark, Apache Geode to leverage their community base while assisting in > their use cases with Concerted. We had a discussion with founders of Apache > Tajo and they showed interest in using Concerted for some of their use > cases. > = Background = > Relational databases were built with the cost of physical memory in mind. > The cost is no longer very relevant and physical memory is now available on > demand. Another driving factor behind Concerted is that there is a paradigm > shift with big data coming into picture. Disk IO speeds are more of a > bottleneck than ever before. Combining the read dominance of analytical > workload with the speed of in memory structures, Concerted fits the current > scene. Also, supporting OLAP workloads with in memory support for faster > read constant queries and joins will be useful. > > = Rationale = > As explained above, large analytical workloads need an in memory > lightweight engine which supports massive read concurrency, ground level > support for aggregations and analytics, extreme scalability and high read > performance, along with the engine being very light itself. Concerted aims > to solve these needs. Concerted is designed and built with three goals as > objectives: > > > Performance > To provide high performance access to data from a large number of rows, > Concerted uses efficient representation and in memory indexing of data > coupled with high performance transactions, custom transactions and > lightweight locking and lockless techniques and an intelligent locking > manager. > > Scalability > Concerted is built with extreme concurrency and scalability in mind. > > Efficiency > Concerted aims to give expected performance under vast variety of > workloads and aims to have as low footprint as possible. > > = Initial Goals = > The initial goal is to leverage an existing code base and invest in > building a community around the project. We anticipate a lot of initial > restructuring of the existing code so that it becomes easi
Re: [VOTE] Accept Concerted into the Apache Incubator
On Fri, Oct 9, 2015 at 6:55 PM, Atri Sharma wrote: > Hi all, > > Following the discussion about Concerted I would like to call a vote for > accepting Concerted as a new incubator project. > > The proposal text is included below, and available on the wiki: > > https://wiki.apache.org/incubator/ConcertedProposal > > The vote is open for 72 hours: > > [ ] +1 accept Concerted in the Incubator > [ ] ±0 > [ ] -1 (please give reason) +1 (binding) Thanks, Roman. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] Accept Concerted into the Apache Incubator
+1 (binding) This is exciting!Thanks for putting this together Atri. -- Lars From: Atri Sharma To: general@incubator.apache.org Sent: Friday, October 9, 2015 8:55 AM Subject: [VOTE] Accept Concerted into the Apache Incubator Hi all, Following the discussion about Concerted I would like to call a vote for accepting Concerted as a new incubator project. The proposal text is included below, and available on the wiki: https://wiki.apache.org/incubator/ConcertedProposal The vote is open for 72 hours: [ ] +1 accept Concerted in the Incubator [ ] ±0 [ ] -1 (please give reason) Regards, Atri = Abstract = Concerted is an in memory write less read more engine aimed to provide extreme read performance with very high degree of concurrency and scalability and focus on minimizing own resource footprint. = Proposal = Concerted is built on the principal that a new type of workload is dominating the scene and is now needed to be supported. These are the large data set analytical workloads being analyzed or used on large clusters or high power machines. Large analytical workloads depend on the ability to query large data sets efficiently and in high concurrency while maintaining semantics such as immediate consistency. An in memory engine designed to support extreme read queries while providing support for aggregation through various features (such as multidimensional representation of tuples) will accelerate many usecases around large scale analytics. Concerted believes that best understanding of user application lies with user application developer. The need for massive read scaling should be on demand and should be flexible to the level that user can decide as to which representation and access of data suits his/her current requirements. Hence, Concerted is not built in a traditional client/server model. Concerted provides users with an API which can be used to load, read, update and delete data. User chooses which data structure has to be used for his current requirements. All API access is covered by Concerted's internal systems like lock manager, transaction manager and cache manager which ensure that reads scale to high level in every API call. Concerted is a Do It Yourself in memory platform for making in memory supporting engines. The use case we think of is supporting big data warehouses like Hive, but there are endless use cases for a custom, highly scalable in memory platform. The goal of this proposal is to leverage an existing code base available on Github and licensed under the Apache License 2.0 to build a community around the project. Currently the community consists of existing hackers of Concerted as well as people who have been following and associated with the project since a while as well as database experts who are excited about building a project like this. We are hoping that entering into Apache would help us attract more contributors as well as connect with existing big data projects like Apache Hive, Apache HAWQ, Apache Storm, Apache Tajo, Apache Spark, Apache Geode to leverage their community base while assisting in their use cases with Concerted. We had a discussion with founders of Apache Tajo and they showed interest in using Concerted for some of their use cases. = Background = Relational databases were built with the cost of physical memory in mind. The cost is no longer very relevant and physical memory is now available on demand. Another driving factor behind Concerted is that there is a paradigm shift with big data coming into picture. Disk IO speeds are more of a bottleneck than ever before. Combining the read dominance of analytical workload with the speed of in memory structures, Concerted fits the current scene. Also, supporting OLAP workloads with in memory support for faster read constant queries and joins will be useful. = Rationale = As explained above, large analytical workloads need an in memory lightweight engine which supports massive read concurrency, ground level support for aggregations and analytics, extreme scalability and high read performance, along with the engine being very light itself. Concerted aims to solve these needs. Concerted is designed and built with three goals as objectives: Performance To provide high performance access to data from a large number of rows, Concerted uses efficient representation and in memory indexing of data coupled with high performance transactions, custom transactions and lightweight locking and lockless techniques and an intelligent locking manager. Scalability Concerted is built with extreme concurrency and scalability in mind. Efficiency Concerted aims to give expected performance under vast variety of workloads and aims to have as low footprint as possible. = Initial Goals = The initial goal is to leverage an existing code base and invest in building a community around the project. We anticipate a lot of initial restructuring of the existing code so that it becomes easier to in
Re: [VOTE] Accept Concerted into the Apache Incubator
+1 (binding) Thank you, Atri. --Chris Nauroth On 10/9/15, 8:55 AM, "Atri Sharma" wrote: >Hi all, > >Following the discussion about Concerted I would like to call a vote for >accepting Concerted as a new incubator project. > >The proposal text is included below, and available on the wiki: > >https://wiki.apache.org/incubator/ConcertedProposal > >The vote is open for 72 hours: > >[ ] +1 accept Concerted in the Incubator >[ ] ±0 >[ ] -1 (please give reason) > >Regards, > >Atri > >= Abstract = > >Concerted is an in memory write less read more engine aimed to provide >extreme read performance with very high degree of concurrency and >scalability and focus on minimizing own resource footprint. > >= Proposal = >Concerted is built on the principal that a new type of workload is >dominating the scene and is now needed to be supported. These are the >large >data set analytical workloads being analyzed or used on large clusters or >high power machines. Large analytical workloads depend on the ability to >query large data sets efficiently and in high concurrency while >maintaining >semantics such as immediate consistency. An in memory engine designed to >support extreme read queries while providing support for aggregation >through various features (such as multidimensional representation of >tuples) will accelerate many usecases around large scale analytics. > >Concerted believes that best understanding of user application lies with >user application developer. The need for massive read scaling should be on >demand and should be flexible to the level that user can decide as to >which >representation and access of data suits his/her current requirements. >Hence, Concerted is not built in a traditional client/server model. >Concerted provides users with an API which can be used to load, read, >update and delete data. User chooses which data structure has to be used >for his current requirements. All API access is covered by Concerted's >internal systems like lock manager, transaction manager and cache manager >which ensure that reads scale to high level in every API call. > >Concerted is a Do It Yourself in memory platform for making in memory >supporting engines. The use case we think of is supporting big data >warehouses like Hive, but there are endless use cases for a custom, highly >scalable in memory platform. > >The goal of this proposal is to leverage an existing code base available >on >Github and licensed under the Apache License 2.0 to build a community >around the project. Currently the community consists of existing hackers >of >Concerted as well as people who have been following and associated with >the >project since a while as well as database experts who are excited about >building a project like this. We are hoping that entering into Apache >would >help us attract more contributors as well as connect with existing big >data >projects like Apache Hive, Apache HAWQ, Apache Storm, Apache Tajo, Apache >Spark, Apache Geode to leverage their community base while assisting in >their use cases with Concerted. We had a discussion with founders of >Apache >Tajo and they showed interest in using Concerted for some of their use >cases. >= Background = >Relational databases were built with the cost of physical memory in mind. >The cost is no longer very relevant and physical memory is now available >on >demand. Another driving factor behind Concerted is that there is a >paradigm >shift with big data coming into picture. Disk IO speeds are more of a >bottleneck than ever before. Combining the read dominance of analytical >workload with the speed of in memory structures, Concerted fits the >current >scene. Also, supporting OLAP workloads with in memory support for faster >read constant queries and joins will be useful. > >= Rationale = >As explained above, large analytical workloads need an in memory >lightweight engine which supports massive read concurrency, ground level >support for aggregations and analytics, extreme scalability and high read >performance, along with the engine being very light itself. Concerted aims >to solve these needs. Concerted is designed and built with three goals as >objectives: > > >Performance >To provide high performance access to data from a large number of >rows, >Concerted uses efficient representation and in memory indexing of data >coupled with high performance transactions, custom transactions and >lightweight locking and lockless techniques and an intelligent locking >manager. > >Scalability >Concerted is built with extreme concurrency and scalability in mind. > >Efficiency >Concerted aims to give expected performance under vast variety of >workloads and aims to have as low footprint as possible. > >= Initial Goals = >The initial goal is to leverage an existing code base and invest in >building a community around the project. We anticipate a lot of initial >restructuring of the existing code so that it becomes easier to include >new >contributors and minimize
Re: [VOTE] Accept Concerted into the Apache Incubator
Thanks for clarifying. +1 (binding) Julian On Fri, Oct 9, 2015 at 2:09 PM, Atri Sharma wrote: > Hi, > > Please find answers below: > > 1) The main source code on Github wasn't updated for a while. However, the > original and main core was written in 2013 and has been open source since > then. As we discussed earlier current code base is only starting point for > complete development and will be first integrated with silo work done > independent and then used as starting implementation. > > 2) The JNI native API when optimized can provide great performance ( I have > written an application using it and it is on production systems for many > years). I think we can still provide a high performance API to the C++ core > and that is something I am personally working on right now. > On 10 Oct 2015 02:31, "Julian Hyde" wrote: > >> I have agreed to be a mentor to Concerted and I think it is an >> interesting idea. I am inclined to vote for it entering the incubator. >> >> However since the project has not released any source code yet, there >> are a couple of questions I'd like to get answered for the record: >> >> 1. How many lines of existing code are there? What is their approximate >> age? >> >> 2. Concerted is in C/C++ but you mention interfacing with JVM-based >> products like Hive. How you would interface with other languages? Is >> it a goal of the project to create APIs to other languages such as >> Java? Would access from those languages be as efficient as native >> access? >> >> I apologize that I didn't bring these up in the discussion thread. >> >> Julian >> >> >> On Fri, Oct 9, 2015 at 11:53 AM, Ayrton Gomesz >> wrote: >> > +1 >> > @henry.saputra thanks man >> > On Oct 9, 2015 5:50 PM, "Henry Saputra" wrote: >> > >> >> +1 (binding) >> >> Good luck guys! >> >> >> >> On Fri, Oct 9, 2015 at 8:55 AM, Atri Sharma wrote: >> >> > Hi all, >> >> > >> >> > Following the discussion about Concerted I would like to call a vote >> for >> >> > accepting Concerted as a new incubator project. >> >> > >> >> > The proposal text is included below, and available on the wiki: >> >> > >> >> > https://wiki.apache.org/incubator/ConcertedProposal >> >> > >> >> > The vote is open for 72 hours: >> >> > >> >> > [ ] +1 accept Concerted in the Incubator >> >> > [ ] ±0 >> >> > [ ] -1 (please give reason) >> >> > >> >> > Regards, >> >> > >> >> > Atri >> >> > >> >> > = Abstract = >> >> > >> >> > Concerted is an in memory write less read more engine aimed to provide >> >> > extreme read performance with very high degree of concurrency and >> >> > scalability and focus on minimizing own resource footprint. >> >> > >> >> > = Proposal = >> >> > Concerted is built on the principal that a new type of workload is >> >> > dominating the scene and is now needed to be supported. These are the >> >> large >> >> > data set analytical workloads being analyzed or used on large >> clusters or >> >> > high power machines. Large analytical workloads depend on the ability >> to >> >> > query large data sets efficiently and in high concurrency while >> >> maintaining >> >> > semantics such as immediate consistency. An in memory engine designed >> to >> >> > support extreme read queries while providing support for aggregation >> >> > through various features (such as multidimensional representation of >> >> > tuples) will accelerate many usecases around large scale analytics. >> >> > >> >> > Concerted believes that best understanding of user application lies >> with >> >> > user application developer. The need for massive read scaling should >> be >> >> on >> >> > demand and should be flexible to the level that user can decide as to >> >> which >> >> > representation and access of data suits his/her current requirements. >> >> > Hence, Concerted is not built in a traditional client/server model. >> >> > Concerted provides users with an API which can be used to load, read, >> >> > update and delete data. User chooses which data structure has to be >> used >> >> > for his current requirements. All API access is covered by Concerted's >> >> > internal systems like lock manager, transaction manager and cache >> manager >> >> > which ensure that reads scale to high level in every API call. >> >> > >> >> > Concerted is a Do It Yourself in memory platform for making in memory >> >> > supporting engines. The use case we think of is supporting big data >> >> > warehouses like Hive, but there are endless use cases for a custom, >> >> highly >> >> > scalable in memory platform. >> >> > >> >> > The goal of this proposal is to leverage an existing code base >> available >> >> on >> >> > Github and licensed under the Apache License 2.0 to build a community >> >> > around the project. Currently the community consists of existing >> hackers >> >> of >> >> > Concerted as well as people who have been following and associated >> with >> >> the >> >> > project since a while as well as database experts who are excited >> about >> >> > building a project like this.
Re: [VOTE] Accept Concerted into the Apache Incubator
Hi, Please find answers below: 1) The main source code on Github wasn't updated for a while. However, the original and main core was written in 2013 and has been open source since then. As we discussed earlier current code base is only starting point for complete development and will be first integrated with silo work done independent and then used as starting implementation. 2) The JNI native API when optimized can provide great performance ( I have written an application using it and it is on production systems for many years). I think we can still provide a high performance API to the C++ core and that is something I am personally working on right now. On 10 Oct 2015 02:31, "Julian Hyde" wrote: > I have agreed to be a mentor to Concerted and I think it is an > interesting idea. I am inclined to vote for it entering the incubator. > > However since the project has not released any source code yet, there > are a couple of questions I'd like to get answered for the record: > > 1. How many lines of existing code are there? What is their approximate > age? > > 2. Concerted is in C/C++ but you mention interfacing with JVM-based > products like Hive. How you would interface with other languages? Is > it a goal of the project to create APIs to other languages such as > Java? Would access from those languages be as efficient as native > access? > > I apologize that I didn't bring these up in the discussion thread. > > Julian > > > On Fri, Oct 9, 2015 at 11:53 AM, Ayrton Gomesz > wrote: > > +1 > > @henry.saputra thanks man > > On Oct 9, 2015 5:50 PM, "Henry Saputra" wrote: > > > >> +1 (binding) > >> Good luck guys! > >> > >> On Fri, Oct 9, 2015 at 8:55 AM, Atri Sharma wrote: > >> > Hi all, > >> > > >> > Following the discussion about Concerted I would like to call a vote > for > >> > accepting Concerted as a new incubator project. > >> > > >> > The proposal text is included below, and available on the wiki: > >> > > >> > https://wiki.apache.org/incubator/ConcertedProposal > >> > > >> > The vote is open for 72 hours: > >> > > >> > [ ] +1 accept Concerted in the Incubator > >> > [ ] ±0 > >> > [ ] -1 (please give reason) > >> > > >> > Regards, > >> > > >> > Atri > >> > > >> > = Abstract = > >> > > >> > Concerted is an in memory write less read more engine aimed to provide > >> > extreme read performance with very high degree of concurrency and > >> > scalability and focus on minimizing own resource footprint. > >> > > >> > = Proposal = > >> > Concerted is built on the principal that a new type of workload is > >> > dominating the scene and is now needed to be supported. These are the > >> large > >> > data set analytical workloads being analyzed or used on large > clusters or > >> > high power machines. Large analytical workloads depend on the ability > to > >> > query large data sets efficiently and in high concurrency while > >> maintaining > >> > semantics such as immediate consistency. An in memory engine designed > to > >> > support extreme read queries while providing support for aggregation > >> > through various features (such as multidimensional representation of > >> > tuples) will accelerate many usecases around large scale analytics. > >> > > >> > Concerted believes that best understanding of user application lies > with > >> > user application developer. The need for massive read scaling should > be > >> on > >> > demand and should be flexible to the level that user can decide as to > >> which > >> > representation and access of data suits his/her current requirements. > >> > Hence, Concerted is not built in a traditional client/server model. > >> > Concerted provides users with an API which can be used to load, read, > >> > update and delete data. User chooses which data structure has to be > used > >> > for his current requirements. All API access is covered by Concerted's > >> > internal systems like lock manager, transaction manager and cache > manager > >> > which ensure that reads scale to high level in every API call. > >> > > >> > Concerted is a Do It Yourself in memory platform for making in memory > >> > supporting engines. The use case we think of is supporting big data > >> > warehouses like Hive, but there are endless use cases for a custom, > >> highly > >> > scalable in memory platform. > >> > > >> > The goal of this proposal is to leverage an existing code base > available > >> on > >> > Github and licensed under the Apache License 2.0 to build a community > >> > around the project. Currently the community consists of existing > hackers > >> of > >> > Concerted as well as people who have been following and associated > with > >> the > >> > project since a while as well as database experts who are excited > about > >> > building a project like this. We are hoping that entering into Apache > >> would > >> > help us attract more contributors as well as connect with existing big > >> data > >> > projects like Apache Hive, Apache HAWQ, Apache Storm, Apache Tajo, > Apache > >> > Spark,
Re: [VOTE] Accept Concerted into the Apache Incubator
I have agreed to be a mentor to Concerted and I think it is an interesting idea. I am inclined to vote for it entering the incubator. However since the project has not released any source code yet, there are a couple of questions I'd like to get answered for the record: 1. How many lines of existing code are there? What is their approximate age? 2. Concerted is in C/C++ but you mention interfacing with JVM-based products like Hive. How you would interface with other languages? Is it a goal of the project to create APIs to other languages such as Java? Would access from those languages be as efficient as native access? I apologize that I didn't bring these up in the discussion thread. Julian On Fri, Oct 9, 2015 at 11:53 AM, Ayrton Gomesz wrote: > +1 > @henry.saputra thanks man > On Oct 9, 2015 5:50 PM, "Henry Saputra" wrote: > >> +1 (binding) >> Good luck guys! >> >> On Fri, Oct 9, 2015 at 8:55 AM, Atri Sharma wrote: >> > Hi all, >> > >> > Following the discussion about Concerted I would like to call a vote for >> > accepting Concerted as a new incubator project. >> > >> > The proposal text is included below, and available on the wiki: >> > >> > https://wiki.apache.org/incubator/ConcertedProposal >> > >> > The vote is open for 72 hours: >> > >> > [ ] +1 accept Concerted in the Incubator >> > [ ] ±0 >> > [ ] -1 (please give reason) >> > >> > Regards, >> > >> > Atri >> > >> > = Abstract = >> > >> > Concerted is an in memory write less read more engine aimed to provide >> > extreme read performance with very high degree of concurrency and >> > scalability and focus on minimizing own resource footprint. >> > >> > = Proposal = >> > Concerted is built on the principal that a new type of workload is >> > dominating the scene and is now needed to be supported. These are the >> large >> > data set analytical workloads being analyzed or used on large clusters or >> > high power machines. Large analytical workloads depend on the ability to >> > query large data sets efficiently and in high concurrency while >> maintaining >> > semantics such as immediate consistency. An in memory engine designed to >> > support extreme read queries while providing support for aggregation >> > through various features (such as multidimensional representation of >> > tuples) will accelerate many usecases around large scale analytics. >> > >> > Concerted believes that best understanding of user application lies with >> > user application developer. The need for massive read scaling should be >> on >> > demand and should be flexible to the level that user can decide as to >> which >> > representation and access of data suits his/her current requirements. >> > Hence, Concerted is not built in a traditional client/server model. >> > Concerted provides users with an API which can be used to load, read, >> > update and delete data. User chooses which data structure has to be used >> > for his current requirements. All API access is covered by Concerted's >> > internal systems like lock manager, transaction manager and cache manager >> > which ensure that reads scale to high level in every API call. >> > >> > Concerted is a Do It Yourself in memory platform for making in memory >> > supporting engines. The use case we think of is supporting big data >> > warehouses like Hive, but there are endless use cases for a custom, >> highly >> > scalable in memory platform. >> > >> > The goal of this proposal is to leverage an existing code base available >> on >> > Github and licensed under the Apache License 2.0 to build a community >> > around the project. Currently the community consists of existing hackers >> of >> > Concerted as well as people who have been following and associated with >> the >> > project since a while as well as database experts who are excited about >> > building a project like this. We are hoping that entering into Apache >> would >> > help us attract more contributors as well as connect with existing big >> data >> > projects like Apache Hive, Apache HAWQ, Apache Storm, Apache Tajo, Apache >> > Spark, Apache Geode to leverage their community base while assisting in >> > their use cases with Concerted. We had a discussion with founders of >> Apache >> > Tajo and they showed interest in using Concerted for some of their use >> > cases. >> > = Background = >> > Relational databases were built with the cost of physical memory in mind. >> > The cost is no longer very relevant and physical memory is now available >> on >> > demand. Another driving factor behind Concerted is that there is a >> paradigm >> > shift with big data coming into picture. Disk IO speeds are more of a >> > bottleneck than ever before. Combining the read dominance of analytical >> > workload with the speed of in memory structures, Concerted fits the >> current >> > scene. Also, supporting OLAP workloads with in memory support for faster >> > read constant queries and joins will be useful. >> > >> > = Rationale = >> > As explained above, large analyt
Re: [VOTE] Accept Concerted into the Apache Incubator
+1 @henry.saputra thanks man On Oct 9, 2015 5:50 PM, "Henry Saputra" wrote: > +1 (binding) > Good luck guys! > > On Fri, Oct 9, 2015 at 8:55 AM, Atri Sharma wrote: > > Hi all, > > > > Following the discussion about Concerted I would like to call a vote for > > accepting Concerted as a new incubator project. > > > > The proposal text is included below, and available on the wiki: > > > > https://wiki.apache.org/incubator/ConcertedProposal > > > > The vote is open for 72 hours: > > > > [ ] +1 accept Concerted in the Incubator > > [ ] ±0 > > [ ] -1 (please give reason) > > > > Regards, > > > > Atri > > > > = Abstract = > > > > Concerted is an in memory write less read more engine aimed to provide > > extreme read performance with very high degree of concurrency and > > scalability and focus on minimizing own resource footprint. > > > > = Proposal = > > Concerted is built on the principal that a new type of workload is > > dominating the scene and is now needed to be supported. These are the > large > > data set analytical workloads being analyzed or used on large clusters or > > high power machines. Large analytical workloads depend on the ability to > > query large data sets efficiently and in high concurrency while > maintaining > > semantics such as immediate consistency. An in memory engine designed to > > support extreme read queries while providing support for aggregation > > through various features (such as multidimensional representation of > > tuples) will accelerate many usecases around large scale analytics. > > > > Concerted believes that best understanding of user application lies with > > user application developer. The need for massive read scaling should be > on > > demand and should be flexible to the level that user can decide as to > which > > representation and access of data suits his/her current requirements. > > Hence, Concerted is not built in a traditional client/server model. > > Concerted provides users with an API which can be used to load, read, > > update and delete data. User chooses which data structure has to be used > > for his current requirements. All API access is covered by Concerted's > > internal systems like lock manager, transaction manager and cache manager > > which ensure that reads scale to high level in every API call. > > > > Concerted is a Do It Yourself in memory platform for making in memory > > supporting engines. The use case we think of is supporting big data > > warehouses like Hive, but there are endless use cases for a custom, > highly > > scalable in memory platform. > > > > The goal of this proposal is to leverage an existing code base available > on > > Github and licensed under the Apache License 2.0 to build a community > > around the project. Currently the community consists of existing hackers > of > > Concerted as well as people who have been following and associated with > the > > project since a while as well as database experts who are excited about > > building a project like this. We are hoping that entering into Apache > would > > help us attract more contributors as well as connect with existing big > data > > projects like Apache Hive, Apache HAWQ, Apache Storm, Apache Tajo, Apache > > Spark, Apache Geode to leverage their community base while assisting in > > their use cases with Concerted. We had a discussion with founders of > Apache > > Tajo and they showed interest in using Concerted for some of their use > > cases. > > = Background = > > Relational databases were built with the cost of physical memory in mind. > > The cost is no longer very relevant and physical memory is now available > on > > demand. Another driving factor behind Concerted is that there is a > paradigm > > shift with big data coming into picture. Disk IO speeds are more of a > > bottleneck than ever before. Combining the read dominance of analytical > > workload with the speed of in memory structures, Concerted fits the > current > > scene. Also, supporting OLAP workloads with in memory support for faster > > read constant queries and joins will be useful. > > > > = Rationale = > > As explained above, large analytical workloads need an in memory > > lightweight engine which supports massive read concurrency, ground level > > support for aggregations and analytics, extreme scalability and high read > > performance, along with the engine being very light itself. Concerted > aims > > to solve these needs. Concerted is designed and built with three goals as > > objectives: > > > > > > Performance > > To provide high performance access to data from a large number of > rows, > > Concerted uses efficient representation and in memory indexing of data > > coupled with high performance transactions, custom transactions and > > lightweight locking and lockless techniques and an intelligent locking > > manager. > > > > Scalability > > Concerted is built with extreme concurrency and scalability in mind. > > > > Efficiency > > Concerted aims to give expected per
Re: [VOTE] Accept Concerted into the Apache Incubator
+1 (binding) Good luck guys! On Fri, Oct 9, 2015 at 8:55 AM, Atri Sharma wrote: > Hi all, > > Following the discussion about Concerted I would like to call a vote for > accepting Concerted as a new incubator project. > > The proposal text is included below, and available on the wiki: > > https://wiki.apache.org/incubator/ConcertedProposal > > The vote is open for 72 hours: > > [ ] +1 accept Concerted in the Incubator > [ ] ±0 > [ ] -1 (please give reason) > > Regards, > > Atri > > = Abstract = > > Concerted is an in memory write less read more engine aimed to provide > extreme read performance with very high degree of concurrency and > scalability and focus on minimizing own resource footprint. > > = Proposal = > Concerted is built on the principal that a new type of workload is > dominating the scene and is now needed to be supported. These are the large > data set analytical workloads being analyzed or used on large clusters or > high power machines. Large analytical workloads depend on the ability to > query large data sets efficiently and in high concurrency while maintaining > semantics such as immediate consistency. An in memory engine designed to > support extreme read queries while providing support for aggregation > through various features (such as multidimensional representation of > tuples) will accelerate many usecases around large scale analytics. > > Concerted believes that best understanding of user application lies with > user application developer. The need for massive read scaling should be on > demand and should be flexible to the level that user can decide as to which > representation and access of data suits his/her current requirements. > Hence, Concerted is not built in a traditional client/server model. > Concerted provides users with an API which can be used to load, read, > update and delete data. User chooses which data structure has to be used > for his current requirements. All API access is covered by Concerted's > internal systems like lock manager, transaction manager and cache manager > which ensure that reads scale to high level in every API call. > > Concerted is a Do It Yourself in memory platform for making in memory > supporting engines. The use case we think of is supporting big data > warehouses like Hive, but there are endless use cases for a custom, highly > scalable in memory platform. > > The goal of this proposal is to leverage an existing code base available on > Github and licensed under the Apache License 2.0 to build a community > around the project. Currently the community consists of existing hackers of > Concerted as well as people who have been following and associated with the > project since a while as well as database experts who are excited about > building a project like this. We are hoping that entering into Apache would > help us attract more contributors as well as connect with existing big data > projects like Apache Hive, Apache HAWQ, Apache Storm, Apache Tajo, Apache > Spark, Apache Geode to leverage their community base while assisting in > their use cases with Concerted. We had a discussion with founders of Apache > Tajo and they showed interest in using Concerted for some of their use > cases. > = Background = > Relational databases were built with the cost of physical memory in mind. > The cost is no longer very relevant and physical memory is now available on > demand. Another driving factor behind Concerted is that there is a paradigm > shift with big data coming into picture. Disk IO speeds are more of a > bottleneck than ever before. Combining the read dominance of analytical > workload with the speed of in memory structures, Concerted fits the current > scene. Also, supporting OLAP workloads with in memory support for faster > read constant queries and joins will be useful. > > = Rationale = > As explained above, large analytical workloads need an in memory > lightweight engine which supports massive read concurrency, ground level > support for aggregations and analytics, extreme scalability and high read > performance, along with the engine being very light itself. Concerted aims > to solve these needs. Concerted is designed and built with three goals as > objectives: > > > Performance > To provide high performance access to data from a large number of rows, > Concerted uses efficient representation and in memory indexing of data > coupled with high performance transactions, custom transactions and > lightweight locking and lockless techniques and an intelligent locking > manager. > > Scalability > Concerted is built with extreme concurrency and scalability in mind. > > Efficiency > Concerted aims to give expected performance under vast variety of > workloads and aims to have as low footprint as possible. > > = Initial Goals = > The initial goal is to leverage an existing code base and invest in > building a community around the project. We anticipate a lot of initial > restructuring of the existing code so that it bec
Re: [VOTE] Accept Concerted into the Apache Incubator
+1 (non-binding) Pavel 2015-10-09 17:55 GMT+02:00 Atri Sharma : > Hi all, > > Following the discussion about Concerted I would like to call a vote for > accepting Concerted as a new incubator project. > > The proposal text is included below, and available on the wiki: > > https://wiki.apache.org/incubator/ConcertedProposal > > The vote is open for 72 hours: > > [ ] +1 accept Concerted in the Incubator > [ ] ±0 > [ ] -1 (please give reason) > > Regards, > > Atri > > = Abstract = > > Concerted is an in memory write less read more engine aimed to provide > extreme read performance with very high degree of concurrency and > scalability and focus on minimizing own resource footprint. > > = Proposal = > Concerted is built on the principal that a new type of workload is > dominating the scene and is now needed to be supported. These are the large > data set analytical workloads being analyzed or used on large clusters or > high power machines. Large analytical workloads depend on the ability to > query large data sets efficiently and in high concurrency while maintaining > semantics such as immediate consistency. An in memory engine designed to > support extreme read queries while providing support for aggregation > through various features (such as multidimensional representation of > tuples) will accelerate many usecases around large scale analytics. > > Concerted believes that best understanding of user application lies with > user application developer. The need for massive read scaling should be on > demand and should be flexible to the level that user can decide as to which > representation and access of data suits his/her current requirements. > Hence, Concerted is not built in a traditional client/server model. > Concerted provides users with an API which can be used to load, read, > update and delete data. User chooses which data structure has to be used > for his current requirements. All API access is covered by Concerted's > internal systems like lock manager, transaction manager and cache manager > which ensure that reads scale to high level in every API call. > > Concerted is a Do It Yourself in memory platform for making in memory > supporting engines. The use case we think of is supporting big data > warehouses like Hive, but there are endless use cases for a custom, highly > scalable in memory platform. > > The goal of this proposal is to leverage an existing code base available on > Github and licensed under the Apache License 2.0 to build a community > around the project. Currently the community consists of existing hackers of > Concerted as well as people who have been following and associated with the > project since a while as well as database experts who are excited about > building a project like this. We are hoping that entering into Apache would > help us attract more contributors as well as connect with existing big data > projects like Apache Hive, Apache HAWQ, Apache Storm, Apache Tajo, Apache > Spark, Apache Geode to leverage their community base while assisting in > their use cases with Concerted. We had a discussion with founders of Apache > Tajo and they showed interest in using Concerted for some of their use > cases. > = Background = > Relational databases were built with the cost of physical memory in mind. > The cost is no longer very relevant and physical memory is now available on > demand. Another driving factor behind Concerted is that there is a paradigm > shift with big data coming into picture. Disk IO speeds are more of a > bottleneck than ever before. Combining the read dominance of analytical > workload with the speed of in memory structures, Concerted fits the current > scene. Also, supporting OLAP workloads with in memory support for faster > read constant queries and joins will be useful. > > = Rationale = > As explained above, large analytical workloads need an in memory > lightweight engine which supports massive read concurrency, ground level > support for aggregations and analytics, extreme scalability and high read > performance, along with the engine being very light itself. Concerted aims > to solve these needs. Concerted is designed and built with three goals as > objectives: > > > Performance > To provide high performance access to data from a large number of rows, > Concerted uses efficient representation and in memory indexing of data > coupled with high performance transactions, custom transactions and > lightweight locking and lockless techniques and an intelligent locking > manager. > > Scalability > Concerted is built with extreme concurrency and scalability in mind. > > Efficiency > Concerted aims to give expected performance under vast variety of > workloads and aims to have as low footprint as possible. > > = Initial Goals = > The initial goal is to leverage an existing code base and invest in > building a community around the project. We anticipate a lot of initial > restructuring of the existing code so that it becomes easier to
Re: [VOTE] Accept Concerted into the Apache Incubator
+1 (non-binding) Amol On Fri, Oct 9, 2015 at 8:55 AM, Atri Sharma wrote: > Hi all, > > Following the discussion about Concerted I would like to call a vote for > accepting Concerted as a new incubator project. > > The proposal text is included below, and available on the wiki: > > https://wiki.apache.org/incubator/ConcertedProposal > > The vote is open for 72 hours: > > [ ] +1 accept Concerted in the Incubator > [ ] ±0 > [ ] -1 (please give reason) > > Regards, > > Atri > > = Abstract = > > Concerted is an in memory write less read more engine aimed to provide > extreme read performance with very high degree of concurrency and > scalability and focus on minimizing own resource footprint. > > = Proposal = > Concerted is built on the principal that a new type of workload is > dominating the scene and is now needed to be supported. These are the large > data set analytical workloads being analyzed or used on large clusters or > high power machines. Large analytical workloads depend on the ability to > query large data sets efficiently and in high concurrency while maintaining > semantics such as immediate consistency. An in memory engine designed to > support extreme read queries while providing support for aggregation > through various features (such as multidimensional representation of > tuples) will accelerate many usecases around large scale analytics. > > Concerted believes that best understanding of user application lies with > user application developer. The need for massive read scaling should be on > demand and should be flexible to the level that user can decide as to which > representation and access of data suits his/her current requirements. > Hence, Concerted is not built in a traditional client/server model. > Concerted provides users with an API which can be used to load, read, > update and delete data. User chooses which data structure has to be used > for his current requirements. All API access is covered by Concerted's > internal systems like lock manager, transaction manager and cache manager > which ensure that reads scale to high level in every API call. > > Concerted is a Do It Yourself in memory platform for making in memory > supporting engines. The use case we think of is supporting big data > warehouses like Hive, but there are endless use cases for a custom, highly > scalable in memory platform. > > The goal of this proposal is to leverage an existing code base available on > Github and licensed under the Apache License 2.0 to build a community > around the project. Currently the community consists of existing hackers of > Concerted as well as people who have been following and associated with the > project since a while as well as database experts who are excited about > building a project like this. We are hoping that entering into Apache would > help us attract more contributors as well as connect with existing big data > projects like Apache Hive, Apache HAWQ, Apache Storm, Apache Tajo, Apache > Spark, Apache Geode to leverage their community base while assisting in > their use cases with Concerted. We had a discussion with founders of Apache > Tajo and they showed interest in using Concerted for some of their use > cases. > = Background = > Relational databases were built with the cost of physical memory in mind. > The cost is no longer very relevant and physical memory is now available on > demand. Another driving factor behind Concerted is that there is a paradigm > shift with big data coming into picture. Disk IO speeds are more of a > bottleneck than ever before. Combining the read dominance of analytical > workload with the speed of in memory structures, Concerted fits the current > scene. Also, supporting OLAP workloads with in memory support for faster > read constant queries and joins will be useful. > > = Rationale = > As explained above, large analytical workloads need an in memory > lightweight engine which supports massive read concurrency, ground level > support for aggregations and analytics, extreme scalability and high read > performance, along with the engine being very light itself. Concerted aims > to solve these needs. Concerted is designed and built with three goals as > objectives: > > > Performance > To provide high performance access to data from a large number of rows, > Concerted uses efficient representation and in memory indexing of data > coupled with high performance transactions, custom transactions and > lightweight locking and lockless techniques and an intelligent locking > manager. > > Scalability > Concerted is built with extreme concurrency and scalability in mind. > > Efficiency > Concerted aims to give expected performance under vast variety of > workloads and aims to have as low footprint as possible. > > = Initial Goals = > The initial goal is to leverage an existing code base and invest in > building a community around the project. We anticipate a lot of initial > restructuring of the existing code so that it becomes
[VOTE] Accept Concerted into the Apache Incubator
Hi all, Following the discussion about Concerted I would like to call a vote for accepting Concerted as a new incubator project. The proposal text is included below, and available on the wiki: https://wiki.apache.org/incubator/ConcertedProposal The vote is open for 72 hours: [ ] +1 accept Concerted in the Incubator [ ] ±0 [ ] -1 (please give reason) Regards, Atri = Abstract = Concerted is an in memory write less read more engine aimed to provide extreme read performance with very high degree of concurrency and scalability and focus on minimizing own resource footprint. = Proposal = Concerted is built on the principal that a new type of workload is dominating the scene and is now needed to be supported. These are the large data set analytical workloads being analyzed or used on large clusters or high power machines. Large analytical workloads depend on the ability to query large data sets efficiently and in high concurrency while maintaining semantics such as immediate consistency. An in memory engine designed to support extreme read queries while providing support for aggregation through various features (such as multidimensional representation of tuples) will accelerate many usecases around large scale analytics. Concerted believes that best understanding of user application lies with user application developer. The need for massive read scaling should be on demand and should be flexible to the level that user can decide as to which representation and access of data suits his/her current requirements. Hence, Concerted is not built in a traditional client/server model. Concerted provides users with an API which can be used to load, read, update and delete data. User chooses which data structure has to be used for his current requirements. All API access is covered by Concerted's internal systems like lock manager, transaction manager and cache manager which ensure that reads scale to high level in every API call. Concerted is a Do It Yourself in memory platform for making in memory supporting engines. The use case we think of is supporting big data warehouses like Hive, but there are endless use cases for a custom, highly scalable in memory platform. The goal of this proposal is to leverage an existing code base available on Github and licensed under the Apache License 2.0 to build a community around the project. Currently the community consists of existing hackers of Concerted as well as people who have been following and associated with the project since a while as well as database experts who are excited about building a project like this. We are hoping that entering into Apache would help us attract more contributors as well as connect with existing big data projects like Apache Hive, Apache HAWQ, Apache Storm, Apache Tajo, Apache Spark, Apache Geode to leverage their community base while assisting in their use cases with Concerted. We had a discussion with founders of Apache Tajo and they showed interest in using Concerted for some of their use cases. = Background = Relational databases were built with the cost of physical memory in mind. The cost is no longer very relevant and physical memory is now available on demand. Another driving factor behind Concerted is that there is a paradigm shift with big data coming into picture. Disk IO speeds are more of a bottleneck than ever before. Combining the read dominance of analytical workload with the speed of in memory structures, Concerted fits the current scene. Also, supporting OLAP workloads with in memory support for faster read constant queries and joins will be useful. = Rationale = As explained above, large analytical workloads need an in memory lightweight engine which supports massive read concurrency, ground level support for aggregations and analytics, extreme scalability and high read performance, along with the engine being very light itself. Concerted aims to solve these needs. Concerted is designed and built with three goals as objectives: Performance To provide high performance access to data from a large number of rows, Concerted uses efficient representation and in memory indexing of data coupled with high performance transactions, custom transactions and lightweight locking and lockless techniques and an intelligent locking manager. Scalability Concerted is built with extreme concurrency and scalability in mind. Efficiency Concerted aims to give expected performance under vast variety of workloads and aims to have as low footprint as possible. = Initial Goals = The initial goal is to leverage an existing code base and invest in building a community around the project. We anticipate a lot of initial restructuring of the existing code so that it becomes easier to include new contributors and minimize ramp up time. We plan to approach this refactoring in a fully transparent, community-driven way thus starting to practice the "Apache Way" governance model from the get go. Various contributors are getting indiv
