Re: [ANNOUNCE] Apache Phoenix 4.8.0 released
I am wondering why many of these interesting features are not listed in official release notes. https://phoenix.apache.org/release_notes.html Best, Afshin On Aug 12, 2016, at 1:25 PM, Ankit Singhal mailto:an...@apache.org>> wrote: Apache Phoenix enables OLTP and operational analytics for Hadoop through SQL support and integration with other projects in the ecosystem such as Spark, HBase, Pig, Flume, MapReduce and Hive. We're pleased to announce our 4.8.0 release which includes: - Local Index improvements[1] - Integration with hive[2] - Namespace mapping support[3] - VIEW enhancements[4] - Offset support for paged queries[5] - 130+ Bugs resolved[6] - HBase v1.2 is also supported ( with continued support for v1.1, v1.0 & v0.98) - Many performance enhancements(related to StatsCache, distinct, Serial query with Stats etc)[6] The release is available in source or binary form here [7]. Release artifacts are signed with the following key: https://people.apache.org/keys/committer/ankit.asc Thanks, The Apache Phoenix Team [1] https://issues.apache.org/jira/browse/PHOENIX-1734 [2] https://issues.apache.org/jira/browse/PHOENIX-2743 [3] https://issues.apache.org/jira/browse/PHOENIX-1311 [4] https://issues.apache.org/jira/browse/PHOENIX-1508 [5] https://issues.apache.org/jira/browse/PHOENIX-2722 [6] https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12334393&projectId=12315120 [7] https://phoenix.apache.org/download.html
Hbase federated cluster for messages
Hello, I am not sure how to do it but I have to configure federated cluster with hbase to store huge amount of messages (client to client) (40% writes, 60% reads). Does somebody have any idea or examples how to configure it? Of course we can configure hdfs in a federated mode but as for me it isn't suitable for hbase. If we want to save message from client 1 to client 2 in the hbase cluster then how hbase know in which namespace it have to save it? Which namenode will be responsible for that message? How we can read client messages? Give me any ideas, please Sincerely, Alexandr
Re: Hbase federated cluster for messages
As far as I know, HBase doesn't support spreading tables across namespaces; you'd have to point it at one namenode at a time. I've heard of people trying to run multiple HBase instances in order to get access to all their HDFS data, but it doesn't tend to be much fun. -Dima On Fri, Aug 19, 2016 at 11:51 AM, Alexandr Porunov < alexandr.poru...@gmail.com> wrote: > Hello, > > I am not sure how to do it but I have to configure federated cluster with > hbase to store huge amount of messages (client to client) (40% writes, 60% > reads). Does somebody have any idea or examples how to configure it? > > Of course we can configure hdfs in a federated mode but as for me it isn't > suitable for hbase. If we want to save message from client 1 to client 2 in > the hbase cluster then how hbase know in which namespace it have to save > it? Which namenode will be responsible for that message? How we can read > client messages? > > Give me any ideas, please > > Sincerely, > Alexandr > -- -Dima
Re: Hbase federated cluster for messages
Hi Dima, But isn't it a bottleneck then? Our throughput limited by a single namenode server? Sincerely, Alexandr On Fri, Aug 19, 2016 at 9:57 PM, Dima Spivak wrote: > As far as I know, HBase doesn't support spreading tables across namespaces; > you'd have to point it at one namenode at a time. I've heard of people > trying to run multiple HBase instances in order to get access to all their > HDFS data, but it doesn't tend to be much fun. > > -Dima > > On Fri, Aug 19, 2016 at 11:51 AM, Alexandr Porunov < > alexandr.poru...@gmail.com> wrote: > > > Hello, > > > > I am not sure how to do it but I have to configure federated cluster with > > hbase to store huge amount of messages (client to client) (40% writes, > 60% > > reads). Does somebody have any idea or examples how to configure it? > > > > Of course we can configure hdfs in a federated mode but as for me it > isn't > > suitable for hbase. If we want to save message from client 1 to client 2 > in > > the hbase cluster then how hbase know in which namespace it have to save > > it? Which namenode will be responsible for that message? How we can read > > client messages? > > > > Give me any ideas, please > > > > Sincerely, > > Alexandr > > > > > > -- > -Dima >
Re: Hbase federated cluster for messages
>> I am not sure how to do it but I have to configure federated cluster with >> hbase to store huge amount of messages (client to client) (40% writes, 60% >> reads). Any particular reason for federated cluster? How huge is huge amount and what is the message size? -Vladimir On Fri, Aug 19, 2016 at 11:57 AM, Dima Spivak wrote: > As far as I know, HBase doesn't support spreading tables across namespaces; > you'd have to point it at one namenode at a time. I've heard of people > trying to run multiple HBase instances in order to get access to all their > HDFS data, but it doesn't tend to be much fun. > > -Dima > > On Fri, Aug 19, 2016 at 11:51 AM, Alexandr Porunov < > alexandr.poru...@gmail.com> wrote: > > > Hello, > > > > I am not sure how to do it but I have to configure federated cluster with > > hbase to store huge amount of messages (client to client) (40% writes, > 60% > > reads). Does somebody have any idea or examples how to configure it? > > > > Of course we can configure hdfs in a federated mode but as for me it > isn't > > suitable for hbase. If we want to save message from client 1 to client 2 > in > > the hbase cluster then how hbase know in which namespace it have to save > > it? Which namenode will be responsible for that message? How we can read > > client messages? > > > > Give me any ideas, please > > > > Sincerely, > > Alexandr > > > > > > -- > -Dima >
Re: Hbase federated cluster for messages
No. There isn't. But I want to figure out how to configure that type of cluster in the case if there is particular reason. How facebook can handle such a huge amount of ops without federation? I don't think that they just have one namenode server and one standby namenode server. It isn't possible. I am sure that they use federation. On Fri, Aug 19, 2016 at 10:08 PM, Vladimir Rodionov wrote: > >> I am not sure how to do it but I have to configure federated cluster > with > >> hbase to store huge amount of messages (client to client) (40% writes, > 60% > >> reads). > > Any particular reason for federated cluster? How huge is huge amount and > what is the message size? > > -Vladimir > > On Fri, Aug 19, 2016 at 11:57 AM, Dima Spivak > wrote: > > > As far as I know, HBase doesn't support spreading tables across > namespaces; > > you'd have to point it at one namenode at a time. I've heard of people > > trying to run multiple HBase instances in order to get access to all > their > > HDFS data, but it doesn't tend to be much fun. > > > > -Dima > > > > On Fri, Aug 19, 2016 at 11:51 AM, Alexandr Porunov < > > alexandr.poru...@gmail.com> wrote: > > > > > Hello, > > > > > > I am not sure how to do it but I have to configure federated cluster > with > > > hbase to store huge amount of messages (client to client) (40% writes, > > 60% > > > reads). Does somebody have any idea or examples how to configure it? > > > > > > Of course we can configure hdfs in a federated mode but as for me it > > isn't > > > suitable for hbase. If we want to save message from client 1 to client > 2 > > in > > > the hbase cluster then how hbase know in which namespace it have to > save > > > it? Which namenode will be responsible for that message? How we can > read > > > client messages? > > > > > > Give me any ideas, please > > > > > > Sincerely, > > > Alexandr > > > > > > > > > > > -- > > -Dima > > >
Re: Hbase federated cluster for messages
FB has its own "federation". It is a proprietary code, I presume. -Vladimir On Fri, Aug 19, 2016 at 1:22 PM, Alexandr Porunov < alexandr.poru...@gmail.com> wrote: > No. There isn't. But I want to figure out how to configure that type of > cluster in the case if there is particular reason. How facebook can handle > such a huge amount of ops without federation? I don't think that they just > have one namenode server and one standby namenode server. It isn't > possible. I am sure that they use federation. > > On Fri, Aug 19, 2016 at 10:08 PM, Vladimir Rodionov < > vladrodio...@gmail.com> > wrote: > > > >> I am not sure how to do it but I have to configure federated cluster > > with > > >> hbase to store huge amount of messages (client to client) (40% writes, > > 60% > > >> reads). > > > > Any particular reason for federated cluster? How huge is huge amount and > > what is the message size? > > > > -Vladimir > > > > On Fri, Aug 19, 2016 at 11:57 AM, Dima Spivak > > wrote: > > > > > As far as I know, HBase doesn't support spreading tables across > > namespaces; > > > you'd have to point it at one namenode at a time. I've heard of people > > > trying to run multiple HBase instances in order to get access to all > > their > > > HDFS data, but it doesn't tend to be much fun. > > > > > > -Dima > > > > > > On Fri, Aug 19, 2016 at 11:51 AM, Alexandr Porunov < > > > alexandr.poru...@gmail.com> wrote: > > > > > > > Hello, > > > > > > > > I am not sure how to do it but I have to configure federated cluster > > with > > > > hbase to store huge amount of messages (client to client) (40% > writes, > > > 60% > > > > reads). Does somebody have any idea or examples how to configure it? > > > > > > > > Of course we can configure hdfs in a federated mode but as for me it > > > isn't > > > > suitable for hbase. If we want to save message from client 1 to > client > > 2 > > > in > > > > the hbase cluster then how hbase know in which namespace it have to > > save > > > > it? Which namenode will be responsible for that message? How we can > > read > > > > client messages? > > > > > > > > Give me any ideas, please > > > > > > > > Sincerely, > > > > Alexandr > > > > > > > > > > > > > > > > -- > > > -Dima > > > > > >
Re: Hbase federated cluster for messages
I'd +1 what Vladimir says. How much data (in TBs/PBs) and how many files are we talking about here? I'd say that use cases that benefit from HBase don't tend to hit the kind of HDFS file limits that federation seeks to address. -Dima On Fri, Aug 19, 2016 at 2:19 PM, Vladimir Rodionov wrote: > FB has its own "federation". It is a proprietary code, I presume. > > -Vladimir > > > On Fri, Aug 19, 2016 at 1:22 PM, Alexandr Porunov < > alexandr.poru...@gmail.com> wrote: > > > No. There isn't. But I want to figure out how to configure that type of > > cluster in the case if there is particular reason. How facebook can > handle > > such a huge amount of ops without federation? I don't think that they > just > > have one namenode server and one standby namenode server. It isn't > > possible. I am sure that they use federation. > > > > On Fri, Aug 19, 2016 at 10:08 PM, Vladimir Rodionov < > > vladrodio...@gmail.com> > > wrote: > > > > > >> I am not sure how to do it but I have to configure federated cluster > > > with > > > >> hbase to store huge amount of messages (client to client) (40% > writes, > > > 60% > > > >> reads). > > > > > > Any particular reason for federated cluster? How huge is huge amount > and > > > what is the message size? > > > > > > -Vladimir > > > > > > On Fri, Aug 19, 2016 at 11:57 AM, Dima Spivak > > > wrote: > > > > > > > As far as I know, HBase doesn't support spreading tables across > > > namespaces; > > > > you'd have to point it at one namenode at a time. I've heard of > people > > > > trying to run multiple HBase instances in order to get access to all > > > their > > > > HDFS data, but it doesn't tend to be much fun. > > > > > > > > -Dima > > > > > > > > On Fri, Aug 19, 2016 at 11:51 AM, Alexandr Porunov < > > > > alexandr.poru...@gmail.com> wrote: > > > > > > > > > Hello, > > > > > > > > > > I am not sure how to do it but I have to configure federated > cluster > > > with > > > > > hbase to store huge amount of messages (client to client) (40% > > writes, > > > > 60% > > > > > reads). Does somebody have any idea or examples how to configure > it? > > > > > > > > > > Of course we can configure hdfs in a federated mode but as for me > it > > > > isn't > > > > > suitable for hbase. If we want to save message from client 1 to > > client > > > 2 > > > > in > > > > > the hbase cluster then how hbase know in which namespace it have to > > > save > > > > > it? Which namenode will be responsible for that message? How we can > > > read > > > > > client messages? > > > > > > > > > > Give me any ideas, please > > > > > > > > > > Sincerely, > > > > > Alexandr > > > > > > > > > > > > > > > > > > > > > -- > > > > -Dima > > > > > > > > > > -- -Dima
Re: Hbase federated cluster for messages
We are talking about facebook. So, there are 25 TB per month. 15 billion messages with 1024 bytes and 120 billion messages with 100 bytes per month. I thought that they used only hbase to handle such a huge data If they used their own implementation of hbase then I haven't questions. Sincerely, Alexandr On Sat, Aug 20, 2016 at 1:39 AM, Dima Spivak wrote: > I'd +1 what Vladimir says. How much data (in TBs/PBs) and how many files > are we talking about here? I'd say that use cases that benefit from HBase > don't tend to hit the kind of HDFS file limits that federation seeks to > address. > > -Dima > > On Fri, Aug 19, 2016 at 2:19 PM, Vladimir Rodionov > > wrote: > > > FB has its own "federation". It is a proprietary code, I presume. > > > > -Vladimir > > > > > > On Fri, Aug 19, 2016 at 1:22 PM, Alexandr Porunov < > > alexandr.poru...@gmail.com> wrote: > > > > > No. There isn't. But I want to figure out how to configure that type of > > > cluster in the case if there is particular reason. How facebook can > > handle > > > such a huge amount of ops without federation? I don't think that they > > just > > > have one namenode server and one standby namenode server. It isn't > > > possible. I am sure that they use federation. > > > > > > On Fri, Aug 19, 2016 at 10:08 PM, Vladimir Rodionov < > > > vladrodio...@gmail.com> > > > wrote: > > > > > > > >> I am not sure how to do it but I have to configure federated > cluster > > > > with > > > > >> hbase to store huge amount of messages (client to client) (40% > > writes, > > > > 60% > > > > >> reads). > > > > > > > > Any particular reason for federated cluster? How huge is huge amount > > and > > > > what is the message size? > > > > > > > > -Vladimir > > > > > > > > On Fri, Aug 19, 2016 at 11:57 AM, Dima Spivak > > > > wrote: > > > > > > > > > As far as I know, HBase doesn't support spreading tables across > > > > namespaces; > > > > > you'd have to point it at one namenode at a time. I've heard of > > people > > > > > trying to run multiple HBase instances in order to get access to > all > > > > their > > > > > HDFS data, but it doesn't tend to be much fun. > > > > > > > > > > -Dima > > > > > > > > > > On Fri, Aug 19, 2016 at 11:51 AM, Alexandr Porunov < > > > > > alexandr.poru...@gmail.com> wrote: > > > > > > > > > > > Hello, > > > > > > > > > > > > I am not sure how to do it but I have to configure federated > > cluster > > > > with > > > > > > hbase to store huge amount of messages (client to client) (40% > > > writes, > > > > > 60% > > > > > > reads). Does somebody have any idea or examples how to configure > > it? > > > > > > > > > > > > Of course we can configure hdfs in a federated mode but as for me > > it > > > > > isn't > > > > > > suitable for hbase. If we want to save message from client 1 to > > > client > > > > 2 > > > > > in > > > > > > the hbase cluster then how hbase know in which namespace it have > to > > > > save > > > > > > it? Which namenode will be responsible for that message? How we > can > > > > read > > > > > > client messages? > > > > > > > > > > > > Give me any ideas, please > > > > > > > > > > > > Sincerely, > > > > > > Alexandr > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > -Dima > > > > > > > > > > > > > > > > > > -- > -Dima >
Re: Hbase federated cluster for messages
You can easily store that much data as long as you don't have small files, which is typically why people turn to federation. -Dima On Friday, August 19, 2016, Alexandr Porunov wrote: > We are talking about facebook. So, there are 25 TB per month. 15 billion > messages with 1024 bytes and 120 billion messages with 100 bytes per month. > > I thought that they used only hbase to handle such a huge data If they used > their own implementation of hbase then I haven't questions. > > Sincerely, > Alexandr > > On Sat, Aug 20, 2016 at 1:39 AM, Dima Spivak > wrote: > > > I'd +1 what Vladimir says. How much data (in TBs/PBs) and how many files > > are we talking about here? I'd say that use cases that benefit from HBase > > don't tend to hit the kind of HDFS file limits that federation seeks to > > address. > > > > -Dima > > > > On Fri, Aug 19, 2016 at 2:19 PM, Vladimir Rodionov < > vladrodio...@gmail.com > > > > > wrote: > > > > > FB has its own "federation". It is a proprietary code, I presume. > > > > > > -Vladimir > > > > > > > > > On Fri, Aug 19, 2016 at 1:22 PM, Alexandr Porunov < > > > alexandr.poru...@gmail.com > wrote: > > > > > > > No. There isn't. But I want to figure out how to configure that type > of > > > > cluster in the case if there is particular reason. How facebook can > > > handle > > > > such a huge amount of ops without federation? I don't think that they > > > just > > > > have one namenode server and one standby namenode server. It isn't > > > > possible. I am sure that they use federation. > > > > > > > > On Fri, Aug 19, 2016 at 10:08 PM, Vladimir Rodionov < > > > > vladrodio...@gmail.com > > > > > wrote: > > > > > > > > > >> I am not sure how to do it but I have to configure federated > > cluster > > > > > with > > > > > >> hbase to store huge amount of messages (client to client) (40% > > > writes, > > > > > 60% > > > > > >> reads). > > > > > > > > > > Any particular reason for federated cluster? How huge is huge > amount > > > and > > > > > what is the message size? > > > > > > > > > > -Vladimir > > > > > > > > > > On Fri, Aug 19, 2016 at 11:57 AM, Dima Spivak < > dspi...@cloudera.com > > > > > > wrote: > > > > > > > > > > > As far as I know, HBase doesn't support spreading tables across > > > > > namespaces; > > > > > > you'd have to point it at one namenode at a time. I've heard of > > > people > > > > > > trying to run multiple HBase instances in order to get access to > > all > > > > > their > > > > > > HDFS data, but it doesn't tend to be much fun. > > > > > > > > > > > > -Dima > > > > > > > > > > > > On Fri, Aug 19, 2016 at 11:51 AM, Alexandr Porunov < > > > > > > alexandr.poru...@gmail.com > wrote: > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > I am not sure how to do it but I have to configure federated > > > cluster > > > > > with > > > > > > > hbase to store huge amount of messages (client to client) (40% > > > > writes, > > > > > > 60% > > > > > > > reads). Does somebody have any idea or examples how to > configure > > > it? > > > > > > > > > > > > > > Of course we can configure hdfs in a federated mode but as for > me > > > it > > > > > > isn't > > > > > > > suitable for hbase. If we want to save message from client 1 to > > > > client > > > > > 2 > > > > > > in > > > > > > > the hbase cluster then how hbase know in which namespace it > have > > to > > > > > save > > > > > > > it? Which namenode will be responsible for that message? How we > > can > > > > > read > > > > > > > client messages? > > > > > > > > > > > > > > Give me any ideas, please > > > > > > > > > > > > > > Sincerely, > > > > > > > Alexandr > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > -Dima > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > -Dima > > > -- -Dima
