well it is still network attached, If you allocate enough heap to fit the whole thing in memory (in hbase/conf/hbase-env.sh) you could probably eliminate this as a possible reason
On Sat, Sep 6, 2014 at 2:43 AM, Vikas Agarwal <[email protected]> wrote: > EBS but with new generation SSD not magnetic one. > > > On Sat, Sep 6, 2014 at 12:11 PM, Alex Kamil <[email protected]> wrote: > >> do you use EBS or ephemeral storage, I found EBS performance to be >> somewhat unpredictable >> >> >> On Sat, Sep 6, 2014 at 2:37 AM, Vikas Agarwal <[email protected]> >> wrote: >> >>> Hbase is 0.98.0 >>> Phoenix is 4.0 >>> >>> >>> On Sat, Sep 6, 2014 at 12:04 PM, Vikas Agarwal <[email protected]> >>> wrote: >>> >>>> Yes, that is why it is a trouble for me. However, on contrary, HBase >>>> shell is also on the same machine and same environment, so if it is an >>>> issue of resource (CPU or memory) it should have affected the HBase too, >>>> but HBase is able to give me results within 0.0150 seconds. :( >>>> >>>> No, I haven't tested it outside AWS. I guess, it should not be the case >>>> due to much better performance by native HBase query on HBase shell. >>>> >>>> >>>> On Sat, Sep 6, 2014 at 11:59 AM, James Taylor <[email protected]> >>>> wrote: >>>> >>>>> Something is up in your environment. What version of Phoenix and HBase >>>>> are you using and in what environment? Have you tried this locally, >>>>> outside of AWS to compare? >>>>> >>>>> Take a look at our perf numbers, generated more-or-less daily, and >>>>> which run over more data that what you're testing against: >>>>> >>>>> http://phoenix-bin.github.io/client/performance/phoenix-20140904095313.htm >>>>> >>>>> Some of these are point queries and they take in the neighborhood of >>>>> 0.01 seconds. >>>>> >>>>> Thanks, >>>>> James >>>>> >>>>> On Fri, Sep 5, 2014 at 10:48 PM, Vikas Agarwal <[email protected]> >>>>> wrote: >>>>> > Missed to mention that count query (posted in my last mail) is also >>>>> taking >>>>> > very long time to return the count. >>>>> > >>>>> > >>>>> > On Sat, Sep 6, 2014 at 11:17 AM, Vikas Agarwal < >>>>> [email protected]> >>>>> > wrote: >>>>> >> >>>>> >> As I mentioned, schema is nothing but bunch of fields (some being >>>>> >> integers, longs and text) along with primary key (row key) and I am >>>>> making >>>>> >> simple query to get result for a particular primary key, nothing >>>>> more than >>>>> >> that. >>>>> >> >>>>> >> 0: jdbc:phoenix:localhost> SELECT count(1) FROM table_name; >>>>> >> >>>>> >> +------------+ >>>>> >> >>>>> >> | COUNT(1) | >>>>> >> >>>>> >> +------------+ >>>>> >> >>>>> >> | 4667515 | >>>>> >> >>>>> >> +------------+ >>>>> >> >>>>> >> 1 row selected (132.11 seconds) >>>>> >> >>>>> >> >>>>> >> >>>>> >> On Sat, Sep 6, 2014 at 11:09 AM, Puneet Kumar Ojha >>>>> >> <[email protected]> wrote: >>>>> >>> >>>>> >>> If you can share the schema,data type,cardinality of each >>>>> dimension and >>>>> >>> usual queries, I can help to design a schema with performance of >>>>> less than 1 >>>>> >>> sec using Phoenix. >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> Thanks >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> ------ Original message------ >>>>> >>> >>>>> >>> From: James Taylor >>>>> >>> >>>>> >>> Date: Sat, Sep 6, 2014 10:15 AM >>>>> >>> >>>>> >>> To: user; >>>>> >>> >>>>> >>> Subject:Re: Phoenix response time >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> Vikas, >>>>> >>> Please post your schema and query. >>>>> >>> Thanks, >>>>> >>> James >>>>> >>> >>>>> >>> On Fri, Sep 5, 2014 at 9:18 PM, Vikas Agarwal < >>>>> [email protected]> >>>>> >>> wrote: >>>>> >>> > Ours is also a single node setup right now and as of now there >>>>> are less >>>>> >>> > than >>>>> >>> > 1 million rows which is expected to grow around 100m at minimum. >>>>> >>> > >>>>> >>> > I am aware of secondary indexes but when I am querying on >>>>> primary/row >>>>> >>> > key, >>>>> >>> > why would it take so much time? >>>>> >>> > >>>>> >>> > I am directly querying using sqlline for Phoenix and hbase shell >>>>> for >>>>> >>> > HBase >>>>> >>> > query. I am not expecting to do any fine tuning for such small >>>>> dataset. >>>>> >>> > I am >>>>> >>> > assumimg a minimum performance level out of the box. >>>>> >>> > >>>>> >>> > On Friday, September 5, 2014, yeshwanth kumar < >>>>> [email protected]> >>>>> >>> > wrote: >>>>> >>> >> >>>>> >>> >> hi vikas, >>>>> >>> >> >>>>> >>> >> we used phoenix on a 4 core/23Gb machine, as a single node >>>>> setup. >>>>> >>> >> used HDP 2.1 >>>>> >>> >> our table has 50-70M rows, >>>>> >>> >> select on that table took less than 2 seconds. >>>>> >>> >> Aggregation queries took less than 8 seconds. >>>>> >>> >> for achieving good performance we created secondary index on the >>>>> >>> >> table. >>>>> >>> >> >>>>> >>> >> make sure you finetuned hbase, >>>>> >>> >> enabling compression on the data makes a difference in response. >>>>> >>> >> if u distribute the data and load over all regions in hbase, >>>>> >>> >> look at the performance tips mentioned in phoenix blog >>>>> >>> >> >>>>> >>> >> -yeshwanth >>>>> >>> >> >>>>> >>> >> >>>>> >>> >> >>>>> >>> >> Cheers, >>>>> >>> >> Yeshwanth >>>>> >>> >> >>>>> >>> >> >>>>> >>> >> >>>>> >>> >> On Fri, Sep 5, 2014 at 5:42 PM, Vikas Agarwal < >>>>> [email protected]> >>>>> >>> >> wrote: >>>>> >>> >>> >>>>> >>> >>> Hi, >>>>> >>> >>> >>>>> >>> >>> Preface: We are testing phoenix using Hortonworks distribution >>>>> for >>>>> >>> >>> HBase >>>>> >>> >>> on Amazon EC2 instance (r3.large, 2 CPU/15 GB RAM). >>>>> >>> >>> >>>>> >>> >>> With contrast to performance benchmarks, I found Phoenix to be >>>>> very >>>>> >>> >>> slow >>>>> >>> >>> in querying even on primary key or row key. So, tried to >>>>> increase the >>>>> >>> >>> RAM >>>>> >>> >>> for HBase and Phoenix and increasing the CPU and RAM by >>>>> upgrading the >>>>> >>> >>> EC2 >>>>> >>> >>> machine type to r3.xlarge (4 CPU, 30 GB RAM). Results were >>>>> like this: >>>>> >>> >>> >>>>> >>> >>> Time takes in returning result of query on row key: >>>>> >>> >>> With Storm running and very less RAM available: 50 sec >>>>> >>> >>> >>>>> >>> >>> With Storm stopped and RAM available to Phoenix and HBase: 18 >>>>> sec >>>>> >>> >>> >>>>> >>> >>> With new machine of next higher category (4 CPU and 30 GB >>>>> RAM): 8 sec >>>>> >>> >>> >>>>> >>> >>> Pure HBase query by row key with Storm stopped and (2 CPU, 15 >>>>> GB >>>>> >>> >>> RAM): >>>>> >>> >>> 0.0150 seconds. :) >>>>> >>> >>> >>>>> >>> >>> So, the difference seems to be many fold of what native HBase >>>>> is >>>>> >>> >>> providing to us. I am not able to understand how it can be >>>>> possible? >>>>> >>> >>> What I >>>>> >>> >>> am missing here? >>>>> >>> >>> >>>>> >>> >>> -- >>>>> >>> >>> Regards, >>>>> >>> >>> Vikas Agarwal >>>>> >>> >>> 91 – 9928301411 >>>>> >>> >>> >>>>> >>> >>> InfoObjects, Inc. >>>>> >>> >>> Execution Matters >>>>> >>> >>> http://www.infoobjects.com >>>>> >>> >>> 2041 Mission College Boulevard, #280 >>>>> >>> >>> Santa Clara, CA 95054 >>>>> >>> >>> +1 (408) 988-2000 Work >>>>> >>> >>> +1 (408) 716-2726 Fax >>>>> >>> >> >>>>> >>> >> >>>>> >>> > >>>>> >>> > >>>>> >>> > -- >>>>> >>> > Regards, >>>>> >>> > Vikas Agarwal >>>>> >>> > 91 – 9928301411 >>>>> >>> > >>>>> >>> > InfoObjects, Inc. >>>>> >>> > Execution Matters >>>>> >>> > http://www.infoobjects.com >>>>> >>> > 2041 Mission College Boulevard, #280 >>>>> >>> > Santa Clara, CA 95054 >>>>> >>> > +1 (408) 988-2000 Work >>>>> >>> > +1 (408) 716-2726 Fax >>>>> >>> > >>>>> >>> > >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> -- >>>>> >> Regards, >>>>> >> Vikas Agarwal >>>>> >> 91 – 9928301411 >>>>> >> >>>>> >> InfoObjects, Inc. >>>>> >> Execution Matters >>>>> >> http://www.infoobjects.com >>>>> >> 2041 Mission College Boulevard, #280 >>>>> >> Santa Clara, CA 95054 >>>>> >> +1 (408) 988-2000 Work >>>>> >> +1 (408) 716-2726 Fax >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > -- >>>>> > Regards, >>>>> > Vikas Agarwal >>>>> > 91 – 9928301411 >>>>> > >>>>> > InfoObjects, Inc. >>>>> > Execution Matters >>>>> > http://www.infoobjects.com >>>>> > 2041 Mission College Boulevard, #280 >>>>> > Santa Clara, CA 95054 >>>>> > +1 (408) 988-2000 Work >>>>> > +1 (408) 716-2726 Fax >>>>> >>>> >>>> >>>> >>>> -- >>>> Regards, >>>> Vikas Agarwal >>>> 91 – 9928301411 >>>> >>>> InfoObjects, Inc. >>>> Execution Matters >>>> http://www.infoobjects.com >>>> 2041 Mission College Boulevard, #280 >>>> Santa Clara, CA 95054 >>>> +1 (408) 988-2000 Work >>>> +1 (408) 716-2726 Fax >>>> >>>> >>> >>> >>> -- >>> Regards, >>> Vikas Agarwal >>> 91 – 9928301411 >>> >>> InfoObjects, Inc. >>> Execution Matters >>> http://www.infoobjects.com >>> 2041 Mission College Boulevard, #280 >>> Santa Clara, CA 95054 >>> +1 (408) 988-2000 Work >>> +1 (408) 716-2726 Fax >>> >>> >> > > > -- > Regards, > Vikas Agarwal > 91 – 9928301411 > > InfoObjects, Inc. > Execution Matters > http://www.infoobjects.com > 2041 Mission College Boulevard, #280 > Santa Clara, CA 95054 > +1 (408) 988-2000 Work > +1 (408) 716-2726 Fax > >
