Hi, Deepa, Sorry, looks like this message somehow got neglected..
Here is a suggestion for tracking down the memory usage. -- add '-v 10' to your command line for ibis you should be able to see Warning message before the crash about requesting a large amounts of memory. Roughly, if you have two tables with 1 million rows each, and you want to generate a full cross product (e.g., a join), you have to be prepared to deal with 1 million times 1 million rows. Looks like you are providing some restrictions on the tables, which should reduce the number of rows in the cross-product table. If you can run the selection conditions on each table separately, you should be able to get a upper bound on the number of rows in the cross product table. Hope this helps. john On 5/25/12 1:49 PM, Deepa Sahni wrote: > The number of rows in one million. I used ardea utility to load the > data from a .csv file. > > Examples of queries I ran are given below(I am including the output > because it includes the query too ). Although from what I could see > any query with these columns in the where clause is failing. > > ./benchmark_q2.sh: line 46: 1923 Killed > /home/deepa/Downloads/fastbit-ibis1.3.0/examples/ibis -d > /home/deepa/Downloads/fastbit-ibis1.3.0/tmp -q "select count(*) where > K2=2 and K4=3" > > > ./benchmark_q2.sh: line 48: 1963 Killed > /home/deepa/Downloads/fastbit-ibis1.3.0/examples/ibis -d > /home/deepa/Downloads/fastbit-ibis1.3.0/tmp -q "select count(*) where > K2=2 and K10=3" > > > ./benchmark_q2.sh: line 50: 2019 Segmentation fault > /home/deepa/Downloads/fastbit-ibis1.3.0/examples/ibis -d > /home/deepa/Downloads/fastbit-ibis1.3.0/tmp -q "select count(*) where > K2=2 and K100=3" > > Also if you think it does need more memory, could you give some > specifics. In the sense if I want to fix the issues how much more > memory should I make available to it. Does it need more main memory or > hard disk space? > > Thanks, > Deepa > ./benchmark_q2.sh: line 52: 2047 Segmentation fault > /home/deepa/Downloads/fastbit-ibis1.3.0/examples/ibis -d > /home/deepa/Downloads/fastbit-ibis1.3.0/tmp -q "select count(*) where > K2=2 and K10K=3" > > > On Fri, May 25, 2012 at 1:20 PM, K. John Wu <[email protected] > <mailto:[email protected]>> wrote: > > Hi, Deepa, > > I would guess that you are using data generated by setqgen. Would you > mind tell us how many rows are there? What are the exact queries > you've used? > > I would assume that your join is a little too big to be held in 1GB of > memory. If you give me the about information, I would be able to > confirm this speculation. > > John > > > > On 5/25/12 1:07 PM, Deepa Sahni wrote: > > > > Hi John, > > > > I have attached the data partition details of all the columns. The > > issue I am having is with the following columns. > > > > Column Name Issue > > -------------------------------------------------- > > k4 killed > > k10 killed > > k10k segmentation fault > > k100 segmentation fault > > > > In addition to this I am running FastBit on ubuntu(running on top of > > VMPlayer) and the ram available to it is 1GB and hard disk is 20GB. > > If you need any other information I will provide that too. > > > > Thanks, > > Deepa > > > > > > > > On Fri, May 25, 2012 at 9:36 AM, K. John Wu <[email protected] > <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>>> wrote: > > > > Yes, you are probably encounter memory shortage. Since you > did not > > give any size information, there is no way for me to verify your > > statement;-) > > > > John > > > > > > On 5/25/12 1:09 AM, Deepa Sahni wrote: > > > Hi, > > > > > > I need some help debugging an issue. FYI I am running > FastBit on > > > ubuntu. > > > > > > Whenever I run queries with certain columns in the where > clause > > I get > > > the following errors. > > > > > > for column K100 : Process killed > > > for column K10 :Segmentation Fault > > > > > > I did not create indexes on any of the columns, since my > > understanding > > > was that FastBit creates indexes as and when needed. As I > ran my > > > queries indexes got created. But for the columns mentioned > above I > > > repeatedly get the same error for all select statements > with these > > > columns in the where clause. > > > > > > I tried deleting all the other indexes to clear up some > memory for > > > these(K100, K10) column indexes(I think this might be > happening > > > because of memory shortage). It did not fix the problem. > Could some > > > shed some light on what might be causing this error or how to > > debug or > > > fix this. > > > > > > > > > Thanks in advance, > > > Deepa > > > > > > > > > > > > > > > > > > _______________________________________________ > > > FastBit-users mailing list > > > [email protected] > <mailto:[email protected]> > <mailto:[email protected] > <mailto:[email protected]>> > > > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > > > > > > _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
