On 02/28/2012 04:29 PM, Frank Chang wrote:
Good morning, We have a SQLITE C++ application which tries to find the
intersection between the blobs in separate sqlite database tables(which we call
subgraphs) ,containing record numbers. When we profile the code below we find
that the top profiler user is sqlite3BTreeMoveToUnpacked. I have attached the
profiler outputs whivh we obtained using SQLITE 3.7.10.
We were wondering if it is possible to reduce the number of times
sqlite3BTreeMoveToUnpacked is called with an SQLITE C++ application? Thank you.
MoveToUnpacked() is the routine that seeks for a specified
rowid within a b-tree structure. In the code below it will
be called once for each call to sqlite3_blob_open() or
sqlite3_blob_reopen().
Are you working through the blobs in a table in rowid order?
If so, you would be better off with a SELECT statement that
iterates through multiple rows. Otherwise, I think you're stuck
with the seeks, and the overhead they impose.
Dan.
void cIntersectingDedupe::GetSubGraphBlob(sSUBGRAPHINFO *SubGraph_,unsigned
long *SubGraphBlob_,
int *Size_) {
int Size;
// Grab a BLOB and put it into a unsigned long buffer. As the BLOB contains
record numbers,
// we will never see a value of zero. Thus, we use 0 to mark the end of the
array.
//
// Note that we can have BLOBs of size 0, though. If we used DistillSubGraph to
// dedupe the subgraph, some records will be consolidated into others. The
// donor record's BLOB gets zapped because all of it's BLOB was rolled into the
// donee (All your BLOB are belong to us!)
// First time, open the BLOB for real, else we can re-open (faster):
if (SubGraph_->hBlob==0)
sqlite3_blob_open(SubGraph_->Database,"main","AggregatedData","Rows",SubGraph_->IteratorPos+1,0,&SubGraph_->hBlob);
else
sqlite3_blob_reopen(SubGraph_->hBlob,SubGraph_->IteratorPos+1);
Size=sqlite3_blob_bytes(SubGraph_->hBlob)/sizeof(unsigned long);
sqlite3_blob_read(SubGraph_->hBlob,SubGraphBlob_,Size*sizeof(unsigned long),0);
SubGraphBlob_[Size]=0;
if (Size_!=0)
*Size_=Size;
}
void cIntersectingDedupe::IntersectBlobs(sSUBGRAPHINFO *SubGraph_,unsigned long
*IntersectionBlob_,
unsigned long *SubGraphBlob_) {
int Pos1,Pos2,PosOut;
GetSubGraphBlob(SubGraph_,SubGraphBlob_);
// Perform the intersection. We walk though the two blobs, if the blobs contain
the same
// value, that value is copied to PosOut in Blob_, else, the blob that is
'behind' is
// incremented so it can 'catch up' to the other:
Pos1=Pos2=PosOut=0;
while (IntersectionBlob_[Pos1]!=0&& SubGraphBlob_[Pos2]!=0) {
if (IntersectionBlob_[Pos1]==SubGraphBlob_[Pos2]) {
IntersectionBlob_[PosOut++]=IntersectionBlob_[Pos1++];
Pos2++;
} else if (IntersectionBlob_[Pos1]<SubGraphBlob_[Pos2]) {
Pos1++;
} else {
Pos2++;
}
}
IntersectionBlob_[PosOut]=0;
}
_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
