[Mono-dev] Performance problem with System.Data
I think i've catched a huge perf. problem with System.Data https://bugzilla.novell.com/show_bug.cgi?id=357947 _ Ce message et les eventuels documents joints peuvent contenir des informations confidentielles. Au cas ou il ne vous serait pas destine, nous vous remercions de bien vouloir le supprimer et en aviser immediatement l'expediteur. Toute utilisation de ce message non conforme a sa destination, toute diffusion ou publication, totale ou partielle et quel qu'en soit le moyen est formellement interdite. Les communications sur internet n'etant pas securisees, l'integrite de ce message n'est pas assuree et la societe emettrice ne peut etre tenue pour responsable de son contenu. ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] Performance problem with System.Data
Hi, A long pending patch from Senga, which I have to work it soon or Varadhan, which will change the System.Data code from Array to Red-Black tree implementation. It doesn't apply directly with the current tree. Need modifications. Thanks Nagappan 2008/2/1 Hubert FONGARNAND <[EMAIL PROTECTED]>: > I think i've catched a huge perf. problem with System.Data > > https://bugzilla.novell.com/show_bug.cgi?id=357947 > _ > Ce message et les eventuels documents joints peuvent contenir des > informations confidentielles. Au cas ou il ne vous serait pas destine, nous > vous remercions de bien vouloir le supprimer et en aviser immediatement > l'expediteur. Toute utilisation de ce message non conforme a sa destination, > toute diffusion ou publication, totale ou partielle et quel qu'en soit le > moyen est formellement interdite. Les communications sur internet n'etant > pas securisees, l'integrite de ce message n'est pas assuree et la societe > emettrice ne peut etre tenue pour responsable de son contenu. > > ___ > Mono-devel-list mailing list > Mono-devel-list@lists.ximian.com > http://lists.ximian.com/mailman/listinfo/mono-devel-list > > -- Linux Desktop Testing Project - http://ldtp.freedesktop.org http://nagappanal.blogspot.com rbtree_updated_patch Description: Binary data ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] Performance problem with System.Data
Title: RE: [Mono-dev] Performance problem with System.Data Hey Hubert, You hit a serious design issue in sys.Data: indeed the performance of DataView is really poor. The reason is as following: every time a change operation is performed on some row, the DataView does not have enough knowledge how that will affect its index, thus DataView rebuilds it. (May be in case with Delete it's clear, but there are many other cases, see OnRowChanged() ). When designing current index implementation I planned to extend the index with events, similar to those DataTable has (i.e. RowChanged, RowDeleted etc). The DataView will need to subscribe to Index events instead of those of DataTable (like it does now). When DataTable changes, it should notify all its indices, so DataView's index will fire a relevant event to its DataView, eliminating the need for UpdateIndex. But when the index implementation has been submitted and everything worked, including the DataView, I did not have a chance to rewrite DataView and left it mostly untouched. Having all this in mind, fixing only the Delete() scenario won't solve the issue in general. Regards, Konstantin Triger -Original Message- From: [EMAIL PROTECTED] on behalf of Hubert FONGARNAND Sent: Fri 2/1/2008 13:08 To: mono-devel-list@lists.ximian.com Subject: [Mono-dev] Performance problem with System.Data I think i've catched a huge perf. problem with System.Data https://bugzilla.novell.com/show_bug.cgi?id=357947 _ Ce message et les eventuels documents joints peuvent contenir des informations confidentielles. Au cas ou il ne vous serait pas destine, nous vous remercions de bien vouloir le supprimer et en aviser immediatement l'expediteur. Toute utilisation de ce message non conforme a sa destination, toute diffusion ou publication, totale ou partielle et quel qu'en soit le moyen est formellement interdite. Les communications sur internet n'etant pas securisees, l'integrite de ce message n'est pas assuree et la societe emettrice ne peut etre tenue pour responsable de son contenu. ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] Performance problem with System.Data
Title: RE: [Mono-dev] Performance problem with System.Data Hey Nagappan, Can you please explain 1. How RBTree implementation will solve the issue in the bug? 2. Why do you think RBTree implementation will be superior over Array in performance? Regards, Konstantin Triger -Original Message- From: [EMAIL PROTECTED] on behalf of Nagappan A Sent: Fri 2/1/2008 17:42 To: Hubert FONGARNAND Cc: mono-devel-list@lists.ximian.com Subject: Re: [Mono-dev] Performance problem with System.Data Hi, A long pending patch from Senga, which I have to work it soon or Varadhan, which will change the System.Data code from Array to Red-Black tree implementation. It doesn't apply directly with the current tree. Need modifications. Thanks Nagappan 2008/2/1 Hubert FONGARNAND <[EMAIL PROTECTED]>: > I think i've catched a huge perf. problem with System.Data > > https://bugzilla.novell.com/show_bug.cgi?id=357947 > _ > Ce message et les eventuels documents joints peuvent contenir des > informations confidentielles. Au cas ou il ne vous serait pas destine, nous > vous remercions de bien vouloir le supprimer et en aviser immediatement > l'expediteur. Toute utilisation de ce message non conforme a sa destination, > toute diffusion ou publication, totale ou partielle et quel qu'en soit le > moyen est formellement interdite. Les communications sur internet n'etant > pas securisees, l'integrite de ce message n'est pas assuree et la societe > emettrice ne peut etre tenue pour responsable de son contenu. > > ___ > Mono-devel-list mailing list > Mono-devel-list@lists.ximian.com > http://lists.ximian.com/mailman/listinfo/mono-devel-list > > -- Linux Desktop Testing Project - http://ldtp.freedesktop.org http://nagappanal.blogspot.com ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] Performance problem with System.Data
Hi Kosta, RBTree implementation is not directly related to this bug, but I was trying to say, in general about the performance of System.Data. In general RBTree performance is much better than Array based. As per Senganal's test result, for adding 1 million records, it took 40 minutes. With RBTree implementation, he was able to do them in seconds. Adding senganal in CC. Thanks Nagappan 2008/2/2 Konstantin Triger <[EMAIL PROTECTED]>: > Hey Nagappan, > > Can you please explain > 1. How RBTree implementation will solve the issue in the bug? > 2. Why do you think RBTree implementation will be superior over Array in > performance? > > Regards, > Konstantin Triger > -- Linux Desktop Testing Project - http://ldtp.freedesktop.org http://nagappanal.blogspot.com ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] Performance problem with System.Data
Hi Nagappan, As far as I know, when adding many records, the suggested usage of DataTable is [BeginLoadData -> add records -> EndLoadData]. In this case the performance of both implementation should be roughly similar, but the memory footprint of RBTree will be much higher. Can you please post the Senganal’s test code? Regards, Konstantin Triger From: Nagappan A [mailto:[EMAIL PROTECTED] Sent: Saturday, February 02, 2008 11:39 PM To: Konstantin Triger Cc: Hubert FONGARNAND; mono-devel-list@lists.ximian.com; [EMAIL PROTECTED] Subject: Re: [Mono-dev] Performance problem with System.Data Hi Kosta, RBTree implementation is not directly related to this bug, but I was trying to say, in general about the performance of System.Data. In general RBTree performance is much better than Array based. As per Senganal's test result, for adding 1 million records, it took 40 minutes. With RBTree implementation, he was able to do them in seconds. Adding senganal in CC. Thanks Nagappan 2008/2/2 Konstantin Triger <[EMAIL PROTECTED]>: Hey Nagappan, Can you please explain 1. How RBTree implementation will solve the issue in the bug? 2. Why do you think RBTree implementation will be superior over Array in performance? Regards, Konstantin Triger -- Linux Desktop Testing Project - http://ldtp.freedesktop.org http://nagappanal.blogspot.com ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] Performance problem with System.Data
Hi Kosta,
I haven't executed / compiled any of the programs :) FYI.
Attaching what ever with me.
Thanks
Nagappan
On Feb 3, 2008 12:02 AM, Konstantin Triger <[EMAIL PROTECTED]> wrote:
> Hi Nagappan,
>
>
>
> As far as I know, when adding many records, the suggested usage of
> DataTable is [BeginLoadData -> add records -> EndLoadData]. In this case the
> performance of both implementation should be roughly similar, but the memory
> footprint of RBTree will be much higher.
>
>
>
> Can you please post the Senganal's test code?
>
>
>
> Regards,
>
> Konstantin Triger
>
> *From:* Nagappan A [mailto:[EMAIL PROTECTED]
> *Sent:* Saturday, February 02, 2008 11:39 PM
> *To:* Konstantin Triger
> *Cc:* Hubert FONGARNAND; mono-devel-list@lists.ximian.com;
> [EMAIL PROTECTED]
> *Subject:* Re: [Mono-dev] Performance problem with System.Data
>
>
>
> Hi Kosta,
>
> RBTree implementation is not directly related to this bug, but I was
> trying to say, in general about the performance of System.Data.
>
> In general RBTree performance is much better than Array based. As per
> Senganal's test result, for adding 1 million records, it took 40 minutes.
> With RBTree implementation, he was able to do them in seconds.
>
> Adding senganal in CC.
>
> Thanks
> Nagappan
>
> 2008/2/2 Konstantin Triger <[EMAIL PROTECTED]>:
>
> Hey Nagappan,
>
> Can you please explain
> 1. How RBTree implementation will solve the issue in the bug?
> 2. Why do you think RBTree implementation will be superior over Array in
> performance?
>
>
>
> Regards,
> Konstantin Triger
>
>
> --
> Linux Desktop Testing Project - http://ldtp.freedesktop.org
> http://nagappanal.blogspot.com
>
--
Linux Desktop Testing Project - http://ldtp.freedesktop.org
http://nagappanal.blogspot.com
using System;
using System.Data;
using System.Data.Common;
using System.Data.SqlClient;
class tmp {
public static void Main (String[] args)
{
DataSet ds = new DataSet ();
DataTable table = ds.Tables.Add ("Table");
table.Columns.Add ("ID", typeof (int));
table.Columns.Add ("Value", typeof (int));
table.PrimaryKey = new DataColumn[] {table.Columns [0]};
long t = DateTime.Now.Ticks;
for (int i=0; i < 100; i++) {
try {
DataRow row = table.NewRow ();
row ["ID"] = i;
row ["Value"] = 1000;
table.Rows.Add (row);
} catch (Exception e) {
}
}
t = DateTime.Now.Ticks - t;
Console.WriteLine ((float)t/(1000));
t = DateTime.Now.Ticks;
ds.EnforceConstraints = false;
ds.EnforceConstraints = true;
t = DateTime.Now.Ticks - t;
Console.WriteLine ((float)t/(1000));
}
}
using System;
using System.Data;
class datatest {
static void Main( string[] args )
{
int nRows = Convert.ToInt32( args[0] );
DataTable table = new DataTable();
DataColumn newColumn = new DataColumn( "a", typeof(Int32) );
table.Columns.Add( newColumn );
newColumn = new DataColumn( "b", typeof(Int32) );
table.Columns.Add( newColumn );
table.BeginLoadData();
object[] dataArray = new object[2];
for ( int i=0; iusing System;
using System.Data;
using System.Data.Common;
using System.Data.SqlClient;
class tmp {
public static void Main (String[] args)
{
DataSet ds = new DataSet ();
DataTable table1 = ds.Tables.Add ("Table1");
DataTable table2 = ds.Tables.Add ("Table2");
table1.Columns.Add ("ID", typeof (int));
table1.Columns.Add ("Value", typeof (int));
table2.Columns.Add ("PID", typeof (int));
table2.Columns.Add ("Value", typeof (int));
table1.PrimaryKey = new DataColumn[] {table1.Columns [0]};
table2.Constraints.Add ("fk", table1.Columns [0], table2.Columns [0]);
long t = DateTime.Now.Ticks;
for (int i=0; i < 100; i++) {
try {
DataRow prow = table1.NewRow ();
prow ["ID"] = i;
prow ["Value"] = 1000;
table1.Rows.Add (prow);
} catch (Exception e) {
}
}
Console.WriteLine (((float)(t-DateTime.Now.Ticks))/1000);
for (int i=0; i < 100; i++) {
try {
DataRow crow = table2.NewRow ();
crow ["PID"] = i;
crow ["Value"] = -1000;
table2.Rows.Add (crow);
crow = table2.NewRow ();
crow ["PID"] = i;
crow ["Value"] = 1000;
table2.Rows.Add (crow);
crow = table2.NewRow ();
crow ["PID"] = i;
crow ["Value"] = 2000;
table2.Rows.Add (crow);
} catch (Exception e) {
}
}
Console.WriteLine (((float)(t-DateTime.Now.Ticks))/1000);
Console.WriteLine ( "Finished creating the tables");
table1.Rows [0][0] = 500;
Console.Write
Re: [Mono-dev] Performance problem with System.Data
Hello, My major concern with an RBTree implementation is memory consumption. Since you must create a node for each record, for a DataTable with several hundreds thousands records memory size of RBTree may reach several MB. Since it’s common that DataTable has several constraints/views, memory consumption may easily reach hundred MB or more... In addition, this will pose an additional pressure on GC. Note that one of design goals behind array based architecture of both DataContainers and indices was to minimize memory consumption due to the calculations above. I think that we must run many different tests/scenarios before we go with index datastructure redesign. Regards, Konstantin Triger From: senganal thirunavakarasu [mailto:[EMAIL PROTECTED] Sent: Sunday, February 03, 2008 6:29 PM To: Nagappan A Cc: Konstantin Triger; Hubert FONGARNAND; mono-devel-list@lists.ximian.com Subject: Re: [Mono-dev] Performance problem with System.Data Hi Its been quite a while (~1.5yrs) and i dont remember all the details , but will try and explain whatever i remember.. Well, the difference is when u have constraints on the datatable.. loading lots of data is inherently slower in an array based storage as adding new data would mean inserting data and not just appending it to the end.. ofcourse, if u disable all constraints and load all the data and then enable the constraints, things should be fine .. but on testing .net implementation, both the usecases (with and without the constraints) were fast as compared the mono implementation.. using a RBTree, inserting/searching are not expensive operations and hence perform reasonably well in both the cases.. infact, its prob a lil bit slower than the array implementation when the constraints are not enforced but thats negligibly small .. the RBTree implementation that i had was pretty much a basic implementation straight out of cormen book.. the idea was to check if there will be any significant effect.. It definitely could be better implemented, which is why i had not checked it in at that time.. anyways, an RBTree or any balanced tree based implementation would definitely be faster than the array based implementation when it comes to loading/modifying the data in dataTable.. hope that helps.. cheers senganal On Feb 3, 2008 4:03 AM, Nagappan A <[EMAIL PROTECTED]> wrote: Hi Kosta, I haven't executed / compiled any of the programs :) FYI. Attaching what ever with me. Thanks Nagappan On Feb 3, 2008 12:02 AM, Konstantin Triger <[EMAIL PROTECTED]> wrote: Hi Nagappan, As far as I know, when adding many records, the suggested usage of DataTable is [BeginLoadData -> add records -> EndLoadData]. In this case the performance of both implementation should be roughly similar, but the memory footprint of RBTree will be much higher. Can you please post the Senganal's test code? Regards, Konstantin Triger From: Nagappan A [mailto:[EMAIL PROTECTED] Sent: Saturday, February 02, 2008 11:39 PM To: Konstantin Triger Cc: Hubert FONGARNAND; mono-devel-list@lists.ximian.com; [EMAIL PROTECTED] Subject: Re: [Mono-dev] Performance problem with System.Data Hi Kosta, RBTree implementation is not directly related to this bug, but I was trying to say, in general about the performance of System.Data. In general RBTree performance is much better than Array based. As per Senganal's test result, for adding 1 million records, it took 40 minutes. With RBTree implementation, he was able to do them in seconds. Adding senganal in CC. Thanks Nagappan 2008/2/2 Konstantin Triger <[EMAIL PROTECTED]>: Hey Nagappan, Can you please explain 1. How RBTree implementation will solve the issue in the bug? 2. Why do you think RBTree implementation will be superior over Array in performance? Regards, Konstantin Triger -- Linux Desktop Testing Project - http://ldtp.freedesktop.org http://nagappanal.blogspot.com -- Linux Desktop Testing Project - http://ldtp.freedesktop.org http://nagappanal.blogspot.com ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] Performance problem with System.Data
Hi Its been quite a while (~1.5yrs) and i dont remember all the details , but will try and explain whatever i remember.. Well, the difference is when u have constraints on the datatable.. loading lots of data is inherently slower in an array based storage as adding new data would mean inserting data and not just appending it to the end.. ofcourse, if u disable all constraints and load all the data and then enable the constraints, things should be fine .. but on testing .net implementation, both the usecases (with and without the constraints) were fast as compared the mono implementation.. using a RBTree, inserting/searching are not expensive operations and hence perform reasonably well in both the cases.. infact, its prob a lil bit slower than the array implementation when the constraints are not enforced but thats negligibly small .. the RBTree implementation that i had was pretty much a basic implementation straight out of cormen book.. the idea was to check if there will be any significant effect.. It definitely could be better implemented, which is why i had not checked it in at that time.. anyways, an RBTree or any balanced tree based implementation would definitely be faster than the array based implementation when it comes to loading/modifying the data in dataTable.. hope that helps.. cheers senganal On Feb 3, 2008 4:03 AM, Nagappan A <[EMAIL PROTECTED]> wrote: > Hi Kosta, > > I haven't executed / compiled any of the programs :) FYI. > > Attaching what ever with me. > > Thanks > Nagappan > > > On Feb 3, 2008 12:02 AM, Konstantin Triger <[EMAIL PROTECTED]> wrote: > > > Hi Nagappan, > > > > > > > > As far as I know, when adding many records, the suggested usage of > > DataTable is [BeginLoadData -> add records -> EndLoadData]. In this case the > > performance of both implementation should be roughly similar, but the memory > > footprint of RBTree will be much higher. > > > > > > > > Can you please post the Senganal's test code? > > > > > > > > Regards, > > > > Konstantin Triger > > > > *From:* Nagappan A [mailto:[EMAIL PROTECTED] > > *Sent:* Saturday, February 02, 2008 11:39 PM > > *To:* Konstantin Triger > > *Cc:* Hubert FONGARNAND; mono-devel-list@lists.ximian.com; > > [EMAIL PROTECTED] > > *Subject:* Re: [Mono-dev] Performance problem with System.Data > > > > > > > > Hi Kosta, > > > > RBTree implementation is not directly related to this bug, but I was > > trying to say, in general about the performance of System.Data. > > > > In general RBTree performance is much better than Array based. As per > > Senganal's test result, for adding 1 million records, it took 40 minutes. > > With RBTree implementation, he was able to do them in seconds. > > > > Adding senganal in CC. > > > > Thanks > > Nagappan > > > > 2008/2/2 Konstantin Triger <[EMAIL PROTECTED]>: > > > > Hey Nagappan, > > > > Can you please explain > > 1. How RBTree implementation will solve the issue in the bug? > > 2. Why do you think RBTree implementation will be superior over Array in > > performance? > > > > > > > > Regards, > > Konstantin Triger > > > > > > -- > > Linux Desktop Testing Project - http://ldtp.freedesktop.org > > http://nagappanal.blogspot.com > > > > > > -- > Linux Desktop Testing Project - http://ldtp.freedesktop.org > http://nagappanal.blogspot.com > ___ Mono-devel-list mailing list Mono-devel-list@lists.ximian.com http://lists.ximian.com/mailman/listinfo/mono-devel-list
Re: [Mono-dev] Performance problem with System.Data
Hi as i said before, my whole reason for trying out the RBTree implementation was to be compare with .net performance for the usecases mentioned (some kind of sorted order for the data) before.. i was trying to see if using a balanced tree would give me comparable results. on the other-hand, there is always a tradeoff with memory .. i dont remember seeing any major difference in memory consumption between the basic implementation and .net (i mean in orders) for large datasets.. infact, if mem consumption is a big issue, then u could have a mixed implementation (messy to maintain ofcourse) and use the tree based storage purely for sorted data and the basic array implementation for others.. i guess this is something that needs to tested and figured out for different usecases and compared with .net.. cheers senganal On Feb 5, 2008 5:19 AM, Konstantin Triger <[EMAIL PROTECTED]> wrote: > Hello, > > > > My major concern with an RBTree implementation is memory consumption. > Since you must create a node for each record, for a DataTable with several > hundreds thousands records memory size of RBTree may reach several MB. Since > it's common that DataTable has several constraints/views, memory consumption > may easily reach hundred MB or more... In addition, this will pose an > additional pressure on GC. > > > > Note that one of design goals behind array based architecture of both > DataContainers and indices was to minimize memory consumption due to the > calculations above. > > > > I think that we must run many different tests/scenarios before we go with > index datastructure redesign. > > > > Regards, > > Konstantin Triger > > *From:* senganal thirunavakarasu [mailto:[EMAIL PROTECTED] > *Sent:* Sunday, February 03, 2008 6:29 PM > *To:* Nagappan A > *Cc:* Konstantin Triger; Hubert FONGARNAND; > mono-devel-list@lists.ximian.com > > *Subject:* Re: [Mono-dev] Performance problem with System.Data > > > > Hi > > Its been quite a while (~1.5yrs) and i dont remember all the details , but > will try and explain whatever i remember.. > > Well, the difference is when u have constraints on the datatable.. loading > lots of data is inherently slower in an array based storage as adding new > data would mean inserting data and not just appending it to the end.. > ofcourse, if u disable all constraints and load all the data and then enable > the constraints, things should be fine .. but on testing .net > implementation, both the usecases (with and without the constraints) were > fast as compared the mono implementation.. > > using a RBTree, inserting/searching are not expensive operations and hence > perform reasonably well in both the cases.. infact, its prob a lil bit > slower than the array implementation when the constraints are not enforced > but thats negligibly small .. > > the RBTree implementation that i had was pretty much a basic > implementation straight out of cormen book.. the idea was to check if there > will be any significant effect.. It definitely could be better implemented, > which is why i had not checked it in at that time.. anyways, an RBTree or > any balanced tree based implementation would definitely be faster than the > array based implementation when it comes to loading/modifying the data in > dataTable.. > > hope that helps.. > > cheers > senganal > > On Feb 3, 2008 4:03 AM, Nagappan A <[EMAIL PROTECTED]> wrote: > > Hi Kosta, > > I haven't executed / compiled any of the programs :) FYI. > > Attaching what ever with me. > > Thanks > Nagappan > > > > On Feb 3, 2008 12:02 AM, Konstantin Triger <[EMAIL PROTECTED]> wrote: > > Hi Nagappan, > > > > As far as I know, when adding many records, the suggested usage of > DataTable is [BeginLoadData -> add records -> EndLoadData]. In this case the > performance of both implementation should be roughly similar, but the memory > footprint of RBTree will be much higher. > > > > Can you please post the Senganal's test code? > > > > Regards, > > Konstantin Triger > > *From:* Nagappan A [mailto:[EMAIL PROTECTED] > *Sent:* Saturday, February 02, 2008 11:39 PM > *To:* Konstantin Triger > *Cc:* Hubert FONGARNAND; mono-devel-list@lists.ximian.com; > [EMAIL PROTECTED] > *Subject:* Re: [Mono-dev] Performance problem with System.Data > > > > Hi Kosta, > > RBTree implementation is not directly related to this bug, but I was > trying to say, in general about the performance of System.Data. > > In general RBTree performance is much better than Array based. As per > Senganal's test result, for adding 1 million records, it took 40 minutes. > With RBTree implementation,
