An arraylist containing bytes would have far worse performance (it also does doubling internally except now you would be dealing with references instead of of single bytes). A List<byte> also will do doubling internally (if you don't set it large enough initially) but will do it dealing with a byte [] internally (a memorystream will work in a similar fashion). If you require fast random access this would probably be the fastest. Another problem will exist in that you would be dealing with large objects (they get garbage colelcted differently) and this can actually open a DOS attack.
The best long term solution is probably to make smaller byte arrays and link them together. This would be extremely good if you are not changing data but simply appending to it. So instead of managing an array that is 100 mb you hand 2000 smaller arrays. Think for a minute about having an array chunks .. each chunk is n bytes so ... byte [][] chunks You could calculate locations rather easily .. chunk = n / chunksize then offset = a modulus n % chunksize. This would allow you to continue adding items without ever having full copy operations. I think someone has already mentioned a similar strategy but this is the way to go. If you ran out of initial chunks, you could easily resize this array (copying 1000-2000 items as opposed to 10000000) Cheers, Greg On 7/6/06, dave wanta <[EMAIL PROTECTED]> wrote:
Hi Erick, hmm... this is interesting. I wouldn't have to worry about constantly expanding the byte array, and in this case I don't need to worry about accessing any internal bytes, just the end of the overall byte array to check for terminating characters. (thinking out loud) I wonder if even something quick and dirty of an ArrayList of byte arrays would work? Simply as a quick prototype? Cheers! Dave ----- Original Message ----- From: "Erick Thompson" <[EMAIL PROTECTED]> To: <[email protected]> Sent: Thursday, July 06, 2006 3:26 PM Subject: Re: [ADVANCED-DOTNET] trim byte array > It's the copy that gets pricey, which is going to be hard to avoid in this situation. A list just won't cut it with this size. I would suggest a new class that uses a list of byte[], and then uses the length of the internal byte array to determine the right array to touch in the indexer. I have to head out of the office, but I could throw together some code later if you need me to. It makes random access slower, but then you don't need to ever copy the data until the final step when you spit out the array, which should speed it up a lot. > > Erick > > > > > -----Original Message----- > > From: Discussion of advanced .NET topics. > > [mailto:[EMAIL PROTECTED] On Behalf Of > > gregory young > > Sent: Thursday, July 06, 2006 1:00 PM > > To: [email protected] > > Subject: Re: [ADVANCED-DOTNET] trim byte array > > > > Dave this is how list<byte> would work as well ... you can > > however set an initial size if you know it to avoid these doublings. > > > > Cheers, > > > > Greg > > > > > > On 7/6/06, dave wanta <[EMAIL PROTECTED]> wrote: > > > > > > Hi all, > > > > > > btw, just fyi...if you Reflector a MemoryStream, it doubles it's > > > internal buffer every time it exceeds it's length. > > > > > > The MemoryStream is really clean from a programmatic > > standpoint, it's > > > just not the performance I really want. > > > > > > Thanks for all the suggestions guys! I really appreciate it. > > > > > > Cheers! > > > Dave > > > > > > =================================== > > > This list is hosted by DevelopMentor http://www.develop.com > > > > > > View archives and manage your subscription(s) at > > > http://discuss.develop.com > > > > > > > > > > > -- > > If knowledge can create problems, it is not through ignorance > > that we can solve them. > > > > Isaac Asimov > > > =================================== This list is hosted by DevelopMentor� http://www.develop.com View archives and manage your subscription(s) at http://discuss.develop.com
-- If knowledge can create problems, it is not through ignorance that we can solve them. Isaac Asimov
