On Mon, Feb 04, 2002 at 01:38:26PM -0500, Gianni Johansson wrote:
> > > > > FSBucketFactory.FSBucket.resetWrite() doesn't work (NOP) and
> > > > > FSBucketFactory.FSBucket.size() returns the buffer size, not the
> > > > > amount of data in the bucket.
> > > >
> > > > Yeah, I know. It's rather an unreasonable interface imho. I have
> > > > never needed to use either of those methods. Do you really need them?
> > >
> > > Yes. They are used by client code. You use Bucket.size() yourself in
> > > InternalClient. I don't think Bucket was ever intended to be used by
> > > core code (but I'm sure I will hear from Oskar if I am wrong ;-) ).
> >
> > But the way I use Bucket.size(), the value I want is the total buffer
> > size (i.e. after it's filled).
> No what you want is the amount of data in the bucket. There is no gaurantee
> that the bucket isn't bigger.
When you obtain a Bucket from BucketFactory you are supposed to know
beforehand the length of the data. So actually there is such a
guarantee.
> > I don't see why you wouldn't want it
> > that way. If you've partially filled it and you want to know how much,
> > count the bytes yourself.
> You don't always know the size of the data when you allocate the bucket.
You do if you're using BucketFactory.
> Scenario:
> Client code running inside Nodes JVM wants to use Node.bf for temp buckets.
> It allocates buckets which are big enough to handle the largest anticipated
> amount of data, but count on .size() to tell how much data was actually
> downloaded.
Why wouldn't you know the data length in advance?
> // This works with the current semantics.
> Bucket meta = Node.bf.makeBucket(MAX_BUCKETSIZE);
> Bucket data = Node.bf.makeBucket(MAX_BUCKETSIZE);
> Request req = new GetRequest(5, someCoolURL, meta, data);
> if (factory.getClient(req).blockingRun() == Request.DONE) {
> System.err.println("bytes of meta data: " + meta.size());
> System.err.println("bytes of data: " + data.size());
> }
The above is bad design in the client library. Whether you're doing
FNP, FCP, or InternalClient, the data lengths are available before any
writing to Buckets actually needs to occur. So the client lib should be
changed to allow allocation of the Bucket after the lengths are known.
Relying on a MAX_BUCKETSIZE parameter like that is an obvious sign of a
design deficienecy.
> > > How do you handle restart after DNV's in InternalClient without
> > > resetWrite()?
> But how does it work now? Does it work?
> >
> > The FSBucket semantics effectively imply a resetWrite() call immediately
> > before getOutputStream().
> This forces you to open and close the OutputStream just to do a reset in
> the case where you know that the current data isn't valid (e.g. after DNV).
>
> That seems ugly if you need to mark the bucket empty, but don't nescessarily
> have more data to write into it.
How many ways can I state that when you obtain an OutputStream through
getOutputStream(), it writes from the beginning of the Bucket each time?
What is ugly about calling getOutputStream() each time you need to
overwrite the Bucket? Are you telling me you like to hold an open
OutputStream, call resetWrite(), and then keep using that same
OutputStream to write from the beginning again?
> > What would be a bitch to implement would be making it append if you
> > don't call resetWrite(). And I don't believe that's needed anywhere in
> > our codebase anyway (i.e., we always want to resetWrite()).
> >
> You should be keeping track of the amount of data written into the Bucket
> anyway, so you can support Bucket.size() correctly. Once you have that data,
> why isn't appending anything more than seeking to an offset when the stream
> opens?
I don't agree that I should be keeping track of the amount of data
written in. But at any rate, there is no seek operation in the FS.
Only read and write. And if you open the stream starting at a different
offset it changes the IV for the crypto.
> If you're not already keeping track of the amount of data written into the
> bucket does that mean that if I allocate a 256K bucket and then write 3 bytes
> into it, the next call to getInputStream() will return a stream 256K long
> with only 3 bytes of valid data, and the rest garbage?
Yes.
> Buckets are not for our codebase. They are for client code.
>
> If you want to use something with different semantics in the internals of
> your DS implementation, that's fine. Just don't call that thing a Bucket,
> make sure InternalClient supports real Bucket semantics, and don't give out
> your objects from BucketFactorys.
It's not just in the internals of the DS implementation, there is a real
need for temp storage objects that work exactly the way I've implemented
the FSBuckets throughout the node, particularly in the FCP states as
well as InternalClient.
I'm still not convinced there is a need for temp storage objects that
conform to the historical Bucket semantics. Those semantics are
inspired by the way ext2fs/FAT/etc. files work and they do not
carry over well to the general semantics of temp storage. In other
words to implement those semantics you pretty much have to use
java.io.File or do some pretty crazy stuff.
However, if there really is a need for both kinds of semantics then yes,
there should be two interfaces. And BucketFactory is not for the old
semantics. If it were it wouldn't require you to predeclare the size.
I'm sorry if this is causing trouble but last year when I was putting
together the 0.4 node we needed temp storage objects that worked my way
and it seemed pretty certain the other semantics weren't going to be at
all useful.
-tc
_______________________________________________
Devl mailing list
Devl at freenetproject.org
http://lists.freenetproject.org/mailman/listinfo/devl
