sasmith-14 commented on issue #884:
URL: https://github.com/apache/wayang/issues/884#issuecomment-6882781093

   Hi @zkaoudi, Thanks for taking a look!
   
   I think you're right that the line is correct when the dataset size is 
known. The case I was worried about is when it isn't: `datasetSize` defaults to 
`UNKNOWN_DATASET_SIZE` (0), and `DataQuanta.sample(...)` leaves it at 0 unless 
the caller passes a size. In that case `sampleSize >= 0` is always true, so if 
I'm reading it right the operator returns the whole input even for a small 
sample.
   
   As a small example, a reservoir sample of 5 over 10 elements with no dataset 
size set returns all 10 instead of 5, which is what the test I added checks.
   
   I noticed `JavaRandomSampleOperator` already guards this with 
`isDataSetSizeKnown()` and falls back to the input size when it's unknown, so I 
tried to match that in the reservoir operator. I may well be missing something 
about how datasetSize gets set in practice though, so please let me know if 
you'd prefer a different approach or if this isn't actually a problem.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to