Answers Inlined. Cheers, Abdullah.
Amoudi, Abdullah. On Tue, Jan 26, 2016 at 12:15 AM, Mike Carey <[email protected]> wrote: > So my remaining Q's (before we e-mail anyone) would be: > 1. Do we possibly drop some data? (Still trying to > make sure I understand the implications of your first > point below.) > No but we don't know when the record was actually persisted and so, there is a chance that we record a file as has been read completely while there is a small window that allows a crash before persistence. > 2. Do we (silently) ignore faulty records for now? > Yes. > 3. Do we (silently) ignore duplicate records for now? > Yes. > Thx! > Mike > > On 1/24/16 5:48 AM, abdullah alamoudi wrote: > >> Yes, those two are fixed at this stage. >> I am not sure I understand well what you mean by semantics but I will >> explain how the filesystem feed will behave. >> The user will have to provide the following (directories' locations, >> expression to match files names against, formats of records). Once the >> adapter is connected, it will start reading from the specified directories >> until the directories are deleted or the feed is disconnected. >> duplicate records will be skipped. >> >> When there are no more files in the specified directories, the feed >> adapter >> will push the buffered records to storage then wait for more events from >> the file system. >> >> Things that are not yet done properly: >> 1. at-least-once semantics. >> 2. logging of faulty records. >> 3. logging of duplicate records. >> >> 2 &3 are easy to implement correctly while 1 will take about a week to get >> right with a good design and implementation. >> Should I draft a new email? >> >> >> Amoudi, Abdullah. >> >> On Sun, Jan 24, 2016 at 3:14 AM, Mike Carey <[email protected]> wrote: >> >> Q: So are both of those (1 & 2) fixed? And could you clarify the >>> semantics that file feeds will have at this stage? :-) >>> Thx! >>> >>> On 1/22/16 11:40 PM, abdullah alamoudi wrote: >>> >>> I think we are ready if they don't care much about at-least-once >>>> semantics. >>>> >>>> Other than that, everything is ready. >>>> The last communication we had with them was a promise to fix: >>>> 1. Duplicate key exceptions >>>> 2. Pushing last few records to storage. >>>> >>>> Cheers >>>> Abdullah >>>> Cool. Are we ready to ping Wisconsin (Condor) again soon...? >>>> (When did you last interact w/them - and where were things left...?) >>>> >>>> On 1/22/16 8:11 AM, abdullah alamoudi wrote: >>>> >>>> It looks like I will not be able to attend the weekly meeting today so I >>>> >>>>> am >>>>> sending my status here: >>>>> 1. Addressed Young-Seok's comments on the Upsert change. >>>>> 2. Completed streaming of Couchbase inserts into AsterixDB. >>>>> 3. Completed the implementation of the flush () operation on all >>>>> IFrameWriters. >>>>> 4. Removed the ExternalLookup operator and merged it with >>>>> UnnestMapOperator >>>>> 5. Created a proposal for Feed changes. >>>>> >>>>> Cheers, >>>>> Abdullah >>>>> >>>>> >>>>> >>>>> >
