Re: DataSourceV2 sync tomorrow

2018-11-15 Thread Ryan Blue
Below are my notes from the sync yesterday. Thanks to everyone that participated! For the format of this sync, I think it would help to make a small change. Since we have so many people and it take a long time to introduce everyone, let’s try to get to the content faster by not doing the round of

Re: DataSourceV2 sync tomorrow

2018-11-14 Thread Ryan Blue
The live stream link for this is https://stream.meet.google.com/stream/6be59d80-04c7-44dc-9042-4f3b597fc8ba Some people said that it didn't work last time. I'm not sure why that would happen, but I don't use these much so I'm no expert. If you can't join the live stream, then feel free to join

Re: DataSourceV2 sync tomorrow

2018-11-14 Thread Ryan Blue
Jamison, I've added you to the invite. If anyone else wants to be invited, please send me a request. You can send it directly to me to avoid too many messages on this thread. On Wed, Nov 14, 2018 at 8:57 AM Jamison Bennett wrote: > Hi Spark Team, > > I am interested in joining this meeting

Re: DataSourceV2 sync tomorrow

2018-11-14 Thread Jamison Bennett
Hi Spark Team, I am interested in joining this meeting because I am interested in the data source v2 APIs. I couldn't find information about this meeting, so could someone please share the link? Thanks, Jamison Bennett Cloudera Software Engineer jamison.benn...@cloudera.com 515 Congress Ave,

Re: DataSourceV2 sync tomorrow

2018-11-13 Thread Arun Mahadevan
IMO, the currentOffset should not be optional. For continuous mode I assume this offset gets periodically check pointed (so mandatory) ? For the micro batch mode the currentOffset would be the start offset for a micro-batch. And if the micro-batch could be executed without knowing the 'latest'

Re: DataSourceV2 sync tomorrow

2018-11-13 Thread Cody Koeninger
Am I the only one for whom the livestream link didn't work last time? Would like to be able to at least watch the discussion this time around. On Tue, Nov 13, 2018 at 6:01 PM Ryan Blue wrote: > > Hi everyone, > I just wanted to send out a reminder that there’s a DSv2 sync tomorrow at > 17:00

DataSourceV2 sync tomorrow

2018-11-13 Thread Ryan Blue
Hi everyone, I just wanted to send out a reminder that there’s a DSv2 sync tomorrow at 17:00 PST, which is 01:00 UTC. Here are some of the topics under discussion in the last couple of weeks: - Read API for v2 - see Wenchen’s doc