Podling Iceberg Report Reminder - April 2020

2020-03-25 Thread jmclean 
Dear podling,

This email was sent by an automated system on behalf of the Apache
Incubator PMC. It is an initial reminder to give you plenty of time to
prepare your quarterly board report.

The board meeting is scheduled for Wed, 15 April 2020, 10:30 am PDT.
The report for your podling will form a part of the Incubator PMC
report. The Incubator PMC requires your report to be submitted 2 weeks
before the board meeting, to allow sufficient time for review and
submission (Wed, April 01).

Please submit your report with sufficient time to allow the Incubator
PMC, and subsequently board members to review and digest. Again, the
very latest you should submit your report is 2 weeks prior to the board
meeting.

Candidate names should not be made public before people are actually
elected, so please do not include the names of potential committers or
PPMC members in your report.

Thanks,

The Apache Incubator PMC

Submitting your Report

--

Your report should contain the following:

*   Your project name
*   A brief description of your project, which assumes no knowledge of
the project or necessarily of its field
*   A list of the three most important issues to address in the move
towards graduation.
*   Any issues that the Incubator PMC or ASF Board might wish/need to be
aware of
*   How has the community developed since the last report
*   How has the project developed since the last report.
*   How does the podling rate their own maturity.

This should be appended to the Incubator Wiki page at:

https://cwiki.apache.org/confluence/display/INCUBATOR/April2020

Note: This is manually populated. You may need to wait a little before
this page is created from a template.

Note: The format of the report has changed to use markdown.

Mentors
---

Mentors should review reports for their project(s) and sign them off on
the Incubator wiki page. Signing off reports shows that you are
following the project - projects that are not signed may raise alarms
for the Incubator PMC.

Incubator PMC

Re: Shall we start a regular community sync up?

2020-03-25 Thread Jun Ma 
Hi Ryan,

Thanks for driving the sync up meeting. Could you please add Fan Diao(
fan.dia...@gmail.com) and myself to the invitation list?

Thanks,
Jun Ma

On Mon, Mar 23, 2020 at 9:57 PM OpenInx  wrote:

> Hi Ryan
>
> I received your invitation. Some guys from our Flink teams also want to
> join the hangouts  meeting. Do we need
> also send an extra invitation to them ?  Or could them just join the
> meeting with entering the meeting address[1] ?
>
> If need so, please let the following guys in:
> 1. ykt...@gmail.com
> 2. imj...@gmail.com
> 3. yuzhao@gmail.com
>
> BTW,  I've written a draft to discuss in the meeting [2],  anyone could
> enrich the topics want to discuss.
>
> Thanks.
>
> [1]. https://meet.google.com/_meet/xdx-rknm-uvm
> [2].
> https://docs.google.com/document/d/1wXTHGYhc7sDhP5DxlByba0S5YguNLWwY98FAp6Tx2mw/edit#
>
> On Mon, Mar 23, 2020 at 5:35 AM Ryan Blue 
> wrote:
>
>> I invited everyone that replied to this thread and the people that were
>> on the last invite.
>>
>> If you have specific topics you'd like to put on the agenda, please send
>> them to me!
>>
>> On Sun, Mar 22, 2020 at 2:28 PM Ryan Blue  wrote:
>>
>>> Let's go with Wednesday. I'll send out an invite.
>>>
>>> On Sun, Mar 22, 2020 at 1:36 PM John Zhuge  wrote:
>>>
 5-5:30 pm work for me. Prefer Wednesdays.

 On Sun, Mar 22, 2020 at 1:33 PM Romin Parekh 
 wrote:

> Hi folks,
>
> Both times slots work for me next week. Can we confirm a day?
>
> Thanks,
> Romin
>
> Sent from my iPhone
>
> > On Mar 20, 2020, at 11:38 PM, Jun H.  wrote:
> >
> > The schedule works for me.
> >
> >> On Thu, Mar 19, 2020 at 6:55 PM Junjie Chen <
> chenjunjied...@gmail.com> wrote:
> >>
> >> The same time works for me as well.
> >>
> >>> On Fri, Mar 20, 2020 at 9:43 AM Gautam 
> wrote:
> >>>
> >>> 5 / 5:30pm any day of next week works for me.
> >>>
> >>> On Thu, Mar 19, 2020 at 6:07 PM 李响  wrote:
> 
>  5 or 5:30 PM (UTC-7, is it PDT now) in any day works for me.
> Looking forward to it 8-)
> 
>  On Fri, Mar 20, 2020 at 8:17 AM RD  wrote:
> >
> > Same time works for me too!
> >
> > On Thu, Mar 19, 2020 at 4:45 PM Xabriel Collazo Mojica
>  wrote:
> >>
> >> 5pm or 5:30pm PT  any day next week would work for me.
> >>
> >> Thanks for restoring the community sync up!
> >>
> >> Xabriel J Collazo Mojica  |  Sr Computer Scientist II  |  Adobe
> >>
> >> On 3/18/20, 6:45 PM, "justin_cof...@apple.com on behalf of
> Justin Q Coffey"  j...@apple.com.INVALID> wrote:
> >>
> >>Any chance we could actually do 5:30pm PST?  I'm a bit of a
> lurker, but this roadmap is important to mine and I have a daily at 5pm 
> :(.
> >>
> >>-Justin
> >>
> >>> On Mar 18, 2020, at 6:43 PM, Saisai Shao <
> sai.sai.s...@gmail.com> wrote:
> >>>
> >>> 5pm PST in any day works for me.
> >>>
> >>> Looking forward to it.
> >>>
> >>> Thanks
> >>> Saisai
> >>
> >>
> >>
> 
> 
>  --
> 
>    李响 Xiang Li
> 
>  手机 cellphone ：+86-136-8113-8972
>  邮件 e-mail  ：wate...@gmail.com
> >>
> >>
> >>
> >> --
> >> Best Regards
>


 --
 John Zhuge

>>>
>>>
>>> --
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>>>
>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>
>

Re: Shall we start a regular community sync up?

2020-03-25 Thread Ryan Blue 
Will do.

On Wed, Mar 25, 2020 at 6:36 PM Jun Ma  wrote:

> Hi Ryan,
>
> Thanks for driving the sync up meeting. Could you please add Fan Diao(
> fan.dia...@gmail.com) and myself to the invitation list?
>
> Thanks,
> Jun Ma
>
> On Mon, Mar 23, 2020 at 9:57 PM OpenInx  wrote:
>
>> Hi Ryan
>>
>> I received your invitation. Some guys from our Flink teams also want to
>> join the hangouts  meeting. Do we need
>> also send an extra invitation to them ?  Or could them just join the
>> meeting with entering the meeting address[1] ?
>>
>> If need so, please let the following guys in:
>> 1. ykt...@gmail.com
>> 2. imj...@gmail.com
>> 3. yuzhao@gmail.com
>>
>> BTW,  I've written a draft to discuss in the meeting [2],  anyone could
>> enrich the topics want to discuss.
>>
>> Thanks.
>>
>> [1]. https://meet.google.com/_meet/xdx-rknm-uvm
>> [2].
>> https://docs.google.com/document/d/1wXTHGYhc7sDhP5DxlByba0S5YguNLWwY98FAp6Tx2mw/edit#
>>
>> On Mon, Mar 23, 2020 at 5:35 AM Ryan Blue 
>> wrote:
>>
>>> I invited everyone that replied to this thread and the people that were
>>> on the last invite.
>>>
>>> If you have specific topics you'd like to put on the agenda, please send
>>> them to me!
>>>
>>> On Sun, Mar 22, 2020 at 2:28 PM Ryan Blue  wrote:
>>>
 Let's go with Wednesday. I'll send out an invite.

 On Sun, Mar 22, 2020 at 1:36 PM John Zhuge  wrote:

> 5-5:30 pm work for me. Prefer Wednesdays.
>
> On Sun, Mar 22, 2020 at 1:33 PM Romin Parekh 
> wrote:
>
>> Hi folks,
>>
>> Both times slots work for me next week. Can we confirm a day?
>>
>> Thanks,
>> Romin
>>
>> Sent from my iPhone
>>
>> > On Mar 20, 2020, at 11:38 PM, Jun H.  wrote:
>> >
>> > The schedule works for me.
>> >
>> >> On Thu, Mar 19, 2020 at 6:55 PM Junjie Chen <
>> chenjunjied...@gmail.com> wrote:
>> >>
>> >> The same time works for me as well.
>> >>
>> >>> On Fri, Mar 20, 2020 at 9:43 AM Gautam 
>> wrote:
>> >>>
>> >>> 5 / 5:30pm any day of next week works for me.
>> >>>
>> >>> On Thu, Mar 19, 2020 at 6:07 PM 李响  wrote:
>> 
>>  5 or 5:30 PM (UTC-7, is it PDT now) in any day works for me.
>> Looking forward to it 8-)
>> 
>>  On Fri, Mar 20, 2020 at 8:17 AM RD  wrote:
>> >
>> > Same time works for me too!
>> >
>> > On Thu, Mar 19, 2020 at 4:45 PM Xabriel Collazo Mojica
>>  wrote:
>> >>
>> >> 5pm or 5:30pm PT  any day next week would work for me.
>> >>
>> >> Thanks for restoring the community sync up!
>> >>
>> >> Xabriel J Collazo Mojica  |  Sr Computer Scientist II  |  Adobe
>> >>
>> >> On 3/18/20, 6:45 PM, "justin_cof...@apple.com on behalf of
>> Justin Q Coffey" > j...@apple.com.INVALID> wrote:
>> >>
>> >>Any chance we could actually do 5:30pm PST?  I'm a bit of a
>> lurker, but this roadmap is important to mine and I have a daily at 5pm 
>> :(.
>> >>
>> >>-Justin
>> >>
>> >>> On Mar 18, 2020, at 6:43 PM, Saisai Shao <
>> sai.sai.s...@gmail.com> wrote:
>> >>>
>> >>> 5pm PST in any day works for me.
>> >>>
>> >>> Looking forward to it.
>> >>>
>> >>> Thanks
>> >>> Saisai
>> >>
>> >>
>> >>
>> 
>> 
>>  --
>> 
>>    李响 Xiang Li
>> 
>>  手机 cellphone ：+86-136-8113-8972
>>  邮件 e-mail  ：wate...@gmail.com
>> >>
>> >>
>> >>
>> >> --
>> >> Best Regards
>>
>
>
> --
> John Zhuge
>


 --
 Ryan Blue
 Software Engineer
 Netflix

>>>
>>>
>>> --
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>>>
>>

-- 
Ryan Blue
Software Engineer
Netflix

Re: [Discuss] Merge spark-3 branch into master

2020-03-25 Thread Saisai Shao 
Hi Ryan,

As mentioned in the meeting, would you please point me out the way to make
some submodules excluded from consistent-versions plugin.

Thanks
Saisai

Anton Okolnychyi  于2020年3月18日周三 上午4:14写道：

> I am +1 on having spark-2 and spark-3 modules as well.
>
> On 7 Mar 2020, at 15:03, RD  wrote:
>
> I'm +1 to separate modules for spark-2 and spark-3, after the 0.8 release.
> I think it would be a big change in organizations to adopt Spark-3 since
> that brings in Scala-2.12 which is binary incompatible to previous Scala
> versions. Hence this adoption could take a lot of time. I know in our
> company we have no near term plans to move to Spark 3.
>
> -Best,
> R.
>
> On Thu, Mar 5, 2020 at 6:33 PM Saisai Shao  wrote:
>
>> I was thinking that if it is possible to limit version lock plugin to
>> only iceberg core related subprojects., seems like current
>> consistent-versions plugin doesn't allow to do so. So not sure if there're
>> some other plugins which could provide similar functionality with more
>> flexibility?
>>
>>  Any suggestions on this?
>>
>> Best regards,
>> Saisai
>>
>> Saisai Shao  于2020年3月5日周四 下午3:12写道：
>>
>>> I think the requirement of supporting different version should be quite
>>> common. As Iceberg is a table format which should be adapted to different
>>> engines like Hive, Flink, Spark. To support different versions is a real
>>> problem, Spark is just one case, Hive, Flink could also be the case if the
>>> interface is changed across major versions. Also version lock may have
>>> problems when several engines coexisted in the same build, as they will
>>> transiently introduce lots of dependencies which may be conflicted, it may
>>> be hard to figure out one version which could satisfy all, and usually they
>>> only confined to a single module.
>>>
>>>  So I think we should figure out a way to support such scenario, not
>>> just maintaining branches one by one.
>>>
>>> Ryan Blue  于2020年3月5日周四 上午2:53写道：
>>>
 I think the key is that this wouldn't be using the same published
 artifacts. This work would create a spark-2.4 artifact and a spark-3.0
 artifact. (And possibly a spark-common artifact.)

 It seems reasonable to me to have those in the same build instead of in
 separate branches, as long as the Spark dependencies are not leaked outside
 of the modules. That said, I'd rather have the additional checks that
 baseline provides in general since this is a short-term problem. It would
 just be nice if we could have versions that are confined to a single
 module. The Nebula plugin that baseline uses claims to support that, but I
 couldn't get it to work.

 On Wed, Mar 4, 2020 at 6:38 AM Saisai Shao 
 wrote:

> Just think a bit on this. I agree that generally introducing different
> versions of same dependencies could be error prone. But I think the case
> here should not lead to  issue:
>
> 1.  These two sub-modules spark-2 and spark-3 are isolated, they're
> not dependent on either.
> 2. They can be differentiated by names when generating jars, also they
> will not be relied by other modules in Iceberg.
>
> So this dependency issue should not be the case here. And in Maven it
> could be achieved easily. Please correct me if wrong.
>
> Best regards,
> Saisai
>
> Saisai Shao  于2020年3月4日周三 上午10:01写道：
>
>> Thanks Matt,
>>
>> If branching is the only choice, then we would potentially have two
>> *master* branches until spark-3 is vastly adopted. That will somehow
>> increase the maintenance burden and lead to inconsistency. IMO I'm OK 
>> with
>> the branching way, just think that we should have a clear way to keep
>> tracking of two branches.
>>
>> Best,
>> Saisai
>>
>> Matt Cheah  于2020年3月4日周三 上午9:50写道：
>>
>>> I think it’s generally dangerous and error-prone to try to support
>>> two versions of the same library in the same build, in the same 
>>> published
>>> artifacts. This is the stance that Baseline
>>>  + Gradle Consistent
>>> Versions 
>>> takes. Gradle Consistent Versions is specifically opinionated towards
>>> building against one version of a library across all modules in the 
>>> build.
>>>
>>>
>>>
>>> I would think that branching would be the best way to build and
>>> publish against multiple versions of a dependency.
>>>
>>>
>>>
>>> -Matt Cheah
>>>
>>>
>>>
>>> *From: *Saisai Shao 
>>> *Reply-To: *"dev@iceberg.apache.org" 
>>> *Date: *Tuesday, March 3, 2020 at 5:45 PM
>>> *To: *Iceberg Dev List 
>>> *Cc: *Ryan Blue 
>>> *Subject: *Re: [Discuss] Merge spark-3 branch into master
>>>
>>>
>>>
>>> I didn't realized that Gradle cannot support two different versions
>>> in one