This vote passes with 3 binding +1 votes, 5 non-binding votes, and no -1
votes.

Thanks all!

+1 votes (binding):
Wenchen Fan
Reynold Xin
Cheng Liang


+1 votes (non-binding):
Xiao Li
Weichen Xu
Vaquar khan
Liwei Lin
Dongjoon Hyun


On Tue, Oct 17, 2017 at 12:30 AM, Dongjoon Hyun <dongjoon.h...@gmail.com>
wrote:

> +1
>
> On Sun, Oct 15, 2017 at 11:43 PM, Cheng Lian <lian.cs....@gmail.com>
> wrote:
>
>> +1
>>
>> On 10/12/17 20:10, Liwei Lin wrote:
>>
>> +1 !
>>
>> Cheers,
>> Liwei
>>
>> On Thu, Oct 12, 2017 at 7:11 PM, vaquar khan <vaquar.k...@gmail.com>
>> wrote:
>>
>>> +1
>>>
>>> Regards,
>>> Vaquar khan
>>>
>>> On Oct 11, 2017 10:14 PM, "Weichen Xu" <weichen...@databricks.com>
>>> wrote:
>>>
>>> +1
>>>
>>> On Thu, Oct 12, 2017 at 10:36 AM, Xiao Li <gatorsm...@gmail.com> wrote:
>>>
>>>> +1
>>>>
>>>> Xiao
>>>>
>>>> On Mon, 9 Oct 2017 at 7:31 PM Reynold Xin <r...@databricks.com> wrote:
>>>>
>>>>> +1
>>>>>
>>>>> One thing with MetadataSupport - It's a bad idea to call it that
>>>>> unless adding new functions in that trait wouldn't break source/binary
>>>>> compatibility in the future.
>>>>>
>>>>>
>>>>> On Mon, Oct 9, 2017 at 6:07 PM, Wenchen Fan <cloud0...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I'm adding my own +1 (binding).
>>>>>>
>>>>>> On Tue, Oct 10, 2017 at 9:07 AM, Wenchen Fan <cloud0...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I'm going to update the proposal: for the last point, although the
>>>>>>> user-facing API (`df.write.format(...).option(...).mode(...).save()`)
>>>>>>> mixes data and metadata operations, we are still able to separate them 
>>>>>>> in
>>>>>>> the data source write API. We can have a mix-in trait `MetadataSupport`
>>>>>>> which has a method `create(options)`, so that data sources can mix in 
>>>>>>> this
>>>>>>> trait and provide metadata creation support. Spark will call this 
>>>>>>> `create`
>>>>>>> method inside `DataFrameWriter.save` if the specified data source has 
>>>>>>> it.
>>>>>>>
>>>>>>> Note that file format data sources can ignore this new trait and
>>>>>>> still write data without metadata(it doesn't have metadata anyway).
>>>>>>>
>>>>>>> With this updated proposal, I'm calling a new vote for the data
>>>>>>> source v2 write path.
>>>>>>>
>>>>>>> The vote will be up for the next 72 hours. Please reply with your
>>>>>>> vote:
>>>>>>>
>>>>>>> +1: Yeah, let's go forward and implement the SPIP.
>>>>>>> +0: Don't really care.
>>>>>>> -1: I don't think this is a good idea because of the following
>>>>>>> technical reasons.
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> On Tue, Oct 3, 2017 at 12:03 AM, Wenchen Fan <cloud0...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> After we merge the infrastructure of data source v2 read path, and
>>>>>>>> have some discussion for the write path, now I'm sending this email to 
>>>>>>>> call
>>>>>>>> a vote for Data Source v2 write path.
>>>>>>>>
>>>>>>>> The full document of the Data Source API V2 is:
>>>>>>>> https://docs.google.com/document/d/1n_vUVbF4KD3gxTmkNEon5qdQ
>>>>>>>> -Z8qU5Frf6WMQZ6jJVM/edit
>>>>>>>>
>>>>>>>> The ready-for-review PR that implements the basic infrastructure
>>>>>>>> for the write path:
>>>>>>>> https://github.com/apache/spark/pull/19269
>>>>>>>>
>>>>>>>>
>>>>>>>> The Data Source V1 write path asks implementations to write a
>>>>>>>> DataFrame directly, which is painful:
>>>>>>>> 1. Exposing upper-level API like DataFrame to Data Source API is
>>>>>>>> not good for maintenance.
>>>>>>>> 2. Data sources may need to preprocess the input data before
>>>>>>>> writing, like cluster/sort the input by some columns. It's better to 
>>>>>>>> do the
>>>>>>>> preprocessing in Spark instead of in the data source.
>>>>>>>> 3. Data sources need to take care of transaction themselves, which
>>>>>>>> is hard. And different data sources may come up with a very similar
>>>>>>>> approach for the transaction, which leads to many duplicated codes.
>>>>>>>>
>>>>>>>> To solve these pain points, I'm proposing the data source v2
>>>>>>>> writing framework which is very similar to the reading framework, i.e.,
>>>>>>>> WriteSupport -> DataSourceV2Writer -> DataWriterFactory -> DataWriter.
>>>>>>>>
>>>>>>>> Data Source V2 write path follows the existing FileCommitProtocol,
>>>>>>>> and have task/job level commit/abort, so that data sources can 
>>>>>>>> implement
>>>>>>>> transaction easier.
>>>>>>>>
>>>>>>>> We can create a mix-in trait for DataSourceV2Writer to specify the
>>>>>>>> requirement for input data, like clustering and ordering.
>>>>>>>>
>>>>>>>> Spark provides a very simple protocol for uses to connect to data
>>>>>>>> sources. A common way to write a dataframe to data sources:
>>>>>>>> `df.write.format(...).option(...).mode(...).save()`.
>>>>>>>> Spark passes the options and save mode to data sources, and
>>>>>>>> schedules the write job on the input data. And the data source should 
>>>>>>>> take
>>>>>>>> care of the metadata, e.g., the JDBC data source can create the table 
>>>>>>>> if it
>>>>>>>> doesn't exist, or fail the job and ask users to create the table in the
>>>>>>>> corresponding database first. Data sources can define some options for
>>>>>>>> users to carry some metadata information like partitioning/bucketing.
>>>>>>>>
>>>>>>>>
>>>>>>>> The vote will be up for the next 72 hours. Please reply with your
>>>>>>>> vote:
>>>>>>>>
>>>>>>>> +1: Yeah, let's go forward and implement the SPIP.
>>>>>>>> +0: Don't really care.
>>>>>>>> -1: I don't think this is a good idea because of the following
>>>>>>>> technical reasons.
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>
>>>
>>
>>
>

Reply via email to