Re: Introducing Apache Gluten(incubating), a middle layer to offload Spark to native engine

[Holden Karau]

On Wed, Apr 10, 2024 at 9:54 PM Binwei Yang  wrote:

>
> Gluten currently already support Velox backend and Clickhouse backend.
> data fusion support is also proposed but no one worked on it.
>
> Gluten isn't a POC. It's under actively developing but some companies
> already used it.
>
>
> On 2024/04/11 03:32:01 Dongjoon Hyun wrote:
> > I'm interested in your claim.
> >
> > Could you elaborate or provide some evidence for your claim, *a door for
> > all native libraries*, Binwei?
> >
> > For example, is there any POC for that claim? Maybe, did I miss something
> > in that SPIP?
>
I think the concern here is there are multiple different layers to get from
Spark -> Native code and ideally any changes we introduce in Spark would be
for common functionality that is useful across them (e.g. data fusion comet
& gluten & photon*, etc.)


* Photon being harder to guess at since it's closed source.

> >
> > Dongjoon.
> >
> > On Wed, Apr 10, 2024 at 8:19 PM Binwei Yang  wrote:
> >
> > >
> > > The SPIP is not for current Gluten, but open a door for all native
> > > libraries and accelerators support.
> > >
> > > On 2024/04/11 00:27:43 Weiting Chen wrote:
> > > > Yes, the 1st Apache release(v1.2.0) for Gluten will be in September.
> > > > For Spark version support, currently Gluten v1.1.1 support Spark3.2
> and
> > > 3.3.
> > > > We are planning to support Spark3.4 and 3.5 in Gluten v1.2.0.
> > > > Spark4.0 support for Gluten is depending on the release schedule in
> > > Spark community.
> > > >
> > > > On 2024/04/09 07:14:13 Dongjoon Hyun wrote:
> > > > > Thank you for sharing, Weiting.
> > > > >
> > > > > Do you think you can share the future milestone of Apache Gluten?
> > > > > I'm wondering when the first stable release will come and how we
> can
> > > > > coordinate across the ASF communities.
> > > > >
> > > > > > This project is still under active development now, and doesn't
> have
> > > a
> > > > > stable release.
> > > > > > https://github.com/apache/incubator-gluten/releases/tag/v1.1.1
> > > > >
> > > > > In the Apache Spark community, Apache Spark 3.2 and 3.3 is the end
> of
> > > > > support.
> > > > > And, 3.4 will have 3.4.3 next week and 3.4.4 (another EOL release)
> is
> > > > > scheduled in October.
> > > > >
> > > > > For the SPIP, I guess it's applicable for Apache Spark 4.0.0 only
> if
> > > there
> > > > > is something we need to do from Spark side.
> > > > >
> > > > > Thanks,
> > > > > Dongjoon.
> > > > >
> > > > >
> > > > > On Mon, Apr 8, 2024 at 11:19 PM WeitingChen <
> weitingc...@apache.org>
> > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > We are excited to introduce a new Apache incubating project
> called
> > > Gluten.
> > > > > > Gluten serves as a middleware layer designed to offload Spark to
> > > native
> > > > > > engines like Velox or ClickHouse.
> > > > > > For more detailed information, please visit the project
> repository at
> > > > > > https://github.com/apache/incubator-gluten
> > > > > >
> > > > > > Additionally, a new Spark SPIP related to Spark + Gluten
> > > collaboration has
> > > > > > been proposed at
> https://issues.apache.org/jira/browse/SPARK-47773.
> > > > > > We eagerly await feedback from the Spark community.
> > > > > >
> > > > > > Thanks,
> > > > > > Weiting.
> > > > > >
> > > > > >
> > > > >
> > > >
> > > > -
> > > > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> > > >
> > > >
> > >
> > > -
> > > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> > >
> > >
> >
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

-- 
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  
YouTube Live Streams: https://www.youtube.com/user/holdenkarau

Re: Introducing Apache Gluten(incubating), a middle layer to offload Spark to native engine

[Binwei Yang]

Gluten currently already support Velox backend and Clickhouse backend. data 
fusion support is also proposed but no one worked on it.

Gluten isn't a POC. It's under actively developing but some companies already 
used it.


On 2024/04/11 03:32:01 Dongjoon Hyun wrote:
> I'm interested in your claim.
> 
> Could you elaborate or provide some evidence for your claim, *a door for
> all native libraries*, Binwei?
> 
> For example, is there any POC for that claim? Maybe, did I miss something
> in that SPIP?
> 
> Dongjoon.
> 
> On Wed, Apr 10, 2024 at 8:19 PM Binwei Yang  wrote:
> 
> >
> > The SPIP is not for current Gluten, but open a door for all native
> > libraries and accelerators support.
> >
> > On 2024/04/11 00:27:43 Weiting Chen wrote:
> > > Yes, the 1st Apache release(v1.2.0) for Gluten will be in September.
> > > For Spark version support, currently Gluten v1.1.1 support Spark3.2 and
> > 3.3.
> > > We are planning to support Spark3.4 and 3.5 in Gluten v1.2.0.
> > > Spark4.0 support for Gluten is depending on the release schedule in
> > Spark community.
> > >
> > > On 2024/04/09 07:14:13 Dongjoon Hyun wrote:
> > > > Thank you for sharing, Weiting.
> > > >
> > > > Do you think you can share the future milestone of Apache Gluten?
> > > > I'm wondering when the first stable release will come and how we can
> > > > coordinate across the ASF communities.
> > > >
> > > > > This project is still under active development now, and doesn't have
> > a
> > > > stable release.
> > > > > https://github.com/apache/incubator-gluten/releases/tag/v1.1.1
> > > >
> > > > In the Apache Spark community, Apache Spark 3.2 and 3.3 is the end of
> > > > support.
> > > > And, 3.4 will have 3.4.3 next week and 3.4.4 (another EOL release) is
> > > > scheduled in October.
> > > >
> > > > For the SPIP, I guess it's applicable for Apache Spark 4.0.0 only if
> > there
> > > > is something we need to do from Spark side.
> > > >
> > > > Thanks,
> > > > Dongjoon.
> > > >
> > > >
> > > > On Mon, Apr 8, 2024 at 11:19 PM WeitingChen 
> > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > We are excited to introduce a new Apache incubating project called
> > Gluten.
> > > > > Gluten serves as a middleware layer designed to offload Spark to
> > native
> > > > > engines like Velox or ClickHouse.
> > > > > For more detailed information, please visit the project repository at
> > > > > https://github.com/apache/incubator-gluten
> > > > >
> > > > > Additionally, a new Spark SPIP related to Spark + Gluten
> > collaboration has
> > > > > been proposed at https://issues.apache.org/jira/browse/SPARK-47773.
> > > > > We eagerly await feedback from the Spark community.
> > > > >
> > > > > Thanks,
> > > > > Weiting.
> > > > >
> > > > >
> > > >
> > >
> > > -
> > > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> > >
> > >
> >
> > -
> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> >
> >
> 

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Introducing Apache Gluten(incubating), a middle layer to offload Spark to native engine

[Dongjoon Hyun]

I'm interested in your claim.

Could you elaborate or provide some evidence for your claim, *a door for
all native libraries*, Binwei?

For example, is there any POC for that claim? Maybe, did I miss something
in that SPIP?

Dongjoon.

On Wed, Apr 10, 2024 at 8:19 PM Binwei Yang  wrote:

>
> The SPIP is not for current Gluten, but open a door for all native
> libraries and accelerators support.
>
> On 2024/04/11 00:27:43 Weiting Chen wrote:
> > Yes, the 1st Apache release(v1.2.0) for Gluten will be in September.
> > For Spark version support, currently Gluten v1.1.1 support Spark3.2 and
> 3.3.
> > We are planning to support Spark3.4 and 3.5 in Gluten v1.2.0.
> > Spark4.0 support for Gluten is depending on the release schedule in
> Spark community.
> >
> > On 2024/04/09 07:14:13 Dongjoon Hyun wrote:
> > > Thank you for sharing, Weiting.
> > >
> > > Do you think you can share the future milestone of Apache Gluten?
> > > I'm wondering when the first stable release will come and how we can
> > > coordinate across the ASF communities.
> > >
> > > > This project is still under active development now, and doesn't have
> a
> > > stable release.
> > > > https://github.com/apache/incubator-gluten/releases/tag/v1.1.1
> > >
> > > In the Apache Spark community, Apache Spark 3.2 and 3.3 is the end of
> > > support.
> > > And, 3.4 will have 3.4.3 next week and 3.4.4 (another EOL release) is
> > > scheduled in October.
> > >
> > > For the SPIP, I guess it's applicable for Apache Spark 4.0.0 only if
> there
> > > is something we need to do from Spark side.
> > >
> > > Thanks,
> > > Dongjoon.
> > >
> > >
> > > On Mon, Apr 8, 2024 at 11:19 PM WeitingChen 
> wrote:
> > >
> > > > Hi all,
> > > >
> > > > We are excited to introduce a new Apache incubating project called
> Gluten.
> > > > Gluten serves as a middleware layer designed to offload Spark to
> native
> > > > engines like Velox or ClickHouse.
> > > > For more detailed information, please visit the project repository at
> > > > https://github.com/apache/incubator-gluten
> > > >
> > > > Additionally, a new Spark SPIP related to Spark + Gluten
> collaboration has
> > > > been proposed at https://issues.apache.org/jira/browse/SPARK-47773.
> > > > We eagerly await feedback from the Spark community.
> > > >
> > > > Thanks,
> > > > Weiting.
> > > >
> > > >
> > >
> >
> > -
> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> >
> >
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Re: Introducing Apache Gluten(incubating), a middle layer to offload Spark to native engine

[Binwei Yang]

The SPIP is not for current Gluten, but open a door for all native libraries 
and accelerators support.

On 2024/04/11 00:27:43 Weiting Chen wrote:
> Yes, the 1st Apache release(v1.2.0) for Gluten will be in September.
> For Spark version support, currently Gluten v1.1.1 support Spark3.2 and 3.3.
> We are planning to support Spark3.4 and 3.5 in Gluten v1.2.0.
> Spark4.0 support for Gluten is depending on the release schedule in Spark 
> community.
> 
> On 2024/04/09 07:14:13 Dongjoon Hyun wrote:
> > Thank you for sharing, Weiting.
> > 
> > Do you think you can share the future milestone of Apache Gluten?
> > I'm wondering when the first stable release will come and how we can
> > coordinate across the ASF communities.
> > 
> > > This project is still under active development now, and doesn't have a
> > stable release.
> > > https://github.com/apache/incubator-gluten/releases/tag/v1.1.1
> > 
> > In the Apache Spark community, Apache Spark 3.2 and 3.3 is the end of
> > support.
> > And, 3.4 will have 3.4.3 next week and 3.4.4 (another EOL release) is
> > scheduled in October.
> > 
> > For the SPIP, I guess it's applicable for Apache Spark 4.0.0 only if there
> > is something we need to do from Spark side.
> > 
> > Thanks,
> > Dongjoon.
> > 
> > 
> > On Mon, Apr 8, 2024 at 11:19 PM WeitingChen  wrote:
> > 
> > > Hi all,
> > >
> > > We are excited to introduce a new Apache incubating project called Gluten.
> > > Gluten serves as a middleware layer designed to offload Spark to native
> > > engines like Velox or ClickHouse.
> > > For more detailed information, please visit the project repository at
> > > https://github.com/apache/incubator-gluten
> > >
> > > Additionally, a new Spark SPIP related to Spark + Gluten collaboration has
> > > been proposed at https://issues.apache.org/jira/browse/SPARK-47773.
> > > We eagerly await feedback from the Spark community.
> > >
> > > Thanks,
> > > Weiting.
> > >
> > >
> > 
> 
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> 
> 

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Introducing Apache Gluten(incubating), a middle layer to offload Spark to native engine

[Weiting Chen]

Yes, the 1st Apache release(v1.2.0) for Gluten will be in September.
For Spark version support, currently Gluten v1.1.1 support Spark3.2 and 3.3.
We are planning to support Spark3.4 and 3.5 in Gluten v1.2.0.
Spark4.0 support for Gluten is depending on the release schedule in Spark 
community.

On 2024/04/09 07:14:13 Dongjoon Hyun wrote:
> Thank you for sharing, Weiting.
> 
> Do you think you can share the future milestone of Apache Gluten?
> I'm wondering when the first stable release will come and how we can
> coordinate across the ASF communities.
> 
> > This project is still under active development now, and doesn't have a
> stable release.
> > https://github.com/apache/incubator-gluten/releases/tag/v1.1.1
> 
> In the Apache Spark community, Apache Spark 3.2 and 3.3 is the end of
> support.
> And, 3.4 will have 3.4.3 next week and 3.4.4 (another EOL release) is
> scheduled in October.
> 
> For the SPIP, I guess it's applicable for Apache Spark 4.0.0 only if there
> is something we need to do from Spark side.
> 
> Thanks,
> Dongjoon.
> 
> 
> On Mon, Apr 8, 2024 at 11:19 PM WeitingChen  wrote:
> 
> > Hi all,
> >
> > We are excited to introduce a new Apache incubating project called Gluten.
> > Gluten serves as a middleware layer designed to offload Spark to native
> > engines like Velox or ClickHouse.
> > For more detailed information, please visit the project repository at
> > https://github.com/apache/incubator-gluten
> >
> > Additionally, a new Spark SPIP related to Spark + Gluten collaboration has
> > been proposed at https://issues.apache.org/jira/browse/SPARK-47773.
> > We eagerly await feedback from the Spark community.
> >
> > Thanks,
> > Weiting.
> >
> >
> 

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Introducing Apache Gluten(incubating), a middle layer to offload Spark to native engine

[Dongjoon Hyun]

Thank you for sharing, Weiting.

Do you think you can share the future milestone of Apache Gluten?
I'm wondering when the first stable release will come and how we can
coordinate across the ASF communities.

> This project is still under active development now, and doesn't have a
stable release.
> https://github.com/apache/incubator-gluten/releases/tag/v1.1.1

In the Apache Spark community, Apache Spark 3.2 and 3.3 is the end of
support.
And, 3.4 will have 3.4.3 next week and 3.4.4 (another EOL release) is
scheduled in October.

For the SPIP, I guess it's applicable for Apache Spark 4.0.0 only if there
is something we need to do from Spark side.

Thanks,
Dongjoon.


On Mon, Apr 8, 2024 at 11:19 PM WeitingChen  wrote:

> Hi all,
>
> We are excited to introduce a new Apache incubating project called Gluten.
> Gluten serves as a middleware layer designed to offload Spark to native
> engines like Velox or ClickHouse.
> For more detailed information, please visit the project repository at
> https://github.com/apache/incubator-gluten
>
> Additionally, a new Spark SPIP related to Spark + Gluten collaboration has
> been proposed at https://issues.apache.org/jira/browse/SPARK-47773.
> We eagerly await feedback from the Spark community.
>
> Thanks,
> Weiting.
>
>

Introducing Apache Gluten(incubating), a middle layer to offload Spark to native engine

[WeitingChen]

Hi all,

We are excited to introduce a new Apache incubating project called Gluten.
Gluten serves as a middleware layer designed to offload Spark to native
engines like Velox or ClickHouse.
For more detailed information, please visit the project repository at
https://github.com/apache/incubator-gluten

Additionally, a new Spark SPIP related to Spark + Gluten collaboration has
been proposed at https://issues.apache.org/jira/browse/SPARK-47773.
We eagerly await feedback from the Spark community.

Thanks,
Weiting.