Re: maven 3.6.1 removed from apache maven repo

[Felix Cheung]

(Hmm, what is spark-...@apache.org?)


From: Sean Owen 
Sent: Tuesday, September 3, 2019 11:58:30 AM
To: Xiao Li 
Cc: Tom Graves ; spark-...@apache.org 

Subject: Re: maven 3.6.1 removed from apache maven repo

It's because build/mvn only queries ASF mirrors, and they remove non-current 
releases from mirrors regularly (we do the same).
This may help avoid this in the future: 
https://github.com/apache/spark/pull/25667

On Tue, Sep 3, 2019 at 1:41 PM Xiao Li 
mailto:lix...@databricks.com>> wrote:
Hi, Tom,

To unblock the build, I merged the upgrade to master. 
https://github.com/apache/spark/pull/25665

Thanks!

Xiao


On Tue, Sep 3, 2019 at 10:58 AM Tom Graves  wrote:
It looks like maven 3.6.1 was removed from the repo - see SPARK-28960.  It 
looks like they pushed 3.6.2,  but I don't see any release notes on the maven 
page for it 3.6.2

Seems like we had this happen before, can't remember if it was maven or 
something else, anyone remember or know if they are about to release 3.6.2?

Tom


--
[Databricks Summit - Watch the 
talks]

Re: maven 3.6.1 removed from apache maven repo

[Sean Owen]

It's because build/mvn only queries ASF mirrors, and they remove
non-current releases from mirrors regularly (we do the same).
This may help avoid this in the future:
https://github.com/apache/spark/pull/25667

On Tue, Sep 3, 2019 at 1:41 PM Xiao Li  wrote:

> Hi, Tom,
>
> To unblock the build, I merged the upgrade to master.
> https://github.com/apache/spark/pull/25665
>
> Thanks!
>
> Xiao
>
>
> On Tue, Sep 3, 2019 at 10:58 AM Tom Graves 
> wrote:
>
>> It looks like maven 3.6.1 was removed from the repo - see SPARK-28960.
>> It looks like they pushed 3.6.2,  but I don't see any release notes on the
>> maven page for it 3.6.2
>>
>> Seems like we had this happen before, can't remember if it was maven or
>> something else, anyone remember or know if they are about to release 3.6.2?
>>
>> Tom
>>
>
>
> --
> [image: Databricks Summit - Watch the talks]
> 
>

Re: maven 3.6.1 removed from apache maven repo

[Xiao Li]

Hi, Tom,

To unblock the build, I merged the upgrade to master.
https://github.com/apache/spark/pull/25665

Thanks!

Xiao


On Tue, Sep 3, 2019 at 10:58 AM Tom Graves 
wrote:

> It looks like maven 3.6.1 was removed from the repo - see SPARK-28960.  It
> looks like they pushed 3.6.2,  but I don't see any release notes on the
> maven page for it 3.6.2
>
> Seems like we had this happen before, can't remember if it was maven or
> something else, anyone remember or know if they are about to release 3.6.2?
>
> Tom
>


-- 
[image: Databricks Summit - Watch the talks]

maven 3.6.1 removed from apache maven repo

[Tom Graves]

It looks like maven 3.6.1 was removed from the repo - see SPARK-28960.  It 
looks like they pushed 3.6.2,  but I don't see any release notes on the maven 
page for it 3.6.2
Seems like we had this happen before, can't remember if it was maven or 
something else, anyone remember or know if they are about to release 3.6.2?
Tom

Re: Why two netty libs?

[Shixiong(Ryan) Zhu]

Yep, historical reasons. And Netty 4 is under another namespace, so we can
use Netty 3 and Netty 4 in the same JVM.

On Tue, Sep 3, 2019 at 6:15 AM Sean Owen  wrote:

> It was for historical reasons; some other transitive dependencies needed
> it.
> I actually was just able to exclude Netty 3 last week from master.
> Spark uses Netty 4.
>
> On Tue, Sep 3, 2019 at 6:59 AM Jacek Laskowski  wrote:
> >
> > Hi,
> >
> > Just noticed that Spark 2.4.x uses two netty deps of different versions.
> Why?
> >
> > jars/netty-all-4.1.17.Final.jar
> > jars/netty-3.9.9.Final.jar
> >
> > Shouldn't one be excluded or perhaps shaded?
> >
> > Pozdrawiam,
> > Jacek Laskowski
> > 
> > https://about.me/JacekLaskowski
> > The Internals of Spark SQL https://bit.ly/spark-sql-internals
> > The Internals of Spark Structured Streaming
> https://bit.ly/spark-structured-streaming
> > The Internals of Apache Kafka https://bit.ly/apache-kafka-internals
> > Follow me at https://twitter.com/jaceklaskowski
> >
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
> --

Best Regards,
Ryan

Re: Why two netty libs?

[Sean Owen]

It was for historical reasons; some other transitive dependencies needed it.
I actually was just able to exclude Netty 3 last week from master.
Spark uses Netty 4.

On Tue, Sep 3, 2019 at 6:59 AM Jacek Laskowski  wrote:
>
> Hi,
>
> Just noticed that Spark 2.4.x uses two netty deps of different versions. Why?
>
> jars/netty-all-4.1.17.Final.jar
> jars/netty-3.9.9.Final.jar
>
> Shouldn't one be excluded or perhaps shaded?
>
> Pozdrawiam,
> Jacek Laskowski
> 
> https://about.me/JacekLaskowski
> The Internals of Spark SQL https://bit.ly/spark-sql-internals
> The Internals of Spark Structured Streaming 
> https://bit.ly/spark-structured-streaming
> The Internals of Apache Kafka https://bit.ly/apache-kafka-internals
> Follow me at https://twitter.com/jaceklaskowski
>

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Why two netty libs?

[Jacek Laskowski]

Hi,

Just noticed that Spark 2.4.x uses two netty deps of different versions.
Why?

jars/netty-all-4.1.17.Final.jar
jars/netty-3.9.9.Final.jar

Shouldn't one be excluded or perhaps shaded?

Pozdrawiam,
Jacek Laskowski

https://about.me/JacekLaskowski
The Internals of Spark SQL https://bit.ly/spark-sql-internals
The Internals of Spark Structured Streaming
https://bit.ly/spark-structured-streaming
The Internals of Apache Kafka https://bit.ly/apache-kafka-internals
Follow me at https://twitter.com/jaceklaskowski

Re: [SS] KafkaSource doesn't use KafkaSourceInitialOffsetWriter for initial offsets?

[Jacek Laskowski]

Hi Devs,

Thanks all for a very prompt response! That was insanely quick. Merci
beaucoup! :)

Pozdrawiam,
Jacek Laskowski

https://about.me/JacekLaskowski
The Internals of Spark SQL https://bit.ly/spark-sql-internals
The Internals of Spark Structured Streaming
https://bit.ly/spark-structured-streaming
The Internals of Apache Kafka https://bit.ly/apache-kafka-internals
Follow me at https://twitter.com/jaceklaskowski



On Mon, Aug 26, 2019 at 4:05 PM Jungtaek Lim  wrote:

> Thanks! The patch is here: https://github.com/apache/spark/pull/25583
>
> On Mon, Aug 26, 2019 at 11:02 PM Gabor Somogyi 
> wrote:
>
>> Just checked this and it's a copy-paste :) It works properly when
>> KafkaSourceInitialOffsetWriter used. Pull me in if review needed.
>>
>> BR,
>> G
>>
>>
>> On Mon, Aug 26, 2019 at 3:57 PM Jungtaek Lim  wrote:
>>
>>> Nice finding! I don't see any reason to not use
>>> KafkaSourceInitialOffsetWriter from KafkaSource, as they're identical. I
>>> guess it was copied and pasted sometime before and not addressed yet.
>>> As you haven't submit a patch, I'll submit a patch shortly, with
>>> mentioning credit. I'd close mine and wait for your patch if you plan to do
>>> it. Please let me know.
>>>
>>> Thanks,
>>> Jungtaek Lim (HeartSaVioR)
>>>
>>>
>>> On Mon, Aug 26, 2019 at 8:03 PM Jacek Laskowski  wrote:
>>>
 Hi,

 Just found out that KafkaSource [1] does not
 use KafkaSourceInitialOffsetWriter (of KafkaMicroBatchStream) [2] for
 initial offsets.

 Any reason for that? Should I report an issue? Just checking out as I'm
 with 2.4.3 exclusively and have no idea what's coming for 3.0.

 [1]
 https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala#L102

 [2]
 https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala#L281

 Pozdrawiam,
 Jacek Laskowski
 
 https://about.me/JacekLaskowski
 The Internals of Spark SQL https://bit.ly/spark-sql-internals
 The Internals of Spark Structured Streaming
 https://bit.ly/spark-structured-streaming
 The Internals of Apache Kafka https://bit.ly/apache-kafka-internals
 Follow me at https://twitter.com/jaceklaskowski


>>>
>>> --
>>> Name : Jungtaek Lim
>>> Blog : http://medium.com/@heartsavior
>>> Twitter : http://twitter.com/heartsavior
>>> LinkedIn : http://www.linkedin.com/in/heartsavior
>>>
>>
>
> --
> Name : Jungtaek Lim
> Blog : http://medium.com/@heartsavior
> Twitter : http://twitter.com/heartsavior
> LinkedIn : http://www.linkedin.com/in/heartsavior
>