RE: [EXTERNAL] Re: [Spark vulnerability] replace jackson-mapper-asl

2022-12-15 Thread haibo.w...@morganstanley.com 
Hi Owen

As confirmed with our firm appsec team, given the library is still being used 
in spark3.3.1. Also I can see the dependency as below:
https://github.com/apache/spark/blob/v3.3.1/pom.xml#L1784

Something misunderstanding? appreciate if you could clarify more, thanks.

Regards
Harper

From: Sean Owen 
Sent: Wednesday, December 14, 2022 10:27 PM
To: Wang, Harper (FRPPE) 
Cc: user@spark.apache.org
Subject: Re: [EXTERNAL] Re: [Spark vulnerability] replace jackson-mapper-asl

The CVE you mention seems to affect jackson-databind, not jackson-mapper-asl.  
3.3.1 already uses databind 2.13.x which is not affected.

On Wed, Dec 14, 2022 at 8:20 AM 
haibo.w...@morganstanley.com 
mailto:haibo.w...@morganstanley.com>> wrote:
Thanks Owen for prompt response
sorry, forgot to mention, it’s latest spark version 3.3.1
Both below spark-py image  or pypi are good to use for us, but both have same 
Jackson-mapper-asl dependencies.

https://hub.docker.com/layers/apache/spark-py/3.3.1/images/sha256-0d4fd8bcb2ad63a35c9ba5be278a3a34c28fc15e898307e458d501a7e11d6d51?context=explore
https://pypi.org/project/pyspark/

Regards
Harper


From: Sean Owen mailto:sro...@gmail.com>>
Sent: Wednesday, December 14, 2022 9:32 PM
To: Wang, Harper (FRPPE) 
mailto:haibo.w...@morganstanley.com>>
Cc: user@spark.apache.org
Subject: [EXTERNAL] Re: [Spark vulnerability] replace jackson-mapper-asl

What Spark version are you referring to? If it's an unsupported version, no, no 
plans to update it.
What image are you referring to?

On Wed, Dec 14, 2022 at 7:14 AM 
haibo.w...@morganstanley.com 
mailto:haibo.w...@morganstanley.com>> wrote:
Hi All

Hope you are doing well.

Writing this email for an vulnerable issue: CVE-2018-14721
apache/spark-py: 
gav://org.codehaus.jackson:jackson-mapper-asl:1.9.13,CVE-2018-14721,1.8.10-cloudera.2,1.5.0
 <= Version <= 1.9.13

We are trying to bring in above image into our firm, but due to the vulnerable 
issue, pyspark is not allowed, understand  the version was stopped maintaining 
in 2013, wondering any plan to replace the Jackson-mapper-asl or any 
workaround? thanks

Regards
Harper Wang
Morgan Stanley | Corporate & Funding Technology
Kerry Parkside | 1155 Fang Dian Road, Pudong New Area
201204 Shanghai
haibo.w...@morganstanley.com



NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or 
views contained herein are not intended to be, and do not constitute, advice 
within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and 
Consumer Protection Act. By communicating with Morgan Stanley you acknowledge 
that you have read, understand and consent, (where applicable), to the Morgan 
Stanley General Disclaimers found at 
http://www.morganstanley.com/disclaimers/terms. The entire content of this 
email message and any files attached to it may be sensitive, confidential, 
subject to legal privilege and/or otherwise protected from disclosure.


NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or 
views contained herein are not intended to be, and do not constitute, advice 
within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and 
Consumer Protection Act. By communicating with Morgan Stanley you acknowledge 
that you have read, understand and consent, (where applicable), to the Morgan 
Stanley General Disclaimers found at 
http://www.morganstanley.com/disclaimers/terms. The entire content of this 
email message and any files attached to it may be sensitive, confidential, 
subject to legal privilege and/or otherwise protected from disclosure.


NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or 
views contained herein are not intended to be, and do not constitute, advice 
within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and 
Consumer Protection Act. By communicating with Morgan Stanley you acknowledge 
that you have read, understand and consent, (where applicable), to the Morgan 
Stanley General Disclaimers found at 
http://www.morganstanley.com/disclaimers/terms. The entire content of this 
email message and any files attached to it may be sensitive, confidential, 
subject to legal privilege and/or otherwise protected from disclosure.

Re: [EXTERNAL] Re: [Spark vulnerability] replace jackson-mapper-asl

2022-12-15 Thread Sean Owen 
Please read the CVE you mention. It is not a CVE about the library you are
referencing.
https://nvd.nist.gov/vuln/detail/CVE-2018-14721


On Thu, Dec 15, 2022 at 7:52 PM haibo.w...@morganstanley.com <
haibo.w...@morganstanley.com> wrote:

> Hi Owen
>
>
>
> As confirmed with our firm appsec team, given the library is still being
> used in spark3.3.1. Also I can see the dependency as below:
>
> https://github.com/apache/spark/blob/v3.3.1/pom.xml#L1784
>
>
>
> Something misunderstanding? appreciate if you could clarify more, thanks.
>
>
>
> Regards
>
> Harper
>
>
>
> *From:* Sean Owen 
> *Sent:* Wednesday, December 14, 2022 10:27 PM
> *To:* Wang, Harper (FRPPE) 
> *Cc:* user@spark.apache.org
> *Subject:* Re: [EXTERNAL] Re: [Spark vulnerability] replace
> jackson-mapper-asl
>
>
>
> The CVE you mention seems to affect jackson-databind, not
> jackson-mapper-asl.  3.3.1 already uses databind 2.13.x which is not
> affected.
>
>
>
> On Wed, Dec 14, 2022 at 8:20 AM haibo.w...@morganstanley.com <
> haibo.w...@morganstanley.com> wrote:
>
> Thanks Owen for prompt response
>
> sorry, forgot to mention, it’s latest spark version 3.3.1
>
> Both below spark-py image  or pypi are good to use for us, but both have
> same Jackson-mapper-asl dependencies.
>
>
>
>
> https://hub.docker.com/layers/apache/spark-py/3.3.1/images/sha256-0d4fd8bcb2ad63a35c9ba5be278a3a34c28fc15e898307e458d501a7e11d6d51?context=explore
>
> https://pypi.org/project/pyspark/
>
>
>
> Regards
>
> Harper
>
>
>
>
>
> *From:* Sean Owen 
> *Sent:* Wednesday, December 14, 2022 9:32 PM
> *To:* Wang, Harper (FRPPE) 
> *Cc:* user@spark.apache.org
> *Subject:* [EXTERNAL] Re: [Spark vulnerability] replace jackson-mapper-asl
>
>
>
> What Spark version are you referring to? If it's an unsupported version,
> no, no plans to update it.
>
> What image are you referring to?
>
>
>
> On Wed, Dec 14, 2022 at 7:14 AM haibo.w...@morganstanley.com <
> haibo.w...@morganstanley.com> wrote:
>
> Hi All
>
>
>
> Hope you are doing well.
>
>
>
> Writing this email for an vulnerable issue: CVE-2018-14721
>
> apache/spark-py:
> gav://org.codehaus.jackson:jackson-mapper-asl:1.9.13,CVE-2018-14721,1.8.10-cloudera.2,1.5.0
> <= Version <= 1.9.13
>
>
>
> We are trying to bring in above image into our firm, but due to the
> vulnerable issue, pyspark is not allowed, understand  the version was
> stopped maintaining in 2013, wondering any plan to replace the
> Jackson-mapper-asl or any workaround? thanks
>
>
>
> Regards
>
> Harper Wang
>
> *Morgan Stanley | Corporate & Funding Technology*Kerry Parkside |
> 1155 Fang Dian Road, Pudong New Area
> 201204 Shanghai
> haibo.w...@morganstanley.com
>
>
>
> --
>
> NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> opinions or views contained herein are not intended to be, and do not
> constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall
> Street Reform and Consumer Protection Act. By communicating with Morgan
> Stanley you acknowledge that you have read, understand and consent, (where
> applicable), to the Morgan Stanley General Disclaimers found at
> http://www.morganstanley.com/disclaimers/terms. The entire content of
> this email message and any files attached to it may be sensitive,
> confidential, subject to legal privilege and/or otherwise protected from
> disclosure.
>
>
> --
>
> NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> opinions or views contained herein are not intended to be, and do not
> constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall
> Street Reform and Consumer Protection Act. By communicating with Morgan
> Stanley you acknowledge that you have read, understand and consent, (where
> applicable), to the Morgan Stanley General Disclaimers found at
> http://www.morganstanley.com/disclaimers/terms. The entire content of
> this email message and any files attached to it may be sensitive,
> confidential, subject to legal privilege and/or otherwise protected from
> disclosure.
>
>
> --
> NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> opinions or views contained herein are not intended to be, and do not
> constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall
> Street Reform and Consumer Protection Act. By communicating with Morgan
> Stanley you acknowledge that you have read, understand and consent, (where
> applicable), to the Morgan Stanley General Disclaimers found at
> http://www.morganstanley.com/disclaimers/terms. The entire content of
> this email message and any files attached to it may be sensitive,
> confidential, subject to legal privilege and/or otherwise protected from
> disclosure.
>
>

UNSUBSCRIBE

2022-12-15 Thread prashanth t 
UNSUBSCRIBE

Re: Spark-on-Yarn ClassNotFound Exception

2022-12-15 Thread scrypso 
Hmm, did you mean spark.*driver*.extraClassPath? That is very odd then - if
you check the logs directory for the driver (on the cluster) I think there
should be a launch container log, where you can see the exact command used
to start the JVM (at the very end), and a line starting "export CLASSPATH"
- you can double check that your jar looks to be included correctly there.
If it is I think you have a really "interesting" issue on your hands!

- scrypso

On Wed, Dec 14, 2022, 05:17 Hariharan  wrote:

> Hi scrypso,
>
> Thanks for the help so far, and I think you're definitely on to something
> here. I tried loading the class as you suggested with the code below:
>
> try {
> 
> Thread.currentThread().getContextClassLoader().loadClass(MyS3ClientFactory.class.getCanonicalName());
> logger.info("Loaded custom class");
> } catch (ClassNotFoundException e) {
> logger.error("Unable to load class", e);
> }
> return spark.read().option("mode", 
> "DROPMALFORMED").format("avro").load();
>
> I am able to load the custom class as above
> *2022-12-14 04:12:34,158 INFO  [Driver] utils.S3Reader - Loaded custom
> class*
>
> But the spark.read code below it tries to initialize the s3 client and is
> not able to load the same class.
>
> I tried adding
> *--conf spark.executor.extraClassPath=myjar*
>
> as well, but no luck :-(
>
> Thanks again!
>
> On Tue, Dec 13, 2022 at 10:09 PM scrypso  wrote:
>
>> I'm on my phone, so can't compare with the Spark source, but that looks
>> to me like it should be well after the ctx loader has been set. You could
>> try printing the classpath of the loader
>> Thread.currentThread().getThreadContextClassLoader(), or try to load your
>> class from that yourself to see if you get the same error.
>>
>> Can you see which thread is throwing the exception? If it is a different
>> thread than the "main" application thread it might not have the thread ctx
>> loader set correctly. I can't see any of your classes in the stacktrace - I
>> assume that is because of your scrubbing, but it could also be because this
>> is run in separate thread without ctx loader set.
>>
>> It also looks like Hadoop is caching the FileSystems somehow - perhaps
>> you can create the S3A filesystem yourself and hope it picks that up? (Wild
>> guess, no idea if that works or how hard it would be.)
>>
>>
>> On Tue, Dec 13, 2022, 17:29 Hariharan  wrote:
>>
>>> Thanks for the response, scrypso! I will try adding the extraClassPath
>>> option. Meanwhile, please find the full stack trace below (I have
>>> masked/removed references to proprietary code)
>>>
>>> java.lang.RuntimeException: java.lang.RuntimeException:
>>> java.lang.ClassNotFoundException: Class foo.bar.MyS3ClientFactory not found
>>> at
>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2720)
>>> at
>>> org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:888)
>>> at
>>> org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:542)
>>> at
>>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469)
>>> at
>>> org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
>>> at
>>> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
>>> at
>>> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
>>> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
>>> at
>>> org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$checkAndGlobPathIfNecessary$1(DataSource.scala:752)
>>> at scala.collection.immutable.List.map(List.scala:293)
>>> at
>>> org.apache.spark.sql.execution.datasources.DataSource$.checkAndGlobPathIfNecessary(DataSource.scala:750)
>>> at
>>> org.apache.spark.sql.execution.datasources.DataSource.checkAndGlobPathIfNecessary(DataSource.scala:579)
>>> at
>>> org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:408)
>>> at
>>> org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:228)
>>> at
>>> org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:210)
>>> at scala.Option.getOrElse(Option.scala:189)
>>> at
>>> org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:210)
>>>
>>> Thanks again!
>>>
>>> On Tue, Dec 13, 2022 at 9:52 PM scrypso  wrote:
>>>
 Two ideas you could try:

 You can try spark.driver.extraClassPath as well. Spark loads the user's
 jar in a child classloader, so Spark/Yarn/Hadoop can only see your classes
 reflectively. Hadoop's Configuration should use the thread ctx classloader,
 and Spark should set that to the loader that loads your jar. The
 extraClassPath option just adds jars directly to the Java command that
 creates the driver/executor.

 I can't immediately tell how your error might arise,

Re: Query regarding Apache spark version 3.0.1

2022-12-15 Thread Sean Owen 
Do you mean, when is branch 3.0.x EOL? It was EOL around the end of 2021.
But there were releases 3.0.2 and 3.0.3 beyond 3.0.1, so not clear what you
mean by support for 3.0.1.

On Thu, Dec 15, 2022 at 9:53 AM Pranav Kumar (EXT)
 wrote:

> Hi Team,
>
>
>
> Could you please help us to know when version 3.0.1 for Apache spark is
> going to be EOS? Till when we are going to get fixes for the version 3.0.1.
>
>
>
> Regards,
>
> Pranav
>
>
>

Query regarding Apache spark version 3.0.1

2022-12-15 Thread Pranav Kumar (EXT) 
Hi Team,

Could you please help us to know when version 3.0.1 for Apache spark is going 
to be EOS? Till when we are going to get fixes for the version 3.0.1.

Regards,
Pranav