Re: How can I get the same spark context in two different python processes

sion.builder.appName("spark").getOrCreate() > > t1 = Thread(target=my_func, args=(context,)) > t1.start() > > t2 = Thread(target=my_func, args=(context,)) > t2.start() > > On Tue, Dec 13, 2022 at 4:10 PM Jack Goodson > wrote: > >> Hi Kevin, >> >

Re: How can I get the same spark context in two different python processes

definitely not something that we test or > support, especially in a scenario you described. > > If you want to achieve concurrent execution, multithreading is normally > more than sufficient and avoids problems with the context. > > > > On 12/13/22 00:40, Kevin Su wrote: > >

Re: How can I get the same spark context in two different python processes

could connect to that kernel. > > But in the end, this is like Spark Connect :) > > > On Mon, Dec 12, 2022 at 2:55 PM Kevin Su wrote: > >> Also, is there any way to workaround this issue without using Spark >> connect? >> >> Kevin Su 於 2022年12月12日 週一 下午2:

Re: How can I get the same spark context in two different python processes

Also, is there any way to workaround this issue without using Spark connect? Kevin Su 於 2022年12月12日 週一 下午2:52寫道： > nvm, I found the ticket. > Also, is there any way to workaround this issue without using Spark > connect? > > Kevin Su 於 2022年12月12日 週一 下午2:42寫道： > >&

How can I get the same spark context in two different python processes

Hey there, How can I get the same spark context in two different python processes? Let’s say I create a context in Process A, and then I want to use python subprocess B to get the spark context created by Process A. How can I achieve that? I've tried pyspark.sql.SparkSession.builder.appName("spark

How to run spark benchmark on standalone cluster?

Hi all, I want to run spark benchmark on a standalone cluster, and I have changed the DataSourceReadBenchmark.scala setting. (Remove "spark.master") --- a/sql/core/src/test /scala/org/apache/spark/sql/execution/benchmark/DataSourceReadBenchmark.scala +++ b/sql/core/src/test /scala/org/apache/spar

Fail to run benchmark in Github Action

Hi all, I try to run a benchmark test in GitHub action in my fork, and I faced the below error. https://github.com/pingsutw/spark/runs/2867617238?check_suite_focus=true java.lang.AssertionError: assertion failed: spark.test.home is not set! 23799

Re: Java Code Style

tyle in this case. Sometimes it's a matter of judgment > when to differ from a standard style for better readability. > > On Sat, Feb 20, 2021 at 8:53 AM Pis Kevin wrote: > >> Hi, >> >> >> >> I use google java code style in intellj idea. But when I refor

Java Code Style

Hi, I use google java code style in intellj idea. But when I reformat the following codes, its inconsistent with the code in spark. Before reformat: [cid:image001.png@01D707DA.2FACD5C0] After reformat: [cid:image002.png@01D707DA.2FACD5C0] Why? And how to fix the issue.

RE: Kicking off the process around Spark 2.2.1

Any update on expected 2.2.1 (or 2.3.0) release process? From: Felix Cheung [mailto:felixcheun...@hotmail.com] Sent: Thursday, October 26, 2017 10:04 AM To: Sean Owen ; Holden Karau Cc: dev@spark.apache.org Subject: Re: Kicking off the process around Spark 2.2.1 Yes! I can take on RM for 2.2.1.

Re: Welcoming Saisai (Jerry) Shao as a committer

Congratulations, Jerry! On Tue, Aug 29, 2017 at 6:35 AM, Meisam Fathi wrote: > Congratulations, Jerry! > > Thanks, > Meisam > > On Tue, Aug 29, 2017 at 1:13 AM Wang, Carson > wrote: > >> Congratulations, Saisai! >> >> >> -Original Message- >> From: Matei Zaharia [mailto:matei.zaha...@gm

Re: Welcoming Hyukjin Kwon and Sameer Agarwal as committers

Thanks for all of your hard work, Hyukjin and Sameer. Congratulations!! 2017년 8월 8일 (화) 오전 9:44, Hyukjin Kwon 님이 작성: > Thank you all. Will do my best! > > 2017-08-08 8:53 GMT+09:00 Holden Karau : > >> Congrats! >> >> On Mon, Aug 7, 2017 at 3:54 PM Bryan Cutler wrote: >> >>> Great work Hyukjin a

Re: welcoming Xiao Li as a committer

Congratulations Xiao!! Sent from my iPhone > On Oct 4, 2016, at 3:59 AM, Tarun Kumar wrote: > > Congrats Xiao. > > Thanks > Tarun >> On Tue, 4 Oct 2016 at 12:57 PM, Cheng Lian wrote: >> Congratulations!!! >> >> >> Cheng >> >> On Tue, Oct 4, 2016 at 1:46 PM, Reynold Xin wrote: >> Hi all, >

RE: regression: no longer able to use HDFS wasbs:// path for additional python files on LIVY batch submit

Great. Thanks for the pointer. I see the fix is in 2.0.1-rc4. Will there be a 1.6.3? If so, how are fixes considered for backporting? From: Steve Loughran [mailto:ste...@hortonworks.com] Sent: Monday, October 3, 2016 5:40 AM To: Kevin Grealish Cc: Apache Spark Dev Subject: Re: regression: no

regression: no longer able to use HDFS wasbs:// path for additional python files on LIVY batch submit

I'm seeing a regression when submitting a batch PySpark program with additional files using LIVY. This is YARN cluster mode. The program files are placed into the mounted Azure Storage before making the call to LIVY. This is happening from an application which has credentials for the storage and

Re: tpcds for spark2.0

29 21:17 GMT+08:00 Olivier Girardot : > I have the same kind of issue (not using spark-sql-perf), just trying to > deploy 2.0.0 on mesos. > I'll keep you posted as I investigate > > > > On Wed, Jul 27, 2016 1:06 PM, kevin kiss.kevin...@gmail.com wrote: > >> hi,all:

tpcds for spark2.0

hi,all: I want to have a test about tpcds99 sql run on spark2.0. I user https://github.com/databricks/spark-sql-perf about the master version ,when I run :val tpcds = new TPCDS (sqlContext = sqlContext) I got error: scala> val tpcds = new TPCDS (sqlContext = sqlContext) error: missing or invalid

Re: spark2.0 how to use sparksession and StreamingContext same time

thanks a lot Terry 2016-07-26 12:03 GMT+08:00 Terry Hoo : > Kevin, > > Try to create the StreamingContext as following: > > val ssc = new StreamingContext(spark.sparkContext, Seconds(2)) > > > > On Tue, Jul 26, 2016 at 11:25 AM, kevin wrote: > >> hi,all: &

spark2.0 how to use sparksession and StreamingContext same time

hi,all: I want to read data from kafka and regist as a table then join a jdbc table. My sample like this : val spark = SparkSession .builder .config(sparkConf) .getOrCreate() val jdbcDF = spark.read.format("jdbc").options(Map("url" -> "jdbc:mysql://master1:3306/demo", "drive

Re: Odp.: spark2.0 can't run SqlNetworkWordCount

------- > *Od:* kevin > *Wysłane:* 25 lipca 2016 11:33 > *Do:* user.spark; dev.spark > *Temat:* spark2.0 can't run SqlNetworkWordCount > > hi,all: > I download spark2.0 per-build. I can run SqlNetworkWordCount test use : > bin/run-example org.apache.spark.exa

Re: where I can find spark-streaming-kafka for spark2.0

1.6. There is also Kafka 0.10 > support in > > dstream. > > > > On July 25, 2016 at 10:26:49 AM, Andy Davidson > > (a...@santacruzintegration.com) wrote: > > > > Hi Kevin > > > > Just a heads up at the recent spark summit in S.F. There was a > presen

spark2.0 can't run SqlNetworkWordCount

hi,all: I download spark2.0 per-build. I can run SqlNetworkWordCount test use : bin/run-example org.apache.spark.examples.streaming.SqlNetworkWordCount master1 but when I use spark2.0 example source code SqlNetworkWordCount.scala and build it to a jar bao with dependencies ( JDK 1.8 AND SCALA

Re: where I can find spark-streaming-kafka for spark2.0

I have compile it from source code 2016-07-25 12:05 GMT+08:00 kevin : > hi,all : > I try to run example org.apache.spark.examples.streaming.KafkaWordCount , > I got error : > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/spark/streami

where I can find spark-streaming-kafka for spark2.0

hi,all : I try to run example org.apache.spark.examples.streaming.KafkaWordCount , I got error : Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/streaming/kafka/KafkaUtils$ at org.apache.spark.examples.streaming.KafkaWordCount$.main(KafkaWordCount.scala:57) at org.apache

spark1.6.2 ClassNotFoundException: org.apache.parquet.hadoop.ParquetOutputCommitter

hi,all: I build spark 1.6.2 frpm source with : ./make-distribution.sh --name "hadoop2.7.1" --tgz "-Pyarn,hadoop-2.6,parquet-provided,hive,hive-thriftserver" -DskipTests -Dhadoop.version=2.7.1 when I try to run : ./bin/run-example sql.RDDRelation or ./spark-shell I met the error with :(but I can r

Re: New Spark json endpoints

Thank you all for the feedback. I’ve created a corresponding JIRA ticket at https://issues.apache.org/jira/browse/SPARK-10565, updated with a summary of this thread. From: Mark Hamstra Date: Thursday, September 17, 2015 at 8:00 AM To: Imran Rashid Cc: Kevin Chen , "dev@spark.apach

Re: New Spark json endpoints

Just wanted to bring this email up again in case there were any thoughts. Having all the information from the web UI accessible through a supported json API is very important to us; are there any objections to us adding a v2 API to Spark? Thanks! From: Kevin Chen Date: Friday, September 11

New Spark json endpoints

any specific reason that these fields are not exposed in the public API? If not, would it be reasonable to add them to the json blobs, possibly in a future /api/v2 API? Thank you, Kevin Chen smime.p7s Description: S/MIME cryptographic signature

Re: Deserializing JSON into Scala objects in Java code

Marcelo and Christopher, Thanks for your help! The problem turned out to arise from a different part of the code (we have multiple ObjectMappers), but because I am not very familiar with Jackson I had thought there was a problem with the Scala module. Thank you again, Kevin From: Christopher

Re: Deserializing JSON into Scala objects in Java code

PM, "Marcelo Vanzin" wrote: >Hi Kevin, > >How did you try to use the Scala module? Spark has this code when >setting up the ObjectMapper used to generate the output: > > >mapper.registerModule(com.fasterxml.jackson.module.scala.DefaultScalaModul >e) > >A

Deserializing JSON into Scala objects in Java code

for Scala Seqs, but got no luck. * Deserialization works if the Scala class does not have any Seq fields, and works if the fields are Java Lists instead of Seqs. Thanks very much for your help! Kevin Chen smime.p7s Description: S/MIME cryptographic signature

Re: Change for submitting to yarn in 1.3.1

ctions via this channel. Chester Sent from my iPad On May 12, 2015, at 20:54, Patrick Wendell wrote: Hey Kevin and Ron, So is the main shortcoming of the launcher library the inability to get an app ID back from YARN? Or are there other issues here that fundame

Re: Change for submitting to yarn in 1.3.1

aunches it. I am still reading the newest code, and we are still researching options to move forward. If there are alternatives, we'd like to know. Kevin Markey On 05/11/2015 01:36 AM, Mridul Muralidharan wrote: That works when it is launched from same process - which is unfortunate

Re: enum-like types in Spark

() as the key instead. Oh well. But it works and seems to work well. Kevin On 03/05/2015 09:49 PM, Mridul Muralidharan wrote: I have a strong dislike for java enum's due to the fact that they are not stable across JVM's - if it undergoes serde, you end up with unpredictable result

Re: [VOTE] Release Apache Spark 1.1.1 (RC1)

for added latency. Not sure if added latency a function of 1.0 vs 1.1 or 1.0 vs 1.1.1 changes, as we've never tested with 1.1.0. But thought I'd share the results. (This is somewhat disappointing.) Kevin Markey On 11/17/2014 11:42 AM, Debasish Das wrote: Andrew, I put up 1.1.1 br

Re: [VOTE] Release Apache Spark 1.0.0 (RC11)

+1 Built -Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0 Ran current version of one of my applications on 1-node pseudocluster (sorry, unable to test on full cluster). yarn-cluster mode Ran regression tests. Thanks Kevin On 05/28/2014 09:55 PM, Krishna Sankar wrote: +1 Pulled & built on Mac

Re: [VOTE] Release Apache Spark 1.0.0 (RC10)

Thank you, all! This is quite helpful. We have been arguing how to handle this issue across a growing application. Unfortunately the Hadoop FileSystem java doc should say all this but doesn't! Kevin On 05/22/2014 01:48 PM, Aaron Davidson wrote: In Spark 0.9.0 and 0.9.1, we stopped

Re: [VOTE] Release Apache Spark 1.0.0 (RC10)

ue ONLY to my user error. The FS closed exception only effects the cleanup of the staging directory, not the final success or failure. I've not yet tested the effect of changing my application's initialization, use, or closing of FileSystem. Thanks again. Kevin On 05/22/2014 01

Re: [VOTE] Release Apache Spark 1.0.0 (RC10)

r error. I did not attempt to test my fix without the patch, so I can't pass judgment on the patch. Although this is merely a pseudocluster based test -- I can't reconfigure our cluster with RC-10 -- I'll now change my vote to... +1. Thanks all who helped. Kevin On 05/21/20

Re: [VOTE] Release Apache Spark 1.0.0 (RC10)

rectory. Also, where before Yarn would report the running program as "RUNNING", it only reported this application as "ACCEPTED". It appeared to run two containers when the first instance never reported that it was RUNNING. I will post a separate note to the USER list about

Re: [VOTE] Release Apache Spark 0.9.1 (RC3)

ve various ASM exclusions from other libraries, recompiling and redeploying. But I'd incur the wrath of the rest of my team doing that, especially after a full day of tracking down yet another (totally unrelated) library conflict. Thanks for this maintenance release. Kevin Markey On 03/3

Re: Spark 0.9.1 release

es can handle it, but the main build goes to Maven Central via our corporate Artifactory mirror. If suddenly we had a Spark 0.9.1 with a shaded ASM, it would have no negative impact on us. Only a positive impact. I just wish that all users of ASM would read FAQ entry 15!!! Thanks Kevin On 03/2

Re: Spark 0.9.1 release

only 6 weeks away! (For those not following 782, according to Jira comments, the SBT build shades it, but it is the Maven build that ends up in Maven Central.) Thanks Kevin Markey On 03/19/2014 06:07 PM, Tathagata Das wrote: Hello everyone, Since the release of Spark 0.9, we have recei

Re: Spark 0.9.1 release

1051 is essential! I'm not sure about the others, but anything that adds stability to Spark/Yarn would be helpful. Kevin Markey On 03/20/2014 01:12 PM, Tom Graves wrote: I'll pull [SPARK-1053] Should not require SPARK_YARN_APP_JAR when running on YARN - JIRA and [SPARK-105

Re: [DISCUSS] Necessity of Maven *and* SBT Build in Spark

the Maven build should be given more priority than at present. It seems a bit odd, if a Maven project can be automatically generated from SBT, that it would take 1 year for ASM shading in Maven to catch up with SBT. Thanks Kevin Markey SBT appears to have syntax for both, just like Maven. Su