BUG: spark.readStream .schema(staticSchema) not receiving schema information

Hi, version: spark-3.0.0-preview2-bin-hadoop2.7 As you can see from the code : STEP 1: I create a object of type static frame which holds all the information to the datasource (csv files). STEP 2: Then I create a variable called staticSchema assigning the information of the schema from the

Re: Release Manager's official `branch-3.0` Assessment?

Now the end of March is just around the corner. I'm not qualified to say (and honestly don't know) where we are, but if we were intended to be in blocker mode it doesn't seem to work; lots of developments still happen, and priority/urgency doesn't seem to be applied to the sequence of reviewing.

Re: OFF TOPIC LIST CRITERIA

OK *user support. user@ is DONE !!!* I reported a work around to an existing bug actually to the experienced user. and "the experienced user" was "not aware" of the setting in the log4j.properties so he learned something new too. Clearly neither were you. Also it may surprise some people but

Re: OFF TOPIC LIST CRITERIA

BCC user, dev, and I encourage others to not reply. I said _dev@_ is not for user support. user@ is. You heard that yesterday, too, and not to cross-post. You actually got answers to several questions, despite their tone, from experienced developers of the project. Messages like yours are, I

OFF TOPIC LIST CRITERIA

Sean Owen says the criteria of these two emailing list is not help to support some body who is new but for people who have been using the software for a long time. He is implying I think that I should only send email when I find bugs so that I can help him in his work. A one way street. He is

Re: spark.readStream.schema(??)

I found another bug. Backbutton.co.uk ¯\_(ツ)_/¯ ♡۶Java♡۶RMI ♡۶ Make Use Method {MUM} makeuse.org On Sat, 28 Mar 2020 at 02:12, Zahid Rahman wrote: > > I have sorted the error anyway because I am the best there is. > It is downhill for me from here. > > There is

spark.readStream.schema(??)

version: spark-3.0.0-preview2-bin-hadoop2.7 The syntax checker objects to the following argument which is what I am supposed to enter. .schema(staticSchema) However when I provide the following argument it works but I don't think that is correct. What is the correct argument for this case ?

Re: what a plava !

- dev@, which is more for project devs to communicate. Cross-posting is discouraged too. The book isn't from the Spark OSS project, so not really the place to give feedback here. I don't quite understand the context of your other questions, but would elaborate them in individual, clear emails

what a plava !

I was very impressed with the amount of material available from https://github.com/databricks/Spark-The-Definitive-Guide/ Over 450+ * megabytes.*

Re: Programmatic: parquet file corruption error

Thanks Wenchen. SOLVED! KINDA! I removed all dependencies from the pom.xml in my IDE so I wouldn't be picking up any libraries from maven repository. I *instead* included the libraries (jar) from the *spark download* of *spark-3.0.0-preview2-bin-hadoop2.7* This way I am using the *same

[DISCUSS][K8s] Copy files securely to the pods or containers.

Hello All, The issue SPARK-23153 lets us copy any file to the pod/container, by first copying it to a hadoop supported filesystem e.g. HDFS, s3, cos etc. This is especially useful if, the files have to be copied to large number of pods/nodes.

Re: Programmatic: parquet file corruption error

Running Spark application with an IDE is not officially supported. It may work under some cases but there is no guarantee at all. The official way is to run interactive queries with spark-shell or package your application to a jar and use spark-submit. On Thu, Mar 26, 2020 at 4:12 PM Zahid Rahman

Re: BUG: take with SparkSession.master[url]

~/spark-3.0.0-preview2-bin-hadoop2.7$ sbin/start-slave.sh spark:// 192.168.0.38:7077 ~/spark-3.0.0-preview2-bin-hadoop2.7$ sbin/start-master.sh Backbutton.co.uk ¯\_(ツ)_/¯ ♡۶Java♡۶RMI ♡۶ Make Use Method {MUM} makeuse.org On Fri, 27 Mar 2020 at 06:12, Zahid Rahman

Re: BUG: take with SparkSession.master[url]

sbin/start-master.sh sbin/start-slave.sh spark://192.168.0.38:7077 Backbutton.co.uk ¯\_(ツ)_/¯ ♡۶Java♡۶RMI ♡۶ Make Use Method {MUM} makeuse.org On Fri, 27 Mar 2020 at 05:59, Wenchen Fan wrote: > Your Spark cluster, spark://192.168.0.38:7077, how is it deployed if

Re: BUG: take with SparkSession.master[url]

Your Spark cluster, spark://192.168.0.38:7077, how is it deployed if you just include Spark dependency in IntelliJ? On Fri, Mar 27, 2020 at 1:54 PM Zahid Rahman wrote: > I have configured in IntelliJ as external jars > spark-3.0.0-preview2-bin-hadoop2.7/jar > > not pulling anything from maven.