Spark driver thread

Hi, Does a Spark driver always works as single threaded? If yes, does it mean asking for more than one vCPU for the driver is wasteful? Thanks, James

Re: Can't get Spark to interface with S3A Filesystem with correct credentials

Thanks for the input Steven and Hariharan. I think this ended up being a combination of bad configuration with the credential providers I was using *and* using the wrong set of credentials for the test data I was trying to access. I was able to get this working with both hadoop 2.8 and 3.1 by

Re: Hostname :BUG

Talking about copy and paste Larry Tesler The *inventor* of *cut*/*copy* & *paste*, find & replace past away last week age 74. Backbutton.co.uk ¯\_(ツ)_/¯ ♡۶Java♡۶RMI ♡۶ Make Use Method {MUM} makeuse.org On Thu, 5 Mar 2020 at 07:01, Zahid Rahman wrote: > Please

Re: Stateful Structured Spark Streaming: Timeout is not getting triggered

Yes that was it! It seems it only works if input data is continuously flowing. I had stopped the input job because I had enough data but it seems timeouts work only if the data is continuously fed. Not sure why it's designed that way. Makes it a bit harder to write unit/integration tests BUT I

Re: Read Hive ACID Managed table in Spark

You can try using have warehouse connector https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.5/integrating-hive/content/hive_hivewarehouseconnector_for_handling_apache_spark_data.html On Thu, Mar 5, 2020, 6:51 AM Chetan Khatri wrote: > Just followup, if anyone has worried on this before > >

Re: Read Hive ACID Managed table in Spark

Just followup, if anyone has worried on this before On Wed, Mar 4, 2020 at 12:09 PM Chetan Khatri wrote: > Hi Spark Users, > I want to read Hive ACID managed table data (ORC) in Spark. Can someone > help me here. > I've tried, https://github.com/qubole/spark-acid but no success. > > Thanks >

Re: SPARK Suitable IDE

Holden Karau writes: > I work in emacs with ensime. the ensime project was stoped and the project archived. its successor "metals" works well for scala >=2.12 any good ressource to setup ensime with emacs ? can't wait overall spark community goes on scala 2.12 -- nicolas paris

Re: SPARK Suitable IDE

There are indications on the Internet that Jupyter Notebook offers an advantage when working with SPARK technologies. I was wondering if there was any substance to these claims , if there was substance to the claims then I would proceed to get comfortable with Jupyter Notebook. Regards Zahid