Re: Override jars in spark submit

Hi Naresh, You could use "--conf spark.driver.extraClassPath=". Note that the jar will not be shipped to the executors, if its a class that is needed on the executors as well you should provide "--conf spark.executor.extraClassPath=". Note that if you do provide executor extraclasspath the jar

connecting spark with mysql

Hi everyone, I tried to manipulate MySQL tables from spark, I do not want to move these tables from MySQL to spark, as these tables can easily get very big. It is ideal that the data stays in the database where it was stored. For me, spark is only used to speed up the read and write process

Re: Spark SQL

Just to make it more clear, Spark sql uses hive metastore and run queries using its own engine and not uses hive execution engine. Please correct me if it’s not true. On Mon, Jun 10, 2019 at 2:29 PM Russell Spitzer wrote: > Spark can use the HiveMetastore as a catalog, but it doesn't use

Override jars in spark submit

Hello All, How can we override jars in spark submit? We have hive-exec-spark jar which is available as part of default spark cluster jars. We wanted to override above mentioned jar in spark submit with latest version jar. How do we do that ? Thank you, Naresh -- Thanks, Naresh

Re: Announcing Delta Lake 0.2.0

Hi We are using Delta features. The only problem we faced till now is Hive can not read DELTA outputs by itself (even if the Hive metastore is shared). However, if we create hive external table pointing to the folder (and with Vacuum), it can read the data. Other than that, the feature looks

Re: Announcing Delta Lake 0.2.0

Hi Gourav, Thanks for the suggestion. Please open a Github issue at https://github.com/delta-io/delta/issues to describe your use case and requirements for "external tables" so we can better track this feature and also get feedback from the community. Regards, Liwen On Wed, Jun 19, 2019 at

Re: Announcing Delta Lake 0.2.0

Hi, does Delta support external tables? I think that most users will be needing this. Regards, Gourav On Wed, Jun 19, 2019 at 8:04 PM Liwen Sun wrote: > We are delighted to announce the availability of Delta Lake 0.2.0! > > To try out Delta Lake 0.2.0, please follow the Delta Lake

Re: Announcing Delta Lake 0.2.0

Hi, this is fantastic :) Regards, Gourav Sengupta On Wed, Jun 19, 2019 at 8:04 PM Liwen Sun wrote: > We are delighted to announce the availability of Delta Lake 0.2.0! > > To try out Delta Lake 0.2.0, please follow the Delta Lake Quickstart: > https://docs.delta.io/0.2.0/quick-start.html > >

Announcing Delta Lake 0.2.0

We are delighted to announce the availability of Delta Lake 0.2.0! To try out Delta Lake 0.2.0, please follow the Delta Lake Quickstart: https://docs.delta.io/0.2.0/quick-start.html To view the release notes: https://github.com/delta-io/delta/releases/tag/v0.2.0 This release introduces two main

pyspark cached dataframe shows deserialized at StorageLevel

Hi, Spark document describes "Since the data is always serialized on the Python side, all the constants use the serialized formats.". http://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.StorageLevel But when I cached dataframe and looked StorageLevel, it shows that cached

RE: tcps oracle connection from spark

Connecting to Oracle from Spark using the TPCS protocol works OK for me. Maybe try to turn debug on with -Djavax.net.debug=all? See also: https://blogs.oracle.com/dev2dev/ssl-connection-to-oracle-db-using-jdbc%2c-tlsv12%2c-jks-or-oracle-wallets Regards, L. From: Richard Xin Sent: Wednesday,

[webinar] TFX Chicago Taxi example on Mini Kubeflow (MiniKF)

Hi all, I would like to invite you to our webinar "Kubeflow Pipelines on-prem". It will take place on Friday, June 21 at 9am Pacific Time. You can register here: https://zoom.us/webinar/register/WN_j_HJbkISTluMckyyr706eg We are going to demonstrate the end-to-end TFX Chicago Taxi example

Unsubscribe

-- Tushar Marne 9011062432

Re: Ask for ARM CI for spark

Thanks for your reply. As I said before, I met some problem of build or test for spark on aarch64 server, so it will be better to have the ARM CI to make sure the spark is compatible for AArch64 platforms. I’m from OpenLab team(https://openlabtesting.org/ ,a community to do open source project

RE: Unable to run simple spark-sql

Hi Raymond, I cross checked hive/conf/hive-site.xml and spark2/conf/hive-site.xml Same value is being shown by Ambari Hive config. Seems correct value here: hive.metastore.warehouse.dir /apps/hive/warehouse Problem : Spark trying to create a local directory under the home