[no subject]

Unsubscribe

[no subject]

Unsubscribe

[no subject]

Unsubscribe

[no subject]

Unsubscribe

Spark-submit --files option help

Hi all, I am trying to pass some configuration files via spark-submit command in cluster mode. >From logs I can see the files are transferred to each executors. But how to build the absolute path of the file in the code? Can anyone plz guide on it with some references. Appreciate your help on

Applying schema dynamically in dataframe

Hi dev, Currently I have a scenario where I am reading the data from Kafka using spark dataframe. Multiple data sources ingest the data into kafka same topic. My dataframe now has got json string from multiple sources. I have got key column from Kafka which can be used to identify which source

Clarification on the commit protocol

Hi team, Just wanted to understand. Is DBIO commit protocol available in open source spark version ?

Re: Does dataframe spark API write/create a single file instead of directory as a result of write operation.

t uses the CSV file for further processing. > > On Sat, 22 Feb 2020 at 14:31, rahul c wrote: > >> Hi Kshitij, >> >> There are option to suppress the metadata files from get created. >> Set the below properties and try. >> >> 1) To disable the transaction logs

Re: Does dataframe spark API write/create a single file instead of directory as a result of write operation.

Hi Kshitij, There are option to suppress the metadata files from get created. Set the below properties and try. 1) To disable the transaction logs of spark "spark.sql.sources.commitProtocolClass = org.apache.spark.sql.execution.datasources.SQLHadoopMapReduceCommitProtocol". This will help to

Requesting for a support

Hi team, I am very new to the community I have experience in using spark framework. I would like to help the community any way I can. Regards Timmanna