Re: Disable/Remove datasources in Spark

to > read any datasource he can reach. > > > Aditya wrote: > > 2. But I am not able to figure out how to "disable" all other data > sources > > - > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >

Disable/Remove datasources in Spark

Hi, I am trying to force all users to use only 1 datasource (A custom datasource I plan to write) to read/write data. So, I was looking at the DataSource api in Spark: 1. I was able to figure out how to create my own Datasource (Reference

Re: convert java dataframe to pyspark dataframe

requirement is but you will need to > implement your custom version of PySpark API to get all functionality you > need and control on JVM side. > > > On 31/03/2021 06:49, Aditya Singh wrote: > > Thanks a lot Khalid for replying. > > I have one question though. The approach tou sh

Re: convert java dataframe to pyspark dataframe

will be performed. So also wanted to ask if this is feasible and if yes do we need to send some special jars to executors so that it can execute udfs on the dataframe. On Wed, 31 Mar 2021 at 3:37 AM, Khalid Mammadov wrote: > Hi Aditya, > > > I think you original question was as ho

Re: convert java dataframe to pyspark dataframe

in the last email. I want to access the same spark session across java and pyspark. So how can we share the spark context and in turn spark session, across java and pyspark. Regards, Aditya On Fri, 26 Mar 2021 at 6:49 PM, Sean Owen wrote: > The problem is that both of these are not shar

convert java dataframe to pyspark dataframe

Hi All, I am a newbie to spark and trying to pass a java dataframe to pyspark. Foloowing link has details about what I am trying to do:- https://stackoverflow.com/questions/66797382/creating-pysparks-spark-context-py4j-java-gateway-object Can someone please help me with this? Thanks,

unsubscribe

unsubscribe

Re: unsubscribe

On Tue, Jan 29, 2019, 10:14 AM Charles Nnamdi Akalugwu unsubscribe >

Unsubscribe

Merging Schema while reading Parquet files

quot;mergeSchema" - is not working here because my base path has multiple directories under which files are residing. Can someone suggest me some effective solution here? Regards, Aditya Borde

Unsubscribe

Unsubscribe - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Spark Executor Lost issue

is using quite large memory than it asked; and yarn kills the executor. Regards, Sushrut Ikhar https://about.me/sushrutikhar <https://about.me/sushrutikhar?promo=email_sig> On Wed, Sep 28, 2016 at 12:17 PM, Aditya <aditya.calangut...@augmentiq.co.in <mailto:adi

Re: Spark Executor Lost issue

: Thanks Sushrut for the reply. Currently I have not defined spark.default.parallelism property. Can you let me know how much should I set it to? Regards, Aditya Calangutkar On Wednesday 28 September 2016 12:22 PM, Sushrut Ikhar wrote: Try with increasing the parallelism by repartitioning

Spark Executor Lost issue

I have a spark job which runs fine for small data. But when data increases it gives executor lost error.My executor and driver memory are set at its highest point. I have also tried increasing--conf spark.yarn.executor.memoryOverhead=600but still not able to fix the problem. Is there any other

Re: Spark Yarn Cluster with Reference File

On Friday 23 September 2016 07:29 PM, ABHISHEK wrote: Thanks for your response Aditya and Steve. Steve: I have tried specifying both /tmp/filename in hdfs and local path but it didn't work. You may be write that Kie session is configured to access files from Local path. I have attached code h

Re: Spark Yarn Cluster with Reference File

Hi Abhishek, From your spark-submit it seems your passing the file as a parameter to the driver program. So now it depends what exactly you are doing with that parameter. Using --files option it will be available to all the worker nodes but if in your code if you are referencing using the

Re: How to specify file

Hi Sea, For using Spark SQL you will need to create DataFrame from the file and then execute select * on dataframe. In your case you will need to do something like this JavaRDD DF = context.textFile("path"); JavaRDD rowRDD3 = DF.map(new Function() {

Re: Spark RDD and Memory

into an issue where my DAG is very long and all the data does not fits into memory and at some point all my executors gets lost. On Friday 23 September 2016 12:15 PM, Aditya wrote: Hi Datta, Thanks for the reply. If I havent cached any rdd and the data that is being loaded into memory after

Re: Spark RDD and Memory

er 2016 at 18:09, Hanumath Rao Maduri <hanu@gmail.com <mailto:hanu@gmail.com>> wrote: Hello Aditya, After an intermediate action has been applied you might want to call rdd.unpersist() to let spark know that this rdd is no longer required. Thanks,

Spark RDD and Memory

Hi, Suppose I have two RDDs val textFile = sc.textFile("/user/emp.txt") val textFile1 = sc.textFile("/user/emp1.xt") Later I perform a join operation on above two RDDs val join = textFile.join(textFile1) And there are subsequent transformations without including textFile and textFile1 further

SparkStreaming is not working with SparkLauncher

I manually submitting jar file from command line. It looks like I am missing something. Any help is highly appreciable. Thanks, -- Aditya

Re: JavaRDD to DataFrame fails with null pointer exception in 1.6.0

Check if schema is generated correctly. On Wednesday 17 August 2016 10:15 AM, sudhir patil wrote: Tested with java 7 & 8 , same issue on both versions. On Aug 17, 2016 12:29 PM, "spats" > wrote: Cannot convert JavaRDD to

Re: [Spark 2.0] ClassNotFoundException is thrown when using Hive

Try using --files /path/of/hive-site.xml in spark-submit and run. On Thursday 18 August 2016 05:26 PM, Diwakar Dhanuskodi wrote: Hi Can you cross check by providing same library path in --jars of spark-submit and run . Sent from Samsung Mobile. Original message From:

SparkStreaming source code

Hi, I need to set up source code of Spark Streaming for exploring purpose. Can any one suggest the link for the Spark Streaming source code? Regards, Aditya Calangutkar - To unsubscribe e-mail: user-unsubscr

Spark Summit 2014 Day 2 Video Streams?

I attended yesterday on ustream.tv, but can't find the links to today's streams anywhere. help! -- Aditya Varun Chadha | http://www.adichad.com | +91 81308 02929 (M)