Question regarding kryo and java encoders in datasets

Hi All, Good day! I am using spark 2.4 and referring https://spark.apache.org/docs/latest/rdd-programming-guide.html#rdd-persistence Bean class: public class EmployeeBean implements Serializable { private Long id; private String name; private Long salary; private Integer

Re: Add column value in the dataset on the basis of a condition

Thanks, Yunus. It solved my problem. Regards, Devender From: Shahab Yunus Sent: Tuesday, December 18, 2018 8:27:51 PM To: Devender Yadav Cc: user@spark.apache.org Subject: Re: Add column value in the dataset on the basis of a condition Sorry Devender, I hit

Add column value in the dataset on the basis of a condition

Hi All, useful code: public class EmployeeBean implements Serializable { private Long id; private String name; private Long salary; private Integer age; // getters and setters } Relevant spark code: SparkSession spark =

Create dataframe from RDBMS table using JDBC

Hi All, I am using Spak 1.6.2 Which is suitable way to create dataframe from RDBMS table. DataFrame df = sqlContext.read().format("jdbc").options(options).load(); or DataFrame df = sqlContext.read().jdbc(url, table, properties); Regards, Devender

Re: Arraylist is empty after JavaRDD.foreach

Hi Franke, I want to convert DataFrame to JSON String. Regards, Devender From: Jörn Franke <jornfra...@gmail.com> Sent: Monday, April 24, 2017 11:15:08 PM To: Devender Yadav Cc: user@spark.apache.org Subject: Re: Arraylist is empty after JavaRDD.foreach

How to convert DataFrame to JSON String in Java 7

Hi All, How can I convert DataFrame to JSON String in Java 7. I am using Spark 1.6.3 I don't want to print on console. I need to return JSON return to another method. Thanks for your attention! Regards, Devender NOTE: This message may contain

Arraylist is empty after JavaRDD.foreach

Hi All, I am using Spark 1.6.2 and Java 7. Sample json (total 100 records): {"name":"dev","salary":1,"occupation":"engg","address":"noida"} {"name":"karthik","salary":2,"occupation":"engg","address":"noida"} Useful code: final List> jsonData = new

Re: How to maintain order of key-value in DataFrame same as JSON?

Thanks Hemanth for a quick reply. From: Hemanth Gudela <hemanth.gud...@qvantel.com> Sent: Monday, April 24, 2017 6:37:48 PM To: Devender Yadav; user@spark.apache.org Subject: Re: How to maintain order of key-value in DataFrame same as JSON? Hi, One option

How to maintain order of key-value in DataFrame same as JSON?

Hi All, Sample JSON data: {"name": "dev","salary": 100,"occupation": "engg","address": "noida"} {"name": "karthik","salary": 200,"occupation": "engg","address": "blore"} Spark Java code: DataFrame df = sqlContext.read().json(jsonPath); df.printSchema(); df.show(false); Output: root |--

Partitioning in spark while reading from RDBMS via JDBC

Hi All, I am running spark in cluster mode and reading data from RDBMS via JDBC. As per spark docs, these partitioning parameters describe how to partition the table when reading in parallel from multiple

How to insert nano seconds in the TimestampType in Spark

Hi All, I am using spark version - 1.6.1 I have a text table in hive having `timestamp` datatype with nanoseconds precision. Hive Table Schema: c_timestamp timestamp Hive Table data: hive> select * from tbl1; OK 00:00:00.1 12:12:12.123456789