Re: SparkSQL to read XML Blob data to create multiple rows

2017-07-08 Thread Amol Talap
Hi Zhang & All, Thanks so much for your earlier response. I am trying to get final solution. We could parse the data successfully however I am getting Nullpointerexception while mapping it back. Can you please suggest on below findings ? spark-shell --packages com.databricks:spark-xml_2.11:0.4.1

Glue-like Functionality

2017-07-08 Thread Benjamin Kim
Has anyone seen AWS Glue? I was wondering if there is something similar going to be built into Spark Structured Streaming? I like the Data Catalog idea to store and track any data source/destination. It profiles the data to derive the scheme and data types. Also, it does some sort-of automated

Event time aggregation is possible in Spark Streaming ?

2017-07-08 Thread Swapnil Chougule
Hello, I want to know whether event time aggregation in spark streaming. I could see it's possible in structured streaming. As I am working on conventional spark streaming, I need event time aggregation in it. I checked but didn't get any relevant documentation. Thanks in advance Regards,