Re: Equivalent of emptyDataFrame in StructuredStreaming

ch is error friendly). Even better, is there a cleaner way to create an empty stream. val emptyErrorStream = (spark:SparkSession) => { implicit val sqlC = spark.sqlContext MemoryStream[DataError].toDS() } Cheers, Arun On Mon, Nov 5, 2018 at 2:41 PM Arun Manivannan wrote: > Hi, > &

Equivalent of emptyDataFrame in StructuredStreaming

Hi, This is going to come off as a silly question with a stream being unbounded but this is problem that I have (created for myself). I am trying to build an ETL pipeline and I have a bunch of stages. val pipelineStages = List( new AddRowKeyStage(EvergreenSchema), new WriteToHBaseStage(hBase

JIRA access

Hi, I would like to attempt SPARK-20592 . Can I please have access to the JIRA so that I could assign it to myself. My user id is : arunodhaya80 Cheers, Arun

Alter table concatenate - unsupported

Hi, Good morning. I am using Spark batch to process and ingest extracts of several RDBMS tables/Filebased systems arriving in regular intervals into a Datalake as ORC backed Hive tables. Considering that the input data file size, file count, row count and feature counts vary quite a lot, I am una