Re: Spark Streaming Code

To get any meaningful answers you may want to provide the information/context as much as possible. e.g. Spark version, which behavior/output was expected (and why you think) and how it behaves actually. On Sun, Mar 29, 2020 at 3:37 AM Siva Samraj wrote: > Hi Team, > > Need help on windowing &

Re: Spark structural streaming sinks output late

Yes, I am also facing the same issue. Did you figured out? On Tue, 9 Jul 2019, 7:25 pm Kamalanathan Venkatesan, < kamalanatha...@in.ey.com> wrote: > Hello, > > > > I have below spark structural streaming code and I was expecting the > results to be printed on the console every 10 seconds. But, I

Spark Streaming Code

Hi Team, Need help on windowing & watermark concept. This code is not working as expected. package com.jiomoney.streaming import org.apache.spark.sql.SparkSession import org.apache.spark.sql.functions._ import org.apache.spark.sql.streaming.ProcessingTime object SlingStreaming { def

Beautiful Spark Code

You will be please to learn that Mr. Mathew Powers have seen to my needs and answers all my questions. He has seen to all my needs. Mr Powers has shut me up !!! Mr Powers has made Google search stackoverflow and user@spark.apache.org redundant. That is all you guys and girl had to do , point

Re: BUG: spark.readStream .schema(staticSchema) not receiving schema information

Thanks for the tip! But if the first thing you come across Is somebody using the trim function to strip away spaces in /etc/hostnames like so from : 127.0.0.1 hostname local To 127.0.0.1hostnamelocal Then there is a log error message showing the outcome of unnecessarily using the trim

Re: Best Practice: Evaluate Expression from Spark DataFrame Column

Is there a way to pass column as a String to expr function in spark? val sampleDF = Seq( (8, 1, "bat", "NUM IS NOT NULL AND FLAG!=0"), (64, 0, "mouse", "NUM IS NOT NULL AND FLAG!=0"), (-27, 1, "horse" , "NUM IS NOT NULL AND FLAG!=0"), (1, 0, "miki", "NUM IS NOT NULL AND FLAG!=0 AND WORD

Re: spark structured streaming GroupState returns weird values from sate

Ok, I will try to create some simple code to reproduce, if I can. Problem is that I am adding this code in an existing big project with several dependencies with spark streaming older version(2.2) on root level etc. Also, I observed that there is @Experimental on GroupState class. What state is

Re: spark structured streaming GroupState returns weird values from sate

I have't heard known issue for this - that said, this may require new investigation which is not possible or require huge effort without simple reproducer. Contributors (who are basically volunteers) may not want to struggle to reproduce from your partial information - I'd recommend you to spend

Re: spark structured streaming GroupState returns weird values from sate

Sorry for typos , correcting them below On Sat, Mar 28, 2020 at 4:39 PM Srinivas V wrote: > Sorry I was just changing some names not to send exact names. Please > ignore that. I am really struggling with this since couple of days. Can > this happen due to > 1. some of the values being null or >

Re: spark structured streaming GroupState returns weird values from sate

Sorry I was just changing some names not to send exact names. Please ignore that. I am really struggling with this sine couple of days. Can this happen due to 1. some of the values being null or 2.UTF8 issue ? Or some sterilization/ deserilization issue ? 3. Not enough memory ? I am using same

Re: OFF TOPIC LIST CRITERIA

Would it be surprising to know that I know the difference between a user issue and a bug and feature. I have commercial experience and training in safety critical software (aerospace). Believe me that is an entirely different world. With entirely different standards. Obviously if I want an

Re: OFF TOPIC LIST CRITERIA

Hi THough Sean requested a non-reply, but this is sincerely crazy. @Zahid - what Sean said is actually mentioned in Spark's website - u...@spark.incubator.apache.org -- for usage questions, help, and announcements.