Re: Visualizing Spark Streaming data

Hey Jeffrey. Thanks for reply. I already have something similar, I use Grafana and Graphite, and for simple metric streaming we've got all set-up right. My question is about interactive patterns. For instance, dynamically choose an event to monitor, dynamically choose group-by field or any sort

Re: Visualizing Spark Streaming data

Management Award NetworkWorld 10 Startups to Watch EMA Most Notable Vendor On Fri, Mar 20, 2015 at 1:06 AM, Harut Martirosyan harut.martiros...@gmail.com wrote: Hey Jeffrey. Thanks for reply. I already have something similar, I use Grafana and Graphite, and for simple metric streaming

Standalone Scheduler VS YARN Performance

What is performance overhead caused by YARN, or what configurations are being changed when the app is ran through YARN? The following example: sqlContext.sql(SELECT dayStamp(date), count(distinct deviceId) AS c FROM full GROUP BY dayStamp(date) ORDER BY c DESC LIMIT 10) .collect() runs on shell

Re: RDD Persistance synchronization

, like, if a persisted block is lost or otherwise unavailable later. On Sun, Mar 29, 2015 at 9:07 AM, Harut Martirosyan harut.martiros...@gmail.com wrote: Hi. rdd.persist() rdd.count() rdd.transform()... is there a chance transform() runs before persist() is complete? -- RGRDZ

Re: Parallel actions from driver

This is exactly my case also, it worked, thanks Sean. On 26 March 2015 at 23:35, Sean Owen so...@cloudera.com wrote: You can do this much more simply, I think, with Scala's parallel collections (try .par). There's nothing wrong with doing this, no. Here, something is getting caught in your

RDD Persistance synchronization

Hi. rdd.persist() rdd.count() rdd.transform()... is there a chance transform() runs before persist() is complete? -- RGRDZ Harut

Spark SQL: Day of month from Timestamp

Hi guys. Basically, we had to define a UDF that does that, is there a built in function that we can use for it? -- RGRDZ Harut

Simple but faster data streaming

Hi guys. Is there a more lightweight way of stream processing with Spark? What we want is a simpler way, preferably with no scheduling, which just streams the data to destinations multiple. We extensively use Spark Core, SQL, Streaming, GraphX, so it's our main tool and don't want to add new

Custom metrics in py-spark 3

Hello. We’re successfully exporting technical metrics to prometheus using built-in capabilities of Spark 3, but we need to add custom business metrics as well using python. Seems like there’s no documentation for that. Thanks.

Create table before inserting in SQL

I have a resultset (defined in SQL), and I want to insert it into my JDBC database using only SQL, not dataframes API. If the table existed, I would create a table using JDBC in spark SQL and then insert into it, but I can't create a table if it doesn't exist in JDBC database. How to do that

Re: Create table before inserting in SQL

Generally, the problem is that I don’t find a way to automatically create a JDBC table in the JDBC database when I want to insert data into it using Spark SQL only, not DataFrames API. > On 2 Feb 2023, at 21:22, Harut Martirosyan > wrote: > > Hi, thanks for the reply. > >

Re: Create table before inserting in SQL

ble for any monetary damages arising from such > loss, damage or destruction. > > > > On Wed, 1 Feb 2023 at 19:33, Harut Martirosyan <mailto:harut.martiros...@gmail.com>> wrote: >> I have a resultset (defined in SQL), and I want to insert it into my

Re: Create table before inserting in SQL

uthor will in no case be liable for any monetary damages arising from such > loss, damage or destruction. > > > > On Thu, 2 Feb 2023 at 17:26, Harut Martirosyan <mailto:harut.martiros...@gmail.com>> wrote: >> Generally, the problem is that I don’t find a way to automatically