Spark SQL - Applying transformation on a struct inside an array

Hi everyone,I'm currently trying to create a generic transformation mecanism on a Dataframe to modify an arbitrary column regardless of the underlying the schema. It's "relatively" straightforward for complex types like struct> to apply an arbitrary UDF on the column and replace the data "inside" t

Nominal Attribute

NominalAttribute in MLib is used to represent categorical data internally. It is barely documented though and has a number of limitations: for example, it supports only integer and string data. Is there any current effort to expose it (and categorical data handling in general) to the users, or is

REST api for monitoring Spark Streaming

Hi everyone, Trying to monitoring our streaming application using Spark REST interface only to found that there is no such thing for Streaming. I wonder if anyone already working on this or I should just start implementing my own one? -- BR Peter Chan ---

Re: Test fails when compiling spark with tests

There are some flaky tests that occasionally fail, my first recommendation would be to re-run the test suite. Another thing to check is if there are any applications listening to spark's default ports. Btw, what is your environment like? In case it is windows, I don't think tests are regularly run