welcoming Xiao Li as a committer

Hi all, Xiao Li, aka gatorsmile, has recently been elected as an Apache Spark committer. Xiao has been a super active contributor to Spark SQL. Congrats and welcome, Xiao! - Reynold

access spark thrift server from another spark session

I built spark data frame/dataset on top of several hive tables, and then registered dataframe/dataset as temporary tables, as well as exposed the temporary table through spark thrift server. Now the question is, this temporary table is only visible to the same spark session. How do I make the

Re: Nominal Attribute

There are plans...but not concrete ones yet: https://issues.apache.org/jira/browse/SPARK-8515 I agree categorical data handling is a pain point and that we need to improve it! On Tue, Sep 13, 2016 at 4:45 PM, Danil Kirsanov wrote: > NominalAttribute in MLib is used to

RE: regression: no longer able to use HDFS wasbs:// path for additional python files on LIVY batch submit

Great. Thanks for the pointer. I see the fix is in 2.0.1-rc4. Will there be a 1.6.3? If so, how are fixes considered for backporting? From: Steve Loughran [mailto:ste...@hortonworks.com] Sent: Monday, October 3, 2016 5:40 AM To: Kevin Grealish Cc: Apache Spark Dev

Re: Running Spark master/slave instances in non Daemon mode

Hi Mike, I can imagine the trouble that daemonization is causing and I think that having non-forking start script is a good idea. A simple, non-intrusive, fix could be to change the "spark-daemon.sh" script to conditionally omit the "nohup &". Personally, I think the semantically correct approach

Re: java.util.NoSuchElementException when serializing Map with default value

Hi Kabeer, which version of Spark are you using? I can't reproduce the error in latest Spark master. regards, --Jakob - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Catalyst - ObjectType for Encoders

Sounds good to me, I'll keep that in mind and pay attention to updates. Shall I make a pull request from my fork, or would someone else prefer to change those function signatures themselves? Thanks, Alek On Fri, Sep 30, 2016 at 2:25 PM Michael Armbrust wrote: > I'd be

Re: regression: no longer able to use HDFS wasbs:// path for additional python files on LIVY batch submit

On 1 Oct 2016, at 02:49, Kevin Grealish > wrote: I’m seeing a regression when submitting a batch PySpark program with additional files using LIVY. This is YARN cluster mode. The program files are placed into the mounted Azure Storage