Re: Possible Hive problem with Spark 2.0.0 preview.

[Arun Allamsetty]

Hi Doug,

If you look at the API docs here:
http://home.apache.org/~pwendell/spark-releases/spark-2.0.0-preview-docs/api/scala/index.html#org.apache.spark.sql.hive.HiveContext,
you'll see
Deprecate* (Since version 2.0.0)* Use
SparkSession.builder.enableHiveSupport instead
So you probably need to use that.

Arun

On Thu, May 19, 2016 at 3:44 PM, Michael Armbrust 
wrote:

> 1. “val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)”
>> doesn’t work because “HiveContext not a member of
>> org.apache.spark.sql.hive”  I checked the documentation, and it looks like
>> it should still work for spark-2.0.0-preview-bin-hadoop2.7.tgz
>>
>
> HiveContext has been deprecated and moved to a 1.x compatibility package,
> which you'll need to include explicitly.  Docs have not been updated yet.
>
>
>> 2. I also tried the new spark session, ‘spark.table(“db.table”)’, it
>> fails with a HDFS permission denied can’t write to “/user/hive/warehouse”
>>
>
> Where are the HDFS configurations located?  We might not be propagating
> them correctly any more.
>

Re: Requesting feedback for PR for SPARK-11962

[Arun Allamsetty]

Hi,

Since the 2.0.0 branch has been created and is now nearing feature freeze,
can SPARK-11962 get some love please. If we can decide if this should go
into 2.0.0 or 2.1.0, that would be great. Personally, I feel it can totally
go into 2.0.0 as the code is pretty much ready (except for the one bug that
I need your help with).

Thanks,
Arun

On Fri, Apr 29, 2016 at 1:00 PM, Arun Allamsetty 
wrote:

> Hi,
>
> I have submitted a PR for SPARK-11962 (
> https://github.com/apache/spark/pull/12708). I have most of it ready
> except I am not able to pin point the cause for a particular bug.
>
> In the PR I've added two major methods to Row, `attempt` and `getOption`.
> The former returns a `Try` while the latter returns an `Option`.
>
>- `attempt` was added after a comment was made in PR #10247
><https://github.com/apache/spark/pull/10247>, where it was suggested I
>not return `None` when certain exceptions are thrown. But in my
>opinion, throwing exceptions from a function which returns an `Option`
>is not a good use case.
>- I am not in love with the method name, `attempt`. Would welcome
>suggestions. I wanted to use `try` but it's a keyword.
>- So about the failing tests (which shouldn't fail in my opinion), in
>`RowTest`, all new tests testing for `ClassCastException` fail. I have
>modified tests as comments in the code blocks (with TODOs) which work, but
>should behave the same way as the actual test from what I can see. I am
>not sure what I am doing wrong in the code here.
>
> Would appreciate some feedback.
>
> Thanks,
>
> Arun
>
> P. S. I have sent similar emails to the dev mailing list before but I
> don't see them in the dev archives. I am guessing they were not received as
> I was not subscribed to the dev list. Hopefully this would work as I have
> subscribed now.
>

Requesting feedback for PR for SPARK-11962

[Arun Allamsetty]

Hi,

I have submitted a PR for SPARK-11962 (
https://github.com/apache/spark/pull/12708). I have most of it ready except
I am not able to pin point the cause for a particular bug.

In the PR I've added two major methods to Row, `attempt` and `getOption`.
The former returns a `Try` while the latter returns an `Option`.

   - `attempt` was added after a comment was made in PR #10247
   , where it was suggested I
   not return `None` when certain exceptions are thrown. But in my opinion,
   throwing exceptions from a function which returns an `Option` is not a
   good use case.
   - I am not in love with the method name, `attempt`. Would welcome
   suggestions. I wanted to use `try` but it's a keyword.
   - So about the failing tests (which shouldn't fail in my opinion), in
   `RowTest`, all new tests testing for `ClassCastException` fail. I have
   modified tests as comments in the code blocks (with TODOs) which work, but
   should behave the same way as the actual test from what I can see. I am
   not sure what I am doing wrong in the code here.

Would appreciate some feedback.

Thanks,

Arun

P. S. I have sent similar emails to the dev mailing list before but I don't
see them in the dev archives. I am guessing they were not received as I was
not subscribed to the dev list. Hopefully this would work as I have
subscribed now.