Re: [Spark SQL] Nanoseconds in Timestamps are set as Microseconds

Seems like a bug we should fix? I agree some form of truncation makes more sense. On Thu, Jun 1, 2017 at 1:17 AM, Anton Okolnychyi wrote: > Hi all, > > I would like to ask what the community thinks regarding the way how Spark > handles nanoseconds in the Timestamp

Re: [Spark SQL] Nanoseconds in Timestamps are set as Microseconds

Then let me provide a PR so that we can discuss an alternative way 2017-06-02 8:26 GMT+02:00 Reynold Xin : > Seems like a bug we should fix? I agree some form of truncation makes more > sense. > > > On Thu, Jun 1, 2017 at 1:17 AM, Anton Okolnychyi < >

Re: Spark Issues on ORC

On 26 May 2017, at 19:02, Dong Joon Hyun > wrote: Hi, All. Today, while I’m looking over JIRA issues for Spark 2.2.0 in Apache Spark. I noticed that there are many unresolved community requests and related efforts over `Feature parity for

Re: SQL TIMESTAMP semantics vs. SPARK-18350

Hi, We would like to solve the problem of interoperability of existing data, and that is the main use case for having table-level control. Spark should be able to read timestamps written by Impala or Hive and at the same time read back its own data. These have different semantics, so having a

Re: Spark Issues on ORC

Thank you for confirming, Steve. I removes the dependency of SPARK-20799 on SPARK-20901. Bests, Dongjoon. From: Steve Loughran Date: Friday, June 2, 2017 at 4:42 AM To: Dong Joon Hyun Cc: Apache Spark Dev Subject: Re: Spark

stuck on one of the jobs in spark streaming app

Hello I have a spark streaming app which consume kafka messages using kafka 0.9 directStream and EsSpark saveToES. almost every time after submit it, it ran fine then after several hours one of the jobs just stuck and keep running. Has anybody seen the same/similar issue? The stack trace is

[VOTE] Apache Spark 2.2.0 (RC3)

Please vote on releasing the following candidate as Apache Spark version 2.2.0. The vote is open until Tues, June 6th, 2017 at 12:00 PST and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 2.2.0 [ ] -1 Do not release this package because ...

Re: [VOTE] Apache Spark 2.2.0 (RC2)

This vote fails. Following shortly with RC3 On Thu, Jun 1, 2017 at 8:28 PM, Reynold Xin wrote: > Again (I've probably said this more than 10 times already in different > threads), SPARK-18350 has no impact on whether the timestamp type is with > timezone or without

Re: [VOTE] Apache Spark 2.2.0 (RC3)

I'm -1 on this. I merged a PR to master/2.2 today and break the build. I'm really sorry for the trouble and I should not be so aggressive when merging PRs. The actual reason is some misleading comments in the code and a bug in Spark's testing framework

Re: SQL TIMESTAMP semantics vs. SPARK-18350

Hi Zoltan, I don't fully understand your proposal for table-specific timestamp type semantics. I think it will be helpful to everyone in this conversation if you can identify the expected behavior for a few concrete scenarios. Suppose we have a Hive metastore table hivelogs with a column named

Re: [VOTE] Apache Spark 2.2.0 (RC3)

This should probably fail the vote. I'll follow up with an RC4. On Fri, Jun 2, 2017 at 4:11 PM, Wenchen Fan wrote: > I'm -1 on this. > > I merged a PR to master/2.2 > today and break the build. I'm really sorry for the trouble

spark messing up handling of native dependency code?

Hi, There is a weird problem with spark when handling native dependency code: I want to use a library (JAI) with spark to parse some spatial raster files. Unfortunately, there are some strange issues. JAI only works when running via the build tool i.e. `sbt run` when executed in spark. When

Re: spark messing up handling of native dependency code?

Maybe not related, but in general geotools are not thread safe,so using from workers is most likely a gamble. On 06/03/2017 01:26 AM, Georg Heiler wrote: > Hi, > > There is a weird problem with spark when handling native dependency code: > I want to use a library (JAI) with spark to parse some