non-deprecation compiler warnings are upgraded to build errors now

Hi all, FYI, we just merged a patch that fails a build if there is a scala compiler warning (if it is not deprecation warning). In the past, many compiler warnings are actually caused by legitimate bugs that we need to address. However, if we don't fail the build with warnings, people don't pay a

RE: Package Release Annoucement: Spark SQL on HBase "Astro"

Yes, but not all SQL-standard insert variants . From: Debasish Das [mailto:debasish.da...@gmail.com] Sent: Wednesday, July 22, 2015 7:36 PM To: Bing Xiao (Bing) Cc: user; dev; Yan Zhou.sc Subject: Re: Package Release Annoucement: Spark SQL on HBase "Astro" Does it also support insert operations

Re: Package Release Annoucement: Spark SQL on HBase "Astro"

Does it also support insert operations ? On Jul 22, 2015 4:53 PM, "Bing Xiao (Bing)" wrote: > We are happy to announce the availability of the Spark SQL on HBase > 1.0.0 release. > http://spark-packages.org/package/Huawei-Spark/Spark-SQL-on-HBase > > The main features in this package, dubbed “As

Where to find Spark-project-hive

Hi guys, I’m trying to patch hive thrift server part related to HIVE-7620. I saw in spark is pulling a private fork of hive under spark-project hive name. Any idea where I can find the source code of it? Thanks~ 马晓宇 / Xiaoyu Ma hzmaxia...@corp.netease.com

PySpark addPyFile for directories

I am running into an inconvenience while developing which I think could be fixed by extending addPyFile. I am working on a pyspark project which has a primary entry point plus several modules. The effect of this is that unless the code is copied to the cluster in the PYTHONPATH or zipped and shippe

Package Release Annoucement: Spark SQL on HBase "Astro"

We are happy to announce the availability of the Spark SQL on HBase 1.0.0 release. http://spark-packages.org/package/Huawei-Spark/Spark-SQL-on-HBase The main features in this package, dubbed "Astro", include: * Systematic and powerful handling of data pruning and intelligent scan, based

Re: PySpark on PyPi

I agree with everything Justin just said. An additional advantage of publishing PySpark's Python code in a standards-compliant way is the fact that we'll be able to declare transitive dependencies (Pandas, Py4J) in a way that pip can use. Contrast this with the current situation, where df.toPandas(

Re: Expression.resolved unmatched with the correct values in catalyst?

Ok, thanks, I understood why this happened. best regards, // maropu On Wed, Jul 22, 2015 at 10:26 PM, Takeshi Yamamuro wrote: > The move prevents some errors though, all the errors cannot be gone. > For example, in o.a.s.sql.catalyst.analysis.*Suite, > > The case ' > https://github.com/maropu/

Re: PySpark on PyPi

// + *Davies* for his comments // + Punya for SA For development and CI, like Olivier mentioned, I think it would be hugely beneficial to publish pyspark (only code in the python/ dir) on PyPI. If anyone wants to develop against PySpark APIs, they need to download the distribution and do a lot of

Re: Expression.resolved unmatched with the correct values in catalyst?

The move prevents some errors though, all the errors cannot be gone. For example, in o.a.s.sql.catalyst.analysis.*Suite, The case ' https://github.com/maropu/spark/commit/961b5e99e2136167f175598ed36585987cc1e236 ' causes 3 errors. AnalysisSuite: - analyze project *** FAILED *** AnalysisErrorSuite:

Fixed number of partitions in RangePartitioner

Hi all: I am developing an algorithm that needs to put together elements with the same key as much as possible but with always using a fixed number of partitions. To do that, this algorithm sorts by key the elements. The problem is that the number of distinct keys influences in the number of

Deleted unreleased version 1.6.0 from JIRA by mistake

Hi all, The unreleased version 1.6.0 has was removed from JIRA due to my misoperation. I've added it back, but JIRA tickets that once targeted to 1.6.0 now have empty target version/s. If you found tickets that should have targeted to 1.6.0, please help marking the target version/s field back

Re: What is the difference between SlowSparkPullRequestBuilder and SparkPullRequestBuilder?

Hi Andrew, I understand that there is no difference currently. Thanks, Yu - -- Yu Ishikawa -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/What-is-the-difference-between-SlowSparkPullRequestBuilder-and-SparkPullRequestBuilder-tp13377p13380.html S

Re: What is the difference between SlowSparkPullRequestBuilder and SparkPullRequestBuilder?

Hi Yu, As it stands today, they are identical except for trigger mechanism. When you say "test this please" or push a commit, SparkPullRequestBuilder is the one that's running the tests. SlowSparkPullRequestBuilder, however, is not used by default, but only triggered when you say "slow test please