Noticed that Nan’s PR is not related to SQL, but the JDBC test suites got
executed. Then I checked PRs of all those Jenkins builds that failed because of
the JDBC suites, it turns out that none of them touched SQL code. The JDBC
code is only contained in the assembly file when the
Hi,
How do you package an app with modified spark?
In seems sbt would resolve the dependencies, and use the official spark
release.
Thank you!
Larry
I found 0.13.1 artifacts in maven:
http://search.maven.org/#artifactdetails%7Corg.apache.hive%7Chive-metastore%7C0.13.1%7Cjar
However, Spark uses groupId of org.spark-project.hive, not org.apache.hive
Can someone tell me how it is supposed to work ?
Cheers
On Mon, Jul 28, 2014 at 7:44 AM,
This sounds more like a user list https://spark.apache.org/community.html
question. This is the dev list, where people discuss things related to
contributing code and such to Spark.
On Mon, Jul 28, 2014 at 10:15 AM, jitendra shelar
jitendra.shelar...@gmail.com wrote:
Hi,
I am new to spark.
Yes, it is published. As of previous versions, at least, hive-exec
included all of its dependencies *in its artifact*, making it unusable
as-is because it contained copies of dependencies that clash with
versions present in other artifacts, and can't be managed with Maven
mechanisms.
I am not
Talked with Owen offline. He confirmed that as of 0.13, hive-exec is still
uber jar.
Right now I am facing the following error building against Hive 0.13.1 :
[ERROR] Failed to execute goal on project spark-hive_2.10: Could not
resolve dependencies for project
It would be great if the hive team can fix that issue. If not, we'll
have to continue forking our own version of Hive to change the way it
publishes artifacts.
- Patrick
On Mon, Jul 28, 2014 at 9:34 AM, Ted Yu yuzhih...@gmail.com wrote:
Talked with Owen offline. He confirmed that as of 0.13,
Owen helped me find this:
https://issues.apache.org/jira/browse/HIVE-7423
I guess this means that for Hive 0.14, Spark should be able to directly
pull in hive-exec-core.jar
Cheers
On Mon, Jul 28, 2014 at 9:55 AM, Patrick Wendell pwend...@gmail.com wrote:
It would be great if the hive team
Where and how is that fork being maintained? I'm not seeing an obviously
correct branch or tag in the main asf hive repo github mirror.
On Mon, Jul 28, 2014 at 9:55 AM, Patrick Wendell pwend...@gmail.com wrote:
It would be great if the hive team can fix that issue. If not, we'll
have to
Yeah so we need a model for this (Mark - do you have any ideas?). I
did this in a personal github repo. I just did it quickly because
dependency issues were blocking the 1.0 release:
https://github.com/pwendell/hive/tree/branch-0.12-shaded-protobuf
I think what we want is to have a semi official
AFAIK, according a recent talk, Hulu team in China has built Spark SQL
against Hive 0.13 (or 0.13.1?) successfully. Basically they also
re-packaged Hive 0.13 as what the Spark team did. The slides of the talk
hasn't been released yet though.
On Tue, Jul 29, 2014 at 1:01 AM, Ted Yu
I've heard from Cloudera that there were hive internal changes between
0.12 and 0.13 that required code re-writing. Over time it might be
possible for us to integrate with hive using API's that are more
stable (this is the domain of Michael/Cheng/Yin more than me!). It
would be interesting to see
Getting and maintaining our own branch in the main asf hive repo is a
non-starter or isn't workable?
On Mon, Jul 28, 2014 at 10:17 AM, Patrick Wendell pwend...@gmail.com
wrote:
Yeah so we need a model for this (Mark - do you have any ideas?). I
did this in a personal github repo. I just did
Exactly, forgot to mention Hulu team also made changes to cope with those
incompatibility issues, but they said that’s relatively easy once the
re-packaging work is done.
On Tue, Jul 29, 2014 at 1:20 AM, Patrick Wendell pwend...@gmail.com wrote:
I've heard from Cloudera that there were hive
On Mon, Jul 28, 2014 at 4:29 AM, Larry Xiao xia...@sjtu.edu.cn wrote:
On 7/28/14, 3:41 PM, shijiaxin wrote:
There is a VertexPartition in the EdgePartition,which is created by
EdgePartitionBuilder.toEdgePartition.
and There is also a ShippableVertexPartition in the VertexRDD.
These two
Gents,
It seem that until recently, building via sbt was a documented process in
the 0.9 overview:
http://spark.apache.org/docs/0.9.0/
The section on building mentions using sbt/sbt assembly. However in the
latest overview:
http://spark.apache.org/docs/latest/index.html
There¹s no mention of
Hi Steve,
I had the opportunity to ask this question at the Summit to Andrew Orr.
He mentioned that with 1.0 the recommended build tool is with maven. sbt
is however still supported. You will notice that the dependencies are now
completely handled within the maven pom.xml: the SparkBuild.scala
Hi Yu,
I could help translating Spark documentation to Japanese. Please let me
know if you need.
Best,
Ken
On Mon, Jul 28, 2014 at 1:03 AM, Yu Ishikawa [via Apache Spark Developers
List] ml-node+s1001551n7546...@n3.nabble.com wrote:
Hello Patrick,
Thank you for your replying.
I checked
Yeah for packagers we officially recommend using maven. Spark's
dependency graph is very complicated and Maven and SBT use different
conflict resolution strategies, so we've opted to official support
Maven.
SBT is still around though and it's used more often by day-to-day developers.
- Patrick
You can use publish-local in sbt.
If you want to be more careful, you can give Spark a different version
number and use that version number in your app.
On Mon, Jul 28, 2014 at 4:33 AM, Larry Xiao xia...@sjtu.edu.cn wrote:
Hi,
How do you package an app with modified spark?
In seems sbt
After manually copying hive 0.13.1 jars to local maven repo, I got the
following errors when building spark-hive_2.10 module :
[ERROR]
/homes/xx/spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala:182:
type mismatch;
found : String
required: Array[String]
[ERROR]
I was looking for a class where reflection-related code should reside.
I found this but don't think it is the proper class for bridging
differences between hive 0.12 and 0.13.1:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
Cheers
On Mon, Jul 28, 2014 at 3:41
A few things:
- When we upgrade to Hive 0.13.0, Patrick will likely republish the
hive-exec jar just as we did for 0.12.0
- Since we have to tie into some pretty low level APIs it is unsurprising
that the code doesn't just compile out of the box against 0.13.0
- ScalaReflection is for
The larger goal is to get a clean compile test in the environment I have
to use. As near as I can tell, tests fail in parquet because parquet was
only added in Hive 0.13. There could well be issues in later meta-stores,
but one thing at a time...
- SteveN
On 7/28/14, 17:22, Michael
https://issues.apache.org/jira/browse/INFRA-8116
Just a heads up, the github mirroring is running behind. You can
follow that JIRA to keep up to date on the fix.
In the mean time you can use the Apache git itself:
https://git-wip-us.apache.org/repos/asf/spark.git
Some people have reported
bq. Either way its unclear to if there is any reason to use reflection to
support multiple versions, instead of just upgrading to Hive 0.13.0
Which Spark release would this Hive upgrade take place ?
I agree it is cleaner to upgrade Hive dependency vs. introducing reflection.
Cheers
On Mon, Jul
+1 Tested on standalone and yarn clusters
2014-07-28 14:59 GMT-07:00 Tathagata Das tathagata.das1...@gmail.com:
Let me add my vote as well.
Did some basic tests by running simple projects with various Spark
modules. Tested checksums.
+1
On Sun, Jul 27, 2014 at 4:52 PM, Matei Zaharia
hi, haoyuan, thanks for replying.
2014-07-21 16:29 GMT+08:00 Haoyuan Li haoyuan...@gmail.com:
Qingyang,
Aha. Got it.
800MB data is pretty small. Loading from Tachyon does have a bit of extra
overhead. But it will have more benefit when the data size is larger. Also,
if you store the
+1 (non-binding)
Tested this on Mac OS X.
On Mon, Jul 28, 2014 at 6:52 PM, Andrew Or and...@databricks.com wrote:
+1 Tested on standalone and yarn clusters
2014-07-28 14:59 GMT-07:00 Tathagata Das tathagata.das1...@gmail.com:
Let me add my vote as well.
Did some basic tests by running
+1
Tested basic spark-shell and pyspark operations and MLlib examples on a Mac.
On Mon, Jul 28, 2014 at 8:29 PM, Mubarak Seyed spark.devu...@gmail.com wrote:
+1 (non-binding)
Tested this on Mac OS X.
On Mon, Jul 28, 2014 at 6:52 PM, Andrew Or and...@databricks.com wrote:
+1 Tested on
NOTICE and LICENSE files look good
Hashes and sigs look good
No executable in the source distribution
Compile source and run standalone
+1
- Henry
On Fri, Jul 25, 2014 at 4:08 PM, Tathagata Das
tathagata.das1...@gmail.com wrote:
Please vote on releasing the following candidate as Apache Spark
Hi devs,
I don't know if this is going to help, but if you can watch vote on the
ticket, it might help ASF INFRA prioritize and triage it faster:
https://issues.apache.org/jira/browse/INFRA-8116
Please do. Thanks!
On Mon, Jul 28, 2014 at 5:41 PM, Patrick Wendell pwend...@gmail.com wrote:
32 matches
Mail list logo
