Re: Silly question about building Spark 1.4.1
Thanks Dean… I was building based on the information found on the Spark 1.4.1 documentation. So I have to ask the following: Shouldn’t the examples be updated to reflect Hadoop 2.6 or are the vendors’ distro not up to 2.6 and that’s why its still showing 2.4? Also I’m trying to build with support for Scala 2.11 Are there any known issues between Scala 2.11 and Hive and hive-thrift server? Dean, the reason I asked about needed to specify the Hive and Hive-Thriftserver options is that at the end of the build I see the following: “ [INFO] Spark Project SQL .. SUCCESS [02:06 min] [INFO] Spark Project ML Library ... SUCCESS [02:23 min] [INFO] Spark Project Tools SUCCESS [ 13.305 s] [INFO] Spark Project Hive . SUCCESS [01:55 min] [INFO] Spark Project REPL . SUCCESS [ 40.488 s] [INFO] Spark Project YARN . SUCCESS [ 38.793 s] [INFO] Spark Project Assembly . SUCCESS [01:10 min] [INFO] Spark Project External Twitter . SUCCESS [ 14.907 s] [INFO] Spark Project External Flume Sink .. SUCCESS [ 21.748 s] [INFO] Spark Project External Flume ... SUCCESS [ 31.754 s] [INFO] Spark Project External MQTT SUCCESS [ 17.921 s] [INFO] Spark Project External ZeroMQ .. SUCCESS [ 18.037 s] [INFO] Spark Project External Kafka ... SUCCESS [ 41.941 s] [INFO] Spark Project Examples . SUCCESS [01:56 min] [INFO] Spark Project External Kafka Assembly .. SUCCESS [ 24.806 s] [INFO] Spark Project YARN Shuffle Service . SUCCESS [ 5.204 s] [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 22:40 min [INFO] Finished at: 2015-07-20T12:54:23-07:00 [INFO] Final Memory: 109M/2332M [INFO] “ Granted this may be something completely different which is why the next time I do a build, I’m going to capture the stderr/stdout to a file. Thx for the quick response. On Jul 20, 2015, at 1:11 PM, Ted Yu yuzhih...@gmail.com wrote: In master (as well as 1.4.1) I don't see hive profile in pom.xml I do find hive-provided profile, though. FYI On Mon, Jul 20, 2015 at 1:05 PM, Dean Wampler deanwamp...@gmail.com mailto:deanwamp...@gmail.com wrote: hadoop-2.6 is supported (look for profile XML in the pom.xml file). For Hive, add -Phive -Phive-thriftserver (See http://spark.apache.org/docs/latest/sql-programming-guide.html#hive-tables http://spark.apache.org/docs/latest/sql-programming-guide.html#hive-tables) for more details. dean Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition http://shop.oreilly.com/product/0636920033073.do (O'Reilly) Typesafe http://typesafe.com/ @deanwampler http://twitter.com/deanwampler http://polyglotprogramming.com http://polyglotprogramming.com/ On Mon, Jul 20, 2015 at 2:55 PM, Michael Segel msegel_had...@hotmail.com mailto:msegel_had...@hotmail.com wrote: Sorry, Should have sent this to user… However… it looks like the docs page may need some editing? Thx -Mike Begin forwarded message: From: Michael Segel msegel_had...@hotmail.com mailto:msegel_had...@hotmail.com Subject: Silly question about building Spark 1.4.1 Date: July 20, 2015 at 12:26:40 PM MST To: d...@spark.apache.org mailto:d...@spark.apache.org Hi, I’m looking at the online docs for building spark 1.4.1 … http://spark.apache.org/docs/latest/building-spark.html http://spark.apache.org/docs/latest/building-spark.html I was interested in building spark for Scala 2.11 (latest scala) and also for Hive and JDBC support. The docs say: “ To produce a Spark package compiled with Scala 2.11, use the -Dscala-2.11 property: dev/change-version-to-2.11.sh http://change-version-to-2.11.sh/ mvn -Pyarn -Phadoop-2.4 -Dscala-2.11 -DskipTests clean package “ So… Is there a reason I shouldn’t build against hadoop-2.6 ? If I want to add the Thirft and Hive support, is it possible? Looking at the Scala build, it looks like hive support is being built? (Looking at the stdout messages…) Should the docs be updated? Am I missing something? (Dean W. can confirm, I am completely brain dead. ;-) Thx -Mike PS. Yes I can probably download a prebuilt image, but I’m a glutton for punishment. ;-)
Re: Silly question about building Spark 1.4.1
hadoop-2.6 is supported (look for profile XML in the pom.xml file). For Hive, add -Phive -Phive-thriftserver (See http://spark.apache.org/docs/latest/sql-programming-guide.html#hive-tables) for more details. dean Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition http://shop.oreilly.com/product/0636920033073.do (O'Reilly) Typesafe http://typesafe.com @deanwampler http://twitter.com/deanwampler http://polyglotprogramming.com On Mon, Jul 20, 2015 at 2:55 PM, Michael Segel msegel_had...@hotmail.com wrote: Sorry, Should have sent this to user… However… it looks like the docs page may need some editing? Thx -Mike Begin forwarded message: *From: *Michael Segel msegel_had...@hotmail.com *Subject: **Silly question about building Spark 1.4.1* *Date: *July 20, 2015 at 12:26:40 PM MST *To: *d...@spark.apache.org Hi, I’m looking at the online docs for building spark 1.4.1 … http://spark.apache.org/docs/latest/building-spark.html I was interested in building spark for Scala 2.11 (latest scala) and also for Hive and JDBC support. The docs say: “ To produce a Spark package compiled with Scala 2.11, use the -Dscala-2.11 property: dev/change-version-to-2.11.sh mvn -Pyarn -Phadoop-2.4 -Dscala-2.11 -DskipTests clean package “ So… Is there a reason I shouldn’t build against hadoop-2.6 ? If I want to add the Thirft and Hive support, is it possible? Looking at the Scala build, it looks like hive support is being built? (Looking at the stdout messages…) Should the docs be updated? Am I missing something? (Dean W. can confirm, I am completely brain dead. ;-) Thx -Mike PS. Yes I can probably download a prebuilt image, but I’m a glutton for punishment. ;-)
Re: Silly question about building Spark 1.4.1
In master (as well as 1.4.1) I don't see hive profile in pom.xml I do find hive-provided profile, though. FYI On Mon, Jul 20, 2015 at 1:05 PM, Dean Wampler deanwamp...@gmail.com wrote: hadoop-2.6 is supported (look for profile XML in the pom.xml file). For Hive, add -Phive -Phive-thriftserver (See http://spark.apache.org/docs/latest/sql-programming-guide.html#hive-tables) for more details. dean Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition http://shop.oreilly.com/product/0636920033073.do (O'Reilly) Typesafe http://typesafe.com @deanwampler http://twitter.com/deanwampler http://polyglotprogramming.com On Mon, Jul 20, 2015 at 2:55 PM, Michael Segel msegel_had...@hotmail.com wrote: Sorry, Should have sent this to user… However… it looks like the docs page may need some editing? Thx -Mike Begin forwarded message: *From: *Michael Segel msegel_had...@hotmail.com *Subject: **Silly question about building Spark 1.4.1* *Date: *July 20, 2015 at 12:26:40 PM MST *To: *d...@spark.apache.org Hi, I’m looking at the online docs for building spark 1.4.1 … http://spark.apache.org/docs/latest/building-spark.html I was interested in building spark for Scala 2.11 (latest scala) and also for Hive and JDBC support. The docs say: “ To produce a Spark package compiled with Scala 2.11, use the -Dscala-2.11 property: dev/change-version-to-2.11.sh mvn -Pyarn -Phadoop-2.4 -Dscala-2.11 -DskipTests clean package “ So… Is there a reason I shouldn’t build against hadoop-2.6 ? If I want to add the Thirft and Hive support, is it possible? Looking at the Scala build, it looks like hive support is being built? (Looking at the stdout messages…) Should the docs be updated? Am I missing something? (Dean W. can confirm, I am completely brain dead. ;-) Thx -Mike PS. Yes I can probably download a prebuilt image, but I’m a glutton for punishment. ;-)
Fwd: Silly question about building Spark 1.4.1
Sorry, Should have sent this to user… However… it looks like the docs page may need some editing? Thx -Mike Begin forwarded message: From: Michael Segel msegel_had...@hotmail.com Subject: Silly question about building Spark 1.4.1 Date: July 20, 2015 at 12:26:40 PM MST To: d...@spark.apache.org Hi, I’m looking at the online docs for building spark 1.4.1 … http://spark.apache.org/docs/latest/building-spark.html http://spark.apache.org/docs/latest/building-spark.html I was interested in building spark for Scala 2.11 (latest scala) and also for Hive and JDBC support. The docs say: “ To produce a Spark package compiled with Scala 2.11, use the -Dscala-2.11 property: dev/change-version-to-2.11.sh mvn -Pyarn -Phadoop-2.4 -Dscala-2.11 -DskipTests clean package “ So… Is there a reason I shouldn’t build against hadoop-2.6 ? If I want to add the Thirft and Hive support, is it possible? Looking at the Scala build, it looks like hive support is being built? (Looking at the stdout messages…) Should the docs be updated? Am I missing something? (Dean W. can confirm, I am completely brain dead. ;-) Thx -Mike PS. Yes I can probably download a prebuilt image, but I’m a glutton for punishment. ;-)