Re: Handling stale PRs
Shout-out to Michael and other Spark SQL contributors for really trimming down the number of open/stale Spark SQL PRs https://spark-prs.appspot.com/#sql. As of right now, the least recently updated open Spark SQL PR goes back only 11 days. Nice work! Nick On Mon Dec 08 2014 at 2:58:08 PM Nicholas Chammas nicholas.cham...@gmail.com wrote: I recently came across this blog post, which reminded me of this thread. How to Discourage Open Source Contributions http://danluu.com/discourage-oss/ We are currently at 320+ open PRs, many of which haven't been updated in over a month. We have quite a few PRs that haven't been touched in 3-5 months. *If you have the time and interest, please hop on over to the Spark PR Dashboard https://spark-prs.appspot.com/, sort the PRs by least-recently-updated, and update them where you can.* I share the blog author's opinion that letting PRs go stale discourages contributions, especially from first-time contributors, and especially more so when the PR author is waiting on feedback from a committer or contributor. I've been thinking about simple ways to make it easier for all of us to chip in on controlling stale PRs in an incremental way. For starters, would it help if an automated email went out to the dev list once a week that a) reported the number of stale PRs, and b) directly linked to the 5 least recently updated PRs? Nick On Sat Aug 30 2014 at 3:41:39 AM Nicholas Chammas nicholas.cham...@gmail.com wrote: On Tue, Aug 26, 2014 at 2:02 AM, Patrick Wendell pwend...@gmail.com wrote: it's actually precedurally difficult for us to close pull requests Just an FYI: Seems like the GitHub-sanctioned work-around to having issues-only permissions is to have a second, issues-only repository https://help.github.com/articles/issues-only-access-permissions. Not a very attractive work-around... Nick
Data source interface for making multiple tables available for query
Hi, Data source ddl.scala, CREATE TEMPORARY TABLE makes one table at time available to temp tables, how about the case if multiple/all tables from some data source needs to be available for query, just like hive tables. I think we also need that interface to connect such data sources. Please comment. Regards, Ramana
Use mvn to build Spark 1.2.0 failed
Hi all, Today download Spark source from http://spark.apache.org/downloads.html page, and I use ./make-distribution.sh --tgz -Phadoop-2.2 -Pyarn -DskipTests -Dhadoop.version=2.2.0 -Phive to build the release, but I encountered an exception as follow: [INFO] --- build-helper-maven-plugin:1.8:add-source (add-scala-sources) @ spark-parent --- [INFO] Source directory: /home/q/spark/spark-1.2.0/src/main/scala added. [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ spark-parent --- [INFO] [INFO] Reactor Summary: [INFO] [INFO] Spark Project Parent POM .. FAILURE [1.015s] [INFO] Spark Project Networking .. SKIPPED [INFO] Spark Project Shuffle Streaming Service ... SKIPPED [INFO] Spark Project Core SKIPPED [INFO] Spark Project Bagel ... SKIPPED [INFO] Spark Project GraphX .. SKIPPED [INFO] Spark Project Streaming ... SKIPPED [INFO] Spark Project Catalyst SKIPPED [INFO] Spark Project SQL . SKIPPED [INFO] Spark Project ML Library .. SKIPPED [INFO] Spark Project Tools ... SKIPPED [INFO] Spark Project Hive SKIPPED [INFO] Spark Project REPL SKIPPED [INFO] Spark Project YARN Parent POM . SKIPPED [INFO] Spark Project YARN Stable API . SKIPPED [INFO] Spark Project Assembly SKIPPED [INFO] Spark Project External Twitter SKIPPED [INFO] Spark Project External Flume Sink . SKIPPED [INFO] Spark Project External Flume .. SKIPPED [INFO] Spark Project External MQTT ... SKIPPED [INFO] Spark Project External ZeroMQ . SKIPPED [INFO] Spark Project External Kafka .. SKIPPED [INFO] Spark Project Examples SKIPPED [INFO] Spark Project YARN Shuffle Service SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 1.644s [INFO] Finished at: Mon Dec 22 10:56:35 CST 2014 [INFO] Final Memory: 21M/481M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) on project spark-parent: Error finding remote resources manifests: /home/q/spark/spark-1.2.0/target/maven-shared-archive-resources/META-INF/NOTICE (No such file or directory) - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException but the NOTICE file is in the download spark release: [wyp@spark /home/q/spark/spark-1.2.0]$ ll total 248 drwxrwxr-x 3 1000 1000 4096 Dec 10 18:02 assembly drwxrwxr-x 3 1000 1000 4096 Dec 10 18:02 bagel drwxrwxr-x 2 1000 1000 4096 Dec 10 18:02 bin drwxrwxr-x 2 1000 1000 4096 Dec 10 18:02 conf -rw-rw-r-- 1 1000 1000 663 Dec 10 18:02 CONTRIBUTING.md drwxrwxr-x 3 1000 1000 4096 Dec 10 18:02 core drwxrwxr-x 3 1000 1000 4096 Dec 10 18:02 data drwxrwxr-x 4 1000 1000 4096 Dec 10 18:02 dev drwxrwxr-x 3 1000 1000 4096 Dec 10 18:02 docker drwxrwxr-x 7 1000 1000 4096 Dec 10 18:02 docs drwxrwxr-x 4 1000 1000 4096 Dec 10 18:02 ec2 drwxrwxr-x 4 1000 1000 4096 Dec 10 18:02 examples drwxrwxr-x 8 1000 1000 4096 Dec 10 18:02 external drwxrwxr-x 5 1000 1000 4096 Dec 10 18:02 extras drwxrwxr-x 4 1000 1000 4096 Dec 10 18:02 graphx -rw-rw-r-- 1 1000 1000 45242 Dec 10 18:02 LICENSE -rwxrwxr-x 1 1000 1000 7941 Dec 10 18:02 make-distribution.sh drwxrwxr-x 3 1000 1000 4096 Dec 10 18:02 mllib drwxrwxr-x 5 1000 1000 4096 Dec 10 18:02 network -rw-rw-r-- 1 1000 1000 22559 Dec 10 18:02 NOTICE -rw-rw-r-- 1 1000 1000 49002 Dec 10 18:02 pom.xml drwxrwxr-x 4 1000 1000 4096 Dec 10 18:02 project drwxrwxr-x 6 1000 1000 4096 Dec 10 18:02 python -rw-rw-r-- 1 1000 1000 3645 Dec 10 18:02 README.md drwxrwxr-x 5 1000 1000 4096 Dec 10 18:02 repl drwxrwxr-x 2 1000 1000 4096 Dec 10 18:02 sbin drwxrwxr-x 2 1000 1000 4096 Dec 10 18:02 sbt -rw-rw-r-- 1 1000 1000 7804 Dec 10 18:02 scalastyle-config.xml drwxrwxr-x 6 1000 1000 4096 Dec 10 18:02 sql drwxrwxr-x 3 1000 1000 4096 Dec 10 18:02 streaming drwxrwxr-x 3 1000 1000 4096 Dec 10 18:02 tools -rw-rw-r-- 1 1000 1000 838 Dec 10 18:02 tox.ini drwxrwxr-x 5
