Re: Docker image to build Spark/Spark doc
Me and my colleagues built one for running spark builds on circleci. The images are at https://hub.docker.com/r/palantirtechnologies/circle-spark-python/ (circle-spark-r if you want to build sparkr). Dockerfiles for those images can be found at https://github.com/palantir/spark/tree/master/dev/docker-images On Wed, 10 Oct 2018 at 15:31, Sean Owen wrote: > You can just build it with Maven or SBT as in the docs. I don't know of a > docker image but there isn't much to package. > > On Wed, Oct 10, 2018, 1:10 AM assaf.mendelson > wrote: > >> Hi all, >> I was wondering if there was a docker image to build spark and/or spark >> documentation >> >> The idea would be that I would start the docker image, supplying the >> directory with my code and a target directory and it would simply build >> everything (maybe with some options). >> >> Any chance there is already something like that which is working and >> tested? >> >> Thanks, >> Assaf >> >> >> >> >> -- >> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ >> >> - >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >>
Re: [VOTE] Apache Spark 2.1.0 (RC1)
-1 since https://issues.apache.org/jira/browse/SPARK-17213 is a correctness regression from 2.0 release. The commit that caused it is 776d183c82b424ef7c3cae30537d8afe9b9eee83. Robert From: Reynold Xin Date: Tuesday, November 29, 2016 at 1:25 AM To: "dev@spark.apache.org" Subject: [VOTE] Apache Spark 2.1.0 (RC1) Please vote on releasing the following candidate as Apache Spark version 2.1.0. The vote is open until Thursday, December 1, 2016 at 18:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 2.1.0 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ The tag to be voted on is v2.1.0-rc1 (80aabc0bd33dc5661a90133156247e7a8c1bf7f5) The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-2.1.0-rc1-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-1216/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-2.1.0-rc1-docs/ === How can I help test this release? === If you are a Spark user, you can help us test this release by taking an existing Spark workload and running on this release candidate, then reporting any regressions. === What should happen to JIRA tickets still targeting 2.1.0? === Committers should look at those and triage. Extremely important bug fixes, documentation, and API tweaks that impact compatibility should be worked on immediately. Everything else please retarget to 2.1.1 or 2.2.0. smime.p7s Description: S/MIME cryptographic signature
Re: critical bugs to be fixed in Spark 2.0.1?
SPARK-16991 (https://github.com/apache/spark/pull/14661) would be nice Robert From: Reynold Xin Date: Monday, August 22, 2016 at 8:14 PM To: "dev@spark.apache.org" Subject: critical bugs to be fixed in Spark 2.0.1? We should work on a 2.0.1 release soon, since we have found couple critical bugs in 2.0.0. Are there any critical bugs outstanding that we should address in 2.0.1? smime.p7s Description: S/MIME cryptographic signature
[1.6] Coalesce/binary operator on casted named column
Hi Spark devs, I have been debugging failing unit test in our application and it led me to believe that the bug is in spark itself. The exception I am getting is org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to dataType on unresolved object, tree: unresolvedalias(cast(string_field#164 as string)) at org.apache.spark.sql.catalyst.analysis.UnresolvedAlias.dataType(unresolved.scala:295) at org.apache.spark.sql.catalyst.expressions.Coalesce$$anonfun$checkInputDataTypes$1.apply(nullExpressions.scala:49) at org.apache.spark.sql.catalyst.expressions.Coalesce$$anonfun$checkInputDataTypes$1.apply(nullExpressions.scala:49) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at org.apache.spark.sql.catalyst.expressions.Coalesce.checkInputDataTypes(nullExpressions.scala:49) at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:62) at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:57) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:319) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:319) at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:53) at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:318) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$5.apply(TreeNode.scala:316) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$5.apply(TreeNode.scala:316) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:265) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47) at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273) at scala.collection.AbstractIterator.to(Iterator.scala:1157) at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265) at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157) at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252) at scala.collection.AbstractIterator.toArray(Iterator.scala:1157) at org.apache.spark.sql.catalyst.trees.TreeNode.transformChildren(TreeNode.scala:305) at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:316) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$5.apply(TreeNode.scala:316) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$5.apply(TreeNode.scala:316) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:265) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47) at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273) at scala.collection.AbstractIterator.to(Iterator.scala:1157) at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265) at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157) at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252) at scala.collection.AbstractIterator.toArray(Iterator.scala:1157) at org.apache.spark.sql.catalyst.trees.TreeNode.transformChildren(TreeNode.scala:305) at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:316) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$5.apply(TreeNode.scala:316) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$5.apply(TreeNode.scala:316) at or
