RE: [VOTE] Release Apache Spark 1.4.0 (RC4)
+1 -Original Message- From: Patrick Wendell [mailto:pwend...@gmail.com] Sent: Wednesday, June 03, 2015 1:47 PM To: dev@spark.apache.org Subject: Re: [VOTE] Release Apache Spark 1.4.0 (RC4) He all - a tiny nit from the last e-mail. The tag is v1.4.0-rc4. The exact commit and all other information is correct. (thanks Shivaram who pointed this out). On Tue, Jun 2, 2015 at 8:53 PM, Patrick Wendell wrote: > Please vote on releasing the following candidate as Apache Spark version > 1.4.0! > > The tag to be voted on is v1.4.0-rc3 (commit 22596c5): > https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= > 22596c534a38cfdda91aef18aa9037ab101e4251 > > The release files, including signatures, digests, etc. can be found at: > http://people.apache.org/~pwendell/spark-releases/spark-1.4.0-rc4-bin/ > > Release artifacts are signed with the following key: > https://people.apache.org/keys/committer/pwendell.asc > > The staging repository for this release can be found at: > [published as version: 1.4.0] > https://repository.apache.org/content/repositories/orgapachespark- > / > [published as version: 1.4.0-rc4] > https://repository.apache.org/content/repositories/orgapachespark-1112 > / > > The documentation corresponding to this release can be found at: > http://people.apache.org/~pwendell/spark-releases/spark-1.4.0-rc4-docs > / > > Please vote on releasing this package as Apache Spark 1.4.0! > > The vote is open until Saturday, June 06, at 05:00 UTC and passes if a > majority of at least 3 +1 PMC votes are cast. > > [ ] +1 Release this package as Apache Spark 1.4.0 [ ] -1 Do not > release this package because ... > > To learn more about Apache Spark, please see http://spark.apache.org/ > > == What has changed since RC3 == > In addition to may smaller fixes, three blocker issues were fixed: > 4940630 [SPARK-8020] [SQL] Spark SQL conf in spark-defaults.conf make > metadataHive get constructed too early > 6b0f615 [SPARK-8038] [SQL] [PYSPARK] fix Column.when() and otherwise() > 78a6723 [SPARK-7978] [SQL] [PYSPARK] DecimalType should not be > singleton > > == How can I help test this release? == If you are a Spark user, you > can help us test this release by taking a Spark 1.3 workload and > running on this release candidate, then reporting any regressions. > > == What justifies a -1 vote for this release? == This vote is > happening towards the end of the 1.4 QA period, so -1 votes should > only occur for significant regressions from 1.3.1. > Bugs already present in 1.3.X, minor regressions, or bugs related to > new features will not block this release. - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
RE: [VOTE] Release Apache Spark 1.4.0 (RC2)
Good catch! BTW, SPARK-6784 is duplicate to SPAKR-7790, didn't notice we changed the title of SPARK-7853.. -Original Message- From: Cheng, Hao [mailto:hao.ch...@intel.com] Sent: Monday, May 25, 2015 4:47 PM To: Sean Owen; Patrick Wendell Cc: dev@spark.apache.org Subject: RE: [VOTE] Release Apache Spark 1.4.0 (RC2) Add another Blocker issue, just created! It seems a regression. https://issues.apache.org/jira/browse/SPARK-7853 -Original Message- From: Sean Owen [mailto:so...@cloudera.com] Sent: Monday, May 25, 2015 3:37 PM To: Patrick Wendell Cc: dev@spark.apache.org Subject: Re: [VOTE] Release Apache Spark 1.4.0 (RC2) We still have 1 blocker for 1.4: SPARK-6784 Make sure values of partitioning columns are correctly converted based on their data types CC Davies Liu / Adrian Wang to check on the status of this. There are still 50 Critical issues tagged for 1.4, and 183 issues targeted for 1.4 in general. Obviously almost all of those won't be in 1.4. How do people want to deal with those? The field can be cleared, but do people want to take a pass at bumping a few to 1.4.1 that really truly are supposed to go into 1.4.1? On Sun, May 24, 2015 at 8:22 AM, Patrick Wendell wrote: > Please vote on releasing the following candidate as Apache Spark version > 1.4.0! > > The tag to be voted on is v1.4.0-rc2 (commit 03fb26a3): > https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=03fb26a > 3e50e00739cc815ba4e2e82d71d003168 > > The release files, including signatures, digests, etc. can be found at: > http://people.apache.org/~pwendell/spark-releases/spark-1.4.0-rc2-bin/ > > Release artifacts are signed with the following key: > https://people.apache.org/keys/committer/pwendell.asc > > The staging repository for this release can be found at: > [published as version: 1.4.0] > https://repository.apache.org/content/repositories/orgapachespark-1103 > / > [published as version: 1.4.0-rc2] > https://repository.apache.org/content/repositories/orgapachespark-1104 > / > > The documentation corresponding to this release can be found at: > http://people.apache.org/~pwendell/spark-releases/spark-1.4.0-rc2-docs > / > > Please vote on releasing this package as Apache Spark 1.4.0! > > The vote is open until Wednesday, May 27, at 08:12 UTC and passes if a > majority of at least 3 +1 PMC votes are cast. > > [ ] +1 Release this package as Apache Spark 1.4.0 [ ] -1 Do not > release this package because ... > > To learn more about Apache Spark, please see http://spark.apache.org/ > > == What has changed since RC1 == > Below is a list of bug fixes that went into this RC: > http://s.apache.org/U1M > > == How can I help test this release? == If you are a Spark user, you > can help us test this release by taking a Spark 1.3 workload and > running on this release candidate, then reporting any regressions. > > == What justifies a -1 vote for this release? == This vote is > happening towards the end of the 1.4 QA period, so -1 votes should > only occur for significant regressions from 1.3.1. > Bugs already present in 1.3.X, minor regressions, or bugs related to > new features will not block this release. > > - > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For > additional commands, e-mail: dev-h...@spark.apache.org > - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
RE: Unsupported Catalyst types in Parquet
By adding a flag in SQLContext, I have modified #3822 to include nanoseconds now. Since passing too many flags is ugly, now I need the whole SQLContext, so that we can put more flags there. Thanks, Daoyuan From: Michael Armbrust [mailto:mich...@databricks.com] Sent: Tuesday, December 30, 2014 10:43 AM To: Alessandro Baretta Cc: Wang, Daoyuan; dev@spark.apache.org Subject: Re: Unsupported Catalyst types in Parquet Yeah, I saw those. The problem is that #3822 truncates timestamps that include nanoseconds. On Mon, Dec 29, 2014 at 5:14 PM, Alessandro Baretta mailto:alexbare...@gmail.com>> wrote: Michael, Actually, Adrian Wang already created pull requests for these issues. https://github.com/apache/spark/pull/3820 https://github.com/apache/spark/pull/3822 What do you think? Alex On Mon, Dec 29, 2014 at 3:07 PM, Michael Armbrust mailto:mich...@databricks.com>> wrote: I'd love to get both of these in. There is some trickiness that I talk about on the JIRA for timestamps since the SQL timestamp class can support nano seconds and I don't think parquet has a type for this. Other systems (impala) seem to use INT96. It would be great to maybe ask on the parquet mailing list what the plan is there to make sure that whatever we do is going to be compatible long term. Michael On Mon, Dec 29, 2014 at 8:13 AM, Alessandro Baretta mailto:alexbare...@gmail.com>> wrote: Daoyuan, Thanks for creating the jiras. I need these features by... last week, so I'd be happy to take care of this myself, if only you or someone more experienced than me in the SparkSQL codebase could provide some guidance. Alex On Dec 29, 2014 12:06 AM, "Wang, Daoyuan" mailto:daoyuan.w...@intel.com>> wrote: Hi Alex, I'll create JIRA SPARK-4985 for date type support in parquet, and SPARK-4987 for timestamp type support. For decimal type, I think we only support decimals that fits in a long. Thanks, Daoyuan -Original Message- From: Alessandro Baretta [mailto:alexbare...@gmail.com<mailto:alexbare...@gmail.com>] Sent: Saturday, December 27, 2014 2:47 PM To: dev@spark.apache.org<mailto:dev@spark.apache.org>; Michael Armbrust Subject: Unsupported Catalyst types in Parquet Michael, I'm having trouble storing my SchemaRDDs in Parquet format with SparkSQL, due to my RDDs having having DateType and DecimalType fields. What would it take to add Parquet support for these Catalyst? Are there any other Catalyst types for which there is no Catalyst support? Alex
RE: Unsupported Catalyst types in Parquet
Hi Alex, I'll create JIRA SPARK-4985 for date type support in parquet, and SPARK-4987 for timestamp type support. For decimal type, I think we only support decimals that fits in a long. Thanks, Daoyuan -Original Message- From: Alessandro Baretta [mailto:alexbare...@gmail.com] Sent: Saturday, December 27, 2014 2:47 PM To: dev@spark.apache.org; Michael Armbrust Subject: Unsupported Catalyst types in Parquet Michael, I'm having trouble storing my SchemaRDDs in Parquet format with SparkSQL, due to my RDDs having having DateType and DecimalType fields. What would it take to add Parquet support for these Catalyst? Are there any other Catalyst types for which there is no Catalyst support? Alex - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
RE: object xxx is not a member of package com
I think you can place the jar in lib/ in SPARK_HOME, and then compile without any change to your class path. This could be a temporary way to include your jar. You can also put them in your pom.xml. Thanks, Daoyuan -Original Message- From: flyson [mailto:m_...@msn.com] Sent: Wednesday, December 03, 2014 11:23 AM To: d...@spark.incubator.apache.org Subject: object xxx is not a member of package com Hello everyone, Could anybody tell me how to import and call the 3rd party java classes from inside spark? Here's my case: I have a jar file (the directory layout is com.xxx.yyy.zzz) which contains some java classes, and I need to call some of them in spark code. I used the statement "import com.xxx.yyy.zzz._" on top of the impacted spark file and set the location of the jar file in the CLASSPATH environment, and use ".sbt/sbt assembly" to build the project. As a result, I got an error saying "object xxx is not a member of package com". I thought that could be related to the library dependencies, but couldn't figure it out. Any suggestion/solution from you would be appreciated! By the way in the scala console, if the :cp is used to point to the jar file, I can import the classes from the jar file. Thanks! -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/object-xxx-is-not-a-member-of-package-com-tp9619.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
