Re: [VOTE] SPARK 2.3.2 (RC1)
+1 On Wed, Jul 11, 2018 at 1:31 AM John Zhuge wrote: > +1 > > On Sun, Jul 8, 2018 at 1:30 AM Saisai Shao wrote: > >> Please vote on releasing the following candidate as Apache Spark version >> 2.3.2. >> >> The vote is open until July 11th PST and passes if a majority +1 PMC >> votes are cast, with a minimum of 3 +1 votes. >> >> [ ] +1 Release this package as Apache Spark 2.3.2 >> [ ] -1 Do not release this package because ... >> >> To learn more about Apache Spark, please see http://spark.apache.org/ >> >> The tag to be voted on is v2.3.2-rc1 >> (commit 4df06b45160241dbb331153efbb25703f913c192): >> https://github.com/apache/spark/tree/v2.3.2-rc1 >> >> The release files, including signatures, digests, etc. can be found at: >> https://dist.apache.org/repos/dist/dev/spark/v2.3.2-rc1-bin/ >> >> Signatures used for Spark RCs can be found in this file: >> https://dist.apache.org/repos/dist/dev/spark/KEYS >> >> The staging repository for this release can be found at: >> https://repository.apache.org/content/repositories/orgapachespark-1277/ >> >> The documentation corresponding to this release can be found at: >> https://dist.apache.org/repos/dist/dev/spark/v2.3.2-rc1-docs/ >> >> The list of bug fixes going into 2.3.2 can be found at the following URL: >> https://issues.apache.org/jira/projects/SPARK/versions/12343289 >> >> PS. This is my first time to do release, please help to check if >> everything is landing correctly. Thanks ^-^ >> >> FAQ >> >> = >> How can I help test this release? >> = >> >> If you are a Spark user, you can help us test this release by taking >> an existing Spark workload and running on this release candidate, then >> reporting any regressions. >> >> If you're working in PySpark you can set up a virtual env and install >> the current RC and see if anything important breaks, in the Java/Scala >> you can add the staging repository to your projects resolvers and test >> with the RC (make sure to clean up the artifact cache before/after so >> you don't end up building with a out of date RC going forward). >> >> === >> What should happen to JIRA tickets still targeting 2.3.2? >> === >> >> The current list of open tickets targeted at 2.3.2 can be found at: >> https://issues.apache.org/jira/projects/SPARK and search for "Target >> Version/s" = 2.3.2 >> >> Committers should look at those and triage. Extremely important bug >> fixes, documentation, and API tweaks that impact compatibility should >> be worked on immediately. Everything else please retarget to an >> appropriate release. >> >> == >> But my bug isn't fixed? >> == >> >> In order to make timely releases, we will typically not hold the >> release unless the bug in question is a regression from the previous >> release. That being said, if there is something which is a regression >> that has not been correctly targeted please ping me or a committer to >> help target the issue. >> > > > -- > John >
Re: Unable to alter partition. The transaction for alter partition did not commit successfully.
I am reading data from Kafka topics using create stream and pushing it to hive by using dataframes. The job seems to run fine for the 5-6 hours and then it fails with the above exception. On Wednesday, May 30, 2018, 3:31:10 PM PDT, naresh Goud wrote: What are you doing? Give more details o what are you doing On Wed, May 30, 2018 at 12:58 PM Arun Hive wrote: Hi While running my spark job component i am getting the following exception. Requesting for your help on this:Spark core version - spark-core_2.10-2.1.1 Spark streaming version -spark-streaming_2.10-2.1.1 Spark hive version -spark-hive_2.10-2.1.1 2018-05-28 00:08:04,317 [streaming-job-executor-2] ERROR (Hive.java:1883) - org.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter partition. The transaction for alter partition did not commit successfully. at org.apache.hadoop.hive.ql.metadata.Hive.alterPartition(Hive.java:573) at org.apache.hadoop.hive.ql.metadata.Hive.alterPartition(Hive.java:546) at org.apache.hadoop.hive.ql.metadata.Hive.alterPartitionSpec(Hive.java:1915) at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:1875) at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1407) at org.apache.hadoop.hive.ql.metadata.Hive.loadDynamicPartitions(Hive.java:1593) at sun.reflect.GeneratedMethodAccessor123.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.sql.hive.client.Shim_v1_2.loadDynamicPartitions(HiveShim.scala:831) at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadDynamicPartitions$1.apply$mcV$sp(HiveClientImpl.scala:693) at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadDynamicPartitions$1.apply(HiveClientImpl.scala:691) at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadDynamicPartitions$1.apply(HiveClientImpl.scala:691) at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:279) at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:226) at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:225) at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:268) at org.apache.spark.sql.hive.client.HiveClientImpl.loadDynamicPartitions(HiveClientImpl.scala:691) at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadDynamicPartitions$1.apply$mcV$sp(HiveExternalCatalog.scala:823) at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadDynamicPartitions$1.apply(HiveExternalCatalog.scala:811) at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadDynamicPartitions$1.apply(HiveExternalCatalog.scala:811) at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97) at org.apache.spark.sql.hive.HiveExternalCatalog.loadDynamicPartitions(HiveExternalCatalog.scala:811) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.sideEffectResult$lzycompute(InsertIntoHiveTable.scala:319) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.sideEffectResult(InsertIntoHiveTable.scala:221) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.doExecute(InsertIntoHiveTable.scala:407) at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114) at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114) at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132) at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113) at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92) at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92) at org.apache.spark.sql.DataFrameWriter.insertInto(DataFrameWriter.scala:263) at org.apache.spark.sql.DataFrameWriter.insertInto(DataFrameWriter.scala:243) - - - at org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$1.apply(JavaDStreamLike.scala:272) at org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$1.apply(JavaDStreamLike.scala:272) at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:627) at
Re: [VOTE] SPARK 2.3.2 (RC1)
+1 On Sun, Jul 8, 2018 at 1:30 AM Saisai Shao wrote: > Please vote on releasing the following candidate as Apache Spark version > 2.3.2. > > The vote is open until July 11th PST and passes if a majority +1 PMC votes > are cast, with a minimum of 3 +1 votes. > > [ ] +1 Release this package as Apache Spark 2.3.2 > [ ] -1 Do not release this package because ... > > To learn more about Apache Spark, please see http://spark.apache.org/ > > The tag to be voted on is v2.3.2-rc1 > (commit 4df06b45160241dbb331153efbb25703f913c192): > https://github.com/apache/spark/tree/v2.3.2-rc1 > > The release files, including signatures, digests, etc. can be found at: > https://dist.apache.org/repos/dist/dev/spark/v2.3.2-rc1-bin/ > > Signatures used for Spark RCs can be found in this file: > https://dist.apache.org/repos/dist/dev/spark/KEYS > > The staging repository for this release can be found at: > https://repository.apache.org/content/repositories/orgapachespark-1277/ > > The documentation corresponding to this release can be found at: > https://dist.apache.org/repos/dist/dev/spark/v2.3.2-rc1-docs/ > > The list of bug fixes going into 2.3.2 can be found at the following URL: > https://issues.apache.org/jira/projects/SPARK/versions/12343289 > > PS. This is my first time to do release, please help to check if > everything is landing correctly. Thanks ^-^ > > FAQ > > = > How can I help test this release? > = > > If you are a Spark user, you can help us test this release by taking > an existing Spark workload and running on this release candidate, then > reporting any regressions. > > If you're working in PySpark you can set up a virtual env and install > the current RC and see if anything important breaks, in the Java/Scala > you can add the staging repository to your projects resolvers and test > with the RC (make sure to clean up the artifact cache before/after so > you don't end up building with a out of date RC going forward). > > === > What should happen to JIRA tickets still targeting 2.3.2? > === > > The current list of open tickets targeted at 2.3.2 can be found at: > https://issues.apache.org/jira/projects/SPARK and search for "Target > Version/s" = 2.3.2 > > Committers should look at those and triage. Extremely important bug > fixes, documentation, and API tweaks that impact compatibility should > be worked on immediately. Everything else please retarget to an > appropriate release. > > == > But my bug isn't fixed? > == > > In order to make timely releases, we will typically not hold the > release unless the bug in question is a regression from the previous > release. That being said, if there is something which is a regression > that has not been correctly targeted please ping me or a committer to > help target the issue. > -- John
[ANNOUNCE] Apache Spark 2.2.2
We are happy to announce the availability of Spark 2.2.2! Apache Spark 2.2.2 is a maintenance release, based on the branch-2.2 maintenance branch of Spark. We strongly recommend all 2.2.x users to upgrade to this stable release. The release notes are available at http://spark.apache.org/releases/spark-release-2-2-2.html To download Apache Spark 2.2.2 visit http://spark.apache.org/downloads.html. This version of Spark is also available on Maven and PyPI. We would like to acknowledge all community members for contributing patches to this release.
