Re: [E] Re: SPARK-19547
Hi Pankaj, >> After the second consumer group comes upDo you mean a second consumer starts >> with the same consumer group as the first ? createDirectStream is overloaded. One of the method, doesn't need you to specify partitions of a topic. Cheers - Sree On Thursday, June 8, 2017 9:56 AM, "Rastogi, Pankaj" wrote: Hi, Thank you for your reply! You got it right! I am trying to run multiple streams using the same consumer, so that I can distribute different partitions among different instances of the consumer group. I don¹t want to provide the list of partitions in createDirectStream API. If I do that then it will become difficult to handle consumer failure as those partitions won¹t be ready by any consumer. Also I will have to handle addition of new partitions. I wanted to see if I can use partition rebalance feature. Pankaj On 6/8/17, 8:24 AM, "Cody Koeninger" wrote: >Can you explain in more detail what you mean by "distribute Kafka >topics among different instances of same consumer group"? > >If you're trying to run multiple streams using the same consumer >group, it's already documented that you shouldn't do that. > >On Thu, Jun 8, 2017 at 12:43 AM, Rastogi, Pankaj > wrote: >> Hi, >> I have been trying to distribute Kafka topics among different >>instances of >> same consumer group. I am using KafkaDirectStream API for creating >>DStreams. >> After the second consumer group comes up, Kafka does partition >>rebalance and >> then Spark driver of the first consumer dies with the following >>exception: >> >> java.lang.IllegalStateException: No current assignment for partition >> myTopic_5-0 >> at >> >>org.apache.kafka.clients.consumer.internals.SubscriptionState.assignedSta >>te(SubscriptionState.java:264) >> at >> >>org.apache.kafka.clients.consumer.internals.SubscriptionState.needOffsetR >>eset(SubscriptionState.java:336) >> at >> >>org.apache.kafka.clients.consumer.KafkaConsumer.seekToEnd(KafkaConsumer.j >>ava:1236) >> at >> >>org.apache.spark.streaming.kafka010.DirectKafkaInputDStream.latestOffsets >>(DirectKafkaInputDStream.scala:197) >> at >> >>org.apache.spark.streaming.kafka010.DirectKafkaInputDStream.compute(Direc >>tKafkaInputDStream.scala:214) >> at >> >>org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonf >>un$1$$anonfun$apply$7.apply(DStream.scala:341) >> at >> >>org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonf >>un$1$$anonfun$apply$7.apply(DStream.scala:341) >> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) >> at >> >>org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonf >>un$1.apply(DStream.scala:340) >> at >> >>org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonf >>un$1.apply(DStream.scala:340) >> at >> >>org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(D >>Stream.scala:415) >> at >> >>org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply( >>DStream.scala:335) >> at >> >>org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply( >>DStream.scala:333) >> at scala.Option.orElse(Option.scala:257) >> at >> >>org.apache.spark.streaming.dstream.DStream.getOrCompute(DStream.scala:330 >>) >> at >> >>org.apache.spark.streaming.dstream.TransformedDStream$$anonfun$6.apply(Tr >>ansformedDStream.scala:42) >> at >> >>org.apache.spark.streaming.dstream.TransformedDStream$$anonfun$6.apply(Tr >>ansformedDStream.scala:42) >> at >> >>scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.sca >>la:244) >> at >> >>scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.sca >>la:244) >> at scala.collection.immutable.List.foreach(List.scala:318) >> at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) >> at scala.collection.AbstractTraversable.map(Traversable.scala:105) >> at >> >>org.apache.spark.streaming.dstream.TransformedDStream.compute(Transformed >>DStream.scala:42) >> at >> >>org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonf >>un$1$$anonfun$apply$7.apply(DStream.scala:341) >> at >> >>org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonf >>un$1$$anonfun$apply$7.apply(DStream.scala:341) >> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) >> at >> >>org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonf >>un$1.apply(DStream.scala:340) >> at >> >>org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonf >>un$1.apply(DStream.scala:340) >> at >> >>org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(D >>Stream.scala:415) >> at >> >>org.apache.spark.streaming.dstream.TransformedDStream.createRDDWithLocalP >>roperties(TransformedDStream.scala:65) >> at >> >>org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply( >>DStream.scala:335) >> at >> >>org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply( >>DStream.sca
Re: [discuss] ending support for Java 6?
If there is any possibility of getting the download counts,then we can use it as EOS criteria as well.Say, if download counts are lower than 30% (or another number) of Life time highest,then it qualifies for EOS. Thanking you. With Regards Sree On Thursday, April 30, 2015 2:22 PM, Sree V wrote: Hi Team, Should we take this opportunity to layout and evangelize a pattern for EOL of dependencies.I propose, we follow the official EOL of java, python, scala, .And add say 6-12-24 months depending on the popularity. Java 6 official EOL Feb 2013Add 6-12 monthsAug 2013 - Feb 2014 official End of Support for Java 6 in SparkAnnounce 3-6 months prior to EOS. Thanking you. With Regards Sree On Thursday, April 30, 2015 1:41 PM, Marcelo Vanzin wrote: As for the idea, I'm +1. Spark is the only reason I still have jdk6 around - exactly because I don't want to cause the issue that started this discussion (inadvertently using JDK7 APIs). And as has been pointed out, even J7 is about to go EOL real soon. Even Hadoop is moving away (I think 2.7 will be j7-only). Hive 1.1 is already j7-only. And when Hadoop moves away from something, it's an event worthy of headlines. They're still on Jetty 6! As for pyspark, https://github.com/apache/spark/pull/5580 should get rid of the last incompatibility with large assemblies, by keeping the python files in separate archives. If we remove support for Java 6, then we don't need to worry about the size of the assembly anymore. On Thu, Apr 30, 2015 at 1:32 PM, Sean Owen wrote: > I'm firmly in favor of this. > > It would also fix https://issues.apache.org/jira/browse/SPARK-7009 and > avoid any more of the long-standing 64K file limit thing that's still > a problem for PySpark. -- Marcelo - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [discuss] ending support for Java 6?
Hi Team, Should we take this opportunity to layout and evangelize a pattern for EOL of dependencies.I propose, we follow the official EOL of java, python, scala, .And add say 6-12-24 months depending on the popularity. Java 6 official EOL Feb 2013Add 6-12 monthsAug 2013 - Feb 2014 official End of Support for Java 6 in SparkAnnounce 3-6 months prior to EOS. Thanking you. With Regards Sree On Thursday, April 30, 2015 1:41 PM, Marcelo Vanzin wrote: As for the idea, I'm +1. Spark is the only reason I still have jdk6 around - exactly because I don't want to cause the issue that started this discussion (inadvertently using JDK7 APIs). And as has been pointed out, even J7 is about to go EOL real soon. Even Hadoop is moving away (I think 2.7 will be j7-only). Hive 1.1 is already j7-only. And when Hadoop moves away from something, it's an event worthy of headlines. They're still on Jetty 6! As for pyspark, https://github.com/apache/spark/pull/5580 should get rid of the last incompatibility with large assemblies, by keeping the python files in separate archives. If we remove support for Java 6, then we don't need to worry about the size of the assembly anymore. On Thu, Apr 30, 2015 at 1:32 PM, Sean Owen wrote: > I'm firmly in favor of this. > > It would also fix https://issues.apache.org/jira/browse/SPARK-7009 and > avoid any more of the long-standing 64K file limit thing that's still > a problem for PySpark. -- Marcelo - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: Tungsten + Flink
I agree, Ewan. We should also look into combining both Flink and Spark into one.This eases the industry adaptation instead. Thanking you. With Regards Sree On Wednesday, April 29, 2015 3:21 AM, Ewan Higgs wrote: Hi all, A quick question about Tungsten. The announcement of the Tungsten project is on the back of Hadoop Summit in Brussels where some of the Flink devs were giving talks [1] on how Flink manages memory using byte arrays and the like to avoid the overhead of all the Java types[2]. Is there an opportunity for code reuse here? Spark and Flink may have different needs in some respects, but they work fundamentally towards the same goal so I imagine there could be come worthwhile collaboration. -Ewan [1] http://2015.hadoopsummit.org/brussels/speaker/?speaker=MrtonBalassi http://2015.hadoopsummit.org/brussels/speaker/?speaker=AljoschaKrettek [2] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=53741525 https://flink.apache.org/news/2015/03/13/peeking-into-Apache-Flinks-Engine-Room.html - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [RESULT] [VOTE] Release Apache Spark 1.2.2
YARN Shuffle Service . SKIPPED
[INFO]
[INFO] BUILD FAILURE
[INFO]
[INFO] Total time: 23:58 min
[INFO] Finished at: 2015-04-17T15:58:27-07:00
[INFO] Final Memory: 151M/1924M
[INFO]
[ERROR] Failed to execute goal
net.alchim31.maven:scala-maven-plugin:3.2.0:compile (scala-compile-first) on
project spark-streaming-flume_2.10: Execution scala-compile-first of goal
net.alchim31.maven:scala-maven-plugin:3.2.0:compile failed. CompileFailed ->
[Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :spark-streaming-flume_2.10
Thanking you.
With Regards
Sree
On Friday, April 17, 2015 3:18 PM, Sree V
wrote:
Hi Sean,
This is from build log. I made a master branch build earlier on this
machine.Do you think, it needs a clean up of .m2 folder, that you suggested in
onetime earlier ?Giving it another try, while you take a look at this.
[INFO] --- scala-maven-plugin:3.2.0:compile (scala-compile-first) @
spark-streaming-flume_2.10 ---
[WARNING] Zinc server is not available at port 3030 - reverting to normal
incremental compile
[INFO] Using incremental compilation
[INFO] compiler plugin:
BasicArtifact(org.scalamacros,paradise_2.10.4,2.0.1,null)
[INFO] Compiling 6 Scala sources and 1 Java source to
/root/sources/github/spark/external/flume/target/scala-2
.10/classes...
[ERROR]
/root/sources/github/spark/external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumeBatchFe
tcher.scala:22: object Throwables is not a member of package
com.google.common.base
[ERROR] import com.google.common.base.Throwables
[ERROR] ^
[ERROR]
/root/sources/github/spark/external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumeBatchFe
tcher.scala:59: not found: value Throwables
[ERROR] Throwables.getRootCause(e) match {
[ERROR] ^
[ERROR]
/root/sources/github/spark/external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumePolling
InputDStream.scala:26: object util is not a member of package com.google.common
[ERROR] import com.google.common.util.concurrent.ThreadFactoryBuilder
[ERROR] ^
[ERROR]
/root/sources/github/spark/external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumePolling
InputDStream.scala:69: not found: type ThreadFactoryBuilder
[ERROR] Executors.newCachedThreadPool(new
ThreadFactoryBuilder().setDaemon(true).
[ERROR] ^
[ERROR]
/root/sources/github/spark/external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumePolling
InputDStream.scala:76: not found: type ThreadFactoryBuilder
[ERROR] new ThreadFactoryBuilder().setDaemon(true).setNameFormat("Flume
Receiver Thread - %d").build())
[ERROR] ^
[ERROR] 5 errors found
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM ... SUCCESS [ 15.894 s]
[INFO] Spark Project Networking ... SUCCESS [ 20.801 s]
[INFO] Spark Project Shuffle Streaming Service SUCCESS [ 18.111 s]
[INFO] Spark Project Core . SUCCESS [08:09 min]
[INFO] Spark Project Bagel SUCCESS [ 43.592 s]
[INFO] Spark Project GraphX ... SUCCESS [01:55 min]
[INFO] Spark Project Streaming SUCCESS [03:02 min]
[INFO] Spark Project Catalyst . SUCCESS [02:59 min]
[INFO] Spark Project SQL .. SUCCESS [03:09 min]
[INFO] Spark Project ML Library ... SUCCESS [03:24 min]
[INFO] Spark Project Tools SUCCESS [ 24.816 s]
[INFO] Spark Project Hive . SUCCESS [02:14 min]
[INFO] Spark Project REPL . SUCCESS [01:12 min]
[INFO] Spark Project YARN Parent POM .. SUCCESS [ 6.080 s]
[INFO] Spark Project YARN Stable API .. SUCCESS [01:27 min]
[INFO] Spark Project Assembly . SUCCESS [01:22 min]
[INFO] Spark Project External Twitter . SUCCESS [ 35.881 s]
[INFO] Spark Project External Flume Sink .. SUCCESS [ 39.561 s]
[INFO] Spark Project External Flume ...
Re: [RESULT] [VOTE] Release Apache Spark 1.2.2
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :spark-streaming-flume_2.10 Thanking you. With Regards Sree On Friday, April 17, 2015 1:08 AM, Sean Owen wrote: Sree that doesn't show any error, so it doesn't help. I built with the same flags when I tested and it succeeded. On Fri, Apr 17, 2015 at 8:53 AM, Sree V wrote: > Sorry, I couldn't catch up before closing the voting.If it still counts, mvn > package fails (1). And didn't run test (2). So, -1.1.mvn -Phadoop-2.4 > -Pyarn -Phive -Phive-0.13.1 -Dhadoop.version=2.6.0 -DskipTests clean package > 2. mvn -Phadoop-2.4 -Pyarn -Phive -Phive-0.13.1 -Dhadoop.version=2.6.0 test > Error: > [INFO] Spark Project External Flume Sink .. SUCCESS [ 39.561 > s] > [INFO] Spark Project External Flume ... FAILURE [ 11.212 > s] > [INFO] Spark Project External MQTT SKIPPED > [INFO] Spark Project External ZeroMQ .. SKIPPED > [INFO] Spark Project External Kafka ... SKIPPED > [INFO] Spark Project Examples . SKIPPED > [INFO] Spark Project YARN Shuffle Service . SKIPPED > > > Thanking you. > > With Regards > Sree > > > On Thursday, April 16, 2015 3:42 PM, Patrick Wendell > wrote: > > > I'm gonna go ahead and close this now - thanks everyone for voting! > > This vote passes with 7 +1 votes (6 binding) and no 0 or -1 votes. > > +1: > Mark Hamstra* > Reynold Xin > Kirshna Sankar > Sean Owen* > Tom Graves* > Joseph Bradley* > Sean McNamara* > > 0: > > -1: > > Thanks! > - Patrick > > On Thu, Apr 16, 2015 at 3:27 PM, Sean Owen wrote: >> No, of course Jenkins runs tests. The way some of the tests work, they >> need the build artifacts to have been created first. So it runs "mvn >> ... -DskipTests package" then "mvn ... test" >> >> On Thu, Apr 16, 2015 at 11:09 PM, Sree V wrote: >>> In my effort to vote for this release, I found these along: >>> >>> This is from jenkins. It uses "-DskipTests". >>> >>> [centos] $ >>> /home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.0.5/bin/mvn >>> -Dhadoop.version=2.0.0-mr1-cdh4.1.2 -Dlabel=centos -DskipTests clean package >>> >>> We build on our locals / servers using same flag. >>> >>> >>> Usually, for releases we build with running all the tests, as well. and at >>> some level of code coverage. >>> >>> Are we by-passing it ? >>> >>> >>> >>> Thanking you. >>> >>> With Regards >>> Sree >>> >>> >>> >>> On Wednesday, April 15, 2015 3:32 PM, Sean McNamara >>> wrote: >>> >>> >>> Ran tests on OS X >>> >>> +1 >>> >>> Sean >>> >>> >>>> On Apr 14, 2015, at 10:59 PM, Patrick Wendell wrote: >>>> >>>> I'd like to close this vote to coincide with the 1.3.1 release, >>>> however, it would be great to have more people test this release >>>> first. I'll leave it open for a bit longer and see if others can give >>>> a +1. >>>> >>>> On Tue, Apr 14, 2015 at 9:55 PM, Patrick Wendell >>>> wrote: >>>>> +1 from me ass well. >>>>> >>>>> On Tue, Apr 7, 2015 at 4:36 AM, Sean Owen wrote: >>>>>> I think that's close enough for a +1: >>>>>> >>>>>> Signatures and hashes are good. >>>>>> LICENSE, NOTICE still check out. >>>>>> Compiles for a Hadoop 2.6 + YARN + Hive profile. >>>>>> >>>>>> JIRAs with target version = 1.2.x look legitimate; no blockers. >>>>>> >>>>>> I still observe several Hive test failures with: >>>>>> mvn -Phadoop-2.4 -Pyarn -Phive -Phive-0.13.1 -Dhadoop.version=2.6.0 >>>>>> -DskipTests clean package; mvn -Phadoop-2.4 -Pyarn -Phive >>>>>> -Phive-0.13.1 -Dhadoop.version=2.6.0 test >>>>>> .. though again I think these are not regressions but known issues in >>>>>> older branches. >>>>>> >>>>>> FYI there are 16 Critical issues still open for 1.2.x: >>>>>> >>>>>> SPARK-6209,ExecutorClassLoader can
Re: [RESULT] [VOTE] Release Apache Spark 1.2.2
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :spark-streaming-flume_2.10 Thanking you. With Regards Sree On Friday, April 17, 2015 1:08 AM, Sean Owen wrote: Sree that doesn't show any error, so it doesn't help. I built with the same flags when I tested and it succeeded. On Fri, Apr 17, 2015 at 8:53 AM, Sree V wrote: > Sorry, I couldn't catch up before closing the voting.If it still counts, mvn > package fails (1). And didn't run test (2). So, -1.1.mvn -Phadoop-2.4 > -Pyarn -Phive -Phive-0.13.1 -Dhadoop.version=2.6.0 -DskipTests clean package > 2. mvn -Phadoop-2.4 -Pyarn -Phive -Phive-0.13.1 -Dhadoop.version=2.6.0 test > Error: > [INFO] Spark Project External Flume Sink .. SUCCESS [ 39.561 > s] > [INFO] Spark Project External Flume ... FAILURE [ 11.212 > s] > [INFO] Spark Project External MQTT SKIPPED > [INFO] Spark Project External ZeroMQ .. SKIPPED > [INFO] Spark Project External Kafka ... SKIPPED > [INFO] Spark Project Examples . SKIPPED > [INFO] Spark Project YARN Shuffle Service . SKIPPED > > > Thanking you. > > With Regards > Sree >
Re: [RESULT] [VOTE] Release Apache Spark 1.2.2
Sorry, I couldn't catch up before closing the voting.If it still counts, mvn package fails (1). And didn't run test (2). So, -1.1.mvn -Phadoop-2.4 -Pyarn -Phive -Phive-0.13.1 -Dhadoop.version=2.6.0 -DskipTests clean package 2. mvn -Phadoop-2.4 -Pyarn -Phive -Phive-0.13.1 -Dhadoop.version=2.6.0 test Error: [INFO] Spark Project External Flume Sink .. SUCCESS [ 39.561 s] [INFO] Spark Project External Flume ... FAILURE [ 11.212 s] [INFO] Spark Project External MQTT SKIPPED [INFO] Spark Project External ZeroMQ .. SKIPPED [INFO] Spark Project External Kafka ... SKIPPED [INFO] Spark Project Examples . SKIPPED [INFO] Spark Project YARN Shuffle Service . SKIPPED Thanking you. With Regards Sree On Thursday, April 16, 2015 3:42 PM, Patrick Wendell wrote: I'm gonna go ahead and close this now - thanks everyone for voting! This vote passes with 7 +1 votes (6 binding) and no 0 or -1 votes. +1: Mark Hamstra* Reynold Xin Kirshna Sankar Sean Owen* Tom Graves* Joseph Bradley* Sean McNamara* 0: -1: Thanks! - Patrick On Thu, Apr 16, 2015 at 3:27 PM, Sean Owen wrote: > No, of course Jenkins runs tests. The way some of the tests work, they > need the build artifacts to have been created first. So it runs "mvn > ... -DskipTests package" then "mvn ... test" > > On Thu, Apr 16, 2015 at 11:09 PM, Sree V wrote: >> In my effort to vote for this release, I found these along: >> >> This is from jenkins. It uses "-DskipTests". >> >> [centos] $ >> /home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.0.5/bin/mvn >> -Dhadoop.version=2.0.0-mr1-cdh4.1.2 -Dlabel=centos -DskipTests clean package >> >> We build on our locals / servers using same flag. >> >> >> Usually, for releases we build with running all the tests, as well. and at >> some level of code coverage. >> >> Are we by-passing it ? >> >> >> >> Thanking you. >> >> With Regards >> Sree >> >> >> >> On Wednesday, April 15, 2015 3:32 PM, Sean McNamara >> wrote: >> >> >> Ran tests on OS X >> >> +1 >> >> Sean >> >> >>> On Apr 14, 2015, at 10:59 PM, Patrick Wendell wrote: >>> >>> I'd like to close this vote to coincide with the 1.3.1 release, >>> however, it would be great to have more people test this release >>> first. I'll leave it open for a bit longer and see if others can give >>> a +1. >>> >>> On Tue, Apr 14, 2015 at 9:55 PM, Patrick Wendell >>> wrote: >>>> +1 from me ass well. >>>> >>>> On Tue, Apr 7, 2015 at 4:36 AM, Sean Owen wrote: >>>>> I think that's close enough for a +1: >>>>> >>>>> Signatures and hashes are good. >>>>> LICENSE, NOTICE still check out. >>>>> Compiles for a Hadoop 2.6 + YARN + Hive profile. >>>>> >>>>> JIRAs with target version = 1.2.x look legitimate; no blockers. >>>>> >>>>> I still observe several Hive test failures with: >>>>> mvn -Phadoop-2.4 -Pyarn -Phive -Phive-0.13.1 -Dhadoop.version=2.6.0 >>>>> -DskipTests clean package; mvn -Phadoop-2.4 -Pyarn -Phive >>>>> -Phive-0.13.1 -Dhadoop.version=2.6.0 test >>>>> .. though again I think these are not regressions but known issues in >>>>> older branches. >>>>> >>>>> FYI there are 16 Critical issues still open for 1.2.x: >>>>> >>>>> SPARK-6209,ExecutorClassLoader can leak connections after failing to >>>>> load classes from the REPL class server,Josh Rosen,In Progress,4/5/15 >>>>> SPARK-5098,Number of running tasks become negative after tasks >>>>> lost,,Open,1/14/15 >>>>> SPARK-4888,"Spark EC2 doesn't mount local disks for i2.8xlarge >>>>> instances",,Open,1/27/15 >>>>> SPARK-4879,Missing output partitions after job completes with >>>>> speculative execution,Josh Rosen,Open,3/5/15 >>>>> SPARK-4568,Publish release candidates under $VERSION-RCX instead of >>>>> $VERSION,Patrick Wendell,Open,11/24/14 >>>>> SPARK-4520,SparkSQL exception when reading certain columns from a >>>>> parquet file,sadhan sood,Open,1/21/15 >>>>> SPARK-4514,SparkContext localProperties does not inherit property >>>>> updates across thread reuse,Josh Rosen,Open,3/31/15 >>&g
Re: [VOTE] Release Apache Spark 1.2.2
In my effort to vote for this release, I found these along: This is from jenkins. It uses "-DskipTests". [centos] $ /home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.0.5/bin/mvn -Dhadoop.version=2.0.0-mr1-cdh4.1.2 -Dlabel=centos -DskipTests clean packageWe build on our locals / servers using same flag. Usually, for releases we build with running all the tests, as well. and at some level of code coverage. Are we by-passing it ? Thanking you. With Regards Sree On Wednesday, April 15, 2015 3:32 PM, Sean McNamara wrote: Ran tests on OS X +1 Sean > On Apr 14, 2015, at 10:59 PM, Patrick Wendell wrote: > > I'd like to close this vote to coincide with the 1.3.1 release, > however, it would be great to have more people test this release > first. I'll leave it open for a bit longer and see if others can give > a +1. > > On Tue, Apr 14, 2015 at 9:55 PM, Patrick Wendell wrote: >> +1 from me ass well. >> >> On Tue, Apr 7, 2015 at 4:36 AM, Sean Owen wrote: >>> I think that's close enough for a +1: >>> >>> Signatures and hashes are good. >>> LICENSE, NOTICE still check out. >>> Compiles for a Hadoop 2.6 + YARN + Hive profile. >>> >>> JIRAs with target version = 1.2.x look legitimate; no blockers. >>> >>> I still observe several Hive test failures with: >>> mvn -Phadoop-2.4 -Pyarn -Phive -Phive-0.13.1 -Dhadoop.version=2.6.0 >>> -DskipTests clean package; mvn -Phadoop-2.4 -Pyarn -Phive >>> -Phive-0.13.1 -Dhadoop.version=2.6.0 test >>> .. though again I think these are not regressions but known issues in >>> older branches. >>> >>> FYI there are 16 Critical issues still open for 1.2.x: >>> >>> SPARK-6209,ExecutorClassLoader can leak connections after failing to >>> load classes from the REPL class server,Josh Rosen,In Progress,4/5/15 >>> SPARK-5098,Number of running tasks become negative after tasks >>> lost,,Open,1/14/15 >>> SPARK-4888,"Spark EC2 doesn't mount local disks for i2.8xlarge >>> instances",,Open,1/27/15 >>> SPARK-4879,Missing output partitions after job completes with >>> speculative execution,Josh Rosen,Open,3/5/15 >>> SPARK-4568,Publish release candidates under $VERSION-RCX instead of >>> $VERSION,Patrick Wendell,Open,11/24/14 >>> SPARK-4520,SparkSQL exception when reading certain columns from a >>> parquet file,sadhan sood,Open,1/21/15 >>> SPARK-4514,SparkContext localProperties does not inherit property >>> updates across thread reuse,Josh Rosen,Open,3/31/15 >>> SPARK-4454,Race condition in DAGScheduler,Josh Rosen,Reopened,2/18/15 >>> SPARK-4452,Shuffle data structures can starve others on the same >>> thread for memory,Tianshuo Deng,Open,1/24/15 >>> SPARK-4356,Test Scala 2.11 on Jenkins,Patrick Wendell,Open,11/12/14 >>> SPARK-4258,NPE with new Parquet Filters,Cheng Lian,Reopened,4/3/15 >>> SPARK-4194,Exceptions thrown during SparkContext or SparkEnv >>> construction might lead to resource leaks or corrupted global >>> state,,In Progress,4/2/15 >>> SPARK-4159,"Maven build doesn't run JUnit test suites",Sean >>> Owen,Open,1/11/15 >>> SPARK-4106,Shuffle write and spill to disk metrics are >>> incorrect,,Open,10/28/14 >>> SPARK-3492,Clean up Yarn integration code,Andrew Or,Open,9/12/14 >>> SPARK-3461,Support external groupByKey using >>> repartitionAndSortWithinPartitions,Sandy Ryza,Open,11/10/14 >>> SPARK-2984,FileNotFoundException on _temporary directory,,Open,12/11/14 >>> SPARK-2532,Fix issues with consolidated shuffle,,Open,3/26/15 >>> SPARK-1312,Batch should read based on the batch interval provided in >>> the StreamingContext,Tathagata Das,Open,12/24/14 >>> >>> On Sun, Apr 5, 2015 at 7:24 PM, Patrick Wendell wrote: Please vote on releasing the following candidate as Apache Spark version 1.2.2! The tag to be voted on is v1.2.2-rc1 (commit 7531b50): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=7531b50e406ee2e3301b009ceea7c684272b2e27 The list of fixes present in this release can be found at: http://bit.ly/1DCNddt The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-1.2.2-rc1/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-1082/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-1.2.2-rc1-docs/ Please vote on releasing this package as Apache Spark 1.2.2! The vote is open until Thursday, April 08, at 00:30 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.2.2 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/
Re: how long does it takes for full build ?
Found it, Ted. Thank you.https://amplab.cs.berkeley.edu/jenkins/job/Spark-1.2-Maven-pre-YARN/hadoop.version=2.0.0-mr1-cdh4.1.2,label=centos/354/consoleFull We locally build with "-DskipTests" and on our jenkins as well. Thanking you. With Regards Sree On Thursday, April 16, 2015 1:04 PM, Ted Yu wrote: You can find the command at the beginning of the console output: [centos] $ /home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.0.5/bin/mvn -DHADOOP_PROFILE=hadoop-2.4 -Dlabel=centos -DskipTests -Phadoop-2.4 -Pyarn -Phive clean package On Thu, Apr 16, 2015 at 12:42 PM, Sree V wrote: > 1. > 40 min+ to 1hr+, from jenkins. > I didn't find the commands of the job. Does it require a login ? > > Part of the console output: > > > git checkout -f 3ae37b93a7c299bd8b22a36248035bca5de3422f > > git rev-list de4fa6b6d12e2bee0307ffba2abfca0c33f15e45 # timeout=10 > Triggering Spark-Master-Maven-pre-YARN ? 2.0.0-mr1-cdh4.1.2,centos > <https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-pre-YARN/hadoop.version=2.0.0-mr1-cdh4.1.2,label=centos/> > Triggering Spark-Master-Maven-pre-YARN ? 1.0.4,centos > <https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-pre-YARN/hadoop.version=1.0.4,label=centos/> > > How to find the commands of these 'triggers' ? > I am interested, whether these named triggers use -DskipTests or not. > > 2. > This page, gives examples all with -DskipTests only. > http://spark.apache.org/docs/1.2.0/building-spark.html > > > 3. > For casting VOTE to release 1.2.2-rc1, > I am running 'mvn clean package' on spark 1.2.2-rc1 with oralce jdk8_40 on > centos7. > This is stuck at, from last night. i.e. almost 12 hours. > ... > ExternalSorterSuite: > - empty data stream > - few elements per partition > - empty partitions with spilling > - empty partitions with spilling, bypass merge-sort > > Any pointers ? > > Thanking you. > > With Regards > Sree > > > > On Thursday, April 16, 2015 12:01 PM, Ted Yu > wrote: > > > You can get some idea by looking at the builds here: > > > https://amplab.cs.berkeley.edu/jenkins/view/Spark/job/Spark-1.2-Maven-with-YARN/HADOOP_PROFILE=hadoop-2.4,label=centos/ > > Cheers > > On Thu, Apr 16, 2015 at 11:56 AM, Sree V > wrote: > > Hi Team, > How long does it takes for a full build 'mvn clean package' on spark > 1.2.2-rc1 ? > > > Thanking you. > > With Regards > Sree > > > > >
Re: how long does it takes for full build ?
+ ShaneHi Shane, Would you address 1. please ? Thanking you. With Regards Sree On Thursday, April 16, 2015 12:46 PM, Sree V wrote: 1. 40 min+ to 1hr+, from jenkins.I didn't find the commands of the job. Does it require a login ? Part of the console output: > git checkout -f 3ae37b93a7c299bd8b22a36248035bca5de3422f > git rev-list de4fa6b6d12e2bee0307ffba2abfca0c33f15e45 # timeout=10 Triggering Spark-Master-Maven-pre-YARN ? 2.0.0-mr1-cdh4.1.2,centos Triggering Spark-Master-Maven-pre-YARN ? 1.0.4,centosHow to find the commands of these 'triggers' ?I am interested, whether these named triggers use -DskipTests or not. 2. This page, gives examples all with -DskipTests only. http://spark.apache.org/docs/1.2.0/building-spark.html 3.For casting VOTE to release 1.2.2-rc1, I am running 'mvn clean package' on spark 1.2.2-rc1 with oralce jdk8_40 on centos7.This is stuck at, from last night. i.e. almost 12 hours. ...ExternalSorterSuite: - empty data stream - few elements per partition - empty partitions with spilling - empty partitions with spilling, bypass merge-sort Any pointers ? Thanking you. With Regards Sree On Thursday, April 16, 2015 12:01 PM, Ted Yu wrote: You can get some idea by looking at the builds here: https://amplab.cs.berkeley.edu/jenkins/view/Spark/job/Spark-1.2-Maven-with-YARN/HADOOP_PROFILE=hadoop-2.4,label=centos/ Cheers On Thu, Apr 16, 2015 at 11:56 AM, Sree V wrote: Hi Team, How long does it takes for a full build 'mvn clean package' on spark 1.2.2-rc1 ? Thanking you. With Regards Sree
Re: how long does it takes for full build ?
1. 40 min+ to 1hr+, from jenkins.I didn't find the commands of the job. Does it require a login ? Part of the console output: > git checkout -f 3ae37b93a7c299bd8b22a36248035bca5de3422f > git rev-list de4fa6b6d12e2bee0307ffba2abfca0c33f15e45 # timeout=10 Triggering Spark-Master-Maven-pre-YARN ? 2.0.0-mr1-cdh4.1.2,centos Triggering Spark-Master-Maven-pre-YARN ? 1.0.4,centosHow to find the commands of these 'triggers' ?I am interested, whether these named triggers use -DskipTests or not. 2. This page, gives examples all with -DskipTests only. http://spark.apache.org/docs/1.2.0/building-spark.html 3.For casting VOTE to release 1.2.2-rc1, I am running 'mvn clean package' on spark 1.2.2-rc1 with oralce jdk8_40 on centos7.This is stuck at, from last night. i.e. almost 12 hours. ...ExternalSorterSuite: - empty data stream - few elements per partition - empty partitions with spilling - empty partitions with spilling, bypass merge-sort Any pointers ? Thanking you. With Regards Sree On Thursday, April 16, 2015 12:01 PM, Ted Yu wrote: You can get some idea by looking at the builds here: https://amplab.cs.berkeley.edu/jenkins/view/Spark/job/Spark-1.2-Maven-with-YARN/HADOOP_PROFILE=hadoop-2.4,label=centos/ Cheers On Thu, Apr 16, 2015 at 11:56 AM, Sree V wrote: Hi Team, How long does it takes for a full build 'mvn clean package' on spark 1.2.2-rc1 ? Thanking you. With Regards Sree
how long does it takes for full build ?
Hi Team, How long does it takes for a full build 'mvn clean package' on spark 1.2.2-rc1 ? Thanking you. With Regards Sree
Re: start-slaves.sh uses local path from master on remote slave nodes
https://issues.apache.org/jira/browse/SPARK-967 Hi Team, The reporter hasn't replied for the suggested change for this issue. Also, there is a work around suggested. Change or Work-A-Round, so we can close this issue ? Thanking you. With Regards Sree
Re: Streamline contribution process with update to Contribution wiki, JIRA rules
Hi Sean, This is not the first time, I am hearing it. I agree with JIRA suggestion. In most of the companies that I worked, we have 'no status', 'no type', when a jira is created. And we set both, in sprint planning meetings. I am not sure, how easy it would be for apache jira. As any change might effect every apache project. Thanking you. With Regards Sree On Monday, April 13, 2015 4:20 PM, Nicholas Chammas wrote: Wow, I had an open email draft to whine (yet again) about our open PR count and provide some suggestions. Will redirect that to the JIRA Sean created. Sweet! Nick On Mon, Apr 13, 2015 at 7:05 PM Patrick Wendell wrote: > Would just like to encourage everyone who is active in day-to-day > development to give feedback on this (and I will do same). Sean has > spent a lot of time looking through different ways we can streamline > our dev process. > > - Patrick > > On Mon, Apr 13, 2015 at 3:59 PM, Sean Owen wrote: > > Pardon, I wanted to call attention to a JIRA I just created... > > > > https://issues.apache.org/jira/browse/SPARK-6889 > > > > ... in which I propose what I hope are some changes to the > > contribution process wiki that could help a bit with the flood of > > reviews and PRs. I'd be grateful for your thoughts and comments there, > > as it's my current pet issue. > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > > For additional commands, e-mail: dev-h...@spark.apache.org > > > > - > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > >
Re: [VOTE] Release Apache Spark 1.3.1 (RC3)
+1builds - checktests - checkinstalls and sample run - check Thanking you. With Regards Sree On Friday, April 10, 2015 11:07 PM, Patrick Wendell wrote: Please vote on releasing the following candidate as Apache Spark version 1.3.1! The tag to be voted on is v1.3.1-rc2 (commit 3e83913): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=3e8391327ba586eaf54447043bd526d919043a44 The list of fixes present in this release can be found at: http://bit.ly/1C2nVPY The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-1.3.1-rc3/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-1088/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-1.3.1-rc3-docs/ The patches on top of RC2 are: [SPARK-6851] [SQL] Create new instance for each converted parquet relation [SPARK-5969] [PySpark] Fix descending pyspark.rdd.sortByKey. [SPARK-6343] Doc driver-worker network reqs [SPARK-6767] [SQL] Fixed Query DSL error in spark sql Readme [SPARK-6781] [SQL] use sqlContext in python shell [SPARK-6753] Clone SparkConf in ShuffleSuite tests [SPARK-6506] [PySpark] Do not try to retrieve SPARK_HOME when not needed... Please vote on releasing this package as Apache Spark 1.3.1! The vote is open until Tuesday, April 14, at 07:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.3.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: SPARK-5364
Thank you, Raynold. Thanking you. With Regards Sree On Sunday, April 12, 2015 11:18 AM, Reynold Xin wrote: I closed it. Thanks. On Sun, Apr 12, 2015 at 11:08 AM, Sree V wrote: > Hi, > I was browsing through the JIRAs and found this can be closed.If anyone > who has edit permissions on Spark JIRA, please close this. > https://issues.apache.org/jira/browse/SPARK-5364 > It is OpenIts Pull Request already merged > Its parent and grand parent Resolved > > > Thanking you. > > With Regards > Sree Vaddi >
Re: SPARK-5364
Hi, I was browsing through the JIRAs and found this can be closed.If anyone who has edit permissions on Spark JIRA, please close this. https://issues.apache.org/jira/browse/SPARK-5364 It is OpenIts Pull Request already merged Its parent and grand parent Resolved Thanking you. With Regards Sree Vaddi
