Re: Java : Testing RDD aggregateByKey

Same here, repartition(0) throws IllegalArgument (What I would have expected for ) , but aggregateByKey(zeroValue, 0, seqFunc, combFunc) is not throwing any exception nor logging any error message. The only consequence is an empty RDD. El sáb, 21 de ago. de 2021 a la(s) 07:45, Jacek Laskowski

Re: Java : Testing RDD aggregateByKey

Hi Pedro, > Anyway, maybe the behavior is weird, I could expect that repartition to zero was not allowed or at least warned instead of just discarting all the data . Interesting... scala> spark.version res3: String = 3.1.2 scala> spark.range(5).repartition(0)

Re: Java : Testing RDD aggregateByKey

Hi, I'm sorry , the problem was really silly: In the test the number of partitions were zero (it was a division of the original number of partitions of the RDD source and in the test that number was just one) and that's why the test was failing. Anyway, maybe the behavior is weird, I could expect

Re: Java : Testing RDD aggregateByKey

Hi Pedro, No idea what might be causing it. Do you perhaps have some code to reproduce it locally? Pozdrawiam, Jacek Laskowski https://about.me/JacekLaskowski "The Internals Of" Online Books Follow me on https://twitter.com/jaceklaskowski

Java : Testing RDD aggregateByKey

Context: spark-core_2.12-3.1.1 Testing with maven and eclipse. I'm modifying a project and a test stops working as expected. The difference is in the parameters passed to the function aggregateByKey of JavaPairRDD. JavaSparkContext is created this way: new JavaSparkContext(new SparkConf()