ashishmgofficial edited a comment on issue #2149: URL: https://github.com/apache/hudi/issues/2149#issuecomment-706110206
@bvaradar Thanks for noticing it. I think that solved the previous error but producing following error now : ``` 20/10/09 10:32:09 INFO AppInfoParser: Kafka version : 2.0.0 20/10/09 10:32:09 INFO AppInfoParser: Kafka commitId : 3402a8361b734732 20/10/09 10:32:09 INFO InternalKafkaConsumer: Initial fetch for spark-executor-null airflow.public.motor_crash_violation_incidents-0 0 20/10/09 10:32:09 INFO Metadata: Cluster ID: cA3sXVaIR-qlM1MPNNYnCw 20/10/09 10:32:09 ERROR Executor: Exception in task 0.3 in stage 2.0 (TID 5) java.lang.RuntimeException: Error while decoding: java.lang.NegativeArraySizeException createexternalrow(input[0, bigint, true], input[1, string, false].toString, input[2, int, false], input[3, int, true], input[4, string, true].toString, input[5, string, true].toString, input[6, int, true], input[7, string, true].toString, input[8, bigint, false], StructField(_ts_ms,LongType,true), StructField(_op,StringType,false), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)) at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.fromRow(ExpressionEncoder.scala:305) at org.apache.hudi.AvroConversionUtils$$anonfun$1.apply(AvroConversionUtils.scala:44) at org.apache.hudi.AvroConversionUtils$$anonfun$1.apply(AvroConversionUtils.scala:44) at scala.collection.Iterator$$anon$11.next(Iterator.scala:410) at scala.collection.Iterator$$anon$11.next(Iterator.scala:410) at scala.collection.Iterator$$anon$10.next(Iterator.scala:394) at scala.collection.Iterator$class.foreach(Iterator.scala:891) at scala.collection.AbstractIterator.foreach(Iterator.scala:1334) at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48) at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310) at scala.collection.AbstractIterator.to(Iterator.scala:1334) at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302) at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1334) at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289) at scala.collection.AbstractIterator.toArray(Iterator.scala:1334) at org.apache.spark.rdd.RDD$$anonfun$take$1$$anonfun$31.apply(RDD.scala:1409) at org.apache.spark.rdd.RDD$$anonfun$take$1$$anonfun$31.apply(RDD.scala:1409) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:123) at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1405) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NegativeArraySizeException at org.apache.spark.unsafe.types.UTF8String.getBytes(UTF8String.java:297) at org.apache.spark.unsafe.types.UTF8String.toString(UTF8String.java:1226) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificSafeProjection.createExternalRow_0_0$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificSafeProjection.apply(Unknown Source) at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.fromRow(ExpressionEncoder.scala:302) ... 28 more ``` I think this error earlier also in the same thread when i was trying to add _hoodie_is_deleted field ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org