prashanthpdesai opened a new issue #1653: URL: https://github.com/apache/incubator-hudi/issues/1653
- Have you gone through our [FAQs](https://cwiki.apache.org/confluence/display/HUDI/FAQ)? Yes - Join the mailing list to engage in conversations and get faster support at dev-subscr...@hudi.apache.org. - If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly. **Describe the problem you faced** org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of range with no configured reset policy for partitions: {enriched-output-changelog-0=0} A clear and concise description of the problem. Running Deltastreamer to consume from partitioned(12) compacted topic and persisting the data in MapR Platform , while consuming we are getting this exception of Offset being out of range and our Pod is being terminated . we did try running in continuous mode with MOR storage type initially , did face the offset exception and we could able to achieve it running by cleaning up the Kafka topic and restarted our pod . Our requirement slightly changed so we don't need to run it continuous , so scheduling mini batch (every 2 hours) so facing same offset exception now. Downstream consumers doesn't require to consume real time so switched to COPY_ON_WRITE and non continuous mode with checkpoint in place to run in mini batch. **To Reproduce** Steps to reproduce the behavior: 1. 2. 3. 4. **Expected behavior** the Files need to be cut equally more likely default size(128MB) and persist into our HDFS location. **Environment Description** * Hudi version : 0.5.2 * Spark version : 2.2.1 * Hive version : * Hadoop version : maps 6.01 and Hadoop 2.7 * Storage (HDFS/S3/GCS..) : HDFS * Running on Docker? (yes/no) : yes **Additional context** Add any other context about the problem here. **Stacktrace** ```Add the stacktrace of the error.``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
