[GitHub] [incubator-hudi] amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException
amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException URL: https://github.com/apache/incubator-hudi/issues/741#issuecomment-504209411 Hive sync is also complete and i can query the table as well.. Thanks! for your quick response. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-hudi] amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException
amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException URL: https://github.com/apache/incubator-hudi/issues/741#issuecomment-504180560 I am part of mailing list.I will ad to slack channel soon. It is working now.I can see the output files. I am working on hive sync now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-hudi] amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException
amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException URL: https://github.com/apache/incubator-hudi/issues/741#issuecomment-504160915 I am rebuilding it.Will update the status once it is complete. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-hudi] amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException
amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException URL: https://github.com/apache/incubator-hudi/issues/741#issuecomment-504147822 Please bear with me for too many questions..If it is cloudera distribution i shouldn't been asking these many question. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-hudi] amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException
amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException URL: https://github.com/apache/incubator-hudi/issues/741#issuecomment-504147148  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-hudi] amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException
amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException URL: https://github.com/apache/incubator-hudi/issues/741#issuecomment-504145084 Is it because of MapR Hadoop distribution? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-hudi] amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException
amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException URL: https://github.com/apache/incubator-hudi/issues/741#issuecomment-504144982 I am getting below error. Caused by: com.uber.hoodie.exception.HoodieException: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: BlockAlignedAvroParquetWriter does not support scheme maprfs 19/06/20 14:00:32 WARN Utils: Service 'SparkUI' could not bind on port 4240. Attempting port 4241. 19/06/20 14:00:32 WARN Utils: Service 'SparkUI' could not bind on port 4241. Attempting port 4242. 19/06/20 14:00:35 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME. ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. 19/06/20 14:01:03 WARN SparkContext: Using an existing SparkContext; some configuration may not take effect. 19/06/20 14:01:24 WARN TaskSetManager: Lost task 0.0 in stage 21.0 (TID 23, dbsls0327.uhc.com, executor 2): java.lang.RuntimeException: com.uber.hoodie.exception.HoodieException: com.uber.hoodie.exception.HoodieException: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: BlockAlignedAvroParquetWriter does not support scheme maprfs at com.uber.hoodie.func.LazyIterableIterator.next(LazyIterableIterator.java:123) at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43) at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) at org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:371) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1055) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1029) at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:969) at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1029) at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:760) at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334) at org.apache.spark.rdd.RDD.iterator(RDD.scala:285) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:108) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: com.uber.hoodie.exception.HoodieException: com.uber.hoodie.exception.HoodieException: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: BlockAlignedAvroParquetWriter does not support scheme maprfs at com.uber.hoodie.func.CopyOnWriteLazyInsertIterable.computeNext(CopyOnWriteLazyInsertIterable.java:106) at com.uber.hoodie.func.CopyOnWriteLazyInsertIterable.computeNext(CopyOnWriteLazyInsertIterable.java:43) at com.uber.hoodie.func.LazyIterableIterator.next(LazyIterableIterator.java:121) ... 20 more Caused by: com.uber.hoodie.exception.HoodieException: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: BlockAlignedAvroParquetWriter does not support scheme maprfs at com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:148) at com.uber.hoodie.func.CopyOnWriteLazyInsertIterable.computeNext(CopyOnWriteLazyInsertIterable.java:102) ... 22 more Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: BlockAlignedAvroParquetWriter does not support scheme maprfs at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:146) ... 23 more Caused by: java.lang.IllegalArgumentException: BlockAlignedAvroParquetWriter does not support scheme maprfs at com.uber.hoodie.common.io.storage.HoodieWrapperFileSystem.getHoodieScheme(HoodieWrapperFileSystem.java:110) at com.uber.hoodie.common.io.storage.HoodieWrapperFileSystem.convertToHoodiePath(HoodieWrapperFileSystem.java:86)
[GitHub] [incubator-hudi] amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException
amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException URL: https://github.com/apache/incubator-hudi/issues/741#issuecomment-504137883 Thanks! I will check and revert back... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-hudi] amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException
amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException URL: https://github.com/apache/incubator-hudi/issues/741#issuecomment-504123629  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-hudi] amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException
amaranathv commented on issue #741: java.lang.NoClassDefFoundError:
com/uber/hoodie/exception/HoodieException
URL: https://github.com/apache/incubator-hudi/issues/741#issuecomment-504122108
$ cat 20190618132401.commit
{
"partitionToWriteStats" : {
"2018/08/31" : [ {
"fileId" : "86f06040-14e7-4c0c-989b-72b2b59fe119-0",
"path" :
"file:/mapr/user/avenka23/delta-streamer/output7/2018/08/31/86f06040-14e7-4c0c-989b-72b2b59fe119-0_0-20-21_20190618132401.parquet",
"prevCommit" : "null",
"numWrites" : 197,
"numDeletes" : 0,
"numUpdateWrites" : 0,
"numInserts" : 197,
"totalWriteBytes" : 442914,
"totalWriteErrors" : 0,
"tempPath" : null,
"partitionPath" : "2018/08/31",
"totalLogRecords" : 0,
"totalLogFilesCompacted" : 0,
"totalLogSizeCompacted" : 0,
"totalUpdatedRecordsCompacted" : 0,
"totalLogBlocks" : 0,
"totalCorruptLogBlock" : 0,
"totalRollbackBlocks" : 0,
"fileSizeInBytes" : 442914
} ]
},
"compacted" : false,
"extraMetadataMap" : {
"ROLLING_STAT" : "{\n \"partitionToRollingStats\" : {\n
\"2018/08/31\" : {\n \"86f06040-14e7-4c0c-989b-72b2b59fe119-0\" : {\n
\"fileId\" : \"86f06040-14e7-4c0c-989b-72b2b59fe119-0\",\n\"inserts\"
: 197,\n\"upserts\" : 0,\n\"deletes\" : 0,\n
\"totalInputWriteBytesToDisk\" : 0,\n\"totalInputWriteBytesOnDisk\" :
442914\n }\n}\n },\n \"actionType\" : \"commit\"\n}",
"deltastreamer.checkpoint.key" : "1560610046000"
},
"extraMetadata" : {
"ROLLING_STAT" : "{\n \"partitionToRollingStats\" : {\n
\"2018/08/31\" : {\n \"86f06040-14e7-4c0c-989b-72b2b59fe119-0\" : {\n
\"fileId\" : \"86f06040-14e7-4c0c-989b-72b2b59fe119-0\",\n\"inserts\"
: 197,\n\"upserts\" : 0,\n\"deletes\" : 0,\n
\"totalInputWriteBytesToDisk\" : 0,\n\"totalInputWriteBytesOnDisk\" :
442914\n }\n}\n },\n \"actionType\" : \"commit\"\n}",
"deltastreamer.checkpoint.key" : "1560610046000"
},
"totalScanTime" : 0,
"totalCreateTime" : 446,
"totalUpsertTime" : 0,
"totalCompactedRecordsUpdated" : 0,
"totalLogFilesCompacted" : 0,
"totalLogFilesSize" : 0,
"fileIdAndRelativePaths" : {
"86f06040-14e7-4c0c-989b-72b2b59fe119-0" :
"file:/mapr/user/avenka23/delta-streamer/output7/2018/08/31/86f06040-14e7-4c0c-989b-72b2b59fe119-0_0-20-21_20190618132401.parquet"
},
"totalRecordsDeleted" : 0,
"totalLogRecordsCompacted" : 0
}avenka23@dbsld0031:/mapr/user/avenka23/delta-streamer/output7/.hoodie
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException
amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException URL: https://github.com/apache/incubator-hudi/issues/741#issuecomment-504121832 sorry..i have .hoodie This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-hudi] amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException
amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException URL: https://github.com/apache/incubator-hudi/issues/741#issuecomment-504121478 No This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-hudi] amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException
amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException URL: https://github.com/apache/incubator-hudi/issues/741#issuecomment-504120148 avenka23@dbsld0031:/mapr/user/avenka23/delta-streamer/output8/2018/08/31 $ ls -al total 2 drwxr-x---. 2 avenka23 dce 2 Jun 19 16:42 . drwxr-x---. 3 avenka23 dce 1 Jun 19 16:42 .. -rw-r--r--. 1 avenka23 dce 93 Jun 19 16:42 .hoodie_partition_metadata -rw-r--r--. 1 avenka23 dce 12 Jun 19 16:42 ..hoodie_partition_metadata.crc I am seeing only these files in the output path. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-hudi] amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException
amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException URL: https://github.com/apache/incubator-hudi/issues/741#issuecomment-503756975 it is printing what ever i had attached in previous reply. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-hudi] amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException
amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException URL: https://github.com/apache/incubator-hudi/issues/741#issuecomment-503756796 Spark status shows as success but i didn't see any output files in the target directory and also i didn't see any logs that shows sample records and other details . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-hudi] amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException
amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException URL: https://github.com/apache/incubator-hudi/issues/741#issuecomment-503678997 Can you please help on this to complete the POC. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-hudi] amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException
amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException URL: https://github.com/apache/incubator-hudi/issues/741#issuecomment-503164955 I am getting invalid data path now. InValid data paths=[file:///mapr/user/avenka23/delta-streamer/output5/2018/08/31/fe957979-67f8-4b68-a371-6c789e81339c-0_0-21-23_20190618093523.parquet] Is there any property needs to be set or environment variables? I am trying to mimic the docker demo using deltastreamer command line to ingest data from json into hoodie data set. $ spark-submit --master yarn --class com.uber.hoodie.utilities.deltastreamer.HoodieDeltaStreamer `ls /mapr/user/avenka23/hoodie/incubator-hudi/packaging/hoodie-utilities-bundle/target/hoodie-utilities-bundle*-SNAPSHOT.jar` --props file:///mapr/user/avenka23/delta-streamer/config/dfs-source.properties --schemaprovider-class com.uber.hoodie.utilities.schema.FilebasedSchemaProvider --source-class com.uber.hoodie.utilities.sources.JsonDFSSource --source-ordering-field ts --target-base-path file:///mapr/user/avenka23/delta-streamer/output5 --target-table stock_ticks_cow --storage-type COPY_ON_WRITE 19/06/18 09:35:00 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME. ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. 19/06/18 09:35:19 WARN SparkContext: Using an existing SparkContext; some configuration may not take effect. 19/06/18 09:35:28 WARN HoodieTable: InValid data paths=[file:///mapr/user/avenka23/delta-streamer/output5/2018/08/31/fe957979-67f8-4b68-a371-6c789e81339c-0_0-21-23_20190618093523.parquet] This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-hudi] amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException
amaranathv commented on issue #741: java.lang.NoClassDefFoundError: com/uber/hoodie/exception/HoodieException URL: https://github.com/apache/incubator-hudi/issues/741#issuecomment-502815062 Can you please help me on this one? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
