Hi all, I ran into carbondata and started trying it out. I am following this page https://carbondata.apache.org/quick-start-guide.html.
So far I have downloaded apache-carbondata-2.3.0-bin-spark3.1.1-hadoop2.7.2.jar. I am trying to use it with Spark 3.1.3 (I imagine the micro version difference should not matter). So far I have tried this with OpenJDK 8 and OpenJDK 11 and in both instances, create table works however when I attempt to load data from CSV it fails with the exception below. This is a different csv from what is called out on that quick-start page. Here is how I am launching Carbondata *spark-shell --conf spark.sql.extensions=org.apache.spark.sql.CarbonExtensions --jars <location to the above jar>* I am getting failures on both OpenJDK 11 and OpenJDK 8. Open JDK 8 fails with a segfault. I am running on a MacBook pro. The OpenJDK 11 errors seem to indicate that is not supported but JDK 8 seems to crash and exit. Please see the errors below, can someone point out what I am doing wrong? Error on OpenJDK 8 when I try to load data from a CSV file ================================================== # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x0000000104cbd7bb, pid=17765, tid=0x000000000000bd03 # # JRE version: OpenJDK Runtime Environment (8.0_282) (build 1.8.0_282-bre_2021_01_20_16_06-b00) # Java VM: OpenJDK 64-Bit Server VM (25.282-b00 mixed mode bsd-amd64 compressed oops) # Problematic frame: # V [libjvm.dylib+0x5667bb] Error on OpenJDK 11 when I try to load data from a CSV file ================================================== 22/04/17 23:57:06 ERROR CarbonFactDataHandlerColumnar: Error in producer java.lang.reflect.InaccessibleObjectException: Unable to make public void jdk.internal.ref.Cleaner.clean() accessible: module java.base does not "exports jdk.internal.ref" to unnamed module @5115e1e6 at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:340) at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:280) at java.base/java.lang.reflect.Method.checkCanSetAccessible(Method.java:198) at java.base/java.lang.reflect.Method.setAccessible(Method.java:192) at org.apache.carbondata.core.memory.UnsafeMemoryManager.destroyDirectByteBuffer(UnsafeMemoryManager.java:232) at org.apache.carbondata.core.datastore.page.LVByteBufferColumnPage.ensureMemory(LVByteBufferColumnPage.java:125) at org.apache.carbondata.core.datastore.page.LVByteBufferColumnPage.putBytes(LVByteBufferColumnPage.java:97) at org.apache.carbondata.core.datastore.page.LocalDictColumnPage.putBytes(LocalDictColumnPage.java:139) at org.apache.carbondata.core.datastore.page.ColumnPage.putData(ColumnPage.java:413) at org.apache.carbondata.processing.store.TablePage.convertToColumnarAndAddToPages(TablePage.java:241) at org.apache.carbondata.processing.store.TablePage.addRow(TablePage.java:201) at org.apache.carbondata.processing.store.CarbonFactDataHandlerColumnar.processDataRows(CarbonFactDataHandlerColumnar.java:397) at org.apache.carbondata.processing.store.CarbonFactDataHandlerColumnar.access$500(CarbonFactDataHandlerColumnar.java:60) at org.apache.carbondata.processing.store.CarbonFactDataHandlerColumnar$Producer.call(CarbonFactDataHandlerColumnar.java:637) at org.apache.carbondata.processing.store.CarbonFactDataHandlerColumnar$Producer.call(CarbonFactDataHandlerColumnar.java:614) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Error on OpenJDK 11 when I try to insert a single record ================================================== 22/04/18 09:46:10 ERROR CarbonFactDataHandlerColumnar: Error in producer java.lang.reflect.InaccessibleObjectException: Unable to make public void jdk.internal.ref.Cleaner.clean() accessible: module java.base does not "exports jdk.internal.ref" to unnamed module @cc4787f at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:340) at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:280) at java.base/java.lang.reflect.Method.checkCanSetAccessible(Method.java:198) at java.base/java.lang.reflect.Method.setAccessible(Method.java:192) at org.apache.carbondata.core.memory.UnsafeMemoryManager.destroyDirectByteBuffer(UnsafeMemoryManager.java:232) at org.apache.carbondata.core.datastore.page.LVByteBufferColumnPage.ensureMemory(LVByteBufferColumnPage.java:125) at org.apache.carbondata.core.datastore.page.LVByteBufferColumnPage.putBytes(LVByteBufferColumnPage.java:97) at org.apache.carbondata.core.datastore.page.LocalDictColumnPage.putBytes(LocalDictColumnPage.java:139) at org.apache.carbondata.core.datastore.page.ColumnPage.putData(ColumnPage.java:413) at org.apache.carbondata.processing.store.TablePage.convertToColumnarAndAddToPages(TablePage.java:241) at org.apache.carbondata.processing.store.TablePage.addRow(TablePage.java:201) at org.apache.carbondata.processing.store.CarbonFactDataHandlerColumnar.processDataRows(CarbonFactDataHandlerColumnar.java:397) at org.apache.carbondata.processing.store.CarbonFactDataHandlerColumnar.access$500(CarbonFactDataHandlerColumnar.java:60) at org.apache.carbondata.processing.store.CarbonFactDataHandlerColumnar$Producer.call(CarbonFactDataHandlerColumnar.java:637) at org.apache.carbondata.processing.store.CarbonFactDataHandlerColumnar$Producer.call(CarbonFactDataHandlerColumnar.java:614) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) 22/04/18 09:46:10 ERROR CarbonFactDataHandlerColumnar: Problem while writing the carbon data file Thanks, --Sandeep
