Hi all:
I used spark to read a parquet file and than save it as carbondata file,
but I found that *my program didn't stop* after loading data successfully, I
saw log output as following:
*main -[MyUbuntu-64][myubuntu][Thread-1]Total time taken to write
dictionary file is: 40728
main -[MyUbuntu-64][myubuntu][Thread-1]Data load is successful for
default.parquet_to_carbondata2
main -main compaction need status is false
main -main Successfully deleted the lock file
/data/carbon_data/default/parquet_to_carbondata2/meta.lock*
According to the log above, I think that the data load was successful, but
why the programe hanged?
My program is :
/val warehouseLocation = Constants.SPARK_WAREHOUSE
val storeLocation = Constants.CARBON_FILES
CarbonProperties.getInstance()
.addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
"yyyy-MM-dd HH:mm:ss")
.addProperty(CarbonCommonConstants.CARBON_DATE_FORMAT, "yyyy-MM-dd")
import org.apache.spark.sql.CarbonSession._
val spark = SparkSession
.builder()
.appName("TestCarbonData")
.master("local[2]")
.config("spark.sql.warehouse.dir", warehouseLocation)
.getOrCreateCarbonSession(storeLocation, Constants.METASTORE_DB)
spark.conf.set("spark.sql.parquet.binaryAsString", true)
val testParquet2 = spark.read.parquet("file:///data/cbd_test.parquet")
spark.sql("""DROP TABLE IF EXISTS parquet_to_carbondata2""")
testParquet2.show()
testParquet2.printSchema()
println("testParquet2: " + testParquet2.count())
testParquet2.write
.format("carbondata")
.option("tableName", "parquet_to_carbondata2")
.option("tempCSV", "false")
.option("compress", "true")
.option("single_pass", "true")
.option("partitionCount", "2")
.option("table_blocksize", "8")
.option("dictionary_exclude", "ftype,chan")
.mode(SaveMode.Overwrite)
.save()
spark.stop()/
*I use command 'jstack' to trace this program, the messages are as
following:*
"Attach Listener" #438 daemon prio=9 os_prio=0 tid=0x00007f91e80a3800
nid=0x1c44 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"DestroyJavaVM" #435 prio=5 os_prio=0 tid=0x00007f9240015000 nid=0x191e
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"nioEventLoopGroup-11-1" #96 prio=10 os_prio=0 tid=0x00007f920009d000
nid=0x19a3 runnable [0x00007f9160df9000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000067a853b30> (a
io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x000000065fa85800> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006602e3ab8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:746)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:391)
at
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"threadDeathWatcher-9-1" #91 daemon prio=1 os_prio=0 tid=0x00007f919c0d4000
nid=0x199e sleeping[0x00007f9168389000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152)
at
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"ForkJoinPool-2-worker-11" #65 daemon prio=5 os_prio=0
tid=0x00007f919c001800 nid=0x1979 waiting on condition [0x00007f916b3ef000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000067aa126b8> (a
scala.concurrent.forkjoin.ForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Locked ownable synchronizers:
- None
"BoneCP-pool-watch-thread" #54 daemon prio=5 os_prio=0
tid=0x00007f92427d2800 nid=0x1966 waiting on condition [0x00007f91bc738000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000065fa82cf8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x000000067a87f9d0> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"BoneCP-keep-alive-scheduler" #53 daemon prio=5 os_prio=0
tid=0x00007f92427a8800 nid=0x1965 waiting on condition [0x00007f91bc839000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000067ac1da68> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"com.google.common.base.internal.Finalizer" #52 daemon prio=5 os_prio=0
tid=0x00007f92427a7000 nid=0x1964 in Object.wait() [0x00007f91bc93a000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000065fa82ce0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x000000065fa82ce0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at com.google.common.base.internal.Finalizer.run(Finalizer.java:127)
Locked ownable synchronizers:
- None
"BoneCP-pool-watch-thread" #51 daemon prio=5 os_prio=0
tid=0x00007f9242518800 nid=0x1963 waiting on condition [0x00007f91bcc3b000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000065f801780> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x000000066020ed18> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"BoneCP-keep-alive-scheduler" #50 daemon prio=5 os_prio=0
tid=0x00007f924251b800 nid=0x1962 waiting on condition [0x00007f91bcd3c000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000065f7973b0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"com.google.common.base.internal.Finalizer" #49 daemon prio=5 os_prio=0
tid=0x00007f92410bd000 nid=0x1961 in Object.wait() [0x00007f91bce3d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000065f8017b0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x000000065f8017b0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at com.google.common.base.internal.Finalizer.run(Finalizer.java:127)
Locked ownable synchronizers:
- None
"derby.rawStoreDaemon" #48 daemon prio=5 os_prio=0 tid=0x00007f9242331800
nid=0x1960 in Object.wait() [0x00007f91bd13e000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.derby.impl.services.daemon.BasicDaemon.rest(Unknown
Source)
- locked <0x000000066043f498> (a
org.apache.derby.impl.services.daemon.BasicDaemon)
at org.apache.derby.impl.services.daemon.BasicDaemon.run(Unknown Source)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Timer-1" #47 daemon prio=5 os_prio=0 tid=0x00007f9242314800 nid=0x195f in
Object.wait() [0x00007f91bd43f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000065f768008> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked <0x000000065f768008> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers:
- None
"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f9240161000
nid=0x1934 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f9240144000
nid=0x1933 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f9240141800
nid=0x1932 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f924013f800
nid=0x1931 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f924013d800
nid=0x1930 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
tid=0x00007f924013c000 nid=0x192f waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f9240104800 nid=0x192e in
Object.wait() [0x00007f9219424000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x000000065f3ff8d0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
Locked ownable synchronizers:
- None
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f9240102000
nid=0x192d in Object.wait() [0x00007f9219525000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
- locked <0x000000065f3d4538> (a java.lang.ref.Reference$Lock)
Locked ownable synchronizers:
- None
"VM Thread" os_prio=0 tid=0x00007f92400fd000 nid=0x192c runnable
"Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f9240026800
nid=0x191f runnable
"Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f9240028000
nid=0x1920 runnable
"Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f924002a000
nid=0x1921 runnable
"Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f924002b800
nid=0x1922 runnable
"Gang worker#4 (Parallel GC Threads)" os_prio=0 tid=0x00007f924002d800
nid=0x1923 runnable
"Gang worker#5 (Parallel GC Threads)" os_prio=0 tid=0x00007f924002f000
nid=0x1924 runnable
"Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f924008b000
nid=0x1927 runnable
"Gang worker#0 (Parallel CMS Threads)" os_prio=0 tid=0x00007f9240087000
nid=0x1925 runnable
"Gang worker#1 (Parallel CMS Threads)" os_prio=0 tid=0x00007f9240088800
nid=0x1926 runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007f9240164800 nid=0x1935
waiting on condition
JNI global references: 341
--
View this message in context:
http://apache-carbondata-mailing-list-archive.1130556.n5.nabble.com/Program-didn-t-stop-after-loading-successfully-tp12001.html
Sent from the Apache CarbonData Mailing List archive mailing list archive at
Nabble.com.
