Fokko commented on pull request #1972:
URL: https://github.com/apache/iceberg/pull/1972#issuecomment-751397938
Able to read the results with Spark:
```
MacBook-Pro-van-Fokko:incubator-iceberg fokkodriesprong$ spark-sql
--packages org.apache.iceberg:iceberg-spark3:0.10.0 --conf
"spark.sql.catalog.hive_prod=org.apache.iceberg.spark.SparkCatalog" --conf
"spark.sql.catalog.hive_prod.type=hive" --conf
"spark.sql.catalog.hive_prod.uri=thrift://localhost:9083"
spark-sql> SELECT * FROM hive_prod.default.test;
20/12/26 22:07:46 INFO BaseMetastoreTableOperations: Refreshing table
metadata from new version:
file:/tmp/test/metadata/00004-ed964aeb-8924-492b-8225-eb2102c22f03.metadata.json
20/12/26 22:07:46 INFO BaseMetastoreCatalog: Table loaded by catalog:
hive_prod.default.test
20/12/26 22:07:47 INFO MemoryStore: Block broadcast_0 stored as values in
memory (estimated size 315.8 KiB, free 366.0 MiB)
20/12/26 22:07:48 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes
in memory (estimated size 27.4 KiB, free 366.0 MiB)
20/12/26 22:07:48 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory
on 192.168.1.113:56438 (size: 27.4 KiB, free: 366.3 MiB)
20/12/26 22:07:48 INFO SparkContext: Created broadcast 0 from broadcast at
SparkScanBuilder.java:138
20/12/26 22:07:48 INFO MemoryStore: Block broadcast_1 stored as values in
memory (estimated size 40.0 B, free 366.0 MiB)
20/12/26 22:07:48 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes
in memory (estimated size 116.0 B, free 366.0 MiB)
20/12/26 22:07:48 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory
on 192.168.1.113:56438 (size: 116.0 B, free: 366.3 MiB)
20/12/26 22:07:48 INFO SparkContext: Created broadcast 1 from broadcast at
SparkScanBuilder.java:139
20/12/26 22:07:48 INFO V2ScanRelationPushDown:
Pushing operators to hive_prod.default.test
Pushed Filters:
Post-Scan Filters:
Output: word#0
20/12/26 22:07:48 INFO TableScan: Scanning table hive_prod.default.test
snapshot 4784653135298349855 created at 2020-12-26 22:07:38.530 with filter true
20/12/26 22:07:49 INFO CodeGenerator: Code generated in 226.493784 ms
20/12/26 22:07:49 INFO SparkContext: Starting job: main at
NativeMethodAccessorImpl.java:0
20/12/26 22:07:49 INFO DAGScheduler: Got job 0 (main at
NativeMethodAccessorImpl.java:0) with 1 output partitions
20/12/26 22:07:49 INFO DAGScheduler: Final stage: ResultStage 0 (main at
NativeMethodAccessorImpl.java:0)
20/12/26 22:07:49 INFO DAGScheduler: Parents of final stage: List()
20/12/26 22:07:49 INFO DAGScheduler: Missing parents: List()
20/12/26 22:07:49 INFO DAGScheduler: Submitting ResultStage 0
(MapPartitionsRDD[3] at main at NativeMethodAccessorImpl.java:0), which has no
missing parents
20/12/26 22:07:49 INFO MemoryStore: Block broadcast_2 stored as values in
memory (estimated size 7.1 KiB, free 366.0 MiB)
20/12/26 22:07:49 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes
in memory (estimated size 3.5 KiB, free 366.0 MiB)
20/12/26 22:07:49 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory
on 192.168.1.113:56438 (size: 3.5 KiB, free: 366.3 MiB)
20/12/26 22:07:49 INFO SparkContext: Created broadcast 2 from broadcast at
DAGScheduler.scala:1223
20/12/26 22:07:49 INFO DAGScheduler: Submitting 1 missing tasks from
ResultStage 0 (MapPartitionsRDD[3] at main at NativeMethodAccessorImpl.java:0)
(first 15 tasks are for partitions Vector(0))
20/12/26 22:07:49 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
20/12/26 22:07:49 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID
0, 192.168.1.113, executor driver, partition 0, PROCESS_LOCAL, 11598 bytes)
20/12/26 22:07:49 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
...
20/12/26 22:07:50 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0).
1580 bytes result sent to driver
20/12/26 22:07:50 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID
0) in 1132 ms on 192.168.1.113 (executor driver) (1/1)
20/12/26 22:07:50 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks
have all completed, from pool
20/12/26 22:07:50 INFO DAGScheduler: ResultStage 0 (main at
NativeMethodAccessorImpl.java:0) finished in 1,235 s
20/12/26 22:07:50 INFO DAGScheduler: Job 0 is finished. Cancelling potential
speculative or zombie tasks for this job
20/12/26 22:07:50 INFO TaskSchedulerImpl: Killing all running tasks in stage
0: Stage finished
20/12/26 22:07:50 INFO DAGScheduler: Job 0 finished: main at
NativeMethodAccessorImpl.java:0, took 1,274538 s
hands
yes
my
go
are
typing
words
Time taken: 4.501 seconds, Fetched 7 row(s)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
