Ling Jinjiang created FLUME-3059: ------------------------------------ Summary: Hive sink failed in hive 2.1.0 Key: FLUME-3059 URL: https://issues.apache.org/jira/browse/FLUME-3059 Project: Flume Issue Type: Bug Components: Sinks+Sources Affects Versions: v1.7.0 Reporter: Ling Jinjiang Priority: Minor
Hive sink failed in hive 2.1.0 As I want to use hive sink to load data into hive 2.1.0, then I configure it with the example in flume user guide. But something wrong happened, the error message is here: 2017-02-23 18:49:09,079 ERROR org.apache.flume.SinkRunner: Unable to deliver event. Exception follows. org.apache.flume.EventDeliveryException: org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri='thrift://127.0.0.1:9083', database='logsdb', table='weblogs', partitionVals=[asia, test, 17-02-23-18-40] } at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:268) at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri='thrift://127.0.0.1:9083', database='logsdb', table='weblogs', partitionVals=[asia, test, 17-02-23-18-40] } at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:99) at org.apache.flume.sink.hive.HiveSink.getOrCreateWriter(HiveSink.java:344) at org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:296) at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:254) ... 3 more Caused by: org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri='thrift://127.0.0.1:9083', database='logsdb', table='weblogs', partitionVals=[asia, test, 17-02-23-18-40] } at org.apache.flume.sink.hive.HiveWriter.newConnection(HiveWriter.java:380) at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:86) ... 6 more Caused by: org.apache.hive.hcatalog.streaming.InvalidTable: Invalid table db:logsdb, table:weblogs: 'transactional' property is not set on Table at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.checkEndPoint(HiveEndPoint.java:340) at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:312) at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:278) at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnectionImpl(HiveEndPoint.java:215) at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnection(HiveEndPoint.java:192) at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnection(HiveEndPoint.java:105) at org.apache.flume.sink.hive.HiveWriter$8.call(HiveWriter.java:376) at org.apache.flume.sink.hive.HiveWriter$8.call(HiveWriter.java:373) at org.apache.flume.sink.hive.HiveWriter$11.call(HiveWriter.java:425) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) It's becasue that https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-TableProperties and https://issues.apache.org/jira/browse/HIVE-11716. So since HIVE-11716 the "transactional" must be "true" when creating table like this: create table weblogs ( id int , msg string ) partitioned by (continent string, country string, time string) clustered by (id) into 5 buckets stored as orc TBLPROPERTIES ("transactional"="true"); And some configuration must be setted in hive to support transaction. But the example in flume user guide of hive sink does not mention these. I think it's nessecory to specify the hive transaction configuration in the guide; -- This message was sent by Atlassian JIRA (v6.3.15#6346)