BTW, I am using a BucketingSink and a DateTimeBucketer. Do I need to set any
other property to move the files from .pending state.
BucketingSink<String> sink = new
BucketingSink<String>("hdfs://localhost:8020/flinktwitter/");sink.setBucketer(new
DateTimeBucketer<String>("yyyy-MM-dd--HHmm"));
On Friday, September 1, 2017, 5:03:46 PM PDT, Krishnanand Khambadkone
<kkhambadk...@yahoo.com> wrote:
This message is eligible for Automatic Cleanup! (kkhambadk...@yahoo.com) Add
cleanup rule | More info
Hi, I have written a small program that uses a Twitter input stream and a
HDFS output sink. When the files are written to HDFS each part file in the
directory has a .pending extension. I am able to cat the file and see the
tweet text. Is this normal for the part files to have .pending extension.
-rw-r--r-- 3 user supergroup 46399 2017-09-01 16:35
/flinktwitter/2017-09-01--1635/_part-0-95.pending
-rw-r--r-- 3 user supergroup 54861 2017-09-01 16:35
/flinktwitter/2017-09-01--1635/_part-0-96.pending
-rw-r--r-- 3 user supergroup 41878 2017-09-01 16:35
/flinktwitter/2017-09-01--1635/_part-0-97.pending
-rw-r--r-- 3 user supergroup 42813 2017-09-01 16:35
/flinktwitter/2017-09-01--1635/_part-0-98.pending
-rw-r--r-- 3 user supergroup 42887 2017-09-01 16:35
/flinktwitter/2017-09-01--1635/_part-0-99.pending
