Saket created FLUME-3066:
----------------------------
Summary: Twitter Streaming using flume in hdfs
Key: FLUME-3066
URL: https://issues.apache.org/jira/browse/FLUME-3066
Project: Flume
Issue Type: Bug
Affects Versions: v1.7.0
Reporter: Saket
I was trying to stream twitter data in HDFS using flume.Succeffuly configured
agent but the data population in HDFS doesnt seems proper please help.
flume agent config:
TwitterAgent.sources= Twitter
TwitterAgent.channels= MemChannel
TwitterAgent.sinks=HDFS
TwitterAgent.sources.Twitter.type =
org.apache.flume.source.twitter.TwitterSource
TwitterAgent.sources.Twitter.channels=MemChannel
TwitterAgent.sources.Twitter.consumerKey=xxxxx
TwitterAgent.sources.Twitter.consumerSecret=xxxx
TwitterAgent.sources.Twitter.accessToken=xxxx
TwitterAgent.sources.Twitter.accessTokenSecret= xxxx
TwitterAgent.sinks.HDFS.channel=MemChannel
TwitterAgent.sinks.HDFS.type=hdfs
TwitterAgent.sinks.HDFS.hdfs.path=hdfs://localhost:8020/user/xxx/flume/events
TwitterAgent.sinks.HDFS.hdfs.fileType=DataStream
TwitterAgent.sinks.HDFS.hdfs.writeformat=Text
TwitterAgent.sinks.HDFS.hdfs.batchSize=1000
TwitterAgent.sinks.HDFS.hdfs.rollSize=0
TwitterAgent.sinks.HDFS.hdfs.rollCount=10000
TwitterAgent.sinks.HDFS.hdfs.rollInterval=600
TwitterAgent.channels.MemChannel.type=memory
TwitterAgent.channels.MemChannel.capacity=10000
TwitterAgent.channels.MemChannel.transactionCapacity=100
=======================Twitter file in HDFS==============
Objavro.schema�
{"type":"record","name":"Doc","doc":"adoc","fields":[{"name":"id","type":"string"},{"name":"user_friends_count","type":["int","null"]},{"name":"user_location","type":["string","null"]},{"name":"user_description","type":["string","null"]},{"name":"user_statuses_count","type":["int","null"]},{"name":"user_followers_count","type":["int","null"]},{"name":"user_name","type":["string","null"]},{"name":"user_screen_name","type":["string","null"]},{"name":"created_at","type":["string","null"]},{"name":"text","type":["string","null"]},{"name":"retweet_count","type":["long","null"]},{"name":"retweeted","type":["boolean","null"]},{"name":"in_reply_to_user_id","type":["long","null"]},{"name":"source","type":["string","null"]},{"name":"in_reply_to_status_id","type":["long","null"]},{"name":"media_url_https","type":["string","null"]},{"name":"expanded_url","type":["string","null"]}]}��rm���R��d�q �#�$838697771628544001Jぽやしみ��F$おやすみ世界eZ_XL1200C(2017-03-06T05:28:25Z�30ゲームから頑張ってはめ続けたのにやめて
https://t.co/SmtQDJHAdz�<a href="http://twitter.com/download/iphone";
rel="nofollow">Twitter for
iPhone</a>^https://pbs.twimg.com/media/C6OnJgRVUAExHzQ.jpg�https://twitter.com/eZ_XL1200C/status/838697771628544001/photo/1��rm���R��d�q �#
Objavro.schema�
{"type":"record","name":"Doc","doc":"adoc","fields":[{"name":"id","type":"string"},{"name":"user_friends_count","type":["int","null"]},{"name":"user_location","type":["string","null"]},{"name":"user_description","type":["string","null"]},{"name":"user_statuses_count","type":["int","null"]},{"name":"user_followers_count","type":["int","null"]},{"name":"user_name","type":["string","null"]},{"name":"user_screen_name","type":["string","null"]},{"name":"created_at","type":["string","null"]},{"name":"text","type":["string","null"]},{"name":"retweet_count","type":["long","null"]},{"name":"retweeted","type":["boolean","null"]},{"name":"in_reply_to_user_id","type":["long","null"]},{"name":"source","type":["string","null"]},{"name":"in_reply_to_status_id","type":["long","null"]},{"name":"media_url_https","type":["string","null"]},{"name":"expanded_url","type":["string","null"]}]}I��
$
��b�u�St�XTޥ$838697775802040320��Exénekakavshi(2017-03-06T05:28:26Z�SUPER
on est en alerte orange ils nous sortent quoi??? "faut pas s'approcher des
toits les tuiles c fragile encore" mdrrrrr mrci fdp�<a
href="http://twitter.com/download/iphone"; rel="nofollow">Twitter for
iPhone</a>$838697775797792768� �full time working mom. convinced
Runner. canadian at heart and #fantasydiamondrace Team holder. عبد الله
الهرري��� running falahrunningfalah(2017-03-06T05:28:26Z�Heute läufts "wir
brauchen dringendst Pasta zu Mittag" gut im Büro.�<a
href="http://twitter.com"; rel="nofollow">Twitter Web
Client</a>$838697775818747904�|*دولة الكويت�من بغإني صرٺ #لعيونه ـ فدآ
ومن تركنيء جعل ربيء يحفظه 💯
�m��Nضــّمـآ آلـمـشــاعـر
8vvvp(2017-03-06T05:28:26Z�RT @Sm_F__16: -
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
