[ https://issues.apache.org/jira/browse/FLUME-3066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898648#comment-15898648 ]
Saket commented on FLUME-3066: ------------------------------ Any one please helpis issue.h me to resolve t > Twitter Streaming using flume in hdfs > ------------------------------------- > > Key: FLUME-3066 > URL: https://issues.apache.org/jira/browse/FLUME-3066 > Project: Flume > Issue Type: Question > Affects Versions: v1.7.0 > Reporter: Saket > > I was trying to stream twitter data in HDFS using flume.Succeffuly configured > agent but the data population in HDFS doesnt seems proper please help. > flume agent config: > TwitterAgent.sources= Twitter > TwitterAgent.channels= MemChannel > TwitterAgent.sinks=HDFS > TwitterAgent.sources.Twitter.type = > org.apache.flume.source.twitter.TwitterSource > TwitterAgent.sources.Twitter.channels=MemChannel > TwitterAgent.sources.Twitter.consumerKey=xxxxx > TwitterAgent.sources.Twitter.consumerSecret=xxxx > TwitterAgent.sources.Twitter.accessToken=xxxx > TwitterAgent.sources.Twitter.accessTokenSecret= xxxx > TwitterAgent.sinks.HDFS.channel=MemChannel > TwitterAgent.sinks.HDFS.type=hdfs > TwitterAgent.sinks.HDFS.hdfs.path=hdfs://localhost:8020/user/xxx/flume/events > TwitterAgent.sinks.HDFS.hdfs.fileType=DataStream > TwitterAgent.sinks.HDFS.hdfs.writeformat=Text > TwitterAgent.sinks.HDFS.hdfs.batchSize=1000 > TwitterAgent.sinks.HDFS.hdfs.rollSize=0 > TwitterAgent.sinks.HDFS.hdfs.rollCount=10000 > TwitterAgent.sinks.HDFS.hdfs.rollInterval=600 > TwitterAgent.channels.MemChannel.type=memory > TwitterAgent.channels.MemChannel.capacity=10000 > TwitterAgent.channels.MemChannel.transactionCapacity=100 > =======================Twitter file in HDFS============== > Objavro.schema� > {"type":"record","name":"Doc","doc":"adoc","fields":[{"name":"id","type":"string"},{"name":"user_friends_count","type":["int","null"]},{"name":"user_location","type":["string","null"]},{"name":"user_description","type":["string","null"]},{"name":"user_statuses_count","type":["int","null"]},{"name":"user_followers_count","type":["int","null"]},{"name":"user_name","type":["string","null"]},{"name":"user_screen_name","type":["string","null"]},{"name":"created_at","type":["string","null"]},{"name":"text","type":["string","null"]},{"name":"retweet_count","type":["long","null"]},{"name":"retweeted","type":["boolean","null"]},{"name":"in_reply_to_user_id","type":["long","null"]},{"name":"source","type":["string","null"]},{"name":"in_reply_to_status_id","type":["long","null"]},{"name":"media_url_https","type":["string","null"]},{"name":"expanded_url","type":["string","null"]}]}��rm���R��d�q�#�$838697771628544001Jぽやしみ��F$おやすみ世界eZ_XL1200C(2017-03-06T05:28:25Z�30ゲームから頑張ってはめ続けたのにやめて > https://t.co/SmtQDJHAdz�<a href="http://twitter.com/download/iphone" > rel="nofollow">Twitter for > iPhone</a>^https://pbs.twimg.com/media/C6OnJgRVUAExHzQ.jpg�https://twitter.com/eZ_XL1200C/status/838697771628544001/photo/1��rm���R��d�q�# > Objavro.schema� > {"type":"record","name":"Doc","doc":"adoc","fields":[{"name":"id","type":"string"},{"name":"user_friends_count","type":["int","null"]},{"name":"user_location","type":["string","null"]},{"name":"user_description","type":["string","null"]},{"name":"user_statuses_count","type":["int","null"]},{"name":"user_followers_count","type":["int","null"]},{"name":"user_name","type":["string","null"]},{"name":"user_screen_name","type":["string","null"]},{"name":"created_at","type":["string","null"]},{"name":"text","type":["string","null"]},{"name":"retweet_count","type":["long","null"]},{"name":"retweeted","type":["boolean","null"]},{"name":"in_reply_to_user_id","type":["long","null"]},{"name":"source","type":["string","null"]},{"name":"in_reply_to_status_id","type":["long","null"]},{"name":"media_url_https","type":["string","null"]},{"name":"expanded_url","type":["string","null"]}]}I�� > $ > ��b�u�St�XTޥ$838697775802040320��Exénekakavshi(2017-03-06T05:28:26Z�SUPER > on est en alerte orange ils nous sortent quoi??? "faut pas s'approcher des > toits les tuiles c fragile encore" mdrrrrr mrci fdp�<a > href="http://twitter.com/download/iphone" rel="nofollow">Twitter for > iPhone</a>$838697775797792768� �full time working mom. convinced > Runner. canadian at heart and #fantasydiamondrace Team holder. عبد الله > الهرري��� running falahrunningfalah(2017-03-06T05:28:26Z�Heute läufts > "wir brauchen dringendst Pasta zu Mittag" gut im Büro.�<a > href="http://twitter.com" rel="nofollow">Twitter Web > Client</a>$838697775818747904�|*دولة الكويت�من بغإني صرٺ #لعيونه ـ فدآ > ومن تركنيء جعل ربيء يحفظه 💯 > �m��Nضــّمـآ آلـمـشــاعـر > 8vvvp(2017-03-06T05:28:26Z�RT @Sm_F__16: - -- This message was sent by Atlassian JIRA (v6.3.15#6346)