[
https://issues.apache.org/jira/browse/FLUME-3066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898648#comment-15898648
]
Saket edited comment on FLUME-3066 at 3/7/17 2:58 AM:
------------------------------------------------------
Any one please help me to resolve this issue.
Please let me know if any further info is required.
Thanx Saket.
was (Author: peta1):
Any one please help me to resolve this issue.
> Twitter Streaming using flume in hdfs
> -------------------------------------
>
> Key: FLUME-3066
> URL: https://issues.apache.org/jira/browse/FLUME-3066
> Project: Flume
> Issue Type: Question
> Affects Versions: v1.7.0
> Reporter: Saket
>
> I was trying to stream twitter data in HDFS using flume.Succeffuly configured
> agent but the data population in HDFS doesnt seems proper please help.
> flume agent config:
> TwitterAgent.sources= Twitter
> TwitterAgent.channels= MemChannel
> TwitterAgent.sinks=HDFS
> TwitterAgent.sources.Twitter.type =
> org.apache.flume.source.twitter.TwitterSource
> TwitterAgent.sources.Twitter.channels=MemChannel
> TwitterAgent.sources.Twitter.consumerKey=xxxxx
> TwitterAgent.sources.Twitter.consumerSecret=xxxx
> TwitterAgent.sources.Twitter.accessToken=xxxx
> TwitterAgent.sources.Twitter.accessTokenSecret= xxxx
> TwitterAgent.sinks.HDFS.channel=MemChannel
> TwitterAgent.sinks.HDFS.type=hdfs
> TwitterAgent.sinks.HDFS.hdfs.path=hdfs://localhost:8020/user/xxx/flume/events
> TwitterAgent.sinks.HDFS.hdfs.fileType=DataStream
> TwitterAgent.sinks.HDFS.hdfs.writeformat=Text
> TwitterAgent.sinks.HDFS.hdfs.batchSize=1000
> TwitterAgent.sinks.HDFS.hdfs.rollSize=0
> TwitterAgent.sinks.HDFS.hdfs.rollCount=10000
> TwitterAgent.sinks.HDFS.hdfs.rollInterval=600
> TwitterAgent.channels.MemChannel.type=memory
> TwitterAgent.channels.MemChannel.capacity=10000
> TwitterAgent.channels.MemChannel.transactionCapacity=100
> =======================Twitter file in HDFS==============
> Objavro.schema�
> {"type":"record","name":"Doc","doc":"adoc","fields":[{"name":"id","type":"string"},{"name":"user_friends_count","type":["int","null"]},{"name":"user_location","type":["string","null"]},{"name":"user_description","type":["string","null"]},{"name":"user_statuses_count","type":["int","null"]},{"name":"user_followers_count","type":["int","null"]},{"name":"user_name","type":["string","null"]},{"name":"user_screen_name","type":["string","null"]},{"name":"created_at","type":["string","null"]},{"name":"text","type":["string","null"]},{"name":"retweet_count","type":["long","null"]},{"name":"retweeted","type":["boolean","null"]},{"name":"in_reply_to_user_id","type":["long","null"]},{"name":"source","type":["string","null"]},{"name":"in_reply_to_status_id","type":["long","null"]},{"name":"media_url_https","type":["string","null"]},{"name":"expanded_url","type":["string","null"]}]}��rm���R��d�q �#�$838697771628544001Jぽやしみ��F$おやすみ世界eZ_XL1200C(2017-03-06T05:28:25Z�30ゲームから頑張ってはめ続けたのにやめて
> https://t.co/SmtQDJHAdz�<a href="http://twitter.com/download/iphone";
> rel="nofollow">Twitter for
> iPhone</a>^https://pbs.twimg.com/media/C6OnJgRVUAExHzQ.jpg�https://twitter.com/eZ_XL1200C/status/838697771628544001/photo/1��rm���R��d�q �#
> Objavro.schema�
> {"type":"record","name":"Doc","doc":"adoc","fields":[{"name":"id","type":"string"},{"name":"user_friends_count","type":["int","null"]},{"name":"user_location","type":["string","null"]},{"name":"user_description","type":["string","null"]},{"name":"user_statuses_count","type":["int","null"]},{"name":"user_followers_count","type":["int","null"]},{"name":"user_name","type":["string","null"]},{"name":"user_screen_name","type":["string","null"]},{"name":"created_at","type":["string","null"]},{"name":"text","type":["string","null"]},{"name":"retweet_count","type":["long","null"]},{"name":"retweeted","type":["boolean","null"]},{"name":"in_reply_to_user_id","type":["long","null"]},{"name":"source","type":["string","null"]},{"name":"in_reply_to_status_id","type":["long","null"]},{"name":"media_url_https","type":["string","null"]},{"name":"expanded_url","type":["string","null"]}]}I��
> $
> ��b�u�St�XTޥ$838697775802040320��Exénekakavshi(2017-03-06T05:28:26Z�SUPER
> on est en alerte orange ils nous sortent quoi??? "faut pas s'approcher des
> toits les tuiles c fragile encore" mdrrrrr mrci fdp�<a
> href="http://twitter.com/download/iphone"; rel="nofollow">Twitter for
> iPhone</a>$838697775797792768� �full time working mom. convinced
> Runner. canadian at heart and #fantasydiamondrace Team holder. عبد الله
> الهرري��� running falahrunningfalah(2017-03-06T05:28:26Z�Heute läufts
> "wir brauchen dringendst Pasta zu Mittag" gut im Büro.�<a
> href="http://twitter.com"; rel="nofollow">Twitter Web
> Client</a>$838697775818747904�|*دولة الكويت�من بغإني صرٺ #لعيونه ـ فدآ
> ومن تركنيء جعل ربيء يحفظه 💯
> �m��Nضــّمـآ آلـمـشــاعـر
> 8vvvp(2017-03-06T05:28:26Z�RT @Sm_F__16: -
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
