Twitter Connector-Spark Streaming
Hi. I am writing twitter connector using spark streaming. but it fetched the random tweets. Is there any way to receive the tweets of a particular account? I made an app on twitter and used the credentials as given below. def managingCredentials(): Option[twitter4j.auth.Authorization]= { object auth{ val config = new twitter4j.conf.ConfigurationBuilder() .setOAuthConsumerKey("") .setOAuthConsumerSecret("") .setOAuthAccessToken("") .setOAuthAccessTokenSecret("") .build } val twitter_auth = new TwitterFactory(auth.config) val a = new twitter4j.auth.OAuthAuthorization(auth.config) val atwitter : Option[twitter4j.auth.Authorization] = Some(twitter_auth.getInstance(a).getAuthorization()) atwitter } Thanks :) -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Twitter-Connector-Spark-Streaming-tp24078.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Twitter Connector-Spark Streaming
TwitterUtils.createStream takes a 3rd argument which is a filter, once you provide these, it will only fetch tweets of such. Thanks Best Regards On Thu, Jul 30, 2015 at 4:19 PM, Sadaf wrote: > Hi. > I am writing twitter connector using spark streaming. but it fetched the > random tweets. > Is there any way to receive the tweets of a particular account? > > I made an app on twitter and used the credentials as given below. > > def managingCredentials(): Option[twitter4j.auth.Authorization]= > { > object auth{ > val config = new twitter4j.conf.ConfigurationBuilder() > .setOAuthConsumerKey("") > .setOAuthConsumerSecret("") > .setOAuthAccessToken("") > .setOAuthAccessTokenSecret("") > .build > } > val twitter_auth = new TwitterFactory(auth.config) > val a = new twitter4j.auth.OAuthAuthorization(auth.config) > val atwitter : Option[twitter4j.auth.Authorization] = > Some(twitter_auth.getInstance(a).getAuthorization()) > atwitter > } > > Thanks :) > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Twitter-Connector-Spark-Streaming-tp24078.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
Re: Twitter Connector-Spark Streaming
You can create a custom receiver and then inside it you can write yourown piece of code to receive data, filter them etc before giving it to spark. Thanks Best Regards On Thu, Jul 30, 2015 at 6:49 PM, Sadaf Khan wrote: > okay :) > > then is there anyway to fetch the tweets specific to my account? > > Thanks in anticipation :) > > On Thu, Jul 30, 2015 at 6:17 PM, Akhil Das > wrote: > >> Owh, this one fetches the public tweets, not the one specific to your >> account. >> >> Thanks >> Best Regards >> >> On Thu, Jul 30, 2015 at 6:11 PM, Sadaf Khan >> wrote: >> >>> yes. but can you please tell me how to mention a specific user account >>> in filter? >>> I want to fetch my tweets, tweets of my followers and the tweets of >>> those whom i followed. >>> So in short i want to fatch the tweets of my account only. >>> >>> Recently i have used >>>val tweets >>> =TwitterUtils.createStream(ssc,atwitter,Nil,StorageLevel.MEMORY_AND_DISK_2) >>> >>> >>> Any response will be very much appreciated. :) >>> >>> Thanks. >>> >>> >>> On Thu, Jul 30, 2015 at 5:20 PM, Akhil Das >>> wrote: >>> >>>> TwitterUtils.createStream takes a 3rd argument which is a filter, once >>>> you provide these, it will only fetch tweets of such. >>>> >>>> Thanks >>>> Best Regards >>>> >>>> On Thu, Jul 30, 2015 at 4:19 PM, Sadaf wrote: >>>> >>>>> Hi. >>>>> I am writing twitter connector using spark streaming. but it fetched >>>>> the >>>>> random tweets. >>>>> Is there any way to receive the tweets of a particular account? >>>>> >>>>> I made an app on twitter and used the credentials as given below. >>>>> >>>>> def managingCredentials(): Option[twitter4j.auth.Authorization]= >>>>> { >>>>> object auth{ >>>>> val config = new twitter4j.conf.ConfigurationBuilder() >>>>> .setOAuthConsumerKey("") >>>>> .setOAuthConsumerSecret("") >>>>> .setOAuthAccessToken("") >>>>> .setOAuthAccessTokenSecret("") >>>>> .build >>>>> } >>>>> val twitter_auth = new TwitterFactory(auth.config) >>>>> val a = new twitter4j.auth.OAuthAuthorization(auth.config) >>>>> val atwitter : Option[twitter4j.auth.Authorization] = >>>>> Some(twitter_auth.getInstance(a).getAuthorization()) >>>>> atwitter >>>>> } >>>>> >>>>> Thanks :) >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> View this message in context: >>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Twitter-Connector-Spark-Streaming-tp24078.html >>>>> Sent from the Apache Spark User List mailing list archive at >>>>> Nabble.com. >>>>> >>>>> - >>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>>> For additional commands, e-mail: user-h...@spark.apache.org >>>>> >>>>> >>>> >>> >> >
Re: Twitter Connector-Spark Streaming
You will have to write your own consumer for pulling your custom feeds, and then you can do a union (customfeedDstream.union(twitterStream)) with the twitter stream api. Thanks Best Regards On Tue, Aug 4, 2015 at 2:28 PM, Sadaf Khan wrote: > Thanks alot :) > > One more thing that i want to ask is that i have used twitters streaming > api.and it seems that the above solution uses rest api. how can i used both > simultaneously ? > > Any response will be much appreciated :) > Regards > > On Tue, Aug 4, 2015 at 1:51 PM, Akhil Das > wrote: > >> Yes you can, when you start the application in the first batch you just >> need to pull all the tweets from your account. You need to look into the >> API for that. Have a look at this >> https://dev.twitter.com/rest/reference/get/statuses/user_timeline >> >> Thanks >> Best Regards >> >> On Tue, Aug 4, 2015 at 1:47 PM, Sadaf Khan wrote: >> >>> Hi. >>> You were really helpful for me last time :) and i have done with the >>> last problem. >>> I wanna ask you one more question. Now my connector is showing the >>> tweets that occurs after running the program. Is there any way to fetch all >>> old tweets since when the account was created? >>> >>> I will be thankful to you for you kind response. >>> >>> On Thu, Jul 30, 2015 at 6:49 PM, Sadaf Khan >>> wrote: >>> >>>> thanks alot for this help :) >>>> >>>> On Thu, Jul 30, 2015 at 6:41 PM, Akhil Das >>>> wrote: >>>> >>>>> You can create a custom receiver and then inside it you can write >>>>> yourown piece of code to receive data, filter them etc before giving it to >>>>> spark. >>>>> >>>>> Thanks >>>>> Best Regards >>>>> >>>>> On Thu, Jul 30, 2015 at 6:49 PM, Sadaf Khan >>>>> wrote: >>>>> >>>>>> okay :) >>>>>> >>>>>> then is there anyway to fetch the tweets specific to my account? >>>>>> >>>>>> Thanks in anticipation :) >>>>>> >>>>>> On Thu, Jul 30, 2015 at 6:17 PM, Akhil Das < >>>>>> ak...@sigmoidanalytics.com> wrote: >>>>>> >>>>>>> Owh, this one fetches the public tweets, not the one specific to >>>>>>> your account. >>>>>>> >>>>>>> Thanks >>>>>>> Best Regards >>>>>>> >>>>>>> On Thu, Jul 30, 2015 at 6:11 PM, Sadaf Khan >>>>>>> wrote: >>>>>>> >>>>>>>> yes. but can you please tell me how to mention a specific user >>>>>>>> account in filter? >>>>>>>> I want to fetch my tweets, tweets of my followers and the tweets of >>>>>>>> those whom i followed. >>>>>>>> So in short i want to fatch the tweets of my account only. >>>>>>>> >>>>>>>> Recently i have used >>>>>>>>val tweets >>>>>>>> =TwitterUtils.createStream(ssc,atwitter,Nil,StorageLevel.MEMORY_AND_DISK_2) >>>>>>>> >>>>>>>> >>>>>>>> Any response will be very much appreciated. :) >>>>>>>> >>>>>>>> Thanks. >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Jul 30, 2015 at 5:20 PM, Akhil Das < >>>>>>>> ak...@sigmoidanalytics.com> wrote: >>>>>>>> >>>>>>>>> TwitterUtils.createStream takes a 3rd argument which is a filter, >>>>>>>>> once you provide these, it will only fetch tweets of such. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> Best Regards >>>>>>>>> >>>>>>>>> On Thu, Jul 30, 2015 at 4:19 PM, Sadaf >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi. >>>>>>>>>> I am writing twitter connector using spark streaming. but it >>>>>>>>>> fetched the >>>>>>>>>> random tweets. >>>>>>>>>> Is there any way to receive the tweets of a particular account? >>>>>>>>>> >>>>>>>>>> I made an app on twitter and used the credentials as given below. >>>>>>>>>> >>>>>>>>>> def managingCredentials(): Option[twitter4j.auth.Authorization]= >>>>>>>>>> { >>>>>>>>>> object auth{ >>>>>>>>>> val config = new twitter4j.conf.ConfigurationBuilder() >>>>>>>>>> .setOAuthConsumerKey("") >>>>>>>>>> .setOAuthConsumerSecret("") >>>>>>>>>> .setOAuthAccessToken("") >>>>>>>>>> .setOAuthAccessTokenSecret("") >>>>>>>>>> .build >>>>>>>>>> } >>>>>>>>>> val twitter_auth = new TwitterFactory(auth.config) >>>>>>>>>> val a = new twitter4j.auth.OAuthAuthorization(auth.config) >>>>>>>>>> val atwitter : Option[twitter4j.auth.Authorization] = >>>>>>>>>> Some(twitter_auth.getInstance(a).getAuthorization()) >>>>>>>>>> atwitter >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> Thanks :) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> View this message in context: >>>>>>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Twitter-Connector-Spark-Streaming-tp24078.html >>>>>>>>>> Sent from the Apache Spark User List mailing list archive at >>>>>>>>>> Nabble.com. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> - >>>>>>>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>>>>>>>> For additional commands, e-mail: user-h...@spark.apache.org >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
Re: Twitter Connector-Spark Streaming
gt; yourown piece of code to receive data, filter them etc before giving >>>>>>>>>> it to >>>>>>>>>> spark. >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> Best Regards >>>>>>>>>> >>>>>>>>>> On Thu, Jul 30, 2015 at 6:49 PM, Sadaf Khan < >>>>>>>>>> sa...@platalytics.com> wrote: >>>>>>>>>> >>>>>>>>>>> okay :) >>>>>>>>>>> >>>>>>>>>>> then is there anyway to fetch the tweets specific to my account? >>>>>>>>>>> >>>>>>>>>>> Thanks in anticipation :) >>>>>>>>>>> >>>>>>>>>>> On Thu, Jul 30, 2015 at 6:17 PM, Akhil Das < >>>>>>>>>>> ak...@sigmoidanalytics.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Owh, this one fetches the public tweets, not the one specific >>>>>>>>>>>> to your account. >>>>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>>>> Best Regards >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Jul 30, 2015 at 6:11 PM, Sadaf Khan < >>>>>>>>>>>> sa...@platalytics.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> yes. but can you please tell me how to mention a specific user >>>>>>>>>>>>> account in filter? >>>>>>>>>>>>> I want to fetch my tweets, tweets of my followers and the >>>>>>>>>>>>> tweets of those whom i followed. >>>>>>>>>>>>> So in short i want to fatch the tweets of my account only. >>>>>>>>>>>>> >>>>>>>>>>>>> Recently i have used >>>>>>>>>>>>>val tweets >>>>>>>>>>>>> =TwitterUtils.createStream(ssc,atwitter,Nil,StorageLevel.MEMORY_AND_DISK_2) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Any response will be very much appreciated. :) >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Jul 30, 2015 at 5:20 PM, Akhil Das < >>>>>>>>>>>>> ak...@sigmoidanalytics.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> TwitterUtils.createStream takes a 3rd argument which is a >>>>>>>>>>>>>> filter, once you provide these, it will only fetch tweets of >>>>>>>>>>>>>> such. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>> Best Regards >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, Jul 30, 2015 at 4:19 PM, Sadaf >>>>>>>>>>>>> > wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi. >>>>>>>>>>>>>>> I am writing twitter connector using spark streaming. but it >>>>>>>>>>>>>>> fetched the >>>>>>>>>>>>>>> random tweets. >>>>>>>>>>>>>>> Is there any way to receive the tweets of a particular >>>>>>>>>>>>>>> account? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I made an app on twitter and used the credentials as given >>>>>>>>>>>>>>> below. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> def managingCredentials(): >>>>>>>>>>>>>>> Option[twitter4j.auth.Authorization]= >>>>>>>>>>>>>>> { >>>>>>>>>>>>>>> object auth{ >>>>>>>>>>>>>>> val config = new twitter4j.conf.ConfigurationBuilder() >>>>>>>>>>>>>>> .setOAuthConsumerKey("") >>>>>>>>>>>>>>> .setOAuthConsumerSecret("") >>>>>>>>>>>>>>> .setOAuthAccessToken("") >>>>>>>>>>>>>>> .setOAuthAccessTokenSecret("") >>>>>>>>>>>>>>> .build >>>>>>>>>>>>>>> } >>>>>>>>>>>>>>> val twitter_auth = new TwitterFactory(auth.config) >>>>>>>>>>>>>>> val a = new >>>>>>>>>>>>>>> twitter4j.auth.OAuthAuthorization(auth.config) >>>>>>>>>>>>>>> val atwitter : Option[twitter4j.auth.Authorization] = >>>>>>>>>>>>>>> Some(twitter_auth.getInstance(a).getAuthorization()) >>>>>>>>>>>>>>> atwitter >>>>>>>>>>>>>>> } >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks :) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> View this message in context: >>>>>>>>>>>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Twitter-Connector-Spark-Streaming-tp24078.html >>>>>>>>>>>>>>> Sent from the Apache Spark User List mailing list archive at >>>>>>>>>>>>>>> Nabble.com. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - >>>>>>>>>>>>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>>>>>>>>>>>>> For additional commands, e-mail: user-h...@spark.apache.org >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >