Re: Twitter Connector-Spark Streaming
Hi Sadaf
Which version of spark are you using? And whats in the spark-env.sh file?
I think you are using both SPARK_CLASSPATH (which is deprecated) and
spark.executor.extraClasspath (may be set in spark-defaults.sh file).
Thanks
Best Regards
On Wed, Aug 5, 2015 at 6:22 PM, Sadaf Khan sa...@platalytics.com wrote:
Hi Akhil,
I know you are a big knowledge base of spark streaming.
I wan your little help more.
I've done the twitter streaming using twitter's streaming user api and
spark streaming. this runs successfully on my local machine. but when i run
this program on cluster in local mode. it just run successfully for the
very first time. later on it gives the following exception.
Exception in thread main org.apache.spark.SparkException: Found both
spark.executor.extraClassPath and SPARK_CLASSPATH. Use only the former.
and spark class path is unset already!!
I have to make a new checkpoint directory each time to make it run
successfully. otherwise it shows above exception.
Can you please again help me to resolve this issue?
Thanks :)
On Tue, Aug 4, 2015 at 4:33 PM, Sadaf Khan sa...@platalytics.com wrote:
thanks for being a helping hand before. :)
one more point i want to discuss is that i now used Twitter Rest Api and
successfully fetch the home timeline. but now it is showing 20 recent
tweets,
Is there any way to get 3200 (max rate limit) last tweets?
Thanks in anticipation :)
On Tue, Aug 4, 2015 at 2:05 PM, Sadaf Khan sa...@platalytics.com
wrote:
thanks alot
On Tue, Aug 4, 2015 at 2:00 PM, Akhil Das ak...@sigmoidanalytics.com
wrote:
You will have to write your own consumer for pulling your custom
feeds, and then you can do a union
(customfeedDstream.union(twitterStream))
with the twitter stream api.
Thanks
Best Regards
On Tue, Aug 4, 2015 at 2:28 PM, Sadaf Khan sa...@platalytics.com
wrote:
Thanks alot :)
One more thing that i want to ask is that i have used twitters
streaming api.and it seems that the above solution uses rest api. how
can i
used both simultaneously ?
Any response will be much appreciated :)
Regards
On Tue, Aug 4, 2015 at 1:51 PM, Akhil Das ak...@sigmoidanalytics.com
wrote:
Yes you can, when you start the application in the first batch you
just need to pull all the tweets from your account. You need to look
into
the API for that. Have a look at this
https://dev.twitter.com/rest/reference/get/statuses/user_timeline
Thanks
Best Regards
On Tue, Aug 4, 2015 at 1:47 PM, Sadaf Khan sa...@platalytics.com
wrote:
Hi.
You were really helpful for me last time :) and i have done with
the last problem.
I wanna ask you one more question. Now my connector is showing the
tweets that occurs after running the program. Is there any way to
fetch all
old tweets since when the account was created?
I will be thankful to you for you kind response.
On Thu, Jul 30, 2015 at 6:49 PM, Sadaf Khan sa...@platalytics.com
wrote:
thanks alot for this help :)
On Thu, Jul 30, 2015 at 6:41 PM, Akhil Das
ak...@sigmoidanalytics.com wrote:
You can create a custom receiver and then inside it you can write
yourown piece of code to receive data, filter them etc before giving
it to
spark.
Thanks
Best Regards
On Thu, Jul 30, 2015 at 6:49 PM, Sadaf Khan
sa...@platalytics.com wrote:
okay :)
then is there anyway to fetch the tweets specific to my account?
Thanks in anticipation :)
On Thu, Jul 30, 2015 at 6:17 PM, Akhil Das
ak...@sigmoidanalytics.com wrote:
Owh, this one fetches the public tweets, not the one specific
to your account.
Thanks
Best Regards
On Thu, Jul 30, 2015 at 6:11 PM, Sadaf Khan
sa...@platalytics.com wrote:
yes. but can you please tell me how to mention a specific user
account in filter?
I want to fetch my tweets, tweets of my followers and the
tweets of those whom i followed.
So in short i want to fatch the tweets of my account only.
Recently i have used
val tweets
=TwitterUtils.createStream(ssc,atwitter,Nil,StorageLevel.MEMORY_AND_DISK_2)
Any response will be very much appreciated. :)
Thanks.
On Thu, Jul 30, 2015 at 5:20 PM, Akhil Das
ak...@sigmoidanalytics.com wrote:
TwitterUtils.createStream takes a 3rd argument which is a
filter, once you provide these, it will only fetch tweets of
such.
Thanks
Best Regards
On Thu, Jul 30, 2015 at 4:19 PM, Sadaf sa...@platalytics.com
wrote:
Hi.
I am writing twitter connector using spark streaming. but it
fetched the
random tweets.
Is there any way to receive the tweets of a particular
account?
I made an app on twitter and used the credentials as given
below.
def managingCredentials():
Option[twitter4j.auth.Authorization]=
{
object auth{
val config = new twitter4j.conf.ConfigurationBuilder()
.setOAuthConsumerKey()
.setOAuthConsumerSecret()
.setOAuthAccessToken()
.setOAuthAccessTokenSecret()
.build
}
Re: Twitter Connector-Spark Streaming
You will have to write your own consumer for pulling your custom feeds, and
then you can do a union (customfeedDstream.union(twitterStream)) with the
twitter stream api.
Thanks
Best Regards
On Tue, Aug 4, 2015 at 2:28 PM, Sadaf Khan sa...@platalytics.com wrote:
Thanks alot :)
One more thing that i want to ask is that i have used twitters streaming
api.and it seems that the above solution uses rest api. how can i used both
simultaneously ?
Any response will be much appreciated :)
Regards
On Tue, Aug 4, 2015 at 1:51 PM, Akhil Das ak...@sigmoidanalytics.com
wrote:
Yes you can, when you start the application in the first batch you just
need to pull all the tweets from your account. You need to look into the
API for that. Have a look at this
https://dev.twitter.com/rest/reference/get/statuses/user_timeline
Thanks
Best Regards
On Tue, Aug 4, 2015 at 1:47 PM, Sadaf Khan sa...@platalytics.com wrote:
Hi.
You were really helpful for me last time :) and i have done with the
last problem.
I wanna ask you one more question. Now my connector is showing the
tweets that occurs after running the program. Is there any way to fetch all
old tweets since when the account was created?
I will be thankful to you for you kind response.
On Thu, Jul 30, 2015 at 6:49 PM, Sadaf Khan sa...@platalytics.com
wrote:
thanks alot for this help :)
On Thu, Jul 30, 2015 at 6:41 PM, Akhil Das ak...@sigmoidanalytics.com
wrote:
You can create a custom receiver and then inside it you can write
yourown piece of code to receive data, filter them etc before giving it to
spark.
Thanks
Best Regards
On Thu, Jul 30, 2015 at 6:49 PM, Sadaf Khan sa...@platalytics.com
wrote:
okay :)
then is there anyway to fetch the tweets specific to my account?
Thanks in anticipation :)
On Thu, Jul 30, 2015 at 6:17 PM, Akhil Das
ak...@sigmoidanalytics.com wrote:
Owh, this one fetches the public tweets, not the one specific to
your account.
Thanks
Best Regards
On Thu, Jul 30, 2015 at 6:11 PM, Sadaf Khan sa...@platalytics.com
wrote:
yes. but can you please tell me how to mention a specific user
account in filter?
I want to fetch my tweets, tweets of my followers and the tweets of
those whom i followed.
So in short i want to fatch the tweets of my account only.
Recently i have used
val tweets
=TwitterUtils.createStream(ssc,atwitter,Nil,StorageLevel.MEMORY_AND_DISK_2)
Any response will be very much appreciated. :)
Thanks.
On Thu, Jul 30, 2015 at 5:20 PM, Akhil Das
ak...@sigmoidanalytics.com wrote:
TwitterUtils.createStream takes a 3rd argument which is a filter,
once you provide these, it will only fetch tweets of such.
Thanks
Best Regards
On Thu, Jul 30, 2015 at 4:19 PM, Sadaf sa...@platalytics.com
wrote:
Hi.
I am writing twitter connector using spark streaming. but it
fetched the
random tweets.
Is there any way to receive the tweets of a particular account?
I made an app on twitter and used the credentials as given below.
def managingCredentials(): Option[twitter4j.auth.Authorization]=
{
object auth{
val config = new twitter4j.conf.ConfigurationBuilder()
.setOAuthConsumerKey()
.setOAuthConsumerSecret()
.setOAuthAccessToken()
.setOAuthAccessTokenSecret()
.build
}
val twitter_auth = new TwitterFactory(auth.config)
val a = new twitter4j.auth.OAuthAuthorization(auth.config)
val atwitter : Option[twitter4j.auth.Authorization] =
Some(twitter_auth.getInstance(a).getAuthorization())
atwitter
}
Thanks :)
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Twitter-Connector-Spark-Streaming-tp24078.html
Sent from the Apache Spark User List mailing list archive at
Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
Re: Twitter Connector-Spark Streaming
You can create a custom receiver and then inside it you can write yourown
piece of code to receive data, filter them etc before giving it to spark.
Thanks
Best Regards
On Thu, Jul 30, 2015 at 6:49 PM, Sadaf Khan sa...@platalytics.com wrote:
okay :)
then is there anyway to fetch the tweets specific to my account?
Thanks in anticipation :)
On Thu, Jul 30, 2015 at 6:17 PM, Akhil Das ak...@sigmoidanalytics.com
wrote:
Owh, this one fetches the public tweets, not the one specific to your
account.
Thanks
Best Regards
On Thu, Jul 30, 2015 at 6:11 PM, Sadaf Khan sa...@platalytics.com
wrote:
yes. but can you please tell me how to mention a specific user account
in filter?
I want to fetch my tweets, tweets of my followers and the tweets of
those whom i followed.
So in short i want to fatch the tweets of my account only.
Recently i have used
val tweets
=TwitterUtils.createStream(ssc,atwitter,Nil,StorageLevel.MEMORY_AND_DISK_2)
Any response will be very much appreciated. :)
Thanks.
On Thu, Jul 30, 2015 at 5:20 PM, Akhil Das ak...@sigmoidanalytics.com
wrote:
TwitterUtils.createStream takes a 3rd argument which is a filter, once
you provide these, it will only fetch tweets of such.
Thanks
Best Regards
On Thu, Jul 30, 2015 at 4:19 PM, Sadaf sa...@platalytics.com wrote:
Hi.
I am writing twitter connector using spark streaming. but it fetched
the
random tweets.
Is there any way to receive the tweets of a particular account?
I made an app on twitter and used the credentials as given below.
def managingCredentials(): Option[twitter4j.auth.Authorization]=
{
object auth{
val config = new twitter4j.conf.ConfigurationBuilder()
.setOAuthConsumerKey()
.setOAuthConsumerSecret()
.setOAuthAccessToken()
.setOAuthAccessTokenSecret()
.build
}
val twitter_auth = new TwitterFactory(auth.config)
val a = new twitter4j.auth.OAuthAuthorization(auth.config)
val atwitter : Option[twitter4j.auth.Authorization] =
Some(twitter_auth.getInstance(a).getAuthorization())
atwitter
}
Thanks :)
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Twitter-Connector-Spark-Streaming-tp24078.html
Sent from the Apache Spark User List mailing list archive at
Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
