> Hi Sadaf
>
> Which version of spark are you using? And whats in the spark-env.sh file?
> I think you are using both SPARK_CLASSPATH (which is deprecated) and
> spark.executor.extraClasspath (may be set in spark-defaults.sh file).
>
> Thanks
> Best Regards
>
> On Wed, Aug 5, 2015 at 6:22 PM, Sadaf Khan <sa...@platalytics.com> wrote:
>
>> Hi Akhil,
>>
>> I know you are a big knowledge base of spark streaming.
>> I wan your little help more.
>> I've done the twitter streaming using twitter's streaming user api and
>> spark streaming. this runs successfully on my local machine. but when i run
>> this program on cluster in local mode. it just run successfully for the
>> very first time. later on it gives the following exception.
>>
>> "Exception in thread "main" org.apache.spark.SparkException: Found both
>> spark.executor.extraClassPath and SPARK_CLASSPATH. Use only the former."
>>
>> and spark class path is unset already!!
>> I have to make a new checkpoint directory each time to make it run
>> successfully. otherwise it shows above exception.
>>
>> Can you please again help me to resolve this issue?
>> Thanks :)
>>
>> On Tue, Aug 4, 2015 at 4:33 PM, Sadaf Khan <sa...@platalytics.com> wrote:
>>
>>> thanks for being a helping hand before. :)
>>>
>>> one more point i want to discuss is that i now used Twitter Rest Api and
>>> successfully fetch the home timeline. but now it is showing 20 recent
>>> tweets,
>>> Is there any way to get 3200 (max rate limit) last tweets?
>>>
>>> Thanks in anticipation :)
>>>
>>>
>>> On Tue, Aug 4, 2015 at 2:05 PM, Sadaf Khan <sa...@platalytics.com>
>>> wrote:
>>>
>>>> thanks alot
>>>>
>>>> On Tue, Aug 4, 2015 at 2:00 PM, Akhil Das <ak...@sigmoidanalytics.com>
>>>> wrote:
>>>>
>>>>> You will have to write your own consumer for pulling your custom
>>>>> feeds, and then you can do a union
>>>>> (customfeedDstream.union(twitterStream))
>>>>> with the twitter stream api.
>>>>>
>>>>> Thanks
>>>>> Best Regards
>>>>>
>>>>> On Tue, Aug 4, 2015 at 2:28 PM, Sadaf Khan <sa...@platalytics.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks alot :)
>>>>>>
>>>>>> One more thing that i want to ask is that i have used twitters
>>>>>> streaming api.and it seems that the above solution uses rest api. how
>>>>>> can i
>>>>>> used both simultaneously ?
>>>>>>
>>>>>> Any response will be much appreciated :)
>>>>>> Regards
>>>>>>
>>>>>> On Tue, Aug 4, 2015 at 1:51 PM, Akhil Das <ak...@sigmoidanalytics.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Yes you can, when you start the application in the first batch you
>>>>>>> just need to pull all the tweets from your account. You need to look
>>>>>>> into
>>>>>>> the API for that. Have a look at this
>>>>>>> https://dev.twitter.com/rest/reference/get/statuses/user_timeline
>>>>>>>
>>>>>>> Thanks
>>>>>>> Best Regards
>>>>>>>
>>>>>>> On Tue, Aug 4, 2015 at 1:47 PM, Sadaf Khan <sa...@platalytics.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi.
>>>>>>>> You were really helpful for me last time :) and i have done with
>>>>>>>> the last problem.
>>>>>>>> I wanna ask you one more question. Now my connector is showing the
>>>>>>>> tweets that occurs after running the program. Is there any way to
>>>>>>>> fetch all
>>>>>>>> old tweets since when the account was created?
>>>>>>>>
>>>>>>>> I will be thankful to you for you kind response.
>>>>>>>>
>>>>>>>> On Thu, Jul 30, 2015 at 6:49 PM, Sadaf Khan <sa...@platalytics.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> thanks alot for this help :)
>>>>>>>>>
>>>>>>>>> On Thu, Jul 30, 2015 at 6:41 PM, Akhil Das <
>>>>>>>>> ak...@sigmoidanalytics.com> wrote:
>>>>>>>>>
>>>>>>>>>> You can create a custom receiver and then inside it you can write
>>>>>>>>>> yourown piece of code to receive data, filter them etc before giving
>>>>>>>>>> it to
>>>>>>>>>> spark.
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> Best Regards
>>>>>>>>>>
>>>>>>>>>> On Thu, Jul 30, 2015 at 6:49 PM, Sadaf Khan <
>>>>>>>>>> sa...@platalytics.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> okay :)
>>>>>>>>>>>
>>>>>>>>>>> then is there anyway to fetch the tweets specific to my account?
>>>>>>>>>>>
>>>>>>>>>>> Thanks in anticipation :)
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Jul 30, 2015 at 6:17 PM, Akhil Das <
>>>>>>>>>>> ak...@sigmoidanalytics.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Owh, this one fetches the public tweets, not the one specific
>>>>>>>>>>>> to your account.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> Best Regards
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Jul 30, 2015 at 6:11 PM, Sadaf Khan <
>>>>>>>>>>>> sa...@platalytics.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> yes. but can you please tell me how to mention a specific user
>>>>>>>>>>>>> account in filter?
>>>>>>>>>>>>> I want to fetch my tweets, tweets of my followers and the
>>>>>>>>>>>>> tweets of those whom i followed.
>>>>>>>>>>>>> So in short i want to fatch the tweets of my account only.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Recently i have used
>>>>>>>>>>>>> val tweets
>>>>>>>>>>>>> =TwitterUtils.createStream(ssc,atwitter,Nil,StorageLevel.MEMORY_AND_DISK_2)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Any response will be very much appreciated. :)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Jul 30, 2015 at 5:20 PM, Akhil Das <
>>>>>>>>>>>>> ak...@sigmoidanalytics.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> TwitterUtils.createStream takes a 3rd argument which is a
>>>>>>>>>>>>>> filter, once you provide these, it will only fetch tweets of
>>>>>>>>>>>>>> such.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>> Best Regards
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Jul 30, 2015 at 4:19 PM, Sadaf <sa...@platalytics.com
>>>>>>>>>>>>>> > wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi.
>>>>>>>>>>>>>>> I am writing twitter connector using spark streaming. but it
>>>>>>>>>>>>>>> fetched the
>>>>>>>>>>>>>>> random tweets.
>>>>>>>>>>>>>>> Is there any way to receive the tweets of a particular
>>>>>>>>>>>>>>> account?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I made an app on twitter and used the credentials as given
>>>>>>>>>>>>>>> below.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> def managingCredentials():
>>>>>>>>>>>>>>> Option[twitter4j.auth.Authorization]=
>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>> object auth{
>>>>>>>>>>>>>>> val config = new twitter4j.conf.ConfigurationBuilder()
>>>>>>>>>>>>>>> .setOAuthConsumerKey("****")
>>>>>>>>>>>>>>> .setOAuthConsumerSecret("****")
>>>>>>>>>>>>>>> .setOAuthAccessToken("****")
>>>>>>>>>>>>>>> .setOAuthAccessTokenSecret("****")
>>>>>>>>>>>>>>> .build
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>> val twitter_auth = new TwitterFactory(auth.config)
>>>>>>>>>>>>>>> val a = new
>>>>>>>>>>>>>>> twitter4j.auth.OAuthAuthorization(auth.config)
>>>>>>>>>>>>>>> val atwitter : Option[twitter4j.auth.Authorization] =
>>>>>>>>>>>>>>> Some(twitter_auth.getInstance(a).getAuthorization())
>>>>>>>>>>>>>>> atwitter
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks :)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> View this message in context:
>>>>>>>>>>>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Twitter-Connector-Spark-Streaming-tp24078.html
>>>>>>>>>>>>>>> Sent from the Apache Spark User List mailing list archive at
>>>>>>>>>>>>>>> Nabble.com.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>>>>>>>>>>>>>>> For additional commands, e-mail: user-h...@spark.apache.org
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
