Dear All, I am facing problem with Spark Twitter Streaming code, When ever twitter4j throws exception, i am unable to catch that exception. Could anyone help me catching that exception.
Here is Pseudo Code: SparkConf sparkConf = new SparkConf().setMaster("local[2]").setAppName("Test"); // SparkConf sparkConf = new SparkConf().setMaster("yarn-client").setAppName("Test"); // SparkTwitterStreaming sss = new SparkTwitterStreaming(); final int batchIntervalInSec = 60; // choose long interval to reduce // num // files created Duration batchInterval = Durations.seconds(batchIntervalInSec); JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, batchInterval); try{ JavaReceiverInputDStream<Status> receiverStream = null; receiverStream = TwitterUtils.createStream(jssc, a2, params); JavaDStream<String> tweets = receiverStream.map(new Function<Status, String>() { public String call(Status tweet) throws Exception { String json = ""; Gson gson = new Gson(); jssc.start(); jssc.awaitTermination(); } I have no issue with streaming the twitter data, When ever twitter account had expired, i have to catch this exception and do work around for this exception. Here is the exception trace: INFO spark.streaming.receiver.BlockGenerator - Started BlockGenerator INFO spark.streaming.scheduler.ReceiverTracker - Registered receiver for stream 0 from 172.16.28.183:34829 INFO spark.streaming.receiver.ReceiverSupervisorImpl - Starting receiver INFO spark.streaming.twitter.TwitterReceiver - Twitter receiver started INFO spark.streaming.receiver.ReceiverSupervisorImpl - Called receiver onStart INFO spark.streaming.receiver.ReceiverSupervisorImpl - Waiting for receiver to be stopped INFO twitter4j.TwitterStreamImpl - Establishing connection. INFO twitter4j.TwitterStreamImpl - 401:Authentication credentials ( https://dev.twitter.com/pages/auth) were missing or incorrect. Ensure that you have set valid consumer key/secret, access token/secret, and the system clock is in sync. <html>\n<head>\n<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>\n<title>Error 401 Unauthorized</title> </head> <body> <h2>HTTP ERROR: 401</h2> <p>Problem accessing '/1.1/statuses/filter.json'. Reason: <pre> Unauthorized</pre> </body> </html> INFO twitter4j.TwitterStreamImpl - Waiting for 10000 milliseconds WARN spark.streaming.receiver.ReceiverSupervisorImpl - Restarting receiver with delay 2000 ms: Error receiving tweets 401:Authentication credentials (https://dev.twitter.com/pages/auth) were missing or incorrect. Ensure that you have set valid consumer key/secret, access token/secret, and the system clock is in sync. <html>\n<head>\n<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>\n<title>Error 401 Unauthorized</title> </head> <body> <h2>HTTP ERROR: 401</h2> <p>Problem accessing '/1.1/statuses/filter.json'. Reason: <pre> Unauthorized</pre> </body> </html> Relevant discussions can be found on the Internet at: http://www.google.co.jp/search?q=944a924a or http://www.google.co.jp/search?q=24fd66dc TwitterException{exceptionCode=[944a924a-24fd66dc], statusCode=401, message=null, code=-1, retryAfter=-1, rateLimitStatus=null, version=3.0.3} at twitter4j.internal.http.HttpClientImpl.request(HttpClientImpl.java:177) at twitter4j.internal.http.HttpClientWrapper.request(HttpClientWrapper.java:61) at twitter4j.internal.http.HttpClientWrapper.post(HttpClientWrapper.java:98) at twitter4j.TwitterStreamImpl.getFilterStream(TwitterStreamImpl.java:304) at twitter4j.TwitterStreamImpl$7.getStream(TwitterStreamImpl.java:292) at twitter4j.TwitterStreamImpl$TwitterStreamConsumer.run(TwitterStreamImpl.java:462) Please let me know if you need some more clarity my question. Thanks, Sony.