Hi All,
I have a requirement to call a rest service url for 300k customer ids. Things I have tried so far is custid_rdd = sc.textFile('file:////Users/zzz/CustomerID_SC/Inactive User Hashed LCID List.csv') #getting all the customer ids and building adds profile_rdd = custid_rdd.map(lambda r: getProfile(r.split(',')[0])) profile_rdd.count() #getprofile is the method to do the http call def getProfile(cust_id): api_key = 'txt' api_secret = 'yuyuy' profile_uri = 'https://profile.localytics.com/x1/customers/{}' customer_id = cust_id if customer_id is not None: data = requests.get(profile_uri.format(customer_id), auth=requests.auth.HTTPBasicAuth(api_key, api_secret)) # print json.dumps(data.json(), indent=4) return data when I print the json dump of the data i see it returning results from the rest call. But the count never stops. Is there an efficient way of dealing this? Some post says we have to define a batch size etc but don't know how. Appreciate your help Regards, Amit