Re: Calling HTTP Rest APIs from Spark Job

Hi Sean, Thanks for great answer. What I am trying to do is to use something like Scala Future (cats-effect IO) to do concurrent calls. Was understanding if any limitation thresholds to make those calls. On Thu, May 14, 2020 at 7:28 PM Sean Owen wrote: > No, it means # HTTP calls = # executor s

Re: Calling HTTP Rest APIs from Spark Job

No, it means # HTTP calls = # executor slots. But even then, you're welcome to, say, use thread pools to execute even more concurrently as most are I/O bound. Your code can do what you want. On Thu, May 14, 2020 at 6:14 PM Chetan Khatri wrote: > > Thanks, that means number of executor = number of

Re: Calling HTTP Rest APIs from Spark Job

Thanks, that means number of executor = number of http calls, I can make. I can't boost more number of http calls in single executors, I mean - I can't go beyond the threashold of number of executors. On Thu, May 14, 2020 at 6:26 PM Sean Owen wrote: > Default is not 200, but the number of execut

Re: Calling HTTP Rest APIs from Spark Job

Default is not 200, but the number of executor slots. Yes you can only simultaneously execute as many tasks as slots regardless of partitions. On Thu, May 14, 2020, 5:19 PM Chetan Khatri wrote: > Thanks Sean, Jerry. > > Default Spark DataFrame partitions are 200 right? does it have > relationshi

Re: Calling HTTP Rest APIs from Spark Job

Thanks Sean, Jerry. Default Spark DataFrame partitions are 200 right? does it have relationship with number of cores? 8 cores - 4 workers. is not it like I can do only 8 * 4 = 32 http calls. Because in Spark number of partitions = number cores is untrue. Thanks On Thu, May 14, 2020 at 6:11 PM Se

Re: Calling HTTP Rest APIs from Spark Job

Yes any code that you write in code that you apply with Spark runs in the executors. You would be running as many HTTP clients as you have partitions. On Thu, May 14, 2020 at 4:31 PM Jerry Vinokurov wrote: > > I believe that if you do this within the context of an operation that is > already par

Re: Calling HTTP Rest APIs from Spark Job

I believe that if you do this within the context of an operation that is already parallelized such as a map, the work will be distributed to executors and they will do it in parallel. I could be wrong about this as I never investigated this specific use case, though. On Thu, May 14, 2020 at 5:24 P

Re: Calling HTTP Rest APIs from Spark Job

Thanks for the quick response. I am curious to know whether would it be parallel pulling data for 100+ HTTP request or it will only go on Driver node? the post body would be part of DataFrame. Think as I have a data frame of employee_id, employee_name now the http GET call has to be made for each

Re: Calling HTTP Rest APIs from Spark Job

Hi Chetan, You can pretty much use any client to do this. When I was using Spark at a previous job, we used OkHttp, but I'm sure there are plenty of others. In our case, we had a startup phase in which we gathered metadata via a REST API and then broadcast it to the workers. I think if you need al

Calling HTTP Rest APIs from Spark Job

Hi Spark Users, How can I invoke the Rest API call from Spark Code which is not only running on Spark Driver but distributed / parallel? Spark with Scala is my tech stack. Thanks