Re: Custom Data Source for getting data from Rest based services

We extensively use pubmed & clinical trial databases for our work and it involves making large amount of parametric rest api queries, usually if the data download is large the requests get timed out ad we have to run queries in very small batches . We also extensively use large number(thousands)

Re: Custom Data Source for getting data from Rest based services

If you need Java code, you can have a look @: https://github.com/jgperrin/net.jgp.labs.spark.datasources and: https://databricks.com/session/extending-apache-sparks-ingestion-building-your-own-java-data-source

Re: Custom Data Source for getting data from Rest based services

Hi Sourav, Looks like this would be a good utility for the development of large scale data driven product based on Data services. We are an early stage startup called Climformatics and we are building a customized high resolution climate prediction tool. This effort requires synthesis of large

Re: Custom Data Source for getting data from Rest based services

It would be great if you can elaborate on the bulk provisioning use case. Regards, Sourav On Sun, Nov 26, 2017 at 11:53 PM, shankar.roy wrote: > This would be a useful feature. > We can leverage it while doing bulk provisioning. > > > > > -- > Sent from:

Re: Custom Data Source for getting data from Rest based services

@sathich Here are my thoughts on your points - 1. Yes this should be able to handle any complex json structure returned by the target rest API. Essentially what it would be returning is Rows of that complex structure. Then one can use Spark SQL to further flatten it using the functions like

Re: Custom Data Source for getting data from Rest based services

This would be a useful feature. We can leverage it while doing bulk provisioning. -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Custom Data Source for getting data from Rest based services

Hi Sourav, This is quite an useful addition to the spark family, this is a usecase that comes more often than talked about. * to get a 3rd party mapping data(geo coordinates) , * access database data through rest. * download data from from bulk data api service It will be really useful to

Custom Data Source for getting data from Rest based services

Hi All, Need your thoughts/inputs on a custom Data Source for accessing Rest based services in parallel using Spark. Many a times for business applications (batch oriented) one has to call a target Rest service for a high number of times (with different set of values of parameters/KV pairs).