Hi,
We are currently working on a solution architecture to solve IoT workloads on
Spark. Therefore, I am interested in getting to know whether it is considered
an Anti-Pattern in Spark to get records from a database and make a ReST call to
an external server with that data. This external server may and will be the
bottleneck -- but from a Spark point of view: is it possibly harmful to open
connections and wait for their responses for vast amounts of rows?
In the same manner: is calling an external library (instead of making a ReST
call) for any row possibly problematic?
How to rather embed a C++ library in this workflow: is it best to make a
function having a JNI call to run it natively -- iff we know we are single
threaded then? Or is there a better way to include C++ code in Spark jobs?
Many thanks in advance.
Kind regards, Daniel.
