If you're able to read the data in as a DataFrame, perhaps you can use a BroadcastHashJoin so that way you can join to that table presuming its small enough to distributed? Here's a handy guide on a BroadcastHashJoin: https://docs.cloud.databricks.com/docs/latest/databricks_guide/index.html#04%20SQL,%20DataFrames%20%26%20Datasets/05%20BroadcastHashJoin%20-%20scala.html
HTH! On Thu, Nov 3, 2016 at 8:53 AM Jain, Nishit <nja...@underarmour.com> wrote: > I have a lookup table in HANA database. I want to create a spark broadcast > variable for it. > What would be the suggested approach? Should I read it as an data frame > and convert data frame into broadcast variable? > > Thanks, > Nishit >
