Re: Lookup / Access of master data in spark streaming

2015-10-06 Thread Olivier Girardot
>it something can be done but should not be done ? >>> >>> Regards, >>> Amit >>> >>> From: Tathagata Das <t...@databricks.com> >>> Date: Thursday, April 9, 2015 at 3:13 PM >>> To: amit assudani <aassud...@impetus.co

Re: Lookup / Access of master data in spark streaming

2015-10-05 Thread Tathagata Das
be joined with 100s of Gbs of master data continuously. Is >>it something can be done but should not be done ? >> >> Regards, >> Amit >> >> From: Tathagata Das <t...@databricks.com> >> Date: Thursday, April 9, 2015 at 3:13 PM >> To: amit as

Lookup / Access of master data in spark streaming

2015-04-09 Thread Amit Assudani
Hi Friends, I am trying to solve a use case in spark streaming, I need help on getting to right approach on lookup / update the master data. Use case ( simplified ) I've a dataset of entity with three attributes and identifier/row key in a persistent store. Each attribute along with row key

Re: Lookup / Access of master data in spark streaming

2015-04-09 Thread Tathagata Das
Responses inline. Hope they help. On Thu, Apr 9, 2015 at 8:20 AM, Amit Assudani aassud...@impetus.com wrote: Hi Friends, I am trying to solve a use case in spark streaming, I need help on getting to right approach on lookup / update the master data. Use case ( simplified ) I’ve a

Re: Lookup / Access of master data in spark streaming

2015-04-09 Thread Amit Assudani
: Lookup / Access of master data in spark streaming Responses inline. Hope they help. On Thu, Apr 9, 2015 at 8:20 AM, Amit Assudani aassud...@impetus.commailto:aassud...@impetus.com wrote: Hi Friends, I am trying to solve a use case in spark streaming, I need help on getting to right approach