Re: 0.14 to 1.x Low-Level application migration questions

2019-06-05 Thread Thunder Stumpges
OK, I think we've figured out what is happening. SamzaContainer.apply() builds "serdeStreams calling StreamConfig.getSerdeStreams which uses StreamConfig.streamIdToSystemStream. In StreamConfig.streamIdToSystemStream the "stream" property of SystemStream is being set using getPhysicalName(). This m

Re: 0.14 to 1.x Low-Level application migration questions

2019-06-05 Thread Thunder Stumpges
Thanks, I think this all makes sense. I was able to get my application up and running with the exception of the Output SystemStream not finding its Serde. For a little more background, here's how we are set up: We are moving from YARN to standalone (running in Kubernetes). I think we have this pa

Re: 0.14 to 1.x Low-Level application migration questions

2019-06-03 Thread Prateek Maheshwari
Hi Thunder, I'm assuming you're talking about the low level (StreamTask) API here, since the High Level API has stronger requirements for I/O systems/streams. > How much IS picked up from config. All of the system, stream and store properties can still be specified in configuration. Properties sp

0.14 to 1.x Low-Level application migration questions

2019-06-01 Thread Thunder Stumpges
Hey guys, I'm following the guide here: http://samza.apache.org/releases/1.0.0 In step 3 it says: "In Samza 1.0, a Samza application’s input, output, and processing-task should be specified in code, rather than in config. " How much IS picked up from config? Will all the configuration of the sys