TrackStateByKey operation for Python

2016-06-03 Thread cmbendre
Hi, I need to build streaming application with Spark in Python since my codebase is in python. UpdateStateBykey is not able to scale with the data that i have. I got to know that the new api "trackStateByKey" is very efficient, but it is only available for Scala. Is there any way i can use

Spark UI only shows lines belonging to py4j lib

2016-05-06 Thread cmbendre
Hi, I am using spark 1.6.1 for my streaming jobs with Kinesis connector. Without any extra configurations put, the jobs run fine but they only show one line in UI. I used to see the actual line number in Python script in previous version. Please see the screenshot to understand what i mean.