subject:"Spark streaming error \- Query terminated with exception\: assertion failed\: Invalid batch\: a#660,b#661L,c#662,d#663,,… 26 more fields \!= b#1291L"

Re: Spark streaming error - Query terminated with exception: assertion failed: Invalid batch: a#660,b#661L,c#662,d#663,,… 26 more fields != b#1291L

2019-04-01 Thread Shixiong(Ryan) Zhu

Could you try to use $”a” rather than df(“a”)? The latter one sometimes doesn’t work. On Thu, Mar 21, 2019 at 10:41 AM kineret M wrote: > I try to read a stream using my custom data source (v2, using spark 2.3), > and it fails *in the second iteration* with the following exception while >

Spark streaming error - Query terminated with exception: assertion failed: Invalid batch: a#660,b#661L,c#662,d#663,,… 26 more fields != b#1291L

2019-03-21 Thread kineret M

I try to read a stream using my custom data source (v2, using spark 2.3), and it fails *in the second iteration* with the following exception while reading prune columns:Query [id=xxx, runId=yyy] terminated with exception: assertion failed: Invalid batch: a#660,b#661L,c#662,d#663,,... 26 more