Each JSON object needs to be on a single line since this is the boundary
the TextFileInputFormat uses when splitting up large files.
On Wed, Dec 24, 2014 at 12:34 PM, elliott cordo elliottco...@gmail.com
wrote:
I have generally been impressed with the way jsonFile eats just about
any json data model.. but getting this error when i try to ingest this
file: Unexpected close marker ']': expected '}
Here are the commands from the pyspark shell:
from pyspark.sql import HiveContext
hiveContext = HiveContext(sc)
f = hiveContext.jsonFile(sample.json)
Here is some sample json:
{wf_session: [
{id:6021fb91-c9ec-4019-9ab9-f628aee8d259,machine_id:b45c8c4a-7e8e-442d-8d49-fb7c32e2d813,session_id:d65ca338-c6b8-4bff-93b1-7f2364726fb7,event_at:2014-12-19T15:55:31.373Z,screen:x,type:1,time_left_secs:1},
{id:7e696c19-3ba4-4469-be28-5ef1f0c03d63,machine_id:b45c8c4a-7e8e-442d-8d49-fb7c32e2d813,session_id:d65ca338-c6b8-4bff-93b1-7f2364726fb7,event_at:2014-12-19T15:55:32.385Z,screen:x,type:2,ad_unit_id:null,spot_started_at:2014-12-19T15:55:12.364Z,spot_ended_at:2014-12-19T15:55:32.385Z,spot_duration_secs:20,impression_count:0,impressions:[],engagement_count:0,engagements:[]},
{id:68a43006-09bc-4c18-af55-1ebdc0e041a3,machine_id:b45c8c4a-7e8e-442d-8d49-fb7c32e2d813,session_id:d65ca338-c6b8-4bff-93b1-7f2364726fb7,event_at:2014-12-19T15:55:32.375Z,screen:x,type:3,duration_secs:20,to_ad_unit_id:developmentbea1f3a4-be08-4119-b9f4-7}
] }
Any help would be appreciated! :) Merry Xmas!
