Johan Brodin created BEAM-1386: ---------------------------------- Summary: Job hangs without warnings after reading ~20GB of gz csv Key: BEAM-1386 URL: https://issues.apache.org/jira/browse/BEAM-1386 Project: Beam Issue Type: New Feature Components: sdk-py Affects Versions: 0.5.0 Environment: Running on Google Dataflow with 'n1-standard-8' machines. Reporter: Johan Brodin Assignee: Ahmet Altay
When running the job it works fine up until 20GB or around 23 million rows from a gzip:ed csv file (total size 43M rows). Halted the job so the statistic from it seam to disappeared, but here it is the id "2017-02-03_04_25_41-15296331815975218867". Is there any built in limitations to file size? Should I try to break the file up into several smaller files? Could the issue be related to the workers disk size? -- This message was sent by Atlassian JIRA (v6.3.15#6346)