i must say i am disappointed in the errors, when i run the same code in a different project it works, so it must be some limitation that exists that i don't know chaim
On Tue, Jan 16, 2018 at 8:22 PM, Lukasz Cwik <[email protected]> wrote: > Look at the worker logs. This page shows how to log information and how to > find what was logged: > https://cloud.google.com/dataflow/pipelines/logging#cloud-dataflow-worker-log-example > The worker logs contain a lot of information written by Dataflow and also by > your code. Note that you may need to change log levels to get enough > information: > https://cloud.google.com/dataflow/pipelines/logging#SettingLevels > > Also good to take a look at this generic troubleshooting information: > https://cloud.google.com/dataflow/pipelines/troubleshooting-your-pipeline > > On Mon, Jan 15, 2018 at 12:18 AM, Chaim Turkel <[email protected]> wrote: >> >> Hi, >> I have a fairly simple pipeline that create daily snapshots of my >> data, and it sometimes fails, but the reason is not obvious: >> >> >> (863777e448a29a5c): Workflow failed. Causes: (863777e448a298ff): >> >> S41:Account_audit/BigQueryIO.Write/BatchLoads/SinglePartitionsReshuffle/GroupByKey/Read+Account_audit/BigQueryIO.Write/BatchLoads/SinglePartitionsReshuffle/GroupByKey/GroupByWindow+Account_audit/BigQueryIO.Write/BatchLoads/SinglePartitionsReshuffle/ExpandIterable+Account_audit/BigQueryIO.Write/BatchLoads/SinglePartitionWriteTables/ParMultiDo(WriteTables)+Account_audit/BigQueryIO.Write/BatchLoads/SinglePartitionWriteTables/ParMultiDo(WriteTables).WriteTablesMainOutput/extract >> table name >> +Account_audit/BigQueryIO.Write/BatchLoads/SinglePartitionWriteTables/ParMultiDo(WriteTables).WriteTablesMainOutput/count/GroupByKey+Account_audit/BigQueryIO.Write/BatchLoads/SinglePartitionWriteTables/ParMultiDo(WriteTables).WriteTablesMainOutput/count/Combine.GroupedValues/Partial+Account_audit/BigQueryIO.Write/BatchLoads/SinglePartitionWriteTables/ParMultiDo(WriteTables).WriteTablesMainOutput/count/GroupByKey/Reify+Account_audit/BigQueryIO.Write/BatchLoads/SinglePartitionWriteTables/ParMultiDo(WriteTables).WriteTablesMainOutput/count/GroupByKey/Write+Account_audit/BigQueryIO.Write/BatchLoads/SinglePartitionWriteTables/WithKeys/AddKeys/Map+Account_audit/BigQueryIO.Write/BatchLoads/SinglePartitionWriteTables/Window.Into()/Window.Assign+Account_audit/BigQueryIO.Write/BatchLoads/SinglePartitionWriteTables/GroupByKey/Reify+Account_audit/BigQueryIO.Write/BatchLoads/SinglePartitionWriteTables/GroupByKey/Write >> failed., (a412b0e93a586d57): A work item was attempted 4 times without >> success. Each time the worker eventually lost contact with the >> service. The work item was attempted on: >> dailysnapshotoptions-chai-01142358-ef6c-harness-qw6s, >> dailysnapshotoptions-chai-01142358-ef6c-harness-j09b, >> dailysnapshotoptions-chai-01142358-ef6c-harness-3t9m, >> dailysnapshotoptions-chai-01142358-ef6c-harness-372t >> >> is there any way to get more information? >> >> the job is: >> >> >> >> https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-01-14_23_58_54-6125672650598375925?project=ordinal-ember-163410&organizationId=782381653268 >> >> >> chaim >> >> -- >> >> >> Loans are funded by FinWise Bank, a Utah-chartered bank located in Sandy, >> Utah, member FDIC, Equal Opportunity Lender. Merchant Cash Advances are >> made by Behalf. For more information on ECOA, click here >> <https://www.behalf.com/legal/ecoa/>. For important information about >> opening a new account, review Patriot Act procedures here >> <https://www.behalf.com/legal/patriot/>. Visit Legal >> <https://www.behalf.com/legal/> to review our comprehensive program terms, >> conditions, and disclosures. > > -- Loans are funded by FinWise Bank, a Utah-chartered bank located in Sandy, Utah, member FDIC, Equal Opportunity Lender. Merchant Cash Advances are made by Behalf. For more information on ECOA, click here <https://www.behalf.com/legal/ecoa/>. For important information about opening a new account, review Patriot Act procedures here <https://www.behalf.com/legal/patriot/>. Visit Legal <https://www.behalf.com/legal/> to review our comprehensive program terms, conditions, and disclosures.
