Re: Unable to send JSON to BigQuery

2019-07-03 Thread Nicolas Delsaux
Well, if you take a look at my schema, the error is subtle, but obvious (once I've added the tests and modified the code). I've set "Consent" to be of typ "record", not "RECORD". Yes, it was a case issue. So I've modified code in BigQueryUtils to use uppercased type in all cases, AND an exce

Re: Unable to send JSON to BigQuery

2019-07-03 Thread Denes Arvay
Yes, and please attach the test cases too. Does this mean that your original issue hasn't been resolved yet by adding the "mode" fields? On Wed, Jul 3, 2019, 19:27 Nicolas Delsaux wrote: > So I have a simple test that replicate the bug. Do I have to open the > issue in Apache JIRA (I already hav

Re: Unable to send JSON to BigQuery

2019-07-03 Thread Nicolas Delsaux
So I have a simple test that replicate the bug. Do I have to open the issue in Apache JIRA (I already have access to) ? Le 03/07/2019 à 11:28, Denes Arvay a écrit : Hi Nicolas, It seems that NiFi expects to have the "mode" field being present, even though based on the BigQuery doc [1] it's op

Re: Unable to send JSON to BigQuery

2019-07-03 Thread Nicolas Delsaux
I'm ivnestigating the same way. I've added the mode field everywhere, but still have the issue. I'll try to create a minimal reproducing schema for your ticket (by running unit tests) Le 03/07/2019 à 11:28, Denes Arvay a écrit : Hi Nicolas, It seems that NiFi expects to have the "mode" fiel

Re: Unable to send JSON to BigQuery

2019-07-03 Thread Denes Arvay
Hi Nicolas, It seems that NiFi expects to have the "mode" field being present, even though based on the BigQuery doc [1] it's optional. I'd suggest trying adding it to every name-type pair with its default value "NULLABLE". (i.e. { "name": "Consent", "type": "record", *"mode": "NULLABLE"*, "field

Unable to send JSON to BigQuery

2019-07-03 Thread Nicolas Delsaux
 I'm using Apache Nifi 1.9.2 and trying to post JSON content to a BigQuery table. There seems to be something wrong, sicne I get 2019-07-03 08:35:24,964 ERROR [Timer-Driven Process Thread-8] o.a.n.p.gcp.bigquery.PutBigQueryBatch PutBigQueryBatch[id=b2b1c6bf-016b-1000-e8c9-b3f9fb5b417e] null: