Jim,
I tried to use Jolt for this but I found in the doc that if you try to
set an empty array or map to null or the empty string it will retain
the empty array or map (no idea why). Since you know the name of the
fields (and I assume want to keep the schema intact) you can use
ScriptedTransformRecord to get at those fields by name and set them to
null.
Regards,
Matt
On Mon, May 29, 2023 at 10:38 AM James McMahon <jsmcmah...@gmail.com> wrote:
>
> I have incoming JSON data that begins like this, and that I am trying to
> flatten with FlattenJSON v1.16.3:
>
> {
> "meta" : {
> "view" : {
> "id" : "kku6-nxdu",
> "name" : "Demographic Statistics By Zip Code",
> "assetType" : "dataset",
> "attribution" : "Department of Youth and Community Development (DYCD)",
> "averageRating" : 0,
> "category" : "City Government",
> "createdAt" : 1311775554,
> "description" : "Demographic statistics broken down by zip code",
> "displayType" : "table",
> "downloadCount" : 1017553,
> "hideFromCatalog" : false,
> "hideFromDataJson" : false,
> "indexUpdatedAt" : 1536596131,
> "newBackend" : true,
> "numberOfComments" : 3,
> "oid" : 4208790,
> "provenance" : "official",
> "publicationAppendEnabled" : false,
> "publicationDate" : 1372266760,
> "publicationGroup" : 238846,
> "publicationStage" : "published",
> "rowClass" : "",
> "rowsUpdatedAt" : 1372266747,
> "rowsUpdatedBy" : "uurm-7z6x",
> "tableId" : 942474,
> "totalTimesRated" : 0,
> "viewCount" : 70554,
> "viewLastModified" : 1652135219,
> "viewType" : "tabular",
> "approvals" : [ {
> "reviewedAt" : 1372266760,
> "reviewedAutomatically" : true,
> "state" : "approved",
> "submissionId" : 1064760,
> "submissionObject" : "public_audience_request",
> "submissionOutcome" : "change_audience",
> "submittedAt" : 1372266760,
> "workflowId" : 2285,
> "submissionDetails" : {
> "permissionType" : "READ"
> },
> "submissionOutcomeApplication" : {
> "failureCount" : 0,
> "status" : "success"
> },
> "submitter" : {
> "id" : "5fuc-pqz2",
> "displayName" : "NYC OpenData"
> }
> } ],
> "clientContext" : {
> "clientContextVariables" : [ ],
> "inheritedVariables" : { }
> },
> "columns" : [ {
> "id" : -1,
> "name" : "sid",
> "dataTypeName" : "meta_data",
> "fieldName" : ":sid",
> "position" : 0,
> "renderTypeName" : "meta_data",
> "format" : { },
> "flags" : [ "hidden" ]
> }, { .....
>
> This is my configuration of my FlattenJson processor:
> Separator .
> Flatten mode normal
> Ignore Reserved Characters false
> Return Type flatten
> Character Set UTF-8
> Pretty Print JSON true
>
> Those lines in red appear in my output like this:
> "meta.view.clientContext.clientContextVariables" : [ ],
> "meta.view.clientContext.inheritedVariables" : {
>
> },
>
> I don't want to preserve the empty list and empty map. I want to set the
> values for these keys to the empty string or null is acceptable.
>
> Can I do this natively through the FlattenJson configuration? If not, what
> would be the most efficient means to post-process to what I seek in my flow?
>
> Thanks in advance for any help.
