[jira] [Commented] (NIFI-4269) GetMongo should be able to provide pure JSON as well as "extended JSON"
[ https://issues.apache.org/jira/browse/NIFI-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155945#comment-16155945 ] ASF GitHub Bot commented on NIFI-4269: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2063 > GetMongo should be able to provide pure JSON as well as "extended JSON" > --- > > Key: NIFI-4269 > URL: https://issues.apache.org/jira/browse/NIFI-4269 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen > Fix For: 1.4.0 > > > GetMongo currently only supports "extended JSON." That results in ugly JSON > like this: > { > "timestamp": { > "$date": LONG_TIMESTAMP > } > } > instead of something cleaner like this: > { > "timestamp": "2017-08-07T00:00:00Z" > } > GetMongo should be refactored to support both modes of JSON serialization. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4269) GetMongo should be able to provide pure JSON as well as "extended JSON"
[ https://issues.apache.org/jira/browse/NIFI-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155941#comment-16155941 ] ASF GitHub Bot commented on NIFI-4269: -- Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/2063 +1 LGTM, built and ran unit tests, and tried with a live MongoDB instance, verified both JSON modes produce the expected output. Thanks for the improvement! Merging to master > GetMongo should be able to provide pure JSON as well as "extended JSON" > --- > > Key: NIFI-4269 > URL: https://issues.apache.org/jira/browse/NIFI-4269 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen > > GetMongo currently only supports "extended JSON." That results in ugly JSON > like this: > { > "timestamp": { > "$date": LONG_TIMESTAMP > } > } > instead of something cleaner like this: > { > "timestamp": "2017-08-07T00:00:00Z" > } > GetMongo should be refactored to support both modes of JSON serialization. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4269) GetMongo should be able to provide pure JSON as well as "extended JSON"
[ https://issues.apache.org/jira/browse/NIFI-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155943#comment-16155943 ] ASF subversion and git services commented on NIFI-4269: --- Commit 527ce0b4efe7e23f95f89f57ace7944aafee8df0 in nifi's branch refs/heads/master from [~mike.thomsen] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=527ce0b ] NIFI-4269 Added the ability to serialize Mongo documents to a clean JSON view instead of just extended JSON. NIFI-4269 incorporated changes from the code review. Signed-off-by: Matthew BurgessThis closes #2063 > GetMongo should be able to provide pure JSON as well as "extended JSON" > --- > > Key: NIFI-4269 > URL: https://issues.apache.org/jira/browse/NIFI-4269 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen > > GetMongo currently only supports "extended JSON." That results in ugly JSON > like this: > { > "timestamp": { > "$date": LONG_TIMESTAMP > } > } > instead of something cleaner like this: > { > "timestamp": "2017-08-07T00:00:00Z" > } > GetMongo should be refactored to support both modes of JSON serialization. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4269) GetMongo should be able to provide pure JSON as well as "extended JSON"
[ https://issues.apache.org/jira/browse/NIFI-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155944#comment-16155944 ] ASF subversion and git services commented on NIFI-4269: --- Commit 527ce0b4efe7e23f95f89f57ace7944aafee8df0 in nifi's branch refs/heads/master from [~mike.thomsen] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=527ce0b ] NIFI-4269 Added the ability to serialize Mongo documents to a clean JSON view instead of just extended JSON. NIFI-4269 incorporated changes from the code review. Signed-off-by: Matthew BurgessThis closes #2063 > GetMongo should be able to provide pure JSON as well as "extended JSON" > --- > > Key: NIFI-4269 > URL: https://issues.apache.org/jira/browse/NIFI-4269 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen > > GetMongo currently only supports "extended JSON." That results in ugly JSON > like this: > { > "timestamp": { > "$date": LONG_TIMESTAMP > } > } > instead of something cleaner like this: > { > "timestamp": "2017-08-07T00:00:00Z" > } > GetMongo should be refactored to support both modes of JSON serialization. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4269) GetMongo should be able to provide pure JSON as well as "extended JSON"
[ https://issues.apache.org/jira/browse/NIFI-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153588#comment-16153588 ] ASF GitHub Bot commented on NIFI-4269: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2063 @mattyb149 Do you think you'll have a chance to review this this week? > GetMongo should be able to provide pure JSON as well as "extended JSON" > --- > > Key: NIFI-4269 > URL: https://issues.apache.org/jira/browse/NIFI-4269 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen > > GetMongo currently only supports "extended JSON." That results in ugly JSON > like this: > { > "timestamp": { > "$date": LONG_TIMESTAMP > } > } > instead of something cleaner like this: > { > "timestamp": "2017-08-07T00:00:00Z" > } > GetMongo should be refactored to support both modes of JSON serialization. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4269) GetMongo should be able to provide pure JSON as well as "extended JSON"
[ https://issues.apache.org/jira/browse/NIFI-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145666#comment-16145666 ] ASF GitHub Bot commented on NIFI-4269: -- Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/2063 Will take a look this week, working towards a hard deadline at the moment :) > GetMongo should be able to provide pure JSON as well as "extended JSON" > --- > > Key: NIFI-4269 > URL: https://issues.apache.org/jira/browse/NIFI-4269 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen > > GetMongo currently only supports "extended JSON." That results in ugly JSON > like this: > { > "timestamp": { > "$date": LONG_TIMESTAMP > } > } > instead of something cleaner like this: > { > "timestamp": "2017-08-07T00:00:00Z" > } > GetMongo should be refactored to support both modes of JSON serialization. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4269) GetMongo should be able to provide pure JSON as well as "extended JSON"
[ https://issues.apache.org/jira/browse/NIFI-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145118#comment-16145118 ] ASF GitHub Bot commented on NIFI-4269: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2063 @mattyb149 Can you take a look? Thanks. > GetMongo should be able to provide pure JSON as well as "extended JSON" > --- > > Key: NIFI-4269 > URL: https://issues.apache.org/jira/browse/NIFI-4269 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen > > GetMongo currently only supports "extended JSON." That results in ugly JSON > like this: > { > "timestamp": { > "$date": LONG_TIMESTAMP > } > } > instead of something cleaner like this: > { > "timestamp": "2017-08-07T00:00:00Z" > } > GetMongo should be refactored to support both modes of JSON serialization. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4269) GetMongo should be able to provide pure JSON as well as "extended JSON"
[ https://issues.apache.org/jira/browse/NIFI-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138325#comment-16138325 ] ASF GitHub Bot commented on NIFI-4269: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2063 @mattyb149 Ok, changes committed and at least the French build ran to completion. > GetMongo should be able to provide pure JSON as well as "extended JSON" > --- > > Key: NIFI-4269 > URL: https://issues.apache.org/jira/browse/NIFI-4269 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen > > GetMongo currently only supports "extended JSON." That results in ugly JSON > like this: > { > "timestamp": { > "$date": LONG_TIMESTAMP > } > } > instead of something cleaner like this: > { > "timestamp": "2017-08-07T00:00:00Z" > } > GetMongo should be refactored to support both modes of JSON serialization. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4269) GetMongo should be able to provide pure JSON as well as "extended JSON"
[ https://issues.apache.org/jira/browse/NIFI-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137637#comment-16137637 ] ASF GitHub Bot commented on NIFI-4269: -- Github user MikeThomsen commented on a diff in the pull request: https://github.com/apache/nifi/pull/2063#discussion_r134629298 --- Diff: nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/GetMongo.java --- @@ -151,17 +172,34 @@ public ValidationResult validate(final String subject, final String value, final return propertyDescriptors; } -private ObjectMapper mapper = new ObjectMapper(); +private ObjectMapper mapper; //Turn a list of Mongo result documents into a String representation of a JSON array -private String buildBatch(List documents) throws IOException { -List docs = new ArrayList<>(); -for (Document document : documents) { -String asJson = document.toJson(); -docs.add(mapper.readValue(asJson, Map.class)); +private String buildBatch(List documents, String jsonTypeSetting) throws IOException { +StringBuilder builder = new StringBuilder(); +for (int index = 0; index < documents.size(); index++) { +Document document = documents.get(index); +String asJson; +if (jsonTypeSetting.equals(JSON_TYPE_NORMAL)) { +asJson = mapper.writerWithDefaultPrettyPrinter().writeValueAsString(document); --- End diff -- You know, I actually think this might have not worked for me so I'm going to remove it for now. > GetMongo should be able to provide pure JSON as well as "extended JSON" > --- > > Key: NIFI-4269 > URL: https://issues.apache.org/jira/browse/NIFI-4269 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen > > GetMongo currently only supports "extended JSON." That results in ugly JSON > like this: > { > "timestamp": { > "$date": LONG_TIMESTAMP > } > } > instead of something cleaner like this: > { > "timestamp": "2017-08-07T00:00:00Z" > } > GetMongo should be refactored to support both modes of JSON serialization. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4269) GetMongo should be able to provide pure JSON as well as "extended JSON"
[ https://issues.apache.org/jira/browse/NIFI-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137246#comment-16137246 ] ASF GitHub Bot commented on NIFI-4269: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2063 Thanks for the feedback @mattyb149 . I'll start working on these shortly (got some other stuff I have to do at the moment). > GetMongo should be able to provide pure JSON as well as "extended JSON" > --- > > Key: NIFI-4269 > URL: https://issues.apache.org/jira/browse/NIFI-4269 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen > > GetMongo currently only supports "extended JSON." That results in ugly JSON > like this: > { > "timestamp": { > "$date": LONG_TIMESTAMP > } > } > instead of something cleaner like this: > { > "timestamp": "2017-08-07T00:00:00Z" > } > GetMongo should be refactored to support both modes of JSON serialization. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4269) GetMongo should be able to provide pure JSON as well as "extended JSON"
[ https://issues.apache.org/jira/browse/NIFI-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136827#comment-16136827 ] ASF GitHub Bot commented on NIFI-4269: -- Github user mattyb149 commented on a diff in the pull request: https://github.com/apache/nifi/pull/2063#discussion_r134485578 --- Diff: nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/GetMongo.java --- @@ -120,12 +123,30 @@ public ValidationResult validate(final String subject, final String value, final .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR) .build(); +static final String JSON_TYPE_EXTENDED = "Extended"; +static final String JSON_TYPE_NORMAL = "Normal"; +static final AllowableValue JSON_EXTENDED = new AllowableValue(JSON_TYPE_EXTENDED, "Extended JSON", --- End diff -- Perhaps replace the developer-specific doc (toJson() from the Java driver, e.g.) with user-level info about [MongoDB Extended JSON](https://docs.mongodb.com/manual/reference/mongodb-extended-json/)? This is a suggestion not a requirement :) > GetMongo should be able to provide pure JSON as well as "extended JSON" > --- > > Key: NIFI-4269 > URL: https://issues.apache.org/jira/browse/NIFI-4269 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen > > GetMongo currently only supports "extended JSON." That results in ugly JSON > like this: > { > "timestamp": { > "$date": LONG_TIMESTAMP > } > } > instead of something cleaner like this: > { > "timestamp": "2017-08-07T00:00:00Z" > } > GetMongo should be refactored to support both modes of JSON serialization. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4269) GetMongo should be able to provide pure JSON as well as "extended JSON"
[ https://issues.apache.org/jira/browse/NIFI-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136828#comment-16136828 ] ASF GitHub Bot commented on NIFI-4269: -- Github user mattyb149 commented on a diff in the pull request: https://github.com/apache/nifi/pull/2063#discussion_r134486575 --- Diff: nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/GetMongo.java --- @@ -120,12 +123,30 @@ public ValidationResult validate(final String subject, final String value, final .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR) .build(); +static final String JSON_TYPE_EXTENDED = "Extended"; +static final String JSON_TYPE_NORMAL = "Normal"; +static final AllowableValue JSON_EXTENDED = new AllowableValue(JSON_TYPE_EXTENDED, "Extended JSON", +"Use MongoDB's \"extended JSON\". This is the JSON generated with toJson() on a MongoDB Document from the Java driver"); +static final AllowableValue JSON_NORMAL = new AllowableValue(JSON_TYPE_NORMAL, "Normal JSON", +"Use Jackson to convert the MongoDB Document into a representation that uses only standard JSON"); +static final PropertyDescriptor JSON_TYPE = new PropertyDescriptor.Builder() +.allowableValues(JSON_EXTENDED, JSON_NORMAL) +.defaultValue(JSON_TYPE_EXTENDED) +.displayName("JSON Type") +.name("json-type") +.description("By default, MongoDB's Java driver returns \"extended JSON\". Some of the features of this variant of JSON" + +" may cause problems for other JSON parsers that expect only standard JSON types and conventions. This configuration setting " + +" controls whether to use extended JSON or provide a clean view that conforms to standard JSON.") +.expressionLanguageSupported(false) +.build(); --- End diff -- Another nitpick :) Although there are only two allowable values and the default is one of them, for consistency this field should be required(true). Also do you think it would be helpful to add some/all of this description to the processor's CapabilityDescription too? That description is pretty thin, and this is a nice added feature so might be good to draw more attention to it :) > GetMongo should be able to provide pure JSON as well as "extended JSON" > --- > > Key: NIFI-4269 > URL: https://issues.apache.org/jira/browse/NIFI-4269 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen > > GetMongo currently only supports "extended JSON." That results in ugly JSON > like this: > { > "timestamp": { > "$date": LONG_TIMESTAMP > } > } > instead of something cleaner like this: > { > "timestamp": "2017-08-07T00:00:00Z" > } > GetMongo should be refactored to support both modes of JSON serialization. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4269) GetMongo should be able to provide pure JSON as well as "extended JSON"
[ https://issues.apache.org/jira/browse/NIFI-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136829#comment-16136829 ] ASF GitHub Bot commented on NIFI-4269: -- Github user mattyb149 commented on a diff in the pull request: https://github.com/apache/nifi/pull/2063#discussion_r134487464 --- Diff: nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/GetMongo.java --- @@ -151,17 +172,34 @@ public ValidationResult validate(final String subject, final String value, final return propertyDescriptors; } -private ObjectMapper mapper = new ObjectMapper(); +private ObjectMapper mapper; //Turn a list of Mongo result documents into a String representation of a JSON array -private String buildBatch(List documents) throws IOException { -List docs = new ArrayList<>(); -for (Document document : documents) { -String asJson = document.toJson(); -docs.add(mapper.readValue(asJson, Map.class)); +private String buildBatch(List documents, String jsonTypeSetting) throws IOException { +StringBuilder builder = new StringBuilder(); +for (int index = 0; index < documents.size(); index++) { +Document document = documents.get(index); +String asJson; +if (jsonTypeSetting.equals(JSON_TYPE_NORMAL)) { +asJson = mapper.writerWithDefaultPrettyPrinter().writeValueAsString(document); --- End diff -- Should there be a third JSON Type option of ("Standard Pretty-Printed JSON")? For speed and/or size concerns, we wouldn't want to force pretty-printing, but it might be nice as an option. An additional property is probably overkill, especially if the Extended JSON output isn't pretty-printed, but if they could each be pretty printed, then perhaps an additional property (defaulted to false) is the right way to go. > GetMongo should be able to provide pure JSON as well as "extended JSON" > --- > > Key: NIFI-4269 > URL: https://issues.apache.org/jira/browse/NIFI-4269 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen > > GetMongo currently only supports "extended JSON." That results in ugly JSON > like this: > { > "timestamp": { > "$date": LONG_TIMESTAMP > } > } > instead of something cleaner like this: > { > "timestamp": "2017-08-07T00:00:00Z" > } > GetMongo should be refactored to support both modes of JSON serialization. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4269) GetMongo should be able to provide pure JSON as well as "extended JSON"
[ https://issues.apache.org/jira/browse/NIFI-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135968#comment-16135968 ] ASF GitHub Bot commented on NIFI-4269: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2063 @mattyb149 @pvillard31 @bbende @joewitt Could one of you take a look at this and give me some feedback? I can fix the merge conflicts that came up since I submitted this. > GetMongo should be able to provide pure JSON as well as "extended JSON" > --- > > Key: NIFI-4269 > URL: https://issues.apache.org/jira/browse/NIFI-4269 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen > > GetMongo currently only supports "extended JSON." That results in ugly JSON > like this: > { > "timestamp": { > "$date": LONG_TIMESTAMP > } > } > instead of something cleaner like this: > { > "timestamp": "2017-08-07T00:00:00Z" > } > GetMongo should be refactored to support both modes of JSON serialization. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4269) GetMongo should be able to provide pure JSON as well as "extended JSON"
[ https://issues.apache.org/jira/browse/NIFI-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16116711#comment-16116711 ] ASF GitHub Bot commented on NIFI-4269: -- GitHub user MikeThomsen opened a pull request: https://github.com/apache/nifi/pull/2063 NIFI-4269 Added the ability to serialize Mongo documents to a clean J… …SON view instead of just extended JSON. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/MikeThomsen/nifi NIFI-4269 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2063.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2063 commit 4363a0d82e44966227033962130c51c659ee7021 Author: Mike ThomsenDate: 2017-08-07T14:59:45Z NIFI-4269 Added the ability to serialize Mongo documents to a clean JSON view instead of just extended JSON. > GetMongo should be able to provide pure JSON as well as "extended JSON" > --- > > Key: NIFI-4269 > URL: https://issues.apache.org/jira/browse/NIFI-4269 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen > > GetMongo currently only supports "extended JSON." That results in ugly JSON > like this: > { > "timestamp": { > "$date": LONG_TIMESTAMP > } > } > instead of something cleaner like this: > { > "timestamp": "2017-08-07T00:00:00Z" > } > GetMongo should be refactored to support both modes of JSON serialization. -- This message was sent by Atlassian JIRA (v6.4.14#64029)