[GitHub] nifi pull request #3193: NIFI-5854 Added TimeUnit enhancements (microseconds...
GitHub user alopresto opened a pull request: https://github.com/apache/nifi/pull/3193 NIFI-5854 Added TimeUnit enhancements (microseconds, decimal parsing) Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/alopresto/nifi NIFI-5854 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3193.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3193 commit 1d4ecd3e4c078e67a6a04c1ffda48d635cf66fb7 Author: Andy LoPresto Date: 2018-11-30T04:17:58Z NIFI-5854 Added skeleton logic to convert decimal time units. Added helper methods. Added unit tests. commit 50b7b6436454b3b0d7e36fd1f2139dd1f10b9067 Author: Andy LoPresto Date: 2018-11-30T04:25:04Z NIFI-5854 [WIP] Cleaned up logic. Resolved failing unit tests due to error message change. commit fc16cf37439e1bee5afe58adbfe61c31bc77df3f Author: Andy LoPresto Date: 2018-11-30T04:48:21Z NIFI-5854 [WIP] All helper method unit tests pass. commit a0865667f03ad9e1a0ad46277045fa364720cd98 Author: Andy LoPresto Date: 2018-11-30T05:15:27Z NIFI-5854 [WIP] FormatUtils#getPreciseTimeDuration() now handles all tested inputs correctly. Added unit tests. commit d2031d5a06713b69f1323215afe87efc8d93428f Author: Andy LoPresto Date: 2018-11-30T19:39:34Z NIFI-5854 [WIP] FormatUtils#getTimeDuration() still using long. Added unit tests. Renamed existing unit tests to reflect method under test. commit ed51f48a3bee3750864b1415b5e85b42422ecaaf Author: Andy LoPresto Date: 2018-11-30T20:13:57Z NIFI-5854 FormatUtils#getTimeDuration() returns long but now accepts decimal inputs. Added @Deprecation warnings (will update callers where possible). All unit tests pass. ---
[GitHub] nifi-minifi-cpp issue #445: MINIFICPP-681 - Add content hash processor
Github user alopresto commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/445 You can also look at [`CryptographicHashContent`](https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/CryptographicHashContent.java) and [`HashService`](https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/security/util/crypto/HashService.java) in NiFi to see how these actions are currently handled. ---
[GitHub] nifi-registry pull request #148: NIFIREG-211 Initial work for adding extenio...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236898636
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
--- End diff --
Ok. Is there any way to craft an instance of a binary with different
metadata that generates the same checksum? Can you point me to the code which
performs this inspection and calculation? Where does the checksum reside in
transmission (i.e. the binary and checksum are sent separately, or it's just
calculated on one end and there is nothing to compare it to?)?
---
[GitHub] nifi-registry pull request #148: NIFIREG-211 Initial work for adding extenio...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236860010
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
+
+
+@ApiModelProperty(value = "The id of this version of the extension
bundle")
+public String getId() {
+return id;
+}
+
+public void setId(String id) {
+this.id = id;
+}
+
+@ApiModelProperty(value = "The id of the extension bundle this version
is for")
+public String getExtensionBundleId() {
+return extensionBundleId;
+}
+
+public void setExtensionBundleId(String extensionBundleId) {
+this.extensionBundleId = extensionBundleId;
+}
+
+@ApiModelProperty(value = "The id of the bucket the extension bundle
belongs to", required = true)
+public String getBucketId() {
+return bucketId;
+}
+
+public void setBucketId(String bucketId) {
+this.bucketId = bucketId;
+}
+
+@ApiModelProperty(value = "The version of the extension bundle")
+public String getVersion() {
+return version;
+}
+
+public void setVersion(String version) {
+this.version = version;
+}
+
+@ApiModelProperty(value = "The optional bundle dependency (i.e.
another bundle this bundle is dependent on)")
+public ExtensionBundleVersionDependency getDependency() {
+return dependency;
+}
+
+public void setDependency(ExtensionBundleVersionDependency dependency)
{
+this.dependency = dependency;
+}
+
+@ApiModelProperty(value = "The timestamp of the create date of this
version")
+public long getTimestamp() {
+return timestamp;
+}
+
+public void setTimestamp(long timestamp) {
+this.timestamp = timestamp;
+}
+
+@ApiModelProperty(value = "The identity that created this version")
+public String getAuthor() {
--- End diff --
Understood. I think that long-term, we probably need to re-evaluate how
that is being done, as the identities between NiFi and NiFi Registry may not
always match.
---
[GitHub] nifi-registry pull request #148: NIFIREG-211 Initial work for adding extenio...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236860247
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
+
+
+@ApiModelProperty(value = "The id of this version of the extension
bundle")
+public String getId() {
+return id;
+}
+
+public void setId(String id) {
+this.id = id;
+}
+
+@ApiModelProperty(value = "The id of the extension bundle this version
is for")
+public String getExtensionBundleId() {
+return extensionBundleId;
+}
+
+public void setExtensionBundleId(String extensionBundleId) {
+this.extensionBundleId = extensionBundleId;
+}
+
+@ApiModelProperty(value = "The id of the bucket the extension bundle
belongs to", required = true)
+public String getBucketId() {
+return bucketId;
+}
+
+public void setBucketId(String bucketId) {
+this.bucketId = bucketId;
+}
+
+@ApiModelProperty(value = "The version of the extension bundle")
+public String getVersion() {
+return version;
+}
+
+public void setVersion(String version) {
+this.version = version;
+}
+
+@ApiModelProperty(value = "The optional bundle dependency (i.e.
another bundle this bundle is dependent on)")
+public ExtensionBundleVersionDependency getDependency() {
+return dependency;
+}
+
+public void setDependency(ExtensionBundleVersionDependency dependency)
{
+this.dependency = dependency;
+}
+
+@ApiModelProperty(value = "The timestamp of the create date of this
version")
+public long getTimestamp() {
+return timestamp;
+}
+
+public void setTimestamp(long timestamp) {
+this.timestamp = timestamp;
+}
+
+@ApiModelProperty(value = "The identity that created this version")
+public String getAuthor() {
+return author;
+}
+
+public void setAuthor(String author) {
+this.author = author;
+}
+
+@ApiModelProperty(value = "The description for this version")
+public String getDescription() {
+return description;
+}
+
+public void setDescription(String description) {
+this.description = description;
+}
+
+@ApiModelProperty(value = "The hex representation of the SHA-256
digest of the binary content for this version")
+public String getSha256Hex() {
+return sha256Hex;
+}
+
+public void setSha256Hex(String sha256Hex) {
[GitHub] nifi-registry pull request #148: NIFIREG-211 Initial work for adding extenio...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236859763
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
--- End diff --
I would like to have a conversation around the cryptographic signatures so
if it's possible to add a new field fairly easily, I'm ok waiting for this to
be in. I am curious if people think there should be any checksum coverage on
the metadata for the extension as well. I like having a calculation over the
binary, but I also think there is value in ensuring the metadata values are
checked as well. Looking for feedback, but this should be independent from this
PR.
---
[GitHub] nifi-registry pull request #148: NIFIREG-211 Initial work for adding extenio...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236840513
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
--- End diff --
How easy is it to add a new field here? I'd like to propose storage of a
cryptographic signature of some sort, in addition to the SHA-256 checksum,
which can be modified (generated over new contents) by a malicious user. What
inputs (data, fields, encoding, etc.) does the SHA-256 cover?
---
[GitHub] nifi-registry pull request #148: NIFIREG-211 Initial work for adding extenio...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236840020
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
+
+
+@ApiModelProperty(value = "The id of this version of the extension
bundle")
+public String getId() {
+return id;
+}
+
+public void setId(String id) {
+this.id = id;
+}
+
+@ApiModelProperty(value = "The id of the extension bundle this version
is for")
+public String getExtensionBundleId() {
+return extensionBundleId;
+}
+
+public void setExtensionBundleId(String extensionBundleId) {
+this.extensionBundleId = extensionBundleId;
+}
+
+@ApiModelProperty(value = "The id of the bucket the extension bundle
belongs to", required = true)
+public String getBucketId() {
+return bucketId;
+}
+
+public void setBucketId(String bucketId) {
+this.bucketId = bucketId;
+}
+
+@ApiModelProperty(value = "The version of the extension bundle")
+public String getVersion() {
+return version;
+}
+
+public void setVersion(String version) {
+this.version = version;
+}
+
+@ApiModelProperty(value = "The optional bundle dependency (i.e.
another bundle this bundle is dependent on)")
+public ExtensionBundleVersionDependency getDependency() {
+return dependency;
+}
+
+public void setDependency(ExtensionBundleVersionDependency dependency)
{
+this.dependency = dependency;
+}
+
+@ApiModelProperty(value = "The timestamp of the create date of this
version")
+public long getTimestamp() {
+return timestamp;
+}
+
+public void setTimestamp(long timestamp) {
+this.timestamp = timestamp;
+}
+
+@ApiModelProperty(value = "The identity that created this version")
+public String getAuthor() {
+return author;
+}
+
+public void setAuthor(String author) {
+this.author = author;
+}
+
+@ApiModelProperty(value = "The description for this version")
+public String getDescription() {
+return description;
+}
+
+public void setDescription(String description) {
+this.description = description;
+}
+
+@ApiModelProperty(value = "The hex representation of the SHA-256
digest of the binary content for this version")
+public String getSha256Hex() {
+return sha256Hex;
+}
+
+public void setSha256Hex(String sha256Hex) {
[GitHub] nifi-registry pull request #148: NIFIREG-211 Initial work for adding extenio...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/148#discussion_r236839456
--- Diff:
nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/extension/ExtensionBundleVersionMetadata.java
---
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.registry.link.LinkableEntity;
+
+import javax.validation.constraints.Min;
+import javax.validation.constraints.NotBlank;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.Objects;
+
+@ApiModel
+@XmlRootElement
+public class ExtensionBundleVersionMetadata extends LinkableEntity
implements Comparable {
+
+@NotBlank
+private String id;
+
+@NotBlank
+private String extensionBundleId;
+
+@NotBlank
+private String bucketId;
+
+@NotBlank
+private String version;
+
+private ExtensionBundleVersionDependency dependency;
+
+@Min(1)
+private long timestamp;
+
+@NotBlank
+private String author;
+
+private String description;
+
+@NotBlank
+private String sha256Hex;
+
+
+@ApiModelProperty(value = "The id of this version of the extension
bundle")
+public String getId() {
+return id;
+}
+
+public void setId(String id) {
+this.id = id;
+}
+
+@ApiModelProperty(value = "The id of the extension bundle this version
is for")
+public String getExtensionBundleId() {
+return extensionBundleId;
+}
+
+public void setExtensionBundleId(String extensionBundleId) {
+this.extensionBundleId = extensionBundleId;
+}
+
+@ApiModelProperty(value = "The id of the bucket the extension bundle
belongs to", required = true)
+public String getBucketId() {
+return bucketId;
+}
+
+public void setBucketId(String bucketId) {
+this.bucketId = bucketId;
+}
+
+@ApiModelProperty(value = "The version of the extension bundle")
+public String getVersion() {
+return version;
+}
+
+public void setVersion(String version) {
+this.version = version;
+}
+
+@ApiModelProperty(value = "The optional bundle dependency (i.e.
another bundle this bundle is dependent on)")
+public ExtensionBundleVersionDependency getDependency() {
+return dependency;
+}
+
+public void setDependency(ExtensionBundleVersionDependency dependency)
{
+this.dependency = dependency;
+}
+
+@ApiModelProperty(value = "The timestamp of the create date of this
version")
+public long getTimestamp() {
+return timestamp;
+}
+
+public void setTimestamp(long timestamp) {
+this.timestamp = timestamp;
+}
+
+@ApiModelProperty(value = "The identity that created this version")
+public String getAuthor() {
--- End diff --
Understanding that "authors" can take many forms, is this a standardized
String representation, or could it be "alopresto", "Andy LoPresto", and "Andy
LoPresto " are all valid and used simultaneously, causing
lookup/resolution issues later? Would using a custom datatype here make sense?
---
[GitHub] nifi pull request #3180: NIFI-5833 Treat GetTwitter API properties as sensit...
GitHub user alopresto opened a pull request: https://github.com/apache/nifi/pull/3180 NIFI-5833 Treat GetTwitter API properties as sensitive Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/alopresto/nifi NIFI-5833 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3180.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3180 commit 9c051560f801867d5fb5c70a16813f990ee54f4f Author: Andy LoPresto Date: 2018-11-21T02:15:40Z NIFI-5833 Marked GetTwitter Consumer Key and Access Token processor properties as sensitive. commit 32a4f3187120dedde6d89f3d02747079e24d1bbd Author: Andy LoPresto Date: 2018-11-21T04:32:08Z NIFI-5833 Added unit test to demonstrate arbitrary decryption of sensitive values regardless of processor property sensitive status. commit 9d1e2be41801703478a5d937a10f7829b89f3252 Author: Andy LoPresto Date: 2018-11-21T04:48:50Z NIFI-5833 Updated GetTwitter documentation with note about 1.9.0+ marking Consumer Key and Access Token as sensitive. ---
[GitHub] nifi issue #3137: NIFI-5797 : EscapedJava for FlattenJson
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/3137 Can you please add some explanation for why this change is submitted, documentation, and at least one unit test? Thank you. ---
[GitHub] nifi issue #3024: NIFI-5595 - Added the CORS filter to the templates/upload ...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/3024 Verified by sending curl POST commands with a client certificate to upload a template. With the `Origin` header set to a remote domain, the request is blocked with the response "Invalid CORS request". Ran `contrib-check` and all tests pass. +1, merging. ---
[GitHub] nifi issue #2936: NIFI-5489: Add expression language support to AMQP process...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2936 There are definitely inconsistencies throughout the project as this wasn't always a firm policy and there are many different contributors to the project. As you noted, we have tried to tighten the reviews and catch this where we can, and hopefully with the completion of [NIFI-5627](https://issues.apache.org/jira/browse/NIFI-5627), we will have a foundation to move forward on enabling EL in passwords for users. ---
[GitHub] nifi issue #2936: NIFI-5489: Add expression language support to AMQP process...
Github user alopresto commented on the issue:
https://github.com/apache/nifi/pull/2936
Hi @lukepfarrar , we have a policy of not evaluating EL in password fields.
Here is [an example of that review
process](https://github.com/apache/nifi/pull/3020#discussion_r219712821) and
the reasoning behind it on another PR.
> Our policy so far has been that passwords do not support expression
language, for a couple reasons:
> 1. How to evaluate if a password `abc${def}` should be interpreted as
`abc` + *the value of(`def`)* or the literal string `abc${def}`
> 1. The variable registry is not designed to store sensitive values
securely, so if a password is stored here, it can be accessed by an
unauthorized user
---
[GitHub] nifi pull request #2956: NIFI-5537 Create Neo4J cypher execution processor
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2956#discussion_r221672221
--- Diff:
nifi-nar-bundles/nifi-neo4j-bundle/nifi-neo4j-processors/src/main/java/org/apache/nifi/processors/neo4j/AbstractNeo4JCypherExecutor.java
---
@@ -0,0 +1,281 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.neo4j;
+
+import java.io.File;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.neo4j.driver.v1.AuthTokens;
+import org.neo4j.driver.v1.Config;
+import org.neo4j.driver.v1.Config.ConfigBuilder;
+import org.neo4j.driver.v1.Config.LoadBalancingStrategy;
+import org.neo4j.driver.v1.Config.TrustStrategy;
+import org.neo4j.driver.v1.Driver;
+import org.neo4j.driver.v1.GraphDatabase;
+
+/**
+ * Abstract base class for Neo4JCypherExecutor processors
+ */
+abstract class AbstractNeo4JCypherExecutor extends AbstractProcessor {
+
+protected static final PropertyDescriptor QUERY = new
PropertyDescriptor.Builder()
+.name("neo4J-query")
+.displayName("Neo4J Query")
+.description("Specifies the Neo4j Query.")
+.required(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor CONNECTION_URL = new
PropertyDescriptor.Builder()
+.name("neo4j-connection-url")
+.displayName("Neo4j Connection URL")
+.description("Neo4J endpoing to connect to.")
+.required(true)
+.defaultValue("bolt://localhost:7687")
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor USERNAME = new
PropertyDescriptor.Builder()
+.name("neo4j-username")
+.displayName("Username")
+.description("Username for accessing Neo4J")
+.required(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PASSWORD = new
PropertyDescriptor.Builder()
+.name("neo4j-password")
+.displayName("Password")
+.description("Password for Neo4J user")
+.required(true)
+.sensitive(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static AllowableValue LOAD_BALANCING_STRATEGY_ROUND_ROBIN = new
AllowableValue(LoadBalancingStrategy.ROUND_ROBIN.name(), "Round Robin", "Round
Robin Strategy");
+
+public static AllowableValue LOAD_BALANCING_STRATEGY_LEAST_CONNECTED =
new AllowableValue(LoadBalancingStrategy.LEAST_CONNECTED.n
[GitHub] nifi issue #3032: NIFI-5612: Support JDBC drivers that return Long for unsig...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/3032 @colindean the releases aren't on a set schedule and are decided by discussion on the d...@nifi.apache.org mailing list, but I think you're right in your feeling that a release is probably coming up soon. This is great work and exciting to get in. Thanks for researching and developing it. ---
[GitHub] nifi pull request #3035: NIFI-5628 Added content length check to OkHttpRepli...
GitHub user alopresto opened a pull request: https://github.com/apache/nifi/pull/3035 NIFI-5628 Added content length check to OkHttpReplicationClient. Added unit tests. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/alopresto/nifi NIFI-5628 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3035.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3035 commit 1baead6f525046a613fc4fe494a0d193776ea70f Author: Andy LoPresto Date: 2018-09-27T01:18:22Z NIFI-5628 Added content length check to OkHttpReplicationClient. Added unit tests. ---
[GitHub] nifi issue #2872: NIFI-5318 Implement NiFi test harness: initial commit of n...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2872 I think there should be a Maven module for the test harness which is disabled by default and can be activated with a flag like `mvn clean test -Ptest-harness`. ---
[GitHub] nifi pull request #2956: NIFI-5537 Create Neo4J cypher execution processor
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2956#discussion_r219954907
--- Diff:
nifi-nar-bundles/nifi-neo4j-bundle/nifi-neo4j-processors/src/test/java/org/apache/nifi/processors/neo4j/TestNeo4JCyperExecutor.java
---
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.neo4j;
+
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.neo4j.driver.v1.Driver;
+import org.neo4j.driver.v1.StatementResult;
+import org.neo4j.driver.v1.Record;
+import org.neo4j.driver.v1.summary.ResultSummary;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.mockito.Answers;
+import org.mockito.Mock;
+import org.mockito.Mockito;
+import org.mockito.junit.MockitoJUnit;
+import org.mockito.junit.MockitoRule;
+
+import java.io.File;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.junit.Assert.assertEquals;
+
+/**
+ * Neo4J Cypher unit tests.
+ */
+public class TestNeo4JCyperExecutor {
--- End diff --
Typo in class name.
---
[GitHub] nifi pull request #2956: NIFI-5537 Create Neo4J cypher execution processor
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2956#discussion_r219953983
--- Diff:
nifi-nar-bundles/nifi-neo4j-bundle/nifi-neo4j-processors/src/main/java/org/apache/nifi/processors/neo4j/AbstractNeo4JCypherExecutor.java
---
@@ -0,0 +1,281 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.neo4j;
+
+import java.io.File;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.neo4j.driver.v1.AuthTokens;
+import org.neo4j.driver.v1.Config;
+import org.neo4j.driver.v1.Config.ConfigBuilder;
+import org.neo4j.driver.v1.Config.LoadBalancingStrategy;
+import org.neo4j.driver.v1.Config.TrustStrategy;
+import org.neo4j.driver.v1.Driver;
+import org.neo4j.driver.v1.GraphDatabase;
+
+/**
+ * Abstract base class for Neo4JCypherExecutor processors
+ */
+abstract class AbstractNeo4JCypherExecutor extends AbstractProcessor {
+
+protected static final PropertyDescriptor QUERY = new
PropertyDescriptor.Builder()
+.name("neo4J-query")
+.displayName("Neo4J Query")
+.description("Specifies the Neo4j Query.")
+.required(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor CONNECTION_URL = new
PropertyDescriptor.Builder()
+.name("neo4j-connection-url")
+.displayName("Neo4j Connection URL")
+.description("Neo4J endpoing to connect to.")
+.required(true)
+.defaultValue("bolt://localhost:7687")
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor USERNAME = new
PropertyDescriptor.Builder()
+.name("neo4j-username")
+.displayName("Username")
+.description("Username for accessing Neo4J")
+.required(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PASSWORD = new
PropertyDescriptor.Builder()
+.name("neo4j-password")
+.displayName("Password")
+.description("Password for Neo4J user")
+.required(true)
+.sensitive(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static AllowableValue LOAD_BALANCING_STRATEGY_ROUND_ROBIN = new
AllowableValue(LoadBalancingStrategy.ROUND_ROBIN.name(), "Round Robin", "Round
Robin Strategy");
+
+public static AllowableValue LOAD_BALANCING_STRATEGY_LEAST_CONNECTED =
new AllowableValue(LoadBalancingStrategy.LEAST_CONNECTED.n
[GitHub] nifi pull request #2956: NIFI-5537 Create Neo4J cypher execution processor
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2956#discussion_r219953428
--- Diff:
nifi-nar-bundles/nifi-neo4j-bundle/nifi-neo4j-processors/src/main/java/org/apache/nifi/processors/neo4j/AbstractNeo4JCypherExecutor.java
---
@@ -0,0 +1,281 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.neo4j;
+
+import java.io.File;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.neo4j.driver.v1.AuthTokens;
+import org.neo4j.driver.v1.Config;
+import org.neo4j.driver.v1.Config.ConfigBuilder;
+import org.neo4j.driver.v1.Config.LoadBalancingStrategy;
+import org.neo4j.driver.v1.Config.TrustStrategy;
+import org.neo4j.driver.v1.Driver;
+import org.neo4j.driver.v1.GraphDatabase;
+
+/**
+ * Abstract base class for Neo4JCypherExecutor processors
+ */
+abstract class AbstractNeo4JCypherExecutor extends AbstractProcessor {
+
+protected static final PropertyDescriptor QUERY = new
PropertyDescriptor.Builder()
+.name("neo4J-query")
+.displayName("Neo4J Query")
+.description("Specifies the Neo4j Query.")
+.required(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor CONNECTION_URL = new
PropertyDescriptor.Builder()
+.name("neo4j-connection-url")
+.displayName("Neo4j Connection URL")
+.description("Neo4J endpoing to connect to.")
+.required(true)
+.defaultValue("bolt://localhost:7687")
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor USERNAME = new
PropertyDescriptor.Builder()
+.name("neo4j-username")
+.displayName("Username")
+.description("Username for accessing Neo4J")
+.required(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PASSWORD = new
PropertyDescriptor.Builder()
+.name("neo4j-password")
+.displayName("Password")
+.description("Password for Neo4J user")
+.required(true)
+.sensitive(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static AllowableValue LOAD_BALANCING_STRATEGY_ROUND_ROBIN = new
AllowableValue(LoadBalancingStrategy.ROUND_ROBIN.name(), "Round Robin", "Round
Robin Strategy");
+
+public static AllowableValue LOAD_BALANCING_STRATEGY_LEAST_CONNECTED =
new AllowableValue(LoadBalancingStrategy.LEAST_CONNECTED.n
[GitHub] nifi pull request #2956: NIFI-5537 Create Neo4J cypher execution processor
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2956#discussion_r219953286
--- Diff:
nifi-nar-bundles/nifi-neo4j-bundle/nifi-neo4j-processors/src/main/java/org/apache/nifi/processors/neo4j/AbstractNeo4JCypherExecutor.java
---
@@ -0,0 +1,281 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.neo4j;
+
+import java.io.File;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.neo4j.driver.v1.AuthTokens;
+import org.neo4j.driver.v1.Config;
+import org.neo4j.driver.v1.Config.ConfigBuilder;
+import org.neo4j.driver.v1.Config.LoadBalancingStrategy;
+import org.neo4j.driver.v1.Config.TrustStrategy;
+import org.neo4j.driver.v1.Driver;
+import org.neo4j.driver.v1.GraphDatabase;
+
+/**
+ * Abstract base class for Neo4JCypherExecutor processors
+ */
+abstract class AbstractNeo4JCypherExecutor extends AbstractProcessor {
+
+protected static final PropertyDescriptor QUERY = new
PropertyDescriptor.Builder()
+.name("neo4J-query")
+.displayName("Neo4J Query")
+.description("Specifies the Neo4j Query.")
+.required(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor CONNECTION_URL = new
PropertyDescriptor.Builder()
+.name("neo4j-connection-url")
+.displayName("Neo4j Connection URL")
+.description("Neo4J endpoing to connect to.")
+.required(true)
+.defaultValue("bolt://localhost:7687")
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor USERNAME = new
PropertyDescriptor.Builder()
+.name("neo4j-username")
+.displayName("Username")
+.description("Username for accessing Neo4J")
+.required(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PASSWORD = new
PropertyDescriptor.Builder()
+.name("neo4j-password")
+.displayName("Password")
+.description("Password for Neo4J user")
+.required(true)
+.sensitive(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static AllowableValue LOAD_BALANCING_STRATEGY_ROUND_ROBIN = new
AllowableValue(LoadBalancingStrategy.ROUND_ROBIN.name(), "Round Robin", "Round
Robin Strategy");
+
+public static AllowableValue LOAD_BALANCING_STRATEGY_LEAST_CONNECTED =
new AllowableValue(LoadBalancingStrategy.LEAST_CONNECTED.n
[GitHub] nifi pull request #2956: NIFI-5537 Create Neo4J cypher execution processor
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2956#discussion_r219953129
--- Diff:
nifi-nar-bundles/nifi-neo4j-bundle/nifi-neo4j-processors/src/main/java/org/apache/nifi/processors/neo4j/AbstractNeo4JCypherExecutor.java
---
@@ -0,0 +1,281 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.neo4j;
+
+import java.io.File;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.neo4j.driver.v1.AuthTokens;
+import org.neo4j.driver.v1.Config;
+import org.neo4j.driver.v1.Config.ConfigBuilder;
+import org.neo4j.driver.v1.Config.LoadBalancingStrategy;
+import org.neo4j.driver.v1.Config.TrustStrategy;
+import org.neo4j.driver.v1.Driver;
+import org.neo4j.driver.v1.GraphDatabase;
+
+/**
+ * Abstract base class for Neo4JCypherExecutor processors
+ */
+abstract class AbstractNeo4JCypherExecutor extends AbstractProcessor {
+
+protected static final PropertyDescriptor QUERY = new
PropertyDescriptor.Builder()
+.name("neo4J-query")
+.displayName("Neo4J Query")
+.description("Specifies the Neo4j Query.")
+.required(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor CONNECTION_URL = new
PropertyDescriptor.Builder()
+.name("neo4j-connection-url")
+.displayName("Neo4j Connection URL")
+.description("Neo4J endpoing to connect to.")
+.required(true)
+.defaultValue("bolt://localhost:7687")
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor USERNAME = new
PropertyDescriptor.Builder()
+.name("neo4j-username")
+.displayName("Username")
+.description("Username for accessing Neo4J")
+.required(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PASSWORD = new
PropertyDescriptor.Builder()
+.name("neo4j-password")
+.displayName("Password")
+.description("Password for Neo4J user")
+.required(true)
+.sensitive(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static AllowableValue LOAD_BALANCING_STRATEGY_ROUND_ROBIN = new
AllowableValue(LoadBalancingStrategy.ROUND_ROBIN.name(), "Round Robin", "Round
Robin Strategy");
+
+public static AllowableValue LOAD_BALANCING_STRATEGY_LEAST_CONNECTED =
new AllowableValue(LoadBalancingStrategy.LEAST_CONNECTED.n
[GitHub] nifi pull request #2956: NIFI-5537 Create Neo4J cypher execution processor
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2956#discussion_r219952904
--- Diff:
nifi-nar-bundles/nifi-neo4j-bundle/nifi-neo4j-processors/src/main/java/org/apache/nifi/processors/neo4j/AbstractNeo4JCypherExecutor.java
---
@@ -0,0 +1,281 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.neo4j;
+
+import java.io.File;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.neo4j.driver.v1.AuthTokens;
+import org.neo4j.driver.v1.Config;
+import org.neo4j.driver.v1.Config.ConfigBuilder;
+import org.neo4j.driver.v1.Config.LoadBalancingStrategy;
+import org.neo4j.driver.v1.Config.TrustStrategy;
+import org.neo4j.driver.v1.Driver;
+import org.neo4j.driver.v1.GraphDatabase;
+
+/**
+ * Abstract base class for Neo4JCypherExecutor processors
+ */
+abstract class AbstractNeo4JCypherExecutor extends AbstractProcessor {
+
+protected static final PropertyDescriptor QUERY = new
PropertyDescriptor.Builder()
+.name("neo4J-query")
+.displayName("Neo4J Query")
+.description("Specifies the Neo4j Query.")
+.required(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor CONNECTION_URL = new
PropertyDescriptor.Builder()
+.name("neo4j-connection-url")
+.displayName("Neo4j Connection URL")
+.description("Neo4J endpoing to connect to.")
+.required(true)
+.defaultValue("bolt://localhost:7687")
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor USERNAME = new
PropertyDescriptor.Builder()
+.name("neo4j-username")
+.displayName("Username")
+.description("Username for accessing Neo4J")
+.required(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PASSWORD = new
PropertyDescriptor.Builder()
+.name("neo4j-password")
+.displayName("Password")
+.description("Password for Neo4J user")
+.required(true)
+.sensitive(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
--- End diff --
Our policy so far has been that passwords do not support expression
language, for a couple reasons:
* How to evaluate if a password `abc${def}` should be interpreted as `abc`
+ *the value of(`def`)* or the literal string `abc${def}`
* The variable registry is not designed to store sensitive values securely,
so if a password is stored here, it can be accessed by an unauthorized user
---
[GitHub] nifi issue #3024: NIFI-5595 - Added the CORS filter to the templates/upload ...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/3024 Reviewing... ---
[GitHub] nifi issue #3020: NIFI-5625: support the variables for the properties of HTT...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/3020 Thanks for your submission. I left some comments, specifically on the `password` fields and a field containing a regex for attribute headers. `GetHTTP` and `PostHTTP` are legacy processors and have been effectively deprecated. `InvokeHTTP` should be used for all remote HTTP operations moving forward. I'll open a separate Jira to mark them as `@Deprecated`. Please also provide new or updated unit test cases for these processors. ---
[GitHub] nifi pull request #3020: NIFI-5625: support the variables for the properties...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3020#discussion_r219714056
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PostHTTP.java
---
@@ -726,7 +737,7 @@ public long getContentLength() {
contentType = StringUtils.isBlank(contentTypeValue) ?
DEFAULT_CONTENT_TYPE : contentTypeValue;
}
-final String attributeHeaderRegex =
context.getProperty(ATTRIBUTES_AS_HEADERS_REGEX).getValue();
+final String attributeHeaderRegex =
context.getProperty(ATTRIBUTES_AS_HEADERS_REGEX).evaluateAttributeExpressions().getValue();
--- End diff --
See comment about `attributeHeaderRegex` above.
---
[GitHub] nifi pull request #3020: NIFI-5625: support the variables for the properties...
Github user alopresto commented on a diff in the pull request: https://github.com/apache/nifi/pull/3020#discussion_r219714049 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PostHTTP.java --- @@ -474,8 +484,8 @@ public void process(final HttpResponse response, final HttpContext httpContext) clientBuilder.setRetryHandler(retryHandler); clientBuilder.disableContentCompression(); -final String username = context.getProperty(USERNAME).getValue(); -final String password = context.getProperty(PASSWORD).getValue(); +final String username = context.getProperty(USERNAME).evaluateAttributeExpressions().getValue(); +final String password = context.getProperty(PASSWORD).evaluateAttributeExpressions().getValue(); --- End diff -- See comment about `password` above. ---
[GitHub] nifi pull request #3020: NIFI-5625: support the variables for the properties...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3020#discussion_r219714033
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PostHTTP.java
---
@@ -183,26 +183,30 @@
.description("How long to wait when attempting to connect to
the remote server before giving up")
.required(true)
.defaultValue("30 sec")
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
.addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
.build();
public static final PropertyDescriptor DATA_TIMEOUT = new
PropertyDescriptor.Builder()
.name("Data Timeout")
.description("How long to wait between receiving segments of
data from the remote server before giving up and discarding the partial file")
.required(true)
.defaultValue("30 sec")
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
.addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
.build();
public static final PropertyDescriptor USERNAME = new
PropertyDescriptor.Builder()
.name("Username")
.description("Username required to access the URL")
.required(false)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.build();
public static final PropertyDescriptor PASSWORD = new
PropertyDescriptor.Builder()
.name("Password")
.description("Password required to access the URL")
.required(false)
.sensitive(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
--- End diff --
See comment about `password` above.
---
[GitHub] nifi pull request #3020: NIFI-5625: support the variables for the properties...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3020#discussion_r219714031
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PostHTTP.java
---
@@ -211,25 +215,29 @@
.required(false)
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.defaultValue(VersionInfo.getUserAgent("Apache-HttpClient",
"org.apache.http.client", HttpClientBuilder.class))
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
.build();
public static final PropertyDescriptor COMPRESSION_LEVEL = new
PropertyDescriptor.Builder()
.name("Compression Level")
.description("Determines the GZIP Compression Level to use
when sending the file; the value must be in the range of 0-9. A value of 0
indicates that the file will not be GZIP'ed")
.required(true)
.addValidator(StandardValidators.createLongValidator(0, 9,
true))
.defaultValue("0")
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
.build();
public static final PropertyDescriptor ATTRIBUTES_AS_HEADERS_REGEX =
new PropertyDescriptor.Builder()
.name("Attributes to Send as HTTP Headers (Regex)")
.description("Specifies the Regular Expression that determines
the names of FlowFile attributes that should be sent as HTTP Headers")
.addValidator(StandardValidators.REGULAR_EXPRESSION_VALIDATOR)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
--- End diff --
Accepting Expression Language in regex fields is difficult. Can you please
share what testing you performed for this field?
---
[GitHub] nifi pull request #3020: NIFI-5625: support the variables for the properties...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3020#discussion_r219712932
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java
---
@@ -714,11 +721,11 @@ private void setSslSocketFactory(OkHttpClient.Builder
okHttpClientBuilder, SSLCo
}
private void setAuthenticator(OkHttpClient.Builder
okHttpClientBuilder, ProcessContext context) {
-final String authUser =
trimToEmpty(context.getProperty(PROP_BASIC_AUTH_USERNAME).getValue());
+final String authUser =
trimToEmpty(context.getProperty(PROP_BASIC_AUTH_USERNAME).evaluateAttributeExpressions().getValue());
// If the username/password properties are set then check if
digest auth is being used
if (!authUser.isEmpty() &&
"true".equalsIgnoreCase(context.getProperty(PROP_DIGEST_AUTH).getValue())) {
-final String authPass =
trimToEmpty(context.getProperty(PROP_BASIC_AUTH_PASSWORD).getValue());
+final String authPass =
trimToEmpty(context.getProperty(PROP_BASIC_AUTH_PASSWORD).evaluateAttributeExpressions().getValue());
--- End diff --
See comment about `password` above.
---
[GitHub] nifi pull request #3020: NIFI-5625: support the variables for the properties...
Github user alopresto commented on a diff in the pull request: https://github.com/apache/nifi/pull/3020#discussion_r219712927 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java --- @@ -632,7 +639,7 @@ public void setUpClient(final ProcessContext context) throws IOException, Unreco } // check the trusted hostname property and override the HostnameVerifier -String trustedHostname = trimToEmpty(context.getProperty(PROP_TRUSTED_HOSTNAME).getValue()); +String trustedHostname = trimToEmpty(context.getProperty(PROP_TRUSTED_HOSTNAME).evaluateAttributeExpressions().getValue()); --- End diff -- See comment about `trustedHostname` above. ---
[GitHub] nifi pull request #3020: NIFI-5625: support the variables for the properties...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3020#discussion_r219712935
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java
---
@@ -947,11 +954,11 @@ private Request configureRequest(final ProcessContext
context, final ProcessSess
Request.Builder requestBuilder = new Request.Builder();
requestBuilder = requestBuilder.url(url);
-final String authUser =
trimToEmpty(context.getProperty(PROP_BASIC_AUTH_USERNAME).getValue());
+final String authUser =
trimToEmpty(context.getProperty(PROP_BASIC_AUTH_USERNAME).evaluateAttributeExpressions().getValue());
// If the username/password properties are set then check if
digest auth is being used
if (!authUser.isEmpty() &&
"false".equalsIgnoreCase(context.getProperty(PROP_DIGEST_AUTH).getValue())) {
-final String authPass =
trimToEmpty(context.getProperty(PROP_BASIC_AUTH_PASSWORD).getValue());
+final String authPass =
trimToEmpty(context.getProperty(PROP_BASIC_AUTH_PASSWORD).evaluateAttributeExpressions().getValue());
--- End diff --
See comment about `password` above.
---
[GitHub] nifi pull request #3020: NIFI-5625: support the variables for the properties...
Github user alopresto commented on a diff in the pull request: https://github.com/apache/nifi/pull/3020#discussion_r219712916 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java --- @@ -367,6 +372,7 @@ + "on the normal truststore hostname verifier. Only valid with SSL (HTTPS) connections.") .addValidator(StandardValidators.NON_EMPTY_VALIDATOR) .required(false) + .expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY) --- End diff -- I'm not very comfortable with this field supporting EL either. Do you have a specific use case that requires it? ---
[GitHub] nifi pull request #3020: NIFI-5625: support the variables for the properties...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3020#discussion_r219712894
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java
---
@@ -318,6 +321,7 @@
.description("The password to be used by the client to
authenticate against the Remote URL.")
.required(false)
.sensitive(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
--- End diff --
See comment about `password` above.
---
[GitHub] nifi pull request #3020: NIFI-5625: support the variables for the properties...
Github user alopresto commented on a diff in the pull request: https://github.com/apache/nifi/pull/3020#discussion_r219712848 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetHTTP.java --- @@ -438,8 +446,8 @@ public void onTrigger(final ProcessContext context, final ProcessSessionFactory clientBuilder.setSslcontext(sslContextService.createSSLContext(ClientAuth.REQUIRED)); } -final String username = context.getProperty(USERNAME).getValue(); -final String password = context.getProperty(PASSWORD).getValue(); +final String username = context.getProperty(USERNAME).evaluateAttributeExpressions().getValue(); +final String password = context.getProperty(PASSWORD).evaluateAttributeExpressions().getValue(); --- End diff -- See comment above about `password` supporting EL. ---
[GitHub] nifi pull request #3020: NIFI-5625: support the variables for the properties...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3020#discussion_r219712821
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetHTTP.java
---
@@ -177,19 +180,22 @@
.name("Username")
.description("Username required to access the URL")
.required(false)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.build();
public static final PropertyDescriptor PASSWORD = new
PropertyDescriptor.Builder()
.name("Password")
.description("Password required to access the URL")
.required(false)
.sensitive(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
--- End diff --
Our policy so far has been that passwords do not support expression
language, for a couple reasons:
1. How to evaluate if a password `abc${def}` should be interpreted as `abc`
+ *the value of(`def`)* or the literal string `abc${def}`
1. The variable registry is not designed to store sensitive values
securely, so if a password is stored here, it can be accessed by an
unauthorized user
---
[GitHub] nifi issue #3018: NIFI-5622 Updated test resource keystores and truststores ...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/3018 I believe @joewitt backed out the OkHttp changes from [NIFI-4806](https://issues.apache.org/jira/browse/NIFI-4806) (my previous note in the Jira) and is instead doing them in [NIFI-5623](https://issues.apache.org/jira/browse/NIFI-5623), but these changes will be necessary then nonetheless. ---
[GitHub] nifi pull request #3018: NIFI-5622 Updated test resource keystores and trust...
GitHub user alopresto opened a pull request: https://github.com/apache/nifi/pull/3018 NIFI-5622 Updated test resource keystores and truststores with Subjec⦠â¦tAlternativeNames to be compliant with RFC 6125. Refactored some test code to be clearer. Renamed some resources to be consistent across modules. Changed passwords to meet new minimum length requirements. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/alopresto/nifi NIFI-5622 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3018.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3018 commit 9b03aa73969ffb95e126873eac5594feb293 Author: Andy LoPresto Date: 2018-09-22T02:26:10Z NIFI-5622 Updated test resource keystores and truststores with SubjectAlternativeNames to be compliant with RFC 6125. Refactored some test code to be clearer. Renamed some resources to be consistent across modules. Changed passwords to meet new minimum length requirements. ---
[GitHub] nifi-registry issue #142: NIFIREG-200 Update dependencies
Github user alopresto commented on the issue: https://github.com/apache/nifi-registry/pull/142 @kevdoran any reason you decided on 18.0 for guava? Looks like the most recent release is 26.0 [Guava](https://github.com/google/guava). ---
[GitHub] nifi pull request #2991: NIFI-3469: multipart request support added to Handl...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2991#discussion_r218868215
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/HandleHttpRequest.java
---
@@ -521,161 +553,221 @@ public void onTrigger(final ProcessContext context,
final ProcessSession session
final long start = System.nanoTime();
final HttpServletRequest request = container.getRequest();
-FlowFile flowFile = session.create();
-try (OutputStream flowFileOut = session.write(flowFile)) {
-StreamUtils.copy(request.getInputStream(), flowFileOut);
-} catch (final IOException e) {
-// There may be many reasons which can produce an IOException
on the HTTP stream and in some of them, eg.
-// bad requests, the connection to the client is not closed.
In order to address also these cases, we try
-// and answer with a BAD_REQUEST, which lets the client know
that the request has not been correctly
-// processed and makes it aware that the connection can be
closed.
-getLogger().error("Failed to receive content from HTTP Request
from {} due to {}",
-new Object[]{request.getRemoteAddr(), e});
-session.remove(flowFile);
-try {
-HttpServletResponse response = container.getResponse();
-response.sendError(Status.BAD_REQUEST.getStatusCode());
-response.flushBuffer();
-container.getContext().complete();
-} catch (final IOException ioe) {
-getLogger().warn("Failed to send HTTP response to {} due
to {}",
-new Object[]{request.getRemoteAddr(), ioe});
+if (!Strings.isNullOrEmpty(request.getContentType()) &&
request.getContentType().contains(MIME_TYPE__MULTIPART_FORM_DATA)) {
+ final long maxRequestSize =
context.getProperty(MAX_REQUEST_SIZE).asLong();
+ request.setAttribute(Request.__MULTIPART_CONFIG_ELEMENT, new
MultipartConfigElement("/tmp", maxRequestSize, maxRequestSize, 0));
--- End diff --
This also opens up a lot of security concerns. We need to be very careful
about how we handle, sanitize, trust, store, and display this data.
Some good starting places for reading:
* https://www.owasp.org/index.php/Deserialization_of_untrusted_data
* https://www.owasp.org/index.php/Unrestricted_File_Upload
* https://www.owasp.org/index.php/Insecure_Temporary_File
* https://www.owasp.org/index.php/Protect_FileUpload_Against_Malicious_File
---
[GitHub] nifi pull request #2983: NIFI-5566 Improve HashContent processor and standar...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2983#discussion_r218132155
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/groovy/org/apache/nifi/security/util/crypto/HashServiceTest.groovy
---
@@ -0,0 +1,457 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License") you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.security.util.crypto
+
+import org.apache.nifi.components.AllowableValue
+import org.bouncycastle.jce.provider.BouncyCastleProvider
+import org.bouncycastle.util.encoders.Hex
+import org.junit.After
+import org.junit.AfterClass
+import org.junit.Before
+import org.junit.BeforeClass
+import org.junit.Test
+import org.junit.runner.RunWith
+import org.junit.runners.JUnit4
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+import java.nio.charset.Charset
+import java.nio.charset.StandardCharsets
+import java.security.Security
+
+@RunWith(JUnit4.class)
+class HashServiceTest extends GroovyTestCase {
+private static final Logger logger =
LoggerFactory.getLogger(HashServiceTest.class)
+static private final String LARGE_FILE_PATH =
"src/test/resources/HashServiceTest/largefile.txt"
--- End diff --
Thanks Kevin. I think this is a good point. I usually do dynamic test files
this way, but in this case I had originally planned for the file to be
persistent. Then when I decided to increase the size to verify the _streaming_
methods, I realized it was a waste to check in a large file with static (and
generateable) text, but I didn't move it. I can make this change.
---
[GitHub] nifi-minifi-cpp issue #398: MINIFICPP-607: Remove host/peer bypass
Github user alopresto commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/398 Thanks @phrocker . LGTM but my C++ is pretty rusty so I'm going to let @apiri or @achristianson give the +1 if they can. ---
[GitHub] nifi issue #2999: NIFI-5589 : Clarify PutMongo documentation
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2999 I looked at the code, and the documentation change is syntactically valid, but I am not familiar enough with MongoDB to assert that the content is correct. You will need a reviewer who is a MongoDB user. There are a few in the community and they look for PRs with _Mongo_ in the title quite regularly. ---
[GitHub] nifi issue #2999: NIFI-5589 : Clarify PutMongo documentation
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2999 Hi @josephxsxn I don't think we have to request that any more as GitHub shows the consolidated diff in one view, and rebasing & force-pushing to a branch that is used for a PR destroys history and can mess up the reviewer comments. The committer rebases & squashes the commits when they merge them. ---
[GitHub] nifi issue #2983: NIFI-5566 Improve HashContent processor and standardize Ha...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2983 Thanks. I may have published the wrong version of the template. I had a process group with processors configured to work with the current behavior, as well as one with the properties configured as you described which will be the behavior after [NIFI-5582](https://issues.apache.org/jira/browse/NIFI-5582) is implemented. ---
[GitHub] nifi issue #2983: NIFI-5566 Improve HashContent processor and standardize Ha...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2983 @thenatog are you seeing behavior that conflicts with the description of expected behavior for configuration 1 if you reproduce the test case Otto described in his PR above? ---
[GitHub] nifi issue #2983: NIFI-5566 Improve HashContent processor and standardize Ha...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2983 Those attributes were added by @ottobackwards in the original PR for this issue. My understanding of the scenarios is as follows: **Flowfile A** *username*: `alopresto` *email*: `alopre...@apache.org` **Flowfile B** *username*: `alopresto` **Flowfile C** *no attributes* ### Processor Configuration 1 (Allow partial): **Fail when no attributes present**: `true` **Missing attribute policy**: `Allow missing attributes` **Flowfile A** -> *success* **Flowfile B** -> *success* **Flowfile C** -> *failure* ### Processor Configuration 2 (Fail on partial): **Fail when no attributes present**: `true` **Missing attribute policy**: `Fail if missing attributes` **Flowfile A** -> *success* **Flowfile B** -> *failure* **Flowfile C** -> *failure* ### Processor Configuration 3 (Allow empty): **Fail when no attributes present**: `false` **Missing attribute policy**: `Allow missing attributes` **Flowfile A** -> *success* **Flowfile B** -> *success* **Flowfile C** -> *success* ### Processor Configuration 4 (Allow empty but fail partial): **Fail when no attributes present**: `false` **Missing attribute policy**: `Fail if missing attributes` **Flowfile A** -> *success* **Flowfile B** -> *failure* **Flowfile C** -> *success* ---
[GitHub] nifi issue #2983: NIFI-5566 Improve HashContent processor and standardize Ha...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2983 Based on @thenatog 's comment on [PR 2980](https://github.com/apache/nifi/pull/2980#issuecomment-419574743) which captured a Java Big Endian Byte Order Mark (BOM) being inserted in UTF-16 strings pre-hash, I added some logic to prevent that here. I closed PR 2980, and this PR can be considered as a standalone solution to [NIFI-5147](https://issues.apache.org/jira/browse/NIFI-5147) and [NIFI-5566](https://issues.apache.org/jira/browse/NIFI-5566). I also opened [NIFI-5582](https://issues.apache.org/jira/browse/NIFI-5582) which depends on this PR. ---
[GitHub] nifi pull request #2980: NIFI-5147 Implement CalculateAttributeHash processo...
Github user alopresto closed the pull request at: https://github.com/apache/nifi/pull/2980 ---
[GitHub] nifi issue #2980: NIFI-5147 Implement CalculateAttributeHash processor
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2980 Thanks for discovering this @thenatog . This is an excellent catch. I've added behavior to catch this, better documentation, and unit tests. However, I added them on the branch that includes [PR 2983](https://github.com/apache/nifi/pull/2983). Let's mark this PR as closed and just review the other one, as it is more complete and addresses this issue. ``` 2018-09-07 21:21:19,784 WARN [Timer-Driven Process Thread-6] o.a.n.security.util.crypto.HashService The charset provided was UTF-16, but Java will insert a Big Endian BOM in the decoded message before hashing, so switching to UTF-16BE 2018-09-07 21:21:19,797 INFO [Timer-Driven Process Thread-9] o.a.n.processors.standard.LogAttribute LogAttribute[id=b15f3209-344d-10a6-4a7b-454530bb72fc] logging for flow file StandardFlowFileRecord[uuid=a4a223fb-aa11-43b9-93a3-d7675c44593c,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1536378604366-1, container=default, section=1], offset=56, length=4],offset=0,name=33467912436349,size=4] [SUCCESS] Standard FlowFile Attributes Key: 'entryDate' Value: 'Fri Sep 07 21:21:19 PDT 2018' Key: 'lineageStartDate' Value: 'Fri Sep 07 21:21:19 PDT 2018' Key: 'fileSize' Value: '4' FlowFile Attribute Map Content Key: 'filename' Value: '33467912436349' Key: 'path' Value: './' Key: 'test_attribute' Value: 'hehe' Key: 'test_attribute_md5_utf16le' Value: '2db0ecc27f7abd29ba95412feb3b5e07' Key: 'uuid' Value: 'a4a223fb-aa11-43b9-93a3-d7675c44593c' [SUCCESS] hehe 2018-09-07 21:21:19,799 INFO [Timer-Driven Process Thread-9] o.a.n.processors.standard.LogAttribute LogAttribute[id=b15f3209-344d-10a6-4a7b-454530bb72fc] logging for flow file StandardFlowFileRecord[uuid=b7459e40-500b-488d-a0dc-3e09ebc6b86e,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1536378604366-1, container=default, section=1], offset=56, length=4],offset=0,name=33467912436349,size=4] [SUCCESS] Standard FlowFile Attributes Key: 'entryDate' Value: 'Fri Sep 07 21:21:19 PDT 2018' Key: 'lineageStartDate' Value: 'Fri Sep 07 21:21:19 PDT 2018' Key: 'fileSize' Value: '4' FlowFile Attribute Map Content Key: 'filename' Value: '33467912436349' Key: 'path' Value: './' Key: 'test_attribute' Value: 'hehe' Key: 'test_attribute_md5_utf16' Value: 'b0ed26b524e0b0606551d78e42b5b7bc' Key: 'uuid' Value: 'b7459e40-500b-488d-a0dc-3e09ebc6b86e' [SUCCESS] hehe 2018-09-07 21:21:19,801 INFO [Timer-Driven Process Thread-9] o.a.n.processors.standard.LogAttribute LogAttribute[id=b15f3209-344d-10a6-4a7b-454530bb72fc] logging for flow file StandardFlowFileRecord[uuid=25c5d1b1-faa4-418d-911c-5c0cea399b83,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1536378604366-1, container=default, section=1], offset=56, length=4],offset=0,name=33467912436349,size=4] [SUCCESS] Standard FlowFile Attributes Key: 'entryDate' Value: 'Fri Sep 07 21:21:19 PDT 2018' Key: 'lineageStartDate' Value: 'Fri Sep 07 21:21:19 PDT 2018' Key: 'fileSize' Value: '4' FlowFile Attribute Map Content Key: 'filename' Value: '33467912436349' Key: 'path' Value: './' Key: 'test_attribute' Value: 'hehe' Key: 'test_attribute_md5_utf16be' Value: 'b0ed26b524e0b0606551d78e42b5b7bc' Key: 'uuid' Value: '25c5d1b1-faa4-418d-911c-5c0cea399b83' [SUCCESS] hehe ``` ---
[GitHub] nifi issue #2983: NIFI-5566 Improve HashContent processor and standardize Ha...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2983 I added a `.keep` file in that directory. ---
[GitHub] nifi issue #2989: NIFI-5366 - Added ContentSecurityPolicyFilter which stops ...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2989 Thanks for making the requested changes. In general, please try to avoid rebasing in PRs as it loses the history of the interactions. The reviewer/committer should handle that when merging to master. +1, merged. ---
[GitHub] nifi pull request #2989: NIFI-5366 - Added ContentSecurityPolicyFilter which...
Github user alopresto commented on a diff in the pull request: https://github.com/apache/nifi/pull/2989#discussion_r215465715 --- Diff: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-jetty/src/main/java/org/apache/nifi/web/server/JettyServer.java --- @@ -502,6 +503,11 @@ private WebAppContext loadWar(final File warFile, final String contextPath, fina // add a filter to set the X-Frame-Options filter webappContext.addFilter(new FilterHolder(FRAME_OPTIONS_FILTER), "/*", EnumSet.allOf(DispatcherType.class)); +// add a filter to set the Content Security Policy frame-ancestors directive +FilterHolder cspFilter = new FilterHolder(new ContentSecurityPolicyFilter()); --- End diff -- From what I can tell by tracing the Jetty code, this is almost equivalent to just assigning the `Filter` directly via `addFilter()`. Do you have a resource or documentation that indicates why using a `FilterHolder` is preferable? ---
[GitHub] nifi issue #2983: NIFI-5566 Improve HashContent processor and standardize Ha...
Github user alopresto commented on the issue:
https://github.com/apache/nifi/pull/2983
@ottobackwards I've run this branch with the full build and all tests
enabled multiple times and I have not encountered that error (just did it again
from scratch with no `.m2` to be sure; 41:24 minute process). Can you manually
verify if that directory and file exist on your machine?
The logic in the test should populate the file when the test is run and
delete the file when the test suite is complete (so we don't just have an
unnecessary 10 MB file checked in to the source).
```
File inputFile = new File(LARGE_FILE_PATH)
// Generates a file with "apachenifi" 10 times per line for 10_000
lines (11 bytes * 10 * 10_000 ~= 1 MiB)
if (!inputFile.exists() || inputFile.length() == 0) {
10_000.times { int i ->
inputFile << "${i.toString().padLeft(5)}: ${"apachenifi " *
10}\n"
}
}
```
```
@AfterClass
static void tearDownOnce() throws Exception {
File largeFile = new File(LARGE_FILE_PATH)
if (largeFile.exists()) {
largeFile.deleteOnExit()
}
}
```
---
[GitHub] nifi issue #2989: NIFI-5366 - Added ContentSecurityPolicyFilter which stops ...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2989 Verified that the header is present on responses. Ran `contrib-check` and all tests pass. +1 if you re-order the dependencies. ---
[GitHub] nifi issue #2989: NIFI-5366 - Added ContentSecurityPolicyFilter which stops ...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2989 Reviewing... ---
[GitHub] nifi pull request #2989: NIFI-5366 - Added ContentSecurityPolicyFilter which...
Github user alopresto commented on a diff in the pull request: https://github.com/apache/nifi/pull/2989#discussion_r215092182 --- Diff: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-security/pom.xml --- @@ -154,5 +154,20 @@ jettison test + +org.springframework +spring-test +5.0.6.RELEASE +test + + +org.eclipse.jetty +jetty-servlet --- End diff -- Let's move the non-`test` dependencies above so they are all together with the `compile` dependencies and the `test` dependencies are together. Not a technical necessity, but good for logical grouping and identification. ---
[GitHub] nifi issue #2987: NIFI-3344 Added property to JoltTransformJSON allowing the...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2987 Thanks for reporting the context around that. If you're comfortable doing it, you can open a Jira to resolve that problem [here](https://issues.apache.org/jira/secure/CreateIssue!default.jspa). If not, let me know and I can do it in a bit. It is probably environment-specific in that the test was not verified on Windows before being re-enabled in [PR 2821](https://github.com/apache/nifi/pull/2821), but not *your environment*-specific in that other Windows users may have the same issue. Either way, I believe it's a legitimate thing to report and we can fix it. Thanks. ---
[GitHub] nifi pull request #2987: NIFI-3344 Added property to JoltTransformJSON allow...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2987#discussion_r215046005
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/JoltTransformJSON.java
---
@@ -130,6 +130,15 @@
.required(true)
.build();
+public static final PropertyDescriptor PRETTY_PRINT = new
PropertyDescriptor.Builder()
+.name(("Pretty Print"))
--- End diff --
It looks like it was added by a different author, and there are no external
references to it, so I believe it can be changed to `private`. @markap14 and
@YolandaMDavis , do you have any reasons it should not be changed? Thanks.
If the change is made, it should also have a `displayName` added for
consistency.
---
[GitHub] nifi pull request #2987: NIFI-3344 Added property to JoltTransformJSON allow...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2987#discussion_r215021556
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/JoltTransformJSON.java
---
@@ -130,6 +130,15 @@
.required(true)
.build();
+public static final PropertyDescriptor PRETTY_PRINT = new
PropertyDescriptor.Builder()
+.name(("Pretty Print"))
--- End diff --
There's an extra set of parentheses here, and the `name` field is usually
formatted like `pretty_print` because it is used in XML object resolution. The
`displayName` field below is used for the human-readable and changeable value
that won't affect object resolution during flow loading.
---
[GitHub] nifi issue #2987: NIFI-3344 Added property to JoltTransformJSON allowing the...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2987 What errors are you encountering when building with tests? ---
[GitHub] nifi pull request #2983: NIFI-5566 Improve HashContent processor and standar...
Github user alopresto commented on a diff in the pull request: https://github.com/apache/nifi/pull/2983#discussion_r214979973 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/HashAttribute.java --- @@ -51,6 +50,9 @@ import org.apache.nifi.processor.util.StandardValidators; /** --- End diff -- It is clearly called out in the documentation for the processor. Perhaps an improvement Jira is needed for an indicator on the canvas and in the *Add Processor* dialog. ---
[GitHub] nifi pull request #2980: NIFI-5147 Implement CalculateAttributeHash processo...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2980#discussion_r214979341
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/security/util/crypto/HashService.java
---
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.security.util.crypto;
+
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import org.apache.commons.codec.binary.Hex;
+import org.apache.commons.codec.digest.DigestUtils;
+import org.bouncycastle.crypto.digests.Blake2bDigest;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * This class provides a generic service for cryptographic hashing. It is
used in
+ * {@link org.apache.nifi.processors.standard.CalculateAttributeHash} and
+ * {@link org.apache.nifi.processors.standard.HashContent}.
+ *
+ * See also:
+ * * {@link HashAlgorithm}
+ */
+public class HashService {
+private static final Logger logger =
LoggerFactory.getLogger(HashService.class);
+
+/**
+ * Returns the hex-encoded hash of the specified value.
+ *
+ * @param algorithm the hash algorithm to use
+ * @param value the value to hash (cannot be {@code null} but can
be an empty String)
+ * @param charset the charset to use
+ * @return the hash value in hex
+ */
+public static String hashValue(HashAlgorithm algorithm, String value,
Charset charset) {
+byte[] rawHash = hashValueRaw(algorithm, value, charset);
+return Hex.encodeHexString(rawHash);
+}
+
+/**
+ * Returns the hex-encoded hash of the specified value. The default
charset ({@code StandardCharsets.UTF_8}) is used.
+ *
+ * @param algorithm the hash algorithm to use
+ * @param value the value to hash (cannot be {@code null} but can
be an empty String)
+ * @return the hash value in hex
+ */
+public static String hashValue(HashAlgorithm algorithm, String value) {
+return hashValue(algorithm, value, StandardCharsets.UTF_8);
+}
+
+/**
+ * Returns the raw {@code byte[]} hash of the specified value.
+ *
+ * @param algorithm the hash algorithm to use
+ * @param value the value to hash (cannot be {@code null} but can
be an empty String)
+ * @param charset the charset to use
+ * @return the hash value in bytes
+ */
+public static byte[] hashValueRaw(HashAlgorithm algorithm, String
value, Charset charset) {
+if (value == null) {
+throw new IllegalArgumentException("The value cannot be null");
+}
+return hashValueRaw(algorithm, value.getBytes(charset));
+}
+
+/**
+ * Returns the raw {@code byte[]} hash of the specified value. The
default charset ({@code StandardCharsets.UTF_8}) is used.
+ *
+ * @param algorithm the hash algorithm to use
+ * @param value the value to hash (cannot be {@code null} but can
be an empty String)
+ * @return the hash value in bytes
+ */
+public static byte[] hashValueRaw(HashAlgorithm algorithm, String
value) {
+return hashValueRaw(algorithm, value, StandardCharsets.UTF_8);
+}
+
+/**
+ * Returns the raw {@code byte[]} hash of the specified value.
+ *
+ * @param algorithm the hash algorithm to use
+ * @param value the value to hash
+ * @return the hash value in bytes
+ */
+public static byte[] hashValueRaw(HashAlgorithm algorithm, byte[]
value) {
+if (algorithm == null) {
+throw new IllegalArgumentException("The hash algorithm cannot
be null");
+}
+if (value == null) {
+throw new IllegalArgumentException("The value can
[GitHub] nifi pull request #2980: NIFI-5147 Implement CalculateAttributeHash processo...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2980#discussion_r214979088
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/security/util/crypto/HashAlgorithm.java
---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.security.util.crypto;
+
+import java.util.Arrays;
+import java.util.List;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.commons.lang3.builder.ToStringStyle;
+
+/**
+ * Enumeration capturing information about the cryptographic hash
algorithms used in
+ * {@link org.apache.nifi.processors.standard.CalculateAttributeHash} and
+ * {@link org.apache.nifi.processors.standard.HashContent} processors.
+ */
+public enum HashAlgorithm {
+
+MD2("MD2", 16, "Cryptographically broken due to collisions"),
+MD5("MD5", 16, "Cryptographically broken due to collisions"),
+SHA1("SHA-1", 20, "Cryptographically broken due to collisions"),
+SHA224("SHA-224", 28, "SHA-2 family"),
+SHA256("SHA-256", 32, "SHA-2 family"),
+SHA384("SHA-384", 48, "SHA-2 family"),
+SHA512("SHA-512", 64, "SHA-2 family"),
+SHA512_224("SHA-512/224", 28, "SHA-2 using SHA-512 with truncated
output"),
+SHA512_256("SHA-512/256", 32, "SHA-2 using SHA-512 with truncated
output"),
+SHA3_224("SHA3-224", 28, "Keccak-based SHA3 family"),
+SHA3_256("SHA3-256", 32, "Keccak-based SHA3 family"),
+SHA3_384("SHA3-384", 48, "Keccak-based SHA3 family"),
+SHA3_512("SHA3-512", 64, "Keccak-based SHA3 family"),
+BLAKE2_160("BLAKE2-160", 20, "Also known as Blake2b"),
+BLAKE2_256("BLAKE2-256", 32, "Also known as Blake2b"),
+BLAKE2_384("BLAKE2-384", 48, "Also known as Blake2b"),
+BLAKE2_512("BLAKE2-512", 64, "Also known as Blake2b");
+
+private final String name;
+private final int digestBytesLength;
+private final String description;
+
+private static final List BROKEN_ALGORITHMS =
Arrays.asList(MD2.name, MD5.name, SHA1.name);
+
+HashAlgorithm(String name, int digestBytesLength, String description) {
+this.name = name;
+this.digestBytesLength = digestBytesLength;
+this.description = description;
+}
+
+public String getName() {
+return name;
+}
+
+public int getDigestBytesLength() {
+return digestBytesLength;
+}
+
+public String getDescription() {
+return description;
+}
+
+/**
+ * Returns {@code true} if this algorithm is considered
cryptographically secure. These determinations were made as of 2018-08-30.
+ *
+ * Current strong algorithms:
+ *
+ * * SHA-224 (SHA2)
+ * * SHA-256 (SHA2)
+ * * SHA-384 (SHA2)
+ * * SHA-512 (SHA2)
+ * * SHA-512/224 (SHA2)
+ * * SHA-512/256 (SHA2)
+ * * SHA3-256
+ * * SHA3-384
+ * * SHA3-512
+ * * Blake2b-256
+ * * Blake2b-384
+ * * Blake2b-512
+ *
+ * Current broken algorithms:
+ *
+ * * MD2
+ * * MD5
+ * * SHA-1
+ *
+ * @return true if the algorithm is considered strong
+ */
+public boolean isStrongAlgorithm() {
+return (!BROKEN_ALGORITHMS.contains(name));
+}
+
--- End diff --
The Blake2 implementations need BouncyCastle and use different API calls
than the other `MessageDigest` instances.
---
[GitHub] nifi pull request #2984: NIFI-5569 Added keywords to Route* and ScanAttribut...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2984#discussion_r214774007
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RouteText.java
---
@@ -655,14 +653,9 @@ public boolean equals(Object obj) {
Group other = (Group) obj;
if (capturedValues == null) {
-if (other.capturedValues != null) {
-return false;
-}
-} else if (!capturedValues.equals(other.capturedValues)) {
-return false;
-}
+return other.capturedValues == null;
+} else return capturedValues.equals(other.capturedValues);
--- End diff --
This was an automatic IDE optimization I wasn't aware of, but the logic
checks out.
---
[GitHub] nifi pull request #2984: NIFI-5569 Added keywords to Route* and ScanAttribut...
GitHub user alopresto opened a pull request: https://github.com/apache/nifi/pull/2984 NIFI-5569 Added keywords to Route* and ScanAttribute processors to im⦠â¦prove discoverability. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/alopresto/nifi NIFI-5569 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2984.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2984 commit a4369b0a781fc15ed03438c1bcf4257942c28586 Author: Andy LoPresto Date: 2018-09-04T02:12:56Z NIFI-5569 Added keywords to Route* and ScanAttribute processors to improve discoverability. ---
[GitHub] nifi issue #2983: NIFI-5566 Improve HashContent processor and standardize Ha...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2983 I created two templates that should help with testing the new behavior and ensuring that upgrading a flow that uses the old components doesn't break it. * [Hash Components (Legacy)](https://gist.github.com/alopresto/80fcabb78be26a04b894721c4f382300) * [Hash Components (New)](https://gist.github.com/alopresto/93ed6907037dce8bf855c9cb4354ea9d) ---
[GitHub] nifi pull request #2983: NIFI-5566 Improve HashContent processor and standar...
GitHub user alopresto opened a pull request: https://github.com/apache/nifi/pull/2983 NIFI-5566 Improve HashContent processor and standardize HashAttribute processor ### This PR should not be merged before [PR 2980](https://github.com/apache/nifi/pull/2980) This PR introduces two new processors `CryptographicHashContent` and `CryptographicHashAttribute` (renamed from `CalculateAttributeHash` in PR 2980), deprecates `HashContent`, and adds documentation to `HashAttribute` indicating the new processor performs the expected behavior. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/alopresto/nifi NIFI-5566 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2983.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2983 commit e2cf56994edd3c72baa346c46c4fab2c03a654fa Author: Otto Fowler Date: 2018-06-18T15:00:38Z NIFI-5147 Add CalculateAttributeHash processor commit 81d7e794e11855b0d44521c55baaae0771c9de44 Author: Otto Fowler Date: 2018-06-26T18:51:06Z add warning and recommendation link commit 808b9b22e4137ce26e7361015a4953dacc0296a1 Author: Otto Fowler Date: 2018-07-01T16:18:08Z per review - added properties to control behavior when attributes that are configured are partially or completely missing - set charset with a property - added tests commit a05de1c62710acf7f91855d5492f01e2ed46e155 Author: Otto Fowler Date: 2018-07-01T16:25:24Z fix assert parameter order commit 66577ddb57ae4fff198608d29ed4490b1543e424 Author: Andy LoPresto Date: 2018-08-31T02:26:53Z NIFI-5147 Added HashAlgorithm enum for CalculateHashAlgorithm and HashContent. Added unit tests. commit 8a594e5b1ec9f189a14d933e7e586e9bc1e04432 Author: Andy LoPresto Date: 2018-08-31T03:07:18Z NIFI-5147 [WIP] Used HashAlgorithm enum in CalculateHashAttribute. Cleaned up typos and descriptions. Added unit test demonstrating missing Blake2 algorithm. commit 39e9cd2325f5f2312daa19d1ccb4128de384 Author: Andy LoPresto Date: 2018-08-31T03:18:44Z NIFI-5147 [WIP] Added logic for Blake2 algorithms (needs refactoring). commit 68db917487c009eb9a3b75ed50d174f809c51e3a Author: Andy LoPresto Date: 2018-08-31T03:31:27Z NIFI-5147 Blake2b limited to 160, 256, 384, and 512 in Bouncy Castle implementation. Updated unit test. commit 5165a41bdb90e32e0e81ae402cbff8a61669dd8b Author: Andy LoPresto Date: 2018-08-31T03:34:28Z NIFI-5147 Finished implementing Blake2 logic. Unit test for all default test vectors passes. commit 6f91e6f9ec3cbbceac5609ca9d495ec259dbc517 Author: Andy LoPresto Date: 2018-08-31T03:44:10Z NIFI-5147 Added unit test demonstrating empty values fail. commit afd6a0cbf4d1b09f8109150051bd28a90d4fda6a Author: Andy LoPresto Date: 2018-08-31T03:45:08Z NIFI-5147 Implemented logic for empty input values. commit 9a47353adc439ca6c6ed4976dfb8327450400f74 Author: Andy LoPresto Date: 2018-08-31T04:17:54Z NIFI-5147 I
[GitHub] nifi issue #2977: NIFI-5562 - Upgraded guava versions from v18.0 to v26.0-jr...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2977 Reviewing... ---
[GitHub] nifi issue #2982: NIFI-4558 - Set JKS as the default keystore type and trust...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2982 Verified that this sets the defaults to "JKS" for both keystore and truststore on `StandardSSLContextService` and `StandardRestrictedSSLContextService`. Ran `contrib-check` and all tests pass. +1, merging. ---
[GitHub] nifi issue #2836: NIFI-5147 Calculate hash attribute redux
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2836 I didn't capture every detail of the issue in the Jira because I planned on writing it myself. However, some of the behavior (empty input should still return a hash, etc.) was standard. Yes, please close this PR and review 2980. ---
[GitHub] nifi issue #2982: NIFI-4558 - Set JKS as the default keystore type and trust...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2982 Reviewing... ---
[GitHub] nifi issue #2836: NIFI-5147 Calculate hash attribute redux
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2836 I started reviewing this PR but realized it did not implement many of the behaviors I had needed in the original ticket. I opened [PR 2980](https://github.com/apache/nifi/pull/2980) which includes this foundation instead. ---
[GitHub] nifi pull request #2980: NIFI-5147 Implement CalculateAttributeHash processo...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2980#discussion_r214265868
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/security/util/crypto/HashAlgorithm.java
---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.security.util.crypto;
+
+import java.util.Arrays;
+import java.util.List;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.commons.lang3.builder.ToStringStyle;
+
+/**
+ * Enumeration capturing information about the cryptographic hash
algorithms used in
+ * {@link org.apache.nifi.processors.standard.CalculateAttributeHash} and
+ * {@link org.apache.nifi.processors.standard.HashContent} processors.
+ */
+public enum HashAlgorithm {
+
+MD2("MD2", 16, "Cryptographically broken due to collisions"),
+MD5("MD5", 16, "Cryptographically broken due to collisions"),
+SHA1("SHA-1", 20, "Cryptographically broken due to collisions"),
+SHA224("SHA-224", 28, "SHA-2 family"),
+SHA256("SHA-256", 32, "SHA-2 family"),
+SHA384("SHA-384", 48, "SHA-2 family"),
+SHA512("SHA-512", 64, "SHA-2 family"),
+SHA512_224("SHA-512/224", 28, "SHA-2 using SHA-512 with truncated
output"),
+SHA512_256("SHA-512/256", 32, "SHA-2 using SHA-512 with truncated
output"),
+SHA3_224("SHA3-224", 28, "Keccak-based SHA3 family"),
+SHA3_256("SHA3-256", 32, "Keccak-based SHA3 family"),
+SHA3_384("SHA3-384", 48, "Keccak-based SHA3 family"),
+SHA3_512("SHA3-512", 64, "Keccak-based SHA3 family"),
+BLAKE2_160("BLAKE2-160", 20, "Also known as Blake2b"),
+BLAKE2_256("BLAKE2-256", 32, "Also known as Blake2b"),
+BLAKE2_384("BLAKE2-384", 48, "Also known as Blake2b"),
+BLAKE2_512("BLAKE2-512", 64, "Also known as Blake2b");
+
+private final String name;
+private final int digestBytesLength;
+private final String description;
+
+private static final List BROKEN_ALGORITHMS =
Arrays.asList(MD2.name, MD5.name, SHA1.name);
+
+HashAlgorithm(String name, int digestBytesLength, String description) {
+this.name = name;
+this.digestBytesLength = digestBytesLength;
+this.description = description;
+}
+
+public String getName() {
+return name;
+}
+
+public int getDigestBytesLength() {
+return digestBytesLength;
+}
+
+public String getDescription() {
+return description;
+}
+
+/**
+ * Returns {@code true} if this algorithm is considered
cryptographically secure. These determinations were made as of 2018-08-30.
+ *
+ * Current strong algorithms:
+ *
+ * * SHA-224 (SHA2)
+ * * SHA-256 (SHA2)
+ * * SHA-384 (SHA2)
+ * * SHA-512 (SHA2)
+ * * SHA-512/224 (SHA2)
+ * * SHA-512/256 (SHA2)
+ * * SHA3-256
--- End diff --
Add SHA3-224 and Blake2b-160.
---
[GitHub] nifi issue #2980: NIFI-5147 Implement CalculateAttributeHash processor
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2980 This encapsulates the changes @ottobackwards made in [PR 2836](https://github.com/apache/nifi/pull/2836), but also: * Adds the *SHA-224, SHA-512/224, SHA-512/256, SHA-3 (SHA3-224, SHA3-256, SHA3-384, SHA3-512)*, and *BLAKE2 (BLAKE2-160, BLAKE2-256, BLAKE2-384, BLAKE2-512)* functions * Moves the hashing functionality into an enum and service which can be reused by `HashContent` * Clearly marks cryptographically broken algorithms as such * Adds unit tests I will open follow-on issues to: 1. Add documentation to `HashAttribute` to explain the different scenarios where these processors are used 1. Refactor `HashContent` to use the `HashService` ---
[GitHub] nifi pull request #2980: NIFI-5147 Implement CalculateAttributeHash processo...
GitHub user alopresto opened a pull request: https://github.com/apache/nifi/pull/2980 NIFI-5147 Implement CalculateAttributeHash processor Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/alopresto/nifi NIFI-5147 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2980.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2980 commit d23c7759fcd8417cfa1a0ea06da41cccbd5b0b03 Author: Otto Fowler Date: 2018-06-18T15:00:38Z NIFI-5147 Add CalculateAttributeHash processor commit 919945083cb4062e56eda6dc97a1cfa01ba059f9 Author: Otto Fowler Date: 2018-06-26T18:51:06Z add warning and recommendation link commit 1e178b7daba7c58202e04fec0eb3b4393669e0a6 Author: Otto Fowler Date: 2018-07-01T16:18:08Z per review - added properties to control behavior when attributes that are configured are partially or completely missing - set charset with a property - added tests commit f9ce58f0982bd108199164d7a474b83a0286011a Author: Otto Fowler Date: 2018-07-01T16:25:24Z fix assert parameter order commit 40ac9f1448c40b76a2c2d33169360520384ddb27 Author: Andy LoPresto Date: 2018-08-31T02:26:53Z NIFI-5147 Added HashAlgorithm enum for CalculateHashAlgorithm and HashContent. Added unit tests. commit c54ca6323b2f0ebca384681c52b810f5404b4e7d Author: Andy LoPresto Date: 2018-08-31T03:07:18Z NIFI-5147 [WIP] Used HashAlgorithm enum in CalculateHashAttribute. Cleaned up typos and descriptions. Added unit test demonstrating missing Blake2 algorithm. commit 40bc74d2d27049d52206480038706e37a9b880bd Author: Andy LoPresto Date: 2018-08-31T03:18:44Z NIFI-5147 [WIP] Added logic for Blake2 algorithms (needs refactoring). commit a7ba56e155efe6f7dac3aff918def35486fe9d2a Author: Andy LoPresto Date: 2018-08-31T03:31:27Z NIFI-5147 Blake2b limited to 160, 256, 384, and 512 in Bouncy Castle implementation. Updated unit test. commit 3a9c5c5aee407798acf24398f0a214b62e8ba177 Author: Andy LoPresto Date: 2018-08-31T03:34:28Z NIFI-5147 Finished implementing Blake2 logic. Unit test for all default test vectors passes. commit 9c682b805f298c53fe1067fe944a4b94791b9fa6 Author: Andy LoPresto Date: 2018-08-31T03:44:10Z NIFI-5147 Added unit test demonstrating empty values fail. commit a928adec9f322635f9333cf5f9a303160a31f3d8 Author: Andy LoPresto Date: 2018-08-31T03:45:08Z NIFI-5147 Implemented logic for empty input values. commit 67468d8498f5b6fca7847baf546521d74967acb4 Author: Andy LoPresto Date: 2018-08-31T04:17:54Z NIFI-5147 Implemented HashService. Added unit tests. commit 65b53e82112c33a568ee34445dc9dad17e1422eb Author: Andy LoPresto Date: 2018-08-31T04:31:46Z NIFI-5147 Implemented unit tests for test vectors, convenience methods, character encoding, and defaults. commit 957628ed38060817c1b90019798213b06a268318 Author: Andy LoPresto Date: 2018-08-31T04:33:01Z NIFI-5147 Refactored CalculateAttributeHash to use H
[GitHub] nifi pull request #2836: NIFI-5147 Calculate hash attribute redux
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2836#discussion_r214223024
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestCalculateAttributeHash.java
---
@@ -0,0 +1,178 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.commons.codec.binary.Hex;
+import org.apache.commons.codec.digest.DigestUtils;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class TestCalculateAttributeHash {
+
+private static final Charset UTF8 = StandardCharsets.UTF_8;
+@Test
+public void testMD2() throws Exception {
+testAllAlgorithm("MD2");
+testParitalAlgorithm("MD2");
+testMissingAlgorithm("MD2");
+}
+
+@Test
+public void testMD5() throws Exception {
+testAllAlgorithm("MD5");
+testParitalAlgorithm("MD5");
+testMissingAlgorithm("MD5");
+}
+
+@Test
+public void testSHA1() throws Exception {
+testAllAlgorithm("SHA-1");
+testParitalAlgorithm("SHA-1");
+testMissingAlgorithm("SHA-1");
+}
+
+@Test
+public void testSHA256() throws Exception {
+testAllAlgorithm("SHA-256");
+testParitalAlgorithm("SHA-256");
+testMissingAlgorithm("SHA-256");
+}
+
+@Test
+public void testSHA384() throws Exception {
+testAllAlgorithm("SHA-384");
+testParitalAlgorithm("SHA-384");
+testMissingAlgorithm("SHA-384");
+}
+
+@Test
+public void testSHA512() throws Exception {
+testAllAlgorithm("SHA-512");
+testParitalAlgorithm("SHA-512");
+testMissingAlgorithm("SHA-512");
+}
+
+public void testAllAlgorithm(String algorithm) {
+final TestRunner runner = TestRunners.newTestRunner(new
CalculateAttributeHash());
+
runner.setProperty(CalculateAttributeHash.HASH_ALGORITHM.getName(), algorithm);
+runner.setProperty("name", String.format("%s_%s", "name",
algorithm));
+runner.setProperty("value", String.format("%s_%s", "value",
algorithm));
+
+final Map attributeMap = new HashMap<>();
+attributeMap.put("name", "abcdefg");
+attributeMap.put("value", "hijklmnop");
+runner.enqueue(new byte[0], attributeMap);
+
+runner.run(1);
+
+runner.assertTransferCount(HashAttribute.REL_FAILURE, 0);
+runner.assertTransferCount(HashAttribute.REL_SUCCESS, 1);
+
+final List success =
runner.getFlowFilesForRelationship(HashAttribute.REL_SUCCESS);
+
+for (final MockFlowFile flowFile : success) {
+
Assert.assertEquals(Hex.encodeHexString(DigestUtils.getDigest(algorithm).digest("abcdefg".getBytes(UTF8))),
+flowFile.getAttribute(String.format("%s_%s", "name",
algorithm)));
+
Assert.assertEquals(Hex.encodeHexString(DigestUtils.getDigest(algorithm).digest("hijklmnop".getBytes(UTF8))),
+flowFile.getAttribute(String.format("%s_%s", "value",
algorithm)));
+}
+}
+
+p
[GitHub] nifi pull request #2836: NIFI-5147 Calculate hash attribute redux
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2836#discussion_r214222959
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestCalculateAttributeHash.java
---
@@ -0,0 +1,178 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import org.apache.commons.codec.binary.Hex;
+import org.apache.commons.codec.digest.DigestUtils;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class TestCalculateAttributeHash {
+
+private static final Charset UTF8 = StandardCharsets.UTF_8;
+@Test
+public void testMD2() throws Exception {
+testAllAlgorithm("MD2");
+testParitalAlgorithm("MD2");
+testMissingAlgorithm("MD2");
+}
+
+@Test
+public void testMD5() throws Exception {
+testAllAlgorithm("MD5");
+testParitalAlgorithm("MD5");
+testMissingAlgorithm("MD5");
+}
+
+@Test
+public void testSHA1() throws Exception {
+testAllAlgorithm("SHA-1");
+testParitalAlgorithm("SHA-1");
+testMissingAlgorithm("SHA-1");
+}
+
+@Test
+public void testSHA256() throws Exception {
+testAllAlgorithm("SHA-256");
+testParitalAlgorithm("SHA-256");
+testMissingAlgorithm("SHA-256");
+}
+
+@Test
+public void testSHA384() throws Exception {
+testAllAlgorithm("SHA-384");
+testParitalAlgorithm("SHA-384");
+testMissingAlgorithm("SHA-384");
+}
+
+@Test
+public void testSHA512() throws Exception {
+testAllAlgorithm("SHA-512");
+testParitalAlgorithm("SHA-512");
+testMissingAlgorithm("SHA-512");
+}
+
+public void testAllAlgorithm(String algorithm) {
+final TestRunner runner = TestRunners.newTestRunner(new
CalculateAttributeHash());
+
runner.setProperty(CalculateAttributeHash.HASH_ALGORITHM.getName(), algorithm);
+runner.setProperty("name", String.format("%s_%s", "name",
algorithm));
+runner.setProperty("value", String.format("%s_%s", "value",
algorithm));
+
+final Map attributeMap = new HashMap<>();
+attributeMap.put("name", "abcdefg");
+attributeMap.put("value", "hijklmnop");
+runner.enqueue(new byte[0], attributeMap);
+
+runner.run(1);
+
+runner.assertTransferCount(HashAttribute.REL_FAILURE, 0);
+runner.assertTransferCount(HashAttribute.REL_SUCCESS, 1);
+
+final List success =
runner.getFlowFilesForRelationship(HashAttribute.REL_SUCCESS);
+
+for (final MockFlowFile flowFile : success) {
+
Assert.assertEquals(Hex.encodeHexString(DigestUtils.getDigest(algorithm).digest("abcdefg".getBytes(UTF8))),
+flowFile.getAttribute(String.format("%s_%s", "name",
algorithm)));
+
Assert.assertEquals(Hex.encodeHexString(DigestUtils.getDigest(algorithm).digest("hijklmnop".getBytes(UTF8))),
+flowFile.getAttribute(String.format("%s_%s", "value",
algorithm)));
+}
+}
+
+public void testParitalAlgorithm(String algorithm) {
--- End diff --
Typo: `testPartialAlgorithm`
---
[GitHub] nifi issue #2976: NIFI-4426 - Replaced Java7 jBCrypt implementation which wa...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2976 Ran `contrib-check` and all tests pass. +1, merging. ---
[GitHub] nifi issue #2973: NIFI-5561 - Add component name filtering to S2S Provenance...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2973 Thanks Pierre. Running a final check and will merge. ---
[GitHub] nifi pull request #2973: NIFI-5561 - Add component name filtering to S2S Pro...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2973#discussion_r214106940
--- Diff:
nifi-nar-bundles/nifi-site-to-site-reporting-bundle/nifi-site-to-site-reporting-task/src/main/java/org/apache/nifi/reporting/SiteToSiteProvenanceReportingTask.java
---
@@ -151,6 +151,25 @@
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.build();
+static final PropertyDescriptor FILTER_COMPONENT_NAME = new
PropertyDescriptor.Builder()
+.name("s2s-prov-task-name-filter")
+.displayName("Component Name to Include")
+.description("Regular expression to filter the provenance events
based on the component name. Only the events matching the regular "
++ "expression will be sent. If no filter is set, all the
events are sent. If multiple filters are set, the filters are cumulative.")
+.required(false)
+.addValidator(StandardValidators.REGULAR_EXPRESSION_VALIDATOR)
--- End diff --
That's an excellent point about the VR scoping that I forgot. I think this
is nominally more "consistent" across the app, but you're right that it's not
as valuable as I expected. Thanks.
---
[GitHub] nifi issue #2976: NIFI-4426 - Replaced Java7 jBCrypt implementation which wa...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2976 Reviewing... ---
[GitHub] nifi pull request #2973: NIFI-5561 - Add component name filtering to S2S Pro...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2973#discussion_r213905518
--- Diff:
nifi-nar-bundles/nifi-site-to-site-reporting-bundle/nifi-site-to-site-reporting-task/src/main/java/org/apache/nifi/reporting/SiteToSiteProvenanceReportingTask.java
---
@@ -151,6 +151,25 @@
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.build();
+static final PropertyDescriptor FILTER_COMPONENT_NAME = new
PropertyDescriptor.Builder()
+.name("s2s-prov-task-name-filter")
+.displayName("Component Name to Include")
+.description("Regular expression to filter the provenance events
based on the component name. Only the events matching the regular "
++ "expression will be sent. If no filter is set, all the
events are sent. If multiple filters are set, the filters are cumulative.")
+.required(false)
+.addValidator(StandardValidators.REGULAR_EXPRESSION_VALIDATOR)
--- End diff --
Should these properties support Expression Language (Variables Only)? I can
imagine a variable being set with a "component name filter" specific to a
process group. Thoughts?
---
[GitHub] nifi pull request #2973: NIFI-5561 - Add component name filtering to S2S Pro...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2973#discussion_r213905074
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-reporting-utils/src/main/java/org/apache/nifi/reporting/util/provenance/ProvenanceEventConsumer.java
---
@@ -241,20 +255,24 @@ private long updateLastEventId(final
List events, final S
private boolean isFilteringEnabled() {
return componentTypeRegex != null || !eventTypes.isEmpty() ||
!componentIds.isEmpty()
-|| componentTypeRegexExclude != null ||
!eventTypesExclude.isEmpty() || !componentIdsExclude.isEmpty();
+|| componentTypeRegexExclude != null ||
!eventTypesExclude.isEmpty() || !componentIdsExclude.isEmpty()
+|| componentNameRegex != null || componentNameRegexExclude
!= null;
--- End diff --
In general, I don't push changing "traditional" code blocks to "new" style
just for the sake of it, but in this case, I think a Java 8-style `.stream()`
construction will make this clearer (some of these fields are `List` and some
are `Pattern`, and the `Pattern` can be empty/blank (which is not checked
here), which would not (logically) enable filtering.
I made two commits ([a regression test on the current
logic](https://github.com/alopresto/nifi/commit/0e717001a6124d6863f21efd18130ba996513e77)
and [a new
implementation](https://github.com/alopresto/nifi/commit/f586dd4f7cf1b0ab33c77e7053028a758ca057d1))
and it still works. Personally, I think this form is clearer and will scale
more gracefully if new properties are added. Let me know your thoughts.
---
[GitHub] nifi issue #2973: NIFI-5561 - Add component name filtering to S2S Provenance...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2973 I created a flow which uses 3 `GenerateFlowFile` processors (named _Component A_, _Component A_, and _Component A (B)_) to evaluate the new filtering behavior. Each generates a flowfile with the content and _source_ attribute indicating the source (for the 2nd processor, the message & attribute refer to _Component A (2)_). I exercised the `SiteToSiteProvenanceReportingTask` with no accept/deny list, then only an accept value, then both. The functionality worked as expected; explicit deny overrode allow (see screenshot).  The template is [available here](https://gist.github.com/alopresto/b78ec5da5482f4af7f88faf7eeefd3b7). ---
[GitHub] nifi issue #2975: NIFI-5526 Improve PutS3Object processor documentation
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2975 Ran `contrib-check` and all tests pass. +1, merging. ---
[GitHub] nifi issue #2975: NIFI-5526 Improve PutS3Object processor documentation
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2975 Reviewing... ---
[GitHub] nifi issue #2972: NIFI-5558 Fixed unit test to avoid contamination from Syst...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2972 Merging. ---
[GitHub] nifi issue #2972: NIFI-5558 Fixed unit test to avoid contamination from Syst...
Github user alopresto commented on the issue:
https://github.com/apache/nifi/pull/2972
The issue that was encountered by some users was that the previous
mechanism for creating a test `NiFiProperties` instance was
`NiFiProperties.createBasicNiFiProperties(String filePath, Map
additionalProperties)`. When provided a `null` file path, the code tried to use
`System.getProperty("nifi.properties.file.path")` to locate the
`nifi.properties` file. On my machine, the test code which was evaluating the
_empty key_ scenario returned an empty `NiFiProperties` instance with only the
overridden properties populated, but on other machines, this code was loading
the properties from `src/test/resources/conf/nifi.properties` where the
`nifi.sensitive.props.key=key`. Because of this, the two passwords were not
equal, and the keys derived from them were different. When you attempt to
decrypt data with the wrong key, a `pad block corrupted` error is common.
The tests still work for me in the following scenarios, so I need a user
who encountered the error to validate that this fixes the issue on their
system.
* via IntelliJ Run/Debug
* via IntelliJ Maven execution
* via command-line Maven execution
* specific test using `mvn clean test
-Dtest=PopularVoteFlowElectionFactoryBeanTest`
* module using `.../nifi-framework-cluster $ mvn clean test`
* full build from root
* with JCE Unlimited Strength Jurisdiction Policies
* without JCE USJP
---
[GitHub] nifi pull request #2972: NIFI-5558 Fixed unit test to avoid contamination fr...
GitHub user alopresto opened a pull request: https://github.com/apache/nifi/pull/2972 NIFI-5558 Fixed unit test to avoid contamination from System property⦠⦠"nifi.properties.file.path" when creating test NiFiProperties instance. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/alopresto/nifi NIFI-5558 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2972.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2972 commit 9c9db43bd157d84ff58e9003f646c4ff38147f56 Author: Andy LoPresto Date: 2018-08-29T02:53:45Z NIFI-5558 Fixed unit test to avoid contamination from System property "nifi.properties.file.path" when creating test NiFiProperties instance. ---
[GitHub] nifi issue #2960: NIFI-5482: Made WriteAheadProvenanceRepository the default...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2960 Ran a flow with multiple `GenerateFlowFile` processors creating flowfiles with a `0 sec` schedule and feeding to a `LogAttribute` processor. Was able to process ~350k flowfiles in about a minute. The provenance queries were successful as well. Verified the documentation now shows WAPR as the default and moved PPR properties lower in docs. Ran `contrib-check` and all tests pass. +1, merging. ---
[GitHub] nifi issue #2960: NIFI-5482: Made WriteAheadProvenanceRepository the default...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2960 Reviewing... ---
[GitHub] nifi issue #2959: NIFI-5540 Provide default `nifi.sensitive.props.key` value...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2959 As discovered by @bbende , I introduced a bug in my commit [f60585a](https://github.com/apache/nifi/commit/f60585a9b6df6b3b28be1eb80a0a60deac6c0493#diff-882d0904bb315ac8a455aec69e6d44d8) as part of [PR 2841](https://github.com/apache/nifi/pull/2841) for [NIFI-5376](https://issues.apache.org/jira/browse/NIFI-5376) which stops cluster startup if no `nifi.sensitive.props.key` value is entered in `nifi.properties`. Previously, a default static value is used (all docs indicate this should *not* be relied on and a custom, unique value should be provided at deployment time), but this is an unintentional regression and was not documented. This fix prints a warning to the log if no value is provided but restores the previous functionality of using the default. ---
[GitHub] nifi pull request #2959: NIFI-5540 Provide default `nifi.sensitive.props.key...
GitHub user alopresto opened a pull request: https://github.com/apache/nifi/pull/2959 NIFI-5540 Provide default `nifi.sensitive.props.key` value if none provided in cluster communications Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/alopresto/nifi NIFI-5540 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2959.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2959 commit 89c24274aa6808dbc1a24a4d1389a693f3ade2c4 Author: Andy LoPresto Date: 2018-08-20T22:24:39Z NIFI-5540 Added unit test to demonstrate missing default sensitive properties key in flow election encryptor creation. commit b97d6c3c7d31928c52b671bd30339b16814b4ade Author: Andy LoPresto Date: 2018-08-21T00:53:18Z NIFI-5540 Added failing unit test and ignored regression test to demonstrate missing default sensitive properties key in flow election encryptor creation. commit 694e70743c41c6b6fb7fc2b527ef339235afc3e6 Author: Andy LoPresto Date: 2018-08-21T00:54:32Z NIFI-5540 Added equality logic to StringEncryptor and utility equality methods to CryptoUtils. commit 229d12b2ecdb431e327aa9e80dd8a6c00e780a23 Author: Andy LoPresto Date: 2018-08-21T01:08:12Z NIFI-5540 Added default sensitive properties key population logic and log warning to StringEncryptor. commit 037ddfa16f709c9ddb9065d0931bca9566e7fa4a Author: Andy LoPresto Date: 2018-08-21T01:09:08Z NIFI-5540 Cleaned up formatting. commit 12c2d7ff4e2e7bb0fdadb40eed6ad594e35db029 Author: Andy LoPresto Date: 2018-08-21T01:11:49Z NIFI-5540 Cleaned up boolean logic. commit d80cc9b2af9d4ac9986eae9f383ee5a5db8511b7 Author: Andy LoPresto Date: 2018-08-21T01:14:47Z NIFI-5540 Added Javadoc to StringEncryptor. commit ca5b12bfa643d1567ca6cb77966f401e65558aa9 Author: Andy LoPresto Date: 2018-08-21T02:41:35Z NIFI-5540 Added unit test for StringEncryptor#equals(). commit 885c17c9689d60129818e9952f47921d16ddadde Author: Andy LoPresto Date: 2018-08-22T02:53:59Z NIFI-5540 Added performance benchmarking unit tests for constantTimeEquals methods for String, byte[], and char[]. commit 85d6a3fe24ded423276f2c77db1537dc1b2fc681 Author: Andy LoPresto Date: 2018-08-22T03:00:31Z NIFI-5540 Fixed checkstyle issue. commit 05db0d127c65e0b1e2de600c3aac893e99f2f925 Author: Andy LoPresto Date: 2018-08-22T03:16:44Z NIFI-5540 Fixed unit tests for default key population. ---
[GitHub] nifi issue #2615: NIFI-5051 Created ElasticSearch lookup service.
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2615 @MikeThomsen I would maybe put a comment in the `src/test/java/.gitignore` file explaining why it's there so someone in the future doesn't see it as a superfluous tooling artifact and remove it, and then your test is silently no longer executed and we don't catch regressions. We've had similar occurrences in some of the other modules. ---
[GitHub] nifi pull request #2615: NIFI-5051 Created ElasticSearch lookup service.
Github user alopresto commented on a diff in the pull request: https://github.com/apache/nifi/pull/2615#discussion_r210029087 --- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/pom.xml --- @@ -127,8 +133,113 @@ 5.6.8 compile + +org.apache.nifi +nifi-avro-record-utils +1.7.0-SNAPSHOT +compile + + +org.apache.nifi +nifi-schema-registry-service-api +compile + + +org.mockito +mockito-all +test + + --- End diff -- If there is nothing in `src/test/java`, the Groovy tests won't be detected unless a plugin references them directly. In this case, the `build-helper-maven-plugin` is accomplishing that. In other locations, the `maven-compiler-plugin` is set to use `groovy-eclipse-compiler` to achieve the same result. ---
[GitHub] nifi issue #2936: NIFI-5489: Add expression language support to AMQP process...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2936 @danieljimenez correct, PRs merged now will be included in the next release. Because our *bug fix* releases (`x.x.1`) do not include feature work, this will be in the next *minor* release, which is `1.8.0`. ---
[GitHub] nifi issue #2919: NIFI-5400 - Changed the hostname verifier from the custom ...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2919 I merged this but made two changes. First, the `nifi-web-utils` tests were not running, because they are Groovy tests and there is nothing in `src/test/java`. Without a file (even empty) in that directory, the Groovy tests do not get picked up (neither compiled nor run). I added the `groovy-eclipse-compiler` plugin to `nifi-web-utils/pom.xml` to ensure this is run. That commit is [5c0232c](https://github.com/alopresto/nifi/commit/5c0232c9dd8009dc69bc5adb1fb1ef7942832911). Second, there was a warning about a duplicate definition of `httpclient` dependency in `nifi-web-utils/pom.xml`. I removed it, and that commit is [5f538c6](https://github.com/alopresto/nifi/commit/5f538c69f1aebc0b6b0d6dbabf0f09c8e9854a57). Both of those commits were rebased onto Nathan's rebased commits as well. A gist demonstrating the issue is [here](https://gist.github.com/alopresto/184f3631ec44a4c036d323d622ea97aa). Ran `contrib-check` and all tests pass. +1, merging. ---
[GitHub] nifi issue #2935: NIFI-5476 Allow TLS toolkit to use externally-signed CA in...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2935 @pepov I've incorporated your PKCS #8 to PKCS #1 code and made some refactorings. Thank you. ---
