[GitHub] nifi issue #2477: NIFI-4839 Adding CLI to nifi-toolkit
Github user aperepel commented on the issue: https://github.com/apache/nifi/pull/2477 @pvillard31 we were discussing a modular design for commands with dynamic discovery/loading (e.g. via Java's ServiceLoader mechanism). This is to support the idea that we should try to incorporate as much reference documentation as possible into command's output, and only add doc pages focusing on the workflow, typical use cases and best practices instead. My $.02 ---
[GitHub] nifi pull request #2465: Broaden support for input, now supports both local ...
Github user aperepel closed the pull request at: https://github.com/apache/nifi/pull/2465 ---
[GitHub] nifi issue #2465: Broaden support for input, now supports both local files _...
Github user aperepel commented on the issue: https://github.com/apache/nifi/pull/2465 Opened against a wrong fork, nm :) ---
[GitHub] nifi pull request #2465: Broaden support for input, now supports both local ...
GitHub user aperepel opened a pull request: https://github.com/apache/nifi/pull/2465 Broaden support for input, now supports both local files _and_ any public URL with a schema recognized by Java runtime. Assume UTF-8 charset by default for now. In theory would be possible to add more (sigh!) switches to specify a charset, but not sure it's worth it until we get further confirmation of the need. You can merge this pull request into a Git repository by running: $ git pull https://github.com/aperepel/nifi import-from-url Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2465.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2465 commit f97df47034346cebf9b5c7bbac2144273c0edc4c Author: Bryan Bende <bbende@...> Date: 2018-01-22T15:31:15Z Initial plumbing for new NiFi Registry toolkit tool - Setting up command framework - Check point for get-flow-version command - Working get-flow-version - Working commands for ListBuckets and CreateBucket - Adding more commands - Adding support for properties files and secure connections with certificates - Adding JLine to make a shell - Cleaning up JLine integration - Adding support for executing single commands, renaming to CLI - Adding command groups to organize commands based on target area - Refactoring how command options are setup - Refactoring how NiFiRegistryClient is created to make it easier to test and to support proxied entities - Implementing custom completer - Working NiFI commands for listing, adding, and updating Registry Clients - Refactoring to create an overall CLI context to pass around with a session, and adding session commands - Making use of session properties file for NiFi and NiFiReg commands - Working import-pg command for NiFi - Adding current user commands and session clear - Reorganizing packages, refactoring to separate API - Adding persistent session - Adding commands for pg-start, pg-stop, and pg-get-vars, and a basic README - Fixes #17 - Only print session info when interactive - Fixes #20 - Making all create commands return a single id value for easy use in automation or copying/pasting - Fixes #19 - Removing name from the options of pg-import since it should use the name from registry - Fixes #16 - Making flow version optional when exporting, if not specified then the latest will be exported - Fixes #18 - Removing flow version arg from import-flow-version, automatically determining next one-up version - Fixes #21 - Removing bucket id from export/import and automatically determining it commit 56b2c2a4edf08ba4e0c8de153fb5d78cdc744212 Author: Andrew Grande <aperepel@...> Date: 2018-02-05T16:42:31Z #24 rename the registry group to `registry` for better UX (#25) commit 0eede186f1a626221271b2671fdfbb9f404177ed Author: Bryan Bende <bbende@...> Date: 2018-02-05T16:48:33Z - Fixing completer unit test - Fixes #29 - Adding pg-get-version, pg-get-all-versions, pg-change-version - Adding info the Context to know if we are in interactive mode commit 0e0b8cbf1ded78f61bfdb040aea43daa8537997e Author: Andrew Grande <aperepel@...> Date: 2018-02-05T22:33:28Z Implemented auto-layout when importing the PG. Will find an available spot on a canvas which doesn't overlap with other components and is as close to the canvas center as possible. commit 7844927a490d14d9b8af3d029e06432eda09b42c Author: Bryan Bende <bbende@...> Date: 2018-02-07T14:42:20Z - Modifying how the process group box is calculated. This closes #31 - Adding command to get the id of a registry client by name - Fixes #28 - Refactoring how results are written to support option of simple or json output - Fixes #30 - Adding pg-set-var command commit aa9dc3fd662eb587fbcf9089a10a3a06c46a5e1f Author: Bryan Bende <bbende@...> Date: 2018-02-09T15:45:03Z Fixes #27 - Added pg-list command commit 6af1505b053036797dd8b7e2662b8715e6379580 Author: Bryan Bende <bbende@...> Date: 2018-02-09T19:26:02Z Fixes #32 - Added getDescription to every command and prints when asking for help on a command commit 3ce95cd433a27a366799f9091971ef3cbf84ed60 Author: Bryan Bende <bbende@...> Date: 2018-02-09T21:49:11Z Adding verbose out to help command to print description for every command commit 0cc85e91c4d7b99e40f4a087507638ddbf09a72c Author: Andrew Grande <aperepel@...> Date: 2018-02-10T15:08:33Z The handling of empty canvas got lost in the merge, causing errors with a new NiFi instance. commit 9dbc6277eb571968826612ed82f6b27100178451 Author: Andrew Grande <aperepel@...> Date: 2018-02-10T21:31:00Z Broaden support for input, now supportes both local files _and_ any public URL with a schema recognized by Java runtime. ---
[GitHub] nifi issue #1886: NIFI-4005 Add support for Azure Shared Access Signature (S...
Github user aperepel commented on the issue: https://github.com/apache/nifi/pull/1886 Thank you, Koji! ---
[GitHub] nifi pull request #1886: NIFI-4005 Add support for Azure Shared Access Signa...
Github user aperepel closed the pull request at: https://github.com/apache/nifi/pull/1886 ---
[GitHub] nifi issue #1885: NIFI-4012 Azure Event Hub UI typos and cleanup
Github user aperepel commented on the issue: https://github.com/apache/nifi/pull/1885 Which version did you have in mind, Pierre? I'd say the number of templates out there with the Azure event hub processor is negligible and the migration requirement could be pointed out in the release doc. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1886: NIFI-4005 Add support for Azure Shared Access Signa...
GitHub user aperepel opened a pull request: https://github.com/apache/nifi/pull/1886 NIFI-4005 Add support for Azure Shared Access Signature (SAS) Tokens - Refactoring and cleanup - Upgraded the client library from 5.0.0 to 5.2.0 You can merge this pull request into a Git repository by running: $ git pull https://github.com/aperepel/nifi NIFI-4005 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1886.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1886 commit 0ae8b2ffdf169724850a583668d3702af5516345 Author: Andrew Grande <apere...@gmail.com> Date: 2017-05-31T20:45:26Z NIFI-4005 Add support for Azure Shared Access Signature (SAS) Tokens. Upgraded the client library from 5.0.0 to 5.2.0 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1636: NIFI-1833 - Azure Storage processors
Github user aperepel commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1636#discussion_r109971519
--- Diff:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/PutAzureBlobStorage.java
---
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.storage;
+
+import java.io.BufferedInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processors.azure.AbstractAzureBlobProcessor;
+import org.apache.nifi.processors.azure.AzureConstants;
+
+import com.microsoft.azure.storage.CloudStorageAccount;
+import com.microsoft.azure.storage.StorageException;
+import com.microsoft.azure.storage.blob.BlobProperties;
+import com.microsoft.azure.storage.blob.CloudBlob;
+import com.microsoft.azure.storage.blob.CloudBlobClient;
+import com.microsoft.azure.storage.blob.CloudBlobContainer;
+
+@Tags({ "azure", "microsoft", "cloud", "storage", "blob" })
+@SeeAlso({ ListAzureBlobStorage.class, FetchAzureBlobStorage.class })
+@CapabilityDescription("Puts content into an Azure Storage Blob")
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@WritesAttributes({ @WritesAttribute(attribute = "azure.container",
description = "The name of the azure container"),
+@WritesAttribute(attribute = "azure.blobname", description = "The
name of the azure blob"),
+@WritesAttribute(attribute = "azure.primaryUri", description =
"Primary location for blob content"),
+@WritesAttribute(attribute = "azure.etag", description = "Etag for
the Azure blob"),
+@WritesAttribute(attribute = "azure.length", description = "Length
of the blob"),
+@WritesAttribute(attribute = "azure.timestamp", description = "The
timestamp in Azure for the blob"),
+@WritesAttribute(attribute = "azure.blobtype", description = "This
is the type of blob and can be either page or block type") })
+public class PutAzureBlobStorage extends AbstractAzureBlobProcessor {
+
+public void onTrigger(final ProcessContext context, final
ProcessSession session) throws ProcessException {
+FlowFile flowFile = session.get();
+if (flowFile == null) {
+return;
+}
+
+final long startNanos = System.nanoTime();
+
+String containerName =
context.getProperty(AzureConstants.CONTAINER).evaluateAttributeExpressions(flowFile).getValue();
+
+String blobPath =
context.getProperty(BLOB).evaluateAttributeExpressions(flowFile).getValue();
+
+try {
+CloudStorageAccount storageAccount =
createStorageConnection(context, flowFile);
+CloudBlobClient blobClient =
storageAccount.createCloudBlobClient();
+CloudBlobContainer container =
blobClient.getContainerReference(conta
[GitHub] nifi pull request #1636: NIFI-1833 - Azure Storage processors
Github user aperepel commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1636#discussion_r109970455
--- Diff:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/ListAzureBlobStorage.java
---
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.storage;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.security.InvalidKeyException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.azure.AzureConstants;
+import org.apache.nifi.processors.azure.storage.utils.BlobInfo;
+import org.apache.nifi.processors.azure.storage.utils.BlobInfo.Builder;
+import org.apache.nifi.processors.standard.AbstractListProcessor;
+
+import com.microsoft.azure.storage.CloudStorageAccount;
+import com.microsoft.azure.storage.OperationContext;
+import com.microsoft.azure.storage.StorageException;
+import com.microsoft.azure.storage.StorageUri;
+import com.microsoft.azure.storage.blob.BlobListingDetails;
+import com.microsoft.azure.storage.blob.BlobProperties;
+import com.microsoft.azure.storage.blob.BlobRequestOptions;
+import com.microsoft.azure.storage.blob.CloudBlob;
+import com.microsoft.azure.storage.blob.CloudBlobClient;
+import com.microsoft.azure.storage.blob.CloudBlobContainer;
+import com.microsoft.azure.storage.blob.CloudBlockBlob;
+import com.microsoft.azure.storage.blob.ListBlobItem;
+
+@TriggerSerially
+@Tags({ "azure", "microsoft", "cloud", "storage", "blob" })
+@SeeAlso({ FetchAzureBlobStorage.class })
+@CapabilityDescription("Lists blobs in an Azure Storage container. Listing
details are attached to an empty FlowFile for use with FetchAzureBlobStorage")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({ @WritesAttribute(attribute = "azure.container",
description = "The name of the azure container"),
+@WritesAttribute(attribute = "azure.blobname", description = "The
name of the azure blob"), @WritesAttribute(attribute = "azure.primaryUri",
description = "Primary location for blob content"),
+@WritesAttribute(attribute = "azure.secondaryUri", description =
"Secondary location for blob content"), @WritesAttribute(attribute =
"azure.etag", description = "Etag for the Azure blob"),
+@WritesAttribute(attribute = "azure.length", description = "Length
of the blob"), @WritesAttribute(attribute = "azure.timestamp", description =
"The timestamp in Azure for the blob"),
+@WritesAttribute(attribute = "mime.type", description = "MimeType
of the content"), @WritesAttribute(attribute = "lang", description = "Language
code for the content"),
+@WritesAttribute(attr
[GitHub] nifi pull request #1636: NIFI-1833 - Azure Storage processors
Github user aperepel commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1636#discussion_r109970180
--- Diff:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/ListAzureBlobStorage.java
---
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.storage;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.security.InvalidKeyException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.azure.AzureConstants;
+import org.apache.nifi.processors.azure.storage.utils.BlobInfo;
+import org.apache.nifi.processors.azure.storage.utils.BlobInfo.Builder;
+import org.apache.nifi.processors.standard.AbstractListProcessor;
+
+import com.microsoft.azure.storage.CloudStorageAccount;
+import com.microsoft.azure.storage.OperationContext;
+import com.microsoft.azure.storage.StorageException;
+import com.microsoft.azure.storage.StorageUri;
+import com.microsoft.azure.storage.blob.BlobListingDetails;
+import com.microsoft.azure.storage.blob.BlobProperties;
+import com.microsoft.azure.storage.blob.BlobRequestOptions;
+import com.microsoft.azure.storage.blob.CloudBlob;
+import com.microsoft.azure.storage.blob.CloudBlobClient;
+import com.microsoft.azure.storage.blob.CloudBlobContainer;
+import com.microsoft.azure.storage.blob.CloudBlockBlob;
+import com.microsoft.azure.storage.blob.ListBlobItem;
+
+@TriggerSerially
+@Tags({ "azure", "microsoft", "cloud", "storage", "blob" })
+@SeeAlso({ FetchAzureBlobStorage.class })
+@CapabilityDescription("Lists blobs in an Azure Storage container. Listing
details are attached to an empty FlowFile for use with FetchAzureBlobStorage")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({ @WritesAttribute(attribute = "azure.container",
description = "The name of the azure container"),
+@WritesAttribute(attribute = "azure.blobname", description = "The
name of the azure blob"), @WritesAttribute(attribute = "azure.primaryUri",
description = "Primary location for blob content"),
+@WritesAttribute(attribute = "azure.secondaryUri", description =
"Secondary location for blob content"), @WritesAttribute(attribute =
"azure.etag", description = "Etag for the Azure blob"),
+@WritesAttribute(attribute = "azure.length", description = "Length
of the blob"), @WritesAttribute(attribute = "azure.timestamp", description =
"The timestamp in Azure for the blob"),
+@WritesAttribute(attribute = "mime.type", description = "MimeType
of the content"), @WritesAttribute(attribute = "lang", description = "Language
code for the content"),
+@WritesAttribute(attr
[GitHub] nifi pull request #1636: NIFI-1833 - Azure Storage processors
Github user aperepel commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1636#discussion_r109969234
--- Diff:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/FetchAzureBlobStorage.java
---
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.storage;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.URISyntaxException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processors.azure.AbstractAzureBlobProcessor;
+import org.apache.nifi.processors.azure.AzureConstants;
+
+import com.microsoft.azure.storage.CloudStorageAccount;
+import com.microsoft.azure.storage.StorageException;
+import com.microsoft.azure.storage.blob.CloudBlob;
+import com.microsoft.azure.storage.blob.CloudBlobClient;
+import com.microsoft.azure.storage.blob.CloudBlobContainer;
+
+@Tags({ "azure", "microsoft", "cloud", "storage", "blob" })
+@CapabilityDescription("Retrieves contents of an Azure Storage Blob,
writing the contents to the content of the FlowFile")
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@WritesAttributes({
+@WritesAttribute(attribute = "azure.length", description = "The length
of the blob fetched")
+})
+public class FetchAzureBlobStorage extends AbstractAzureBlobProcessor {
+public static final List PROPERTIES = Collections
+.unmodifiableList(Arrays.asList(AzureConstants.ACCOUNT_NAME,
AzureConstants.ACCOUNT_KEY, AzureConstants.CONTAINER, BLOB));
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+return PROPERTIES;
+}
+
+@Override
+public void onTrigger(ProcessContext context, ProcessSession session)
throws ProcessException {
+FlowFile flowFile = session.get();
+if (flowFile == null) {
+return;
+}
+
+final long startNanos = System.nanoTime();
+
+String containerName =
context.getProperty(AzureConstants.CONTAINER).evaluateAttributeExpressions(flowFile).getValue();
+String blobPath =
context.getProperty(BLOB).evaluateAttributeExpressions(flowFile).getValue();
+
+try {
+CloudStorageAccount storageAccount =
createStorageConnection(context, flowFile);
+CloudBlobClient blobClient =
storageAccount.createCloudBlobClient();
+CloudBlobContainer container =
blobClient.getContainerReference(containerName);
+
+final Map<String, String> attributes = new HashMap<>();
+final CloudBlob blob =
container.getBlockBlobReference(blobPath);
+
+// TODO - we may be able do fancier things with ranges and
+// distribution of download over threads, investigate
+flowFile = session.write(flowFile, new OutputStreamCallback() {
--- End diff --
Done
---
If your project
[GitHub] nifi pull request #1636: NIFI-1833 - Azure Storage processors
Github user aperepel commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1636#discussion_r109969044
--- Diff:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/FetchAzureBlobStorage.java
---
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.storage;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.URISyntaxException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processors.azure.AbstractAzureBlobProcessor;
+import org.apache.nifi.processors.azure.AzureConstants;
+
+import com.microsoft.azure.storage.CloudStorageAccount;
+import com.microsoft.azure.storage.StorageException;
+import com.microsoft.azure.storage.blob.CloudBlob;
+import com.microsoft.azure.storage.blob.CloudBlobClient;
+import com.microsoft.azure.storage.blob.CloudBlobContainer;
+
+@Tags({ "azure", "microsoft", "cloud", "storage", "blob" })
+@CapabilityDescription("Retrieves contents of an Azure Storage Blob,
writing the contents to the content of the FlowFile")
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@WritesAttributes({
+@WritesAttribute(attribute = "azure.length", description = "The length
of the blob fetched")
+})
+public class FetchAzureBlobStorage extends AbstractAzureBlobProcessor {
+public static final List PROPERTIES = Collections
--- End diff --
Done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1636: NIFI-1833 - Azure Storage processors
Github user aperepel commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1636#discussion_r109968848
--- Diff:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/AbstractAzureProcessor.java
---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure;
+
+import java.net.URISyntaxException;
+import java.security.InvalidKeyException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+
+import com.microsoft.azure.storage.CloudStorageAccount;
+
+public abstract class AbstractAzureProcessor extends AbstractProcessor {
+
+public static final Relationship REL_SUCCESS = new
Relationship.Builder().name("success").description("All successfully processed
FlowFiles are routed to this relationship").build();
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure").description("Unsuccessful operations
will be transferred to the failure relationship.").build();
+public static final Set RELATIONSHIPS =
Collections.unmodifiableSet(new HashSet<>(Arrays.asList(REL_SUCCESS,
REL_FAILURE)));
+
+protected CloudStorageAccount createStorageConnection(ProcessContext
context) {
+final String accountName =
context.getProperty(AzureConstants.ACCOUNT_NAME).evaluateAttributeExpressions().getValue();
+final String accountKey =
context.getProperty(AzureConstants.ACCOUNT_KEY).evaluateAttributeExpressions().getValue();
+return createStorageConnectionFromNameAndKey(accountName,
accountKey);
+}
+
+protected CloudStorageAccount createStorageConnection(ProcessContext
context, FlowFile flowFile) {
+final String accountName =
context.getProperty(AzureConstants.ACCOUNT_NAME).evaluateAttributeExpressions(flowFile).getValue();
+final String accountKey =
context.getProperty(AzureConstants.ACCOUNT_KEY).evaluateAttributeExpressions(flowFile).getValue();
+return createStorageConnectionFromNameAndKey(accountName,
accountKey);
+}
+
+private CloudStorageAccount
createStorageConnectionFromNameAndKey(String accountName, String accountKey) {
+final String storageConnectionString =
String.format(AzureConstants.FORMAT_DEFAULT_CONNECTION_STRING, accountName,
accountKey);
+try {
+return
createStorageAccountFromConnectionString(storageConnectionString);
+} catch (InvalidKeyException | IllegalArgumentException |
URISyntaxException e) {
+throw new IllegalArgumentException(e);
+}
+}
+
+/**
+ * Validates the connection string and returns the storage account.
The connection string must be in the Azure connection string format.
+ *
+ * @param storageConnectionString
+ *Connection string for the storage service or the emulator
+ * @return The newly created CloudStorageAccount object
+ *
+ */
+protected static CloudStorageAccount
createStorageAccountFromConnectionString(String storageConnectionString) throws
IllegalArgumentException, URISyntaxException, InvalidKeyException {
+CloudStorageAccount storageAccount;
+try {
+storageAccount =
CloudStorageAccount.parse(storageConnectionString);
+} catch (IllegalArgumentException | URISyntaxException e) {
+throw e;
+} catch (InvalidKeyException e) {
--- End diff --
Done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not hav
[GitHub] nifi pull request #1636: NIFI-1833 - Azure Storage processors
Github user aperepel commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1636#discussion_r109968697
--- Diff:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/AbstractAzureProcessor.java
---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure;
+
+import java.net.URISyntaxException;
+import java.security.InvalidKeyException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+
+import com.microsoft.azure.storage.CloudStorageAccount;
+
+public abstract class AbstractAzureProcessor extends AbstractProcessor {
+
+public static final Relationship REL_SUCCESS = new
Relationship.Builder().name("success").description("All successfully processed
FlowFiles are routed to this relationship").build();
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure").description("Unsuccessful operations
will be transferred to the failure relationship.").build();
+public static final Set RELATIONSHIPS =
Collections.unmodifiableSet(new HashSet<>(Arrays.asList(REL_SUCCESS,
REL_FAILURE)));
+
+protected CloudStorageAccount createStorageConnection(ProcessContext
context) {
+final String accountName =
context.getProperty(AzureConstants.ACCOUNT_NAME).evaluateAttributeExpressions().getValue();
+final String accountKey =
context.getProperty(AzureConstants.ACCOUNT_KEY).evaluateAttributeExpressions().getValue();
+return createStorageConnectionFromNameAndKey(accountName,
accountKey);
+}
+
+protected CloudStorageAccount createStorageConnection(ProcessContext
context, FlowFile flowFile) {
+final String accountName =
context.getProperty(AzureConstants.ACCOUNT_NAME).evaluateAttributeExpressions(flowFile).getValue();
+final String accountKey =
context.getProperty(AzureConstants.ACCOUNT_KEY).evaluateAttributeExpressions(flowFile).getValue();
+return createStorageConnectionFromNameAndKey(accountName,
accountKey);
+}
+
+private CloudStorageAccount
createStorageConnectionFromNameAndKey(String accountName, String accountKey) {
+final String storageConnectionString =
String.format(AzureConstants.FORMAT_DEFAULT_CONNECTION_STRING, accountName,
accountKey);
+try {
+return
createStorageAccountFromConnectionString(storageConnectionString);
+} catch (InvalidKeyException | IllegalArgumentException |
URISyntaxException e) {
+throw new IllegalArgumentException(e);
+}
+}
+
+/**
+ * Validates the connection string and returns the storage account.
The connection string must be in the Azure connection string format.
+ *
+ * @param storageConnectionString
+ *Connection string for the storage service or the emulator
+ * @return The newly created CloudStorageAccount object
+ *
+ */
+protected static CloudStorageAccount
createStorageAccountFromConnectionString(String storageConnectionString) throws
IllegalArgumentException, URISyntaxException, InvalidKeyException {
--- End diff --
Done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1636: NIFI-1833 - Azure Storage processors
Github user aperepel commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1636#discussion_r109968145
--- Diff:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/AbstractAzureProcessor.java
---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure;
+
+import java.net.URISyntaxException;
+import java.security.InvalidKeyException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+
+import com.microsoft.azure.storage.CloudStorageAccount;
+
+public abstract class AbstractAzureProcessor extends AbstractProcessor {
+
+public static final Relationship REL_SUCCESS = new
Relationship.Builder().name("success").description("All successfully processed
FlowFiles are routed to this relationship").build();
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure").description("Unsuccessful operations
will be transferred to the failure relationship.").build();
+public static final Set RELATIONSHIPS =
Collections.unmodifiableSet(new HashSet<>(Arrays.asList(REL_SUCCESS,
REL_FAILURE)));
--- End diff --
Neither private, protected nor package-default will work, compilation
errors. Needs to be public.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1636: NIFI-1833 - Azure Storage processors
Github user aperepel commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1636#discussion_r109966858
--- Diff:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/AbstractAzureBlobProcessor.java
---
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+
+public abstract class AbstractAzureBlobProcessor extends
AbstractAzureProcessor {
+
+public static final PropertyDescriptor BLOB = new
PropertyDescriptor.Builder().name("Blob").description("The filename of the
blob").addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+
.expressionLanguageSupported(true).required(true).defaultValue("${azure.blobname}").build();
+
+public static final List properties = Collections
--- End diff --
Done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1636: NIFI-1833 - Azure Storage processors
Github user aperepel commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1636#discussion_r109812402
--- Diff:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/ListAzureBlobStorage.java
---
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.storage;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.security.InvalidKeyException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.azure.AzureConstants;
+import org.apache.nifi.processors.azure.storage.utils.BlobInfo;
+import org.apache.nifi.processors.azure.storage.utils.BlobInfo.Builder;
+import org.apache.nifi.processors.standard.AbstractListProcessor;
+
+import com.microsoft.azure.storage.CloudStorageAccount;
+import com.microsoft.azure.storage.OperationContext;
+import com.microsoft.azure.storage.StorageException;
+import com.microsoft.azure.storage.StorageUri;
+import com.microsoft.azure.storage.blob.BlobListingDetails;
+import com.microsoft.azure.storage.blob.BlobProperties;
+import com.microsoft.azure.storage.blob.BlobRequestOptions;
+import com.microsoft.azure.storage.blob.CloudBlob;
+import com.microsoft.azure.storage.blob.CloudBlobClient;
+import com.microsoft.azure.storage.blob.CloudBlobContainer;
+import com.microsoft.azure.storage.blob.CloudBlockBlob;
+import com.microsoft.azure.storage.blob.ListBlobItem;
+
+@TriggerSerially
+@Tags({ "azure", "microsoft", "cloud", "storage", "blob" })
+@SeeAlso({ FetchAzureBlobStorage.class })
+@CapabilityDescription("Lists blobs in an Azure Storage container. Listing
details are attached to an empty FlowFile for use with FetchAzureBlobStorage")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({ @WritesAttribute(attribute = "azure.container",
description = "The name of the azure container"),
+@WritesAttribute(attribute = "azure.blobname", description = "The
name of the azure blob"), @WritesAttribute(attribute = "azure.primaryUri",
description = "Primary location for blob content"),
+@WritesAttribute(attribute = "azure.secondaryUri", description =
"Secondary location for blob content"), @WritesAttribute(attribute =
"azure.etag", description = "Etag for the Azure blob"),
+@WritesAttribute(attribute = "azure.length", description = "Length
of the blob"), @WritesAttribute(attribute = "azure.timestamp", description =
"The timestamp in Azure for the blob"),
+@WritesAttribute(attribute = "mime.type", description = "MimeType
of the content"), @WritesAttribute(attribute = "lang", description = "Language
code for the content"),
+@WritesAttribute(attr
[GitHub] nifi pull request #1636: NIFI-1833 - Azure Storage processors
Github user aperepel commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1636#discussion_r109800464
--- Diff:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/ListAzureBlobStorage.java
---
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.storage;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.security.InvalidKeyException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.azure.AzureConstants;
+import org.apache.nifi.processors.azure.storage.utils.BlobInfo;
+import org.apache.nifi.processors.azure.storage.utils.BlobInfo.Builder;
+import org.apache.nifi.processors.standard.AbstractListProcessor;
+
+import com.microsoft.azure.storage.CloudStorageAccount;
+import com.microsoft.azure.storage.OperationContext;
+import com.microsoft.azure.storage.StorageException;
+import com.microsoft.azure.storage.StorageUri;
+import com.microsoft.azure.storage.blob.BlobListingDetails;
+import com.microsoft.azure.storage.blob.BlobProperties;
+import com.microsoft.azure.storage.blob.BlobRequestOptions;
+import com.microsoft.azure.storage.blob.CloudBlob;
+import com.microsoft.azure.storage.blob.CloudBlobClient;
+import com.microsoft.azure.storage.blob.CloudBlobContainer;
+import com.microsoft.azure.storage.blob.CloudBlockBlob;
+import com.microsoft.azure.storage.blob.ListBlobItem;
+
+@TriggerSerially
+@Tags({ "azure", "microsoft", "cloud", "storage", "blob" })
+@SeeAlso({ FetchAzureBlobStorage.class })
+@CapabilityDescription("Lists blobs in an Azure Storage container. Listing
details are attached to an empty FlowFile for use with FetchAzureBlobStorage")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({ @WritesAttribute(attribute = "azure.container",
description = "The name of the azure container"),
+@WritesAttribute(attribute = "azure.blobname", description = "The
name of the azure blob"), @WritesAttribute(attribute = "azure.primaryUri",
description = "Primary location for blob content"),
+@WritesAttribute(attribute = "azure.secondaryUri", description =
"Secondary location for blob content"), @WritesAttribute(attribute =
"azure.etag", description = "Etag for the Azure blob"),
+@WritesAttribute(attribute = "azure.length", description = "Length
of the blob"), @WritesAttribute(attribute = "azure.timestamp", description =
"The timestamp in Azure for the blob"),
+@WritesAttribute(attribute = "mime.type", description = "MimeType
of the content"), @WritesAttribute(attribute = "lang", description = "Language
code for the content"),
+@WritesAttribute(attr
[GitHub] nifi pull request #1636: NIFI-1833 - Azure Storage processors
Github user aperepel commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1636#discussion_r109800429
--- Diff:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/ListAzureBlobStorage.java
---
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.storage;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.security.InvalidKeyException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.azure.AzureConstants;
+import org.apache.nifi.processors.azure.storage.utils.BlobInfo;
+import org.apache.nifi.processors.azure.storage.utils.BlobInfo.Builder;
+import org.apache.nifi.processors.standard.AbstractListProcessor;
+
+import com.microsoft.azure.storage.CloudStorageAccount;
+import com.microsoft.azure.storage.OperationContext;
+import com.microsoft.azure.storage.StorageException;
+import com.microsoft.azure.storage.StorageUri;
+import com.microsoft.azure.storage.blob.BlobListingDetails;
+import com.microsoft.azure.storage.blob.BlobProperties;
+import com.microsoft.azure.storage.blob.BlobRequestOptions;
+import com.microsoft.azure.storage.blob.CloudBlob;
+import com.microsoft.azure.storage.blob.CloudBlobClient;
+import com.microsoft.azure.storage.blob.CloudBlobContainer;
+import com.microsoft.azure.storage.blob.CloudBlockBlob;
+import com.microsoft.azure.storage.blob.ListBlobItem;
+
+@TriggerSerially
+@Tags({ "azure", "microsoft", "cloud", "storage", "blob" })
+@SeeAlso({ FetchAzureBlobStorage.class })
+@CapabilityDescription("Lists blobs in an Azure Storage container. Listing
details are attached to an empty FlowFile for use with FetchAzureBlobStorage")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({ @WritesAttribute(attribute = "azure.container",
description = "The name of the azure container"),
+@WritesAttribute(attribute = "azure.blobname", description = "The
name of the azure blob"), @WritesAttribute(attribute = "azure.primaryUri",
description = "Primary location for blob content"),
+@WritesAttribute(attribute = "azure.secondaryUri", description =
"Secondary location for blob content"), @WritesAttribute(attribute =
"azure.etag", description = "Etag for the Azure blob"),
+@WritesAttribute(attribute = "azure.length", description = "Length
of the blob"), @WritesAttribute(attribute = "azure.timestamp", description =
"The timestamp in Azure for the blob"),
+@WritesAttribute(attribute = "mime.type", description = "MimeType
of the content"), @WritesAttribute(attribute = "lang", description = "Language
code for the content"),
+@WritesAttribute(attr
[GitHub] nifi pull request #1636: NIFI-1833 - Azure Storage processors
Github user aperepel commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1636#discussion_r109797526
--- Diff:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/ListAzureBlobStorage.java
---
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.storage;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.security.InvalidKeyException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.azure.AzureConstants;
+import org.apache.nifi.processors.azure.storage.utils.BlobInfo;
+import org.apache.nifi.processors.azure.storage.utils.BlobInfo.Builder;
+import org.apache.nifi.processors.standard.AbstractListProcessor;
+
+import com.microsoft.azure.storage.CloudStorageAccount;
+import com.microsoft.azure.storage.OperationContext;
+import com.microsoft.azure.storage.StorageException;
+import com.microsoft.azure.storage.StorageUri;
+import com.microsoft.azure.storage.blob.BlobListingDetails;
+import com.microsoft.azure.storage.blob.BlobProperties;
+import com.microsoft.azure.storage.blob.BlobRequestOptions;
+import com.microsoft.azure.storage.blob.CloudBlob;
+import com.microsoft.azure.storage.blob.CloudBlobClient;
+import com.microsoft.azure.storage.blob.CloudBlobContainer;
+import com.microsoft.azure.storage.blob.CloudBlockBlob;
+import com.microsoft.azure.storage.blob.ListBlobItem;
+
+@TriggerSerially
+@Tags({ "azure", "microsoft", "cloud", "storage", "blob" })
+@SeeAlso({ FetchAzureBlobStorage.class })
+@CapabilityDescription("Lists blobs in an Azure Storage container. Listing
details are attached to an empty FlowFile for use with FetchAzureBlobStorage")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({ @WritesAttribute(attribute = "azure.container",
description = "The name of the azure container"),
+@WritesAttribute(attribute = "azure.blobname", description = "The
name of the azure blob"), @WritesAttribute(attribute = "azure.primaryUri",
description = "Primary location for blob content"),
+@WritesAttribute(attribute = "azure.secondaryUri", description =
"Secondary location for blob content"), @WritesAttribute(attribute =
"azure.etag", description = "Etag for the Azure blob"),
+@WritesAttribute(attribute = "azure.length", description = "Length
of the blob"), @WritesAttribute(attribute = "azure.timestamp", description =
"The timestamp in Azure for the blob"),
+@WritesAttribute(attribute = "mime.type", description = "MimeType
of the content"), @WritesAttribute(attribute = "lang", description = "Language
code for the content"),
+@WritesAttribute(attr
[GitHub] nifi pull request #1636: NIFI-1833 - Azure Storage processors
Github user aperepel commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1636#discussion_r109796618
--- Diff:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/AbstractAzureProcessor.java
---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure;
+
+import java.net.URISyntaxException;
+import java.security.InvalidKeyException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+
+import com.microsoft.azure.storage.CloudStorageAccount;
+
+public abstract class AbstractAzureProcessor extends AbstractProcessor {
+
+public static final Relationship REL_SUCCESS = new
Relationship.Builder().name("success").description("All FlowFiles that are
received are routed to success").build();
+protected static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure").description("Any failed fetches will be
transferred to the failure relation.").build();
--- End diff --
Updated
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1636: NIFI-1833 - Azure Storage processors
Github user aperepel commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1636#discussion_r109749394
--- Diff: nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/pom.xml
---
@@ -57,5 +81,10 @@ language governing permissions and limitations under the
License. -->
${powermock.version}
test
+
+org.apache.nifi
+nifi-standard-processors
--- End diff --
Done, verified changes with a live Azure account.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1636: NIFI-1833 - Azure Storage processors
Github user aperepel commented on a diff in the pull request: https://github.com/apache/nifi/pull/1636#discussion_r109749346 --- Diff: nifi-nar-bundles/nifi-azure-bundle/nifi-azure-nar/pom.xml --- @@ -35,6 +35,12 @@ nifi-azure-processors 1.2.0-SNAPSHOT + + +org.apache.nifi +nifi-standard-services-api-nar --- End diff -- Done, verified changes with a live Azure account. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1636: NIFI-1833 - Azure Storage processors
Github user aperepel commented on the issue: https://github.com/apache/nifi/pull/1636 Initial PR submission, will inquire about checkstyle rules --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1636: NIFI-1833 - Azure Storage processors
GitHub user aperepel opened a pull request: https://github.com/apache/nifi/pull/1636 NIFI-1833 - Azure Storage processors Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/aperepel/nifi NIFI-1833-pr Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1636.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1636 commit 1dcdd76783f0eac126cff69fbbb689439433dd86 Author: Simon Elliston Ball <si...@simonellistonball.com> Date: 2016-05-01T23:35:34Z NIFI-1833 - Azure Storage processors --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
