[GitHub] nifi issue #2420: NIFI-4731
Github user danieljimenez commented on the issue: https://github.com/apache/nifi/pull/2420 There are two other PRs that supersede this one. ---
[GitHub] nifi issue #3019: [NIFI-4731][NIFI-4933] Big Query processor
Github user danieljimenez commented on the issue: https://github.com/apache/nifi/pull/3019 LGTM ---
[GitHub] nifi issue #2682: NIFI-4731: BQ Processors and GCP library update.
Github user danieljimenez commented on the issue: https://github.com/apache/nifi/pull/2682 Hey @pvillard31, I'm 100% okay with that. I'll even add you to my fork if you want. I wont be able to circle back to this for a couple of weeks at least. Thanks for all your help. ---
[GitHub] nifi issue #2682: NIFI-4731: BQ Processors and GCP library update.
Github user danieljimenez commented on the issue: https://github.com/apache/nifi/pull/2682 Is there anything I can do to ensure this makes it to 1.8? Thanks! ---
[GitHub] nifi issue #2682: NIFI-4731: BQ Processors and GCP library update.
Github user danieljimenez commented on the issue: https://github.com/apache/nifi/pull/2682 Plus I have no way to test. ---
[GitHub] nifi issue #2682: NIFI-4731: BQ Processors and GCP library update.
Github user danieljimenez commented on the issue: https://github.com/apache/nifi/pull/2682 I'd rather not, the scope of this keeps moving. I'd like to keep it focused on master right now. ---
[GitHub] nifi issue #2936: NIFI-5489: Add expression language support to AMQP process...
Github user danieljimenez commented on the issue: https://github.com/apache/nifi/pull/2936 FWIW I am currently using a SNAPSHOT from 01782bf on a real RabbitMQ server. ---
[GitHub] nifi issue #2936: NIFI-5489: Add expression language support to AMQP process...
Github user danieljimenez commented on the issue: https://github.com/apache/nifi/pull/2936 @mosermw Port added sir. :) ---
[GitHub] nifi issue #2936: NIFI-5489: Add expression language support to AMQP process...
Github user danieljimenez commented on the issue: https://github.com/apache/nifi/pull/2936 @mosermw I can, I just enabled the fields most likely to be variables for most use cases. If you'd like me to add that to this PR, I certainly can. ---
[GitHub] nifi issue #2936: NIFI-5489: Add expression language support to AMQP process...
Github user danieljimenez commented on the issue: https://github.com/apache/nifi/pull/2936 Missed a spot evaluating the attribute expressions. Fixed. ---
[GitHub] nifi issue #2936: NIFI-5489: Add expression language support to AMQP process...
Github user danieljimenez commented on the issue: https://github.com/apache/nifi/pull/2936 Awesome, thanks! That would mean it's targeted for 1.8.0, correct? ---
[GitHub] nifi issue #2936: NIFI-5489: Add expression language support to AMQP process...
Github user danieljimenez commented on the issue: https://github.com/apache/nifi/pull/2936 After rerunning the Travis failures are gone. ---
[GitHub] nifi issue #2682: NIFI-4731: BQ Processors and GCP library update.
Github user danieljimenez commented on the issue: https://github.com/apache/nifi/pull/2682 @pvillard31 thanks, I've added that and force pushed an update. ---
[GitHub] nifi issue #2682: NIFI-4731: BQ Processors and GCP library update.
Github user danieljimenez commented on the issue: https://github.com/apache/nifi/pull/2682 I'm not sure about the RAT plugin, but this is ready otherwise. ---
[GitHub] nifi issue #2936: NIFI-5489: Add expression language support to AMQP process...
Github user danieljimenez commented on the issue: https://github.com/apache/nifi/pull/2936 No idea what the travis failures are, they are definitely not related to this change. ---
[GitHub] nifi pull request #2936: NIFI-5489: Add expression language support to AMQP ...
GitHub user danieljimenez opened a pull request: https://github.com/apache/nifi/pull/2936 NIFI-5489: Add expression language support to AMQP processors HOST, V⦠â¦HOST and USER Fields. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [x] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [x] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/danieljimenez/nifi NIFI-5489-amqp-attribute-expression-support Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2936.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2936 commit 1a9c9d5cd8759125ec344a9ad389b282fb40267d Author: Daniel Jimenez Date: 2018-08-04T16:11:28Z NIFI-5489: Add expression language support to AMQP processors HOST, VHOST and USER Fields. ---
[GitHub] nifi pull request #2682: NIFI-4731: BQ Processors and GCP library update.
Github user danieljimenez commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2682#discussion_r206191230
--- Diff:
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/bigquery/PutBigQueryBatch.java
---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.gcp.bigquery;
+
+import com.google.cloud.bigquery.BigQuery;
+import com.google.cloud.bigquery.FormatOptions;
+import com.google.cloud.bigquery.Job;
+import com.google.cloud.bigquery.JobInfo;
+import com.google.cloud.bigquery.Schema;
+import com.google.cloud.bigquery.TableDataWriteChannel;
+import com.google.cloud.bigquery.TableId;
+import com.google.cloud.bigquery.WriteChannelConfiguration;
+import com.google.common.collect.ImmutableList;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.LogLevel;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.gcp.storage.DeleteGCSObject;
+import org.apache.nifi.processors.gcp.storage.PutGCSObject;
+import org.apache.nifi.util.StringUtils;
+
+import java.nio.ByteBuffer;
+import java.nio.channels.Channels;
+import java.nio.channels.ReadableByteChannel;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * A processor for batch loading data into a Google BigQuery table
+ */
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@Tags({"google", "google cloud", "bq", "bigquery"})
+@CapabilityDescription("Batch loads flow files to a Google BigQuery
table.")
+@SeeAlso({PutGCSObject.class, DeleteGCSObject.class,
PutBigQueryStream.class})
+
+@WritesAttributes({
+@WritesAttribute(attribute = BigQueryAttributes.DATASET_ATTR,
description = BigQueryAttributes.DATASET_DESC),
+@WritesAttribute(attribute = BigQueryAttributes.TABLE_NAME_ATTR,
description = BigQueryAttributes.TABLE_NAME_DESC),
+@WritesAttribute(attribute = BigQueryAttributes.TABLE_SCHEMA_ATTR,
description = BigQueryAttributes.TABLE_SCHEMA_DESC),
+@WritesAttribute(attribute = BigQueryAttributes.SOURCE_TYPE_ATTR,
description = BigQueryAttributes.SOURCE_TYPE_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.IGNORE_UNKNOWN_ATTR, description =
BigQueryAttributes.IGNORE_UNKNOWN_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.CREATE_DISPOSITION_ATTR, description =
BigQueryAttributes.CREATE_DISPOSITION_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.WRITE_DISPOSITION_ATTR, description =
BigQueryAttributes.WRITE_DISPOSITION_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.MAX_BADRECORDS_ATTR, description =
BigQueryAttributes.MAX_BADRECORDS_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.JOB_CREATE_TIME_ATTR, description =
BigQueryAttributes.JOB_CREATE_TIME_DESC),
+@WritesAttribute(attribute = BigQueryAttributes.JOB_END_TIME_ATTR,
description = BigQueryAttributes.JOB_END_TIME_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.JOB_START_TIME_ATTR, description =
BigQueryAttributes.JOB_START_TIME_DESC),
+@Writ
[GitHub] nifi pull request #2682: NIFI-4731: BQ Processors and GCP library update.
Github user danieljimenez commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2682#discussion_r206158466
--- Diff:
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/bigquery/PutBigQueryBatch.java
---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.gcp.bigquery;
+
+import com.google.cloud.bigquery.BigQuery;
+import com.google.cloud.bigquery.FormatOptions;
+import com.google.cloud.bigquery.Job;
+import com.google.cloud.bigquery.JobInfo;
+import com.google.cloud.bigquery.Schema;
+import com.google.cloud.bigquery.TableDataWriteChannel;
+import com.google.cloud.bigquery.TableId;
+import com.google.cloud.bigquery.WriteChannelConfiguration;
+import com.google.common.collect.ImmutableList;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.LogLevel;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.gcp.storage.DeleteGCSObject;
+import org.apache.nifi.processors.gcp.storage.PutGCSObject;
+import org.apache.nifi.util.StringUtils;
+
+import java.nio.ByteBuffer;
+import java.nio.channels.Channels;
+import java.nio.channels.ReadableByteChannel;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * A processor for batch loading data into a Google BigQuery table
+ */
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@Tags({"google", "google cloud", "bq", "bigquery"})
+@CapabilityDescription("Batch loads flow files to a Google BigQuery
table.")
+@SeeAlso({PutGCSObject.class, DeleteGCSObject.class,
PutBigQueryStream.class})
+
+@WritesAttributes({
+@WritesAttribute(attribute = BigQueryAttributes.DATASET_ATTR,
description = BigQueryAttributes.DATASET_DESC),
+@WritesAttribute(attribute = BigQueryAttributes.TABLE_NAME_ATTR,
description = BigQueryAttributes.TABLE_NAME_DESC),
+@WritesAttribute(attribute = BigQueryAttributes.TABLE_SCHEMA_ATTR,
description = BigQueryAttributes.TABLE_SCHEMA_DESC),
+@WritesAttribute(attribute = BigQueryAttributes.SOURCE_TYPE_ATTR,
description = BigQueryAttributes.SOURCE_TYPE_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.IGNORE_UNKNOWN_ATTR, description =
BigQueryAttributes.IGNORE_UNKNOWN_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.CREATE_DISPOSITION_ATTR, description =
BigQueryAttributes.CREATE_DISPOSITION_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.WRITE_DISPOSITION_ATTR, description =
BigQueryAttributes.WRITE_DISPOSITION_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.MAX_BADRECORDS_ATTR, description =
BigQueryAttributes.MAX_BADRECORDS_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.JOB_CREATE_TIME_ATTR, description =
BigQueryAttributes.JOB_CREATE_TIME_DESC),
+@WritesAttribute(attribute = BigQueryAttributes.JOB_END_TIME_ATTR,
description = BigQueryAttributes.JOB_END_TIME_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.JOB_START_TIME_ATTR, description =
BigQueryAttributes.JOB_START_TIME_DESC),
+@Writ
[GitHub] nifi issue #2682: NIFI-4731: BQ Processors and GCP library update.
Github user danieljimenez commented on the issue: https://github.com/apache/nifi/pull/2682 Hi, sorry for the rebases. This is ready now. ---
[GitHub] nifi pull request #2682: NIFI-4731: BQ Processors and GCP library update.
Github user danieljimenez commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2682#discussion_r204721738
--- Diff:
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/bigquery/BigQueryAttributes.java
---
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.gcp.bigquery;
+
+/**
+ * Attributes associated with the BigQuery processors
+ */
+public class BigQueryAttributes {
+private BigQueryAttributes() {}
+
+public static final String DATASET_ATTR = "bq.dataset";
--- End diff --
@zenfenan After looking at this, that doesn't seem the match the defaults
used by the other gcp processors. i.e. `gcs.bucket`.
---
[GitHub] nifi issue #2826: NIFI-4535 Set the Page Title to the name of the Root Proce...
Github user danieljimenez commented on the issue: https://github.com/apache/nifi/pull/2826 Three cheers.\ ---
[GitHub] nifi pull request #2682: NIFI-4731: BQ Processors and GCP library update.
Github user danieljimenez commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2682#discussion_r189218048
--- Diff:
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/bigquery/BigQueryAttributes.java
---
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.gcp.bigquery;
+
+/**
+ * Attributes associated with the BigQuery processors
+ */
+public class BigQueryAttributes {
+private BigQueryAttributes() {}
+
+public static final String DATASET_ATTR = "bq.dataset";
--- End diff --
That sounds okay to me, I'll try and cover that in the rebase.
---
[GitHub] nifi pull request #2682: NIFI-4731: BQ Processors and GCP library update.
Github user danieljimenez commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2682#discussion_r186428592
--- Diff:
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/AbstractGCPProcessor.java
---
@@ -36,14 +36,14 @@
*/
public abstract class AbstractGCPProcessor<
CloudService extends Service,
-CloudServiceRpc,
-CloudServiceOptions extends HttpServiceOptions<CloudService,
CloudServiceRpc, CloudServiceOptions>> extends AbstractProcessor {
+CloudServiceOptions extends ServiceOptions<CloudService,
CloudServiceOptions>> extends AbstractProcessor {
public static final PropertyDescriptor PROJECT_ID = new
PropertyDescriptor
.Builder().name("gcp-project-id")
.displayName("Project ID")
.description("Google Cloud Project ID")
.required(true)
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
--- End diff --
I added variable registry support to the GCP project ID.
---
[GitHub] nifi pull request #2682: NIFI-4731: BQ Processors and GCP library update.
GitHub user danieljimenez opened a pull request: https://github.com/apache/nifi/pull/2682 NIFI-4731: BQ Processors and GCP library update. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ X] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X ] Is your initial contribution a single, squashed commit? ### For code changes: - [X ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ X] Have you written or updated unit tests to verify your changes? - [NA] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [NA] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [NA] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-ssembly? - [NA] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [X] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/danieljimenez/nifi bigquery-1.7 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2682.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2682 commit fc372089a16eec95533d9eb8dfeb17b4822b9c4b Author: Michal Misiewicz <misiewicz.michall@...> Date: 2018-04-29T08:01:34Z NIFI-4731: BQ Processors and GCP library update. ---
