[GitHub] [nifi] turcsanyip commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
turcsanyip commented on code in PR #6504: URL: https://github.com/apache/nifi/pull/6504#discussion_r1039539797 ## nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/utils/PollableAsanaObjectFetcher.java: ## @@ -0,0 +1,40 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.asana.utils; + +import static java.util.Collections.emptyIterator; + +import java.util.Iterator; + +public abstract class PollableAsanaObjectFetcher implements AsanaObjectFetcher { + +private Iterator pending; + +public PollableAsanaObjectFetcher() { +pending = emptyIterator(); +} + +@Override +public AsanaObject fetchNext() { +if (!pending.hasNext()) { +pending = poll(); +} +return pending.hasNext() ? pending.next() : null; +} + +protected abstract Iterator poll(); Review Comment: The current call chain is `fetchNext()` => `poll()` => `refreshObjects()` => `fetchXXX()`. I would somehow try to consolidate these various names because they refer to the same process. E.g. `fetchNext()` => `fetch()` => `fetchObjects()` => `fetchXXX()` Not sure but `Pollable-` may not be relevant any more due to the on-the-fly processing and also because it should process a single iteration. Simply `Abstract-` might be a better name. ## nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/utils/PollableAsanaObjectFetcher.java: ## @@ -0,0 +1,40 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.asana.utils; + +import static java.util.Collections.emptyIterator; + +import java.util.Iterator; + +public abstract class PollableAsanaObjectFetcher implements AsanaObjectFetcher { + +private Iterator pending; + +public PollableAsanaObjectFetcher() { +pending = emptyIterator(); +} + +@Override +public AsanaObject fetchNext() { +if (!pending.hasNext()) { +pending = poll(); +} +return pending.hasNext() ? pending.next() : null; +} Review Comment: The `Iterator` should be retrieved only once per `onTrigger` call. The returned `Iterator` can iterate over all the currently available Asana objects. Now we have a second call (when the first `Iterator` has no more elements) but it unnecessarily queries the same Asana objects again which items will be filtered out on the NiFi side. ## nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java: ## @@ -0,0 +1,405 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language gov
[GitHub] [nifi] turcsanyip commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
turcsanyip commented on code in PR #6504: URL: https://github.com/apache/nifi/pull/6504#discussion_r1035214769 ## nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services-api/src/main/java/org/apache/nifi/controller/asana/AsanaClient.java: ## @@ -0,0 +1,195 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.controller.asana; + +import com.asana.models.Attachment; +import com.asana.models.Project; +import com.asana.models.ProjectMembership; +import com.asana.models.ProjectStatus; +import com.asana.models.Section; +import com.asana.models.Story; +import com.asana.models.Tag; +import com.asana.models.Task; +import com.asana.models.Team; +import com.asana.models.User; + +import java.util.Map; + +/** + * This interface represents a client to Asana REST server, with some basic filtering options built in. + */ +public interface AsanaClient { +/** + * Find & retrieve an Asana project by its name. If multiple projects match, returns the first. + * If there is no match, then {@link RuntimeException} is thrown. Note that constant ordering Review Comment: ```suggestion * If there is no match, then {@link AsanaClientException} is thrown. Note that constant ordering ``` Also in the javadoc of `getTeamByName()` and `getSectionByName()`. ## nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services-api-nar/src/main/resources/META-INF/NOTICE: ## @@ -0,0 +1,108 @@ +nifi-asana-services-api-nar +Copyright 2015-2022 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +=== +Apache Software License v2 +=== + +The following binary components are provided under the Apache Software License v2 + + (ASLv2) Apache Commons IO +The following NOTICE information applies: + Apache Commons IO + Copyright 2002-2022 The Apache Software Foundation Review Comment: Thanks for updating the `NOTICE` file, overall it looks good! `Commons IO` is not present in the services-api-nar so it should be removed. I forgot to mention earlier but the entries need to be added in the assembly nar as well (`nifi-assembly/NOTICE`). This "global" NOTICE file is kind of a union of all entries in the nar. Could you please copy those items to that NOTICE file which are not present there yet? ## nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java: ## @@ -0,0 +1,393 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.asana; + +import static java.lang.String.format; +import static java.util.Collections.singletonMap; +import static java.util.stream.Collectors.groupingBy; +import static java.util.stream.Collectors.joining; +import static org.apache.commons.collections4.ListUtils.partition; +import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_EVENTS; +import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_MEMBERS; +import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_ATTACHMENTS; +import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_PROJECT_STATUS_UPDATES; +import static org.apache.nifi.processors.asana.AsanaObjectType.AV_COLLECT_
[GitHub] [nifi] turcsanyip commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
turcsanyip commented on code in PR #6504: URL: https://github.com/apache/nifi/pull/6504#discussion_r1027996972 ## nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java: ## @@ -0,0 +1,430 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.asana; + +import com.google.common.collect.Lists; +import com.google.common.collect.Sets; +import org.apache.http.entity.ContentType; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.Stateful; +import org.apache.nifi.annotation.behavior.TriggerSerially; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.AllowableValue; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.state.Scope; +import org.apache.nifi.components.state.StateMap; +import org.apache.nifi.controller.asana.AsanaClient; +import org.apache.nifi.controller.asana.AsanaClientServiceApi; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.flowfile.attributes.CoreAttributes; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.processors.asana.utils.AsanaObject; +import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher; +import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher; +import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher; +import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher; +import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher; +import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher; +import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher; +import org.apache.nifi.processors.asana.utils.AsanaTagFetcher; +import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher; +import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher; +import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher; +import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher; +import org.apache.nifi.processors.asana.utils.AsanaUserFetcher; +import org.apache.nifi.reporting.InitializationException; + +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; + +@TriggerSerially +@Stateful(scopes = {Scope.LOCAL}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.") +@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN) +@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.") +@Tags({"asana", "source", "connector", "ingest"}) +@CapabilityDescription("This processor collects data from Asana") +public class GetAsanaObject extends AbstractProcessor { + +protected static final String ASANA_GID = "asana.gid"; +protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks"; +protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments"; +protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects"; +protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags"; +protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users"; +protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members"; +protected static final String AV_NAME_COLLE
[GitHub] [nifi] turcsanyip commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
turcsanyip commented on code in PR #6504: URL: https://github.com/apache/nifi/pull/6504#discussion_r1027320316 ## nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services-api/pom.xml: ## @@ -0,0 +1,39 @@ + + +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd";> +4.0.0 + + +org.apache.nifi +nifi-asana-bundle +1.19.0-SNAPSHOT + + +nifi-asana-services-api +jar + + + +org.apache.nifi +nifi-api +1.19.0-SNAPSHOT + Review Comment: `nifi-api`'s version and scope are defined in `dependecyManagement` in parent pom and they do not need to be specified here. ```suggestion org.apache.nifi nifi-api ``` ## nifi-nar-bundles/nifi-asana-bundle/nifi-asana-services/pom.xml: ## @@ -0,0 +1,64 @@ + + +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd";> +4.0.0 + + +org.apache.nifi +nifi-asana-bundle +1.19.0-SNAPSHOT + + +nifi-asana-services +jar + + + +org.apache.nifi +nifi-api +1.19.0-SNAPSHOT + + +org.apache.nifi +nifi-mock +1.19.0-SNAPSHOT +test + + +com.squareup.okhttp3 +mockwebserver +test + + +commons-io +commons-io + + +com.asana +asana + Review Comment: The `asana` dependency should be `provided` here because it is available from `nifi-asana-services-api-nar` which is the parent nar of this nar (jar-s from the parent nar will be loaded first and they are available in the child nar-s). ```suggestion com.asana asana provided ``` ## nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/pom.xml: ## @@ -0,0 +1,69 @@ + + +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd";> +4.0.0 + + +org.apache.nifi +nifi-asana-bundle +1.19.0-SNAPSHOT + + +nifi-asana-processors +jar + + + +org.apache.nifi +nifi-api +1.19.0-SNAPSHOT + + +org.apache.nifi +nifi-mock +1.19.0-SNAPSHOT +test + + +commons-io +commons-io + + +org.apache.commons +commons-collections4 +4.4 + + +com.asana +asana + + +org.apache.nifi +nifi-utils +1.19.0-SNAPSHOT +compile + + +org.apache.nifi +nifi-asana-services-api + Review Comment: Similar to `nifi-asana-services`' pom: - no version needed for `nifi-api` - `compile` scope for `nifi-utils` is redundant - `asana` and `nifi-asana-services-api` should be provided ## nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java: ## @@ -0,0 +1,457 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.asana; + +import static java.lang.String.format; +import static java.util.Collections.singletonMap; +import static java.util.stream.Collectors.groupingBy; +import static java.util.stream.Collectors.joining; +import static org.apache.commons.collections4.ListUtils.partition; + +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Arrays; +import
[GitHub] [nifi] turcsanyip commented on a diff in pull request #6504: NIFI-10618: Add Asana connector
turcsanyip commented on code in PR #6504: URL: https://github.com/apache/nifi/pull/6504#discussion_r994985443 ## nifi-nar-bundles/nifi-asana-bundle/nifi-asana-processors/src/main/java/org/apache/nifi/processors/asana/GetAsanaObject.java: ## @@ -0,0 +1,430 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.asana; + +import com.google.common.collect.Lists; +import com.google.common.collect.Sets; +import org.apache.http.entity.ContentType; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.Stateful; +import org.apache.nifi.annotation.behavior.TriggerSerially; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.AllowableValue; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.state.Scope; +import org.apache.nifi.components.state.StateMap; +import org.apache.nifi.controller.asana.AsanaClient; +import org.apache.nifi.controller.asana.AsanaClientServiceApi; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.flowfile.attributes.CoreAttributes; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.processors.asana.utils.AsanaObject; +import org.apache.nifi.processors.asana.utils.AsanaObjectFetcher; +import org.apache.nifi.processors.asana.utils.AsanaProjectEventFetcher; +import org.apache.nifi.processors.asana.utils.AsanaProjectFetcher; +import org.apache.nifi.processors.asana.utils.AsanaProjectMembershipFetcher; +import org.apache.nifi.processors.asana.utils.AsanaProjectStatusAttachmentFetcher; +import org.apache.nifi.processors.asana.utils.AsanaProjectStatusFetcher; +import org.apache.nifi.processors.asana.utils.AsanaStoryFetcher; +import org.apache.nifi.processors.asana.utils.AsanaTagFetcher; +import org.apache.nifi.processors.asana.utils.AsanaTaskAttachmentFetcher; +import org.apache.nifi.processors.asana.utils.AsanaTaskFetcher; +import org.apache.nifi.processors.asana.utils.AsanaTeamFetcher; +import org.apache.nifi.processors.asana.utils.AsanaTeamMemberFetcher; +import org.apache.nifi.processors.asana.utils.AsanaUserFetcher; +import org.apache.nifi.reporting.InitializationException; + +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; + +@TriggerSerially +@Stateful(scopes = {Scope.LOCAL}, description = "Fingerprints of items in the last successful query are stored in order to enable incremental loading and change detection.") +@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN) +@WritesAttribute(attribute = GetAsanaObject.ASANA_GID, description = "Global ID of the object in Asana.") +@Tags({"asana", "source", "connector", "ingest"}) +@CapabilityDescription("This processor collects data from Asana") +public class GetAsanaObject extends AbstractProcessor { + +protected static final String ASANA_GID = "asana.gid"; +protected static final String AV_NAME_COLLECT_TASKS = "asana-collect-tasks"; +protected static final String AV_NAME_COLLECT_TASK_ATTACHMENTS = "asana-collect-task-attachments"; +protected static final String AV_NAME_COLLECT_PROJECTS = "asana-collect-projects"; +protected static final String AV_NAME_COLLECT_TAGS = "asana-collect-tags"; +protected static final String AV_NAME_COLLECT_USERS = "asana-collect-users"; +protected static final String AV_NAME_COLLECT_PROJECT_MEMBERS = "asana-collect-project-members"; +protected static final String AV_NAME_COLLEC