I have created a Jira - NIFI-5156 <https://issues.apache.org/jira/browse/NIFI-5156> and made the necessary changes to the current nifi-gcp-bundle. I have tested the components - GCS Processors & GCP Credentials Service. They all work as expected. Anyone want to give it a look and review the changes? PR #2680 <https://github.com/apache/nifi/pull/2680>
- Sivaprasanna On Thu, May 3, 2018 at 9:48 PM, Sivaprasanna <sivaprasanna...@gmail.com> wrote: > I'm interested although it might consume some time since I don't know how > big it is going to be. And I suppose it is better to capture it in a > separate Jira. The summary could be upgrade and refactor GCP processor > codebase or something like that. We could then make NIFI-5133 a dependent > of the new Jira. Thoughts? > > Meanwhile, anyone who has any other recommendations, feel free to share. :) > > - > Sivaprasanna > > On Thu, May 3, 2018 at 9:43 PM, Joe Witt <joe.w...@gmail.com> wrote: > >> Sivaprasanna >> >> Ok makes sense. Are you in a position of interest/expertise/time to >> make those changes as well? >> >> Thanks >> >> On Thu, May 3, 2018 at 12:11 PM, Sivaprasanna <sivaprasanna...@gmail.com> >> wrote: >> > Hi. As I had mentioned, upgrading to the latest version of the library >> is >> > not as simple as I thought. Google Cloud team introduced many breaking >> > changes. Many of the APIs (classes & methods) have been >> > scrapped/replaced/modified/refactored/renamed. >> > >> > In short, a simple change of version may demand changes on the >> processor's >> > code, especially on the AbstractProcessors (AbstractGCS, AbstractGCP) >> which >> > may pose backward compatibility issues, I'm afraid. >> > >> > Thanks, >> > Sivaprasanna >> > >> > On Thu, May 3, 2018 at 9:26 PM, Joe Witt <joe.w...@gmail.com> wrote: >> > >> >> Sivaprasanna >> >> >> >> I might not completely follow but is there a 3rd option to upgrade to >> >> a more recent library and solve the use of the proper jars >> >> problem(smaller nar)? >> >> >> >> Thanks >> >> >> >> On Thu, May 3, 2018 at 11:51 AM, Sivaprasanna < >> sivaprasanna...@gmail.com> >> >> wrote: >> >> > Hi >> >> > >> >> > I've started the initial works on implementing Google Cloud Pub/Sub >> >> > processors. The associated Jira ID is NIFI-5133 >> >> > <https://issues.apache.org/jira/browse/NIFI-5133>. This will go to >> the >> >> > existing GCP bundle which currently has only the storage processors. >> Upon >> >> > some inspection, I noticed the following: >> >> > >> >> > - As of now, the bundle uses google-cloud >> >> > <http://mvnrepository.com/artifact/com.google.cloud/google-cloud> >> as >> >> its >> >> > dependency which is like uber/fat jar that contains most of the >> Google >> >> > Cloud's client library SDKs including storage, bigquery, pubsub, >> etc. >> >> The >> >> > main point is it is using a very older version (0.8.0) >> >> > - I thought of using google-cloud-bom >> >> > <http://mvnrepository.com/artifact/com.google.cloud/google- >> cloud-bom> >> >> in >> >> > the bundle's POM >> >> > <https://github.com/apache/nifi/blob/master/nifi-nar- >> >> bundles/nifi-gcp-bundle/pom.xml> >> >> > and then use the required artifacts in the processor's POM >> >> > <https://github.com/apache/nifi/blob/master/nifi-nar- >> >> bundles/nifi-gcp-bundle/nifi-gcp-processors/pom.xml>. >> >> > The benefit is, it will help us reduce the overall size of the >> NAR. >> >> > >> >> > When I tried to do #2, I realized this is not a simple version change >> >> but a >> >> > change that brings backward compatibility issues. Ex: Some APIs used >> in >> >> the >> >> > older version i.e. 0.8.0 have now been entirely scrapped and moved to >> >> > different library. We can do either two things: >> >> > >> >> > 1. User the Pub/Sub APIs from the older version but the problem is >> >> it's >> >> > very old and the problem of upgrading would soon catchup with us. >> >> > 2. Or we can continue to use the older version of >> google-cloud-storage >> >> > only for the storage processors and introduce the #2 mentioned >> above >> >> but I >> >> > don't think then the new processors can't properly extend the >> existing >> >> > AbstractGCPProcessor >> >> > <https://github.com/apache/nifi/blob/master/nifi-nar- >> >> bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/ >> >> org/apache/nifi/processors/gcp/AbstractGCPProcessor.java>. >> >> > >> >> > >> >> > A quick glance on the processor code and the POM would help you >> >> understand >> >> > my concern. >> >> > >> >> > I'm stuck up here so any help & guidance in this regard is very much >> >> > appreciated. :) >> >> > >> >> > Thanks, >> >> > >> >> > Sivaprasanna >> >> >> > >
