Re: [PR] NIFI-10222: Add schema registry service for Apicurio [nifi]
exceptionfactory closed pull request #7913: NIFI-10222: Add schema registry service for Apicurio URL: https://github.com/apache/nifi/pull/7913 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-10222: Add schema registry service for Apicurio [nifi]
takraj commented on code in PR #7913: URL: https://github.com/apache/nifi/pull/7913#discussion_r1379847196 ## nifi-nar-bundles/nifi-apicurio-bundle/nifi-apicurio-nar/src/main/resources/META-INF/NOTICE: ## @@ -0,0 +1,101 @@ +nifi-airtable-nar +Copyright 2014-2023 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +** +Apache Software License v2 +** + + (ASLv2) Jackson Annotations +The following NOTICE information applies: + Jackson Annotations + Copyright (c) 2019 Tatu Saloranta + + (ASLv2) Jackson Core +The following NOTICE information applies: + Jackson Core + Copyright (c) Tatu Saloranta + + (ASLv2) Jackson Databind +The following NOTICE information applies: + Jackson Databind + Copyright (c) 2007- Tatu Saloranta, tatu.salora...@iki.fi + + Jackson is a high-performance, Free/Open Source JSON processing library. + It was originally written by Tatu Saloranta (tatu.salora...@iki.fi), and has + been in development since 2007. + + It is currently developed by a community of developers, as well as supported + commercially by FasterXML.com. + + (ASLv2) Caffeine (com.github.ben-manes.caffeine:caffeine:jar:3.1.8 - https://github.com/ben-manes/caffeine) +The following NOTICE information applies: + Caffeine (caching library) + Copyright Ben Manes + + (ASLv2) Error Prone Annotations +The following NOTICE information applies: + Error Prone Annotations + Copyright 2015 The Error Prone Authors + + (ASLv2) Apache Commons IO +The following NOTICE information applies: + Apache Commons IO + Copyright 2002-2016 The Apache Software Foundation + + (ASLv2) Apache Avro +The following NOTICE information applies: + Apache Avro + Copyright 2009-2017 The Apache Software Foundation + + (ASLv2) Apache Commons Compress +The following NOTICE information applies: + Apache Commons Compress + Copyright 2002-2014 The Apache Software Foundation + + The files in the package org.apache.commons.compress.archivers.sevenz + were derived from the LZMA SDK, version 9.20 (C/ and CPP/7zip/), + which has been placed in the public domain: + + "LZMA SDK is placed in the public domain." (https://www.7-zip.org/sdk.html) + + (ASLv2) Apache Commons CSV +The following NOTICE information applies: + Apache Commons CSV + Copyright 2005-2016 The Apache Software Foundation + + (ASLv2) Apache Commons Lang +The following NOTICE information applies: + Apache Commons Lang + Copyright 2001-2017 The Apache Software Foundation + + This product includes software from the Spring Framework, + under the Apache License 2.0 (see: StringUtils.containsWhitespace()) + + (ASLv2) Apache Commons Text +The following NOTICE information applies: + Apache Commons Text + Copyright 2001-2018 The Apache Software Foundation + + +Eclipse Distribution License 1.0 Review Comment: OK. This was only something I've spotted, but I'm not sure if it is needed to be included or not. ## nifi-nar-bundles/nifi-apicurio-bundle/nifi-apicurio-nar/src/main/resources/META-INF/NOTICE: ## @@ -0,0 +1,101 @@ +nifi-airtable-nar +Copyright 2014-2023 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +** +Apache Software License v2 +** + + (ASLv2) Jackson Annotations +The following NOTICE information applies: + Jackson Annotations + Copyright (c) 2019 Tatu Saloranta + + (ASLv2) Jackson Core +The following NOTICE information applies: + Jackson Core + Copyright (c) Tatu Saloranta + + (ASLv2) Jackson Databind +The following NOTICE information applies: + Jackson Databind + Copyright (c) 2007- Tatu Saloranta, tatu.salora...@iki.fi + + Jackson is a high-performance, Free/Open Source JSON processing library. + It was originally written by Tatu Saloranta (tatu.salora...@iki.fi), and has + been in development since 2007. + + It is currently developed by a community of developers, as well as supported + commercially by FasterXML.com. + + (ASLv2) Caffeine (com.github.ben-manes.caffeine:caffeine:jar:3.1.8 - https://github.com/ben-manes/caffeine) +The following NOTICE information applies: + Caffeine (caching library) + Copyright Ben Manes + + (ASLv2) Error Prone Annotations +The following NOTICE information applies: + Error Prone Annotations + Copyright 2015 The Error Prone Authors + + (ASLv2) Apache Commons IO +The following NOTICE information applies: + Apache Commons IO + Copyright 2002-2016 The Apache Software Foundation + + (ASLv2) Apache Avro +The
Re: [PR] NIFI-10222: Add schema registry service for Apicurio [nifi]
Lehel44 commented on code in PR #7913:
URL: https://github.com/apache/nifi/pull/7913#discussion_r1378866814
##
nifi-nar-bundles/nifi-apicurio-bundle/nifi-apicurio-schema-registry-service/src/main/java/org/apache/nifi/apicurio/schemaregistry/ApicurioSchemaRegistry.java:
##
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.apicurio.schemaregistry;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import
org.apache.nifi.apicurio.schemaregistry.client.ApicurioSchemaRegistryClient;
+import
org.apache.nifi.apicurio.schemaregistry.client.CachingSchemaRegistryClient;
+import org.apache.nifi.apicurio.schemaregistry.client.SchemaRegistryApiClient;
+import org.apache.nifi.apicurio.schemaregistry.client.SchemaRegistryClient;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.SchemaField;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.schemaregistry.services.SchemaRegistry;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.SchemaIdentifier;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.OptionalInt;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+@Tags({"schema", "registry", "apicurio", "avro"})
+@CapabilityDescription("Provides a Schema Registry that interacts with the
Apicurio Schema Registry so that those Schemas that are stored in the Apicurio
Schema "
++ "Registry can be used in NiFi. When a Schema is looked up by name by
this registry, it will find a Schema in the Apicurio Schema Registry with their
names.")
+public class ApicurioSchemaRegistry extends AbstractControllerService
implements SchemaRegistry {
+
+private static final Set schemaFields =
EnumSet.of(SchemaField.SCHEMA_NAME, SchemaField.SCHEMA_TEXT,
+SchemaField.SCHEMA_TEXT_FORMAT, SchemaField.SCHEMA_IDENTIFIER,
SchemaField.SCHEMA_VERSION);
+
+static final PropertyDescriptor SCHEMA_REGISTRY_URL = new
PropertyDescriptor.Builder()
+.name("schema-registry-url")
+.displayName("Schema Registry URL")
Review Comment:
Wow, okay, that makes sense!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-10222: Add schema registry service for Apicurio [nifi]
exceptionfactory commented on code in PR #7913:
URL: https://github.com/apache/nifi/pull/7913#discussion_r1378749889
##
nifi-nar-bundles/nifi-apicurio-bundle/nifi-apicurio-schema-registry-service/src/main/java/org/apache/nifi/apicurio/schemaregistry/ApicurioSchemaRegistry.java:
##
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.apicurio.schemaregistry;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import
org.apache.nifi.apicurio.schemaregistry.client.ApicurioSchemaRegistryClient;
+import
org.apache.nifi.apicurio.schemaregistry.client.CachingSchemaRegistryClient;
+import org.apache.nifi.apicurio.schemaregistry.client.SchemaRegistryApiClient;
+import org.apache.nifi.apicurio.schemaregistry.client.SchemaRegistryClient;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.SchemaField;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.schemaregistry.services.SchemaRegistry;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.SchemaIdentifier;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.OptionalInt;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+@Tags({"schema", "registry", "apicurio", "avro"})
+@CapabilityDescription("Provides a Schema Registry that interacts with the
Apicurio Schema Registry so that those Schemas that are stored in the Apicurio
Schema "
++ "Registry can be used in NiFi. When a Schema is looked up by name by
this registry, it will find a Schema in the Apicurio Schema Registry with their
names.")
+public class ApicurioSchemaRegistry extends AbstractControllerService
implements SchemaRegistry {
+
+private static final Set schemaFields =
EnumSet.of(SchemaField.SCHEMA_NAME, SchemaField.SCHEMA_TEXT,
+SchemaField.SCHEMA_TEXT_FORMAT, SchemaField.SCHEMA_IDENTIFIER,
SchemaField.SCHEMA_VERSION);
+
+static final PropertyDescriptor SCHEMA_REGISTRY_URL = new
PropertyDescriptor.Builder()
+.name("schema-registry-url")
+.displayName("Schema Registry URL")
Review Comment:
Good question, the recent addition of
[NIFI-12139](https://issues.apache.org/jira/browse/NIFI-12139) provides new API
methods to migrate existing property names to new property names. Part of this
capability provides the opportunity to unify the property name and display
name, which makes it more straightforward to translate component configuration
through the UI to programmatic calls. Previously, the displayName provided a
way to rename properties without breaking existing flows, but with the new
migrate properties capability, that will no longer be necessary. For new
components, there is the opportunity to keep these values the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-10222: Add schema registry service for Apicurio [nifi]
Lehel44 commented on PR #7913: URL: https://github.com/apache/nifi/pull/7913#issuecomment-174594 Thanks for the comments everyone, added all the requested changes and tested it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-10222: Add schema registry service for Apicurio [nifi]
Lehel44 commented on code in PR #7913:
URL: https://github.com/apache/nifi/pull/7913#discussion_r1378738385
##
nifi-nar-bundles/nifi-apicurio-bundle/nifi-apicurio-schema-registry-service/src/main/java/org/apache/nifi/apicurio/schemaregistry/ApicurioSchemaRegistry.java:
##
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.apicurio.schemaregistry;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import
org.apache.nifi.apicurio.schemaregistry.client.ApicurioSchemaRegistryClient;
+import
org.apache.nifi.apicurio.schemaregistry.client.CachingSchemaRegistryClient;
+import org.apache.nifi.apicurio.schemaregistry.client.SchemaRegistryApiClient;
+import org.apache.nifi.apicurio.schemaregistry.client.SchemaRegistryClient;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.SchemaField;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.schemaregistry.services.SchemaRegistry;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.SchemaIdentifier;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.OptionalInt;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+@Tags({"schema", "registry", "apicurio", "avro"})
+@CapabilityDescription("Provides a Schema Registry that interacts with the
Apicurio Schema Registry so that those Schemas that are stored in the Apicurio
Schema "
++ "Registry can be used in NiFi. When a Schema is looked up by name by
this registry, it will find a Schema in the Apicurio Schema Registry with their
names.")
+public class ApicurioSchemaRegistry extends AbstractControllerService
implements SchemaRegistry {
+
+private static final Set schemaFields =
EnumSet.of(SchemaField.SCHEMA_NAME, SchemaField.SCHEMA_TEXT,
+SchemaField.SCHEMA_TEXT_FORMAT, SchemaField.SCHEMA_IDENTIFIER,
SchemaField.SCHEMA_VERSION);
+
+static final PropertyDescriptor SCHEMA_REGISTRY_URL = new
PropertyDescriptor.Builder()
+.name("schema-registry-url")
+.displayName("Schema Registry URL")
Review Comment:
Where does this new change come from?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-10222: Add schema registry service for Apicurio [nifi]
exceptionfactory commented on code in PR #7913:
URL: https://github.com/apache/nifi/pull/7913#discussion_r1378323529
##
nifi-nar-bundles/nifi-apicurio-bundle/nifi-apicurio-schema-registry-service/pom.xml:
##
@@ -0,0 +1,71 @@
+
+
+http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
https://maven.apache.org/xsd/maven-4.0.0.xsd";>
+4.0.0
+
+
+org.apache.nifi
+nifi-apicurio-bundle
+2.0.0-SNAPSHOT
+
+
+nifi-apicurio-schema-registry-service
+jar
+
+
+org.apache.nifi
+nifi-api
+provided
Review Comment:
This is the default setting based on parent configuration, so this is not
needed.
```suggestion
```
##
nifi-nar-bundles/nifi-apicurio-bundle/nifi-apicurio-schema-registry-service/src/main/java/org/apache/nifi/apicurio/schemaregistry/util/SchemaUtils.java:
##
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.apicurio.schemaregistry.util;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.avro.Schema;
+import org.apache.avro.SchemaParseException;
+import org.apache.nifi.avro.AvroTypeUtil;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.SchemaIdentifier;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.UncheckedIOException;
+
+public class SchemaUtils {
+private static final ObjectMapper OBJECT_MAPPER = setObjectMapper();
+
+private SchemaUtils() {
+}
+
+public static RecordSchema createRecordSchema(final InputStream
schemaStream, final String name, final int version) throws
SchemaNotFoundException, IOException {
+final JsonNode schemaNode = OBJECT_MAPPER.readTree(schemaStream);
+final String schemaText = schemaNode.toString();
+
+try {
+final Schema avroSchema = new Schema.Parser().parse(schemaText);
+final SchemaIdentifier schemaId = SchemaIdentifier.builder()
+.name(name)
+.version(version)
+.build();
+return AvroTypeUtil.createSchema(avroSchema, schemaText, schemaId);
+} catch (final SchemaParseException spe) {
+throw new SchemaNotFoundException("Obtained Schema with name " +
name
++ " from Apicurio Schema Registry but the Schema Text that
was returned is not a valid Avro Schema");
+}
+}
+
+public static int getVersionAttribute(InputStream in) {
+final JsonNode metadataNode;
+try {
+metadataNode = OBJECT_MAPPER.readTree(in);
+} catch (IOException e) {
+throw new UncheckedIOException("Failed to read HTTP response input
stream", e);
+}
+return Integer.parseInt(metadataNode.get("version").asText());
+}
+
+public static ResultAttributes getResultAttributes(InputStream in) {
+final JsonNode jsonNode;
+try {
+jsonNode = OBJECT_MAPPER.readTree(in);
+} catch (IOException e) {
+throw new UncheckedIOException("Failed to read HTTP response input
stream", e);
Review Comment:
Similar to above, I recommend changing the message to read `Failed to read
result properties from HTTP response stream`
##
nifi-nar-bundles/nifi-apicurio-bundle/nifi-apicurio-schema-registry-service/src/main/java/org/apache/nifi/apicurio/schemaregistry/ApicurioSchemaRegistry.java:
##
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENS
Re: [PR] NIFI-10222: Add schema registry service for Apicurio [nifi]
exceptionfactory commented on code in PR #7913: URL: https://github.com/apache/nifi/pull/7913#discussion_r1378322735 ## nifi-nar-bundles/nifi-apicurio-bundle/nifi-apicurio-nar/pom.xml: ## @@ -0,0 +1,44 @@ + + +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> + +nifi-apicurio-bundle +org.apache.nifi +2.0.0-SNAPSHOT + +4.0.0 + +nifi-apicurio-nar + +nar + +true +true + + + +org.apache.nifi +nifi-apicurio-schema-registry-service +2.0.0-SNAPSHOT + + +org.apache.nifi +nifi-standard-services-api-nar Review Comment: Yes, this should use the `nifi-standard-shared-nar` following the pattern of recent changes in many modules, which will allow this bundle to use common versions of shared libraries. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-10222: Add schema registry service for Apicurio [nifi]
Lehel44 commented on code in PR #7913: URL: https://github.com/apache/nifi/pull/7913#discussion_r1378254243 ## nifi-nar-bundles/nifi-apicurio-bundle/nifi-apicurio-schema-registry-service/src/main/java/org/apache/nifi/apicurio/schemaregistry/client/ApicurioSchemaRegistryClient.java: ## @@ -0,0 +1,66 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.apicurio.schemaregistry.client; + +import org.apache.nifi.apicurio.schemaregistry.util.SchemaUtils.ResultAttributes; +import org.apache.nifi.schema.access.SchemaNotFoundException; +import org.apache.nifi.serialization.record.RecordSchema; + +import java.io.IOException; +import java.io.InputStream; +import java.net.URI; + +import static org.apache.nifi.apicurio.schemaregistry.util.SchemaUtils.createRecordSchema; Review Comment: I like it because it makes the code more concise :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-10222: Add schema registry service for Apicurio [nifi]
Lehel44 commented on code in PR #7913:
URL: https://github.com/apache/nifi/pull/7913#discussion_r1378253719
##
nifi-nar-bundles/nifi-apicurio-bundle/nifi-apicurio-schema-registry-service/src/main/java/org/apache/nifi/apicurio/schemaregistry/client/CachingSchemaRegistryClient.java:
##
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.apicurio.schemaregistry.client;
+
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.LoadingCache;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import java.time.Duration;
+
+public class CachingSchemaRegistryClient implements SchemaRegistryClient {
+private final SchemaRegistryClient client;
+private final LoadingCache nameCache;
+private final LoadingCache, RecordSchema>
nameVersionCache;
Review Comment:
Thanks for the suggestion. We have precedents in RestSchemaRegistry and
GlueSchemaRegistry but I think this makes sense and viable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-10222: Add schema registry service for Apicurio [nifi]
Lehel44 commented on code in PR #7913:
URL: https://github.com/apache/nifi/pull/7913#discussion_r1378246162
##
nifi-nar-bundles/nifi-apicurio-bundle/nifi-apicurio-schema-registry-service/src/main/java/org/apache/nifi/apicurio/schemaregistry/util/SchemaUtils.java:
##
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.apicurio.schemaregistry.util;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.avro.Schema;
+import org.apache.avro.SchemaParseException;
+import org.apache.nifi.avro.AvroTypeUtil;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.SchemaIdentifier;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.UncheckedIOException;
+
+public class SchemaUtils {
+private static final ObjectMapper OBJECT_MAPPER = setObjectMapper();
+
+private SchemaUtils() {
+}
+
+public static RecordSchema createRecordSchema(final InputStream
schemaStream, final String name, final int version) throws
SchemaNotFoundException, IOException {
+final JsonNode schemaNode = OBJECT_MAPPER.readTree(schemaStream);
+final String schemaText = schemaNode.toString();
+
+try {
+final Schema avroSchema = new Schema.Parser().parse(schemaText);
+final SchemaIdentifier schemaId = SchemaIdentifier.builder()
+.name(name)
+.version(version)
+.build();
+return AvroTypeUtil.createSchema(avroSchema, schemaText, schemaId);
+} catch (final SchemaParseException spe) {
+throw new SchemaNotFoundException("Obtained Schema with name " +
name
++ " from Apicurio Schema Registry but the Schema Text that
was returned is not a valid Avro Schema");
+}
+}
+
+public static int getVersionAttribute(InputStream in) {
+final JsonNode metadataNode;
+try {
+metadataNode = OBJECT_MAPPER.readTree(in);
+} catch (IOException e) {
+throw new UncheckedIOException("Failed to read HTTP response input
stream", e);
Review Comment:
Thanks for the suggestion! This is a commonly used pattern in NiFi.
ProcessExceptions are related to FlowFile handling in my understanding and
according to its JavaDoc. I think this one fits better here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-10222: Add schema registry service for Apicurio [nifi]
Lehel44 commented on code in PR #7913: URL: https://github.com/apache/nifi/pull/7913#discussion_r1377764130 ## nifi-nar-bundles/nifi-apicurio-bundle/nifi-apicurio-nar/src/main/resources/META-INF/NOTICE: ## @@ -0,0 +1,101 @@ +nifi-airtable-nar +Copyright 2014-2023 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +** +Apache Software License v2 +** + + (ASLv2) Jackson Annotations +The following NOTICE information applies: + Jackson Annotations + Copyright (c) 2019 Tatu Saloranta + + (ASLv2) Jackson Core +The following NOTICE information applies: + Jackson Core + Copyright (c) Tatu Saloranta + + (ASLv2) Jackson Databind +The following NOTICE information applies: + Jackson Databind + Copyright (c) 2007- Tatu Saloranta, tatu.salora...@iki.fi + + Jackson is a high-performance, Free/Open Source JSON processing library. + It was originally written by Tatu Saloranta (tatu.salora...@iki.fi), and has + been in development since 2007. + + It is currently developed by a community of developers, as well as supported + commercially by FasterXML.com. + + (ASLv2) Caffeine (com.github.ben-manes.caffeine:caffeine:jar:3.1.8 - https://github.com/ben-manes/caffeine) +The following NOTICE information applies: + Caffeine (caching library) + Copyright Ben Manes + + (ASLv2) Error Prone Annotations +The following NOTICE information applies: + Error Prone Annotations + Copyright 2015 The Error Prone Authors + + (ASLv2) Apache Commons IO +The following NOTICE information applies: + Apache Commons IO + Copyright 2002-2016 The Apache Software Foundation + + (ASLv2) Apache Avro +The following NOTICE information applies: + Apache Avro + Copyright 2009-2017 The Apache Software Foundation + + (ASLv2) Apache Commons Compress +The following NOTICE information applies: + Apache Commons Compress + Copyright 2002-2014 The Apache Software Foundation + + The files in the package org.apache.commons.compress.archivers.sevenz + were derived from the LZMA SDK, version 9.20 (C/ and CPP/7zip/), + which has been placed in the public domain: + + "LZMA SDK is placed in the public domain." (https://www.7-zip.org/sdk.html) + + (ASLv2) Apache Commons CSV +The following NOTICE information applies: + Apache Commons CSV + Copyright 2005-2016 The Apache Software Foundation + + (ASLv2) Apache Commons Lang +The following NOTICE information applies: + Apache Commons Lang + Copyright 2001-2017 The Apache Software Foundation + + This product includes software from the Spring Framework, + under the Apache License 2.0 (see: StringUtils.containsWhitespace()) + + (ASLv2) Apache Commons Text +The following NOTICE information applies: + Apache Commons Text + Copyright 2001-2018 The Apache Software Foundation + + +Eclipse Distribution License 1.0 + + +The following binary components are provided under the Eclipse Distribution License 1.0. + +(EDL 1.0) Jakarta Activation API (jakarta.activation:jakarta.activation-api:jar:1.2.2) +(EDL 1.0) Jakarta XML Binding API (jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.3) + + +The MIT License Review Comment: I don't see examples mentioning Mit License in the LICENSE files. Am I missing anything? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-10222: Add schema registry service for Apicurio [nifi]
Lehel44 commented on code in PR #7913: URL: https://github.com/apache/nifi/pull/7913#discussion_r1377763639 ## nifi-nar-bundles/nifi-apicurio-bundle/nifi-apicurio-nar/src/main/resources/META-INF/NOTICE: ## @@ -0,0 +1,101 @@ +nifi-airtable-nar +Copyright 2014-2023 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +** +Apache Software License v2 +** + + (ASLv2) Jackson Annotations +The following NOTICE information applies: + Jackson Annotations + Copyright (c) 2019 Tatu Saloranta + + (ASLv2) Jackson Core +The following NOTICE information applies: + Jackson Core + Copyright (c) Tatu Saloranta + + (ASLv2) Jackson Databind +The following NOTICE information applies: + Jackson Databind + Copyright (c) 2007- Tatu Saloranta, tatu.salora...@iki.fi + + Jackson is a high-performance, Free/Open Source JSON processing library. + It was originally written by Tatu Saloranta (tatu.salora...@iki.fi), and has + been in development since 2007. + + It is currently developed by a community of developers, as well as supported + commercially by FasterXML.com. + + (ASLv2) Caffeine (com.github.ben-manes.caffeine:caffeine:jar:3.1.8 - https://github.com/ben-manes/caffeine) +The following NOTICE information applies: + Caffeine (caching library) + Copyright Ben Manes + + (ASLv2) Error Prone Annotations +The following NOTICE information applies: + Error Prone Annotations + Copyright 2015 The Error Prone Authors + + (ASLv2) Apache Commons IO +The following NOTICE information applies: + Apache Commons IO + Copyright 2002-2016 The Apache Software Foundation + + (ASLv2) Apache Avro +The following NOTICE information applies: + Apache Avro + Copyright 2009-2017 The Apache Software Foundation + + (ASLv2) Apache Commons Compress +The following NOTICE information applies: + Apache Commons Compress + Copyright 2002-2014 The Apache Software Foundation + + The files in the package org.apache.commons.compress.archivers.sevenz + were derived from the LZMA SDK, version 9.20 (C/ and CPP/7zip/), + which has been placed in the public domain: + + "LZMA SDK is placed in the public domain." (https://www.7-zip.org/sdk.html) + + (ASLv2) Apache Commons CSV +The following NOTICE information applies: + Apache Commons CSV + Copyright 2005-2016 The Apache Software Foundation + + (ASLv2) Apache Commons Lang +The following NOTICE information applies: + Apache Commons Lang + Copyright 2001-2017 The Apache Software Foundation + + This product includes software from the Spring Framework, + under the Apache License 2.0 (see: StringUtils.containsWhitespace()) + + (ASLv2) Apache Commons Text +The following NOTICE information applies: + Apache Commons Text + Copyright 2001-2018 The Apache Software Foundation + + +Eclipse Distribution License 1.0 Review Comment: I don't see examples mentioning Eclipse Distribution in the LICENSE files. Am I missing anything? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-10222: Add schema registry service for Apicurio [nifi]
ChrisSamo632 commented on code in PR #7913: URL: https://github.com/apache/nifi/pull/7913#discussion_r1373309714 ## nifi-nar-bundles/pom.xml: ## @@ -114,6 +114,7 @@ nifi-py4j-bundle nifi-compress-bundle nifi-opentelemetry-bundle +nifi-apicurio-bundle Review Comment: The `nar` also needs adding to the `nifi-assembly` if going to be included in the build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-10222: Add schema registry service for Apicurio [nifi]
ChrisSamo632 commented on code in PR #7913:
URL: https://github.com/apache/nifi/pull/7913#discussion_r1373301484
##
nifi-nar-bundles/nifi-apicurio-bundle/nifi-apicurio-schema-registry-service/src/main/java/org/apache/nifi/apicurio/schemaregistry/client/ApicurioSchemaRegistryClient.java:
##
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.apicurio.schemaregistry.client;
+
+import
org.apache.nifi.apicurio.schemaregistry.util.SchemaUtils.ResultAttributes;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URI;
+
+import static
org.apache.nifi.apicurio.schemaregistry.util.SchemaUtils.createRecordSchema;
+import static
org.apache.nifi.apicurio.schemaregistry.util.SchemaUtils.getResultAttributes;
+import static
org.apache.nifi.apicurio.schemaregistry.util.SchemaUtils.getVersionAttribute;
+
+public class ApicurioSchemaRegistryClient implements SchemaRegistryClient {
+private final SchemaRegistryApiClient apiClient;
+
+public ApicurioSchemaRegistryClient(SchemaRegistryApiClient apiClient) {
+this.apiClient = apiClient;
+}
+
+@Override
+public RecordSchema getSchema(String schemaName) throws IOException,
SchemaNotFoundException {
+final URI searchUri = apiClient.buildSearchUri(schemaName);
+final InputStream searchResultStream =
apiClient.retrieveResponse(searchUri);
+final ResultAttributes attributes =
getResultAttributes(searchResultStream);
Review Comment:
Fetching `attributes` looks like common code that could be shared with
`getSchema` below - refactor to a separate method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-10222: Add schema registry service for Apicurio [nifi]
ChrisSamo632 commented on code in PR #7913:
URL: https://github.com/apache/nifi/pull/7913#discussion_r1373300281
##
nifi-nar-bundles/nifi-apicurio-bundle/nifi-apicurio-schema-registry-service/src/main/java/org/apache/nifi/apicurio/schemaregistry/client/ApicurioSchemaRegistryClient.java:
##
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.apicurio.schemaregistry.client;
+
+import
org.apache.nifi.apicurio.schemaregistry.util.SchemaUtils.ResultAttributes;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URI;
+
+import static
org.apache.nifi.apicurio.schemaregistry.util.SchemaUtils.createRecordSchema;
+import static
org.apache.nifi.apicurio.schemaregistry.util.SchemaUtils.getResultAttributes;
+import static
org.apache.nifi.apicurio.schemaregistry.util.SchemaUtils.getVersionAttribute;
+
+public class ApicurioSchemaRegistryClient implements SchemaRegistryClient {
+private final SchemaRegistryApiClient apiClient;
+
+public ApicurioSchemaRegistryClient(SchemaRegistryApiClient apiClient) {
+this.apiClient = apiClient;
+}
+
+@Override
+public RecordSchema getSchema(String schemaName) throws IOException,
SchemaNotFoundException {
+final URI searchUri = apiClient.buildSearchUri(schemaName);
+final InputStream searchResultStream =
apiClient.retrieveResponse(searchUri);
+final ResultAttributes attributes =
getResultAttributes(searchResultStream);
+
+final URI metaDataUri =
apiClient.buildMetaDataUri(attributes.groupId(), attributes.artifactId());
+final InputStream metadataResultStream =
apiClient.retrieveResponse(metaDataUri);
+final int version = getVersionAttribute(metadataResultStream);
+
+final URI schemaUri = apiClient.buildSchemaUri(attributes.groupId(),
attributes.artifactId());
+final InputStream schemaResultStream =
apiClient.retrieveResponse(schemaUri);
+
+return createRecordSchema(schemaResultStream, attributes.name(),
version);
Review Comment:
Seems to be common code that could be its own method (repeated in
`getSchema`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-10222: Add schema registry service for Apicurio [nifi]
ChrisSamo632 commented on code in PR #7913:
URL: https://github.com/apache/nifi/pull/7913#discussion_r1373298694
##
nifi-nar-bundles/nifi-apicurio-bundle/nifi-apicurio-schema-registry-service/src/main/java/org/apache/nifi/apicurio/schemaregistry/client/ApicurioSchemaRegistryClient.java:
##
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.apicurio.schemaregistry.client;
+
+import
org.apache.nifi.apicurio.schemaregistry.util.SchemaUtils.ResultAttributes;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URI;
+
+import static
org.apache.nifi.apicurio.schemaregistry.util.SchemaUtils.createRecordSchema;
+import static
org.apache.nifi.apicurio.schemaregistry.util.SchemaUtils.getResultAttributes;
+import static
org.apache.nifi.apicurio.schemaregistry.util.SchemaUtils.getVersionAttribute;
+
+public class ApicurioSchemaRegistryClient implements SchemaRegistryClient {
+private final SchemaRegistryApiClient apiClient;
+
+public ApicurioSchemaRegistryClient(SchemaRegistryApiClient apiClient) {
+this.apiClient = apiClient;
+}
+
+@Override
+public RecordSchema getSchema(String schemaName) throws IOException,
SchemaNotFoundException {
+final URI searchUri = apiClient.buildSearchUri(schemaName);
+final InputStream searchResultStream =
apiClient.retrieveResponse(searchUri);
+final ResultAttributes attributes =
getResultAttributes(searchResultStream);
+
+final URI metaDataUri =
apiClient.buildMetaDataUri(attributes.groupId(), attributes.artifactId());
+final InputStream metadataResultStream =
apiClient.retrieveResponse(metaDataUri);
+final int version = getVersionAttribute(metadataResultStream);
+
+final URI schemaUri = apiClient.buildSchemaUri(attributes.groupId(),
attributes.artifactId());
+final InputStream schemaResultStream =
apiClient.retrieveResponse(schemaUri);
+
+return createRecordSchema(schemaResultStream, attributes.name(),
version);
+}
+
+
+@Override
+public RecordSchema getSchema(final String schemaName, final int version)
throws IOException, SchemaNotFoundException {
+final URI searchUri = apiClient.buildSearchUri(schemaName);
+final InputStream searchResultStream =
apiClient.retrieveResponse(searchUri);
Review Comment:
`InputStream`s should be closed after they're no longer needed, otherwise
could lead to resource leakage
##
nifi-nar-bundles/nifi-apicurio-bundle/nifi-apicurio-schema-registry-service/src/main/java/org/apache/nifi/apicurio/schemaregistry/client/ApicurioSchemaRegistryClient.java:
##
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.apicurio.schemaregistry.client;
+
+import
org.apache.nifi.apicurio.schemaregistry.util.SchemaUtils.ResultAttributes;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URI;
+
+import static
org.apache.nifi.apicurio.schemaregistry.util.SchemaUtils.createRecordSchema;
+import static
org.apache.nifi.apicurio.schemaregistry.util.SchemaUtils.getResultAttributes;
+import static
org.apache.nifi.apicurio.sch
Re: [PR] NIFI-10222: Add schema registry service for Apicurio [nifi]
ChrisSamo632 commented on code in PR #7913:
URL: https://github.com/apache/nifi/pull/7913#discussion_r1373292964
##
nifi-nar-bundles/nifi-apicurio-bundle/nifi-apicurio-schema-registry-service/src/main/java/org/apache/nifi/apicurio/schemaregistry/util/SchemaUtils.java:
##
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.apicurio.schemaregistry.util;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.avro.Schema;
+import org.apache.avro.SchemaParseException;
+import org.apache.nifi.avro.AvroTypeUtil;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.SchemaIdentifier;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.UncheckedIOException;
+
+public class SchemaUtils {
+private static final ObjectMapper OBJECT_MAPPER = setObjectMapper();
+
+private SchemaUtils() {
+}
+
+public static RecordSchema createRecordSchema(final InputStream
schemaStream, final String name, final int version) throws
SchemaNotFoundException, IOException {
+final JsonNode schemaNode = OBJECT_MAPPER.readTree(schemaStream);
+final String schemaText = schemaNode.toString();
+
+try {
+final Schema avroSchema = new Schema.Parser().parse(schemaText);
+final SchemaIdentifier schemaId = SchemaIdentifier.builder()
+.name(name)
+.version(version)
+.build();
+return AvroTypeUtil.createSchema(avroSchema, schemaText, schemaId);
+} catch (final SchemaParseException spe) {
+throw new SchemaNotFoundException("Obtained Schema with name " +
name
Review Comment:
At least log the `SchemaParseException`, it might well contain useful
information such as details of what's invalid in the schema being parsed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-10222: Add schema registry service for Apicurio [nifi]
ChrisSamo632 commented on code in PR #7913: URL: https://github.com/apache/nifi/pull/7913#discussion_r1373291035 ## nifi-nar-bundles/nifi-apicurio-bundle/nifi-apicurio-schema-registry-service/src/main/java/org/apache/nifi/apicurio/schemaregistry/client/ApicurioSchemaRegistryClient.java: ## @@ -0,0 +1,66 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.apicurio.schemaregistry.client; + +import org.apache.nifi.apicurio.schemaregistry.util.SchemaUtils.ResultAttributes; +import org.apache.nifi.schema.access.SchemaNotFoundException; +import org.apache.nifi.serialization.record.RecordSchema; + +import java.io.IOException; +import java.io.InputStream; +import java.net.URI; + +import static org.apache.nifi.apicurio.schemaregistry.util.SchemaUtils.createRecordSchema; Review Comment: Personally not a fan of such static imports, but I don't think it blocks the PR being merged -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-10222: Add schema registry service for Apicurio [nifi]
ChrisSamo632 commented on code in PR #7913:
URL: https://github.com/apache/nifi/pull/7913#discussion_r1373285994
##
nifi-nar-bundles/nifi-apicurio-bundle/nifi-apicurio-schema-registry-service/src/main/java/org/apache/nifi/apicurio/schemaregistry/client/CachingSchemaRegistryClient.java:
##
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.apicurio.schemaregistry.client;
+
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.LoadingCache;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.nifi.serialization.record.RecordSchema;
+
+import java.time.Duration;
+
+public class CachingSchemaRegistryClient implements SchemaRegistryClient {
+private final SchemaRegistryClient client;
+private final LoadingCache nameCache;
+private final LoadingCache, RecordSchema>
nameVersionCache;
Review Comment:
Seems a bit confusing to have a single `Cache Size` property but then have
two caches - would it be better to combine these together, passing the
`OptionalInt` from the `ApicuriSchemaRegistry` then using a default value (e.g.
`-1` that wouldn't be valid as input to the service normally) to keep
everything in a single cache, e.g. `nameVersionCache` - would that make more
sense, or do we have precendent of multiple caches elsewhere already?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-10222: Add schema registry service for Apicurio [nifi]
ChrisSamo632 commented on code in PR #7913: URL: https://github.com/apache/nifi/pull/7913#discussion_r1373274561 ## nifi-nar-bundles/nifi-apicurio-bundle/nifi-apicurio-nar/pom.xml: ## @@ -0,0 +1,44 @@ + + +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> + +nifi-apicurio-bundle +org.apache.nifi +2.0.0-SNAPSHOT + +4.0.0 + +nifi-apicurio-nar + +nar + +true +true + + + +org.apache.nifi +nifi-apicurio-schema-registry-service +2.0.0-SNAPSHOT + + +org.apache.nifi +nifi-standard-services-api-nar Review Comment: Should this be the (new) `nifi-standard-shared-nar` instead? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-10222: Add schema registry service for Apicurio [nifi]
takraj commented on code in PR #7913:
URL: https://github.com/apache/nifi/pull/7913#discussion_r1372811693
##
nifi-nar-bundles/nifi-apicurio-bundle/nifi-apicurio-nar/src/main/resources/META-INF/NOTICE:
##
@@ -0,0 +1,101 @@
+nifi-airtable-nar
Review Comment:
```suggestion
nifi-apicurio-nar
```
##
nifi-nar-bundles/nifi-apicurio-bundle/nifi-apicurio-nar/src/main/resources/META-INF/NOTICE:
##
@@ -0,0 +1,101 @@
+nifi-airtable-nar
+Copyright 2014-2023 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+**
+Apache Software License v2
+**
+
+ (ASLv2) Jackson Annotations
+The following NOTICE information applies:
+ Jackson Annotations
+ Copyright (c) 2019 Tatu Saloranta
+
+ (ASLv2) Jackson Core
+The following NOTICE information applies:
+ Jackson Core
+ Copyright (c) Tatu Saloranta
+
+ (ASLv2) Jackson Databind
+The following NOTICE information applies:
+ Jackson Databind
+ Copyright (c) 2007- Tatu Saloranta, tatu.salora...@iki.fi
+
+ Jackson is a high-performance, Free/Open Source JSON processing library.
+ It was originally written by Tatu Saloranta (tatu.salora...@iki.fi), and
has
+ been in development since 2007.
+
+ It is currently developed by a community of developers, as well as
supported
+ commercially by FasterXML.com.
+
+ (ASLv2) Caffeine (com.github.ben-manes.caffeine:caffeine:jar:3.1.8 -
https://github.com/ben-manes/caffeine)
+The following NOTICE information applies:
+ Caffeine (caching library)
+ Copyright Ben Manes
+
+ (ASLv2) Error Prone Annotations
+The following NOTICE information applies:
+ Error Prone Annotations
+ Copyright 2015 The Error Prone Authors
+
+ (ASLv2) Apache Commons IO
+The following NOTICE information applies:
+ Apache Commons IO
+ Copyright 2002-2016 The Apache Software Foundation
+
+ (ASLv2) Apache Avro
+The following NOTICE information applies:
+ Apache Avro
+ Copyright 2009-2017 The Apache Software Foundation
+
+ (ASLv2) Apache Commons Compress
+The following NOTICE information applies:
+ Apache Commons Compress
+ Copyright 2002-2014 The Apache Software Foundation
+
+ The files in the package org.apache.commons.compress.archivers.sevenz
+ were derived from the LZMA SDK, version 9.20 (C/ and CPP/7zip/),
+ which has been placed in the public domain:
+
+ "LZMA SDK is placed in the public domain."
(https://www.7-zip.org/sdk.html)
+
+ (ASLv2) Apache Commons CSV
+The following NOTICE information applies:
+ Apache Commons CSV
+ Copyright 2005-2016 The Apache Software Foundation
+
+ (ASLv2) Apache Commons Lang
+The following NOTICE information applies:
+ Apache Commons Lang
+ Copyright 2001-2017 The Apache Software Foundation
+
+ This product includes software from the Spring Framework,
+ under the Apache License 2.0 (see: StringUtils.containsWhitespace())
+
+ (ASLv2) Apache Commons Text
+The following NOTICE information applies:
+ Apache Commons Text
+ Copyright 2001-2018 The Apache Software Foundation
+
+
+Eclipse Distribution License 1.0
Review Comment:
Missing from `LICENSE` file.
##
nifi-nar-bundles/nifi-apicurio-bundle/nifi-apicurio-schema-registry-service/src/test/java/util/SchemaUtilsTest.java:
##
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package util;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.apicurio.schemaregistry.util.SchemaUtils;
+import
org.apache.nifi.apicurio.schemaregistry.util.SchemaUtils.ResultAttributes;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.io.InputStream;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+class SchemaUtilsTest {
+
+@Test
+void testCreateRecordSchema() throws SchemaNotFoundException, IOException {
+final In
[PR] NIFI-10222: Add schema registry service for Apicurio [nifi]
Lehel44 opened a new pull request, #7913: URL: https://github.com/apache/nifi/pull/7913 # Summary [NIFI-10222](https://issues.apache.org/jira/browse/NIFI-10222) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [ ] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
