Re: Required work to close out 1.10
FWIW I'm 100% in favor of waiting until we can get both fully working on Java 11. On Mon, Jul 29, 2019 at 3:44 PM Jeff wrote: > Another thought might be to get NiFi Registry builds on Java 11 working as > well for the NiFi Registry 0.5.0 release. With the NiFi Java 11 build > compatibility close to being merged, it should be much more simple to get > NiFi Registry building on Java 11. It would be good to release versions > for both projects to support Java 11 builds close together. > > We could release NiFi 1.10.0 with Java 11 build compatibility, and then > release NiFi Registry 0.6.0 with Java 11 build compatibility, but I'm not > sure what other features would be included in NiFi Registry 0.6.0 other > than the Java 11 build compatibility. I realize that might push back the > NiFi Registry 0.5.0 release timeframe back, probably weeks. > > On Mon, Jul 29, 2019 at 3:29 PM Andy LoPresto > wrote: > > > There are also a number of new features around encrypted repositories and > > encrypted configuration values that would be good to get in. Some have > > active PRs up right now, and others are close to being posted. > > > > Andy LoPresto > > alopre...@apache.org > > alopresto.apa...@gmail.com > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > > > On Jul 29, 2019, at 12:27 PM, Bryan Bende wrote: > > > > > > There are two really helpful bug fixes [1][2] related to version > > > controlling flows that I would like to see make it in to 1.10.0, but > > > they are dependent on releasing registry 0.5.0 first, since part of > > > the fixes are in the registry flow-diff code which NiFi depends on. > > > > > > So I was hoping to release registry 0.5.0 first, then nifi 1.10.0. > > > > > > If we took that approach I think there are still a few remaining items > > > before we can make a registry release. > > > > > > [1] https://issues.apache.org/jira/browse/NIFI-6025 > > > [2] https://issues.apache.org/jira/browse/NIFI-6314 > > > > > > On Mon, Jul 29, 2019 at 3:12 PM Mike Thomsen > > wrote: > > >> > > >> This thread is to just get a discussion started on what tickets need > to > > be > > >> closed out before we should do 1.10. > > >> > > >> For graph there are two essential bug fixes: > > >> > > >> https://github.com/apache/nifi/pull/3571 > > >> https://github.com/apache/nifi/pull/3572 > > >> > > >> I think Jeff's Java 11 build support should also be included if > > possible: > > >> > > >> https://github.com/apache/nifi/pull/3404/ > > >> > > >> If anyone needs some help on a review, feel free to ping me on GitHub. > > >> > > >> Mike > > > > >
Re: Required work to close out 1.10
Another thought might be to get NiFi Registry builds on Java 11 working as well for the NiFi Registry 0.5.0 release. With the NiFi Java 11 build compatibility close to being merged, it should be much more simple to get NiFi Registry building on Java 11. It would be good to release versions for both projects to support Java 11 builds close together. We could release NiFi 1.10.0 with Java 11 build compatibility, and then release NiFi Registry 0.6.0 with Java 11 build compatibility, but I'm not sure what other features would be included in NiFi Registry 0.6.0 other than the Java 11 build compatibility. I realize that might push back the NiFi Registry 0.5.0 release timeframe back, probably weeks. On Mon, Jul 29, 2019 at 3:29 PM Andy LoPresto wrote: > There are also a number of new features around encrypted repositories and > encrypted configuration values that would be good to get in. Some have > active PRs up right now, and others are close to being posted. > > Andy LoPresto > alopre...@apache.org > alopresto.apa...@gmail.com > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > On Jul 29, 2019, at 12:27 PM, Bryan Bende wrote: > > > > There are two really helpful bug fixes [1][2] related to version > > controlling flows that I would like to see make it in to 1.10.0, but > > they are dependent on releasing registry 0.5.0 first, since part of > > the fixes are in the registry flow-diff code which NiFi depends on. > > > > So I was hoping to release registry 0.5.0 first, then nifi 1.10.0. > > > > If we took that approach I think there are still a few remaining items > > before we can make a registry release. > > > > [1] https://issues.apache.org/jira/browse/NIFI-6025 > > [2] https://issues.apache.org/jira/browse/NIFI-6314 > > > > On Mon, Jul 29, 2019 at 3:12 PM Mike Thomsen > wrote: > >> > >> This thread is to just get a discussion started on what tickets need to > be > >> closed out before we should do 1.10. > >> > >> For graph there are two essential bug fixes: > >> > >> https://github.com/apache/nifi/pull/3571 > >> https://github.com/apache/nifi/pull/3572 > >> > >> I think Jeff's Java 11 build support should also be included if > possible: > >> > >> https://github.com/apache/nifi/pull/3404/ > >> > >> If anyone needs some help on a review, feel free to ping me on GitHub. > >> > >> Mike > >
Re: Record path API chooses CHOICE[STRING, RECORD] when the field is missing
Mike, What Record Reader is being used here? The problem appears to be due to the Record Reader itself assigning that as the field type. I created a dummy unit test to verify the RecordPath stuff is correct: @Test public void testFromEmail() { final List fields = new ArrayList<>(); fields.add(new RecordField("firstName", RecordFieldType.STRING.getDataType())); fields.add(new RecordField("lastName", RecordFieldType.STRING.getDataType())); fields.add(new RecordField("creationDateTime", RecordFieldType.TIMESTAMP.getDataType(), true)); final RecordSchema schema = new SimpleRecordSchema(fields); final Map values = new HashMap<>(); values.put("firstName", "John"); values.put("lastName", "Doe"); values.put("creationDateTime", new Timestamp(System.currentTimeMillis())); final Record record = new MapRecord(schema, values); final Optional optionalFieldValue = RecordPath.compile("/creationDateTime").evaluate(record).getSelectedFields().findFirst(); final FieldValue fieldValue = optionalFieldValue.get(); System.out.println(fieldValue.getField()); } Which prints out the correct field type: RecordField[name=creationDateTime, dataType=TIMESTAMP:-MM-dd HH:mm:ss, nullable=true] So I presume the Record Reader may not be properly applying the schema to the Record that it returns. Thanks -Mark On Jul 28, 2019, at 10:19 AM, Mike Thomsen mailto:mikerthom...@gmail.com>> wrote: I have a simple avro schema in a test case that looks like this: { "type": "record", "name": "PersonRecord", "fields": [ { "name": "firstName", "type": "string" }, { "name": "lastName", "type": "string" }, { "name": "creationDateTime", "type": [ "null", "type": "long", "logicalType": "timestamp-millis" }] ] } Then I try something like this... RecordPath path = recordPathCache.getCompiled("/creationDateTime"); RecordPathResult rp = path.evaluate(targetRecord); Optional nodeField = rp.getSelectedFields().findFirst(); if (!nodeField.isPresent()) { throw new ProcessException("..."); } FieldValue fieldValue = nodeField.get(); //fieldValue.getField() is a Choice of String, Record Is there a way to get the correct field type here? I assume that Choice[String, Record] default here was done to facilitate schema inference. Thanks, Mike
Re: Required work to close out 1.10
There are also a number of new features around encrypted repositories and encrypted configuration values that would be good to get in. Some have active PRs up right now, and others are close to being posted. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jul 29, 2019, at 12:27 PM, Bryan Bende wrote: > > There are two really helpful bug fixes [1][2] related to version > controlling flows that I would like to see make it in to 1.10.0, but > they are dependent on releasing registry 0.5.0 first, since part of > the fixes are in the registry flow-diff code which NiFi depends on. > > So I was hoping to release registry 0.5.0 first, then nifi 1.10.0. > > If we took that approach I think there are still a few remaining items > before we can make a registry release. > > [1] https://issues.apache.org/jira/browse/NIFI-6025 > [2] https://issues.apache.org/jira/browse/NIFI-6314 > > On Mon, Jul 29, 2019 at 3:12 PM Mike Thomsen wrote: >> >> This thread is to just get a discussion started on what tickets need to be >> closed out before we should do 1.10. >> >> For graph there are two essential bug fixes: >> >> https://github.com/apache/nifi/pull/3571 >> https://github.com/apache/nifi/pull/3572 >> >> I think Jeff's Java 11 build support should also be included if possible: >> >> https://github.com/apache/nifi/pull/3404/ >> >> If anyone needs some help on a review, feel free to ping me on GitHub. >> >> Mike
Re: Required work to close out 1.10
There are two really helpful bug fixes [1][2] related to version controlling flows that I would like to see make it in to 1.10.0, but they are dependent on releasing registry 0.5.0 first, since part of the fixes are in the registry flow-diff code which NiFi depends on. So I was hoping to release registry 0.5.0 first, then nifi 1.10.0. If we took that approach I think there are still a few remaining items before we can make a registry release. [1] https://issues.apache.org/jira/browse/NIFI-6025 [2] https://issues.apache.org/jira/browse/NIFI-6314 On Mon, Jul 29, 2019 at 3:12 PM Mike Thomsen wrote: > > This thread is to just get a discussion started on what tickets need to be > closed out before we should do 1.10. > > For graph there are two essential bug fixes: > > https://github.com/apache/nifi/pull/3571 > https://github.com/apache/nifi/pull/3572 > > I think Jeff's Java 11 build support should also be included if possible: > > https://github.com/apache/nifi/pull/3404/ > > If anyone needs some help on a review, feel free to ping me on GitHub. > > Mike
Required work to close out 1.10
This thread is to just get a discussion started on what tickets need to be closed out before we should do 1.10. For graph there are two essential bug fixes: https://github.com/apache/nifi/pull/3571 https://github.com/apache/nifi/pull/3572 I think Jeff's Java 11 build support should also be included if possible: https://github.com/apache/nifi/pull/3404/ If anyone needs some help on a review, feel free to ping me on GitHub. Mike