Send kea-dev mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.isc.org/mailman/listinfo/kea-dev
or, via email, send a message with subject or body 'help' to
[email protected]
You can reach the person managing the list at
[email protected]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of kea-dev digest..."
Today's Topics:
1. Re: update on this ....FRe: thoughts on 3601 (Stephen Morris)
2. Re: update on this ....FRe: thoughts on 3601 (Thomas Markwalder)
3. Re: update on this ....FRe: thoughts on 3601 (Tomek Mrugalski)
4. Re: update on this ....FRe: thoughts on 3601 (Thomas Markwalder)
----------------------------------------------------------------------
Message: 1
Date: Fri, 30 Oct 2015 12:42:26 +0000
From: Stephen Morris <[email protected]>
To: [email protected]
Subject: Re: [kea-dev] update on this ....FRe: thoughts on 3601
Message-ID: <[email protected]>
Content-Type: text/plain; charset=windows-1252
On 30/10/15 10:57, Thomas Markwalder wrote:
> Currently, it allows one to receive current schema data from
> a file containing prior schema records. For example
> if the current schema is 3.0 and Kea is given a 1.0 lease file:
>
>
> 1.0 file --> VersionedCSVFile --> 3.0 data ---> MemfileLeaseMgr
>
> This will occur automatically whenever MemfileLeaseMgr loads the
> lease file(s). Currently, this happens inside the MemfileLeaseMgr
> constructor.
>
> At this point, however, the physical lease file(s) are still 1.0 form.
> The header row has the list of 1.0 columns, the data rows have 1.0 data.
> Any new records written to the file will be in 3.0 form. Operationally,
> this is fine although not optimal. VersionedCSVFile can load files that
> have mixed generation data. In fact, such a file could contain 1.0, 2.0,
> and 3.0 records and it will load correctly. Whenever the file is loaded,
> older schema records are upgraded, current schema records pass thru. Kea
> always gets all records in current schema form.
>
> The twist is how do we physically convert the files?
>
> The most obvious answer is to let LFC take care of it. LFC will convert
> them automatically. Old schema records go in, current scheme lease files
> are written out. Of course this only happens if LFC is enabled or they
> run it manually.
Is this really a problem?
AIUI, the idea is that when Kea reads a line from the lease file, the
number of columns determines what memfile schema was used to write that
line. If the number of columns is fewer that that required by the
current version of Kea, defaults are applied to obtain the missing
values. If the number of columns is greater, the additional information
is ignored. (Presumably this could occur if someone upgraded Kea, ran it
for a bit, then reverted back to the previous version.)
If LFC is enabled, then the files will get converted to the new format
when it first runs. But Kea does not "need" the file to be converted
and will work perfectly well if LFC is disabled.
> 1. We log an error message, stating they should enable LFC or run it
> manually.
>
> We make sure document by example how to do this for the purposes of
> upgrading.
> (We should probably do this anyway)
>
> 2. We trigger a one-shot run of LFC when Memfile lease manager starts up
> IF we
> detect an upgrade is needed AND LFC is disabled.
I'd suggest just logging a warning message if we detect a schema upgrade
and LFC is disabled.
Of course, this assumes that any version of Kea will always be able to
read files created with any previous version of Kea.
Stephen
------------------------------
Message: 2
Date: Fri, 30 Oct 2015 08:58:17 -0400
From: Thomas Markwalder <[email protected]>
To: [email protected]
Subject: Re: [kea-dev] update on this ....FRe: thoughts on 3601
Message-ID: <[email protected]>
Content-Type: text/plain; charset=windows-1252
On 10/30/15 8:42 AM, Stephen Morris wrote:
> On 30/10/15 10:57, Thomas Markwalder wrote:
>
>> Currently, it allows one to receive current schema data from
>> a file containing prior schema records. For example
>> if the current schema is 3.0 and Kea is given a 1.0 lease file:
>>
>>
>> 1.0 file --> VersionedCSVFile --> 3.0 data ---> MemfileLeaseMgr
>>
>> This will occur automatically whenever MemfileLeaseMgr loads the
>> lease file(s). Currently, this happens inside the MemfileLeaseMgr
>> constructor.
>>
>> At this point, however, the physical lease file(s) are still 1.0 form.
>> The header row has the list of 1.0 columns, the data rows have 1.0 data.
>> Any new records written to the file will be in 3.0 form. Operationally,
>> this is fine although not optimal. VersionedCSVFile can load files that
>> have mixed generation data. In fact, such a file could contain 1.0, 2.0,
>> and 3.0 records and it will load correctly. Whenever the file is loaded,
>> older schema records are upgraded, current schema records pass thru. Kea
>> always gets all records in current schema form.
>>
>> The twist is how do we physically convert the files?
>>
>> The most obvious answer is to let LFC take care of it. LFC will convert
>> them automatically. Old schema records go in, current scheme lease files
>> are written out. Of course this only happens if LFC is enabled or they
>> run it manually.
> Is this really a problem?
>
> AIUI, the idea is that when Kea reads a line from the lease file, the
> number of columns determines what memfile schema was used to write that
> line. If the number of columns is fewer that that required by the
> current version of Kea, defaults are applied to obtain the missing
> values. If the number of columns is greater, the additional information
> is ignored. (Presumably this could occur if someone upgraded Kea, ran it
> for a bit, then reverted back to the previous version.)
You're correct, operationally it isn't a problem Kea can work with
mixed-generation
files, across reloads/restarts indefinitely. The files would look a
bit unusual upon
visual inspection or importation into a spreadsheet, but only to the
uninformed eye.
Minor point. currently VersionedCSVFile considers an input file with
more columns
than are defined as invalid. Primarily because I didn't think a newer
schema might
remove columns. One could address this by expanding the meta-data for each
column to include an upgrade flag such as "delete" or "ignore". I'll
look into it.
> If LFC is enabled, then the files will get converted to the new format
> when it first runs. But Kea does not "need" the file to be converted
> and will work perfectly well if LFC is disabled.
100% correct.
>> 1. We log an error message, stating they should enable LFC or run it
>> manually.
>>
>> We make sure document by example how to do this for the purposes of
>> upgrading.
>> (We should probably do this anyway)
>>
>> 2. We trigger a one-shot run of LFC when Memfile lease manager starts up
>> IF we
>> detect an upgrade is needed AND LFC is disabled.
> I'd suggest just logging a warning message if we detect a schema upgrade
> and LFC is disabled.
I tend to agree. On an tangential note a control channel command to
run LFC is probably a good thing to have.
> Of course, this assumes that any version of Kea will always be able to
> read files created with any previous version of Kea.
That is intent at least. I suppose if we reach a point where the
schema or format of the file change radically we might need to employ
a different strategy.
> Stephen
> _______________________________________________
> kea-dev mailing list
> [email protected]
> https://lists.isc.org/mailman/listinfo/kea-dev
------------------------------
Message: 3
Date: Fri, 30 Oct 2015 22:13:02 +0900
From: Tomek Mrugalski <[email protected]>
To: [email protected]
Subject: Re: [kea-dev] update on this ....FRe: thoughts on 3601
Message-ID: <[email protected]>
Content-Type: text/plain; charset=windows-1252
On 30/10/15 19:57, Thomas Markwalder wrote:
> So there is a minor twist to the VersionedCSVFile approach.
>
> Currently, it allows one to receive current schema data from
> a file containing prior schema records. For example
> if the current schema is 3.0 and Kea is given a 1.0 lease file:
>
> 1.0 file --> VersionedCSVFile --> 3.0 data ---> MemfileLeaseMgr
>
> This will occur automatically whenever MemfileLeaseMgr loads the
> lease file(s). Currently, this happens inside the MemfileLeaseMgr
> constructor.
>
> At this point, however, the physical lease file(s) are still 1.0 form.
> The header row has the list of 1.0 columns, the data rows have 1.0 data.
> Any new records written to the file will be in 3.0 form. Operationally,
> this is fine although not optimal. VersionedCSVFile can load files that
> have mixed generation data. In fact, such a file could contain 1.0, 2.0,
> and 3.0 records and it will load correctly. Whenever the file is loaded,
> older schema records are upgraded, current schema records pass thru. Kea
> always gets all records in current schema form.
>
> The twist is how do we physically convert the files?
Do I understand it correctly that once you load 1.0 file into Kea
version that expects 3.0 data, it will append 3.0 data to a file that
has 1.0 header? If that is so, I think it's a very bad idea. It would
make the lease file inconsistent and difficult to process by any
software that happens to not be Kea. That was one of the reasons why we
decided to use CSV in the first place: to be able to process the file by
other software.
I think that once the version inconsistency is detected, the software
should rewrite the whole CSV file to a new format. This will introduce
an extended startup time after Kea upgrade, but that's perfectly ok.
Software upgrades are infrequent (and planned) events.
> The most obvious answer is to let LFC take care of it. LFC will convert
> them automatically. Old schema records go in, current scheme lease files
> are written out. Of course this only happens if LFC is enabled or they
> run it manually.
>
> Keeping in mind the VerionedCSVFile can expose the knowledge that the data
> required updating, what should we do if we detect at start up that
> LFC is disabled AND the files need upgrading?
>
> 1. We log an error message, stating they should enable LFC or run it
> manually.
>
> We make sure document by example how to do this for the purposes of
> upgrading.
> (We should probably do this anyway)
>
> 2. We trigger a one-shot run of LFC when Memfile lease manager starts up
> IF we detect an upgrade is needed AND LFC is disabled.
I would go even one step further and refuse to start the server until
the LFC upgrade run is completed. It's much better to have couple
seconds downtime that is planned and avoidable if needed. Does 10
seconds pause bothers you? Then don't upgrade at this time. The
alternative, to have an inconsistent database, is much worse.
On a related note, I think having Kea the ability to upgrade on load is
fine, but we still should provide kea-admin upgrade capability, even if
it only calls kea-lfc. This is purely from operational perspective.
Without it, one could not upgrade the database without running the DHCP
service. Update and providing service are two different operations and
there may be valid scenarios, where you want to just do the upgrade, but
not run the service.
Tomek
------------------------------
Message: 4
Date: Fri, 30 Oct 2015 09:38:00 -0400
From: Thomas Markwalder <[email protected]>
To: [email protected]
Subject: Re: [kea-dev] update on this ....FRe: thoughts on 3601
Message-ID: <[email protected]>
Content-Type: text/plain; charset=windows-1252
On 10/30/15 9:13 AM, Tomek Mrugalski wrote:
> On 30/10/15 19:57, Thomas Markwalder wrote:
>> So there is a minor twist to the VersionedCSVFile approach.
>>
>> Currently, it allows one to receive current schema data from
>> a file containing prior schema records. For example
>> if the current schema is 3.0 and Kea is given a 1.0 lease file:
>>
>> 1.0 file --> VersionedCSVFile --> 3.0 data ---> MemfileLeaseMgr
>>
>> This will occur automatically whenever MemfileLeaseMgr loads the
>> lease file(s). Currently, this happens inside the MemfileLeaseMgr
>> constructor.
>>
>> At this point, however, the physical lease file(s) are still 1.0 form.
>> The header row has the list of 1.0 columns, the data rows have 1.0 data.
>> Any new records written to the file will be in 3.0 form. Operationally,
>> this is fine although not optimal. VersionedCSVFile can load files that
>> have mixed generation data. In fact, such a file could contain 1.0, 2.0,
>> and 3.0 records and it will load correctly. Whenever the file is loaded,
>> older schema records are upgraded, current schema records pass thru. Kea
>> always gets all records in current schema form.
>>
>> The twist is how do we physically convert the files?
> Do I understand it correctly that once you load 1.0 file into Kea
> version that expects 3.0 data, it will append 3.0 data to a file that
> has 1.0 header?
Yes, that is what would happen if the server began appending to
the 1.0 file.
> If that is so, I think it's a very bad idea. It would
> make the lease file inconsistent and difficult to process by any
> software that happens to not be Kea. That was one of the reasons why we
> decided to use CSV in the first place: to be able to process the file by
> other software.
I didn't say it was necessarily a good thing, I said it would be operate.
> I think that once the version inconsistency is detected, the software
> should rewrite the whole CSV file to a new format. This will introduce
> an extended startup time after Kea upgrade, but that's perfectly ok.
> Software upgrades are infrequent (and planned) events.
>
>> The most obvious answer is to let LFC take care of it. LFC will convert
>> them automatically. Old schema records go in, current scheme lease files
>> are written out. Of course this only happens if LFC is enabled or they
>> run it manually.
>>
>> Keeping in mind the VerionedCSVFile can expose the knowledge that the data
>> required updating, what should we do if we detect at start up that
>> LFC is disabled AND the files need upgrading?
>>
>> 1. We log an error message, stating they should enable LFC or run it
>> manually.
>>
>> We make sure document by example how to do this for the purposes of
>> upgrading.
>> (We should probably do this anyway)
>>
>> 2. We trigger a one-shot run of LFC when Memfile lease manager starts up
>> IF we detect an upgrade is needed AND LFC is disabled.
> I would go even one step further and refuse to start the server until
> the LFC upgrade run is completed. It's much better to have couple
> seconds downtime that is planned and avoidable if needed. Does 10
> seconds pause bothers you? Then don't upgrade at this time. The
> alternative, to have an inconsistent database, is much worse.
Making the server not process requests until LFC is complete on a one shot
basis may be harder than one might think. Though I suppose there might be
other states when we would want servers to drop requests until some
condition
clears. Would need to investigate this.
Also keep in mind how LFC works. Before the actual process is invoked
the server copies the current lease file to the copy file, so the 1.0
file goes to 1.0.copy.
Then it opens a new file, which will be 3.0 and hence all records
written to it would be
3.0. In other words, just by doing the setup prior to LFC running, Kea
starts working
with a current schema lease file.
I think it might be close enough to have it start LFC with the smallest
possible
interval. It might even be possible to kick it without the timer, I'd
have to look at that
more closely.
> On a related note, I think having Kea the ability to upgrade on load is
> fine, but we still should provide kea-admin upgrade capability, even if
> it only calls kea-lfc. This is purely from operational perspective.
> Without it, one could not upgrade the database without running the DHCP
> service. Update and providing service are two different operations and
> there may be valid scenarios, where you want to just do the upgrade, but
> not run the service.
We could make kea-admin call LFC for this. In reality, we ought to be
able to do that
through kea-admin anyway. I don't have an issue with that. It still keeps
all of the conversion know-how in one place. I would be opposed to
implementing a
secondary script type of conversion for this, unless of we discard the
idea of automatic
conversion.
> Tomek
>
> _______________________________________________
> kea-dev mailing list
> [email protected]
> https://lists.isc.org/mailman/listinfo/kea-dev
------------------------------
_______________________________________________
kea-dev mailing list
[email protected]
https://lists.isc.org/mailman/listinfo/kea-dev
End of kea-dev Digest, Vol 19, Issue 17
***************************************
