[
https://issues.apache.org/jira/browse/PIO-45?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15851751#comment-15851751
]
Pat Ferrel commented on PIO-45:
-------------------------------
$sets should never be dropped regardless of time-window, neither should $unset,
or $delete since they all affect aggregated properties. They are immune to the
time-window and only affected by compression. They are, in effect, permanent.
The values of property aggregates over time change due to aggregating different
values and the test should alway be, do the properties have the same value when
compressed as they have if queried as uncompressed aggregated values. So
looking at them from a fixed time they would look exactly the same compressed
or not. Strictly speaking the property compression setting could be taken out
of the timeWindow. Deduping does not apply to these events either since this
may affect aggregate state. So the only thing that affects them is compression.
This is not true of usage events which decay out of the DB and are not queried
for in aggregate, only as collections.
So yes the issues seem to be:
1) arrays seem to be compressed wrong
2) $sets and the other property/state changing events seem to be dropped by
time-wiindow.
There is a release in a weeks, so if you can get the fix in, that would be
great. Thanks for looking at this [~emergentorder]
> SelfCleaningDatasource erases all data
> --------------------------------------
>
> Key: PIO-45
> URL: https://issues.apache.org/jira/browse/PIO-45
> Project: PredictionIO
> Issue Type: Bug
> Affects Versions: 0.10.0-incubating
> Reporter: Pat Ferrel
> Assignee: Alexander Merritt
> Priority: Blocker
> Fix For: 0.11.0
>
> Attachments: import_handmade_simple.py,
> sample-time-window-and-downsample-data.txt
>
>
> as integrated into the UR, in the integration-test, the SelfCleaningDataset
> erases all data. This feature works fine in the AML version of PIO.
> Although not tested one could assume that this would be true with any other
> Datasource in other templates.
> [~emergentorder] can you check to see if the PIO merge was done correctly.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
