[jira] [Updated] (YARN-4025) Deal with byte representations of Longs in writer code
[ https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4025: --- Fix Version/s: 2.9.0 > Deal with byte representations of Longs in writer code > -- > > Key: YARN-4025 > URL: https://issues.apache.org/jira/browse/YARN-4025 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Sangjin Lee > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-4025-YARN-2928.001.patch, > YARN-4025-YARN-2928.002.patch, YARN-4025-YARN-2928.003.patch, > YARN-4025-YARN-2928.004.patch > > > Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl > code. There seem to be some places in the code where there are conversions > between Long to byte[] to String for easier argument passing between function > calls. Then these values end up being converted back to byte[] while storing > in hbase. > It would be better to pass around byte[] or the Longs themselves as > applicable. > This may result in some api changes (store function) as well in adding a few > more function calls like getColumnQualifier which accepts a pre-encoded byte > array. It will be in addition to the existing api which accepts a String and > the ColumnHelper to return a byte[] column name instead of a String one. > Filing jira to track these changes. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4025) Deal with byte representations of Longs in writer code
[
https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated YARN-4025:
--
Attachment: YARN-4025-YARN-2928.004.patch
v.4 patch posted.
- implemented proper handling of a null column prefix
- added more javadoc to clarify several places
- renamed the test from {{TestHBaseTimelineWriterImpl}} to
{{TestHBaseTimelineStorage}}
- clarified and made explicit the no-limit split
- fixed javadoc comments for the value separator
- added some logging statements
This should address most of the review comments. I stopped short of renaming
the {{readResults()}} method. We can treat that method as the default
{{readResults()}} method, and the other one as the one for having raw
(non-string) components. I added more javadoc to clarify that point. Let me
know.
Deal with byte representations of Longs in writer code
--
Key: YARN-4025
URL: https://issues.apache.org/jira/browse/YARN-4025
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Vrushali C
Assignee: Sangjin Lee
Attachments: YARN-4025-YARN-2928.001.patch,
YARN-4025-YARN-2928.002.patch, YARN-4025-YARN-2928.003.patch,
YARN-4025-YARN-2928.004.patch
Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl
code. There seem to be some places in the code where there are conversions
between Long to byte[] to String for easier argument passing between function
calls. Then these values end up being converted back to byte[] while storing
in hbase.
It would be better to pass around byte[] or the Longs themselves as
applicable.
This may result in some api changes (store function) as well in adding a few
more function calls like getColumnQualifier which accepts a pre-encoded byte
array. It will be in addition to the existing api which accepts a String and
the ColumnHelper to return a byte[] column name instead of a String one.
Filing jira to track these changes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4025) Deal with byte representations of Longs in writer code
[
https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated YARN-4025:
--
Attachment: YARN-4025-YARN-2928.003.patch
v.3 patch posted.
- added more javadoc to explain the format of the event column name
- changed the signature of {{readResultsHavingCompoundQualifers()}} to make the
key type ({{byte[][]}}) explicit
- updated the event/application table to reflect the new separator value
Deal with byte representations of Longs in writer code
--
Key: YARN-4025
URL: https://issues.apache.org/jira/browse/YARN-4025
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Vrushali C
Assignee: Sangjin Lee
Attachments: YARN-4025-YARN-2928.001.patch,
YARN-4025-YARN-2928.002.patch, YARN-4025-YARN-2928.003.patch
Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl
code. There seem to be some places in the code where there are conversions
between Long to byte[] to String for easier argument passing between function
calls. Then these values end up being converted back to byte[] while storing
in hbase.
It would be better to pass around byte[] or the Longs themselves as
applicable.
This may result in some api changes (store function) as well in adding a few
more function calls like getColumnQualifier which accepts a pre-encoded byte
array. It will be in addition to the existing api which accepts a String and
the ColumnHelper to return a byte[] column name instead of a String one.
Filing jira to track these changes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4025) Deal with byte representations of Longs in writer code
[ https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-4025: - Assignee: Sangjin Lee (was: Vrushali C) Deal with byte representations of Longs in writer code -- Key: YARN-4025 URL: https://issues.apache.org/jira/browse/YARN-4025 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Vrushali C Assignee: Sangjin Lee Attachments: YARN-4025-YARN-2928.001.patch Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl code. There seem to be some places in the code where there are conversions between Long to byte[] to String for easier argument passing between function calls. Then these values end up being converted back to byte[] while storing in hbase. It would be better to pass around byte[] or the Longs themselves as applicable. This may result in some api changes (store function) as well in adding a few more function calls like getColumnQualifier which accepts a pre-encoded byte array. It will be in addition to the existing api which accepts a String and the ColumnHelper to return a byte[] column name instead of a String one. Filing jira to track these changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4025) Deal with byte representations of Longs in writer code
[
https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated YARN-4025:
--
Attachment: YARN-4025-YARN-2928.002.patch
v.2 patch posted.
- reconciled with YARN-3906 (application table)
- updated how {{HBaseTimelineReaderImpl}} reads events
- made {{TestHBaseTimelineWriterImpl}} verify the results from reading timeline
entities using {{HBaseTimelineReaderImpl}} in addition to direct verification
using the HBase schema
- simplified some asserts in the test
- fixed a bug in {{HBaseTimelineReaderImpl}} on reading config
One major change I did is that now {{TestHBaseTimelineWriterImpl}} verifies the
timeline entities read by {{HBaseTimelineReaderImpl}} as well. This provides a
nice benefit of verifying correctness of {{HBaseTimelineReaderImpl}}. It
uncovered a bug in the process. :)
Deal with byte representations of Longs in writer code
--
Key: YARN-4025
URL: https://issues.apache.org/jira/browse/YARN-4025
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Vrushali C
Assignee: Sangjin Lee
Attachments: YARN-4025-YARN-2928.001.patch,
YARN-4025-YARN-2928.002.patch
Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl
code. There seem to be some places in the code where there are conversions
between Long to byte[] to String for easier argument passing between function
calls. Then these values end up being converted back to byte[] while storing
in hbase.
It would be better to pass around byte[] or the Longs themselves as
applicable.
This may result in some api changes (store function) as well in adding a few
more function calls like getColumnQualifier which accepts a pre-encoded byte
array. It will be in addition to the existing api which accepts a String and
the ColumnHelper to return a byte[] column name instead of a String one.
Filing jira to track these changes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4025) Deal with byte representations of Longs in writer code
[ https://issues.apache.org/jira/browse/YARN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-4025: - Attachment: YARN-4025-YARN-2928.001.patch Attaching patch v001. I have updated the HBaseTimelineWriter to avoid unnecessary conversions from byte[] to Strings. To allow that, I have added APIs that accept byte[] in ColumnHelper#getColumnQualifier and EntityColumnPrefix.#store . I also added a EntityColumnPrefix#readResultsHavingCompoundColumnQualifiers and ColumnHelper#readResultsHavingCompoundColumnQualifiers to help read results that have compound columns. The unit test has an example of how to read the Timeline Event's compound column qualifier. I have also updated the unit test to remove an internal call to another unit test, I moved that second unit to be a @ Test unit test itself. Last but not the least, I have updated the Separators for values/compound qualifiers to be = instead of ? to help easier implementation of reader code since splitting on ? is not easy due to ? being a wildcard. Would appreciate your review. Deal with byte representations of Longs in writer code -- Key: YARN-4025 URL: https://issues.apache.org/jira/browse/YARN-4025 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Vrushali C Assignee: Vrushali C Attachments: YARN-4025-YARN-2928.001.patch Timestamps are being stored as Longs in hbase by the HBaseTimelineWriterImpl code. There seem to be some places in the code where there are conversions between Long to byte[] to String for easier argument passing between function calls. Then these values end up being converted back to byte[] while storing in hbase. It would be better to pass around byte[] or the Longs themselves as applicable. This may result in some api changes (store function) as well in adding a few more function calls like getColumnQualifier which accepts a pre-encoded byte array. It will be in addition to the existing api which accepts a String and the ColumnHelper to return a byte[] column name instead of a String one. Filing jira to track these changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
