[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805305#action_12805305 ] Doug Cutting commented on AVRO-160: --- I am okay adding length to blocks, but it should be do

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805297#action_12805297 ] Philip Zeyliger commented on AVRO-160: -- > Isn't that more of a different encoder/decoder

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805287#action_12805287 ] Scott Carey commented on AVRO-160: -- bq. To be clear, I think you mean "size in bytes (after

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805252#action_12805252 ] Philip Zeyliger commented on AVRO-160: -- Hi Scott, I don't have strong opinions either w

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805150#action_12805150 ] Scott Carey commented on AVRO-160: -- Any thoughts on a possible change to the file format to

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12804653#action_12804653 ] Scott Carey commented on AVRO-160: -- bq. No, the spec currently says each block is prefixed b

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12795870#action_12795870 ] Doug Cutting commented on AVRO-160: --- > you have removed the count of the number of objects,

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12795752#action_12795752 ] Jeff Hammerbacher commented on AVRO-160: Hey Doug, To clarify, for the block format,

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12795624#action_12795624 ] Doug Cutting commented on AVRO-160: --- Andrew: sorry, I committed this before I saw your comm

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12795598#action_12795598 ] Jeff Hammerbacher commented on AVRO-160: bq. Some kind of flush method that forces wr

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12795529#action_12795529 ] Andrew Purtell commented on AVRO-160: - Some quick comments from over on HBASE-2055: - I

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12795476#action_12795476 ] Philip Zeyliger commented on AVRO-160: -- +1! > file format should be friendly to streami

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12795387#action_12795387 ] Philip Zeyliger commented on AVRO-160: -- Looked over the patch again. Looks good. Synch

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12795161#action_12795161 ] Doug Cutting commented on AVRO-160: --- Jeff> The most recent patch seems to write both the le

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793805#action_12793805 ] Jeff Hammerbacher commented on AVRO-160: Hey Doug, The most recent patch seems to wr

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793421#action_12793421 ] Philip Zeyliger commented on AVRO-160: -- Thanks for addressing my comments. Some minor n

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12792370#action_12792370 ] Philip Zeyliger commented on AVRO-160: -- Took a look at the patch. I hadn't read the old

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791790#action_12791790 ] Doug Cutting commented on AVRO-160: --- BTW, you need to 'svn cp src/java/org/apache/avro/file

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790896#action_12790896 ] Doug Cutting commented on AVRO-160: --- > there is a collision > the file was written corruptl

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790879#action_12790879 ] Scott Carey commented on AVRO-160: -- {quote}Who's silently skipping blocks?{quote} Code that

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790840#action_12790840 ] Doug Cutting commented on AVRO-160: --- Scott> I'm concerned about silent data loss during pro

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790540#action_12790540 ] Philip Zeyliger commented on AVRO-160: -- Doug> The count of items and bytes per block pro

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790516#action_12790516 ] Philip Zeyliger commented on AVRO-160: -- bq. This problem is really an HDFS problem. A fu

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790504#action_12790504 ] Scott Carey commented on AVRO-160: -- {quote}The other thing we're ditching is computed metada

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790429#action_12790429 ] Doug Cutting commented on AVRO-160: --- Philip> To be clear, you're preserving appendability a

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790405#action_12790405 ] Scott Carey commented on AVRO-160: -- I agree, a simple format for 80%+ of the use cases is a

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790395#action_12790395 ] Philip Zeyliger commented on AVRO-160: -- I like the simpler model. I'm pretty confident

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790324#action_12790324 ] Jeff Hammerbacher commented on AVRO-160: Not crazy. Eric Anderson, who works on the D

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790320#action_12790320 ] Doug Cutting commented on AVRO-160: --- I'm now having second thoughts about the current propo

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769519#action_12769519 ] Doug Cutting commented on AVRO-160: --- > At what size does the metadata block overhead repres

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769378#action_12769378 ] Scott Carey commented on AVRO-160: -- {quote} Yes, if we kept a global block index, we could a

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769302#action_12769302 ] Doug Cutting commented on AVRO-160: --- > It is useful to leave open the option for index type

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768975#action_12768975 ] Scott Carey commented on AVRO-160: -- {quote}For mapreduce, we need to be able to seek to an a

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768905#action_12768905 ] Doug Cutting commented on AVRO-160: --- > You could also store an offset pointer to the schema

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768871#action_12768871 ] Doug Cutting commented on AVRO-160: --- > Perhaps this type should not be optimized for random

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768839#action_12768839 ] Scott Carey commented on AVRO-160: -- Sounds to me like this file type is trying to be too man

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768806#action_12768806 ] Philip Zeyliger commented on AVRO-160: -- Ok, that makes sense. For some reason, I though

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768781#action_12768781 ] Doug Cutting commented on AVRO-160: --- > Is that not the case? No, that would not work. You

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768766#action_12768766 ] Philip Zeyliger commented on AVRO-160: -- bq. Note that the only change permitted to a sch

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768517#action_12768517 ] Doug Cutting commented on AVRO-160: --- > I think you'll also have to get rid of getCount(), n

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768117#action_12768117 ] Philip Zeyliger commented on AVRO-160: -- I think you'll also have to get rid of getCount(

[jira] Commented: (AVRO-160) file format should be friendly to streaming

[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767943#action_12767943 ] Matt Massie commented on AVRO-160: -- +1 > file format should be friendly to streaming >