[jira] [Created] (KAFKA-1532) Move CRC32 to AFTER the payload
Julian Morrison created KAFKA-1532: -- Summary: Move CRC32 to AFTER the payload Key: KAFKA-1532 URL: https://issues.apache.org/jira/browse/KAFKA-1532 Project: Kafka Issue Type: Improvement Components: core, producer Reporter: Julian Morrison Assignee: Jun Rao Priority: Minor To support streaming a message of known length but unknown content, take the CRC32 out of the message header and make it a message trailer. Then client libraries can calculate it after streaming the message to Kafka, without materializing the whole message in RAM. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1532) Move CRC32 to AFTER the payload
[ https://issues.apache.org/jira/browse/KAFKA-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14056792#comment-14056792 ] Julian Morrison commented on KAFKA-1532: Is "Kafka itself fully materializes messages in memory" unavoidable? (I am assuming writing it to file buffers is obvious.) > Move CRC32 to AFTER the payload > --- > > Key: KAFKA-1532 > URL: https://issues.apache.org/jira/browse/KAFKA-1532 > Project: Kafka > Issue Type: Improvement > Components: core, producer >Reporter: Julian Morrison >Assignee: Jun Rao >Priority: Minor > > To support streaming a message of known length but unknown content, take the > CRC32 out of the message header and make it a message trailer. Then client > libraries can calculate it after streaming the message to Kafka, without > materializing the whole message in RAM. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1532) Move CRC32 to AFTER the payload
[ https://issues.apache.org/jira/browse/KAFKA-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14056816#comment-14056816 ] Julian Morrison commented on KAFKA-1532: Streaming it to a scratch file (never sync'd, unlinked after use) avoids blocking the log - the blocking operation becomes "it's verified in the scratch file, copy it to the log." > Move CRC32 to AFTER the payload > --- > > Key: KAFKA-1532 > URL: https://issues.apache.org/jira/browse/KAFKA-1532 > Project: Kafka > Issue Type: Improvement > Components: core, producer >Reporter: Julian Morrison >Assignee: Jun Rao >Priority: Minor > > To support streaming a message of known length but unknown content, take the > CRC32 out of the message header and make it a message trailer. Then client > libraries can calculate it after streaming the message to Kafka, without > materializing the whole message in RAM. -- This message was sent by Atlassian JIRA (v6.2#6252)
