[
https://issues.apache.org/jira/browse/PIG-3453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13813659#comment-13813659
]
Mridul Jain commented on PIG-3453:
----------------------------------
Dmitriy,
Interesting point regarding Trident and SummingBird. I haven't looked into the
complete details of SummingBird; but just out of curiosity wanted to know the
relaxation in data delivery guarantees that you have done. Any pointers? The
reason I am asking is that, we faced an odd problem (or atleast I thought it
was odd) in Storm transactional topologies and was wondering if the same
existed in Trident as well:
https://groups.google.com/forum/#!searchin/storm-user/mridul/storm-user/yX3giAO3ZWs/nfx8ofzSlhcJ.
As we were running on transactional topologies, I created an optional patch,
where you could use batches but still have no strict commit orders. Haven't
submitted the patch because it was for code which is supposed to be deprecated.
That patch alone worked well for us because we didn't pay for the overhead on
account of earlier batch failures.Trident does mention IBatchSpout, which says
batches without transactions, though I have to explore it a bit. I am
interested in knowing other optimizations in this direction..even if on Vanilla
Storm.
Another thing I wanted to know: We generally do batch(microbatches in Storm)
processing in Storm than tuple by tuple processing, as it provides high
throughput(especially when interacting with external systems like db updates
etc) as well as fits in logically well when we work at batch level. Batch
semantics is critical for us .....does SummingBird support batches. May be I
need to go over the details of SummingBird....
> Implement a Storm backend to Pig
> --------------------------------
>
> Key: PIG-3453
> URL: https://issues.apache.org/jira/browse/PIG-3453
> Project: Pig
> Issue Type: New Feature
> Affects Versions: 0.13.0
> Reporter: Pradeep Gollakota
> Assignee: Jacob Perkins
> Labels: storm
> Fix For: 0.13.0
>
> Attachments: storm-integration.patch
>
>
> There is a lot of interest around implementing a Storm backend to Pig for
> streaming processing. The proposal and initial discussions can be found at
> https://cwiki.apache.org/confluence/display/PIG/Pig+on+Storm+Proposal
--
This message was sent by Atlassian JIRA
(v6.1#6144)
