[
https://issues.apache.org/jira/browse/IMPALA-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated IMPALA-8780:
---------------------------------
Description:
Implement {{BufferedPlanRootSink}} so that {{FlushFinal}} blocks until all rows
are fetched. The implementation should use the {{RowBatchQueue}} introduced by
IMPALA-8779. By blocking in {{FlushFinal}} all non-coordinator fragments will
be closed if all results fit in the {{RowBatchQueue}}.
{{BufferedPlanRootSink::Send}} should enqueue each given {{RowBatch}} onto the
queue and then return. If the queue is full, it should block until there is
more space left in the queue. {{BufferedPlanRootSink::GetNext}} reads from the
queue and then fills in the given {{QueryResultSet}} by using the {{DataSink}}
{{ScalarExprEvaluator}}-s. Since the producer thread can call
{{BufferedPlanRootSink::Close}} while the consumer is calling
{{BufferedPlanRootSink::GetNext}} the two methods need to be synchronized so
that the {{DataSink}} {{MemTracker}}-s are not closed while {{GetNext}} is
running.
The implementation of {{BufferedPlanRootSink}} should remain the same
regardless of whether a {{std::queue}} backed {{RowBatchQueue}} or a
{{BufferedTupleStream}} backed {{RowBatchQueue}} is used.
{{BufferedPlanRootSink}} and {{BlockingPlanRootSink}} are similar in the sense
that {{BlockingPlanRootSink}} buffers one {{RowBatch}}, so for queries that
return under 1024 rows, all non-coordinator fragments are closed immediately as
well. The advantage of {{BufferedPlanRootSink}} is that allows buffering for 1+
{{RowBatch}}-es.
was:Implement {{BufferedPlanRootSink}} so that {{FlushFinal}} blocks until
all rows are fetched. The implementation should use the {{RowBatchQueue}}
introduced by IMPALA-8779. By blocking in {{FlushFinal}} all non-coordinator
fragments will be closed if all results fit in the {{RowBatchQueue}}.
> Implementation of BufferedPlanRootSink where FlushFinal blocks until all rows
> are fetched
> -----------------------------------------------------------------------------------------
>
> Key: IMPALA-8780
> URL: https://issues.apache.org/jira/browse/IMPALA-8780
> Project: IMPALA
> Issue Type: Sub-task
> Components: Backend
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Priority: Major
>
> Implement {{BufferedPlanRootSink}} so that {{FlushFinal}} blocks until all
> rows are fetched. The implementation should use the {{RowBatchQueue}}
> introduced by IMPALA-8779. By blocking in {{FlushFinal}} all non-coordinator
> fragments will be closed if all results fit in the {{RowBatchQueue}}.
> {{BufferedPlanRootSink::Send}} should enqueue each given {{RowBatch}} onto
> the queue and then return. If the queue is full, it should block until there
> is more space left in the queue. {{BufferedPlanRootSink::GetNext}} reads from
> the queue and then fills in the given {{QueryResultSet}} by using the
> {{DataSink}} {{ScalarExprEvaluator}}-s. Since the producer thread can call
> {{BufferedPlanRootSink::Close}} while the consumer is calling
> {{BufferedPlanRootSink::GetNext}} the two methods need to be synchronized so
> that the {{DataSink}} {{MemTracker}}-s are not closed while {{GetNext}} is
> running.
> The implementation of {{BufferedPlanRootSink}} should remain the same
> regardless of whether a {{std::queue}} backed {{RowBatchQueue}} or a
> {{BufferedTupleStream}} backed {{RowBatchQueue}} is used.
> {{BufferedPlanRootSink}} and {{BlockingPlanRootSink}} are similar in the
> sense that {{BlockingPlanRootSink}} buffers one {{RowBatch}}, so for queries
> that return under 1024 rows, all non-coordinator fragments are closed
> immediately as well. The advantage of {{BufferedPlanRootSink}} is that allows
> buffering for 1+ {{RowBatch}}-es.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
