ueries on
> hive tables (not inserts ), capture the query output and process it?
>
> Regards
> Vivek
>
>
>
> --
> Sent from: http://apache-apex-users-list.78494.x6.nabble.com/
>
Hi
I would like to know if there is any recommended way to execute queries on
hive tables (not inserts ), capture the query output and process it?
Regards
Vivek
--
Sent from: http://apache-apex-users-list.78494.x6.nabble.com/
Thanks Ram. This is helpful.
On Fri, Dec 23, 2016 at 9:30 PM, Munagala Ramanath
wrote:
> There are multiple approaches, each with its own tradeoffs. Here is first
> step:
>
> A1. Create a pair of in-memory non-transient queues to hold the tuples
> (non-transient
>because we want them to
There are multiple approaches, each with its own tradeoffs. Here is first
step:
A1. Create a pair of in-memory non-transient queues to hold the tuples
(non-transient
because we want them to be checkpointed and restored on recovery
from failure).
A2. Create a separate thread that waits for a
Dear Team,
I am performing a batch operation. As per the use case, I cannot commit the
transaction from output operator to external system unless I process all
the batch data. Once all the data is processed, output operator will get a
bulking collection Object as single tuple. Output Operator is n
Thank you everyone for your support !!
On Tue, Dec 6, 2016 at 9:57 PM, Vlad Rozov wrote:
> I'd recommend to use additional output port solution outlined by Bhupesh.
> There are few Apex applications on the field that leverage that solution.
>
> Thank you,
>
> Vlad
>
>
> On 12/4/16 11:45, Visha
I'd recommend to use additional output port solution outlined by
Bhupesh. There are few Apex applications on the field that leverage that
solution.
Thank you,
Vlad
On 12/4/16 11:45, Vishal Agrawal wrote:
Thank you Bhupesh and Ram. Appreciate your quick response.
I see ThrottlingStatsListene
Once per second: If you change the log level to DEBUG you'll see messages
like this in the AppMaster log:
2016-12-05 07:49:24,010 DEBUG
com.datatorrent.examples.throttle.ThrottlingStatsListener: Operator 1 min
window 6360643434897736168 max window 6360643434897736196
2016-12-05 07:49:24,011 DEBUG
Thank you Bhupesh and Ram. Appreciate your quick response.
I see ThrottlingStatsListener.processStats() method gets called whenever
new stats are received from the operators.
How frequently these stats are sent by the operators? Is it end of every
window?
Thanks,
Vishal
On Sat, Dec 3, 2016 at
To further clarify Bhupesh's comment, suppose you determine in window N in
the input operator the
data reading phase is complete and send the control tuple on the dedicated
port to the output
operator in window N+1. If the downstream operators (including the output
operator) P_i are
processing resp
Hi Vishal,
A window is processed by an operator only when the previous window is
completely processed. When you send the control tuple in a new window, you
can be sure that all previous windows have been processed.
That is the reason I asked you to send the control tuple in a new window.
For shut
Thank you Bhupesh.
Another catch is just because input operator has processed last record
doesn't mean all the intermediate operators have processed it as well. How
can I ensure that all the operators have processed all the records before
performing the write operation.
Also is there a way to shu
Hi Vishal,
The support for such operations is currently being enhanced in Apex.
For now, you can do the following:
- Have an additional output port in your input operator as well as an
input port in the "Writer" operator.
- Once the Input operator has read and emitted all the data that it wante
Hi,
I am performing a batch operation. My input operator is reading multiple
files line by line and then there are bunch of operators manipulating the
records to evaluate result.
My output operator is supposed to write the final result to external system
once all the records from each of the files
14 matches
Mail list logo
