I don't think an operator needs a specific appender. An appender can be
dynamically assigned to an application designer, application master and
container.
Thanks,
Sergey
On Mon, Apr 10, 2017 at 6:26 PM, Munagala Ramanath
wrote:
> I don't have one, I thought that was what the intent of the prop
[
https://issues.apache.org/jira/browse/APEXCORE-575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pramod Immaneni updated APEXCORE-575:
-
Fix Version/s: 3.6.0
> Improve application relaunch time.
>
It is a callback on the operator thread.
Thank you,
Vlad
On 4/10/17 20:00, Ganelin, Ilya wrote:
Neat – idle would be when not in process, begin window, or end window? Is that
its own event loop or is it a periodic callback?
- Ilya Ganelin
On 4/10/17, 1:58 PM, "Pramod Immaneni" wrote:
Neat – idle would be when not in process, begin window, or end window? Is that
its own event loop or is it a periodic callback?
- Ilya Ganelin
On 4/10/17, 1:58 PM, "Pramod Immaneni" wrote:
You can also emit when the system is idling by implementing the
IdleTimeHandler interface in you
[
https://issues.apache.org/jira/browse/APEXCORE-697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vlad Rozov closed APEXCORE-697.
---
Resolution: Invalid
The JIRA is way too abstract. If there is a concrete unit test that should use
I don't have one, I thought that was what the intent of the proposal was,
but looks like
I misunderstood. After re-reading some of the earlier responses, I
understand the
proposal better.
Ram
On Mon, Apr 10, 2017 at 5:39 PM, Vlad Rozov wrote:
> I don't see a use case where an individual opera
I don't see a use case where an individual operators need to define a
specific appender, can you provide one?
Thank you,
Vlad
On 4/10/17 16:53, Munagala Ramanath wrote:
Yes, totally agree, it would be helpful to have a detailed use case and/or
a detailed spec
of the desired capabilities -- no
Yes, totally agree, it would be helpful to have a detailed use case and/or
a detailed spec
of the desired capabilities -- not necessarily a complete spec but with
enough detail to
understand why existing capabilities are inadequate.
Ram
On Mon, Apr 10, 2017 at 4:43 PM, Vlad Rozov wrote:
> It wi
It will be good to understand a use case where an operator needs a
specific appender.
IMO, an operator designer defines *what* should be logged and dev-ops
team defines *where* to log.
Thank you,
Vlad
On 4/10/17 16:27, Munagala Ramanath wrote:
Yes, I understand, I was just wondering if indi
How this functionality of log4j can be used in the Yarn environment? It
sounds like a more complicated solutions compared to providing an
ability to programmatically add an appender at launch time and/or run-time.
Thank you,
Vlad
On 4/10/17 16:18, Munagala Ramanath wrote:
http://logging.apac
Yes, I understand, I was just wondering if individual operators could
define the appenders
they potentially need at compile time and then the operator callbacks could
simply
check the desired runtime condition and add the appropriate appender.
Or are we saying there are scenarios where we absolute
http://logging.apache.org/log4j/1.2/faq.html#3.6
Log4j has the ability to dynamically reload changes to the properties file
via the
*configureAndWatch()* method of *PropertyConfigurator*. If this is already
baked in, could
devops simply change the properties file if we provide an enhanced set of
a
Yes, it makes sense. We can add/remove an run-time appender dynamically.
Thanks,
Sergey
On Mon, Apr 10, 2017 at 2:30 PM, Pramod Immaneni
wrote:
> As we are already doing dynamic log level changes and are talking about
> dynamically adding appenders, does it make sense and technically feasibl
[
https://issues.apache.org/jira/browse/APEXCORE-575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pramod Immaneni resolved APEXCORE-575.
--
Resolution: Fixed
> Improve application relaunch time.
> -
[
https://issues.apache.org/jira/browse/APEXCORE-575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15963582#comment-15963582
]
ASF GitHub Bot commented on APEXCORE-575:
-
Github user asfgit closed the pull re
Github user asfgit closed the pull request at:
https://github.com/apache/apex-core/pull/422
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is e
As we are already doing dynamic log level changes and are talking about
dynamically adding appenders, does it make sense and technically feasible
to apply a delta log4j configuration dynamically (which can include log
levels + appeneders + ?) on top of the static configuration provided by the
syste
Ram,
Really the new appender class must extend the abstract class
AppenderSkeleton. And in order to add a new appender programmatically in
Java, some code in Apex should call the following log4j method:
org.apache.log4j.Logger.getRootLogger().addAppender(Appender newAppender)
The general idea of
It will require application recompilation and repackaging. The proposed
functionality is for dev-ops to be able to route application logging to
a preferred destination without recompiling applications. It is run-time
configuration vs compile time hardcoded appender.
Thank you,
Vlad
On 4/10/17
You can also emit when the system is idling by implementing the
IdleTimeHandler interface in your operator.
On Mon, Apr 10, 2017 at 1:06 PM, Amol Kekre wrote:
> Not yet, But we could leverage internal structures of Apex as they do same
> thing. For example in container local streams. There is a
Not yet, But we could leverage internal structures of Apex as they do same
thing. For example in container local streams. There is a catch though -
the queue read by main thread will only happen when another data tuple
arrives in process call, or control tuple arrives for start or end window.
Thks
Thanks, Amol – that makes sense and was the solution I’d arrived at. I just was
trying to avoid the delay between the data being ready and emitting it. Has
anyone built a solution where it emits from the parent as soon as it’s ready in
the child (assuming I don’t care about order).
- Ilya Gane
Ilya,
This constraint was introduced as allowing two threads to emit data creates
lots of bad situations
1. The emit is triggered between end_window and begin_window. This was a
critical blocker
2. Order no longer guaranteed, upon replay getting wrong order of events
within a window. This was somet
Hello – I’ve got an operator that runs a cleanup thread (separate from the main
event loop) and triggers a callback when an item is removed from an internal
data structure. I would like for this callback to emit data from one of the
operator’s ports, but I run into the following Exception:
(Fro
[
https://issues.apache.org/jira/browse/APEXCORE-609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15963340#comment-15963340
]
ASF GitHub Bot commented on APEXCORE-609:
-
GitHub user PramodSSImmaneni opened a
GitHub user PramodSSImmaneni opened a pull request:
https://github.com/apache/apex-core/pull/507
APEXCORE-609 Backpressure when spooling is disabled.
The publisher is suspended if ahead of subscriber by maximum number of
blocks. This doesn't yet address the issue of blocks being ov
Sandesh created APEXCORE-697:
Summary: Use MiniYARNCluster for unit tests
Key: APEXCORE-697
URL: https://issues.apache.org/jira/browse/APEXCORE-697
Project: Apache Apex Core
Issue Type: Improveme
You can do it in a trivial derived class without changing the base class.
Ram
On Mon, Apr 10, 2017 at 11:19 AM, Vlad Rozov
wrote:
> Does not the proposal to use Logger.addAppender() requires modifications
> to used operators code?
>
> Thank you,
>
> Vlad
>
> On 4/10/17 10:58, Munagala Ramanath
Does not the proposal to use Logger.addAppender() requires modifications
to used operators code?
Thank you,
Vlad
On 4/10/17 10:58, Munagala Ramanath wrote:
People can currently do this by simply implementing the Appender interface
and adding it
with Logger.addAppender() in the setup method. W
People can currently do this by simply implementing the Appender interface
and adding it
with Logger.addAppender() in the setup method. Why do we need something
more elaborate ?
Ram
On Mon, Apr 10, 2017 at 10:30 AM, Sergey Golovko
wrote:
> The configuration of a log4j appender via log4j configu
The configuration of a log4j appender via log4j configuration file is a
static configuration that cannot be disabled/enabled and managed
dynamically by an application designer. The programmatic approach will
allow an application designer to specify which of the available log4j
appenders should be
Depends on a use case. Consider join->aggregate->topN pipeline. My point
was that even for reader->aggregator->writer there is a way to better
handle resources.
Thank you,
Vlad
On 4/10/17 09:07, Thomas Weise wrote:
Yes, but source is not important for the resource allocation aspect because
i
Yes, but source is not important for the resource allocation aspect because
it is a reader that does not hold a lot of resources. The big ticket items
are join and topN, and they need to be allocated at the same time if you
don't have a swap space.
On Mon, Apr 10, 2017 at 8:56 AM, Vlad Rozov wro
For the second pipeline, source can be de-allocated as soon as join gets
all data and join can be de-allocated as soon as topN gets all data.
Note that topN (and sink) does not need to be allocated before join
starts emitting data.
Thank you,
Vlad
On 4/10/17 08:48, Thomas Weise wrote:
The p
The pipeline depends on the resource availability. It could be:
( source -> join -> writer ) - - -> ( reader -> topN -> sink)
or
(source -> join -> topN -> sink)
The second case does not allow you do deallocate join (join and topN are
active at the same time).
On Mon, Apr 10, 2017 at 8:37 AM,
It is important. The generic pipeline proposed is (... -> writer) --->
(reader -> join -> writer) ---> (reader -> ...), where reader->
aggregator -> writer becomes a common pattern for a single stage processing.
Thank you,
Vlad
On 4/10/17 08:31, Thomas Weise wrote:
Where the data comes from
Where the data comes from isn't important for this discussion. The scenario
is join -> topN
With intermediate files it is: ( join -> writer ) - - -> ( reader -> topN )
On Mon, Apr 10, 2017 at 8:26 AM, Vlad Rozov wrote:
> In your example join is both consumer and producer, is not it? Where does
[
https://issues.apache.org/jira/browse/APEXCORE-408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15963035#comment-15963035
]
Pramod Immaneni commented on APEXCORE-408:
--
How about widening the scope to als
In your example join is both consumer and producer, is not it? Where
does it get data from? Join is not an input operator.
Thank you,
Vlad
On 4/10/17 08:13, Thomas Weise wrote:
In this example join/writer produces the data, reader/topN consumes. You
cannot deallocate producer before all data
Great, for some reason I was under the impression it were changes sent from
individual operators. I will add to the JIRA to add more trigger points
from where changes can happen.
On Mon, Apr 10, 2017 at 8:19 AM, Thomas Weise wrote:
> That sounds very similar to
> https://issues.apache.org/jira/b
That sounds very similar to
https://issues.apache.org/jira/browse/APEXCORE-408 ?
On Mon, Apr 10, 2017 at 8:13 AM, Pramod Immaneni
wrote:
> Currently, DAG topological changes are limited to repartitioning of
> existing operators after stats update and client directed logical plan
> changes via w
Even now operators need to be aware of connected ports as it may need to
do complex processing before emitting on a port that is not connected.
From an operator point of view, closing output ports simply means that
the operator guarantees that it will not emit data on that port. It is
engine t
Currently, DAG topological changes are limited to repartitioning of
existing operators after stats update and client directed logical plan
changes via web services. We have seen the need for more dynamic plan
changes especially from batch use cases and other scenarios.
I wanted to start a discussi
In this example join/writer produces the data, reader/topN consumes. You
cannot deallocate producer before all data has been drained. When using
files, join/writer can be deallocated when all data was flushed to the
files and allocation of consumer can wait until that occurred, if the space
isn't a
With additional file readers/writers the pipeline of a single stage
becomes the 3 operator use case I described. With ability to open/close
ports, platform can optimize it by re-allocating resources from readers
to writers.
Thank you,
Vlad
On 4/10/17 07:44, Thomas Weise wrote:
In streaming
In streaming there is a stream (surprise), in a space constraint batch
case, we can have additional file writers/readers between the operators.
Modules can in fact be used to support pipeline reuse, but they must be
added/removed dynamically to support stages with on-demand resource
allocation.
T
Do you suggest that in a streaming use case join operator also pass data
to downstream using files or that there are two different join operators
one for streaming and one for batch? If not, it means that the join
operator needs to emit data to a separate file output operator, so it
still needs
I also think that this sort of stages/flow control does not belong into the
operator. Why should the operator code contain logic to open and close
ports? Then they could not be reused in an application where the same
functions are used on smaller windows in a pipeline fashion.
On Mon, Apr 10, 201
Right, I did not mean that APEXCORE-408 can not support in-memory
streams, my point was that it is preferable to pass data between stages
by not using disk/HDFS and that use cases for the proposal and
APEXCORE-408 are the same or very close to each other.
I filed https://issues.apache.org/jira
[
https://issues.apache.org/jira/browse/APEXCORE-696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vlad Rozov updated APEXCORE-696:
Description: Ability to open and close output ports should help with
supporting batch use cases an
I don't think this fully covers the the scenario of limited resources. You
describe a case of 3 operators, but when you consider just 2 operators that
both have to hold a large data set in memory, then the suggested approach
won't work. Let's say the first operator is outer join and the second
oper
Vlad Rozov created APEXCORE-696:
---
Summary: Provide an ability to open and close output ports
Key: APEXCORE-696
URL: https://issues.apache.org/jira/browse/APEXCORE-696
Project: Apache Apex Core
+1 Apex engine does not own log4j config file - it is provided either by
Hadoop or an application. Hadoop log4j config does not necessarily meet
application logging requirements, but if log4j is provided by an
application designer, who can only specify what to log, it may not meet
operations re
GitHub user deepak-narkhede reopened a pull request:
https://github.com/apache/apex-malhar/pull/601
APEXMALHAR-2431 Create Kinesis Input operator which emits byte array
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/deepak-narkh
[
https://issues.apache.org/jira/browse/APEXMALHAR-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15962635#comment-15962635
]
ASF GitHub Bot commented on APEXMALHAR-2431:
GitHub user deepak-narkhede
[
https://issues.apache.org/jira/browse/APEXMALHAR-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15962634#comment-15962634
]
ASF GitHub Bot commented on APEXMALHAR-2431:
Github user deepak-narkhede
Github user deepak-narkhede closed the pull request at:
https://github.com/apache/apex-malhar/pull/601
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the f
57 matches
Mail list logo
