[jira] [Updated] (APEXCORE-655) Support LATEST as archetype version when creating a project
[ https://issues.apache.org/jira/browse/APEXCORE-655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-655: Priority: Minor (was: Major) > Support LATEST as archetype version when creating a project > > > Key: APEXCORE-655 > URL: https://issues.apache.org/jira/browse/APEXCORE-655 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Thomas Weise >Priority: Minor > > Rather than keep on replacing the Apex archetype version number in various > examples, Maven supports LATEST to refer to the latest released archetype > version. > This requires a change to decouple resolution of Apex dependency version from > archetype version to replace the Apex version when the archetype is created > vs. when the project is created. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Reopened] (APEXCORE-634) Apex Platform unable to set unifier attributes for modules in DAG
[ https://issues.apache.org/jira/browse/APEXCORE-634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov reopened APEXCORE-634: - > Apex Platform unable to set unifier attributes for modules in DAG > - > > Key: APEXCORE-634 > URL: https://issues.apache.org/jira/browse/APEXCORE-634 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Deepak Narkhede >Assignee: Deepak Narkhede > Fix For: 3.6.0 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (APEXCORE-635) Proposal: Manage memory to avoid memory copy and garbage collection
[ https://issues.apache.org/jira/browse/APEXCORE-635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15888527#comment-15888527 ] Vlad Rozov commented on APEXCORE-635: - bq. Yes, Agree, One array of byte only treat as one object. But as the memory is relative big, it probably will not allocated in TLA( Thread Local Area ), which means it need to get lock for each allocation, so the allocation is not a very light operation. Is your concern GC or allocating new byte array? From what I understand, it was GC. bq. Output.getBuffer() probably not work in our case. The getBuffer() returns the reference of Output's internal buffer, which was used for putting the serialized bytes. As we need to share one instance of Output for multiple tuples, the buffer was reused when serialize next tuple. Which cause problem no matter we set the output's position to zero or not. If set position to zero, which means clean the previous serialized data. If not changed position, then the serialized data of next tuple concat to the previous one, but we don't know the boundary of each tuple, and the buffer keeping on growing which means output have to keep on allocate new big memory and copy the old data(if the initial buffer size is not very big). Output.getBuffer() works. Tuples are serialized one at a time and after each serialization, bytes from the shared Output buffer are copied to the SerializedTuple that allocates a new buffer each time. It avoids extra copy that you refer to. bq. Yes, the original prototype reset the buffer when it should not. But as I pointed in this proposal, the buffer will be reset after the data sent to socket. That why need to override the write() method. The original prototype has a serious bug, so the results of benchmarking are not reliable. Prior to implementing the proposed approach, I suggest to implement a prototype in stages and have a prove that the approach will provide a significant performance improvement. Without such prove, the complexity of the proposal is not justifiable. bq. Yes, agree. But I think it was worthy to change StreamCode, as suggested in the proposal, it's not easy to implemented the customized reusable serialization without memory copy with the previous interface. And it doesn't compatible with kryo's interface. The current implementation(DefaultStatefulStreamCodec) of StatefulStreamCodec seems not natural for me. The pairs are cleared after first tuple. And for each other tuples, the pairs need to checked, and an instance of DataStatePair need to created to wrapper state(which is null except first tuple) and data. I agree that the interface is not optimal and needs to be changed, but it is not possible to change it without breaking backward compatibility. It means that new interface needs to be introduced and BufferServer will need to deal with 3 different codec interfaces. Before this is done, I'd like to have a prove that there is significant performance benefit that justifies the complexity. bq. The netlet client need to aware of the memory management mechanism in order to reset the memory. But the code is maintained in same class (current BufferServerPublisher; an extended class in proposal), so this should be ok. The serialization formation don't have to change. it can compatible with the previous one. But if we want to take advantage of reserve() function of SerializationBuffer to avoid another extra copy, the format need to be changed to avoid use variable length. Again, I'd like to see a staged prototype that proves that there is a performance benefit and that it is possible to avoid tight code coupling between Codec, Publisher and Tuple serialization format. Overall, I am still (n) on the approach and I think that it needs to be revisited when netlet->netty migration is done. Netty has it's own memory allocator and it will be necessary to see how it can be integrated with Kryo and other serializers. > Proposal: Manage memory to avoid memory copy and garbage collection > --- > > Key: APEXCORE-635 > URL: https://issues.apache.org/jira/browse/APEXCORE-635 > Project: Apache Apex Core > Issue Type: Wish >Reporter: bright chen >Assignee: bright chen > > Manage memory to avoid memory copy and garbage collection > The aim of this proposal is to reuse the memory to avoid the garbage > collection and avoid unnecessary memory copy to increase the performance. In > this proposal the term serde means serialization and deserialization. It’s > same as codec. > Currently, apex by default use DefaultStatefulStreamCodec for serde, which > extends Kryo and optimize it by replace class by class id. And application > developer can optimize serializer by implement interface StreamCodec. > First, let’s look into the default codec
[jira] [Created] (APEXCORE-518) Update codestyle-config project pom.xml
Vlad Rozov created APEXCORE-518: --- Summary: Update codestyle-config project pom.xml Key: APEXCORE-518 URL: https://issues.apache.org/jira/browse/APEXCORE-518 Project: Apache Apex Core Issue Type: Task Reporter: Vlad Rozov Assignee: Vlad Rozov Priority: Trivial -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXCORE-518) Update codestyle-config project pom.xml
[ https://issues.apache.org/jira/browse/APEXCORE-518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459045#comment-15459045 ] Vlad Rozov commented on APEXCORE-518: - IMO, codestyle-config project is completely independent of Apache Apex core and may even reside in a different git repository. It is not a sub-module of Apex core. I don't see why is it necessary to combine changes to the Apache Apex parent pom and codestyle-config pom together. > Update codestyle-config project pom.xml > --- > > Key: APEXCORE-518 > URL: https://issues.apache.org/jira/browse/APEXCORE-518 > Project: Apache Apex Core > Issue Type: Task >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Trivial > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (APEXCORE-521) Upgrade Apex core pom.xml to the latest Apache pom parent
Vlad Rozov created APEXCORE-521: --- Summary: Upgrade Apex core pom.xml to the latest Apache pom parent Key: APEXCORE-521 URL: https://issues.apache.org/jira/browse/APEXCORE-521 Project: Apache Apex Core Issue Type: Task Reporter: Vlad Rozov Assignee: Thomas Weise Priority: Trivial -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (APEXCORE-506) Add trailing whitespace check to Apex checkstyle
[ https://issues.apache.org/jira/browse/APEXCORE-506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov resolved APEXCORE-506. - Resolution: Implemented Fix Version/s: 3.5.0 > Add trailing whitespace check to Apex checkstyle > > > Key: APEXCORE-506 > URL: https://issues.apache.org/jira/browse/APEXCORE-506 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Trivial > Fix For: 3.5.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (APEXCORE-522) Promote singleton usage pattern for String2String, Long2String and other StringCodecs
[ https://issues.apache.org/jira/browse/APEXCORE-522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-522: Description: String2String codec does not hold state, a single instance of the same String2String class may be reused. > Promote singleton usage pattern for String2String, Long2String and other > StringCodecs > - > > Key: APEXCORE-522 > URL: https://issues.apache.org/jira/browse/APEXCORE-522 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Minor > > String2String codec does not hold state, a single instance of the same > String2String class may be reused. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (APEXCORE-522) Promote singleton usage pattern for String2String, Long2String and other StringCodecs
Vlad Rozov created APEXCORE-522: --- Summary: Promote singleton usage pattern for String2String, Long2String and other StringCodecs Key: APEXCORE-522 URL: https://issues.apache.org/jira/browse/APEXCORE-522 Project: Apache Apex Core Issue Type: Improvement Reporter: Vlad Rozov Assignee: Vlad Rozov Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (APEXCORE-515) Refresh tokens failing in some scenarios with a login failure message
[ https://issues.apache.org/jira/browse/APEXCORE-515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov resolved APEXCORE-515. - Resolution: Fixed Fix Version/s: 3.5.0 > Refresh tokens failing in some scenarios with a login failure message > - > > Key: APEXCORE-515 > URL: https://issues.apache.org/jira/browse/APEXCORE-515 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Pramod Immaneni >Assignee: Pramod Immaneni > Fix For: 3.5.0 > > > In some scenarios the token refresh to allow applications to run without > shutting down is failing with the following exception > java.io.IOException: Login failure for x from keytab > at > org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1146) > at > com.datatorrent.stram.security.StramUserLogin.refreshTokens(StramUserLogin.java:96) > at > com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:623) > at > com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:313) > Caused by: javax.security.auth.login.LoginException: Unable to obtain > password from user > > at > com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:897) > at > com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760) > at > com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > javax.security.auth.login.LoginContext.invoke(LoginContext.java:755) > at > javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) > at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) > at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680) > at java.security.AccessController.doPrivileged(Native Method) > at > javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) > at javax.security.auth.login.LoginContext.login(LoginContext.java:587) > at > org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1135) > ... 3 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (APEXCORE-524) Add support for custom maven repository to ClassPathResolverTest.testManifestClassPathResolver
Vlad Rozov created APEXCORE-524: --- Summary: Add support for custom maven repository to ClassPathResolverTest.testManifestClassPathResolver Key: APEXCORE-524 URL: https://issues.apache.org/jira/browse/APEXCORE-524 Project: Apache Apex Core Issue Type: Improvement Reporter: Vlad Rozov Assignee: Vlad Rozov Priority: Minor ClassPathResolverTest.testManifestClassPathResolver fails when maven is configured with non-default repository location (-Dmaven.repo.local). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (APEXCORE-521) Upgrade Apex core pom.xml to the latest Apache pom parent
[ https://issues.apache.org/jira/browse/APEXCORE-521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov resolved APEXCORE-521. - Resolution: Done Fix Version/s: 3.5.0 > Upgrade Apex core pom.xml to the latest Apache pom parent > - > > Key: APEXCORE-521 > URL: https://issues.apache.org/jira/browse/APEXCORE-521 > Project: Apache Apex Core > Issue Type: Task >Reporter: Vlad Rozov >Assignee: Thomas Weise >Priority: Trivial > Fix For: 3.5.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (APEXCORE-531) Enable System.out/System.err check for *Test
Vlad Rozov created APEXCORE-531: --- Summary: Enable System.out/System.err check for *Test Key: APEXCORE-531 URL: https://issues.apache.org/jira/browse/APEXCORE-531 Project: Apache Apex Core Issue Type: Task Reporter: Vlad Rozov Assignee: Vlad Rozov Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (APEXCORE-527) Minor changes in LocalStramChildLauncher to help with unit test failures
Vlad Rozov created APEXCORE-527: --- Summary: Minor changes in LocalStramChildLauncher to help with unit test failures Key: APEXCORE-527 URL: https://issues.apache.org/jira/browse/APEXCORE-527 Project: Apache Apex Core Issue Type: Improvement Reporter: Vlad Rozov Assignee: Vlad Rozov Priority: Minor - Catch Error in addition to Exception, so both are properly logged - Update childContainers in the container thread -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (APEXCORE-535) Node.teardown() should try to gracefully shutdown exectutor service
Vlad Rozov created APEXCORE-535: --- Summary: Node.teardown() should try to gracefully shutdown exectutor service Key: APEXCORE-535 URL: https://issues.apache.org/jira/browse/APEXCORE-535 Project: Apache Apex Core Issue Type: Improvement Reporter: Vlad Rozov Assignee: Vlad Rozov Priority: Minor Fix For: 3.5.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (APEXCORE-535) Node.teardown() should try to gracefully shutdown exectutor service
[ https://issues.apache.org/jira/browse/APEXCORE-535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-535: Description: Forceful shutdown of executor service leads to InterruptedException if asynchronous (was: Forceful shutdown of executor service leads to InterruptedException if asynchronious ) > Node.teardown() should try to gracefully shutdown exectutor service > --- > > Key: APEXCORE-535 > URL: https://issues.apache.org/jira/browse/APEXCORE-535 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Minor > Fix For: 3.5.0 > > > Forceful shutdown of executor service leads to InterruptedException if > asynchronous -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (APEXCORE-535) Node.teardown() should try to gracefully shutdown exectutor service
[ https://issues.apache.org/jira/browse/APEXCORE-535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-535: Description: Forceful shutdown of executor service leads to InterruptedException if asynchronious > Node.teardown() should try to gracefully shutdown exectutor service > --- > > Key: APEXCORE-535 > URL: https://issues.apache.org/jira/browse/APEXCORE-535 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Minor > Fix For: 3.5.0 > > > Forceful shutdown of executor service leads to InterruptedException if > asynchronious -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (APEXCORE-535) Node.teardown() should try to gracefully shutdown exectutor service
[
https://issues.apache.org/jira/browse/APEXCORE-535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vlad Rozov updated APEXCORE-535:
Description:
Forceful shutdown of the executor service leads to InterruptedException if
asynchronous checkpointing is in progress:
{noformat}
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1281)
at java.lang.Thread.join(Thread.java:1355)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:456)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:678)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:661)
at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:639)
at
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:305)
at org.apache.hadoop.fs.FileSystem.primitiveCreate(FileSystem.java:1011)
at
org.apache.hadoop.fs.DelegateToFileSystem.createInternal(DelegateToFileSystem.java:85)
at
org.apache.hadoop.fs.ChecksumFs$ChecksumFSOutputSummer.(ChecksumFs.java:344)
at org.apache.hadoop.fs.ChecksumFs.createInternal(ChecksumFs.java:390)
at
org.apache.hadoop.fs.AbstractFileSystem.create(AbstractFileSystem.java:575)
at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:676)
at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:672)
at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
at org.apache.hadoop.fs.FileContext.create(FileContext.java:672)
at
com.datatorrent.common.util.AsyncFSStorageAgent.copyToHDFS(AsyncFSStorageAgent.java:118)
at
com.datatorrent.stram.engine.Node$CheckpointHandler.call(Node.java:667)
at
com.datatorrent.stram.engine.Node$CheckpointHandler.call(Node.java:656)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-09-14 22:02:51,463 [Thread-2109] WARN util.Shell run - Error reading the
error stream
java.io.IOException: Stream closed
at java.io.BufferedReader.ensureOpen(BufferedReader.java:115)
at java.io.BufferedReader.readLine(BufferedReader.java:310)
at java.io.BufferedReader.readLine(BufferedReader.java:382)
at org.apache.hadoop.util.Shell$1.run(Shell.java:431)
{noformat}
was:Forceful shutdown of executor service leads to InterruptedException if
asynchronous
> Node.teardown() should try to gracefully shutdown exectutor service
> ---
>
> Key: APEXCORE-535
> URL: https://issues.apache.org/jira/browse/APEXCORE-535
> Project: Apache Apex Core
> Issue Type: Improvement
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Minor
> Fix For: 3.5.0
>
>
> Forceful shutdown of the executor service leads to InterruptedException if
> asynchronous checkpointing is in progress:
> {noformat}
> java.lang.InterruptedException
> at java.lang.Object.wait(Native Method)
> at java.lang.Thread.join(Thread.java:1281)
> at java.lang.Thread.join(Thread.java:1355)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:456)
> at org.apache.hadoop.util.Shell.run(Shell.java:379)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:678)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:661)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:639)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:305)
> at
> org.apache.hadoop.fs.FileSystem.primitiveCreate(FileSystem.java:1011)
> at
> org.apache.hadoop.fs.DelegateToFileSystem.createInternal(DelegateToFileSystem.java:85)
> at
> org.apache.hadoop.fs.ChecksumFs$ChecksumFSOutputSummer.(ChecksumFs.java:344)
> at org.apache.hadoop.fs.ChecksumFs.createInternal(ChecksumFs.java:390)
> at
> org.apache.hadoop.fs.AbstractFileSystem.create(AbstractFileSystem.java:575)
> at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:676)
> at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:672)
> at
[jira] [Updated] (APEXMALHAR-2251) Replace dependency on com.datatorrent.netlet.util.Slice
[ https://issues.apache.org/jira/browse/APEXMALHAR-2251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXMALHAR-2251: --- Description: Dependency on com.datatorrent.netlet.util.Slice should be replaced with either java.nio.ByteBuffer or internal malhar class that provides similar functionality. (was: The ) Issue Type: Improvement (was: Bug) Summary: Replace dependency on com.datatorrent.netlet.util.Slice (was: Slice hashCode should not depend on offset) > Replace dependency on com.datatorrent.netlet.util.Slice > --- > > Key: APEXMALHAR-2251 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2251 > Project: Apache Apex Malhar > Issue Type: Improvement >Reporter: bright chen > > Dependency on com.datatorrent.netlet.util.Slice should be replaced with > either java.nio.ByteBuffer or internal malhar class that provides similar > functionality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXMALHAR-2255) Use latest java sdk for couchbase
[ https://issues.apache.org/jira/browse/APEXMALHAR-2255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510868#comment-15510868 ] Vlad Rozov commented on APEXMALHAR-2255: Is the request to upgrade JDK or upgrade couchbase client library? It seems that it is later. Please update title accordingly and specify upgrade version. Is it backward compatible in a sense that it can be used to connect to older version of couchbase server? > Use latest java sdk for couchbase > - > > Key: APEXMALHAR-2255 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2255 > Project: Apache Apex Malhar > Issue Type: Test >Reporter: Priyanka Gugale > > Right now the Couchbase connector in Malhar uses couchbase-client library > which is outdated. We should instead use java-client library from couchbase > to connect to couchbase server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXMALHAR-2251) Replace dependency on com.datatorrent.netlet.util.Slice
[ https://issues.apache.org/jira/browse/APEXMALHAR-2251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512077#comment-15512077 ] Vlad Rozov commented on APEXMALHAR-2251: Do you mean Direct or Heap? Majority of the code should work with ByteBuffer and not worry whether it is Direct or Heap allocated. > Replace dependency on com.datatorrent.netlet.util.Slice > --- > > Key: APEXMALHAR-2251 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2251 > Project: Apache Apex Malhar > Issue Type: Improvement >Reporter: bright chen > > Dependency on com.datatorrent.netlet.util.Slice should be replaced with > either java.nio.ByteBuffer or internal malhar class that provides similar > functionality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXCORE-539) DefaultAttributeMap is not thread safe
[
https://issues.apache.org/jira/browse/APEXCORE-539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514021#comment-15514021
]
Vlad Rozov commented on APEXCORE-539:
-
I see the following exception after I changed DefaultAttributeMap to store and
compare thread with the current thread:
{noformat}
2016-09-22 10:36:46,849 [container-0] ERROR stram.StramLocalCluster run -
Container container-0 failed
java.util.ConcurrentModificationException: current thread
Thread[container-0,5,main] existing thread Thread[main,5,main]
at
com.datatorrent.api.Attribute$AttributeMap$DefaultAttributeMap.put(Attribute.java:209)
at
com.datatorrent.stram.engine.StreamingContainer.setup(StreamingContainer.java:160)
at
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainer.run(StramLocalCluster.java:178)
at
com.datatorrent.stram.StramLocalCluster$LocalStramChildLauncher.run(StramLocalCluster.java:269)
at java.lang.Thread.run(Thread.java:745)
{noformat}
Why not to use ConcurrentHashMap instead of HashMap?
> DefaultAttributeMap is not thread safe
> --
>
> Key: APEXCORE-539
> URL: https://issues.apache.org/jira/browse/APEXCORE-539
> Project: Apache Apex Core
> Issue Type: Bug
>Reporter: Vlad Rozov
>
> DefaultAttributeMap put() and get() methods may be called from different
> threads and current implementation that uses HashMap is not thread safe.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (APEXCORE-539) DefaultAttributeMap is not thread safe
Vlad Rozov created APEXCORE-539: --- Summary: DefaultAttributeMap is not thread safe Key: APEXCORE-539 URL: https://issues.apache.org/jira/browse/APEXCORE-539 Project: Apache Apex Core Issue Type: Bug Reporter: Vlad Rozov DefaultAttributeMap put() and get() methods may be called from different threads and current implementation that uses HashMap is not thread safe. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (APEXCORE-538) Log raw data when RPC message fails to de-serialize
Vlad Rozov created APEXCORE-538: --- Summary: Log raw data when RPC message fails to de-serialize Key: APEXCORE-538 URL: https://issues.apache.org/jira/browse/APEXCORE-538 Project: Apache Apex Core Issue Type: Improvement Reporter: Vlad Rozov Assignee: Vlad Rozov Fix For: 3.2.2, 3.3.1, 3.4.1, 3.5.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (APEXCORE-516) StramLocalCluster should always use loopback address for buffer server location
[
https://issues.apache.org/jira/browse/APEXCORE-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15440431#comment-15440431
]
Vlad Rozov edited comment on APEXCORE-516 at 8/27/16 2:17 AM:
--
Example of bad configurion from builds.apache.org:
{noformat}
InetAddress.getLocalHost():
214.175.197.104.bc.googleusercontent.com/104.197.175.214
hostname
jenkins-beam2.apache.org
hostname -i
104.197.175.214
hostname -I
10.128.0.2 172.17.0.1
{noformat}
was (Author: vrozov):
Example of bad configurion from builds.apache.org:
{noformat}
InetAddress.getLocalHost():
214.175.197.104.bc.googleusercontent.com/104.197.175.214
hostname
jenkins-beam2.apache.org
/bin/hostname -i
104.197.175.214
hostname -I
10.128.0.2 172.17.0.1
{noformat}
> StramLocalCluster should always use loopback address for buffer server
> location
> ---
>
> Key: APEXCORE-516
> URL: https://issues.apache.org/jira/browse/APEXCORE-516
> Project: Apache Apex Core
> Issue Type: Improvement
> Environment: Jenkins Apache (builds.apache.org using beam label)
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Minor
>
> With incorrectly configured environments where InetAddress.getLocalHost()
> returns resolvable address, but that address is not among active interfaces
> of the host where buffer server is deployed, neither buffer server publish or
> subscriber will be able to connect to the buffer server. In most cases, this
> issue should be solved by properly configuring the cluster or node where
> buffer server is deployed. This fix should address only a case where Apex is
> deployed into an automatically provisioned environment for running unit and
> integration tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (APEXCORE-516) StramLocalCluster should always use loopback address for buffer server location
[
https://issues.apache.org/jira/browse/APEXCORE-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15440431#comment-15440431
]
Vlad Rozov commented on APEXCORE-516:
-
Example of bad configurion from builds.apache.org:
{noformat}
InetAddress.getLocalHost():
214.175.197.104.bc.googleusercontent.com/104.197.175.214
hostname
jenkins-beam2.apache.org
/bin/hostname -i
104.197.175.214
hostname -I
10.128.0.2 172.17.0.1
{noformat}
> StramLocalCluster should always use loopback address for buffer server
> location
> ---
>
> Key: APEXCORE-516
> URL: https://issues.apache.org/jira/browse/APEXCORE-516
> Project: Apache Apex Core
> Issue Type: Improvement
> Environment: Jenkins Apache (builds.apache.org using beam label)
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Minor
>
> With incorrectly configured environments where InetAddress.getLocalHost()
> returns resolvable address, but that address is not among active interfaces
> of the host where buffer server is deployed, neither buffer server publish or
> subscriber will be able to connect to the buffer server. In most cases, this
> issue should be solved by properly configuring the cluster or node where
> buffer server is deployed. This fix should address only a case where Apex is
> deployed into an automatically provisioned environment for running unit and
> integration tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (APEXCORE-516) StramLocalCluster should always use loopback address for buffer server location
Vlad Rozov created APEXCORE-516: --- Summary: StramLocalCluster should always use loopback address for buffer server location Key: APEXCORE-516 URL: https://issues.apache.org/jira/browse/APEXCORE-516 Project: Apache Apex Core Issue Type: Improvement Environment: Jenkins Apache (builds.apache.org using beam label) Reporter: Vlad Rozov Assignee: Vlad Rozov Priority: Minor With incorrectly configured environments where InetAddress.getLocalHost() returns resolvable address, but that address is not among active interfaces of the host where buffer server is deployed, neither buffer server publish or subscriber will be able to connect to the buffer server. In most cases, this issue should be solved by properly configuring the cluster or node where buffer server is deployed. This fix should address only a case where Apex is deployed into an automatically provisioned environment for running unit and integration tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (APEXCORE-169) instantiating DTLoggerFactory during test causes incorrect logging behavior
[ https://issues.apache.org/jira/browse/APEXCORE-169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-169: Assignee: Vlad Rozov > instantiating DTLoggerFactory during test causes incorrect logging behavior > --- > > Key: APEXCORE-169 > URL: https://issues.apache.org/jira/browse/APEXCORE-169 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Vlad Rozov >Assignee: Vlad Rozov > > After DTLoggerFactory is instantiated in maven build, log levels in > log4j.properties are ignored. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (APEXCORE-519) Add support for DIGEST enabled hadoop web services environment
[ https://issues.apache.org/jira/browse/APEXCORE-519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov resolved APEXCORE-519. - Resolution: Fixed > Add support for DIGEST enabled hadoop web services environment > -- > > Key: APEXCORE-519 > URL: https://issues.apache.org/jira/browse/APEXCORE-519 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Pramod Immaneni >Assignee: Pramod Immaneni > > Add support for Apex to work with DIGEST enabled Hadoop web services. Work > would have common elements with BASIC authentication, APEXCORE-517 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (APEXCORE-519) Add support for DIGEST enabled hadoop web services environment
[ https://issues.apache.org/jira/browse/APEXCORE-519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov reopened APEXCORE-519: - > Add support for DIGEST enabled hadoop web services environment > -- > > Key: APEXCORE-519 > URL: https://issues.apache.org/jira/browse/APEXCORE-519 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Pramod Immaneni >Assignee: Pramod Immaneni > > Add support for Apex to work with DIGEST enabled Hadoop web services. Work > would have common elements with BASIC authentication, APEXCORE-517 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (APEXCORE-519) Add support for DIGEST enabled hadoop web services environment
[ https://issues.apache.org/jira/browse/APEXCORE-519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov resolved APEXCORE-519. - Resolution: Implemented > Add support for DIGEST enabled hadoop web services environment > -- > > Key: APEXCORE-519 > URL: https://issues.apache.org/jira/browse/APEXCORE-519 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Pramod Immaneni >Assignee: Pramod Immaneni > > Add support for Apex to work with DIGEST enabled Hadoop web services. Work > would have common elements with BASIC authentication, APEXCORE-517 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXCORE-510) Enforce DefaultOutputPort.emit() or Sink.put() thread affinity
[ https://issues.apache.org/jira/browse/APEXCORE-510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526970#comment-15526970 ] Vlad Rozov commented on APEXCORE-510: - There are two possible solutions to the issue: 1. Implement check in the Sink. 2. Allow ports that don't call DefaultPort.setup() to skip the check. > Enforce DefaultOutputPort.emit() or Sink.put() thread affinity > -- > > Key: APEXCORE-510 > URL: https://issues.apache.org/jira/browse/APEXCORE-510 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Vlad Rozov > > Apex platform assumes that an operator interacts with the platform using the > dedicated operator thread. Currently, operators may create worker threads and > emit tuples from a worker thread. This leads to undefined behavior and hard > to find bugs, so it should be possible to enforce that > DefaultOutputPort.emit() and/or Sink.put() are called on the operator thread. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (APEXCORE-565) Ability to activate and de-activate operators in the DAG
Vlad Rozov created APEXCORE-565: --- Summary: Ability to activate and de-activate operators in the DAG Key: APEXCORE-565 URL: https://issues.apache.org/jira/browse/APEXCORE-565 Project: Apache Apex Core Issue Type: New Feature Reporter: Vlad Rozov Currently, there is no ability to mark operators/streams as inactive and activate or de-activate them at run-time based on a condition or event. It should be possible to statically populate DAG, but mark some operators as inactive, so stram app master will not allocate physical nodes for inactive operators and will not request containers from Yarn or any other resource manager. This will allow to save resources and better support batch operations in Apex. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (APEXCORE-568) whitelist used dependencies
Vlad Rozov created APEXCORE-568: --- Summary: whitelist used dependencies Key: APEXCORE-568 URL: https://issues.apache.org/jira/browse/APEXCORE-568 Project: Apache Apex Core Issue Type: Improvement Reporter: Vlad Rozov Priority: Minor Dependencies should be either explicitly declared in the section of pom.xml or, in case, we rely on a library that is a dependency of an explicitly declared dependency, such library should be whitelisted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXCORE-560) Logical plan is not changed when all physical partitions of operator are removed from DAG
[ https://issues.apache.org/jira/browse/APEXCORE-560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15591998#comment-15591998 ] Vlad Rozov commented on APEXCORE-560: - Why is it necessary to remove an operator from the logical plan? A logical plan does not correspond one to one to the physical plan. For example, unifiers are present only in the physical plan. > Logical plan is not changed when all physical partitions of operator are > removed from DAG > - > > Key: APEXCORE-560 > URL: https://issues.apache.org/jira/browse/APEXCORE-560 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Bhupesh Chawda > > Throwing a ShutdownException() from an input operator removes them from the > physical plan, but can still be seen in the logical plan. Ideally the > corresponding logical operator must also be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXCORE-560) Logical plan is not changed when all physical partitions of operator are removed from DAG
[ https://issues.apache.org/jira/browse/APEXCORE-560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15600183#comment-15600183 ] Vlad Rozov commented on APEXCORE-560: - Such approach may lead to a confusion and hard to debug issues. IMO, it will be better to mark some parts of the DAG as inactive and allow activation/de-activation of operators in the logical DAG. > Logical plan is not changed when all physical partitions of operator are > removed from DAG > - > > Key: APEXCORE-560 > URL: https://issues.apache.org/jira/browse/APEXCORE-560 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Bhupesh Chawda > > Throwing a ShutdownException() from an input operator removes them from the > physical plan, but can still be seen in the logical plan. Ideally the > corresponding logical operator must also be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (APEXCORE-564) We should be able to remove loggers level settings
[ https://issues.apache.org/jira/browse/APEXCORE-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603449#comment-15603449 ] Vlad Rozov edited comment on APEXCORE-564 at 10/24/16 10:43 PM: No, it does not. Setting log level to null in com.datatorrent.stram.util.LoggerUtil does not reset log level to the original level, it will be set to parent's level. For example, if log4j.properties has root logger set to ERROR and com.datatorrent set to INFO, applying "com.datatorrent.\*:DEBUG" pattern will set log level for all com.datatorrent.* loggers to debug. Applying "com.datatorrent.\*:null" pattern does not reset level to INFO as expected, it will reset it to the root level that is ERROR. was (Author: vrozov): No, it does not. Setting log level to null in com.datatorrent.stram.util.LoggerUtil does not reset log level to the original level, it will be set to parent's level. For example, if log4j.properties has root logger set to ERROR and com.datatorrent set to INFO, applying "com.datatorrent.*:DEBUG" pattern will set log level for all com.datatorrent.* loggers to debug. Applying "com.datatorrent.*:null" pattern does not reset level to INFO as expected, it will reset it to the root level that is ERROR. > We should be able to remove loggers level settings > -- > > Key: APEXCORE-564 > URL: https://issues.apache.org/jira/browse/APEXCORE-564 > Project: Apache Apex Core > Issue Type: New Feature >Reporter: Priyanka Gugale >Assignee: Priyanka Gugale >Priority: Minor > Original Estimate: 48h > Remaining Estimate: 48h > > StramWebServices has api to setLoggersLevel, but there is not way to remove > the already set loggers level. There should be an api to remove loggers level > which was set earlier. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXCORE-564) We should be able to remove loggers level settings
[ https://issues.apache.org/jira/browse/APEXCORE-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603449#comment-15603449 ] Vlad Rozov commented on APEXCORE-564: - No, it does not. Setting log level to null in com.datatorrent.stram.util.LoggerUtil does not reset log level to the original level, it will be set to parent's level. For example, if log4j.properties has root logger set to ERROR and com.datatorrent set to INFO, applying "com.datatorrent.*:DEBUG" pattern will set log level for all com.datatorrent.* loggers to debug. Applying "com.datatorrent.*:null" pattern does not reset level to INFO as expected, it will reset it to the root level that is ERROR. > We should be able to remove loggers level settings > -- > > Key: APEXCORE-564 > URL: https://issues.apache.org/jira/browse/APEXCORE-564 > Project: Apache Apex Core > Issue Type: New Feature >Reporter: Priyanka Gugale >Assignee: Priyanka Gugale >Priority: Minor > Original Estimate: 48h > Remaining Estimate: 48h > > StramWebServices has api to setLoggersLevel, but there is not way to remove > the already set loggers level. There should be an api to remove loggers level > which was set earlier. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (APEXCORE-510) Enforce DefaultOutputPort.emit() or Sink.put() thread affinity
[ https://issues.apache.org/jira/browse/APEXCORE-510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov resolved APEXCORE-510. - Resolution: Implemented > Enforce DefaultOutputPort.emit() or Sink.put() thread affinity > -- > > Key: APEXCORE-510 > URL: https://issues.apache.org/jira/browse/APEXCORE-510 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Vlad Rozov >Assignee: Sanjay M Pujare > > Apex platform assumes that an operator interacts with the platform using the > dedicated operator thread. Currently, operators may create worker threads and > emit tuples from a worker thread. This leads to undefined behavior and hard > to find bugs, so it should be possible to enforce that > DefaultOutputPort.emit() and/or Sink.put() are called on the operator thread. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (APEXCORE-510) Enforce DefaultOutputPort.emit() or Sink.put() thread affinity
[ https://issues.apache.org/jira/browse/APEXCORE-510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-510: Assignee: Sanjay M Pujare > Enforce DefaultOutputPort.emit() or Sink.put() thread affinity > -- > > Key: APEXCORE-510 > URL: https://issues.apache.org/jira/browse/APEXCORE-510 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Vlad Rozov >Assignee: Sanjay M Pujare > > Apex platform assumes that an operator interacts with the platform using the > dedicated operator thread. Currently, operators may create worker threads and > emit tuples from a worker thread. This leads to undefined behavior and hard > to find bugs, so it should be possible to enforce that > DefaultOutputPort.emit() and/or Sink.put() are called on the operator thread. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXCORE-564) We should be able to remove loggers level settings
[ https://issues.apache.org/jira/browse/APEXCORE-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15596825#comment-15596825 ] Vlad Rozov commented on APEXCORE-564: - The required functionality is not supported by com.datatorrent.stram.util.LoggerUtil and it is not straightforward to support resetting pattern. Note that setLoggersLevel has a cumulative effect. It means that calling com.datatorrent.stram.util.LoggerUtil.changeLoggersLevel() with empty map does not impact patterns and each time com.datatorrent.stram.util.LoggerUtil.changeLoggersLevel() is called, entries are added to the patterns and they are never removed, except for the case where a new pattern specified overwrites existing patterns. Please provide detailed specification how the functionality will be supported. > We should be able to remove loggers level settings > -- > > Key: APEXCORE-564 > URL: https://issues.apache.org/jira/browse/APEXCORE-564 > Project: Apache Apex Core > Issue Type: New Feature >Reporter: Priyanka Gugale >Assignee: Priyanka Gugale >Priority: Minor > Original Estimate: 48h > Remaining Estimate: 48h > > StramWebServices has api to setLoggersLevel, but there is not way to remove > the already set loggers level. There should be an api to remove loggers level > which was set earlier. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXMALHAR-2263) Offsets in AbstractFileInputOperator should be long rather than int
[ https://issues.apache.org/jira/browse/APEXMALHAR-2263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583599#comment-15583599 ] Vlad Rozov commented on APEXMALHAR-2263: Agree with [~thw] that introducing orphaned properties breaks semantic versioning even though it will avoid japicmp from breaking the build. > Offsets in AbstractFileInputOperator should be long rather than int > --- > > Key: APEXMALHAR-2263 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2263 > Project: Apache Apex Malhar > Issue Type: Bug > Components: adapters other >Reporter: Munagala V. Ramanath >Assignee: Matt Zhang > > Offsets in AbstractFileInputOperator use the int type which means files with > more that (2**31 -1) records will cause overflows and mysterious failures. > Should be changed to use long. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (APEXCORE-572) Remove dependency on hadoop-common test.jar
[ https://issues.apache.org/jira/browse/APEXCORE-572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-572: Description: The dependency on hadoop-common test-jar can be avoided by implementing a few helper methods. It is not necessary to have such dependency. > Remove dependency on hadoop-common test.jar > --- > > Key: APEXCORE-572 > URL: https://issues.apache.org/jira/browse/APEXCORE-572 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Minor > > The dependency on hadoop-common test-jar can be avoided by implementing a few > helper methods. It is not necessary to have such dependency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (APEXCORE-563) Have a pointer to container log file name and offset in stram events that deliver a container or operator failure event.
[ https://issues.apache.org/jira/browse/APEXCORE-563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-563: Issue Type: New Feature (was: Bug) > Have a pointer to container log file name and offset in stram events that > deliver a container or operator failure event. > > > Key: APEXCORE-563 > URL: https://issues.apache.org/jira/browse/APEXCORE-563 > Project: Apache Apex Core > Issue Type: New Feature >Reporter: Sanjay M Pujare >Assignee: Priyanka Gugale > > The default DailyRollingFileAppender does not take into account of how many > backup files to keep and it will result in unbounded growth of log files, > especially for long running applications. > The below is an interesting add-on to the default DailyRollingFileAppender > that supports maxBackupIndex. > http://wiki.apache.org/logging-log4j/DailyRollingFileAppender -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (APEXCORE-461) Hidden ports in extended operators appear in the application package
[ https://issues.apache.org/jira/browse/APEXCORE-461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-461: Assignee: Bhupesh Chawda > Hidden ports in extended operators appear in the application package > > > Key: APEXCORE-461 > URL: https://issues.apache.org/jira/browse/APEXCORE-461 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Bhupesh Chawda >Assignee: Bhupesh Chawda > Fix For: 3.5.0 > > > Ports which are hidden due to overriding in the sub classes of the operator > should not be shown in the final app package. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (APEXCORE-525) DefaultStatefulStreamCodec not implementing newInstance method keep using old codec.
[ https://issues.apache.org/jira/browse/APEXCORE-525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-525: Priority: Minor (was: Major) > DefaultStatefulStreamCodec not implementing newInstance method keep using old > codec. > > > Key: APEXCORE-525 > URL: https://issues.apache.org/jira/browse/APEXCORE-525 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Tushar Gosavi >Assignee: Tushar Gosavi >Priority: Minor > > Custom stream codec extended from DefaultStatefulStreamCodec but not > overriding newInstance method keep on using default > DefaultStatefulStreamCodec without any warning to end user, causing wrong > result in the application. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (APEXCORE-525) DefaultStatefulStreamCodec not implementing newInstance method keep using old codec.
[ https://issues.apache.org/jira/browse/APEXCORE-525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-525: Issue Type: Improvement (was: Bug) > DefaultStatefulStreamCodec not implementing newInstance method keep using old > codec. > > > Key: APEXCORE-525 > URL: https://issues.apache.org/jira/browse/APEXCORE-525 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Tushar Gosavi >Assignee: Tushar Gosavi > > Custom stream codec extended from DefaultStatefulStreamCodec but not > overriding newInstance method keep on using default > DefaultStatefulStreamCodec without any warning to end user, causing wrong > result in the application. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (APEXCORE-525) Subclass aware DefaultStatefulStreamCodec.newInstance() implementation
[ https://issues.apache.org/jira/browse/APEXCORE-525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov resolved APEXCORE-525. - Resolution: Implemented Fix Version/s: 3.5.0 > Subclass aware DefaultStatefulStreamCodec.newInstance() implementation > -- > > Key: APEXCORE-525 > URL: https://issues.apache.org/jira/browse/APEXCORE-525 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Tushar Gosavi >Assignee: Tushar Gosavi >Priority: Minor > Fix For: 3.5.0 > > > Custom stream codec extended from DefaultStatefulStreamCodec but not > overriding newInstance method keep on using default > DefaultStatefulStreamCodec without any warning to end user, causing wrong > result in the application. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXCORE-148) Auto metric: metric aggregations test is failing
[ https://issues.apache.org/jira/browse/APEXCORE-148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688055#comment-15688055 ] Vlad Rozov commented on APEXCORE-148: - Any update on the JIRA? I suggest lowering the priority. > Auto metric: metric aggregations test is failing > > > Key: APEXCORE-148 > URL: https://issues.apache.org/jira/browse/APEXCORE-148 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Chandni Singh >Priority: Critical > > From the build log : > AutoMetricTest.testMetricsAggregations:279 progress expected:<2> but was:<1> -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (APEXCORE-251) Journal output stream is null error message
[
https://issues.apache.org/jira/browse/APEXCORE-251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vlad Rozov reassigned APEXCORE-251:
---
Assignee: Vlad Rozov
> Journal output stream is null error message
> ---
>
> Key: APEXCORE-251
> URL: https://issues.apache.org/jira/browse/APEXCORE-251
> Project: Apache Apex Core
> Issue Type: Task
>Reporter: Chetan Narsude
>Assignee: Vlad Rozov
>
> A simple checkout and test run prints that message a gazillion times as
> WARN.
> {code}
> 2015-11-07 07:30:05,047 [master] WARN stram.Journal write - Journal output
> stream is null. Skipping write to the WAL.
> {code}
> Looks like it could be a debug message or something. Make it less verbose.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (APEXCORE-557) Upgrade netlet dependency to 1.3.0
[ https://issues.apache.org/jira/browse/APEXCORE-557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov resolved APEXCORE-557. - Resolution: Fixed Fix Version/s: 3.5.0 > Upgrade netlet dependency to 1.3.0 > -- > > Key: APEXCORE-557 > URL: https://issues.apache.org/jira/browse/APEXCORE-557 > Project: Apache Apex Core > Issue Type: Task >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Trivial > Fix For: 3.5.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (APEXCORE-510) Enforce DefaultOutputPort.emit() or Sink.put() thread affinity
[ https://issues.apache.org/jira/browse/APEXCORE-510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-510: Fix Version/s: 3.5.0 > Enforce DefaultOutputPort.emit() or Sink.put() thread affinity > -- > > Key: APEXCORE-510 > URL: https://issues.apache.org/jira/browse/APEXCORE-510 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Vlad Rozov >Assignee: Sanjay M Pujare > Fix For: 3.5.0 > > > Apex platform assumes that an operator interacts with the platform using the > dedicated operator thread. Currently, operators may create worker threads and > emit tuples from a worker thread. This leads to undefined behavior and hard > to find bugs, so it should be possible to enforce that > DefaultOutputPort.emit() and/or Sink.put() are called on the operator thread. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (APEXCORE-519) Add support for DIGEST enabled hadoop web services environment
[ https://issues.apache.org/jira/browse/APEXCORE-519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-519: Fix Version/s: 3.5.0 > Add support for DIGEST enabled hadoop web services environment > -- > > Key: APEXCORE-519 > URL: https://issues.apache.org/jira/browse/APEXCORE-519 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Pramod Immaneni >Assignee: Pramod Immaneni > Fix For: 3.5.0 > > > Add support for Apex to work with DIGEST enabled Hadoop web services. Work > would have common elements with BASIC authentication, APEXCORE-517 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (APEXCORE-505) setup and activate calls in operator block heartbeat loop in container
[
https://issues.apache.org/jira/browse/APEXCORE-505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vlad Rozov updated APEXCORE-505:
Fix Version/s: 3.5.0
> setup and activate calls in operator block heartbeat loop in container
> --
>
> Key: APEXCORE-505
> URL: https://issues.apache.org/jira/browse/APEXCORE-505
> Project: Apache Apex Core
> Issue Type: Bug
>Reporter: Ashwin Chandra Putta
>Assignee: Sandesh
> Fix For: 3.5.0
>
>
> The setup and activate calls in the operators block the StreamingContainer
> heartbeat loop to send heartbeats to application master. As a result, if
> activation/setup takes more than the heartbeat timeout for any given operator
> within a container, the app master ends up killing the container for
> heartbeat timeout even though container is active.
> To test this: I created simple test application and added sleep for 40
> seconds in setup or activate calls. The application master shows the heart
> beat timeout message and kills the container with operator. Please find the
> stack trace on the container while it was active as follows:
> sleep in activate callback
> ===
> {code}
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode):
> "Attach Listener" daemon prio=10 tid=0x7f8060ac nid=0x6d2f waiting on
> condition [0x]
>java.lang.Thread.State: RUNNABLE
> "1/randomGenerator:RandomNumberGenerator" prio=10 tid=0x7f8060b9b800
> nid=0x65a2 waiting on condition [0x7f8050bc8000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at com.datatorrent.sample.MyOperator.activate(MyOperator.java:63)
> at com.datatorrent.stram.engine.Node.activate(Node.java:619)
> at
> com.datatorrent.stram.engine.GenericNode.activate(GenericNode.java:205)
> at
> com.datatorrent.stram.engine.StreamingContainer.setupNode(StreamingContainer.java:1336)
> at
> com.datatorrent.stram.engine.StreamingContainer.access$100(StreamingContainer.java:130)
> at
> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1396)
> "ProcessWideEventLoop" prio=10 tid=0x7f8060afb800 nid=0x658b runnable
> [0x7f8050cc9000]
>java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x0007d186a380> (a
> com.datatorrent.netlet.OptimizedEventLoop$SelectedSelectionKeySet)
> - locked <0x0007d185a4a0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0007d185a070> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> at
> com.datatorrent.netlet.OptimizedEventLoop.runEventLoop(OptimizedEventLoop.java:185)
> at
> com.datatorrent.netlet.OptimizedEventLoop.runEventLoop(OptimizedEventLoop.java:157)
> at
> com.datatorrent.netlet.DefaultEventLoop.run(DefaultEventLoop.java:156)
> at java.lang.Thread.run(Thread.java:745)
> "Dispatcher-0" daemon prio=10 tid=0x7f8060ae8000 nid=0x6586 waiting on
> condition [0x7f8050dca000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0007dc5fa118> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> net.engio.mbassy.bus.AbstractSyncAsyncMessageBus$1.run(AbstractSyncAsyncMessageBus.java:51)
> at java.lang.Thread.run(Thread.java:745)
> "org.apache.hadoop.hdfs.PeerCache@389ed39" daemon prio=10
> tid=0x7f8060abf800 nid=0x6577 waiting on condition [0x7f8050fcc000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:255)
> at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46)
> at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124)
> at java.lang.Thread.run(Thread.java:745)
> "IPC Parameter Sending Thread #0" daemon prio=10 tid=0x7f8060a2f800
> nid=0x6572 waiting on condition [0x7f80510cd000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at
[jira] [Updated] (APEXCORE-476) Window Id calculation in ToString method in SubscribeRequestTuple & GenericRequestTuple is wrong
[ https://issues.apache.org/jira/browse/APEXCORE-476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-476: Fix Version/s: 3.5.0 > Window Id calculation in ToString method in SubscribeRequestTuple & > GenericRequestTuple is wrong > > > Key: APEXCORE-476 > URL: https://issues.apache.org/jira/browse/APEXCORE-476 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Sandesh >Assignee: Sandesh >Priority: Minor > Fix For: 3.5.0 > > > Window Id calculation should be > (long)basecond << 32 | windowId, instead it is (long)basesecond | windowdId -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (APEXCORE-557) Upgrade netlet dependency to 1.3.0
Vlad Rozov created APEXCORE-557: --- Summary: Upgrade netlet dependency to 1.3.0 Key: APEXCORE-557 URL: https://issues.apache.org/jira/browse/APEXCORE-557 Project: Apache Apex Core Issue Type: Task Reporter: Vlad Rozov Assignee: Vlad Rozov Priority: Trivial -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (APEXCORE-476) Window Id calculation in ToString method in SubscribeRequestTuple & GenericRequestTuple is wrong
[ https://issues.apache.org/jira/browse/APEXCORE-476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov resolved APEXCORE-476. - Resolution: Fixed > Window Id calculation in ToString method in SubscribeRequestTuple & > GenericRequestTuple is wrong > > > Key: APEXCORE-476 > URL: https://issues.apache.org/jira/browse/APEXCORE-476 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Sandesh >Assignee: Sandesh >Priority: Minor > > Window Id calculation should be > (long)basecond << 32 | windowId, instead it is (long)basesecond | windowdId -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (APEXCORE-583) Buffer Server LogicalNode should not be reused by Subscribers
[ https://issues.apache.org/jira/browse/APEXCORE-583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov resolved APEXCORE-583. - Resolution: Fixed Fix Version/s: 3.6.0 > Buffer Server LogicalNode should not be reused by Subscribers > - > > Key: APEXCORE-583 > URL: https://issues.apache.org/jira/browse/APEXCORE-583 > Project: Apache Apex Core > Issue Type: Bug > Components: Buffer Server >Reporter: Vlad Rozov >Assignee: Vlad Rozov > Fix For: 3.6.0 > > > Currently LogicalNode supports only one Subscriber and it can't be reused by > a new Subscriber. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXCORE-559) Support quit which should behave the same as "exit"
[ https://issues.apache.org/jira/browse/APEXCORE-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15726935#comment-15726935 ] Vlad Rozov commented on APEXCORE-559: - I think it is not necessary to initialize ~/.dt/clirc with the support for quit. It's user preference to add such support. Apex CLI supports ctrl-D, that is 4-times shorter than exit or quit :) > Support quit which should behave the same as "exit" > --- > > Key: APEXCORE-559 > URL: https://issues.apache.org/jira/browse/APEXCORE-559 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Sanjay M Pujare >Priority: Trivial > > Most CLIs accept both quit and exit and we should enhance the Apex CLI to > accept "quit" which should have the same semantics as "exit". > apex> quit > Invalid command 'quit'. Type "help" for list of commands -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (APEXCORE-591) SubscribeRequestTuple has wrong buffer size when mask is zero
Vlad Rozov created APEXCORE-591: --- Summary: SubscribeRequestTuple has wrong buffer size when mask is zero Key: APEXCORE-591 URL: https://issues.apache.org/jira/browse/APEXCORE-591 Project: Apache Apex Core Issue Type: Bug Components: Buffer Server Reporter: Vlad Rozov Assignee: Vlad Rozov Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXCORE-587) Allow access to DAG through the Launcher API
[ https://issues.apache.org/jira/browse/APEXCORE-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730893#comment-15730893 ] Vlad Rozov commented on APEXCORE-587: - I agree, it should be OK to move getDAG() to the application handle returned by the launchApp() API. It may be possible to use prepareDAG() to get newly created DAG, but it does need to support preparing more than one DAG/Application. I would also consider using a different class to provide such functionality in case DAG creation/preparing does not depend on the Launcher implementation. Currently, there is only one class that implements DAG, and delegating DAG creation to different Launcher implementations is not necessary. > Allow access to DAG through the Launcher API > > > Key: APEXCORE-587 > URL: https://issues.apache.org/jira/browse/APEXCORE-587 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Thomas Weise >Priority: Minor > > It is useful for clients to access the DAG without launching it. The existing > methods from EmbeddedLauncher (getDAG, prepareDAG) should be available > through the parent class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (APEXCORE-585) Latency should be calculated only after the first window has been complete
[ https://issues.apache.org/jira/browse/APEXCORE-585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-585: Priority: Minor (was: Major) > Latency should be calculated only after the first window has been complete > -- > > Key: APEXCORE-585 > URL: https://issues.apache.org/jira/browse/APEXCORE-585 > Project: Apache Apex Core > Issue Type: Bug >Reporter: David Yan >Assignee: David Yan >Priority: Minor > Fix For: 3.6.0 > > > From the latency unit test, it tries to calculate the latency by looking at > the difference between current window id and upstream's current window id > when the end window stats is not available. But when the current window id is > 0 (first window has not been complete), it would return some wacky latency > value because the current window id is 0. > This would not be an issue if the window id was not a simple sequence. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (APEXCORE-589) Validate health of MiniYARNCluster in StramMiniClusterTest
Vlad Rozov created APEXCORE-589: --- Summary: Validate health of MiniYARNCluster in StramMiniClusterTest Key: APEXCORE-589 URL: https://issues.apache.org/jira/browse/APEXCORE-589 Project: Apache Apex Core Issue Type: Improvement Reporter: Vlad Rozov Priority: Minor In case MiniYARNCluster is started on a node with the limited disk space (more than 90% which is the default value for yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage), it will start and be in STARTED state, but NodeManager will go into UNHEALTHY state and there will be no available NodeManagers for the Yarn mini cluster. This leads to obscure test failures with a hard to identify reason for the problem. It should be possible to check NM and RM health when Yarn mini cluster starts and fails the test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (APEXCORE-598) Embedded mode execution does not use APPLICATION_PATH for checkpointing
[ https://issues.apache.org/jira/browse/APEXCORE-598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov resolved APEXCORE-598. - Resolution: Fixed Fix Version/s: 3.6.0 > Embedded mode execution does not use APPLICATION_PATH for checkpointing > --- > > Key: APEXCORE-598 > URL: https://issues.apache.org/jira/browse/APEXCORE-598 > Project: Apache Apex Core > Issue Type: Bug >Affects Versions: 3.5.0 >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Minor > Fix For: 3.6.0 > > > Checkpoints are written to a default location, and when tests execute in > parallel they will interfere with each other. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (APEXCORE-598) Embedded mode execution does not use APPLICATION_PATH for checkpointing
[ https://issues.apache.org/jira/browse/APEXCORE-598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-598: Priority: Minor (was: Major) > Embedded mode execution does not use APPLICATION_PATH for checkpointing > --- > > Key: APEXCORE-598 > URL: https://issues.apache.org/jira/browse/APEXCORE-598 > Project: Apache Apex Core > Issue Type: Bug >Affects Versions: 3.5.0 >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Minor > Fix For: 3.6.0 > > > Checkpoints are written to a default location, and when tests execute in > parallel they will interfere with each other. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (APEXCORE-583) Buffer Server LogicalNode should not be reused by Subscribers
[ https://issues.apache.org/jira/browse/APEXCORE-583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-583: Assignee: Vlad Rozov > Buffer Server LogicalNode should not be reused by Subscribers > - > > Key: APEXCORE-583 > URL: https://issues.apache.org/jira/browse/APEXCORE-583 > Project: Apache Apex Core > Issue Type: Bug > Components: Buffer Server >Reporter: Vlad Rozov >Assignee: Vlad Rozov > > Currently LogicalNode supports only one Subscriber and it can't be reused by > a new Subscriber. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (APEXCORE-583) Buffer Server LogicalNode should not be reused by Subscribers
Vlad Rozov created APEXCORE-583: --- Summary: Buffer Server LogicalNode should not be reused by Subscribers Key: APEXCORE-583 URL: https://issues.apache.org/jira/browse/APEXCORE-583 Project: Apache Apex Core Issue Type: Bug Components: Buffer Server Reporter: Vlad Rozov Currently LogicalNode supports only one Subscriber and it can't be reused by a new Subscriber. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXCORE-581) Delivery of Custom Control Tuples
[ https://issues.apache.org/jira/browse/APEXCORE-581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716101#comment-15716101 ] Vlad Rozov commented on APEXCORE-581: - Agree, GenericNode needs to orchestrate tuples the same way what it currently does. The same applies to the Buffer Server. I foresee code changes both to GenericNode and to the Buffer Server needed to support custom control tuples. > Delivery of Custom Control Tuples > - > > Key: APEXCORE-581 > URL: https://issues.apache.org/jira/browse/APEXCORE-581 > Project: Apache Apex Core > Issue Type: Sub-task >Reporter: David Yan > > The behavior should be as follow: > - The control tuples should only be sent to downstream at streaming window > boundaries > - The control tuples should be sent to all partitions downstream > - The control tuples should be sent in the same order of arrival. > - Within a streaming window, do not send the same control tuple twice, even > if the same control tuple is received multiple times within that window. This > is possible if the operator has two input ports. (The LinkedHashMap should be > easily able to ensure both order and uniqueness.) > - The delivery of control tuples needs to stop at DelayOperator. > - When a streaming window is committed, remove the associated LinkedHashMap > that belong to windows with IDs that are less than the committed window > - It's safe to assume the control tuples are rare enough and can fit in memory > This will involve an additional MessageType to represent a custom control > tuple. > We probably need to have a data structure (possibly a LinkedHashMap) per > streaming window that stores the control tuple in the buffer server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXCORE-592) Returning description field in defaultProperties during apex cli call get-app-package-info
[ https://issues.apache.org/jira/browse/APEXCORE-592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15805158#comment-15805158 ] Vlad Rozov commented on APEXCORE-592: - [~ashwinchandrap] Attributes are not application/operator/port specific. They are defined by the Apex platform. A proprietary GUI like DataTorrent gateway UI can provide tooltips for attributes without additional Apex API. Attributes description specified in a property.xml, should be ignored. In the future, attributes defined by the Apex platform may be extended to include additional meta information, but this is beyond this JIRA. [~ajaygit158] The output relies on an application designer to provide properties.xml file that has operators properties description, while operator properties are defined by an operator designer. It may be OK as a short term, but IMO, it will be better to have a solution where operator designer provides the info. > Returning description field in defaultProperties during apex cli call > get-app-package-info > -- > > Key: APEXCORE-592 > URL: https://issues.apache.org/jira/browse/APEXCORE-592 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Ajay Gupta >Priority: Minor > Original Estimate: 24h > Remaining Estimate: 24h > > Currently for an operator property, only the value field is returned. The > operator property can have a description describing the property. This task > is created to return this filed for that operator property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (APEXCORE-504) Possible race condition in StreamingContainerAgent.getStreamCodec()
[
https://issues.apache.org/jira/browse/APEXCORE-504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vlad Rozov updated APEXCORE-504:
Assignee: Vlad Rozov
> Possible race condition in StreamingContainerAgent.getStreamCodec()
> ---
>
> Key: APEXCORE-504
> URL: https://issues.apache.org/jira/browse/APEXCORE-504
> Project: Apache Apex Core
> Issue Type: Bug
>Affects Versions: 3.3.0, 3.2.1, 3.4.0
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>
> StreamingContainerAgent.getStreamCodec() may be called from multiple RPC
> heartbeat processing threads for the same inputPortMeta (for an input port of
> a partitioned operator) that leads to a race condition in the following code:
> {noformat}
> StreamCodec codec = inputPortMeta.getValue(PortContext.STREAM_CODEC);
> if (codec == null) {
> // it cannot be this object that gets returned. Depending on this
> value is dangerous
> codec = inputPortMeta.getPortObject().getStreamCodec();
> if (codec != null) {
> // don't create codec multiple times - it will assign a new
> identifier
> inputPortMeta.getAttributes().put(PortContext.STREAM_CODEC, codec);
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (APEXCORE-605) Suppress bootstrap compiler warning
Vlad Rozov created APEXCORE-605:
---
Summary: Suppress bootstrap compiler warning
Key: APEXCORE-605
URL: https://issues.apache.org/jira/browse/APEXCORE-605
Project: Apache Apex Core
Issue Type: Improvement
Reporter: Vlad Rozov
Assignee: Vlad Rozov
Priority: Minor
Apex uses animal-sniffer-maven-plugin to enforce 1.7 API compatibility. The
following warning caused by {{1.7}} compiler plugin option
should be suppressed.
{noformat}
[WARNING] bootstrap class path not set in conjunction with -source 1.7
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (APEXCORE-592) Returning description field in defaultProperties during apex cli call get-app-package-info
[ https://issues.apache.org/jira/browse/APEXCORE-592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15761614#comment-15761614 ] Vlad Rozov commented on APEXCORE-592: - QUEUE_CAPACITY is an attribute and not a property of the port. Attributes are defined by the Apex platform and do not change across ports. What is the significance of providing the description for attributes? > Returning description field in defaultProperties during apex cli call > get-app-package-info > -- > > Key: APEXCORE-592 > URL: https://issues.apache.org/jira/browse/APEXCORE-592 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Ajay Gupta > Original Estimate: 24h > Remaining Estimate: 24h > > Currently for an operator property, only the value field is returned. The > operator property can have a description describing the property. This task > is created to return this filed for that operator property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (APEXCORE-572) Remove dependency on hadoop-common test.jar
[ https://issues.apache.org/jira/browse/APEXCORE-572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov resolved APEXCORE-572. - Resolution: Fixed Fix Version/s: 3.6.0 > Remove dependency on hadoop-common test.jar > --- > > Key: APEXCORE-572 > URL: https://issues.apache.org/jira/browse/APEXCORE-572 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Minor > Fix For: 3.6.0 > > > The dependency on hadoop-common test-jar can be avoided by implementing a few > helper methods. It is not necessary to have such dependency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (APEXCORE-573) Need to return activation window id for physical operators in StramWebServices
[ https://issues.apache.org/jira/browse/APEXCORE-573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-573: Priority: Minor (was: Major) > Need to return activation window id for physical operators in StramWebServices > -- > > Key: APEXCORE-573 > URL: https://issues.apache.org/jira/browse/APEXCORE-573 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Sanjay M Pujare >Assignee: Sanjay M Pujare >Priority: Minor > > It will be useful to return to the caller the window id the physical operator > (re)started from in > com.datatorrent.stram.webapp.StramWebServices.getOperatorInfo(int) since this > information could be useful in debugging the application. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXCORE-573) Need to return activation window id for physical operators in StramWebServices
[ https://issues.apache.org/jira/browse/APEXCORE-573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781261#comment-15781261 ] Vlad Rozov commented on APEXCORE-573: - I don't see an immediate need to fix this JIRA. It should stay open till somebody picks it up. > Need to return activation window id for physical operators in StramWebServices > -- > > Key: APEXCORE-573 > URL: https://issues.apache.org/jira/browse/APEXCORE-573 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Sanjay M Pujare >Assignee: Sanjay M Pujare >Priority: Minor > > It will be useful to return to the caller the window id the physical operator > (re)started from in > com.datatorrent.stram.webapp.StramWebServices.getOperatorInfo(int) since this > information could be useful in debugging the application. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (APEXCORE-592) Returning description field in defaultProperties during apex cli call get-app-package-info
[ https://issues.apache.org/jira/browse/APEXCORE-592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-592: Priority: Minor (was: Major) > Returning description field in defaultProperties during apex cli call > get-app-package-info > -- > > Key: APEXCORE-592 > URL: https://issues.apache.org/jira/browse/APEXCORE-592 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Ajay Gupta >Priority: Minor > Original Estimate: 24h > Remaining Estimate: 24h > > Currently for an operator property, only the value field is returned. The > operator property can have a description describing the property. This task > is created to return this filed for that operator property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXCORE-559) Support quit which should behave the same as "exit"
[
https://issues.apache.org/jira/browse/APEXCORE-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15726860#comment-15726860
]
Vlad Rozov commented on APEXCORE-559:
-
As apex supports aliases, it is quite easy to add "quit" as the alias for
"exit" by adding {noformat}alias quit exit{noformat} to clirc in the ~/.dt.
IMO, It will be better to document how this should be done and not to hardcode
"quit".
> Support quit which should behave the same as "exit"
> ---
>
> Key: APEXCORE-559
> URL: https://issues.apache.org/jira/browse/APEXCORE-559
> Project: Apache Apex Core
> Issue Type: Improvement
>Reporter: Sanjay M Pujare
>Priority: Trivial
>
> Most CLIs accept both quit and exit and we should enhance the Apex CLI to
> accept "quit" which should have the same semantics as "exit".
> apex> quit
> Invalid command 'quit'. Type "help" for list of commands
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (APEXCORE-587) Allow access to DAG through the Launcher API
[ https://issues.apache.org/jira/browse/APEXCORE-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730697#comment-15730697 ] Vlad Rozov commented on APEXCORE-587: - IMO, those methods don't belong to the Launcher or EmbeddedLauncher class. It should be possible to use the same Launcher instance to start more than one application and the current interface of the Launcher class fully support this. In contrast, getDAG() assumes that there is one to one relationship between DAG/LogicalPlan and Launcher. It will be better to have a different class that deals with DAG creation and cloning. prepareDAG() may still be part of the Launcher API, but it needs to change to support multiple applications. > Allow access to DAG through the Launcher API > > > Key: APEXCORE-587 > URL: https://issues.apache.org/jira/browse/APEXCORE-587 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Thomas Weise >Priority: Minor > > It is useful for clients to access the DAG without launching it. The existing > methods from EmbeddedLauncher (getDAG, prepareDAG) should be available > through the parent class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXCORE-587) Allow access to DAG through the Launcher API
[ https://issues.apache.org/jira/browse/APEXCORE-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731060#comment-15731060 ] Vlad Rozov commented on APEXCORE-587: - JIRA is not assigned :). > Allow access to DAG through the Launcher API > > > Key: APEXCORE-587 > URL: https://issues.apache.org/jira/browse/APEXCORE-587 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Thomas Weise >Priority: Minor > > It is useful for clients to access the DAG without launching it. The existing > methods from EmbeddedLauncher (getDAG, prepareDAG) should be available > through the parent class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXCORE-558) Do not use yellow color to display command strings in help output
[ https://issues.apache.org/jira/browse/APEXCORE-558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15726969#comment-15726969 ] Vlad Rozov commented on APEXCORE-558: - For the CLI help bold should be sufficient. > Do not use yellow color to display command strings in help output > - > > Key: APEXCORE-558 > URL: https://issues.apache.org/jira/browse/APEXCORE-558 > Project: Apache Apex Core > Issue Type: Bug > Environment: MacOS, Linux >Reporter: Sanjay M Pujare >Priority: Minor > > Apex CLI Help output (at least on MacOS and Linux terminals) shows command > strings in yellow which is extremely hard to read and reduces usability of > the CLI. Should be changed to red or purple or just be bolded in black. > In the following output "alias" and "begin-macro" are in yellow and hard to > read. > apex> help > GLOBAL COMMANDS EXCEPT WHEN CHANGING LOGICAL PLAN: > alias alias-name command > Create a command alias > begin-macro name > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXCORE-592) Returning description field in defaultProperties during apex cli call get-app-package-info
[ https://issues.apache.org/jira/browse/APEXCORE-592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15764552#comment-15764552 ] Vlad Rozov commented on APEXCORE-592: - It depends on requirements. I'd like to understand how attributes and properties description are populated and what is the significance of providing attributes description. > Returning description field in defaultProperties during apex cli call > get-app-package-info > -- > > Key: APEXCORE-592 > URL: https://issues.apache.org/jira/browse/APEXCORE-592 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Ajay Gupta >Priority: Minor > Original Estimate: 24h > Remaining Estimate: 24h > > Currently for an operator property, only the value field is returned. The > operator property can have a description describing the property. This task > is created to return this filed for that operator property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (APEXCORE-679) Build fails with hadoop 2.7.3 dependency
[ https://issues.apache.org/jira/browse/APEXCORE-679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-679: Priority: Minor (was: Critical) Issue Type: Improvement (was: Bug) Summary: Build fails with hadoop 2.7.3 dependency (was: Apex core 3.5.0 build failing with hadoop 2.7.3 dependency) As far as I can see this is a compile time only issue as all undefined symbols are static final constants and their definition did not change (only constant name were changed). At run-time, even when an application is started on hadoop-2.7.x cluster, the required values should be present in the yarn conf files. Lover priority as apex compiles against hadoop-2.6 by default. > Build fails with hadoop 2.7.3 dependency > - > > Key: APEXCORE-679 > URL: https://issues.apache.org/jira/browse/APEXCORE-679 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Chinmay Kolhatkar >Priority: Minor > > 1. I downloaded apache-apex-core-3.5.0-source-release.tar.gz > 2. Extracted the tar > 3. Ran following command to build apex core: > mvn clean package -DskipTests -Dhadoop.version=2.7.3 > (NOTE: I have overridden hadoop version to 2.7.3) > I get following compilation error: > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-compiler-plugin:3.3:compile (default-compile) > on project apex-engine: Compilation failure: Compilation failure: > [ERROR] > /home/chinmay/files/apache-apex-core-3.5.0/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[159,87] > cannot find symbol > [ERROR] symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT > [ERROR] location: class org.apache.hadoop.yarn.conf.YarnConfiguration > [ERROR] > /home/chinmay/files/apache-apex-core-3.5.0/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[586,120] > cannot find symbol > [ERROR] symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_KEY > [ERROR] location: class org.apache.hadoop.yarn.conf.YarnConfiguration > [ERROR] > /home/chinmay/files/apache-apex-core-3.5.0/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[586,173] > cannot find symbol > [ERROR] symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT > [ERROR] location: class org.apache.hadoop.yarn.conf.YarnConfiguration > [ERROR] -> [Help 1] > It should compile with 2.7.x version of hadoop. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-679) Build fails with hadoop 2.7.x dependency
[ https://issues.apache.org/jira/browse/APEXCORE-679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-679: Summary: Build fails with hadoop 2.7.x dependency (was: Build fails with hadoop 2.7.3 dependency) > Build fails with hadoop 2.7.x dependency > - > > Key: APEXCORE-679 > URL: https://issues.apache.org/jira/browse/APEXCORE-679 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Chinmay Kolhatkar >Priority: Minor > > 1. I downloaded apache-apex-core-3.5.0-source-release.tar.gz > 2. Extracted the tar > 3. Ran following command to build apex core: > mvn clean package -DskipTests -Dhadoop.version=2.7.3 > (NOTE: I have overridden hadoop version to 2.7.3) > I get following compilation error: > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-compiler-plugin:3.3:compile (default-compile) > on project apex-engine: Compilation failure: Compilation failure: > [ERROR] > /home/chinmay/files/apache-apex-core-3.5.0/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[159,87] > cannot find symbol > [ERROR] symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT > [ERROR] location: class org.apache.hadoop.yarn.conf.YarnConfiguration > [ERROR] > /home/chinmay/files/apache-apex-core-3.5.0/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[586,120] > cannot find symbol > [ERROR] symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_KEY > [ERROR] location: class org.apache.hadoop.yarn.conf.YarnConfiguration > [ERROR] > /home/chinmay/files/apache-apex-core-3.5.0/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[586,173] > cannot find symbol > [ERROR] symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT > [ERROR] location: class org.apache.hadoop.yarn.conf.YarnConfiguration > [ERROR] -> [Help 1] > It should compile with 2.7.x version of hadoop. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (APEXCORE-593) apex cli get-app-package-info could not retrieve properties defined in properties.xml
[ https://issues.apache.org/jira/browse/APEXCORE-593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov resolved APEXCORE-593. - Resolution: Fixed Fix Version/s: 3.6.0 > apex cli get-app-package-info could not retrieve properties defined in > properties.xml > - > > Key: APEXCORE-593 > URL: https://issues.apache.org/jira/browse/APEXCORE-593 > Project: Apache Apex Core > Issue Type: Bug >Affects Versions: 3.5.0 >Reporter: Yogi Devendra >Priority: Minor > Fix For: 3.6.0 > > > If application defines properties in properties.xml; such properties should > be available to populateDAG() method when invoked from processAppDirectory() > in get-app-package-info. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-656) Upgrade org.apache.httpcomponents.httpclient
[ https://issues.apache.org/jira/browse/APEXCORE-656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-656: Issue Type: Dependency upgrade (was: Improvement) > Upgrade org.apache.httpcomponents.httpclient > > > Key: APEXCORE-656 > URL: https://issues.apache.org/jira/browse/APEXCORE-656 > Project: Apache Apex Core > Issue Type: Dependency upgrade >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Trivial > Fix For: 3.6.0 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXMALHAR-2466) Upgrade org.couchbase.mock.CouchbaseMock
[
https://issues.apache.org/jira/browse/APEXMALHAR-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vlad Rozov updated APEXMALHAR-2466:
---
Labels: contrib easyfix newbie (was: )
> Upgrade org.couchbase.mock.CouchbaseMock
>
>
> Key: APEXMALHAR-2466
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2466
> Project: Apache Apex Malhar
> Issue Type: Dependency upgrade
>Reporter: Vlad Rozov
>Priority: Minor
> Labels: contrib, easyfix, newbie
>
> There must be no dependencies on SNAPSHOT versions, but contrib pom.xml
> depends on 0.8-SNAPSHOT version of org.couchbase.mock.CouchbaseMock
> {noformat}
>
> org.couchbase.mock
> CouchbaseMock
> 0.8-SNAPSHOT
> test
> jar
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (APEXMALHAR-2466) Upgrade org.couchbase.mock.CouchbaseMock
Vlad Rozov created APEXMALHAR-2466:
--
Summary: Upgrade org.couchbase.mock.CouchbaseMock
Key: APEXMALHAR-2466
URL: https://issues.apache.org/jira/browse/APEXMALHAR-2466
Project: Apache Apex Malhar
Issue Type: Dependency upgrade
Reporter: Vlad Rozov
Priority: Minor
There must be no dependencies on SNAPSHOT versions, but contrib pom.xml depends
on 0.8-SNAPSHOT version of org.couchbase.mock.CouchbaseMock
{noformat}
org.couchbase.mock
CouchbaseMock
0.8-SNAPSHOT
test
jar
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (APEXCORE-685) Update contributor guidelines
[ https://issues.apache.org/jira/browse/APEXCORE-685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15949307#comment-15949307 ] Vlad Rozov commented on APEXCORE-685: - https://lists.apache.org/thread.html/489d1141bb3351d5f1bcd2dc13118692869797a1cc81e88ff03fdb23@%3Cdev.apex.apache.org%3E > Update contributor guidelines > - > > Key: APEXCORE-685 > URL: https://issues.apache.org/jira/browse/APEXCORE-685 > Project: Apache Apex Core > Issue Type: Task > Components: Website >Reporter: Thomas Weise > > Additions that have been discussed on the mailing list. > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (APEXCORE-686) AppPackage fails when .apa contains zero length stored entry
Vlad Rozov created APEXCORE-686:
---
Summary: AppPackage fails when .apa contains zero length stored
entry
Key: APEXCORE-686
URL: https://issues.apache.org/jira/browse/APEXCORE-686
Project: Apache Apex Core
Issue Type: Bug
Reporter: Vlad Rozov
Assignee: Vlad Rozov
Fix For: 3.6.0
JarInputStream/ZipInputStream does not support valid formats of .jar/.zip file
{noformat}
java.util.zip.ZipException: only DEFLATED entries can have EXT descriptor
at java.util.zip.ZipInputStream.readLOC(ZipInputStream.java:309)
at java.util.zip.ZipInputStream.getNextEntry(ZipInputStream.java:121)
at java.util.jar.JarInputStream.(JarInputStream.java:83)
at java.util.jar.JarInputStream.(JarInputStream.java:62)
at com.datatorrent.stram.client.AppPackage.(AppPackage.java:215)
at com.datatorrent.stram.client.AppPackage.(AppPackage.java:196)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (APEXCORE-504) Possible race condition in StreamingContainerAgent.getStreamCodec()
[
https://issues.apache.org/jira/browse/APEXCORE-504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vlad Rozov resolved APEXCORE-504.
-
Resolution: Fixed
Fix Version/s: 3.4.1
3.6.0
3.2.2
3.3.1
> Possible race condition in StreamingContainerAgent.getStreamCodec()
> ---
>
> Key: APEXCORE-504
> URL: https://issues.apache.org/jira/browse/APEXCORE-504
> Project: Apache Apex Core
> Issue Type: Bug
>Affects Versions: 3.3.0, 3.2.1, 3.4.0, 3.5.0
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
> Fix For: 3.3.1, 3.2.2, 3.6.0, 3.4.1
>
>
> StreamingContainerAgent.getStreamCodec() may be called from multiple RPC
> heartbeat processing threads for the same inputPortMeta (for an input port of
> a partitioned operator) that leads to a race condition in the following code:
> {noformat}
> StreamCodec codec = inputPortMeta.getValue(PortContext.STREAM_CODEC);
> if (codec == null) {
> // it cannot be this object that gets returned. Depending on this
> value is dangerous
> codec = inputPortMeta.getPortObject().getStreamCodec();
> if (codec != null) {
> // don't create codec multiple times - it will assign a new
> identifier
> inputPortMeta.getAttributes().put(PortContext.STREAM_CODEC, codec);
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (APEXMALHAR-2461) Fix dependencies on libraries licensed under Category X
Vlad Rozov created APEXMALHAR-2461: -- Summary: Fix dependencies on libraries licensed under Category X Key: APEXMALHAR-2461 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2461 Project: Apache Apex Malhar Issue Type: Bug Reporter: Vlad Rozov Fix For: 3.8.0 There are few samples and the benchmark application in Malhar that depend on libraries licensed under Category X. All such dependencies need to be either optional, be replaced with libraries that are compatible with Apache license or be removed. Any newly introduced dependency should be either compatible with the Apache license or be optional. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-663) Application restart not working.
[ https://issues.apache.org/jira/browse/APEXCORE-663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-663: Priority: Major (was: Critical) > Application restart not working. > > > Key: APEXCORE-663 > URL: https://issues.apache.org/jira/browse/APEXCORE-663 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Tushar Gosavi >Assignee: Sandesh > Fix For: 3.6.0 > > > Application restart is not working. Steps to reproduce > - launch PiDemo > - Wait for some time (till recovery windowId reaches 120 >) > - kill the app > - Try to relaunch using 'launch -originalAppId' -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (APEXCORE-593) apex cli get-app-package-info could not retrieve properties defined in properties.xml
[ https://issues.apache.org/jira/browse/APEXCORE-593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov reassigned APEXCORE-593: --- Assignee: Vikram Patil > apex cli get-app-package-info could not retrieve properties defined in > properties.xml > - > > Key: APEXCORE-593 > URL: https://issues.apache.org/jira/browse/APEXCORE-593 > Project: Apache Apex Core > Issue Type: Bug >Affects Versions: 3.5.0 >Reporter: Yogi Devendra >Assignee: Vikram Patil >Priority: Minor > Fix For: 3.6.0 > > > If application defines properties in properties.xml; such properties should > be available to populateDAG() method when invoked from processAppDirectory() > in get-app-package-info. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (APEXCORE-683) Apex client should support application packages on HDFS
Vlad Rozov created APEXCORE-683:
---
Summary: Apex client should support application packages on HDFS
Key: APEXCORE-683
URL: https://issues.apache.org/jira/browse/APEXCORE-683
Project: Apache Apex Core
Issue Type: Improvement
Reporter: Vlad Rozov
Assignee: Vlad Rozov
apex client supports only local file system:
{noformat}
hdfs dfs -ls hdfs:///user/vrozov
Found 4 items
drwx-- - vrozov supergroup 0 2016-12-24 16:00
hdfs:///user/vrozov/.Trash
drwxrwxrwx - vrozov supergroup 0 2017-02-28 19:20
hdfs:///user/vrozov/.flink
-rw-r--r-- 3 vrozov supergroup 18862977 2017-03-25 09:20
hdfs:///user/vrozov/appPackage.apa
drwxrwxrwx - vrozov supergroup 0 2016-10-25 21:49
hdfs:///user/vrozov/datatorrent
[vrozov@node0 apex]$ apex
Apex CLI 3.6.0-SNAPSHOT 25.03.2017 @ 09:17:45 PDT rev: b4a4e05 branch: master
apex> get-app-package-info hdfs:///user/vrozov/appPackage.apa
java.io.FileNotFoundException: hdfs:/user/vrozov/appPackage.apa (No such file
or directory)
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.(ZipFile.java:215)
at java.util.zip.ZipFile.(ZipFile.java:145)
at java.util.jar.JarFile.(JarFile.java:154)
at java.util.jar.JarFile.(JarFile.java:118)
at com.datatorrent.stram.client.AppPackage.(AppPackage.java:185)
at com.datatorrent.stram.client.AppPackage.(AppPackage.java:258)
at
com.datatorrent.stram.cli.ApexCli.newAppPackageInstance(ApexCli.java:467)
at
com.datatorrent.stram.cli.ApexCli$GetAppPackageInfoCommand.execute(ApexCli.java:3502)
at com.datatorrent.stram.cli.ApexCli$3.run(ApexCli.java:1520)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (APEXCORE-687) Update docs, change supported Hadoop version to 2.6
[ https://issues.apache.org/jira/browse/APEXCORE-687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-687: Priority: Minor (was: Major) > Update docs, change supported Hadoop version to 2.6 > --- > > Key: APEXCORE-687 > URL: https://issues.apache.org/jira/browse/APEXCORE-687 > Project: Apache Apex Core > Issue Type: Documentation > Components: Documentation >Reporter: Dean Lockgaard >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-676) Show description for DefaultProperties only when user requests it
[ https://issues.apache.org/jira/browse/APEXCORE-676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-676: Priority: Minor (was: Major) Fix Version/s: 3.6.0 > Show description for DefaultProperties only when user requests it > - > > Key: APEXCORE-676 > URL: https://issues.apache.org/jira/browse/APEXCORE-676 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Sandesh >Assignee: Ajay Gupta >Priority: Minor > Fix For: 3.6.0 > > > Apex CLI shows description for default properties, this feature was added > recently as a part of > https://issues.apache.org/jira/browse/APEXCORE-592 > But description should be shown only when a user requests it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (APEXCORE-676) Show description for DefaultProperties only when user requests it
[ https://issues.apache.org/jira/browse/APEXCORE-676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov resolved APEXCORE-676. - Resolution: Fixed > Show description for DefaultProperties only when user requests it > - > > Key: APEXCORE-676 > URL: https://issues.apache.org/jira/browse/APEXCORE-676 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Sandesh >Assignee: Ajay Gupta >Priority: Minor > Fix For: 3.6.0 > > > Apex CLI shows description for default properties, this feature was added > recently as a part of > https://issues.apache.org/jira/browse/APEXCORE-592 > But description should be shown only when a user requests it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (APEXCORE-668) Remove CustomControlTuple type from Buffer Server
[ https://issues.apache.org/jira/browse/APEXCORE-668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15940617#comment-15940617 ] Vlad Rozov commented on APEXCORE-668: - Please prototype the proposed solution and benchmark it. Publish the results to the JIRA. > Remove CustomControlTuple type from Buffer Server > - > > Key: APEXCORE-668 > URL: https://issues.apache.org/jira/browse/APEXCORE-668 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Bhupesh Chawda >Assignee: Pramod Immaneni > > Following argument made in favour of removing CustomControlTuple type from > Buffer server: > Regarding creating custom control tuple as a separate type in bufferserver: I > don't think that should be the case since currently this tuple is a > passthrough for bufferserver just like any data tuple and there is no action > bufferserver takes for this tuple. So why should it be a special type for > bufferserver? Window markers are different because bufferserver uses that > knowledge. As far as bufferserver is concerned this is just data and the > differentiation should be left to the payload. > This might mean adding a byte to the data payload to differentiate between > regular data or control tuple which either side, publisher and subscriber > understand, but bufferserver doesn't really care about it. > There is a performance argument to be made that an extra byte can be saved by > overloading the buffer server message type, but then we should support it > more explicitly. For example, you could say that the data tuple type has a > message type range from 0xf0-0xff, where the first four bits being all 1's > denotes it's a data tuple and the last four bits can be configured by the > publisher/subscriber to sub-differentiate the data type. This is just an > example, you could go with lesser number of bits. But the point is buffer > server will only check the first four bits in this case to determine it is a > data tuple and deal with it accordingly, there will be no control tuple > definition as far as bufferserver is concerned. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (APEXCORE-680) Review container heartbeat timeout log level messages
Vlad Rozov created APEXCORE-680: --- Summary: Review container heartbeat timeout log level messages Key: APEXCORE-680 URL: https://issues.apache.org/jira/browse/APEXCORE-680 Project: Apache Apex Core Issue Type: Sub-task Reporter: Vlad Rozov Assignee: Hitesh Kapoor Priority: Trivial -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (APEXCORE-662) Raise StramEvent for heartbeat miss
[ https://issues.apache.org/jira/browse/APEXCORE-662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov resolved APEXCORE-662. - Resolution: Fixed Fix Version/s: 3.6.0 > Raise StramEvent for heartbeat miss > --- > > Key: APEXCORE-662 > URL: https://issues.apache.org/jira/browse/APEXCORE-662 > Project: Apache Apex Core > Issue Type: Task >Reporter: Tushar Gosavi >Assignee: Hitesh Kapoor >Priority: Minor > Fix For: 3.6.0 > > > One of the cause of container kill is heartbeat miss. we can raise an > StramEvent in case of heartbeat miss to make debugging easier. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
