[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...
GitHub user pvillard31 opened a pull request: https://github.com/apache/nifi/pull/347 NIFI-1755 Fixed remote process group status counts by only considering connected remote ports You can merge this pull request into a Git repository by running: $ git pull https://github.com/pvillard31/nifi NIFI-1755 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/347.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #347 commit 147f528eb8e5e7a4da5f6e49c5f16657227d5e58 Author: Pierre Villard Date: 2016-04-13T09:31:23Z NIFI-1755 Fixed remote process group status counts by only considering connected remote ports --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...
Github user olegz commented on the pull request: https://github.com/apache/nifi/pull/347#issuecomment-211126901 @pvillard31 Changes look good, but was wondering if you could add some tests --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...
Github user pvillard31 commented on the pull request: https://github.com/apache/nifi/pull/347#issuecomment-211283811 @olegz I agree a unit test would be nice, but I must admit that I am not sure to see how to do it. It involves RPG, remote ports, remote processors and I don't see how I can mock the whole remote flow to get the expected testing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...
Github user olegz commented on the pull request: https://github.com/apache/nifi/pull/347#issuecomment-211324090 @pvillard31 right, not the simplest thing to do, but let me dig as I believe I have something. Will let you know --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...
Github user olegz commented on the pull request:
https://github.com/apache/nifi/pull/347#issuecomment-211417082
@pvillard31 I was gonna play with it a bit more and possibly create the
test, but getting tied up with other things. Yet I was able to dig up one of my
experiments that I used to play with RPGs in unit test environment, and since
you already familiar with FlowController, you may make sense of it. Anyway, let
me know.
```java
System.setProperty("nifi.properties.file.path",
"/Users/ozhurakousky/dev/nifi-integration/src/main/resources/nifi.properties");
NiFiProperties properties = NiFiProperties.getInstance();
RingBufferEventRepository repository = new
RingBufferEventRepository(1);
UserService us = mock(UserService.class);
AuditService as = mock(AuditService.class);
FlowController controller =
FlowController.createStandaloneInstance(repository, properties, us, as, null);
ProcessGroup senderGroup = controller.createProcessGroup("SENDER");
senderGroup.setName("SENDER");
RemoteProcessGroup remoteProcessGroup =
controller.createRemoteProcessGroup("SENDER_REMOTE",
"http://localhost:8080/nifi";);
senderGroup.addRemoteProcessGroup(remoteProcessGroup);
// INPUT PORT (DESTINATIOIN)
StandardRemoteProcessGroupPortDescriptor inputPortDescriptor = new
StandardRemoteProcessGroupPortDescriptor();
inputPortDescriptor.setId("67b1950c-554e-4cb6-92fb-7e313b015e8c");
inputPortDescriptor.setName("in");
remoteProcessGroup.setInputPorts(Collections.
singleton(inputPortDescriptor));
RemoteGroupPort inputPort =
remoteProcessGroup.getInputPort("67b1950c-554e-4cb6-92fb-7e313b015e8c");
inputPort.setProcessGroup(senderGroup);
System.out.println(inputPort.isRunning());
System.out.println(inputPort.isTargetRunning());
System.out.println(inputPort.isValid());
// SOURCE
ProcessorNode source =
controller.createProcessor(GenerateFlowFile.class.getName(), "A");
source.setProperty("File Size", "10 b");
source.setScheduldingPeriod("2 sec");
source.setProcessGroup(senderGroup);
senderGroup.addProcessor(source);
// CONNECTION
Connection connection =
controller.createConnection("MyConnectionId", "MyConnectionName", source,
remoteProcessGroup.getInputPort("67b1950c-554e-4cb6-92fb-7e313b015e8c"),
Collections.singletonList("success"));
connection.setProcessGroup(senderGroup);
senderGroup.addConnection(connection);
inputPort.addConnection(connection);
remoteProcessGroup.startTransmitting();
// inputPort.verifyCanStart();
// senderGroup.addInputPort(inputPort);
// senderGroup.startInputPort(inputPort);
//controller.startTransmitting(remoteProcessGroup.getInputPort("in"));
senderGroup.startProcessor(source);
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...
Github user pvillard31 commented on the pull request:
https://github.com/apache/nifi/pull/347#issuecomment-212876061
@olegz Thanks for the code. However if I try to run it, it will try to
connect to localhost:8080/nifi and will fail:
594 [NiFi Site-to-Site Connection Pool Maintenance] WARN
org.apache.nifi.remote.client.socket.EndpointConnectionPool -
EndpointConnectionPool[Cluster URL=http://localhost:8080/nifi] Unable to
refresh Remote Group's peers due to {}
java.net.ConnectException: Connection refused
It makes sense since there is no NiFi instance running. Did you have in
mind to really have an instance running or to simulate the remote end in some
ways?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...
Github user olegz commented on the pull request: https://github.com/apache/nifi/pull/347#issuecomment-212995061 @pvillard31 let me dig some more. Yes you have to have the "other side" running but I had some other code bits to simulate that. I'll post here once I find --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...
Github user olegz commented on the pull request: https://github.com/apache/nifi/pull/347#issuecomment-213020240 @pvillard31 Actually I totally forgot about the effort I started a while back. There is actually a branch for it https://github.com/olegz/nifi/tree/int-test/nifi-integration-tests. Not much there but S2S test is (totally headless). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...
Github user olegz commented on the pull request: https://github.com/apache/nifi/pull/347#issuecomment-216612204 @pvillard31 Are you planning to add tests for this? If not I can take a crack at it. LMK. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...
Github user pvillard31 commented on the pull request: https://github.com/apache/nifi/pull/347#issuecomment-216631894 @olegz I must admit I tried to use your code as mentioned in your link and got it working but I didn't find a way to validate the change since it seems to simulate S2S without all the code around about statistics and so on. Everything is quite "nested" and I didn't find a way to get the expected result. (Speaking of that, I saw a new JIRA NIFI-1820 (in addition to NIFI-1331) to address this part). If you want to have a look, that would be great (and I am sure I'll learn a lot!). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...
Github user olegz commented on the pull request: https://github.com/apache/nifi/pull/347#issuecomment-216635676 Sure, I'll give it a shot --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...
Github user olegz commented on the pull request:
https://github.com/apache/nifi/pull/347#issuecomment-217000289
@pvillard31 below is the test you can use to validate this issue. Basically
if you execute it on current master it will fail the assertions and then with
your changes it will succeed.
Also keep in mind that you would need a second PR (one as an update to this
one and another for 0.x branch that is essentially 0.7.0-SNAPSHOT). The reason
being is that _UserService_ was migrated to _KeyService_ so the test below
won't compile on 0.x where you still have to use UserService. I am sure you'll
figure it out. Also, feel free to change the test class/method names to be more
reflective what's being tested.
Cheers
```java
package org.apache.nifi.controller;
import static org.junit.Assert.assertEquals;
import static org.mockito.Mockito.mock;
import java.net.URL;
import java.util.Collections;
import org.apache.nifi.admin.service.AuditService;
import org.apache.nifi.admin.service.KeyService;
import org.apache.nifi.controller.repository.RepositoryStatusReport;
import org.apache.nifi.controller.repository.RingBufferEventRepository;
import org.apache.nifi.controller.repository.StandardFlowFileEvent;
import org.apache.nifi.controller.repository.StandardRepositoryStatusReport;
import org.apache.nifi.controller.status.ProcessGroupStatus;
import org.apache.nifi.groups.ProcessGroup;
import org.apache.nifi.groups.RemoteProcessGroup;
import org.apache.nifi.groups.RemoteProcessGroupPortDescriptor;
import org.apache.nifi.provenance.MockProvenanceEventRepository;
import org.apache.nifi.remote.RemoteGroupPort;
import org.apache.nifi.remote.StandardRemoteProcessGroupPortDescriptor;
import org.apache.nifi.util.NiFiProperties;
import org.junit.After;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Test;
public class RPGStatusTest {
private final KeyService userService = mock(KeyService.class);
private final AuditService auditService = mock(AuditService.class);
private volatile FlowController controller;
@BeforeClass
public static void beforeClass() {
try {
URL url =
ClassLoader.getSystemClassLoader().getResource("nifi.properties");
System.setProperty("nifi.properties.file.path", url.getFile());
} catch (Exception e) {
throw new IllegalStateException("Failed to discover
nifi.properties at the root of the classpath", e);
}
}
@Before
public void before() {
NiFiProperties properties = NiFiProperties.getInstance();
properties.setProperty(NiFiProperties.PROVENANCE_REPO_IMPLEMENTATION_CLASS,
MockProvenanceEventRepository.class.getName());
properties.setProperty(NiFiProperties.STATE_MANAGEMENT_CONFIG_FILE,
"src/test/resources/state-management.xml");
properties.setProperty(NiFiProperties.STATE_MANAGEMENT_LOCAL_PROVIDER_ID,
"local-provider");
properties.setProperty(NiFiProperties.REMOTE_INPUT_HOST,
"localhost");
properties.setProperty("nifi.remote.input.secure", "false");
RingBufferEventRepository repository = new
RingBufferEventRepository(1);
this.controller =
FlowController.createStandaloneInstance(repository,
NiFiProperties.getInstance(),
this.userService, this.auditService, null);
}
@After
public void after() {
this.controller.shutdown(false);
}
@Test
public void testPr347() throws Exception {
ProcessGroup receiverGroup = controller.createProcessGroup("SITE");
NiFiTestUtils.setControllerRootGroup(this.controller,
receiverGroup);
RemoteProcessGroup remoteProcessGroup =
controller.createRemoteProcessGroup("SENDER_REMOTE","http://foo:1234/nifi";);
receiverGroup.addRemoteProcessGroup(remoteProcessGroup);
String inputPortId = "inputId";
StandardRemoteProcessGroupPortDescriptor inputPortDescriptor = new
StandardRemoteProcessGroupPortDescriptor();
inputPortDescriptor.setId(inputPortId);
inputPortDescriptor.setName("inputPort");
remoteProcessGroup.setInputPorts(Collections.
singleton(inputPortDescriptor));
RemoteGroupPort inputPort =
remoteProcessGroup.getInputPort(inputPortId);
inputPort.setProcessGroup(receiverGroup);
String outputPortId = "outputId";
StandardRemoteProcessGroupPortDescriptor outputPortDescriptor = new
StandardRemoteProcessGroupPortDescriptor();
outputPortDescriptor.setId(outputPortId);
outputPortDescriptor.setName("outputPport");
remoteProcessGroup.setOutputPorts(Collections.
s
[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...
Github user pvillard31 commented on the pull request: https://github.com/apache/nifi/pull/347#issuecomment-218867090 @olegz Thanks Oleg! Not exactly what I had in mind but it does the trick :) (I was trying to set up an environment with two simple processors sending flow files to RPG with different frequencies and ensure the correct values of counters in RPG status). This PR is updated for master branch and I just sent a PR for 0.x branch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...
Github user olegz commented on the pull request: https://github.com/apache/nifi/pull/347#issuecomment-219489958 merging --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1755 Fixed remote process group status cou...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/347 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
