Re: [DISCUSS] On HBase client retries (NIFI-6197)

[Josh Elser]

So, do we have a real issue here? Or are we just overlapping two 
discussions about the same code-change with different context?


My impression from Bryan is, now that he has the bigger context of why 
implicit retries are important for HBase, that he is not against setting 
this value higher (although, he also originally said this [1]), but just 
not the 1.x default of 35 retries (which *I* will say is too high). Am I 
reading too much into what you've said, Bryan?


I understand that folks are busy and this is probably not the highest 
priority. I'm just trying to figure out if we're actually still in 
disagreement. My gut says that we're not.


Re-stating goals: I think everyone is trying to make sure that the user 
of NiFi with HBase in the mix has a great user experience OOTB. That 
requires finding a balance between "fail fast when something is 
outwardly messed up" and "don't fail up when HBase can implicitly ride 
over a transient/automatically-recoverable state".


[1] https://github.com/apache/nifi/pull/3425#issuecomment-487962996

On 2/19/20 11:14 AM, Lars Francke wrote:

That's what I suggested[1] and Bryan rejected[2]

[1] <https://github.com/apache/nifi/pull/3425#issuecomment-482711295>
[2] <https://github.com/apache/nifi/pull/3425#issuecomment-491984116>


On Wed, Feb 19, 2020 at 4:29 PM Josh Elser  wrote:


Thanks for sharing the code, Bryan! I lazily did not go digging.

Looking into #onEnabled, we could change this to create its own
Connection with a very low 1 or 2 retries with a very short retry
duration. Throw that one away after we did our sanity check, set a high
retry amount, and then create a new Connection for all of the
puts/gets/scans/whatever the service will do.

I think NiFi, in general, can err on the side of "lower" client retries
(both in number and duration of backoff) than a normal client since it
can implicitly buffer+retry flowfiles that fail.

WDYT, Lars?

On 2/19/20 9:24 AM, Bryan Bende wrote:

We do already expose the ability to configure the retries in the
controller service [1], it was just a debate about what the default
value should be. Currently it is set to 1 because we believed it was
better to fail fast during the initial configuration of the service. A
user can easily set it back to 15, or whatever HBase client normally
does. I even suggested a compromise of making the default value 7,
which was half way between the two extremes, but that was considered
unacceptable, even though based on what you said in your original
message, most cases work on the first retry, so 7 would have covered
those.

We can also expose a property for the RPC timeout, but I think the
retries is more the issue here.

We can also definitely improve the docs, but I would still lean
towards the default retires being something lower, and then the docs
can explain that after ensuring the service can be successfully
started that this value can be increased to 15.

[1]

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-hbase_2-client-service-bundle/nifi-hbase_2-client-service/src/main/java/org/apache/nifi/hbase/HBase_2_ClientService.java#L134-L140


On Tue, Feb 18, 2020 at 8:31 PM Josh Elser  wrote:


We could certainly implement some kind of "sanity check" via HBase code,
but I think the thing that is missing is the logical way to validate
this in NiFi itself.

Something like...

```
Configuration conf = HBaseConfiguration.create();
conf.setInt("hbase.rpc.timeout", 5000);
try (Connection conn = ConnectionFactory.create(conf)) {
 // do sanity check
}
Configuration conf2 = new Configuration(conf);
conf2.setInt("hbase.rpc.timeout", 25000);
try (Connection conn = ConnectionFactory.create(conf2)) {
 // do real hbase-y stuff
}
```

Maybe instead of requiring an implicit way to do this (requiring NiFi
code changes), we could solve the problem at the "human level": create
docs that walk people through how to push a dummy record through the
service/processor with the low configuration of timeouts and retries?
Make the "sanity check" a human operation and just expose the ability to
set timeout/retries via the controller service?

On 2/18/20 4:36 PM, Lars Francke wrote:

Hi,

Josh, thanks for bringing it up here again.
I'm happy to revive the PR with whatever the outcome of this thread is.
It came up today because another client complained about how "unstable"
HBase is on NiFi.

@Josh: As the whole issue is only the initial connect can we have a
different timeout setting there? I have to admit I don't know.

Cheers,
Lars

On Tue, Feb 18, 2020 at 8:11 PM Pierre Villard <

pierre.villard...@gmail.com>

wrote:


Good point, I don't think we can do that on a controller service.

Le mar. 18 févr. 2020 à 11:06, Bryan Bende  a

écrit :



That could make it a little better, but I can't remember, can we
t

Re: [DISCUSS] On HBase client retries (NIFI-6197)

[Josh Elser]

Thanks for sharing the code, Bryan! I lazily did not go digging.

Looking into #onEnabled, we could change this to create its own 
Connection with a very low 1 or 2 retries with a very short retry 
duration. Throw that one away after we did our sanity check, set a high 
retry amount, and then create a new Connection for all of the 
puts/gets/scans/whatever the service will do.


I think NiFi, in general, can err on the side of "lower" client retries 
(both in number and duration of backoff) than a normal client since it 
can implicitly buffer+retry flowfiles that fail.


WDYT, Lars?

On 2/19/20 9:24 AM, Bryan Bende wrote:

We do already expose the ability to configure the retries in the
controller service [1], it was just a debate about what the default
value should be. Currently it is set to 1 because we believed it was
better to fail fast during the initial configuration of the service. A
user can easily set it back to 15, or whatever HBase client normally
does. I even suggested a compromise of making the default value 7,
which was half way between the two extremes, but that was considered
unacceptable, even though based on what you said in your original
message, most cases work on the first retry, so 7 would have covered
those.

We can also expose a property for the RPC timeout, but I think the
retries is more the issue here.

We can also definitely improve the docs, but I would still lean
towards the default retires being something lower, and then the docs
can explain that after ensuring the service can be successfully
started that this value can be increased to 15.

[1] 
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-hbase_2-client-service-bundle/nifi-hbase_2-client-service/src/main/java/org/apache/nifi/hbase/HBase_2_ClientService.java#L134-L140

On Tue, Feb 18, 2020 at 8:31 PM Josh Elser  wrote:


We could certainly implement some kind of "sanity check" via HBase code,
but I think the thing that is missing is the logical way to validate
this in NiFi itself.

Something like...

```
Configuration conf = HBaseConfiguration.create();
conf.setInt("hbase.rpc.timeout", 5000);
try (Connection conn = ConnectionFactory.create(conf)) {
// do sanity check
}
Configuration conf2 = new Configuration(conf);
conf2.setInt("hbase.rpc.timeout", 25000);
try (Connection conn = ConnectionFactory.create(conf2)) {
// do real hbase-y stuff
}
```

Maybe instead of requiring an implicit way to do this (requiring NiFi
code changes), we could solve the problem at the "human level": create
docs that walk people through how to push a dummy record through the
service/processor with the low configuration of timeouts and retries?
Make the "sanity check" a human operation and just expose the ability to
set timeout/retries via the controller service?

On 2/18/20 4:36 PM, Lars Francke wrote:

Hi,

Josh, thanks for bringing it up here again.
I'm happy to revive the PR with whatever the outcome of this thread is.
It came up today because another client complained about how "unstable"
HBase is on NiFi.

@Josh: As the whole issue is only the initial connect can we have a
different timeout setting there? I have to admit I don't know.

Cheers,
Lars

On Tue, Feb 18, 2020 at 8:11 PM Pierre Villard 
wrote:


Good point, I don't think we can do that on a controller service.

Le mar. 18 févr. 2020 à 11:06, Bryan Bende  a écrit :


That could make it a little better, but I can't remember, can we
terminate on a controller service?

The issue here would be on first time enabling the the HBase client
service, so before even getting to a processor.

On Tue, Feb 18, 2020 at 2:00 PM Pierre Villard
 wrote:


Bryan,

I didn't follow the whole discussion so I apologize if I'm saying

something

stupid here. Now that we have the possibility to terminate threads in a
processor, would that solve the issue?

Pierre

Le mar. 18 févr. 2020 à 10:52, Bryan Bende  a écrit

:



Hi Josh,

The problem isn't so much about the retries within the flow, its more
about setting up the service for the first time.

A common scenario for users was the following:

- Create a new HBase client service
- Enter some config that wasn't quite correct, possibly hostnames

that

weren't reachable from nifi as one example
- Enable service and enter retry loop
- Attempt to disable service to fix config, but have to wait 5+ mins
for the retries to finish

Maybe a lazy initialization of the connection on our side would help
here, although it would just be moving the problem until later (i.e.
service immediately enables because nothing is happening, then they
find out about config problems later when a flow file hits an hbase
processor).

I guess the ideal scenario would be to have different logic for
initializing the connection vs. using it, so that there wouldn't be
retries during initialization.

-Bryan



On Tue, Feb 18

Re: [DISCUSS] On HBase client retries (NIFI-6197)

[Josh Elser]

We could certainly implement some kind of "sanity check" via HBase code, 
but I think the thing that is missing is the logical way to validate 
this in NiFi itself.


Something like...

```
Configuration conf = HBaseConfiguration.create();
conf.setInt("hbase.rpc.timeout", 5000);
try (Connection conn = ConnectionFactory.create(conf)) {
  // do sanity check
}
Configuration conf2 = new Configuration(conf);
conf2.setInt("hbase.rpc.timeout", 25000);
try (Connection conn = ConnectionFactory.create(conf2)) {
  // do real hbase-y stuff
}
```

Maybe instead of requiring an implicit way to do this (requiring NiFi 
code changes), we could solve the problem at the "human level": create 
docs that walk people through how to push a dummy record through the 
service/processor with the low configuration of timeouts and retries? 
Make the "sanity check" a human operation and just expose the ability to 
set timeout/retries via the controller service?


On 2/18/20 4:36 PM, Lars Francke wrote:

Hi,

Josh, thanks for bringing it up here again.
I'm happy to revive the PR with whatever the outcome of this thread is.
It came up today because another client complained about how "unstable"
HBase is on NiFi.

@Josh: As the whole issue is only the initial connect can we have a
different timeout setting there? I have to admit I don't know.

Cheers,
Lars

On Tue, Feb 18, 2020 at 8:11 PM Pierre Villard 
wrote:


Good point, I don't think we can do that on a controller service.

Le mar. 18 févr. 2020 à 11:06, Bryan Bende  a écrit :


That could make it a little better, but I can't remember, can we
terminate on a controller service?

The issue here would be on first time enabling the the HBase client
service, so before even getting to a processor.

On Tue, Feb 18, 2020 at 2:00 PM Pierre Villard
 wrote:


Bryan,

I didn't follow the whole discussion so I apologize if I'm saying

something

stupid here. Now that we have the possibility to terminate threads in a
processor, would that solve the issue?

Pierre

Le mar. 18 févr. 2020 à 10:52, Bryan Bende  a écrit

:



Hi Josh,

The problem isn't so much about the retries within the flow, its more
about setting up the service for the first time.

A common scenario for users was the following:

- Create a new HBase client service
- Enter some config that wasn't quite correct, possibly hostnames

that

weren't reachable from nifi as one example
- Enable service and enter retry loop
- Attempt to disable service to fix config, but have to wait 5+ mins
for the retries to finish

Maybe a lazy initialization of the connection on our side would help
here, although it would just be moving the problem until later (i.e.
service immediately enables because nothing is happening, then they
find out about config problems later when a flow file hits an hbase
processor).

I guess the ideal scenario would be to have different logic for
initializing the connection vs. using it, so that there wouldn't be
retries during initialization.

-Bryan



On Tue, Feb 18, 2020 at 1:21 PM Josh Elser 

wrote:


Hiya!

LarsF brought this up in the apache-hbase slack account and it

caught my

eye. Sending a note here since the PR is closed where this was

being

discussed before[1].

I understand Bryan's concerns that misconfiguration of an HBase
processor with a high number of retries and back-off can create a
situation in which the processing of a single FlowFile will take a

very

long time to hit the onFailure state.

However, as an HBase developer, I can confidently state that
hbase.client.retries=1 will create scenarios in which you'll be

pushing

a FlowFile through a retry loop inside of NiFi for things which

should

be implicitly retried inside of the HBase client.

For example, if a Region is being moved between two RegionServers

and an

HBase processor is trying to read/write to that Region, the client

will

see an exception. This is a "retriable" exception in HBase-parlance
which means that HBase client code would automatically re-process

that

request (looking for the new location of that Region first). In

most

cases, the subsequent RPC would succeed and the caller is

non-the-wiser

and the whole retry logic took 1's of milliseconds.

My first idea was also what Lars' had suggested -- can we come up

with a

sanity check to validate "correct" configuration for the processor
before we throw the waterfall of data at it? I can respect if

processors

don't have a "good" hook to do such a check.

What _would_ be the ideal semantics from NiFi's? perspective? We

have

the ability to implicitly retry operations and also control the

retry

backoff values. Is there something more we could do from the HBase

side,

given what y'all have seen from the battlefield?

Thanks!

- Josh

[1] https://github.com/apache/nifi/pull/3425

[DISCUSS] On HBase client retries (NIFI-6197)

[Josh Elser]

Hiya!

LarsF brought this up in the apache-hbase slack account and it caught my 
eye. Sending a note here since the PR is closed where this was being 
discussed before[1].


I understand Bryan's concerns that misconfiguration of an HBase 
processor with a high number of retries and back-off can create a 
situation in which the processing of a single FlowFile will take a very 
long time to hit the onFailure state.


However, as an HBase developer, I can confidently state that 
hbase.client.retries=1 will create scenarios in which you'll be pushing 
a FlowFile through a retry loop inside of NiFi for things which should 
be implicitly retried inside of the HBase client.


For example, if a Region is being moved between two RegionServers and an 
HBase processor is trying to read/write to that Region, the client will 
see an exception. This is a "retriable" exception in HBase-parlance 
which means that HBase client code would automatically re-process that 
request (looking for the new location of that Region first). In most 
cases, the subsequent RPC would succeed and the caller is non-the-wiser 
and the whole retry logic took 1's of milliseconds.


My first idea was also what Lars' had suggested -- can we come up with a 
sanity check to validate "correct" configuration for the processor 
before we throw the waterfall of data at it? I can respect if processors 
don't have a "good" hook to do such a check.


What _would_ be the ideal semantics from NiFi's? perspective? We have 
the ability to implicitly retry operations and also control the retry 
backoff values. Is there something more we could do from the HBase side, 
given what y'all have seen from the battlefield?


Thanks!

- Josh

[1] https://github.com/apache/nifi/pull/3425

Re: [DISCUSS] Tar + Gzip vs. Zip

[Josh Elser]

On 6/25/18 11:34 PM, Andy LoPresto wrote:

Hi folks,

I do not want to start a long-running argument or entrenched battle. 
However, having just performed the RM duties for the latest release, I 
believe I have identified a resource inefficiency in the fact that we 
generate, upload, host, and distribute two compressed archives of the 
binary which are functionally equivalent. For 1.7.0, both the .tar.gz 
and .zip files are 1.2 GB (1_224_352_000 bytes for tar.gz vs. 
1_224_392_000 bytes for zip). The time to build and sign these is 
substantial, but the true cost comes in uploading and hosting them. 
While the fabled extension registry will save all of us from this 
burden, it isn’t arriving tomorrow, and I think we could drastically 
improve this before the next release.


I have no personal preference between the two formats. In earlier days, 
there were platform inconsistencies and the tools weren’t available on 
all systems, but now they are pretty standard for all users. This [1] is 
an interesting article I found which had some good info on the origins, 
and here are some additional resources for anyone interested [2][3]. I 
don’t care which we pick, but I propose removing one of the options for 
the build going forward (toolkit as well).


That said, if someone has a good reason that both are necessary, I would 
love to hear it. I didn’t find anything on the Apache Release Policy 
which stated we must offer both, but maybe I missed it. Thanks.


I'm not aware of any ASF policy. I think it mostly stems from default 
convention you get out of the maven-assembly-plugin.



[1] https://itsfoss.com/tar-vs-zip-vs-gz/
[2] https://superuser.com/a/1257441/40003
[3] https://superuser.com/a/173995/40003
[4] https://www.apache.org/legal/release-policy.html#artifacts


Andy LoPresto
alopre...@apache.org 
/alopresto.apa...@gmail.com /
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

Re: [DISCUSS] Official Apache NiFi Docker Image

[Josh Elser]

Thanks for the pointer, Aldrin. I learned something new today :)

Sounds like you all have a great handle on this.

/me goes back to lurking

Aldrin Piri wrote:

I think with some of the general ideas we discussed, we are tracking
appropriately with currently accepted guidance. Of particular note is our
discussed terms of tagging/versions of the images that would be created
exclusively and solely with community releases.

I base this interpretation off LEGAL-270 [1].

[1] https://issues.apache.org/jira/browse/LEGAL-270

On Fri, Jan 6, 2017 at 11:36 AM, Josh Elser  wrote:


Cool, regex to work on the recent "rel/*" tagging scheme would work out
well!

I'd also give one word of caution about licensing (I don't think the ASF
as a whole has figured out how to navigate this). Essentially, a docker
image is a "convenience binary" and may contain GPL'ed things. Thus, this
would violate the traditional policy.

I don't bring that up to dissuade Jeremy or anyone from taking up the
work, just to make sure that you all are aware that this has been a broad
area of discussion :)


Aldrin Piri wrote:


Thanks for the info, Josh.  Looks like these are building directly off of
the ASF Github mirrors.

Looks like a few projects have navigated the process [1] and INFRA would
likely be the folks to make that happen.  Need to understand a bit more
about what is appropriate in the context of ASF.  But, at minimum, we
could
create automated builds that only build release tags using a regex [2].

[1]
https://issues.apache.org/jira/browse/INFRA-12019?jql=text%
20~%20%22dockerhub%22%20and%20project%20%3D%20INFRA
[2] https://docs.docker.com/docker-hub/builds/

On Thu, Jan 5, 2017 at 2:07 PM, Josh Elser   wrote:

FYI, there's also an "apache" space on dockerhub[1]. Sadly, I'm a little

unclear on how a project would actually go about pushing stuff there.
Might
be some docs floating around or a ping to infra.

[1] https://hub.docker.com/u/apache/


Jeremy Dyer wrote:

Team,

I wanted to discuss getting an official Apache NiFi Docker image similar
to
other Apache projects like storm [1], httpd [2], thrift [3], etc.

Official Docker images are hosted at http://www.dockerhub.com and made
available to the Docker runtime of end users without them having to
build
the images themselves. The process of making a Docker image "official",
meaning that it is validated and reviewed by a community of Docker folks
for security flaws, best practices, etc, works very closely to how our
standard contribution process to NiFi works today. We as a community
would
create our Dockerfile(s) and review them just like we review any JIRA
today
and then commit that against our codebase.

There is an additional step from there in that once we have a commit
against our codebase we would need an "ambassador" (I happily volunteer
to
handle this if there are no objections) who would open a Github Pull
Request against the official docker image repo [4]. Once that PR has
successfully been reviewed by the official repo folks it would be hosted
on
Dockerhub and readily available to end users.

In my mind the steps required to reach this goal would be.
1. Create NiFi, MiNiFi, MiNiFi-CPP JIRAs for creating the initial folder
structure and baseline Dockerfiles in each repo. I also volunteer myself
to
take this on as well.
2. Once JIRA is completed, reviewed, and community thumbs up is given I
will request the Dockerhub repo handle of "library/apachenifi" with the
maintainer of that repos contact email as
2a). I suggest we follow the naming structure like
"library/apachenifi:nifi-1.1.0", "library/apachenifi:minifi-0.1.0",
"libraryapachenifi:minifi-cpp-0.1.0". This makes our official image
much
more clean than having 3 separate official images for each subproject.
3) I will open a PR against [4] with our community Dockerfiles
4) After each release I will continue to open pull requests against [4]
to
ensure the latest releases are present.

Please let me know your thoughts.

[1] - https://hub.docker.com/r/library/storm/
[2] - https://hub.docker.com/_/httpd/
[3] - https://hub.docker.com/_/thrift/
[4] - https://github.com/docker-library/official-images

Thanks,
Jeremy Dyer

Re: [DISCUSS] Official Apache NiFi Docker Image

[Josh Elser]

Cool, regex to work on the recent "rel/*" tagging scheme would work out 
well!


I'd also give one word of caution about licensing (I don't think the ASF 
as a whole has figured out how to navigate this). Essentially, a docker 
image is a "convenience binary" and may contain GPL'ed things. Thus, 
this would violate the traditional policy.


I don't bring that up to dissuade Jeremy or anyone from taking up the 
work, just to make sure that you all are aware that this has been a 
broad area of discussion :)


Aldrin Piri wrote:

Thanks for the info, Josh.  Looks like these are building directly off of
the ASF Github mirrors.

Looks like a few projects have navigated the process [1] and INFRA would
likely be the folks to make that happen.  Need to understand a bit more
about what is appropriate in the context of ASF.  But, at minimum, we could
create automated builds that only build release tags using a regex [2].

[1]
https://issues.apache.org/jira/browse/INFRA-12019?jql=text%20~%20%22dockerhub%22%20and%20project%20%3D%20INFRA
[2] https://docs.docker.com/docker-hub/builds/

On Thu, Jan 5, 2017 at 2:07 PM, Josh Elser  wrote:


FYI, there's also an "apache" space on dockerhub[1]. Sadly, I'm a little
unclear on how a project would actually go about pushing stuff there. Might
be some docs floating around or a ping to infra.

[1] https://hub.docker.com/u/apache/


Jeremy Dyer wrote:


Team,

I wanted to discuss getting an official Apache NiFi Docker image similar
to
other Apache projects like storm [1], httpd [2], thrift [3], etc.

Official Docker images are hosted at http://www.dockerhub.com and made
available to the Docker runtime of end users without them having to build
the images themselves. The process of making a Docker image "official",
meaning that it is validated and reviewed by a community of Docker folks
for security flaws, best practices, etc, works very closely to how our
standard contribution process to NiFi works today. We as a community would
create our Dockerfile(s) and review them just like we review any JIRA
today
and then commit that against our codebase.

There is an additional step from there in that once we have a commit
against our codebase we would need an "ambassador" (I happily volunteer to
handle this if there are no objections) who would open a Github Pull
Request against the official docker image repo [4]. Once that PR has
successfully been reviewed by the official repo folks it would be hosted
on
Dockerhub and readily available to end users.

In my mind the steps required to reach this goal would be.
1. Create NiFi, MiNiFi, MiNiFi-CPP JIRAs for creating the initial folder
structure and baseline Dockerfiles in each repo. I also volunteer myself
to
take this on as well.
2. Once JIRA is completed, reviewed, and community thumbs up is given I
will request the Dockerhub repo handle of "library/apachenifi" with the
maintainer of that repos contact email as
2a). I suggest we follow the naming structure like
"library/apachenifi:nifi-1.1.0", "library/apachenifi:minifi-0.1.0",
"libraryapachenifi:minifi-cpp-0.1.0". This makes our official image much
more clean than having 3 separate official images for each subproject.
3) I will open a PR against [4] with our community Dockerfiles
4) After each release I will continue to open pull requests against [4] to
ensure the latest releases are present.

Please let me know your thoughts.

[1] - https://hub.docker.com/r/library/storm/
[2] - https://hub.docker.com/_/httpd/
[3] - https://hub.docker.com/_/thrift/
[4] - https://github.com/docker-library/official-images

Thanks,
Jeremy Dyer

Re: [DISCUSS] Official Apache NiFi Docker Image

[Josh Elser]

FYI, there's also an "apache" space on dockerhub[1]. Sadly, I'm a little 
unclear on how a project would actually go about pushing stuff there. 
Might be some docs floating around or a ping to infra.


[1] https://hub.docker.com/u/apache/

Jeremy Dyer wrote:

Team,

I wanted to discuss getting an official Apache NiFi Docker image similar to
other Apache projects like storm [1], httpd [2], thrift [3], etc.

Official Docker images are hosted at http://www.dockerhub.com and made
available to the Docker runtime of end users without them having to build
the images themselves. The process of making a Docker image "official",
meaning that it is validated and reviewed by a community of Docker folks
for security flaws, best practices, etc, works very closely to how our
standard contribution process to NiFi works today. We as a community would
create our Dockerfile(s) and review them just like we review any JIRA today
and then commit that against our codebase.

There is an additional step from there in that once we have a commit
against our codebase we would need an "ambassador" (I happily volunteer to
handle this if there are no objections) who would open a Github Pull
Request against the official docker image repo [4]. Once that PR has
successfully been reviewed by the official repo folks it would be hosted on
Dockerhub and readily available to end users.

In my mind the steps required to reach this goal would be.
1. Create NiFi, MiNiFi, MiNiFi-CPP JIRAs for creating the initial folder
structure and baseline Dockerfiles in each repo. I also volunteer myself to
take this on as well.
2. Once JIRA is completed, reviewed, and community thumbs up is given I
will request the Dockerhub repo handle of "library/apachenifi" with the
maintainer of that repos contact email as
2a). I suggest we follow the naming structure like
"library/apachenifi:nifi-1.1.0", "library/apachenifi:minifi-0.1.0",
"libraryapachenifi:minifi-cpp-0.1.0". This makes our official image much
more clean than having 3 separate official images for each subproject.
3) I will open a PR against [4] with our community Dockerfiles
4) After each release I will continue to open pull requests against [4] to
ensure the latest releases are present.

Please let me know your thoughts.

[1] - https://hub.docker.com/r/library/storm/
[2] - https://hub.docker.com/_/httpd/
[3] - https://hub.docker.com/_/thrift/
[4] - https://github.com/docker-library/official-images

Thanks,
Jeremy Dyer

Re: [DISCUSS] Slack team for Apache NiFi

[Josh Elser]

ASF's chat bot has the ability to automatically record conversations and 
mail them to a list. As long as you remember to turn it on before 
starting the conversation (wink), it's not too bad.


http://www.apache.org/dev/asfbot.html#meetings

Bryan Bende wrote:

My only concern with any kind of chat/channel is the loss of searchable
public content.
Right now the mailing list posts show up in Google searches and can be
found through other various tools, which is valuable for people searching
for information related to an issue they are seeing.
If people start transitioning to chat-based communication, all the
conversations there will likely only be found through that tool, and will
probably not be organized into individual threads of questions.

I have not used Slack or Gitter much so I can't really say which is better,
but if I remember correctly Gitter only required you to sign-in with your
GitHub account and after that you could go into the community without
approval, which seems more open to me, but I really don't know.

On Mon, Oct 10, 2016 at 2:28 PM, John Wiesel  wrote:


Good evening,

I am a new member to the community so I do not know much about the current
needs.
Still I'd like to point at gitter.im which has an open-sourced IRC bridge
(https://irc.gitter.im/) for the heavy IRC users.
Overall imho a good Slack alternative, its aim is to provide chatrooms for
developers.

Best
John


On Mon, Oct 10, 2016 at 4:53 PM, Matt Burgess
wrote:


All,

I'd like to revisit the idea of having (and promoting) a Slack team
for the Apache NiFi community. We broached the subject in an email
thread a while back [1]. The email lists are great and IRC per se is
still popular in the open-source community, but I think for folks that
are more comfortable in the Slack environment, this would be another
good avenue for the community to get together.

We could perhaps add integrations with the email list(s), IRC
channel(s), GitHub, etc. as sort of a one-stop shop for NiFi
communication, or just use it for online collaboration.

I'm thinking it would be invite-only (to avoid bots, spam, etc.) but
that it would be fully open to the community, just an email to an
admin (such as myself) to request an invitation (kind of like a manual
version of the Apache Mailing List subscribe bot). The code of conduct
would be the Apache Software Foundation's CoC [2], as it is a
"communication channel used by our communities". I have seen this same
invite-only approach work well for other ASF projects with Slack
teams.

I grabbed a URL [3] just in case the community would like to proceed
(nifi.slack.com is taken, but if someone in the community owns it and
wants to use that instead, it's all good). The PMC would be
owners/admins, my email address (mattyb...@apache.org) could be the
destination for invite requests, and we could discuss what
integrations, access rights (creating new channels, e.g.), etc. are
appropriate.


Thanks,
Matt

[1] https://mail-archives.apache.org/mod_mbox/nifi-users/201511.
mbox/%3CCA+w4d5RJvxvFEnGVsexCSmEvzL_TxG_Ne2KEiFsrGEapCwoOAQ@
mail.gmail.com%3E

[2] https://www.apache.org/foundation/policies/conduct

[3] https://apachenifi.slack.com

Re: Fwd: Your Distro

[Josh Elser]

They own the mailing list software (thus, the name "infrastructure").

*@nifi.apache.org manages the community and software you know as Apache 
Nifi.


Paul Nahay wrote:

Thanks, although why would they be willing to help me?

-Original Message-----
From: Josh Elser [mailto:josh.el...@gmail.com]
Sent: Thursday, March 24, 2016 6:02 PM
To: Paul Nahay
Cc: dev@nifi.apache.org
Subject: Re: Fwd: Your Distro

Sorry you're having problems, but if you want help figuring it out, please 
contact infrastruc...@apache.org

Paul Nahay wrote:

Yes. I don't understand it.

See screenshot for my settings, which have never changed.

Paul

-Original Message-----
From: Josh Elser [mailto:josh.el...@gmail.com]
Sent: Thursday, March 24, 2016 10:28 AM
To: Paul Nahay
Subject: Re: Fwd: Your Distro

The mail list server appears to disagree :)

Paul Nahay wrote:

According to my webmail settings, I HAVE been sending as flat text, not as HTML!


Paul Nahay
1013 Rosemere Ave.
Silver Spring, MD 20904-3008
301-680-3825 (home)
301-806-9265 (cell)
pna...@sprynet.com


-Original Message-----

From: Josh Elser
Sent: Mar 24, 2016 12:44 AM
To: dev@nifi.apache.org
Cc: pna...@sprynet.com
Subject: Re: Fwd: Your Distro

Matthew Clarke wrote:

-- Forwarded message --
From: Paul Nahay
Date: Wed, Mar 23, 2016 at 10:31 AM
Subject: Your Distro
To: Matthew Clarke


This is the 2nd time I've gotten an email I sent to
dev@nifi.apache.org returned to me:



Hi. This is the qmail-send program at apache.org.
I'm afraid I wasn't able to deliver your message to the following addresses.
This is a permanent error; I've given up. Sorry it didn't work out.

:
ezmlm-reject: fatal: Sorry, I don't accept messages of MIME
Content-Type 'text/html' (#5.2.3)


As it says: you're sending HTML emails which the mail-list program
doesn't accept.

In other words, configure your mail client to send plaintext emails
instead of HTML "rich content" message. If you still have trouble
sending email, you should contact infrastruct...@apache.org for
assistance in configuring your mail client.

Re: Fwd: Your Distro

[Josh Elser]

Sorry you're having problems, but if you want help figuring it out, 
please contact infrastruc...@apache.org


Paul Nahay wrote:

Yes. I don't understand it.

See screenshot for my settings, which have never changed.

Paul

-Original Message-----
From: Josh Elser [mailto:josh.el...@gmail.com]
Sent: Thursday, March 24, 2016 10:28 AM
To: Paul Nahay
Subject: Re: Fwd: Your Distro

The mail list server appears to disagree :)

Paul Nahay wrote:

According to my webmail settings, I HAVE been sending as flat text, not as HTML!


Paul Nahay
1013 Rosemere Ave.
Silver Spring, MD 20904-3008
301-680-3825 (home)
301-806-9265 (cell)
pna...@sprynet.com


-Original Message-----

From: Josh Elser
Sent: Mar 24, 2016 12:44 AM
To: dev@nifi.apache.org
Cc: pna...@sprynet.com
Subject: Re: Fwd: Your Distro

Matthew Clarke wrote:

-- Forwarded message --
From: Paul Nahay
Date: Wed, Mar 23, 2016 at 10:31 AM
Subject: Your Distro
To: Matthew Clarke


This is the 2nd time I've gotten an email I sent to
dev@nifi.apache.org returned to me:



Hi. This is the qmail-send program at apache.org.
I'm afraid I wasn't able to deliver your message to the following addresses.
This is a permanent error; I've given up. Sorry it didn't work out.

:
ezmlm-reject: fatal: Sorry, I don't accept messages of MIME
Content-Type 'text/html' (#5.2.3)


As it says: you're sending HTML emails which the mail-list program
doesn't accept.

In other words, configure your mail client to send plaintext emails
instead of HTML "rich content" message. If you still have trouble
sending email, you should contact infrastruct...@apache.org for
assistance in configuring your mail client.

Re: Fwd: Your Distro

[Josh Elser]

Matthew Clarke wrote:

-- Forwarded message --
From: Paul Nahay
Date: Wed, Mar 23, 2016 at 10:31 AM
Subject: Your Distro
To: Matthew Clarke


This is the 2nd time I've gotten an email I sent to
dev@nifi.apache.org
returned to me:



Hi. This is the qmail-send program at apache.org.
I'm afraid I wasn't able to deliver your message to the following addresses.
This is a permanent error; I've given up. Sorry it didn't work out.

:
ezmlm-reject: fatal: Sorry, I don't accept messages of MIME
Content-Type 'text/html' (#5.2.3)



As it says: you're sending HTML emails which the mail-list program 
doesn't accept.


In other words, configure your mail client to send plaintext emails 
instead of HTML "rich content" message. If you still have trouble 
sending email, you should contact infrastruct...@apache.org for 
assistance in configuring your mail client.

Re: Pull Request Comments and JIRA

[Josh Elser]

I know this was working for Accumulo in September. Maybe it broke since 
then? Probably worth a note/JIRA to infra to ask.


Bryan Bende wrote:

Does anyone know why in-line comments on pull-requests don't post back to
the JIRA?

Comments on the overall pull request do post back. I feel like this might
have been something that worked during incubation and no longer works, but
could be wrong.