[jira] [Commented] (CAMEL-11656) Support default directory sorter for FileConsumer
[
https://issues.apache.org/jira/browse/CAMEL-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122682#comment-16122682
]
Claus Ibsen commented on CAMEL-11656:
-
There is already a lot of options on the file component so we have to be a bit
careful to not keep adding new stuff.
> Support default directory sorter for FileConsumer
> -
>
> Key: CAMEL-11656
> URL: https://issues.apache.org/jira/browse/CAMEL-11656
> Project: Camel
> Issue Type: Wish
> Components: camel-core
>Affects Versions: 2.19.0
>Reporter: Viktor Khoroshko
> Fix For: Future
>
>
> Hello,
> The current way of sorting consumed files isn't very flexible due to how it's
> implemented:
> Files are sorted AFTER they're added to in progress repository what in
> combination with *maxMessagesPerPoll* set to be > 0 and
> *eagerLimitMaxMessagesPerPoll* set to false forces the
> *removeExcessiveInProgressFiles* call - which in case of persistent
> idempotent repository used causes redundant calls to a database.
> This is not only the issue.
> In my case I have a logic in a custom file filter that a file shouldn't be
> processed if a file with a same prefix is already in progress while still
> those files should be processed in a specified order.
> The issue is that since sorting is performed after each file is added to in
> progress repository and before added there a file should be accepted by a
> filter but it will be not - as a previous file with a same prefix has
> already been added to in progress repo.
> *Example*: files - test.001, test.002, test.003
> If test.003 is first in a returned file list then it will be added to in
> progress repository and next files will not be accepted due to the same
> prefix.
> Default sorting would solve this issue.
> It would be great if default sorter can be specified for *FileConsumer* in
> the pollDirectory method:
> {code:java}
> log.trace("Polling directory: {}", directory.getPath());
> File[] dirFiles = directory.listFiles();
> if (dirFiles == null || dirFiles.length == 0) {
> // no files in this directory to poll
> if (log.isTraceEnabled()) {
> log.trace("No files found in directory: {}",
> directory.getPath());
> }
> return true;
> } else {
> // we found some files
> if (log.isTraceEnabled()) {
> log.trace("Found {} in directory: {}", dirFiles.length,
> directory.getPath());
> }
> }
> List files = Arrays.asList(dirFiles);
>// sort there?
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CAMEL-11656) Support default directory sorter for FileConsumer
[
https://issues.apache.org/jira/browse/CAMEL-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16152550#comment-16152550
]
Claus Ibsen commented on CAMEL-11656:
-
What do you mean by default sorter?
> Support default directory sorter for FileConsumer
> -
>
> Key: CAMEL-11656
> URL: https://issues.apache.org/jira/browse/CAMEL-11656
> Project: Camel
> Issue Type: Wish
> Components: camel-core
>Affects Versions: 2.19.0
>Reporter: Viktor Khoroshko
>Priority: Minor
> Fix For: Future
>
>
> Hello,
> The current way of sorting consumed files isn't very flexible due to how it's
> implemented:
> Files are sorted AFTER they're added to in progress repository what in
> combination with *maxMessagesPerPoll* set to be > 0 and
> *eagerLimitMaxMessagesPerPoll* set to false forces the
> *removeExcessiveInProgressFiles* call - which in case of persistent
> idempotent repository used causes redundant calls to a database.
> This is not only the issue.
> In my case I have a logic in a custom file filter that a file shouldn't be
> processed if a file with a same prefix is already in progress while still
> those files should be processed in a specified order.
> The issue is that since sorting is performed after each file is added to in
> progress repository and before added there a file should be accepted by a
> filter but it will be not - as a previous file with a same prefix has
> already been added to in progress repo.
> *Example*: files - test.001, test.002, test.003
> If test.003 is first in a returned file list then it will be added to in
> progress repository and next files will not be accepted due to the same
> prefix.
> Default sorting would solve this issue.
> It would be great if default sorter can be specified for *FileConsumer* in
> the pollDirectory method:
> {code:java}
> log.trace("Polling directory: {}", directory.getPath());
> File[] dirFiles = directory.listFiles();
> if (dirFiles == null || dirFiles.length == 0) {
> // no files in this directory to poll
> if (log.isTraceEnabled()) {
> log.trace("No files found in directory: {}",
> directory.getPath());
> }
> return true;
> } else {
> // we found some files
> if (log.isTraceEnabled()) {
> log.trace("Found {} in directory: {}", dirFiles.length,
> directory.getPath());
> }
> }
> List files = Arrays.asList(dirFiles);
>// sort there?
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CAMEL-11656) Support default directory sorter for FileConsumer
[
https://issues.apache.org/jira/browse/CAMEL-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16245700#comment-16245700
]
Viktor Khoroshko commented on CAMEL-11656:
--
Just sort list of files of found alphabetically by default in the code
placeholder "// sort there" from above description.
> Support default directory sorter for FileConsumer
> -
>
> Key: CAMEL-11656
> URL: https://issues.apache.org/jira/browse/CAMEL-11656
> Project: Camel
> Issue Type: Wish
> Components: camel-core
>Affects Versions: 2.19.0
>Reporter: Viktor Khoroshko
>Priority: Minor
> Fix For: Future
>
>
> Hello,
> The current way of sorting consumed files isn't very flexible due to how it's
> implemented:
> Files are sorted AFTER they're added to in progress repository what in
> combination with *maxMessagesPerPoll* set to be > 0 and
> *eagerLimitMaxMessagesPerPoll* set to false forces the
> *removeExcessiveInProgressFiles* call - which in case of persistent
> idempotent repository used causes redundant calls to a database.
> This is not only the issue.
> In my case I have a logic in a custom file filter that a file shouldn't be
> processed if a file with a same prefix is already in progress while still
> those files should be processed in a specified order.
> The issue is that since sorting is performed after each file is added to in
> progress repository and before added there a file should be accepted by a
> filter but it will be not - as a previous file with a same prefix has
> already been added to in progress repo.
> *Example*: files - test.001, test.002, test.003
> If test.003 is first in a returned file list then it will be added to in
> progress repository and next files will not be accepted due to the same
> prefix.
> Default sorting would solve this issue.
> It would be great if default sorter can be specified for *FileConsumer* in
> the pollDirectory method:
> {code:java}
> log.trace("Polling directory: {}", directory.getPath());
> File[] dirFiles = directory.listFiles();
> if (dirFiles == null || dirFiles.length == 0) {
> // no files in this directory to poll
> if (log.isTraceEnabled()) {
> log.trace("No files found in directory: {}",
> directory.getPath());
> }
> return true;
> } else {
> // we found some files
> if (log.isTraceEnabled()) {
> log.trace("Found {} in directory: {}", dirFiles.length,
> directory.getPath());
> }
> }
> List files = Arrays.asList(dirFiles);
>// sort there?
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CAMEL-11656) Support default directory sorter for FileConsumer
[
https://issues.apache.org/jira/browse/CAMEL-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16247312#comment-16247312
]
Claus Ibsen commented on CAMEL-11656:
-
Okay people is welcome to contribute with a new option to turn on pre sorting.
@UriParam("label = "consumer")
boolean preSort = false
Then you can turn this on, in your use-case.
Also mind that we should try to implement this for the camel-ftp component as
well.
> Support default directory sorter for FileConsumer
> -
>
> Key: CAMEL-11656
> URL: https://issues.apache.org/jira/browse/CAMEL-11656
> Project: Camel
> Issue Type: Wish
> Components: camel-core
>Affects Versions: 2.19.0
>Reporter: Viktor Khoroshko
>Priority: Minor
> Fix For: 2.21.0
>
>
> Hello,
> The current way of sorting consumed files isn't very flexible due to how it's
> implemented:
> Files are sorted AFTER they're added to in progress repository what in
> combination with *maxMessagesPerPoll* set to be > 0 and
> *eagerLimitMaxMessagesPerPoll* set to false forces the
> *removeExcessiveInProgressFiles* call - which in case of persistent
> idempotent repository used causes redundant calls to a database.
> This is not only the issue.
> In my case I have a logic in a custom file filter that a file shouldn't be
> processed if a file with a same prefix is already in progress while still
> those files should be processed in a specified order.
> The issue is that since sorting is performed after each file is added to in
> progress repository and before added there a file should be accepted by a
> filter but it will be not - as a previous file with a same prefix has
> already been added to in progress repo.
> *Example*: files - test.001, test.002, test.003
> If test.003 is first in a returned file list then it will be added to in
> progress repository and next files will not be accepted due to the same
> prefix.
> Default sorting would solve this issue.
> It would be great if default sorter can be specified for *FileConsumer* in
> the pollDirectory method:
> {code:java}
> log.trace("Polling directory: {}", directory.getPath());
> File[] dirFiles = directory.listFiles();
> if (dirFiles == null || dirFiles.length == 0) {
> // no files in this directory to poll
> if (log.isTraceEnabled()) {
> log.trace("No files found in directory: {}",
> directory.getPath());
> }
> return true;
> } else {
> // we found some files
> if (log.isTraceEnabled()) {
> log.trace("Found {} in directory: {}", dirFiles.length,
> directory.getPath());
> }
> }
> List files = Arrays.asList(dirFiles);
>// sort there?
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CAMEL-11656) Support default directory sorter for FileConsumer
[
https://issues.apache.org/jira/browse/CAMEL-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16262297#comment-16262297
]
Önder Sezgin commented on CAMEL-11656:
--
[~oceansize] i am not sure if default sorting at line you mentioned would solve
the issue.
because AFAIK {{directory.listFiles()}} already lists the files alphabetically.
> Support default directory sorter for FileConsumer
> -
>
> Key: CAMEL-11656
> URL: https://issues.apache.org/jira/browse/CAMEL-11656
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
>Affects Versions: 2.19.0
>Reporter: Viktor Khoroshko
>Priority: Minor
> Fix For: 2.21.0
>
>
> Hello,
> The current way of sorting consumed files isn't very flexible due to how it's
> implemented:
> Files are sorted AFTER they're added to in progress repository what in
> combination with *maxMessagesPerPoll* set to be > 0 and
> *eagerLimitMaxMessagesPerPoll* set to false forces the
> *removeExcessiveInProgressFiles* call - which in case of persistent
> idempotent repository used causes redundant calls to a database.
> This is not only the issue.
> In my case I have a logic in a custom file filter that a file shouldn't be
> processed if a file with a same prefix is already in progress while still
> those files should be processed in a specified order.
> The issue is that since sorting is performed after each file is added to in
> progress repository and before added there a file should be accepted by a
> filter but it will be not - as a previous file with a same prefix has
> already been added to in progress repo.
> *Example*: files - test.001, test.002, test.003
> If test.003 is first in a returned file list then it will be added to in
> progress repository and next files will not be accepted due to the same
> prefix.
> Default sorting would solve this issue.
> It would be great if default sorter can be specified for *FileConsumer* in
> the pollDirectory method:
> {code:java}
> log.trace("Polling directory: {}", directory.getPath());
> File[] dirFiles = directory.listFiles();
> if (dirFiles == null || dirFiles.length == 0) {
> // no files in this directory to poll
> if (log.isTraceEnabled()) {
> log.trace("No files found in directory: {}",
> directory.getPath());
> }
> return true;
> } else {
> // we found some files
> if (log.isTraceEnabled()) {
> log.trace("Found {} in directory: {}", dirFiles.length,
> directory.getPath());
> }
> }
> List files = Arrays.asList(dirFiles);
>// sort there?
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CAMEL-11656) Support default directory sorter for FileConsumer
[
https://issues.apache.org/jira/browse/CAMEL-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16262302#comment-16262302
]
Önder Sezgin commented on CAMEL-11656:
--
and maybe {{sortBy}} option maybe used? doesn't it work for you?
> Support default directory sorter for FileConsumer
> -
>
> Key: CAMEL-11656
> URL: https://issues.apache.org/jira/browse/CAMEL-11656
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
>Affects Versions: 2.19.0
>Reporter: Viktor Khoroshko
>Priority: Minor
> Fix For: 2.21.0
>
>
> Hello,
> The current way of sorting consumed files isn't very flexible due to how it's
> implemented:
> Files are sorted AFTER they're added to in progress repository what in
> combination with *maxMessagesPerPoll* set to be > 0 and
> *eagerLimitMaxMessagesPerPoll* set to false forces the
> *removeExcessiveInProgressFiles* call - which in case of persistent
> idempotent repository used causes redundant calls to a database.
> This is not only the issue.
> In my case I have a logic in a custom file filter that a file shouldn't be
> processed if a file with a same prefix is already in progress while still
> those files should be processed in a specified order.
> The issue is that since sorting is performed after each file is added to in
> progress repository and before added there a file should be accepted by a
> filter but it will be not - as a previous file with a same prefix has
> already been added to in progress repo.
> *Example*: files - test.001, test.002, test.003
> If test.003 is first in a returned file list then it will be added to in
> progress repository and next files will not be accepted due to the same
> prefix.
> Default sorting would solve this issue.
> It would be great if default sorter can be specified for *FileConsumer* in
> the pollDirectory method:
> {code:java}
> log.trace("Polling directory: {}", directory.getPath());
> File[] dirFiles = directory.listFiles();
> if (dirFiles == null || dirFiles.length == 0) {
> // no files in this directory to poll
> if (log.isTraceEnabled()) {
> log.trace("No files found in directory: {}",
> directory.getPath());
> }
> return true;
> } else {
> // we found some files
> if (log.isTraceEnabled()) {
> log.trace("Found {} in directory: {}", dirFiles.length,
> directory.getPath());
> }
> }
> List files = Arrays.asList(dirFiles);
>// sort there?
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CAMEL-11656) Support default directory sorter for FileConsumer
[
https://issues.apache.org/jira/browse/CAMEL-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16262499#comment-16262499
]
ASF GitHub Bot commented on CAMEL-11656:
onders86 opened a new pull request #2108: CAMEL-11656 - add preSort option
URL: https://github.com/apache/camel/pull/2108
even though i believe this issue is not related to sorting file list
beforehand, i am just opening the PR to discuss further and revise...
@davsclaus could you review?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Support default directory sorter for FileConsumer
> -
>
> Key: CAMEL-11656
> URL: https://issues.apache.org/jira/browse/CAMEL-11656
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
>Affects Versions: 2.19.0
>Reporter: Viktor Khoroshko
>Priority: Minor
> Fix For: 2.21.0
>
>
> Hello,
> The current way of sorting consumed files isn't very flexible due to how it's
> implemented:
> Files are sorted AFTER they're added to in progress repository what in
> combination with *maxMessagesPerPoll* set to be > 0 and
> *eagerLimitMaxMessagesPerPoll* set to false forces the
> *removeExcessiveInProgressFiles* call - which in case of persistent
> idempotent repository used causes redundant calls to a database.
> This is not only the issue.
> In my case I have a logic in a custom file filter that a file shouldn't be
> processed if a file with a same prefix is already in progress while still
> those files should be processed in a specified order.
> The issue is that since sorting is performed after each file is added to in
> progress repository and before added there a file should be accepted by a
> filter but it will be not - as a previous file with a same prefix has
> already been added to in progress repo.
> *Example*: files - test.001, test.002, test.003
> If test.003 is first in a returned file list then it will be added to in
> progress repository and next files will not be accepted due to the same
> prefix.
> Default sorting would solve this issue.
> It would be great if default sorter can be specified for *FileConsumer* in
> the pollDirectory method:
> {code:java}
> log.trace("Polling directory: {}", directory.getPath());
> File[] dirFiles = directory.listFiles();
> if (dirFiles == null || dirFiles.length == 0) {
> // no files in this directory to poll
> if (log.isTraceEnabled()) {
> log.trace("No files found in directory: {}",
> directory.getPath());
> }
> return true;
> } else {
> // we found some files
> if (log.isTraceEnabled()) {
> log.trace("Found {} in directory: {}", dirFiles.length,
> directory.getPath());
> }
> }
> List files = Arrays.asList(dirFiles);
>// sort there?
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CAMEL-11656) Support default directory sorter for FileConsumer
[
https://issues.apache.org/jira/browse/CAMEL-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16262500#comment-16262500
]
ASF GitHub Bot commented on CAMEL-11656:
GitHub user onders86 opened a pull request:
https://github.com/apache/camel/pull/2108
CAMEL-11656 - add preSort option
even though i believe this issue is not related to sorting file list
beforehand, i am just opening the PR to discuss further and revise...
@davsclaus could you review?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/onders86/camel CAMEL-11656
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/camel/pull/2108.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2108
commit b7e51f1a3c1aec4be3b5ae546a8ef74ef25f3e7b
Author: onders86
Date: 2017-11-22T11:52:41Z
CAMEL-11656 - add preSort option
> Support default directory sorter for FileConsumer
> -
>
> Key: CAMEL-11656
> URL: https://issues.apache.org/jira/browse/CAMEL-11656
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
>Affects Versions: 2.19.0
>Reporter: Viktor Khoroshko
>Priority: Minor
> Fix For: 2.21.0
>
>
> Hello,
> The current way of sorting consumed files isn't very flexible due to how it's
> implemented:
> Files are sorted AFTER they're added to in progress repository what in
> combination with *maxMessagesPerPoll* set to be > 0 and
> *eagerLimitMaxMessagesPerPoll* set to false forces the
> *removeExcessiveInProgressFiles* call - which in case of persistent
> idempotent repository used causes redundant calls to a database.
> This is not only the issue.
> In my case I have a logic in a custom file filter that a file shouldn't be
> processed if a file with a same prefix is already in progress while still
> those files should be processed in a specified order.
> The issue is that since sorting is performed after each file is added to in
> progress repository and before added there a file should be accepted by a
> filter but it will be not - as a previous file with a same prefix has
> already been added to in progress repo.
> *Example*: files - test.001, test.002, test.003
> If test.003 is first in a returned file list then it will be added to in
> progress repository and next files will not be accepted due to the same
> prefix.
> Default sorting would solve this issue.
> It would be great if default sorter can be specified for *FileConsumer* in
> the pollDirectory method:
> {code:java}
> log.trace("Polling directory: {}", directory.getPath());
> File[] dirFiles = directory.listFiles();
> if (dirFiles == null || dirFiles.length == 0) {
> // no files in this directory to poll
> if (log.isTraceEnabled()) {
> log.trace("No files found in directory: {}",
> directory.getPath());
> }
> return true;
> } else {
> // we found some files
> if (log.isTraceEnabled()) {
> log.trace("Found {} in directory: {}", dirFiles.length,
> directory.getPath());
> }
> }
> List files = Arrays.asList(dirFiles);
>// sort there?
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CAMEL-11656) Support default directory sorter for FileConsumer
[
https://issues.apache.org/jira/browse/CAMEL-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16262512#comment-16262512
]
ASF GitHub Bot commented on CAMEL-11656:
oscerd commented on issue #2108: CAMEL-11656 - add preSort option
URL: https://github.com/apache/camel/pull/2108#issuecomment-346350161
In my opinion it can be good to have this, but we need to avoid the options
number explosion in this component. Lets wait for @davsclaus
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Support default directory sorter for FileConsumer
> -
>
> Key: CAMEL-11656
> URL: https://issues.apache.org/jira/browse/CAMEL-11656
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
>Affects Versions: 2.19.0
>Reporter: Viktor Khoroshko
>Priority: Minor
> Fix For: 2.21.0
>
>
> Hello,
> The current way of sorting consumed files isn't very flexible due to how it's
> implemented:
> Files are sorted AFTER they're added to in progress repository what in
> combination with *maxMessagesPerPoll* set to be > 0 and
> *eagerLimitMaxMessagesPerPoll* set to false forces the
> *removeExcessiveInProgressFiles* call - which in case of persistent
> idempotent repository used causes redundant calls to a database.
> This is not only the issue.
> In my case I have a logic in a custom file filter that a file shouldn't be
> processed if a file with a same prefix is already in progress while still
> those files should be processed in a specified order.
> The issue is that since sorting is performed after each file is added to in
> progress repository and before added there a file should be accepted by a
> filter but it will be not - as a previous file with a same prefix has
> already been added to in progress repo.
> *Example*: files - test.001, test.002, test.003
> If test.003 is first in a returned file list then it will be added to in
> progress repository and next files will not be accepted due to the same
> prefix.
> Default sorting would solve this issue.
> It would be great if default sorter can be specified for *FileConsumer* in
> the pollDirectory method:
> {code:java}
> log.trace("Polling directory: {}", directory.getPath());
> File[] dirFiles = directory.listFiles();
> if (dirFiles == null || dirFiles.length == 0) {
> // no files in this directory to poll
> if (log.isTraceEnabled()) {
> log.trace("No files found in directory: {}",
> directory.getPath());
> }
> return true;
> } else {
> // we found some files
> if (log.isTraceEnabled()) {
> log.trace("Found {} in directory: {}", dirFiles.length,
> directory.getPath());
> }
> }
> List files = Arrays.asList(dirFiles);
>// sort there?
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CAMEL-11656) Support default directory sorter for FileConsumer
[
https://issues.apache.org/jira/browse/CAMEL-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263870#comment-16263870
]
ASF GitHub Bot commented on CAMEL-11656:
onders86 commented on issue #2108: CAMEL-11656 - add preSort option
URL: https://github.com/apache/camel/pull/2108#issuecomment-346538996
yes sure.updated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Support default directory sorter for FileConsumer
> -
>
> Key: CAMEL-11656
> URL: https://issues.apache.org/jira/browse/CAMEL-11656
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
>Affects Versions: 2.19.0
>Reporter: Viktor Khoroshko
>Priority: Minor
> Fix For: 2.21.0
>
>
> Hello,
> The current way of sorting consumed files isn't very flexible due to how it's
> implemented:
> Files are sorted AFTER they're added to in progress repository what in
> combination with *maxMessagesPerPoll* set to be > 0 and
> *eagerLimitMaxMessagesPerPoll* set to false forces the
> *removeExcessiveInProgressFiles* call - which in case of persistent
> idempotent repository used causes redundant calls to a database.
> This is not only the issue.
> In my case I have a logic in a custom file filter that a file shouldn't be
> processed if a file with a same prefix is already in progress while still
> those files should be processed in a specified order.
> The issue is that since sorting is performed after each file is added to in
> progress repository and before added there a file should be accepted by a
> filter but it will be not - as a previous file with a same prefix has
> already been added to in progress repo.
> *Example*: files - test.001, test.002, test.003
> If test.003 is first in a returned file list then it will be added to in
> progress repository and next files will not be accepted due to the same
> prefix.
> Default sorting would solve this issue.
> It would be great if default sorter can be specified for *FileConsumer* in
> the pollDirectory method:
> {code:java}
> log.trace("Polling directory: {}", directory.getPath());
> File[] dirFiles = directory.listFiles();
> if (dirFiles == null || dirFiles.length == 0) {
> // no files in this directory to poll
> if (log.isTraceEnabled()) {
> log.trace("No files found in directory: {}",
> directory.getPath());
> }
> return true;
> } else {
> // we found some files
> if (log.isTraceEnabled()) {
> log.trace("Found {} in directory: {}", dirFiles.length,
> directory.getPath());
> }
> }
> List files = Arrays.asList(dirFiles);
>// sort there?
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CAMEL-11656) Support default directory sorter for FileConsumer
[
https://issues.apache.org/jira/browse/CAMEL-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263928#comment-16263928
]
ASF GitHub Bot commented on CAMEL-11656:
davsclaus commented on issue #2108: CAMEL-11656 - add preSort option
URL: https://github.com/apache/camel/pull/2108#issuecomment-346549840
Maybe we can improved the documentation a bit, something like:
```
When pre-sort is enabled then the consumer will sort the file and directory
names during polling, that was retrieved from the file system. You may want to
do this in case you need to operate on the files in a sorted order. The
pre-sort is executed before the consumer starts to filter, and accept files to
process by Camel. This option is default disabled.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Support default directory sorter for FileConsumer
> -
>
> Key: CAMEL-11656
> URL: https://issues.apache.org/jira/browse/CAMEL-11656
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
>Affects Versions: 2.19.0
>Reporter: Viktor Khoroshko
>Priority: Minor
> Fix For: 2.21.0
>
>
> Hello,
> The current way of sorting consumed files isn't very flexible due to how it's
> implemented:
> Files are sorted AFTER they're added to in progress repository what in
> combination with *maxMessagesPerPoll* set to be > 0 and
> *eagerLimitMaxMessagesPerPoll* set to false forces the
> *removeExcessiveInProgressFiles* call - which in case of persistent
> idempotent repository used causes redundant calls to a database.
> This is not only the issue.
> In my case I have a logic in a custom file filter that a file shouldn't be
> processed if a file with a same prefix is already in progress while still
> those files should be processed in a specified order.
> The issue is that since sorting is performed after each file is added to in
> progress repository and before added there a file should be accepted by a
> filter but it will be not - as a previous file with a same prefix has
> already been added to in progress repo.
> *Example*: files - test.001, test.002, test.003
> If test.003 is first in a returned file list then it will be added to in
> progress repository and next files will not be accepted due to the same
> prefix.
> Default sorting would solve this issue.
> It would be great if default sorter can be specified for *FileConsumer* in
> the pollDirectory method:
> {code:java}
> log.trace("Polling directory: {}", directory.getPath());
> File[] dirFiles = directory.listFiles();
> if (dirFiles == null || dirFiles.length == 0) {
> // no files in this directory to poll
> if (log.isTraceEnabled()) {
> log.trace("No files found in directory: {}",
> directory.getPath());
> }
> return true;
> } else {
> // we found some files
> if (log.isTraceEnabled()) {
> log.trace("Found {} in directory: {}", dirFiles.length,
> directory.getPath());
> }
> }
> List files = Arrays.asList(dirFiles);
>// sort there?
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CAMEL-11656) Support default directory sorter for FileConsumer
[
https://issues.apache.org/jira/browse/CAMEL-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16264336#comment-16264336
]
ASF GitHub Bot commented on CAMEL-11656:
onders86 closed pull request #2108: CAMEL-11656 - add preSort option
URL: https://github.com/apache/camel/pull/2108
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/camel-core/src/main/docs/file-component.adoc
b/camel-core/src/main/docs/file-component.adoc
index 8aa0247103e..294ced12221 100644
--- a/camel-core/src/main/docs/file-component.adoc
+++ b/camel-core/src/main/docs/file-component.adoc
@@ -69,7 +69,7 @@ with the following path and query parameters:
| *directoryName* | *Required* The starting directory | | File
|===
- Query Parameters (80 parameters):
+ Query Parameters (81 parameters):
[width="100%",cols="2,5,^1,2",options="header"]
|===
@@ -82,6 +82,7 @@ with the following path and query parameters:
| *moveFailed* (consumer) | Sets the move failure expression based on Simple
language. For example to move files into a .error subdirectory use: .error.
Note: When moving the files to the fail location Camel will handle the error
and will not pick up the file again. | | String
| *noop* (consumer) | If true the file is not moved or deleted in any way.
This option is good for readonly data or for ETL type requirements. If
noop=true Camel will set idempotent=true as well to avoid consuming the same
files over and over again. | false | boolean
| *preMove* (consumer) | Expression (such as File Language) used to
dynamically set the filename when moving it before processing. For example to
move in-progress files into the order directory set this value to order. | |
String
+| *preSort* (consumer) | When pre-sort is enabled then the consumer will sort
the file and directory names during polling that was retrieved from the file
system. You may want to do this in case you need to operate on the files in a
sorted order. The pre-sort is executed before the consumer starts to filter and
accept files to process by Camel. This option is default=false meaning
disabled. | false | boolean
| *recursive* (consumer) | If a directory will look for files in all the
sub-directories as well. | false | boolean
| *sendEmptyMessageWhenIdle* (consumer) | If the polling consumer did not poll
any files you can enable this option to send an empty message (no body)
instead. | false | boolean
| *directoryMustExist* (consumer) | Similar to startingDirectoryMustExist but
this applies during polling recursive sub directories. | false | boolean
diff --git
a/camel-core/src/main/java/org/apache/camel/component/file/FileConsumer.java
b/camel-core/src/main/java/org/apache/camel/component/file/FileConsumer.java
index 5e3a180bc50..aa3dcfafc2e 100644
--- a/camel-core/src/main/java/org/apache/camel/component/file/FileConsumer.java
+++ b/camel-core/src/main/java/org/apache/camel/component/file/FileConsumer.java
@@ -21,11 +21,13 @@
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Arrays;
+import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
+import java.util.stream.Collectors;
import org.apache.camel.Exchange;
import org.apache.camel.Message;
@@ -84,6 +86,9 @@ protected boolean pollDirectory(String fileName,
List> fileLis
}
}
List files = Arrays.asList(dirFiles);
+if (getEndpoint().isPreSort()) {
+Collections.sort(files, (a, b) ->
a.getAbsoluteFile().compareTo(a.getAbsoluteFile()));
+}
for (File file : dirFiles) {
// check if we can continue polling in files
diff --git
a/camel-core/src/main/java/org/apache/camel/component/file/GenericFileEndpoint.java
b/camel-core/src/main/java/org/apache/camel/component/file/GenericFileEndpoint.java
index ec4c0c5c6e6..cc7c7020689 100644
---
a/camel-core/src/main/java/org/apache/camel/component/file/GenericFileEndpoint.java
+++
b/camel-core/src/main/java/org/apache/camel/component/file/GenericFileEndpoint.java
@@ -111,6 +111,8 @@
protected boolean recursive;
@UriParam(label = "consumer")
protected boolean delete;
+@UriParam(label = "consumer")
+protected boolean preSort;
@UriParam(label = "consumer,filter")
protected int maxMessagesPerPoll;
@UriParam(label = "consumer,filter", defaultValue = "true")
@@ -414,6 +416,20 @@ public void setAntFilterCaseSensitive(boolean
antFilterCaseSensitive) {
public GenericFileFilter getAntFilter() {
return antFilter;
}
+
+public boolean isPreSort() {
+return p
[jira] [Commented] (CAMEL-11656) Support default directory sorter for FileConsumer
[
https://issues.apache.org/jira/browse/CAMEL-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16264337#comment-16264337
]
ASF GitHub Bot commented on CAMEL-11656:
Github user onders86 closed the pull request at:
https://github.com/apache/camel/pull/2108
> Support default directory sorter for FileConsumer
> -
>
> Key: CAMEL-11656
> URL: https://issues.apache.org/jira/browse/CAMEL-11656
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
>Affects Versions: 2.19.0
>Reporter: Viktor Khoroshko
>Priority: Minor
> Fix For: 2.21.0
>
>
> Hello,
> The current way of sorting consumed files isn't very flexible due to how it's
> implemented:
> Files are sorted AFTER they're added to in progress repository what in
> combination with *maxMessagesPerPoll* set to be > 0 and
> *eagerLimitMaxMessagesPerPoll* set to false forces the
> *removeExcessiveInProgressFiles* call - which in case of persistent
> idempotent repository used causes redundant calls to a database.
> This is not only the issue.
> In my case I have a logic in a custom file filter that a file shouldn't be
> processed if a file with a same prefix is already in progress while still
> those files should be processed in a specified order.
> The issue is that since sorting is performed after each file is added to in
> progress repository and before added there a file should be accepted by a
> filter but it will be not - as a previous file with a same prefix has
> already been added to in progress repo.
> *Example*: files - test.001, test.002, test.003
> If test.003 is first in a returned file list then it will be added to in
> progress repository and next files will not be accepted due to the same
> prefix.
> Default sorting would solve this issue.
> It would be great if default sorter can be specified for *FileConsumer* in
> the pollDirectory method:
> {code:java}
> log.trace("Polling directory: {}", directory.getPath());
> File[] dirFiles = directory.listFiles();
> if (dirFiles == null || dirFiles.length == 0) {
> // no files in this directory to poll
> if (log.isTraceEnabled()) {
> log.trace("No files found in directory: {}",
> directory.getPath());
> }
> return true;
> } else {
> // we found some files
> if (log.isTraceEnabled()) {
> log.trace("Found {} in directory: {}", dirFiles.length,
> directory.getPath());
> }
> }
> List files = Arrays.asList(dirFiles);
>// sort there?
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CAMEL-11656) Support default directory sorter for FileConsumer
[
https://issues.apache.org/jira/browse/CAMEL-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16264437#comment-16264437
]
ASF GitHub Bot commented on CAMEL-11656:
oscerd commented on issue #2108: CAMEL-11656 - add preSort option
URL: https://github.com/apache/camel/pull/2108#issuecomment-346639149
@onders86 is there a specific reason to backport to 2.20.x? This is a new
option, so usually we avoid new options in minor releases
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Support default directory sorter for FileConsumer
> -
>
> Key: CAMEL-11656
> URL: https://issues.apache.org/jira/browse/CAMEL-11656
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
>Affects Versions: 2.19.0
>Reporter: Viktor Khoroshko
>Priority: Minor
> Fix For: 2.21.0
>
>
> Hello,
> The current way of sorting consumed files isn't very flexible due to how it's
> implemented:
> Files are sorted AFTER they're added to in progress repository what in
> combination with *maxMessagesPerPoll* set to be > 0 and
> *eagerLimitMaxMessagesPerPoll* set to false forces the
> *removeExcessiveInProgressFiles* call - which in case of persistent
> idempotent repository used causes redundant calls to a database.
> This is not only the issue.
> In my case I have a logic in a custom file filter that a file shouldn't be
> processed if a file with a same prefix is already in progress while still
> those files should be processed in a specified order.
> The issue is that since sorting is performed after each file is added to in
> progress repository and before added there a file should be accepted by a
> filter but it will be not - as a previous file with a same prefix has
> already been added to in progress repo.
> *Example*: files - test.001, test.002, test.003
> If test.003 is first in a returned file list then it will be added to in
> progress repository and next files will not be accepted due to the same
> prefix.
> Default sorting would solve this issue.
> It would be great if default sorter can be specified for *FileConsumer* in
> the pollDirectory method:
> {code:java}
> log.trace("Polling directory: {}", directory.getPath());
> File[] dirFiles = directory.listFiles();
> if (dirFiles == null || dirFiles.length == 0) {
> // no files in this directory to poll
> if (log.isTraceEnabled()) {
> log.trace("No files found in directory: {}",
> directory.getPath());
> }
> return true;
> } else {
> // we found some files
> if (log.isTraceEnabled()) {
> log.trace("Found {} in directory: {}", dirFiles.length,
> directory.getPath());
> }
> }
> List files = Arrays.asList(dirFiles);
>// sort there?
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CAMEL-11656) Support default directory sorter for FileConsumer
[
https://issues.apache.org/jira/browse/CAMEL-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16264688#comment-16264688
]
ASF GitHub Bot commented on CAMEL-11656:
onders86 commented on issue #2108: CAMEL-11656 - add preSort option
URL: https://github.com/apache/camel/pull/2108#issuecomment-346682083
@oscerd No there is not. I thought we would need to backport because this
was a wish since 2.19.0. I dont mind reverting. Let me know.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Support default directory sorter for FileConsumer
> -
>
> Key: CAMEL-11656
> URL: https://issues.apache.org/jira/browse/CAMEL-11656
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
>Affects Versions: 2.19.0
>Reporter: Viktor Khoroshko
>Priority: Minor
> Fix For: 2.21.0
>
>
> Hello,
> The current way of sorting consumed files isn't very flexible due to how it's
> implemented:
> Files are sorted AFTER they're added to in progress repository what in
> combination with *maxMessagesPerPoll* set to be > 0 and
> *eagerLimitMaxMessagesPerPoll* set to false forces the
> *removeExcessiveInProgressFiles* call - which in case of persistent
> idempotent repository used causes redundant calls to a database.
> This is not only the issue.
> In my case I have a logic in a custom file filter that a file shouldn't be
> processed if a file with a same prefix is already in progress while still
> those files should be processed in a specified order.
> The issue is that since sorting is performed after each file is added to in
> progress repository and before added there a file should be accepted by a
> filter but it will be not - as a previous file with a same prefix has
> already been added to in progress repo.
> *Example*: files - test.001, test.002, test.003
> If test.003 is first in a returned file list then it will be added to in
> progress repository and next files will not be accepted due to the same
> prefix.
> Default sorting would solve this issue.
> It would be great if default sorter can be specified for *FileConsumer* in
> the pollDirectory method:
> {code:java}
> log.trace("Polling directory: {}", directory.getPath());
> File[] dirFiles = directory.listFiles();
> if (dirFiles == null || dirFiles.length == 0) {
> // no files in this directory to poll
> if (log.isTraceEnabled()) {
> log.trace("No files found in directory: {}",
> directory.getPath());
> }
> return true;
> } else {
> // we found some files
> if (log.isTraceEnabled()) {
> log.trace("Found {} in directory: {}", dirFiles.length,
> directory.getPath());
> }
> }
> List files = Arrays.asList(dirFiles);
>// sort there?
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CAMEL-11656) Support default directory sorter for FileConsumer
[
https://issues.apache.org/jira/browse/CAMEL-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16264745#comment-16264745
]
ASF GitHub Bot commented on CAMEL-11656:
onders86 commented on issue #2108: CAMEL-11656 - add preSort option
URL: https://github.com/apache/camel/pull/2108#issuecomment-346692161
@oscerd , reverted.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Support default directory sorter for FileConsumer
> -
>
> Key: CAMEL-11656
> URL: https://issues.apache.org/jira/browse/CAMEL-11656
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
>Affects Versions: 2.19.0
>Reporter: Viktor Khoroshko
>Priority: Minor
> Fix For: 2.21.0
>
>
> Hello,
> The current way of sorting consumed files isn't very flexible due to how it's
> implemented:
> Files are sorted AFTER they're added to in progress repository what in
> combination with *maxMessagesPerPoll* set to be > 0 and
> *eagerLimitMaxMessagesPerPoll* set to false forces the
> *removeExcessiveInProgressFiles* call - which in case of persistent
> idempotent repository used causes redundant calls to a database.
> This is not only the issue.
> In my case I have a logic in a custom file filter that a file shouldn't be
> processed if a file with a same prefix is already in progress while still
> those files should be processed in a specified order.
> The issue is that since sorting is performed after each file is added to in
> progress repository and before added there a file should be accepted by a
> filter but it will be not - as a previous file with a same prefix has
> already been added to in progress repo.
> *Example*: files - test.001, test.002, test.003
> If test.003 is first in a returned file list then it will be added to in
> progress repository and next files will not be accepted due to the same
> prefix.
> Default sorting would solve this issue.
> It would be great if default sorter can be specified for *FileConsumer* in
> the pollDirectory method:
> {code:java}
> log.trace("Polling directory: {}", directory.getPath());
> File[] dirFiles = directory.listFiles();
> if (dirFiles == null || dirFiles.length == 0) {
> // no files in this directory to poll
> if (log.isTraceEnabled()) {
> log.trace("No files found in directory: {}",
> directory.getPath());
> }
> return true;
> } else {
> // we found some files
> if (log.isTraceEnabled()) {
> log.trace("Found {} in directory: {}", dirFiles.length,
> directory.getPath());
> }
> }
> List files = Arrays.asList(dirFiles);
>// sort there?
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
