[jira] [Commented] (BEAM-9270) Unable to use ElasticsearchIO for indexpatterns: index-*
[ https://issues.apache.org/jira/browse/BEAM-9270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17548423#comment-17548423 ] Danny McCormick commented on BEAM-9270: --- This issue has been migrated to https://github.com/apache/beam/issues/20055 > Unable to use ElasticsearchIO for indexpatterns: index-* > > > Key: BEAM-9270 > URL: https://issues.apache.org/jira/browse/BEAM-9270 > Project: Beam > Issue Type: Bug > Components: io-java-elasticsearch >Affects Versions: 2.19.0 >Reporter: Krishnaiah Narukulla >Priority: P3 > > ElasticsearchIO input doesnot work with index patterns with wildcard. for > example: index-*. I It works well with single index. It becomes problem if > we need to query multiple indices using index pattern. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (BEAM-9270) Unable to use ElasticsearchIO for indexpatterns: index-*
[ https://issues.apache.org/jira/browse/BEAM-9270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17136889#comment-17136889 ] Beam JIRA Bot commented on BEAM-9270: - This issue was marked "stale-P2" and has not received a public comment in 14 days. It is now automatically moved to P3. If you are still affected by it, you can comment and move it back to P2. > Unable to use ElasticsearchIO for indexpatterns: index-* > > > Key: BEAM-9270 > URL: https://issues.apache.org/jira/browse/BEAM-9270 > Project: Beam > Issue Type: Bug > Components: io-java-elasticsearch >Affects Versions: 2.19.0 >Reporter: Krishnaiah Narukulla >Priority: P3 > > ElasticsearchIO input doesnot work with index patterns with wildcard. for > example: index-*. I It works well with single index. It becomes problem if > we need to query multiple indices using index pattern. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9270) Unable to use ElasticsearchIO for indexpatterns: index-*
[ https://issues.apache.org/jira/browse/BEAM-9270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17122385#comment-17122385 ] Beam JIRA Bot commented on BEAM-9270: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Unable to use ElasticsearchIO for indexpatterns: index-* > > > Key: BEAM-9270 > URL: https://issues.apache.org/jira/browse/BEAM-9270 > Project: Beam > Issue Type: Bug > Components: io-java-elasticsearch >Affects Versions: 2.19.0 >Reporter: Krishnaiah Narukulla >Priority: P2 > Labels: stale-P2 > > ElasticsearchIO input doesnot work with index patterns with wildcard. for > example: index-*. I It works well with single index. It becomes problem if > we need to query multiple indices using index pattern. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9270) Unable to use ElasticsearchIO for indexpatterns: index-*
[ https://issues.apache.org/jira/browse/BEAM-9270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17033930#comment-17033930 ] Ludovic Boutros commented on BEAM-9270: --- Hi [~krisnaru], as [~echauchot] already pointed it, there is an ongoing complete refactoring of the module which should be able to handle this type of usage in a better way. That said, I did not have any time to test it but would an alias on the indices work ? > Unable to use ElasticsearchIO for indexpatterns: index-* > > > Key: BEAM-9270 > URL: https://issues.apache.org/jira/browse/BEAM-9270 > Project: Beam > Issue Type: Bug > Components: io-java-elasticsearch >Affects Versions: 2.19.0 >Reporter: Krishnaiah Narukulla >Priority: Major > > ElasticsearchIO input doesnot work with index patterns with wildcard. for > example: index-*. I It works well with single index. It becomes problem if > we need to query multiple indices using index pattern. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9270) Unable to use ElasticsearchIO for indexpatterns: index-*
[ https://issues.apache.org/jira/browse/BEAM-9270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17033716#comment-17033716 ] Etienne Chauchot commented on BEAM-9270: I did some more testing: the problem is really restricted to only the size estimation. Once got a proper estimation, then the read *index** works fine > Unable to use ElasticsearchIO for indexpatterns: index-* > > > Key: BEAM-9270 > URL: https://issues.apache.org/jira/browse/BEAM-9270 > Project: Beam > Issue Type: Bug > Components: io-java-elasticsearch >Affects Versions: 2.19.0 >Reporter: Krishnaiah Narukulla >Priority: Major > > ElasticsearchIO input doesnot work with index patterns with wildcard. for > example: index-*. I It works well with single index. It becomes problem if > we need to query multiple indices using index pattern. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9270) Unable to use ElasticsearchIO for indexpatterns: index-*
[ https://issues.apache.org/jira/browse/BEAM-9270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17033655#comment-17033655 ] Etienne Chauchot commented on BEAM-9270: Hi [~krisnaru], thannks for raising this ! I can indeed reproduce this bug. What happens is that: Beam Read needs to evaluate the size of the data it needs to read to know how to split for parallel read. To get the size of data the ElasticsearchIO calls the stats API of ES. Right now the IO uses a low level ES REST client. So it gets a json object from ES cluster listing all the indices available in the cluster. The IO then searches for the index name provided by the user in the json. In that case it does not find a match with the "*" character and considers that the provided index does not exist and that the size is 0 and returns no document. To fix that it would require to update the json parsing by manually expanding the wildcard. But there is an ongoing complete refactoring of ElasticsearchIO (see the thread on the dev mailing list) that would allow to use high level ES client and that will unlock many ES features. So either we do a fix to match the index name in the stats using a regExpr or we wait for the new ElasticsearchIO. > Unable to use ElasticsearchIO for indexpatterns: index-* > > > Key: BEAM-9270 > URL: https://issues.apache.org/jira/browse/BEAM-9270 > Project: Beam > Issue Type: Bug > Components: io-java-elasticsearch >Affects Versions: 2.19.0 >Reporter: Krishnaiah Narukulla >Priority: Major > > ElasticsearchIO input doesnot work with index patterns with wildcard. for > example: index-*. I It works well with single index. It becomes problem if > we need to query multiple indices using index pattern. -- This message was sent by Atlassian Jira (v8.3.4#803005)
