[GitHub] [flink] rmetzger commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-801122674 Very nice! Thanks a lot for your efforts! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-796665243 I was concerned that this would happen to you. A new hire in our company is facing the same issue. Looks like Azure is having some issues with cryptocurrency mining on their Infra, but they lack the proper infra to detect it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-794477315 It failed again: https://dev.azure.com/rmetzger/Flink/_build/results?buildId=8955=logs=9401bf33-03c4-5a24-83fe-e51d75db73ef=72901ab2-7cd0-57be-82b1-bca51de20fba This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-794286699 https://dev.azure.com/rmetzger/Flink/_build/results?buildId=8955=results This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-794285369 Here's how to setup azure: https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-794233139 Have you setup your CI with the password as well, and verified the change? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-793577861 > Why the s3 access key is need to be added? Because some S3 end to end tests work against the real S3 service, not some mock or "fake" service. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-793576430 Sorry, I meant ``` SECRET_GLUE_SCHEMA_ACCESS_KEY: $[variables.IT_CASE_GLUE_SCHEMA_ACCESS_KEY] SECRET_GLUE_SCHEMA_SECRET_KEY: $[variables.IT_CASE_GLUE_SCHEMA_SECRET_KEY] ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-793560044 The `-z` approach should work: https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/common_s3.sh#L25 but it's worth a try. Can you also add ``` SECRET_S3_ACCESS_KEY: $[variables.IT_CASE_S3_ACCESS_KEY] SECRET_S3_SECRET_KEY: $[variables.IT_CASE_S3_SECRET_KEY] ``` to `build-apache-repo.yml`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-793518186 Oh, I didn't notice that the main CI isn't working either. The condition looks fine. Maybe the problem is that we are not setting the environment variables here: https://github.com/apache/flink/blob/master/tools/azure-pipelines/build-apache-repo.yml#L51 The build-apache-repo.yml definition is used for triggering the PR builds. Maybe azure is putting the variable name if the variable is not defined (hence the test is executed in the PR Ci build because the variable is replaced by the variable name, and the test is executed on my personal ci, because there the correct variable is set). > Why we need to pass the value of $IT_CASE_GLUE_SCHEMA_ACCESS_KEY to $SECRET_GLUE_SCHEMA_ACCESS_KEY? These are the secret variables I have defined in my personal CI:  The definitions in `azure-pipelines.yml` get the defined variables (for example `variables.IT_CASE_GLUE_SCHEMA_ACCESS_KEY` and store them in an internal, temporary variable (for example `SECRET_GLUE_SCHEMA_ACCESS_KEY`), then in `jobs-template.yml` we make the environment variable available again, under the original name: `IT_CASE_GLUE_SCHEMA_ACCESS_KEY: $(SECRET_GLUE_SCHEMA_ACCESS_KEY)`. Why this weird transformation from "defined variable" to "internal variable" back to "environment variable"? I'm not 100% sure anymore, but I remember that I've struggled a lot setting up the secrets. What I know is that you need to manually declare the secret variables on the respective Azure tasks like this: ```yaml env: IT_CASE_S3_BUCKET: $(SECRET_S3_BUCKET) IT_CASE_S3_ACCESS_KEY: $(SECRET_S3_ACCESS_KEY) IT_CASE_S3_SECRET_KEY: $(SECRET_S3_SECRET_KEY) IT_CASE_GLUE_SCHEMA_ACCESS_KEY: $(SECRET_GLUE_SCHEMA_ACCESS_KEY) IT_CASE_GLUE_SCHEMA_SECRET_KEY: $(SECRET_GLUE_SCHEMA_SECRET_KEY) ``` ... otherwise they won't work. Afaik using the same name for the variable also didn't seem to have worked. What we know is that this approach works well for the S3 tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-793459808 My commit needs to be included into the PR so that it ends up in master. If this environment variable is not available (for example here during CI verification, or with personal Azure accounts that don't have this secret setup), it will be ignored. If the environment variable is set (for example on apache/flink master, my personal CI account or maybe your personal CI account), the tests are executing. What we need to figure out is why the env variables are not picked up from your test code in my CI environment. It seems that they are set, because the bash script "decided" to run the test. I recommend you to debug / resolve this issue on your personal azure CI account. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-793006791 There are no secrets setup for the main CI (which runs the pull request validation). We can not do this, because people could steal our credentials by opening a pull request exporting the secrets from the env variables. My Azure account has the secrets set up. I would recommend you to do the same on your personal Azure account (It's free). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-792650616 https://dev.azure.com/rmetzger/Flink/_build/results?buildId=8949=results This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-791962050 https://dev.azure.com/rmetzger/Flink/_build/results?buildId=8944=results This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-791455142 Also note that you need to include this commit if you want to make the secrets forwarding work: https://github.com/rmetzger/flink/commit/697c40cad14f42119604b3e754c4a52ede4a5c82 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-791348220 Looks like the test failed: https://dev.azure.com/rmetzger/Flink/_build/results?buildId=8940=logs=9401bf33-03c4-5a24-83fe-e51d75db73ef=72901ab2-7cd0-57be-82b1-bca51de20fba This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-791200406 Both CIs are green, however, the e2e test didn't execute on my CI. I'll quickly try to fix that. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-790878966 I pushed your rebased branch also again: https://dev.azure.com/rmetzger/Flink/_build/results?buildId=8938=results This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-790878304 yes, rebasing to the latest master should fix the issue! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-790381549 Thanks for your review @dannycranmer! Before we can merge this PR, we need to make sure that CI is passing without credentials (this is the case in the PR CI validation), and with credentials (my personal CI). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-790375946 I pushed this PR to my personal azure, where the CI credentials are provided: https://dev.azure.com/rmetzger/Flink/_build/results?buildId=8929=results This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-790375151 Looks like the e2e test is failing. Can you write the e2e test in a way that it only executes if the credentials are available? (personal Azure accounts won't have credentials) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-768331002 Thanks a lot for the clarification This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-766724268 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-767029789 You don't need to open a PR for every fix, you can just keep (force) pushing to the branch. Can you close the old PR? I'll review the PR tomorrow. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
rmetzger commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-766724268 What's the relationship of this PR to https://github.com/apache/flink/pull/14490 ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
