[GitHub] [flink] LinyuYao1021 commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
LinyuYao1021 commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-795179693 This is my pipeline, directly to run my commit to see what's the problem: - https://dev.azure.com/yaolinyu3547/yaoliny.flink/_build/results?buildId=1=results This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] LinyuYao1021 commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
LinyuYao1021 commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-794508193 > It failed again: https://dev.azure.com/rmetzger/Flink/_build/results?buildId=8955=logs=9401bf33-03c4-5a24-83fe-e51d75db73ef=72901ab2-7cd0-57be-82b1-bca51de20fba It's still because credentials can't be extracted. How did you success last time? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] LinyuYao1021 commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
LinyuYao1021 commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-794239401 > Have you setup your CI with the password as well, and verified the change? Not yet, I haven't used Azure before. Could you quickly walk me through what to do to set up CI? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] LinyuYao1021 commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
LinyuYao1021 commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-794238904 > password Not yet, I haven't used Azure before. Could you quickly walk me through what to do to set up CI? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] LinyuYao1021 commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
LinyuYao1021 commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-794169266 Hi Robert, main CI passed now. Would you please bring the latest commit to your personal CI to verify it and then we can close this PR? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] LinyuYao1021 commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
LinyuYao1021 commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-793624549 > SECRET_GLUE_SCHEMA_ACCESS_KEY: $[variables.IT_CASE_GLUE_SCHEMA_ACCESS_KEY] > SECRET_GLUE_SCHEMA_SECRET_KEY: $[variables.IT_CASE_GLUE_SCHEMA_SECRET_KEY] Added This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] LinyuYao1021 commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
LinyuYao1021 commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-793572293 > https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/common_s3.sh#L25 Why the s3 access key is need to be added? > The `-z` approach should work: https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/common_s3.sh#L25 > but it's worth a try. > > Can you also add > > ``` > SECRET_S3_ACCESS_KEY: $[variables.IT_CASE_S3_ACCESS_KEY] > SECRET_S3_SECRET_KEY: $[variables.IT_CASE_S3_SECRET_KEY] > ``` > > to `build-apache-repo.yml`? Why the s3 access key is needed for my case? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] LinyuYao1021 commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
LinyuYao1021 commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-793563537 > > It looks like the GSR test is running even when the creds are not set. Can we try setting: > > ``` > > if [ -n "$IT_CASE_GLUE_SCHEMA_ACCESS_KEY" ] && [ -n "$IT_CASE_GLUE_SCHEMA_SECRET_KEY" ]; then > > run_test "AWS Glue Schema Registry nightly end-to-end test" "$END_TO_END_DIR/test-scripts/test_glue_schema_registry.sh" > > fi > > ``` > > > > > > to > > ``` > > if [ "$IT_CASE_GLUE_SCHEMA_ACCESS_KEY" ] && [ "$IT_CASE_GLUE_SCHEMA_SECRET_KEY" ]; then > > run_test "AWS Glue Schema Registry nightly end-to-end test" "$END_TO_END_DIR/test-scripts/test_glue_schema_registry.sh" > > fi > > ``` > > Updated After diving into this, I find that `-n` means whether a string is empty, which is same with the length is zero. So we can use it for the condition check. The problem is we should use `[[ ]]` instead of `[ ]` to ensure > The `-z` approach should work: https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/common_s3.sh#L25 > but it's worth a try. > > Can you also add > > ``` > SECRET_S3_ACCESS_KEY: $[variables.IT_CASE_S3_ACCESS_KEY] > SECRET_S3_SECRET_KEY: $[variables.IT_CASE_S3_SECRET_KEY] > ``` > > to `build-apache-repo.yml`? Yes, I've added in the latest commit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] LinyuYao1021 commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
LinyuYao1021 commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-793552614 > What we know is that this approach works well for the S3 tests. Okay, I got it. Theoretically, variables transformation should work. Then the problem should still come from condition check. I've put the following code back to package in the latest commit. ``` SECRET_S3_ACCESS_KEY: $[variables.IT_CASE_S3_ACCESS_KEY] SECRET_S3_SECRET_KEY: $[variables.IT_CASE_S3_SECRET_KEY] ``` Meanwhile, I changed ``` if [ -n "$IT_CASE_GLUE_SCHEMA_ACCESS_KEY" ] && [ -n "$IT_CASE_GLUE_SCHEMA_SECRET_KEY" ]; then run_test "AWS Glue Schema Registry nightly end-to-end test" "$END_TO_END_DIR/test-scripts/test_glue_schema_registry.sh" fi ``` to ``` if [[ -n "$IT_CASE_GLUE_SCHEMA_ACCESS_KEY" ]] && [[ -n "$IT_CASE_GLUE_SCHEMA_SECRET_KEY" ]]; then run_test "AWS Glue Schema Registry nightly end-to-end test" "$END_TO_END_DIR/test-scripts/test_glue_schema_registry.sh" fi ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] LinyuYao1021 commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
LinyuYao1021 commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-793481182 > My commit needs to be included into the PR so that it ends up in master. > If this environment variable is not available (for example here during CI verification, or with personal Azure accounts that don't have this secret setup), it will be ignored. > If the environment variable is set (for example on apache/flink master, my personal CI account or maybe your personal CI account), the tests are executing. > > What we need to figure out is why the env variables are not picked up from your test code in my CI environment. It seems that they are set, because the bash script "decided" to run the test. I recommend you to debug / resolve this issue on your personal azure CI account. Currently, both main CI and your CI are not working. For main CI, it doesn't skip my e2e test. For your one, it can't pick the env variables. I dive deep in the secret forwarding and have a question about this [part](https://github.com/rmetzger/flink/commit/697c40cad14f42119604b3e754c4a52ede4a5c82#diff-7915b9b726a397ae7ba6af7b9703633d21c031ebf21682f3ee7e6a4ec52837a5R59). Why we need to pass the value of `$IT_CASE_GLUE_SCHEMA_ACCESS_KEY` to `$SECRET_GLUE_SCHEMA_ACCESS_KEY`? My understanding for secret forwarding is that we will have env variable `$SECRET_GLUE_SCHEMA_ACCESS_KEY` on personal CI and pass it to `$IT_CASE_GLUE_SCHEMA_ACCESS_KEY. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] LinyuYao1021 commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
LinyuYao1021 commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-793146783 > There are no secrets setup for the main CI (which runs the pull request validation). We can not do this, because people could steal our credentials by opening a pull request exporting the secrets from the env variables. > My Azure account has the secrets set up. > I would recommend you to do the same on your personal Azure account (It's free). So do you mean that I shouldn't include your [commit](https://github.com/rmetzger/flink/commit/697c40cad14f42119604b3e754c4a52ede4a5c82) to include secret forwarding in main CI? This commit is only for your or my personal testing? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] LinyuYao1021 commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
LinyuYao1021 commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-792941032 > https://dev.azure.com/rmetzger/Flink/_build/results?buildId=8949=results Hi Robert, can I check with you how's the secret forwarding working and would it affect the env variable of main CI? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] LinyuYao1021 commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
LinyuYao1021 commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-792643073 > It looks like the GSR test is running even when the creds are not set. Can we try setting: > > ``` > if [ -n "$IT_CASE_GLUE_SCHEMA_ACCESS_KEY" ] && [ -n "$IT_CASE_GLUE_SCHEMA_SECRET_KEY" ]; then > run_test "AWS Glue Schema Registry nightly end-to-end test" "$END_TO_END_DIR/test-scripts/test_glue_schema_registry.sh" > fi > ``` > > to > > ``` > if [ "$IT_CASE_GLUE_SCHEMA_ACCESS_KEY" ] && [ "$IT_CASE_GLUE_SCHEMA_SECRET_KEY" ]; then > run_test "AWS Glue Schema Registry nightly end-to-end test" "$END_TO_END_DIR/test-scripts/test_glue_schema_registry.sh" > fi > ``` Updated This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] LinyuYao1021 commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
LinyuYao1021 commented on pull request #14737:
URL: https://github.com/apache/flink/pull/14737#issuecomment-792069594
>
https://dev.azure.com/rmetzger/Flink/_build/results?buildId=8944=results
Hi Robert, the CI failed because it ran my e2e test. Is it because I added
the secrets forwarding work?
```
2021-03-06T03:20:28.9621676Z
org.apache.flink.client.program.ProgramInvocationException: The main method
caused an error: Exception occurred while fetching or registering schema
definition =
{"type":"record","name":"User","namespace":"org.apache.flink.glue.schema.registry.test","fields":[{"name":"name","type":"string"},{"name":"favorite_number","type":["int","null"]},{"name":"favorite_color","type":["string","null"]}]},
schema name = gsr-input-stream
2021-03-06T03:20:28.9625458Z at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
2021-03-06T03:20:28.9629120Z at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
2021-03-06T03:20:28.9631833Z at
org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
2021-03-06T03:20:28.9634712Z at
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
2021-03-06T03:20:28.9637432Z at
org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
2021-03-06T03:20:28.9640102Z at
org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
2021-03-06T03:20:28.9642641Z at
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
2021-03-06T03:20:28.9645230Z at
org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
2021-03-06T03:20:28.9647741Z at
org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
2021-03-06T03:20:28.9651870Z Caused by:
com.amazonaws.services.schemaregistry.exception.AWSSchemaRegistryException:
Exception occurred while fetching or registering schema definition =
{"type":"record","name":"User","namespace":"org.apache.flink.glue.schema.registry.test","fields":[{"name":"name","type":"string"},{"name":"favorite_number","type":["int","null"]},{"name":"favorite_color","type":["string","null"]}]},
schema name = gsr-input-stream
2021-03-06T03:20:28.9655740Z at
com.amazonaws.services.schemaregistry.common.AWSSchemaRegistryClient.getORRegisterSchemaVersionId(AWSSchemaRegistryClient.java:190)
2021-03-06T03:20:28.9658830Z at
com.amazonaws.services.schemaregistry.serializers.GlueSchemaRegistrySerializationFacade.getOrRegisterSchemaVersion(GlueSchemaRegistrySerializationFacade.java:84)
2021-03-06T03:20:28.9662027Z at
com.amazonaws.services.schemaregistry.serializers.avro.AWSAvroSerializer.registerSchema(AWSAvroSerializer.java:59)
2021-03-06T03:20:28.9665706Z at
org.apache.flink.glue.schema.registry.test.GSRKinesisPubsubClient.sendMessage(GSRKinesisPubsubClient.java:103)
2021-03-06T03:20:28.9669840Z at
org.apache.flink.glue.schema.registry.test.GlueSchemaRegistryExampleTest.main(GlueSchemaRegistryExampleTest.java:73)
2021-03-06T03:20:28.9672485Z at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2021-03-06T03:20:28.9693855Z at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2021-03-06T03:20:28.9694668Z at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2021-03-06T03:20:28.9695132Z at
java.lang.reflect.Method.invoke(Method.java:498)
2021-03-06T03:20:28.9695580Z at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
2021-03-06T03:20:28.9695953Z ... 8 more
2021-03-06T03:20:28.9696928Z Caused by:
com.amazonaws.services.schemaregistry.exception.AWSSchemaRegistryException:
Failed to get schemaVersionId by schema definition for schema name =
gsr-input-stream
2021-03-06T03:20:28.9697831Z at
com.amazonaws.services.schemaregistry.common.AWSSchemaRegistryClient.getSchemaVersionIdByDefinition(AWSSchemaRegistryClient.java:136)
2021-03-06T03:20:28.9698500Z at
com.amazonaws.services.schemaregistry.common.AWSSchemaRegistryClient.getORRegisterSchemaVersionId(AWSSchemaRegistryClient.java:167)
2021-03-06T03:20:28.9698930Z ... 17 more
2021-03-06T03:20:28.9702818Z Caused by:
software.amazon.awssdk.core.exception.SdkClientException: Unable to load
credentials from any of the providers in the chain
AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(),
EnvironmentVariableCredentialsProvider(),
WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(),
ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) :
[SystemPropertyCredentialsProvider(): Unable to load credentials from
[GitHub] [flink] LinyuYao1021 commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
LinyuYao1021 commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-791255225 > Both CIs are green, however, the e2e test didn't execute on my CI. I'll quickly try to fix that. Thanks Robert. Please give me a feedback when it's working. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] LinyuYao1021 commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
LinyuYao1021 commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-790872203 > Thanks for your review @dannycranmer! > > Before we can merge this PR, we need to make sure that CI is passing without credentials (this is the case in the PR CI validation), and with credentials (my personal CI). Hi Robert, I added `if` condition to make e2e test only runs when credentials are available. Then CI keeps failing due to [[FAIL] 'Run kubernetes session test (default input)' failed after 1 minutes and 58 seconds! Test exited with exit code 1](https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=14148=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529=2236). Do you think it can fix the issue by merging the latest commit? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] LinyuYao1021 commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
LinyuYao1021 commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-766989857 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] LinyuYao1021 commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
LinyuYao1021 commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-766991327 > What's the relationship of this PR to #14490 ? It's the same one but fixing the compiling error. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] LinyuYao1021 commented on pull request #14737: [FLINK-19667] Add AWS Glue Schema Registry integration
LinyuYao1021 commented on pull request #14737: URL: https://github.com/apache/flink/pull/14737#issuecomment-766989857 It's the same one but fixing the compiling error. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
