Hi +Reuven Lax <re...@google.com> - I saw your post when researching the error above: https://stackoverflow.com/a/69111493 :)
As we are on Flink 1.13.1 in the middle of a move, it would be tough to have to downgrade Flink to downgrade Beam here. Is the warning something of concern or can we expect the data to be delivered (and ignore this warning for now, and wait for the next release and then upgrade to it)? Best, Tobi On Thu, Sep 23, 2021 at 2:51 PM Kaymak, Tobias <tobias.kay...@ricardo.ch> wrote: > Hello, > I am using the GCP IOs to connect to BigQuery and Bigtable. I seem to have > improved the situation by including Bigtable and BigQuery in my POM > with the exact same version as the Beam 2.32.0 SDK. However I still get a > lot of these errors: > > io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference > cleanQueue > SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=3, target= > bigquerystorage.googleapis.com:443} was not shutdown properly!!! ~*~*~* > Make sure to call shutdown()/shutdownNow() and wait until > awaitTermination() returns true. > java.lang.RuntimeException: ManagedChannel allocation site > at > io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:93) > at > io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:53) > at > io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:44) > at > io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:615) > at > io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:261) > at > com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel(InstantiatingGrpcChannelProvider.java:327) > at > com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.access$1700(InstantiatingGrpcChannelProvider.java:74) > at > com.google.api.gax.grpc.InstantiatingGrpcChannelProvider$1.createSingleChannel(InstantiatingGrpcChannelProvider.java:220) > at com.google.api.gax.grpc.ChannelPool.create(ChannelPool.java:72) > at > com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel(InstantiatingGrpcChannelProvider.java:227) > at > com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel(InstantiatingGrpcChannelProvider.java:210) > at > com.google.api.gax.rpc.ClientContext.create(ClientContext.java:169) > at > com.google.cloud.bigquery.storage.v1beta2.stub.GrpcBigQueryWriteStub.create(GrpcBigQueryWriteStub.java:138) > at > com.google.cloud.bigquery.storage.v1beta2.stub.BigQueryWriteStubSettings.createStub(BigQueryWriteStubSettings.java:145) > at > com.google.cloud.bigquery.storage.v1beta2.BigQueryWriteClient.<init>(BigQueryWriteClient.java:128) > at > com.google.cloud.bigquery.storage.v1beta2.BigQueryWriteClient.create(BigQueryWriteClient.java:109) > at > org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.newBigQueryWriteClient(BigQueryServicesImpl.java:1279) > at > org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.access$800(BigQueryServicesImpl.java:139) > at > org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.<init>(BigQueryServicesImpl.java:510) > at > org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.<init>(BigQueryServicesImpl.java:455) > at > org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.getDatasetService(BigQueryServicesImpl.java:172) > at > org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn.finishBundle(WriteTables.java:284) > at > org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn$DoFnInvoker.invokeFinishBundle(Unknown > Source) > at > org.apache.beam.runners.core.SimpleDoFnRunner.finishBundle(SimpleDoFnRunner.java:242) > at > org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.finishBundle(DoFnRunnerWithMetricsUpdate.java:89) > at > org.apache.beam.runners.core.SimplePushbackSideInputDoFnRunner.finishBundle(SimplePushbackSideInputDoFnRunner.java:125) > at > org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.invokeFinishBundle(DoFnOperator.java:894) > at > org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.checkInvokeFinishBundleByTime(DoFnOperator.java:887) > at > org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.lambda$open$2(DoFnOperator.java:529) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1419) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$16(StreamTask.java:1410) > at > org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) > at > org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90) > at > org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsNonBlocking(MailboxProcessor.java:359) > at > org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:323) > at > org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:202) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:681) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:636) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:647) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:620) > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) > at java.base/java.lang.Thread.run(Unknown Source) > > dependency-tree: > > ch.ricardo.di:di-beam:jar:2.14.0 > +- org.apache.beam:beam-sdks-java-core:jar:2.32.0:compile > | +- org.apache.beam:beam-model-pipeline:jar:2.32.0:compile > | +- org.apache.beam:beam-model-job-management:jar:2.32.0:compile > | +- org.apache.beam:beam-vendor-bytebuddy-1_11_0:jar:0.1:compile > | +- org.apache.beam:beam-vendor-grpc-1_36_0:jar:0.2:compile > | +- org.apache.beam:beam-vendor-guava-26_0-jre:jar:0.1:compile > | +- com.google.code.findbugs:jsr305:jar:3.0.2:compile > | +- com.google.errorprone:error_prone_annotations:jar:2.3.4:compile > | +- com.fasterxml.jackson.core:jackson-core:jar:2.12.1:compile > | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.12.1:compile > | +- com.fasterxml.jackson.core:jackson-databind:jar:2.12.1:compile > | +- org.apache.avro:avro:jar:1.8.2:compile > | | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile > | | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile > | | +- com.thoughtworks.paranamer:paranamer:jar:2.7:compile > | | +- org.apache.commons:commons-compress:jar:1.8.1:compile > | | \- org.tukaani:xz:jar:1.5:compile > | \- org.xerial.snappy:snappy-java:jar:1.1.8.4:compile > +- > org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.32.0:compile > | +- org.checkerframework:checker-qual:jar:3.10.0:compile > | +- org.apache.beam:beam-runners-core-java:jar:2.32.0:compile > | | \- org.apache.beam:beam-sdks-java-fn-execution:jar:2.32.0:compile > | | \- org.apache.beam:beam-model-fn-execution:jar:2.32.0:compile > | +- org.apache.beam:beam-sdks-java-expansion-service:jar:2.32.0:compile > | | \- org.apache.beam:beam-runners-java-fn-execution:jar:2.32.0:compile > | +- > org.apache.beam:beam-sdks-java-extensions-google-cloud-platform-core:jar:2.32.0:compile > | | +- com.google.cloud.bigdataoss:gcsio:jar:2.2.2:compile > | | \- > com.google.apis:google-api-services-cloudresourcemanager:jar:v1-rev20210331-1.31.0:compile > | +- > org.apache.beam:beam-runners-core-construction-java:jar:2.32.0:compile > | | \- io.github.classgraph:classgraph:jar:4.8.104:compile > | +- org.apache.beam:beam-sdks-java-extensions-arrow:jar:2.32.0:compile > | +- com.google.cloud.bigdataoss:util:jar:2.2.2:compile > | | +- > com.google.api-client:google-api-client-jackson2:jar:1.31.3:compile > | | +- > com.google.apis:google-api-services-iamcredentials:jar:v1-rev20201022-1.31.0:compile > | | \- com.google.flogger:google-extensions:jar:0.5.1:compile > | +- com.google.flogger:flogger-system-backend:jar:0.6:compile > | | \- com.google.flogger:flogger:jar:0.6:compile > | +- com.google.api:gax:jar:1.63.0:compile > | +- com.google.api:gax-grpc:jar:1.63.0:compile > | +- com.google.api:gax-httpjson:jar:0.80.0:compile > | +- com.google.api:api-common:jar:1.10.1:compile > | +- > com.google.apis:google-api-services-healthcare:jar:v1beta1-rev20210407-1.31.0:compile > | +- > com.google.apis:google-api-services-pubsub:jar:v1-rev20210322-1.31.0:compile > | +- > com.google.apis:google-api-services-storage:jar:v1-rev20210127-1.31.0:compile > | +- com.google.auth:google-auth-library-credentials:jar:0.25.2:compile > | +- com.google.auth:google-auth-library-oauth2-http:jar:0.25.2:compile > | +- com.google.cloud:google-cloud-bigquerystorage:jar:1.17.0:compile > | | +- > com.google.api.grpc:proto-google-cloud-bigquerystorage-v1alpha2:jar:0.117.0:compile > | | +- > com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1:jar:0.117.0:compile > | | +- org.json:json:jar:20200518:compile > | | +- > com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1beta1:jar:0.117.0:compile > | | +- > com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1beta2:jar:0.117.0:compile > | | \- > com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1:jar:1.17.0:compile > | +- com.google.cloud.bigtable:bigtable-client-core:jar:1.19.1:compile > | | +- com.google.api.grpc:grpc-google-common-protos:jar:2.1.0:compile > | | +- > com.google.api.grpc:grpc-google-cloud-bigtable-admin-v2:jar:1.22.0:compile > | | +- io.opencensus:opencensus-contrib-grpc-util:jar:0.24.0:compile > | | \- io.dropwizard.metrics:metrics-core:jar:3.2.6:compile > | +- com.google.cloud:google-cloud-core:jar:1.94.6:compile > | +- com.google.cloud:google-cloud-core-grpc:jar:1.94.6:compile > | +- > com.google.cloud.datastore:datastore-v1-proto-client:jar:1.6.3:compile > | | \- > com.google.http-client:google-http-client-protobuf:jar:1.39.2:compile > | +- com.google.cloud:google-cloud-firestore:jar:2.2.5:compile > | | \- > com.google.cloud:proto-google-cloud-firestore-bundle-v1:jar:2.2.5:compile > | +- com.google.cloud:google-cloud-pubsublite:jar:0.13.2:compile > | +- com.google.cloud:google-cloud-pubsub:jar:1.112.0:compile > | +- com.google.cloud:google-cloud-spanner:jar:6.2.0:compile > | | +- > com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:6.2.0:compile > | | +- > com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:6.2.0:compile > | | +- > com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:6.2.0:compile > | | \- > com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:6.2.0:compile > | +- com.google.code.gson:gson:jar:2.8.6:compile > | +- > com.google.http-client:google-http-client-jackson2:jar:1.39.2:compile > | +- io.grpc:grpc-alts:jar:1.37.0:compile > | +- io.grpc:grpc-api:jar:1.37.0:compile > | +- io.grpc:grpc-auth:jar:1.37.0:compile > | +- io.grpc:grpc-core:jar:1.37.0:compile > | +- io.grpc:grpc-context:jar:1.37.0:compile > | +- io.grpc:grpc-grpclb:jar:1.37.0:compile > | +- io.grpc:grpc-netty:jar:1.37.0:compile > | | +- io.netty:netty-codec-http2:jar:4.1.52.Final:compile > | | | \- io.netty:netty-codec-http:jar:4.1.52.Final:compile > | | \- io.netty:netty-handler-proxy:jar:4.1.52.Final:runtime > | | \- io.netty:netty-codec-socks:jar:4.1.52.Final:runtime > | +- io.grpc:grpc-netty-shaded:jar:1.37.0:compile > | +- io.grpc:grpc-stub:jar:1.37.0:compile > | +- com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:1.94.0:compile > | +- > com.google.api.grpc:grpc-google-cloud-pubsublite-v1:jar:0.13.2:compile > | +- com.google.guava:guava:jar:30.1-jre:compile > | +- org.hamcrest:hamcrest:jar:2.1:compile > | +- org.apache.httpcomponents:httpclient:jar:4.5.10:compile > | +- org.apache.httpcomponents:httpcore:jar:4.4.12:compile > | +- io.netty:netty-handler:jar:4.1.52.Final:compile > | | +- io.netty:netty-common:jar:4.1.52.Final:compile > | | +- io.netty:netty-resolver:jar:4.1.52.Final:compile > | | +- io.netty:netty-buffer:jar:4.1.52.Final:compile > | | +- io.netty:netty-transport:jar:4.1.52.Final:compile > | | \- io.netty:netty-codec:jar:4.1.52.Final:compile > | +- io.netty:netty-tcnative-boringssl-static:jar:2.0.33.Final:compile > | +- > com.google.api.grpc:proto-google-cloud-bigquerystorage-v1:jar:1.17.0:compile > | +- > com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta2:jar:0.117.0:compile > | +- > com.google.api.grpc:proto-google-cloud-bigtable-admin-v2:jar:1.22.0:compile > | +- > com.google.api.grpc:proto-google-cloud-bigtable-v2:jar:1.22.0:compile > | +- > com.google.api.grpc:proto-google-cloud-datastore-v1:jar:0.89.0:compile > | +- > com.google.api.grpc:proto-google-cloud-firestore-v1:jar:2.2.5:compile > | +- com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:1.94.0:compile > | +- > com.google.api.grpc:proto-google-cloud-pubsublite-v1:jar:0.13.2:compile > | +- > com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:6.2.0:compile > | +- com.google.api.grpc:proto-google-cloud-spanner-v1:jar:6.2.0:compile > | +- com.google.api.grpc:proto-google-common-protos:jar:2.1.0:compile > | +- com.google.protobuf:protobuf-java:jar:3.15.8:compile > | +- org.apache.arrow:arrow-memory-core:jar:4.0.0:compile > | +- org.apache.arrow:arrow-vector:jar:4.0.0:compile > | | +- org.apache.arrow:arrow-format:jar:4.0.0:compile > | | \- com.google.flatbuffers:flatbuffers-java:jar:1.12.0:compile > | \- org.threeten:threetenbp:jar:1.4.4:compile > +- com.google.api-client:google-api-client:jar:1.31.1:compile > | +- com.google.oauth-client:google-oauth-client:jar:1.31.2:compile > | \- > com.google.http-client:google-http-client-apache-v2:jar:1.39.2:compile > +- > com.google.apis:google-api-services-bigquery:jar:v2-rev20210410-1.31.0:compile > +- com.google.http-client:google-http-client:jar:1.39.2:compile > | +- com.google.j2objc:j2objc-annotations:jar:1.3:compile > | +- io.opencensus:opencensus-api:jar:0.28.0:compile > | \- io.opencensus:opencensus-contrib-http-util:jar:0.28.0:compile > +- joda-time:joda-time:jar:2.10.10:compile > +- org.slf4j:slf4j-api:jar:1.7.30:compile > +- org.slf4j:slf4j-jdk14:jar:1.7.30:runtime > +- org.hamcrest:hamcrest-core:jar:2.1:compile > +- org.hamcrest:hamcrest-library:jar:2.1:compile > +- junit:junit:jar:4.13.1:compile > +- org.apache.beam:beam-runners-direct-java:jar:2.32.0:runtime > +- org.apache.beam:beam-sdks-java-io-kafka:jar:2.32.0:compile > | +- org.springframework:spring-expression:jar:4.3.18.RELEASE:compile > | | \- org.springframework:spring-core:jar:4.3.18.RELEASE:compile > | +- io.confluent:kafka-avro-serializer:jar:5.3.2:compile > | | +- io.confluent:common-config:jar:5.3.2:compile > | | \- io.confluent:common-utils:jar:5.3.2:compile > | | \- com.101tec:zkclient:jar:0.10:compile > | \- io.confluent:kafka-schema-registry-client:jar:5.3.2:compile > +- org.apache.kafka:kafka-clients:jar:2.6.0:compile > | +- com.github.luben:zstd-jni:jar:1.4.4-7:compile > | \- org.lz4:lz4-java:jar:1.7.1:compile > +- com.google.protobuf:protobuf-java-util:jar:3.6.1:compile > +- ch.ricardo.schema:proto-schemas:jar:3.1.68:compile > | \- javax.annotation:javax.annotation-api:jar:1.3.2:compile > +- org.mockito:mockito-core:jar:3.7.7:test > | +- net.bytebuddy:byte-buddy:jar:1.10.19:test > | +- net.bytebuddy:byte-buddy-agent:jar:1.10.19:test > | \- org.objenesis:objenesis:jar:3.1:test > +- com.google.auto.value:auto-value-annotations:jar:1.6.2:compile > +- com.google.auto.value:auto-value:jar:1.6.2:provided > +- org.projectlombok:lombok:jar:1.16.22:compile > +- com.jayway.jsonpath:json-path:jar:2.3.0:compile > | \- net.minidev:json-smart:jar:2.3:compile > | \- net.minidev:accessors-smart:jar:1.2:compile > | \- org.ow2.asm:asm:jar:5.0.4:compile > +- org.apache.commons:commons-lang3:jar:3.8.1:compile > +- com.google.cloud:google-cloud-bigquery:jar:1.127.12:compile > | +- com.google.api.grpc:proto-google-iam-v1:jar:1.0.12:compile > | +- com.google.http-client:google-http-client-gson:jar:1.39.2:compile > | +- com.google.cloud:google-cloud-core-http:jar:1.94.6:compile > | +- > com.google.http-client:google-http-client-appengine:jar:1.39.2:compile > | +- commons-logging:commons-logging:jar:1.2:compile > | +- commons-codec:commons-codec:jar:1.15:compile > | +- org.checkerframework:checker-compat-qual:jar:2.5.5:compile > | +- com.google.guava:failureaccess:jar:1.0.1:compile > | \- > com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile > +- com.google.cloud:google-cloud-bigtable:jar:1.22.0:compile > | +- com.google.api.grpc:grpc-google-cloud-bigtable-v2:jar:1.22.0:compile > | +- org.conscrypt:conscrypt-openjdk-uber:jar:2.5.1:compile > | +- org.codehaus.mojo:animal-sniffer-annotations:jar:1.20:runtime > | +- com.google.android:annotations:jar:4.1.1.4:runtime > | +- io.perfmark:perfmark-api:jar:0.23.0:runtime > | +- io.grpc:grpc-protobuf:jar:1.37.0:compile > | \- io.grpc:grpc-protobuf-lite:jar:1.37.0:compile > +- org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.32.0:compile > +- org.jetbrains:annotations:jar:20.1.0:compile > +- com.google.cloud:google-cloud-bigtable-emulator:jar:0.128.2:test > \- commons-io:commons-io:jar:2.6:compile > > Best, > Tobi > > On Thu, Sep 23, 2021 at 6:15 AM Tomo Suzuki <suzt...@google.com> wrote: > >> From the package name, I think you want to focus on grpc-netty-shaded >> artifact rather than netty. >> >> Stacktrace and the dependency tree would be helpful to understand why the >> discrepancy occurred. >> >> Regards, >> Tomo >> >> On Wed, Sep 22, 2021 at 12:21 Reuven Lax <re...@google.com> wrote: >> >>> Are you using any of the GCP IOs in your pipeline? >>> >>> On Wed, Sep 22, 2021 at 9:11 AM Kaymak, Tobias <tobias.kay...@ricardo.ch> >>> wrote: >>> >>>> Hello, >>>> >>>> while upgrading our cluster and pipelines to the new visions I noticed: >>>> >>>> java.lang.NoClassDefFoundError: >>>> io/grpc/netty/shaded/io/netty/channel/FailedChannelFuture >>>> >>>> so I checked the maven dependency tree and found that the: >>>> >>>> >>>> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.32.0:compile >>>> >>>> dependency is using netty 4.1.52 - while Flink 1.13.1 seems to use >>>> 4.1.49. >>>> >>>> Is there a way to fix this? >>>> >>>> Best, >>>> Tobi >>>> >>> -- >> Regards, >> Tomo >> >