DanielMorales9 commented on issue #32746:
URL: https://github.com/apache/beam/issues/32746#issuecomment-2407056553
I noticed, however, that I have not provided the triggering frequency. I
will check if it changes something and let you know.
I also found that the writer hangs indefinitely while trying to update the
manifest file, but not timing out. Not sure why is that🤔
```
Operation ongoing in bundle for at least 02h10m00s without completing
Current user step name:
Managed.ManagedTransform/ManagedSchemaTransformProvider.ManagedSchemaTransform/IcebergWriteSchemaTransformProvider.IcebergWriteSchemaTransform/IcebergIO.WriteRows/Write
Rows to Destinations/AppendFilesToTables/Append metadata updates to tables
Time spent in this step(millis): 1728598024092
Processing times in each step(millis)
Step name:
Managed.ManagedTransform/ManagedSchemaTransformProvider.ManagedSchemaTransform/IcebergWriteSchemaTransformProvider.IcebergWriteSchemaTransform/MapElements/Map
Time spent in this step: IntSummaryStatistics{count=90, sum=124, min=0,
average=1.377778, max=53}
Step name:
Managed.ManagedTransform/ManagedSchemaTransformProvider.ManagedSchemaTransform/IcebergWriteSchemaTransformProvider.IcebergWriteSchemaTransform/IcebergIO.WriteRows/Write
Rows to Destinations/AppendFilesToTables/Append metadata updates to tables
Time spent in this step: IntSummaryStatistics{count=90, sum=17162211,
min=34058, average=190691.233333, max=996066}
Step name:
Managed.ManagedTransform/ManagedSchemaTransformProvider.ManagedSchemaTransform/IcebergWriteSchemaTransformProvider.IcebergWriteSchemaTransform/IcebergIO.WriteRows/Write
Rows to Destinations/AppendFilesToTables/Group metadata updates by
table/ReadStream
Time spent in this step: IntSummaryStatistics{count=942, sum=1470, min=0,
average=1.560510, max=234}
at [email protected]/java.net.SocketInputStream.socketRead0(Native Method)
at
[email protected]/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
at
[email protected]/java.net.SocketInputStream.read(SocketInputStream.java:168)
at
[email protected]/java.net.SocketInputStream.read(SocketInputStream.java:140)
at
[email protected]/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:484)
at
[email protected]/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478)
at
[email protected]/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
at
[email protected]/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1459)
at
[email protected]/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1070)
at
[email protected]/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
at
[email protected]/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)
at
[email protected]/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
at
[email protected]/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:789)
at
[email protected]/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:724)
at
[email protected]/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1615)
at
[email protected]/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520)
at
[email protected]/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527)
at
[email protected]/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:334)
at
app//com.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:36)
at
app//com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:152)
at
app//com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)
at
app//com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012)
at
app//com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:525)
at
app//com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:466)
at
app//com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeMedia(AbstractGoogleClientRequest.java:490)
at
app//com.google.api.services.storage.Storage$Objects$Get.executeMedia(Storage.java:6522)
at
app//com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.openStream(GoogleCloudStorageReadChannel.java:933)
at
app//com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.openContentChannel(GoogleCloudStorageReadChannel.java:724)
at
app//com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.performLazySeek(GoogleCloudStorageReadChannel.java:715)
at
app//com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.read(GoogleCloudStorageReadChannel.java:307)
at
app//com.google.cloud.hadoop.fs.gcs.GoogleHadoopFSInputStream.read(GoogleHadoopFSInputStream.java:142)
at [email protected]/java.io.DataInputStream.read(DataInputStream.java:149)
at
app//org.apache.iceberg.hadoop.HadoopStreams$HadoopSeekableInputStream.read(HadoopStreams.java:123)
at
app//org.apache.iceberg.avro.AvroIO$AvroInputStreamAdapter.read(AvroIO.java:117)
at
app//org.apache.avro.file.DataFileReader.openReader(DataFileReader.java:65)
at
app//org.apache.iceberg.avro.AvroIterable.newFileReader(AvroIterable.java:100)
at app//org.apache.iceberg.avro.AvroIterable.iterator(AvroIterable.java:76)
at
app//org.apache.iceberg.io.CloseableIterable$7$1.<init>(CloseableIterable.java:188)
at
app//org.apache.iceberg.io.CloseableIterable$7.iterator(CloseableIterable.java:187)
at
app//org.apache.iceberg.ManifestFiles.copyManifestInternal(ManifestFiles.java:312)
at
app//org.apache.iceberg.ManifestFiles.copyAppendManifest(ManifestFiles.java:264)
at
app//org.apache.iceberg.MergingSnapshotProducer.copyManifest(MergingSnapshotProducer.java:288)
at
app//org.apache.iceberg.MergingSnapshotProducer.add(MergingSnapshotProducer.java:279)
at app//org.apache.iceberg.MergeAppend.appendManifest(MergeAppend.java:68)
at
app//org.apache.beam.sdk.io.iceberg.AppendFilesToTables$AppendFilesToTablesDoFn.processElement(AppendFilesToTables.java:93)
at
app//org.apache.beam.sdk.io.iceberg.AppendFilesToTables$AppendFilesToTablesDoFn$DoFnInvoker.invokeProcessElement(Unknown
Source)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
