Fokko commented on issue #7519: FLINK-11378 - Allow writing to Hadoop compatible Filesystems via Hado… URL: https://github.com/apache/flink/pull/7519#issuecomment-468696725 We're running the GCS connector using the HDFS interface. We're having some issues with performance, after firing up the logging we seem some renaming going on. Would this be something that we can get rid of, or should we implement the StreamingFileSink with GCS ourselves? Any guidance? A snipper from the logs: ``` Successfully copied gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=14/.part-239-6.inprogress.648a93c1-61f5-4a0e-aa50-cc6685c62050 to gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=14/part-239-6 deleteObjects([gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=14/.part-239-6.inprogress.648a93c1-61f5-4a0e-aa50-cc6685c62050]) getItemInfo: gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=15/part-219-7/: exists: no listObjectNames(fokko-events-raw, yyyymmdd=2019-03-01/hh=15/part-219-7/, /, 1) listStorageObjectsAndPrefixes(fokko-events-raw, yyyymmdd=2019-03-01/hh=15/part-219-7/, /, false, 1) createListRequest(fokko-events-raw, yyyymmdd=2019-03-01/hh=15/part-219-7/, /, false, 1) listStorageObjectsAndPrefixesPage({bucket=fokko-events-raw, delimiter=/, includeTrailingDelimiter=false, maxResults=2, prefix=yyyymmdd=2019-03-01/hh=15/part-219-7/}, 1) Successfully deleted gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=14/.part-239-6.inprogress.648a93c1-61f5-4a0e-aa50-cc6685c62050 at generation 1551452605129306 Successfully copied gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=14/.part-294-6.inprogress.806455c4-9a36-49f8-9ae9-6e00b90bbdd2 to gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=14/part-294-6 deleteObjects([gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=14/.part-294-6.inprogress.806455c4-9a36-49f8-9ae9-6e00b90bbdd2]) Successfully copied gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=15/.part-33-7.inprogress.fec75b00-723c-452e-8041-d8608c9ad07a to gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=15/part-33-7 deleteObjects([gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=15/.part-33-7.inprogress.fec75b00-723c-452e-8041-d8608c9ad07a]) Successfully deleted gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=14/.part-294-6.inprogress.806455c4-9a36-49f8-9ae9-6e00b90bbdd2 at generation 1551452603846557 Successfully deleted gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=15/.part-33-7.inprogress.fec75b00-723c-452e-8041-d8608c9ad07a at generation 1551452602868674 getItemInfo(gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=14/.part-33-6.inprogress.27ca9b20-6f8a-476c-9a0d-0f97852b595d) getObject(gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=14/.part-33-6.inprogress.27ca9b20-6f8a-476c-9a0d-0f97852b595d) getItemInfo(gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=14/.part-33-6.inprogress.27ca9b20-6f8a-476c-9a0d-0f97852b595d/) getObject(gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=14/.part-33-6.inprogress.27ca9b20-6f8a-476c-9a0d-0f97852b595d/) Successfully copied gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=15/.part-219-7.inprogress.79dbbc8c-8c89-49c7-989f-4a6460eb3b25 to gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=15/part-219-7 deleteObjects([gs://fokko-events-raw/yyyymmdd=2019-03-01/hh=15/.part-219-7.inprogress.79dbbc8c-8c89-49c7-989f-4a6460eb3b25]) ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
