#general
@mo.aladham: @mo.aladham has joined the channel
#random
@mo.aladham: @mo.aladham has joined the channel
#troubleshooting
@mo.aladham: @mo.aladham has joined the channel
@laxman: Hi, anyone facing issues with Segment purging with GCS as deep store.
@laxman: We are facing the following issue
```2020/12/22 02:35:25.329 ERROR [SegmentDeletionManager] [pool-6-thread-1] Had trouble deleting directories:
java.io.IOException: java.io.IOException: java.net.URISyntaxException: Expected scheme-specific part at index 5: file:
at org.apache.pinot.plugin.filesystem.GcsPinotFS.listFiles(GcsPinotFS.java:322) ~[pinot-gcs-0.6.0-shaded.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
at org.apache.pinot.controller.helix.core.SegmentDeletionManager.removeAgedDeletedSegments(SegmentDeletionManager.java:233) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
at org.apache.pinot.controller.helix.core.retention.RetentionManager.postprocess(RetentionManager.java:89) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
at org.apache.pinot.controller.helix.core.periodictask.ControllerPeriodicTask.postprocess(ControllerPeriodicTask.java:131) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
at org.apache.pinot.controller.helix.core.periodictask.ControllerPeriodicTask.processTables(ControllerPeriodicTask.java:97) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
at org.apache.pinot.controller.helix.core.periodictask.ControllerPeriodicTask.runTask(ControllerPeriodicTask.java:68) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
at org.apache.pinot.core.periodictask.BasePeriodicTask.run(BasePeriodicTask.java:120) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
at org.apache.pinot.core.periodictask.PeriodicTaskScheduler.lambda$start$0(PeriodicTaskScheduler.java:73) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.io.IOException: java.net.URISyntaxException: Expected scheme-specific part at index 5: file:
at org.apache.pinot.plugin.filesystem.GcsPinotFS.getBase(GcsPinotFS.java:135) ~[pinot-gcs-0.6.0-shaded.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
at org.apache.pinot.plugin.filesystem.GcsPinotFS.normalizeToDirectoryPrefix(GcsPinotFS.java:105) ~[pinot-gcs-0.6.0-shaded.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
at org.apache.pinot.plugin.filesystem.GcsPinotFS.listFiles(GcsPinotFS.java:307) ~[pinot-gcs-0.6.0-shaded.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
... 13 more
Caused by: java.net.URISyntaxException: Expected scheme-specific part at index 5: file:
at java.net.URI$Parser.fail(URI.java:2913) ~[?:?]
at java.net.URI$Parser.failExpecting(URI.java:2919) ~[?:?]
at java.net.URI$Parser.parse(URI.java:3119) ~[?:?]
at java.net.URI.<init>(URI.java:685) ~[?:?]
at java.net.URI.<init>(URI.java:786) ~[?:?]
at org.apache.pinot.plugin.filesystem.GcsPinotFS.getBase(GcsPinotFS.java:133) ~[pinot-gcs-0.6.0-shaded.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
at org.apache.pinot.plugin.filesystem.GcsPinotFS.normalizeToDirectoryPrefix(GcsPinotFS.java:105) ~[pinot-gcs-0.6.0-shaded.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
at org.apache.pinot.plugin.filesystem.GcsPinotFS.listFiles(GcsPinotFS.java:307) ~[pinot-gcs-0.6.0-shaded.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
... 13 more```
@laxman: Had gone through this codepath multiple times but I couldn’t see any issue with config or code.
@dlavoie: The exceptions complains of a malformed URI
```
java.net.URISyntaxException: Expected scheme-specific part at index 5: file:```
Can you share your non sensitive gcs configuration?
@laxman: Yeah. GCS path is also there in the above stacktrace.
@laxman: ```ERROR [SegmentDeletionManager] [pool-6-thread-1] Had trouble deleting directories: ```
@laxman: Segments are getting uploaded up to gcs cleanly.
Also, they are moved to Deleted_Segments too in gcs.
But, they are not getting deleted from there after the retention period is over
@laxman: controller config here (masked some data)
```root@pinot-controller-0:/opt/pinot# cat /var/pinot/controller/config/pinot-controller.conf
controller.helix.cluster.name=my-views
controller.port=9000
controller.zk.str=zookeeper.host.svc.cluster.local:2181/pinot
pinot.set.instance.id.to.hostname=true
controller.data.dir=
controller.enable.split.commit=true
controller.local.temp.dir=/var/pinot/controller/data/temp
=org.apache.pinot.plugin.filesystem.GcsPinotFS
pinot.controller.storage.factory.gs.projectId=XXXXXXXXX
pinot.controller.storage.factory.gs.gcpKey=XXXXXXXX.json
pinot.controller.segment.fetcher.protocols=file,http,gs
pinot.controller.segment.fetcher.gs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher```
@fx19880617: @elon.azoulay have you seen similar issue or any resolution is recommended?
@elon.azoulay: Checking, I think I know why, will update shortly
@elon.azoulay: Working on a fix, will update shortly
@elon.azoulay: thanks for catching this @laxman! - we are affected by this as well
@fx19880617: Thanks Elon!
@laxman: Thanks for the update @elon.azoulay. Curious to know where this bug is in the code. Also, is there workaround?
#getting-started
@hello: @hello has joined the channel
#feat-partial-upsert
@yupeng: @yupeng has joined the channel
@g.kishore: @g.kishore has joined the channel
@jackie.jxt: @jackie.jxt has joined the channel
@qiaochu: @qiaochu has joined the channel
@tingchen: @tingchen has joined the channel
@yupeng: starting for the design
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
