[jira] [Commented] (STORM-3871) Storm blobstore leak space
[
https://issues.apache.org/jira/browse/STORM-3871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759001#comment-17759001
]
Richard Zowalla commented on STORM-3871:
Does this happen in Storm 2.5.0 or 2.6.0-SNAPSHOT?
> Storm blobstore leak space
> --
>
> Key: STORM-3871
> URL: https://issues.apache.org/jira/browse/STORM-3871
> Project: Apache Storm
> Issue Type: Bug
> Components: blobstore
>Affects Versions: 2.3.0
> Environment: Storm 2.3.0 in Kubernetes 1.21
> Storm nimbus have 3 empty dir volumes mapped to
> /var/lib/kubelet/pods/.../volumes in a 10GB /var limit.
> Storm has default blob cleanup strategy.
>Reporter: Antoine Tran
>Priority: Major
>
> We observe after 5 days, with 3 nimbus deployed in HA, that the blostore
> keeps growing.
>
> Usually we submit topologies each 10min and it ends in around 10 min. The
> expected result is that the blobstore cleans itself so that topologies older
> than days are removed. But this is not the case, as seen below (done the
> 2022-06-03, expected result: only blob from the current day should appear.
> current behavior: blobs from 2 days ago are still here)
>
> {noformat}
> /space/StormApp/current/bin/storm blobstore list 2>&1 | grep
> o.a.s.c.Blobstore | sort -k7 | perl -anE '$date=localtime($F[6]/1000);shift
> @F;say join " ", $date, @F'
> Wed Jun 1 14:12:43 2022 [main] INFO o.a.s.c.Blobstore -
> dep-apeframework-3.11.3-8f132aef-b401-4fa6-b07e-3588073da824.jar
> 1654092763312 [o::r--, o::rwa, u:class
> org.apache.storm.security.auth.NimbusPrincipal:rwa]
> Wed Jun 1 14:13:43 2022 [main] INFO o.a.s.c.Blobstore -
> dep-apeframework-3.11.3-ae74967a-0def-44a1-8d1b-53a255900239.jar
> 1654092823330 [o::r--, o::rwa, u:class
> org.apache.storm.security.auth.NimbusPrincipal:rwa]
> Wed Jun 1 14:33:00 2022 [main] INFO o.a.s.c.Blobstore -
> dep-apeframework-3.11.3-8fa7edcb-5f8a-4a79-8f8d-8c8fd50dbc9e.jar
> 1654093980305 [o::r--, o::rwa, u:class
> org.apache.storm.security.auth.NimbusPrincipal:rwa]
> Wed Jun 1 15:14:19 2022 [main] INFO o.a.s.c.Blobstore -
> dep-apeframework-3.11.3-9168e429-2cd6-433c-90d9-6db19d564824.jar
> 1654096459069 [o::r--, o::rwa, u:class
> org.apache.storm.security.auth.NimbusPrincipal:rwa]
> Wed Jun 1 16:13:49 2022 [main] INFO o.a.s.c.Blobstore -
> dep-apeframework-3.11.3-9ed7a9fd-2819-4784-a953-c04a65981c59.jar
> 1654100029294 [o::r--, o::rwa, u:class
> org.apache.storm.security.auth.NimbusPrincipal:rwa]
> Wed Jun 1 22:53:29 2022 [main] INFO o.a.s.c.Blobstore -
> dep-apeframework-3.11.3-124b606a-0d2b-44a1-9885-4bb1152a98e7.jar
> 1654124009759 [o::r--, o::rwa, u:class
> org.apache.storm.security.auth.NimbusPrincipal:rwa]
> Wed Jun 1 23:23:59 2022 [main] INFO o.a.s.c.Blobstore -
> dep-apeframework-3.11.3-be7e85b7-3cf1-495a-83a7-6a28da33af57.jar
> 1654125839685 [o::r--, o::rwa, u:class
> org.apache.storm.security.auth.NimbusPrincipal:rwa]
> Wed Jun 1 23:33:43 2022 [main] INFO o.a.s.c.Blobstore -
> dep-apeframework-3.11.3-f799a103-d544-4a31-a680-1d0625fab2b9.jar
> 1654126423302 [o::r--, o::rwa, u:class
> org.apache.storm.security.auth.NimbusPrincipal:rwa]
> Thu Jun 2 01:13:49 2022 [main] INFO o.a.s.c.Blobstore -
> dep-apeframework-3.11.3-3edcef03-f7d6-400e-8076-30dc0e5c4bff.jar
> 1654132429251 [o::r--, o::rwa, u:class
> org.apache.storm.security.auth.NimbusPrincipal:rwa]
> Thu Jun 2 02:03:29 2022 [main] INFO o.a.s.c.Blobstore -
> dep-apeframework-3.11.3-807000a6-0416-4821-b67d-a1e5e433edce.jar
> 1654135409776 [o::r--, o::rwa, u:class
> org.apache.storm.security.auth.NimbusPrincipal:rwa]
> Thu Jun 2 02:13:30 2022 [main] INFO o.a.s.c.Blobstore -
> dep-apeframework-3.11.3-4f7e7439-7788-4b74-81a2-4d6433caed4b.jar
> 1654136010028 [o::r--, o::rwa, u:class
> org.apache.storm.security.auth.NimbusPrincipal:rwa]
> Thu Jun 2 03:03:59 2022 [main] INFO o.a.s.c.Blobstore -
> dep-apeframework-3.11.3-038e17f3-7acd-4c19-a212-a095e6c7adba.jar
> 1654139039870 [o::r--, o::rwa, u:class
> org.apache.storm.security.auth.NimbusPrincipal:rwa]
> Thu Jun 2 07:43:30 2022 [main] INFO o.a.s.c.Blobstore -
> dep-apeframework-3.11.3-b4fb1761-4f97-4ac7-8785-85e2aefbb66c.jar
> 1654155810868 [o::r--, o::rwa, u:class
> org.apache.storm.security.auth.NimbusPrincipal:rwa]
> Thu Jun 2 08:33:49 2022 [main] INFO o.a.s.c.Blobstore -
> dep-apeframework-3.11.3-ac2c4004-d561-44ad-beca-0883ad8980c5.jar
> 1654158829119 [o::r--, o::rwa, u:class
> org.apache.storm.security.auth.NimbusPrincipal:rwa]
> Thu Jun 2 09:44:00 2022 [main] INFO o.a.s.c.Blobstore -
> dep-apeframework-3.11.3-31f8b9b4-8f79-47ef-8728-46127baf04a2.jar
> 1654163040830 [o::r--, o::rwa, u:class
> org.apache.storm.security.auth.NimbusPrincipal:rwa]
> Thu Jun 2 09:53:00 2022 [main] INFO o.a.s.c.Blobstore -
>
[jira] [Commented] (STORM-3871) Storm blobstore leak space
[
https://issues.apache.org/jira/browse/STORM-3871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17545891#comment-17545891
]
Antoine Tran commented on STORM-3871:
-
Workaround:
{quote}
{code:java}
#!/bin/bash
declare -i nowminus30m="$(date -d '30 minutes ago' +%s)000"
/space/StormApp/current/bin/storm blobstore list 2>&1| grep
'o.a.s.c.Blobstore'>blobstore
sort -k7 blobstore>sortedblobstore
while IFS= read -r line
do
declare -i d=$(echo $line | awk '{ print $7}')
name=$(echo $line | awk '{ print $6}' )
echo "$d ? $nowminus30m"
if [ $d -lt $nowminus30m ]; then
echo " delete $name $line "
/space/StormApp/current/bin/storm blobstore delete $name
else
echo "keep $name $line"
fi
done < "sortedblobstore"
rm -f blobstore sortedblobstore
{code}
{quote}
> Storm blobstore leak space
> --
>
> Key: STORM-3871
> URL: https://issues.apache.org/jira/browse/STORM-3871
> Project: Apache Storm
> Issue Type: Bug
> Components: blobstore
>Affects Versions: 2.3.0
> Environment: Storm 2.3.0 in Kubernetes 1.21
> Storm nimbus have 3 empty dir volumes mapped to
> /var/lib/kubelet/pods/.../volumes in a 10GB /var limit.
> Storm has default blob cleanup strategy.
>Reporter: Antoine Tran
>Priority: Major
>
> We observe after 5 days, with 3 nimbus deployed in HA, that the blostore
> keeps growing.
>
> Usually we submit topologies each 10min and it ends in around 10 min. The
> expected result is that the blobstore cleans itself so that topologies older
> than days are removed. But this is not the case, as seen below (done the
> 2022-06-03, expected result: only blob from the current day should appear.
> current behavior: blobs from 2 days ago are still here)
>
> {noformat}
> /space/StormApp/current/bin/storm blobstore list 2>&1 | grep
> o.a.s.c.Blobstore | sort -k7 | perl -anE '$date=localtime($F[6]/1000);shift
> @F;say join " ", $date, @F'
> Wed Jun 1 14:12:43 2022 [main] INFO o.a.s.c.Blobstore -
> dep-apeframework-3.11.3-8f132aef-b401-4fa6-b07e-3588073da824.jar
> 1654092763312 [o::r--, o::rwa, u:class
> org.apache.storm.security.auth.NimbusPrincipal:rwa]
> Wed Jun 1 14:13:43 2022 [main] INFO o.a.s.c.Blobstore -
> dep-apeframework-3.11.3-ae74967a-0def-44a1-8d1b-53a255900239.jar
> 1654092823330 [o::r--, o::rwa, u:class
> org.apache.storm.security.auth.NimbusPrincipal:rwa]
> Wed Jun 1 14:33:00 2022 [main] INFO o.a.s.c.Blobstore -
> dep-apeframework-3.11.3-8fa7edcb-5f8a-4a79-8f8d-8c8fd50dbc9e.jar
> 1654093980305 [o::r--, o::rwa, u:class
> org.apache.storm.security.auth.NimbusPrincipal:rwa]
> Wed Jun 1 15:14:19 2022 [main] INFO o.a.s.c.Blobstore -
> dep-apeframework-3.11.3-9168e429-2cd6-433c-90d9-6db19d564824.jar
> 1654096459069 [o::r--, o::rwa, u:class
> org.apache.storm.security.auth.NimbusPrincipal:rwa]
> Wed Jun 1 16:13:49 2022 [main] INFO o.a.s.c.Blobstore -
> dep-apeframework-3.11.3-9ed7a9fd-2819-4784-a953-c04a65981c59.jar
> 1654100029294 [o::r--, o::rwa, u:class
> org.apache.storm.security.auth.NimbusPrincipal:rwa]
> Wed Jun 1 22:53:29 2022 [main] INFO o.a.s.c.Blobstore -
> dep-apeframework-3.11.3-124b606a-0d2b-44a1-9885-4bb1152a98e7.jar
> 1654124009759 [o::r--, o::rwa, u:class
> org.apache.storm.security.auth.NimbusPrincipal:rwa]
> Wed Jun 1 23:23:59 2022 [main] INFO o.a.s.c.Blobstore -
> dep-apeframework-3.11.3-be7e85b7-3cf1-495a-83a7-6a28da33af57.jar
> 1654125839685 [o::r--, o::rwa, u:class
> org.apache.storm.security.auth.NimbusPrincipal:rwa]
> Wed Jun 1 23:33:43 2022 [main] INFO o.a.s.c.Blobstore -
> dep-apeframework-3.11.3-f799a103-d544-4a31-a680-1d0625fab2b9.jar
> 1654126423302 [o::r--, o::rwa, u:class
> org.apache.storm.security.auth.NimbusPrincipal:rwa]
> Thu Jun 2 01:13:49 2022 [main] INFO o.a.s.c.Blobstore -
> dep-apeframework-3.11.3-3edcef03-f7d6-400e-8076-30dc0e5c4bff.jar
> 1654132429251 [o::r--, o::rwa, u:class
> org.apache.storm.security.auth.NimbusPrincipal:rwa]
> Thu Jun 2 02:03:29 2022 [main] INFO o.a.s.c.Blobstore -
> dep-apeframework-3.11.3-807000a6-0416-4821-b67d-a1e5e433edce.jar
> 1654135409776 [o::r--, o::rwa, u:class
> org.apache.storm.security.auth.NimbusPrincipal:rwa]
> Thu Jun 2 02:13:30 2022 [main] INFO o.a.s.c.Blobstore -
> dep-apeframework-3.11.3-4f7e7439-7788-4b74-81a2-4d6433caed4b.jar
> 1654136010028 [o::r--, o::rwa, u:class
> org.apache.storm.security.auth.NimbusPrincipal:rwa]
> Thu Jun 2 03:03:59 2022 [main] INFO o.a.s.c.Blobstore -
> dep-apeframework-3.11.3-038e17f3-7acd-4c19-a212-a095e6c7adba.jar
> 1654139039870 [o::r--, o::rwa, u:class
> org.apache.storm.security.auth.NimbusPrincipal:rwa]
> Thu Jun 2 07:43:30 2022 [main] INFO o.a.s.c.Blobstore -
> dep-apeframework-3.11.3-b4fb1761-4f97-4ac7-8785-85e2aefbb66c.jar
> 1654155810868 [o::r--, o::rwa, u:class
> org.apache.storm.security.auth.NimbusPrincipal:rwa]
> Thu Jun
