Hi Garrett,
> On Dec 29, 2018, at 2:16 AM, Garrett D'Amore <[email protected]> wrote: > > Looking at that cde in aggsum_init: > > + as->as_buckets = kmem_zalloc(boot_ncpus * sizeof (aggsum_bucket_t), > > <https://bitbucket.org/dilos/dilos-illumos/commits/3a2d8a1b18794077eff4c504197c1d6f9d7ee487#Lusr/src/uts/common/fs/zfs/aggsum.cT90> > + KM_SLEEP); > > <https://bitbucket.org/dilos/dilos-illumos/commits/3a2d8a1b18794077eff4c504197c1d6f9d7ee487#Lusr/src/uts/common/fs/zfs/aggsum.cT91> > > The panic you encountered makes me thing boot_ncpus is an invalid value at > this point (negative one?) > > This may be highly specific to SPARC initialization flow. yes, it is issue what i have found by myself on sun4u too :) on sun4v we have another logic/specific with boot_ncpus identification. but sparc bulid is very long - about 9h - for checks. -Igor > > - Garrett > > On Dec 27, 2018, 11:13 AM -0800, Igor Kozhukhov <[email protected]>, wrote: >> >> Hi All, >> >> I have found issue where SPARC sun4u sunfire v240 has panic on boot with >> commit: >> >> https://bitbucket.org/dilos/dilos-illumos/commits/3a2d8a1b18794077eff4c504197c1d6f9d7ee487 >> >> dump: >> >> Rebooting with command: boot -kvd >> Boot device: /pci@1c,600000/scsi@2/disk@0,0:a File and args: -kvd >> module /platform/sun4u/kernel/sparcv9/unix: text at [0x1000000, 0x10d37ad] >> data at 0x1800000 >> module /platform/sun4u/kernel/sparcv9/genunix: text at [0x10d37c0, >> 0x12f3de7] data at 0x1858380 >> module /platform/SUNW,Sun-Fire-V240/kernel/misc/sparcv9/platmod: text at >> [0x12f3de8, 0x12f491f] data at 0x18ae918 >> module /platform/sun4u/kernel/cpu/sparcv9/SUNW,UltraSPARC-IIIi: text at >> [0x12f4980, 0x1306faf] data at 0x18aef80 >> Loading kmdb... >> module /platform/sun4u/kernel/misc/sparcv9/kmdbmod: text at [0x1307000, >> 0x138c187] data at 0x19199d0 >> module /kernel/misc/sparcv9/ctf: text at [0x138c1a0, 0x13963b7] data at >> 0x1953fc0 >> / >> Welcome to kmdb >> kmdb: unable to determine terminal type: assuming `vt100' >> Loaded modules: [ unix krtld genunix ] >> [1]> moddebug/W 0x80000001 >> moddebug: 0 = 0x80000001 >> [1]> :c >> OP: 1: param 'dump_plat_mincpu' was '0xffffffff' in module: '<null string>'. >> now it is set to '0x0'. >> OP: 1: param 'dump_bzip2_level' was '0x1' in module: '<null string>'. >> now it is set to '0x1'. >> DilOS Version 2.0.1.12-4D 64-bit >> Copyright (c) 2011-2018, DilOS. All rights reserved. >> load 'fs/fifofs' id 8 loaded @ 0x13f4d38/0x19580a8 size 14432/184 >> Returned from _info, retval = 1 >> load 'fs/specfs' id 7 loaded @ 0x13f0000/0x1958000 size 19768/168 >> Returned from _info, retval = 1 >> load 'fs/devfs' id 10 loaded @ 0x13f90f0/0x1958ca0 size 17160/584 >> Returned from _info, retval = 1 >> load 'fs/procfs' id 11 loaded @ 0x7be00000/0x1958f0c size 113512/1448 >> Returned from _info, retval = 1 >> load 'misc/swapgeneric' id 12 loaded @ 0x13fd3f8/0x19594c8 size 7208/400 >> Returned from _info, retval = 1 >> load 'sched/TS' id 14 loaded @ 0x7be1c000/0x1959658 size 12448/1984 >> Returned from _info, retval = 1 >> installing TS, module id 14. >> breakpoint before calling TS:_init() >> _init >> Loaded modules: [ specfs ] >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> load 'sched/TS_DPTBL' id 15 loaded @ 0x7be1bb68/0x1959e38 size 240/2096 >> Returned from _info, retval = 1 >> installing TS_DPTBL, module id 15. >> breakpoint before calling TS_DPTBL:_init() >> _init >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> Returned from _init, val = 0 >> Returned from _init, val = 0 >> Ethernet address = 0:3:ba:9f:17:61 >> installing swapgeneric, module id 12. >> breakpoint before calling swapgeneric:_init() >> _init >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> Returned from _init, val = 0 >> load 'misc/kcf' id 18 loaded @ 0x7bf1c000/0x19672c8 size 123136/3176 >> Returned from _info, retval = 1 >> load 'misc/sha1' id 19 loaded @ 0x7bf3c000/0x1970da8 size 20392/784 >> Returned from _info, retval = 1 >> load 'crypto/swrand' id 17 loaded @ 0x7bf1a000/0x19668a8 size 4256/2592 >> Returned from _info, retval = 1 >> load 'sys/doorfs' id 21 loaded @ 0x7bf46000/0x1971158 size 20792/312 >> Returned from _info, retval = 1 >> load 'misc/tlimod' id 23 loaded @ 0x7bf3a100/0x1984100 size 6824/88 >> Returned from _info, retval = 1 >> load 'strmod/rpcmod' id 22 loaded @ 0x7bf4c000/0x1971380 size 97832/77184 >> Returned from _info, retval = 1 >> load 'misc/idmap' id 20 loaded @ 0x7bf42000/0x19710e8 size 14560/112 >> Returned from _info, retval = 1 >> load 'sched/SDC' id 24 loaded @ 0x7bf64000/0x19841f8 size 5080/1184 >> Returned from _info, retval = 1 >> load 'misc/sha2' id 25 loaded @ 0x7bf66000/0x1984698 size 47600/1232 >> Returned from _info, retval = 1 >> load 'misc/skein' id 26 loaded @ 0x7bf72000/0x1984b68 size 26312/888 >> Returned from _info, retval = 1 >> load 'misc/edonr' id 27 loaded @ 0x7bf7a000/0x1984ee0 size 14688/88 >> Returned from _info, retval = 1 >> load 'fs/zfs' id 16 loaded @ 0x7be20000/0x195a668 size 1021464/49728 >> Returned from _info, retval = 1 >> OP: 1: param 'zfs_nocacheflush' was '0x0' in module: 'zfs'. >> now it is set to '0x1'. >> OP: 1: param 'zfs_arc_max' was '0x0' in module: 'zfs'. >> now it is set to '0x80000000'. >> OP: 1: param 'zfs_arc_min' was '0x0' in module: 'zfs'. >> now it is set to '0x80000000'. >> load 'drv/rootnex' id 28 loaded @ 0x7bf8a000/0x7023eedc size 6480/632 >> Returned from _info, retval = 1 >> load 'drv/options' id 29 loaded @ 0x7bf63e28/0x7023f164 size 336/192 >> Returned from _info, retval = 1 >> load 'drv/sad' id 30 loaded @ 0x7bf4b138/0x7023f224 size 3632/568 >> Returned from _info, retval = 1 >> load 'drv/pseudo' id 31 loaded @ 0x7bf19618/0x7023f45c size 2312/568 >> Returned from _info, retval = 1 >> load 'drv/clone' id 32 loaded @ 0x7bf3bba8/0x7023f694 size 976/568 >> Returned from _info, retval = 1 >> load 'misc/scsi' id 34 loaded @ 0x7bf9c000/0x1984f38 size 124712/15288 >> Returned from _info, retval = 1 >> load 'drv/scsi_vhci' id 33 loaded @ 0x7bf8c000/0x7023f8cc size 64272/496 >> Returned from _info, retval = 1 >> load 'misc/scsi_vhci/scsi_vhci_f_asym_sun' id 35 loaded @ >> 0x7bf653d8/0x7023fc0a size 3064/208 >> Returned from _info, retval = 1 >> load 'misc/scsi_vhci/scsi_vhci_f_asym_lsi' id 36 loaded @ >> 0x7bfbe000/0x7023fcda size 6520/632 >> Returned from _info, retval = 1 >> load 'misc/scsi_vhci/scsi_vhci_f_asym_emc' id 37 loaded @ >> 0x7bf1b0a0/0x1988af0 size 3528/232 >> Returned from _info, retval = 1 >> load 'misc/scsi_vhci/scsi_vhci_f_sym_emc' id 38 loaded @ >> 0x7bf89e20/0x1988bd8 size 424/208 >> Returned from _info, retval = 1 >> load 'misc/scsi_vhci/scsi_vhci_f_sym_hds' id 39 loaded @ >> 0x7bf9bb10/0x1988ca8 size 1176/200 >> Returned from _info, retval = 1 >> load 'misc/scsi_vhci/scsi_vhci_f_sym' id 40 loaded @ 0x7bf719f0/0x1988d70 >> size 1064/336 >> Returned from _info, retval = 1 >> load 'misc/scsi_vhci/scsi_vhci_f_tpgs' id 41 loaded @ 0x7be1f0a0/0x1988ec0 >> size 2480/192 >> Returned from _info, retval = 1 >> load 'misc/cmlb' id 43 loaded @ 0x7bfe0000/0x1989bb8 size 22064/2064 >> Returned from _info, retval = 1 >> load 'drv/sd' id 42 loaded @ 0x7bfc0000/0x1988f80 size 129136/3128 >> Returned from _info, retval = 1 >> load 'drv/glm' id 44 loaded @ 0x7bfe6000/0x198a3c8 size 61784/3040 >> Returned from _info, retval = 1 >> load 'drv/pcisch' id 45 loaded @ 0x7ba00000/0x198afa8 size 84112/4696 >> Returned from _info, retval = 1 >> load 'drv/rmc_comm' id 47 loaded @ 0x7bff6000/0x198c258 size 12864/736 >> Returned from _info, retval = 1 >> load 'drv/pmugpio' id 48 loaded @ 0x7bf458e0/0x198c538 size 1800/336 >> Returned from _info, retval = 1 >> load 'tod/todm5819p_rmc' id 46 loaded @ 0x7bff5158/0x198c200 size 2624/88 >> Returned from _info, retval = 1 >> installing todm5819p_rmc, module id 46. >> installing rmc_comm, module id 47. >> breakpoint before calling rmc_comm:_init() >> _init >> Loaded modules: [ pcisch scsi_vhci sd zfs ] >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> Returned from _init, val = 0 >> installing pmugpio, module id 48. >> breakpoint before calling pmugpio:_init() >> _init >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> Returned from _init, val = 0 >> breakpoint before calling todm5819p_rmc:_init() >> _init >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> WARNING: todm5819p_rmc: kernel debugger detected: hardware watchdog disabled >> Returned from _init, val = 0 >> mem = 4194304K (0x100000000) >> avail mem = 4149420032 >> installing rootnex, module id 28. >> breakpoint before calling rootnex:_init() >> _init >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> Returned from _init, val = 0 >> root nexus = Sun Fire V240 >> installing options, module id 29. >> breakpoint before calling options:_init() >> _init >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> Returned from _init, val = 0 >> installing pseudo, module id 31. >> breakpoint before calling pseudo:_init() >> _init >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> Returned from _init, val = 0 >> pseudo0 at root >> pseudo0 is /pseudo >> installing clone, module id 32. >> breakpoint before calling clone:_init() >> _init >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> Returned from _init, val = 0 >> installing scsi_vhci, module id 33. >> installing scsi, module id 34. >> breakpoint before calling scsi:_init() >> _init >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> Returned from _init, val = 0 >> breakpoint before calling scsi_vhci:_init() >> _init >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> Returned from _init, val = 0 >> installing scsi_vhci_f_asym_sun, module id 35. >> breakpoint before calling scsi_vhci_f_asym_sun:_init() >> _init >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> Returned from _init, val = 0 >> installing scsi_vhci_f_asym_lsi, module id 36. >> breakpoint before calling scsi_vhci_f_asym_lsi:_init() >> _init >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> Returned from _init, val = 0 >> installing scsi_vhci_f_asym_emc, module id 37. >> breakpoint before calling scsi_vhci_f_asym_emc:_init() >> _init >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> Returned from _init, val = 0 >> installing scsi_vhci_f_sym_emc, module id 38. >> breakpoint before calling scsi_vhci_f_sym_emc:_init() >> _init >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> Returned from _init, val = 0 >> installing scsi_vhci_f_sym_hds, module id 39. >> breakpoint before calling scsi_vhci_f_sym_hds:_init() >> _init >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> Returned from _init, val = 0 >> installing scsi_vhci_f_sym, module id 40. >> breakpoint before calling scsi_vhci_f_sym:_init() >> _init >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> Returned from _init, val = 0 >> installing scsi_vhci_f_tpgs, module id 41. >> breakpoint before calling scsi_vhci_f_tpgs:_init() >> _init >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> Returned from _init, val = 0 >> scsi_vhci0 at root >> scsi_vhci0 is /scsi_vhci >> installing zfs, module id 16. >> installing specfs, module id 7. >> installing fifofs, module id 8. >> breakpoint before calling fifofs:_init() >> _init >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> Returned from _init, val = 0 >> breakpoint before calling specfs:_init() >> _init >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> Returned from _init, val = 0 >> installing swrand, module id 17. >> installing kcf, module id 18. >> breakpoint before calling kcf:_init() >> _init >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> Returned from _init, val = 0 >> installing sha1, module id 19. >> breakpoint before calling sha1:_init() >> _init >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> Returned from _init, val = 0 >> breakpoint before calling swrand:_init() >> _init >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> Returned from _init, val = 0 >> installing idmap, module id 20. >> installing doorfs, module id 21. >> breakpoint before calling doorfs:_init() >> _init >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> Returned from _init, val = 0 >> installing rpcmod, module id 22. >> installing tlimod, module id 23. >> breakpoint before calling tlimod:_init() >> _init >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> Returned from _init, val = 0 >> breakpoint before calling rpcmod:_init() >> _init >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> Returned from _init, val = 0 >> breakpoint before calling idmap:_init() >> _init >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> Returned from _init, val = 0 >> installing SDC, module id 24. >> breakpoint before calling SDC:_init() >> _init >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> Returned from _init, val = 0 >> installing sha2, module id 25. >> breakpoint before calling sha2:_init() >> _init >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> Returned from _init, val = 0 >> installing skein, module id 26. >> breakpoint before calling skein:_init() >> _init >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> Returned from _init, val = 0 >> installing edonr, module id 27. >> breakpoint before calling edonr:_init() >> _init >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> Returned from _init, val = 0 >> breakpoint before calling zfs:_init() >> _init >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> :c >> >> panic[cpu1]/thread=180e000: vmem_xalloc(): size overflow >> >> Warning - stack not written to the dumpbuf >> 000000000180ad60 genunix:vmem_xalloc+7c4 (30000018d98, ffffffffffffffc8, 0, >> 0, 1, 0) >> %l0-3: 0000000000002000 0000000000000000 0000030000018db8 ffffffffffffffff >> %l4-7: 0000000000000001 0000000000080000 ffffffffffffffc0 0000000000000001 >> 000000000180aea0 genunix:vmem_alloc+1b0 (0, ffffffffffffffc8, 0, 0, f0, 0) >> %l0-3: 0000030000018d98 0000030000018db8 ffffffffffffffc0 0000000000002006 >> %l4-7: 0000000000000000 0000000000000000 0000000000003000 000003000004a760 >> 000000000180af60 genunix:kmem_alloc+194 (ffffffffffffffc8, 0, 80, 0, >> 103a204, 0) >> %l0-3: 0000000000000100 000003000004a588 00000000000000d0 00000000012934c8 >> %l4-7: 00000000000000d0 0000000000000000 0000000000000000 0000000000000000 >> 000000000180b010 genunix:kmem_zalloc+f0 (ffffffffffffffc8, 0, 6, 0, 103a1fc, >> 0) >> %l0-3: ffffffffffffffc8 0000000000000004 00000000018243b0 00000000012934c8 >> %l4-7: 000003000146fba0 0000000001834000 0000000000000000 0000000000000720 >> 000000000180b0c0 zfs:aggsum_init+48 (701be8e0, 0, 7be27c60, 7bf04e88, 1, 3) >> %l0-3: 0000000001824000 0000000001814508 000000000180b158 0000000001824000 >> %l4-7: 000003000146fcb0 0000000001834000 0000000000000000 0000000000000090 >> 000000000180b170 zfs:arc_state_init+1ec (0, 3, 0, 0, 1, 195de68) >> %l0-3: 000000000195df48 000000000195dc00 000000000195dc00 000000000195dc00 >> %l4-7: 000000000195dc00 000000000195dc00 000000000195dc00 000000000195df48 >> 000000000180b220 zfs:arc_init+22c (300005bb608, 1000000, ffffffffffffe000, >> 0, 3fff8, fffffffffffc0008) >> %l0-3: 000000000195a7c8 000000000181c800 00000000701bec00 0000000020000000 >> %l4-7: 000000000183e000 000000000186c000 0000000000000000 000000000180b1f0 >> 000000000180b2e0 zfs:dmu_init+44 (7bf11bc0, 30001692000, 2000, 1963ba8, >> 20000, 8000) >> %l0-3: 0000000001000000 0000000000008000 0000000000040000 0000000000002000 >> %l4-7: 0000000000030000 00000000701fedd0 00000000701bedb8 000000000180b368 >> 000000000180b390 zfs:spa_init+e4 (3, 1, 0, 185e400, 1fff, 12889fc) >> %l0-3: 0000000001868800 0000000000000000 0000060010c50f72 00000000018688b0 >> %l4-7: 0000000000000001 00000300000edac0 0000000000000000 000000000180b380 >> 000000000180b440 zfs:_init+8 (12f2d00, 30000424650, 10, 10cf530, 0, 0) >> %l0-3: 000000000195d800 00000300000b7048 000003000042457b 0000030000424640 >> %l4-7: 00000000018460e0 000000000183a340 0000000000000000 000007fff0017180 >> 000000000180b4f0 genunix:modinstall+94 (7becd9a0, 1, 12f2add, 180b700, 0, 0) >> %l0-3: 00000300000b7728 00000000018ae000 00000000012f2c00 0000000000000000 >> %l4-7: 0000000000000000 0000000000000020 0000000000000000 0000000000000001 >> 000000000180b5b0 genunix:mod_hold_installed_mod+a4 (300000b7728, 0, 0, >> 180b71c, 81010100, ff00) >> %l0-3: 0000000000000007 00000300000458c8 00000300014ec000 00000300014ea770 >> %l4-7: 0000000000000000 0000000001895798 00000300016a2000 0000000000000004 >> 000000000180b660 genunix:modrload+94 (13fe990, 1843ec8, 0, 0, 103a204, 0) >> %l0-3: 0000030000424578 0000000000000007 00000000000000d0 00000000012934c8 >> %l4-7: 0000000000000000 000003000041e498 0000000000000004 0000000000000001 >> 000000000180b720 genunix:modload+c (13fe990, 1843ec8, 3, 186aeb8, 186aec0, >> 3000004d288) >> %l0-3: 000003000004d088 0000000000000003 0000000000002000 0000000000002006 >> %l4-7: 0000000000000000 000000000181a0a8 0000000000003000 000003000004d260 >> 000000000180b7d0 swapgeneric:rootconf+44 (1843c00, 0, 4, 0, 1, 26) >> %l0-3: 00000000013fe800 00000300000b7a48 0000000001818a78 0000000001824000 >> %l4-7: 00000300014ea660 0000000001834000 0000000000000000 0000000000000720 >> 000000000180b880 unix:stubs_common_code+70 (30000371000, 0, 4, 0, 1, 26) >> %l0-3: 000000000180b139 000000000180b201 0000000000000000 0000000001812e38 >> %l4-7: 0000000000000000 0000000001818a88 00000000000002b0 0000000000000001 >> 000000000180b940 genunix:vfs_mountroot+58 (3000007d088, fa0, >> fffffffffffffff8, 0, 0, 0) >> %l0-3: 000000000186c000 000003000007d0e8 ffffffffffffffff 0000000001812e38 >> %l4-7: 000003000007d0f0 000000000186c000 0000000000000000 0000000000000000 >> 000000000180ba00 genunix:main+118 (180bb80, 180bec0, 0, 0, f0059684, 0) >> %l0-3: 000000000183a340 0000000000000001 00000000018bac00 000000000180bad0 >> %l4-7: 00000000018460e0 000000000183a340 0000000000000000 000007fff0017180 >> >> panic: entering debugger (no dump device, continue to reboot) >> kmdb: target stopped at: >> kmdb_enter: ta %icc, 0x7d >> [1]> >> as you can see, we have panic with load zfs module. >> >> if i revert commit - boot is fine. >> question is: how to try fix it? >> ideas are welcome. >> >> i have no panic on SPARC sun4v T5220, but has panic on sun4 sun fire v240 >> >> best regards, >> -Igor >> > > openzfs <https://openzfs.topicbox.com/latest> / openzfs-developer / see > discussions <https://openzfs.topicbox.com/groups/developer> + participants > <https://openzfs.topicbox.com/groups/developer/members> + delivery options > <https://openzfs.topicbox.com/groups/developer/subscription>Permalink > <https://openzfs.topicbox.com/groups/developer/T86a13d435170569e-Me8f3d630c0934625d33a8511> ------------------------------------------ openzfs: openzfs-developer Permalink: https://openzfs.topicbox.com/groups/developer/T86a13d435170569e-Mcd74fcbd9477818f1094ce35 Delivery options: https://openzfs.topicbox.com/groups/developer/subscription
