Ali, Ah I see.
Those are the pre-allocated log files by zookeeper. 64M is the default size. The Message Router repo contains the Docker related set up files and an initial state snapshot. This initial state records a number of pre-configured topics whose API keys have been configured into other components. Those log files of zookeeper are part of this initial state. So we need those files. We have this initial state snapshot here in git for version control, i.e. adding new pre-configured topics bumps version. In those log files, other than the first couple dozen bytes or so, it is all 0x00. Which is why the git clone size being so small, due to the benefit of compression. Given that you are mostly concerned with the git sizes, these mostly-0 log files are not making much trouble there. In future releases, as Message Router evolves to better support for topic provisioning, the need for initial state will go away too. In the meanwhile, we could try to reduce the default logging size (which may have penalties associated) or move the log files to a different place say the raw file repo of ONAP Nexus. But since Message Router is used by many other components, doing these kind of changes for only couple hundred K bytes seems not very rewarding with respect to the impact. What do you think? Regards, Lusheng On Apr 21, 2017, at 9:57 AM, Ali Al-Shabibi <ali.al-shabibi at onlab.us<mailto:ali.al-shabibi at onlab.us>> wrote: Hi Lusheng, I got then but running a find on the entire onap code and listing the contents of all directories then sorting by file size. So the size you see in the list is the size you should be observing on disk unless something has changed in the meantime. Cheers. -- Ali On Apr 20, 2017, 14:59 -0500, JI, LUSHENG (LUSHENG) <lji at research.att.com<mailto:lji at research.att.com>>, wrote: Hi Ali, As I was looking into the DCAE Message Router log files you mentioned, I am wondering if you could clarify how you got the size numbers? For example, when I tried to clone the dcae/demo/startup/message-router repo, I can see the total size is about 236KB, which is vastly different from your numbers (~67M) ... Cloning into 'dcae-org.openecomp.dcae.demo.startup.message-router'... remote: Total 221 (delta 0), reused 221 (delta 0) Receiving objects: 100% (221/221), 236.65 KiB | 441.00 KiB/s, done. Resolving deltas: 100% (69/69), done. Given that git compresses a lot of things, after the cloning, a du also shows that the overall size for the while cloned dir is 394kB (394920dcae-org.openecomp.dcae.demo.startup.message-router). Regards, Lusheng On Apr 12, 2017, at 9:50 AM, Ali Al-Shabibi <ali.al-shabibi at onlab.us<mailto:ali.al-shabibi at onlab.us>> wrote: Hi Catherine, Here are the top-25 offenders. After the log files the others files aren?t all that big. -rw-r--r-- 1 ash staff 424102999<tel:424102999> Apr 6 13:42 ./sdnc/oam/installation/sdnc/src/main/resources/distribution-karaf-0.5.1-Boron-SR1.tar.gz -rw-r--r-- 1 ash staff 67108880<tel:67108880> Apr 6 13:42 ./dcae/demo/startup/message-router/docker_files/data-zookeeper/version-2/log.125 -rw-r--r-- 1 ash staff 67108880<tel:67108880> Apr 6 13:42 ./dcae/demo/startup/message-router/docker_files/data-zookeeper/version-2/log.103 -rw-r--r-- 1 ash staff 67108880<tel:67108880> Apr 6 13:42 ./dcae/demo/startup/message-router/docker_files/data-zookeeper/version-2/log.1 -rw-r--r-- 1 ash staff 3416772<tel:3416772> Apr 6 13:42 ./sdc/openecomp-ui/resources/images/ecomp/ASDC_Sprite.png -rw-r--r-- 1 ash staff 3413794<tel:3413794> Apr 6 13:42 ./sdc/catalog-ui/app/styles/images/welcome/ss-01.png -rw-r--r-- 1 ash staff 3386285<tel:3386285> Apr 6 13:42 ./sdc/catalog-ui/app/styles/images/welcome/sprite.png -rw-r--r-- 1 ash staff 3376741<tel:3376741> Apr 6 13:42 ./sdc/catalog-ui/app/styles/images/resource-icons/vl.png -rw-r--r-- 1 ash staff 3088201<tel:3088201> Apr 6 13:42 ./sdc/catalog-ui/app/styles/images/welcome/SD&C_Welcome15.jpg -rw-r--r-- 1 ash staff 3082618<tel:3082618> Apr 6 13:42 ./sdc/catalog-ui/app/styles/images/welcome/SD&C_Welcome15_b.jpg -rw-r--r-- 1 ash staff 2906163<tel:2906163> Apr 6 13:42 ./sdc/catalog-ui/app/styles/images/welcome/bg/004.png -rw-r--r-- 1 ash staff 2804026<tel:2804026> Apr 6 13:42 ./sdc/catalog-ui/app/styles/images/welcome/bg/002.png -rw-r--r-- 1 ash staff 2406510<tel:2406510> Apr 6 13:42 ./sdc/openecomp-be/tools/swagger-ui/swagger-ui.js -rw-r--r-- 1 ash staff 2127872<tel:2127872> Apr 6 13:42 ./policy/engine/packages/base/src/files/install/servers/console/bin/sql/log.h2.db -rw-r--r-- 1 ash staff 1785443<tel:1785443> Apr 6 13:42 ./dcae/dmaapbc/www/swagger/swagger-ui.js -rw-r--r-- 1 ash staff 1556910<tel:1556910> Apr 6 13:42 ./portal/ecomp-portal-FE/client/bower_components/angular-scenario/angular-scenario.js -rw-r--r-- 1 ash staff 1545132<tel:1545132> Apr 6 13:42 ./sdc/catalog-be/src/main/resources/swagger/swagger-ui.js -rwxr-xr-x 1 ash staff 1478697<tel:1478697> Apr 6 13:42 ./sdnc/oam/admportal/lib/antlr4-4.5.1.jar -rw-r--r-- 1 ash staff 1470296<tel:1470296> Apr 6 13:42 ./sdc/catalog-ui/app/styles/images/welcome/bg/003.png -rw-r--r-- 1 ash staff 1444224<tel:1444224> Apr 6 13:42 ./vid/vid/src/main/webapp/app/fusion/external/angular-1.5/angular-scenario.js -rw-r--r-- 1 ash staff 1444224<tel:1444224> Apr 6 13:42 ./ui/dmaapbc/dcae_dmaapbc_webapp/src/main/webapp/app/fusion/external/angular-1.5/angular-scenario.js -rw-r--r-- 1 ash staff 1444224<tel:1444224> Apr 6 13:42 ./policy/engine/ecomp-sdk-app/src/main/webapp/app/fusion/external/angular-1.5/angular-scenario.js -rw-r--r-- 1 ash staff 1444224<tel:1444224> Apr 6 13:42 ./ecompsdkos/ecomp-sdk/sdk-app/src/main/webapp/app/fusion/external/angular-1.5/angular-scenario.js -rw-r--r-- 1 ash staff 1412745<tel:1412745> Apr 6 13:42 ./sdc/sdc-os-chef/sdc-backend/chef-repo/cookbooks/sdc-normatives/files/default/get-pip.py -rw-r--r-- 1 ash staff 1412744<tel:1412744> Apr 6 13:42 ./sdc/asdctool/src/main/resources/es-resources/get-pip.py Cheers. -- Ali On 10 Apr 2017 15:56 -0500, Lefevre, Catherine <cl664y at intl.att.com<mailto:cl664y at intl.att.com>>, wrote: Hi Ali, Sure ? will do Thank you again for your feedback Best regards Catherine From: Ali Al-Shabibi [mailto:ali.al-shab...@onlab.us] Sent: Monday, April 10, 2017 10:53 PM To: onap-discuss at lists.onap.org<mailto:onap-discuss at lists.onap.org>; JI, LUSHENG <lji at research.att.com<mailto:lji at research.att.com>>; Lefevre, Catherine <cl664y at intl.att.com<mailto:cl664y at intl.att.com>> Subject: RE: [onap-discuss] Large files in repos Hi Catherine, I looked very briefly, I can look again and provide a list of top offenders to you soon. Feel free to remind me if I take too long to get back to you. Cheers. -- Ali On 10 Apr 2017 14:43 -0500, Lefevre, Catherine <cl664y at intl.att.com<mailto:cl664y at intl.att.com>>, wrote: Good morning Ali, We confirm that the files stored in the SDN-C Gerrit repositories will be reviewed and potentially stored in the Nexus repo https://jira.onap.org/browse/SDNC-7<https://urldefense.proofpoint.com/v2/url?u=https-3A__jira.onap.org_browse_SDNC-2D7&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=66ObImPAUA0o2f1hTGknnnv5ScXvX8EnREJCPHHBY5M&m=1nK1fCS0pfxSsQh4pzaA7EiGy9hLfhASVWnBShopFAw&s=Dps5t1OXw6pw48JVx19ooGO6t5E5h8HNksvZI_DrVlI&e=> Concerning DCAE large files, these are necessary due to the message router. Did you identify any other large file that you would like that we review? Many thanks & regards Catherine From: onap-discuss-bounces at lists.onap.org<mailto:onap-discuss-bounces at lists.onap.org> [mailto:onap-discuss-boun...@lists.onap.org] On Behalf Of Ali Al-Shabibi Sent: Thursday, April 06, 2017 9:34 PM To: onap-discuss at lists.onap.org<mailto:onap-discuss at lists.onap.org> Subject: [onap-discuss] Large files in repos Hi again, I noticed that there are quite large files checked in to a few repos in ONAP. The entire source base is about 1.8GB which is rather large. I know that you probably don?t want to download all the repos but sdnc on it?s own is 427MB. For example, in the sdnc the following file is 404MB: sdnc/oam/installation/sdnc/src/main/resources/distribution-karaf-0.5.1-Boron-SR1.tar.gz Then in the dcae repos there are three log files of 64MB each cae/demo/startup/message-router/docker_files/data-zookeeper/version-2/ I would be happy to help reducing the size of these repos just to make it easier to download. Cheers. -- Ali -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.onap.org/pipermail/onap-discuss/attachments/20170421/6dbe6e42/attachment-0001.html>
