Hello, On 06 Mar 2014, at 09:13, Marcel Offermans <[email protected]> wrote:
> Hello Damien, > > On 05 Mar 2014, at 12:20 pm, Damien Martin-Guillerez > <[email protected]> wrote: > >> After a disk quota excess, our ACE server started to behave weird: when a >> new target arrives with only one bundle in the deployment list, the >> deployment fails with "[ERROR] 11:35:02 (deployment) Stream does not contain >> a valid Jar / java.io.IOException: Unknown/unexpected status code: 500”. >> When looking at the server log, I get : >> 2014.03.05 12:14:03 WARNING - Bundle: org.apache.ace.deployment.servlet - >> 500:Could not deliver package - org.apache.felix.log.LogException: >> org.apache.ace.deployment.servlet.AceRestException: 500:Could not deliver >> package >> … >> caused by a connection refused. > > If at some point the server could no longer write to disk because of those > quota, and you're getting error messages when a target tries to deploy an > update, a few things could be wrong. > > When a target polls, the server first fetches the deployment repository, > looks up the version (or versions) and calculates the (delta of) bundles that > need to be sent. It could be that this deployment repository is corrupt. On > disk it is stored as a gzip'd XML file in the bundle cache of the repository > bundle. So that's one place to look. Another possibility is that one of the > artifacts or the index in the OBR is corrupt. The index (repository.xml in > the store folder) you can always delete and ACE will recreate it. If any of > the artifacts is corrupt, try deleting it via ACE and then upload it again. Ok I’ve deleted the repository.xml file and it was regenerated but the bug persisted. I tried going into the bundle cache data’s folders. Here are the bundle that contains data: bundle23/data org.apache.ace.client.repository.impl.jar bundle26/data org.apache.ace.configurator.useradmin.task.jar bundle3/data org.apache.felix.useradmin.filestore-1.0.1.jar bundle40/data org.apache.ace.log.server.store.file.jar bundle5/data org.apache.felix.prefs-1.0.4.jar bundle50/data org.apache.ace.repository.impl.jar bundle52/data org.apache.ace.resourceprocessor.useradmin.jar bundle59/data org.apache.ace.webui.vaadin.jar bundle6/data org.apache.felix.configadmin-1.8.0.jar I suppose that the org.apache.ace.repository.impl bundle is the one containing the potentially broken data. I went in it and there is two folders: repos and tmp. Deleting tmp made no change, deleting repos cleared all the links data that I would like to keep (If I cannot do it, I will simply drop everything using the REST API). Restoring the repos folder recovers all the informations I need. The data in the repository is non understandable: $ ls -R repos/ repos/: org.apache.ace.server.repository.factory.07b73c8e-821f-4a35-bcff-634e4c6a6130 org.apache.ace.server.repository.factory.288e53af-700c-4167-843e-31602e6f9e1d org.apache.ace.server.repository.factory.6c7e85b1-f0b3-4a56-9e50-fea310d2be6f org.apache.ace.server.repository.factory.c74d457c-6595-4a55-9ab3-f2b3749b1558 org.apache.ace.server.repository.factory.ce71c886-9d19-439a-9b26-8b8cfde9d6da org.apache.ace.server.repository.factory.fdc95335-d67c-4992-b667-38809696d153 repos/org.apache.ace.server.repository.factory.07b73c8e-821f-4a35-bcff-634e4c6a6130: 1 repos/org.apache.ace.server.repository.factory.288e53af-700c-4167-843e-31602e6f9e1d: 1 109 119 129 139 149 159 169 179 189 199 208 218 228 238 248 258 268 278 31 41 51 61 71 81 91 10 11 12 13 14 15 16 17 18 19 2 209 219 229 239 249 259 269 279 32 42 52 62 72 82 92 100 110 120 130 140 150 160 170 180 190 20 21 22 23 24 25 26 27 28 33 43 53 63 73 83 93 101 111 121 131 141 151 161 171 181 191 200 210 220 230 240 250 260 270 280 34 44 54 64 74 84 94 102 112 122 132 142 152 162 172 182 192 201 211 221 231 241 251 261 271 281 35 45 55 65 75 85 95 103 113 123 133 143 153 163 173 183 193 202 212 222 232 242 252 262 272 282 36 46 56 66 76 86 96 104 114 124 134 144 154 164 174 184 194 203 213 223 233 243 253 263 273 283 37 47 57 67 77 87 97 105 115 125 135 145 155 165 175 185 195 204 214 224 234 244 254 264 274 284 38 48 58 68 78 88 98 106 116 126 136 146 156 166 176 186 196 205 215 225 235 245 255 265 275 29 39 49 59 69 79 89 99 107 117 127 137 147 157 167 177 187 197 206 216 226 236 246 256 266 276 3 4 5 6 7 8 9 108 118 128 138 148 158 168 178 188 198 207 217 227 237 247 257 267 277 30 40 50 60 70 80 90 repos/org.apache.ace.server.repository.factory.6c7e85b1-f0b3-4a56-9e50-fea310d2be6f: 1 109 119 129 139 149 159 169 179 189 199 208 218 228 238 248 258 268 32 42 52 62 72 82 92 10 11 12 13 14 15 16 17 18 19 2 209 219 229 239 249 259 269 33 43 53 63 73 83 93 100 110 120 130 140 150 160 170 180 190 20 21 22 23 24 25 26 27 34 44 54 64 74 84 94 101 111 121 131 141 151 161 171 181 191 200 210 220 230 240 250 260 270 35 45 55 65 75 85 95 102 112 122 132 142 152 162 172 182 192 201 211 221 231 241 251 261 271 36 46 56 66 76 86 96 103 113 123 133 143 153 163 173 183 193 202 212 222 232 242 252 262 272 37 47 57 67 77 87 97 104 114 124 134 144 154 164 174 184 194 203 213 223 233 243 253 263 28 38 48 58 68 78 88 98 105 115 125 135 145 155 165 175 185 195 204 214 224 234 244 254 264 29 39 49 59 69 79 89 99 106 116 126 136 146 156 166 176 186 196 205 215 225 235 245 255 265 3 4 5 6 7 8 9 107 117 127 137 147 157 167 177 187 197 206 216 226 236 246 256 266 30 40 50 60 70 80 90 108 118 128 138 148 158 168 178 188 198 207 217 227 237 247 257 267 31 41 51 61 71 81 91 repos/org.apache.ace.server.repository.factory.c74d457c-6595-4a55-9ab3-f2b3749b1558: 1 10 11 12 13 14 15 16 17 18 19 2 20 21 22 3 4 5 6 7 8 9 repos/org.apache.ace.server.repository.factory.ce71c886-9d19-439a-9b26-8b8cfde9d6da: repos/org.apache.ace.server.repository.factory.fdc95335-d67c-4992-b667-38809696d153: The first file is an xml file the others file are binary files (non-zip files). > >> Before filing any bug reports, I’d like to do two things: >> 1. try updating the ACE server to latest trunk version (I’m running >> trunk revision 1555357). Anyway, the latest jenkins build is failed (#213 >> https://builds.apache.org/job/ACE-trunk/) and I can’t figure out a way to >> simply update the server without crushing the bundle cache of the server. Is >> latest trunk version safe to use? Is there’s a way to update without losing >> the data? > > In theory it's a matter of "updating" the bundles. Right now we don't have a > standard way to do that. We are shipping releases based on a bnd feature that > creates an executable jar, and the current release of bnd has an issue that > prevents bundles from being updated (if you just replace the old jar with the > new one and launch again). This has been fixed and will be part of the next > bnd, so once that is released, we will start using that. See: > https://github.com/bndtools/bnd/issues/424 Ok too bad :( >> 2. Try to refresh / reload data from the OBR. Is that possible? Or >> should the data always be fresh? Or maybe there’s some file that might be >> broken but I cannot find it (the single deployed bundle on the repository is >> correct). > > See above, if the bundle is okay, it could still be the "repository.xml" file > that is broken. > > Greetings, Marcel > Thank you — Damien
