Hi Evans, thanks a lot for the feedback, it was exactly what I needed. The simpler the better is definitely a good advice in this use case, I'll try this week another rollout/rollback and report back :)
Luca On Thu, Apr 9, 2020 at 8:09 PM Evans Ye <[email protected]> wrote: > > Hi Luca, > > Thanks for reporting back and let us know how it goes. > I don't have the exactly HDFS with QJM HA upgrade experience. The experience > I had was 0.20 non-HA upgrade to 2.0 non-HA and then enable QJM HA, which was > back in 2014. > > Regarding to rollback, I think you're right: > > it is possible to rollback to HDFS’ state before the upgrade in case of > unexpected problems. > > My previous experience is the same that the rollback is merely a snapshot > before the upgrade. If you've gone far, then rollback cost more data lost... > Our runbook is if our sanity check failed during upgrade downtime, we perform > the rollback immediately. > > Regarding to that FSImage hole issue, I've experienced it as well. > I managed to fix it by manually edit the FSImage with offline image viewer[1] > and delete that missing editLog in FSImage. That actually brought my cluster > back with a little number of missing blocks. > > Our experience says that the more the steps, the more the chance you failed > the upgrade. We did good on dozen times of testing, DEV cluster, STAGING > cluster, but still got missing blocks when upgrading Production... > > The suggestion is to get your production in good shape first(the less > decommissioned, offline DNs, disk failures, the better). > Also, maybe you can switch to non-HA mode and do the upgrade to simplify the > things? > > Not many helps but please let us know if any progress. > Last one, have you reached out to Hadoop community? the authors should know > the most :) > > - Evans > > [1] > https://hadoop.apache.org/docs/r2.8.5/hadoop-project-dist/hadoop-hdfs/HdfsImageViewer.html > > Luca Toscano <[email protected]> 於 2020年4月8日 週三 21:03 寫道： >> >> Hi everybody, >> >> most of the bugs/issues/etc.. that I found while upgrading from CDH 5 >> to BigTop 1.4 are fixed, I am now testing (as suggested also in here) >> upgrade/rollback procedures for HDFS (all written in >> https://phabricator.wikimedia.org/T244499, will add documentation >> about this at the end I promise). >> >> I initially followed [1][2] in my Test cluster, choosing the Rolling >> upgrade, but when I tried to rollback (after days since the initial >> upgrade) I ended up in an inconsistent state and I wasn't able to >> recover the previous HDFS state. I didn't save the exact error >> messages but the situation was more or less the following: >> >> FS-Image-rollback (created at the time of the upgrade) - up to transaction X >> FS-Image-current - up to transaction Y, with Y = X + 10000 (number >> totally made up for the example) >> QJM cluster: first available transaction Z = X + 10000 + 1 >> >> When I tried to rolling rollback, the Namenode complained about a hole >> in the transaction log, namely at X + 1, so it refused to start. I >> tried to force a regular rollback, but the Namenode refused again >> saying that there was no available FS Image to roll back to. I checked >> in the Hadoop code and indeed the Namenode saves the fs image with >> different naming/path in case of a rolling upgrade or a regular >> upgrade. Both cases make sense, especially the first one since there >> was indeed a hole between the last transaction of the >> FS-Image-rollback and the first available transaction to reply on the >> QJM cluster. I chose the rolling upgrade initially since it was >> appealing: it promises to bring back the Namenodes to their previous >> versions, but keeping the data modified between upgrade and rollback. >> >> I then found [3], in which it is said that with QJM everything is more >> complicated, and a regular rollback is the only option available. What >> I think this mean is that due to the Edit log spread among multiple >> nodes, a rollback that keeps data between upgrade and rollback is not >> available, so worst case scenario the data modified during that >> timeframe is lost. Not a big deal in my case, but I want to triple >> check with you if this is the correct interpretation or if there is >> another tutorial/guide/etc.. that I haven't read with a different >> procedure :) >> >> Is my interpretation correct? If not, is there anybody with experience >> in HDFS upgrades that could shed some light on the subject? >> >> Thanks in advance! >> >> Luca >> >> >> >> [1] >> https://hadoop.apache.org/docs/r2.8.5/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Upgrade_and_Rollback >> [2] >> https://hadoop.apache.org/docs/r2.8.5/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html >> [3] >> https://hadoop.apache.org/docs/r2.8.5/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html#HDFS_UpgradeFinalizationRollback_with_HA_Enabled
