[ https://issues.apache.org/jira/browse/SOLR-17460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17884426#comment-17884426 ]
Arda commented on SOLR-17460: ----------------------------- To successfully migrate the collection with all data intact, I followed these steps: *1. Prepare the Collection Configuration:* First, I retrieved the Solr collection's configuration files from the Solr 7.0 cluster and modified them to be compatible with Solr 8.4. I then created the new collection on the Solr 8.4 cluster without any data, using the updated configuration. *2. Copy Shard Core Nodes Locally:* I copied each shard’s core nodes from the HDFS path of the Solr 7.0 cluster to a local directory. Using the Solr Web UI, I identified which nodes corresponded to each shard and replica for the collection. Note that the core node paths may differ for each collection. Here’s an example of the commands used: _# Shard 1, 2, and 3_ hdfs dfs --copyToLocal <solrCollectionPath>/core_node1 <localCorePath> _# Shard 1_ hdfs dfs --copyToLocal <solrCollectionPath>/core_node2 <localCorePath> _# Shard 2_ hdfs dfs --copyToLocal <solrCollectionPath>/core_node3 <localCorePath> _# Shard 3_ _# Replica nodes_ hdfs dfs --copyToLocal <solrCollectionPath>/core_node6 <localCorePath> _# Replica 1_ hdfs dfs --copyToLocal <solrCollectionPath>/core_node8 <localCorePath> _# Replica 2_ hdfs dfs --copyToLocal <solrCollectionPath>/core_node10 <localCorePath> _# Replica 3_ After copying the core node files locally, I transferred them to the Solr 8.4 cluster. *3. Copy Shard Core Nodes to HDFS on the New Cluster:* I copied the shard core node files into the appropriate HDFS directory on the new Solr 8.4 cluster. *Important:* The shard core nodes must be copied into the exact corresponding location. For example, if shard 1 was stored in *_core_node1_* in the old cluster and is now assigned to *_core_node5_* in the new cluster, you must copy the data from *_core_node1_* to *_core_node5._* Example commands: _# Copy shard 1, 2, and 3 core nodes_ hdfs dfs -put <localCorePath>/core_node1 <newSolrCollectionPath>/core_node5 _# Shard 1_ hdfs dfs -put <localCorePath>/core_node2 <newSolrCollectionPath>/core_node6 _# Shard 2_ hdfs dfs -put <localCorePath>/core_node3 <newSolrCollectionPath>/core_node7 _# Shard 3_ _# Copy replica core nodes_ hdfs dfs -put <localCorePath>/core_node6 <newSolrCollectionPath>/core_node11 _# Replica 1_ hdfs dfs -put <localCorePath>/core_node8 <newSolrCollectionPath>/core_node12 _# Replica 2_ hdfs dfs -put <localCorePath>/core_node10 <newSolrCollectionPath>/core_node9 _# Replica 3_ *4. Adjust Ownership in HDFS:* I changed the ownership of the collection’s HDFS path on the new cluster to ensure Solr had the necessary permissions to access the data. hdfs dfs -chown -R solr:solr <newSolrCollectionPath> To verify that the files were correctly copied, I compared the file sizes on both clusters using the following command: hdfs dfs -du -s -h -v -x <collectionPath> *5. Reload the Collection:* Finally, I reloaded the collection via the Solr Web UI on the new Solr 8.4 cluster. To confirm the successful migration of all data, I queried the collection and verified that the document count matched the original. With these steps, I was able to migrate the entire collection along with all data. You can check the document count by running a simple query to verify that the migration was successful. > Error During Collection Migration from Solr 7.0 to Solr 8.4: Missing Files > and Shard Restoration Failures > --------------------------------------------------------------------------------------------------------- > > Key: SOLR-17460 > URL: https://issues.apache.org/jira/browse/SOLR-17460 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: hdfs, SolrCloud > Affects Versions: 7.0, 8.4 > Reporter: Arda > Priority: Minor > Labels: backup, restore > > I was attempting to migrate a collection with 3 shards from a Solr 7.0 > cluster to a Solr 8.4 cluster. The data is stored in HDFS. I followed the > backup-restore process but encountered issues with two of the shards during > the restoration. > h1. *Migration Process:* > *1-* *Backup Command:* To avoid timeouts, I initiated the backup with an > async parameter: > curl -k --negotiate -u : > 'https://<solrNode>:<solrPort>/solr/admin/collections?action=BACKUP&name=<backupName>&collection=<solrCollectionName>x&location=<hdfsPath>& > async=12346' > *2- Copy Backup to Local:* After the backup, I copied the data from HDFS to > the local filesystem: > hdfs dfs --copyToLocal <backupPath> <localPath> > *3- Transfer Backup to New Cluster:* I then copied the backup files from the > older Solr node to the newer one: > scp -pr <localPath> <username>@<ip>:<localPathDestination> > *4- Prepare New HDFS Path:* On the new Solr cluster, I created a new > directory in HDFS and adjusted ownership: > hdfs dfs -mkdir <pathName2> > hdfs dfs -chown solr:solr <pathName2> > *5- Copy Backup to New HDFS Location:* I transferred the backup data from > local to the new HDFS path. Before that, I deleted > "<str>queryDocAuthorization</str>" parts from solrconfig.xml file to become > compatible with the newer version. > hdfs dfs --copyFromLocal <localPathDestination> <pathName2> > *6- Restore Collection:* Finally, I ran the restore command: > curl -k --negotiate -u : > 'https://<solrNode>:<solrPort>/solr/admin/collections?action=RESTORE&name=<backupName>&collection=<solrCollectionName>x&location=<hdfs_path>& > async=12345' > h1. > *Issue:* > After the restore process completed, I found that two of the shards could not > be restored. The logs displayed the following errors: > *Error During Shard Restoration:* > ERROR [c: <solrCollectionName> s: shard2 r:core_node5 x: : > <solrCollectionName>_shard2_replica_n4] o.a.s.h.RequestHandlerBase > org.apache.solr.common. SolrException: Error CREATEing SolrCore > '<solrCollectionName>_shard2_replica_n4': Unable to create core > [:<solrCollectionName>_shard2_replica_n4] Caused by: > org.apache.solr.handler.component.QueryDocAuthorizationComponent..... > *FileNotFoundException and Index Corruption:* > WARN > (parallelCoreAdminExecutor-6-thread-7-processing-n:<solrNode>:<solrPort>_solrx:<solrCollectionName>_shard2 > _replica_n1 <numbers> RESTORECORE) [x:<solrCollectionName>_shard2_replica_n1] > o.a.s.h. RestoreCore Could not switch to restored index. Rolling back to the > current index => org.apache.lucene.index.CorruptindexException: Unexpected > file read error while reading index. > (resource=BufferedChecksumIndexInput(segments_1g9dk)) > Caused by: java.io. FileNotFoundException: File does not exist: > hdfs://<hdfsPath>/core_node2/data/restore/<fileName> > It appears that Solr is looking for a file in HDFS that doesn't exist, > despite no manual deletions being made. I cannot determine why these specific > shards failed to restore, or why the system is unable to locate the required > files. > Expected Behavior: > The backup and restore process should complete without errors, and all shards > should be restored successfully to the new cluster. > Actual Behavior: > Two shards failed to restore, with errors related to missing files and index > corruption. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org