Hi, I have been working on a persistent log storage solution for Autotest and want to discuss my thoughts with you, seek your advice and investigate if prior work/solution exist on this front. As of now Autotest in its default config stores the logs locally on the results server. We don't get redundancy as well as very large storage capacity in this configuration. To tackle this issue I am thinking of implementing a variant of ResultsArchiver that archives the log files and stores them on HDFS.
The proposed changes are like this - 1. Config file will dictate whether to use local storage or HDFS. 2. All HDFS related configs will be in the global config file. 3. ResultsArchiver's HDFS implementation can either use python libraries or wrap command line tools to push a file on HDFS. I am even planning to explore HttpFS for Hadoop. 4. For reading the files, currently Apache file handler handles the file rendering. We can use HttpFS for accessing the files directly from HDFS and this will need some alteration to the file urls. I think this can be achieved by some rewrite rules. 5. Another solution which will be better performance-wise but harder to implement is to cache the files locally and then deliver them through Apache file handler as we are doing now. The details of this implementation are yet to be sorted out, again your feedback will be valuable here. Has this storage problem's solution been attempted in the past or do we have any existing solution inside Autotest already that I might have missed? If not, then does my proposed plan look good and will it be something we would like to see in Autotest? Please let me know your views and any questions that you might have. Thanks & regards, Dev
_______________________________________________ Autotest-kernel mailing list [email protected] https://www.redhat.com/mailman/listinfo/autotest-kernel
