Hi , I need some suggestions related to setting up scrapers with Airflow.
Issue is lets say i have a website to scrape that has some 3k links. I want to divide this over 3 batches of 1k each and all these have to run on different days . What can be the best approach in this case? If we can do some conditional parameter basis scheduling. Example I have excel as my data source so next to every url i can mention like Batch no ... now basis batch number if we can schedule differently? Hope this makes some sense. Please suggest .
