[isatrader]

I found that pulling data is not as simple as it sounds.  Its easy until you find that there are market holidays which vary by market, that cryptos and forex (?) have data on weekends where stocks and commodities do not.  In the end I decided to just ignore market holidays, it was simpler.  So for market breadth purposes I make a copy of the indices and fill in the gaps caused by market holidays.  Another piece of fun is that sometimes stock data can be slow, even perhaps for individual stocks.  So I bulk download end of day data.  Then at some point overnight I have a script that checks for stocks missing data and it tries to update that data individually.  If a stock fails to be updated that way 10 times in a row then I assume that stock ticker is dead.

The smaller cap stocks seem to have more data issues than the larger ones.  Perhaps they simply don't trade on some days?

When free data was available Yahoo had a reasonable API, Google was second.  When they stopped I found it took me a while to find a provider at an affordable price.  Unsure whether some were commercial feeds rather than for individuals, but the costs could escalate quickly.

As for sector and group info.  There is some free data for some markets that I can find from time to time.  But each source has its own format. I could get the data from my existing data provider, but I would have to update my package to one four times the price, so I'm not keen to go that route right now.

Nice clear charts.

My developer has been working on it since Christmas, so is a big endeavour, but is initially only going to be focused on the US markets, as data costs for a commercial licence are at least ten times that of those for personal use per month. So am currently negotiating with a number of data providers to try and get what I need and to make the product viable for me. As the first quote I got came in at $1250 a month, which is just for open, high, low, close, volume data. So was concerned that I wouldn't be able to get it done, but have since found a couple of providers willing to do it in the hundreds, not thousands a month. As redistributing data on a website is very different price wise than personal use. But hopefully, with the higher price, also comes more sanitised data for errors, like you've mentioned. But we'll see in time with that.