[pcabc]

My developer has been working on it since Christmas, so is a big endeavour, but is initially only going to be focused on the US markets, as data costs for a commercial licence are at least ten times that of those for personal use per month. So am currently negotiating with a number of data providers to try and get what I need and to make the product viable for me. As the first quote I got came in at $1250 a month, which is just for open, high, low, close, volume data. So was concerned that I wouldn't be able to get it done, but have since found a couple of providers willing to do it in the hundreds, not thousands a month. As redistributing data on a website is very different price wise than personal use. But hopefully, with the higher price, also comes more sanitised data for errors, like you've mentioned. But we'll see in time with that.

That is a massive leap there, I really hope it works well.  I suppose the problem is that things are either tailored for individual traders or institutions, and even the smallest institution will have deep pockets.  But that is supposition?  Starting with the US market has got to make sense, it will get the most attention in the English and likely non-English speaking worlds.  The performance in the UK is nowhere near it no matter how nice it would be to include it, and even if UK performance had matched or exceeded then for the aforementioned reasons the US would need to be the focus.  Plus, at least my screening experience is that very good Stage Analysis screens can be rare beasts - you really do need to trawl a really wide pool and the US pool of stocks is the largest.

I'm unsurprised by the amount of effort.  Over the years I've put many many man hours in my database / screener.  Yes, my screener 'happened' rather than was planned.  But if I had to start from fresh tomorrow it would take a massive chunk of time.  Plus - what I do has less pressure to look nice and I can tolerate quirks as its not public or customer facing.

Frustratingly I've implemented additional Mansfield RS breadth measures that I mentioned in earlier posts - but this morning - coincident with a minor tweak that I soon reversed, it stopped working.  No new breadth calcs - at least the ones I wanted, appeared in my database.  I think I'm low on ram, likely am for what I am doing.  Low and behold, a couple of hours later it seems to be working - nothing to do with any fixes I tried.  But only after having ordered more RAM.  So either I've splashed out on quadrupling my RAM for only a marginal benefit and I will kick myself for a marginal purchase - or it will give me a massive speedup and I will kick myself for not doing it sooner.  There is a hint that it was the web browser making the very slow to process request might be the issue?

(if anyone suggests I'm a fool for spending more money on RAM when it was working with the same amount of RAM hours earlier, I note that machines can really bog down and become unusable if they run low on RAM and swap a lot.  I think my machine may have been marginal and just tipped over the edge.  Furthermore, there is not much I could do about memory usage without a large investment of time and inconvenience, cutting capability and then I'd probably need more RAM anyway.)

It will be interesting to see how what you are doing scales? Hopefully cloud will help?  And your developer is extremely likely to know more about this than my amateur efforts.  On my personal machine here, as its operating both as the client machine, web server and database server (as this seemed a simple way to do things) I think that provided its not doing the Mansfield RS calculations it could likely serve more than a single user if I were so inclined and data licenses were no issue.  But it definitely starts to bog down with the breadth calcs.  Furthermore, it was hard to see if it was memory, CPU or IO limited.  It seemed to be frustratingly, relying on few threads, to although I've an 8-core 16 thread machine more cores did not necessarily help with speed.

So, hopefully your endeavors go well, and I've fixed my issues and increasing the RAM gives me a good speedup.