Made it go away by refreshing cursor settings inside Windows mouse settings tab, which made the problem disappear for couple hours but it just came back again. It didn't happen the first 1-2 days I had the 6600 installed. One more thing is, this has just recently started happening.
Tried both the beta and the latest, both seem to do it. I didn't use DDU, just uninstalled Nvidia drivers and went on with the AMD set. This has been happening ever since I plugged out my 3060 Ti to put in my RX 6600 (I don't game anymore). While looking up on the issue over Reddit, I noticed it becomes completely unseen when I highlight some texts over the Reddit pages.
So my cursor seems to disappear a good chunk of the time when I highlight search bars.
