In many cases, this has much to do with the app itself, certain settings in Windows 11/10, or your network connection. Thankfully, there are ways to increase the Download Speed in Xbox app ...
Xbox's Cloud Gaming and remote play services are seeing excessively long load times as holiday demand hits servers.