The XR650 is a great bike; I imagine you had plenty 'o fun riding it. Of the 3 Japanese big-bore singles, I've owned 2: the klr650 and now my DR650 and I've ridden a friend's XR--seemed very capable, much lighter, especially compared with the KLR.
But I've never ascribed to the "not riding hard enough if you didn't fall over at least once a ride". I don't own dual-purpose bikes to ride hard and push limits to the point I'm likely to fall every ride. I guess I get it, some guys love the speed, the adrenline...the risk, too. I enjoy a little of that, but then I get tired of it quickly. I ride to get to places with beautiful views, to moto-camp, take pictures!, get to a trailhead to hike, find a place to read a book, to enjoy the forests and deserts of Oregon...and to get back home in one piece with no pain and no broken bike...that's a good ride for me!
And I like to ride alone, and to remote places. I always have those two factors in mind--alone and remote. So I ride in a way and choose levels of difficulty that match the bike I'm on, and match my desire to not injure myself while I might be hours away from anyone (I take a few more risks on my KLX300 as it weighs much less than the DR650).
I have a friend I ride with who is really good off-road. But we understand each other...he'll take off and lose me for awhile, then take a water / navigation / nature break while I catch up. He pushes me sometimes and I like the challenge, but I think he also likes knowing that I'm coming up behind him in case something happens to him.