Or, indeed, if they're actually members of your family. If you've named and linked them, and as long as you're not both trying to play the same game at once, they could be just about anywhere.
In an interview with Ars Technica, Microsoft Xbox Chief Marketing and Strategy Officer Yusuf Mehdi drops a lot more detail on how Xbox One's very controversial digital strategy will actually work. It's well worth reading in its entirety, but the critical section for those looking to share games is this:
"For another, they don't have to live in the primary owner's house—I could name a friend that lives 3,000 miles away as one of my "family members" Mehdi said. You'll be able to link other Xbox Live accounts as having shared access to your library when you first set up a system and will also be able to add them later on (though specific details of how you manage these relationships is still not being discussed). The only limitation, it seems, is that only one person can be playing the shared copy of a single game at any given time.
Thanks to Jarrard for the tip!