I think it better if you declare the boundaries before you’re in private with him, knowing that some men think that agreement to go back to “somebody’s place” is a guarantee of sex.
Seems pretty simple. You get asked to go to “his place”; you tell him you’d like to, but you’re not into sex on the first/second/third date or a more vague “this early in a relationship”.
If he bolts, then you know you just saved yourself from having problems with an asshole. If he agrees, and you change your mind later, everyone’s happy.