My Guy Stays At My Apartment When I'm Not There

Leo and Rebecca Buzz Question....Me and my man have been dating for about 4 months now and things are really good. He's actually been staying over my apartment every now and then. HERE'S THE THING...the other day for the first time, I went to work AND HE STAYED IN MY APARTMENT ALONE WITHOUT ME THERE....He's kind of been making it a habit of doing that lately and HONESTLY I'm not COOL WITH HIM BEING AT MY APARTMENT WHEN IM NOT THERE...How do I tell him without hurting his feelings or making him mad about it...>???