Please do, buyers are reluctant to discuss a home or do a thorough investigation with the seller nearby. You might actually say something which could cost you a sale. You want the buyer to feel comfortable and take their time looking at your house. I know it sometimes is an inconvenience but leaving your home for a brief period of time is part of the home selling process.