As with a traditional face-to-face study, you ask people to do tasks that take them through key areas of the product. Because the respondents do the study on their own, the tasks need to be very simple and easy to understand. I use the same tasks from the live study in order to gather corroborating evidence.
Aside from being able to collect large amounts of feedback, there are other benefits to the automated approach. For example, you can segment participants to analyze how different types of users performed. Or you can have everyone do the same set of tasks but on different designs.
Some limitations of this approach
Overall, a remote unattended study can be a great complement to traditional face-to-face studies because of the large amount of data this method produces. But there are caveats and drawbacks. Your study needs to be well thought out and piloted to make sure you are asking the right questions. You can't test proprietary designs. You also need to contend with the "cheater-repeater" syndrome where participants lie about themselves to get in so they can collect the incentive. (However, there are ways of identifying these folks and tossing them out.) As with other web-based research methods (such as A/B split testing), this approach tells you what people did but not why they did it or how they felt. So, to get a complete picture, you should balance this automated method with traditional face-to-face research.