Crowd contributions to Dataset Creation
24/12/14 18:29
Dear study participant/Mechanical Turk Worker,
Thank you for doing our HITs. We also appreciate you following this link to find out more about us.
We are a group composed of researchers at multiple universities around the world who work in small teams. We are interested in improving Internet technology, especially search engines that help people find images, videos or music.
We publish HITs on Mechanical Turk because we would like to gather the judgments, opinions, or insights of a large and diverse group of people. We need many different perspectives in order to make our techniques better. In turn, we hope that our research will benefit Internet users at large.
Our HITs have gotten better over the years because of people’s feedback, both the positive and the negative. We include a comment box at the bottom or our HITs. Please leave your comments there for us. We also encourage you to participate in MTurk fora. However, the most direct way to make sure we get your feedback is to use the comment box.
We use excerpts from your answers in our scientific research. Your responses are used for the stated purposes of the study. We are careful to disassociate all of your responses from you personally. Once we have collected your responses, we disassociate them with your worker ID, and anything else that might identify you.
For many of our HITs there are no “right” or “wrong” answers. Instead we are asking for your judgment or opinion. Often there will be a question that asks for an answer written as one or more complete sentences. This is because we need to see that you are reflecting on your answers, and also we need to understand clearly what you are telling us. If a question requests a complete-sentence answer, and you don’t answer with a complete sentence, your answer is genuinely not useful to us. We try to offer a margin for error, but we can’t afford to pay for large quantities of responses that we cannot use.
If you find yourself giving the same answer to the questions that ask you to write complete sentences, then something is wrong. These questions are posed that the answers are not all identical. Please stop to consider whether you really understand the HIT.
We are not infallible, so sometimes something goes awry on our side. As mentioned already, we include a comment box at the bottom of our HITs, and we encourage you to put a comment there if you suspect that something is wrong with the HIT. We look at all comments individually, and keep a close watch out from problems that we may have unwittingly caused ourselves. We try our best not to penalize you for them, and take the time to discuss unclear cases with you by email. In email discussion, we respond best to factual statements of what went wrong from your perspective. Messages formulated as threats are less likely to lead to successful resolution.
Our budget is limited, so often our rewards are lower than they might otherwise be. We appreciate that you keep in mind that we are non-profit, and we understand if you prefer to do HITs for which your earn a higher reward. Please comment in the comment box at the bottom of the HIT, if you want to give us feedback on the reward levels, or any other aspect of our HITs.
We try to introduce an element of intellectual challenge in our HITs, and also an element of fun. We hope that you realize how much we value you as a study participant contributing our research.
We may take a bit longer to review your HITs than other requestors since we are a small team. However, we try to be as quick as we can, which we consider to be within three days. Thank you for your patience.
Although we very much appreciate Mechanical Turk, and the contributions of the community of Mechanical Turk Workers, in general, we have some criticisms of the way that Amazon runs Mechanical Turk. We would be able to offer more HITs if Amazon were to support us with better best practices for quality control and pricing, and help to mediate on those occasions that we encounter a tough call.
Take away either the requestors or the workers, and Mechanical Turk will cease to exist. It doesn’t make sense to us that Amazon does not view us all as equal parties in the MTurk community.
While still hoping that Amazon will reform MTurk, in the meantime we make our best effort to offer you a small stream of interesting HITs, and to maintain your trust as our study participants, and workers working on our HITs.
Thanks you and best regards,
Martha Larson
Delft University of Technology
Thank you for doing our HITs. We also appreciate you following this link to find out more about us.
We are a group composed of researchers at multiple universities around the world who work in small teams. We are interested in improving Internet technology, especially search engines that help people find images, videos or music.
We publish HITs on Mechanical Turk because we would like to gather the judgments, opinions, or insights of a large and diverse group of people. We need many different perspectives in order to make our techniques better. In turn, we hope that our research will benefit Internet users at large.
Our HITs have gotten better over the years because of people’s feedback, both the positive and the negative. We include a comment box at the bottom or our HITs. Please leave your comments there for us. We also encourage you to participate in MTurk fora. However, the most direct way to make sure we get your feedback is to use the comment box.
We use excerpts from your answers in our scientific research. Your responses are used for the stated purposes of the study. We are careful to disassociate all of your responses from you personally. Once we have collected your responses, we disassociate them with your worker ID, and anything else that might identify you.
For many of our HITs there are no “right” or “wrong” answers. Instead we are asking for your judgment or opinion. Often there will be a question that asks for an answer written as one or more complete sentences. This is because we need to see that you are reflecting on your answers, and also we need to understand clearly what you are telling us. If a question requests a complete-sentence answer, and you don’t answer with a complete sentence, your answer is genuinely not useful to us. We try to offer a margin for error, but we can’t afford to pay for large quantities of responses that we cannot use.
If you find yourself giving the same answer to the questions that ask you to write complete sentences, then something is wrong. These questions are posed that the answers are not all identical. Please stop to consider whether you really understand the HIT.
We are not infallible, so sometimes something goes awry on our side. As mentioned already, we include a comment box at the bottom of our HITs, and we encourage you to put a comment there if you suspect that something is wrong with the HIT. We look at all comments individually, and keep a close watch out from problems that we may have unwittingly caused ourselves. We try our best not to penalize you for them, and take the time to discuss unclear cases with you by email. In email discussion, we respond best to factual statements of what went wrong from your perspective. Messages formulated as threats are less likely to lead to successful resolution.
Our budget is limited, so often our rewards are lower than they might otherwise be. We appreciate that you keep in mind that we are non-profit, and we understand if you prefer to do HITs for which your earn a higher reward. Please comment in the comment box at the bottom of the HIT, if you want to give us feedback on the reward levels, or any other aspect of our HITs.
We try to introduce an element of intellectual challenge in our HITs, and also an element of fun. We hope that you realize how much we value you as a study participant contributing our research.
We may take a bit longer to review your HITs than other requestors since we are a small team. However, we try to be as quick as we can, which we consider to be within three days. Thank you for your patience.
Although we very much appreciate Mechanical Turk, and the contributions of the community of Mechanical Turk Workers, in general, we have some criticisms of the way that Amazon runs Mechanical Turk. We would be able to offer more HITs if Amazon were to support us with better best practices for quality control and pricing, and help to mediate on those occasions that we encounter a tough call.
Take away either the requestors or the workers, and Mechanical Turk will cease to exist. It doesn’t make sense to us that Amazon does not view us all as equal parties in the MTurk community.
While still hoping that Amazon will reform MTurk, in the meantime we make our best effort to offer you a small stream of interesting HITs, and to maintain your trust as our study participants, and workers working on our HITs.
Thanks you and best regards,
Martha Larson
Delft University of Technology