Games are a common research method for researching human-robot connections, because they are found to be interactive and still have a decent level of control for the researcher and the participants (Shahid, Krahmer, & Swerts, 2014).
Control is important to create an as real as possible situation in the experiment. The reason why this method is chosen for this specific study, is because it gives an indication where between playing alone and playing with a friend the Child Robot Interaction (CRI) falls (Shahid, Krahmer, & Swerts, 2014).
The researchers found it relevant include the chance of differences between the Dutch and Pakistani culture (Shahid, Krahmer, & Swerts, 2014). They thought that in general there are differences between cultures in how they deal with robots, as a result of exposure to them in their daily life. This difference exists, because of whether the culture is individualistic or collectivistic. Another reason why the researchers found it important to add cultural differences to this research, is that there have been previous studies that already show that culture differences lead to different interactions with robots and that the same psychological findings in cross-cultural human to human interactions could also be applicable for human to robot interactions (Shahid, Krahmer, & Swerts, 2014).
Independent variables (Shahid, Krahmer, & Swerts, 2014):
Partner (alone, iCat, friend), Nominal
Culture (Pakistani, Dutch), Nominal
Age (8 years old, 12 years old), Ordinal
Dependent variable (Shahid, Krahmer, & Swerts, 2014):
Mean fun score (1= not fun at all; 7= a lot of fun), Ratio
This set-up represents a between-subject factorial research design (Shahid, Krahmer, & Swerts, 2014).
How does the condition the child is in, influence his/her expressiveness?
The conditions in this case are: Partner, Culture, Age, Gender, Winning or losing (Shahid, Krahmer, & Swerts, 2014). These are based on the first study and the research question of study 2.
Triangulation is when multiple research methods are used to research one research question (Treadwell, 2016). According to Treadwell (2016), when the different methods conclude the same outcome, the researchers can be more certain of their findings (Treadwell, 2016). In the article of Shahid, Krahmer, & Swerts (2014) it is the case that the different methods result in the same outcome. Even though this is the case, the writers find it important to have done triangulation, because every method reveals other information about the matter (Shahid, Krahmer, & Swerts, 2014).
In the third study there were 3 coders for all the data (Shahid, Krahmer, & Swerts, 2014). There was one researcher with two independent assistants, one from the Netherlands and one from Pakistan. The researcher explained in detail how the coding was supposed to be done. The assistants did the coding independently after they had no questions anymore about the coding scheme. After the first round of coding the coders measured the inter-rater reliability (Shahid, Krahmer, & Swerts, 2014). This measurement is needed to test whether the different coders coded similarly (Treadwell, 2016).
Once the coders identified differences in the coded data, they designed another coding scheme (Shahid, Krahmer, & Swerts, 2014). This way the Observer reliability was ensured, which means that all coders agree about their coding scheme (Treadwell, 2016).
These two aspects increase the reliability of the third study in the article of Shahid, Krahmer, & Swerts (2014). The Inter-rater reliability was measured and afterwards their Observer reliability was ensured (Shahid, Krahmer, & Swerts, 2014).
In the article of Shahid, Krahmer, & Swerts (2014) is stated that a game was chosen as research method. One of the reasons for this was to make the situation as much as possible like the real world (Shahid, Krahmer, & Swerts, 2014). However, this external validity can never completely be reached through experiments. External validity indicated whether the researcher actually measured the real world which is studied (Treadwell, 2016).
The article also does not state anything about the Panel validity of the first study in the article (Shahid, Krahmer, & Swerts, 2014). This means that it is unsure whether the studys validity was not tested by a panel or relevant researchers (Treadwell, 2016). We cannot assume it happened, which is not good for the validity of the first study of the article.
Since an experiment can never completely replicate real life situations and nothing is stated about Panel validity, the validity of the first study of the article is questionable.
It is stated, in the first study, that the parents and teachers gave their consent for the research in the place of their child, since the children were still too young to decide (Shahid, Krahmer, & Swerts, 2014). This consent indicates that the parents were informed beforehand, knew that the recordings of the game would be used for this research and that the ethical standards were upheld (Shahid, Krahmer, & Swerts, 2014).
Before starting the actual experiment the children were explained in their own language what was expected of them and how the game worked (Shahid, Krahmer, & Swerts, 2014). This was to make sure the comprehension of the experiment was as high as possible. Comprehension is one of the three elements that form Informed Consent (Treadwell, 2016).
The fact that the participants of the second and third study re-watch the video recordings of the children playing the game is also indicating that the ethical standards are upheld (Shahid, Krahmer, & Swerts, 2014). The participants, by watching the video footage, also see whether the children are treated well during the experiment. This is a side-effect of the second and third study (Shahid, Krahmer, & Swerts, 2014).
The paragraphs above all show a positive image of the ethical standards in this research: the ethical standards are met.
To draw the conclusion, I find it most important to check whether the ethical standard are met. In the research the participants were informed beforehand and there was official consent from the parents and teachers of the children. Furthermore, the participants of the second and third study also had a control function on how the researchers treated the children in the first study. In my opinion, the ethical standards are met.
In terms of reliability and validity, the article acceptable. The researchers calculated the Inter-rater reliability and they also ensured the Observer reliability. This shows that the reliability is actually good. The validity is less good, because an experiment can never completely capture a real life situation. Another reason why the validity in this article is rather low, is because it is lacking information about Panel validity.