(IJCRSEE) International Journal of Cognitive Research in Science, Engineering and Education
Vol. 2, No.1, 2014.
RELIABILITY OF THE DYNAMIC OCCUPATIONAL THERAPY COGNITIVE ASSESSMENT FOR CHILDREN (DOTCA-CH): THAI VERSION OF ORIENTATION, SPATIAL PERCEPTION, AND THINKING OPERATIONS SUBTESTS
Thikamporn Rangson, Master’s degree student, Occupational Therapy, Faculty of Associated Medical Sciences, Chiang Mai University, Thailand, E-mail: fah_fahnoi@hotmail.com
Dr. Supaporn Chinchai, Occupational Therapy Department, Faculty of Associated medical Sciences, Chiang Mai University, Thailand, E-mail: jeab_dr@yahoo.com
Dr. Sarinya Sriphetcharawut, Occupational Therapy Department, Faculty of Associated medical Sciences, Chiang Mai University, Thailand, E-mail: srisarinya@yahoo.com
Dr. Suchitporn Lersilp, Occupational Therapy Department, Faculty of Associated medical Sciences,
Chiang Mai University, Thailand, E-mail: suchitporn.l@cmu.ac.th
Received: April, 28. 2014.
Accepted: May, 22.2014.
Original Article
UDK 159.937.24.072-053.5
Abstract. The Dynamic Occupational Therapy Cognitive Assessment for Children (DOTCA-Ch) is a tool for finding out about cognitive problems in school-aged children. However, the DOTCA-Ch was developed in English for Western children. For this reason, it’s not appropriate for Thai children because of the differences of culture and language. The objectives of this study were aimed at translating the DOTCA-Ch in Orientation, Spatial Perception, and Thinking Operations sub tests to a Thai version on a World Health Organization back-translation process, and to examine its internal consistency, inter-rater reliability and test-retest reliability. Participants consisted of 38 intellectually impaired and learning disabled individuals between the ages of 6–12 years. Results from this study revealed high internal consistency in the Orientation sub test (α=.83) Spatial Perception sub test (α=.82) and Thinking Operations sub test (α=.82); high inter-rater reliability in the Orientation sub test (ICC =.83), Spatial Perception sub test (ICC =.84) and Thinking Operations sub test (ICC =.74); and high test-retest reliability in the Orientation sub test (ICC =.84), Spatial Perception sub test (ICC =.86), and Thinking Operations sub test (ICC =.85). These results indicate that the Thai version of the Orientation, Spatial Perception, and Thinking Operations sub test might be used as an appropriate assessment tool for Thai children, based on psychometric evidence including internal consistency, inter-rater reliability and test-retest reliability. However, additional study of other psychometric properties, including, predictive validity, concurrent reliability, and inter-rater reliability during the mediation process of this assessment tool needs to be carried out.
Keywords: Cognitive, Orientation, Spatial Perception, Thinking Operations, DOTCA-Ch.
1. INTRODUCTION
Cognition is the high level ability of the brain that can help children to learn to be adapted to any situation. This ability enables children to have complex motor planning skills, problem-solving skills, recognition skills, decision-making skills and memory process (Wittayakorn, 2004 & Shettleworth, 2010). Cognition is especially important for younger school-age children aged 6-12 years who are in the "concrete operations" stage of Piaget's cognitive development theory so that they are able to use logical and coherent actions in thinking and solving problems to understand and explain the concepts of things. For this reason, their cognitive function can make possible wider social interactions and development of more academic skills needed to learn to read, write, calculate, and manipulate their hands in daily life activities (Chinchai, 2000 & Encyclopedia of Children`s Health, 2014). Indeed, cognitive function is an important element in their daily lives, both in school and at home (Josman, 2005). However, children who have cognitive dysfunction will have problems with self-control, social participation, study, and participation in activities in their school and community (Katz, 2005). Teachers and school therapists should know about their students’ cognitive dysfunction, so that students with cognitive dysfunction can be rapidly determined and provided with the appropriate intervention (Katz, 2007).
All students who study in Thai government schools use Thai as a first language and English as a second language. However, there is the absence of standardized assessment tools in Thai to evaluate cognitive functions in school-age students. When therapists considered that their pediatric clients need to be evaluated for cognitive function, the therapists will use non-standardized tests to evaluate the client’s performance. Besides using non-standardized tests, they sometimes adapt other standardized tests that were not developed to evaluate the cognitive function of children directly, for example: the Developmental Test of Visual Perception - 2nd edition for visual perception evaluation, and the Beery-Buktenica Developmental Test of Visual-Motor Integration – 6th edition for visual motor integration evaluation. When we explored the standardized assessment of cognitive function, we found that many researchers utilized the Dynamic Occupational Therapy Cognitive Assessment for Children (DOTCA-Ch) (Rodger et al, 2005).
The DOTCA-Ch, a criterion-referenced assessment tool, was developed to directly evaluate cognitive function in children ages 6 – 12 years. It is implemented as a dynamic assessment to enable the identification of children’s learning potential and their thinking strategies (Katz et. al., 2004). Because the DOTCA-Ch serves as a foundation for the cognitive evaluation of children in an efficient and friendly format, and has the unique feature of the dynamic evaluation testing procedure, therapists use this assessment to find out about the cognitive problems of children and use the results to confidently plan intervention strategies tailor-made for the individual child under treatment (Katz, 2004 & Nowack, 2005). But the DOTCA-Ch was developed in western countries and that means that the therapists in Thailand are not familiar with it. In addition, some therapists try to use the DOTCA-Ch on Thai children but find it ineffective due to cultural and linguistic differences.
In this study, we were concerned with developing a Thai version of the DOTCA-Ch. It is under a process of back-translation in the manual, scoring sheet, format and instruction and interpretation of data that are in English and studies about the psychometric properties, including internal consistency, inter-rater reliability and test-retest reliability. We chose to study three sub tests including orientation, spatial perception and thinking operations. These three sub tests were chosen because orientation is an important skill of basic cognition that can be developed to meta-cognition, especially orientation of time. Munkhetwit (2008) reported that many studies showed trends indicating that orientation of time was a common problem for children. Because orientation of time relates to the memory of time and is a dynamic process, which continually happens, there is a need for an input process, a storage process, and a recall process. In addition, problems of memory of time can be an indicator of memory impairment that has an influence on human occupations. The second sub test, spatial perception, is an important skill in activities of daily life. If children have a problem with spatial perception, they will have problems with using tools and with coordinating their bodily movements. The third sub test, thinking operations, is a basic skill for information processing that can help children to learn and understand the meaning of information, and to decide how to respond to that information (Munkhetwit, 2008). All of these sub tests are necessary to assess a child’s development of learning strategies and cognitive skills. The results of this research will be useful for therapists in Thailand in that it provides them with a standardized assessment methodology to accurately evaluate the cognitive abilities of Thai children.
The objectives of this research were as follows:
2.MATERIALS AND METHODS
The DOTCA-Ch was designed to provide a measurement of cognitive intervention need for children who are referred for treatment as a result of possible developmental, cognitive, or academic and learning difficulties, and for children with brain injuries as well as mild intellectually impaired children (Katz et. al., 2004). For this reason, in this study the criteria of participants in purposive sampling were the students who were evaluated by medical and educational screening tests as mild learning problem students. Also, these participants did not have any physical, visual, or hearing disabilities. Participants in this study were 38 children who were 6-12 years old, 18 male students (47.36%) and 20 female students (52.64%), who were selected by purposive selection. The test group was made up of representatives from 17 learning disability students and 21 mild intellectual disability students. The average age of the participants was 8 years and 7 months.
The methodology of this study consisted of three phases as is presented in Figure 1. The first phase was a process of translation and back translation of the assessment tool “DOTCA-Ch”, a dynamic criterion-referenced assessment of cognitive abilities and learning potential for children with cognitive and learning difficulties. In the first phase, there were five steps which referred to WHO's back translation process (WHO, 2014) as follows:
Step 1: The researcher asked for permission from the Maddak Corporation in USA, who is owned of DOTCA-Ch, for the translation assessment tool DOTCA-Ch.
Step 2: The target of the language translation procedure was to be able to translate the assessment tool from English to Thai. The characteristics of the translators were expertise in English and Thai, experience with the DOTCA-Ch assessment method, knowledge of occupational therapy, and expertise in developmental theory and performance of cognitive domains Orientation, Spatial Perception and Thinking Operation.
Step 3: The Thai translation of DOTCA-Ch was back-translated into English by two experts from the Faculty of Western Language and the Faculty of Humanity of Chiang Mai University. This English version was then checked and proofed by the selected experts. The characteristics of the experts were the fact that they were fluent in English and Thai, had not worked in occupational therapy, and had not used the DOTCA-Ch assessment tool.
Step 4: The content of Thai version of the DOTCA-Ch was cross-checked with the English version and corrected and hereby amended as needed.
Step 5: The researcher trained two research assistants in Thai DOTCA-Ch Orientation, Spatial Perception and Thinking Operation sub tests before conducting a preliminary trial with three children to achieve testing proficiency.
The second phase of the study was pilot use of the method. This phase was the trial use of a Thai version of the DOTCA-Ch by the researcher and two research assistants to test the students with cognition and learning disability. During the trial use, the researcher and research assistants took notes about the problems of using this assessment tool. In the subsequent phase, the researcher started by contacting the Wat-Gukham School, Sanpatong District, Chiang Mai Province to get written permission for its use as a research area. After that, the researcher and research assistants administered the Thai DOTCA-Ch to the children with the cognitive disability who was studying in the Wat-Gukham School. The duration of the assessment was approximately 45 minutes. However, during the test sessions, if the children complained, were tired or had a lack of concentration, they were able to bring to an end to rest for about 2 or 3 minutes. In addition, research assistants took notes and commented when they saw any mistakes or problems and wrote their suggestions after they had tried to use the DOTCA-Ch with seven students. Then, the researcher rectified and improved the content of the Thai DOTCA-Ch assessment tool according to the recorded comments and suggestions.
Finally, the third phase was the process of checking on the accuracy of the assessment tool by determining the internal consistency, accuracy between examiners and stability in repeating the test. In this phase, the researcher started by contacting the Wat Chang Kian School and Kawila Anugul School to get written permission for their use as experimental areas. Then, the researcher used the modified Thai DOTCA-Ch to appraise 38 children, 17 learning disability students from the Wat Chang Kian School and 21 mild intellectual impairment students from the Kawila Anugul School. When the assessment was completed, the researcher determined the accuracy of inter-rater reliability between one of research assistants and the researcher who knew about the DOTCA-Ch and had previously used the Thai DOTCA-Ch version of the Orientation, Spatial Perception and Thinking Operation sub tests. Next, the researcher analyzed the data to assess internal consistency by the Conbach’s alpha coefficient and the intraclass correlation coefficient. Additionally, the researcher used the Thai DOTCA-Ch to appraise the same group of children to determine the test-retest reliability. The period between the test and the retest was 2 weeks.
Figure 1. The process in the methodology of this study
Note: Adapted from the WHO back translation process (WHO, 2014)
3. RESULTS AND DISCUSSION
In this study, the results indicated that the Thai DOTCA-Ch version of the Orientation, Spatial Perception and Thinking Operation sub tests had an overall highly significant internal consistency coefficient of .82 (Table 1). Each sub test also had high internal consistency coefficients of between .81 - .84. The overall inter-rater reliability was .80. In each sub test, the inter-rater reliability was between .72 - .84. This result implied that the assessment tool had a good to excellent level of reliability. The overall test-retest reliability was.85. In each sub-test, the inter-rater reliability was between .82 - .87, which meant that the Thai DOTCA-Ch version had excellent reliability.
Table 1. The internal consistency of the Thai DOTCA-Ch version of the Orientation, Spatial Perception and Thinking Operation sub tests by the Cronbach’s alpha coefficient, Inter-rater Reliability and the Test-retest Reliability
Note: OP = Orientation for Place, OT = Orientation for Time, SP1 = Directions on Child's Body, SP2 = Spatial Relations between Children. and Objects in Near Space, SP3 = Spatial Relations on a Picture, CA = Categorization, RU = ROC Unstructured, RS = ROC Structured, PS1 = Pictorial Sequence A, PS2 = Pictorial Sequence B, GS1 = Geometrical Sequence A, GS2 = Geometrical Sequence B
All three sub-tests had intra-class correlation coefficients between .72 - .86 and had an overall inter-class correlation coefficient of .80. According to the criteria of Cicchetti & Sparrow (1981), the Thai DOTCA-Ch version of the Orientation, Spatial Perception and Thinking Operation sub tests had excellent reliability. For the Spatial Perception sub test, the inter-rater reliability had the highest intra-class correlation coefficient which was similar to the DOTCA-Ch original version (ICC = .95) (Katz et. al., 2004). The value of inter-rater reliability of the Spatial Perception sub test was higher than the Orientation and Thinking Operation sub tests. These results might have been the result of a number of factors, such as that the two examiners might not have truly understood this assessment tool and had not gained adequate experience in using it in the assessment trials. Additionally, the methods of evaluation and scoring of the children came from them answering “yes” or “no” questions. This result may be related to the study of Ziviani et al. (2004) which revealed that a dichotomous scale assessment would have had a higher reliability than a multiple-scale assessment or an ordinal scale assessment. For example, if the child answered correctly, he would get 1point. On the other hand, if incorrect, he would get 0 points. This testing approach was unproblematic for the examiners in scoring so that it may have led to a lower reliability. The Thinking Operation sub test had a lower reliability than the other sub test . This was similar to the findings of Katz et al. (2004) who studied the original version of the DOTCA-Ch and showed that the inter-rater reliability was .87; this value indicated a lower reliability than Orientation and Spatial Perception sub tests. The reason for this result might be the different characteristics of the two examiners; one might have been stricter or more flexible in the observation of children’s behaviors than the other. Since the Thinking Operation sub test information is derived from observing the student’s performance and from interviewing the students, it is possible that the examiners could have evaluated student performance in the interview differently. These factors may have led to the lower reliability of the Thinking Operation sub test. However, all three sub tests had reliabilities ranging from good to excellent levels that were similar to the original version of the DOTCA-Ch. For this reason, the Thai DOTCA-Ch version of the Orientation, Spatial Perception and Thinking Operation sub test could be considered to have a high inter-rater reliability (Cicchetti and Sparrow, 1981).
From the analysis of the test-retest reliability, the results showed that the intra-class correlation co-efficient was between .82 - .87, and the overall intra-class correlation co-efficient was .87. This meant that the assessment tool had excellent reliability, according to the criteria of the reliability co-efficient of Cicchetti & Sparrow (1981).In addition, there was an interval of 2 weeks between the test to the retest evaluation so that most likely, this duration did not have an impact on the cognitive development of the children. According to the theory of Piaget, cognitive development can be categorized into four levels, including a sensorimotor stage, preoperational stage, concrete operational stage and the formal operational stage. The basic levels will develop the higher levels, and this process takes more than 2 years to complete. As a consequence, the period of 2 weeks did not cause any discernable differences in development. The lower test-retest reliability of the orientation sub test might have been caused by the characteristics of the questions for which the children needed to provide answers by explanation their reasons. For instance, in the question of place orientation, “Where do you live? Explain in detail about where do you live.”, the child might answer differently each time it is requested. The scoring of these questions had three levels: 0 point, 1 point, and 2 points. Zero points meant the child did not know the answer, 1 point meant the child answer correctly by choosing from multiple choices, or the examiner explained the question in more detail, and 2 points meant the child answered correctly by themselves without being given choices, or without any explanation from the examiner. As a result, the answers from the retest might have been from memory of the first answer. In fact, the child might not truly have got the question and answered with the examiner’s explanations or choices which gave 1 point. However, when they were given a retest, they might have gotten 2 points because they remembered the choices or the explanations from the first test. Thus, although they performed the identical, in each type of question they got different points for each. This may have affected the test-retest reliability of the Orientation sub test slightly less than the Spatial Perception and Thinking Operation sub test. However, all these sub tests had high test-retest reliabilities. These results were similar to those obtained by Ziviani et al. (2004), who found that the test-retest reliability of the original version of the DOTCA-Ch was high to very high. On the other hand Daibes (2012) found that regardless of the fact that 53% of the assessment items of the original version of the DOTCA-Ch had a high test-retest reliability, 19% of the assessment items had a low test-retest reliability.
All of the above results indicated that the Thai DOTCA-Ch versions of Orientation, Spatial Perception and Thinking Operation sub tests had the psychometric properties of consistency to be good assessment tools. Indeed, although the assessments were repeated, the results were no different. These results were not only focused on the end product of the Thai DOTCA-Ch versions but also gave guidelines for researchers in other countries in which English is a second language to develop the standardized test in their own language by the WHO back translation process.
A limitation of this research was that all tests of reliability came from the score points before using mediations. In future, the researcher may study the norms in Thai children and other tests of the validity of this assessment tool, such as, predictive validity, construct validity and concurrent validity. Moreover, interestingly, future research can study the psychometric properties of the DOTCA-Ch after using mediation.
CONCLUSIONS
This research showed the process of cognitive assessment tool development in the context of a country where English is a second language. This systematic process developed a new assessment methodology which had excellent reliability. Suitable cognitive assessment was enabled that overcame any language barrier so that the cognitive problems of children could be accurately identified.
ACKNOWLEDGEMENTS
This project was funded by The Graduate School, Chiang Mai University, Thailand.
Conflict of interests
Authors declare no conflict of interest.
REFERENCES
Chinchai, S. (2000). Basic principle of occupational therapy. Chiang Mai: Occupational therapy department, Chiang Mai University.
Cicchetti, D. V., & Sparrow, S. S. (1981). Developing criteria for establishing interrater reliability of specific items: Applications to assessment of adaptive behavior. American Journal of Mental Deficiency, 86, 127-137.
Donaghy, S., & Wass, P. J. (1998). Inter-rater reliability of functional assessment in a brain injury programme. Archive of Physical Medicine and Rehabilitation, 9, 1231-1236.
Daibes J.S.A. (2012). The Dynamic Occupational Therapy Cognitive Assessment for children DOTCA-Ch: Pilot study of inter-rater and Test-retest reliability (Dissertion). Palestine: Nables, AnNajah National University.
Encyclopedia of Children's Health (2014). Cognitive development. On-line. Available from Internet, http://www.healthofchildren.com/C/Cognitive-Development.html, accessed 16 May 2014.
Hinkle, D. E., Wiersma, W., & Jurs, S. G. (2003). Applied Statistic for the Behavioral Sciences. 5th ed. Boston: Boston: Houghton Mifflin.
Josman, N. (2005). The dynamic interactional model in schizophrenia. Cognition and occupation across the life span: Models for Intervention in Occupational Therapy. 2nd ed. Bethesda: American Occupational Therapy Association.
Katz, N. (2005). Cognition & Occupation across the Life Span. 2nd ed. Bethesda: The American occupational Therapy. : Association Inc.
Katz, N., Golstand, S., Traub Bar-Ilan R, , & Parush,S. (2007). The dynamic Occupational Therapy Cognitive Assessment for Children (DOTCA-Ch): A new instrument for assessing learning potential. American Journal of Occupational Therapy, 61, 41-52.
Katz, N., Kizony, R. & Parush, S. (2002). Visuomotor organisation and thinking operations performance of school-age Ethiopian, Bedouin, and mainstream Israeli children. The Occupational Therapy Journal of Research, 22,34–43.
Katz, N., Parush, S., & Traub Bar-Ilan R, (2004). DOTCA-Ch - Dynamic Occupational Therapy Cognitive Assessment for children manual. Pequannock NJ. : Maddak Inc.
Lefrancois, G. R. (2006). Theories of Human Learning. Belmont, CA: Thomson Higher Education.
Munkhetwit, P. (2008). Occupational therapy interventions for persons with cognitive-perceptual dysfunction. Chiang Mai. : Darawan Publishing.
Nowack, R. A. (2005). Performance Comparison of Boy Versus Girls between the Ages of 9-12 on the DOTCA-Ch: A pilot Study. NY: Thesis Touro College School of Health Sciences Occupational Therapy Program.
Rodger S., Daley E., Hughes K. & Ziviani J. (2005). Dynamic Occupational Therapy Cognitive Assessment for Children: Perceived utility in Australian occupational therapy practice. Australian Occupational Therapy Journal, 52, 337– 34.
Shettleworth, S. J. (2010). Cognition, Evolution and Behavior. 2nd edition. New York: Oxford University Press.
Wittayakorn, S. (2004). Occupational therapy in children: clinical evaluation. Chiang Mai: Occupational therapy department, Chiang Mai University.
World Health Organization. (2014). Process of translation and adaptation of instruments. Online. Available from Internet, http://www.who.int/substance_abuse/research_tools/translation/en/index.html, accessed 15 January 2014.
Ziviani, J., Rodger, S., Pacheco, P., Rotsey, L., Smith, A., & Katz, N. (2004). The Dynamic Occupational Therapy Cognitive Assessment for Children (DOTCA Ch): Pilot Study of Inter-rater and Test Retest Reliability. New Zeland Journal of Occupational Therapy, 51(2).