The Effects of Automated Writing Evaluation (AWE)+Human Feedback on the Quality of Argumentative Writing by Chinese EFL Learners
DOI:
https://doi.org/10.17507/jltr.1603.20Keywords:
AWE+human feedback, writing quality, holistic scores, cohesion, coherenceAbstract
Nowadays, Chinese EFL learners are increasingly using automated writing evaluation (AWE) to provide feedback on their writing. However, AWE is relatively inadequate for providing elaborate feedback on the global aspects of writing that require human judgment. Thus, how to combine AWE with human feedback is a significant issue worth exploring. Regarding the effects of AWE+human feedback, cohesion and coherence are rarely studied. Thus, this study aimed to address that gap. This research employed a quasi-experimental study with a quantitative method to address AWE+human feedback. It also aimed to compare the effects of traditional feedback with AWE+human feedback modes, that is, teacher-only, AWE+teacher (AT), and AWE+peer+teacher (APT), on EFL learners’ writing quality in terms of holistic score, cohesion, and coherence. A total of 90 EFL learners from three intact classes of English major were randomly assigned to the control and experimental groups. The control group received only teacher feedback. The experimental groups received AT feedback and APT feedback in their writing process, respectively. Two instruments, iWrite and Coh-Metrix, were used to collect the data. Results showed that all feedback types improved holistic scores, coherence, and cohesion, with the APT model producing the most significant improvements across these dimensions. The APT group demonstrated particularly high holistic scores. It enhanced sentence-level coherence and lexical cohesion, suggesting that integrating AWE, peer, and teacher feedback provides a comprehensive and effective approach to developing writing proficiency.
References
Alharbi, W. (2023). AI in the foreign language classroom: A pedagogical overview of automated writing assistance tools. Education Research International, 2023(1), 4253331. https://doi.org/10.1155/2023/4253331
Barrot, J. S. (2023). Using automated written corrective feedback in the writing classrooms: Effects on L2 writing accuracy. Computer Assisted Language Learning, 36(4), 584-607. https://doi.org/10.1080/09588221.2021.1936071
Burstein, J., Chodorow, M., & Leacock, C. (2003, August). CriterionSM Online Essay Evaluation: An Application for Automated Evaluation of Student Essays. IAAI, 3-10. Retrieved October 31, 2024, from Google Scholar.
Chen, G., & Gong, M.Y. (2022). Duo yuan fan kui mo shi cu jin shen du xue xi de xing dong yan jiu [An Action Research on Multiple Feedback Models for Deeper Learning]. Wai yu jiao xue, 43(3), 60-66.
Chen, H. H. J., Chiu, S. T. L., & Liao, P. (2009). Analyzing the grammar feedback of two automated writing evaluation systems: My Access and Criterion. Ying yu jiao xue qi kan, 33(2), 1–43. https://doi.org/10.6330/ETL.2009.33.2.01
Chen, M., & Cui, Y. (2022). The effects of AWE and peer feedback on cohesion and coherence in continuation writing. Journal of Second Language Writing, 57, 100915. https://doi.org/10.1016/j.jslw.2022.100915
Cheng, X., Zhang, L. J., & Yan, Q. (2021). Exploring teacher written feedback in EFL writing classrooms: Beliefs and practices in interaction. Language Teaching Research. https://doi.org/10.1177/13621688211057665
Cotos, E., Link, S., & Huffman, S. (2017). Effects of DDL technology on genre learning. Language Learning & Technology, 21(3),104 – 130. https://orcid.org/0000-0002-5586-1495
Crossley, S. A., Kyle, K., & McNamara, D. S. (2016). The development and use of cohesive devices in L2 writing and their relations to judgments of essay quality. Journal of Second Language Writing, 32, 1-16. https://doi.org/10.1016/j.jslw.2016.01.003
Dikli, S. (2006). An overview of automated scoring of essays. The Journal of Technology, Learning and Assessment, 5(1). Retrieved October30, 2024, from http://www.jtla.org
Ferris, D. R., Pezone, S., Tade, C. R., & Tinti, S. (1997). Teacher commentary on student writing: Descriptions & implications. Journal of second language writing, 6(2), 155-182. https://doi.org/10.1016/S1060-3743(97)90032-1
Fu, Q. K., Zou, D., Xie, H., & Cheng, G. (2024). A review of AWE feedback: Types, learning outcomes, and implications. Computer Assisted Language Learning, 37(1-2), 179-221. https://doi.org/10.1080/09588221.2022.2033787.
Guo, K., Pan, M., Li, Y., & Lai, C. (2024). Effects of an AI-supported approach to peer feedback on university EFL students' feedback quality and writing ability. The Internet and Higher Education, 63, 100962. https://doi.org/10.1016/j.iheduc.2024.100962.
Han, T., & Sari, E. (2024). An investigation on the use of automated feedback in Turkish EFL students’ writing classes. Computer Assisted Language Learning, 37(4), 961-985. https://doi.org/10.1080/09588221.2022.2067179
Hassanzadeh, M., & Fotoohnejad, S. (2021). Implementing an automated feedback program for a Foreign Language writing course: A learner‐centric study: Implementing an AWE tool in a L2 class. Journal of Computer Assisted Learning, 37(5), 1494-1507. https://orcid.org/0000-0003-1510-1149
Huang, J., & Zhang, W.X. (2014). Duo yuan fan kui dui da xue sheng ying yu zuo wen xiu gai de ying xiang yan jiu [The Impact of the Integrated Feedback on Students’ Writing Revision]. Zhong guo wai yu, 11(1), 51-56.
Hyland, K., & Hyland, F. (2006). Feedback on second language students' writing. Language teaching, 39(2), 83-101. https://doi.org/10.1017/S0261444806003399
Keh, C. L. (1990). Feedback in the Writing Process: A Model and Methods for Implementation. ELT Journal, 4(4), 294-304. https://doi.org/10.1093/elt/44.4.294
Lee, I. (2002). Teaching coherence to ESL students: A classroom inquiry. Journal of second language writing, 11(2), 135-159. https://doi.org/10.1016/S1060-3743 (02)00065-6
Liao, H. C. (2016). Enhancing the grammatical accuracy of EFL writing by using an AWE-assisted process approach. System, 62, 77-92. https://doi.org/10.1016/j.system.2016.02.007
Liao, L.Y. (2022). Automated Evaluation or Human Evaluation? Optimal Means of Evaluation in EFL Writing Assessment. [Unpublished master’s thesis]. University of Macau.
Link, S., Mehrzad, M., & Rahimi, M. (2022). Impact of automated writing evaluation on teacher feedback, student revision, and writing improvement. Computer Assisted Language Learning, 35(4), 605-634. https://doi.org/10.1080/09588221.2020.1743323
Liu, M., & Shao, C. (2024). The Effects of Integrated Feedback based on AWE on English Writing of Chinese EFL Learners. Scalable Computing: Practice and Experience, 25(2), 987-996. https://doi.org/10.12694/scpe.v25i2.2617
Liu, M., & Braine, G. (2005). Cohesive features in argumentative writing produced by Chinese undergraduates. System, 33(4), 623-636. https://doi.org/10.1016/j.system.2005.02.002
Liu, S. L. (2022). A Research on the Blended Evaluation Mode in College English Writing Course. Journal of Language Teaching and Research, 13(4), 763-771. https://doi.org/10.17507/jltr.1304.09
McNamara, D. S., & Kintsch, W. (1996). Learning from texts: Effects of prior knowledge and text coherence. Discourse processes, 22(3), 247-288. https://doi.org/10.1080/01638539609544975
Plakans, L., and Gebril, A. (2017). Exploring the relationship of organization and connection with scores in integrated writing assessment. Assessing Writing, 31, 98-112. https://doi.org/10.1016/j.asw.2016.08.005.
Ramaprasad, A. (1983). On the definition of feedback. Behavioral science, 28(1), 4-13. https://doi.org/10.1002/bs.3830280103
Ranalli, J. (2018). Automated written corrective feedback: How well can students make use of it? Computer Assisted Language Learning, 31(7), 653-674. https://doi.org/10.1080/09588221.2018.1428994
Ranalli, J. (2021). L2 student engagement with automated feedback on writing: Potential for learning and issues of trust. Journal of Second Language Writing, 52, 100816. https://doi.org/10.1016/j.jslw.2021.100816
Reynolds, D. W. (2001). Language in the balance: Lexical repetition as a function of topic, cultural background, and writing development. Language Learning, 51(3), 437-476. https://doi.org/10.1111/0023-8333.00161
Shang, H. F. (2022). Exploring online peer feedback and automated corrective feedback on EFL writing performance. Interactive Learning Environments, 30(1), 4-16. https://doi.org/10.1080/10494820.2019.1629601
Stevenson, M., & Phakiti, A. (2014). The effects of computer-generated feedback on the quality of writing. Assessing Writing, 19, 51-65. https://doi.org/10.1016/j.asw.2013.11.007
Tan, S., Cho, Y. W., & Xu, W. (2023). Exploring the effects of automated written corrective feedback, computer-mediated peer feedback and their combination mode on EFL learner’s writing performance. Interactive Learning Environments, 31(10), 7276-7286. https://doi.org/10.1080/10494820.2022.2066137
Todd, R. W., Khongput, S., & Darasawang, P. (2007). Coherence, cohesion and comments on students’ academic essays. Assessing writing, 12(1), 10-25. https://doi.org/10.1016/j.asw.2007.02.002
Wang, Y., Luo, X., Liu, C. C., Tu, Y. F., & Wang, N. (2022). An integrated automatic writing evaluation and SVVR approach to improve students’ EFL writing performance. Sustainability, 14(18), 11586. https://doi.org/10.3390/su141811586.
Warschauer, M., & Grimes, D. (2008). Automated writing assessment in the classroom. Pedagogies: An International Journal, 3(1), 22-36. https://doi.org/10.1080/15544800701771580
Weigle, S. C. (2007). Teaching writing teachers about assessment. Journal of Second Language Writing, 16(3), 194-209. https://doi.org/10.1016/j.jslw.2007.07.004
Wilson, J., & Roscoe, R. D. (2020). Automated writing evaluation and feedback: Multiple metrics of efficacy. Journal of Educational Computing Research, 58(1), 87-125. https:// doi.org/10.1177/0735633119830764
Witte, S.P, & Faigley, L. (1981). Coherence, cohesion, and writing quality. College Composition & Communication, 32(2), 189-204. https://doi.org/10.58680/ccc198115912
Xu, H. Y. (2020). Ji yu iWrie2.0 xie zuo ping yue xi tong de liang zhong fan kui fang shi dui da xue sheng ying yu xie zuo ying xiang de shi zheng yan jiu [An Empirical Study on the Influence of Two Feedback Modes on College English Writing ——Based on iWrite2.0 Writing Evaluation System], [Unpublished master’s thesis], Jiangxi shi fan da xue.
Xu, T. S., Zhang, L. J., & Gaffney, J. S. (2022). Examining the relative effects of task complexity and cognitive demands on students’ writing in a second language. Studies in Second Language Acquisition, 44(2), 483-506. https://doi.org/10.1017/S0272263121000310
Yu, S. (2021). Feedback-giving practice for L2 writing teachers: Friend or foe? Journal of Second Language Writing, 52, 100798. https://doi.org/10.1016/j.jslw.2021.100798
Zhai, N., & Ma, X. (2022). Automated writing evaluation (AWE) feedback: A systematic investigation of college students’ acceptance. Computer Assisted Language Learning, 35(9), 2817-2842. https://doi.org/ 10.1080/09588221.2021.1897019
