Wouldn’t it be great to have a large data set of annotated reflective writings written in various languages? We might get there but let’s start with a bit of brainstorming.
At first sight, the task to create a multiple languages reflective writing corpus looks like a relatively straightforward task. For example, we could use a set of English reflective writings and let them translate into several languages. The problem here is that we would not learn anything about the essence of reflection, for example, whether there is anything language/culture-specific to reflection or not. Another idea which is quick to execute is that everyone interested can just contact friends (or friends of friends) from Uni and ask them whether they want to provide a reflective writing that they have written during their study time in exchange for a small payment for research purposes. The problem here is that we would not know whether the outcomes of the analysis of these reflective writings is telling us more about our sample choice or about reflection.
These two examples already show that the task is tricky. There is much to consider and many decisions to be made.
We all can go out and just collect data, but with some planning and collaboration we might get better data. I believe that it is therefore important to bring together many potential stakeholders and to brainstorm what we need to consider to create a corpus of comparative reflective writings in multiple languages. What I mean with multiple languages is that the aim is to create a corpus of two or more languages consisting of reflective writings. Each language set of reflective writings is built according to the same standards. Overall the aim is to have data set of different languages that are similar or comparable.
This brings me to the brainstorming task. Imagine that we all received a good amount of time and money to study reflective thinking across the world in order to better understand the variety of reflective thinking expressed in writings.
“What do we need to consider to create and annotate a publicly available data set of comparable HE students’ reflective writings of multiple languages?”
Think about this question. Then jot down a list of your considerations (keep them short and add an explanation to each). Once you have created your list, sort the items according to your priority. But don’t stop there. Send them to me. I am more than happy to discuss your points with you so please send them to me. And remember, brainstorming is about creativity, so there are not right or wrong answers.
Such data set would be of great value for various purposes. One of them is that we may discover that we all reflect in the same way, but also if we discover that there are differences then this should help us to learn from each other. Personally, I would be very interested to know whether feelings play more of a role in some languages than in others and also the importance of expressing a personal perspective.
I set out this idea of a data set of reflective writings in many languages in my thesis a while ago. It was one of my limitations as in this work I only focused on English student writings. My hope was and is ‘that this research can inspire further research to evaluate the potential of automated detection of reflection across languages’ . In my 2019 paper, I proposed that the specific method that I used in this paper can be of guidance to set standards regarding the annotation of the data set (method and theory). The paper presented several ideas from of using a ‘standardized evaluation method, its proposal of reflective writing categories that are common to many models, its focus on model validity, and its reliability’ .
I summarised some of the problems that I had when creating my large data set of reflective writings in this Google group post a while ago here: https://groups.google.com/d/msg/wred-general/9Fpbp2sQ0K8/m8y-GbioBgAJ. This was in response to the blog post of Ming Liu and Simon Buckingham Shum of the UTS team calling for contributions to a reflective writing data set for machine learning. See here: http://wa.utscic.edu.au/2018/09/14/building-a-reflective-writing-corpus-for-analytics-research/
I updated my list of problems or considerations when creating a reflective data set taking especially my work in the 2019 paper into account . They are:
- What is the use case of the data set? My research was about the analysis of texts. Manual content analysis is the number one method to analyse reflective writing and from the analysis of the manual content analysis of reflective writings, I developed the coding scheme for my data set. Although analysis and assessment are close they do differ and thus a data set for the assessment of reflective writings may be different from an analysis data set.
- This also raises questions regarding what the central constituents of a reflective writing are. This relates very much to the theory of reflection.
- What unit of analysis is useful?
- What is the right size?
- What standards should be followed?
- What languages would be included?
- Are there important subject differences that need to be considered?
- Are there any demographic variables to consider?
- What research would be needed to carry out first in order to create a sound data set?
- Once there is a data set, how can others use it?
This list gives a good summary of points to consider when creating a large data set suitable for machine learning. Now, in addition to this, what do we need to consider when creating a comparable data set of reflective writings in multiple languages?
References
{70228:7AXQ9KEA};{70228:PA2WGP2T};{70228:PA2WGP2T}
apa
default
asc
no
595
%7B%22status%22%3A%22success%22%2C%22updateneeded%22%3Afalse%2C%22instance%22%3A%22zotpress-4b51d0c8791d6d67d72cb9caf1dbf04a%22%2C%22meta%22%3A%7B%22request_last%22%3A0%2C%22request_next%22%3A0%2C%22used_cache%22%3Atrue%7D%2C%22data%22%3A%5B%7B%22key%22%3A%22PA2WGP2T%22%2C%22library%22%3A%7B%22id%22%3A70228%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Ullmann%22%2C%22parsedDate%22%3A%222019-05%22%2C%22numChildren%22%3A2%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%202em%3B%20text-indent%3A-2em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3EUllmann%2C%20T.%20D.%20%282019%29.%20Automated%20Analysis%20of%20Reflection%20in%20Writing%3A%20Validating%20Machine%20Learning%20Approaches.%20%3Ci%3EInternational%20Journal%20of%20Artificial%20Intelligence%20in%20Education%3C%5C%2Fi%3E%2C%20%3Ci%3E29%3C%5C%2Fi%3E%282%29%2C%20217%26%23×2013%3B257.%20%3Ca%20href%3D%27https%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1007%5C%2Fs40593-019-00174-2%27%3Ehttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1007%5C%2Fs40593-019-00174-2%3C%5C%2Fa%3E%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Automated%20Analysis%20of%20Reflection%20in%20Writing%3A%20Validating%20Machine%20Learning%20Approaches%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Thomas%20Daniel%22%2C%22lastName%22%3A%22Ullmann%22%7D%5D%2C%22abstractNote%22%3A%22Reflective%20writing%20is%20an%20important%20educational%20practice%20to%20train%20reflective%20thinking.%20Currently%2C%20researchers%20must%20manually%20analyze%20these%20writings%2C%20limiting%20practice%20and%20research%20because%20the%20analysis%20is%20time%20and%20resource%20consuming.%20This%20study%20evaluates%20whether%20machine%20learning%20can%20be%20used%20to%20automate%20this%20manual%20analysis.%20The%20study%20investigates%20eight%20categories%20that%20are%20often%20used%20in%20models%20to%20assess%20reflective%20writing%2C%20and%20the%20evaluation%20is%20based%20on%2076%20student%20essays%20%285%2C080%20sentences%29%20that%20are%20largely%20from%20third-%20and%20second-year%20health%2C%20business%2C%20and%20engineering%20students.%20To%20test%20the%20automated%20analysis%20of%20reflection%20in%20writings%2C%20machine%20learning%20models%20were%20built%20based%20on%20a%20random%20sample%20of%2080%25%20of%20the%20sentences.%20These%20models%20were%20then%20tested%20on%20the%20remaining%2020%25%20of%20the%20sentences.%20Overall%2C%20the%20standardized%20evaluation%20shows%20that%20five%20out%20of%20eight%20categories%20can%20be%20detected%20automatically%20with%20substantial%20or%20almost%20perfect%20reliability%2C%20while%20the%20other%20three%20categories%20can%20be%20detected%20with%20moderate%20reliability%20%28Cohen%27s%20%5Cu03ba%20ranges%20between%20.53%20and%20.85%29.%20The%20accuracies%20of%20the%20automated%20analysis%20were%20on%20average%2010%25%20lower%20than%20the%20accuracies%20of%20the%20manual%20analysis.%20These%20findings%20enable%20reflection%20analytics%20that%20is%20immediate%20and%20scalable.%22%2C%22date%22%3A%22May%202019%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%2210.1007%5C%2Fs40593-019-00174-2%22%2C%22ISSN%22%3A%221560-4292%2C%201560-4306%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Flink.springer.com%5C%2Farticle%5C%2F10.1007%5C%2Fs40593-019-00174-2%22%2C%22collections%22%3A%5B%2262P4KVI8%22%2C%22HF5DUT2B%22%5D%2C%22dateModified%22%3A%222019-06-06T18%3A12%3A23Z%22%7D%7D%2C%7B%22key%22%3A%227AXQ9KEA%22%2C%22library%22%3A%7B%22id%22%3A70228%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Ullmann%22%2C%22parsedDate%22%3A%222015%22%2C%22numChildren%22%3A2%7D%2C%22bib%22%3A%22%3Cdiv%20class%3D%5C%22csl-bib-body%5C%22%20style%3D%5C%22line-height%3A%202%3B%20padding-left%3A%202em%3B%20text-indent%3A-2em%3B%5C%22%3E%5Cn%20%20%3Cdiv%20class%3D%5C%22csl-entry%5C%22%3EUllmann%2C%20T.%20D.%20%282015%29.%20%3Ci%3EAutomated%20detection%20of%20reflection%20in%20texts.%20A%20machine%20learning%20based%20approach%3C%5C%2Fi%3E%20%28PhD%20Thesis%29.%20The%20Open%20University.%20Retrieved%20from%20%3Ca%20href%3D%27http%3A%5C%2F%5C%2Foro.open.ac.uk%5C%2F45402%5C%2F%27%3Ehttp%3A%5C%2F%5C%2Foro.open.ac.uk%5C%2F45402%5C%2F%3C%5C%2Fa%3E%3C%5C%2Fdiv%3E%5Cn%3C%5C%2Fdiv%3E%22%2C%22data%22%3A%7B%22itemType%22%3A%22thesis%22%2C%22title%22%3A%22Automated%20detection%20of%20reflection%20in%20texts.%20A%20machine%20learning%20based%20approach%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Thomas%20Daniel%22%2C%22lastName%22%3A%22Ullmann%22%7D%5D%2C%22abstractNote%22%3A%22Promoting%20reflective%20thinking%20is%20an%20important%20educational%20goal.%20A%20common%20educational%20practice%20is%20to%20provide%20opportunities%20for%20learners%20to%20express%20their%20reflective%20thoughts%20in%20writing.%20The%20analysis%20of%20such%20text%20with%20regard%20to%20reflection%20is%20mainly%20a%20manual%20task%20that%20employs%20the%20principles%20of%20content%20analysis.%20%5Cn%5CnConsidering%20the%20amount%20of%20text%20produced%20by%20online%20learning%20systems%2C%20tools%20that%20automatically%20analyse%20text%20with%20regard%20to%20reflection%20would%20greatly%20benefit%20research%20and%20practice.%20%5Cn%5CnPrevious%20research%20has%20explored%20the%20potential%20of%20dictionary-based%20approaches%20that%20automatically%20map%20keywords%20to%20categories%20associated%20with%20reflection.%20Other%20automated%20methods%20use%20manually%20constructed%20rules%20to%20gauge%20insight%20from%20text.%20Machine%20learning%20has%20shown%20potential%20for%20classifying%20text%20with%20regard%20to%20reflection-related%20constructs.%20However%2C%20not%20much%20is%20known%20of%20whether%20machine%20learning%20can%20be%20used%20to%20reliably%20analyse%20text%20with%20regard%20to%20the%20categories%20of%20reflective%20writing%20models.%20%20%5Cn%5CnThis%20thesis%20investigates%20the%20reliability%20of%20machine%20learning%20algorithms%20to%20detect%20reflective%20thinking%20in%20text.%20In%20particular%2C%20it%20studies%20whether%20text%20segments%20from%20student%20writings%20can%20be%20analysed%20automatically%20to%20detect%20the%20presence%20%28or%20absence%29%20of%20reflective%20writing%20model%20categories.%20%5Cn%5CnA%20synthesis%20of%20the%20models%20of%20reflective%20writing%20is%20performed%20to%20determine%20the%20categories%20frequently%20used%20to%20analyse%20reflective%20writing.%20For%20each%20of%20these%20categories%2C%20several%20machine%20learning%20algorithms%20are%20evaluated%20with%20regard%20to%20their%20ability%20to%20reliably%20detect%20reflective%20writing%20categories.%20%5Cn%5CnThe%20evaluation%20finds%20that%20many%20of%20the%20categories%20can%20be%20predicted%20reliably.%20The%20automated%20method%2C%20however%2C%20%20does%20not%20achieve%20the%20same%20level%20of%20reliability%20as%20humans%20do.%22%2C%22thesisType%22%3A%22PhD%20Thesis%22%2C%22university%22%3A%22The%20Open%20University%22%2C%22date%22%3A%222015%22%2C%22language%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Foro.open.ac.uk%5C%2F45402%5C%2F%22%2C%22collections%22%3A%5B%22NAGIUMXW%22%2C%225XSZTTP2%22%5D%2C%22dateModified%22%3A%222019-02-06T12%3A19%3A23Z%22%7D%7D%5D%7D
Ullmann, T. D. (2019). Automated Analysis of Reflection in Writing: Validating Machine Learning Approaches.
International Journal of Artificial Intelligence in Education,
29(2), 217–257.
https://doi.org/10.1007/s40593-019-00174-2
Ullmann, T. D. (2015).
Automated detection of reflection in texts. A machine learning based approach (PhD Thesis). The Open University. Retrieved from
http://oro.open.ac.uk/45402/