eContent Quality
I met with Johan Thorbiörnson (KTH, Stockholm) today. We discussed the problem of defining eContent Quality.
I think that it not possible to give any clear specification of what is good and what is not good. The field is changing so rapidly. New ideas on how to use the web, and new devices come up with such a phase that even the best eContent today may become obsolete tomorrow.
That eContent is good which is being used and which leads to results. Search engines and tracking tools help us to find out the winners in this competition.
While it seems difficult to define in specific terms what the characteristics of good eContent are, one may approach this by discussing examples.
Johan Thorbiörnson is heading a large scale operation in Sweden. He has about 10 000 (ten thousand!) students who study mathematics and other courses offered by his organization.
We agreed in that the most valuable part of this kind of educational eContent is the assessment system. Students must be able to practice, to get intelligent automatic feed-back, and the instructors must be able to see what their students are doing.
When dealing with such large numbers of students as what JT is doing, it is not possible for the instructors to have much individual interaction with the students. The process needs to be automated.
There are three levels of mastery of a topic:
- Basic understanding of the facts. This can be automatically tested with multiple choice questions or with simple questions where the answer is a string.
- Technical mastery. This can be tested automatically with problems that require sophisticated computations based on the subject matter.
- In depth mastery. Here students must produce a presentation in which the student shows that he or she has mastered both the concepts and the techniques of the learning unit, and that he or she can use them.
To test in depth mastery of a topic is tricky. In a small course, like the courses given at universities or in a high school class, the instructor can test in depth mastery of students individually. If you are teaching thousands of students that is not anymore possible. The solution that JT is using is to split students into groups of four and assign each group a task. The students are then supposed to discuss together and come up with a solution of a complicated problem or write a presentation. The students will, in such a process, teach each other. The role of the instructors is then simply to supervise this activity. In this way we could teach very large numbers of students.
Technorati Tags: Trackback URL for this post:
- Le blog de Mika.Seppala
- Ajouter un commentaire
- 1818 lectures
Quality: a systemic
Quality: a systemic affair
1. Purpose of education
Education, and education in mathematics in particular, is meant to improve human thinking.
But, "before we can make it better, we need to know more of what it is" (Houghton, 2004).
2. Bloom's taxonomy
It is a well-developed paradigm for classifying educational objectives, in the cognitive, affective and psychomotor domains.
“It is used by curriculum planners, administrators, researchers, and classroom teachers at all levels of education" (Anderson & Sosniak, 1994).
It is a de facto standard, and even more after its revised version (Anderson & Kathwohl, 2001), with the title "A Taxonomy for Learning, Teaching, and assessing".
Here we will focus on the cognitive domain, where the classification is based on six (hierarchical) levels of complexity:
This poses the problem (cf. Don Allen's contribution) of how high can eLearning aspire to impact in the thinking-learning hierarchy.
3. Who defines quality?
Quality is/will be defined by quality agencies, which operate at European, state, and regional levels in an organized network. We may hope to influence the definition of protocols for assessing quality of eLearning. Quality assessments, however, are systemic by its very nature: they look at the parts, how they fit and interact together, and how they serve the purpose of the object/program/organization.
4. Three main layers
There are three main layers to take into account:
The Macro-level provides context and conditions (cf. Anders Sanne) for the Meso-level, which in turn is built from the micro-level by aggregation.
Communication is an intrinsic feature of any "quality brand".
5. Research perspective?
It may be relevant to JEM to adopt a research perspective focused on contributing to build a theoretically sound quality framework with a due consideration of empirical evidence.
In such an endeavour it is required to take into account the customer/market viewpoints and political/social directives and structures.
My perspective on e-Content
My perspective on e-Content quality is a bit different than that of Johan Thorbiörnson (KTH, Stockholm). My experience is with distant learners, and we are counting our students in tens rather than thousands. The subjects are the same as taught on campus, but adapted for distant learners. The syllabuses and the exams are the same as in the campus courses.
e-Content is not context free
All learning takes place in a context. I therefore state that it is necessary to look at the context in which the e-Content are used. We use a bundle of learning materials and activities designed as a whole:
Of this only the videos and the forums qualify as e-Content. To say something fair about the quality, it is necessary to know something about the context in which the e-Content is designed for and used in. But, it is of course not a good idea to make the e-Content more context dependent than necessary. As Mika states, the “e-Content is good which is being used”, and I may add reused.
An example:
When recording our video lessons, the teachers tried not to refer to numbered sections, theorems or pages in the textbook. This has proven to be a good strategy as textbooks come in new editions all the time.
Metadata and quality
Videos and material with a lot of pictures, drawings or mathematical notation, are not a trivial task for search engines. Supplying the e-Content with metadata will make it more reusable, and thus raise the quality in some sense. Metadata may also be used to say something about the context for which the e-Content is designed.
Measuring quality is not easy
We cannot measure the quality of our video lessons by simply comparing the distant learners with the ordinary students on campus. Even if their written final exam is exactly the same, the two groups of students are not comparable. The distant learners are older, they are part time students, and their primary subject of interest is not mathematics. The distant learners have a very low rate of failure, but the dropout rate before the exam is large.
To find out something about the quality of our e-Content, we simply asked the students what they think. The students describe the streamed video recordings very positively. The following quotation is typical:
EContent Quality The six
EContent Quality
The six phases of mathematical competence from the (US) National Research Council book "Adding it Up" indicate the following levels in hierarchical form.
• Mathematics Proficiency
• Conceptual understanding
• Procedural fluency
• Strategic competence
• Adaptive reasoning
• Productive disposition
Good EContent can possibly address the first three of these. The first is roughly the same as Basic Mathematics and the third is roughly Technical Mastery. The online teaching of conceptual understanding is possible through videos with practice problems and applets. The last three are more difficult for eContent and likewise for classroom teaching.
All of the said let me outline necessary conditions for EContent Quality. First the feature set of the EContent must be complete. There must be text, video lectures and problems, interactive applets, practice testing, and online testing. In addition, there needs to be feedback loops from testing events to the EContent. Students must have immediate feedback for errors and recommended remedies to their errors.
On a higher plane, the materials overall must be engaging and attractive to students (and even faculty). This is essential to keep them coming back and in fact to be engaged to an extent where they may exceed their own expectations. The word “engaging” currently forms a central point of all pedagogies, and is regarded as the first essential step to student learning.
Now to measure whether the quality of the EContent competes with the traditional teaching/learning modes, we begin with an analogy. Consider the EContent and the Traditional as two factories each producing a product – a student completing the same material. To compare these products there must be crossover at the next course level. That is, students taking the EContent must switch to the traditional mode for the next course and have similar success rates as the continuing traditional students. This validates the quality at a meta-level, that is to say independent of how precisely it was delivered.
For example, when we began teaching WebCalC (Calculus I), our online calculus more than ten years ago, students were asked to take the same common midterm exams. Grade averages were competitive with the traditional mode. However, when they took the next course (Calculus II) offered only in the traditional mode, we measured their performance compared with all students. The grade point ratio of the WebCalC students, now fully integrated into a multitude of traditional sections, was slightly higher than for the traditional students. This validated our online course and its quality irrefutably. So the new factory (EContent) produced a product of quality undiminished in comparison with the traditional factory.
To my mind this is the best way to validate EContent; it is certainly the only way to command the respect of our technologically reluctant colleagues and administrators.
Of course, exit satisfaction surveys are a vehicle to determine satisfaction with the EContent. Students tend to be sincere and honest in such surveys, and much can be learned from their views – provided the right questions are asked.
eContent Quality. First, I
eContent Quality.
First, I would like to point out that eContent quality and e-learning quality are not the same kind of thing. A good eContent is an element of the e-learning model or activity and thus it is needed in e-learning quality. Perhaps eContent could still be good in a bad e-learning system. For example, think of a marvellous interactive web material used without the needed feed back in on-line distance calculus.
A good eContent needs to be used in some “suitable” learning model, unless it was created for a very simple self-learning activity (which in fact is a learning model too). Its efficacy depends, not only on their own quality, but also on the learning model chosen and the adaptation to the student characteristics. Thus, the efficiency of the eContent is not only content-dependent. At least we have two other factors: e-learning model and student profile. The teaching staff is part of the learning model in web based e-learning (much more than in face-to-face).
For instance, web-based distance students, in general, do not like debates or collaborative learning activities, they have very little time for their studies and they prefer a model and an eContent which facilitates them to work individually, and as personalised as possible).
As for the quality of mathematical eContent itself, it is necessary to follow common, standardised criteria, such as:
(1) International quality standards and specifications (like the ones in LORI) - Content Quality (veracity, accuracy and an appropriate level of detail) - Learning Goal Alignment (alignment among learning goals, assessments, and learner characteristics) - Feedback and Adaptation (adaptive content or personalized feedback ) - Motivation (stimulate the interest of learners) - Usability (easy navigation and easy interface) - Accessibility (support for learners with disabilities) - Reusability (ability to port between different courses or learning contexts without modification) - Standards compliance (adherence to international standards and specifications)
(2) Use of specific mathematical software but adapted to the goals of learning and not of contents adapted to the software.
(3) Learning activities according to the mathematical background of the student. Also, tedious manual calculations should be replaced by activities of application to real cases, in the case of the students of Technical studies.
(4) Continuous assessment should be part of the material itself, as a learning element too.
Finally, the degree of satisfaction of the different agents: students and teaching staff must be a fundamental part of the continuous process of evaluation of the used eContent and guides for its continuous improvement. For this to be feasible, teamwork of the different specialists needs to exist (professors of and-learning, editors, web-experts in education, usability, accessibility, etc).
The new role of the teacher-author as part of a network of authors of contents working collaboratively is not sufficiently accepted nor has it an effective model of organization. On the other hand, the author’s tools that make use of the educational technologies based on standards of e-learning are still not sufficiently usable. In my opinion, these two are necessary working directions for the program of eContent Quality: a new model for collaborative development of eContent and the usable tools to do it.