Skip To Content

Technology Enhanced Knowledge Research Institute (TEKRI)

TEKRI blogs

Study links student cheating to whether a course is popular or disliked

Jon Dron's blog - October 8, 2017 - 11:16

We already know that extrinsically motivated students (mainly those driven by grades and testing) are far more likely to cheat than those who are more intrinsically motivated. I bookmarked yet another example of this effect just the other day but there are hundreds if not thousands of research papers that confirm this in many different ways. And, as this article reaffirms, we already know that mastery learning approaches (that focus on supporting control, appropriate levels of challenge, and, ideally, social engagement) tend to make cheating far less likely, because they tend to better support intrinsic motivation. Hardly anyone cheats if they are doing stuff they love to do, unless some strong extrinsic force overrides it (like grades, rewards, punishments, hard-to-meet deadlines, etc). 

This research reveals another interesting facet of the problem that exactly accords with what self-determination theory would predict: that, whether or not the pedagogy is sensible (supportive of intrinsic motivation) or dumb (extrinsically driven), a student’s dislike of a course appears to predict an increased likelihood of cheating. This is pretty obvious when you think about it. If someone does not like a course then, by definition, they are not intrinsically motivated and, if they are still taking it despite that, the only motivation they can possibly have left is extrinsic.

The increased chances of cheating on disliked courses, whether or not mastery learning techniques are used, is completely unsurprising because it ain’t what you do, it’s the way that you do it. If mastery learning techniques are not working then it probably means that we are simply not using them very well. Most likely there is not enough support, or not enough learner control, or insufficient social engagement, or not enough/too much challenge, or there’s too much pressure, or something along those lines. It is actually much more difficult and usually far more time consuming to teach well using techniques that respect learner autonomy and individual needs than it is to follow the objectivist instructivist path, at least in an institutional environment that deeply embeds extrinsic motivation at its very core, so it is not surprising that it quite often fails.  It is also very possible that the problem is almost entirely due to the surrounding educational ecosystem. For instance if it is one that forces students down institutionally-determined paths whether or not they are ready, whether or not it matters to them, or if not enough time is allowed for it, or if the stakes for failure are high, then even well-designed courses with enthusiastic, supportive, skilled, well-informed, compassionate, unpressured teachers are not likely to help that much.

Some people will take a pragmatic lesson from this to look more carefully for cheating on courses that they know to be disliked. That’s not the solution. Others will look at those courses and try to find ways to make them more likeable. That’s much better. But really, once we have done that, we need to be wondering about why anyone would be taking a course that they dislike in the first place. And that points to a central problem with our educational systems and the tightly coupled teaching and accreditation that they embed deep in their bones. Given enough time, support, and skilled tuition, almost anyone can learn almost anything, and love doing so. We live in a time of plenty, where there are usually countless resources, people, and methods to learn almost anything, in almost any practical way, so it makes no sense that people should still be forced to learn in ways that they dislike, at inappropriate times, and at an inappropriate pace. If they do, it is because (one way or another) we make them do so, and that’s the root of the problem. We – the educators and, above all, the educational system – are the cause of cheating, as much as we are the victims of it. And we are the ones that should fix it.

The original paywalled paper can be found here.

Address of the bookmark: https://www.insidehighered.com/news/2017/10/06/study-links-student-cheating-whether-course-popular-or-disliked

Originally posted at: https://landing.athabascau.ca/bookmarks/view/2762299/study-links-student-cheating-to-whether-a-course-is-popular-or-disliked

This was actually accepted for an IEEE conference and then published

Jon Dron's blog - October 6, 2017 - 21:06

I invite you to draw your own conclusions about this paywalled paper and the amount of quality control and editorial input that goes into IEEE publications nowadays. Here’s the abstract, which is one of the more coherent passages in the paper:

Abstract—The momentum contemplate evaluates the relationship among online social recreations and the e-learning utilization by look at the impact of social, subjective and teaching nearness on e-learning use between female understudies by method for playing on the web social diversions. This study utilizes an exploratory research plan, comfort test procedure. The outcomes propose that all scales are basically related with E- learning use. It is found that E-learning uses is emphatically tremendous and has a direct related with social nearness. The relationship between E-learning use and psychological nearness has a decidedly strong enormous connection; in like manner, the relationship between E-learning use and teaching nearness has an emphatically strong colossal connection. The disclosures inferred that the characteristic of online social amusements; both intellectual and teaching nearness impact E-learning utilization.

There’s not enough research about female understudies. I’m glad that someone is filling that gap. It’s well worth what otherwise appear to be the subscription fees IEEE is charging (US$33 in case you were wondering) . 

Address of the bookmark: http://ieeexplore.ieee.org/document/8052647/

Originally posted at: https://landing.athabascau.ca/bookmarks/view/2760723/this-was-actually-accepted-for-an-ieee-conference-and-then-published

The NGDLE: We Are the Architects | EDUCAUSE

Jon Dron's blog - October 3, 2017 - 21:43

A nice overview of where the NGDLE concept was earlier this year. We really need to be thinking about this at AU because the LMS alone will not take us where we need to be. One of the nice things about this article is that it talks quite clearly about the current and future roles of existing LMSs, placing them quite neatly within the general ecosystem implied by the NGDLE.

The article calls me out on my prediction that the acronym would not catch on though, in my defence, I think it would have been way more popular with a better acronym! The diagram is particularly useful as a means to understand the general concept at, if not a glance, then at least pretty quickly…

Address of the bookmark: https://er.educause.edu/articles/2017/7/the-ngdle-we-are-the-architects

Originally posted at: https://landing.athabascau.ca/bookmarks/view/2752680/the-ngdle-we-are-the-architects-educause

The return of the weblog – Ethical Tech

Jon Dron's blog - September 28, 2017 - 09:58

Blogs have evolved a bit over the past 20 years or so, and diversified. The always terrific Ben Werdmuller here makes the distinction between thinkpieces (what I tend to think of as vaguely equivalent to keynote presentations at a conference, less than a journal article, but carefully composed and intended as a ‘publication’) and weblogging (kind of what I am doing here when I bookmark interesting things I have been reading, or simply a diary of thoughts and observations). Among the surprisingly large number of good points that he makes in such a short post is that a weblog is best seen as a single evolving entity, not as a bunch of individual posts:

Blogging is distinct from journalism or formal writing: you jot down your thoughts and hit “publish”. And then you move on. There isn’t an editorial process, and mistakes are an accepted part of the game. It’s raw.

A consequence of this frequent, short posting is that the product isn’t a single post: it’s the weblog itself. Your website becomes a single stream of consciousness, where one post can build on another. The body of knowledge that develops is a reflection of your identity; a database of thoughts that you’ve put out into the world.

This is in contrast to a series of thinkpieces, which are individual articles that live by themselves. With a thinkpiece, you’re writing an editorial; with a blog, you’re writing the book of you, and how you think.

This is a good distinction. I also think that, especially in the posts of popular bloggers like Ben, the blog is also comprised of the comments, trackbacks, and pings that develop around it, as well as tweets, pins, curations, and connections made in other social media. Ideas evolve in the web of commentary and become part of the thing itself. The post is a catalyst and attractor, but it is only part of the whole, at least when it is popular enough to attract commentary.

This distributed and cooperative literary style can also be seen in other forms of interactive publication and dialogue – a Slashdot or Reddit thread, for instance, can sometimes be an incredibly rich source of knowledge, as can dialogue around a thinkpiece, or (less commonly) the comments section of online newspaper articles. What makes the latter less commonly edifying is that their social form tends to be that of the untarnished set, perhaps with a little human editorial work to weed out the more evil or stupid comments: basically, what matters is the topic, not the person. Untarnished sets are a magnet for trolls, and their impersonal nature that obscures the individual can lead to flaming, stupidity, and extremes of ill-informed opinion that crowd out the good stuff. Sites like Slashdot, StackExchange, and Reddit are also mostly set-based, but they use the crowd and an algorithm (a collective) to modulate the results, usually far more effectively than human editors, as well as to provide shape and structure to dialogues, so that dialogues become useful and informative. At least, they do when they work: none are close to perfect (though Slashdot, when used well, is closer than the rest because its algorithms and processes are far more evolved and far more complex, and individuals have far more control over the modulation) but the results can often be amazingly rich.

Blogs, though, tend to develop the social form of a network, with the blogger(s) at the centre. It’s a more intimate dialogue, more personal, yet also more public as they are almost always out in the open web, demanding no rituals of joining in order to participate, no membership, no commitment other than to the person writing the blog. Unlike dedicated social networks there is no exclusion, no pressure to engage, no ulterior motives of platforms trying to drive engagement, less trite phatic dialogue, more purpose, far greater ownership and control. There are plenty of exceptions that prove the rule and plenty of ways this egalitarian structure can be subverted (I have to clean out a lot of spam from my own blogs, for instance) but, as a tendency, it makes blogs still very relevant and valuable, and may go some way to explaining why around a quarter of all websites now run on WordPress, the archetypal blogging platform.

Address of the bookmark: https://words.werd.io/the-return-of-the-weblog-f6b702a7cf99

Originally posted at: https://landing.athabascau.ca/bookmarks/view/2740999/the-return-of-the-weblog-%E2%80%93-ethical-tech

Instagram uses 'I will rape you' post as Facebook ad in latest algorithm mishap

Jon Dron's blog - September 26, 2017 - 14:44

Another in a long line of algorithm fails from the Facebook stable, this time from Instagram…

This is a postcard from our future when AI and robots rule the planet. Intelligence without wisdom is a very dangerous thing. See my recent post on Amazon’s unnerving bomb-construction recommendations for some thoughts on this kind of problem, and how it relates to attempts by some researchers and developers to use learning analytics beyond its proper boundaries.

 

Address of the bookmark: https://www.theguardian.com/technology/2017/sep/21/instagram-death-threat-facebook-olivia-solon

Original page

Infants make more attempts to achieve a goal when they see adults persist

Jon Dron's blog - September 24, 2017 - 18:02

A straightforward and briefly reported study that supports the rather obvious hypothesis that quite young (15-month-old) children can and do learn from observing adults, at least in the short term. The twist here is that adults in the study were deliberately trying to model an attitude (grit) more than a distinct behaviour, in an attempt to teach the kids to do the same.

It is fair to say that the researchers demonstrated to the kids that persevering with problems after initially failure can lead to desirable results, and that the kids appeared to be more inclined to do the same after watching adults doing so: this accords well with the title of the paper. I’m not sure that the adults adequately demonstrated grit, though. I don’t know about you, but I actually enjoy solving problems and positively relish the failures that teach me how to succeed. In fact, in many situations (programming, for example) I deliberately make things fail in order to understand how they do so, and that’s part of the fun, even though (and partly because) I may curse and fume when the process fails to enlighten me. Same for many commercially available puzzles, from Rubik’s Cubes to letter-sliding games. Seems to me that grit involves more than doing something enjoyable on the way to achieving some anticipated goal that matters to us. It’s often about doing unenjoyable things, sometimes for goals we don’t even find particularly interesting or worthwhile, often over a prolonged period. That’s not what was happening here. This is interesting, though, if only to confirm that really quite young kids are able to see others as beings like themselves, and to transfer the lessons of stories that they construct about what they perceive others to be doing into actions they then take themselves.

The brief timeframe of the study means that it doesn’t show whether this is how grit is actually learned over time.  The extent to which lessons persist depends on a great many things, including prior experience, repetition, who is repeating it, success in the short term, effectiveness of the attitude in overcoming meaningful challenges in the long term, social value of the attitude, current context, and counter-examples over time. Outside an experimental context we pick up attitudes and sentiments from kids as much as they do from us, from one another, and from the world at large. There are usually very many others around us who are all engaged in a rich reciprocal dance with us through which we collectively construct our various intersecting cultures and subcultures, including our attitudes and values. Also, life is seldom so neatly structured and categorized that a lesson can be so directly transferred from one context to another. At least, such cases are not the interesting ones. Though the experimenters tried to make the tasks a bit different, the study was really set up to highlight the similarities, and to lead to results that would please the children.  In real life, we usually need to connect one situation with another that is quite different, separated by time, and to choose between competing strategies to deal with it, often with others around us that are adopting different approaches, all of which will influence us. Often, we are not even particularly interested in the outcomes. It’s much harder to do experiments that reflect that reality. In fact, it’s probably impossible, at least without adopting the ethical precepts of Josef Mengele. The researchers laudably note a range of other limitations, including cultural differences, beliefs of children about adults, task specific issues, and so on, and make no extravagant claims that it can be generalized further. Indeed it cannot.

That said, this is good evidence for something that I believe is not a bad idea: that teachers (formal or otherwise) should act as they hope their students will act. A very large part of the role of a teacher is to model how people in their field (or society at large, in the case of younger kids) think and behave, to enact and demonstrate their approaches and attitudes,  perhaps more than to pass on the facts, skills, and technologies of their discipline, or to provide support for gaining such knowledge.

Bearing that in mind, while there is value in ‘grit’ and I don’t want to knock it too much, I think there are other attitudes that might matter a whole lot more, especially those that enable us to not just stick with stuff we don’t enjoy but to find pleasure and meaning in it. Passion is way more useful than grit, in the long run. Caring, too. Teachers that light fires in students’ hearts achieve way more than those that simply show them how to stick at things they hate.

Abstract

Persistence, above and beyond IQ, is associated with long-term academic outcomes. To look at the effect of adult models on infants’ persistence, we conducted an experiment in which 15-month-olds were assigned to one of three conditions: an Effort condition in which they saw an adult try repeatedly, using various methods, to achieve each of two different goals; a No Effort condition in which the adult achieved the goals effortlessly; or a Baseline condition. Infants were then given a difficult, novel task. Across an initial study and two preregistered experiments (N = 262), infants in the Effort condition made more attempts to achieve the goal than did infants in the other conditions. Pedagogical cues modulated the effect. The results suggest that adult models causally affect infants’ persistence and that infants can generalize the value of persistence to novel tasks.

Address of the bookmark: http://science.sciencemag.org/content/357/6357/1290

Original page

No, you aren't a 'visual' learner

Jon Dron's blog - September 21, 2017 - 17:16

It’s a damning indictment of our collective resistance to truth that the point of this article still has to be restated, yet again. Amazingly, 93% of the general public and 76% of educators still erroneously believe that we should be taught in ways that match our learning styles. I assume this is so in the US – unless things have changed recently, the percentages, for teachers at least, are even worse in some other countries where the idea has been pushed harder from the top down, such as the UK and Netherlands. To be quite clear: this belief is not supported by any compelling evidence at all.

The fact that it is false (or, at best, no more provable than, and just as likely as, astrology) doesn’t mean that designing for learning styles necessarily a terrible idea, inasmuch as it can encourage reflective practice on the part of teachers and can even result in quite useful outcomes. As the article puts it:

“If you’re trying to vary what you do in the classroom to respect different styles, variation in instruction is probably a good thing, anyway,” he says. But rather than formatting lessons differently for auditory, kinetic or visual learners, he and Macdonald suggest that teachers tweak their instruction based on content.

“I think it really depends on your objectives for the lesson,” Macdonald says. “Some types of content really lend themselves to visual presentation … if you’re teaching maps, that’s got to be visual. If you’re teaching music, those are [the] types of things that need to be auditory.

“But if your goal is to get a multifaceted exposure to certain content, it can be helpful to weave in all different types of modalities.”

That thinking about learning styles can be a useful design tool is a fair point, and one that I have often made myself (including in quite some detail in my first book), though it’s a happy side effect of a mistake, rather than a consequence of a good theory. Using star signs would probably work just as well.  I am not convinced that content should always lead design either: objectives-driven teaching is not the only fruit and, for some expansive subject areas and pedagogies, it is positively (positivistly?) harmful. But, notwithstanding its constraints and limitations, at least it is not based on a fiction.

There are many risks to using a false world model, even if it has some practical value or plausible results (pre-Copernican geocentric astronomy was better than Copernicus’s own theory at predicting movements of planets), not least of which being that it blinds us to real possibilities and leads us in worthless, wasteful, or even harmful directions. Even when the consequences include better teaching, it’s a terrible lesson to teach someone that they are a visual (or sensing, or whatever nonsense the particular theory suggests) learner. No they are not. They might have some habits, reinforced patterns, or preferences, sure. But that just means they need to try a bit harder to extend themselves and to learn to use some alternative approaches because they are definitely going to have to use them at some point when there’s no teacher in control of things but themselves, and nothing to fit their preferred style available. My learning style is and should be whatever the hell I need.

I’ve mentioned before that I believe a better (if less attractive) term would be ‘being-taught habits’ because one of the least savoury aspects of the whole learning styles gestalt is that it actually has little to do with learning, and everything to do with achieving better indoctrination; of asserting the power of the teacher (at least, it would if it worked). For that kind of thing, we’d learn more from the sciences and arts of the advertising industry than from any snake oil learning style theory. We might equally learn from preachers and religions: they are mostly pretty good at making people think and behave the way they wish.

There are other ways to gain the useful side-effects of designing for learning styles that do not rely on falsehoods, or that make no claims that they match reality one way or the other – de Bono’s Thinking Hats, for instance, or design-based research. And it doesn’t take much to make learning style theories less dumb. I am personally quite fond of Gordon Pask’s serialist/holist model, despite coming perilously close to a learning styles theory at times, because it describes a continuum of learning strategies, without suggesting too much (OK, fair enough, Pask slipped here and there) that such strategies be fixed, habitual, or generally preferred by particular learners.  They are simply perspectives we can choose as and when it is helpful to do so. However, if possible, when designing learning activities, we should use approaches that are based as much as we are able on how the world is, not how we think it should be. From that perspective, learning styles are a potentially dangerous and time-consuming dead end.

Address of the bookmark: http://theweek.com/articles/725352/no-arent-visual-learner

Original page

Amazon helps and teaches bomb makers

Jon Dron's blog - September 19, 2017 - 15:30

Amazon’s recommender algorithm works pretty well: if people start to gather together ingredients needed for making a thermite bomb, Amazon helpfully suggests other items that may be needed to make it, including hardware like ball bearings, switches, and battery cables. What a great teacher!

It is disturbing that this seems to imply that there are enough people ordering such things for the algorithm to recognize a pattern. However, it would seem remarkably dumb for a determined terrorist to leave such a (figuratively and literally) blazing trail behind them, so it is just as likely to be the result of a very slightly milder form of idiot, perhaps a few Trump voters playing in their backyards. It’s a bit worrying, though, that the ‘wisdom’ of the crowd might suggest uses of and improvements to some stupid kids’ already dangerous backyard experiments that could make them way more risky, and potentially deadly.

Building intelligent systems is not too hard, as long as the activity demanding intelligence can be isolated and kept within a limited context or problem domain. Computers can beat any human at Go, Chess, or Checkers. They can drive cars more safely and more efficiently than people (as long as there are not too many surprises or ethical dilemmas to overcome, and as long as no one tries deliberately to fool them). In conversation, as long as the human conversant keeps within a pre-specified realm of expertise, they can pass the Turing Test. They are even remarkably much better than humans at identifying, from a picture, whether someone is gay or not. But it is really hard to make them wise. This latest fracas is essentially a species of the same problem as that reported last week of Facebook offering adverts targeted at haters of Jews. It’s crowd-based intelligence, without the wisdom to discern the meaning and value of what the crowd (along with the algorithm) chooses. Crowds (more accurately, collectives) are never wise: they can be smart, they can be intelligent, they can be ignorant, they can be foolish, they can even (with a really smart algorithm to assist) be (or at least do) good; but they cannot be wise. Nor can AIs that use them.

Human wisdom is a result of growing up as a human being, with human needs, desires, and interests, in a human society, with all the complexity, purpose, meaning, and value that it entails. An AI that can even come close to that is at best decades away, and may never be possible, at least not at scale, because computers are not people: they will always be treated differently, and have different needs (there’s an interesting question to explore as to whether they can evolve a different kind of machine-oriented wisdom, but let’s not go there – SkyNet beckons!). We do need to be working on artificial wisdom, to complement artificial intelligence, but we are not even close yet. Right now, we need to be involving people in such things to a much greater extent: we need to build systems that informate, that enhance our capabilities as human beings, rather than that automate and diminish them. It might not be a bad idea, for instance, for Amazon’s algorithms to learn to report things like this to real human beings (though there are big risks of error, reinforcement of bias, and some fuzzy boundaries of acceptability that it is way too easy to cross) but it would definitely be a terrible idea for Amazon to preemptively automate prevention of such recommendations.

There are lessons here for those working in the field of learning analytics, especially those that are trying to take the results in order to automate the learning process, like Knewton and its kin. Learning, and that subset of learning that is addressed in the field of education in particular, is about living in a human society, integrating complex ideas, skills, values, and practices in a world full of other people, all of them unique and important. It’s not about learning to do, it’s about learning to be. Some parts of teaching can be automated, for sure, just as shopping for bomb parts can be automated. But those are not the parts that do the most good, and they should be part of a rich, social education, not of a closed, value-free system.

Address of the bookmark: http://www.alphr.com/politics/1007077/amazon-reviewing-algorithms-that-promoted-bomb-materials

Original page

 

Update: it turns out that the algorithm was basing its recommendations on things used by science teachers and people that like to make homemade fireworks, so this is nothing like as sinister as it at first seemed. Nonetheless, the point still stands. Collective stupidity is just as probable as collective intelligence, possibly more so, and wisdom can never be expected from an algorithm, no matter how sophisticated.

Analytic thinking undermines religious belief while intelligence undermines social conservatism, study suggests

Jon Dron's blog - September 18, 2017 - 11:37

‘Suggests’ is the operative word in the title here. The title is a sensationalist interpretation of an inconclusive and careful study, and I don’t think this is what the authors of the study mean to say at all. Indeed, they express caution in numerous ways, noting small effect sizes, lack of proof of causality, large overlaps between groups, and many other reasons for extremely critical interpretation of the evidence:

“We would like to warn readers to resist the temptation to draw conclusions that suit their ideological worldviews,” Saribay told PsyPost. “One must not think in terms of profiles or categories of people and also not draw simple causal conclusions as our data do not speak to causality. Instead, it’s better to focus on how certain ideological tendencies may serve psychological needs, such as the need to simplify the world and conserve cognitive energy.”

This is suitably cautious and very much at odds with the title of the PsyPost article.

The study itself finds some confirmatory evidence that, in the US (and only in the US):

  •     Religion may be embedded more in Type 1 intuitions relative to politics.
  •     Processing liberal political arguments may require cognitive ability.
  •     Religious belief should be predicted uniquely by analytic cognitive style.
  •     Conservatism should be uniquely predicted by cognitive ability.

It is important to note, however, that ‘prediction’ in this instance has a very precise meaning of implying slightly increased odds of correlation between these factors, not that there is a causal connection one way or the other. The study simply adds a little more evidence to an already fairly substantial body of proof that cognitively challenged people, especially those more inclined to intuition than to reason (the two are statistically correlated), are somewhat more likely to be drawn both to religion and to right wing politics. Much as I would like it to imply the inverse – that intelligence and rationality are a cure for religion and right wing beliefs – there is absolutely nothing in this research to suggest that.

Part of the motivation for the study is the researchers’ observation of the growing antagonism to intelligence, expertise, evidence, and truth that is revealed in Trump’s victory, Brexit, ISIL, man-made climate change denial, and so on. While such evils are no doubt fuelled and sustained by (not to put too fine a point on it) stupid people in search of simple solutions to complex problems, it would be foolish (stupid, even) and highly inaccurate to suggest that all (or even a majority) of those exhibiting such attitudes and beliefs are stupid, or driven by intuition rather than reason, or both. As the study’s authors rightly observe, the value of this study is its contribution to understanding some of the complexity of the problem and should not be used to extrapolate exactly the same kind of simplified caricatures that cause it in the first place:

“…a more balanced understanding can only be reached via continued empirical research. Human beings may sometimes benefit from cognitive simplification of a complex and at times scary world of constant change and uncertainty. It does seem that certain aspects of religion and conservative ideology serve to deal with this, in slightly different ways. This is the direction that evidence points to thus far. However, researchers of course must resist this very need to simplify the world beyond a certain level.”

The original study can be found at http://www.sciencedirect.com/science/article/pii/S019188691730226X

Address of the bookmark: http://www.psypost.org/2017/09/analytic-thinking-undermines-religious-belief-intelligence-undermines-social-conservatism-study-suggests-49655

Original page

Highly praised children are more inclined to cheat

Jon Dron's blog - September 15, 2017 - 13:26

The title of this Alphr article is a little misleading because the point the article rightly makes is that it all depends on the type of praise given. It reports on research from the University of Toronto that confirms (yet again) what should be obvious: praising learners for who they are (‘you’re so smart’) is a really bad idea, while praising what they do (‘you did that well’) is not normally a bad idea. The issue, though, is essentially one of intrinsic vs extrinsic motivation. By praising the person for being a particular way you are positioning that as the purpose, rather than a side-effect, of the activity, and positioning yourself as the arbiter, so disempowering the learner. By praising the behaviour, you are offering useful feedback on performance that empowers the recipient to choose whether and how to do such things again, as well as supporting needs for relatedness (it shows you care) and competence (it helps them improve). Both forms of praise contribute to feelings of self-esteem, but only one supports intrinsic motivation. 

The nice twist in these particular studies (here and here) is that the researchers were looking at effects on morality. They found that ability praise (teling them they are smart) is very strongly correlated with a propensity to cheat. Exactly as theory would predict, kids who have been told that they are smart are significantly more likely to respond to the extrinsic motivation (the need to live up to expectations when given ability praise) by cheating, when given the opportunity. Interestingly, praising the behaviour (performance praise) has little or no effect on likelihood of cheating when compared with those given no praise at all: it is only when an expectation is set that the children are perceived as smart that cheating behaviour increases. It is also interesting, if tangential, that boys appeared to be way more likely to cheat than girls under all the conditions though, once primed by ability praise, girls were more likely to cheat than boys that had received no praise or performance praise.

The lesson is nothing like as simple as remembering to just praise the action, not the person. Praising behaviours can, when used badly, be just as disempowering as praising the person. For instance, while in some senses it might be possible to view grades as a kind of abbreviated praise (or punishment, which amounts to much the same thing) for a behaviour, there’s a critical difference: the fact that it will be graded is known in advance by the learner. This is compounded by the fact that the grade matters to them, often more than the performance of the activity itself. Thus, achieving the grade becomes the goal, not the consequence of the behaviour, and it reinforces the power of the grader to determine the behaviour of the learner, with a consequent loss of learner autonomy. That shift from intrinsic to extrinsic motivation is the big issue here, not the praise itself. There are lots of ways to give both performance praise and ability praise that are not coercive. They are only harmful when used to manipulate behaviour.

Address of the bookmark: http://www.alphr.com/science/1007043/highly-praised-children-are-more-inclined-to-cheat

Original page

Learning as Artifact Creation

elearnspace (George Siemens) - September 14, 2017 - 12:04

Digitization is deceptive in that the deep impact isn’t readily observable. Remember when MOOCs were going to transform higher education? Or when personalized learning was going to do away with instructors? Going back about a century ago, audio, then video, was also going to disrupt education. All of these trends have been window dressing – a facade more reflective of the interests of those who advocate for them rather than a substantive departure from established norms.

Yet, change is happening, often under the radar of enthusiasts because it’s harder to sell a technology product or draw clicks to a website when being nuanced and contextual. Education is an idea/information-based process. How information is accessed, created, and shared is revealing about the future of learning. Essentially, follow information in order to understand the future of higher education. Today, information is networked and digital. University transformation and proposed innovation should align to this reality to have a broad impact – notably on student learning and the development of knowledge in society.

In 2004, I tried to respond to the network/digitization alpha trend by describing the new actions and experiences that were available to learners: Connectivism: A Learning Theory for the Digital Age. This article and work with Stephen Downes formed subsequent development of MOOCs and learning analytics.

Connectivism was presented as a theory that described how learning happened in networks, complex ambiguous information spaces, digital environments, and the increased opportunities of the participative web. Unfortunately, much of that theory remains undeveloped. Details regarding cognitive processes, teacher actions, learner mindsets, design models, and social interaction remain rudimentary. I’m confident that these will be developed over time, but progress has been slow. As a result, connectivism is something people cite rather than engage and develop into a more complex theory or framework of learning. But, whether connectivism or by some other name, a social networked model of learning is our future.

Enter the artifact…

One aspect of connectivism that has great potential for development is the role of the artifact in learning. With CCK08, we found fascinating activities arising due to student created artifacts. One student creates an image to detail the architecture of the course. Another updates it and adds to it. Another comes by and critiques it. The artifact serves as a social learning object. This process reflects my earlier point: big trends unfold behind the scenes over time and in education, they map and mirror to what people do with information that is digital and networked.

Here’s an example: Education over the past several centuries has been defined by the centrality of the instructor and the actions of a learner in relationship to what the instructor knows. There have always been voices that challenged this model – Dewey, Illich, Freire, Montessori – but the system of learning that defines our society is modeled on the assumption of learners needing to duplicate what instructors already know. Learning artifacts – a paper or a test – were largely held between the instructor and student. Group work and class presentations brought others into the relationship, but the message was still clear: the instructor and the content were central, all else was held in their orbit.

Then the internet happened. And later the web. Small groups of people could share without a mediator. You didn’t need a publisher to blast your thoughts to a bulletin board. Yahoo groups, Friendster, and other early social software didn’t fully live up to the vision of the mesh, but they enabled communication. Content creation was still largely the domain of experts or people in positions of control. Britannica and newspapers were still gatekeepers. Then, the late 1990′s rolled around and Blogger made self-publishing reasonably accessible to anyone.

We could now create artifacts, not only talk about them.

A stunning period web innovation occurred between 2000-2005: delicious, myspace, many blog platforms, flickr, wikis, etc. The gates were opened and everyone was a content creator and everything was subject to user creation. Everything was a possible social artifact. Take and share a picture. Post your thoughts on a blog. Tag and share valuable resources. The web had its velveteen rabbit moment and became real to people who had previously been unable to easy share their creative artifacts. Eventually we were blessed with the ugly stepchildren of this movement (Twitter, Facebook) that enabled flow of creative artifacts but in themselves where not primarily generative technologies.

Educationally, this provided new opportunities for students. That class lecture that didn’t make sense? There is a better resource online. That stats textbook that is confusing? There is a MOOC for that. Don’t like a class? Tell the web. Don’t like your instructor? Tell the web (rate my prof). Have an important thought to share? Upload a video to youtube. An awesome song? Upload. Share. A terrific painting you’ve been working on? Upload. Share.

Consider the impact of these opportunities on education and how poorly the higher education system has responded. Consider our curriculum as a self-contained coherent resource. The goal of education? Teach this container to the students. What happens when you add artifact creation? The entire curriculum can shift. If I lecture on the development of open learning and open source technologies, I’m presenting my voice, my priorities, my values. If someone comes along and says “what about the power structure and the bias that underpins this content”? Bam. It’s a new course. Someone creates a video reacting to a lecture I delivered? Bam. It’s a new course. This doesn’t always happen on grand scales. Often the artifact has a limited impact – a brief detour in a new conversation and learning direction for students. The aggregate of these artifacts is significant because it places students in a new mindset, one defined by personal autonomy and agency.

All of this is obvious. It’s mainly about the permissions that technology enables: namely, to write ourselves and our values into any curriculum and learning interaction. The impact that this has on the learning experience is not well understood. We have theories of community (community of inquiry, community of practice). We have many theories of content and content interaction (including transactional distance). There is something about the artifact that is unique in its ability to make every learner a teacher, every contribution a redirection of learning, every interaction a reaction and augmentation.

In one of our LINK grants we currently exploring the power of artifacts as redirective entities (an NSF grant titled COCOA). How does creating a blog post, video, or meme contribute to enlarging the curriculum? How do artifacts contribute to, and take away from, the course content? What is a well-designed artifact? What causes resonance between learner resources being shared and students that respond to those resources?

E-Learn 2017, Vancouver, 17-20 October – last day of cheaper registration rates

Jon Dron's blog - September 13, 2017 - 09:11

Today is the final day to get the discount rate if you are planning on coming to E-Learn in Vancouver this year (US$455 today vs US$495 from tomorrow onwards).

It promises to be quite a big event this year, with an estimated 900+ concurrent sessions, 100+ posters, and three lunchtime SIGs (including a new one on sustainable learning technologies), not to mention some fine keynotes and networking events.  Annoyingly, it clashes with ICDE in Toronto this year but, IMHO, E-Learn is a better conference for those working and researching in online education, and it’s a much better location. I may be a little biased, being both a resident of Vancouver and local co-chair of the conference, but there are some very good reasons I chose to be both those things!

I have attended almost all E-Learn (and its predecessor, WebNet) conferences for nearly 20 years now because it tends to attract some great people, provides an excellently diverse and blended mix of technical and pedagogical perspectives, gives plentiful chances to engage with both early-career researchers and those at the top of the field, usually picks great locations, is well-organized, and focuses solely on adult online learning (mainly higher education but also some from industry, healthcare, government, etc). The acceptance rate (1-in-3 to 1-in-4) is high enough to attract diverse papers that can be off the wall and interesting (especially from younger researchers who don’t know what’s impossible yet so sometimes achieve it), but low enough to exclude utter rubbish. If that kind of thing interests you, this is the conference for you!

I hope to see you there.

Address of the bookmark: https://www.aace.org/conf/elearn/registration/

Original page

Learning Analytics Courses

elearnspace (George Siemens) - September 11, 2017 - 10:48

After about a year of planning, we can finally announce the following courses on edX focusing on learning analytics. The intent of these courses is to eventually lead into a MicroMasters and then advance placement in an in-development Masters of Science in Learning Analytics at UTA. Each course runs about three weeks and we’ve tried to settle on prominent analytics tools for educational data so the experience is one where skills can immediately be applied.

We’ve structured these courses to provide an intro to analytics in education (a good compliment to the courses is our recently released Handbook of Learning Analytics – free download):

  • Learning Analytics Fundamentals
  • Social Network Analysis
  • Cluster Analysis
  • Predictive Modeling in Learning Analytics
  • Feature Engineering for Improving Learning Environments
  • Connecting Learning Data to Improve Instructional Design
  • Natural Language Processing and Natural Language Understanding in Educational Research
  • Knowledge Inference and Structure Discovery for Education
  • Multi-modal Learning Analytics
  • We have exceptional instructors – world leaders in the field. We are, however, well aware of the gender imbalance. We had five faculty (women) who ended up dropping out due to existing commitments. If you’d like to help right this imbalance, email me and let me know courses or topics that you’d like to instruct in the LA domain.

    Open Education: Please give

    elearnspace (George Siemens) - September 8, 2017 - 07:44

    In about a month, David Wiley and I are teaching this course on edX: Introduction to Open Education. As we are both firm adherents to social and participatory pedagogical models (i.e. we like it when others do our work), we need some help. Specifically, we’d love to have faculty/researchers/practitioners provide short 3-5 minute reflections on one or more of the following topics:

    Week 1: Why Open Matters
    Week 2: Copyright, The Public Domain, and The Commons
    Week 3: The 5R Activities and the Creative Commons Licenses
    Week 4: Creating, Finding and Using OER
    Week 5: Research on the Impact of OER Adoption
    Week 6: The Next Battles for Openness: Data, Algorithms, and Competency Mapping

    The process:
    1. Create a short video/tutorial or any other artifact (if we have yours by Sept 14, we’ll include it in the course) responding to any of the above weekly topics
    2. Upload your creation to some site where we can access/download it
    3. Email me a link (gsiemens, gmail) or share on Twitter using #openedmooc or leave a link in the comments
    4. Sit back and enjoy the feeling of accomplishment that will wash over you knowing you’ve made the world a better place.

    Not true: coding bootcamps you can take online are an "oxymoron" no one has yet solved

    Jon Dron's blog - September 1, 2017 - 19:50

    A Quartz article that claims (accurately) that p-learning bootcamps dominate for those learning programming and other technical skills and (inaccurately) that the reason for that is that e-learning is much less engaging. In fact, there’s a sneaky and almost unnoticeable sleight of hand here, because what is actually claimed is that online learning can be less engaging and, based on that indubitable fact, extrapolates from the particular to the general, asserting that all online learning suffers the same way.

    Nonsense.

    Yes, there is a lot of rubbish online learning and, in fairness, even a well-evolved establishment like Athabasca University has some of it, at least for some people some of the time. But that’s not at all surprising because every university (online or not) presents the same problem and, if you are trying to make a single learning design work for everyone, you are sure to make it too complex for some and too boring for others (personalization technologies and intelligent personal learning designs notwithstanding). Athabasca has a lot less of it thanks to its extremely rigorous quality assurance processes, but it would be crazy to imagine that everything it does is perfect for every learner at every time, as much as it would be crazy to imagine that everyone at a bricks and mortar institution gets a wonderful learning experience every time. Crazier, in fact.

    The thing is, it ain’t what you do, it’s the way that you do it, that’s what gets results. It’s not that online learning is less effective (countless studies prove otherwise), it’s simply that it tends to be done in a way that gives control and flexibility to learners. The immersion of physical bootcamps does have one major and very distinctive benefit: that it pulls people out of everything else and forces them to engage for a lot of hours in the day. The ‘bootcamp’ part of it ensures that they are well and truly immersed, with no way to back out apart from backing out completely  It’s not that the learning experience is any better – very far from it in most cases. Most bootcamps I have seen use inane pedagogies that would not pass muster even in a conventional university, let alone somewhere like Athabasca University that actually pays attention to such things. It’s just that people are there and they have to do stuff (a lot of social pressure is involved, as well as loss-aversion) so they wind up learning a lot simply because they put in the hours, and that happens simply because they are enrolled on a bootcamp and cannot get away.Not dissimilar to the ways traditional universities work, as it happens, just a lot more intense.

    Online learning gives people more choice and greater control so, if they are not innately fascinated or they have not very single-mindedly put aside enough time then, of course, they wind up learning less quickly because they put in less time over a longer period. Duh. It’s not rocket science. This is not about teaching effectiveness or smart learning designs, it is simply about stopping people from being distracted and doing other things. The solution, if a solution is needed, is for online learners to block out the time and drop the distractions. I can imagine plenty of learning designs for online learning that would make that happen – simply making it real-time and using smart tools for desktop sharing, real-time interaction, and monitoring of progress would achieve much the same results, as long as the ground rules are fully understood by all concerned. I was at a conference the other day that did pretty much that. Such an approach doesn’t happen much, of course, for all the reasons people go for online learning in the first place, inasmuch as such methods take away the control and flexibility that make it so appealing. On the other hand, perhaps there is a place for such techniques. It seems there is a market and, as long as expectations are carefully managed (you don’t distract yourself with reading emails and engaging in social media, you pledge to be available, you block out your calendar) it might work pretty well. But why bother? Seems to me that online learning is better precisely because of the control it gives people. If they need extrinsic motivation to force them to learn then that’s the problem that should be solved before enrolling on any courses, and it will do them a world of good in many other situations too.

     

    Address of the bookmark: https://qz.com/1064814/the-awkward-irony-of-not-being-able-to-take-a-good-coding-bootcamp-online/

    Original page

    Making the community the curriculum | Dave Cormier

    Jon Dron's blog - August 30, 2017 - 17:48

    The always wonderful Dave Cormier is writing a book (open, of course) about rhizomatic learning and, as you might expect given Dave’s eclectic and rich range of skills (from uber-tech-guru to uber-learning-guru) not to mention his cutting edge knowledge (this is someone so far ahead of trends that he actually invented the term ‘MOOC’) it’s brilliant stuff. Though it is a work in progress and still a bit raw in places, there are clues that this is not your common or garden e-book right from the opening chapter, Why we work together – cheating as learning which introduces the radical idea that people are pretty good at helping other people to learn while, in the process, learning themselves. Other chapters are equally charmingly named: Learning in a Time of Abundance, Five tips for slackers for keeping track of digital stuff and One person’s guide to evaluating educational technologies. What comes through most strongly in this is a vision of where we are going – where we must be going – in a world of increasing connection and increasingly connective technologies. In all, it provides an extremely practical, achievable, and pragmatic way of going about that without breaking everything in sight, very well grounded in theory, and very entertainingly (and very clearly) presented.

    I’d not noticed this work in progress till now and am very glad that I found it. Highly recommended reading for anyone in education, edtech, or who is simply interested in learning or how technology changes us, and how to manage that change. I really look forward to seeing the finished or, at least, the published product. My sense is that this will always be an evolving book because that’s pretty much the nature of the beast, and so it will continue to be relevant for a long time to come.

    Address of the bookmark: https://davecormier.pressbooks.com/

    Original page

    SpyStudent: hidden wireless video live transmission camera

    Jon Dron's blog - August 25, 2017 - 13:12

    Who does not know the problems with the driving test or studies testing? You have not time to learn and have more important things to do! And suddenly, the date for the exam or test in a few days.If your exam is important to you and you do not know what you should do otherwise, then you are right with us! Do not despair, we have something for you that can help you!  “

    This fabulous offer from dron.si (who knew that my surname was a thing in Slovenia?)  allows you, for a mere €374.17, to be the proud owner of the SpyStudent kit, a camera, mic, earpiece, and wireless transmission system made for “those who do not like to learn for the test”.

    You can easily and undetectably shove the various bits of transmitter down your underpants, assuming you weigh more than 300 kilos, you wear exceptionally baggy clothing, and you have no fear of numerous forms of radiation in your nether regions. You might be a little challenged to find a way to shove the ‘wireless spy earpiece’ that, from the picture, seems to be made for elephants, down into your ear canal, let alone ever hope to get it out again (I hope it holds its charge!) but that’s part of the fun.  Anyway, I am sure you can put up with a little inconvenience for a device that enables you (with the aid of your BMW-owning accomplice outside the building who you really hope knows the answers) to cheat on any exam or test with impunity.

    Obviously, exam invigilators have never seen microphones before so you’re fine on that count, and they never bother to look for people muttering into their shirts, holding up exam papers to their chests, or tilting their heads as though listening to large black objects shoved into their ears. And exams are normally taken in open fields, so the range won’t be a problem

    The man in the illustration is taking a break from his more usual activities of molesting small children/terrorism/voting for Trump, to enjoy cheating on his driving test. Sadly, he also cheated on the ‘holding your pen’ test, so it’s not going to end well:

     

    Don’t forget this crucial advice, though…

    You go into the examination room and you try to keep quiet as if nothing had.

    I hope nothing had.

    Address of the bookmark: http://www.dron.si/en/brezzicne-ip-kamere/i_424_hidden-wireless-video-live-transmission-camera-spystudent-button-camera-power-pack

    Original page

    Strategies for successful learning at AU

    Jon Dron's blog - August 15, 2017 - 17:02

    Earlier today I responded to a prospective student who was, amongst other things, seeking advice on strategies for success on a couple of our self-paced programming courses. My response was just a stream of consciousness off the top of my head but I think it might be useful to others. Here, then, with some very light editing to remove references to specific courses, are a few fairly random thoughts on how to succeed on a self-paced online programming course (and, for the most part, other courses) at Athabasca University. In no particular order:

    • Try to make sure that people close to you know what you are doing and, ideally, are supportive. Other people can really help, not just for the mechanical stuff but for the emotional support. Online learning, especially the self-paced form we use, can feel a bit isolating at times, but there are lots of ways to close the gap and they aren’t all found in the course materials and processes. Find support wherever you can.
    • Make a schedule and try to keep to it, but don’t blame yourself if your deadlines slip a bit here and there – just adjust the plan. The really important thing is that you should feel in control of the process. Having such control is one of the huge benefits of our way of teaching, but you need to take ownership of the process yourself in order to experience the benefits.
    • If the course provides forums or other social engagement try to proactively engage in them. Again, other people really help.
    • You will have way more freedom than those in traditional classrooms, who have to follow a teacher simply because of the nature of physics. However, that freedom is a two-edged sword as you can sometimes be swamped with choices and not know which way to go. If you are unsure, don’t be afraid to ask for help. But do take advantage of the freedom. Set your own goals. Look for the things that excite you and explore further. Take breaks if you are getting tired. Play. Take control of the learning process and enjoy the ride.
    • Enjoy the challenges. Sometimes it will be hard, and you should expect that, especially in programming courses like these. Programming can be very frustrating at times – after 35 years of programming I can still spend days on a problem that turns out to involve a misplaced semi-colon! Accept that, and accept that even the most intractable problems will eventually be solved (and it is a wonderful feeling when you do finally get it to work). Make time to sleep on it. If you’re stuck, ask for help.
    • Get your work/life/learning balance right. Be realistic in your aspirations and expect to spend many hours a week on this, but make sure you make time to get away from it.
    • Keep a learning journal, a reflective diary of what you have done and how you have addressed the struggles, even if the course itself doesn’t ask for one. There are few more effective ways to consolidate and connect your learning than to reflect on it, and it can help to mark your progress: good to read when your motivation is flagging.
    • Get used to waiting for responses and find other things to learn in the meantime. Don’t stop learning because you are waiting – move on to something else, practice something you have already done, or reflect on what you have been doing so far.
    • Programming is a performance skill that demands constant and repeated practice. You just need to do it, get it wrong, do it again, and again, and again, until it feels like second nature. In many ways it is like learning a musical instrument or maybe even driving. It’s not something you can learn simply by reading or by being told, you really have to immerse yourself in doing it. Make up your own challenges if you run out of things to do.
    • Don’t just limit yourself to what we provide. Find forums and communities with appropriate interests. I am a big fan of StackOverflow.com for help and inspiration from others, though relevant subreddits can be useful and there are many other sites and systems dedicated to programming. Find one or two that make sense to you. Again, other people can really help.

    Online learning can be great fun as long as you are aware of the big differences, primarily relating to control and personal agency. Our role is to provide a bit of structure and a supportive environment to enable you to learn, rather than to tell you stuff and make you do things, which can be disconcerting at first if you are used to traditional classroom learning. This puts more pressure on you, and more onus on you to organize and manage your own learning, but don’t ever forget that you are not ever really alone – we are here to help.

    In summary, I think it really comes down to three big things, all of which are really about motivation, and all of which are quite different when learning online compared to face-to-face:

    1. Autonomy – you are in control, but you must take responsibility for your own learning. You can always delegate control to us (or others) when the going gets hard or choices are hard to make, but you are always free to take it back again, and there will be no one standing over you making you do stuff apart from yourself.
    2. Competence – there are few things more satisfying than being able to do more today than you could do yesterday. We provide some challenges and we try to keep them difficult-but-achievable at every stage along the way, but it is a great idea for you to also seek your own challenges, to play, to explore, to discover, especially if the challenges we offer are too difficult or too boring. Reflection can help a lot with this, as a means to recognize what, how, and why you have learned.
    3. Relatedness – never forget the importance of other people. You don’t have to interact with them if you don’t want to do so (that’s another freedom we offer), but it is at the very least helpful to think about how you belong in our community, your own community, and the broader community of learners and programmers, and how what and how you are learning can affect others (directly or indirectly).

    This advice is by no means comprehensive! If you have other ideas or advice, or things that have worked for you, or things that you disagree with, do feel free to share them in the comments.

    Professor Jon Dron | Beyond Busy

    Jon Dron's blog - July 13, 2017 - 12:13

    An interview with me by Graham Allcott, author of the bestselling How to be a productivity ninja and other books, for his podcast series Beyond Busy, and as part of the research for his next book. In it I ramble a lot about issues like social media, collective intelligence, motivation, technology, education, leadership, and learning, and Graham makes some incisive comments and asks some probing questions. The interview was conducted on the landing of the Grand Hotel, Brighton, last year.

    Address of the bookmark: http://getbeyondbusy.com/e/35495d7ba89876L/?platform=hootsuite

    Original page

    SCIS makes a great showing at HCI 2017, Vancouver

    Jon Dron's blog - July 12, 2017 - 18:47

     

    I had the pleasure to gatecrash the HCI 2017 conference in Vancouver today, which gave me the chance to see Dr Ali Dewan present three excellent papers in a row (two with his name on them) on a variety of themes, as well as a great paper written and presented by one of our students, Miao-Han Chang.

    Both did superb jobs of presenting to a receptive crowd. Ali got particular acclaim from the audience for the first work he presented  (Combinatorial Auction based Mechanism Design for Course Offering Determination
    by Anton Vassiliev, Fuhua Lin & M. Ali Akber Dewan) for its broad applicability in many areas beyond scheduling courses. 

    Athabasca, and especially the School of Computing and Information Systems, has made a great showing at this prestigious conference, with contributions not just from Ali and Miao-Han, but also from Oscar (Fuhua) Lin, Dunwei Wen, Maiga Chang and Vive Kumar. Kurt Reifferscheid and Xiaokun Zhang also had a paper in the proceedings but were sadly not able to attend to present it.

     

    Jon and Ali at the Vancouver Conference Centre after Ali’s marathon presentation stint. I detect a look of relief on Ali’s face!

     

    Papers

    • Combinatorial Auction based Mechanism Design for Course Offering Determination
      Anton Vassiliev, Fuhua Lin, M. Ali Akber Dewan, Athabasca University, Canada
    • Enhance the Use of Medical Wearables through Meaningful Data Analytics
      Kurt Reifferscheid, Xiaokun Zhang, Athabasca University, Canada
    • Classification of Artery and Vein in Retinal Fundus Images Based on the Context-Dependent Features
      Yang Yan, Changchun Normal University, P.R. China; Dunwei Wen, M. Ali Akber Dewan, Athabasca University, Canada; Wen-Bo Huang, Changchun Normal University, P.R. China
    • ECG Identification Based on PCA-RPROP
      Jinrun Yu, Yujuan Si, Xin Liu, Jilin University, P.R. China; Dunwei Wen, Athabasca University, Canada; Tengfei Luo, Jilin University, P.R. China; Liuqi Lang, Zhuhai College of Jilin University, P.R. China
    • Usability Evaluation Plan for Online Annotation and Student Clustering System – A Tunisian University Case
      Miao-Han Chang, Athabasca University, Canada; Rita Kuo, New Mexico Institute of Mining and Technology, United States; Fathi Essalmi, University of Kairouan, Tunisia; Maiga Chang, Vive Kumar, Athabasca University, Canada; Hsu-Yang Kung, National Pingtung University of Science and Technology, Taiwan