Project evaluation

On Monday 25th June I went to an ALT workshop on evaluating projects.

I chose to concentrate on the evaluation of a pilot of the e-portfolio software PebblePad. It looks like we will pilot with staff and a few students in the Autumn Semester 2007 and then have further pilots based on using it within modules in the Spring Semester 2008. If all goes well we can roll it out to all in the Autumn Semester 2008.

Some interesting points were made including:

  • Who is the audience for the project?
  • Is there more than one audience?
  • How will the evaluation differ for different audiences?

This got me thinking about how to show that the project was working to all stakeholders including managers, staff and students. It looks like I will need at least two different sorts of evaluation instrument, managers are more interested in quantitative evaluations which say what percentage of people said what whereas the users of the project are more interested in qualitative evaluation involving issues around learning, ease of use, workload etc.

It looks as if we need to show that the project is successful two these different audiences but there will be some overlap as getting a good evaluation against the student and staff criteria will mean that more people will want to use the software and therefore it is more likely to be deemed a success by managers.

A recent conference

Last Thursday I went to the Blended Learning Conference in Hatfield where I gave a presentation on using blogs for learning.

Here is the abstract of the talk, if you want more information please contact me by commenting on this post.

The use of social software is becoming more popular in higher education as a means of engaging students in their learning, and there is some evidence that students are aided in their learning as a consequence of engaging with this technology (Glogoff 2005). However it is clear that it is not the use of technology that is important in enabling the learner to learn but the way that the technology is used to support appropriate pedagogy (Oblinger & Hawkins 2006).

There are a number of learning principles that may be applied to the use of technology in a blended learning context (Chickering & Ehrmann 1995, McLoughlin 2000). It is the degree to which these, or equivalent, learning principles are incorporated into the student learning environment that determines how effective the learning tools might be.

This presentation is based on an action research project which used blogs and wikis to encourage students to be reflective about their work and to feedback to themselves, their peers and to their tutor. Students taking part in the project are able to take advantage of the technology for learning because its use is appropriately blended with a number of well established learning principles.

The presentation will discuss the following:

  • how the use of the technology was linked to learning principles to support effective learning;
  • how the use of these tools promoted good practice in the student’s learning;
  • what were the student’s perceptions about the how the tools aided their learning;
  • how effective was the use of these tools in encouraging learning;

A number of examples of using these tools will be discussed and from these a series of recommendations for effective practice will be made.

References:

Chickering, Arthur and Stephen C. Ehrmann (1996), “Implementing the Seven Principles: Technology as Lever,” AAHE Bulletin, October, pp. 3-6

Available at http://www.tltgroup.org/programs/seven.html (accessed February 9th 2007)

Glogoff, S. (2005) Instructional blogging: Promoting interactivity, student-centered learning, and peer input. Innovate 1 (5).
http://www.innovateonline.info/index.php?view=article&id=126 (accessed January 11, 2007)

McLoughlin, C. (2000). Beyond the halo effect: Investigating the quality of student learning online. Southern Cross University. Proceedings Moving Online Conference, pp.141-154.

Oblinger, D & Hawkins, B. (2006) The Myth about No Significant Difference, Educause Review, November/December 2006

Some interesting ideas about web 2.0 software

At a conference today about e-learning [details here in Rod’s post] I found some interesting explanations, in video form, for some web 2.0 technologies:

At the Connectivism Conference - a few reflections

Finally found time to join the conference, even then only the recording. Started with Diana Oblinger’s talk which so far is re-iterating things we all know already but done in a very engaging way. At first there does not seem to be a focus on Higher Education as I was expecting.

I was thinking are we going to talk about the relevance of this for student learning? Then we came to the section on implications.
Implications

  • distributed cognition

don’t see how this works - I agree that expertise can reside in a number of different people/places but is this actually distributing intelligence and learning - is learning not in the mind of the beholder?  Seems to be about aiding cognition by using activities which are supported by technology and allowing students to work in communities. Information can be shared but can understanding?

  • any place is a learning space

technology enhances the environment - flexibility is the key - so we need ubiquitous computing on campus and off - libraries are changing, but not always i the way that people want - wireless access that works for all

  • net savvy

our students and ourselves - need to be able to evaluate information - key for students just because they are IT literate it does not mean that they are e-learning literate - are there issues about providing open information? People are more aware of the implications of providing and using information now but what about stuff they published in the past before they became net savvy?

What do we need to do now?

Widening definition of learning - does our model reflect present circumstances - do we need to use communication and networks more? How do our students define learning.

Develop skills about use of images and other ways of representing information rather than just using text?

Re-assess and unlearn - get rid of debilitating prejudices - CPD

Not about preserving the past but creating the future - agreed

Trying out on-line discussions with academic staff

I have spent some time over the last two weeks trying to get a discussion group going using WebCT about e-learning to give participants:

  • some experience of things from the student side;
  • an insight into some of the issues about e-learning;
  • an opportunity to generate some guidelines about promoting on-line participation

We deliberately made this a low volume discussion expecting two  sessions of activity a week, one at the start and one at the end, each of about 30mins

So far participation has not been high although all but two participants have posted something and I am wondering whether to do try to make an intervention in an attempt to improve participation.

I am also wondering how things could have been made to work better. Perhaps it was naive of me to assume that tutors would have the necessary skills to organise themselves for on-line activity and I needed to be more prescriptive about it.

More later ….

The digital divide

Here is a link to Jacob Neilsen’s comments about the digital divide.

In it he puts forward three types of divide:

Economic - which he maintains is no longer an issue as the developed countries will be able to afford computers for all their citizens within the next five years and poor developing countries will need at least twenty years before their citizens can afford computers. Of course there are initiatives like the 100 dollar laptop scheme that might help with this.

Usability - he maintains that for many computers are simply too difficult to use and that this much more of an issue than access.

Empowerment - even if people can access and can use computers many of them do not feel empowered to use them effectively and take part in on-line communities. He quotes the 90:9:1 rule.

Although I can’t say that I agree with all of his comments I do think he has a point worth considering.

University 2.0

I have been thinking for some time about the implications of web 2.0 for universities.

Web 2.0 is about people creating their own information on the web and sharing their ideas with one another in order to create further information. If information is available to all and the individual can make their own decisions about what material they access and how they do it what does that leave for the University to do?

It seems that universities are going one of two ways:

  1. they are setting up systems which are designed to control student’s access to materials by using software like WebCT as a managed learning environment (MLE);
  2. they are allowing students to make their own decisions about information access, putting information in the public domain and encouraging the use of software like blogs and wikis;

Of course the situation is not as clear cut as this because universities consist of individuals and there is still an opportunity for academics and students to work autonomously and make their own decisions about information access outside the “official” university system. However this path can only be taken by those who are sufficiently e-literate to know about and be able to use the “non-official” systems.

Are we in danger of creating a bipartite structure where some of us have access to a range of technologies and others do not? If so who are going to be the haves? Do we really have digital natives who can use the digital systems to enhance their learning? Perhaps the situation is not so clear cut and it depends on a subtle interaction between academics who are expert in the academic use of the technologies for learning and students who are confident and expert enough to use these technologies.

I have seen no evidence of digital natives who are also digital learners. Most students need to develop the skills of using e-technologies appropriately for learning. Is this because I just don’t understand they way that students learn with technology and that I am trying to use a “traditional” model of learning in an on-line situation? Or is there actually no difference between the way that people learn with traditional technologies such as paper and pen and with computer systems? If people construct their own meanings and one of the ways they do this is through social interaction does it really matter how they undertake this social interaction? Is the advantage of e-learning that it provides more opportunity for access to learning and information rather than it providing a “new” way of learning?

There area number of people who have something to say about university 2.0. These include:

Is connectivism a theory of learning?

I was reading George Siemens’ blog post Externalising Knowledge which led me to read this article by Bijdrage van Pløn Verhagen which seems to argue that connectivism is not a theory but a pedagogy. Seimens has written an extensive article where he defends the ideas of connectivism bringing forward a number of arguments based on developments which have taken place since his 2004 article.

This is just my first impression as the arguments are long and involved and I will need time to read them properly before I feel confident to make further comment. I also need to re-read Stephen Downes’ article on connective knowledge.
So far I am struggling with the essential differences between social constructivism and connectivism but as Siemens remarks this may be because I have a fluid idea of the meaning of social constructivism. However I do think that whatever the theory we use learning is about constructing a meaning for ourselves and others through interaction and the ability to interact is an essential skill. In the twenty first century interacting through technology will become an essential life skill, if it is not already so.

As to whether the meaning we construct resides solely in ourselves or whether there can be a greater meaning that we have as a consequence of our interactions is something about which I am not sure.

Who is the expert now?

I am not sure that this is really about e-learning just that e-learning seems to highlight the issue.

Alex Reid has a post at his Digital Digs blog where he says:

I know my students and colleagues view me as an “expert” in new media, but how could I be an expert in something I did for the first time only a few months ago, something that may not have even existed a year ago?

I’m just keeping ahead of the curve, my “expertise” coming from my ability to tap into a network of information. In other words, my expertise doesn’t come from inside, from my authority, but from outside, from my intersection with the network.

It seems that anybody who knows a little more than others is considered to be an expert and that too many people rely on these “experts”. Of course it is easier to ask somebody else to do something for you than to find out for yourself.

Recently I had a request to set up access to a discussion area where tutors could watch how an “expert moderator” went about their work. I was a little uneasy about doing this because of the model of learning that this implies and have tried to get people to go along with a more participative system where they take part in a discussion and reflect on how the moderation helped or hidered the learning.

How do we know who is reading our posts

I was reading an interesting post about the nature of blogging by Vicki Davis where she said:

The point is not whether you screw up, the point is whether you keep playing. I had one reader - myself — for a while. Then I had three. Then I had seven until mid January. (I started blogging in December.) Now, I have a lot more than that.

This made me think about how you know how many readers you have. I have added some counters to my blog so that I can see how many hits I have but I have no idea how many actual readers there are, if any :-)

Perhaps I could ask readers just to say hello in a comment on this post? What do you think?

If you are one of the few reading this blog please let me know.