Assessing pupils work for #digitalstudies – my examination of badges and levels

Today saw a fast and fantastic discussion take place on twitter between a number of people on the topic of how to assess work within #digitalstudies. It began with @misterel asking how we would be showing progress by pupils – a very reasonable question. To answer it though I will be taking a sidetrack down my recent computer games playing experience.

Two of the last major computer games I have played have been the original Mass Effect and Modern Warfare 2 (I know I am slightly behind … I blame parenting and Minecraft :-) ). Modern Warfare 2 firstly is a classic example of an on the rails shooter. Yes you can run from one side of the path to another side and do a teensy amount of sight seeing but if you don’t follow Captain British Stereotype the next story trigger won’t trigger and nothing happens. You exist in a very strictly designed world where you need to progress through the levels and options as set out for you by the game designers. I know for those who are reading this I am probably stating the obvious but hear me out.

Mass Effect on the other hand is less of an on the rails shooter. You are presented with some degree of freedom in the choices you make and can see some effect of those choices. Choose to ask someone nicely and they might offer you something out of kindness or be horrible and they respond out of fear. Once you are on a mission there may be some form of ‘rails’ but some freedom persists and once out of a mission you can choose to sidestep optional missions and focus only on the most important ones. You are still following what the game designers have decreed but there is a lot more flexibility in the order of choices you can make.

Having mentioned Minecraft as an aside perhaps I can also bring that back in as well and say that even though it’s a classic open sandbox game where you are free to make your own choices you are still bound by certain rails and rules. In survival mode youcan’t break through the adminium blocks at the bottom of the level and you can’t build higher than a certain level.

So how do games like MW2 on the one hand and Mass Effect and Minecraft on the other hand apply to assessment and learning in education. I think that MW2 is an analogy for National Curriculum levels and Mass Effect and Minecraft are an analogy for badges as an assessment method.

As in a game pupils need to make progress through the tasks they have been set and they and the teachers should be able to show that progress. Levels as a concept can work as it allows a pupil to see where they are, what skills they have gained so far and be able to see where they should go next. However the definitions for the levels (and I am thinking just for ICT here) were so arcane i can remember spending forever translating them into pupil speak (and wondering why they weren’t in pupil speak in the first place).

Like an on the rails shooter I also thought that levels were restrictive of a pupils progress and ensured that even with separate strands of levels a pupil was in essence working through the same levels as everyone else. As I think @mberry put it today – pupils may be climbing different mountains (but sometimes the same foothills). As soon as a pupils is wanting to leapfrog aspects of the levelling criteria it becomes very hard to pin them down to a number which shows the ‘progress’ the levelling system is meant to show.

Mass Effect and Minecraft are therefore analogies of a better approach to learning and assessment. Pupils remain within a semi-walled garden as even with freedom of choice in learning paths teachers must still provide the guidance and structure to make sure that pupils are continuing to learn in a good way. But pupils are presented (like the missions in a game) with a choice of things they wish to proceed with. There may still be some order to it as i can think of a situation in Starcraft 2 where I was unable to proceed to a next mission unless I had completed a specific previous mission bit there is still some flexibility.

Badges to me reward a pupils choice, naturally promote flexibility, provide encouragement to proceed down a learning path but also allow pupils to choose a different path from some of their peers. I have always believed that the goal with this new focus on programming is not to turn every pupil into a programmer but give every pupil the chance and option to take that path and then provide other alternate paths such as digital law or digital authoring.

#digitalstudies is a portfolio based subject. At the beginning of a year I feel that pupils should be shown what they should ideally be producing by the end of the year. However in a portfolio such as the multimedia one I am working on pupils could either be focussing on using digital tools to design and create beautiful digital works of art in imagery, audio or video but doing less on the programming side or they could be doing a small amount of digital creativity but focussing more on coding an awesome multimedia website. This approach I think is a reflection of a typical startup where each person has a particular strength and focusses on that strength but is able to work with others. Badges suit this type of work approach as it allows each pupil to work towards their own path which may be independent to someone else even in their same group. Levels I think are highly individualistic and don’t adapt naturally to work done in a group whereas a badge I think could be easier to assign. Pupils then show their progress through an accumulation of badges which could start to indicate both to the pupil and the teacher what their natural ‘discipline’ within #digitalstudies is.

Although I had a go at MW2 earlier I do remember spending quite a bit of time trying to nail 3 stars in as many specops missions as I could. Even though I knew that others were also working towards it it was a badge for me and I really wanted it. I think this could also apply to how a pupil could feel motivated in class.

How does APP and AfL come into this? Although I have focussed on badges I do believe strongly in formative assessment. Badges although summative as well are I think a better summative assessment method than levels. Pupils would read a number or grade and ignore the comments so often my feedback is only formative and includes no summative grades. From my own understanding of AfL I think that formative assessment is a necessary part of AfL and therefore if you are doing the formative assessment properly you are naturally driving AfL forward as well.

APP though although I have never used it being at an independent school I think is too bureaucratic and too structured. It contains a pupil too much within a top down methodology and therefore lacks the flxibility of badge based assessment.

Badges therefore in conclusion provides the flexibility, gives the pupil a measure of independent choice in what they wish to learn and show progress in and ultimately provides a learning and assessment experience which I think comes closest to what exists in the workplace which is ultimately what we are preparing the pupils for.

About these ads

14 thoughts on “Assessing pupils work for #digitalstudies – my examination of badges and levels”

  1. @paulwill – not ignorance at all sir! a very searching and relevant question.Your question about what pupils are learning is apt. Yes this is a very experiential course through what pupils do but there should be within each project opportunities for theoretical learning through concepts and issues which pupils have to research, write about and hopefully apply through what they are doing in their work.As for a list of competencies there is (although not very public) a list of simple competencies for each strand. I wonder though whether having a master list of competencies (eg a PoS) isn’t necessarily the best approach. Instead each project contains their own competencies and the teacher then is able to choose a series of projects which then provide the competencies they want to teach. In a sense I am thinking why not move away from nationally standardising the learning in #digitalstudies towards a highly individual based approach. After all when I look at someone coming for a job I don’t compare their skills to a national average, I look at that person on their own merits.Having said that we do have a way of providing assessment which if it works to a full extent could provide that individualised assessment but also provide a way of standardising pupils achievements and that is through digital badges. We are working on a single issuing engine for the badges and this therefore would give us a simple way of seeing exactly what a pupil can do and what they still have left to do. Developing digital badges will give us the means of identifying what we are going to assess.#digitalstudies I hope will be developed both on a school by school basis and as a collaborative enterprise. I want to encourage schools to work on their own interpretations and then feed those back into the national debate on its approach. In a sense its like the open source approach whereby different schools can ‘fork’ the subject but ultimately still pass back features and updates to the master approach.I think with the controls being lifted by Gove earlier in the year coupled with the fact that I work at an independent school I am really trying to see what can be done very differently. There are very few ‘holy cows’ in setting up #digitalstudies. Now at a state school level this can be harder to do but as long as we have a DfE who are providing the freedoms to do these sorts of things I am going to take those opportunities and do as much as possible with them.Paul, thanks again for the comment. I really do enjoy being able to thrash out ideas about the subject and I hope what I have said helps.All the bestBrian

  2. Forgive my ignorance…..the method of problem/project based learning is a sound methodology for experiential learning. Agreed. But what are pupils learning? The 4 strands that are listed as far as digital studies are clear as titles and definitions, but do we not need to develop a list of competencies? What badge specifically would be awarded for this particular competency of this particular strand etc.?This is starting to sound a little like levels/competencies, but all assessment is a measure of something against something. We therefore need to decide what to measure before we can measure it.Are people expecting that DS will be developed on a school by school basis or a collaborative enterprise? I’m coming to this topic a little late!

  3. Hi @mrstucke – I think #digitalstudies needs to hang a sign out front saying high level critique is very welcome and so definitely thank you very much for your comment. I can see three points you have made. Firstly the balance between portfolio and curriculum and what they will learn, secondly the badges and lastly the length of time for portfolios.On the first point about skills versus curriculum: One of the original ‘planks’ of #digitalstudies was the fact that this is a skills based course. However I think we should hopefully be able to ensure that sufficient ‘curriculum’ comes through as well. In starting to thrash out some ideas for one of the first portfolios there are already things like checking legality of what you do. Now obviously this may involve some ‘skills’ but largely this is going to involve the pupil learning the in’s and out’s of something like copyright law and creative commons. So there is going to be that mixture but a definite requirement for #digitalstudies will be that need to signpost what it is the pupils are learning. Portfolios therefore still remain key as although they will be providing some structure in terms of the skills that pupils will be needing to demonstrate they should also define the learning as well.On the second point about badges I agree with you about going for bronze, silver, gold levels of badges. If pupils are doing lets say CSS in Year 7, 8 and 9 it fits quite reasonably. However what about schools which go from primary through to to secondary? A lot of answers I am pretty sure will come from Chris’s conference call with Mozilla’s openbadges initiative.On the issue of length I think you may be right. When #digitalstudies kicked off the year long focus was a key element to it for a number of reasons. I know Chris has started thinking about term long projects now as well so I think there should be scope for both. The key should be flexibility so that schools can spend lets say a term on a portfolio then two terms, then a year then back to two terms etc. Although I need to dig up what Nick has written again I would say that the student led nature of this should hopefully also come through strong use of group work as well.many thanksBrian

  4. Hi All,First a massive well done on the work that’s gone into #DigitalStudies so far – your enthusiasm and efforts are remarkable.Please don’t take these as criticisms but some devil’s advocate questions having spent an hour or so reading through the wiki and posts on here and all with my hat on of an ICT leader thinking about our KS3 curriculum for next year. :)I posted in here specifically because one of my big Qs is highlighted by assessment discussions: “What are the skills / topics that a ‘course’ of Digital Studies covers?” As an outsider looking in at the moment the idea of portfolios and projects is clear, along with some tools that will be used. But the good old fashioned curriculum seems to be unclear at the moment. I still think that school leaders, governors and parents will want to know what the pupils will be learning in this subject. Gove’s current position is a great opportunity for schools to effectively write their own curricula. And perhaps I’m being traditional but I think these need to come first, and then the how of delivery comes second. What do you think?On the badges front I think there’s a great opportunity here and I’m really keen to look into this further. I’m the man in charge of data across our school, but even I hate levels. I mean, what does a level 5c in ICT mean? And in particular what does it mean to a student or parent!?? Would love to see badges based on specific parts of the curriculum, maybe at gold/silver/bronze level or something similar. And putting my Maths teacher hat on it would be wonderful to be able to look at my Maths class and see what spreadsheet / data handling badges they have gained in ICT – would be fantastic for planning work. Better still I think it might simplify the process of covering more of the traditional ICT curriculum in other subjects, hence freeing up more time in Digital Studies or wherever for the more specialist side of things like programming. Our technology and art departments do photoshop work for example, I’d be delighted to put together a few image editing badges and effectively verify the work that students are doing elsewhere around school – letting them use the work from other subjects in their portfolios.I think there’s a great opportunity here, but I think what I’m getting at is that a big annual portfolio / project might not be ideal for many schools especially as ICT time on timetables is squeezed. Getting the brains on this project together to really thrash out what skills knowledge and understanding we should be providing at this age would be amazing then looking at how it’s delivered, whether that’s through portfolio projects as suggested, through student led projects like Nick Jackson recently blogged about or across subjects like Dai Barnes is beginning to work on.That really is a bit of a ramble but I hope it’s helpful!Dan

  5. @teachesict Hi Nic, hardly a ramble at all. Your point about motivating pupils to move outside their comfort zone is very important. Just as much as we should accept that some pupils are naturally going to gravitate towards a set of skills such as programming or creative stuff we should be pushing these pupils to gain skills outside their own skill set. Something I took from Steve Jobs’s autobiography was that compared to other technology companies where people were almost entirely focussed on the engineering at Apple they saw themselves not only as engineers but also as artists. I think that’s the key to innovation and why focussing purely on computer science at KS3 is a bad idea.

  6. @ianinsheffield – yeah largerama did hit the nail on the head – progressive or levelled badges may be needed for #digitalstudies. I wonder whether I need to go play some more modern warfare to come up with (pinch) a way of figuring out how it could be done :-)As for pupil’s making comparisons with each other I am pretty happy with that as it is pretty natural as you say. But I think as long as we are aware of this we should be able to manage it. Your point about the motivation for the pupil is very important and we you are right – the learning must be the motivation and the badge the reward for undergoing that learning. The answer to that may simply have to be leaving it in the hands of the teacher in the class to make sure that happens.thanks again Ian :-)brian

  7. @largerama you have definitely hit on a small problem with #digitalstudies which I am trying to think through. It’s a positive problem though which is good.The portfolio I ‘should’ be working on for next year’s year 7′s at my school shouldn’t necessarily be something which is always done in year 7 every year. I think either myself or any other teacher should feel free to pick and choose whichever portfolio they intend to do each year so pupils could start with a game design portfolio in year 7 and then do lets say multimedia in year 7.However if both portfolios have similar skills or badges how do you show progression from one year to the next? So let’s say the pupils do image editing in both portfolios. We may very well need a way to show levelling up within the badges so if they have already got a badge in a skill and those skills are being used again in a future portfolio they can ‘improve’ that badge.I think two things need to happen – one someone from #digitalstudies needs to have a chat with mozilla’s @openbadges and two we need to see if we can bring a developer on board who can build an online system to help manage the possible complexity of awarding badges so that it does remain simple for pupils and teachers.

  8. Great post Brian and really interesting follow up posts from Miles, Ian and Nick which have raised some thought provoking issues with regards to the badges. I thought I would add my thoughts as well!I believe that the badges will be an essential part of #digitalstudies rather than levels as it will allow pupils to choose their own paths based on their own skills and interests. It will also allow, where appropriate, similarities to the ‘real world’ where teams solve problems by delegating sub tasks according to the strengths of the individual members of the teams. As pupils progress, it will become clear which pupils are the programmers, the designers, the web page creators and their strengths can be utilised rather than overlooked as is the danger with the current assessment systems.That said, we also have to be very careful of those pupils that feel a badge is unattainable or is for a topic that they currently have no interest in. We have to ensure that pupils do attempt badges outside of their comfort zone and do not pigeon hole themselves as programmers, designers or web page creators. To this end, as has been discussed in the earlier comments badges should have degrees of difficulty themselves in order to encourage a pupil to at least have an initial look at a new skill or challenge – after all “we don’t know what we don’t know”.I realise that this comment has been a ramble and not as clear as the idea in my head. For more info please look at http://teachesict.wordpress.com/2012/04/07/assessing-digitalstudies-grades-or

  9. Brian, thanks so much for such a fulsome response; I can now see much more clearly how student choice and learning path flexibility are addressed.On the issue of progression, I think the scout sleeve analogy (I was in Scouts too!) works well in describing breadth of experience, but here I’d suggest Nick has a point that there is also room for badging at different levels, indicative of greater development within a particular skill set. Incorporating depth was something I was keen to include in the project I’ve been working on (http://ianinsheffield.wordpress.com/2011/10/02/questing-in-or-non-formal-lear… i.e. the facility to recognise different levels of expertise within a particular skill domain.I’m right with you as far as comparative progress is concerned. Provided each student is being challenged and progressing at a rate appropriate to his/her needs, then surely that’s what really matters? My only slight concern here is that with progress being so visible (that’s the whole point of the badge after all), the students themselves will make the comparisons, as they invariably do. Now one might argue that a competitive element is healthy; as you said the motivation is there to strive to increase one’s badge portfolio. However we must be wary of the degree to which that motivation then becomes solely extrinsic, with the badge becoming the focus, rather than the learning which leads to it. I’ve already been pondering this here http://ianinsheffield.wordpress.com/2011/12/29/no-ive-not-had-a-good-xmas-and… following a cautionary note offered by @JohnMcLear at a #currymeet of all places!But having said all that, I’m very much in the ‘nothing ventured’ camp and exploring alternative mechanisms by which achievement can be recognised is so important at the moment.

  10. I wasn’t privvy to the discussion on badges and so what has come to my mind after reading this you may have already covered:First of all this is amazing work by all of you involved and I will continue to add my little titbits and offer assistance where I can but on the badges front, it strikes me there are questions:Could some badges be worth more than others? i.e. harder to get/more work to get/longer process/deeper thinking/show greater skill or understandingCould some badges be part of other badges a progression (bit of a) badge along the way to a full badge? e.g. full web design badge is not achieved until whole website done but you can achieve a CSS manipulation badgeIf either or both of these can be the case then it begins to make sense to consider putting badges into ‘bands’ or ‘levels’ and hence making any conversion to levels easier for any school that requires it. I don’t know if I am upsetting the apple cart with this here or indeed like Ian, playing devil’s advocate but it’s what came to mind.

  11. Hi @IaninsheffieldI think we all welcome a little bit of devil’s advocate. #digitalstudies is becoming a serious option for schools and therefore to be taken seriously we need to definitely respond to any constructive critique such as yours.I think the flexibility from using badges comes through what badges pupils might be able to work towards. To start with the portfolios / projects may be quite heavily structured and pupils do take a fairly similar line of progress through the work. However I envisage as an example that within lets say a group of pupils who are working together on a website you have a pupil who is concentrating heavily on producing good quality imagery which meets the client brief, is designed well showing effective annotations and is then produced with great skills showing good use of advanced features and is optimised effectively for use on the web. In the exact same group you have a pupil who is instead working on the CSS and HTML which will pull in the imagery, display it on the page in the right way and create code which is compact, well written and does the job.Using traditional levels you may be able to distinguish the two pupils but I feel that with the way levels are written it may prove cumbersome to do so and the teacher will simply assign a general level which fails to acknowledge the radical difference between the two pupils. By having two different badges available – one for the graphic creation and one for the CSS and HTML and assigning those separately to the two pupils I think is a much neater and more relevant way of acknowledging the skills they have shown.So how does one show progress? Another analogy to use is that of the scout uniform sleeve. When I attended scouts in South Africa way back in the day we had a semi-militarised uniform so I have no idea what goes on now but our uniforms allowed us to sew our badges on our sleeves whenever we achieved a skill. These sleeves didn’t show levels or progression charts, they simply showed the amount of badges a boy had and let me tell you this, boy were we motivated to gain more badges to show off on our sleeves. I can remember feeling slightly embarrassed by my collection at one point and going through the little book we had of activities to do to gain badges to see which ones I could do to gain more badges.So my first point about showing progress is that the collection of badges simply becomes your means for assessing progress. It is an easy way to show how much a pupil has done and because the full list of badges is made available before the portfolio starts they can see what they can still do. Tying into what I said further up about different types of badges pupils can also concentrate on collecting badges to suit their own skills (and this is where the teacher can obviously direct pupils who are unsure of their own strengths).But what about levels? My simple answer to that is how many technology companies ‘level’ their employees one against another? I don’t think many if any do and yet we are preparing pupils for hopefully technology roles one day. A badge becomes a simple metaphor for the acquisition of skills which can easily then be translated into that first CV.So I think ultimately comparative progress is largely irrelevant for #digitalstudies but certainly one can track the chronology of progress simply through when badges are assigned.All the bestBrian

  12. Hi Brian. Just catching up with what you guys have been discussing on the Badges front. Fascinating!I’m definitely in favour of this alternative way of recognising and celebrating achievements, but with your forbearance I’m going to play devil’s advocate for a moment. Can I ask for a little clarification of how you think Badges might provide more ‘flexibility’ and ‘increased choice?’ Since the Badge represents a token picked up on *completion* of a learning journey, I was just thinking that flexibility and choice might kick in at the commencement of that journey. Or to put it another way, the Badge is the destination, whereas flexibility comes from the routes available?I too think that Badges might provide a good way of revealing progress, especially to a wider audience than with current systems, but haven’t yet resolved how best to show *rate* or chronology of progress … or doesn’t that matter? Or even comparative progress … or doesn’t that matter either?Be good to hear your thoughts, as I’m wrestling with these issues internally at the moment … and wrestling with yourself is no fun at all! ;-)

  13. @mberry thanks for the comment. I like the use of the terms spatial and temporal to describe different learning methods as even looking at the meaning of the word spatial it implies the flexibility and choice which is needed within learning which a pupil is unable to use under a temporal lesson.Thanks for the link as well – certainly looks useful.

  14. Great post Brian, and a really interesting conversation yesterday.We include a lecture on ‘non linear narrative’ as part of our first year ‘creativity and computing’ module, which touches on some of the issues raised here, as well as giving us an excuse to play some old school text adventures (many of which were on rails, of course). The analogies often divide into spatial (non linear) vs temporal (linear). Much learning design assumes a single path, a ‘learning journey’, or at least a common destination, whereas in reality learning is, I think, mcuh more often about exploring a landscape than reaching a destination.Christopher Alexander’s classic ‘A city is not a tree’, see eg http://www.rudi.net/pages/8755, is worth a read, keeping the ideas of both learning and curriculum design at the back of your mind whilst you do.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s