UC Davis Sustainable Agriculture & Food Systems (SA&FS)

Screen Shot 2014-03-14 at 4.52.42 PM

Learner Driven Badge Project

Preface

The information contained in this project appendix was gathered from the original project proposal that was funded in 2012 and interviews with project leaders through the end of 2014.  This appendix and the overall design principles database from the DPD project does not reflect further evolution of the project or developments after the final interview that took place in 2014. As of the final interview, the UC Davis Sustainable Agriculture & Food Systems (SA&FS) had issued badges but ultimately ceased development of its badge system. Based on this information, we have classified the SA&FS badge system as a suspended (rather than an implemented or partial) badge system..

Summary

The Sustainable Agriculture & Food Systems is a new interdisciplinary major that integrates a portfolio system featuring digital badges. UC Davis’s Agricultural Sustainability Institute (ASI) and College of Agricultural and Environmental Sciences developed “a model of learning, participation, and assessment focused around high-level ‘core competencies’ that bridge classroom and real-world experiences, academic investigations and concrete skills. We believe this model has the ability to train leaders who will transform the food system, changing the way each of us eats and lives” (DML Stage 1 Proposal).

A distinguishing feature of the Sustainable Agriculture & Food Systems badge system is that students have a role in defining the criteria for the badges they attempt to earn. While badges that represent named “core competencies” are identified and include some program-related requirements, the the criteria for all of the other badges in the system are not defined. Students each develop their own criteria, and in claiming a badge, open up both their achievements and their criteria to feedback by peers and mentors (DML Proposal).

The new major aims to create graduates who can lead in shaping the future of agriculture. “ASI carried out a Delphi survey of academics, practitioners and students that identified which skills, content and experiences were most essential to prepare students to lead in the field of Sustainable Agriculture and Food Systems”. The new program intends to create a system to ensure that program requirements are being met “while preserving the individualized, dynamic nature” of each student’s achievements in order to match the needs of the careers in the field and the range of learning opportunities available (DML Proposal).

In addition to explicitly allowing students to “find and represent learning experiences outside the university, to help them build learner identities and communities, and to clarify learner goals and gaps”, badges can be “a system that responds not only to the diversity of learners and diversity of problems to be solved, but also to the process of learning in the real and changing world” (DML Proposal).

Coinciding badge and curriculum development

The interdisciplinary Sustainable Agriculture and Food Systems major is new to UC Davis, and the programs that created it were able to think about their new opportunities for recognizing learning as they decided what learning they hoped to emphasize. Designing a competency-based recognition system was appealing because SA&FS organizers felt the skills important to the future of agriculture and food are not exclusive to one course or experience.

The idea to use badges was not part of the program from the very beginning. Joanna Normoyle became familiar with badges through following the Digital Media and Learning community, which is deeply connected to badges, through Mozilla, the MacArthur foundation and the DML Badges for Lifelong Learning competition, from which SA&FS eventually received a grant. In conversations about the cluster of interrelated goals UC Davis had for the new major, badges helped advance multiple connected goals. Normoyle reflects, “It’s not just a bunch of words on a page that nobody can wrap their head around…It’s a system that you can start setting goals around, and start mapping experiences onto, and thinking about how your portfolio experiences fit.” With badges, the portfolio component of the major fit with several of these other goals, including complex social elements like encouraging student interaction (DPD Follow-up Interview).

SA&FS created formal curriculum requirements and identified seven high-level competencies. As the plan for badges developed and through preparing their proposal for the DML grant competition, the team decided each of the competencies would be represented by a badge. They intended the main badges in the system would cover the seven competencies:

  • Systems Thinking
  • Experimentation and Inquiry
  • Interpersonal Communication
  • Understanding Values
  • Strategic Management
  • Civic Engagement
  • Personal Development

The badge program is designed as complementary to the formal degree not as a replacement, a structure that allowed SA&FS to recognize diverse achievements recorded in a portfolio (DPD Initial Interview).

The badge and portfolio tool rolled out to students during the Fall 2013 term and featuring the SA&FS-developed competency badges. The project intends to add the capability for students to create their own badges in Skill, Knowledge, and Experience categories (HASTAC Q&A).

The competencies that SA&FS articulated in the beginning haven’t changed as they were enacted, and the core course offerings and design philosophy of the major remains as intended (DPD Follow-up Interview). The nature of the curriculum is a highly adaptive curriculum, that really involves the students in customizing and iterating the implementation over years of operation. Normoyle says that as they are building out its components one by one, it “has this feeling of a co-evolution” between students and faculty. “Implementation is definitely in co-design with the badge system, but the content or vision we started with hasn’t shifted too much.”

In some ways, then, the curriculum was set before the work of building badges started, but many important decisions about day-to-day activity within the system were made as various pieces were designed and implemented. This design process has elements of both styles of building badge systems identified by the DPD Project, however overall SA&FS is situated more cleanly with the collection of projects building badges, tools and curriculum together. Like the other projects in that category, SA&FS had broad goals for how the system would work and refined elements of the curriculum, tools and social practices around the badges in pursuit of those goals.

Timeline

The earliest stages of planning the SA&FS major at UC Davis occurred before project leaders knew about Mozilla’s Open badges. However, the program’s coordinator of experiential learning, Joanna Normoyle, was keeping up with the Digital Media and Learning community, and as the team was thinking about how to devise a system that would fit well with the program’s goals, she introduced the idea of using badges to tie several elements of the program together. “We had goals in the program, and they hadn’t quite come together yet in everybody’s minds… This system is one tool in the toolkit of how to address them.”

Before SA&FS had put together the final descriptions of desired competencies, Normoyle had to think about the experiential learning that would be such an important component of the major, and how to connect students to learning opportunities, connect them together in cohorts, develop social learning, peer to peer learning and mentoring in the major.

The construction of this badge system is a continuing multi-year process that will have UC Davis implementing pieces at a time as they are ready and as practices are established around the technology platform’s features. Another reason for the slow deployment is simply the extended period of time each student is expected to spend moving through the badge system. Normoyle reports, “It will take a while to get the system fully up and running, because it will take a long time for the students to go through all the experiences and ..process” (DPD Follow-up Interview).

Evolving Practices and Design Principles

What follows is a list of practices as they relate to the general and more specific design principles in each category of practice. The headings name a (a) General Principle, (b) Specific principle, (c) Specific practice. The paragraphs below each heading detail the project’s (a) intended practice, (b) enacted practice, and (c) how that practice relates to the specific and general principles.

Design Principles for Recognizing Learning

The SA&FS program developed ambitious plans for the scope of the learning to be recognized by badges, covering students’ experiences in classes and internships as well as life experiences outside of formal education. The program aimed to recognize learning achievement of potentially hard-to-assess competencies that had been identified as key outcomes of the major. On top of the portfolio-based competency recognition program, SA&FS also intended to build a social layer around the badges that would help students in the program see and respond to the experiences and achievements of their peers.

Align credential to standards > Use standards internal to community > Align badges to identified competencies

Intended, enacted, formal

The core badges SA&FS intended to make available for students consist of one for each of the seven competencies. Each is defined, but SA&FS notes that “The skills that build toward this overarching set of competencies are varied, and are somewhat self-determined by the students using the badge system” (DML Stage 2 Proposal). By self-determined, the project means that students “contribute criteria for fulfilling a given badge, and publicly display these definitions, opening their reflection to feedback from peers and mentors” alongside the evidence students claim meets the criteria (DML Stage 1 Proposal).

Joanna Normoyle also notes that UC Davis as a campus, as part of a move toward outcomes-based learning and accreditation has had all the undergraduate majors start to define “student learning outcomes to the majors.” SA&FS now has started talking about what students gain through the major as learning outcomes as well as competencies, negotiating this connection by training students to ask themselves questions like, “which of my outcomes did this experience help me build competency in” (DPD Follow-up Interview). They intend for students to be able to make arguments about their competencies or outcomes through reference to their portfolio, including the badges.

As of the beginning of 2014, the implementation of the badges component of the portfolio system is approaching completion, and the core badges. The alignment of badges to the community’s defined outcomes remains a strong element of the existing and forthcoming features. Students’ negotiation of how their experiences matches up with the competencies and outcomes will occur partly in reflection activities after completing courses, internships, and other experiences and partly in a later stage of portfolio curation, tailoring the alignment to the particular badges students will try for.

This is an example of aligning badges to internal community standards, contrasted from efforts to show how students meet wider national standards, for instance. One of the implications of this type of implementation of the “align credential to standards” principle is illustrated by the extra steps students take to describe and justify their portfolio selections against these standard definitions. These processes, which are intended to be partly available to other community members, are part of collectively defining community standards in a grassroots fashion.

Use badges to map learning trajectory > Provide routes or pathways > Use badges to map student progress

Intended, enacted, formal

One of the important functions of the badge system is to allow students to track their progress toward developing the program’s competencies, across a wide range of experiences and combinations of experiences, from classroom work to internships. The project’s original proposal to the DML competition called it “a system that guides, organizes and visualizes the many experiences, motivations, achievements and feedback loops that contribute to a skill or competency” (DML Stage 1 Proposal).

In the intended system, as students are working toward adding experiences to their portfolios, at the end of each experience, SA&FS intends for them to fill out a final evaluation of the experience on the portfolio system. They intend this to be part reflection, but more like “using a Likert scale to rate how they thought the experience went, and it gives them an opportunity to do a competency assessment across the range of those seven outcomes.” Students will ask themselves which of the outcomes their experience helped them build competency in, and they can record a few notes connecting that outcome to the context of the experience (DPD Follow-up Interview).

SA&FS intends that students will be able to “use the analytics from the system, basically the information..provided through self assessment..and be able to filter through a faceted navigation interface” and see which experiences they had previously identified as contributing toward each outcome so they can curate which experiences will show up as evidence behind the badges they try for (DPD Follow-up Interview).

An early sketch of the visual sorting tool shows one way intended for students to view their progress, with different types of experiences as different icons sorted by what competencies students said they helped them develop (See Figure 1).


Figure 1. Sketch of e-portfolio intended design showing badges sorted by competency (SA&FS Badges Project Concept Sketch)

As of early 2014, some of the portfolio system features are in place, though the screens for students to reflect on experiences immediately after completion is still in development. The final system will be an example of “use badges to map learning trajectory for the students, both in terms of forward-looking processes like goal-setting, but also in reflective and portfolio-building steps where students consider what they have accomplished.

Use badges as a means of external communication of learning > Public portfolios

Intended, not enacted, not formal

An important component of the SA&FS badge system is the portfolio that students use to display their work and how their different experiences fit together in various ways. The initial proposal stipulated that the proposed student portfolio “encourages personalization, … allowing them to display the experiences, events, achievements and moments that they believe best demonstrate their learning and expertise (DML Stage 1 Proposal).  Some of the portfolios’ audience is internal to the UC Davis community, but portfolios also allow students to share with the public, including future employers and collaborators.

As building proceeded and many features of the portfolio were simplified, the customisability of the display remained an important intent:

The goal is to always have a public facing aspect of the portfolio. The question for us is how deep that goes into the portfolio….They set the permissions on who sees what on everything in their portfolio, even to the degree of [what is visible] to their peers and to faculty members. (DPD Follow-up Interview)

Construction continues through early 2014, adding features for student portfolio-builders. The idea is to let students build layers of private workspace, feedback opportunities, local community artifacts and “some you’re ready to share out to potential employers so that you’re really in control of your identity and reputation all throughout the process” (DPD Follow-up Interview).The intent to use portfolios this way is an example of the principle “use badges as a means of external communication of learning” beyond simply providing badges themselves. The intended portfolio experience offers a rich opportunity to explore a student’s accomplishments. As the project has been suspended, this principle has not been fully carried out.”

Promote discovery > Discover learning opportunities > Peer discovery

Intended, not enacted, not formal

From the beginning, the project was clear that helping connect students with experiences where they could develop key competencies would be an integral feature of the badge system, saying in their initial proposal, “the badge system is built explicitly to help students find and represent learning experiences outside the university” (DML Stage 1 Proposal). Joanna Normoyle later described this as an intent to build the portfolio system to feature a “discoverability aspect,” so that as students are browsing their peers’ portfolios they could find badges that appeal to them and see what experiences led their peers to gain the represented competencies (DPD Follow-up Interview).

As initially implemented in winter 2013-14, the sharing and discovery features are not active, because of a strong initial focus on the student-faculty interaction, which SA&FS intends to be the first interactive layer of students portfolio-building process. While the layers associated with sharing artifacts more widely across university networks and the public are coming, building strong features specifically to enhance discoverability is dependent on future funding (DPD Follow-up Interview). Normoyle, detailing why this practice remains an important intent for the badge system, said, “I can tell students about internships, and I send them emails all the time, and they don’t really care what I have to say about it. They want to talk to their friends about what they’ve done that’s really cool.” They get this information in person as is, but “they don’t really have the ability to have those conversations all the time,” so putting this on the Web 2.0 style portfolio system they intend could put students “in the driver’s seat” in discovering opportunities (DPD Follow-up Interview).

Promote discovery > Discover learners > Help colleagues at other institutions discover learners

Intended, not enacted, not formal

While there are no existing plans to implement the practice, SA&FS has considered how the system might function if it is adopted more widely than the UC Davis community, spreading to other institutions. The focus is on developing the internal system, but the program indicates they have thought about how exporting the model could help build an ecosystem of discovering learners, saying

we also envision working within the national network of similar institutions (INFAS) many of which support majors like SA&FS. We’d like to see if we can not only spread the adoption to other schools and programs like ours, but interconnect them so that students and faculty can share and provide a community of support around portfolio building and badge earning across institutions. We imagine, students in our programs going on to graduate work within that network, or perhaps taking key positions in the regions where these schools are located. (HASTAC Q&A)

According to Joanna Normoyle, this follows a traditional networking path for undergraduates, who get connections to professors interested in taking them on at the graduate level through the professors they knew at their first school. As part of exploring how badging might spread to similar programs across the country, SA&FS will keep this idea on file. A practice implemented along these lines would be an example of building a system to facilitate the discovery of learners. As the badge system has been suspended, this principle has not been fully carried out.

Award formal academic credit for badges > Award college credit for completing curricular units and activities

Intended, not enacted, not formal

The project planned to design a badge system as part of their interdisciplinary major in Sustainable Agriculture and Food Systems. As students complete activities in the college curriculum, they would by extension receive formal credit for their engagement with the subject. However, because the badge system did not fully take off, the principle has not been carried into practice.

Design Principles for Assessing Learning in Digital Badge Systems

The Sustainable Agriculture & Food Systems program aims to recognize learning through student portfolios containing badges and evidence to support them. Within each course, internship, or educational experience, instructors are responsible for assessing work to their own standards. Students then submit work to their portfolios, identify examples corresponding to their desired badges, and this work is judged by their mentors as to whether or not it fulfills the criteria of each specific badge.

Involve students at a granular level > Learning pathways and badge design > Have students design and apply for their own badges.

Intended, not enacted, not formal

The seven main competencies form the core of the the SA&FS badge system, but SA&FS also intended for students to be able to create their own badges to complement this set. The program aimed to allow students to select portfolio pieces that match the criteria of a badge, whether that badge represents one of the core program competencies or is self-designed. SA&FS intended for students to be the first assessors of their own work, followed by program experts who validate their decision afterward. Below, some early design sketches show a method for students to define badges for Skills, Knowledge or Experience.


Figure 2. Badge builder shows how SA&FS intended for students to be able to define and earn badges of their own design (SA&FS Badge Project Concept Sketch).

As planned, upon rollout, learners may categorize, name, and define criteria for their own badges, customizing them to their needs and experience (DPD Interview). In an interview, Joanna Normoyle stressed that this is the way the UC Davis Agriculture Sustainability Institute runs the offline components of the program, and “it would be disingenuous to move into this space in any other way.” This fits with an active learning disposition and the way humans are “predisposed to discover things in the world,” as opposed to a passive approach that Normoyle feels may be an effect of the design of many online learning environments (DPD Follow-up Interview). Through allowing students to design their own credentials and to perform their own self-assessment as the first step to earning a badge, SA&FS is trying to create an environment where “we instill leadership, engagement, and drive in our students” (DPD Follow-up Interview).

As the badges are just beginning to be implemented, as of January 2014, and the student-designed badges feature is not the first type of badge to go live, the technical and social practices around student design processes have yet to be fleshed out. The principle remains important to the program, and will likely become part of SA&FS’s formal practices when the program is fully implemented.

Use leveled badge systems > Competency Levels > Assessing levels of achievement

Intended, not enacted, not formal

SA&FS intended for assessments to get more involved as students progress through the program and develop their badged competencies. As students develop their skills, knowledge, and competencies, they indicate their level of understanding from novice to expert, with 3 intermediate gradations. In this plan, self-evaluation would become the first level of assessment. As they move forward, SA&FS envisioned that faculty, then peers would join the assessment process so that when students assert expert level, peers and mentors can provide summative assessment to validate that achievement.


Figure 3. Sketch of intended design for student’s level selection. (SA&FS Badge Project Concept Sketch)

Reinforcing the idea SA&FS intended rules that would build assessment requirements, like, “A user can only rate themselves ‘expert’ once they have received at least 5 reviews in the competency” (SA&FS Badges Design Spreadsheet).

As the badge system begins operation in early 2014, the levels are not yet a feature of the portfolio tool. SA&FS indicates that more information about the leveled assessments function will be available after students have been earning and peer-assessing badges for several terms (DPD Follow-up Interview). Joanna Normoyle indicates that the need to fully think through these ideas before implementing them, saying “This concept for us around competency development has been challenging to convey, and it was one of the the really challenging conversation with the original tech partners we were working with, because they just could not get that there was no endpoint. You’ve never really arrived; you’re never really done with systems thinking.” Even as the project has moved onto a web developer who understands their goals more completely, Normoyle said, “I still don’t think we’ve quite solved the question…We don’t want students to have a sense that they’re trying to get somewhere” (DPD Follow-up Interview).

As the program develops, SA&FS will implement some display of development in a competency and attempt to negotiate the challenges of assessing and recognizing levels of competencies. This approach is important for its clarity of distinction from binary models of competency recognition. It represents an implementation of “competency levels” that rejects hard boundaries and finish lines, where the richness of the information in the portfolios and the variety of intended assessments will provide an example of how digital badges present an opportunity to do competency based education differently.

Align assessment activities to standards: create measurable learning objectives > Internal standards > Assessing work with competencies in mind

Intended, enacted, formal

SA&FS intended to recognize learning around community standards: the major’s identified competencies and the university’s push toward defined student outcomes. This recognition focus implies matching assessments. As of early 2014, SA&FS has built some rubrics, but as explored next under “enhance validity with expert judgment,” the program prefers a more personal approach involving dialogue about portfolio objects and experiences when possible.

SA&FS is implementing assessments targeted toward its community standards each time new assessments are needed, and will continue to build out the badge system’s assessment set with this strategy. As much of the work students will put in their portfolios is already part of college experience, there are assessments already involved in the process, that are necessary even without considering badges. Some of the badge system’s work will involve capturing these existing data streams, where existing assessments are already serving the university’s goals (J. Normoyle, Personal Communication, 26 February 2014).

Enhance validity with expert judgment > Use human experts > Promotion of holistic assessment with advisor support

Intended, not enacted, not formal

Through validated self-assessment of e-portfolios, SA&FS hopes to promote a holistic assessment program. Students are all assigned an advisor in the program who helps provide formative feedback that helps students develop how they present material to be judged against criteria for particular badges (summative assessment).

Students’ knowledge, skills, and experience come from a variety of classroom and outside work, such as internships. Classroom work assessment is well-developed. SA&FS will see how the opportunity to award badges affects assessments of student performance and learning through internships. The spreadsheet of badge types SA&FS developed shows various badge types issuable for this outside work, like a badge endorsing a student’s skill, as assessed by an outside specialist, or an internship badge that is attached to a particular competency, skill, or issue area within the e-portfolio system.

SA&FS has not yet implemented the competency assessments, but Normoyle stresses that it will follow their existing pattern of students having “back-and-forth mentoring conversations with me and the faculty all throughout their experience in the major” in a recurring process of dialogue over competency development where they propose ideas and refine what they do (DPD Follow-up Interview). When students are assessed on activities that align to competencies in some of the core major courses, there is a competency rubric involved, but Normoyle feels the progress toward overall assessment will come through the one-on-one dialogue processes. She notes that in speaking with the faculty, it seems like faculty still like just sitting down with students and talking with them even when online tools that capture and display information are available. As she periodically waved to students passing by her office on the student farm, Normoyle described how SA&FS would still need to have an incredibly strong offline community, which is complemented by the online tools. Though not yet fully implemented, the assessment practices will incorporate the feedback of students’ subject-specialist professors.

Use e-portfolios > Local to community & Foster discussion around artifacts > E-portfolios

Intended, enacted, formal (Local to community)

Intended, not enacted, not formal (Foster discussion around artifacts)

One of the key features of the badge system is that students focus on the development of a portfolio of content, skills and experiences. SA&FS intended to organize portfolio items around the badge system but enable sorting around other factors as well. See the Nov 2011 SA&FS presentation for a wealth of sketches of how portfolios are intended to be organized and viewed and Figure 1 above for a sketch of the intended competency facet portfolio view. Assessment practices are designed around this centrality of the portfolio. SA&FS advisors intend to provide formative assessment as portfolios develop, followed by peer and faculty reviews.

As enacted, the e-portfolio system has been developed to cooperate with the UC Davis course management platform, which is closely tied to students’ assessed coursework. (DPD Initial Interview). The portfolio system, though not complete as of the beginning of 2014, is designed to give students control over who and when they display information, because in addition to display space, the portfolio is intended to be a workspace for students to collect and reflect upon artifacts representing coursework, internships, jobs, and other experiences.

In the prototype shown below (Figure 4) of an example portfolio piece on an experience, the system allows students to upload a variety of evidence types in a private workspace mode, and then seek feedback from various parties when ready.

Figure 4. Enacted Portfolio development system prototype screenshot.

In the implementation of the portfolio system, the concept of it being “local to community” has been implemented. Even though the portfolio will eventually give students the ability to display their learning to the public or to particular members of the public like prospective employers, there is a stronger focus to date on using the system to support learning while students are in school by enabling conversations with peers and faculty. The local sharing elements are implemented, but much of the conversation components are not yet built, including soliciting reviews from peers. However, the DPD Project feels these “foster discussion around artifacts” elements of the badge system are important enough to SA&FS to already consider them a component of the project’s continuing formal practices because of the number of times the project has mentioned the conversations between students, their peers and faculty as an important part of the system’s function. SA&FS remains dedicated to fully implementing the e-portfolio principles around local community sharing and discussion of artifacts and competencies.

Use formative functions of assessment > Provide a combination of peer and expert feedback > Providing feedback

Intended, not enacted, not formal

SA&FS intended for students to be collect feedback, through the portfolio system from faculty and from their peers: From their proposal, SA&FS described how:

As they develop, learners receive formative feedback from peers, faculty, and community partners with whom they have worked. When a learner has reached what he or she believes is ‘expert’ level in a particular ‘competency,’ members of the community can agree or disagree with the expert designation and provide targeted feedback. Viewers see how many reviews a learner has received, but not all of the review contents. By making the content – not just completion – of learning experiences evident, this integration of self, peer-, and practitioner-assessment can guide future learning as learners recognize gaps and strengths.”

The intended feedback components are not yet active in the implemented system, as of January 2014, but they are continuing through refinement and implementation. Figure 4 above shows a prototype view with some reviews in the implemented system, while Figure 5 below shows the initial design for collecting reviews. The plan for peer assessment is for students to identify competency badges or create the badges they think they qualify for, upload evidence,  and then send in a request. Peers and program experts then validate or deny that request. Faculty can provide comments and start a dialogue with the student, fostering a mentor/mentee relationship (DPD Interview).


Figure 5. Reviews, from design sketches

As this practice inched towards implementation, SA&FS found that peer assessment required more thinking, especially to ensure that the right balance of many factors and assessment techniques is struck, but also to ensure that they “make it something that people want to do” (DPD Follow-up Interview). Joanna Normoyle said a bigger conversation is needed around peer assessment with a better design research process to sort it out: “This is incredibly difficult and messy, and it’s the piece that has to work in order for everything else to work…It’s in getting this kind of feedback that makes everything else in the system feel valuable” (DPD Follow-up Interview). What is clear so far is that peer and expert assessment will serve formative functions, occurring in the middle of students’ learning and portfolio-building process. These assessments will provide points for students to reflect back on and information that will help them choose further learning activities and finalize portfolio elements. When later, students ask themselves whether they’re ready to earn this, they look back at the data points they’ve gotten already to decide if they’re ready to earn the badge (DPD Follow-up Interview).

Promote hard and soft skills > Combination of collaborative learning and discrete skills > Assessing soft skills

Intended, enacted, foraml

The badge system, centered around the program competencies, exists in large part to promote skill development across areas that are not typically recognized by formal higher education credentials, possibly because they are difficult to assess. The competencies, (a) Systems Thinking, (b) Experimentation and Inquiry, (c) Interpersonal Communication, (d) Understanding Values, (e) Strategic Management, (f) Civic Engagement, and (g) Personal Development, are highly desired by schools and employers. In broad strokes, they fit in the realm of “soft skills.”

As enacted, SA&FS’s assessment regime is incomplete, but students can begin building portfolios toward these competencies. This practice represents SA&FS’s commitment to promoting hard and soft skills, and it is an important continuing part of the badge system.

Design Principles for Motivating Learning

User-driven learning and choice is woven throughout the SA&FS undergraduate major and badge project. The badge system intends for students to design and define new badges or select pre-designed badges, as opposed to only pursuing badges that an instructor or supervisor has chosen. The initiative presumes that students are more likely to be motivated to participate in a system that they helped create, leading to more user buy-in.

The badge system rolls out for students over several years beginning in Fall 2013. The intent for the portfolio and achievement system is to motivate students to both think holistically and specialize in their learning, making connections between experiences and competencies. More information will be available as students use the system over the coming terms. The program’s coordinator of experiential learning spoke about motivation, saying “the badges are a way to help them to understand, set goals, get motivated, articulate new ideas. It’s a way of framing and developing [their competencies].”

Utilize different types of assessment > Peer > Peer assessment

Intended, not enacted, not formal

SA&FS intends for students to receive formative feedback as they develop, but when they get to the point of claiming an “expert” level badge in a competency area, the program plans for members of the community to be able to chime in to agree or disagree with the assertion (SA&FS Concept Sketches Presentation). Aggregate figures of endorsement are to be available to portfolio viewers, adding to badge credibility. Students may be motivated to aim for this level of recognition but also may only wish to submit to summative judgment when they truly feel qualified, hopefully motivating them to perform and submit high-quality work.

After badges accumulate in student portfolios, SA&FS will be able to investigate the motivational effect peer assessment has on learning within the major, particularly as students discuss the progress of their portfolios with advisors (DPD Interview).

Speaking for SA&FS, the program’s coordinator of experiential learning, Joanna Normoyle recognized the need for formal research into how students are motivated by the system to back up design decisions around motivation, but said that the project is operating on a strong belief that, “validation matters: seeing and being seen, and it matters differently to different people, but I think fundamentally that when people feel that they’re being recognized as effective in what they’re doing by their community, there are very few things that match that for motivations as human beings…Whatever we can do to provide experiences for our students that tap into that deeper motivation, giving them recognition for being effective, we’ve got to figure out how to do it” (DPD Follow-up Interview). She shared her experience on the student farm, which is a place where “people come and they actually get to do things and demonstrate ability…in front of all their friends…and it’s amazing the sort of changes in people we see even over a quarter or two because of that” (DPD Follow-up Interview).

Peer assessment is not yet enacted in the system but it forms an important underpinning of the motivational local theories of the project and will likely be implemented fully in the future (DPD Follow-up Interview). Solidifying this principle as core to SA&FS’s motivation thinking, Normoyle said “the individual experience of portfolio building and badging is pretty well built and implemented, but we don’t yet have students interacting with each other around the badges they’ve earned, [and]…that’s the critical piece to the motivation, more than the actual badges” (Assessment Strategies for Effective Badge Systems). The designers feel the impact of interacting with their community around their achievements is an essential component of the motivation practices.

Utilize different types of assessment > Self > Self-assessment

Intended, enacted, formal

The project intended to allow students to assert their own level of experience in each competency. Choosing when to self-assess at an “expert” level may motivate students to set and accomplish goals, because of the higher stakes of the community’s judgment that is allowed by that designation.

Normoyle feels that motivation derived from self-assessment is mixed, because “some students are really oriented around reflection. Some start out not really liking it at all, and then they become very reflective people and find it to be a powerful dynamic in their lives” but “some hate it and think it sucks, and it’s super demotivating” (DPD Follow-up Interview). Reflective elements of the badge system are not fully implemented as of January 2014. Participating students have only done one formal reflection, though there is a self-reflective element to all of the participating students’ activity in the portfolio system.

As the team’s thinking evolves, they feel that “there has to be some degree of opt-in” and decisions about how much and how often users must perform formal self-reflective processes, that self-reflection can be motivating if students are in control of the process (DPD Follow-up Interview). “If you create resistance…they’re maybe not going to participate in another way that might be their way in, that might end up getting them into some of the reflection (DPD Follow-up Interview). Because of this, integration of the formal self-reflective elements of the badge system are moving ahead carefully.

Set goals > User-created badges > Self-selected and self-created badges

Intended, enacted, formal

SA&FS hopes that the ability to select and create badges will motivate students to think holistically about the education they gain and how their portfolio could create a cohesive picture of a skilled graduate.

SA&FS intended for the first phase of a student’s interaction with the system to be involved with collecting and tagging experiences, and a second phase where students review their material and curate selections to apply for badges. They will be able to select which options from the seven main competency badges they would like to apply for, in order to supplement their official degree (DPD Follow-up Interview). Besides the competency badges, SA&FS intends for students to have freedom to create custom badges to pursue. In addition, they generated a number of category-level badge definitions of “skill badges”, ‘knowledge badges”, experience badges” and “honor badges” that represent the most important skills, content, and experiences for students in the major to develop or be exposed to, as determined from a survey of stakeholders in the field (DML Stage 1 Proposal).

These badge definitions (skill, knowledge, and experience) serve as templates for students to create their own specific badges, if they like, selecting its name, criteria, and design. Students would then point to evidence of their achievement from their portfolio (DML Proposal).

Students have a private work-in-progress section to collect the badges they wish to earn as well as the section of their portfolio showing finished accomplishments (DPD Interview).

Joanna Normoyle said, “I think it’s going to take time for them to even figure out what they would want to make badgeable,” as students fill out portfolios and slowly explore the features of the portfolio and badge system (DPD Follow-up Interview).

SA&FS hopes that the ability to select and create badges will motivate students to think holistically about the education they gain and how their portfolio could create a cohesive picture of a skilled graduate.

Before the ability to create their own badges to become motivational, Normoyle thinks that the system needs to grow to the point where students would feel they have something to say, like

“we want to carve out what we think is important in what we’re learning. We want to be co-creators of what this curriculum is and share in that articulation.” They’re just not to the point where they understand what that possibility means. But once they do and they can understand what that authorship looks like, and what it looks like to be recognized for being the designer of a badge, [this could change]. (DPD Follow-up Interview)

SA&FS remains committed to the idea that students could create badges that others could earn, but the implementation of this practice depends on many other implementations first to build student understanding of the meaning and value of the system (DPD Follow-up Interview).

Provide outside value for badges > Real-life application of knowledge > Badges represent skills useful in students’ future careers

Intended, enacted, formal

SA&FS chose the competency badges based on what was known about the needs of potential employers in the changing agriculture world. This presents the possibility that badges may be motivating for students because they stand for things that might have more value in their chosen fields than elements of traditional degrees. Normoyle indicated that this is only just beginning to be motivational for students:

“The new students coming are starting to know about the badges, because when they search for the program, they find stuff about badges…They ask about them and say ‘we’re like the only program in the world that gets badges. That’s really cool; when do we get badges?”

A lot of the students coming into the program haven’t built a resume, and some of them haven’t had a job before entering the SA&FS program, so they don’t. Some students are “just grasping what this might be about for them, trying to translate it into what they’ve heard before about what could be useful for them. It’s starting to get really exciting for them” (DPD Follow-up Interview). SA&FS will continue to build student understanding about how they can display and use the badges as the system continues to develop.

Display badges to the public > Learners can choose to share their badges with others > Choice of publicly displayed information

Intended, enacted, formal

The program intended to allow students to show or hide the badges they earn and those they are working towards, which should encourage them to think about how their accomplishments fit together. SA&FS intended for program staff and faculty advisors to be able to see students’ badges in progress as well as completed badges, though public viewers will only see publicly published content. Students will be able to solicit feedback from these experts on this information as they work (DPD Interview).

Normoyle says that giving the students choice about what they share is the way the portfolio is designed:

“It’s the students’ information; they need to be in charge of their information; there isn’t really any other way I would feel comfortable doing it…When it comes to motivation, I want them to share about what they’re doing. Some of them might default to keeping things more private…if they weren’t really pressed [to share]” (DPD Follow-up Interview)

SA&FS has implemented offline, portfolio presentations in person: “At the end of a course, students have to share their portfolios for peer review and a faculty panel, and an “online charette process throughout the quarter”  These are processes that try to get people comfortable to opening up for critique, for constructive support, “because it’s a really important process for people to realize it’s constructive and supportive and not like a tear-down. The way we do that is making it a facilitated process” (DPD Follow-up Interview). In this regard, the program prepares students for the feedback they could receive from public portfolio pieces.

Build outside value for badges > Evidence for outside opportunities > Evidence for potential collaborators

Intended, enacted, formal

SA&FS intended to create a resource that can help open opportunities for students in the future by showing potential collaborators that they would be valuable partners (DML Stage 1 Proposal). Organizers hoped students would recognize this possibility and are motivated to use the portfolio tool to showcase their skills.

Badges could be a sign to those who might collaborate with a student about their fitness for the work. The opening of future opportunities with the audience of the badge system will motivate them to carefully develop their portfolios.

SA&FS has also talked about the system helping students achieve further educational opportunities across institutions, but as a possible example here, Normoyle pointed to the senior capstone project, saying that as:

“part of the process, when you’re thinking about forming teams, you can get to know things about people’s strengths and the experiences that people have had, and if you have access to a system where you can understand the history of other people’s experiences, that helps inform and helps you understand [whether they would be] a person who they would like to work with” (DPD Follow-up Interview)

Normoyle believes that being able to present that about oneself or know that somebody else could be a very useful thing.

“We’re moving into a place where… we’re not just getting hired for jobs anymore. We’re also collaborating with people on building projects and getting funding and doing startups. This is a much more entrepreneurial environment, the 21st Century workplace. It isn’t just about handing your resume to one person…[It’s about] having the pieces and being able to present them, and that’s effectively the mechanism that we’re trying to put in place here, and give the students the tools to do that. ” (DPD Follow-up Interview).

She notes that the program’s graduates might not even go into sustainable agriculture. What the program is trying to develop in students is the capacity to do well in this field or what they need to do well in other modern fields.

Provide incentives > Internships > Offer the possibility of internships

Intended, enacted, formal

SA&FS planned to link their badge system to internships, strengthening the motivation of students and sustaining their interest. In their DML Stage One Proposal, they described, “We hope to build a platform that can connect classroom learning and program structures to internships, student-led projects, fieldwork and lab courses, and self-assessment” (DML Stage One Proposal). Additionally, the internship was built into the formal curriculum of the university. In their HASTAC Q&A, SA&FS explained, “The criteria consist of having completed the core courses of the major, including the senior capstone, having complete the internship requirement of 12 units, and then, depending on the competency, a mixture of other types of badges or work from the student’s portfolio” (HASTAC Q&A). Although SA&FS halted development on their badge system, this practice has been formally carried out, given that the internship requirement is a formal component of the university curriculum.

Set goals > User-determined learning trajectory > Students involved in the process of charting their own learning

Intended, enacted, formal

The initiative intended to design a badge structure in which learners can determine their own trajectory of learning. The DPD Project inferred that this has positive motivational effects on learner engagement. SA&FS explained in their HASTAC Q&A that their “main goal is to support a learning process that is student-driven and outcomes-based, in a way that helps faculty and other mentors and advisers interact effectively with students and create a learning community that can foster and dynamic and networked experiences for everyone in the system” (HASTAC Q&A). To this end, the project set out to create a dynamic, student-determined learning experience, providing them with autonomy to set their own goals within the badge system. Although the badge system has been suspended, this practice has been carried out to an extent in the degree program of the university.

Design Principles for Studying Learning

Besides some initial research aimed at identifying competencies to issue badges around, SA&FS had no formal plans intended from the beginning to study the system.

Improve badge impact > Seeking practice improvement

SA&FS staff think that there are many moments where more data or better research, especially in the realm of design research could be useful to their efforts, but did not intend any specific formal practices in their initial design proposal. The project did submit a separate research grant, which was not funded.

Colin Dixon, a graduate student in the UC Davis School of Education, is dedicated to doing some kind of research in the system. Time and resource constraints have gotten in the way of designing or running a formal research practice of any kind thus far.

There are many questions that the team has about their practices that they are unable to spend time specifically researching. The biggest one for Joanna Normoyle relates to assessment: what is the right mix of what peers, faculty and collaborators like intern hosts have to say about the student, and how does this vary depending on variables like the “scale” of the badge? When it comes to actually demonstrating evidence for the work that you’ve done, the stakes are higher. “If you’re going toward competency based education, who wants to be the person who can really vouch for that?” Figuring out the right mixture of these different data sources to meet the high bar set by the competency claims would be the research question most important (DPD Follow-up Interview). In turn, achieving this mixture, and considering research into finding the right balance depends on the fully functioning system that provides all the data points that can feed those different streams.

Recognizing Principles Assessing Principles Motivating Principles Studying Principles
Specific principles: Specific principles: Specific principles:
About

Loves open education, Open Badges, free culture, Progress of the Useful Arts and Sciences, people-powered politics, and local food production. Coordinator for the badges Design Principles Documentation Project at Indiana University.

3 comments on “UC Davis Sustainable Agriculture & Food Systems (SA&FS)
  1. Merija (Mary) Jirgensons says:

    What happened to the “UC Davis Sustainable Agriculture & Food Systems (SA&FS)” badging system in 2017. I checked the UC U Davis catalog, and it appears the badging program was disbanded. What happened? What were the results? What did you learn from the experience.

    I can find no follow-up. The program was initiated with so much fan-fare. So, I am left wondering what happened????

    • Nate Otto says:

      There have been a good handful of projects studied here that followed the pattern of SA&FS. The vision was strong and ambitious but based on the efforts of only one or two people within an organization. Additionally, the program encountered significant difficulties developing the software platform to perform the portfolio and badging functions and had to make a hard pivot from one contractor to another. The combination of very ambitious recognition goals that required labor-intensive new assessment practices, staff change among the program’s main supporters, and higher-than-expected costs to develop the technology, are the factors that I think led to this program never getting off the ground. -Nate

      • Merija (Mary) Jirgensons says:

        There have been a good handful of projects studied here that followed the pattern of SA&FS. The vision was strong and ambitious but based on the efforts of only one or two people within an organization. Additionally, the program encountered significant difficulties developing the software platform to perform the portfolio and badging functions and had to make a hard pivot from one contractor to another. The combination of very ambitious recognition goals that required labor-intensive new assessment practices, staff change among the program’s main supporters, and higher-than-expected costs to develop the technology, are the factors that I think led to this program never getting off the ground. -Nate

        Reply and comment
        In response to your response

        Thank you for your reply. I will pass the information on to my organization. Your experience is worth keeping in mind.

        I want to ask you: do you think using badges in a university is worthwhile? What do think are important factors to keep in mind when undertaking badging? Just keep in small? Are there learning aspects that badges do well? Or do you think badging is just a fad?

        I appreciate your sharing your experience.

        Mary

1 Pings/Trackbacks for "UC Davis Sustainable Agriculture & Food Systems (SA&FS)"
  1. […] One of the earliest programs to start thinking about digital badges in higher ed is UC Davis’ Sustainable Agriculture & Food Systems (SA&FS) undergraduate program. This case shows how badges can complement an existing credential structure by […]

Leave a Reply

Your email address will not be published. Required fields are marked *

*

Mozilla Open Badges
Digital Badges are web-enabled credentials of learning or accomplishment. -Erin Knight, director of the Badge Alliance
Badges contain detailed claims about learning, links to actual evidence of learning, and they're shareable over the web. -Dan Hickey, DPD Project Lead Investigator
To me, digital badges represent the bridge between formal learning & informal structures. -Alex Halavais, DML research competition winner
Open Badges can help people tell a verifiable story about their accomplishments. -Nate Otto, DPD Project coordinator
Regardless of where you start, it’s more than likely you’ll end up somewhere other than your intended destination. That’s okay. Systems are living things, and your badge system needs to be flexible. You must embrace a bit of chaos in its design. -Carla Casilli, Director of Design + Practice at the Badge Alliance