The information contained in this project appendix was gathered from the original project proposal that was funded in 2012 and interviews with project leaders through the end of 2014. This appendix and the overall design principles database from the DPD project does not reflect further evolution of the project or developments after the final interview that took place in 2014. As of the final interview, Planet Stewards had developed a functional badge system and issued badges. Based on this information, we have classifed the Planet Stewards badge system as an implemented (rather than a partial or suspended) badge system.
Planet Stewards is a badge-driven learning program that guides high school science students through exploration of careers in five categories of Earth sciences. It is a partnership effort by the National Oceanic and Atmospheric Administration (NOAA) and Boise State University’s 3D GameLab, an online learning platform designed around quest-based learning. 3D GameLab developers, headed by Lisa Dawley of GoGo Labs and Chris Haskell of Boise State, partnered with NOAA to offer badges based on NOAA’s educational content. Planet Stewards won a grant through the MacArthur Foundation’s Digital Media and Learning (DML) competition to build a badge system around planet science career pathways. The project involved not only developing the badges, but also the quest-based curriculum around NOAA’s educational content. The program also considered the developing Next Generation Science Standards and the published guiding principles that form the basis of those standards as they decided what science and engineering practices would be emphasized in the career quests.
The team from 3D GameLab worked closely with subject matter experts working in NOAA’s various planet science programs from climate studies to oceanography. The placement of the project within a “career pathways” group of DML winners guided the team to organize the badge system around possible careers for students to explore, so that students could earn a badge for exploring the work dones in each of 15 planet science careers. For each career badge, Planet Stewards defines a set of quests and activities that help students understand the type of work performed in each of the system’s science disciplines.
On the 3D GameLab website, students see their options for quests to attempt, with descriptions, requirements, standards alignment tags, and user ratings. Completing quests unlock further quests, and students earn badges after completing each career’s map of quests. Though many of the learning experiences associated with the badges are online, the program is offered by participating teachers working with their own students. 3D GameLab designed their online platform so that teachers can adapt the basic quest-based curriculum to the needs of their particular classroom and can require students to complete all sorts of assignments that the teacher can assess. The 3D GameLab platform allows teachers to select and organize quests from an available collection and offer them to their students.
Because teachers are an integral part of the program’s operation, 3D GameLab offers training and certification on behalf of NOAA to ensure that these teachers’ students get a good experience and that the badges they earn maintain NOAA’s high standards. After completing a training session and earning their own NOAA certification badge, teachers gain the privilege of cloning the Planet Stewards curriculum into their own classroom quest map and customizing it if necessary. As their students progress through the system, teachers assess their completion of quest assignments and award badges by certifying their completion of each career quest map.
The Planet Stewards project is a collaboration between NOAA’s education effort and 3D GameLab, both initiatives with established infrastructure prior to the project. NOAA made a proposal to the Digital Media and Learning (DML) competition for leveraging its existing education content to recognize high school science students for increasing their scientific literacy in topics covered by the agency. In phase two of the DML competition, NOAA partnered with 3D GameLab, and revised their proposal around quest-based learning, envisioning a focus on rewarding students for progressing toward science standards. The team was placed in a group of projects pursuing career pathways and found a new focus for the Planet Stewards badge system around career exploration.
In the investigation of this initiative, the Design Principles Documentation Project analyzed the proposals and related materials outlining project plans, characterizing practices they intended to use in the badge system designs for recognizing, assessing, motivating, and studying learning. Next, the DPD Project grouped these practices and those identified in other projects and named general principles under which they clustered. In November 2012, the DPD research team interviewed Lisa Dawley, co-creator of 3D GameLab and Peg Steffen, education coordinator in NOAA’s National Ocean Service in order to identify the practices that had begun to be enacted in the badge system.
System development proceeded, and Planet Stewards held its first teacher training session in Spring 2013, following up with bigger sessions in the summer and fall. After these trainings, teachers could begin implementing Planet Stewards in their classrooms however it best fit in their curriculum. The first group of teachers started offering the program throughout the fall and winter of 2013-14. In October 2013, the DPD Project interviewed Lisa Dawley again to characterize how practices had evolved and determine the formal continuing practices of the badge system.
Evolving Practices and Design Principles
What follows is a list of practices as they relate to the general and more specific design principles in each category of practice. The headings name a (a) General Principle, (b) Specific principle, (c) Specific practice. The paragraphs below each heading detail the project’s (a) intended practice, (b) enacted practice, and (c) how that practice relates to the specific and general principles.
Design Principles for Recognizing Learning
Planet Stewards runs on 3D GameLab’s quest-based learning platform, and is accordingly organized around quests, organized along learning pathways that represent career roles and study areas filled by working NOAA scientists. Planet Stewards offers badges for exploration of careers in five areas, corresponding to major components of NOAA’s mission: Climate Science, Freshwater, Marine Life, Oceans and Coasts, and Weather. Students progress through these five tracks of three career badge quests with some degree of choice over which careers they will pursue. In completing the quests needed for each badge, students practice the skills used by scientists in each of these fields and those skills identified by applicable standards covering high school science education.
Contextual Factor > Coinciding badge and curriculum development
NOAA and 3D GameLab partnered to apply for the DML competition grant. Prior to receiving the grant, NOAA had some educational materials covering the areas targeted by Planet Stewards, but the badge project included a lot of work to turn this existing material into quests on 3D GameLab’s platform.
The idea of fifteen badges in five “areas”, with three badges in each area was sketched out in the original proposal, but the team had to then define them in terms of career pathways. This required 3D GameLab to involve NOAA subject area experts to find out what they thought were the relevant career groupings and pathways. Lisa Dawley explains, “The badge was the frame, and then we started drilling down on that, for what content needed to be covered” (DPD Initial Interview). NOAA’s educational director Peg Steffen detailed this process:
We used that career pathways framework, and then we took a look at what kinds of science careers NOAA is good at. We used existing materials, videos, and activities, to weave them into a matrix highlighting career pathways that students might take an interest in pursuing…We also wanted to make sure that this also was relevant to the content that was being covered in the classroom, and since NOAA is a science agency primarily, and they do a lot of applied science so there were lots of opportunities to weave applied science with the careers with the existing materials into the 3d gamelab platform. (DPD Initial Interview)
The DPD project identifies this process as coinciding badge and curriculum development despite the prior existence of some educational content. Molding that content into the 3D GameLab quest system was involved and required significant reconfiguration of material to align to the badges named after careers.
Sheryl Grant’s “buckets” describing stages of development badge projects start in or pass through may illuminate the differences between Planet Stewards’ approach and other projects that coincided badge and curriculum development. Planet Stewards did not start from scratch, because NOAA brought years worth of material developed for disparate education efforts, and the 3D GameLab system already had almost all of the features required to issue badges. Grant would characterize this starting point as a “layered build,” referring to the layering of the badge system upon existing content and technology stack. Either of those two variables may differ between projects.
In contrast, a process of building badges to match previously developed educational content, badges would be named and aligned based on the structure of the existing content, which in this case was not previously tightly aligned to careers of NOAA scientists. See Figure 1 for an example of the three careers selected for the Freshwater career category.
Upon reflection after initial implementation, Lisa Dawley describes the team’s approach as an instructional design process not just a badge design process. Badges represent the educational objectives that you normally start with in an instructional design process. Outcomes in this system align to the badges. She emphasizes that the element of alignment to career pathways took on a stronger role as development progressed from initial ideas to the first design (DPD Follow-up Interview).
Align badges to standards > Use national or international standards > Alignment to National Science Education standards and Next Generation Science standards
Intended, enacted, formal
Before shifting their focus to career pathways upon being assigned into the career pathways DML competition group, the Planet Stewards team intended primarily to target scientific skills students could develop by interacting with NOAA educational material (DML Stage 1 Proposal). Specifically, they intended to recognize skills identified as important for high schoolers by the Next Generation Science Standards and include recognition of these skills where relevant within the career pathways structure. By the time of the second Planet Stewards proposal to the DML competition, the team described their intent to cover “Skills and knowledge… drawn directly from the National Science Education Standards for 9-12 grades. We are also able to adapt content to the Next Generation Science Standards that will be released later this year. Emphasis will be placed on science as inquiry, in which students learn skills such as observation, inference, and experimentation” (DML Stage 2 Proposal). After the assignment into the career pathways group and the 3D GameLab team’s first meeting at NOAA, Planet Stewards solidified on a focus on career badges, with related skills embedded within careers (DPD Initial Interview).
As the team began implementing the system, the NGS standards were not finalized. However, standard development was based on a previously published framework and guiding principles, which named the skills to be included in the standard. Planet Stewards used the guiding principles, which identify science and engineering practices, core concepts and ideas. Upon initial enactment of the badge program, Peg Steffen felt this approach of focusing on the “framework” rather than waiting for the finished standard made more sense for a project like Planet Stewards, because the forthcoming standards will contain performance expectations, which would be tougher to fit content to. In contrast, Peg thought that it would be easy to tie the activities to the named concepts in general, partly because they had been solidified for a while prior to the development of the specific standard (DPD Initial Interview).
As the team planned the career pathway badges, they kept their eye on the framework, “pulling the major pieces from the framework that they wanted to make sure they covered in different badges (DPD Initial Interview). As part of the process, the team contracted with EdGate Correlation Services, whose employees analyzed the badge system and tagged each badge with the elements of the framework. As standards are updated, EdGate plans to help Planet Stewards keep 3D GameLab updated and properly tagged (DPD Initial Interview).
While it is no longer the primary focus of the badge system, the continuing practices represent a clear case of using the principle “align credential to standards.”
Recognize diverse learning> Badges for Knowledge and Skills
Intended, enacted, formal
Planet Stewards intended to help students learn about the skills practiced by scientists working in the various disciplines represented by the career badges. Embedded in the activities corresponding to each badge are tasks that let students to exercise these skills. Planet Stewards planned to recognize these skills primarily through the career badges. Lisa Dawley said, the whole idea of the science career badges “is that the skills are woven in with the content” (DPD Initial Interview).
However, the 3D GameLab system has the capability to award credentials that are not OBI-compliant digital badges as well, as it has done in its other quest-based content programs. These are called “achievements” and are often automated indicators of progress, based on factors like total number of quests completed (DML Stage 2 Proposal). Though not explicitly described in their initial proposals, Planet Stewards enacted several achievements of this sort that appear on students’ score cards to recognize particular discrete skills that they wanted to reinforce that were separate from the recognition embedded in the career badges. For example, one is named “Science Communicator.” How they appear in students’ scorecards is shown in the mockup for the DML proposal shown in Figure 3 below in a separate box from the badges.
Between the skills developed completing the badges and those recognized specifically by “achievements”, Planet Stewards recognizes diverse learning that would be expected from a program covering a wide range of science careers and knowledge.
Use badges to map learning trajectory > Provide routes or pathways > Badge hierarchy
Intended, enacted, formal
From NOAA’s first proposal for a grant to build the Planet Stewards badge system, they had the idea that teaching about careers would be an important component of the system, arguing that, “NOAA science provides unique opportunities for students of all ages to learn more about potential career paths” and providing examples of the wide variety of skills that come together under the job descriptions of various NOAA functions (DML Stage 1 Proposal).
As enacted in the Planet Stewards badge system, the quests and badges provide pathways for students to explore these careers. The collection of badges is designed like “a pool of quests with a flowchart (DPD Follow-up Interview). Quests often open up further quests upon completion, so students are guided through the system, scaffolding their knowledge and skills as they explore different careers, though there is no expectation that every student will complete all 15 career badges (DPD Initial Interview).
The implementation of the pathways has changed slightly from initial designs, in order to reinforce the focus on deepening student knowledge of science careers. Planet Stewards implemented five metabadges awarded for completion of all three career badges within each of the five career areas in the system. Lisa Dawley rejected the characterization of this structure as a hierarchy, or at least insisted that it’s a “minimal hierarchy” and that the focus of the system is more on defining pathways than defining levels of skill or knowledge (DPD Follow-up Interview). None of the maps of quests within one of these areas looks like any other, with quests branching differently depending on the decisions of the NOAA subject experts and instructional designers who developed the pathways (See Figure 2).
The Planet Stewards program, as implemented is a good example of the “provide routes or pathways” specific principle of “use badges to map learning trajectory.” The idea is to show the connections and differences between the different planet science careers, and building pathways that let students progress through them provides some freedom to focus on individuals’ interests while also ensuring that students have ample opportunities to practice the skills they should be learning in high school science.
Seek external backing of credential > Externally endorsed > Backing by NOAA and other agencies
Intended, enacted, formal
NOAA proposed the Planet Stewards badge system based on its educational content and goals for the DML competition. As part of the competition process, NOAA partnered with 3D GameLab’s developers and found that the 3D GameLab quest-based learning platform was a good fit for its goals. From this point, the Planet Stewards team intended to issue badges from the 3D GameLab application but clearly identified as NOAA badges.
As this plan moved toward a working badge system, the team met with subject specialist NOAA scientists to define and focus the badges. Peg Steffen told the DPD Project that the live offices whose scientists fill the positions represented in career badges were responsible for generating and validating the particular badge sets:
The people that were responsible for helping to generate and validate the weather and climate badge sets work for the National Weather Service and Climate Program Office. Since we do that science, we made sure that those people were involved. (DPD Initial Interview)
This close collaboration in development allowed NOAA to be confident that the badges carrying their logo would represent a level of quality and accuracy that they would want to be associated with their agency. From the perspective of the issuer application 3D GameLab, this partnership exemplifies the principle of “seeking external backing of credential” by means of “external endorsement.” NOAA signs the badges with its logo and name. Before the endorsement portion of Mozilla’s Open Badges specification is complete, this will likely be one of the most common forms formal endorsement of badges takes.
Seek external backing of credential > Externally valued > External recognition
Intended, not enacted, not formal
In addition to collaboration with NOAA, the Planet Stewards team also thought about seeking recognition by other outside entities. Possible partners like the College Board are already deeply embedded in providing credentials for high school students. The National Research Council is another example of an organization Planet Stewards felt might be interested in supporting the badge system in some way.
In describing the plan for the badges, Planet Stewards announced an intention to create badges that would be valued credentials of student achievement in important fields, credentials that might be recognized to qualify those students for opportunities:
The badges will convey that the learner is qualified as an inquirer of science, in a variety of areas, and that they have a thorough knowledge of a content area in science. This might qualify them for college and work apprenticeships in ocean, air, and environmental studies, among other fields. Prerequisites include completion of 8th grade. (DML Stage 2 Proposal)
To date, no formal agreements have developed from this intent. The DPD Project records this as an intended practice under the principle “seek external backing of credential” to record that Planet Stewards investigated and continues to think about possibilities for increasing the value of their students’ badges through collaboration with other organizations. The team would still like to pursue opportunities along this line in the future (DPD Follow-up Interview).
Award formal academic credit for badges > Seek college credit for badges
Intended, not enacted, not formal
In one way the Planet Stewards badges can be associated with formal academic credit in the classrooms in which the program runs. Teachers assessing students’ work will incorporate Planet Stewards assignments into class requirements as they deem appropriate. However, the Planet Stewards team also intended to explore the possibility of granting badge-earning high schoolers some college transfer credit awarded by 3D GameLab’s parent university, Boise State (DPD Initial Interview).
No formal plan has developed from the discussions with Boise State to date, but Planet Stewards still intends to investigate this possibility in the future. This represents a strong possible use of the principle “award formal academic credit for badges.”
Determine appropriate lifespan of badges > Never expires > Permanence
Intended, enacted, formal
From their early proposals, Planet Stewards intended to give students a permanent record of their science exploration, saying “badges will exist permanently as they represent a specific era of educational achievement in grades 9-12” (DML Stage 2 Proposal). The initial plans for permanence focused heavily on how the badges would be aligned to science standards and the value of the permanence would be attached to the value of meeting of the credential.
In their integration with the OBI, Planet Stewards implemented this dedication to permanent credentials, but the focus of badges had shifted to career exploration. With their collaboration with EdGate to support correlation of their activities to standards, the evidence pages for badges will be updated with the developing correlations as the science standards are finalized.
Use badges as a means of external communication of learning > Transparency and communication
Intended, not enacted, not formal
Upon their initial partnership 3D GameLab intended to build an associated display widget that could be embedded on other websites to allow students to easily display the badges they earned:
With a heading such as, “I’m a certified scientist!” at the top, clickable badges representing individual areas of achievement, and a “Mozilla Open Badge Certified” statement at bottom, viewers will understand that these badges represent areas of educational achievement aligned to completion of competencies in science education. (DML Stage 2 Proposal)
Upon digging into building for Mozilla Open Badges to implement Planet Stewards, the 3D GameLab team realized many of the desired capabilities for the imagined display widget existed in Mozilla’s badge organization program, the backpack. They decided to train their users to use the backpack for sharing instead of devoting limited development time to duplicating existing functionality. The language Planet Stewards used in the proposal is an example of the principle “Use badges as a means of external communication of learning.” The proposed practice was to support communication of learning was implemented to the point of providing users support in using the backpack tools but was not needed to any greater extent.
Have experts issue badges > Credentialed via external accredited entity > Badges issued by NOAA credentialed teacher
Intended, enacted, formal
Once they had decided on a career exploration framework, Planet Stewards intended to put NOAA’s educational content onto an internet platform and create a badge system that would allow students to earn credentials recognizing them for practicing the skills used in each of the system’s identified careers. However, NOAA scientists would not be available to administer learning experiences themselves or to perform assessments of the wide variety of materials students could turn in to meet the criteria of the different quests. The Planet Stewards team decided that the classroom science teachers leading students through the program would be the ones to assess activity and to award students badges (DPD Initial Interview).
As detailed below in the challenges section, the need to train teachers to hold students to standards consistent with the high value NOAA wanted to maintain for its badges became one of the biggest concerns of the badge system.
As enacted, Planet Stewards runs a quest-based training program for teachers and completed its third session in Fall 2013 (What’s new with NOAA Planet Stewards). The 3D GameLab team continues to seek for methods that will allow the training program to scale up so that more teachers and students have the opportunity to implement Planet Stewards in their classrooms. This is an example of turning teachers into accredited experts and using the recognizing design principle “have experts issue badges.”
Recognize educator learning > Give NOAA-certified teacher badges
Not intended, enacted, formal
As the first round of teacher training concluded, teachers asked Planet Stewards for badges as well. In response, Planet Stewards developed a NOAA-certified teacher badge for those teachers who completed the training. This badge became the requirement that allowed teachers to unlock the ability to clone the Planet Stewards curriculum and offer it to their own classes.
Design Principles for Assessing Learning in Digital Badge Systems
The Planet Stewards badge system utilizes the 3D GameLab online environment, but the program is administered by high school science teachers to their classrooms of students. The quest system allows students to make submissions of various types through the platform, but it does not feature automatic assessment of the submitted items for Planet Stewards, so the classroom teachers are responsible for assessing the completion of quests. In order to not hamper student progress, teachers can mark some quests to not require assessment before students can move forward along the learning pathways.
Align assessment to standards > National/state standards > Assessment activities are aligned to standards
Intended, enacted, formal
From the beginning of 3D GameLab’s involvement with the project, there was a strong focus on using the learning experiences in the Planet Stewards quests to forward students progress toward meeting the goals exemplified by science standards, particularly the National Science Education Standards and the framework underlying the Next Generation Science Standards. The 3D GameLab platform, as shown in Figure 3, which appeared in the project’s second stage DML proposal, has support for standards built in. The player scorecard shows students’ progress toward each component of standards they are targeting, and each quest has the capacity to be tagged so that its completion can mark students’ progress toward various standards components. The inclusion of this capability forced the team to ensure that teachers’ assessments would constitute adequate measurements of these learning claims.
When the badge system began operating in high school classrooms, the project’s shift toward career exploration toned down some of the effort to target standards, but no capabilities were removed from the system. Furthermore, Planet Stewards, through collaboration with EdGate’s standards correlation program continued to strengthen its ability to recognize student progress toward these national standards. Lisa Dawley describes the implemented pattern:
We did indeed create a curriculum matrix that aligned all quests to NGSS. We worked with a company called EdGate to create a digital mapping of these standards, a sort of Rosetta stone that would align our work to all 50 state science standards (personal communication, 20 January 2014).
The full integration with EdGate’s mapping is not yet complete as of January 2014, but it is “one of the next major features in our development queue (personal communication, 20 January 2014).”
Use leveled badge systems > Competency levels & Metabadges > Leveled assessments for leveled badges
Intended, not enacted, not formal (Competency Levels)
Not intended, enacted, formal (Metabadges)
Initially, Planet Stewards intended to focus on standards and design the badges around “levels in achievement of competencies, and will be tied to learner analytics that visually illustrate progress toward completion of national science standards at the 9-12 grade level” (DML Stage 2 Proposal). The hypothetical designs that appeared in the proposal were not thought to be complete. The proposal detailed the process for moving from these initial thoughts to a completed design for badge levels, saying “Levels have yet to be identified, and will require thoughtful discussion between project directors, instructional designers, and subject-matter experts from NOAA (DML Stage 2 Proposal). Before the 3D GameLab and NOAA education teams met to work out the levels, the system focus changed to career exploration, and at the July 2012 workshop at NOAA headquarters, the basic map of the badges based on career exploration was developed.
As enacted, the badge system contains minimal levels because of the focus on careers, none of which are elevated to greater importance than the others. However, there are elements of leveling that remain in the system. Teachers assess artifacts students submit for each quest. Lisa Dawley emphasized that the assessments and badges themselves don’t level, as the higher level badges are simply metabadges for completion of all the careers in a pathway. However, the complexity of what it takes to complete a quest inside the quests that lead to each career badge scaffolds and gets progressively more involved (DPD Follow-up Interview). For example, the career tracks in the Freshwater career category each have badge pathways culminating in an “Epic quest” featuring larger requirements (See Figure 4). As always, teachers design and run their own assessments based on their own curricular goals and quest customization, the default listed quest criteria and their NOAA certification training. As students move through the quests, and the requirements grow more complex, teachers must also adjust assessments to match.
The change in focus from achievement of standards to career exploration necessitated a change in the levels structure imagined for the badge system. At first, the design for levels fell under the DPD Project’s specific principle of “competency levels,” but the eventual career badges and higher level career area specialistbadges are a classic example of “metabadges,” badges awarded for the completion of a particular set of lower level badges.
Enhance validity with expert judgment > Use a combination of human and computer experts > Validation at multiple levels
Intended, enacted, formal
This practice is a function of the quest-based learning environment in 3D GameLab and its interaction with classroom assessment. Each quest has an associated task, something that the students must do or submit. The 3D GameLab team, led by Lisa Dawley of GoGo Labs and Chris Haskell from Boise State University, intended to design a system where teachers would “own all the awarding activity,” where they would have the ability to customize the quests their students encounter to match the skill level, feedback needs, and pacing that works in their own classrooms (DPD Follow-up Interview). Dawley explains, “Each quest has a task, and it’s up to the teacher to determine whether the minimum requirements for that task are met…Either you meet the requirements, as interpreted by the teacher, or you don’t meet the requirements and you need to go back and keep working on it until you do” (DPD Initial Interview).
This pattern was enacted as planned. Teachers determine for themselves “key milestone quests”, synthesizing steps, where teachers would really want to take a look at the work, that should require manual approval. The other less important ones could be automated, so the student could keep leveling up without needing to stop for approval (DPD Initial Interview). Planet Stewards found that some teachers might want to review 75% of the quests students complete, where others opt for a periodic milestone review. Dawley feels that this freedom increases the flexibility for teachers to be able to use the system in a broad range of classroom contexts, but makes it more complex, so some teachers don’t like having to figure out when to review or not. Dawley recounts that some teachers did not even realize they had this power, which had implications for future sessions of the Planet Stewards teacher training (DPD Follow-up Interview). Even where students are allowed to move past a quest without waiting for teacher feedback, the materials they submit are still visible to their teachers and can become part of later human assessments or impromptu formative interventions (DPD Initial Interview).
This practice is an example of the “use a combination of human and computer experts” specific principle of “enhance validity with expert judgment,” as the teachers can decide when and how to apply their time and effort, and when it would be better to let students move through the system more freely.
Enhance validity with expert judgment > Give human experts badges > Assessment of participating teachers
Not intended, enacted, formal
As detailed below under the challenges faced by this badge system, Planet Stewards realized from the beginning of planning to use the 3D GameLab platform that teachers would need help implementing quest-based learning and specialized training to become familiar with the NOAA content. Teachers are identified, training runs for 3 weeks. Teachers perform quests asynchronously, and combine with online live events. Teachers had to finish that and earn their badge, which allowed them to clone the whole Planet Stewards curriculum and start issuing. The assessments teachers encounter in their training quests were intended to take similar forms to the assessments they would later need to perform on submissions made by their students.
Lisa Dawley reports the training assessments were successfully implemented as planned and continue with new and growing groups. As the teacher training program concluded for the first time, Planet Stewards made a consequential change, based on feedback from the participating teachers. The teachers asked for a badge commemorating their accomplishment, and Planet Stewards created one for them. Upon completing the training quests and passing Planet Stewards’ assessments, teachers earn a NOAA certification badge.
This practice is an example of “enhancing validity with expert judgment” by building their experts’ familiarity with quest-based learning and the NOAA content.
Design Principles for Motivating Learning
The Planet Stewards team intends for students participating in this program to have the chance to learn about conservation through questing in the 3D GameLab platform. The website will afford students the chance to interact in a community, sharing what they have learned and collaborating with others in quests. The badge system follows 3D GameLab’s quest-based platform, each new badge “unlocking” further opportunities within the system. The initiative hopes that this feature will encourage students onward while giving them choices about which quests and careers to pursue. The team intended for participating teachers and advanced students to have the opportunity to design quests of their own.
Provide incentives > New activities > New learning activities and privileges
Intended, enacted, formal
3D GameLab’s vision for the quest-based learning platform was to motivate students to engage with learning material by involving elements from games, like user choice, into learning. 3D GameLab co-creator Chris Haskell claims early research shows over 60% of students continue questing even after earning an A grade, illustrating the motivational potential of the platform (The Power of Quest Based Learning). One of the elements that may have motivational potential is the ability for students to choose particular paths through the system, and prioritize what they want to work on first.
The questing platform, enacted as intended, opens up later quests based on the completion of prerequisites. The new activities progress from simple introductory activities to more interesting and involved quests, including some that use NOAA data. In the quest map for the Coastal Manager badge, for instance, students start with video content NOAA has created to get them familiar with estuaries, and move to analyzing real-time data from NOAA’s estuary data collection sites.
This is an example of how the Planet Stewards program may motivate students by opening up new and more interesting learning opportunities as they get further into their learning.
Set goals > User-determined learning trajectory & Provider-determined learning trajectory > Quest paths
Intended, enacted, formal
This practice is related to opening up new learning opportunities and describes the ability for students to see the learning opportunities and badges available in the system and use that knowledge to set goals for themselves. The Planet Stewards system in 3D GameLab was intended to reveal the available quests, though many of them are initially locked to students until they complete prerequisites.
As intended, the Planet Stewards badge system allows students to choose which careers to explore and allows them to see a possible target level (earning the career badge) that they could set for adequate exploration of that career. In addition, the tasks they complete are tied to their classroom grades in ways that their teachers determine, and students may use the quest to track the assignments they still need to complete to meet course requirements. In analyzing how this motivational principle functioned in the badge system’s function with its first testers, Lisa Dawley points to anecdotal evidence already available, saying the team received feedback from students including comments like, “I had no idea that jobs like this even existed,” “Being a marine biologist is not what I thought it was,” “I thought that was a career I was interested in, but now I’m not,” and “I’m interested in ocean science, and I thought I liked this job, but I found a job that I like better” (DPD Follow-up Interview)
From the point of building the career badge quest maps, the Planet Stewards team did not expect students to be able to complete all 15 career badges. They necessarily would decide on a subset to try for, with some guidance and possibly restriction from their classroom teachers (DPD Initial Interview).
Students ability to decide what to pursue and see the steps to achieve it is a good example implementation of the “set goals” motivational principle. Here, both user-determined and provider-determined trajectories come together through the NOAA-created, teacher-mediated, student-completed quest maps.
Recognize identities > Roles within a system > Career exploration choice
Intended, enacted, formal
Although the Planet Stewards team did not articulate it specifically in their proposals in a motivational context, the DPD Project noticed the structure of the principle “recognize identities” as possibly motivating students within this badge system. Before deciding on specific badges, Planet Stewards proposed a focus on “role and identity,” referring to how “3D GameLab allows users to create their own avatar to represent their identity. The quest groups, badge names, and graphics will illustrate growth and achievements in scientific roles over time” (DML Stage 2 Proposal).
As enacted in the badge system, the roles named in badges are those of careers employing adult scientists. Earning a badge with the name of “Ecologist” allows students to try that role on for size, with a badge certifying their accomplishment on their profile. Their reactions, as described above under “set goals” display how high schoolers tried on roles, and in some cases found that they didn’t like the actual work done in their selected scientist role.
The DPD project identifies this as a case where “recognizing identities” could have a motivational effect on students choosing learning content to pursue.
Utilize different types of assessment>Computer>Automated Assessments for small quests
Intended, enacted, formal
As stated in the assessment section, Planet Stewards developed an assessment practice that enabled teachers to assess relevant milestone quests and allow for less relevant tasks to be assessed by the computer system. This smaller scale assessment assisted in motivating learners to proceed to more relevant tasks.
Design Principles for Studying Learning
Research studying learning that occurs in Planet Stewards is closely related to the data that is collected through students participation on the 3D GameLab platform. The designers are focused on improving the badge system, learning how it can best operate in classrooms, and figuring out how to support larger and larger groups of students exploring careers.
Study badge impact > Research of badges > Data mining
Intended, enacted, formal
The 3D GameLab developers had been studying activity in their system informally before partnering with NOAA to create Planet Stewards. They intended to use many of the same techniques with the NOAA badges, planning to implement data and text mining of the system logs to learn about the activity happening in the badge system. Some methods of analysis used in the past include looking at the pattern of user activities and trying to determine patterns behind more successful and less successful users. Data available includes logins, what quests students played, and how long it took to complete questions.
Lisa Dawley mentioned that the team had an eye to developing key performance indicators (KPIs) from the broader pattern recognition efforts, which they would then use a KPI instrument to be able to refer to the same metrics over time. She explained two approaches with building tools that distill data from a system like 3D GameLab: “Either you want to be able to quickly tell what is going on, or you want to be able to look at real-time data,” meaning that desirable queries either answer the most important questions about system activity or provide insight that can be immediately used (DPD Initial Interview).
As enacted on 3D GameLab, organizers have collected logs of activity since the beginning of operation, which can be analyzed at any time. Dawley notes that the data coming out of the system can be particularly valuable in sustaining a project as an important tool to convince funders of its impact. Her function as CEO of GoGo Labs involves reporting to funders and investors on findings from the logs about usage. Dawley elaborated on the benefits gained by collecting data about how long it takes students to complete quests, explaining,“For curricular design, you can create a quest with a guess as for how long it will take, but you don’t know in advance. That takes data.” The team will be collecting this data through a larger pilot program with students in Fall 2013, and afterwards Planet Stewards will be able to share estimates about the number of hours teachers should plan to devote to different parts of the system as each teacher decides how it can fit into their own curricular designs.
As initially enacted, pulling some of the system-wide data still requires a laborious process where programmers manually query the database to measure aggregate activity, but work is underway building an administrator dashboard to access these performance indicators on a realtime basis. The 3D GameLab team has completed a control panel for teachers to make student progress through badges and quests visible for the purposes of their classroom management (DPD Follow-up Interview).
The DPD Project classifies this type of research as studying the impact of badges, or “research OF badges.” Its aim is to determine how the badges work in the system, and what effect they have on students’ lives.
Improve badge impact > Research for badges > Quest analysis
Intended, enacted, formal
Distinguished from research OF badges is the function studying system activity can have to improve the badges and quests in Planet Stewards. Lisa Dawley explained that “we were doing the data mining before the badge piece came in, but now badges give an interesting way to look at the data. It’s another way to sort and look at what’s working and what’s not working” and noted that one new metric available for analysis was the aggregated level of learning outcomes, when previously outcomes had been represented by students’ grades:
Before, we’d look at who got an A, B, C, D or F and ask what were the actions or the click behavior that they took to get there, to the A or to the F, and trying to drive more students in the direction of the A…Now it’s not looking at the outcome as the grade, but the outcome as the badge or no badge. (DPD Initial Interview)
The system delivers feedback on the curriculum through queries like these comparisons for curricular improvement. Planet Stewards administrators can determine which quests are most popular, which get the highest ratings as well as whether they are taking the amount of time expected. They can compare this output to various features of the different quests, like whether they had video or whether it involved project-based learning as part of a process of evaluating the quests and determining whether to revise requirements, content, or descriptions.
Challenges this Project Faced
Planet Stewards is an ambitious badge system in many respects. Particularly, implementing any system nationwide to be administered in many classroom contexts by teachers whose curricular needs varied forced Planet Stewards to figure out how to ensure consistent quality of the badges. A second challenge detailed below shows how the badge system administrators from 3D GameLab were put in the position of technology support for the teachers and students using their system, and that role extended to the software running other components of the wider badge ecosystem.
Training badge awarding teachers: NOAA certification for valid assessments (Recognizing and Assessing)
One of the distinguishing features of the Planet Stewards badge system is that high school teachers across the country gain the privilege of awarding their students high value badges under the widely-recognized NOAA name. However, one of the main consequences of this design decision is that Planet Stewards had to design practices to ensure that badge earners were held to the high standards NOAA desired. This became one of the main system design challenges Planet Stewards faced, and it impacted the team’s decisions around both recognition and assessment practices.
The team from NOAA and 3D GameLab decided that the need for consistent assessments across many classrooms required teacher training and that the ability to offer the Planet Stewards curriculum would be limited to teachers with the training. They proposed a solution tailored to the capabilities of the 3D GameLab platform:
Groups of quests that are associated to awarding the badge will be “locked.” It will require the teacher to complete a pre-training in the use of a quest group, as well as a verification that they are a certified teacher in order to unlock access for her class of students. (DML Stage 2 Proposal)
Certification of teachers would then open up the ability to “clone” the Planet Stewards quests, customize them to the needs of a particular classroom, and then to issue badges.
Implementing the training practices first occurred in Spring 2013 with a pilot group of 20 teachers interested in running the Planet Stewards program. Speaking from 3D GameLab’s perspective, Lisa Dawley described one of the goals of the training program as ensuring badge integrity, saying “when you’re a platform and you’re issuing badges on behalf of someone else, [where the assessment isn’t automatic] you want to make sure that the assessments are valid.” The training also ensures that teachers will be able to support their students’ use of the quest platform, help them work through the content and assignments, and adequately assess the learning claimed by the badges.
While teachers may be initially attracted to either the NOAA content or the 3D GameLab quest-based learning methods, they need to learn about both in order to run a successful program in their classrooms. Through the training program, teachers learn about quest-based learning, instructional design for quests, and how to integrate activities in the online platform into their classes. Dawley says, “We first pull them into this piece, then we introduce them to the Planet Stewards curriculum.”
There remain challenges for the badge system’s training practices. The team determined that the pilot round of training with 20 teachers went well and immediately made plans to follow up with larger groups in the summer and fall. Dawley has an eye to scaling the training system as the next large challenge to face, saying “My next question is ‘ok this works with 20, what happens when you try to scale to 5000, and we haven’t addressed that yet’” (DML Initial Interview). The question remained when the DPD Project checked in with Planet Stewards again in October 2013. As Dawley explains, this “limits the scaling of the Planet Stewards platform, because they can’t just say ‘Kids, go play Planet Stewards. They have to be trained’” (DPD Follow-up Interview). That fall, 70 teachers went through a third round, but the Planet Stewards team is still considering how to scale the teacher training portion of the badge system even more, recognizing that teachers’ ability to attend training sessions synchronously and the associated costs are obstacles to overcome.
For the teachers who went through the training, their user accounts in 3D GameLab gain the technical privileges necessary to run the program. Though not initially intended, Planet Stewards expanded their certification to include awarding teachers an OBI-compliant badge to recognize them as a NOAA-certified educator. This carries NOAA logo and may prove to be an attractive credential. The Planet Stewards team felt that offering a badge for teachers also made the training requirement more clear in terms of representing an actual certification and endorsement to be able to offer the curriculum.
Dawley, L. (2012, November 29). DPD Initial Planet Stewards Interview.
Dawley, L. (2013a, October 6). DPD Follow-up Planet Stewards Interview.
Dawley, L. (2013b, December 2). What’s new with NOAA Planet Stewards? | Planet Stewards. Retrieved from http://planetstewards.wordpress.com/2013/12/02/whats-new-with-noaa-planet-stewards/
Dawley, L. (2014, January 20). A couple final Planet Stewards questions.
Dawley, L. (n.d.). Project Q&A With: Planet Stewards: Personalized Learning in 3D GameLab. HASTAC. Retrieved December 27, 2013, from http://www.hastac.org/dml-badges/planet-stewards
Gillispie, L., & Sims, K. (2012, September 3). Coastal Manager Badge Preview | Planet Stewards. Retrieved from http://planetstewards.wordpress.com/2012/09/03/coastal-manager-badge-preview/
Grant, S. (2013, October 23). 5 Buckets for Badge System Design: “You Are Here.” Retrieved from http://www.hastac.org/blogs/slgrant/2013/10/23/5-buckets-badge-system-design-you-are-here
Haskell, C. (2012a). 3D GameLab from Student Perspective. Retrieved from http://www.youtube.com/watch?v=wsen5rg7Lb0
Haskell, C. (2012b). Teacher Dashboard NOAA PS. Retrieved from http://www.youtube.com/watch?v=ux-IUYX6_qg
Haskell, C. (2013). 3D GameLab – The Power of Quest-Based LearningTM. Retrieved from http://www.youtube.com/watch?v=fwk0oCMWFzY
National Academy of Sciences. (2011, July 19). A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas. Board on Science Education. Retrieved January 21, 2014, from http://sites.nationalacademies.org/dbasse/bose/framework_k12_science/index.htm
NOAA, & GoGo Labs. (2012a). Planet Stewards DML Stage 1 Proposal.
NOAA, & GoGo Labs. (2012b). Planet Stewards DML Stage 2 Proposal.
Planet Stewards. (n.d.-a). Badges & Quests: The Badges are Here! Retrieved January 21, 2014, from http://planetstewards.wordpress.com/badges-quests/
Planet Stewards. (n.d.-b). Planet Stewards Popplet. Retrieved January 20, 2014, from http://popplet.com/app/#/522637
Planet Stewards. (n.d.-c). Project Team. Retrieved January 20, 2014, from http://planetstewards.wordpress.com/project-team/
Sims, K. (2012, August 23). A big day for OBI badges at NOAA. Retrieved from http://planetstewards.wordpress.com/2012/08/23/a-big-day-for-obi-badges-at-noaa/
|Recognizing Principles||Assessing Principles||Motivating Principles||Studying Principles|
|Specific principles:||Specific principles:||Specific principles:|