BuzzMath

Preface

The information contained in this project appendix was gathered from the original project proposal that was funded in 2012 and interviews with project leaders through the end of 2014.  This appendix and the overall design principles database from the DPD project does not reflect further evolution of the project or developments after the final interview that took place in 2014. As of the final interview, BuzzMath had issued badges and developed a functioning badge system. Based on this information, we have classified this badge system as an implemented (rather than a partial or suspended) badge system.

 Summary

BuzzMath is a platform where middle school students develop mathematics competencies corresponding to the Common Core standards. The project issues Open Badges aligned to these standards as students complete activities on buzzmath.com.

The BuzzMath team from the small software development firm Scolab wants to help students in grades 6 through 8 achieve competency in the mathematics standards included in the Common Core State Standards system. To this end, the website guides students through “practice documents” and “challenge documents” targeting each element of the standards (Figure 2). These take the form of sets of assessment questions that progress in difficulty and build on learned concepts as they go. They are designed to complement classroom instruction and assessment. In addition, teachers that use BuzzMath in their classrooms will be able to track their students’ skill development through an integrated dashboard (Figure 3)

Buzz Math Home Page

Figure 1. The Buzz Math Home Page

Middle schoolers could earn up to 15 gold badges across the Common Core math content areas each year, but they will also have the opportunity to earn smaller achievement badges for meeting small goals. BuzzMath will offer “process badges” to recognize soft skills related to success in mathematics by allowing teachers to award these badges.

The Common Core standards are now in the adoption process by most American states. By implementing a badge program to cover this content, BuzzMath hopes to help students navigate through it by providing clear learning pathways and to offer a reliable set of assessments so that teachers can follow their progress confident that they have a valid picture of learning. The availability of a broad standard ensures that BuzzMath content is relevant to a large audience of students, justifying investment in the project.

BuzzMath targets homeschoolers as well as classrooms and earns revenue on a subscription model (Figure 1). Classrooms can access the site for free, but offers premium features under a subscription.

BuzzMath attempts to fill an important niche supporting the implementation of Common Core standards, especially in their early years when not all math teachers have had the time to build a complete suite of standards-targeted assessments of their own. Using a platform like BuzzMath to provide coverage across the standard could allow teachers time to build on their best lessons first. In the realm of studying learning, offering badges along such a well-defined pathway creates the opportunity for a wealth of data, allowing teachers better knowledge of their classrooms and offering BuzzMath opportunities to study the learning system.

Timeline

Scolab, creators of BuzzMath, earned a grant through the 2012 DML Badges for Lifelong Learning competition, sponsored by the MacArthur Foundation.

In the investigation of this initiative, the Design Principles Documentation Project scoured all the competition winners’ proposals in the to describe the practices they intended to use in their badge system designs for recognizing, assessing, motivating, and studying learning. Next, the DPD Project grouped these practices and those identified in other projects and named general principles under which they clustered. The research team interviewed Jean-Philippe Choinière and Jamie Piecora of Scolab in September 2012 to begin the process of identifying the practices that BuzzMath had begun to enact as they were building the system.

Scolab had the idea for an application like BuzzMath on the back burner until winning a grant through the DML competition to develop it. With the grant, they brought on two new programmers and were able to start working on the badges project right away. Planning and development proceeded from April 2012 to April 2013. Development started with the infrastructure, because the content badges required modifications to the database to collect the necessary data. Then the BuzzMath team worked with a teacher in the US to design the content of challenge activities. The DPD Project interviewed BuzzMath again November 2013 as BuzzMath had entered its first full school year of operation and officially launched to determine what practices and principles would become part of the system’s continued formalized operation.

Evolution of Practices and Relation to Principles

What follows is a list of practices as they relate to the general and more specific design principles in each category of practice. The headings name a (a) General Principle, (b) Specific principle, (c) Specific practice. The paragraphs below each heading detail the project’s (a) intended practice, (b) enacted practice, and (c) how that practice relates to the specific and general principles.

Principles for Recognizing Learning in Digital Badge Systems

The most important element of the badge system BuzzMath intended and implemented was alignment to the Common Core State Standards for Mathematics. In addition, BuzzMath designed “process knowledge” badges that classroom teachers could award to recognize positive behaviors and habits related to mathematics success.

Align credential to standards >Use national or international standards >Common Core standards alignment.

Intended, enacted, formal

As a growing number of schools in the US implement the Common Core mathematics standards, the need to recognize student skills defined in the standards grows. The schools themselves are the primary audience of this information, but common standards create more audiences among administrators and state officials for program assessment, and outside of the schools among those who employ or select graduates for further opportunities. Providing learning opportunities in this material this learning is BuzzMath’s key goal.

As enacted at BuzzMath, each of the “gold” badges are directly aligned to Common Core State Standards. Students will have the opportunity to gain up to 15 gold badges per year across the mathematics standard. The alignment of the curriculum together, it represents mastery across the standards. For example, project member Jamie Piecora explains, “Fractions are a main concept of middle school math. When you get a gold badge for fractions, it is an accumulation of all the common core standards associated with fractions concepts. When you achieve all of the gold badges for a grade level, that would represent a full mastery of all of the concepts in the Common Core State Standards” while there is not exactly a one badge per standard equivalency (DPD Interview).

BuzzMath started with the 6th grade level, hiring a curriculum specialist to go through each activity and ensure their coverage of the Common Core at that level was accurate and complete. In order to take the program from where it started with unaligned math exercises to complete alignment for 6th grade, the team had to figure out where their existing activities fit within the standard and then create almost 40 new activities of 10 pages each, a sum that now represents half of the activities available at the 6th grade level. A great deal of time was dedicated to this effort, as Jean-Philippe Choinière describes, because, “A complete Common Core curriculum was…very important because we wanted our gold badges to represent all the skills in [each] topic.” BuzzMath found that completing the alignment yielded much clearer pathways: for students and teachers, the order they should move through the activities is now obvious, and smaller goals are a lot clearer. The alignment allowed them to structure the activities with review and enrichment sections. The activities’ organization is now much more clear to teachers as well, who developers had noticed often assigned material in the order it was presented to them rather then spending the time to figure out how the curriculum was designed.

These are examples of powerful effects of the decision to use a practice fitting with the “Align credential to standards” principle. This practice was one of the core guiding principles of BuzzMath’s system design as a whole.

Use badges to map learning trajectory > Level badges >Granular badges.

Intended, enacted, formalf

While the bronze, silver, and gold badges are intended to represent developing skill in a broad area of middle school math, smaller “process knowledge” and “achievement” badges each represent “represent a discrete skill, competency, or activity for a specific grade level” (DML Proposal).

The Common Core standards are compatible with this approach. Drawing from research by William Schmidt and Richard Houang, the standards aimed to define the curriculum as a sequence of content that reflects the underlying “sequential or hierarchical nature” of the discipline. By delineating granular achievements out of a larger structure, the standards allow BuzzMath to recognize these component competencies as well as the broad strokes.

As initially enacted, BuzzMath’s badge system carries both content mastery badges as well as “stars” for successfully completed activities. Typically, 4-6 stars add up to a bronze content area badge, and then more activities allow the user to level up to silver, then again for gold. BuzzMath only will issue the gold achievements representing mastery of a Common Core standard as a Mozilla OBI Open Badge.

One surprise that BuzzMath encountered in building out the content at the 6th grade level was as Jean-Philippe describes: “at first in my head all the topics would have three content badges (bronze silver and gold), but when we started the alignment to a new structure based on the Common Core system, we saw that some topics, like fractions only had just enough activities to have one or two badges at the 6th grade level, then a lot more in 7th grade”, instead of the three they had planned (DPD Bloom Interview). As a consequence, BuzzMath decided the gold badges would represent the targeted component of the standard, and bronze and silver would be added as they fit.

The team would be interested to explore further granular badges, like a “wood” type for smaller accomplishments or special “platinum” for large accomplishments. For the time being, as enacted, the badge of the same type are roughly comparable, and half of the content areas have all three levels in their progression. Bronze badges require a set of easy activities intended for students to complete in about one class period.

This practice fits in the principle of “Use badges to map learning trajectory” because the levels are designed to guide students through the activities to the point where they earn a gold level badge aligned to a component of the Common Core curriculum.

Use badges to map learning trajectory > Provide routes or pathways >Pathways to learning

Intended, enacted, formal

BuzzMath’s platform, leveled badges and progress bars toward badges are designed to guide students through activities, scaffolding their understanding on the way to gold content badges.

Practice Document

Figure 2. A “Practice Document” takes students through progressively more difficult questions in one content area, like number lines.

Figure 2 shows a progression through one activity on number lines as enacted. Locating numbers on the line, as shown is the first concept in number lines. This understanding is built upon by the following concepts, adding up to the understanding expected by the number lines component of the Common Core Standard.

BuzzMath found that integration with the standard created much better coherent pathways for their students. Jean-Philippe says, “It’s no longer just a big wall of activities to complete… Now we have different groups.” He points toward some feedback they received on this point:

“We have a nice quote from a teacher: ‘It allows my student to focus on a goal that is more specific than a number of stars. I also like how the sections are broken up into review, badge, and enrichment.’” (DPD Bloom Interview)

Buzzmath feels that the alignment creates a nice progression felt it was easy to make it clear to outside audiences what leads to each badge, and there are possibilities to enhance the criteria pages further, perhaps including average time to complete badges.

Determine appropriate lifespan of badges >Never expires> Permanence of badges

Intended, enacted, formal

The badges issued through this system were not intended to expire, as BuzzMath indicated that they should be considered a record of the student’s achievements at the grade level when they were issued: “These badges will represent their mathematical knowledge and skills at each specific grade level.” (DML Proposal). That fact does not change even if a student later forgets a concept. As with all OBI-compliant badges, students will decide if and how to present these badges in their future pursuits.

Jean-Philippe notes, “It’s hard to remove something from the student after they collected it,” wondering if the students would collect badges again if they are designed to expire. BuzzMath explored this option at first, but after talking to teachers and some students, they decided it was not something that looked fun for the students. The team noted that expiration could be an interesting dynamic, to get students to go back and refresh the skill, but they decided to implement this in a refresher section at the beginning of later exercises dependent on the skill instead (DPD Follow Up Interview).

In this way, BuzzMath determined the appropriate lifespan of the badges to be permanent record of the student’s achievement at the time they are issued.

Use badges as a means of external communication of learning > Opportunities outside organization

Intended, not enacted, not formal

There are a number of venues outside of students’ schools that would be interested in their badges, including schools to which they may transfer, college admission officers, and guidance counselors. Badges aligned to a well-recognized standard, “will provide college recruiters, educators, employers, tutors, parents, and scholarship providers with a comprehensive understanding of an individual’s mathematical competencies” (DML Proposal). BuzzMath’s Jean-Philippe comments:

“This becomes more interesting for high school students, but there is less of this for middle school…We will continue to make the badge project evolve, but I want the student to be able to print the evidence in a certificate way, so they can show their parents, or so the teachers can show the parents in the meeting…Privacy issues are a big thing. Maybe 90% of our users are limited by COPPA. We decided to go with basic things at first. We want to explore more.” (DPD Bloom Interview)

One possibility BuzzMath might be interested in exploring would be possible collaboration with other platforms to let students start further along in a similar system like Khan Academy, but the team sees that building partnerships with competitors may be difficult (DPD Bloom Interview).

How this practice evolved illustrates some of the issues involved with implementing the “Use badges as a means of external communication of learning” principle, especially for young earners.

Have experts issue badges >Credentialed via external accredited entity >Award badges through online system

Intended, enacted, formal

BuzzMath intended there to be multiple types of badges within the BuzzMath system: content badges for Common Core standards, process badges to recognize demonstrated positive behaviors that lead to mathematical achievement and success, and activity badges for completing small goals and individual activities on the site. BuzzMath intended to record and serve badges through its online system for both content badges and the process badges to be awarded by teachers (DML Stage 2 Proposal).

As enacted, BuzzMath is the sole awarding entity listed as issuer on OBI badges, so students’ achievement is recognized by “external accredited entity”, an example of the general principle “have experts issue badges.”

Seek external backing of credential > Externally endorsed > Seek Common Core endorsement

Intended, not enacted, not formal

BuzzMath hoped to seek the endorsement of the Common Core State Standards Initiative for its content badges.

BuzzMath is operating without a formal endorsement, feeling that it would be a nice bonus to have an endorsement from the Common Core initiative, but not essential to the value of the badge system. Negotiations on this intent have not occurred.

Principles for Assessing Learning in Digital Badge Systems

BuzzMath runs assessments for each problem and content badges through the learning activities on the buzzmath.com website. Previous to the badges grant and Common Core alignment, BuzzMath had some basic plans to generate challenges, but through building the badge system, these are now much improved. The assessments are almost entirely automated, as was intended from the beginning. However, after looking into the assessment practices, research, and discussions with other educators in the field, the project is modifying the algorithm to reflect these viewpoints and to provide a better experience for students. Assessments allow teachers to track the performance of all their students, tracking not only mastery of concepts along the learning pathway, but also time spent and accuracy for each concept. While the assessments consist of several pages of related mathematical concepts, BuzzMath has taken great care to ensure that the last several pages of each “challenge document” are precisely aligned to the Common Core.

Use leveled badge systems > Competency levels > Leveled assessments for leveled badges

Intended, enacted, formal

The bronze, silver, and gold content badges in the BuzzMath system require assessments that reflect the increasing difficulty of the material. As students progress through BuzzMath challenges, they must be capable of employing previously studied concepts as their understanding of the particular challenge is assessed.

As they got started, BuzzMath realized they needed to make some changes to their assessment practices in the activities to match the needs of different classrooms of students. Assessments are generated from the problem types students encounter in practice documents, but BuzzMath has made significant changes to how they are generated over the first year of operation based on student data and teacher feedback.

At first, BuzzMath thought the best assessments would be custom made for each medal, but it would have been an impossible amount of additional work to create all of them. The team felt the best option for challenges was to remix content students had encountered in practice.

The new model allows BuzzMath to manually select pages that they want to be remixed. Problems are drawn from all the available pages via an algorithm that BuzzMath has frequently adjusted to take into account data like how long ago students had encountered that skill. Bronze challenges currently select 5 pages, silver 6 pages, and gold 7 pages as part of an effort to ensure challenges take students a predetermined of time, although the targeted time had to be significantly lowered from initial assumptions.

One decision in tweaking the algorithm was whether or not to favor content that students repeatedly failed to master in practice. The assumption that this would be useful turned out to be problematic, as Jean-Philippe explains, “maybe a student missed this problem 5 times not because of not understanding the math, but maybe because of the way we explain it.” Forcing students to encounter the same problem again on a higher stakes assessment would not be helpful to their learning (DPD Follow up Interview).

This practice remains an important component of the BuzzMath badge system and represents the assessment principle of assessing in leveled badge systems, specifically aiming to assess developing competency.

Enhance validity with expert judgment >Use a combination of human and computer experts > Protecting validity

Intended, enacted, formal

Sample View

Figure 3. Sample view from a dashboard showing one student’s activity.

BuzzMath intended to design practice and challenge activities so that it is very difficult for students to guess correct answers. Prompts should require answers in varying formats, such as “matching objects, drag and drop, multiple choice, true/false, text input, and ordering” (Stage 2 Proposal), and allowing teachers to see data about how long the student spent on the site and the number of attempts will give them confidence in the validity of the assessment. See Figure 3 for a sample of the teacher dashboard view of one student’s progress and Figure 4 for the visual appearance of available question types.

Answer Formats

Figure 4. A variety of answer formats

BuzzMath implemented this practice as designed. Content badges are awarded through the website as students complete challenge activities, and process knowledge badges are awarded by their classroom. In practice, BuzzMath tried to avoid multiple choice as much as possible. They tried to design their new content with a variety of “response objects”, with manipulatives. They hoped that students would need to think more about the problem before answering it than other systems that focus mainly on multiple choice (DPD Follow Up Interview).

Students have a limited number of attempts to complete challenges, while they have unlimited passes through practice sections. Teachers have access to accuracy statistics across all attempts. This combination of computerized tests designed to stimulate different ways of thinking about concepts and exposure of metrics that could help teachers see patterns of invalid assessment is an example of the principle “Enhance validity with expert judgment.” The best fit for implementing this general principle within BuzzMath’s online learning environment context paired with classroom instruction was naturally in the category of the specific principle “use a combination of human and computer experts.”

Align assessment activities to standards: create measurable learning objectives > Common Core state standards > Align assessments to Common Core

Intended, enacted, formal

As mentioned above, BuzzMath’s alignment to the Common Core is one of the core features of the badge system. This recognition decision guided the decisions about what assessments were needed. BuzzMath worked with a curriculum specialist designing activities to match the skills set out in the standard.

As initially enacted, BuzzMath’s automated content “challenge documents” consisted of several pages of problems algorithmically generated based on problem types that students see in previously completed “practice documents.” As the program has developed through 2013, the algorithm that generates these assessments has been tweaked frequently, based on how students use it to adjust for length while ensuring good standards alignment. Going forward, BuzzMath will focus on making sure that the last several pages of each challenge are well-aligned to the standard, though they will give themselves the freedom to incorporate some other content in the assessments as well (DPD Follow-up Interview).

Their commitment to these practices and those described under “Protecting validity” above demonstrate the principle “align assessment activities to standards.”

Use mastery learning > Judged by a combination of human and computer experts > Assess mastery of content and processes

Intended, enacted, formal

BuzzMath intended to issue both “content badges” and “process badges” (DML Stage 1 Proposal). The team wanted to create each content badge to claim the earner has mastered a particular component of the Common Core standards, like “7th grade fractions,” with automated assessments that credibly back up that claim (See Figure 5). Process knowledge assessments cover skills like peer tutoring, self efficacy, and mathematical communication. Both of these badge types are designed to recognize developing mastery of skills rather than singular accomplishments, but they necessitate different assessment techniques. BuzzMath planned to let classroom teachers assess and award these badges based on their own classroom observation or formal assessments individual teachers may assign for the purpose.

As enacted, the automated system allows students any number of attempts to complete “practice documents” and several attempts to complete “challenge documents.” The students are essentially competing against themselves to improve their badge level, and they have the chance to get instant feedback on incorrect answers. With paid subscriptions, BuzzMath offers a premium answer guide feature that should help students gain formative feedback from failed practice questions. Additionally, teachers can gain access to detailed reports and statistics showing their students’ progress through activities. The implementation of the process knowledge badges is, as planned, largely up to teachers own discretion (DPD Follow-up Interview).

Going forward, it is clear that both the content and process knowledge assessments at BuzzMath are configured around tenets of mastery learning. Their particular implementation of mastery learning requires both automated assessments that provide feedback and human review of students progress and developing soft skills.

Principles for Motivating Learning with Digital Badges

BuzzMath implemented a badge ecosystem to serve as a roadmap to the middle school math requirements that helps students see clearly how to navigate through the concepts available on the website. Before the introduction of badges, BuzzMath found that students were unsure of where to go after completing an activity. With the inclusion of badges, the pathway to leveling up and completing progress bars will be much more easily understood by the user. Making this clear pathway visible has hypothesized effects on learner motivation and ultimately what the student learns. One thing the initiative intended and is still trying to discern is how to offer the privilege of peer tutoring to students who have successfully completed a certain level. Additionally, BuzzMath hopes to reward hard-working students with other privileges, such as entry into math competitions and featured profiles on the buzzmath.com website.

Setting goals >Display of goal trajectory > Provide clear learning pathways

Intended, enacted, formal

One of the primary goals of adding a badge system to BuzzMath’s platform was to make the progression of learning clear to students:

“In most cases, content knowledge badges will be awarded in a progressive manner. Students will first earn a bronze badge that will later be upgraded to silver and then gold as they demonstrate an increased knowledge of each concept.” (DML Stage 2 Proposal)

Leveling up badges from bronze to silver and then gold is an easily understood pathway that is replicated across each of the 15 badge tracks per year of the Common Core.

BuzzMath implemented the badge levels to show the progression between different activities and put clearer objectives in the system. While they would have liked to have duplicated the bronze, silver, gold structure in every badge pathway, some tracks only have two badge challenges (DPD Follow-up Interview).

They will continue to measure and tweak their activities as they learn more about how students move through them, but so far, the team feels that the pathways are one of the main positive aspects of implementing the badge system. Including elements of review within this pathway further strengthens student learning. This practice is an example of the general principle “setting goals” because showing students where they are along a clear trajectory allows them to plan their path ahead and strive for the important milestones and corresponding badges.

BuzzMath believes this practice to be an important component to their badge system that lets students see what progress they have made, how far they have to go toward their goals (and the goals set for them by the Common Core standards). The system will continue to use this principle going forward.

Recognizing different outcomes >Effort-based >Acknowledge related soft skills and process mastery

Intended, enacted, formal

Content Badges

Figure 5. Content badges show mastery of a Common Core component, like 7th Grade Fractions.

In order to reinforce positive behaviors and problem solving techniques, BuzzMath intended to offer classroom teachers the chance to award “process knowledge” badges when they see students exemplifying these behaviors that may lead to improved math achievement. The team defined eight categories of skills in this area that contribute to mathematical success and created badges around these concepts. Badges for self efficacy, peer tutoring, teamwork and problem solving are among the process knowledge badges BuzzMath built to allow teachers to encourage these behaviors (Figure 5).

Another type of badges BuzzMath intended, primarily for motivation was “achievement badges,” which recognize accomplishments like 50 stars earned, directly measuring the quantity of activities completed. They will not turn into OBI compliant badges: “These achievement badges are meant to serve as motivation for students and they will only be visible within BuzzMath” (DML Stage 2 Proposal).

As enacted, BuzzMath allows teachers to use process knowledge badges to reward or motivate students in their classrooms however they want. Achievement badges that indicate levels of commitment to progressing through the material are awarded automatically.

Both of these types of badges are examples of the application of the effort-based motivation specific principle of “recognizing different outcomes.” The soft skill-type process badges establish values within the system and abstract goals to strive for, and the achievement badges in the online system remain to motivate students to keep moving.

Buzz Math 6

Figure 6. Process knowledge badges intended for the BuzzMath system (DML Stage 2 Proposal).

Utilize different types of assessment > Expert > Teachers design their own assessments for process knowledge badges

Intended, enacted, formal

BuzzMath could easily measure students’ interaction with mathematical concepts when they were working alone with the website software, but the online component has no credible way to assess their process knowledge skills like teamwork and mathematical communication. In order to issue badges in these areas, BuzzMath intended to allow participating teachers to decide on their own how to assess and award process knowledge badges to the students in their classrooms. By the time of submitting their revised DML grant proposals, BuzzMath had decided that “Each student will have the opportunity to earn only one of each badge in the set of process knowledge badges each school year” and noted that teachers would include information from their Class Tracking Reports in their evaluation. (DML Stage 2 Proposal).

As enacted, BuzzMath has left the awarding of these badges mostly up to teachers. Some teachers tended to treat them as the result of an assessment, while others would choose appropriate situations that occur in the classroom as opportunities to observe students and award badges based on observed behaviors either during class periods or by reviewing Class Tracking Reports on buzzmath.com. There is a near-identical set of process knowledge badges available at each grade level (the 7th grade badges are pictured in the design prototype in Figure 5).

BuzzMath decided early that the use of in-person expert assessment is necessary to recognize process knowledge with badges. The DPD project finds that practices in the “expert” category of “utilize different types of assessment” may motivate students to further participate in learning activities because of the attractiveness of gaining approval from respected authorities like teachers.

Use different types of assessment Computer Content assessed by computer

Intended, enacted, formal

As previously  discussed, the software used by Buzzmath was capable of assessing learners engagement with mathamatical content. Further, the capacity for such software to track performance on such content provided additional motivations by enabling learners to compete with their previous performances on the content. These processes provide motivation for continued engagement with the mathematical content.

Provide incentives >Peer mentorship>Peer tutoring privilege

Intended, not enacted, not formal

As the badge system took shape, BuzzMath envisioned that students could earn the right to join various learning communities as a peer tutor.

Enacting this practice has challenged BuzzMath to figure out how to inform people of students’ capabilities as potential mentors without compromising privacy as regulated by COPPA. Unfortunately, based on previous experience with implementing this within a similar system, Buzzmath felt that this practice was too resource-intensive to enact. They intended to create a system in which students who collect the gold badge in one topic, should be able to help others, but the question of how to implement this goal remains. BuzzMath points out that in Quebec there is a program sponsored by the government to provide homework tutoring that has partnered with BuzzMath developer Scolab. Scolab plans to build a virtual console where students who need help could interact with tutors. Experience from this project may feed back into BuzzMath.

This intended practice is an example of the motivational principle “provide privileges,” which claims students may be more motivated to engage with learning when opportunities for sharing their knowledge through peer mentorship with others are opened up.

Provide incentives >New activities & prizes>Access to new activities

Intended, not enacted, not formal

Early plans imagined badges could qualify students for opportunities offered internally by BuzzMath, like online math competitions: These competitions will consist of challenging problems that students will complete in a specified time period. The names of the top scorers will be posted on the BuzzMath website leaderboard” (DML Stage 1 Proposal). BuzzMath also imagined online learning communities where students could collaborate to solve challenging problems and assist each other with work (DML Stage 1 Proposal).

As enacted at BuzzMath, badges may motivate students because they show mastery and dedication appreciated by organizations like honor societies. The team continued to investigate running contests with publicly displayed results and contest badges. As of the end of 2013, BuzzMath is planning some class contests, when teachers choose to allow students to complete. Competitions include prizes for best classes or individual students including monetary prizes like gift certificates for Amazon.

The BuzzMath team does not want to focus on practices that bring extrinsic motivation to the forefront, like contests with prizes, but they would like to be able to offer more of these in the future for the enjoyment of students on the side of their more serious progress through the badge system (DPD Follow-up interview). Challenges to introducing this content fall mostly in the realm of needing to allocate resources to build custom non-reusable content for them.

The forthcoming competitions represent a new activity that students can gain access to, which may motivate them to achieve the qualifying level. In addition, prizes provide potential extrinsic motivation. These practices fall under the principle of attempting to motivate student learning by “providing privileges.”

Give badges for small accomplishments to hook in learnersAward badges for effot and progress

Intended, enacted, formal

As described above, Buzzmath awarded participants for multiple levels of performance and engagement. These badges included not only improvement on performance but also demonstrations of effort. These small awards provide additional motivation to continue to proceed through the badge system.

Design Principles for Studying Learning in Digital Badge Systems

No research or evaluation practices were necessarily intended but BuzzMath designed the site to obtain data on student activities and making them available to the teacher, allowing teachers to intervene with struggling students. This characteristic of the activity platform allows for possible research. Despite developing no formal plans initially to study the badges in the system, the BuzzMath team at Scolab constantly reevaluated their designs based on informal investigation of how students moved through the system

Improve badge impact with badge evidence > Research with & for Badges> Tracking link between badges and test performance.

At first, it was unclear to the BuzzMath team what sort of evaluation or research might be useful to feed back into the continued development of the project. With this goal in mind, connecting this program to the other measures of students’ achievement in the Common Core seemed the most relevant.

BuzzMath is interested in researching how earning badges is related to achievement on high-stakes testing. The project is gathering educators to help create the challenges within the BuzzMath curriculum to improve the connection between the badges and test-based measurement. This sort of data and analysis would be convincing to stakeholders that the BuzzMath program is effective.

BuzzMath feels that a research team outside of their team may be able to bring an impartial perspective to this kind of research. However, they note that it is very expensive to do this sort of work, which may require one specialist spending months investigating. BuzzMath hopes to figure out a way to make their system easy and inexpensive for outsiders to study.

Research for badges > Teacher surveys about process badges

After the badge system began operation, BuzzMath began to see that teachers were awarding the process knowledge badge differently. Some classes use them more than others. The team has collected a few anecdotes, but now plans to do surveys to see how teachers award and use these process badges in a more formal sense. They would like to know specifically if teachers like the set of existing process badges or if there are others that should be added. The DPD project records this practice in the “enacted” category because of the anecdotes collected, but not in the formal/continuing category yet, because no formal plans exist to systematize this research method.

Challenge vignettes

Have Your Business Model Ready (Recognizing)

BuzzMath noted one of their key lessons learned while building a badge system as, “Explore how your badge system will integrate with your current business model early in the project, especially if you are a for-profit company” (Project Q&A).
In our follow-up interview Jean-Philippe Choinière expanded on BuzzMath’s selection of a business model:

“When you are a new product, when you try to sell it, your business model can change quickly. In the last year, we changed the business model two times, maybe three. When you switch to only having paid users inside a program, then the number of badges issued will be very limited, and you need a big sales team to sell it. That was the first approach. Today, we are more on a freemium model, teachers can create classes, etc, students don’t have access to detailed solution or text to speech, teachers don’t have access to all the reporting.

BuzzMath reports that this approach is working well. Lots of classes have signed up, with more joining the freemium model each week. In this model, the badges fit well. In previous model, BuzzMath felt they weren’t as easy to integrate.

Resources

BuzzMath. (2012a). DML Stage 1 Proposal. Retrieved December 10, 2013, from http://dml4.dmlcompetition.net/Competition/4/badges-projects.php%3Fid=2852.html

BuzzMath. (2012b). DML Stage 2 Proposal. Retrieved December 10, 2013, from http://dml4.dmlcompetition.net/Competition/4/badges-projects.php%3Fid=3224.html

Choinière, J.-P. (2013a, March 22). Project Q&A With: BuzzMath. Retrieved from http://www.hastac.org/dml-badges/buzzmath

Choinière, J.-P. (2013b, November 20). DPD Follow-up BuzzMath Interview.

Choinière, J.-P., & Piecora, J. (2012, September 13). DPD Initial BuzzMath Interview.

 

 

 

Recognizing Principles Assessing Principles Motivating Principles Studying Principles
Specific principles: Specific principles: Specific principles:
About

Loves open education, Open Badges, free culture, Progress of the Useful Arts and Sciences, people-powered politics, and local food production. Coordinator for the badges Design Principles Documentation Project at Indiana University.

One comment on “BuzzMath
  1. Joshua Quick says:

    The Process Knowledge badges are also based on standards (like the Content Knowledge badges). They are based on the mathematical practices (http://www.corestandards.org/Math/Practice/). This focus on process standards provides additional validity for the use of badges.

    Additionally, the BuzzMath system continues to evolve. As of September 2015, there will be a complete, comprehensive badge system for the entire middle school program (grades 6,7, 8). This means the 7th grade teachers can look back at the badges the students have accumulated to easily assess their understanding of each concept from 6th grade. It gives a comprehensive view of a student’s math accomplishments from the prior year.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

Mozilla Open Badges
Digital Badges are web-enabled credentials of learning or accomplishment. -Erin Knight, director of the Badge Alliance
Badges contain detailed claims about learning, links to actual evidence of learning, and they're shareable over the web. -Dan Hickey, DPD Project Lead Investigator
To me, digital badges represent the bridge between formal learning & informal structures. -Alex Halavais, DML research competition winner
Open Badges can help people tell a verifiable story about their accomplishments. -Nate Otto, DPD Project coordinator
Regardless of where you start, it’s more than likely you’ll end up somewhere other than your intended destination. That’s okay. Systems are living things, and your badge system needs to be flexible. You must embrace a bit of chaos in its design. -Carla Casilli, Director of Design + Practice at the Badge Alliance