The information contained in this project appendix was gathered from the original project proposal that was funded in 2012 and interviews with project leaders through the end of 2014. This appendix and the overall design principles database from the DPD project does not reflect further evolution of the project or developments after the final interview that took place in 2014. As of the final interview, the Providence After School Alliance (PASA) had issued badges and developed a functioning badge system. Based on this information, we have classified the PASA badge system as an implemented (rather than a partial or suspended) badge system.
The Providence After School Alliance (PASA) partners with after school and extracurricular programs to offer quality learning experiences to middle and high school students. In their DML Proposal, PASA described their mission “to expand and improve quality afterschool, summer, and other expanded learning opportunities for the youth of Providence by organizing a sustainable public/private system that contributes to student success and serves as a national model” (Stage 1 DML Proposal). Specifically, PASA supports the operations and infrastructure of the local AfterZone network of organizations that administer learning experiences to middle school student and a similar network of organizations offering programs to high schoolers that includes a social and discovery-based website called The Hub.
PASA, the Providence Public School District, and the AfterZone Site-Based Management Agencies (SMAs) support the operations and programming of the AfterZone network. Employed in 20 cities, the AfterZone model is an after-school program for middle school students that provides youth with hands-on learning experiences with real-world applications.
PASA’s high school program’s participating sites provide expanded learning opportunities (ELOs) to students in high school, enabling community educators to assess out-of-school learning pathways. Expanded learning opportunities also open the possibility for high school students to earn academic credit for learning that takes place outside of school, supporting and strengthening students’ in-school learning. Each ELO is aligned to Common Core standards and approved by the Providence school district. As a part of this initiative, the Hub website provides students a place where they can connect their in-school learning with out-of-school experiences, finding ELOs, blogging, and uploading artifacts to showcase their learning in an online community. The program provides a referral and support tool, offering students resources to prepare for college and life after graduation.
To chart and communicate youth’s informal learning and achievements, PASA implemented the Pathways for Lifelong Learning badge system across programs. Bridging the AfterZone and Hub systems, PASA is “developing badge-supported learning pathways which will reflect, motivate, assess and validate the learning interests of youth through middle school, high school, to graduation, and onward” (Stage 1 DML Proposal). PASA launched the digital badge system to enable learners to communicate their extra-curricular experiences and accomplishments to school educators, staff, post-secondary institutions, and employers. Additionally PASA aims to provide a model for other cities looking to build badge systems in after school or expanded learning networks. They describe that an “online badge system that allows youth to capture their learning when and where it happens will also provide a systemic, replicable model for cities that want a systems approach to building engaged learners” (Stage 1 DML Proposal). While the Hub site is specific to Providence, PASA hopes the model represented by this badge system can serve as resource for other cities interested in implementing similar initiatives.
Evolving Practices and Design Principles
What follows is a list of practices as they relate to the general and more specific design principles in each category of practice. The headings name a (a) General Principle, (b) Specific principle, (c) Specific practice. The paragraphs below each heading detail the project’s (a) intended practice, (b) enacted practice, and (c) how that practice relates to the specific and general principles.
Design Principles for Recognizing Learning
PASA coordinated the badge system implementation across Rhode Island public schools, empowering youth to communicate their extracurricular learning and experiences to organizations, institutions, and potential employers. At the high school level, PASA awards academic credit for earning specific badges, enabling informal learning to translate to formal settings. The different programs vary in decisions made about what learning to recognize and who recognizes that learning. In this context, PASA placed measures to assess program quality to ensure rigor across programs. Additionally, the badges are aligned to relevant state standards and the national Common Core.
Align badges to standards > Use national or international standards > Standards alignment
Intended, enacted, formal
PASA intended to align afterschool programming and extracurricular activities to the nationwide Common Core State Standards and to relevant school and state standards. The project planned to map the badges to the standards of accredited institutions or agencies. In addition, PASA also intended to integrate a program assessment inventory method they had already constructed, the Rhode Island Program Quality Assessment (RIPQA) tool. They aimed to set consistent standards and ensure steady improvement of the quality of the programming in a list of “indicators” set out in the RIPQA. Equipped with this tool, badge issuers and program providers can further strengthen the educational rigor and communicate the quality of youth’s learning experiences across organizations by ensuring that participating programs keep improving.
PASA enacted their plans to assess program improvement with the RIPQA tool, and they noted how various expanded learning opportunity (ELO) programs aligned to elements of the Common Core standards for high school. Challenges emerged in standards alignment, as program partners were caught between conflicting perspectives and directions. Some ELO educators did not want to align program activities to the domain of formal schooling. Conversely, other educators were already covering the standards and noticed this only when the alignment process started. As the badging initiative progressed, a greater number of ELO programs have been aligning to standards (DPD Follow-up Interview). PASA aligned some badges to academic standards, specifically at the high school level. Other badges, such as those designed for middle school, are not likely to be aligned to content standards. When the program applies to be an ELO, the school district would review their application for approval and then determine how the program aligned to the standards (DPD Follow-up Interview). Further, some of the programs are aligned with industry, including areas of arts programming and environment groups. Coordinating with the schools, district curriculum leaders review the curriculum and standards alignment of the ELO programming to ensure a good fit with what high schoolers are learning in the classroom.. Moreover, there is a prospect for alignment and collaboration between programs in other cities for standardization.
This practice of “standards alignment” illustrates the principle of “align credential to standards,” ensuring that the curricula and badges are consistent with a set of standards that can translate across settings. In this way, badges can communicate specific competencies that are recognized, promoting clarity in the interpretation of skills represented by badges.
Award formal academic credit for badges > External collaboration
Intended, enacted, formal
As described in their initial proposals, PASA intended to collaborate with the Providence school district to align after school program curriculum with school-based standards so that the schools could award formal credit to youth for earning specific badges. All programs are run by after school program providers from the community.
In addition, PASA planned to integrate services from an existing partner, national data management tool YouthServices.net, with The Hub to automatically update the profiles of youth to reflect new badges that they qualify for in their field. The system also allows the badge opportunities and achievements to be shared as “news” items on the HubProv homepage and other online networks. The project uses YouthServices primarily to measure assets of the system, such as program attendance, students’ applications, school data, student identification, location of residence, and previous middle school attended. PASA set up a data feed from YouthServices to provide students with real-time updates on their online profiles. The data was embedded in YouthServices from the middle school network, and PASA worked with them to ensure that YouthServices worked with the HubProv platform.
In carrying out the badge system, PASA incorporates YouthServices.net to measure the attendance of students. A pending agreement with youthservice.net will allow PASA to automatically award attendance badges (DPD Initial Interview). The program collaborates with community partners to provide expanded learning opportunities. PASA collaborates with the Providence school district to enable high school students to earn academic credit for their badges at PASA. All programs are run by after school program providers from the community. They continue to use program participation of middle school students in PASA to guide program participation at the high school level.
As shown in PASA’s practice of “external collaboration,” the badge system demonstrated the principle of “award formal academic credit for badge” by enabling high school students to receive formal credit for earning certain badges. Through external partnerships, the project promoted the capacity to pave pathways that make an impact across school and organizational settings.
Use badges to map learning trajectories > Provide routes or pathways >Flexible learner pathways
Intended, not enacted, not formal
A central aspect of the learning process PASA intended involves uncovering which program categories students choose and the reasons for it. The badge infrastructure is categorized differently in middle school and high school programs. Whereas the middle school programs are categorized into arts and sports skills, the infrastructure at the high school level is defined in bigger categories including technology, business entrepreneurship, school and college, jobs, community and culture. These categories are then divided into finer-grained subcategories. As has been the case before adding badges, participating students would be able to look at available programs on the Hub website and apply directly to the program themselves.
Figure 1. PASA example of youth level badges
PASA planned to observes the choices of programs that students make and the reasons behind them, charting the next steps in the pathways, and intended to continue this monitoring while implementing the badge system, tracking students’ pathways.
As the system developed, the project worked toward shedding light on the user behavior of program participants in order to better understand the next steps in the pathways. PASA Deputy Director Alex Molina explained, unpacking a cluster of related functions of the diverse options students pursue:
We’re using that data to make sure we’re giving [students] the right tools to make informed decisions about what they’re interested in. Once they get to high school, they have an idea of what they’re interested in… The data goes online. Young people can then see what they’ve done in the past and what they can do in the future. Even more, we’re working hard with the district and the Office of College and Technical Careers to make sure [students] understand exactly the long term impacts, and how an adult could come in and say, ‘Hey, it looks like you’re really interested in technology. You’ve done these programs in middle school; this in high school. You have these skills. Think about X.’ That’s for youth, and what we hope to do–this is in the planning process with the district right now–is to personalize those experiences, so the next step is as young people go to those pathways. [As] they get those badges, it would unlock those internships and unlock a lot of opportunities” (DPD Follow-up Interview).
The visibility of students’ pathways would allow for possibilities that combine to improve the interaction between students and the various programs they participate in, would also enhance their ability to choose a pathway through different middle and high school experiences that is fun and educational.
This practice shows how PASA would make various learning pathways available in the badge system, as an implementation of the principle “use badges to map learning trajectories” As it stands, the project has yet to continue development on using badges to map out learning trajectories. However, program selection by students offer insight into the steps that they take in ultimately charting the course of their learning pathways.
Recognize educator learning > Students award badges to educators
Intended, not enacted, not formal
PASA also intended for students to award badges to educators and to create the badges themselves. The project would like to enable students to award badges to their community educators and program providers, offering recognition for strengths. PASA said that they planned to facilitate “design session with students to identify the kinds of information that would be valuable and the type of recognition that would be valuable in this relationship” (DPD Initial Interview). In this way, the community can arrive at developing badges that would meaningfully recognize and offer proper credit to educators for their work.
The project is developing the capability for youth to award badges to educators. In the badge system, the only implemented badges are those PASA awards to youth. This practice has not yet been enacted yet, but it remains a possibility on the horizon. Alex Molina raised a point with respect to communicating the value and meaning of a badge within the urban community. He explained:
One thing that we’ve discovered is that young people are really attracted to the experience, and they’re connected to adults, and that’s why we’re trying to ensure there’s a currency behind badges and that it opens doors, because we’re scared that badges for young people and the community doesn’t mean much. . . However, they keep talking about the experience and the connection to adults, so for that reason, what we need to do for the next six months of the year is to make sure, if we do give badges, to open something up and it means something else. Because right now, they’re like “Great, it’s a badge. What can I do with that?” So we need to make sure that it is of value, and we haven’t done that yet, because young people really don’t know the value of a badge, and that’s something we’re working hard to fix (DPD Follow-up Interview).
As Alex described, the main obstacle to implementing higher level practices is that students need to understand how badges work on a personal level before they can contemplate doing more advanced things like awarding badges themselves. As PASA introduces badges to the community, the goal they described for moving forward for this process was “showing the value” of earning a badge and how it could translate into access to additional opportunities and experiences. Through this lens, PASA hopes to reach the point of implementing advanced badge practices like students awarding badges to educators.
This intended but not yet enacted practice represents the principle “recognize educator learning” with digital badges and would provide distinctive credentials for participating adults, but much culture-building work remains before implementing this could become an option.
Have experts issue badges > Credentialed via accredited entity and community > Accredit program providers to award badges
Intended, enacted, formal
As described in their proposals, while central PASA administrators would be the primary badge awarders, the project would also incorporate the capacity for distributed program providers and participating young people to award badges. In this respect, PASA hoped to establish a threshold for an individual to qualify to issue badges.
Besides this, PASA also intended for educators to receive badges in the badging system, which would involve steps in the process of becoming endorsed program partners for PASA. The project discussed the creation of digital badge pathways to receive endorsement, describing the process of manually constructing it and understanding what is involved in building this system (DML Stage Two Proposal). The inclusion of this process strengthens the way the badges are validated.
As enacted, PASA issues all the badges in the program so far. The project has ongoing work on designing peer to peer, peer to instructor, or instructor to student badges for HubProv, the online platform designed for high school youth. As it stands, students demonstrate their skills on the HubProv website and present in a final demonstration session before they can receive a badge (DPD Follow-up Interview).
The practice of accrediting program providers to award badges in PASA’s system is connected to the principle of “have experts issue badges,” in which the project presents a case of “credentialed via accredited entity and community.” Before a badge is issued, a panel of community judges review and assess the learner artifact, and programs are held to the standards that PASA established. In addition, the intent will be for badge awarders outside of PASA’s internal staff to qualify first before they can award badges, though this has not been implemented.
Design Principles for Assessing Learning in Digital Badge Systems
PASA aligned the badge system and curriculum to relevant standards, leveraging the Rhode Island Program Quality Assessment tool to ensure consistency of high quality experiences across programs. At the high school level, students can receive formal credit for earning badges based on their participation and achievements. Besides using a computer scoring system, the project also included peers and a panel of community judges in their plans for the assessment process. Peers were intended to play an important role in assessing the learning of one another, and rubrics have been developed for badge issuers to ensure that the assessment process is consistent with specific criteria.
Align assessment activities to standards: create measurable learning objectives > national/state standards > Standards alignment and earning credit for participation
Intended, enacted, formal
As intended, PASA aimed to align the badge system and assessment process to relevant content standards. Because the after school ten week programs directly results in elective credit, PASA has begun to build ways of ensuring that their assessments meet schools’ standards for awarding credit. By integrating the RIPQA, the project aimed to include program AfterZone and high school program providers in a continual process of quality improvement. The project also intended to align badges in high school programs to the Common Core State Standards. Many programs have applied to be expanded learning opportunities for high school youth, and they would typically outline learning goals and share their curriculum, which PASA would analyze to determine how it aligns with the Common Core State Standards (DPD Follow-up Interview). This process is intended to ensure that a strong case can be made to participating schools that students should earn academic credit.
The enacted practices include the alignment of badges to standards at the high school level. The high school programs are all aligned to the Common Core State Standards and relevant school and state standards, enabling them to earn credit that transfers to their regular classes for participating in after school programs. For different programs, the curricula are aligned to distinct standards depending on the field or area. PASA Deputy Director Alex Molina described,
It begins with standards alignment…[and] culminates when their teacher recommender, their community provider, and panelists come together and assess if their learning has happened. We give them the goals and outcomes, the standards, they review it, and they’re looking, especially for the teachers and community partners, from point A where the students started to the end, if they have hit those points. There’s other things they need to do besides showing those skills. They need to come to the program at least 80% of the time. They need to blog, and to some extent they need to defend their thesis, what have you learned. Those adults come together, they have the rubric, common assessments, and they know what they’re looking for with those things…. At the end, everybody comes to a consensus that the young person met every benchmark: “Here’s proof, here’s evidence. He or she should get credit and a badge.” (DPD Follow-up Interview)
PASA exemplifies the principle of “align assessment activities to standards: create measurable learning objectives” through their practice of “standards alignment and earning credit for participation.” Aligning curricula to standards, PASA illuminates the skills and knowledge to be assessed and justifies the value of the badges and students’ learning to the schools who award them academic credit.
Enhance validity with expert judgement > Use a combination of human and computer experts > Validation by a computer scoring system, peers, and a panel of community judges
Intended, enacted, formal
PASA intended for program providers to set benchmarks and nominate youth for badges. As the process of benchmarking and nomination happened, YouthServices.net could log student progress along a pathway. When students qualify for a badge within their fields, their profiles will be automatically updated to reflect their accomplishment (DPD Follow-up Interview). The project planned to develop more opportunities for peers to use rubrics in assessing blogs about specific skills and challenges. The project envisioned youth as integral to the assessment process in offering feedback to peers.
In their badge system implementation, PASA is currently still in the process of developing peer assessment mechanisms and practices. Alex Molina reported that HubProv serves mainly as a place for students to interact but not truly assess one another, so this functionality is not used to currently award digital badges (DPD Follow-up Interview). The project faces the task of building a culture of use, as it introduces badges to the student community and communicates the value of badges.
As carried out their badge practices, PASA employed a computer scoring system with YouthServices.net to validate certain criteria for badges, including students’ attendance data. The badge issuing process was intended to includes an automated component, and the project is continuing to develop peer assessment and build in assessment by a panel of community judges based on the use of rubrics. As enacted, badges are awarded solely on students demonstration day at the end of each ten-week after school program, though attendance information and blog post data collected online is used in their assessment.
The badging practice of “validation by a computer scoring system, peers, and a panel of community judges” exhibits the principle of “computer experts” and specific principle “use a combination of human and computer experts.” Educators and judges from the community achieve certain qualifications before engaging in the assessment process, appraising student performance based on carefully developed rubrics.
Use rubrics > General rubrics > Rubrics are used to assess learning
Intended, enacted, formal
PASA integrated rubrics in their plans for the assessment of students’ performances, adopting and modifying existing rubrics from Ohio, New Hampshire, statewide material from Rhode Island. PASA intended to further develop and improve the rubrics for better alignment and consistency as they progressed.
As this practice was enacted, PASA employed detailed rubrics to assess artifacts and learning. These rubrics are being adopted and modified from several existing rubrics, including the abovementioned from Ohio, New Hampshire, and Rhode Island. The rubric has been modified and updated to meet student’s learning goals and needs. By adjusting the rubric, PASA worked with the district and community providers to ensure that the rubric is readily interpretable, enabling panels to form a clear basis for understanding the outcomes to assess. Specifically, the students go through a “final demonstration session” before a panel of judges before they can earn a credit-bearing badge (DPD Initial Interview). The program duration is at least ten weeks for credit-awarding ELOs at the high school level. When youth present their work after ten weeks of the program, they are assessed based on a rubric by a panel of school and subject specialist community members, including entrepreneurs, teachers, and peers. The rubric is used to assess whether the student has demonstrated the achievement of learning goals and expectations, before awarding students badge and credit.
Illustrating the principle “use rubrics,” PASA implemented rubrics developed to offer a guide to assess student learning and achievements and ensure assessment consistency for high-value badges that also indicate conferral of school credit.
Use formative functions of assessment > Peer feedback > Assess program quality, accredit experts and issue program level badges
Intended, not enacted, not formal
The project intended to develop a common assessment method based on the RIPQA to ensure that participating programs met the quality levels PASA wanted to achieve, and more specifically, that they keep making progress on many “indicators” of quality (RIPQA). They envisioned a “program endorsement” badge for program providers based on the achievement of progress measured by the RIPQA.
Figure 2. PASA planned to endorse participating program providers
PASA leveraged the RIPQA in assessing program quality in its badge system implementation. The RIPQA is based on a set of indicators for what a quality assessment program would look like. These indicators are organized into five core areas: (1) Health, Safety, and Environment, (2) Relationships, (3) Programming and Activities, (4) Staffing and Professional Development, (5) Administration. As a whole, the assessment method provides a set of guidelines on aspects of a quality program.
This practice fits under formative functions of assessment for PASA’s effort on ensuring that programs make progress toward high marks in the RIPQA. This is not a practice for providing formative feedback for students, but there are to be badges issued under this practice. The project is still continuing development on badges for program providers, primarily issuing badges to students. However, PASA’s practice of implementing the RIPQA demonstrates the process of employing feedback toward the ongoing quality of the programs. As fo the DPD followup interview, this practice was not enacted.
Design Principles for Motivating Learning
PASA employs badges as a way to map student progress, attendance, and achievement of learning outcomes in a variety of after school activities. Badges were intended to be awarded not only by program staff but also by peers, though peer awarding has not been implemented to date. At the same time, the program includes the caveat that the mere addition of a badge to an otherwise unmotivating activity does not automatically increase student motivation; one story detailed below among PASA’s challenges that system designer Kerri Lemoie shared was about a student who did not find the badge a motivating force to complete the activities (DPD Initial Interview).
Give badges for small accomplishments to hook in learners > Acknowledge continuation of interest
Intended, not enacted, not formal
The project intended to initiate learner interest and sustain student engagement by awarding badges for subskills of specific subjects, created by the specific programs and tailored to the learning pathways through which each program led students. PASA explained that the badges “reflect and enhance ongoing behaviors in middle school programs—passion, perseverance, etc.—[and] will act as goals and guides along learning pathways extending through high school and beyond” (DML Stage 1 Proposal). As described in the project proposal, badges can motivate learners to persist in a specific area by reflecting a continuation of interest and promoting their mastery of specific skills or subjects. Further, the experience is rooted in youth choice. PASA explained, “Some youth explore multiple interests or take part in leadership opportunities, but many repeatedly take the same “types” of programs, reflecting an interest in a particular learning pathway. Hub Director Damian Ewens explained:
Imagine a kid takes two or three STEM-related programs. What we’re defining right now is that if they’ve taken three of them in a row, we’re using that as a proxy for interest. By the time they hit eighth grade, if they take three (and attendance expectations and such are met for the program), they would have the opportunity to be the first to apply to our STEM-related programs in the high school level. (DPD Initial Interview)
As students decide on which program to take, they play a central role in paving their own trajectories of learning and charting out their own pathways. Badges could act a means of demonstrating students’ interests and ability in specific content, functioning as milestones in students’ learning and acquisition of skills.
The project has not carried out the practice yet, but hope to look at the skills in the Spring 2014 (DPD Follow-up Interview). They are planning to award the badges on the youth profile, working with the providers to identify where in the program young people would practice granular skills that could serve as the basis for minor badges. PASA aims to plan the skills to recognize together with the school district and with employers in Rhode Island, who are looking for future employees who are able to communicate effectively, persevere, solve problems, and work in a team.
This intended practice has not been enacted yet, but forms an example of the principle “give small badges to hook in learners,” because PASA uses badges to encourage continuing interaction with particular learning pathways. The badge system would then provide guideposts to youth and encourages further growth as youth continue to develop and pursue their interests until they complete 10-week programs and earn the full-value PASA badges.
Provide incentives > New activities and Internships > Give students privileges and new opportunities
Intended, not enacted, not formal
PASA intended to open up privileges to students for earning certain badges. Badges are intended to motivate students by providing access to internships and other resources. The project proposal explained, “As 8th graders and high schoolers attain certain badges they would receive special privileges—8th grade-only programming, paid internships through The Hub, personalized tutoring, etc. As the badge ecosystem develops, opportunities and privileges would expand significantly, supporting healthy behaviors, college access, leadership development and more” (DML Stage 1 Proposal). The projects offers privileges to badge earners to promote youth development outcomes and foster their learning.
As of the DPD followup interview, no formal grant of privileges based on the earning of badges has been implemented. However, badges would act as a motivator by helping provide access to higher-level programs and opportunities. As youth earn specific badges, they will look more qualified to those who approve their application to more advanced ELOs that can then further advance their knowledge and skill mastery. In this way, PASA can indirectly offer them opportunities to gain additional experience in a certain field based on their previous experience (DPD Follow-up Interview). However, plans are moving forward to help young people leverage their after school experience to open new opportunities. Alex Molina reported in December 2013 that through partnering with the school district Office of College and Technical Careers that “the next step is as young people go to those pathways, they get those badges–it would unlock those internships and unlock a lot of opportunities, so we’re at maybe about six months to a year to finalizing that” (DPD Follow-up Interview).
PASA illustrates the principle of “provide incentives” through their practice of giving access to internships and other opportunities, both through informal current practices and their continuing ambitions. These privileges can advance youth’s competency development and open up further ways to apply their learning.
Utilizing different types of assessment > Peer > Feedback on peers’ blog
Intended, enacted, formal
The DPD project noted that PASA’s intended practice of peers endorsing each other’s work was aimed at motivating participation. As youth are encouraged to participate in peer assessment, they can provide one another with peer feedback, which the project intended would then unlock additional privileges, such as new learner activities, within the site.
As enacted, PASA gives capacity for peers to comment on one another’s blogs on HubProv. The project plans on developing the possibility for peers to give a “+1” to indicate a good review or feedback. Additionally, this has the potential to lead to additional privileges with what they can do on the site. While there is student communication on the website, however, PASA would like to increase the volume of interaction between students in providing feedback (DPD Follow-Up Interview).
As illustrated through this practice PASA’s badge system provides an example of the principle “utilizing different types of assessment.” One of the possible effects of students’ interaction on the Hub is that they may be further motivated to learn because of peer feedback and increasing social connections.
Set goals > User-created badges > Users help create badge graphics
Intended, enacted, formal
PASA intended for users to take part in creating the images that are used as badges. By obtaining student input, PASA hoped to enable youth to take greater ownership of their learning experiences and contribute to the program design.
As they built the functioning program and badge system, PASA used input from youth to design the AfterZone brand for middle schoolers. The Hub Youth Team co-created the Hub programs, logo, website and collateral materials. AfterZone and Hub students took part in the graphic design of middle and high school badges with a professional designer from Embolden. The after school system also held badge design competitions and incorporated elements of AfterZone and Hub branding. Alex Molina confirmed this as an important continuing practice of the PASA system, saying “young people are the customers, like any organization you’re going to listen to your customers to inform product design and delivery” (DPD Follow-up Interview).
Illustrating the principle of “setting goals,” PASA implemented the practice of “users help create badges.” By including the input of youth, the project enables them to take greater control of their learning experience, which may increase their connection with the credentials and motivate them to learn and participate more.
Set goals > Display of goal trajectory > Badges mark completed milestones towards goals
Intended, not enacted, not formal
PASA intended to measure students’ progress toward growth. PASA intended to build Progress and Recognition badges to acknowledge achievements along a pathway from which users are awarded points. After users have reached a number of points required by the benchmark, PASA planned to award a Progress badge. This element of the program aimed to recognize skills or competency-based achievements along a path toward a PASA Recognition badge. PASA intended for the badges to recognize students’ efforts en route to a larger goal.
The project works to ensure that badges can connect students’ in-school and out-of-school learning. Youth are informed of choices through discussions with teachers and also on the online portal. On the youth profile, students would see the badges that have already earned and additional badges that they could earn, finding out about skills and competencies that they can continue to develop and grow. PASA has not yet started to award Recognition or Progress badges, but they have been issuing Completion and Skill badges, which students earn during the final demonstration day. Completion badges are awarded for completion of the program, and Skills badges are awarded for mastery of certain skills achieved by the student. Students can see which badges they have earned and which others are available. As described by PASA Deputy Director Alex Molina, “the big picture includes working with the district to create a portfolio that connects students’ in-school and out of school experiences, providing youth with the tools to create their own personal learning pathways” (DPD Follow-up Interview). By showing their badges and achievements, then, students would be able to access other opportunities and apply their learning to institutional or professional contexts.
PASA demonstrates the principle of “setting goals” through their practice of “badges mark completed milestones towards goals.” The project propels learners to persist and grow their abilities further by awarding badges that mark the milestones students reach in developing their skills and competence.
Build outside value for badges > Badges as academic credit > Link badges with formal academic credit
Intended, enacted, formal
The badge development effort offers formal academic credit for the badges that are earned. The DPD team inferred that this practice likely boosts the motivation of learners to earn badges, for they can satisfy school requirements at the same time. In addition, it moves them forward toward completing high school. The practice of connecting badges with formal credit sheds light on the principle “build outside value for badges” and the sub-principle “badges as academic credit.” By developing this principle, PASA builds pathways that enable youth to cultivate their skills while they advance toward graduation.
Utilizing different types of assessment > Computer > Computer-based assessments
Intended, enacted, formal.
PASA employed computer-based assessments that verified specific criteria for a badge through Youthservices.net. As inferred by the DPD team, the automated nature of the scoring system can allow for more immediate feedback, which strengthens the motivation of learners. The project can also derive metrics from the participation of youth on the HubProv website, enabling PASA to adjust their program offerings to connect with learners’ interests. The practice of “computer-based assessments” illustrates the principle “utilizing different types of assessment” and sub-principle “computer.” Through the implementation of this principle, the badge system verifies the learning and participation of youth, sustaining their learning and activity on the platform.
Design Principles for Studying Learning
PASA’s research practices include collecting data on program attendance and user engagement. The type of choices students make can inform further design and iteration of the badge system and map out their learning pathways. As PASA carries out assessments of its programs, the evidence gathered from these assessments can be applied to make adjustments or improvements to the learner experience.
Improve badge impact >Research for badges > Formative program assessment
Intended, enacted, formal
The project intended to develop a common assessment across projects. PASA wanted to implement program assessment to ensure the rigor of students’ experiences and accomplishments. Specifically, the RIPQA tool includes a set of “indicators” from which educators can assess the quality and improvement of PASAs varied participating programs and strengthen the consistency of high quality from program to program.
PASA enacted the practice of using the RIPQA to conduct regular program assessments. The program offers aligned professional development and holds providers to high attendance and enrollment benchmarks. PASA leveraged the assessment evidence to serve formative purposes and enhance the quality of the program experience. The project will employ program assessments to conduct research and iteratively refine the program experience, as noted in the formative assessment practice “assess program quality, accredit experts and issue program level badges.” The project has adjusted the training as they implemented the badge system. PASA Deputy Director Alex Molina described that, in middle school and eventually in high school, they will work on another assessment tool for young people (DPD Follow-up Interview). For middle school, the method is called the Survey of Academic Youth Outcomes Staff Survey (SAYOS) by the National Institute of Out-of-School Time (NIOST), focusing on youth outcomes and the growth of students over time (DPD Follow-up Interview). The tool is not employed at the high school level, and the project is working with NOIST to explore assessment methods and showcase student growth.
The principle of “research with badges and for badges,” referring to efforts to improve badge systems, is embodied in PASA’s practice of “formative program assessment.” Through assessments of program quality, PASA uses the data and evidence to inform the further design and development of the user experience.
PASA faces challenges in motivating learning as well as across the strands of motivating and recognizing learning. The project encountered the challenge of building a culture of use within the urban community. It tried to get youth to recognize the value and meaning of badges, connecting them to opportunities and experiences. PASA believes that its badges in and of themselves are not motivating if attached to an activity that is not inherently motivating. Moreover, PASA works with communicate providers to get across the potential of badges and to carry out its program effectively.
Infrastructure challenges and access to technology
PASA encountered challenges in terms of infrastructure constraints and district-wide access to technology. Alex Molina, PASA Deputy Director, stated “We work with schools that don’t even have computers or laptops, or there’s no wireless access, and young people go home where there’s no access to technology either…. the reality is that we’re working in an urban environment, and [on] the idea of badges, a lot of urban centers or cities are not going to be ready five, ten years down the line. The school district does not even have the infrastructure to do this” (DPD Follow-up Interview). There are considerable difficulties in implementing the badge system if the school infrastructure cannot support it. Molina pointed out, “What I’m scared of is if we don’t fix those structural issues, badges may become another assessment for the haves that have the access to technology. They know how to leverage complex networks, and urban youth are going to get left behind, so that’s one of the challenges we want to address: that badges don’t become another line in the sand that says you can’t access this because your school’s not providing this” (DPD Follow-up Interview). The project met the challenge of ensuring that students have access to opportunities and experiences even with technology hurdles.
Badges do not overcome unmotivating activities by themselves (Motivating Learning)
A compelling point is that badges in and of themselves do not motivate. Kerri Lemoie said, “One of my students didn’t do his weekly task. Another student got a badge for doing his, and I said to the first student, ‘But you’ll get this badge!’ And he still hasn’t done his task. He just wasn’t interested in doing the task. The badge didn’t even matter. It didn’t matter how cool it looked or that his friend got it. He just didn’t care about it” (DPD Initial Interview). This illustrates that offering students a badge for performing an activity that is not intrinsically motivating does not automatically translate into motivation for that task. This challenge relates closely to building a culture of use around badges, as described below.
Building a culture of use in recognizing and motivating learning
A challenge surfaced in building a culture of use in the urban community for recognizing and motivating learning with badges. PASA Deputy Director Alex Molina described, “We’re learning that badges are a cool thing for us adults or those who work with technology, but for a lot of urban youth, it’s not there. How do you convince young people that a badge has a currency when they’re facing other issues–when they’re at a failing school, when they’re competing for jobs with adults? And until a badge gives them a job or really gets them to college, it has no value to them, especially for urban youth, the value is engag[ing] in high quality experiences with an adult that cares” (DPD Follow-up Interview). In the communities, youth face a number of immediate and pressing concerns, and they may not recognize the value and potential of a badge. Molina describes the structural challenges in recognizing the value of the badge and its capacity to effect change and open up concrete pathways for learners to pursue (DPD Follow-up Interview – December 20, 2013). He explained, “Young people sign up for programs not because they’re going to get a badge. They sign up for programs, because they get to work with a cool adult. They get to participate in something that school’s not giving them” (DPD Follow-up Interview). Often, the reasons that youth sign up are based largely on the experiences they gain rather than for the badges they could earn.
PASA faces the challenge in communicating this value to students and building up a culture of use, because they are managing a citywide network of programs, permitting them less face-to-face time with individual students. Molina stated, “From PASA’s perspective, also, we’re not a program, we’re a system that enables people to access those [opportunities], so a lot of times, we don’t have a one-on-one with the student. We allow programs to work with students, so that’s been a difficult thing” (DPD Follow-up Interview). The project works with the programs to deliver the content to young people; they do not tell the programs how to teach the subject, but rather how to work with students in implementing the protocols.
Expanded Learning Opportunity Programs. (n.d.). The Hub. Retrieved January 17, 2014, from http://hubprov.com/elo-programs
Providence After School Alliance. PASA Program Level Badges. (2012). Pathways for Lifelong Learning DML Stage 2 Proposal. Retrieved January 17, 2014, from http://dml4.dmlcompetition.net/fastapps.dev.hri.uci.edu/files/1414/files/PASA_program_level_badges.pdf
Providence After School Alliance. PASA Youth Level Badges. (2012). Pathways for Lifelong Learning DML Stage 2 Proposal. Retrieved January 17, 2014, from http://dml4.dmlcompetition.net/fastapps.dev.hri.uci.edu/files/1414/files/PASA_youth_level_badges.pdf
Providence After School Alliance. Pathways for Lifelong Learning: DML Stage 1 Proposal. (2012). Digital Media and Learning Competition. Retrieved January 17, 2014, from http://dml4.dmlcompetition.net/dml4.dmlcompetition.net/Competition/4/badges-projects.php%3Fid=2773.html
Providence After School Alliance. Pathways for Lifelong Learning: DML Stage 2 Proposal. (2012). Digital Media and Learning Competition. Retrieved January 17, 2014, from http://dml4.dmlcompetition.net/dml4.dmlcompetition.net/Competition/4/badges-projects.php%3Fid=3227.html
|Recognizing Principles||Assessing Principles||Motivating Principles||Studying Principles|
|Specific principles:||Specific principles:||Specific principles:|