Computer Science Student Network (CS2N)

Preface

The information contained in this project appendix was gathered from the original project proposal that was funded in 2012 and interviews with project leaders through the end of 2014.  This appendix and the overall design principles database from the DPD project does not reflect further evolution of the project or developments after the final interview that took place in 2014. As of the final interview,  the Computer Science Student Network (CS2N) had  was still developing its badges and badge system. Based on this information, we have classified the badge system as a partial (rather than an implemented or suspended) badge system.

Summary

Computer Science Student Network (CS2N) is a collaboration between Carnegie-Mellon University (CMU) and the Defense Advanced Research Projects Agency (DARPA)  intended to develop educational systems for computer science and other STEM fields for diverse learning groups ranging from middle school students to adult learners and hobbyists.  CS2N intended to develop a highly-structured badge system that developed a guided pathway for learners to acquire skills in robotics, computer science, and related STEM content areas.  These skills were assessed and developed through the application of artificial intelligence systems designed to track and develop learners’ progression through the program.

CS2N developed a badge system that incorporated multiple paths to different competencies that utilized sophisticated artificial intelligence software in the assessment and development of learners’ skill competency.  The degree of this automaticity differed from its initial conception due to difficulties in implementing such sophisticated software.  Similarly, the diverse range of industry standards used as credentialing for certification and motivational factors proved difficult to unify into a coherent system.  CS2N is continuing to refine their badge system and is in the process designing a more complex badge system (DPD Follow-up Interview).

Evolving Practices and Design Principles

 What follows is a list of practices as they relate to the general and more specific design principles in each category of practice. The headings name a (a) General Principle, (b) Specific principle, (c) Specific practice. Below each heading, we indicate what stages this principle appeared in the project: as an (a) intended practice, (b) enacted practice, or (c) formalized continuing practice.

Design Principles for Recognizing Learning

CS2N used its existing badge system and partnerships to develop a system that rewarded badges that displayed competence in industry valued and mandated skills in computer science, robotics, and related fields.  Badges were designed to display these skills to learners and consumers within the robotics and computer science community and were developed in alignment to relevant engineering standards of practices in and between specific STEM communities.

Use badges to map learning trajectory > Provide routes or pathways > Badges mark progress.

Intended, enacted, formal.

CS2N developed their badge system to incorporate numerous small badges in order to display the development and progress of each individual student while also providing potential motivation by the fact that they were making progress (DML Stage 1 Proposal).  The intention was to develop parallel pathways that would allow students to progress from entry level abilities in programming and engineering skills to industrial level or related, certification of ability.  Although CS2N developed a system that displays a pathway for student progress, the process changed as it developed . As detailed in Figure 1, badges became more aligned to certification standards of industry certification processes, and this alignment was reflected in the progress of students within the CS2N framework (DPD Follow-up Interview).

CS2N 1 Figure 1. Badges shown along a career pathway. (DML Stage 2 Proposal)

Use badges to map learning trajectory > Level badges > Badges for proficiency levels

Intended, enacted, formal.

As stated, the CS2N badge system incorporated a progression of badges that reflected ability in computer science and engineering skill sets.  This progression also mapped the degree of proficiency of individual students in terms of badge levels in computer science, robotics, engineering, and general critical thinking ability.

Recognize educator learning > Grant instructors authority to award badges

Intended, enacted, not formal

CS2N intended the development of an instructor badge that would represent technical as well as pedagogical achievements for individuals to exhibit in a range of activities (DML Stage 1 Proposal).  An instructor who earned the teaching badge would have some administrative access to the CS2N network that would enable them to approve a student’s achievements for a particular badge.  The instructors were intended to have the authority of the affiliate organizations issuing final certification for badges, which would provide additional legitimacy to the issued badges (DML Stage 2 Proposal).

CS2N developed this intended design into a two leveled professional development program that required 36 hours for completion through both online and face-to-face interactions and a final examination that served as a verification of technical and pedagogical ability.  As the project progressed, the instructor endorsement process was not included as a functional aspect of the design due to personnel limitations. (DPD Follow-up Interview).

Seek external backing of credential > Externally endorsed > Align to industry standards

Intended, enacted, formal

The CS2N developed its badge system to incorporate certification programs with existing partnerships of engineering and computer science certification programs.  As stated, the badge system progressed from a small badge constellation to full certification with  these programs. (DPD Follow-up interview).  These partnerships were incorporated into the development and design of the curriculum towards certification and therefore enabled development of externally recognized credentials of robotics and engineering expertise. .This development of badges as credentials for industry valued skill sets was facilitated by the prior existing parterships CS2N had with the robotics and computer science industry

CS2N 2

Figure 2. Progression of CS2N badges from small up to partner-sponsored industry certifications.

Align badges to standards > Use standards internal to community > industry aligned   standards

Intended, enacted, formal.

As stated, CS2N developed a badging system that incorporated relevant industry standards for computer programming, robotics, and engineering practices.  Badges were intended and developed to display individual’s competency with these industry-mandated standards.

Use badges as a means of external communication of learning > Badges represent multiple standards

Intended, enacted, formal.

CS2N had intended to use badges as stand-ins that would communicate a student’s proficiency with engineering and robotics skill sets.  However, the numerous agencies that had aligned themselves with CS2N often employed standards internal to their communities that could not be easily synthesized into a single badge (DPD Follow-up Interview).

In response to these diverse standards, CS2N developed multiple badges that would facilitate recognition of a student’s skill within a particular class of skills represented in particular standards valued by the various agencies allied with CS2N.  Badges were designed to represent and link to the relevant standards (DPD Follow-up Interview).

Have experts issue badges > Credentialed by external accredited entity > Badges awarded by certification agencies

Intended, enacted, formal

CS2N conceived a badge system in which instructors could award badges credentialed by certification entities.  As the badge system developed, the instructors in the CS2N system allowed instructors who had received professional development training through CS2N to issue and acknowledge the achievement of a badge. The current state of this function is currently limited in its applications. As of the DPD follow-up interview, however, the CS2N team intends to extend this function towards a more general, reusable aspect of the badge ecosystem (DPD Follow-up Interview).

Design Principles for Assessing Learning in Digital Badge Systems

CS2N developed a highly-automated system to assess student learners’ competency and skills with robotics and engineering practices.  These automated systems were intended to provide information and feedback on students’ skill levels and mastery of relevant industry- mandated standards.  Although significant automation of assessing learners has occurred within the CS2N system, the degree of automated assessment did not meet the original goal envisioned by the CS2N team.

Use mastery learning > Judged by computers > Assessment through automated programs

Intended, enacted, formal.

CS2N initially proposed a badge system that would utilize computer assessment of performance that would award significantly large-scale badges for achieving mastery of central concepts in computer programming, robotics, or engineering skills (DML Stage 1 Proposal).

As the project developed, there was considerable development towards computer- based judgment  of engineering skills.  CS2N utilized software to track and monitor the progress of students towards skill mastery. Due to limited personnel, however, CS2N was unable to automate the process of assessing skills to the degree initially intended.  As of the DPD follow-up interview, CS2N was in the process of pursuing new avenues for this method of automated assessment processes.

Enhance validity with expert judgment > Use AI totors  and use computer scoring systems > Automated assessment

Intended, enacted, formal.

A major intention of the CS2N program was the development of artificial intelligence tutors and computer scoring software.  CS2N initially intended to incorporate a large scale AI tutor and scoring process into the assessment and awarding of badges.  As the project developed, technological limitations and the difficulties inherent in designing intelligent software to assess large scale projects and performance required intelligent assessment software to be scaled to small assessment and micro-assessment processes (DPD Follow-up Interview).

Use leveled badge systems > Competency levels > Badges awarded for particular skill levels

Intended, enacted, formal

As stated, CS2N developed parallel  leveled badge systems that would allow for the assessment of particular skill levels from limited skill sets to skill mastery in engineering practices that would enable industry certification. Small scale badges were awarded through computer based assessment processes whereas large scale or certification level badges were awarded through additional assessments and data incorporated into small scale badges.  (DPD Follow-up Interview)

Use formative functions of assessment > Peer feedback > Peer judgment of competitive submissions

Intended, not enacted, not formal

CS2N planned for the development of badges for peer assessment through the judgment of peers’ submissions to programming competitions.  Additionally, CS2N planned to implement functions that acknowledged peer feedback that would lead toward a meta badge (DML Stage 1 Proposal).

As the badge system developed, peer feedback was not finalized into badges due to difficulties in determining how peer feedback contributed to level of performance in competitions. Although the system functions for peer feedback were  developed, it has not been implemented in the current badge system. As of the DPD follow up interview, CS2N acknowledged the potential for future implementation of peer feedback functions within the badge system, though no current plans have been proposed.

Design Principles for Motivating Learning

In order to provide significant motivation to learners, CS2N developed a badge system that aligned itself to relevant engineering standards.  The inclusion of relevant standards was intended to provide a path toward certification in particular skills and technologies required for careers in engineering and computer science.  Achievement of badges was a significant step in being certified in industry mandated and desired technologies and, therefore,  recognition as a member of professional engineering communities.

Build outside value for badges > Real-life application of knowledge > Develop industry mandated skills

Intended, enacted, formal.

CS2N developed  a badge system that aligned to industry based standards in computer programming, robotics, and engineering practices.  As with the assessment of student skills, the alignment towards industry standards was intended to develop a value for the badges within the engineering community as signifiers for particular skill levels relevant to the community practices (DPD Follow-up Interview).

Set goals > Display of goal trajectory and provider determined goal trajectory > Display path towards industry certification

Intended, not enacted, not formal

CS2N initially conceived a badge system in which learning trajectories would be set and influenced by industry standards and the paths to achieving those standards would be identifiable to learners (HASTAC Q&A).

As the badge system developed toward a more skill-focused system, the process of developing a badging ecosystem that incorporated multiple industry standards became exceptionally difficult due to the diverse and often specific nature of many industry standards (DPD Follow-up Interview). CS2N is currently developing  pathways that  display  significant achievements for these engineering communities.

Give badges for small accomplishments to hook in learners >  Award badges for beginning learning trajectories

Intended, enacted, formal.

As described in Figure 2, CS2N initially planned to incorporate a badge system that would produce  interest in  robotics design in novice learners by developing numerous small accomplishment badges. As CS2N developed the badge system, the conception of the learning trajectory based on principles of game design that incorporated badges for small accomplishments, which greatly reduced the development of small scale badges in order to prevent unnecessarily doubling up the number of badges awarded at the beginning of learning trajectories  (DPD Follow-up Interview).

Design Principles for Studying Learning

Unlike many of the other projects monitored by the DPD team, CS2N intended to develop a badge system that would collect and develop information on the effects of badges on the motivation and assessment of students learning computer science skills.  The system is intended to collect data on both the effects and functionality of badges on learner’s motivations within the CS2N network and the STEM community in general.  The model is currently being developed toward  a more sophisticated analytics system.

Study badge impact > Research OF badges Model badge effects

Intended, enacted, formal.

CS2N proposed to collect information on how badges influenced learning outcomes within the CS2N programs and the STEM communities beyond. To collect this information, CS2N developed an intuitive model to study how badges influenced motivation and assessment practices.  As of the DPD follow-up interview, the CS2N team is still developing the model toward a more sophisticated, nuanced system that collects more detailed information on how badges influence assessment, motivation, and learning.

Improve badge impact > Research FOR badges > Data collection from system?

Intended, enacted, formal

Another aspect of the CS2N model was to collect information on how random manipulations on badges affected the development of learning outcomes.  These manipulations currently involve random transformations and adjustments to learners earning and displaying particular types of badge (DPD Follow-up Interview). As stated, the model is still in development and is shifting toward a more nuanced system that can can manipulate and collect data on particular badges earned in particular settings and conditions.

Recognizing Principles Assessing Principles Motivating Principles Studying Principles
Specific principles: Specific principles: Specific principles:

Leave a Reply

Your email address will not be published. Required fields are marked *

*

Mozilla Open Badges
Digital Badges are web-enabled credentials of learning or accomplishment. -Erin Knight, director of the Badge Alliance
Badges contain detailed claims about learning, links to actual evidence of learning, and they're shareable over the web. -Dan Hickey, DPD Project Lead Investigator
To me, digital badges represent the bridge between formal learning & informal structures. -Alex Halavais, DML research competition winner
Open Badges can help people tell a verifiable story about their accomplishments. -Nate Otto, DPD Project coordinator
Regardless of where you start, it’s more than likely you’ll end up somewhere other than your intended destination. That’s okay. Systems are living things, and your badge system needs to be flexible. You must embrace a bit of chaos in its design. -Carla Casilli, Director of Design + Practice at the Badge Alliance