Introduction

Capturing instructional practices in Science, Technology, Mathematics, and Engineering (STEM) classrooms is critical to advance our understanding of effective teaching, identify paths to support faculty in improving their instructional practices, and to measure the impact of professional development.

In this website, we provide a new analytical tool – the COPUS Analyzer - aimed at characterizing instructional practices. This tool is based on an established observation protocol, the Classroom Observation Protocol for Undergraduate STEM (COPUS) . This observation protocol captures instructor’s and students’ behaviors that take place during 2-minute time intervals throughout a lecture. The COPUS Analyzer permits the visualization of occurrence and co-occurrence of these behaviors throughout a single or multiple classroom observations for one or several instructors. More importantly, the COPUS Analyzer automatically classifies classroom observations into specific instructional styles, called COPUS Profiles. The profiles represent a range of instructional styles, from teacher-centered to student-centered. They were developed from a data set that comprised 2,008 individual class periods collected from over 500 STEM instructors across at least 25 institutions in the United States.


Get Started!

Follow the following steps to start taking advantage of this tool:

    1. Read the Instructions tab – it is critical to ensure that data are entered correctly
    2. Fill out the Demographics tab – this information is used to enable various analytical tools that are pertinent to your specific needs
    3. Upload your data in the COPUS Analyzer tab.

Disclaimer: Any data submitted within this app will be saved by the research team on a secure server and used for research purposes. As such, it is critical that data uploaded be de-identified. The Institutional Review Board at the University of Nebraska-Lincoln has approved this project.


If you have any questions regarding this tool or this project, please email either Marilyne Stains or Jordan Harshman

This material is based upon work supported by the National Science Foundation under Grant No. CAREER 1552448. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

Instructions

Data Format


Please click on the folling link for a brief video tutorial: Video Tutorial (X minutes)

This tool will accept data in three formats (you can download each template at the top of this page). All are .xlsx files.

    1. The Minute-by-Minute Template (xlsx) (download here) is for instructors/researchers who have COPUS data for every 2-minute intervals of each class. This can be one class for one instructor or multiple classes for multiple instructors. This template is also used if researchers have multiple coders COPUSing the same video(s) in a training context or for inter-rater reliability.
    2. The Summary Template (xlsx) (download here) is for instructors/researchers who have COPUS data for each classroom observation in percent of two-minute intervals. Each of the template files have a pre-filled in entry equivalent to one instructor's observation to exemplify what the data should look like.

The format of the required variables within these files are lenient.

    Time: On a minute-by-minute file, you can enter the actual time blocks (i.e. '9:30AM-9:32AM'), the video time blocks (i.e. '00:00-02:00'), a number representing which time block it is (i.e. 'time block 1' or just '1'), or similar formats.
    Instructor: Can be entered as strings (i.e. 'FRK') or numbers (i.e. instructor IDs, '14'). Please ensure that data entered in this column de-identify the instructor (i.e, if John Smith taught the course do not enter John Smith in this column, rather, use numbers or an acronym).
    Class_number: This refers to the actual class (i.e. 'CHEM 101'), but could also just be a number or letter.
    Class_date: Can be in any date format (i.e. 'April 3, 2017', '04/13/2017', '04-03-17', 'Tuesday, April 3', etc.) or just a numeric identifier (i.e. 'Day 1', '1').

The only important thing to note is that a given instructor can have multiple classes observed on the same date or the same class observed on multiple dates.


Data Upload


Step 1

Bear in mind that once you upload your data, we will save a copy on our secure server for use in our research project. For this reason, make sure that the data you upload is de-identified.

Step 2

On the Demographics page, fill out all the required demographics and whether you are uploading data for single or multiple observations and which format (minute-by-minute template or summary template) you have. Make sure that you are using the correct template for each format.

Step 3

Once the demographics survey is completed, you will be able to click the 'Submit' button. While you are able to see the analyzer tool page, it will not be responsive until the 'Submit' button is clicked. By clicking the submit button, you also agree to have us use your submitted data for our studies

Step 4

After clicking the 'Submit' button, click on the COPUS Analyzer tab. Once on the COPUS Analyzer page, click the 'Browse' button and navigate to the file that you want to upload (excel files only).


Data Analysis


Below is a list of the analyses that you can do with this tool. Please note that depending the format of the data you upload, not all of these options will be available (i.e. if you don’t have minute-by-minute data, we can't produce a minute-by-minute heat map).

Options for minute-by-minute files
    Generate a summary file: You can convert the minute-by-minute into a summary file that shows the percent of 2-minute intervals spent on each behavior for each instructor for each observation. Additionally, each observation will contain a cluster number that corresponds to the instructional profile that best matches that observation. Simply click the Download Summary button to download the summary file.
    Heatmap of COPUS codes You can view and download customizable heat maps that are displayed for each of the 25 COPUS behaviors for each minute in an observation. If you want less/more COPUS behaviors, simply check the boxes to include/remove them. Click the Download HM Behaviors button to download a .png image file of the heatmap currently displayed.
    Heatmap of COPUS categories Heatmap of COPUS categories: You can view and download customizable heat maps that display 8 broad categories (4 student, 4 instructor) for each minute in an observation – this categorization is based on the work from Smith et al. (2014). If you want less/more COPUS collapsed codes, simply check the boxes to include/remove them. Click the Download HM Categories button to download a .png image file of the heatmap currently displayed.
Options for minute-by-minute files
    View percentage of each COPUS behavior for each class of a particular instructor: You can view the aggregate percentages for each class of a selected instructor. Simply click the Download Aggregate button to download a .png image file of the heatmap currently displayed.
    Generate a report for an instructor: If you have uploaded multiple observations per instructor, you can generate a report for each instructor that will include
      a) one graph that shows the percent of each COPUS behavior for all observations (for both minute-by-minute and summary files),
      b) a heat map that shows each COPUS behavior for each minute of class (for minute-by-minute files), and
      c) a heat map that shows each COPUS category for each minute of class (for minute-by-minute files).
    To generate this report in a pdf format, simply click the Download Report button.

Calculate inter-rater reliability


If you have a data set with multiple coders for the same instructor's video(s), you can access inter-rater reliability information. Just use the minute-by-minute template (see example of irr data here) and be sure to answer the question on the demographics that you are using the tool for inter-rater reliability/training. Once you upload your data, you will see two tables that look similar:

    - The top table shows all of the coders for the same video (remember that the same coder and the same video needs to be spelled exactly the same) . For example, if you have 4 coders for the same video, the graph will show the total number of coders who coded each of the 25 behaviors throughout each minute of the class.
    - The bottom table shows only the coders, which you can select in the drop down menu for quick reference when differences arise.

Both of these tables can be exported in a .csv format (opens in Excel).

Finally, this program will calculate percent agreement and Fleiss' kappa for the selected video. The values will show at the below the second table. These measures are good for determining how similar all the coders rated the same video. If the graph isn't big enough to fit the numbers (this can happen for classes that are longer than 50 minutes), simply download each excel file.


Troubleshooting


In order for this tool work effectively, please ensure the following:

    1. You are using the correct template - see Data Format section above.
    2. Variables within the template are in the exact same order as they appear in the template.
    3. You have selected the appropriate radio buttons on the Demographics Tab (combination of minute-by-minute vs. summary data for single vs. multiple observations).
    4. If you have multiple instructors, classes, and dates that should be grouped, they need to be spelled exactly the same (i.e. if your instructor is called 'Jordan' in one row and 'Jordan H' in another, they will be treated as two different instructors or if the class is 'CHEM 101' in one row and 'CHM 101' in another, they will be treated as two separate classes).
    5. Make sure that you do not have any duplicates. For analyzing COPUS data, you will need to provide an instructor name, class, AND date. If you only have instructor name but know that it's the same class on different dates, simply make up a class number/name and dates (but those columns need to be filled). If you're doing inter-rater reliability, you will only need to fill in the observer and instructor columns. If you have multiple observations for the same instructor, simply put the full description in the instructor column (i.e. instead of 'Frank' in the instructor column and 'Day 1' in the date column, put 'Frank Day 1' in the instructor column).
    6. This program does not generate error messages. You can assume that the tool has encountered an error if it remains inactive for longer than 30 seconds. If this happens, check that your data is correct and try re-uploading your data.

Troubleshooting


If something in this tool is not working as you would expect it to, you find inaccuracies, or wish to provide feedback regarding its functioning, please contact Marilyne Stains or Jordan Harshman

Demographics

Please fill out the following fields. All fields are required before you are allowed to use the COPUS analysis tool. Click 'Submit' below when you are finished.

Thanks, your response was submitted successfully!

You may now click on 'COPUS Analyzer' tab to anlayze your COPUS data!

Abbreviations

The following codes are from the Classroom Observation Protocol for Undergraduate STEM (COPUS) paper previously published


Instructor Behaviors

Lec = Lecturing (presenting content, deriving mathematical results, presenting a problem solution, etc.

RtW = Real-time writing on board, doc. projector, etc. (often checked off along with Lec)

FUp = Follow-up/feedback on clicker question or activity to entire class.

PQ = Posing a non-clicker question to students (non-rhetorical).

CQ = Asking a clicker question (mark the entire time the instructor is using a clicker question, not just when first asked).

TAnQ = Listening to and answering student questions with entire class listening.

MG = Moving through class guiding ongoing student work during active learning task.

OoO = One-on-one extended discussion with one or a few individuals, not paying attention to the rest of the class (can be along with MG or TAnQ).

DV = Showing or conducting a demo, experiment, simulation, video, or animation.

Adm = Administration (assign homework, return tests, etc.).

TW = Waiting when there is an opportunity for an instructor to be interacting with or observing/listening to student or group activities and the instructor is not doing so.

TO = Other - explain in comments.


Student Behaviors

L = Listening to instructor/taking notes, etc.

Ind = Individual thinking/problem solving. Only mark when an instructor explicitly asks students to think about a clicker question or another question/problem on their own.

CG = Discuss clicker question in groups of 2 or more students.

WG = Working in groups on worksheet activity.

OG = Other assigned group activity, such as responding to indstructor question.

SAnQ = Student answering a question posed by the instructor with the rest of class listening.

SQ = Student asks question.

WC Engaged in whole class discussion by offering explanations, opinion, judgment, et. to whole class, often facilitated by instructor.

Prd = Making a predication about the outcome of a demo or experiment.

SP = Presentation by student(s)

TQ = Test or quiz

SW = Waiting (instructor late, working on fixing AV problems, instructor otherwise occupied.

SO = Other - explain in comments.


Instructor Categories (Collapsed Codes)

Presenting = Lec, RtW, or DV

Guiding = FUp, PQ, CQ, TAnQ, MG, OoO

Administrative = Adm

Instructor other = TW, TO


Student Categories (Collapsed Codes)

Receiving = L

Talking to class = SAnQ, SQ, WC, SP

Students working = Ind, CG, WG, OG, Prd, TQ

Student other = SW, SO

COPUS Analyzer

IMPORT ------------------

FILTER ------------------

BY CATEGORY ------------------
BY BEHAVIOR ------------------
DOWNLOAD ------------------
Download Summary Download HM Categories Download HM Behaviors Download Aggregate Download Report
BY CATEGORY ------------------

BY BEHAVIOR ------------------

ALL OBSERVATIONS FOR INSTRUCTOR ------------------

INSTRUCTIONAL PROFILE ------------------

COPUS Analyzer

IMPORT ------------------

FILTER ------------------

BY BEHAVIOR ------------------
DOWNLOAD ------------------
Download Aggregate
ALL OBSERVATIONS FOR INSTRUCTOR ------------------

INSTRUCTIONAL PROFILE ------------------

COPUS Analyzer

IMPORT ------------------

BY CATEGORY ------------------
BY BEHAVIOR ------------------
DOWNLOAD ------------------
Download Summary Download HM Categories Download HM Behaviors
BY CATEGORY ------------------

BY BEHAVIOR ------------------

INSTRUCTIONAL PROFILE ------------------

COPUS Analyzer

IMPORT ------------------
INSTRUCTIONAL PROFILE ------------------

COPUS IRR

IMPORT ------------------

FILTER ------------------
RESIZE ------------------

DOWNLOAD ------------------
Download top graph Download bottom graph