2 Rubric and Assessment
- Coursework: 65%
- Final Project: 20%
- All work indicated in each section of the workbook under ‘What you need to do each week’ has to be completed.
All work has to be completed to a satisfactory level, per the criteria below.
By the end of this course, you should be able to do the following:
- Understand how to use the idea of ‘the productive fail’ to use computing power effectively as an historian;
- Understand, plan, and employ concepts of computational reproducibility in the service of history
- Employ web tools to build history in public, collegially
- Weigh the evidence appropriately, understanding how it was created, in order to communicate the compelling story
- Understand the ways digital tools change us as we use them, to create compelling history that is self-reflexive
These objectives map against the criteria for your coursework and final project grades like this:
|1. Productive ‘Fail’ documented|
|2. Reproducibility fully enabled|
|3. Collegiality amply demonstrated|
|4. Evidence carefully and fully engaged with, analyzed|
|5. Self Reflection present in all work||.|
Over the course of the term, you will produce different pieces of work that speak to these different criteria:
- online fail logs to keep track of what you actually do at your computer
- online notebooks that tell the story of what you did, and why, and what you did about it when things didn’t work
- web annotations of each others’ logs/notebooks, and of the readings, and responding to others’ annotations
- storytelling and engagement with the data, its limitations and its strengths
- your growth as a historian over this course.
2.1 Weekly work
Each week, you will complete (with certain caveats, as detailed in the workbook)
- exercises that are appropriate to your tech level. How do you know if it was appropriate? If the exercise was easy, then you do the next one. You push yourself until you get to the point where you are stumped. You will keep a fail log and an online notebook for the exercises. The point here is not to stump you, but to push you from your current comfort level.
- collaborative reading assignments, which you will demonstrate that you have done via collaborative web annotations of either the fail logs, notebooks, or assigned readings. Exercises are organized by modules, and the sequence of modules reflects the sequence of doing digital history.
nb there must be evidence from each module that you have fully engaged with the exercises and the collaborative reading. How to demonstrate this evidence will be covered below. A checklist of what should be completed by when is provided below. Work must be completed in a timely manner. Follow, and keep to, the schedule.
2.1.1 More about the fail log and notebook
You will keep the fail log in a repository on Github that details the work you do for each module. What can go into this? Everything and anything. Files you create. Notes you make to yourself on how to use a particular tool. Random thoughts as they occur. Here is an example for you to refer to.
I call it a ‘fail log’ because more often than not, things don’t work at first. We learn by trying, by failing productively, as it were. In essence, there are many ways to craft digital history; you need to leave yourself breadcrumbs so that you can return to a project after a hiatus and know what it was you were doing/thinking at the time. Caleb McDaniel calls this open notebook history and you should read that post of his before going any further. FYI, here’s my own repository for my research: http://github.com/shawngraham.
You should also keep a narrative of your research that connects the dots and explains the thinking behind what you were trying to do (and what you documented in your fail log); think of this as your lab notebook. A blogging platform is a good way of doing that. FYI, here’s mine: Electric Archaeology; I’ve tried another variation on the model here. There is no one correct way to do this; you will have to work to find a way that feels authentic to you.
The modules in the workbook contain readings, and exercises aimed at various levels of comfort with things digital. That is to say, if you’ve never done anything with your computer other than updating Facebook, there are exercises designed to introduce you to computing; if you are doing a computer science minor on the side, there are exercises designed to challenge you too. The idea is that you work through the exercises until you get stumped. You detail where you were at and how you got stumped, in your narrative in your notebook. You ask each other for help, encourage each other, and discuss the exercises, in our Slack space.
I am not interested so much in your final product as I am interested in your progress. The assessment of the fail log and the notebook is not on whether you end up with a ‘right’ or ‘wrong’ piece, but rather, how you have grown and challenged yourself. Should you be someone for whom the exercises hold no challenge, then I challenge you to write the next exercise yourself. I’m always learning: I would be disappointed if I learned nothing from all of you. Again, the challenge of these modules: you must push yourself beyond your comfort level. That is what I am looking to assess. If I see that you are only trying the minimum level when it is apparent that you could be pushing further, that will be deemed unsatisfactory.
2.1.2 More about the collaborative readings
Digital history is not done in isolation. In contrast to other courses you may have done, I expect you to collaborate, to help each other out, to problem solve, to draw each other’s attention to the important points. In the first instance, annotating the readings and workbook make this experience more like a study group. In the second, I’m quite content that you should help each other out with the exercises. Really?! I hear you exclaim. Yes, but: if you had help working through an exercise (for instance), you would make a public note of the help you needed, who helped you out, and how. How will you do this? You could make a post in your notebook about how you helped someone out - or how they helped you out. You could read someone else’s notebook about their work and perhaps that sparks a thought or idea in you. You could then write about that, and link back to your peer’s original work.
This collaborative community building can also happen in our private Slack space. Maybe you organize folks into a reading group. Maybe you set up a group video-chat session. All these things create a cohesive group, and I will be watching for evidence of this.
Unacknowledged help or unacknowledged collaboration will be grounds for determining unsatisfactory work.
2.2 Final project
The final project will have two parts:
- an analysis (a work of digital history) of the provided digital dataset using at least two or more techniques/tools that you learned in the exercises. This analysis has to be available on the web, with your data and methods fully documented such that someone else could undertake the analysis for themselves.
- a ‘paradata’ essay (or video, or other digital form) that reflects on your growth as a historian over the term with reference to the analysis in 1.
nb the final project may be submitted at any time up to midnight on the very last day of the course. Since the work will live on the web, you submit by sending me an email with ‘Final Project HIST3814o - submitted’ as the subject. I will make an archival copy of that project the following day. More information on the final project will be provided below.
There is no midterm.
Detailed instructions for the Final Project may be found here. Your final project will take as its source material some 1500 editions of The Shawville Equity, an English newspaper published in Western Quebec, from 1883 to the present day. Begin thinking now about the kinds of history or the historical questions for which such information might be suitable.
Paradata’ is a description of the ‘how’ and ‘why’ of what you’re doing (see also the ‘when things go wrong’ section on the detailed instructions). A formal discussion of what paradata do from the London Charter:
Documentation of the evaluative, analytical, deductive, interpretative and creative decisions made in the course of computer-based visualisation should be disseminated in such a way that the relationship between research sources, implicit knowledge, explicit reasoning, and visualisation-based outcomes can be understood.
Your paradata document should have the following sections (as discussed in the London Charter):
Implementation; Aims and Methods; Research Sources; Documentation; Sustainability; and Access
My strategy here is two-fold. I recognize that sometimes, a project fails to come together. I believe in ‘productive failure’, where we learn from what works and what doesn’t work. I want to explicitly signal to you that it is safe to try something here that might not get to where you want it to go - and that’s ok. As an example, let me share with you ‘How I lost the crowd’, a piece where a public digital history work of my own fell to pieces - and how I salvaged worth from it. (As an example of ‘paradata’ it’s not strictly speaking as formal as I want from you.)
2.3 Real Names Policy
You do not need to use your real name or identity on any public-facing work that you do in this course. Nor do you need to explain to me that you wish to use a pseudonym. It is sufficient that you send an email to me with the following message:
‘I would like to use the following username in all public-facing work: xxxxxxxx’
…where xxxxx is the name you have selected. For safety’s sake, if you decide to use a pseudonym, do not use one that you have used on any other website or social media platform.
2.4 Submitting evidence
At the end of each module, fill in the report form and provide the links to:
- appropriate entries in your fail log
- appropriate links in your notebook
- appropriate links to annotations.
The first exercises in the workbook will walk you through setting up your notebook and fail log.
Because of the compressed schedule of a summer course, work that is submitted late will be considered not completed; you will have a get-out-of-jail-card-free for ONE module’s work only. IE, for whatever reason, you couldn’t get module 3 done nor module 4. You soldier on and submit late either module 3 OR module 4, selecting the better of the two and invoke the card with an email to me. Note that it is always possible to consider why things haven’t come together in one’s notebook.
Students should contact Dr. Graham if they are late due to illness or other exigency.
2.5 Schedule and due dates
This course runs from July 4th to August 16th. Weekly reports due by Midnight, Sunday.
- July 4 - 9: Getting Up and Running. Complete ‘Getting Started’ in the Workbook.
- July 10 - 16: Module 1 Open Access Research
- July 17 - 23: Module 2 Finding Data
- July 24 - 30: Module 3 Fixing Data
- July 31 - Aug 6: Module 4 analysis
- Aug 7 - Aug 13: Project Work & Visualization
- Aug 16th, midnight: submit final project
2.5.1 Overview of work to be done/submitted
- Getting Started: Setup, Initial blog post, Annotations
- Module 1: Blog post re readings, blog post re exercises, Fail logs, Annotations
- Module 2: Blog post re readings, blog post re exercises, Fail logs, Annotations
- Module 3: Blog post re readings, blog post re exercises, Fail logs, Annotations
- Module 4: Blog post re readings, blog post exercises, Fail logs, Annotations
- Module 5: Final Blog Post
- Final Project: Paradata document, Project repo, Project itself
2.6 Workbook structure
The workbook may be found at workbook.craftingdigitalhistory.ca. The modules in the course are built around the progressive steps of working with big data:
- Module 1 Principles of open access research and your digital identity
- Module 2 Finding Data
- Module 3 ‘Wrangling’ Data, or getting it into useable shape
- Module 4 Analyzing data, or matching the appropriate tool to the question
- Module 5 Visualizing (graphing, writing, plotting, mapping) data patterns, or communicating the compelling story
These modules are built around the following ideas:
- Identify and define the limitations of useful sources of historical data online
- Compare and employ appropriate tools to clean and manipulate this data with a critical eye to how the tools themselves are theory-laden
- Analyze data using various tools with an awareness of the tendency of tools to push towards various historiographic or epistemic perspectives (ie, the ‘procedural rhetorics’ of various tools)
- Visualize meaningful patterns in the data to write ‘good history’ across multiple platforms, with critical evaluation of the limitations
- Model best practices in open access data management as mandated by SSHRC and other research agencies
- Develop an online scholarly voice to contribute data and reflection to the wider digital history community
2.7 Calculating grades
If you have demonstrated to me that you have satisfactorily met the criteria detailed above in the rubric (which map to the learning objectives for this course), you will receive an A. If you demonstrate that you have satisfied four of the five criteria, then you will receive a B. Three of the five: a C. Two of the five: a D. One of the five: an F.
Note that ‘satisfactorily’ meeting the criteria means completing all of the work that I have set for you. BUT, note also that you have the power to determine how much work that actually is, provided that you can justify why you haven’t pushed further. That is, I will look at your fail logs and your notebooks to determine whether or not the module exercises were challenging for you. It is thus in your best interest to keep careful and complete fail logs and notebooks as you do both the modules AND your final project. I can only grade where the evidence exists.
In exceptional cases, where the work completed goes above and beyond, an A+ may be possible.
We all come to digital work with differing degrees of digital literacy. By framing assessment as ‘satisfactory’ I recognize that each of you are different, and I assess you against your own starting point and your own growth.
You will receive feedback and be in communication with me and with our TA throughout the course; our feedback will help you grow.
The final project is assessed against the same criteria as the coursework. Community participation is assessed through your substantive and generous collaboration via the annotations and in the Slack.
2.7.1 The process of grading coursework
Here are the instructions I give to my TAs on how we grade.
We are looking to see, in each week, that the students have
- done all the required work as listed in the workbook
- done the work in a substantive manner
Each week, we leave comments on posts, and we reply to annotations. By Wednesday we need to provide written feedback via email, explaining what is satisfactory and nudging where things can be improved, and where we also link to where we have responded to the student.
Our feedback each week should encourage dialogue with the student, and be framed within the criteria 1 - 5 below. Keep an eye out for evidence that the students are collaborating, and that they are responding in a meaningful way to each other’s work.
ALL COURSE WORK HAS TO BE COMPLETED, AND ALL WORK HAS TO BE SATISFACTORY, TO GET AN A
We can nudge students to complete things, when we note that they are missing. We can also be human, and adjust accordingly, in special cases.
We do not require students to ‘prove’ illness, or other issues. We will take them at their word.
What constitutes ‘satisfactory’ work? Satisfactory work demonstrates that the student:
1 Understands how to use the idea of ‘the productive fail’ to use computing power effectively as an historian.
2 Understands, plans, and employs concepts of computational reproducibility in the service of history.
3 Employs web tools to build history in public, collegially.
4 Weighs the evidence appropriately, understanding how it was created, in order to communicate the compelling story.
5 Understands the ways digital tools change us as we use them, to create compelling history that is self-reflexive.
Not every one of these criteria apply to each week or module; in aggregate, at the end, all must be present in the body of work that the students have produced over the course.
In the tracking spreadsheet, you simply tick off whether or not the student has done the work, entering a 1 or 0 in the column as appropriate.
** Final grade for coursework **
Missing module work reduces the grade.
Work that is completed, but is done in a perfunctory manner, similarly reduces the grade (as it isn’t satisfactory).
At the end of the course, we assign a final grade, looking for the satisfactory presence of criteria 1 - 5 across their entire opus, and add up accordingly (so 5/5 = A).
We then adjust downward for any missing work, or upwards for anything above and beyond the call.