Savings without sacrifice: a case report on open-source textbook adoption

ABSTRACT Rising textbook costs have prompted the development of open-source textbooks to increase access to education. The purpose of this case report is to examine open-source textbook adoption through the COUP framework (costs, outcomes, use, and perceptions) comparing a semester with a commercial textbook to a semester with an open-source textbook. Students (N = 520) were enrolled in an undergraduate course at a mid-sized public university in the United States. Results indicated that although costs were substantially lower, student learning outcomes and perceptions of quality were similar or better with an open-source textbook. Although students were much more likely to access the open-source textbook electronically, there were no differences in how they reported using the two textbooks to support their learning. Considering the financial savings of open-source textbooks, these findings build on existing empirical support that encourage the adoption of open-source textbooks.

use, and perceptions. The purpose of this case report is to apply the COUP framework in an empirical comparison of the use of an open-source textbook to a commercial textbook in an undergraduate course.

Cost
A clear motivation behind the development of open-source textbooks has been financial given the high costs of commercial textbooks (Ally and Samaka, 2013;Hilton & Wiley, 2011;Perry, 2015). However, college students are savvy to methods of saving money on commercial textbooks by purchasing older editions, sharing textbooks, using course reserve copies at the campus library, downloading illegal electronic copies, or simply not getting the textbook at all (Florida Virtual Campus, 2012;Moxley, 2013). Often cost savings are assumed when examining research findings on open-source textbooks, but not explicitly tested (e.g. Fischer, Hilton, Robinson, & Wiley, 2015;Hilton & Laman, 2012;Hilton, Gaudet, Clark, Robinson, & Wiley, 2013; but see Bliss, Hilton, Wiley, & Thanos, 2013; for an exception). Given that faculty often state that saving students money is the primary attraction of open-source textbooks (Petrides, Jimes, Middleton-Detzner, Walling, & Weiss, 2011), additional evidence of the cost-savings assumption is warranted.
It should be noted that the cost-effectiveness of accessing an electronic textbook depends on where a student is located. In countries in which most students already are paying for internet and have computers, an open-source textbook may be downloaded without additional cost to the student (Bliss & Smith, 2017). However, a student living in a country in which internet and/or access to technology is limited would likely have costs associated with either accessing the open-source textbook electronically or in print (Butcher, 2015).

Outcomes
In general, open textbooks are associated with better or similar student performance outcomes than commercial textbooks. For example, there were higher final examination scores and a lower withdrawal rate in the semester an open-source textbook was adopted compared to the previous semester for the same course with a commercial textbook (Hilton & Laman, 2012). In an examination of multiple courses at a community college, student outcomes in terms of grades and course completion when using opensource textbooks were usually similar or better compared to commercial textbooks (Fischer et al., 2015). However, in many studies, the adoption of open textbooks was part of a larger course redesign such as the inclusion of interactive online learning (Bowen, Chingos, Lack, & Nygren, 2014), greater use of active learning techniques in the classroom (Pawlyshyn, Braddlee, Casper, & Miller, 2013), development of new course objectives (Hilton & Laman, 2012), or an entirely new curriculum (Feldstein et al., 2012). There has been limited examination of outcomes between open source and commercial textbooks in which instructional and assessment methods were similar (Hilton, 2016). Therefore, these previous findings are not helpful for instructors who already use innovative teaching methods and incorporate textbooks as resources for students to apply in active learning activities (e.g. Love, Hodge, Grandgenett, & Swift, 2014) or to prepare for class (e.g. Heiner, Banet, & Wieman, 2014;Parappilly, Siddiqui, Zadnik, Shapter, & Schmidt, 2013). In addition, students' prior academic performance was typically not considered in research studies comparing grades using open-source versus commercial textbooks (Hilton, 2016;see Allen, Guzman-Alvarez, Molinaro, & Larsen, 2015; for an exception). Given that random assignment of different textbooks to students is not tenable, causal claims cannot be made from previous research findings, prompting a need for replication (Fischer et al., 2015). Given these issues, it is necessary to conduct a study comparing learning outcomes in courses with open-source and commercial textbooks in which the instructor and content were held constant and student prior academic performance was considered.

Use
Open access textbooks are typically available electronically without fees, but there are costs if a student wishes read a hard copy (e.g. Hilton & Wiley, 2011). Faculty members often assume open textbooks would be accessed electronically and, based on this assumption, have concerns about learning from digital materials (Belikov & Bodily, 2016). This concern is substantiated given research findings indicating students may be more likely to be distracted when reading from a screen than from paper (Woody, Daniel, & Baker, 2010). Furthermore, college students often indicate a preference for reading from paper as opposed to electronic text (Aharony & Bar-Ilan, in press;Mizrachi, 2015), which could lead to reluctance to use the textbook. A lack of textbook use may be detrimental to learning (Landrum, Gurung, & Spann, 2012). This may be particularly problematic for courses that incorporate active learning because students often need to use the textbook to complete activities (Seaton, Kortemeyer, Bergner, Rayyan, & Pritchard, 2014). Therefore, examining how students access an open-source textbook compared to a commercial textbook (i.e. electronically or paper) is critical to consider in empirical comparisons of these textbooks. Not only should the medium of access be addressed, but also this issue should be considered when examining how the textbook is used in the course. In other words, do students typically read the open-source textbook electronically and the commercial textbook on paper and, if so, does a difference in medium carry over to less use and poorer learning with open-source textbooks compared to commercial textbooks?

Perceptions
Student perceptions of open-source textbooks are of critical importance to examine for multiple reasons. Instructors often choose textbooks based on students' preferences (Durwin & Sherman, 2008). Moreover, students tend to perform better on learning measures of textbooks they rate as high quality (Durwin & Sherman, 2008). Empirical studies of student perceptions of open-source textbooks are limited, but show that students tend to rate their open-source textbooks as the same or better quality as commercial textbooks they have used Cooney, 2016;Hilton et al., 2013;Illowsky et al., 2016). However, in previous studies of student perceptions of open-source textbooks Cooney, 2016;Illowsky et al., 2016), students were asked to compare an open-source textbook they were currently using to the commercial textbooks used in their other courses, which would likely be other disciplines. It is possible that attitudes regarding open-source textbooks were complicated by differences in content. An examination of student perceptions of open-source textbooks compared to student attitudes of commercial textbooks in the same type of course would avoid these complications.

Current study
To date, no study has applied the COUP framework to compare open-source and commercial textbooks between students in the same course from the same instructor. In order to address this gap, this current study compared two semesters of an introductory psychology course: one with a commercial textbook and one with an opensource textbook. This study addressed four research questions: (1) What were the differences in self-reported costs between commercial and opensource textbooks? (2) How did student learning outcomes compare in semesters with commercial versus open-source textbooks? (3) How did students report accessing (electronically or on paper) and using the different types of textbooks in their courses? (4) How did student perceptions towards the required course textbook compare in semesters with commercial versus open-source textbooks?

Method Participants
Participants were students in the author's introduction to psychology courses in Spring 2016 (two courses with 316 students enrolled at the end of the term) and Fall 2016 (one course with 204 students enrolled at the end of the term) at a mid-sized, Midwestern university. In Spring 2016, a commercial textbook was used and in Fall 2016 an opensource textbook was used (see details in Materials). Based on questionnaire responses (see Measures), there was a greater percentage of female students in the semester with an open-source textbook compared to the semester with a commercial textbook (this potential confound is addressed in the Discussion). The percentages of native English speakers, first-generation college students, and freshmen were not significantly different between the two semesters (see Table 1 for descriptive and chi-square statistics).

Materials
The commercial textbook used Spring 2016 was the 11 th edition of Worth Publishers' Psychology in Modules (Myers & DeWall, 2015). This textbook was at the campus bookstore in print, loose-leaf format, but could be ordered in electronic or bound formats. In the preface of the Worth Publishers' textbook, 77 reviewers and consultants from a variety of institutions were listed. The open-source textbook was the 1 st edition of OpenStax College's Psychology (OpenStax College, 2014). A PDF of the textbook as well as the link to its web-based format were posted on the course's learning management system's site (Blackboard). In addition, students were informed that, if they preferred paper, they could order a bound copy through OpenStax or they could print it out on their own paper. In the preface of the OpenStax textbook, 36 peer reviewers from a variety of institutions were listed.

Measures
Grades. To address the research question regarding outcomes, course-level data of student grades were compared. Specifically, the final course percentages and the number of students who withdrew from the course were examined based on course records and institutional records, respectively. Course percentages were based on four exams, weekly quizzes, two writing assignments, and participation. Approximately 15% of the items in the quizzes and exams were from content covered in the textbook, but not in lectures. The exams and quizzes were similar, but not identical, in the two semesters due to slight variations in the content covered in the two textbooks. To gauge the students' academic preparation, the average high school grade point averages of students enrolled at the end of the terms as well as students who withdrew from the courses were obtained from institutional records.

Questionnaire
To address research questions on cost, use, and perceptions, a questionnaire was developed. For cost, there was one item in which students were asked how much they spent on their textbook. There was one item for how the textbook was accessed. There were five items on use based on what the course instructor recommended students use the textbook for. Students were asked to rate on a six-point Likert scale how much they agreed with statements about use (strongly disagree to strongly agree; see Table 2 for types of use). There were seven items regarding quality perception based on features in both the commercial and open-source textbook (see Table 2 for features).
Students were asked to rate on a six-point Likert scale how much they liked each of the textbook features (strongly dislike to strongly like). There were also two open-ended items in which students were asked what they liked and disliked the most about their course textbook. The end of questionnaire asked students to report their gender, first generation college status, native language, and year in school. For the semester with the commercial textbook, approximately 86% of the students responded to the

Results
Descriptive and inferential statistics are reported in Table 1 for chi-square analyses and in Table 2 for t-test analyses for all research questions. Cohen's d effect sizes are stated in the results for statistically significant t-tests, that is p < .05.

Cost
Not surprisingly, students reported spending considerably less on open-source textbooks (including printing expenses or cost of ordering a print version) than commercial textbooks, a highly statistically significant (p < .001) and very large difference (Cohen's d = 2.12).

Outcomes
Based on a t-test of course percentages, students performed slightly better the semester in which an open-source textbook was used compared to the semester in which a commercial textbook was used, a statistically significant (p = .011) and small difference (Cohen's d = .23). However, the average high school grade point average for the students who were enrolled at the end of the term was also slightly higher for the semester with open-source textbooks, a statistically significant (p = .009) and small difference (Cohen's d = .25). In terms of withdrawals, there were proportionally fewer withdrawals for the semester with the open-source textbook compared to the semester with the commercial textbook, a finding that was highly statistically significant (p < .001) with a small effect size Cohen's d = .36 . The average high school grade point average for students who withdrew was not reliably different between the two semesters.

Use
Responses for accessing the two types of textbooks were collapsed into two categories: electronic only and print. Substantially more students accessed the open-source textbook electronically than the commercial textbook. There were no reliable differences between commercial and open-access textbooks on any of the five items regarding use of the textbook.

Perceptions
Based on questionnaire responses, student perceptions of the quality of the two textbooks were generally similar with two exceptions (see Table 2). Visual appeal was marginally (p = .06) rated higher with the commercial textbook than the open-source textbook. In contrast, writing was rated higher with the open-source textbook than the commercial textbook, a statistically significant (p = .029) and small difference (Cohen's d = .23). Student responses to the open-ended items regarding what they liked and disliked the most about their textbook were categorised. As can be noted in Table 3, the most common response for what was liked most about the commercial textbook was regarding the writing followed by its organisation. For the opensource textbook, a theme emerged around responses related to affordability. Responses also suggested that the writing, electronic access, and electronic features were qualities that students enjoyed about the open-source textbook. As can be noted in Table 4, the most common responses for what was disliked about the commercial textbook, were related to its physical print features, especially its size and weight. In contrast, the most common responses for what was disliked about the open-source textbook were related to its electronic nature including a dislike of reading from screens.

Discussion
The purpose of this case report was to compare costs, outcomes, use, and perceptions for an open-source and commercial textbook. Self-reported costs were considerably lower the semester in which an open-source textbook was used. Based on these findings, the savings for a class of 200 students would be $16,382. Indeed, cost was the most valued feature of the open-source textbook. Student grades were somewhat better in the semester in which an open-source textbook was adopted. However, the high school GPAs were also somewhat better for these students, indicating that they were likely better prepared academically. In addition, there was a larger proportion of female students based on questionnaire results in the semester with the open-source textbook compared to the semester with the commercial textbook. This difference in gender distribution is noteworthy because female students, on average, have been found to have slightly higher grades than male students (Voyer & Voyer, 2014). Taken together, there was likely no real effect of open-source textbook adoption on course grades. The number of students who withdrew from the course was substantially higher in the semester with a commercial textbook compared to the semester with an opensource textbook, although the high school GPAs of students who withdrew were similar the two semesters. In terms of use, students were far more likely to solely access the open-source textbook electronically than the commercial textbook. However, there were no differences in how students reported using the textbook. As far as student perceptions of quality, the two textbooks were generally similar with one exception: Students liked how the open-source textbook was written more than the commercial textbook. A discussion of these findings applying the COUP framework follows.

Cost and outcomes
Students spent substantially less on the open-source textbook than the commercial textbook without detriment to learning outcomes. For students who completed the course, their performance appeared to be approximately equivalent after taking into consideration differences in high school grade point averages. The findings regarding student learning outcomes in this study build on others by incorporating previous academic performance and using actual grades, rather than self-reports of learning, as well as making comparisons between two semesters with the same instructor and content (e.g. Fischer et al., 2015;Hilton & Laman, 2012;Pawlyshyn et al., 2013).
In this case report, the withdrawal rate was substantially lower for the semester with an open-source textbook. It is important to note that students withdraw from courses for numerous reasons with the most common being dislike of the instructor and/or content (Hall, Smith, Boeckman, Ramachandran, & Jasin, 2003). In this study, the same instructor taught the same content to the two courses that were compared. Although the students were obviously different in the two semesters examined in this study, the previous academic performance of the students who withdrew was similar both semesters. This is a critical factor that was not addressed in previous findings associating lower withdrawal rates with the adoption of opensource textbooks (Fischer et al., 2015;Hilton & Laman, 2012). Therefore, it is realistic to suspect the difference in withdrawal rate in this study was at least, in some small part, related to the adoption of an open-source textbook. Perhaps if students feel that they are behind in the material and have the option to catch up by reading a textbook at no cost, they may be less likely to withdraw than if they never bought the textbook and need to pay for one to succeed in the course. Alternatively, students who can access course materials at no cost may be afforded better opportunity to engage with the course material reducing the likelihood of getting behind in the course. It should be noted that these possibilities were not empirically tested and are solely conjecture. An interesting avenue for future research would be to examine the reasons students withdraw from courses with opensource versus commercial textbooks.

Use
Students were much more likely to access the open-source textbook electronically than the commercial textbook, perhaps because the open-source textbook was available online without charge. Given that research findings show students prefer to read paper as opposed to electronic texts (Aharony & Bar-Ilan, in press;Mizrachi, 2015), there may be concerns that students would use the open-source textbook less than the commercial textbook. However, the findings indicated no differences in self-reports of using the textbook.

Perceptions
Overall, students had similar perceptions of quality for the commercial and the opensource textbook, with the exception that students liked the writing of the open-source textbook more than the commercial textbook. This finding is noteworthy given previous research indicating positive correlations between writing quality and amount of the textbook read as well as exam scores (Gurung & Martin, 2011). In past research, students generally had positive perceptions of open-source textbooks Cooney, 2016;Illowsky et al., 2016). This study addressed gaps in previous studies by comparing student perceptions of quality between a commercial textbook and an open-source textbook covering similar content. Furthermore, the findings in previous work used global measures, such as asking students how they would rate the overall quality of the textbook compared to other textbooks they have used (e.g. . In contrast, the perception of quality measures in this study addressed specific features (e.g. writing, visuals) to allow for a more nuanced examination.

Conclusion
In this study, the differences in student cost, outcomes, use, and perceptions between an open-source and commercial textbook were examined. As expected, students' selfreported average expense for the open-source textbook was substantially lower than that of the commercial textbook. Importantly, student outcomes in terms of grades were comparable between the two textbooks and the withdrawal rate was lower with the opensource textbook, indicating spending more money on materials did not appear to benefit learning. Although students were much more likely to access the open-source textbook electronically, they used it as much as the commercial textbook that was typically accessed in print. Overall, the findings from this case report indicate that the open-source textbook reduced the financial burden of a college education without negatively influencing student learning, student perceptions of quality, or use of the textbook. Taken together, these findings are helpful for encouraging faculty to adopt open-source textbooks, thereby contributing to a global advocacy effort to promote OERs. Note 1. The review questions had answers at the end of the textbook.