The Life of an Accounting Information Systems Research Course

This paper describes the development, evaluation, and the changes made to a research course for part-time Master of Science students in accounting. The course prepares students for their Master of Science thesis and aims to develop their research skills. On top of this, it managed to overcome barriers between faculty members who were chiefly involved in teaching, and those who mainly conducted research. In the course, students work in teams, closely supervised by faculty members, and go through a research process that ends with the preparation of a research paper. The course consists of 10 steps, which are described and critically discussed. Evaluation scores indicate that students appreciate the course and experience a steep learning curve. Faculty members also experience benefits, despite the extensive preparation time involved.

Accounting information systems; course development; research skills

In this paper, we describe and evaluate the approach followed by faculty members of a Dutch university to strengthen the research interests of their accounting students. We focus in particular on the ‘making of’ and development of an Accounting Information Systems (AIS) course. The main research question we try to address is: how can an AIS-research course be designed that unifies the interests of three parties, namely, (1) program management, who is primarily concerned with meeting quality requirements imposed by accreditation committees; (2) faculty members, who wish to conduct (empirical) research, write research papers, and deliver quality teaching; and (3) students, who wish to write a Master of Science thesis in a limited amount of time, and continue their studies to qualify as a CPA.[1]

Accountants are researchers. The activities involved in the evaluation of annual reports may be a case in point (Carnegie & Napier, [7] ). One of an accountant’s main tasks is to ‘look behind’ the financial figures he/she is presented with, and determine whether these accurately reflect the financial status of an organization, however difficult this may be to assess (Macintosh, [39] ). In order to fulfill this task, it is common practice that accountants interview key stakeholders in an organization, review the quality of information stemming from software systems that support the organization’s daily operations, and check the legitimacy of a variety of financial and non-financial transactions. Many accountants do not seem to realize that correctly assessing an organization’s state of affairs, and thoroughly checking the reliability of the financial information it presents is not that straightforward. They often fail to see that having a research ‘mindset’ can help to achieve greater reliability. This may be due to a lack of exposure to research and/or limited attention in their education to the importance of research in their daily work (Carnegie & Napier, [7] ; Gendron & Spira, [19] ).

The paper aims to provide insight into the AIS-course’s development process and (shifting) content and structure, so that it may serve as a template or exemplar for similar courses that interested readers may wish to develop,[2] even if the structure of the curriculum, the kind of students involved (full-time or part-time), and/or the place of the course within a curriculum differs from what is presented here. We suggest that the timing and content of the course be adapted accordingly, as will be clarified in the final section of the paper. The paper covers the 2010-2015 period, when the authors were actively involved in the course in question.

Evaluation scores of the AIS-research course during this period suggest that it adequately prepared students for the rest of their studies, as well as for particular issues and problems occurring in their daily practice. On top of this, the course seems to have increased faculty coherence: there is no longer a huge divide between ‘those who mainly teach’ and ‘those who mainly conduct research’ in the university department in question. Therefore, we believe that the course may be an interesting alternative to other developments in accounting curricula that try to develop students’ research skills (Hoque, [30] ; Irving, [32] ; Jalbert, [33] ) and aim to increase faculty coherence.

The role of research in accounting education has been the subject of debate (Craig & Amernic, [11] ; Polster, [47] ; Watty, [59] ). There are various reasons why teaching has been separated from research in accounting curricula (Bui & Porter, [5] ). Among others, the requirements placed by universities on tenure-track and post-doc faculty members in terms of research output often does not match (potential) demands for more practice-based research carried out by students and practitioners (Geary, Kutcher, & Porco, [18] ). Ter Bogt and Scapens ([54] ), who examine two accounting and finance faculties, one in a Dutch university, and one in a UK-based university, conclude that the increased focus on research in both universities, using detailed performance management systems, has created anxiety among faculty members about how these systems are used. The authors suggest that this may inhibit creativity in teaching and innovation in accounting research, and result in research that has limited links with the complexities of organizational practice.

Leisyte, Enders, and De Boer ([36] ) claim that changing institutional environments in both the UK and the Netherlands have created a distinction between what is considered to be ‘good’ research and ‘good’ teaching. The authors put forward the view that the two may be increasingly difficult to combine, and argue that there is considerable controversy in the literature regarding whether it can be accomplished, and if so, how. The increased publication demands that are placed on faculty members seem to have resulted in growing workloads and a division within faculties between ‘those who teach’ and ‘those who do research’. The authors conclude that many of the faculty members they interviewed must have had the feeling that they were living in a ‘bad dream’ (Leisyte et al., [36] , p. 634).

Nevertheless, some encouraging examples of integrating research into accounting education are available. Hoque ([30] ) uses research papers to teach public sector accounting. Jalbert ([33] ) describes his experiences with publishing peer-reviewed research, using input from undergraduate accounting and finance students. Irving ([32] ) describes how specific research activities have been integrated in an undergraduate accounting course, using a pedagogical approach that fosters active learning. In the course, students read and discuss excerpts from research papers related to class topics. These excerpts and discussions are intended to enhance students’ understanding of these topics, and aim to develop their awareness of how accounting research and practice are intertwined (see also Hansen, [26] ; [27] ). After they have familiarized themselves with academic research papers and what is considered academic research, students are challenged to complete a research project, in which they have to address specific questions related to the course content, and have to test hypotheses using archival data. Survey results suggest that the majority of the students find that the course substantially improves their knowledge, skills, reading and research abilities.

The course discussed in this paper offers an extension and adaptation of Irving’s ([32] ) approach. As will be explicated in section ‘“Road map” of the AIS-research course’, in the AIS-research course students are explicitly treated as fellow researchers[3] (and not as research assistants) as they become actively involved in a research project that faculty members have devised. Students work together in groups and have to construct a (small) research project of their own based on the project developed by faculty members. They then have to carry out this project themselves and write up their results in a research paper. Among other things, students have to collect their own, as well as their fellow students’ data in the process. No use is made of archival data in any of these projects. Especially these aspects of the course help to substantially improve students’ research skills, as we will see in section ‘Impressions’.

Section ‘‘The ““making of” the AIS-research course’ describes how the AIS course at hand came about. In section ‘“Road map” of the AIS-research course’, the content and structure of the course is set out. In section ‘Impressions’, the course is critically evaluated. Several changes that have been implemented since 2010 are described. The final section contains our main conclusions, lists remaining problems and possible improvements, and discusses how far our conclusions may apply when a course is developed that does not necessarily prepare students for their Master of Science thesis, but aims to strengthen their research interests.

In early 2010, the management team of the university studied in this paper decided to make considerable room in their accounting curriculum for three research courses. They would have to be followed by all students who wished to obtain their Master of Science in Accountancy degree, and become a Certified Public Accountant (CPA) thereafter. The research courses had to cover the main subject areas that Dutch CPAs have to master: AIS, Financial Accounting (FA) and auditing. The university never intended to develop a single, general accounting research course. It was acknowledged that research traditions in AIS, FA, and auditing differed substantially, which made them difficult to combine in a single course. The three courses were developed by separate teams at the same time. Even though the content of the courses differs, their general structure is comparable. The AIS course, in particular, was developed for three reasons. First, there were growing concerns among senior faculty members about the exposure of accounting students to research, and academic research in particular. They had had this concern for a longer period of time, but it had intensified due to the enhanced thesis assessment criteria imposed by accreditation committees (as indicated in note 1), and due to several accounting scandals, such as Ahold, WorldCom and Enron, which had surfaced in previous years (Carnegie & Napier, [7] ). Second, in their evaluations of the accounting program, students had expressed that they wanted to be better prepared for their Master of Science thesis. The university is the market leader in accountancy in the Netherlands (approximately 35% of all accountancy students follow their Master of Science courses there - which amounts to approximately 350 students per academic year[4]). Being the market leader, and the main ‘deliverer’ of future accountants, the university’s program management decided that something had to be done, given the students’ comments. A third reason for the development of the course was an interest expressed by the university’s management team. They wished to create more faculty coherence and place a greater emphasis on research as a faculty activity, instead of as an individual activity. Especially the head of the accounting curriculum at that time thought that a research course could be a vehicle to accomplish this, and hence, fully supported its development.

The idea that the development of a research course might strengthen students’ research interests and increase faculty coherence stemmed from the university’s Part-time Master of Science program in controlling, where such a course had been a success for approximately five years.

In the Netherlands, there are 15 state-accredited[5] universities. Fourteen of these are public institutions that receive an annual budget from the government based on student enrollment numbers and the number of graduates per academic year. One university is privately funded and receives no government support. Having obtained a Bachelor’s degree, most students at Dutch universities go on to complete a Master’s. The Master of Science degree concludes with a written thesis which must be defended in front of an academic panel (as stated in note 1). In finance, economics, and accounting, Master of Science programs usually last one year. They are mostly offered as a full-time program. Only part-time students can enroll in the Master of Science program in accounting at the university in question. Hence, this program lasts two years instead of one. Students usually work from Monday to Thursday, and follow classes on Friday and/or Saturday.

The AIS course covered in this paper is part of the second year of the Master of Science in accounting program. There has been a clear shift in the program from more practice-oriented courses to research-oriented courses from the first to the second year. In the first year, students chiefly follow application-based courses in auditing, FA, AIS, and corporate governance, after having taken various introductory courses on these topics during their undergraduate studies. These courses mainly use methods of direct instruction and a ‘chalk and talk’ approach (Hansen, [27] ). The second year starts with an introductory course on research methods and scientific writing, and one of the three research courses mentioned in section ‘Introduction’ under ‘The “making of” the AIS-research course’. Students can express their preference for one of these courses, but they are encouraged to limit their choice to the topic (AIS, FA, or auditing) they wish to write their Master of Science thesis on. After the completion of the research course, the thesis needs to be written. So, research activities are emphasized in the second year of the Master of Science program at hand. Students have 14 weeks (between 3 and 4 months) to complete the AIS-research course. Nine ECTS (European Credit Transfer System) points are granted when the course is completed.[6] This implies that the course requires approximately 250 study hours. Sixteen sessions, each lasting three hours, are organized to facilitate the students’ learning process.[7]

The development of every course is a journey. Fink ([16] ) presents an iterative process to structure this journey in such a way that a course ultimately contains a clear link between its learning objectives, teaching and learning activities, and feedback and assessment procedures. The development team of the course in question, which consisted of an AIS professor and an associate professor of research methods and methodology, decided to follow Fink’s procedure after several rounds of open-ended discussions had not led to much progress. Fink ([16] , p. 8) states that one of the main questions that needs to be answered before the development of a course can start is: ‘What would distinguish students who have taken this course from students who have not?’ Discussions in the development team led to the conclusion that the ‘uniqueness’ of the AIS course would have to reside in both the outcome and the structure of the course. In terms of its outcome, conducting a research project in small groups together with faculty members, and writing a research paper supervised by faculty members were deemed important. In terms of the course’s structure, the development team wanted to ensure that all steps that are typically involved in conducting (empirical) research and writing a research paper would be properly understood and executed by each student, before they started working on their Master of Science thesis (see also Smith, [52] ). They also wanted to involve other faculty members in the ‘making of’ and teaching of the course, so as to enhance faculty coherence. Some choices had to be made, e.g. in terms of the topical focus of the course, and its placement within the academic year, to realize this. Both the AIS-related topics that were ultimately chosen, and the research approach that was used to frame these, are outlined below. More details on faculty involvement are presented in section ‘Steps followed’.

Around the turn of the century, there had been considerable debate in the information systems (IS) literature on what constituted relevant IS research. Part of the discussion focused on which topics and types of research would be important in the immediate future (McKnight, [42] ). McKnight argues that as consequence of this debate, especially AIS-related research

 …  expanded into such areas as electronic commerce, ethics, system user participation, group or individual decision aid technologies, assurance services, and knowledge management. Reference disciplines for the field are said to include not only computer science for the design science parts of the field, but also psychology, sociology, and philosophy  …  The definition of the AIS domain is still a work in progress  …  . ([42] , p. 86)

Despite the continuing debate on what constitutes (A)IS, it is widely accepted that enterprise resource planning (ERP) systems are an important research topic within AIS (Grabski, Leech, & Schmidt, [23] ; Rom & Rohde, [49] ). Therefore the decision was made to focus the AIS course on ERP-related research, especially since several faculty members found this an interesting topic that they wanted to engage with, or already had some knowledge of. The development team suggested that the focus of the course should remain the same for at least the next 2-3 years in order to stimulate faculty involvement. After 2-3 years, changes in topical focus were possible. These changes could be proposed by the faculty members who were running the course. The AIS professor in charge had to agree with the proposed changes, but was requested by the development team to only agree with such changes if future faculty involvement could be safeguarded.

Grabski et al. ([23] ) assert that in the AIS literature

 …  much has been written about ERP implementation and use. Unfortunately, much of the research (such as the large number of papers on critical success factors) has been survey-based, without strong underlying theory.  …  Unless a research paper is following a design science methodology  …  a strong theoretical development and a rigorous research design need to be utilized. (p. 64)

This prompted the development team to discuss possible research approaches that could be used to frame ERP-related research.

Various research approaches could have been adopted in a course such as this. Of course, the approach chosen for the course had to be in line with the knowledge and research interests of faculty members. Furthermore, the development team did not want to confuse students by confronting them with a large variety of research approaches, given their mostly limited exposure to academic research in the Master of Science program up to that point. Lee and Lings ([35] ) distinguish research approaches[9] based on realism (which argue that there is an objective reality that can be directly accessed and/or observed by researchers, who can ‘purge’ their own views by mixing methods, data sources, and/or other researchers’ views), and more interpretive approaches (suggesting that even though there may be a material world ‘out there’, a researcher’s access to it is severely limited. At best, personal understanding of a phenomenon that can be shared and discussed with others may be achieved through research. Research is therefore very much an [inter-]subjective affair). Following the research interests of most faculty members, the development team decided that the course’s research approach should be based on realism, adopting mainstream views on accounting research (Chua, [10] ; Merchant & Van der Stede, [44] ). Faculty members could decide themselves whether they wanted to be involved in the AIS course, but in order to increase the odds of them getting involved, it was decided to adopt this particular focus. In 2013, the emphasis of the AIS course shifted to accommodate interpretive research approaches and qualitative research methods, which had gradually become more important in (A)IS research (Walsham, [57] ; [58] ).[10] However, it turned out that the course length prohibited an in-depth treatment of this approach. Hence, this initiative was stopped in 2015. Interpretive research approaches and qualitative research methods are still discussed in the course, but only at the students’ request (see also section ‘Steps followed’).

After deciding on a particular topic and research approach, further discussions among the members of the development team, using Fink’s ([16] ) step-by-step guide, resulted in a list of learning objectives that the team found students should meet at the end of the course, given the Master of Science thesis that had to be written afterwards.[11] Not surprisingly, enhancing the research interests of students was one of the main objectives the team identified. It was also acknowledged that the course had to be competencies-based, as research typically combines knowledge (e.g. about what constitutes a ‘good’ theory), skills (e.g. running regressions in a software package such as SPSS), and certain attitudes (e.g. being critical). Following the example set by Van den Brink, Kokke, De Loo, Nederlof, and Verstegen ([56] ), who developed a competencies-based, distance-teaching course in management accounting at the Master of Science level, the development team developed the set of learning objectives presented below. The emphasis on knowledge (K), skills (S) and/or attitudes (A) each objective contains is indicated accordingly.

The development team thought that at the end of the AIS-research course, every student should be able to:

Working in teams of two or three students (to facilitate joint learning and keep the possibility of free-rider problems to a minimum), students have to prepare a research paper to assess what they have learned. The research paper is graded using the same criteria as the Master of Science thesis (see section ‘Group composition’ for details). According to Hansen ([26] , [27] ), this is the most far-reaching proficiency that economics graduates can acquire in a curriculum. In addition, based on the experiences of faculty members who had participated in the research course in the university’s Master of Science in controlling program, a written exam was introduced to assess students’ understanding of the papers they had to discuss (further details about the papers and exams are provided in section ‘Steps followed’). Faculty members found this was the best way to ensure that students would actually read and analyze the papers. According to Hansen ([27] ), such an exam demands a lower level of proficiency than carrying out a research project and writing a research paper. The development team decided to have students take the exam before they actively engaged in the research project of the course. After that, the course structure and a more detailed course outline were developed. These will be discussed next.

At the university in question, it is common practice to either offer plenary sessions/lectures in groups up to 450 students, or to work in smaller groups of 20-30 students. Smaller groups are organized when students have to acquire or develop certain skills and/or attitudes, as is primarily the case here (judging from the learning objectives in section ‘Learning objectives’). The Master of Science in accounting program was offered twice a year by the university in the 2010-2015 period (it has been offered three times per year since).[13] In the aftermath of the financial crisis, student enrollment numbers at the university varied greatly, but most students tended to enroll in the spring. Therefore, in the spring, the development team thought it might be advisable to allow two groups of students to follow the course if the time schedule of faculty members and student enrollment numbers could accommodate this. In the fall, a maximum of one group had to be accommodated. More groups could not be supervised in either period due to faculty involvement in other programs.

After discussions with faculty members who might be involved in the course, and who might (eventually) become course tutors,[14] it was decided that the research course should consist of 10 sets of activities, which will be referred to as ‘steps’ from here onwards. These steps are depicted in Table 1. The last column of Table 1 indicates which learning objectives (mentioned in section ‘Learning objectives’) are covered by which step(s). We have done this to highlight the didactical structure of the course (Fink, [16] ).

Course overview.

Note: Each letter refers to a learning objective. See section ‘Learning objectives’.

The goal of the first two steps in Table 1 is to acquaint students with how to conduct an in-depth treatment and discussion of academic research papers. Closely supervised by two course tutors who have considerable research experience, students work together in teams of two or three to familiarize themselves with these issues. In so doing, they also have to acquaint themselves with the research methods used in the papers. Steps 1-2 end with the written exam discussed in section ‘Chosen assessment’. The main goal of steps 3-10 is the development and execution of a research project, and the subsequent development of the research paper that is part of the course’s assessment procedure. Course tutors are mostly allocated by the AIS professor who supervises the course. Before every run of the course, the professor asks who in the department wishes to take on this role. The professor allocates tutors to particular groups of students provided that they have ample research experience.

We will elaborate on the steps shown in Table 1.

Step 1. Students have to read between 10 and 12 papers on a variety of AIS-related topics. The total number of papers is based on experiences from the Master of Science in controlling program. In that particular program, it had been found that students would usually read two to three papers per session in great detail, but tended to become very superficial if more papers had to be prepared.

Even though special attention is devoted to ERP-related research from step 3 onwards, steps 1 and 2 aim to expose students to the ‘broader field’ of (A)IS research. The AIS professor who was a member of the development team thought that this ‘field’ comprised specific topic areas, which concurred with his own research interests. He personally selected papers within these topic areas, asking the other member of the development team whether he agreed with his choices. Some minor changes were made to the list of papers he had proposed as a consequence. The following topics and papers were chosen for the first run of the course. It should be noted that, as indicated in section ‘Research approach’, the development team suggested that special attention be devoted to ERP-related research in the first 2-3 years. The research papers selected were on:

information quality (Gorla, Somers, & Wong, [21] ; Maas & Matĕjka, [38] ; Maines & Wahlen, [40] ; Nelson, Todd, & Wixom, [46] );

the implementation of ERP systems (Gattiker & Goodhue, [17] ; Hong & Kim, [29] );

organizational culture (Iivari & Huisman, [31] ; Morris & Venkatesh, [45] );

Enterprise Risk Management (ERM) (Beasley, Clune, & Hermanson, [2] ; Gordon, Loeb, & Tseng, [20] );

management control (Henri, [28] ; Widener, [60] ).

Every time the course is offered, course tutors discuss whether the list of topics and research papers needs to be modified, depending on what is trending in the current (A)IS literature, and the research preferences of faculty members. Typically, every year, some minor changes to the mandatory literature are made. All papers are taken from journals listed on the 2010 version of the Academic Journal Guide from the Association of Business Schools (ABS). Each paper has to be presented by a team, and has to be critically discussed by another team. Teams are mainly formed by the students themselves. Each team knows in advance which papers it has to present and discuss. Teams are strongly advised to use Smith’s ([52] , pp. 7-13) approach to structure their presentation and discussion. This approach consists of the following questions:

Why is a paper interesting/important?

Are the outcomes important?

Why is (are) the author(s) writing this paper now?

What is the research problem/question?

What theory (or theories) or theoretical framework underpins (underpin) the research?

Which key bodies of literature is the study predicated on?

Which research method(s) has (have) been chosen?

How has the sample been selected?

How have questions of validity and reliability been addressed?

How have the results been analyzed?

Are the conclusions and recommendations consistent with the findings?

In the plenary session that starts the course, course tutors illustrate how this approach can be used to frame a presentation and critical discussion using two sample papers, and show how these questions relate to research approaches based on realism.

Step 2. Immediately after the teams have read and discussed the papers, a written (individual) exam is organized which lasts one hour. Students are allowed to bring both the papers and their personal notes to the exam. They need to have read and critically discussed all of them beforehand, as the exam would be too long and complicated otherwise. The aim of the exam is to see whether students have grasped the content of the papers, can critically discuss them, and can argue how far the research that has been conducted really supports a paper’s findings. The questions that are asked are in line with Smith’s ([52] ) approach outlined above. Of course, only a limited number of questions can be asked in an exam that lasts an hour. An exam usually contains 4-5 questions, covering 4-5 papers. The questions are either literal replications of Smith’s questions mentioned above, or questions that can be linked to one or more of his questions. Examples of the latter type of questions are: ‘Could you please explain how someone might improve the internal validity of construct W used by author X in paper Y to measure Z?’, and ‘Do you agree with author X’s conclusion about Y in paper Z? Why/Why not?’

Step 3. After completing the proverbial ‘groundwork’ (reading, discussing, and taking the exam about AIS-related research papers in steps 1 and 2), a plenary session is organized about the research project students have to participate in. Course tutors explain why this particular research subject has been chosen, what the focus of the project is, and how the survey that students have to use has been prepared. In addition, organizational and logistical issues are discussed, e.g. about time schedules and how students can find and approach possible, suitable respondents to complete the survey (see also step 5 below). As stated, the course mainly focuses on the (perceived) implementation (effects) of ERP systems in Dutch SMEs, and the factors affecting this. This was an apparent research gap in the literature when the AIS course was developed (Esteves & Bohorquez, [14] ; Grabski et al., [23] ). With the help of another AIS professor, a survey was developed and extensively pretested among faculty members and practitioners. Students on the AIS course had no involvement in this as the survey had to be ready before the course started. Students only distributed the survey later on in the course (as stipulated in step 5). In the 2010 run of the course, it consisted of validated survey instruments on the following topics: past and current organizational performance (Eckartz, Wieringa, & Hillegersberg, [13] ; Shang & Seddon, [50] ), the decision-making processes that typically occur when an ERP system is adopted (Ugrin, [55] ), the perceived success of the implementation of an ERP system (Grabski & Leech, [22] ), business processes supported by an ERP system (Mantakas & Doukas, [41] ), critical success factors for an ERP implementation (Bradley, [4] ), the perceived information and system quality of an ERP system (Hong & Kim, [29] ; Nelson et al., [46] ), and organizational culture (Cameron & Quinn, [6] ). Furthermore, several general questions were included, e.g. about the functional/hierarchical position of a respondent in an organization, and his/her role in the ERP implementation.

Step 4. A research proposal (following Master of Science thesis guidelines) has to be prepared by each team. Of course, the survey instrument imposes limits on what a proposal can focus on: only those topics contained in the survey can be used to frame a proposal. This proposal is presented to and discussed with all other teams and a selection of faculty members (including, but not necessarily limited to, the two course tutors). The teams are the same as those involved in step 1. The proposals are graded by the course tutors. If a proposal has to be improved, students are given the opportunity to consult tutors during office hours if they have concrete questions. Sometimes, tutors ask students to contact them, especially when little progress has been made or when inaccuracies in a proposal remain.

Step 5. Each student has to find four respondents (employees of different firms) to be interviewed to complete the survey, and then distribute the survey. Students are asked to approach either two chief financial officers (CFO) or financial directors, or management accountants of two small or medium-sized (SME) organizations within their own network who have implemented an ERP system in the last three years. They also have to find either two CFOs, or financial directors, or management accountants of two SMEs who have not (yet) implemented an ERP system. As has been stated above, the university in question only has part-time students who combine work and study. Students’ networks can, therefore, mostly be used to find suitable respondents. If a student has difficulties in finding enough suitable respondents, other students, and in some cases, course tutors and other faculty members help them by suggesting respondents from their own network(s).

Step 6. All respondents have to complete the entire survey.[15] To accomplish this, students must be present when a respondent completes a survey, and ask him/her the questions that have been prepared. If there are (for example) 26 students, this yields 104 (26*4) responses/completed surveys.

Step 7. Teams assemble their data in pre-prepared datasheets (in Excel) which have been created by course tutors. When students hand in their data, they also have to hand in the completed paper versions of the surveys, and field notes that they may have taken (describing, for example, the attitude of the respondents when they completed the survey). Occasionally, tutors check whether respondents have indeed been asked by a particular student to participate. Since contact details have to be provided, this can be done fairly easily. Students are informed about these checks, and research ethics in general (Smith, [52] ; see also note 15) in the plenary session that initiates step 3. Despite several follow-up calls, ‘fake’ respondents/responses have been found in only very few cases (in a particular group, in a specific run of the course).

Step 8. Each team receives the data it needs, according to its research proposal, after its datasheet and the surveys have been submitted. A team receives the data of all respondents, not only from the respondents the team members have collected information from. This means that students have to do their best to collect quality data.

Step 9. Each team conducts its own data analyses. Course tutors are willing to organize sessions on statistical analysis or, for example, case research during that time, but only on request. Students have to indicate what they would like the session to be about approximately three days in advance. If nothing is proposed, a session is considered superfluous and is cancelled. Students thus have to regulate part of their learning process themselves (Zimmerman, [61] ).

Step 10. Each team prepares a presentation about its preliminary findings, which are discussed with other teams and faculty members (including, but not necessarily limited to, the two course tutors). Based on the comments they receive, their research paper is finalized and submitted. This usually happens two weeks later. The thesis is then graded by the course tutor. If a paper is considered substandard, a team has two weeks to improve it. If a re-written paper is still substandard, all the team members must follow the entire course again.

In 2013, two faculty members (who had been course tutors) found that the AIS course needed revising.[16] In the revised version of the course that they prepared, part of the literature discussed in steps 1 and 2 (see Table 1) had to be selected by the teams themselves, using the online journal databases (such as Ebsco Business Source Complete, and Elsevier ScienceDirect) the university had access to. In addition, teams did not always know in advance anymore when they had to present and discuss specific papers. This was decided on the spot by the course tutors. Hence, each team had to be prepared to present and discuss each paper, and some teams could be asked to present or review papers in multiple sessions. In addition, in between some of the sessions (when teams were preparing and refining their research proposal), contact via Twitter and Facebook with the course tutors was actively encouraged to reduce the number of one-to-one meetings. Students had to regulate these interactions themselves (Zimmerman, [61] ). This particular course also partially focused on qualitative research methods, in addition to quantitative research methods, which led to a change in supplementary reading materials (e.g. Alvesson, [1] ; Silverman, [51] ). Some of the research papers changed as well, and continued to change as the course was further developed (e.g. the papers on management control, at one point, comprised only Ezzamel & Burns, [15] and Qu & Cooper, [48] ). However, the general structure of the course, as indicated by the steps in Table 1, has always remained the same.

In the next section, we will present the evaluation results of the course.

Below we list and discuss the most important comments received from students and faculty members (including course tutors) who have been involved in the course. We have selected comments that we believe apply to all rounds of the course. After each course, and after every session during the first three runs of the course, impromptu evaluations were carried out involving course tutors and students. Course tutors often took field notes immediately after the evaluations, although this did not always happen in a structured fashion. Standardized evaluation forms were used during most of the runs as well (except for the first run in 2010), which also helped us to identify the most important comments.

Initially, students felt uncomfortable doing academic research. Many did not seem to see any link between the AIS course and their daily work (although most of them did see a link with their Master of Science thesis). Nor were they comfortable having to approach and interview respondents to complete a survey. Some students flatly refused to do this, especially when they were also the accountant of a particular organization. If this happened, they were asked to select different organizations/respondents. Some students needed to be convinced that they could find suitable respondents in their own network, and could also use each other’s networks to find respondents. Furthermore, they needed to be encouraged by course tutors to contact potential respondents early on in the course, as not all respondents might have been willing to participate or reply quickly. During the third round of the course, a number of students had inadvertently approached some of the same respondents that had already been interviewed, as a specific organization was part of their network as well. Although some respondents of these organizations declined to participate in the survey, others took part in it again. These responses had to be removed from the dataset. Therefore, in 2012 (slightly earlier than the development team had expected), the team decided to change the emphasis of the survey from ERP implementations to internal control issues. The same happened in 2015 (after the involvement of the authors in the course had ended). After this the emphasis of the survey shifted towards IT controls, which is still the focus of the survey in 2017.

In addition, the fact that students were more or less ‘forced’ to accept choices made by faculty members (for example, in the construction of the course, the selection of the research topics, and the survey instrument) did not always go down well. In particular, questions were raised about the survey approach, and the fact that it focused only on one major topic (at least, in the 2010-2012 period): ERP implementations in Dutch SMEs. However, after the completion of the course, they stated that these choices helped them focus their own research and manage their learning process. The focus on particular topical areas is, we believe, inherent to, or perhaps even necessary in a course such as this. In 2015, after the involvement of the authors of this paper in the course had ended, the new course tutors introduced an extra session on research methods based on students’ comments. This session is sometimes cancelled (if there is no input from students), but when it is organized, it mainly focuses on other research methods than surveys and interviews that students consider using in their Master of Science thesis. This alleviated the emphasis the course originally had on particular research methods. Also, the changing role of the course tutors during the course tended to cause some problems. As the course progressed, tutors would critically evaluate and discuss the research done by students, but would not give them step-by-step instructions anymore on what they had to do next (cf. Hansen, [27] ). Especially after the written exam, the tutors started to work as coaches, avoiding methods of direct instruction as much as possible (cf. Van den Brink et al., [56] ). Although (most) course tutors felt this was in line with the decision to treat students as fellow researchers (as indicated in section ‘Contribution’), and conformed to the way in which they would be treated when writing their Master of Science thesis, students indicated that they would have liked more concrete suggestions from course tutors throughout the course. However, after students had completed their Master of Science thesis, they often argued that, with hindsight, they were glad they had not received much direction in the second part of the AIS course. Hence, it was decided to keep the role of course tutors as the development team had envisioned, and refrain from methods of direct instruction as much as possible.

In the impromptu evaluation sessions after the first runs of the course, students frequently indicated that they now had a different work perspective: they had a better understanding of the role of research in their work, and felt less inhibited to present their own views when problems occurred. At the end of each run of the course, students were also asked to evaluate it (anonymously) on a scale from 1 to 10, in which a higher number indicates a better grade. The course has an (average) grade of 7.30, and has never received a grade below 7.14. The maximum score was 7.47, which was awarded in the spring of 2012.[17]

We will now present some statistical analyses based on data taken from the standardized (electronic) evaluation forms that the university’s program management distributed after every run of the course. Since these forms are different for each of the three research courses that were developed, we will only focus on the AIS course. Unfortunately, individual evaluation forms have not been kept. Only average scores across all students are available for each run of the course. Questions on the evaluation form include ‘How many hours did you study per week?’, ‘How would you grade tutor X?’, ‘How many sessions did you miss?’ and ‘How would you grade your own learning process?’ Most questions on the evaluation form were measured on a five-point Likert scale, on which a higher score indicated greater satisfaction/quality.

In Table 2, we present Spearman correlation coefficients between the course grade (GRADE), the average number of study hours per week (HOURS), the course tutors’ average grade (TUTOR), the average number of classes missed (MISSED), the average self-assessment score of the students’ own learning process (LEARN) and the average grade for the course location (which was always on campus) (LOCAT). It should be noted that the sample size is small, given that only average scores are available, the number of runs of the course between 2010 and 2015 was limited, and the content of the evaluation form changed after 2010. In the period that the authors of this paper were involved in the course, it was offered nine times, in which the first run was evaluated using a completely different evaluation form, which hindered comparisons with the other runs. Consequently, our sample size is eight (n = 8). The small sample size indicates that we have to use non-parametric tests to analyze the data (Hair, Black, Babin, & Anderson, [25] ).

Spearman correlation coefficients (n = 8).

GRADE = average course grade given by students (on a 1-5 scale; 5 indicates the highest possible grade); HOURS = students’ estimate of average number of study hours per week; TUTOR = course tutors’ average grade given by students (measured on a 1-5 scale; 5 indicates the highest possible grade); MISSED = average number of classes students missed (out of 16); LEARN = self-assessment score of students’ own learning process (measured on a 1-5 scale; 5 indicates the highest possible grade); LOCAT = students’ grade for the quality of the course location on campus (measured on a 1-5 scale; 5 indicates the highest possible grade).

* = significant at 10% level; ** = significant at 5% level; *** = significant at 1% level (all two-tailed).

As can be seen in Table 2, there are three statistically significant correlations at the 5% and 10% level of significance: between GRADE and HOURS (.643), GRADE and TUTOR (.738), and LEARN and MISSED (.743). All correlations are positive, which is only surprising for the correlation between LEARN and MISSED. Apparently, the more sessions students have missed, the higher they rate their own learning process. This suggests that sessions are deemed more useful when they have been missed more often. An alternative, and we feel, slightly less convincing explanation is that when an individual’s learning curve is steep, he/she believes that more sessions may be skipped (since one’s learning effects are considered sufficiently substantial, or the teaching is not deemed good enough).

We also conducted a backward regression, in which GRADE was regressed on all of the other variables in Table 2. This regression was carried out to identify the most parsimonious model fitting the data (Hair et al., [25] ). Given the relatively high and significant correlation between LEARN and MISSED, which signifies that multicollinearity may be present, MISSED was first removed from the set of explanatory variables (Hair et al., [25] ). In that case, only TUTOR remained as a significant explanatory variable in the model (b = .192; p = .051; R2 = .495).[18] The positive sign indicates that when a tutor is graded more favorably, the course is as well. The inclusion of MISSED, and the deletion of LEARN from the regression model, yields the same results. The results have been summarized in Table 3.

Backward regression results for GRADE (with either LEARN or MISSED deleted) (n = 8).

Note: Variable definitions can be found in Table 2. P-values are shown between brackets. The standardized regression coefficient for TUTOR is.704.

* = significant at 10% level; ** = significant at 5% level; *** = significant at 1% level (all two-tailed).

Interestingly, individual evaluation forms containing students’ perceptions about their Master of Science thesis have been kept. On these forms, students are requested to provide a general reflection on the thesis, and their studies as a whole. We divided the students who completed this evaluation form into two groups: students who started working on their AIS thesis after June 2008, but before January 2011 (when the first run of the AIS course ended), and students who started between January 2011 and June 2013. These groups comprise 97 and 126 students respectively. The average time spent on the thesis is 500-600 hours for the first group, and 300-400 hours for the second group. This suggests a considerable decline in throughput time of the thesis after the introduction of the AIS course. The group of students that started between June 2013 and March 2015 (37 students) also had an average throughput time of 300-400 hours. Although causalities cannot be established, given these results, it would appear that the apparent drop in throughput time of the Master of Science thesis coincided with the introduction of the AIS course. In terms of thesis quality, quality increased between January 2011 and June 2013, as compared to the period between June 2008 and January 2011. The average thesis grade rose from 6.56 to 6.98 (on a 1-10 scale; 10 indicates the highest possible grade). The average grade is slightly lower for the group that started with its AIS Master of Science thesis between July 2013 and March 2015 (6.78), but this grade is still higher than for the cohort that started between June 2008 and January 2011. Table 4 lists these findings.

Evaluation scores after completion of the Master of Science thesis.

We can summarize the feedback students provided as follows. Even though the AIS course is deemed valuable by most students, they experienced some difficulties. These chiefly relate to the preset nature of specific elements in the course (the use of surveys, the detection of suitable respondents to interview and take surveys from, and the gradual decrease in methods of direct instruction). With hindsight, students recognize that they underwent a considerable learning process. Once they have completed their Master of Science thesis, students tend to better understand the relevance of the AIS course, also for their daily work. Furthermore, the throughput time of students’ Master of Science theses has decreased, and thesis quality has increased. These results have been recapitulated in Table 5, together with some of the measures that have been taken to alleviate the problems students encountered. Some of these measures are discussed in section ‘Costs and benefits’.

Summary of student feedback, including measures taken to alleviate problems.

Course tutors find the AIS-research course a stimulating course to teach, even though the preparation time and the number of contact hours per week can be quite substantial (including six hours of teaching per week [in most weeks]. It is fair to assume that between one-two hours have to be spent on questions, assessing research proposals, and presentations per team per week. This means that during each run of the course, tutors need approximately three days a week to work on the course). Since two faculty members usually act as course tutors (in order to create an open research setting, in which the importance of debates between students and tutors is stressed), some believe that the workload is too high.

However, at the same time, the learning effects among faculty members seem to be considerable. Faculty members indicate that they are working together more closely, as many of them became involved in the same course and had to work in pairs. Moreover, some faculty members who were not used to doing research (anymore) have become interested in research (again). Those faculty members who felt uncomfortable using different approaches to teaching than methods of direct instruction, or were not used to working in pairs, could follow a special course the university organized after the first three runs of the course. This course boosted the confidence level of most of these tutors.

Intriguingly, course tutors encountered hardly any free-rider problems, which could have been a distinct possibility especially in the steps in section ‘Steps followed’ in which group work had to be carried out. Although groups were sometimes ill-prepared to present or discuss a paper, they tended to withdraw from presenting when this happened, perhaps since they thought their fellow students would not react favorably if they did take the stage. However, when more groups were ill-prepared (which would usually happen in the spring, when students were in their ‘busy season’), and the groups in question knew about this before a session started, they often would present despite their less than optimal preparation. This resulted in meetings which were not very productive. This problem turned out to be difficult to circumvent (see section ‘Final remarks’). As the course progressed, and students’ attention shifted to their own research project, literally no free-rider problems were detected. There were some dysfunctional groups due to personal issues, but these groups were either split once this became clear, or they could continue after a few discussions with their course tutors. It should be kept in mind that Master of Science theses in the Netherlands are individual assignments. Students therefore knew that they had to master the ‘craft’ of academic research themselves, with the thesis lurking in the background. This may also have alleviated potential free-rider problems.

Overall, the level of the research paper that teams of students prepare is reasonable. Grades typically range from 5 to 8 on a 1-10 scale (a 5 being substandard, leading to an additional assignment to improve the paper). Complete rewrites rarely occur, and often involve dysfunctional teams. There have also been fewer retakes for the Master of Science thesis than before the course was initiated. Whereas before the introduction of the AIS course about 10% of the theses had to be rewritten, this now happens in less than 4% of the cases. As a by-product, the course provided faculty members with a unique dataset that has gradually grown in size. Faculty members, in cooperation with students, have started to use this database for publications in ABS-ranked journals.

Table 6 summarizes the results. Some of the ways in which the problems that course tutors signaled have been alleviated are discussed in the following section.

Summary of feedback from faculty members (including course tutors), including measures taken to alleviate problems.

The development team gave itself nine months to develop the course. They did not work on the course full time, however. On average, a day and a half a week was spent on it. This included all activities, from the selection of research papers, the development of the course program (following Fink, [16] ), informal and more formal meetings between the course developers, potential course tutors and associated staff, the preparation and pretest of the survey, the creation of a pool of written exams, and the preparation of the Excel sheets that students could use to assemble their data. Subsequent changes after the course started (e.g. the inclusion of qualitative research methods, the selection of different research papers, and adaptations of the survey) took considerably less time. At most, two weeks were spent on implementing all of these changes.

As can be seen in Tables 5 and 6, there have been substantial benefits in term of students’ skills, the throughput time of Master of Science theses, and the development of databases that can be used for research purposes by faculty members. Faculty coherence has increased (as indicated in section ‘Faculty members’), and there is now a larger pool of tutors that can be called upon to offer the course than when it started. Originally, there were three faculty members (out of eight) who could teach the course. This number has now increased to seven, including two people from other departments. One extra faculty member has been appointed since the start of the course, and one person has retired. This demonstrates that the interest, knowledge, and expertise among faculty members about (AIS) research have gradually grown.

With hindsight, the biggest challenge that the development team met was convincing a rather conservative student population that was very much used to methods of direct instruction to embrace different approaches to teaching that required substantial personal responsibility, without giving any guarantee as to what they would attain or could learn exactly when they did this. Not all students participated willingly in the course, especially in the first two runs. It took a number of runs (and word of mouth) for students’ attitude to change. Course tutors also gradually devoted more attention to the differences and challenges the course posed vis-à-vis other courses in the accounting curriculum in the first sessions of the course, which helped to alleviate some of these problems.

Interestingly, the fact that course tutors were graded by students after each run of the course turned out to have adverse effects. During a particular run of the course, one particular course tutor suddenly indicated that he felt pressured to adapt the course’s teaching methods. He then intentionally started to apply methods of direct instruction again. As a consequence of this incident, in 2013 two course tutors asked the university’s program management to discard eventual negative evaluation scores after they had adapted the course in the way described in section ‘Course revisions’. This request was granted, and has always been granted if the course has subsequently been revised. Program management has found that this decreases the possibility that adverse effects will occur. Two students from the cohort that followed the course after the 2013 revision, are now among its biggest advocates. They feel they have progressed, both as practitioners and researchers, because of the course. They even claim to have become better human beings. These students sometimes open the AIS course, and talk about their own learning process and the relevance of the course, both for work purposes, and for one’s studies (as indicated in Table 6).

The 2017 version of the course, which the authors of this paper are no longer involved in, contains the following topics and (preselected) literature[19]:

ERM (Beasley et al., [2] ; Gordon et al., [20] ; McShane, Nair, & Rustambekov, [43] );

effectiveness of IT (Bhatt & Grover, [3] ; Chae, Koh, & Prybutok, [9] ; Lu & Ramamurthy, [37] );

the implementation of ERP-systems (Hong & Kim, [29] ; Ugrin, [55] );

E-governance (Carter & Bélanger, [8] ; Kim & Lee, [34] ; Teo, Srivastava, & Jiang, [53] ;

organizational culture (Gregory, Harris, Armenakis, & Shook, [24] ; Iivari & Huisman, [31] ; Morris & Venkatesh, [45] ).

A newly appointed AIS professor now supervises the course, but she has asked an assistant professor and a PhD researcher with a lot of research experience to organize it, following their own judgment and research interests.

Even though the AIS course discussed in this paper is highly regarded by faculty members and (most) students, and seems to have enhanced students’ research skills, some issues would be tackled differently if the course were developed today. Institutes that wish to develop courses along similar lines might use these insights in their own development plans.

There are three issues we would like to single out. First, the development team could have included 1-2 students, instead of only faculty members. This might have helped to signal some of the problems that were detected as the course was run. An example was listed in section ‘Costs and benefits’: moving away from methods of direct instruction could have been explained clearly and/or could have been introduced more gradually than the development team originally thought.

Second, course tutors sometimes noticed that teams had a strict division of tasks when preparing their research paper, and consequently failed to check the quality of each other’s work. For example, in a team, one team member might complete the literature review and develop hypotheses, whereas the other member(s) carried out statistical analyses in SPSS and drew inferences. The paper was written accordingly, with often huge differences in quality and writing style, depending on who had done what. In such cases, course tutors had to explicitly inform students that all team members would be fully responsible for their research paper, no matter who had done what. Since the university’s rules and regulations for plagiarism were added to the course outline in 2015, which stipulate that research is a joint activity, this problem seems to have vanished.

Third, in terms of course didactics, it would have been worth incorporating specific issues that accountancy students specializing in AIS research typically meet when writing their Master of Science thesis to a greater extent. An example is that students often seemed to have difficulties formulating accurate research questions that contribute to existing theoretical frameworks and/or working practices. Instead of tackling this problem during the Master of Science thesis, it could be (and actually, now is being) tackled in the AIS-research course.

Even though it could be argued that much of an accountant’s work involves research, this is not often recognized by accountants and accountancy students (Bui & Porter, [5] ). Apparently, the AIS-research course that we have discussed in this paper helps students to recognize the importance and relevance of research in academia as well as in their daily practice. Evaluation scores suggest that the course adequately prepares students for their Master of Science thesis, and helps to spur on their research interests. The throughput time of Master of Science theses has decreased, and thesis quality has increased. Moreover, students seem to be able to discuss various research issues which relate both to their daily work and to their Master of Science thesis with one another, and with faculty members, in a more detailed and in-depth manner than before. Therefore, we believe that the course is an interesting alternative to other developments in accounting curricula that try to develop students’ research skills (Hoque, [30] ; Irving, [32] ; Jalbert, [33] ). In addition, it seems that the ‘gap’ between ‘those who teach’ and ‘those who do research’ in the faculty has decreased (Leisyte et al., [36] ). Course tutors have to be experienced researchers, and as the popularity of the course has grown, faculty members with little research exposure and/or experience have become interested in doing research (again). This is reflected in the size of the pool of course tutors that can now be called upon to teach the course. However, a specific problem remains - even in the 2017 version of the course. Some students tend to read and discuss the research papers they have been allotted superficially, even though they have ample time and are offered considerable training on how to do this. This problem still needs to be adequately tackled.[20]

It is worth noting that a research course such as the one presented here may be useful even if students do not need to write a Master’s thesis to complete their studies, as is the case in many Master’s programs in North America. Writing research papers typically is an important ingredient of these programs, as is scientific thinking and taking responsibility for one’s own learning process. We believe the general structure of the course could, therefore, very much remain the same, even if the type of paper that is required were different - as long as it is research that is ‘on the table’. As stated, students have to structure part of their activities themselves in this course, and the structure and content of the course could be modified to accommodate their independence to a greater or lesser extent. We could envision that under such circumstances, some of the steps mentioned in section ‘Steps followed’ may receive different emphases but will still be relevant.

No potential conflict of interest was reported by the authors.