- A comparative approach to evaluation

in: Online Educa Berlin, 4^th International Conference on Technology Supported Learning, Book of Abstracts, Dec 2 – 4, 1998 Berlin, International WHERE + HOW: Bonn 1998, pg. 287 - 292

In the midst of an institutional rush toward distance education, some crucial issues are in danger of being forgotten. It was felt that there are two critical needs:

1. There is a basic need for a faculty development training program in which new distance education faculty can develop a broader perspective based on general foundations of distance education, on distance education theory, pedagogical models, on positive examples of good practice, critical knowledge and skills in the field.
2. There is also a need for a global perspective among distance education faculty so that they can benefit from the knowledge of how other institutions approach distance education and solve problems, particularly in cross-border and cross-cultural contexts.

Given that need for faculty development and training in distance education, the authors submitted a proposal in 1995 to participate in the "Global Distance Learning Initiative" , a series of grants for research and exploration by the International Council for Open and Distance Education (ICDE) in collarboration with the AT&T Foundation. They were awarded a grant for 1996/97.

A significant part of the ICDE/AT&T grant was dedicated to evaluation. The main part of the evaluation was done externally by Helmut Fritsch of the FernUniversität in Germany, who has already published his findings (http://www.fernuni-hagen.de/ZIFF/evirs.htm).

The extent of the efforts put into the experimental phase in 1996 and 1997 have contributed to the permanency of the offering. The redesigned Seminar January through March 1998 was able to be offered as a self-supporting enterprise with a participant’s fee of USD 580.00 for a maximum of 45 participants. A waiting list for applicants and the participants’ positive feedback about their experience with the virtual seminar resulted in a third seminar being offered September through November 1998.

The results of the final questionnaire for the second seminar in 1998 are available (http://www.fernuni-hagen.de/ZIFF/docudone.htm) and allow a comparison of the participants’ feedbacks to both virtual seminars, one which was supported by a substantial grant, and the other which was self-supporting.

The general strategy of the Seminar was to encourage faculty development in two areas:

1. Theory in distance education. By this we meant the foundations, history, theories, models and the technology-base of distance education.
2. Practice in distance education. By this we meant the applications and the actual process of developing, delivering, supporting, guiding, evaluating and administering distance education courses and programs.

The original objectives of the virtual seminar for professional development in distance education were:

1. Develop a 10 week course entitled "A Virtual Seminar for University Faculty and Administrators: Professional Development in Distance Education".
2. Offer the course via the Internet to a maximum of 45 participants January through March 1997, as a pilot project, prior to a global implementation.
3. Evaluate the seminar outcomes and revise the course.

A decision was made early on to offer the seminar via the Internet using easily accessible and wide-spread computer-based technologies. Given the 10 week syllabus, the ongoing work commitments of the participants, and their location in different time-zones, it was also decided that asynchronous computer communication appeared to be the most appropriate mode of communication, and it was decided that web-based computer conferencing would best to support the structure and objectives of the Seminar.

The web-based conferencing system that was selected was HyperNews, which is a Unix-based "threaded" system. In addition, the general Seminar environment was supported by :

• A "home" page , which led to the various weekly conference discussions, as well as allowed for general announcements;
• A brief self-submitted biography and photograph from each participant;
• An introduction/orientation document; which explained the logistics, schedule and other details of the planned virtual seminar;
• A readings pages, which listed the readings for each week;
• A short tutorial for learning to use the web-based conferencing system;
• A separate computer conference to obtain technical assistance, as well as an e-mail link to the conferencing administrator.

The core professional development strategy of the Seminar is one of combining the idea of master practitioner with that of peer interaction. Each part of the syllabus was achieved by common discussion of a topic supported by interaction with a well qualified expert in the field of distance education. Thus, the design of the Seminar was one of a meeting of peers and not one of a relationship between students and teacher. The Seminar feature of visiting experts was a unique opportunity to interact (in almost real-time) with distinguished scholars and practitioners. It was reasoned that it would be a strong motivator for faculty to persist in the Seminar, since this opportunity would not normally be available elsewhere.

The Seminar Leaders were well aware that they were dealing with qualified professionals who were actively employed in academia, business or government. We were also aware that many of the academics were engaged in a regular teaching calendar. In other words, the participants were a group of working professionals who had commitments other than the Seminar. The Seminar Leaders jointly provided the overall frame for each of the weekly discussions. Both Seminar Leaders had a wide variety of experiences in distance education to share with the participants.

The syllabus for the first Seminar in 1997 was as follows:

Week 1: Introduction and practice with the conferencing system

Week 2: Foundations of Distance Education - with guest expert Börje Holmberg,

Week 3: Institutional Models of Distance Education - with Gary Miller

Week 4: Theories of Distance Education - with Otto Peters

Week 5: Technology of Distance Education - with Tony Bates

Week 6: Orientation into Distance Education Applications and Project Planning

Week 7: Student Support and Assessment

Week 8: Instructional Design and Course Development

Week 9: Technology Applications

Week 10: Project Reports, Summary and Conclusion

The redesigned syllabus for the Seminar in 1998 was as follows:

Pre-Seminar week: Module 1 Introduction and practice with the conferencing system

Week 1 and 2: Module 2 Foundations of Distance Education - with Börje Holmberg in week 2

Week 3 and 4: Module 3 Institutional Models of Distance Education - with Gary Miller in week 4

Week 5 and 6: Module 4 Theories of Distance Education - with Otto Peters in week 6

Week 7 and 8: Module 5 Technology of Distance Education - with Tony Bates in week 8

Week 9 and 10: Module 6 Practical Applications in Distance Education on Student Support, Tutoring and Counseling, Instructional Design and Course Development and other Topics of Interest - with Ulrich Bernath and Eugene Rubin in week 9 and 10

Week 10: Module 7 Summary and Conclusion

The key ingredient of the Seminars, which had the most relevance towards the goals of each Seminar and indicated in both final (positive) evaluations by the participants, was the participation and contribution of the four distinguished experts. Based on the selected readings, participants and experts created, in terms of quantity, almost similar voluminous dialogues and discussions. Surprisingly, the content of the discussions and the issues raised by the participants were significantly different from each other in both Seminars.

The experts also gave positive reports regarding their own experiences in the Seminar. They uniformly reported that the experience had been enjoyable and worth their time and this was supported by the fact they all agreed to participate the second time and now have agreed to be involved in a third Seminar. This information is critical to the continuing success of the Seminar as an ongoing professional development activity, since the "expert model" is at the core of the Seminar’s design.

The global aspect of the Seminar was also important for its success. By being globally accessible via the Internet, the content and interaction allowed participants to differentiate and generalize across cultural borders and among the diverse practices within the field of distance education. It gave depth to the learning and forced the participants to think beyond their own cultural and environmental constraints.

The Seminar was also an example of distance education in practice. One of the primary goals of the Seminar was to inform distance educators about issues related to the practice of distance education. We essentially "practiced what we preached."

The Seminar fee, introduced with the second Seminar had no visible impact on the surmount of applications as well as on the participants’ positive feedback about their experience with the virtual seminar.

Our virtual Seminar experience is reading and writing and demands much of the participant’s time. The written contributions in the asynchronous discussion process differ from a synchronous and flighty chat and are fundamentally different from a conventional seminar. Engaging in a virtual seminar and using computer conferencing is a much more reflective process than face-to-face interaction. One types out one’s thoughts, rereads them, often edits them or adds to them, and sometimes even spell checks them. After carefully inspecting what one has written, they are then submitted for others to read. The meaning of the written word is measured and sincere, and it persists. It can be read and reread by others long after the termination of the seminar.

Conventional seminars do not allow all participants to contribute at once. They usually do not encourage a response from each and every individual. Moreover, it is difficult to attend to and keep track of a long sequence of oral contributions. A long list of written contributions is treated differently. You can stop when you wish and easily compare and contrast various contributions. You can go back and reread for clarification. The asynchronous computer conference is, in a way, a renaissance of reading and writing communication, but incredibly fast and ubiquitous. We can now hope that it will bring us new and extended opportunities for teaching, training and learning, regardless of time and space constraints.

The discussion process in the virtual seminar needs leadership, direction and moderation to best use the opportunities offered by the media and the technology. In particular, it is important to get as much of the activities on the "surface" as possible.

There is also an emotional component to the seminar. We felt that this emotional component was critical to the success of the seminar.

In summary, both evaluations appear to indicate that the virtual seminar met the majority of expectations of their respective participants. There were interesting similarities among the two participant populations, despite their different cultural profiles, which could likely be ascribed to their common interest in the design of our virtual Seminar for professional development in distance education.

The authors: Ulrich Bernath and Eugene Rubin

Reference: Ulrich Bernath, Eugene Rubin, A Virtual Seminar for International Professional Development in Distance Education, in: INFORMATIK FORUM, Vol 12 (1), March 1998, pp. 18-23

Appendix: The following tables show an striking accordance in the feedback given to both virtual seminars:

1997: number of responses = 33

1998: number of responses = 27