Analysis of student interaction in Project-based learning

(1)

Analysis of student interaction in Project-based learning.

Md Jahid Hossain

Master’s thesis

School of Computing Computer Science

May 2021

(2)

ii

UNIVERSITY OF EASTERN FINLAND, Faculty of Science and Forestry, Joensuu School of Computing

Computer Science

Analysis of student interaction in Project-based learning.

Master’s thesis, 76 p., 3 Appendix (16 p.)

Supervisor of the Master’s Thesis: Dr. Mohamed Saqr (Ph.D.), Senior researcher May 2021

Abstract

Network analysis is one of the decent approaches to categorize student’s interaction while solving a group project. The interaction pattern of students can be modeled using a cognitive network. By analyzing the interaction network, it is possible to categorize student’s performance changes over time during a course and project.

Student interaction analysis can help an instructor judge the student better and design the study module based on its necessity. In this research, we have analyzed student’s interaction by using factual data obtained from the Learning Analytics course.

Initially, we have designed a coding framework based on the Community of Inquiry coding template. The whole dataset is coded, and a portion is used to crosscheck the inter-rater reliability of coding. Social Network Analysis (SNA) and Epistemic Network Analysis (ENA) are applied to analyze student’s degree, centrality values, network pattern, interaction pattern, and cognitive connection. Content analysis is done using the developed coding framework and topic-based coding. It shows student’s areas of concentration and quality of interaction. Both qualitative and quantitative method is used in this research. Results of this research are demonstrated using visualization and statistical analysis of data.

Keywords:

Project-based learning, interaction-pattern, cognitive-network, the community of inquiry, inter-rater reliability, Learning analytics, Social network analysis, Epistemic network analysis, content analysis.

(3)

iii

Acknowledgments

This thesis was done at the School of Computing, University of Eastern Finland, during spring 2021.

After a long year, I have finished this thesis work. At this precious moment, I would like to express my gratitude to my family members, my wife Kaniz Ferdousi, and daughter Rumaisa Hossain for their support and blessing. My heartiest and more sincere gratitude to the patron of this work, without whom I could not accomplish this onerous task.

This work is mainly indebted to Dr. Mohammed Saqr, Senior Researcher, School of Computing, University of Eastern Finland, who has supervised the work. It is a great pleasure for me to acknowledge his profound gratitude to his supervisor for constant advice, constructive criticisms, and valuable guidance. His encouragement helped in every stage of accomplishment of this work. His immense help guided me to complete the work and to prepare this thesis. Without his patience, concern, and efforts, this thesis would not have been attainable.

Next, I would like to thank Dr. Oili Kohonen, Coordinator, IMPIT program, The University of Eastern Finland, for her unconditional support, valuable advice, and encouragement during the master’s program. Finally, sincere appreciation goes to my classmates, friends, and colleagues for their encouragement and support during this thesis.

Md Jahid Hossain May 2021

Joensuu, Finland

(4)

iv

List of abbreviations

SNA Social Network Analysis PjBL Project-Based Learning ENA Epistemic Network Analysis LMS Learning Management System

LA Learning Analytics

CoI Community of Inquiry

(5)

v

Chapter 1: Introduction

This chapter briefly describes the research area, thesis planning, objectives of the thesis, organization of this thesis, and a review of the literature associated with this thesis.

1.1 Introduction

Project-based learning is a vital concept representing the socio-cultural perceptions of knowledge in educational environments [1]. This method introduces an ideal platform to provide problem-solving prospects to the students that presently belong to real-world problems. It is necessary to note both the procedure and invention while students are working on a project using an online environment [2]. For an expressive educational practice, interaction is an essential element irrespective of the learning framework; either it can be a traditional classroom environment or an online infrastructure [3]. This research investigates the patterns and the quality of online interaction during Project-based learning.In the past few years, the latest invention and technologies allow analyzing group interaction data from several perspectives [4]. Social network analysis and Epistemic network analysis are significant inventions for interaction analysis in recent years. In most cases, online education’s significance generally depends on the quality of interaction [5]. Due to the inadequacy of social interaction in online platforms, learning may become unsuccessful [6].

Content analysis is another effective method to analyze online learning interactions [7]. By considering participant’s skills compare with communication and teamwork, Social network analysis introduces valuable methods to establish investigation and experiments [8]. Social network analysis is one of the growing widespread methods of studying communication forms between students in online learning environments [9]. Epistemic network analysis is useful for quantitative analysis of interactions and the cognitive connection between the coded interactions [10].

(8)

2

The content of this research is from a graduate course that discovers the experimental and theoretical methodologies to instructional design. This course, fundamentally, is delivered in a blended classroom with a selection of online learning events. In addition to the course activities, an online group project was included in the course.

Even though students could meet face-to-face, but they were instructed to communicate online for the group project.

The interaction data is collected from the Learning Analytics course of the University of Eastern Finland for two consecutive years. In both years’ students were assigned to small groups by the instructor using Learning Management System and distributed a project for each group. Learning Management System (LMS) Moodle records student interactions while solving the group project and other course activities. Before applying Epistemic network analysis and content analysis, all the interactions were coded using a standard coding framework. The coding framework used in this research is developed based on the Community of Inquiry (CoI) coding template [11]. Interactions were also coded using the subject of the interactions.

Social network analysis is applied to analyze the quantitative aspect of student’s interactions. Epistemic network analysis analyzes the interaction pattern, connection- weight among codes, compares different groups by visualization and quantitative analysis.

1.2 The Community of Inquiry model

The Community of Inquiry framework is a prototype for learning methods in online and face-to-face learning environments. It is a functional coding template widely used for coding students’ interaction in each educational platform, which requires three significant learning events, known as teaching presence, cognitive presence, and social presence [11].

The cognitive presence, teaching presence, and social presence establishes the CoI model. The intersection area between cognitive and teaching presences represents

(9)

3

how instructors model their communication medium and students compare their thoughts and share know-how.

Figure 1.1: Community of Inquiry Model

Cognitive and social presence overlays mainly in discourse and knowledge areas and shows students’ cooperation in an educational atmosphere. Teaching and social presence intersection represent the choice of infrastructures like platforms and software to assist student’s learning.

The Community of Inquiry model depends on message or data analysis to find out interactions among participants of a specific community [12]. For efficient coding of messages using CoI to generate meaningful data, a coding scheme is required [13].

The Community of Inquiry framework is applied to more than 500 students registered in a combined online course in different polytechnic schools and universities in Portugal. The result shows that the polytechnic students represent a robust community than the university students. Most of the participants from polytechnic schools represented a reasonable to a high level of understanding [14]. A survey was conducted over 1500 students at two United States higher education institutes in seven different disciplines to identify the applicability of the Community of Inquiry framework by analyzing the learning differences among the disciplines.

Content Selection Environment Setup

Supportive Discourse

(10)

4

Survey results show that science and technical course participants evaluated the community of inquiry presence magnitudes higher than business or management participants [15]. The Community of Inquiry Framework was used to analyze student’s online interactions in a graduate course at the University of Calgary [16].A series of numerous arenas and disciplines use the Community of Inquiry framework [17]. The framework maintains a wide acceptance [18], and more than hundreds of researchers have already used the Community of Inquiry framework since its discovery [19].

1.3 Scope and objective of the thesis

The quick expansion of online learning experiences and educational tools in recent past time [20] has given both challenges and opportunities to online education designers, developers, and teachers using online platforms. The universe of online education is a relatively new field, where group motivation, student-centered interaction, and student-teacher communication might operate differently from traditional face-to-face learning [21]. The number of applicants participating in online learning becomes gradually growing, so teachers need to observe the patterns of involvement and student interaction continuously. Teachers’ feedback followed by students guarantees that all participants, irrespective of background, previous education, and culture, can take advantage of the conversation, communication, and teacher feedback [22]. Effective online schooling requires a proper understanding of how learners from diversified educational backgrounds answer tutor questions, communicate with each other, and understand individual and group forms of interaction. This thesis is mainly concerned with the student’s online interaction. It primarily investigates student’s qualitative cognitive interactions quantitatively using ENA and SNA. By analyzing these interactions, it is possible to identify different aspects of their learning process, which may help an instructor design a course more efficiently. The data used in this research was mainly collected while solving a group project in the Learning Analytics course. Learning Analytics course typically offered once a year by the Faculty of Science and Forestry, University of Eastern Finland.

During this course, every student participates in solving a group project. During

(11)

5

solving this group project, a student can establish several connections to solve the project. Also, a student can communicate regarding different aspects of the project.

To obtain a meaningful insight from this communication is the main objective of this thesis. Visually and statistically, we show the comparison among different group’s interactions and their cognitive networks. The following research questions have been answered extensively by this research work.

RQ.1 How can social network analysis demonstrate student’s connection patterns and performance visually and statistically during the course and group project.

RQ.2 How content analysis can classify the student’s areas of concentration in a group project.

RQ.3 How can ENA demonstrate student’s cognitive connections between codes in different groups by visualization and quantitative analysis.

1.4 Planning of the thesis

The planning of this thesis includes several stages.

• Formulating research proposal and objectives(2-4 weeks): Background study and topic selection, Literature review, Identify and refine research question.

• Data collection (6-8 weeks): Define data collection plan and protocol, collect raw data, and data preparation.

• Data coding (4-6 weeks): Develop a coding framework and data coding.

• Analysis method (2-4 weeks): Explain the data analysis method, state the data analysis tools and software and analyze it.

• Results and discussion (3-5 weeks): Discussion of findings from the analysis.

(12)

6

• Report writing (4-6 weeks): Writing the thesis paper, review by the supervisor, and editing.

• Thesis Submission (1-2 weeks): Submission of the final report for approval.

This thesis typically consists of five sections: interactions in Project-based learning, The Community of inquiry model, Social network analysis, Epistemic network analysis, and Content Analysis. We have collected and studied materials from each section separately and try to integrate them into our research.

1.5 Organization of the thesis

Chapter One introduces the present thesis work area and states the scope and objective. A discussion related to the current research and review from various literature has also been presented. This chapter also focuses on the research questions and organization of this thesis.

Chapter Two is mainly dedicated to a review of project-Based Learning, the benefits and goals of Project-based learning, and the implementation of Project-based learning in the classroom. Also, the success factors of Project-Based Learning are presented.

Chapter Three comprises different types of network analysis with a review of the literature. The core concept of Learning Analytics, Social network analysis and Epistemic network analysis are described in this chapter.

Chapter Four illustrates the data collection procedure, data preparation technique.

The development of the coding framework and data coding method is also shown in chapter four.

Chapter Five describes the Inter-rater reliability using the Simple Classification method and Kohen’s kappa method.

(13)

7

Chapter Six shows the data analysis method. Social network analysis, Epistemic network analyses, and content analysis methods are described in this chapter.

Chapter Seven shows the results of the data analysis and discussion. Social network analysis, Epistemic network analyses, and content analysis are used for data analysis.

Research questions are discussed with the help of data analysis results.

Chapter Eight includes the thesis’s conclusion with present limitations and recommendations for future work.

(14)

8

Chapter 2: Project-Based Learning

This chapter describes a brief review of Project-based learning, the benefits and goals of PjBL, how Project-based learning can apply to the classroom, and the success factors of PjBL.

2.1 Introduction and literature review

Project-based learning (PjBL) is a dynamic student-oriented method of education where students learn by solving problems. The following three productive methods construct this learning environment: education is domain-oriented, students are dynamically associated with the learning cycle, and learners attain their objective using interactions and sharing their know-how and understanding [23]. Student’s self-capability, productive analysis, objective fixation, association, interaction, and replication with practical problems and practices are the key element of project-based learning.

The valuable student’s interaction is the prime concern in all learning environments.

Proper involvement between the participants is mandatory to enabling useful online conversation in project-based learning [24], for example, the instructor’s supervision and response and standard conversation pattern. Project-based learning is a typical prototype of social learning [25]. In project-based learning, participants with various educational backgrounds and expertise work in small groups to achieve a mutual output. Liu and Tsai [26] established group communication and significant features that facilitate online group events.

Furthermore, Project-based learning is applied in collaborative frameworks or at an individual level. According to the idea of Dewey’s [27], social aspects of learning allow students to gain benefits of project-based learning in collaborative frameworks.

Simultaneously, they generate project output and co-construct their knowledge through group interaction and support in collaborative Project-based learning. There are a few challenges, such as syllabus design, project planning, schedule

(15)

9

management, and group cooperation, for converting project plans into general educational frameworks [28]. Generating actual output by resolving real problems in a group becomes crucial for students and creates a real skill, particularly in university education. However, Chang [29] shows conflicting outcomes concerning the role plays by online group conversations on project work in higher education. Earlier research shows that online interaction and educational results may have an optimistic relation.

Project-based learning is a fruitful method for giving several options to participants with various educational backgrounds and levels. Projects can associate participants in better ways rather than traditional learnings, which help achieve efficient education. Projects need to be associated with the syllabus that permits participants to invest their skills and experience in such a way to robust and fruitful education and functional interactions. Project-based learning is a specific practical investigation- oriented education where education content comes from reliable studies and real problems [30], which guide meaningful educational experiences [31]. The cycle of project-based learning includes the steps that students require to produce knowledge by resolving real problems - by inquiring and purifying questions, planning and directing analysis, collecting, investigating, and rendering materials and data, finalizing project tasks, and present the results. The emphasis is on students to accomplish a common objective by teamwork. Participants may face problems in their project that are essential to focus on, creating and illustrating the final output connected to the primary inquiry. Project-based learning requires a dynamic replication and mindful involvement than traditional learning. Project-based learning can prepare students to solve real-world problems, whereas traditional learning is typically limited to academic content.

2.2 Benefits and objectives of Project-based learning (PjBL)

Project-based learning has various advantages to our existing learning frameworks.

Participants in project-based learning obtain different skillsets and expertise by participating in several programs and various contexts. Besides, they build characters

(16)

10

or likelihood to behave in a specific way also generate emotions like self-esteem and assurance [32]. Several pieces of research investigate the consequence of project- based learning on learning outcomes. In research, Noe and Neo [33] showed that participants’ curiosity, complex thinking capabilities, documentation and presentation ability, interaction skills, and the possibility of working efficiently in a group were improved when working on project-based learning events.

Furthermore, Grant [34] conveyed research output, journal papers, artifacts, and exhibits generated in a PjBL event, showing that participants can transfer from beginners to specialists in intelligence and merged their knowledge capabilities in developing the artifacts. Students participating in project-based learning have no restriction to broadcasting evidence and tried to explore the evidence, remained inspired around cooperative work in documentation, and stated optimistic arrogance to teaching antiquity.

Additionally, instructors correspondingly narrate apparent constructive consequences of project-based learning on the education procedure. Examining tutors’ observations of PjBL described that developing participants’ inventive intellectual expertise by way of the advantage achieved, trailed through education and understanding about subject context. Likewise, measuring educators about the practice of project-based learning, Ravitz [35] specified the educators’ powerful motives about using PjBL to learn outside the domain, which makes education modified and diverse and training theoretical context further efficiently. Subsequently, the recognized proof on project- based learning efficiency and tutors’ observations specifies that project-based learning is a framework of teaching and gaining skills about the specific education domain; nevertheless, its goal is to develop the students entirely.

2.3 Implementation of Project-based learning in the classroom

Implementing Project-based learning in educational classes requires additional observations and efforts from the teachers. Mergendoller and Thomas [36]

recommended some significant features in their research; the expert teacher in the

(17)

11

domain of PjBL identifies these features; they found those factors successful in implementing PjBL in the classroom. Those features are as follows,

a) Forming groups and managing them: This segment is for creating groups, founding suitable groups, ensuring all members’ participation and monitoring group performance, and observing and recording the document of group advancement.

b) Setting a good starting point: Introducing students to the problem, enabling them to reflect around the project earlier than the start, clarify exact project output, combinedly accepting the evaluation criteria and features before the project starts. This segment also inspires meaningful work early on the project in designing a research plan and an appropriate research question whereas enabling a sagacity of work.

c) Creating an environment that pressurizes participant’s motivation:

Under this section, responsibility and accountability are transferred from the instructor’s side to the participant’s side during the project’s design phase. Here students decide on self-motivation and learn new skills, theories on their own.

d) Co-ordination and time management: This segment is typically responsible for synchronizing the project timetables with other instructors.

Coordination of timing is necessary among the students and teachers working on a project. Using block scheduling may increase timeliness’ flexibility during project planning and learning when to strict and when to relax a timeline.

e) Maintain external connection: Co-operating and working with experts and colleagues apart from the classroom is vital to solving a critical problem. It may include other educators, parents, and individuals from society who can explain a complex problem and solve it.

f) Uses of technology: Getting maximum benefit from the latest technologies. Use various types of technological resources after evaluating the

(18)

12

appropriateness of applying them for the project. Professionally uses the Internet in discovering appropriate websites and building complex problem-solving skills.

g) Evaluating projects and assessment of students: This concluding segment states the necessity of scoring participants. Scoring can be done by applying several evaluation processes, including personal and team scores and highlighting persons over team presentation. Signifying replication approaches and gathering influential assessment data from the project’s participant improvement strategy might be determined.

From beginning the evidence that Project-based learning adopts through a substantial transformation in learning practices, Krajcik [37] demonstrates the learning process of teachers to focus challenges illustrates by the active interaction of three fundamentals of teaching:

• Instructors associate with counselors and university peoples to interact about thoughts, strategies, and learning events.

• Instructors’ design the plan and produce understandings regarding the possibilities and transform ideology with suitable instruction approaches.

• Educators produce on education via articles, research papers, case study reports, and journals to progress the know-how to facilitate education and learning.

2.4 Success keys of Project-based learning

Academics have acknowledged numerous essential elements for the fruitful application of project-based learning. Whereas PjBL is criticized earlier for not functioning appropriately, the below characteristics may significantly increase a worthwhile project’s possibilities.

(19)

13

a) A simple, practical project that brings into line with participant’s knowledge, expertise, experience, and interests, and involves education undoubtedly well-defined context and skillsets.

b) Organized teamwork with groups of not more than four members, with diversified levels of background, knowledge, experience, and skills and self- dependent characters; group recognition; and personal transparency, created on participant development.

c) Several-group valuation method, through numerous prospects for participants, to obtain comment and criticism and revision of tasks (e.g., standards, thoughtful events); Various educational results (e.g., problem-resolving, context, cooperation); and demonstrations that inspire contribution and communal values (e.g., displays, presentations).

d) Involvement in a specialized education platform and networks, with cooperation and reflection in project-based learning knowledge in school with classmates and other students and subjects in the investigation-oriented learning process.

Project-based learning may successfully apply in educational classrooms by following the above key factors.

(20)

14

Chapter 3: Network Analysis

This chapter explains Learning Analytics, Social network analysis, Epistemic network analysis in detail, including a literature review. Students can gain crucial knowledge and a valuable learning experience through interaction in online learning atmospheres [38].

3.1 Learning analytics (LA)

Learning Analytics represents a method of collecting and analyzing learners’ data to facilitate learning and development of a learning framework [39]. Learning Analytics uses big-data analysis to determine the learning outcomes from an educational perspective. Institutions operating in a distance learning context represent an appropriate model for Learning Analytics use. They already have a massive quantity of data because their students use the internet and online communication as learning and communication medium. Using this data, researchers can analyze the learners’

activity and method of learning to identify a proper learning approach. Drachsler &

Greller [40] have shown a structure of learning analytics in their research. They have distributed the LA structure into the following five significant steps.

• Capturing the data from learners.

• Processing and preparation of data for analysis.

• Analysis and interpretation of data.

• Actions are taken according to the requirements.

• Feedback on the results is collected.

3.1.1 Learning analytics model

Learning Analytics uses a framework of three-phase cycles. Learning Analytics combinedly uses four categories of resources to complete these cycles, as illustrated in the below figure.

(21)

15

In this working method, quality information and accurate data, recent learning theory, and human knowledge may work combinedly to gain students, teachers, and officials’ requirements. Norris et al. [41] imagined analytics could evaluate learner’s proficiency better through personal design plan, recommendation, and

Figure 3.1: LA continuous improvement cycle

foremost execution from everywhere and involvement of students requirements and entirely increase student’s success.

3.1.2 Benefits of learning analytics

Learning Analytics has numerous benefits for different stakeholders. Wong [42]

illustrates the benefits of Learning Analytics for institutes, employees, and learners in his research. Summary of the benefits of Learning Analytics rotates around the below features.

a) Refine learner’s preservation: Through carefully observing learners’

education and perseverance, objectionable learning manners and emotive statuses

(22)

16

can initially recognize learners at risk. Features important to learner’s dropout or retention can readily be acknowledged, and forecast prototype established. The employee can continuously take rapid measures and deliver appropriate support to learners who require essential assistance, like; recommending education materials, consultation, and design personal education plans. Learners’ accomplishment and their preservation level can be improved.

b) Assist in up-to-date decision-making: Learning Analytics can support institutes in the current decision-making process. Data and analysis results are produced from a vast quantity of information is provided to the organizations to help in the latest decision-making process. Like, course improvement plan can be accepted and continue, and distribution of materials based on evidence about the necessity of courses, and categories and regularity of students’ resources based on the study.

c) Reduce cost: Applying Learning Analytics can accelerate the organization’s cost-effectiveness. LA and other student learning management platforms can be merged and work combinedly. Teachers have access to all types of information about students to support them and answer their questions. Students’

evaluation reports, study progress, teacher’s comments can send to administrative officials, guardians, and students using an automated, cost-effective method.

d) Knowing student’s learning behaviors. By investigating various information sources (e.g., LMS, online networks, chatting, and social sites), institutes and teachers may recognize the affairs between learners’ use of materials, study patterns, and students’ results, through which organization can assess and get the feedback about the efficiency of applied pedagogies and design plan for enhancement. Through watching classroom videos, Learning Analytics helps to understand student’s behavior patterns. Syllabus and education resources can design better ways to focus learners’ needs and preferences.

e) Design individual assistance plan for learners: LA can supply learners, sensitive facts and figures about their study behavior and learning structure. Their

(23)

17

learning plan becomes more person-specific, relevant, attractive, and engaging, which simplifies their reproductions and developments during a study module is remains in growth. If a student’s academic results go below up to a certain level, initial warnings become automatedly created and referred to students.

f) Real-time response and intervention: The teacher can observe the time- to-time status during a course. So that he can provide real-time comments to students and personalized instruction made. Students build an academic position in the learner society by the individualized response provided to them. To better understand learner society improvements, teachers can use social network analytics, classify students with lower academic progress or isolated from the class and conversation, and then provide real-time intervention through conversation. Real-time response is particularly significant for organizations, where participants use diverse learning approaches, and social media typically plays the role of a connection medium.

3.2 Social network analysis

Social network analysis (SNA) is a quantitative analysis tool that can visualize and relate interactions between participants within a group in a learning environment [43]. Two primary methods are used in SNA analysis: network visualization and statistical analysis [44]. Visualization gives relationships among participants of social networks by graphs referred to as sociograms; the sociogram represents participants (nodes in SNA terms) in points and relationships as arrows from the origins of the interaction and points to the objective of interaction [45]. Statistical analysis is a mathematical approach to quantify the connectedness and relations of participants in a network [44].

SNA mainly computerized the procedure of abstraction, gathering, assessment, and illustration of learner’s interaction data, and rapidly demonstrating to teachers [46].

In addition to these, it is possible to provide real-time responses to students and teachers. The software produces the visual and statistical results of SNA analysis.

SNA may apply to any interaction analysis for determining the participant’s

(24)

18

involvement in a course or project [47]. Social network analysis is a technique for examining features with computable statistics, which permits investigators to discover the connection between students on a broader perception instead of concentrating on personal qualities [48].

Irrespective of the learning platform, Social network analysis delivers a set of methods and terminology to identify the better meaning of personal, relational interactions in groups [45]. Social network analysis methods gradually illustrate online interaction, enhancing traditional face-to-face connections and meaningful learning [49]. In an undergraduate instructor training event in online peer mentoring sites, social network analysis analyzes the interaction’s structure [50].

3.3 Epistemic network analysis (ENA)

Epistemic Network Analysis (ENA) identifies and measures connections between coded interactions and visualizes them in a network model. This model visualizes the networks’ structure and determines the depth of the connections between codes [10].

ENA can help to compare and analyze the network pattern of different groups by visually and statistically. ENA is considered as a network and content analysis method used by a developing society of Learning Analytics scholars. ENA uses the Singular value decomposition method to visualize the networks and represent the analysis features [51]. Epistemic network analysis is a computational instrument for qualitative data analysis that helps researchers in their research. A vast range of features can model using ENA, designing urban planning with various limitations [52], supporting students in critical thinking, and group problem solving [53]. ENA has some specialized uses like methods where students combine to obtain essential data from the internet [54].

3.3.1 Background of ENA and the epistemic frame theory

ENA is primarily developed based on an epistemic frame theory. While working in a big-data discourse, it fully needs a complete understanding of the community’s

(25)

19

codes; only classify the codes is not sufficient. In epistemic frame mapping, codes are connected in the discourse, and ENA offers the method to modeling those epistemic frames. Activity that happens within a current temporal framework: the earlier activities give a mutual foundation in which actions are clarified, such as individuals behave in reply to the effect that occurred only a few, whereas a previous interaction or stream of the event are defined [55]. Identification of analytical connections between objects which can build up an epistemic frame, the co- occurrences of relationship and object is recognized in an interaction pattern, in a discussion forum mainly in an online chatting event. Epistemic network analysis designs the sequential correlation pattern of codes, the number of interactions among codes, for the interested community in the given dataset. This network method has statistical and theoretical benefits over additional distinctive multivariate designs, wherever interactions among participants are naturally secondary, sometimes even between the underlying participants [56]. Social network analysis and Epistemic network analysis combinedly produce an entire student’s conversation pattern during their project on a broader viewpoint.

ENA primarily develops on four established thoughts regarding human events existing in the real world [57]. Participants are first embedded with cultures, second surrounded in the discourse, thirdly with interaction, and finally in time. All four philosophies integrate into “epistemic frame” theories. The relation between codes is formally explained in an epistemic frame, which is revealed by investigating how codes from a big-data discourse are connected in individual or group discourse. The epistemic frame’s philosophy [57] explores the thoughts that, for better understanding of big-data discourse, need some sense about a specific community’s code, only to classify the community’s codes is not sufficient. ENA delivers a method of building epistemic frames by drawing networks among codes and a model of the epistemic frame of a student or students—and, therefore, a way to explain how education is concurrently embedded in culture, discourse, interaction, and time.

The epistemic frame theory proposes that every community has a culture composed of Skills, Knowledge, Identity, Values, and Epistemology, and integration of these

(26)

20

features form the epistemic frame of that community [58]. Epistemic frame theory states the following three guidelines,

a) An epistemic frame integrates skills, knowledge, values, identity, and epistemology that define a specific community member.

b) By adopting a frame’s features, a person can be a specific community member.

c) Once adopted if a person demonstrates a condition as a member of that community, that community’s epistemic frame is used.

3.3.2 Application of epistemic network analysis

ENA is applicable for various data types, like the interview and survey scripts, specific group data analysis, interactions, and observational data. During the analysis of descriptive information like interactions, surveys, and interviews, ENA assists in recognizing the structure and association, which helps further analyze the primary data. ENA is used in numerous phenomena, such as cognitive networks learners make during critical problem solutions, the connection between neurons in MRI data, cooperation of operative knowledge during a surgical operation, and some more.

Wooldridge [51] summarizes various applications of ENA in his research. ENA was founded based on an educational framework to classify the connections among professional skills, such as engineering design of urban and regional planning, design surgeons’ connection network in an operating room, social gaze coordination, operating error handling, and historical semantics documents. Some application explains that ENA can highlight network pattern except investigating descriptive content so that ENA can be a statistical tool. For descriptive content analysis, ENA can also be used. For example, Analyzing surgical interns’ interaction during the modeling of a laparoscopic surgery permits them to model and evaluate how surgical residents integrate elements essential for successful operative performance.

(27)

21

Chapter 4: Data Collection & Coding

In this chapter, we illustrate data collection and preparation methodology. Data collection needs a well-organized design method, massive effort, endurance, persistence, and other features to complete successful research [59]. The development of the coding framework is also explained here.

4.1 Data collection

We collect this research data from the Learning Analytics course offered by the School of Computing under the Faculty of Science and Forestry, the University of Eastern Finland, for two consecutive terms. During Autumn 2019, 45 students and Autumn 2020, 51 students have participated in the course. Each term, all the students are divided into several groups. LMS application Moodle automatically creates all the groups. Each group is allocated a group project during the course, and it contains 20% of the course weight. Students use the online platform Moodle for all types of communications during the course activities and project.

Moodle is a learning management system (LMS) used by the University of Eastern Finland. While working on the group project, each group created a separate channel for their interactions. Approximately 1700 interactions were collected for two years period.

4.2 Data preparation

Initially, raw data is collected in Microsoft Excel format from the learning management system of the University. Collected data was unorganized and included the following features; Source, Forum, Course, GroupID, FirstPost, Discussion, Parent, UserID, Created, Modified, Subject, Message, Attachment, TotalScore, and Target. Data is prepared based on the requirements of analysis. Keeping these features; Source, Target, UserID, Forum, GroupID, Parent, Created, and Message for analysis, all remaining features were discarded from the data. We have shorted

(28)

22

the interaction data based on GroupID so that all conversations within a group remain in the same place and we can separate easily. Later we have divided the interaction data by group for a detailed analysis.

4.3 Development of the coding framework

The development of coding was one of the significant tasks in this thesis. Wide- ranging literature was reviewed for developing the coding framework. Yiping Lou &

S. Kim MacGregor [1] has presented a coding structure for interaction analysis. H.

Heo, K. Y. Lim & Youngsoo Kim [60] have used a coding template for interaction analysis based on Gunawardena et al. (1997) coding scheme. Diana, A de Leng &

Wolfhagen [61] showed a coding framework for analyzing verbal interaction.

Garrison, Anderson, and Archer [11] illustrated a coding template known as the Community of Inquiry (CoI) coding template. This coding template was extensively studied and used in this research. We have added a few codes to the CoI template depending on the requirement of the data. Typically CoI has three domains, such as cognitive presence, social presence, and teaching presence. We integrated an additional domain, which is common presence; it includes some general categories of codes. The following table illustrates our developed coding framework.

Table 4.1: Coding Template for Project-Based Learning

Group Code Explanation

Common Presence

Questioning Ask any type of question. It may like asking for new information, explanation, verification, or it may become a counter-question.

Disagreement Any member does not agree with the idea, concept, plan, or opinion of another member.

Off-task The discussion is not related to the subject.

Management Discussion about managing a project.

(29)

23

Group Code Explanation

Cognitive Presence

Triggering Event

Identify the particular problem for solving any task.

Exploration Knowledge sharing and discussing ambiguities.

Integration Connect various ideas to create a meaningful solution.

Resolution Create or apply a new idea or theories to solve a problem.

Social Presence

Affective Expression

Anything related to Emotional attachment, any autobiographical descriptions.

Open

Communication

Expressing anything without risk encourages the work of others and acknowledges other’s support or tasks.

Group Cohesion

Inspire the relationship between team members by assisting, helping, and supporting the members.

Teaching Presence

Instructional Management

Course content creation and organization, Discussion topic settlement, Creating groups for project or discussion.

Building understanding

Express personal opinion and understanding, Agreeing and accepting other member’s ideas.

Direct Instruction

Instruction may be in answering questions, focusing on specific topics, or resolving a problem.

4.4 Data coding

For data coding, we use two different methodologies, like coding using the developed coding framework and coding using the topic of interaction.

(30)

24

Firstly we code approximately 1700 data based on our developed coding framework.

Each row of data was coded separately against each developed code. Some data can be represented by a single code, whereas some data needs multiple codes. We have coded all the interactions using the binary value “0” and “1”. When a single code represents a message, “1” is entered under this code, and enter “0” to all the remaining codes. If multiple codes represent a message, we have to enter a value “1”

to each code and enter “0” to all other codes.

Secondly, we code the whole 1700 data again using the topic of the interaction. That means each interaction took place under a specific subject. Each row of messages codes against all possible topics, and every message belongs to one specific topic or subject only. Again “0” and “1” represent which message belongs to which topic.

There is no possibility that multiple topics represent one message. If any message represents by a topic “1” is entered under this topic, enter “0” to all other topics.

Table 4.2 and 4.3 show a pattern of coded student interactions using the developed coding framework and using the interaction topic, respectively.

Table 4.2: Coding Student’s conversation using developed code

UserID \ Student

Messages / Interactions /

Data

Codes

Questioning Disagreement Management Triggering Event Exploration Integration Open Comm. Group Cohesion Instructional Mgt. Building Und. Direct Instruction

42

Friday evening works for me as well. Let’s see what other members say. Friday evening say 5 PM works for everyone?

1 0 0 0 0 0 1 0 0 1 0

16 Now absolutely nobody has excuses. <a0>

0 0 0 0 0 0 1 0 0 0 0

21 Friday at 5 PM is okay for me. 0 0 0 0 0 0 0 0 0 1 0 21 So, if we talk tomorrow, which

medium are we going to use?

Skype? 1 0 0 0 0 0 0 0 0 0 0

(31)

25

Table 4.3: Coding Student’s conversation using topic

UserID \ Student Messages /

Interactions / Data

Topics

Project Discuss. Theory Policy Discussion Ethics Feedback Conclusion Data Analysis Group Discuss. Ques. And Ans.

69 These Gephi files are here for the Presentation

1 0 0 0 0 0 0 0 0

16 Now absolutely nobody has excuses. <a0>

0 0 0 0 0 0 0 1 0

45 Thanks XXXXXX. 0 0 0 0 1 0 0 0 0

21 So, if we talk tomorrow, which medium are we going to use?

Skype? 0 0 0 0 0 0 0 0 1

(32)

26

Chapter 5: Inter-rater Reliability

This chapter explains the inter-rater agreement in the coded data. Two standard methods, The Simple classification method and Cohen’s Kappa method evaluate inter-rater reliability agreement.

5.1 Simple classification method

The inter-rater agreement evaluation delivers quantifying the degree of reliability among multiple coders; they code independently about several specific subject features [62]. Inter-rater reliability analysis determines how much of the coded data have agreement or similarity between coders after removing error or disagreement in codes. For example, a simple classification reliability value of 0.75 indicates that 75% of the coded data have similarities between coders, and 25% is the differences in coding or error among coders. Simple classification reliability value can determine using the below formula;

Simple Classification rate:

The following table shows a pattern of agreement or disagreement,

Table 5.1: Agreement-disagreement pattern for Inter-rater reliability

Message / Data

Code by Rater 1 Code by Rater 2

Agreement / Disagreement

Questioning Open Communicati on Building Understandin g Direct Instruction Questioning Open Communicati on Building Understandin g Direct Instruction

Thanks, XXXXXX.

This was much needed.

0 1 0 0 0 0 1 0 0

Hi 3 of us are in study room.

0 1 1 0 0 1 1 0 1

Where is the location of study room?

1 0 0 0 1 0 0 0 1

Next to lecture room TB 180

0 1 0 1 0 1 0 1 1

Numer of coded data that have an agreement Total number of coded data

(33)

27

When all the codes representing a message or data are similar between the coders, it is known as an agreement. If there is a dissimilarity in the code among coders, it is known as disagreement. Binary value ‘1’ represents an agreement, whereas ‘0’

represents a disagreement. Out of 1700 data, we have taken 200 data as a sample to determine the inter-rater reliability. These 200 data coded by two coders, and in 169 coded data, we found an agreement between the coders.

Simple Classification Reliability rate: = 0.845

In our coding, we found the Simple Classification reliability rate is 84.5%. This percentage is a standard rate of reliability. This reliability indicates we can proceed further with our coding.

5.2 Cohen’s Kappa method

The Kappa is a method of correlation constant. It is one of the most frequently used statistics to assess inter-rater reliability, but it has limitations. Same as correlation coefficients, Kappa can range between −1 to +1, where 0 denotes the rate of agreement that can predict from random chance, and 1 denotes the highest agreement among the coders. Kappa values below 0 are unlikely in practice. Cohen’s suggested static Kappa value is interpreted as below [63].

Table 5.2: Static Kappa value Interpretation

Kappa Value Interpretation of Agreement Percentage of Reliable Data (%)

Below 0 Not in practice Not Applicable

0 – 0.20 No Agreement 0 – 4%

0.21 – 0.39 Minimum Agreement 4 – 15%

169 200

(34)

28

Kappa Value Interpretation of Agreement Percentage of Reliable Data (%)

0.40 – 0.59 Weak Agreement 15 – 35%

0.60 – 0.79 Moderate Agreement 35 – 63%

0.80 – 0.90 Strong Agreement 64 – 81%

Upper 0.90 Almost Perfect Agreement 82 – 100%

Using IBM SPSS version 25 and previously coded 200 data, we determined the Kappa value for each code. We can see the associated Kappa value and interpretation against each code from the following table.

Table 5.3: Kappa value Interpretation with Code and Data Reliability

Code Kappa Value Interpretation of

Agreement

Percentage of Data Reliability

Questioning .976 Almost Perfect 82 – 100%

Disagreement .886 Strong 64 – 81%

Off-task .000 None 0 – 4%

Management .664 Moderate 35 – 63%

Triggering Event .565 Weak 15 – 35%

Exploration .565 Weak 15 – 35%

Integration .745 Moderate 35 – 63%

(35)

29

Code Kappa Value Interpretation of

Agreement

Percentage of Data Reliability

Resolution 1.00 Almost Perfect 82 – 100%

Affective Expression .496 Weak 15 – 35%

Open Communication .844 Strong 64 – 81%

Group Cohesion .745 Moderate 35 – 63%

Instructional Management

.852 Strong 64 – 81%

Building understanding .841 Strong 64 – 81%

Direct Instruction .874 Strong 64 – 81%

From the above table, we can see that out of 14 code; five code has a strong agreement equivalent kappa value ranging between .80 to .90. At the same time, only two code has almost perfect agreement equivalent kappa value .976 and 1.00. For moderate agreement and weak agreement equivalent kappa value, three code belongs to each group. Only one code has 0 kappa value and falls into the None agreement group.

The inter-rated agreement is essential in coding interaction data, as it explains the reliability and goal of coding by several raters and recognizes the value of the coding framework [64].

(36)

30

Chapter 6: Analysis Method

This chapter describes the data analysis methods in detail. The analysis is done using different software and the developed coding framework. We used Gephi 0.9.2 version for Social network analysis and RStudio (readxl, rENA and ggplot2 packages) for Epistemic network analysis. For content analysis, we have used our developed coding framework and the subjects of the interaction.

6.1 SNA analysis method

Social Network Analysis (SNA) is a series of approaches and tools for analyzing relations, communications, and interactions. SNA can research and track online interactions as interaction data can automatically be analyzed. Social network analysis primarily uses degree, weighted degree, eccentricity, closeness-centrality, betweenness-centrality, clustering, and eigen-centrality.

The SNA software generates these feature’s value and used in research result analysis. Patricia & Doran (2011) explain social network analysis features in their research [47], which analyzes the students’ participation in a course or project. The following features are essential for Social network analysis.

Degree: The degree represents a vital score based on the number of direct connections by each student, and it is the most straightforward measure of student connectivity. Degree value includes both in-degree (quantity of incoming connection) and out-degree (quantity of outgoing connection) as specific measures, such as transactional data or account activity. We have used a directed network connection in our visualization.

Closeness Centrality: Closeness centrality counts each student depending on their closeness to other students and teachers’ connectivity. It computes the minimal paths among all students and allocates each student’s value based on the minimal paths. It helps to identify a student who can quickly inspire the whole network.

(37)

31

Betweenness Centrality: Betweenness centrality counts how many times a student lies on the optimal path among other students or the teachers. It indicates which students are ‘ties’ between other students in a network. It typically identifies the students who influence the movement and activity of a group. Maximum betweenness may specify a student or students who play as a motivator, influencer, or maybe a leader.

Eigen Centrality: Like degree, Eigen Centrality counts a student’s influence quantity of connection it has to other students in the network. It also considers how well a student is connected and the number of links the connections have. It measures a student’s extended network and identifies students with impact on all networks, not directly connected networks. It is an excellent overall SNA value to understand the human social network.

6.2 Content analysis method

One effective method to analyze student conversation in an online platform is content analysis [7]. The descriptive analysis of content is limited to classifying the content of the interaction, showing the meaning of a small part of the interaction.

Content analysis is done based on the coding framework and subject of the interaction. Our developed coding framework has four domains: common presence, cognitive presence, social presence, and teaching presence.

As interactions are coded using the developed coding framework and subject of the interaction, the frequency of each coded interaction counts and gives an overall integrated description. After counting the frequency of each code, codes are placed according to their groups. The coded content is gathered and summarized to produce the result of content analysis.

Again frequency of each interaction counts according to their subjects. All the subjects represent the total number of interactions that belong to them. Then the summarized subjects are used to generate the results of content analysis.

(38)

32

The following table illustrates the summarized coded content using the coding framework from Autumn 2019 data,

Table 6.1: Summary of coded content using the coding framework in Autumn 2019

Group Code Code Count No. Of Group Code

Common Presence

Questioning 166

262

Disagreement 11

Off-task 5

Management 80

Cognitive Presence

Triggering Event 43

195

Exploration 49

Integration 73

Resolution 30

Social Presence

Affective Expression 47 Open Communication 170 309

Group Cohesion 92

Teaching Presence

Instructional Management 18 Building understanding 185 264 Direct Instruction 61

The following table illustrates the summarized coded content using the coding framework from Autumn 2020 data,

(39)

33

Table 6.2: Summary of coded content using the coding framework in Autumn 2020

Group Code Code Count No. Of Group Code

Common Presence

Questioning 159

222

Disagreement 6

Off-task 2

Management 55

Triggering Event 32

Cognitive Presence

Exploration 67

149

Integration 39

Resolution 11

Social Presence

Affective Expression 2 Open Communication 205 265

Group Cohesion 58

Teaching Presence

Instructional Management 12 Building understanding 225 262 Direct Instruction 25

We have also coded the data using the subject of the interaction. That means each interaction took place under a specific subject. Each row of messages codes against all possible topics, and every message belongs to one specific topic or subject only.

The following tables demonstrate the subject-based coded summary from Autumn 2019 and Autumn 2020 data separately.

(40)

34

Table 6.3: Summary of subject-based coding in Autumn 2019

Topic / Subject No of interactions

Project Discussion 243

General Group Discussion 203

Feedback 77

Conclusion 69

Policy Discussion 67

Questions and Answers 66

Ethics 54

Theory 44

Data Analysis 22

Table 6.4: Summary of subject-based coding in Autumn 2020

Topic / Subject No of interactions

General Group Discussion 292

Project Discussion 86

Feedback 83

Theory 64

Policy Discussion 58

Questions and Answers 53

Relation of LA with other fields 36

Events and Timeline 24

Analysis of student interaction in Project-based learning