A. Rahnev1 , N. Pavlov2, A. Golev3, M. Stieger4, T. Gardjeva5

1,2,3,5 Faculty of Mathematics and Informatics University of Plovdiv
236, Bulgaria Blvd., 4003 Plovdiv, BULGARIA

4 Center for European University Studies AUSTRIA

Abstract: This paper describes some of the new electronic educational services, provided by the Distributed e-Learning Platform – DisPeL. Learning is empowered though the use of adaptive learning content. Testing examination is facilitated through algorithms to generate unique tests with minimum pre-defined testing questions. Testing examination is supported in multiple languages. These services were successfully used in the Magister’s Program “Business Software Technologies” at the Faculty of Mathematics and Informatics at the Plovdiv University “Paisii Hilendarski”.

AMS Subject Classification: 97U50, 97Q70

1. Introduction

The Distributed e-Learning Platform – DisPeL [5], is an integrated software system for automating the management and execution of the learning process.

In this paper we describe how DisPeL is improving the learning process of Magister’s Program “Business Software Technologies”. We outline the new services of DisPeL which add value to the existing practices in learning and e-learning.

The Magisters Program “Business software technologies” is developed by the faculty of mathematics and informatics at Plovdiv University “Paisii Hilendarski” [4]. It is targeted chiefly towards students with a degree in Economics, Business Administration, Marketing and Management. Partners on the program are “Center for European University Studies – CEUS”, Austria and “World Wide Education – WWEDU”, Austria. The courses are in English language, which is the official language of the Program. The graduates acquire qualification “Informatician”. The new program started in the 2012/13 academic year. Thirty students from Austria, Germany and Croatia were educated. The lectures and the final exams were carried out in Wien, Austria. Courses were oriented towards information technologies and programing and their application in law, banking, accounting, insurance and economics. The courses were prepared especially for the Magister Program.

DisPeL improves the learning in the aforementioned program by providing the following innovative services:

  • Management of the learning process by automating the administration;
  • Adaptive learning content;
  • Electronic testing and assessment;
  • Electronic services to assist the traditional testing examination and assessment.

    2. Overview of the Educational Process

    The Program includes five primary disciplines, three disciplines and a state exam or thesis defense. The length of the program is three semesters [4].

    The material for the primary disciplines is divided in fifteen chapters each, and the material for the optional ones – in seven chapters. Electronic textbooks are created for each of the disciplines.

    A web based information system for e-learning, part of DisPeL, is used to support the Magisters Program. The web based information system enables exchange of information and distant examination within a virtual educational auditorium. The virtual auditorium is a virtual space, which exists in the context of a certain disci- pline. It allows the tutors to add educational materials, examination tests, assign and assess homework assignments, and conduct online exams for the appropriate discipline. The web auditorium improves the efficiency of the communications between lecturers and students. Every user of the system has a personal account and operates in designates classrooms.

    The system contains electronic textbooks and additional electronic materials for the studied disciplines. Every textbook can be activated for a definite period of time.

For that period the students must pass through all the chapters of the textbook and make a overall test on the learned material.

The web system for e-learning is multilingual. Electronic textbooks and testing questions are written in one the official language of the Program, English. The textbooks, the electronic materials and the questions can be translated to a second auxiliary language. In that case the system can generate tests in two languages. For the convenience of the students both versions are available. The tests are assessed in the official language of the program.

3. Administration of the Learning Process

Efficient administration of the learning process is a key factor for the overall success of the education. It leads to improved times for all administrative services, like processing students data, servicing students requests, and providing information to authorities. DisPeL includes an integrated information system that automates the work and the activities at the Registration Office of a University – UnIS. Fully compliant with the legislation in Bulgaria and the special requirements for Universities of the Bulgarian Ministry of Education, UnIS empowers administrators to easily manage the complete cycle of the learning for every student, from enrollment to graduation. It maintains exhaustive data about student’s academic progress – en- rolment for study schedules and the corresponding courses, marks, academic status, participation in international educational programs, accommodation at students’ hall of residence, getting a scholarship, and health insurance status. It automates many otherwise time-consuming operations, like moving a group of students to the next study schedule, filling in the examination records, group enrolment in elective courses, etc.

4. Adaptive Learning Content

We consider adaptability of learning content as a key factor for improving the learning experience for students. We base our consideration on existing research, which has shown that using adaptation can provide better learning environment in electronic learning systems [3]. Further, we adopt the model of Kirschner, who defines an adaptable e-learning system as an interactive system, which personalizes and adapts learning content [6].

Adaptability of learning content is achieved via two methods: automated counselling and progress control. At the end of every chapter of the learning content, the learners are tested on the covered materials. The system validates the answers to the test and uses the result to define hypotheses about the lapses of knowledge in the presented material. With automated counselling, the system automatically constructs targeted learning content. Targeted learning content is a subset of the learning content, which focuses on the identified lapses of knowledge.

Test results are further used to exercise progress control. Progress control does not let learners skip through the learning material without having adequately assimilated the content so far. Progress control ensures that material is covered sequentially, in the order the tutor has devised, and that on every step the learners are making good progress.

The process is described as:

Test → Assessment → Identify Problems → Consult (Provide targeted learning content)

5. Generating Examination Tests

Good and efficient testing examination requires that there is sufficient number of unique testing questions with equal difficulty level. When a large number of testing questions is required, they will imminently vary in content and difficulty. The results might be hard to rate and compared. As a result, the fairness of the exam is compromised, and there is serious risk of not assessing the students properly.

Cheating by looking at neighbors’ workplace (computer or paper) is another problem with testing examination. Invigilators are required for every exam to secure that there will not be cheating. The problem can also be mitigated by a large number of unique testing questions.

Creating many unique testing questions is a highly time-consuming task. In this paper we describe an algorithm in DisPeL, which allows tutors to:

1. Prepare a minimum number of unique testing questions;

2. Based on the prepared data, generate unique testing questions and examination tests for every learner.

The system contains a database with test questions of different types. For every chapter of the e-textbook at least seven test questions are prepared. The closed test questions can have multiple correct and incorrect answers. There must be more than one correct answer and the recommended number of correct answer is at least two. Incorrect answers must be at least three, but it is recommended they are at least five. The more the correct and incorrect answers, the better the output of the algorithm will be. Questions can be used for generating tests for passing between the chapters of a discipline, final test of a textbook, final test for the end of the semester and test for State Exam.

If we know the emplacement of the students in advance, the number of different tests can be minimized [1]. The goal is to prevent cheating. For conducting exams for this Magisters Program there were no available computer rooms and the emplacement of the students wasn’t known in advance.

5.1. Intermediate Tests

At the end of every chapter of electronic textbooks, students are subjected to a test where must give correct answers to at least four questions from each chapter, before they can continue with the next one. The four questions are generated randomly every time when the student takes the test on current chapter. For every test question, one correct and three incorrect answers are randomly selected.

If the questions are not given correct answers, the systems analyses the errors and recommends the students which material he should concentrate on. When the test is taken again, the students get four new random questions.

With minimum number of seven questions per chapter, with an average of two correct and five incorrect answers, the possible variations of the four randomly generated questions is at least 16800 (700 without considering the order of the answers).

5.2. Final Textbook Tests

The learning of a discipline ends with a final online test on the overall material. The system assigns ten questions to every single student. The test can be solved in a predefined period of time. The questions are generated randomly from the questions of all chapters.

The test is conducted online in the web-based front-end of DisPeL and gives 30% of the final mark for the appropriate discipline. The test can be done on a defined schedule, or at any time to the convenience of the students.

5.3. Semester Exams

At the end of each semester there are final attendance exams. An attendance exam can be conducted electronically in a computer room or the test can be done on paper.

We have developed an algorithm to generate testing questions for the final attendance exam for the disciplines. They have fifteen chapters each. From every chapter of the textbook for the discipline the system chooses randomly two questions and thus we have 30 questions which will be used in the final exam. The test for every single student contains 20 questions. First, from the preselected 30 questions, we take one question from every chapter which makes 15 questions. Then, from the remaining questions from every consistent group of 3 chapters we choose one question which makes 5 more questions.

// from every two questions we get one

for (int i = 1; i <= all_questions; i += 2)


rand_n = rand_num.Next(0, 2);

generated_test[t_pos++].question = i + rand_n;

selected_questions[i + rand_n] = true;

// from every 6 questions we get unused one

for (int i = 1; i <= all_questions; i += 6)


rand_n = rand_num.Next(0, 3);
cnum = 0;
for (q_pos = i; q_pos < i + 6; q_pos++)


if (selected_questions[q_pos] == false)

if (cnum == rand_n)




selected_questions[q_pos] = true;

generated_test[t_pos++].question = q_pos;


The 20 chosen questions are randomized. From the possible answers for each question, one correct and three incorrect answers are selected, whose order also can be randomized within the question:

r = correct_answers[pos];
correct_answer = (r == 1) ? 1 : rand_num.Next(1, r + 1);

pos_cr_ans = rand_num.Next(0, 4);
ans.n_answer[pos_cr_ans] = ’C’ + correct_answer.ToString();

nw = wrong_answers[pos];
HashSet<int> w_ans = new HashSet<int>();

for (int i = 1; i <= nw; i++) w_ans.Add(i);

for (int i=0, k = nw; k > nw – 3; k–, i++)


r = rand_num.Next(0, k);

w_ans_3[i] = w_ans.ElementAt(r); w_ans.Remove(w_ans_3[i]);


nw = 0;
for (int i = 0; i < 4; i++)


if (i == pos_cr_ans) continue;
ans.n_answer[i] = ’a’ + w_ans_3[nw++].ToString();


In this way we have a unique test for every student. The test is different enough despite the fact that only 30 questions are used. This decreases significantly the opportunity for cheating. For example:

Q02, a2, C2, a4, a3 Q21, a1, C4, a2, a5 Q11, C1, a4, a1, a3 Q08, C1, a3, a2, a1 Q25, a5, C2, a2, a4 Q22, a1, a2, a3, C1 Q10, a1, C3, a3, a4

Q05, a3, a4, a2, C1 Q01, C1, a3, a2, a1 Q28, a2, a4, C2, a1 Q29, a2, a3, C2, a5 Q13, a2, C1, a3, a4 Q09, a5, a1, C3, a3 Q20, C4, a4, a3, a5

Q15, C2, a2, a6, a3 Q16, a3, a5, C3, a2 Q24, a2, a1, C2, a5 Q04, C2, a5, a1, a2 Q17, a3, C1, a1, a5 Q30, a4, a3, C1, a1

When choosing specific questions for a given test, additional conditions must be checked. For example: when we have two similar questions or if the formulation of a question suggests the answer of another – only one of these questions must be chosen.

When an examination is carrying out with a test on paper, for each generated test a LATEX document must be created to assist the automated formatting and printing of the test on paper. Each document contains the name of the discipline, and the unique identifier of the generated test. The text for the LATEX document is prepared with the help of the following template:


\renewcommand\labelenumi{{\bfseries\theenumi}.} \renewcommand\labelenumii{{\theenumii})}


\def\myheader{\hfill #COURSE_NAME# – Test #TEST_NUM#, page \thepage} \usepackage{fancyhdr}






{ \large \bf

Final test #TEST_DATE#


Student’s name: \dotfill Test \textnumero #TEST_NUM#






The template contains variables which are enclosed with the # symbol. Variables are replaced with a relevant text. Questions are numerated consequently with numbers, and the possible 4 answers are marked with letters from a) to d). The template for creating a question is:

\item {\bf #QUESTION#} % #QUESTION_NUM#


\item #ANSWER_1#               \item #ANSWER_2#

\item #ANSWER_3#                \item #ANSWER_4#


Before printing, all LATEX documents are compiled to pdf format and are checked for problems with pagination. That takes no more than a minute for every single test and it is done during the printing process.

The system can print a list of correct answers of the generated tests for the chosen discipline, which greatly facilitates the assessment of the test. It is not necessary for the reviewers to know about the specific area of the test.

Test No. 1: 1b, 2c, 3d, 4a, 5a, 6c, 7c, 8d, 9c, 10c, 11b, 12b, 13d, 14d, 15d, … Test No. 2: 1c, 2a, 3c, 4d, 5b, 6c, 7d, 8c, 9b, 10b, 11c, 12c, 13d, 14a, 15b, … Test No. 3: 1b, 2d, 3d, 4b, 5b, 6b, 7b, 8c, 9d, 10c, 11b, 12d, 13d, 14d, 15b, … Test No. 4: 1d, 2b, 3c, 4a, 5c, 6b, 7a, 8d, 9c, 10a, 11a, 12b, 13b, 14a, 15d, … Test No. 5: 1a, 2c, 3d, 4c, 5b, 6c, 7d, 8b, 9c, 10d, 11c, 12b, 13d, 14c, 15b, …

The algorithm for generating tests supports parameterization of the disciplines, the number of the chapters for each discipline, the number of the chosen questions for a concrete discipline and the schema for choosing questions for every single test. The same algorithm is used to generate the final tests for all main disciplines and the test for the State Exam of the Magisters Program.

6. State Exam of the Magisters Program

The State Exam for the Magisters Program is defending a diploma thesis or solving a test with 50 questions on all studied material.

The preparation of the State Exam test is performed in a similar way, described in the previous chapter. Unused questions in the exams so far are also used.

For the graduation of the first class of the Magister Program – October, 2013 there were chosen 7 questions for each one of the 5 main disciplines and 5 questions for each one of the 3 selectable disciplines or a total of 50 questions.

The preliminary selection of 30 questions for the final 50 questions for the State Exam is made to:

• Only selected questions are translated to the secondary learning language;

• Students answer to questions with the same level of difficulty;

• The assessment of the tests is faster.

The State Exam for the Magisters Program was conducted in Wien. Two students defended their diploma theses and 25 students took a test. Three students didn’t come to the State Exam due to failing with the eBook education or not coming to the final attendance exams.

7. Conclusion

DisPeL is an evolution to the Distributed e-Testing Cluster (DeTC) and is a standalone, independent e-learning platform. It provides new services for learners and tutors which empower the learning and testing examination processes. DisPeL can be also used to assist traditional paper-based testing examination. The platform was tested and proved in the Magister’s Program “Business Software Technologies” at the Faculty of Mathematics and Informatics at the Plovdiv University “Paisii Hilendarski”. In 2013, thirty students from Austria, Germany and Croatia were enrolled in the program, and 27 graduated successfully. The collected feedback was generally very positive. The success of the platform was boosted by the multi-lingual support for testing questions, which improved the experience for the learners.

DisPeL is also being targeted towards electronic education for business compa- nies who seek to improve the qualification of their staff, as software as a service – a cloud service.


Parts of this work are supported by NPD of Plovdiv University “Paisii Hilendarski”, Bulgaria, grant NI13 FMI–002.


[1]  A. Golev, O. Rahneva, A. Rahnev, Algorithms to minimize the number of unique tests in real group testing examination, Scientific Works, Plovdiv University, 36, No. 3, Mathematics (2009), 39–49.

[2]  National Institute for Education and Training in Education System,

[3]  R. Oppermann, R. Rashev, A. Kinshuk, Adaptability and adaptivity in learning systems, Proceedings on Knowledge Transfer, Volume II (1997), 173–179.

[4]  A. Rahnev, Hr. Krushkov, N. Pavlov, M. Stieger, N. Valchanov, New master program “Business Software Technologies” in faculty of mathematics and informatics of Plovdiv university, Proceedings of “Traditions, Direction, Challenges” – Jubilee National Scientific Conference with International Participation, Smolyan (2012), 173-176.

[5]  A. Rahnev, N. Pavlov, V. Kyurkchiev, Distributed Platform for e-Learning – DisPeL, European International Journal of Science and Technology, 3, No. 1 (2014), 95-109, ISSN: 2304–9693.

[6]  S. Stoyanov, P. Kirschner, Expert concept mapping method for defining the characteristics of adaptive e-learning: ALFANET project case, Educational Technology Research and Development, 52, No. 2 (2004), 41-56.


Kommentar verfassen

Trage deine Daten unten ein oder klicke ein Icon um dich einzuloggen:

Du kommentierst mit Deinem Abmelden /  Ändern )

Google+ Foto

Du kommentierst mit Deinem Google+-Konto. Abmelden /  Ändern )


Du kommentierst mit Deinem Twitter-Konto. Abmelden /  Ändern )


Du kommentierst mit Deinem Facebook-Konto. Abmelden /  Ändern )


Verbinde mit %s