The problem of the quality of education and its objective assessment. Education quality assessment system. Issues for discussion

The phrase "quality of education" in the early nineties of the last century in Russia was more often used by managers than by scientists. This was most likely due to the fact that an article on state control over the quality of education appeared in the Law of the Russian Federation “On Education” (1992 and 1996), which gave rise to a large number of various practices of such control.

The practice of organizing quality control in education, initiated by government decisions, gave impetus to the development of relevant theoretical concepts and was the main factor in the steady increase in the interest of scientists in this problem. It should be noted that the assessment of the quality of education is associated with a number of contradictions.

1. No thesaurus.
2. Stability and variability of education.
3. Highlighting quality vocational education.
4. Theoretical and practical training.
5. Teacher assessment standards and qualification requirements for a specialist.
6. Training in professional disciplines and practice in solving standard professional problems.
7. Declared humanistic and actual technological approaches to assessing the quality of a student's education.
8. Possession of fundamental knowledge by the graduate and poor readiness to fulfill professional duties.

The effectiveness of research on the evaluation of the quality of education mainly depends on what is meant by the quality of education. This concept is interpreted by various researchers rather ambiguously.

In this situation, there are three main approaches to assessing the quality of education. The first is theoretical, within which the study of the problem follows the path of theoretical and methodological research. At the same time, sometimes the paths of transition from the theoretical level to practical development of the methodology for assessing quality and introducing it into the educational process are not clearly visible.

The second, practical approach implies that its representatives follow the path of creating tools (for example, control) for evaluating the training of trainees, without thinking about the conceptual components of the study.

Representatives of the third direction combine theoretical, methodological and practical components in their research. They follow the most difficult path, but it is precisely this approach to this problem that is most acceptable.

However, to date, a unified scientifically based system of indicators of the quality of training of students has not been developed and approved, just as there is no single generally accepted and approved system for assessing the quality of education.

The relevance of the problem of assessing and managing the quality of education is explained by the fact that recent decades higher education has become widespread, which has become a growing concern for the quality and effectiveness educational process.

Management of the development of education as an important component requires the creation of systems for monitoring the quality of education both on a national scale and in the context of the regions of the country.

Thus, at present, there is a need for such a methodology for assessing the quality of education, which would not be based only on grades in a student's diploma, but would objectively assess the actual knowledge of students, their ability to put into practice the acquired knowledge. The methodology for assessing the quality of education should also include the student's propensity for innovation, the desire for further improvement of the individual.

The conceptual and methodological basis for assessing the quality of graduate education should include the following main areas (blocks):

1) the fundamental nature of education, which allows graduates to have a broad outlook in the relevant fields of knowledge;
2) target specialization of education, allowing you to quickly adapt and successfully carry out specific duties;
3) the presence of creative skills and the ability to generate innovations;
4) the ability and ability to implement knowledge and innovation and investment projects in the industrial and social spheres;
5) social and moral qualities of graduates and the level of education in the socio-political and humanitarian spheres.

The assessment of the quality of education in accordance with the presented directions should further imply the substantiation of a system of criteria, methods for calculating a comprehensive, integral assessment, organization and informatization of assessments, which are constituent elements of monitoring the quality of education.

Relevance

The increasing role of education in social progress and the resulting increase in the requirements for its quality predetermine the relevance of implementing a set of measures related to the creation of education quality management systems. The creation of education quality management systems, built mainly on the basis of monitoring the achievements of students and standardizing the content of education, does not give visible results in improving the quality of education and the efficiency of educational institutions. As a consequence, not only among education leaders and research scientists, but also among the general public, an opinion began to form that without improving the quality of teaching, it is impossible to achieve high quality educational outcomes. In this regard, one of the promising areas of educational management is the introduction rational system assessment of the activities of personnel in the field of education.

An analysis of a number of domestic and foreign publications makes it possible to determine the most typical shortcomings of methods for assessing pedagogical activity. The most characteristic drawback of the methods we studied is the predominant qualitative nature of the indicators proposed for evaluation. Undoubtedly, the complete formalization of indicators characterizing the quality of teaching is impossible, in creative activity, which includes the activity of a teacher, non-formalizable aspects are too strong. However, in any case, it is necessary to strive for the possibility of presenting the estimated indicators in the form of objective quantitative data, using substitution procedures, group expert evaluation, etc. Otherwise, we will inevitably face justified criticism in the subjectivity of the assessment given to the teacher by the evaluating person on the basis of personal preferences, with the vagueness of the wording of assessment criteria such as “actively introduces into the educational process ...”, “during classes, shows high quality…” and so on, which is sometimes the fault of individual methods.

In contrast to the disadvantage noted above, a number of techniques are excessively “formalized”. Any act of the teacher's activity, their authors try to present mathematical formula, and the more creativity is manifested in it, the more complex, in their opinion, the formula should be. One of the weaknesses of this approach is generated by the assumption that the overall assessment of the teacher's performance is determined only by quantitative characteristics its constituents. This assumption is quite rigid and does not quite adequately reflect reality. Such important points as lecturing skills, the value of scientific and methodological publications, individual style of activity, etc., fall out of consideration. True, these methods provide for the possibility of some compensation for this shortcoming by granting evaluators the right to award "incentive points" for the high professional quality of the teacher's work. However, this only mitigates, and does not eliminate, this shortcoming. In some cases, it is proposed to evaluate only those elements of activity that can be clearly measured quantitatively, and the final result is determined only on the basis of these particular indicators that do not cover the entire spectrum of the teacher's activities. It also does not allow to obtain an objective comprehensive assessment of the teacher's activities. Therefore, the practical use mathematical models should not cancel, but, on the contrary, assume the use of traditional methods of assessment, and the developed methods should proceed from a rational combination of quantitative and qualitative indicators.

Another significant drawback of the methods is that the evaluation does not take into account the value (axiological) aspect of the evaluation. In them, all types of teacher's activities and individual indicators within the types are assumed to be equivalent, and the generalized assessments and the final assessment of the teacher's activities are calculated as the arithmetic mean of the assessments of individual indicators.

The presence of such a significant number of different methods indicates, on the one hand, the unconditional recognition by the leadership of educational authorities and educational institutions, the pedagogical community of the need to introduce a system for evaluating the performance of teachers, and, on the other hand, indicates the absence of unified approaches to its content and organization. .

Analysis of the socio-cultural situation, state and prospects for the development of the market educational services in the country, requests from parents, students led to the conclusion that the problem of developing theoretical foundations and guidelines on the diagnosis of professional pedagogical activity of the staff has acquired particular relevance and significance due to the following circumstances.

Firstly, this is due to the reform of the education system. The introduction of a new system of remuneration (per capita financing and incentive remuneration), which entails the development of uniform criteria for evaluating pedagogical activity.

Secondly, the processes of democratization of society have led, in particular in the field of education, to the rejection of rigid dogmatic schemes, the appearance of textbooks and teaching aids reflecting the diverse opinions of authors, sometimes diametrically opposed scientific positions. The professional competence of staff in the context of the existence of pluralism of opinions is an actual requirement of modern management in education.

Thirdly, the elimination of the ideological monopoly in society actually destroyed the previously existing system of ideals and social values that occupied an important place in the process of educating the younger generation. Filling the resulting gap is impossible without proper, often non-traditional, pedagogical innovations.

Fourthly, the process of active introduction of modern information technologies which is required by society. As a result, the problem of mastering computer technology and their application in practice by middle and older generation personnel.

Fifth, the realities of the world around us have changed, society itself is changing. This must be taken into account in psychological process aimed at adapting personnel to modern social conditions. And here there is a problem of reorientation of the former goals, objectives, methods of education and upbringing, which, in turn, is associated with a change in the criteria for evaluating pedagogical activity.

In connection with the foregoing, it is necessary to solve the problem of adequate assessment of the level professional activity teachers. It seems necessary to develop such an assessment model and such tools that will make it comprehensive, as objective as possible, covering all stages of the teacher's work. This model, on the one hand, will give school administrations an effective tool for making managerial decisions, and on the other hand, it will provide social protection for teachers.

Analysis of the initial situation

…Quality management in a school begins with work with a person and, above all, with a teacher, and ends with work with personnel, raising their professional level. There are no other ways... (Yu.A. Konarzhevsky)

This academic year, the pedagogical process in MBOU "Secondary School No. 20" was carried out by 62 teachers.

One of the main factors for ensuring the quality of the educational process is both the effective activity of the teacher and the teaching staff as a whole. Therefore, the school creates conditions for the implementation of the personal functions of the teacher, to increase the level of his professional self-development, readiness for innovation. In order to effectively manage the system for improving the professional growth of a teacher, the following types of monitoring are carried out:

monitoring of professional competence;
readiness of teachers to use modern educational technologies;
the readiness of the teaching staff for innovative activities;
monitoring the influence of factors stimulating and hindering the development of a teacher, etc.

However, the current practice is far from perfect, since it does not have clear criteria, there is no unified system for evaluating the activities of teachers. Therefore, the goal of our work in this area is to create a mechanism for assessing the professional skills of a teacher as a means of improving the quality of school education.

Object of study: assessment of the quality of the teacher's work.

Subject of study: indicators, criteria, indicators for assessing the quality of pedagogical work in accordance with GEF and NSOT.

Target: development of a model for evaluating the activities of a teacher in accordance with the second generation of the Federal State Educational Standard and the NSOT.

Tasks:

to analyze the theoretical and methodological conceptual provisions of the system for assessing professional pedagogical activity;
to determine the structure of the system and process of diagnostics of professional pedagogical activity;
theoretically and experimentally substantiate indicators and criteria for assessing the professionalism of personnel, develop recommendations for their implementation in practice;
implement the results of the experiment in educational practice schools;
conduct an examination of the effectiveness of the developed model for assessing professional pedagogical activity.

Resource provision.

Good material and technical base of the school.
Qualified teaching staff of the educational institution.
Sufficient educational and methodological support educational process.

Restrictions:

The inertia of the teaching staff.
Manifestations of the syndrome of "emotional burnout".
Psychophysiological changes and deterioration in the state of health of teachers.
The development of professionally undesirable qualities that adversely affect productive professional activity.
Insufficiently thought out system of motivation of the teaching staff of the school.

Planned results:

Improving the quality of educational services, and as a result, the quality of education at school.
Increasing the positional-value component of the general professional competence of the teacher, in relation to the psychological and pedagogical knowledge and skills that underlie the individualization of education.
The output of the system for assessing the quality of teachers' activities to the level of stable effective functioning.

Implementation stages

I. Organizational

The study of the state of the problem in the theory and practice of the school.
Analysis of psychological and pedagogical literature and literature on the management of the organization.

II. Practical

Selection of evaluation criteria and their justification.
Development of diagnostic tools for assessing pedagogical activity.
Approbation of the model.
Creation of recommendations for the implementation of the model.
The results of the implementation of an innovative project are monitored through a system of observation, diagnostic slices, and testing.

III. Generalizing

Implementation of verification, processing and evaluation of research results.
Implementation of the results of the experiment in the educational practice of the school.
Generalization, systematization, description of the obtained results

Action plan

No. p / p		the date	Responsible
	Organization of the work of the creative team.	September
	Acquaintance with normative documents according to NSOT	September
	The study of the state of the problem in the theory and practice of the school. Analysis of psychological and pedagogical literature and literature on the management of the organization.	September
	Development of the conceptual apparatus of the research program.	September
	Development of the content of the model for assessing pedagogical activity.	September
	Development of a regulatory framework accompanying the project activities.
	Development and approbation of the mechanism for distributing the incentive fund for the remuneration of teachers	September-November
	Selection of evaluation criteria and their justification. Development of diagnostic tools for assessing pedagogical activity
	Monitoring the composition of the teaching staff.	September, November
	Monitoring the influence of factors stimulating and hindering the development of a teacher, etc.
	Monitoring the pedagogical activity of the teacher according to the selected criteria and indicators.	January February March April May
	Conducting an examination of the effectiveness of the developed model for assessing professional pedagogical activity.	January, May

Today, most of the countries of Central and Eastern Europe, including Russia, have developed the framework for monitoring and evaluation policies educational activities as part of the global reform of their countries' education systems. These countries have begun to define norms (standards) in the development of training programs, which is an important stage in the national policy in the field of education and its quality control as an integral part. These norms (standards) are the necessary basis for determining the goals of education, creating a single pedagogical space in the country, thanks to which a single level will be ensured. general education received by young people in different types educational institutions.

However, in general, Russia has not yet taken the necessary measures to create a regular system for evaluating the work of educational institutions and the education system as a whole. It should be noted that there is a fundamental contradiction in this area: on the one hand, the autonomy of educational institutions and the teaching corps from the state in the field of determining training programs is significantly expanding; on the other hand, the autonomy of educational institutions and teachers may conflict with the systematic process of evaluation of the results of their activities by the state.

The successes of the new policy in the field of education are connected with the socio-economic processes taking place in society. Indeed, openness, sharing of responsibility, the right to diversity and matching supply to needs are principles that must first be introduced and implemented in the political and economic sectors in order to be applied then in the field of education.

At education quality assessment the following provisions should be highlighted:

o Quality assessment is not limited to testing students' knowledge (although this remains one of the indicators of the quality of education).

o The assessment of the quality of education is carried out comprehensively, considering the educational institution in all areas of its activity.

Quality assurance or quality management, addressed primarily through the use of quality monitoring, means step-by-step monitoring of the process of obtaining a product to make sure that each of the production steps is optimally performed, which in turn, in theory, prevents the release of defective products.

Taking into account the above concepts, it can be said that the following elements are part of the education quality monitoring system:

o standard setting and operationalization: defining standards;

o operationalization of standards in indicators (measured values);

o establishing a criterion by which it is possible to judge the achievement of standards,

o data collection and evaluation: data collection; evaluation of results,

o actions: taking appropriate measures, evaluating the results of the measures taken in accordance with the standards.

Monitoring the quality of education can be carried out directly in an educational institution (self-assessment, internal monitoring) or through external educational institution service approved, as a rule, by state bodies (external monitoring).

When forming educational standards, it is advisable to be guided by a pluralistic vision of the content and purpose of the standards (both the standards of the content of education and the standards of the final result that students achieve). The standards relating to the conditions that ensure the successful implementation of the standards are defined as the standards for ensuring the "process" of education. An example of such standards is the availability of the required number of textbooks and qualified teachers, appropriate material and technical support for the educational process, etc.

Thus, education is supposed to be evaluated as a result and process of activity of each educational institution from the side of controlling the level of knowledge and skills of students (at the same time teaching staff and external, state bodies), and from the side of control, evaluation of the activities of teachers.

We will speak separately about the quality control of education as the control of the assimilation of knowledge by teachers. Let's just say a few words about assessment of the performance of the teaching staff.

There is no doubt that there is a connection between the educational level of a teacher and the results achieved by his students; moreover, this is the easiest, most simplified and at the same time dangerous way to determine the suitability of a teacher for a position. It must be taken into account that teachers and educational establishments are just an element educational system, and, quite possibly, not the most influential among the many others on which the educational achievements of the student depend. Therefore, when understanding the need to evaluate the performance of a teacher to control the quality of education, it is important to remember that this element has a lesser impact on academic, educational achievements than the family environment or individual characteristics student (inclinations, motivation, etc.).

Quality does not appear suddenly. It needs to be planned. Education quality planning associated with the development of a long-term direction of the educational institution. Strong strategic planning is one of the most important factors in the success of any institution in the education system.

The leading goals of strategic planning are determined not only by the development of a general plan for the development of an educational institution for a certain time period, but also by the comprehension and revision of the main directions of educational services provided by this educational institution, and their compliance with consumer needs and forecasting the development of society in the near and distant future.

Improving the quality of education in the municipality "City of Adygeysk"

is provided through the implementation of the following tasks:

1. Improving the quality of education through ensuring the differentiation and individualization of the content of education at an expanded and in-depth level.

2. Organization of pre-profile education in 9th grades and profile education in 10-11th grades.

3. Shaping positive attitude students to educational work, knowledge, science through the organization of intellectual and cognitive activity in the classroom, outside the school, in the system of additional education.

4. Application of modern pedagogical and educational technologies: recognition as priority technologies with a personal, integrative, information and communication, health-saving, project, cognitive orientation: development of the subject-methodical, material and technical base of the school; improving the system for monitoring the quality of the educational activities of the school; ensuring the reliability of the data of pedagogical and psychological diagnostics.

5. Stimulation of educational institutions that actively implement innovative educational programs, the best professional teachers, class teachers, talented youth.

6. Ensuring organizational and economic changes in the education system that are adequate to the conditions of society.

7. Creation of a unified educational space.

8. Involvement of the public in the educational and upbringing process.

9. Continuity between general and vocational education.

How effective education is, what updates are required in this area, and what should be abandoned - all this is determined through periodic monitoring and evaluation of the educational process. Aspects and nuances of this difficult process, which require huge efforts, painstaking work, deep potential and sincere desire, will be discussed in this article.

Definition of value

The quality of education is a complex characteristic scientific activity and training of students. These are indicators that express the degree of compliance of training with federal state standards and the needs of individuals in whose interests educational activities are carried out. The quality of vocational education is also determined by the degree of achievement of the planned results thematic program. They are evaluated, compared and analyzed.

Why is monitoring the quality of education necessary?

Education monitoring is inside quality assessment systems. It serves as information support for the supervision of current activities. Actually, monitoring is a complex analytical tracking of all processes that determine the quantitative and qualitative changes in the characteristics of educational activities. Its result is a conclusion about the extent to which the achievements, their conditions correspond to the requirements fixed in the regulatory documentation and local acts of the state system.

What does the assessment of the quality of school education include?

The assessment of the quality of education includes checking the implementation of the sanitary and hygienic standards of the educational process of the school, catering, as well as the implementation of measures to ensure the safety of students. For a comprehensive study and analysis of the state of educational activities, its results and conditions, expert methods are used.

To internal evaluation include procedures organized and carried out by the school itself, as a rule, by the administration, teachers, students, as well as with the involvement of parents and the public. The resulting indicators are used to develop operational decisions that underlie school planning. Examples of this type of assessment, which will be mandatory for analytics that determine the quality of education at school, are self-assessment of the subjects of the educational process, the collection of statistical data, school monitoring, evaluation of subject curricula, and parent surveys.

Goals and organization of monitoring and control

As you know, the purpose of monitoring is the collection, generalization and analysis of the collected information regarding the state of the education system of the school. Quality control of studies is based on these data. To achieve the goals set, it is necessary to solve the following tasks:

A mechanism for accurate collection, processing and storage of information about the quality of education should be formed.
Coordination of the activities of all involved monitoring participants has been established.
Points of growth in the dynamics of the results of the educational process were identified and fixed in a timely manner.
Factors that significantly affect the quality of education should be identified and measures should be taken to reduce the effect and eliminate the possible negative consequences of those that do not carry positive dynamics.
Involvement of program and methodological, material and technical, personnel, information and technical, organizational and other bases that improve the quality of education.
Determining the direction in accordance with the results of educational activities school curriculum for the previous academic year, in conjunction with the problems and tasks set for the current period.

Education in modern Russia

Many scientists associate the beginning of the twenty-first century with the advent of the era of innovation. They bring the most important transformations in the educational sphere, which seems to be able to radically change our understanding of its role in modern society. The basis of such innovations is the development of non-standard approaches to the learning process using modern technologies which will greatly improve education in Russia.

The role of the educational process in present stage development of our country is determined in its interaction with the tasks set for the transition to a democratic-legal state, as well as to eliminate the danger of the state lagging behind world trends in the field of economic and social development. Exactly modern education is associated with the growing influence of the quality of human and intellectual capital on social development, with the entire process of accumulation and successive transfer of knowledge. Therefore, modern and future generations need an effective dynamic learning system based on innovative technologies.

Requirements for the quality of education in Russia

The main task of the Russian educational policy is to ensure the modern quality of education on the basis of maintaining its fundamental nature. Compliance with the current and future needs of society, the individual and the state is also important. In the conditions of individualization of education, modern education in Russia should be continuous. Such a requirement is determined by the human need for the need to constantly replenish their own knowledge during their professional activities and the progress of science and technology. The goals and principles that define modern education should be oriented towards preparing students for full and effective participation in the public and professional fields in the current conditions of market relations.

What gives control over the quality of educational activities?

The control and monitoring of the quality of education makes it possible to modernize the system of the Russian educational process in a timely manner. This is laying the foundation at the stage of building the building of an economically and politically effective legal state. It is not so much about controlling the knowledge, skills and abilities of students, but about the quality of the system itself and teaching methods.

At the present stage, improving the quality of education is the introduction global change in the goals and content of programs that will guide students to develop a new model of preparing people for life and for professional activities. They need to develop completely new personal qualities and skills. All this is also dictated by the new requirements for modern specialists.

Evaluative monitoring is the basis for the formation of competent specialists

The modernization of the education system and the introduction of information and communication technologies into the learning process open up A New Look on the quality of professional knowledge. By making learning open, we radically change its properties. They emphasize the freer planning of the learning process, the choice of time and pace, the choice of place, the transition from the principle of "education for life" to a new conceptual concept of "knowledge through life".

Today, most countries pay increased attention issues such as learning effectiveness. Pay attention to the quality control of education. To do this, scientists combine their efforts to develop technologies and tools, methodologies and comparative studies of the effectiveness and quality of the process. By doing so, they create a monitoring system to assess the quality learning activities on a global scale.

Progress according to time

The current system forms an updated model for training specialists. The assessment of the quality of education has brought it to a new level, when it is required to take into account not so much the qualification model of a specialist as his competence. A professional who is knowledgeable in his field is distinguished not only by knowledge, skills, experience, but also by the ability to implement them, bring them to life, act, create and create.

The quality of education in the competence model of a specialist is associated with integrated interdisciplinary requirements for the result of the current process. That is, the qualitative characteristics of the individual come first, which will be formed by modern educational technologies. Such a system is implemented in practice using network learning technologies. First of all, they have spread widely among representatives of those social and age groups that receive in-depth knowledge without leaving their main labor activity. As a result, a more efficient use of modern technological advances in the traditional education system will lead to the fact that the line between full-time, part-time and distance learning. And this, in turn, is the main feature progressive innovative learning modern youth.

The problem of the quality of education as a problem of control and evaluation of educational activities

Today, most of the countries of Central and Eastern Europe, including Russia, have developed the framework for a policy of monitoring and evaluating educational activities as part of a global reform of their countries' education systems. These countries have begun to define norms (standards) in the development of training programs, which is an important stage in the national policy in the field of education and its quality control as an integral part. These norms (standards) are the necessary basis for determining the goals of education, creating a single pedagogical space in the country, thanks to which a single level of general education will be provided for young people in different types of educational institutions.

Ateducation quality assessmentthe following provisions should be highlighted:

Quality assessment is not limited to testing students' knowledge (although this remains one of the indicators of the quality of education).
The assessment of the quality of education is carried out comprehensively, considering the educational institution in all areas of its activity.

Quality assurance or quality management, addressed primarily through the use of quality monitoring, means step-by-step monitoring of the process of obtaining a result to make sure that each step is optimally performed, which in turn theoretically prevents the output of an incompetent student.

Monitoring the quality of education can be carried out directly in an educational institution (self-certification, internal monitoring) or through an external service in relation to an educational institution, approved, as a rule, by state bodies (external monitoring).

Thus, education is expected to be assessed as a result and process of the activities of each educational institution from the side of monitoring the level of knowledge and skills of students (simultaneously by the teaching staff and external government bodies), and from the side of monitoring and evaluating the activities of teachers.

We will speak separately about the quality control of education as the control of the assimilation of knowledge by teachers. Let's just say a few words aboutassessment of the performance of the teaching staff.

There is no doubt that there is a connection between the educational level of a teacher and the results achieved by his students; moreover, this is the easiest, most simplified and at the same time dangerous way to determine the suitability of a teacher for a position. It must be borne in mind that teachers and educational institutions are just an element of the educational system, and, quite possibly, not the most influential among the many others on which the educational achievements of the student depend. Therefore, when understanding the need to evaluate the activities of a teacher to control the quality of education, it is important to remember that this element has a lesser impact on academic and educational achievements than the family environment or individual characteristics of the student (inclinations, motivation, etc.).

The most important principles for monitoring learning(achievement) of students - as one of the main components of the quality of education - are:

Objectivityconsists in the scientifically substantiated content of control tasks, questions, equal, friendly attitude of the teacher to all students, accurate assessment of knowledge and skills adequate to the established criteria. In practice, the objectivity of the controllers, or as they often say in recent times- diagnostic procedures, means that the marks given are the same regardless of the methods and means of control and teachers.

The principle of systematicrequires an integrated approach to diagnosing, in which various forms, methods and means of monitoring, verification, evaluation are used in close relationship and unity, subject to one goal.

When preparing materials for test control, it is necessary to adhere to the following basic rules:

Answers that cannot be justified by students at the time of testing cannot be included.
Wrong answers should be constructed based on common mistakes and must be believable.
The correct answers among all the proposed answers must be placed in random order.
Questions should not repeat the wording of the textbook.
The answers to some questions should not be clues to the answers to others.
Questions should not contain "traps".