Maharashtra Knowledge Corporation Limited (MKCL) has been implementing a Maharashtra Statewide IT-Literacy training programme over last five years under its Maharashtra State Certificate in Information Technology (MS-CIT) brand.
MS-CIT is a 132-hour (3 month, part-time) course in IT literacy in English/ Marathi / Hindi medium offered at more than 3000 Authorised Training Centers (ATC’s) of MKCL spread over all 35 districts and 358 tehsils of the state spanning urban and rural areas.
MS-CIT has emerged as a de-facto standard in IT Literacy in the state and more than 2.6 million learners have undergone this course over last five years. The response to this course is steadily increasing over the years.
MKCL has implemented several measures for quality improvement, some of which are:
1. Novel Curriculum and its continuous upgradation in response to emerging trends in technology and market and exceeding international benchmarks in IT literacy
2. A continuously evolving (ver 23.0) interactive multimedia tutor powered by MKCL Learning Management System (LMS) with rich voice over in English / Marathi / Hindi
3. Quarterly quality assessment of all Authorised Training Centers (ATCs)
4. A fully Web-based management framework for management of Learner life-cycle from admission to certification and management of ATC life-cycle
5. Wider and transparent availability of recent and vital information of all aspects of this entire activity on MKCL Website to all stake-holders and public at large
6. An online supervised examination system at about 1000 Online Examination Centers in the State leading to instant e-Certificates.
The problem was how to conduct a meaningful assessment leading to certification of a large number of trainers; at affordable cost; with high quality, reliability and trustworthiness; with widest accessibility over metropolitan, urban, semi-urban, rural, tribal, and hilly areas of the state; in shortest possible time of 3 months; in a mass-personalised manner leading to personalised assessment rather than mere mass-testing so as to offer developmental inputs to the trainers.
An e-Learning technology-mediated approach was the only inevitable pointer to solve all these complex problem.
e-Learning technology mediated approach
MKCL devised a unique assessment model of Online Certification Test (OnCeT) and designed and developed an enabling technology platform for it in the form of MKCL’s Online Evaluation Framework (OEF).
MKCL’s OnCeT model for certification of IT literacy trainers
Online Teachers’ Certification programme (OnCeT) has been launched to examine the skills and competencies of the Trainers of IT Literacy for using various application software tools effectively and efficiently, through a set of assignments and thereby evaluate the level of their mastery on the subject matter so as to bring about overall improvement in quality of academic delivery of the course.
OnCeT comprises of eight unsupervised weekly tests and a final supervised comprehensive Online examination at the district location.
Type of each of the eight tests includes Practical and Objective type online tests. Each Practical Test comprises of three assignments which are to be solved in unsupervised environment and submitted in one week’s duration and passing threshold is at 50% marks. Each of the Objective tests shall consist of 30 questions to be solved online in a single session of 45 minutes maximum one week delay in submission may be allowed with 20% de-rating of score, thereby requiring 62.5% marks for passing instead of 50%.
Eight tests have a weightage of 80 marks and a final supervised comprehensive Online examination has a weightage of 20 marks.
In all the interim assignments of OnCeT, various skills and competencies of the trainers at various levels have been tested, such as, knowledge, comprehension, application of knowledge and analysis.
MKCL’s Online Evaluation Framework
MKCL’s Online Evaluation Framework (OEF) is a collaborative framework that facilitates Online Assignment Management and conduct of objective type tests with various types of questions.
Assignment Management System features
Authenticated Personal Interfaces: Online Assignment Management System offers authenticated personal interfaces for Trainers, Subject Matter Experts (Faculty), Evaluators (Assistant Faculty), Tutors (Lab Instructor) and administrators. This has been designed and developed with an integrated approach to enable the Subject Matter Expert (Faculty) to post assignments for the trainers based on the course structure.
Assignment Randomisation: By using the feature of randomisation, a set of assignments can be prepared and various assignments can be given to multiple trainers randomly.
Submission and Evaluation: Trainers can download, complete and then submit the assignments online for approval and feedback from the assistant faculty or local tutor which after approval are made available to the evaluators. This functionality is optional and in case of Supervised Final Examinations, assignments can directly be routed to the Evaluators for final evaluation.
Work flow: The automated work flow in the system keeps track of every assignment of individual trainer. Framework also offers the facility to the trainers to get the assignments evaluated from best of the evaluators.
Objective Evaluation System features
OEF also has an inbuilt engine to conduct Objective type tests with functionality offered to evaluators to frame variety of questions to test the trainers.
Different types of questions include multiple choice single correct, multiple choice multiple correct, match the following, fill in the blanks, picture identification, jigsaw puzzle, crosswords, hotspots, second level reasoning and concept map.
By following international QTI standards prescribed by SCORM, to maintain the question repository, different question parameters are stored, such as, difficulty level, type, marks, and associated learning unit.
Continuous up-gradation of the question repository enables the question paper setter to go for varied combinations while designing the paper structure. Also, framework supports random paper generation.
Trainer authentication features such as photograph and signature display on the screen are also best supported by the system.
Some of the important distinguishing features include test attempt record, result, and crash and recovery system.
Admin’s login: Admin creates users to perform specific role. With the specific user name and password provided by admin, the users can access OEF.
Work Distributor’s Login: Work Distributor allocates the work to the Subject Experts (Faculties) and Evaluators (Asst. Faculties). While giving the assignments for Evaluation, Work Distributor decides the number of assignments to be given to one Evaluator.
Subject Expert’s Login: Subject Expert can post the assignment for various topics using the interface. While posting the assignment, s/he can set the start and end dates, extended date if any, maximum marks, number of re-submissions allowed, etc.
Evaluator’s Inbox: Evaluator can download the solved assignment submitted by the Trainer and can Evaluate the same. After evaluation, s/he can enter the marks and personalised corrective feedback for that assignment.
Trainer’s Inbox: Trainer can get a set of assignments to be solved in his/her inbox along with the reference material available for download for each assignment. Using the same interface, solved assignment can be submitted, marks allotted and corrective feedback given by the Evaluator can be seen.
Trainer’s Interface for Objective Test: Trainer can access all the objective Tests by using this interface. Upon selecting any Test paper, trainer can view the start and end dates, duration of test, total questions, etc…
Trainer’s Result sheet: After the evaluation process is complete, the Trainer can get the Result sheet in his/her Login. The Result sheet will have the marks obtained out of maximum in each topic and also
the final percentage of marks and final result.
Difficulties encountered and solutions designed and deployed during the implementation phase
Several difficulties that are addressed during the implementation phase are (a)Failure to submit assignments on-time; (b) Incorrect Assignment Submission by Trainers; (c) Virus Infected Assignment Submission; (d) Natural calamity like floods; (e) Evaluators’ Performance; (f) Re-evaluation requests from Trainers.
Extended Date of Submission was provided for the Trainers to submit the assignment after the original end date. A new feature was added in the OEF, which helps the trainer to view the already submitted assignment. The new feature routes the assignments for re-evaluation automatically, based on the criteria set by the System Admin. Special sessions on distributed classroom were arranged to orient evaluators. Continuous call center support was provided to the Trainers to handle their issues related to assignment submission using OEF. Integrated SMS Gateway technology was used extensively for instant communication and follow up with evaluators.
Spin-offs and future directions
OnCeT Model was found to be generic and can be replicated for conducting IT based Certification Programmes by best using the configurability and customisation capabilities of the system. Online Evaluation Framework can be used for students of the Universities / Colleges offering Distance Education programmes. OEF can cater to the needs of the students undergoing these programmes by offering them an access to interim assignments and also the corrective feedback by remote evaluators.
About 12100 teachers (who appeared for OnCeT), 67 evaluators have contributed to the successful implementation of OnCeT programme and more than 180000 assignments have been submitted and evaluated so far.
OnCeT has become a case study for its unique methodology for skill testing, quality of the assignments, huge number of uploaded assignments, functional use of state-of-the-art e-Learning technology frameworks by thousands of grassroot trainers spread over the state of Maharashtra, and moreover, the characteristic collaborative effort of the evaluators to offer personalised corrective feedback to the trainers.
OnCeT programme, thus, attempts to simultaneously satisfy all the complex parameters affecting the learning services, such as, low cost, high quality, high speed, high touch, wide access and personalised experience to masses.
In essence, Online Certification Test (OnCeT) illustrates effective functional use of eLearning Technologies by grassroot Trainers to test their IT competencies and set an example of avoiding the Digital Divide effectively.