The paper-based exam marking system follows a tedious and time consuming process involving several levels of sorting before bundles of handwritten scripts are delivered to the evaluators to assess the candidates. To make the process simpler and efficient, Rajasthan Public Service Commission (RPSC) has taken an initiative to adopt an on screen marking system that exploits the latest Information Technology tools to achieve the desired results. Akhilesh Mittal, Joint Director IT, RPSC, Ajmer, Government of Rajasthan, tells Elets News Network (ENN) about the system that makes awarding marks to the assesses just a click away.
What is an On Screen Marking System?
On screen marking is the process of applying technology to the process of marking or scoring descriptive answer scripts with a view of improving quality, flexibility, efficiency and transparency while making the scoring process easier for evaluators. It involves several processes which include scanning answer scripts, indexing answer scripts, setting up and creating the marking environment, marking by evaluators and analysis of markers’ performance.
The process to be adopted for an on screen marking system is as follows:
Answer Booklet Collection from RPSC
A resource will be deployed from the firm, who will work with RPSC official responsible for the handover of the answer booklet to firm after separation on main page containing confidential information and RPSC office secret cell would be processed this task. The firm resource will collect the documents from RPSC and will get the documents counted and then reconciled against the performa given by the department. Once the documents are reconciled then an entry into firm‘s inventory system will be made.
Bar Coding or Numbering Answer Sheets
If the answer booklets are not bar coded then a bar code will be affixed on the answer sheet, which will help in the separation of the answer sheets and also will give a unique number to the answer sheet in a particular order for tracking the answer sheets at later point of time. Since the answer scripts are barcoded and numbered, the answer booklets are received and stored in zone wise, venue–wise, subject wise and preparatory work for scanning shall be undertaken.
The answer booklets will be made into packets of 25 or 50 as per decision of RPSC. The packet details along with the details of the answer sheet series is entered into the inventory system. This will help in keeping track of the count as well as locate the answer booklet after the scanning process on need basis.
Preparation of Scanning
The answer booklets will be prepared for scanning by cutting the spine (if the booklet is stitched in middle) or staple will be removed (if stapled in middle), so that the individual sheets of answer booklet can be feed to the scanner. Generally, 25 booklets shall be feed to the scanner once it is prepared for scanning.
Batch Preparation and Scanning
The scanning units then prepare the batches and scan the documents.
Image Processing and Quality Check
The scanned images of the answer booklets are moved to the local server and from there it is pushed to central server located at a three-tier data centre. The images are processed and then the quality of the images is checked before they are moved for indexing. If there are any discrepancies in the images of a particular script or if the image quality is bad then those scripts are rescanned.
The answer booklets will be prepared for scanning by cutting the spine (if the booklet is stitched in middle) or staple will be removed (if stapled in middle), so that the individual sheets of answer booklet can be feed to the scanner
Storage of scanned booklets
Scanned booklets are packed once again in the order of 25 each and the packets are stored in carton boxes. The boxes are numbered and have the details of the packet numbers on them. The carton boxes are handed over to the department for reconciliation. The flow of document management is shown below.
A centrally hosted solution that enables higher security/confidentiality, high levels of data integrity, higher redundancy and most importantly, a complete automation of the process.
Post scanning process:
- Answer scripts are scanned at RPSC premises and the images securely transmitted to an central image server.
- The scanned copies of answer booklets are then distributed electronically and marked On Screen by evaluators.
- The Client software is an application program which is downloaded onto the evaluator’s machine from a secure data center, through a secure network.
- Each evaluator is provided with a username and a strong password. Following successful log in, the examiners can access the key functions of software for which they have been configured.
- Software provides the evaluators with information about the subject, expected number of scripts to be marked by when, along with the tools for marking and a set of support features such as the inbuilt messaging system.
- In addition, software provides the marking supervisor with additional functions allowing them to approve, review and view the marking of team members and to provide support to evaluators through a messaging system.
- The marking screen displays candidate responses, the structure of marks, a mark input tool and a range of other supporting tools. It allows the Evaluator to view and mark each candidate’s response.
- Question-level marks and evaluators’ annotations are captured by the software throughout the marking process, without manual intervention.
- Reporting systems provides access to a wide range of reports covering different aspects of marking, supervision and administration process. Reports are tabular in format and displayed on screen, but can be exported to Microsoft Excel for further analysis and are compatible with other statistical analysis tools.
- Marking window can display range of response types:- Structured Responses: A candidate is expected to write the response in predefined area. The system only displays the predefine area for the evaluator. o Unstructured Responses: the candidate may write their responses anywhere within the answer booklet, for example when writing essays. The system displays all of the candidate’s response and provides the Evaluator with the tools to easily navigate between the images. So it is proposed that the answer booklet will have structured responses only. The firm shall adopt the process of structured answer script evaluation mode
- Marking non text scripts like drawing, geometry etc., can also be marked very easily using:-
- Ruler tool for measuring distances
- Protractor tool for measuring angles
- Multi-line tool, which is very flexible for marking graph or similar questions
- The system supports multiple features marking. For example, separate marks for accuracy and method.
- Marks can be reviewed by the evaluator any time before the response is submitted.
- Supervisor can inspect the marked scripts and post for remark if needed.
- Supervisors can reallocate scripts based on needs.
- Constant and dynamic monitoring of marking process is available through the administrative interface.
- The system also allows for multi way messaging, allowing evaluators to communicate with supervisors for guidance, reporting complaints, etc.
- At the end of the marking process, marked script with item-wise scores, annotations and summary of scores will be re-generated in PDF, submitted to the RPSC and achieved.
To enable a smooth transition for the evaluators, the firm will conduct an in depth training session for the evaluators as per the list given by the department. The training will be provided in a live modular training session explaining all the features of evaluation platform followed by hands on training of few scripts.
Security for the solution is managed at all levels – physical, user level and relevant communication.
- Physical: The servers hosting the OSM software will be located at a secure data centre where access will be controlled and only available to authorized personnel.
- User: User accounts for the administration portal are generated through secure data interfaces. Users gain access to the application through a username and password. The password is automatically generated and can be characterized as strong.
- Communications: communication between applications and the core domain is encrypted to prevent eavesdropping and data tampering. Web service Extension (3.0) has been incorporated into the web services to ensure data integrity, data confidentiality and user authentication. The tool communicates across the secure link using secure socket layer SSL on port 442(HTTPS).
The software should be designed in such a manner that people don’t have direct access to the database. The data is securely hidden behind multiple layers of architecture.
To enable a smooth transition for the evaluators, the firm will conduct an in depth training session for the evaluators as per the list given by the department. The training will be provided in a live modular session.
Evaluation centres will be provided by RPSC based on the number of evaluator required for evaluating the answer booklets with-in the project timelines. The evaluation centre must have required number of computers, uninterrupted power supply, UPS back-up, good lighting, internet connectivity (min 8 Mbps for 25-seater) and preferably air conditioned lab or room.
Sanitisation of Computer: The selected evaluation centre to be handed over to firm team one week in advance from the start date of evaluation under supervision of RPSC. The firm team shall sanitize the computers, check for the min configuration of the system, load required evaluation software and get ready for marking
Setting up of DVR Cameras: In order to maintain high degree of security of the evaluation centres, firm shall deploy DVR Cameras to cover the entire evaluation centre. The recordings of the evaluation shall be done continuously; copy of the same shall be handed over to the department at the end of the evaluation process
Undertaking of secrecy and security of Data: The firm must provide undertaking for maintaining secrecy and security of data and no leakage of data during evaluation or what so ever.
Given the significant number of answer scripts (around 1 lakh), the traditional system is very tedious and error prone which create unnecessary burden of RTI issues and legal cases as scores totaling and re-totaling being also done manually and sometimes unchecked portions are found in answer scripts. RPSC was in search of solutions to resolve these issues since long.
In this regard, RPSC has taken various presentations from firms in the field of on screen marking system which may be fit for RPSC requirements to achieve the following major benefits that include reduced risk, improved security, improved quality, efficiency, faster time to results, ensures innovation and management control.
After that RPSC was of view that such answer keys evaluation process and execution through vendor is very much confidential like printing press and therefore, RPSC in principally agreed to execute the task confidentially only from best firms who are in such field with proper recording of process confidentially.
It is recommended that RPSC must define proper technical and administrative process as per above suggested process for effective implementation.