Are we there yet? Evaluating assessment validation 12 months on

We will be showcasing at the nxt ICVEt Conference 4th August at the ICVET Learning Powerhouse.
We will be discussing what we have found 12 months on with assessment validation.
These are some of the questions we are covering so if there is anything we have missed please let us know. We are trying to make the presentation as interactive as possible!

So how did we go? How will we measure the outcomes? Did we meet the requirements of the VETAB Audit 2005? With the final question: Are we there yet?

Find about what has happened over the last 12 months with stories reflecting on the highs and the lows as well as the learning that took place. Join in a discussion on sharing ideas or strategies that can be used for evaluating assessment validation performance.

The final discussion will highlight the ongoing issues of:
Time versus effort:
How often and how much and how that impacts on delivery and preparation.
Sustaining the motivation and momentum:
How do the processes prepare for constant changes in Training Packages?
Record keeping: How long do we keep the records? 30 years and beyond?
Share your experiences with others at this workshop.

It would be great to hear from you all again!

Thank you for your Participation

Rob and I would like to thank everyone who participated in the Assessment validation Workshop and and for sharing your ideas and comments. Give us some feedback or contact us if you would like more information about either of our projects.

By using this blog we hope to identify if there is interest in developing a network across Institutes to share ideas and strategies for assessment validation.
We believe technology will support us to do this so let us know your thoughts.

Thanks again.

The best assessment validation process/procedure would……”

Final question for the workshop was to come up with ideas that if we could develop the best model for assessment validation what would it be, what would we need and how could we use it...

Here are the responses:

Team work communication
Easy and quick to use
In lead pencil and the pen after review
Clear Assessment items to enable each individual (sorry could not read the rest of this??)
Be fair and equitable
Easy to use, time to do it (paid to do it) and more time in non teaching week.
Team approach
be easy to use
Done by someone else (love this but who would that be ???)
Involve teachers in an ongoing process and continuous learning
User friendly and self evident and self sustaining
Be inclusive of all staff and sustainable
Be creative, innovative, adaptable, interesting.
Teaching materials (we are developing a CD rom for this if you are interested it will be showcased at the Quality awards)
web based online data base updated continually
Paying people who are good at it to do it (Mmm. I wonder if we could the funding for this...)
Create a reliable valid consistent state wide assessment bank. (yes I agree we are trying SWSI anyone interested in this pursuing this?)
Be easy, be fun, and involve everyone associated
Part of professional practice.

Have you though of anymore ideas? Share your ideas with others here...

Describe a successful Assessment Validation strategy

Time to think about the successes in Assessment Validation.

Here are the examples provided from the workshop.

Feature Assessment validation on meeting agenda per term.
Share point trialled assessment for comment.
Check that the events meet the LO or unit of competence.
comparison's with other Institutes.
Writing assessments with others.
None known at present.
We have had very little assessment validation.
Question bank.
Trialling of assessment.
The students involved with the process.
design as a module team and then assessed by another module team.
Timetabled paid meetings for all staff to discuss and do the validation and planning in non teaching weeks.
teacher meeting to discuss what does it mean to be competent? Then focus on the tool.
Website for upload and download of assessments with approval process to validate.

Some great ideas. Currently in SWSI we use the Janison Learning Management System to upload and discuss items. If you would like to know more about this as I think everyone using is using Janison(well I think so?) then contact us and we will show you how it can be done. Leave a comment here...

What do you think can make Assessment Validation successful?

The next question was about what would make assessment validation successful.

Here are the responses:



Support for staff to embarce and understand Assessment Validation
Team Work????
Safe Environment
Trust- sharing
Make the time commitment to evaluate
Clear instructions
Excellent teacher
Agreement by all staff
Idnetification of all stakeholders
Representative from each discpline
Knowledge of modules- evidence
Teacher commitment to quality teaching
Leadership, commitment and staff development to proivde the tools and motivation for success
Commitment for participation
New ideas and new techniques
Greater netwroking between Institutes and greater involvement
Validation Tools
Motivated and ideas encouraged.
Knowledge of materials to be validated
Planned approach and meaningful participation

All important points. I have long supported the concept of a "student free day once a year" This day would be for all full time and part time students to participate in a day of validation that is fully paid for.

If you can think of any more idea please add a comment here....

What do you think you will get from this workshop?

This was the first question we asked at the workshop.


What did everyone want to know? These are the responses:

Ideas and other's experiences of Assessment validation.
Insight into other methods of assessment validation.
Different approaches.
Assessment Validation tips.
Validation techniques.
Ideas on assessment validation.
New ideas and techniques.
How to do assessment validation.
To look at assessment validation from another point of view.
What is happening in other Institutes regarding validation.
How assessment validation can be organised/ implemented in 2 different ways.
Ideas to take back to the college.
Strategies for assessment validation.
Moderation strategies.
An excellent model for assessment validation.
Not sure about this one but here goes....
What to do implement OAY NSI mentor VA Project? (if someone could elaborate on this one that would be great!!)

Overwhelming new (and possibly excellent!) strategies / ideas / experiences about assessment validation was what was wanted. Unfortunately there was limited time to share all the stories. So hopefully this blog will lead to more sharing. We have opened the discussion with our stories....

So if you have a model or experience to share you may want to add a comment here.....

NAVAL by Jo Fuller

This is the picture I like to use of myself!!

Background Information
NAVAL is an acronym for Network Assessment Validation Assessor Leaders. This group was formed from the Community Services faculty, South Western Sydney institute TAFE NSW to further develop assessment validation across the Faculty. The group represents Children’s Services, Nursing, Welfare, Sport and Recreation, Hairdressing, Beauty Care and Aged Care.

The group identified three main priorities:
• A consistent assessment validation process
• Development of an assessment bank
• Training of staff

An action plan was developed for the period 2005-2006 which identified strategies, resource implications, measures, performance results and review tools incorporating reporting and communication strategies.

Monthly meetings have agreed standing items. These include:
• Review of processes including discussions to ensure continual improvement
• Discussion and development of tools that are easily accessed and customised
• Training which utilises CD resources (eg. ANTA TAA certificate IV Toolbox) internet resources (eg. Flexible Learning Framework) and relevant training materials (eg The Assessor DVD Training package)

NAVAL participants are the lead assessors for sections throughout the Faculty. This includes mentoring, provide advice and assistance to staff of the section and ensure an effective monitoring and reporting system to the Faculty, Community Services. The lead assessors develop and implement continual evaluation of assessment validation processes and procedures. NAVAL has a combination of Head Teachers and teachers from the sections but most importantly participants are motivated and interested in assessment validation.

The restructure of SWSI has provided a Faculty structure that incorporates a position of Teaching and Learning Manager who is responsible for the development and monitoring of the assessment validation process. This provides ownership and provides support to the continuation of the processes.

Assessment validation has been implemented and adapted in many ways to suit the local needs of sections within the Faculty. Sections were at differing stages of implementation of the assessment validation process. On investigation a wide selection of forms and processes were being used and it became evident that a needs analysis was needed. Issues that were identified included:
• Time versus effort; was the level of effort producing desired outcomes?
• Resources were limited or non existent. So how do you keep staff motivated?
• Record keeping: how many forms are enough?

NAVAL formed from a common need and interest in developing a sustainable and user friendly system that can be developed to provide a process that is embedded within the practices of the teaching sections.

Key Themes

Current Project
NAVAL has been in existence since May2005 and the interest and participation has meant the progress of the group has far surpassed anyone’s expectations. NAVAL has been well supported by sections and in some circumstances there are 2 section representatives. The Assessment Validation Action Plan has provided clear direction and purpose to the meetings and gives the Network an opportunity to measure success or the opportunity to refine the outcomes.

Strategy
NAVAL is a network of Assessors that participate in the assessment validation process. The philosophy behind the group is to motivate and encourage creative ideas and strategies to effectively manage assessment validation and build a self sustaining process. The leaders have an opportunity to share and discuss issues whilst supporting each other in the challenges that they may face in monitoring the assessment validation process in their sections. The group has direct input into all facets of the process and therefore members have an understanding of the way decisions are made and can take ownership for the continual evaluation and improvement systems used.

Challenges
Challenges for NAVAL occur on two levels. Resources support can often mean the Leaders take on additional duties within their teaching load to participate at meetings, mentor and monitor the section’s activities. This can lead to burnout or being overworked to the point that members may become disillusioned. Important that the network be supportive at all times.

Successes
The Faculty Community Services providing a Staff Development Day later in the year to officially launch the Faculty’s assessment validation training package. This training package will inform staff of the processes and demonstrate the effort and commitment of the NAVAL group.

Future directions
Sustainability is the long term aim of this network to ensure teachers have the support and encouragement to assist in making professional judgements when assessing students. This ensures an equitable and reliable process for all. Systems are being developed to provide an assessment bank of tools, exemplars and marking guides that will further assist teaching sections and help the Faculty and the sections meet the standards required by the Australian Qualifications Training Framework.

CAVAN by Rob Young

This is Rob's story. Did you know Rob can dance?

Background Information

In 2003, the then Construction and Transport Faculty within SWSI embarked on a Reframing the Future project to address AQTF Standard 9 in relation to ‘Assessment Validation’. The project was initiated to meet the requirements of an internal assessment validation process for SWSI staff. Workshops and meetings were held throughout the project to allow teachers/assessors to come together to compare and discuss assessment issues. I was one of two Project Officers who facilitated and managed these sessions.
The focus of the project was to develop staff in relation to assessment validation processes, procedures and requirements with an aim to create usable tools and instruments, which addressed the first stage/year of each trade area.

The staff selected were mainly full-time teachers with most of these being Head Teachers. These members formed the nucleus of a group earmarked to become train-the-trainers for future projects. The outcome of the project resulted in the development of usable tools and instruments, at stage/year 1 level, for the majority of the construction trade areas and several of the transport areas. Tools were developed using a standard format for written and practical tasks, as well as assessment instruments, which were up-loaded to the institutes’ intranet share site, i.e. EKB (Educational Knowledge Bank) this site provided a secure repository for validated assessments, saved as PDFs, and ensured version control for these documents.

To a large extent it was a serendipitous journey, even though there was much pre-planning and an ‘action plan’ to follow. We discovered and uncovered issues we didn’t expect to encounter such as some of the existing transport Category C assessments not complying with course document pass levels, a general lack of funding support by the institute for the program, to being surprised by the ownership of the project taken by some staff members.

The project concluded in November 2003 with a high degree of success, however there were some negatives to come out of it, such as a lack of enthusiasm by some staff to embrace the need for a standardised approach to assessment, difficulty in getting release for some full-time staff, the unavailability of part-time teachers to attend due to demands of their sections and the lack of funding to sustain a planned approach to validation beyond the life of the 2003 project.

Another issue affecting further work throughout 2004 was the fact that most of our trade areas were about to receive new training packages in 2005, which would involve a shift from the current modular-based delivery to a new unit-based approach. We were also about to undergo yet another restructure to become independent faculties.
Therefore, it was agreed that we would hold off till 2005 before undertaking any further work in this area, as all the documents produced would need to be modified in some shape or form. This lead to the application for Reframing the Future funding in 2005, so we could re-develop our assessments, the assessment approach and establish an assessor network consisting of SWSI staff and many of our colleagues from around the state.

Key Themes

Current project:
The new SWSI Construction Faculty was successful in its application for 2005 Reframing the Future funding of a sub-program in ‘Effective Networking’, which has been named CAVN (Construction Assessment Validation Network).
The project is designed to create a sustainable network connection between assessors delivering 5 different training packages. The project is hosted by SWSI and includes stakeholders from several other Sydney-based institutes, regional institutes and relevant industry associations who will form the nucleus of a NSW state-wide assessor network aimed at improving the quality and consistency in assessment for construction.

Strategy
The project builds on the approaches and processes established during the previous SWSI internal validation program in 2003. The project takes the next step required by the AQTF Standards to include assessors from other RTOs by adopting an ‘external validation’ approach.
Many of the previous project’s construction staff, as well as many new full-time and part-time staff, have been included as the current ‘Team Leaders’ for the 2005 project. These team leaders meet formally on a regular basis, i.e. at least once and up to twice per term, to discuss approaches, procedures and processes in relation to the development and recording of assessment outcomes.
The group has adopted an ‘Action Research’ and ‘Action Learning’ approach, which is being facilitated by a trained institute facilitator. This approach allows members to reflect on their individual experiences and share them with the group. The approach has already identified the need for an alternative assessment process to be implemented within sections who are delivering via a day-release mode, as opposed to those delivering in a block-release mode. It has also clearly identified that there does not have to be a one-shoe-fits-all approach to assessment.

Challenges
The greatest challenge will be to initially get consensus across a large number of sections on the degree of unit clustering and then adoption of the developed assessments. Also, sustaining the project beyond 2005 and keeping team leaders motivated when the pressures of day-to-day section life kick in.

Successes
Part of one formal meeting/workshop involved a presentation overview of an instructional design model, known as 4MAT. The presentation was inspiring and provided a link between lesson development, delivery strategies and assessment strategies by honing in on the needs of learners, learner styles and how these relate to positive assessment outcomes. The feedback from the team leaders was very encouraging and many have indicated that they’d like to undertake full Level 1 training in this model.

Future Directions:
The development of an assessor network website has allowed all participants to be involved and share materials and experiences. The ‘CAVN’ website will remain to be used as a development platform where draft documents can be shared and commented on. The long-term aim, with the assistance of our Curriculum Centre, will be to house the validated documents on their Gateway site, so all NSW construction teachers can access the developed materials.

NAVAL and CAVN: TWO STORIES OF ASSESSMENT VALIDATION

This is an overview of the seminar presented at the ICVET Conference.

INTRODUCTION:

Teaching and Learning Managers, Jo Fuller (Community Services) and Rob Young (Construction) were responsible for managing Assessment Validation for their respective Faculties within the Institute. Jo created a group called NAVAL (Network of Assessment Validation Assessor Leaders) this group developed Learning Resources and easy to use processes to support teachers. Rob, from a Construction background was successful in obtaining Reframing the Future funding and worked with a state wide group to develop tools for the release of new training packages. This workshop is about the journey that took place, detailing the challenges and successes of the two processes.

Conclusion
Developing and maintaining an effective assessment validation process has many challenges. Our stories highlight these challenges but they equally highlight there can be many successes. This presentation is about discussion it is about thinking outside the square and of working closely with those willing to converse and exchange ideas on ways to improve and do things in a different way. In no way do we claim that our methods for assessment validation are the way that everyone should follow. What we do suggest is to think creatively on how assessment validation can work for you and your networks.