10/04/2024
We’re excited to announce that the 7th Annual FLIP+ Event is set to take place in the vibrant city of Lisbon, Portugal.
This year, we delve into the theme “The Role of AI in National and Classroom Assessments”, exploring the impact of artificial intelligence and digital technology on education. This annual gathering promises rich insights, discussions, and networking opportunities for all attendees.
These will include e-assessment experiences shared by member institutions and FLIP+ working groups as well as interactive sessions covering topics from the accuracy of AI in marking and assessment design to the revolutionary potential of AI in education. Delve into everything from generative AI applications and speech-to-text integrations to the dual roles of AI in enhancing assessment integrity and anticipating future educational trends. An agenda is attached. Final details will be unveiled in May 2024, ensuring a comprehensive and engaging program for all participants.
Secure your spot at this pivotal gathering by registering here.
We look forward to welcoming you to Lisbon for what promises to be an enriching and transformative experience.
21/02/2024
We are thrilled to announce an exclusive, all-day in-person workshop designed to rejuvenate and redefine the FLIP+ Working Groups. This workshop is an open invitation to all FLIP+ members keen on shaping the future of digital assessments and contributing to the International Item Library (IIL).
Event Details: •
When: Friday, 15th March 2024 •
Where: DEPP, Paris •
Time: All day from 08:30 am (including pre-workshop dinner on the 14th of March)
Why You Should Attend: Following the insightful discussions from our recent FLIP+ webinar for our members on 2nd February, the association is carrying the momentum forward to an in-person workshop. This is not just another meeting; it’s a pivotal moment for collaborative innovation within our association. We will take the opportunity to delve deep into how each of us can contribute to the FLIP+ Working Groups, exploring new content development for the IIL, and discussing the vital role your unique insights and experiences play in our collective success.
The workshop promises to be a day full of engaging discussions, creative brainstorming, and strategic planning. It’s an unparalleled opportunity for you to voice your ideas, share your expertise, and collaborate with fellow FLIP+ members who are equally passionate about the evolution of e-assessment.
We understand the dedication it takes to balance commitments, and we appreciate every level of involvement. Whether you’ve been with us from the start or are looking to join the journey, your participation is invaluable. Together, we can ensure the FLIP+ Working Groups continue to be a beacon of innovation and progress in the field of digital assessment.
Join Us: Don’t miss out on this opportunity to contribute to a significant chapter in FLIP+’s story.
REGISTER HERE to secure your spot and be part of shaping the future of digital assessments.
Your ideas, expertise, and enthusiasm are what make FLIP+ a pioneering force. Let us continue to make a difference, together.
29/01/2024
Since their inception, our Working Groups have been instrumental in shaping the content and technological solutions for Maths, English, IT Development and more. However, the recent years have brought new challenges and priorities, impacting our ability to contribute regularly.
Now, as we look forward to developing fresh content for the International Item Library (IIL), we are eager to redefine our Working Groups and inject new energy into this collaborative journey. And for this, we need your valuable insights and creativity!
Prior to our in-person meeting in Paris on 15th March, we’re hosting an informal yet informative webinar on Friday, 2nd February to discuss the mission of our Working Groups, the vital role of your contributions, and how this all ties into the growth of the IIL. This is a perfect chance to understand the impact of your participation and envision how we can collectively advance our goals.
Can’t attend live? You should still register, as we will be sending out the recording after the webinar to all registrants.
4/11/2023
FLIP+ invites you to an engaging and informative webinar with Irene Custodio, a seasoned expert in the field of high-stakes digital assessments and her colleague Ellen Barrow, both from Pearson – UK. This webinar will shed light on the design and development of inclusive digital assessments, illustrating not only the complexities imbued in achieving inclusive onscreen assessments but encouraging attendees to critically evaluate our understanding of what opportunities digital assessments can bring and what equitable access for all learners might look like.
Date: Friday, 26th January 2024 at 16:00 CET.
The presenters will discuss:
Can’t attend live? You should still register, as we will be sending out the recording after the webinar to all registrants. REGISTER HERE
Irene Custodio is the Digital Assessment Design Lead at Pearson working in the development of valid and fit-for-purpose digital assessment within a high-stakes context, with a focus on user-centred design principles, test-taker experience and accessibility. She has worked for over 15 years in the field of educational research, assessment design and the development of high-stakes digital assessments. She is also experienced in the delivery of large-scale international onscreen assessment.
4/11/2023
In Wales, all children aged 7-14 are required to take national on-screen adaptive assessments in reading, procedural numeracy and numerical reasoning each year. These award-winning calibrated assessments provide a comparison with national standards, but their purpose is formative, delivering trusted data to inform teaching, improve learning and support the next steps for all students.This webinar is a must-attend for anyone interested in the evolving landscape of educational assessment. Gain a comprehensive understanding of how personalized assessments are shaping the future of education in Wales.
Join us on Friday, 19th January 2024 at 16:00 CET for an enlightening FLIP+ webinar presented by Ben Rockliffe from AlphaPlus – UK.
Key Topics:
Can’t attend live? You should still register, as we will be sending out the recording after the webinar to all registrants. REGISTER HERE
Ben Rockliffe is the Deputy director of Assessment at AlphaPlus, UK. In his role, he provides technical input into the design and development of assessment services for AlphaPlus customers. His current significant projects include onscreen adaptive assessments for the Welsh and Scottish Governments.
16/11/2023
Apparently, a chatbot ‘passed‘ the bar exam, which assesses the knowledge and skills every lawyer should have. Does that qualify the chatbot to be a lawyer? Certainly not. Does this result give us any understanding about the chatbot‘s real (not superficial) knowledge and skills? Probably not.
Cesare who is the Head of Research and Development at AQA, one of the leading assessment organisations in the UK, will look at the assessment of AI from a traditional assessment perspective. With the advent of general-purpose large language models, concepts are increasingly borrowed from educational assessment approaches. Machine performance is likened to people performance, often with exaggerated claims about AI achieving superhuman performance. However, machine performance evaluation still mostly relies on tasks that generalise poorly and aren’t underpinned by any clear ‘construct‘, even though researchers interpret test results as if they were indicative of some general ‘latent’ skill. Even in leading companies, formal assessment of machine capabilities is carried out by non-assessment experts.
Moreover, assessments designed for people are liberally modified and adapted to suit machine administration. In summary, the validity of the claims around AI capabilities often relies on tenuous evidence. He argues, with others, that it is time to blur the lines between machine performance evaluation and people assessment (including educational measurement and psychometrics). We must adopt an interdisciplinary approach that places validity back at the core of all assessments – whether we are testing people or machines, or people assisted by machines.
Don’t miss this insightful session that promises to broaden your understanding of the evolving relationship between AI and educational assessment.
Cesare Aloisi is the Head of Research and Development at AQA, one of the leading assessment organisations in the UK. His primary research interest is around validity and the foundations of educational measurement. His most recent work includes the impact of AI on public trust in assessment, AI regulation and explainability, and the validity of AI assessment.
27/10/2023
We invite you to join our upcoming FLIP+ webinar, presented in collaboration with CAEd, the research and technological development centre of the Federal University of Juiz de Fora (UFJF) in Brazil.
The session will be led by Jairo Francisco de Sousa and Carlos Palacios Carvalho da Cunha e Melo from CAEd. They will discuss the application of oral reading fluency tests to 2nd-year primary school pupils across multiple Brazilian states. Moreover, the webinar will introduce the audience to the technology that enables automatic reading correction using artificial intelligence, a significant breakthrough that promises greater flexibility and lower costs in assessment practices.
CAEd is a research and technological development centre of the Federal University of Juiz de Fora (UFJF), Brazil, that has been working in the fields of assessment and public education policy since 2000. With the aim of contributing to the improvement of public education in the various states of Brazil, CAEd focuses on the development of tools and systems for the evaluation and management of basic education.
Don’t miss this chance to gain valuable knowledge.
Can’t attend live? You should still register, as we will be sending out the recording after the webinar to all registrants.
04/05/2023
We are excited to remind you about the upcoming 6th FLIP+ Annual Event, taking place in Cambridge, UK, on the 8th and 9th of June 2023. Don’t miss this incredible opportunity to engage with e-assessment professionals and enthusiasts in our FLIP+ community at:
Cambridge Assessment – The Triangle Building, Shaftesbury Road, Cambridge, CB2 8EA, UK
In case you missed our previous announcement, the event will feature various interactive sessions, including sharing member experiences, workshops on digital country transformations, inclusivity and special educational needs, AI and computational methods, as well as innovation and assessment futures.
The 6th FLIP+ Annual Event is a fantastic community space for knowledge sharing, networking, and collaboration within the e-assessment community. We encourage you to secure your spot by registering now, and the latest, by 23rd May 2023.
If you require an invitation letter to apply for entry into the UK, please feel free to write to us at info@flip-plus.org.
We look forward to seeing you in Cambridge and hosting an unforgettable event. Register today and stay tuned for more updates!
25/04/2023
Join us for another valuable FLIP+ webinar presented by INVALSI, Italy’s National Institute for the Evaluation of the Education and Training System. The webinar will focus on using Robotic Process Automation (RPA) to streamline the quality assessment process for INVALSI’s compulsory computer-based standardized tests for grades 8, 10, and 13.
In this session, we will examine how Selenium, an open-source automation testing tool, can be utilized with Python to simulate human interaction with online TAO tests. This approach minimizes the time and effort needed for repetitive tasks, such as selecting answers, inputting text, and matching options in various question types.
By the end of the webinar, attendees will gain valuable insights into good practices for automating monotonous and time-consuming quality assessment activities, helping to reduce human errors in low value-added tasks. The session will also include a live demonstration of an automated data entry process.
Don’t miss this chance to learn how RPA can enhance quality assessment processes.
Can’t attend live? You should still register, as we will be sending out the recording after the webinar to all registrants.
27/03/2023
Are you ready to learn about another experience of implementing education assessment? An insightful FLIP+ webinar exploring the latest developments in Spain’s new competence-based curriculum and the digitalization of national assessment frameworks and tests.
In this engaging webinar, you’ll learn about:
Don’t miss this opportunity to gain valuable insights into Spain’s experiences in implementing digital assessment initiatives in its evolving education landscape.
The webinar will last an hour, with some time for questions and sharing at the end of the presentation.
Can’t attend live? You should still register, as we will send out the recording after the webinar to all registrants.
20/03/2023
We are thrilled to announce that the 6th FLIP+ Annual Event will take place in person on 8-9th June 2023 in the historic city of Cambridge, UK. The event will be graciously hosted by Cambridge University Press and Assessment.
This two-day gathering provides a unique platform for FLIP+ Association members to share their e-assessment experiences, learn from others, and engage in meaningful discussions on the latest developments in the field. As every year, our event will cover institutional experiences on e-assessment developments, updates on FLIP+ working group activities, and an opportunity to explore various technology tools and solutions, including information on the international ITEM LIBRARY.
Call for Presentations and Workshops
We invite all FLIP+ members to contribute to the event by presenting their e-assessment experiences or proposing a workshop on aspects of digital assessment. Here’s what you can expect:
Day 1: E-assessment Experiences Presentations Reserve a 20-minute slot to present your institutional experiences related to e-assessment developments.
Day 2: Digital Assessment Workshops Propose a workshop that delves into various aspects of digital assessment, offering attendees a more in-depth understanding and hands-on experience.
Submit Your Expression of Interest
If you are interested in contributing to the event, please submit your expression of interest by 31st March 2023, to info@flip-plus.org. Our team will review your submission and get back to you with further details.
Don’t miss this opportunity to connect with fellow FLIP+ Association members, showcase your e-assessment expertise, and contribute to the advancement of e-assessment practices. We look forward to seeing you in Cambridge!
06/03/2023
The Norwegian Directorate for Education and Training (NDET) has, since 2021, completed two mixed-method pilot projects shedding light on accommodations for summative assessment, digital affordances, and exam construct validity.
In this webinar, Kevin Steinman will describe exciting pilot projects in upper secondary public and special needs schools. These projects have resulted in an improved understanding of the assessment experience in subject English for students who are hard of hearing, as well as for those who use sign language.
Given Norway’s separate curriculum in English for students who use sign language, the examination in English has been modified, with items designed to measure receptive competency via listening being replaced with additional reading items. This raises exciting questions to be discussed in this webinar about construct validity, for example, whether the additional reading items “test the (reading) endurance” of students rather than fairly assessing their receptive English competence. Similarly, the use of video with subtitles for hard-of-hearing students following the common English curriculum demonstrates promising new possibilities for digital assessment and universal design.
08/02/2023
In the context of a doctoral research project, a team at the University of Luxembourg will share their research approach to derive a set of heuristics to design child-friendly e-assessments for educational purposes. Using literature review, the empirical work of practitioners in the field of Human-Computer Interaction as well as expert knowledge the team will present the derived set of usability principles specific to e-assessments applications for children between 6 and 8 years. Along with some examples, we illustrate how the principles contribute to better interface design. Besides, we learn how experts can use these principles as an underlying evaluation method, a cost and time-efficient method to uncover problematic elements in an interface and assess its compliance with recognized usability principles. The overall takeaway from this webinar is to provide the audience with the necessary guidance to identify usability problems in their e-assessment application and to introduce expert evaluation as a method for applying the heuristics.
Presenter: Florence Lehnert
16/01/2023
Happy New Year 2023 and may this year bring prosperity, well-being and good health to you and your loved ones.
We are excited to present another FLIP+ webinar series starting with the sharing of experiences related to digital assessments for 6- and 8-year-old students from Norway (27th January), and developing and improving accessibility in student assessments in France (10th February). Details are provided below. All webinars will start at 16:00 CET and last an hour. At the end of the presentations, there will be some time for questions and sharing.
Can’t attend live? You should still register as we will send out the recording after the webinar to all registrants.
27th January 2023 at 16:00 CET: NORWAY
Digital assessments in reading and numeracy for 6- and 8-year-old students – two experiences from Norway
In Norway, the national government provides primary schools with assessment tools to detect students at risk of lagging in basic skills. These consisted of traditional paper-based assessments administered to students with teacher support, which have transited to digital format since 2022.
In this webinar, you will learn about two experiences: the issues encountered in administering these assessments in reading and numeracy to 1st and 3rd graders.
Adaptive, national screening test of reading – possibilities and challenges
The first presentation will give insights into the research and development process of new national adaptive reading screening tests for 1st and 3rd-grade students (aged 6 and 8) in Norway. How were the tests designed to foster the students’ continued desire to read and learn? Did the tests mirror good classroom instruction? This presentation will address these aspects and will show how teachers’ interpretation and their use of test scores have guided the development process from the start. Reactions from stakeholders will be included in the presentation.
Presenter: Bente R. Walgermo is an associate professor at the National Centre for Reading Education and Research at the University of Stavanger and specializes in reading and writing motivation and assessment. She runs several research projects on assessment and is the project leader of the new Norwegian screening test of reading for 1st and 3rd Grade.
Advantages and issues related to developing and administering digital numeracy assessments
The second presentation will discuss the advantages and issues related to developing and administering digital numeracy assessments, and how they are adapted for 1st and 3rd-grade students. What adaptations were made for students who do not yet read well enough to understand written task instructions? Did the assessment tasks enable students to work in much the same way they do in classroom activities during mathematics instruction? What is the impact on the listening ability, concentration, and motor skills of these very young students?
Using data from cognitive labs, pilot studies and the first test administration, this presentation will discuss how students are affected by test design, and how valid and reliable digital numeracy assessments can be developed for young students.
Presenter: Guri A. Nortvedt is a professor at the Department of Teacher Education and School Research at the University of Oslo. Her research focuses on assessment, mathematics teaching and assessment in diverse classrooms and mathematical competence. Nortvedt is the project leader of the Norwegian numeracy mapping test for grade 1 and 3 students.
10th February 2023 at 16:00 CET: FRANCE
Adapting student assessments to develop and improve test accessibility in France
Since 2020, the DEPP within the Ministry of Education in France has dedicated much effort to exploring and improving the access characteristics of digital assessments for all students undertaking the national assessment program. What is accessibility related to assessments? How can we improve access to digital assessments for all students? Is it easier to make a paper-based assessment accessible than a digital one?
Using examples of test adaptations, this webinar will discuss the above questions based on the work carried out by the DEPP team, which focused on three levels:
– the training of the teams that design the assessments;
– the development of digital accessibility based on the W3C international web standard (tests of the digital platform, creation of adaptations such as text alternatives to images for blind students, creation of video supports in sign language, simplification of instructions, etc. )
– the communication with schools to better support students with special educational needs during assessments.
A similar approach is used for paper-based assessments, allowing us to understand better the advantages and limitations of using digital technology for accessibility.
Presenter: Elodie Persem is the Head of the Accessibility, Innovation and Research Pole at the DEPP, Ministry of National Education, Youth and Sports. She is a specialized teacher and develops the accessibility of assessments carried out by the DEPP and develops partnerships to implement research on 21st-century skills. She was a former coordinator of inclusive school training at the Academy of Versailles.
03/05/2022
We are only five weeks away from meeting members of our FLIP+ community for our two-day Annual Event in Paris in June.
Many of you have already registered for the event and expressed interest in sharing your e-assessment experiences with us on Day 1. The highlight of Day 2 is the participatory digital assessment workshop where attendees will dive into case studies on digital assessment, digital tools, the uses of log data, and inclusion. The case studies will be presented by experts in assessment design and assessment research, with insights into item development. The workshop sessions are designed to stimulate discussion and sharing of experiences among the workshop participants.
Other sessions will also be organised to present the activities of FLIP+ working groups as well as to discuss the setting up of the International Item Library Project.
You may find the tentative agenda below, which you will understand is still subject to minor changes, as we await more confirmation details.
The description of the Day 1 presentations will be fine-tuned by the end of May. Please refer to pages 2 – 5 for more details about Day 2 sessions.
Participants will also have ample opportunities to share informally during the welcome reception on 8th June as well as during the exciting social event to discover Paris on the evening of 9th June.
We look forward to meeting as many of you in Paris!
04/04/2022
Here is another exciting FLIP+ webinar from one of our community institutional members AQA Education from the UK. In this presentation, participants will learn about the notion of ‘preparedness’ in the context of digital assessments.
Do young people’s digital immersion in their everyday lives and their increased exposure to remote delivery of the curriculum during Covid-19, imply they are sufficiently prepared for digital assessments? The AQA Education team conducted a trial with 15-16-year-olds in England to find out how increased exposure to digital teaching and learning shaped the response to online screen assessment of this cohort of students.
In this one-hour webinar, the AQA Education team examines the challenges involved in the transition from paper-based to onscreen assessment, and presents findings related to the preparation of students required for this transition to be successful.
29/03/2022
For all members of FLIP+, please note that our General Assembly 2022 will again be held virtually, on Wed 13th April 2022 from 12:00-14:00 CET. An invitation, the meeting agenda and documents have been sent by email to all members.
28/03/2022
It has been two long years since the FLIP+ community members were able to meet up in person at our events. We are therefore excited to announce that the association will be organizing its 5th annual event on the 9th – 10th June 2022, in Paris, kindly hosted by the DEPP (French Ministry of Education).
A draft agenda for the two days is shown below and details will follow as we reach closer to June. However, the intention is to dedicate the first day to sharing member experiences and FLIP+ activities in a plenary session. On the second day, parallel sessions will be organized which will cover an introduction to a process data and digital workshop (by Bryan Maddox, the Executive Director of Assessment MicroAnalytics), a discussion / demo of ongoing FLIP+ IT development & tools, a session dedicated to the setting up of the future International Item Library.
There will be also a welcome reception on Wednesday evening (8th June) as well as an exciting social event to discover Paris, on Thursday evening (9th June).
Participation in the event will be free for FLIP+ members. The registration fee for non-members is €200.
You will receive the access details upon registration and a detailed agenda will be sent by the end of April. For further information please do not hesitate to contact us at info@flip-plus.org.
18/03/2022
We are excited to present to you another FLIP+ webinar, this time on how game-based elements are used to improve the way students are assessed in education. This will be presented to us by the CITO Foundation team from the Netherlands.
Elements from games are increasingly used in educational contexts. For example, by working with levels, badges, or even complete virtual environments in which avatars can move freely, students interact with educational content in a game-based way. The CITO Foundation team is researching the implementation of game-based assessment examining how can game-based elements contribute to a more positive assessment experience without compromising assessment quality. How are these elements used to assess information literacy and the associated problem-solving skills of candidates? The research team built a prototype game specifically targeted at public safety and security officers, who are concerned with administrative law enforcement in the Netherlands. In the game, officers are tasked with evaluating, organizing and combining information from multiple sources to come up with a surveillance plan for their shift.
In this one-hour webinar on the 1st of April at 16:00 CET, the CITO team will demonstrate this game-based environment and discuss the opportunities and risks of future game-based assessment.
21/12/2021
As the new year begins, FLIP+ will kick off its first three webinars for 2022 on January 14. As per our motto, our objective is to share with participants, the ongoing developments in e-assessment within our FLIP+ e-assessment community.
We are pleased therefore to announce the first three webinars to be conducted on Zoom, which will cover France and Italy’s experiences during the four years as they gradually transited to e-assessment, Denmark’s project on an online open-source coding system and OECD’s work on developing a digital, formative assessment platform.
All webinars will start at 16:00 CET and last an hour. At the end of the presentations, there will be some time for questions and sharing.
Jan. 14 – Transition to e-assessment – lessons and perspectives after four years, by DEPP, France and INVALSI, Italy
In this webinar, Roberto Ricci (INVALSI, Italy) and Thierry Rocher (DEPP, France) will share and discuss the opportunities and lessons learnt regarding the implementation of national very-large-scale digital assessment programs, in their respective countries.
In Italy as in France, the transition to e-assessment started off about four/five years ago. The development of these programs faced many implementation challenges, including technological ones, but it also revealed important opportunities.
The webinar will provide a space to discuss the issues at stake in such ambitious programs, both from technological and scientific points of view, as well as from pedagogical and policy perspectives.
Jan. 21 – A tour of an online open-source coding system, by Aarhus University, Denmark
In this webinar, Jeppe Bundsgaard from the Danish School of Education, Aarhus University, will present and give a tour of a new free and open-source online coding system that he has developed. OpenCoding can be used both for human coding and post-test scripted auto-coding. It includes a backend for import of data and export of results, creation of coding settings, management of coders, tasks, coding and flagging, an overview of progress and double coding etc. And it includes a frontend with a potentially interactive view of test takers responses. The system is available at GitHub: https://github.com/openPCI/openCoding
Feb. 4 – Development of a digital, formative assessment platform, by the Organisation for Economic Co-operation and Development, OECD
In this webinar, Mario Piacentini and Emma Linsenmayer from the Directorate for Education and Skills from the OECD will present their work on the Platform for Innovative Learning Assessments (PILA). PILA is a fully open-source solution for formative assessments developed in collaboration with the NGO Know Learning. It provides interactive, customisable assessment experiences and actionable data on what students know and how they learn.
The OECD team will provide an overview of the first PILA module on computational problem solving, as well as introduce the second module in development that focuses on modelling. A demonstration of how teachers can define sequences of tasks and customise individual tasks on the platform will follow. Finally, the team will present the first version of their live-monitoring and performance dashboards, and describe how results from a teacher usability study in Ireland helped to improve the design of the tools.
Information on planned research activities will be shared for feedback and possible collaboration within the FLIP+ network.
18/10/2021
At the end of September, some members of the FLIP+ team participated in the “Beyond Results 2021” two-day workshop, co-hosted by IEA (International Association for the Evaluation of Educational Achievement), DIPF (Leibniz Institute for Research and Information in Education) and ZIB (Centre for International Student Assessment). The main focus of the discussion was the design of cognitive assessments providing log data and process indicators, as well as ways to derive and communicate meaningful insights from them.
Following the workshop, the FLIP+ e-assessment community has taken the responsibility of launching a working group, with the support of DIPF and IEA, to collaborate around these different aspects related to log data and process indicators (research, methods, item design, standards…).
To this effect, we would welcome FLIP+ members and partners interested in participating in this working group, to sign up, by reaching out to us at process-data@flip-plus.org.
The focus of the working group is on theory-driven use and research on process data in educational assessments of and for learning. Emphasis will be given to log data but other sources of process data can be considered as well (e.g., eye tracking, video…). We are aiming for a continuous exchange between different disciplines regarding process data in order to share experience and stimulate collaborative projects.
We look forward to your participation and to what we can create together using e-assessment data. To join the working group, or should you have any clarification questions, please send a mail to process-data@flip-plus.org.
01/06/2021
Have you already registered for our 4th FLIP+ annual event taking place next week on 10th-11thJune, from 12:00 – 14:00 CET?
Join us to learn about the opportunities and challenges for IEA studies (TIMSS, PIRLS, ICILS, ICCS), as institutions experience a transition from paper-based to computer-based assessments (CBA): how are they dealing with item authoring, translation, translation verification, national adaptation and test delivery? What about institutions that choose to administer both paper-based and CBA? How do these assessments change the reporting of the results of the assessments? IEA will share the perspective of how the international study centres coordinate international large-scale assessments.
The DIPF | Leibniz Institute for Research and Information in Education will then briefly present selected infrastructure work and research activities related to computer-based assessments. Reporting from a high-stakes assessment based on TAO, DIPF will share with participants, the development of the CBA Item Builder, an authoring tool for complex items that can be used to implement or extend assessments. Using a mixed-mode assessment as an illustration, you will learn how personal interviews may be replaced by online web-based assessments. In addition, DIPF will share tools for the analyses of log file data, tools and microservices for assessment-related data processing such as NLP tasks and test assembly problems. DIPF will outline ongoing and future work for changing from assessments of learning to assessments for learning.
Other institutions will also briefly present their e-assessment work, including adaptations related to changes due to the pandemic. These include the National Institute for Evaluation (INVALSI) of Italy, the Centre for Educational Assessment (CAEd) attached to the Federal University of Juiz de Fora, Brazil, the National Agency for Education (Skolverket) in Sweden, the National Assessment & Exam Centre (NAEC) of Georgia, the CITO Foundation in the Netherlands and the DEPP from the Ministry of Education in France.
On both days, the FLIP+ team will also provide participants with an update on the ongoing work related to the Item Library, in particular a first design of the minimum viable product created by the CITO team, based on the technical specifications provided by the FLIP+ Item Library team.
19/04/2021
Over a month ago, we presented to you observational insights from recent user experience (UX) studies. Our next webinar coming up this Friday, 23rd April at 16:00 CET, will be addressing this same topic but within the context of findings from a Ph. D. thesis. A team from the University of Luxembourg will present factors influencing pupils’ user experience with a Technology-Based Assessment (TBA) application.
By applying a mixed-methods approach (i.e. user observation, self-developed questionnaire), the team looked into the interplay of different factors on a contextual and personal level, affecting the UX with a TBA application. The findings are based on two case studies in France and Luxembourg and present shortcomings of the current TBA design. We also shed light on measuring childrens’ UX by, for example, discussing weaknesses of the Smileyometer, a prominent scale design for conducting UX research with children.
This research was carried out within the Human-Computer Interaction Research group of the University of Luxembourg and in collaboration with the Luxembourg Centre for Educational Testing (LUCET) and the Department of Student Assessment (DEPP) within French Ministry of Education.
13/04/2021
Our fourth annual event will be held on 10th-11th June 2021, from 12:00 – 14:00 CET on both days.
The theme would centre on countries’ use of technology in assessment in response to the COVID-19 pandemic. Sub-topics to include are high-school assessment, learning losses due to school closures and lockdown; sharing of experiences (templates to monitor changes in what is done in schools / education), digital tools used and content; how can FLIP+ Item Library be used to help in these circumstances.
The agenda on the first day will include country experiences on main e-assessment developments and adaptation due to the pandemic. An update on ongoing work will be shared on the second day, related to the Item Library Content (Maths and English) with, if necessary, parallel breakout sessions according to themes to be defined ahead. A further update will be provided on the platform development (update, standards, technology…). Further details to come soon.
26/03/2021
For all members of FLIP+, please note that as our General Assembly cannot again be held in person this year, the association will organize it virtually on Thu 15th April 2021 from 13:00-14:00 CET. An invitation and agenda will be sent to all members by the beginning of next week.
05/03/2021
Mar 12th – User experience in computer-based assessments, by Dr. Bryan Maddox, Assessment MicroAnalytics™
User Experience (UX) studies are an essential part of good test development. In computer-based assessments, they are all the more important because of the big investments that are necessary in item and platform design, and because of the potential for dimensions of the digital mode of assessment to unexpectedly impact on test performance and validity.
In this FLIP+ webinar, Bryan Maddox will share observational insights from recent UX studies in computer-based assessment. He will describe how UX data is collected, the diverse ways that test takers engage with, and navigate compute-based test items, their cognitive and affective strategies, and their responses to success and failure. This will include evidence from recent UX studies involving data from eye tracking, gesture, facial recognition, and think aloud protocols.
Bryan will highlight some of the strengths and hazards of computer-based assessment design and share tips on how UX studies can strengthen student engagement, support construct validation, and enhance the use of process data.
Mar 19th – The taxonomy and framework for digital assessment items – International Baccalaureate and CITO
In this webinar, Rebecca Hamer, from the International Baccalaureate (IB) and Caroline Jongkamp, from CITO, will present their work related to a framework for categorising digital assessments inspired by Kathleen Scalise’s 2009 taxonomy. The framework builds on several workshops and includes many innovative digital item types.
Participants will be introduced to the framework and some (preliminary) results will be presented. The second phase to validate the framework will be described, with an invitation to all for further contributions to this work. In addition, first thoughts will be shared on whether the framework would help systematise the meta-tagging in the FLIP+ Item Library.
29/01/2021
Jan 29th – Innovations in Digital Assessment, by CITO, Netherlands
In this webinar, the CITO team will show you examples of innovative assessments that they developed for primary and secondary education in the Netherlands.
The constructs of the assessments vary widely (speaking, reading, digital literacy, empathetic ability) but what they have in common is the way they incorporate playfulness and direct measurement into the assessment.
Above all, the discussion will focus on the consequences of these developments in the context of assessment validity and how it affects the (technical functionalities of) the testing environment.
Feb 5th – A new diagnostic reading literacy test and online scoring and reporting tool, by Educational Research Centre (ERC), Ireland
This webinar will be presented by the Educational Research Centre (ERC) and is a collaboration between the ERC and the National Educational Psychological Service (NEPS) in Ireland.
The aim is to share their work in providing a standardised diagnostic test of literacy to schools. This paper-based test, the PPAD-E (Post-Primary Assessment and Diagnosis – English) was developed and piloted by NEPS and standardised by the ERC using a nationally representative sample of Grade 7 students in Oct-Nov 2019. The complete test package includes five subtests: word reading, spelling, reading speed, reading comprehension and writing samples.
The webinar will provide details on why and how the test was developed and standardised. It will also cover the advantages and benefits of the test, its structure and content as well as the training and support materials for teachers. Participants will also be able to view a demo of the scoring and reporting tool (features, functionality, security).
Feb 26th – Assessment of social-emotional skills, by Instituto Ayrton Senna, Brazil
In this webinar, participants will have the privilege of learning about a framework to understand and measure social-emotional skills, developed by the Institute of Ayrton Senna in Brazil. Social-emotional skills are individual characteristics that are manifested in consistent patterns of thoughts, feelings and behaviours. They influence socioeconomic outcomes throughout someone’s life and they are considered pivotal to life and work adaption, for example. This webinar will cover how the institute’s projects tackle the idiosyncrasies of each targeted population.
Can’t attend live? We still invite you to sign up, as a recording of the webinars will be made available to all those who have registered.
14/01/2021
We have had a great turnout for our webinars that were organised in December 2020 and last week. A big shout out to all participants who joined and actively engaged in the discussions.
The upcoming webinars include an update of the development work undertaken by the FLIP+ team with regards to the FLIP+ Maths Item Library, a showcase of innovations in digital assessment by CITO, Netherlands and a new diagnostic reading literacy test and online scoring and reporting tool, to be presented by the Educational Research Centre in Dublin, Ireland.
On December 11th, participants were able to learn about the INVALSI formative testing project, which provides schools with tools for diagnostic (initial) and ongoing assessment of student achievement on the expected proficiency and content goals for the previous school grade. The assessment also aims to provide concrete and operational help to schools, teachers, students and their families to reduce the disadvantages that have been exacerbated as a result of remote teaching. This initiative comes in the light of the pandemic affecting schools, and INVALSI intends to make use of the vast accumulated knowledge gained over more than ten years of census-based standardised tests, in grades 2, 5, 8, 10 and 13.
On December 18th, Denmark presented a joint assessment tool based on the principles of design thinking, which is an approach to tackling problems and thereby gaining so-called 21st century skills. During the webinar, interactive items were shown live and participants were able to learn about examples of tasks and how human and automatic coding are performed. The joint assessment tool presented by the Danish School of Education, Aarhus University, aims to find out to what extent the interventions impact students’ proficiency in design thinking.
Last Friday (January 8th), the FLIP Development team presented its work on e-assessment tools that are being developed in different member institutions. Again, interactive items and tools for 21st century skills, maths and science as well as literacy and numeracy were presented.
Participants to the webinars have expressed great satisfaction with all the content so far and the FLIP+ team is very grateful for all the collaboration and sharing that is taking place within the association.
27/11/2020
During our 3rd FLIP+ annual event 2020, many participants expressed interest in learning about the ongoing developments in e-assessment within our FLIP+ e-assessment community. In response, our first three webinars on Zoom will cover the experiences of Italy, Denmark as well as the items and tools being developed by our FLIP+ Development team. The webinar dates are as follows:
The webinars will last an hour. At the end of the presentations, there will be some time for questions and sharing.
Can’t attend live? You should still register as we will be sending out the recording after the webinar to all registrants.
15/06/2020
During its third international FLIP+ online event last week, the FLIP+ association gathered over 100 participants from 50 different institutions working in the field of educational assessment. Participants joined in from 24 countries as well as from IEA and OECD.
Under the FLIP+ motto of “sharing”, participants were able to learn about the e-assessment experiences of countries like Denmark, Georgia, Lithuania and Norway and gain insights about developments in six other FLIP+ member institutions. Work undertaken this year related to the FLIP+ Item Library was also presented. It should be recalled that the first ideas to set up the item library were initially discussed during the 2019 annual event in Rome. A year later, participants greatly welcomed the live demo of the initial prototype of the Item Library platform which showed where the items would be stored, how they could be accessed, reviewed, worked upon and shared.
Great feedback was received through the active online discussions on both days and through a mini satisfaction survey about the event. Links to test portals, national item resources and articles were shared, and institutions reflected on how to further contribute to the future work of FLIP+. The next milestone will be the definition of FLIP+’s formal collaboration model for item sharing. Following the suggestions of the event participants, FLIP+ will also be organising a series of webinars from September 2020 onwards. This will include topics such as assessment of design thinking (21stC skills), digital tools related to complex maths items, speech recognition and marking, taxonomy of digital assessment items, user experience studies in e-assessment, digitalising exams, translation workflows, and in-house development of open-source proctoring solutions. Tune in to this space for more details.
The presentations and shared resources from the 2-day FLIP+ event are available here:
3rd FLIP Event Online 11-12 June 2020 DAY2 Item Library Platform Prototype Demo
03/06/2020
On the first day of our online event, attendees will be able to hear the e-assessment experiences of FLIP+ member countries. These include the assessment of design thinking in Denmark, piloting e-assessment of items and an online delivery system in Georgia, the Lithuanian experience of transiting to e-assessment of literacy and how Norway is digitalising its national exams. Read on for more details. Other member countries and institutions will also update us on the ongoing e-assessment activities this year. To join our event, please register by completing the registration form here.
Denmark: Assessment of Design Thinking
Two ongoing Danish research projects have constructed teaching units building on the principles of design thinking, which they propose as an approach to tackling problems and thereby gaining so-called 21st century skills. The joint assessment tool will serve to estimate an effect of the interventions on the students’ proficiency in design thinking.
The test measures four dimensions of Design Thinking: EMPATHY (The ability to understand the needs and perceptions of others – end users as well as peer collaborators), IDEATION (The ability to draw on past experience to come up with many, different ideas), MODELING (The ability to understand, construct and critique representations of ideas and information) and PROCESS MANAGEMENT (The ability to prioritize as well as determine temporal order, duration and simultaneity of tasks and activities in order to inform design decisions).
Five new PCI’s are developed: Brainstorm, Voxelcraft, Route, Room, and Gantt. The presentation will give examples of tasks and how human and automatic coding are performed.
Georgia: Assessment for Development
How do we build an assessment that guides educational decision-making on different levels? How can we use technologies to leverage the use of assessment data for the improvement of teaching and learning?
Assessment for Development is an e-assessment project in Georgia which aims to design and generate assessment data that guides the improvement of teaching and learning on different educational levels.
The assessment is fully online. Items are created and delivered to the students via the TAO testing platform. The results are collected and analysed through in-house built e-reporting system. E-reports provide elaborated analysis on the strength and challenges for each student, as well as teaching and learning trends on classroom and school levels.
The project was piloted in February 2020 with 6 000 students so as to trying out the items and online test-delivery system. The next pilot is planned to test the e-reporting module and its compliance with e-delivery and e-scoring modules.
Lithuania: Transition to e-Assessment: a tool to strengthen e-Literacy
Founded by the Ministry of Education, Science and Sport of the Republic of Lithuania, The National Agency for Education (NSA) is tasked with implementing education policies, ensuring the quality of education and monitoring, researching and implementing educational solutions.
The transition towards e-assessment is related to the national context of education and highlights the tradition of paper-based assessment, expectations and reactions of community. The experiences over the past two years and that of the recent lockdown in Lithuania confirmed that the chosen priority for e-assessment technologies was the right solution. However, it is still necessary to keep in mind that there are different approaches of stakeholders, the need to ensure balance between the novelty of technology and communication, strict planning and attention to details. Lithuania struggles not for the bigger numbers of e-assessment participants, but rather to create and implement the national digital strategy with professional and adaptive e-assessment system.
Norway: Digitalising exams in Norway
In this presentation we will show how Norway have digitalised exams since 2007 using a system based on the distribution of PDFs. From 2020 we will start truly digitalising exams using the TAO testing platform. Some of our perspectives in this transformation are outlined here.
How do we engage students and teachers as well as exam authors in developing new digital exams, to ensure the human perspective in digital transformation? We will point out some challenges of digitalisation of exams in different subjects. How do we construct exams that utilize the possibilities in our new digital platform and still maintain the traditions of the subjects and reflect the goals of the new curriculum?
To us, maintaining the qualities of our current exams and ensuring a quality increase on relevant areas such as psychometrics and analytic assessment is crucial and must be reflected in the design of the new framework for exam creation that we are developing.
20/05/2020
Is your institution currently transiting from paper-based to e-assessments, or are you well on the way in your journey to create large-scale e-assessments for students? Would you like to hear how other institutions in this field are building digital assessment tools? Would you like to learn how a group of e-assessment experts are currently creating a digital item library of mathematics items?
The FLIP+ e-assessment association is pleased to invite you to its 3rd international event on 11th – 12th June 2020. Due to the actual global circumstances, the event will be held online via Zoom from 11:45 to 14:30 CET on both days.
The presentations will include country experiences from Denmark, Georgia, Lithuania and Norway, FLIP+ member updates on ongoing developments related to e-assessment progress, the ongoing work carried out by members on both the FLIP+ Item Library and the technological developments related to the platform. The agenda is found here.
If you wish to participate in our online event, please register by completing the registration form. You will receive the access detail upon registration. For further information please do not hesitate to contact us at info@flip-plus.org.
For all members of FLIP+, please note that as our General Assembly cannot be held in person this year, the association has opted to organize it separately during a remote meeting on Thu 25th June 2020 from 13:00-14:00 CET. An invitation and agenda will be sent to all members by the beginning of June.
23/03/2020
Due to the global impact of the unfolding coronavirus (COVID-19) crisis, the FLIP+ association has decided to switch its third in-person international event to a remote meeting on 11th – 12th June 2020.
The virtual space will be used to present country experiences, progress on the work carried out by members on both the FLIP+ Item Library and the technological developments related to the platform.
A link for registering to the event, as well as a more detailed agenda, will be available soon.
11/02/2020
During the 2nd thematic FLIP+ workshop on the 7th February, 2020, held at INVALSI, Rome, participants from eight countries addressed the issues related to the construction of the item library. The goals were to identify members who would work on the public content and the technical implementation of the item library.
Following the presentation of some concrete examples from the Italian context, there was an in-depth discussion between participants related to:
Participants then teamed up in three subgroups and three parallel sessions were organised to discuss: the item library catalogue, the technical creation and implementation of the platform underlying the item library as well as the content itself. A collective summary record of each plenary session will be produced and shared with all participants.
20/12/2019
Following our kick-off item-library workshop at the end of October in Rome, a first 1-day working meeting will be organized on Friday 7th February, 2020 in Rome to begin concrete work on creating the content of the item library (IL).
This will include outlining the IL structure and defining both the conceptual framework underlying the IL as well as the structure of the IL catalogue. To start off the process, participants in this working group will focus on mathematics as this was considered as a transversal need for all FLIP+ members.
The intention is to present the first product of this working meeting during the 3rd international FLIP+ event to be held on 11th-12th June (venue to be announced soon).
Should you wish to participate in this working group on item library content or for more information, please write to us at info@flip-plus.org by 10th January 2020.
15/12/2019
The FLIP+ Steering Committee members met for a one-day meeting in EduPôle, Walferdange-Luxembourg on the 12th of December 2019 to discuss the organization of the planned work for creation of the item library, as well as the agenda of the 3rd international FLIP+ event in June 2020.
During this meeting, the FLIP+ Steering Committee elaborated on three different work packages which would cover targeted actions to be undertaken by three working groups, in order to build the item library. These refer to the item library content, the general organization of the item library and the technology underlying the item library. FLIP+ members will receive an invitation in the coming weeks to participate in these different working groups. Webinars will also be organized to current developments related to e-assessment in member countries.
29/10/2019
The FLIP+ e-association welcomed around 40 participants on Monday 28th October in Rome to its first workshop, with the aim of discussing the creation of its own digital item library which would be shared eventually by all association members. The participants expressed the need for members to share and pool all the knowledge and experience related to concepts, frameworks, format, technology, content, skills existing in their institutional context. This would serve as a basis to identify new interactive items to be built and at a later stage piloting new experiences being tried out in member countries. Discussions included the general principles related to the item library such as the collaborative development of items, the conditions of access and use of the items as well as the use of the items by different stakeholders. The various aspects to be considered for the creation and maintenance of the item library were also addressed, such as the concepts, approaches, frameworks underlying item bank systems and the technology requirements as well as item standards and formats. First ideas were put forward on how to organize the work related to the creation of the item library into different work packages. Participants also had the opportunity to learn about and discover an example of the Italian item repository.
19/09/2019
During the 2nd international FLIP+ event in Rome in June 2019, participants expressed a strong need to create a shared item library for the FLIP+ e-assessment community. To this effect, a one-day kick-off meeting will be organised on Monday 28th October in Rome. The objectives of this first workshop is to address the philosophy and functioning of the item library, the requirements of the underlying technology (definition of security, access, user roles, …), the rules and modalities of sharing, the usage of the items and the implications of this sharing. Following the results of the workshop, a follow-up working meeting will be organised to start the actual creation / sharing of the items.
In order to prepare for the workshop, a support document will be sent out to participants in October, outlining the position of FLIP+ with regards to the philosophy of the item library as described above. Participants are expected to contribute actively by reflecting ahead on aspects such as the contents their institution can potentially add to the library, the contents they expect to benefit from, the expertise or technical know-how they (or their institution) can offer to create the library and its content. The creation of such an item library is an ambitious endeavour which can only succeed through the association’s motto of “sharing”.
16/09/2019
As we announced before the summer holidays, the FLIP+ community is now ready to welcome new members. The annual membership fees are €1000 for an institutional member and €300 for an individual member. To join the association, simply write to the FLIP+ e-assessment secretariat at info@flip-plus.org, providing the following details:
The FLIP+ secretariat will send you an invoice and the fees can be transferred to the bank account of the FLIP+ e-assessment association. Any queries regarding the membership enrolment should be addressed by e-mail info@flip-plus.org to the association’s treasurer, Thierry ROCHER. Please note that the annual membership fee should be settled by the 31st December of the current year.
17/06/2019
The FLIP+ e-assessment community is proud to announce that it is now an officially-established non-profit association.
This is a great milestone for our initiative which grew from informal thoughts and first reflections in 2017, followed by small but bold concrete steps in the years after, and to the launch of our e-assessment association today. It is a formal start to a great adventure with all those who seek to share and collectively develop innovative digital solutions to traditional assessment design, instruments and test administration to improve assessment of and for learning.
Applications for membership will be open as at September 1st 2019.
The Association has three categories of members:
Stay tuned for more details on the statutes of the association and on how to join as a member.
10/06/2019
The second international FLIP+ event was held in Rome on 6th – 7th June 2019. Over 50 participants from educational assessment and technology institutions in 13 countries were present. Representatives from the IEA, OECD and UNESCO also attended the meeting.
The event offered the opportunity for participants to share their experiences related to e-assessment. The FLIP+ team presented updates on e-assessment activities in their countries as well as current and planned technological developments (such as item developments, offline solutions, e-assessment portal, functionalities, item banking, tools, data analytics).
Other than sharing knowledge and experiences, the FLIP+ team facilitated a session to explain the value-added of being part of the FLIP+ e-assessment community and the broad principles underlying the collaborative framework of FLIP+. Participants expressed a strong need for a shared item library for the FLIP+ e-assessment community. In line with this reflection, a separate FLIP+ workshop will be organized in Rome in autumn 2019 to address the modalities and principles underlying the FLIP+ item library. Following the brainstorming session in the first event in Paris in 2018 regarding the governance structure of FLIP+, a session was dedicated this year to presenting the statutes of the future FLIP+ association. In parallel, the FLIP+ developer community held separate conversations to share updates, demos and functionalities.
The presentations from the 2nd international FLIP+ event are available for download:
All participants actively participated in all the sessions and the FLIP+ team strongly believes that this sharing within our community can only contribute to strengthening the world of e-assessment in education. The FLIP+ team will organize its third international event on the 11th-12th June 2020. Watch this space for more detail and for news related to the upcoming FLIP+ activities.
27/03/2019
The FLIP coordinating team met for a one-day meeting on the 25th of March 2019 to draw up the legal aspects of the collaboration. In order to identify the initiative with the idea of openness and common shared interest in the field of electronic assessment, the partnership was renamed the FLIP+ e-assessment community.
During this meeting, the team also discussed the road map priorities related to technical developments and outlined strategies for sharing content. The agenda of the upcoming FLIP+ event in Rome in June 2019 was discussed in detail. Invitations to the event, a draft agenda and the instructions for registration will be sent out by mid-April.
06/03/2019
The second FLIP event will be held in Rome on the 6th – 7th of June 2019. More information and registration form will be available soon!
07/01/2019
A FLIP-developer community was set up in January 2019 in order to exchange information and share experience on technical aspects of the e-assessment, as well as to share progress on joint FLIP developments. Up to now, two joint developments – an offline e-assessment solution and an app solution (both based on TAO) – have been undertaken by three member countries: France, Portugal and Brazil.
07/12/2018
The FLIP coordinating team met in Lisbon on 5th-7th of December 2018 to further discuss the legal and technical aspects of the partnership and to plan the next FLIP event that will take place in Rome in June 2019.
The 1st FLIP event took place in Paris on 18th – 19th of June 2018. Altogether 55 participants from 17 different countries took part in this meeting. This wide interest indicated a promising start to a fruitful and long-lasting community.
This first FLIP event provided a great opportunity for rich inputs and exchanges between participants. Country reports from current FLIP members France, Luxembourg, Italy, Portugal and Brazil were presented.
This was followed by an introduction to an open standard ecosystem IMS and a demonstration of an open source assessment product TAO.
Multiple extended roadmap sessions were then held, addressing issues such as assessment platform functionalities, security and data protection, data analytics and reporting, item banking and technology enhanced items, etc.
A first brainstorming regarding the legal organization of the partnership and the setting-up of a technical community was done during the two community-building sessions. The intention is to draw up a more formal framework that would define FLIP’s governance structure and mode of functioning as well as detailing how costs would be shared.
Feedback from the participants of the event showed an overall satisfaction and interest towards the event and the initiative. All participants responding to the feedback survey found the content of the first FLIP event programme interesting and varied and the organization of the first FLIP event entirely satisfactory. Participants also found the first FLIP event to be professionally useful.
75% of the respondents noted that they were already involved in the FLIP partnership or willing to play an active role in it in the future.
The participants showed most interest towards the following developments: portal for reporting assessment results, (online) and interoperable marking system, and technology-enhanced items.
