Draft questions

As I wrote in Collecting data about engagement - quick questionnaires, quickconversations publics can be engaged at a range of  levels from

1. engagement  with a research project's (prototype) products
   to
2. engagement  with the research process that are generating and modifying  a research project's (prototype) products.

Juxtalearn will collect lots of data of type A before, during and immediately after the scheduled interventions. Our purpose within the 'Evaluating ways of capturing engagement processes' project should focus on collecting data about level B, and data about level A over a longer time period than for Juxatalearn. In the following paragraphs I describe some initial versions of questions that could be asked of people to help us understand engagement with Juxtalearn research. These are initial drafts, and will be developed further, ready for the first sessions which will occur in a week or twos time.

The Publics we will need to focus on include the  teachers and students of the classes within which the Juxtalearn tools and practices are tried. In addition, we need to consider the  representatives of schol management who will influence how the Juxtalearn interventions occur, and any future devlopments within the school resulting from the interventions.

 

Purposes: What are the aims and objectives of this engaged research? 

The purposes identified in my post entitled Collecting data about engagement - quick questionnaires, quick conversations can be rewritten to yield 3  research questions. Here are those reserach questions, along with some suggestions of questions for teachers and students.

1. How  do the publics we have identified understand  the purposes of the Juxtalearn research?
   Questions  for students & teachers
   Why do you think the researchers are running the Juxtalearn  sessions at your school?
   (Don’t be afraid to tell us you don’t know as that is a perfectly valid answer.)
   The above  question could be asked immediately after a Juxtalearn session.
2. How do members of these publics communicate with each other about Juxtalearn's research goals?
   Questions  for students
   Have you discussed the Juxtalearn sessions with your friends or class mates?
   If so, what sort of things have you discussed?Questions  for teachers
   Have you discussed the Juxtalearn sessions with your colleagues?
   If so, what sort of things have you discussed?
   The above  questions could be asked sometime after a Juxtalearn session, and more than once, e.g. a week after, then a week later.
3. How do  members of these publics communicate with the research team about Juxtalearn's research goals?
   This can be answered by the Research Team.

Processes: (1) How does the research involve relevant publics in meaningful ways?  (2) When, and (3) how often, are publics be involved? (4) Where are these interventions likely to take place, and (5) through what mechanisms?
The meaning of the Juxtalearn interventions  to the publics immediately before, during and after the Juxtalearn sessions will be assessed by the Juxtalearn project itself, as part of the project's  evaluation workpackage. A description of the processes will also be recorded by the Juxtalearn researchers, e.g. in terms of both (i) the planned Juxtalearn interventions, and (ii) a record of how the interventions playedout in practice. 
To complement this, we will prompt for reposnses over a longer time frame, e.g. at monthly  intervals after the initial Juxtalearn workshop.  The following are draaft questions intended to find out if the teachers and studnets have made use of the Juxalearn approach outside of the Juxtalearn sessions, and outside of the topic areas used to  introduced to it in the Juxtalearn sessions.

Questions  for students
Have you used video creation to help you learn since attending the Juxtalern session on insert-date-here?
If so, how, and what topics have you used it in connection with?

Questions  for teachers
Have you used video creation to help you teach since attending the Juxtalern session on insert-date-here?
If so, how, and what topics have you used it in connection with?

Participation: measures of how the publics and  researchers participated?

The Juxtalearn project will collect data related to planned Juxtalearn events. We aim to complement this by enquiring about unscripted and unplanned interactions with researchers about the research, initiated by members of the publics, and interactions about the research between members of the publics  not involving Juxtalearn researchers. Data relate to both of these the themes will be collected in reponse to questions listed  under purposes.

To conclude, the previous paragraphs outline some draft questions intended to generate  data from students, teachers and school managers about their views of the engagement with Juxtalearn research. The maechanisms that these questions will be put to these publics is  likely to be through twitter and online surveys, though which will be used iss likely to depend on the preferences of memebers of these  publics themselves.

Participate or participation or participating participants

Andrew refers us to Rick Holliman's six dimensions of engagement.  One of the six is participation, and I want to unpick that because it's an abstract noun that might be a synonym for this difficult-to-define phenomenon that is engagement.

• Handley et al differentiate between participation and engagement that involves “hearts and minds” (Handley et al., 2007 p. 181), which they see in the context of learning situations and Wenger’s communities of practice (Wenger, 1998).
• Wenger defines mutual engagement as a dimension of a community of practice, involving engaged diversity, doing things together, relationships, social complexity, community and maintenance. For Wenger, engagement is a process of community building, social energy and emergent knowledgeability (Wenger, 1998 p. 237). Knowledgeability is the ability to acquire and use knowledge and is a negotiation of meaning, a continual contextual process. This emergent knowledgeability may add value to a project. In fact, Wenger writes that engagement “can be a vehicle for sharing ownership and meaning” (Wenger, 1998 p. 203). He further suggests the value of communities of practice is because they are “organisational assets that represent investments in mutual engagement.”

Some contradictory perceptions of the goodness of participation exist. For instance, Axelrod (2001) writes that participation can increase bureaucracy when in a hierarchical top-down process, such as may exist in UK central government.  In that case, we don't see it as engagement, do we?  That sort of process, we don't want. 

Now there I'm implying that participation is a process, which makes participation the same as process, and not distinct from Holliman's other dimension of engagement: a process. Perhaps engagement requires a process, and that process includes people who participate, and the action of participating is different from abstract participation. Isn't engaging about action, so doesn't engagement require action?

---

Handley, K., Clark, T., Fincham, R. and Sturdy, A. (2007) 'Researching Situated Learning', Management Learning, Vol 38, No 2, pp. 173-191.
Wenger, E. (1998) Communities of Practice: Learning, Meaning, and Identity, Cambridge, Cambridge University Press.

Place (and time) of engagement

People interact in places, and behave and engage differently in different places.  Imagine the different engagement between a lecturer and students in a lecture theatre, and engaged behaviour in a chemistry lab.  Place influences engagement.

Consider place to include material objects and time. Nonaka and Konno described Ba (equivalent to "place" in English) as a shared space for emerging relationships (Nonaka and Konno, 1998). Environment is the place in which people engage, and it includes time and the material objects with which people interact, like science lab benches.

1. Physical proximity enhances learning (Skerlavaj 2006) and the physical structure of a room can allow easy engagement. For example, round tables are more indicative of a democratic approach to listening and decision making than an auditorium with fixed seating facing forward (Block, 2000: 277). Sturdy et al suggested that consultants and clients could cross or blur boundaries by meeting in other than at routine places and times, in liminal spaces, spaces where institutionalised or cultural rules, norms and routines are suspended (Sturdy et al., 2006: 932), like hospital waiting rooms or car parks, or the water-cooler.
2. Materials invite people to do something. Orlikowski (2006: 465) suggests that “the materiality of infrastructures, spaces and technological artefacts structure […] knowledgeability” thus extending place to include other material objects. Materiality is the context in which people interact and engage.  Material objects include shared physical tangibles and intangibles such as time. Objects that are shared and sharable across different key parties are boundary objects (Carlile, 2002, Bechky, 2003, Star and Griesemer, 1989) and can help solve problems. Skovgaard-Smith observed consultants facilitating discussion using flipchart material, which provided a tangible aspect of consultancy service to the group. Thus a place affords combination capability.  
3. Time is another aspect of environment, crucial in combination with space (Maaninen-Olsson and Müllern, 2009). Orlikowski, using a scaffolding metaphor, for knowing suggests that “Scaffolds are emergent – they are erected over time, changing in form and function, as needed to continue supporting the changing scale and scope of the element(s) being built over time” (Orlikowski, 2006: 462) Such scaffolds of knowing afford a temporary stability (Orlikowski, 2006). 

In summary, aspects of environment that are likely to be relevant to understanding engagement are shared place, time and material objects.

---

BECHKY, B. A. (2003) 'Object Lessons: Workplace Artifacts as Representations of Occupational Jurisdiction'. American Journal of Sociology, 109, 720-752.
BLOCK, P. (2000) Flawless consulting: a guide to getting your expertise used, 2 ed, Jossey-Bass/Fpeiffer.
CARLILE, P. R. (2002) 'A Pragmatic View of Knowledge and Boundaries: Boundary Objects in New Product Development'. Organization Science, 13, 442-455.
MAANINEN-OLSSON, E. & MÜLLERN, T. (2009) 'A contextual understanding of projects--The importance of space and time'. Scandinavian Journal of Management, 25, 327-339.
NONAKA, I. & KONNO, N. (1998) 'The Concept of "Ba": building a foundation for knowledge creation '. California Management Review, 40, 40-54.
ORLIKOWSKI, W. J. (2006) 'Material knowing: the scaffolding of human knowledgeability'. European Journal of Information Systems, 15, 460-466.
SKERLAVAJ, M. & DIMÖVSKI, V. (2006) 'Social Network Approach To Organizational Learning'. Journal of Applied Business Research, 22, 89-98.
SMITH, I. S. (2010) Materializing the Organization - The role of consultants in processes of objectification. Academy of Management Annual Conference Montreal, Canada.
STAR, S. L. & GRIESEMER, J. R. (1989) 'Institutional Ecology, 'Translations' and Boundary Objects: Amateurs and Professionals in Berkeley's Museum of Vertebrate Zoology, 1907-39'. Social Studies of Science (Sage), 19, 387-420.
STURDY, A., SCHWARZ, M. & SPICER, A. (2006) 'Guess who's coming to dinner? Structures and uses of liminality in strategic management consultancy'. Human Relations, 59, 929-960.

Previous literature on engagement

I review engagement literature here because Andrew comments:

"who can tell you how to engage in general, becuase as far as I know there has been little work on systematically analysing engagement either qualitatively and quantitaivley"

I'm considering literature that addresses different articulations of engagement:

• involvement,
• participation,
• commitment,
• collaboration and
• motivation.

Previous research on engagement focused on outcomes and products and seems one-sided, focusing on for example,

• employee engagement with work (Saks, 2006, Schaufeli et al., 2006),
• student engagement with learning (Handley et al., 2007, Robinson and Hullinger, 2008, Arbaugh, 2000),
• client engagement with child welfare services (Yatchmenoff, 2005)
• customer engagement with a brand (Mollen, 2010).

These approaches apparently use engagement as a one-way relationship, rather than share knowledge.

However, consultancy practitioner literature advises consultants how to engage to with clients to get and share information. Axelrod (2002) suggested four underlying principles that would produce an engaged organisation:

• widening the circle of involvement through people and ideas,
• connecting people to each other,
• creating communities for action and
• embracing democratic principles.

Block advised setting out the room in a manner than encouraged people to engage with each other.  i.e. adapt the conditions of the setting or environment and you'll get different behaviour.

Involvement
How does engagement differ from involvement?
Barki et al suggested involvement is a separate construct from participation and refers to a psychological state, although they do not elaborate on what that state might be.  For Saks, job involvement relates to self-image, and to how employees perform their jobs (Saks, 2006 p. 602).    Axelrod, (2004) states that involvement is “working with others to get things done” thus implying that involvement and engagement may be similar phenomena (Axelrod et al., 2004 p. iv). In conclusion, in the sense that involvement is working with other people, it is highly relevant to engagement.

Commitment
The term engagement has the sense of 'engagement with' someone, so implies some form of relationship that might require commitment. Commitment may affect sense making and the social structure (Weick, 1995). An organisational context with visibility (behaviour is public), volition (with an element of choice) and irrevocability (behaviour cannot be undone) “should generate stronger commitment” (Weick, 1995 p. 159). Nevertheless, commitment is “also a liability because it reduces flexibility, learning and adaption”. However McCormick, who developed a survey tool to measure the impact that participation in large-group intervention had on attitudes and beliefs of participants, found engagement increased commitment (McCormick, 1999) suggesting engagement leads to commitment. Perhaps we can collect data on commitment as a measure of engagement.

Collaboration
Collaboration may also be related to engagement. Collaboration is what organisations do together and is closely related to cooperation (Huxham, 1993).  Cooperation and collaboration both mean “something to do with working together,” (Huxham, 1993 p. 5) which is similar to Axelrod’s involvement being the art of bringing people together (Axelrod, 2001). Huxham uses the concept of “collaborative advantage” (Kanter, 1994) to solve problems together (Huxham and Vangen, 2005). Collaborative advantage, meaning being a good partner, arises from organisations pooling resources and expertise for a common aim, creating synergy. A reason to discard Huxham’s collaborative model is that it focuses on cooperative relationships that have complementary rather than shared goals.

And it depends whether we're researching engagement between individuals or between organisations.  If the stakeholders for this research project are individuals, then data from a collaborative model of organisations is not relevant.

Motivation
Marcum, comparing motivation with engagement, points out that people choose to be engaged. He reviews literature on engagement from learning theory, information management and philosophy, concluding that “An engagement mindset offers a more useful model for cultivating mutually beneficial working relationships with staff and colleagues ... engagement is based on learning and involvement” (Marcum, 1999  p. 46). Marcum’s perspective concurs with Hartwick and Barki’s findings that participation and involvement depend on whether IT system use is mandatory or voluntary (Hartwick and Barki, 1994). Previous discussion explored different constructions of engagement, but did not identify an existing conceptual framework for the phenomenon. We still do not understand the quality of the engagement, what produces it, how and whose engagement leads to effectiveness.

Participation I look at participation in a separate posting because it merits special attention.

We need research that looks at the process of engagement and how interaction between people builds commitment

---

ARBAUGH, J. B. (2000) 'How Classroom Environment and Student Engagement Affect Learning in Internet-based MBA Courses'. Business Communication Quarterly, 63, 9-26.
AXELROD, R. H. (2001) 'Terms of engagement: changing the way we change organizations'. Journal for Quality & Participation, 24, 22.
AXELROD, R. H. (2002) Terms of engagement: changing the way we change organizations, ed, San Francisco, Berrett-Koehler.
AXELROD, R. H., AXELROD, E., BEEDON, J. & JACOBS, C. D. (2004) You don't have to do it alone: how to involve others to get things done, ed, San Francisco, CA, Berrett-Koehler.
AXELROD, R. H. & MACLEOD, H. B. (2002) 'Engaging the Staff'. Health Forum Journal, 45, 22.
BARKI, H. & HARTWICK, J. (1989) 'Rethinking the Concept of User Involvement'. MIS Quarterly, 13, 53-63.
BLOCK, P. (2000) Flawless consulting: a guide to getting your expertise used, 2 ed, Jossey-Bass/Fpeiffer.
HANDLEY, K., CLARK, T., FINCHAM, R. & STURDY, A. (2007) 'Researching Situated Learning'. Management Learning, 38, 173-191.
HARTWICK, J. & BARKI, H. (1994) 'Explaining the Role of User Participation in Information System Use'. Management Science, 40, 440-465.
HUXHAM, C. (1993) 'Pursuing Collaborative Advantage'. Journal of the Operational Research Society, 44, 599-611.
HUXHAM, C. & VANGEN, S. (2005) Managing to collaborate: the theory and practice of collaborative advantage, ed, London; New York, Routledge.
KANTER, R. M. (1994) 'Collaborative Advantage: The Art of Alliances'. Harvard Business Review, 72, 96.
MARCUM, J. W. (1999) 'Out with motivation, in with engagement'. National Productivity Review, 18, 43-46.
MCCORMICK, M. T. (1999) The impact of large-scale participative interventions on participants, Philadelphia, Temple University
MOLLEN, A. (2010) 'Engagement, telepresence and interactivity in online consumer experience: Reconciling scholastic and managerial perspectives'. Journal of business research, 63, 919-925.
ROBINSON, C. C. & HULLINGER, H. (2008) 'New Benchmarks in Higher Education: Student Engagement in Online Learning'. Journal of Education for Business, 84, 101-109.
SAKS, A. M. (2006) 'Antecedents and consequences of employee engagement'. Journal of Managerial Psychology, 21, 600-619.
SAKS, A. M. (2008) 'The Meaning and Bleeding of Employee Engagement: How Muddy Is the Water?'. Industrial & Organizational Psychology: Perspectives on Science and Practice, 1, 40-43.
SCHAUFELI, W. B., BAKKER, A. B. & SALANOVA, M. (2006) 'The Measurement of Work Engagement With a Short Questionnaire: A Cross-National Study'. Educational & Psychological Measurement, 66, 701-716.
WEICK, K. E. (1995) Sensemaking in organizations, ed, Thousand Oaks, Sage Publications.
YATCHMENOFF, D. K. (2005) 'Measuring Client Engagement From the Client’s Perspective in Nonvoluntary Child Protective Services'. Research on Social Work Practice, 15, 84-96.

Collecting data about engagement - quick questionnaires, quick conversations

We are aiming to develop and evaluate a variety of engagement data collection tools for their richness in informing us about engagement changes and effects for teachers, students and researchers. In this post I analyse a definition and some dimensions of engagement with the aim of characterising the kind of  data we should collect so as to give us the rich picture we are aiming for.

Engaged? definition and data

I need to know what engagement is, and as Richard Holliman points out in the Engaging Research blog "Public engagement with research and school-university partnership work means different things to different people". But if I'm working on engagement with the JuxtaLearn project and evaluating engagement, then I need to know what it is, what creates it, what its results are, and I need to know what data to collect in order to evaluate engagement. So I have to know what engagement is before I can evaluate it.
The OU 'Public Engagement with Research Catalyst Team defines engaged research thus:

Excellent public engagement with research is reflected in the different ways that researchers meaningfully connect and share research with various stakeholders, user communities and members of the public. Done well, public engagement with research will generate benefits, changes and effects for all participants as they share knowledge, expertise and skills. Excellence will be demonstrated partly through recognition of the contributions that all participants make to the shaping of research agendas, the processes of conducting research, and in the products of that research.[http://www.open.ac.uk/blogs/per/?page_id=1621]

I read two actions here, where engagement

1.  is reflected
2. generates

"Excellent public engagement with research is reflected..." suggests that we'll recognise engagement when we see it, in the way that I recognise myself when I see myself in the mirror. But the reflection is not me but an image of me. Magritte painted "the Treachery of Images" as a pipe under which he wrote, "Ceci n'est pas une pipe", French for "This is not a pipe", and indeed the painting is merely an image of a pipe, not a pipe. So we do not in the first sentence of the above definition of engagement have a definition of engagement, merely a statement that it will be (passive voice) reflected. I want to know actions: how to engage, what I have to do to engage, and I need to know beforehand what those actions are so I can plan my engagement and others' engagement, not have to recognise it after the event, when it's too late.
 "Done well, public engagement with research will generate benefits,..” This part of the definition tells me that engagement generates, I understand an action that "generates benefits, changes and effects" is an engaged action. Therefore to engage, I must plan to generate benefits, changes and effects. And from the first part of the definition, I will recognise engagement through the (I assume) consequent reflections that results from the action of generating.
What might an engaged relationship be? It might require commitment. a context with visibility (behaviour is public), volition (with an element of choice) and irrevocability (behaviour cannot be undone) “should generate stronger commitment” (Weick, 1995: p159). Benefits and changes must be the consequent "shared knowledge, expertise and skills". Would such context and behaviour generate benefits, changes and effects?
The given definition provides a means to recognise data to collect, but we must accept the data may be wrong data if the definition is flawed. What we may have however, is an opportunity to identify engagement and create a better definition.

Theatre studies workshop

Theatre studies - now would you say that was a subject full of threshold concepts?  Apparently, it is, the theatre studies teacher at an MK school assures us, and he gave us and his sixth formers an example: the difference between naturalism and realism.

We went into a school on Monday, four researchers, nay, five of us, following nearly a dozen kids as they struggled to understand these 'isms'.  Eleven students, either in year 12 or about to start year 12, came along to the workshop day, most of them having studied their teacher, Gary's, pencasts.  He'd put the pencasts together a day or two earlier, and emailed them to read them.  One student was in the classroom reading a pencast off her mobile device.  Another complained that she couldn't watch it and listen on a mobile device, so I guess they could get only the pdf form.  But they did like these pencasts, and saw them as something to watch, read and study independently, without having to be in the classroom, but also that they liked the familiarity of their teachers' voice, a personality that they knew.

Eleven students made up three groups of two fours and a three.  One group of four was all boys, the group of three was all girls, and the last group was mixed.  The adults included:

• the teacher, 
• our lead researcher, Anne,
• the OU research associate, Gill who'd put together the tricky topic tool, about which more anon,
• Liz who once worked for Catcher Media,
• Canan who felt rotten ill and took notes and films all day,
• Aba-Sah from Birmingham who wanted to provide table top software of the storyboard.

The tricky topic tool is a means for the teacher to analyse the stumbling blocks within a threshold concept (Meyer and Land's idea).  It requires the teacher to open a web page with options to pick and choice topics of interest, and then set up quizzes to test the students' understanding.  I've tried it and it takes ages to do because of the thinking, experience and analysis it requires.  The idea is to test students before and after they've worked on their video idea.  So the first activity of the day was for students to take their starting quiz.  No one worried what the mark gained was, just that we had a base line to compare with at the end of the day. 

Then we gave the run down on what the project was about - that the EU funded it and so on, during which they looked blank, bored or fazed.  To enthuse them, we showed a five minute video on juxtaposing, but it confused not enthused, and needs remaking. A new draft is available already at https://vimeo.com/99420903 with the password 'Juxtaposing', and a further version at https://vimeo.com/101430106.  Students were raring to go, ready to write, and though they didn't need the video but were polite about it, and it did give some of them some ideas. Perhaps theatre studies students see themselves as creative already. 

They sat around in their groups on the floor of what I assume is the drama studio, a dark room with black out blinds on the window - a shame because it was a beautiful day, and perhaps we should have opened the blinds in the morning, though later people were using the room to put their performances together. They sat and talked.  Within half an hour, ideas were flowing, multi-coloured penned words represented abstract ideas on the all-girls paper, biro crossings-out and stick men on other papers.  Then things slowed down, with little more action, apart from the odd late comer, student or researcher arriving. Gary told them they had half an hour more, no - only thirty minutes - and they had to be on to pre-production by then, allocating roles and sourcing assets.  They all groaned, but funnily enough within ten minutes were up and off, each student group with an OU borrowed flip video camera, and some with a tripod.  I suspect they were stuck creatively, and what they need is to do something active, try it out, find it doesn't work and return to the storyboard, which to some extent is what they did, except that they didn't return to the storyboard,  One group rather than spoil the storyboard, wrote on the back of the paper (the prompts were originally on the back - why weren't they?  Wasn't the paper printed double sided?) and kept changes in their heads!

The mixed group of four students (L, J, H and M) went down to the hall to use the stage.  Where Aba-Sah and I sat to observe and record (Aba-Sah's recordings are all up in a dropbox), we couldn't hear the students, not even when they were performing, let alone when they discussed.  I went up on the stage to eavesdrop, setting up my flip video camera.  But actually, a lot of the time they were quiet, thinking, not talking.  J tapped her finger nails.  They were pretty nails as were L's, and tapping drew attention to how pretty they were. Perhaps the tapping represented frustration, a not-knowing what to do.
However, eventually, Gary appeared, talked through with them again on what naturalism and realism was, and suggested they found somewhere quieter to record.  That hour was the students' learning hour, I reckon.  That was the hour when they stumbled and fumbled, realising that they didn't realise. 

Perhaps M didn't realise.  He'd taken charge of the tripod and the camera, and standing off stage wasn't in some of the conversations that the other three had.  Watching the recording of a major discussion, I observe he didn't contribute. Didn't he have a lower score on the second quiz?  That would imply that you need to join in the conversation,  at least be present, if you are going to learn.  M was apart; he wasn't present.

We had a hasty lunch on the floor of the drama studio, observed some more video production, then retired to a room full of OU laptops for the students to edit before doing the second quiz.  Gary promised them that those that scored highly would go on a trip, so there was some tension and angst.  Somewhat to their surprise, students who had struggled most with the concepts, students who felt they didn't have the perfect product, a viewable video, were the ones who'd made the biggest progress.   But that outcome pleased Gary and proves to the researchers that the process improves the learning.