Link Search Menu Expand Document
Evaluation in VR

VR object selection and manipulation study checklist

Contribute via Google Docs

Download pdf-version

Version 1.0.0

Table of contents

  1. Method
  2. Participants
  3. Design
  4. Task
  5. Procedure
  6. Targets
  7. Materials
  8. Results and Analysis

Method

  1. Provide an explicit statement of the goal of the study.
  2. Provide supplementary information (e.g., a link or an appendix) to preregistration, if applicable.

Participants

  1. Report the number of participants and if possible, how this sample size was decided.
  2. Describe the aims of recruitment together with demographics (e.g., level of expertise with virtual reality and in the domain, if applicable).

Design

  1. State all independent variables or factors and their levels in an explicit sentence. When there are more than one independent variables or factors, report them and their levels in M×N×P format. Be sure to include all factors.
  2. Report counterbalancing or randomisation of the order of conditions and the method for those. Specify this for each of the independent variables or factors. If none, justify why.
  3. Report the number of repetitions if any, and the number of trials in total for the main independent variable (e.g., if M is a technique, report the number of trials with each technique in N×P×repetitions format).
  4. Define the dependent variables in an explicit sentence (e.g., selection or task completion time, errors, accuracy, and user experience or workload, if applicable). Explain how the values of dependent variables are determined or calculated.

Task

  1. Report the task(s) and possible sub-tasks in detail. Classify the tasks as selection or manipulation, and classify manipulation as translation, rotation, or scaling.
  2. State if the task is based on previous work, and if so, which parameters were modified, if any.

Procedure

  1. Express whether the task is discrete task or serial. If the task is serial, explain the trials through the sequences(e.g., 10 target selections in a 11-target circle after the initial starting selection).
  2. Describe how the participants were trained for the task (if any). Specify the type of the training (e.g., in the task itself, in getting used to a technique in a setting different from the task). Specify also the duration of the training (e.g., a time or number of trials).

Targets

  1. Report the total number of distinct targets.
  2. Report whether distractors were used, and whether they were external to the task. Report the number of external distractors in relation to the targets.
  3. Report the method of arranging targets. This can be reporting a shape that is followed with an even distribution (e.g., on a matrix), or randomisation. Specify the constraints used in randomisation, such as the dimensions of the area in which the targets are laid out (e.g., a virtual cube, a matrix, or a sphere), and the boundary conditions for randomisation (e.g., minimum distance between targets, the number of targets, or removal of occluded targets). Ideally provide a visual depiction in addition to a text-based description.
  4. Report the actual target locations (e.g., in mm and degrees). Do this even when the targets are distributed randomly: log their arrangement and report at least a summary of their locations.
  5. Describe the distance to each target as euclidian distance (e.g., meters) and as angles if applicable (e.g., from the participant’s point-of-view). Explain also how this distance is defined both from the starting point (e.g., in a discrete task it can be from the reset/starting point) and from the end point (e.g., to the nearest point on the target sphere, or to the center of the target sphere).
  6. Report target locations relative to the movement direction (e.g., there were six targets on three axes: frontal, lateral, and vertical, with all together six movement directions: reach and withdraw, left and right, and up and down). Do this numerically.
  7. Report the target shape and size. Specify how the size is defined (e.g., the diameter of the target sphere, or the side×side×side of a cuboid). Ideally provide a visual depiction in addition to a text-based description.
  8. Report the cursor type and size (e.g., a single-pixel ray, or a sphere with a diameter of d), or the size and shape of the object that is manipulated.
  9. Connect target distances with size if applicable (e.g., list the target IDs together with sizes and distances).
  10. Specify how the cursor or manipulated object and the target object sizes are used to define errors, if any are measured (e.g., the size difference if the cursor needs to fit inside the target for successful performance, or the acceptable threshold of correct positioning). Otherwise, state that accuracy is used as a measure and define how it is calculated.

Materials

  1. Describe the physical setting with respect to the participants and their movement (e.g., what posture were they instructed to maintain, were there any controlled or otherwise limited physical motions such as a use of a chinrest, an instruction to stand on the same spot, or resetting the physical posture in a starting position between trials). Ideally provide a visual depiction in addition to a text-based description.
  2. Specify the physical settings in spatial units, such as distance in meters (e.g., where does the participant sit in relation to a table) or angles in degrees (e.g., the range the participant can move their hand on a haptic device).
  3. Describe the virtual setting. Specify at a minimum which default virtual scene was used if that is the case, and if not, with specify and justify the design of the virtual scene (e.g., adding walls to make a room) and the light sources in the space. Describe these with spatial measures (e.g., dimensions of a room with meters or a light source position with degrees). Ideally provide a visual depiction in addition to a text-based description.
  4. Specify user representations in the virtual setting. Specify at least where in the virtual setting the user’s view-point is located (also in relation to the target space), what kind of representations are presented of users or input devices (e.g., as a full-body avatar, a virtual representation of the controller), and how feedback of the user’s actions are given.
  5. Describe the devices used in the study. Detail from virtual reality devices their FoV, refresh rate, and resolution in addition to their brand, type, and version. From the motion tracking or other input devices, specify their spatial and temporal accuracy.
  6. Describe the interaction technique(s) used in the study. The typical details include how the user’s movements are mapped into the virtual setting (e.g., motion gains for cursors and methods for casting rays or pointers) and the mechanism for triggering selection (and if applicable, task completion).

Results and Analysis

  1. Report the effect size of your statistics.
  2. Provide supplementary information (e.g., a link) for accessing data, if open.